We Misunderstood LLMs And It's Costing Us

Microsoft and GitHub recently introduced new usage limits and charges for premium AI models in Copilot. Under the new structure - standard subscriptions now have a strict monthly quota for queries using top-tier models like GPT-4o or Claude 3.5 Sonnet. Once you hit the limit, you either pay extra or fall back to standard base models.

They were completely honest about the reason: the massive operational cost of running advanced frontier models at scale.

Why did GitHub have to do this? Because when given a choice - users automatically select the “smartest”, biggest model available for every single task. We use the most expensive engines to perform basic functions.

The entire world treats AI this way. OpenAI CEO Sam Altman admitted that users simply saying “please” and “thank you” to ChatGPT costs the company tens of millions of dollars in electricity and computing power.

This behavior highlights a clear, ongoing trend. It proves that users fundamentally do not understand what LLMs are.

The Chatbot Illusion #

During the initial AI boom, the entire focus was on the novelty of conversation. We were fascinated by a bot that seemed smart and could answer complex questions. People quickly started using it as a primary source of world knowledge. We began to talk to it like a colleague.

But the chat interface is just a very convincing side effect.

The true goal of this technology is Natural Language Processing (NLP). LLMs are engines designed to understand, transform, and generate text. They are advanced processors - not digital encyclopedias. When we treat them as chatbots, we force them to use “world knowledge” that is merely a statistical byproduct of their training.

What Real AI Work Looks Like #

When you stop treating an LLM as a chat companion, you realize it is an incredibly valuable tool for actual work.

Look at real business use cases:

• Summarization - turning an hour-long meeting transcript into a concise summary with assigned action points.
• Data Extraction - pulling structured, repeatable data points from thousands of unformatted historical documents stored in a data lake.
• Classification - automatically sorting incoming support tickets.
• AI Agents - running complex, multi-step background tasks based on text triggers.

To perform these tasks perfectly - a model does not need to know the entire history of the Roman Empire. It only needs to understand the structure of language.

The Edge Processing Shift #

We are already seeing major tech companies realize this. Look at the recent controversy surrounding Google Chrome. Google was heavily criticized for silently downloading a 4GB Gemini Nano AI model onto users’ devices without explicit consent.

Forcing a massive download in the background is a terrible PR move. But if we look past the execution - we should pay attention to what Google was actually trying to achieve: moving the processing of sensitive data locally to the edge.

They are trying to shift text processing directly onto your local machine. Contrast this with Microsoft Edge, which frequently faces severe backlash for sending users’ active browser data and screen content back to the cloud for processing. Google’s approach aims to process the text on your laptop, ensuring your data never leaves your device.

The Small Model Advantage #

To make “edge processing” a reality - we rely on specialized, small models.

It is true that these local models do not possess the vast “world knowledge” of massive cloud models. But counterintuitively - this is exactly their greatest strength. Giant models often suffer from knowledge bias. They have a strong tendency to hallucinate or make things up because they struggle to distinguish between their internal training data and the specific context provided by the user. If they think they “know” the answer - they will guess rather than process the text.

Small models lack this vast internal knowledge. Because they don’t have a massive database of trivia to fall back on - they rely almost entirely on the data sources provided by the user. If you give them a document to summarize or extract from - they focus strictly on that document. They process rather than guess.

Beyond accuracy, the biggest driver for adopting these local models is Privacy and Compliance. As we saw with the browser data concerns - sending sensitive information to a third-party cloud API is often a massive legal and security risk. When you handle client data, internal financial reports, or healthcare records, data sovereignty is no longer optional.

With strict regulations like the GDPR and the new EU AI Act, you need guarantees. Small models allow you to run powerful LLMs completely isolated from the internet. Your data never leaves your device.

This privacy-driven shift to the edge is becoming too large to ignore. Even Western tech giants - whose primary business model relies on keeping users dependent on expensive cloud APIs - are being forced to adapt. Google maintains its presence with the open-weights Gemma family, and OpenAI recently released GPT-OSS, their first truly local model in years. They are entering this space simply because they cannot afford to let Chinese tech companies completely dominate the rapidly growing local segment.

How This Looks in Practice #

When I design and build AI-driven solutions for clients, ensuring strict data privacy, regulatory compliance, and cost efficiency are always my top priorities.

I practice what I preach in my own daily workflow. I use local models to transcribe hour-long meetings and automatically generate meeting minutes with action points assigned to specific people. All of this happens directly on my machine - ensuring sensitive company discussions are never uploaded to the internet.

I used this exact approach to build an AI agent for the recruitment department at my previous company. The local agent operated in real-time during interviews, tracking the conversation against required criteria. It reminded recruiters if they missed a critical question and handled the administrative paperwork automatically. This allowed the HR team to focus entirely on the candidate. Most importantly - it processed everything locally, ensuring that candidates’ private voice data and sensitive information never touched a random cloud server.

I practiced this exact strategy recently with a client struggling with their massive data lake. Their initial idea was to build a “smart chatbot” on top of their historical data using standard RAG (Retrieval-Augmented Generation). It was expensive and inaccurate. I helped them shift their entire approach. Instead of forcing a giant model to guess answers from messy files on the fly - we deployed cost-effective local models to systematically process the entire data lake beforehand. The small models extracted structured metadata from years of unformatted historical documents.

The result? The data lake became instantly searchable using standard database queries. And when we did connect a chatbot - its performance skyrocketed because it was retrieving highly accurate, structured metadata instead of relying on fuzzy vector search across unstructured text.

Building reliable AI is not about jumping between the latest marketing buzzwords or buying the biggest model on the market. It is a conscious choice. It is about understanding the technology, securing your data, and picking the exact right tool for the job.

If you are tired of the AI hype cycle and want to discuss how to build secure, cost-effective solutions tailored to your actual business needs - let’s connect. I would be happy to talk about how we can make this technology work for you.