Most people today use ChatGPT, Claude, Gemini, Cursor or AI coding assistants every day. They write prompts, get answers and assume that’s enough. But in real products this approach quickly starts to break down: models behave unpredictably, ignore instructions, “forget” context, burn through the API budget and give inconsistent results.
The problem isn’t the quality of the models. The problem is that people use them blind, without understanding how they work under the bonnet.
This article walks through the fundamental principles of how LLMs work: from tokens and attention to context windows, the cost of generation, agents and context engineering. It isn’t an ML course and it isn’t maths. It’s an engineering understanding that directly affects the stability, speed and cost of production systems.
Prompts are not architecture
A common misconception: learn to write good prompts and the model will “just work”. In practice a prompt is only the top layer of control. It’s like the steering wheel in a car: an important part, but without understanding the engine, the brakes and the gearbox you won’t get far.
In production, mistakes are expensive. You can spend weeks building a system that sometimes works and sometimes doesn’t. You can quietly burn hundreds of dollars a day on the API simply because you don’t understand where you need a cache, where a local model, and where a different approach to the architecture.
The difference between “sometimes works” and “works reliably” is an understanding of the fundamental concepts of LLMs.
Tokens: why words are a poor measure
One of the first mistakes is assuming the model works with words. It doesn’t.
A token is the basic unit of text for a language model. A token can be:
- a whole word,
- part of a word,
- a single character,
- or a combination of characters.
The same word can be split into a different number of tokens in different models. For example, a long Russian word might become 3 tokens in one tokeniser, 6 in another and 15–17 in a third.
This matters for three reasons:
- Cost
- Every API charges by the token. The more tokens, the more expensive the request.
- Context window
- The model doesn’t see “text”, it sees a limited number of tokens. In popular models Russian usually needs 1.5–2 times as many tokens as English, because most models were trained predominantly on English-language data.
- Speed
- The more tokens, the slower the processing.
For example, a context window of 128,000 tokens is roughly:
- 250–300 pages of English text,
- or only 100–150 pages of equivalent volume in Russian.
Attention: how the model grasps meaning
Take a simple sentence:
“The senior developer looked at the code, and it broke.”
A person automatically understands that “it” refers to the code, not the developer. The model does the same, through self-attention.
The attention mechanism works like this:
- each token looks at all the preceding tokens,
- calculates the weights of the connections,
- works out which of them matter most for the current step.
But this doesn’t happen just once.
Multi-head attention
Modern models use dozens of attention heads in each of dozens of layers. Each “head” looks at the text from its own angle.
An analogy with code review:
- one “head” looks for syntactic links,
- another tracks data types,
- a third analyses the business logic.
All the heads work in parallel, and their results are combined into a single representation.
Important: the specialisation of attention heads is not set in advance. It forms during training.
Why transformers won
Before transformers, recurrent neural networks were used. They read text sequentially, word by word. That was slow and scaled badly.
Transformers process all tokens at once thanks to the attention mechanism. This:
- dramatically sped up training,
- made it possible to scale models to hundreds of billions of parameters,
- became the basis of all modern LLMs: GPT, Claude, LLaMA, Gemini.
But transformers have a fundamental problem.
Quadratic complexity and the price of long context
Attention connects every token with every other token.
- 2,000 tokens: ~4 million operations
- 4,000 tokens: ~16 million operations
Doubling the context quadruples the computation.
Hence:
- long context = slow,
- long context = expensive,
- contexts of hundreds of thousands or millions of tokens are still an engineering challenge.
That’s why hybrid architectures and new approaches are appearing, but for now transformers remain the de facto standard. And if you work with AI tools today, there’s a transformer under the bonnet.
The context window is the model’s working memory
The context window is what the model sees at once. It helps to think of it as a desk.
- A small desk (4,000 tokens): a laptop and a mug of coffee.
- A big desk (200,000 tokens): monitors, books, notes, headphones.
But the desk gets cluttered fast.
The context holds:
- the system prompt,
- the conversation history,
- the current request,
- the model’s reply (which then also becomes part of the history).
AI assistants add even more:
- tool descriptions,
- open files,
- command output,
- search results,
- the history of actions.
A real example:
five files of 200 lines each + a few terminal commands = 15,000 tokens in a single agent step.
How to tell the context is overflowing
Typical symptoms:
- the reply cuts off mid-sentence,
- the model forgets the initial instructions,
- repeats steps it has already done,
- asks again for files it has already opened.
Most assistants automatically summarise the older part of the conversation. It’s convenient, but dangerous: summarisation loses architectural decisions, the reasons behind changes and the context of actions.
Auto-compaction is a crutch, not a solution.
Prefill and decode: why the answer is typed out word by word
Generating an answer consists of two fundamentally different stages.
1. Prefill
The model processes the entire input prompt in parallel.
500 tokens in a fraction of a second.
2. Decode (generation)
The answer is generated sequentially, token by token.
200 tokens take a few seconds.
Each next token depends on the previous ones. This can’t be parallelised.
KV cache: why the first token is the slowest
Without optimisations, the model would have to recalculate attention for the whole context every time it generates a token.
The solution is the key-value cache:
- at the prefill stage the model saves the attention results,
- at the decode stage it uses the cache,
- only the new token is recalculated.
Hence:
- the first token appears more slowly,
- the rest come faster.
Why output costs more than input
Input is processed in parallel.
Output is generated sequentially and needs a growing cache.
That’s why with most providers:
- output costs 3–5 times more than input.
This isn’t marketing. It follows from the architecture.
The real economics of requests
Take an example:
- system prompt + documentation: 10,000 tokens,
- the user’s question: 100 tokens,
- the model’s answer: 500 tokens.
At typical rates:
- one request: ~a few cents,
- 1,000 requests a day: tens of dollars,
- per month: thousands.
Without optimisation, budgets drain away before you notice.
Caching as a way to save money
A cache lets you:
- write the context once, at a high price,
- read it many times, cheaply.
The saving is up to 70–90% on repeated requests.
This isn’t an optimisation “for later”, it’s a basic architectural necessity.
The model doesn’t learn from your conversations
It’s important to keep these two apart:
- Training: teaching the model (months, GPU clusters, millions of dollars),
- Inference: using the model (seconds, cents).
During inference the model’s weights do not change.
What looks like “learning” is work with context and memory:
- chat history,
- saved facts,
- retrieval of relevant information.
That isn’t learning, it’s sound context engineering.
Temperature, top-p and top-k
A model is a next-token predictor.
Temperature controls the extent to which the model:
- sticks to the most likely options,
- or allows less likely ones.
- T near 0: the most likely token is almost always chosen.
- T towards ∞: chaos.
The practically useful range is 0–1.2.
Top-p and top-k are additional probability filters:
- top-p: a dynamic set of tokens by cumulative probability,
- top-k: a fixed number of options.
These are tools for controlling behaviour, not magic knobs.
LM, reasoning model and agent are different levels
LLM
Stateless. One request, one answer.
Suitable for:
- Q&A,
- analysing a single document,
- generating text.
Reasoning model
Thinks step by step.
More expensive and slower, but:
- fewer mistakes,
- better for complex logic.
Agent
An autonomous system that:
- observes,
- reasons,
- acts,
- keeps state.
It can:
- read and change files,
- run tests,
- deploy code,
- check the result.
The LLM is a consultant.
The reasoning model is an analyst.
The agent is an engineer.
Context engineering matters more than prompts
The prompt is the last line.
The context is everything else.
The hotel booking example shows this clearly:
- a perfect prompt without context gives a poor result,
- the same prompt with the right context gives a precise solution.
Context engineering means:
- memory management,
- retrieval (RAG),
- state,
- tools,
- assembling the request dynamically.
In-context, RAG and fine-tuning: when to use which
In-context learning
- fast,
- no training,
- expensive at scale,
- limited by the size of the context.
RAG
- scalable,
- up-to-date data,
- more complex infrastructure,
- depends on the quality of retrieval.
Fine-tuning
- knowledge is “baked” into the model,
- faster inference,
- expensive and slow,
- suits stable domains.
The right path:
- in-context,
- then RAG,
- and only then fine-tuning.
Why foundation models changed everything
Training a model from scratch costs hundreds of millions of dollars.
Foundation models let you:
- take a ready-made base,
- adapt it to the task.
There are:
- closed models (quality, but dependence),
- open-source models (control, but lower quality).
The key idea:
you’re not teaching the model to understand the world, you’re teaching it to understand your task.
The future: MCP, MoE and security
- MCP standardises how agents connect to systems.
- Mixture of Experts activates only the parts of the model that are needed.
- AI security is becoming critical: prompt injection, shadow AI, supply chain attacks.
Most AI projects fail not because of model quality, but because they lack architecture and security.
The bottom line
LLMs aren’t magic.
They are a complex but understandable engineering system.
Tokens, attention, context, cache, agents, RAG: this isn’t theory.
These are tools that:
- cut costs,
- improve stability,
- make systems predictable.
Once you understand how it all works under the bonnet, you stop being a user.
You become an engineer who builds AI systems deliberately.
Useful articles and discussion here (chat in Russian)
Where the model gets it wrong
An LLM chooses a likely continuation of the text; it doesn’t automatically check the truth. So facts, calculations, code and links need to be checked with separate tools or against the primary source, especially when a mistake affects money, safety or someone’s rights.
