One Model, Many Conversations: LLM Serving, Memory, and Reasoning

Part 3 of How LLMs Actually Run: shared weights, per-sequence state, continuous batching, persistent memory, and inference-time reasoning.

Part 1 separated the model data from the inference runtime. Part 2 followed llama_decode() through a GGML computation graph, CUDA kernels, and the GPU memory hierarchy.

One question remained.

If a model occupies gigabytes of VRAM, does every active user need another copy? If not, where does each conversation live? And if the weights are frozen, how can a reasoning model spend more effort on one problem than another?

The answer is another separation of concerns:

model weights = shared learned behavior
sequence state = one request's working context
serving engine = scheduling, isolation, and resource management

The transformer does not know which human owns a token sequence. The runtime does.

Multiple users do not require multiple models

A serving engine normally loads one copy of the model weights per relevant device placement, not one copy per user.

The weights are read-only during inference and can be shared across requests. Each active sequence needs its own logical state, especially the attention keys and values accumulated for its tokens.

Shared model weights
────────────────────────────────

User A sequence → User A KV cache
User B sequence → User B KV cache
User C sequence → User C KV cache

This resembles many familiar server architectures. A Node.js process does not load another copy of the application executable for every HTTP request. Requests share code while retaining separate stacks, request objects, database transactions, and response state.

The analogy is not exact—the GPU execution and memory-management model are very different—but the ownership boundary is useful. Model weights are shared implementation. Sequence state belongs to a request.

What the KV cache saves

Inside each attention layer, tokens produce key and value vectors. When generating the next token, the new query needs to compare itself with keys from earlier positions and combine the corresponding values.

Without a cache, the runtime would repeatedly recompute those keys and values for the entire prefix:

token 1
tokens 1..2
tokens 1..3
tokens 1..4
...

The KV cache preserves them instead.

earlier token representations
        ↓ computed once
keys and values for every layer
        ↓ retained
next token attends to cached history

This exchanges memory for compute. Long contexts and many concurrent sequences can consume a large amount of KV-cache memory, even though the model weights themselves remain shared.

The cache is also why conversation state is more concrete than “the model remembers.” The runtime retains numerical attention state derived from the tokens in the current context. Remove that context and cache, and the base model has not permanently learned anything from the conversation.

Continuous batching keeps the GPU useful

Autoregressive requests do not stay synchronized.

User A may be generating a long answer. User B may have just submitted a large prompt. User C may finish after the next token. If a server created a fixed batch and waited for every sequence to finish before admitting more work, GPU capacity would be wasted as requests completed at different times.

Continuous batching allows the serving runtime to change the active batch between iterations.

A needs a next token
B needs a next token
C needs a next token
        ↓
dynamic batch
        ↓
one GPU model execution
        ↓
A token
B token
C token

On the next iteration, C may be finished and D may join:

[A, B, C] → [A, B, D]

The runtime maps each input position to the correct logical sequence, gives attention access only to the appropriate context, updates the corresponding KV cache, and routes each sampled token back to the right network response.

This is serving-engine work, not learned model behavior.

The model sees tensor batches and attention boundaries. It does not know that token positions belong to three browsers, two API keys, or one customer running parallel jobs.

Isolation is more than separate arrays

A correct serving layer must preserve several boundaries:

• Token positions must map to the right sequence.
• Attention must not cross from one user’s context into another’s.
• Each sequence must use the correct sampling configuration.
• Stop conditions and cancellation must apply to the correct request.
• Output tokens must be routed to the correct stream.
• KV-cache pages must not expose stale data across tenants.
• Resource limits must prevent one request from exhausting the server.

The model architecture does not provide tenant isolation. The serving system provides it through memory management, scheduling, metadata, and carefully constructed attention inputs.

This is an important security correction. “The model is stateless” does not mean the system serving it has no sensitive state. Prompts, KV caches, logs, traces, request queues, and stored conversation history can all contain data requiring normal isolation and retention controls.

KV-cache management looks surprisingly familiar

Naively allocating one large contiguous KV-cache buffer for every request wastes memory. The server does not know in advance whether a sequence will generate ten tokens or ten thousand, and fragmented free space is difficult to reuse efficiently.

Serving engines can divide KV-cache storage into blocks and map logical sequence positions onto physical blocks. That starts to resemble virtual memory: logical continuity without requiring one physically contiguous allocation.

The PagedAttention paper describes this approach in vLLM. Its importance is broader than one implementation. Better memory utilization allows more active sequences to fit, which allows larger batches, which creates more useful work per model-weight read.

That connection was easy for me to miss:

better KV-cache utilization
        ↓
more concurrent sequences fit in VRAM
        ↓
larger effective batches
        ↓
better reuse of shared model weights
        ↓
higher serving throughput

The serving problem is not only “make matrix multiplication fast.” It is also admission control, memory allocation, scheduling, preemption, cancellation, and fairness around an expensive shared runtime.

Throughput and latency are different objectives

Batching more requests usually improves total tokens per second because a weight fetched from memory can contribute to more sequences. That does not mean every user receives the first token sooner.

A production serving engine balances competing goals:

• Time to first token: how long a new request waits before output begins
• Inter-token latency: how quickly later tokens arrive
• Throughput: total tokens produced across all active requests
• Fairness: whether large prompts or long outputs starve smaller requests
• Memory pressure: whether another sequence can safely be admitted

The best policy depends on the product. An interactive chat values latency differently from an offline summarization queue. A coding assistant streaming one response differs from a batch system processing thousands of documents overnight.

There is no universally optimal batch size. There is an operating point chosen around workload, hardware, and service-level objectives.

Running locally does not train the model

Another misconception appears when a local model remembers earlier messages in a conversation.

It is tempting to say the model learned from the interaction. Normally, it did not.

During inference:

weights stay frozen
context changes
KV cache changes
output tokens accumulate

The prompt, system instructions, retrieved documents, and earlier conversation turns become tokens in the model’s current context. They influence the next output because attention can use them. When that context is removed, the information is gone from the inference state.

Persistent application memory usually comes from ordinary software around the model:

• Store conversation history in a database.
• Retrieve relevant records for a later prompt.
• Use RAG to inject documents into the context.
• Maintain a user profile or task state outside the model.
• Summarize older context and persist the summary.

This separation matters architecturally. Context is a bounded working set, not a durable source of truth. A model’s context window should not become the application database any more than a process heap should become the accounting ledger.

Changing weights requires training

Actual learning changes parameters through optimization.

Training runs examples through the model, measures error against an objective, computes gradients, and updates weights. It requires substantially different memory and computation from inference because intermediate values and optimizer state may need to be retained for backpropagation.

Fine-tuning changes some or all of a pretrained model for a narrower objective. LoRA avoids updating every original weight by learning smaller low-rank adapter matrices. QLoRA combines that approach with a quantized frozen base model; the QLoRA paper describes backpropagating through the frozen quantized model into the adapters.

The operational distinction is:

RAG / stored memory → change what the model can see now
fine-tuning         → change learned behavior in model parameters

They solve different problems. If the requirement is “answer from today’s inventory and policy documents,” external retrieval is usually the correct mechanism. If the requirement is “consistently follow a specialized response behavior,” fine-tuning may be relevant after prompting and evaluation have shown a stable target.

The feed-forward transformer sits inside a loop

The concept that changed my mental model most was reasoning.

I initially imagined inference-time reasoning as running the model several times:

runModel(question)
runModel(question)
runModel(question)

Independent reruns can produce different samples, but they do not automatically build on one another. Each invocation sees the same question and none of the intermediate work.

A useful conceptual model is closer to:

let state = question

while (!done) {
  const next = model(state)
  state += next
}

At the level of one forward pass, a decoder transformer is feed-forward. Inputs move through fixed layers and produce logits. There is no while loop hidden inside a transformer layer.

Autoregressive generation places a loop around that fixed network. Each selected token becomes additional context for the next pass. The system can externalize an intermediate result into the token sequence, attend to it on the next step, extend it, revise direction, or use it to produce a later conclusion.

fixed transformer
      ↑       ↓
growing token context

This does not mean the model reasons like a human. It means sequential token generation can turn a fixed learned function into an iterative computation over an expanding state.

Reasoning tokens are working state

The clearest conceptual mapping I found is:

fixed weights               = learned long-term behavior
context                     = working memory
generated reasoning tokens  = iterative computation

Suppose a problem requires combining several constraints. A direct answer asks the model to map the question to a conclusion in relatively few generation steps. A longer trajectory allows it to emit intermediate representations that later tokens can use.

More steps do not guarantee correctness. The model can amplify a bad premise, wander, or spend tokens restating the problem. The inference strategy, training, verifier, search procedure, tool use, and stopping policy all matter.

The important correction is that useful additional computation must usually share state somehow. That state may be a generated token trajectory, a hidden scratchpad, a search tree, tool results, multiple candidates evaluated by another process, or another architecture-specific mechanism. Simply calling the same stateless endpoint twice is not itself cumulative reasoning.

Inference-time compute is another scaling axis

Traditional model scaling emphasized:

more parameters
more training data
more training compute

Inference-time reasoning adds:

more compute on this individual problem

A serving system can spend additional tokens, generate multiple candidate trajectories, use search, apply verifiers, call tools, or combine these techniques. Recent work describes several distinct test-time scaling regimes rather than one universal “reasoning loop.”

That caveat matters. Proprietary reasoning systems do not necessarily expose their internal reasoning tokens, and visible explanations are not proof of the exact internal computation that produced an answer. Some systems keep intermediate state hidden. Others use external tools, parallel candidates, learned critics, or model-specific mechanisms.

Architecturally, the durable point is narrower: once inference is allowed to spend variable compute while carrying forward useful state, capability is no longer determined only by the frozen weights and the original prompt. The runtime around the model becomes part of the reasoning system.

The stack feels less magical now

I started with the idea that an LLM was an intelligent program sitting behind an API.

The model itself is closer to a giant collection of learned numbers.

The runtime is the program.

The transformer defines how those numbers interact.

The GPU executes the math.

The serving engine shares weights, isolates sequence state, manages KV-cache memory, batches work, and routes tokens back to requests.

Autoregressive generation wraps a loop around a fixed mathematical system. That loop lets generated state become new input, which supports context, language generation, and forms of iterative reasoning.

The abstraction is still remarkable. It feels much less magical once I follow the stack all the way down—and much more like a system engineers can measure, operate, and improve.

Alternate titles for review

• How One LLM Serves Thousands of Conversations
• The Runtime Around the Model: KV Caches, Batching, and Reasoning
• A Model Does Not Remember You: How LLM Context Actually Works
• How Autoregressive Generation Turns a Fixed Model Into an Iterative System