The thing that bothers me about long AI conversations is not just the cost.

It is the waste.

A thread starts with a question. Then clarification, examples, a detour, a correction, a partial conclusion, a new branch. By the time it becomes genuinely useful, it is also bloated: thousands of words, tens of thousands of tokens, and a lot of repeated cognitive scaffolding.

The obvious answer is summarization.

But summarization is lossy.

If you have ever worked inside a long-running thread with an LLM, you know the problem. The summary preserves the headline but not the pressure. It remembers the decision but loses why that decision was fragile. It keeps the “what” and compresses away the “therefore.”

What I am interested in is a stronger idea:

Can we compress thinking without changing the model’s next-token distribution?

Not “same meaning.” Not “shorter summary.” Not even “same words.” Something more like this:

Given a full conversation C,
find a compressed state Z = f(C),
such that the model behaves as if it had seen C.

In probability language:

P(y | C) ≈ P(y | Z)

Where C is the full context, Z is the compressed state, and y is the future continuation. If the approximation becomes exact:

P(y | C) = P(y | Z)

then Z is a sufficient statistic for the conversation C, relative to the model’s future behavior.

That, to me, is the real frontier:

The problem with language as scratchpad

Chain-of-thought unlocked something important. Asking a model to reason step by step often improves performance, especially on math, code, planning, and multi-hop reasoning.

But natural language is a strange medium for thought. It is serial, verbose, and optimized for human communication, not internal computation. A model that writes:

Let us analyze the problem carefully.
First, we should identify the key constraints.
Then we should consider edge cases.

is spending tokens on coherence, politeness, formatting, and exposition. Some of those tokens matter. Many do not.

The hidden question is: which tokens actually move the distribution?

After context C, the next-token distribution is:

P(x_{t+1} | x_1, x_2, ..., x_t)

If a reasoning trace adds tokens r_1, ..., r_n, the model conditions on:

P(y | C, r_1, r_2, ..., r_n)

But if many of those tokens are redundant, maybe there is a shorter z such that:

P(y | C, r_1, ..., r_n) ≈ P(y | C, z)

The sharp target is not “make the reasoning shorter.” It is “make it shorter without changing what the model believes next.”

Perplexity as the lens

Perplexity measures how surprised a model is by a sequence. For tokens x_1, ..., x_N, the cross-entropy is:

H = -1/N Σ log P(x_i | x_<i)
and perplexity is:

PPL(x) = exp(H)

Lower perplexity means the model was less surprised. Now take two contexts, the original long thread C and the compressed state Z. If they produce the same distribution over future outputs, the model’s perplexity on those outputs should match:

PPL(y | C) ≈ PPL(y | Z)

A caution worth stating plainly: perplexity on one observed continuation is the thermometer, not the patient. Two different distributions can assign the same y identical perplexity while diverging everywhere else. The real claim is about the whole distribution over continuations, which is why the stricter version reaches for KL divergence:

D_KL(P_model(. | C) || P_model(. | Z)) ≈ 0

If the KL is near zero, the compressed state is behaviorally close to the original context. “Same perplexity” is the catchy phrase. “Same distribution” is the actual goal.

Three kinds of compression

At least three different problems hide under the word “compression.”

1. Text compression

Ordinary lossless compression: gzip, arithmetic coding, dictionary coding. It satisfies:

decompress(compress(C)) = C

Useful for storage. But it does not solve the context problem unless the model can operate on the compressed form directly. If the model has to decompress the whole thing before reasoning, we have saved storage, not cognition.

2. Semantic compression

Summarization. It tries to preserve human-judged meaning:

meaning(C) ≈ meaning(summary(C))

Useful, but lossy. It tends to drop uncertainty, alternatives, failed branches, source dependencies, and the logical pressure that made a decision meaningful. A normal summary says:

The team decided to do X.

A better state representation says:

Decision: X
Why: A, B, C
Rejected alternatives: Y, Z
Fragility: depends on assumption Q
Open risk: R

The second is still compressed, but it preserves the decision surface.

3. Distribution-preserving compression

This is the interesting one. The goal is not to preserve the text. It is to preserve the model’s behavior:

P(y | C) ≈ P(y | Z)

As an optimization problem:

argmin_Z D_KL(P(. | C) || P(. | Z))
subject to length(Z) << length(C)

Or as a single objective with a compression budget:

min_Z  D_KL(P(. | C) || P(. | Z)) + λ · length(Z)

Compress the thought-state as much as possible while keeping the continuation distribution unchanged. That is the clean form of the dream.

The sufficient statistic view

The best mathematical phrase for this is sufficient statistic.

In statistics, a statistic T(X) is sufficient for a parameter θ if it preserves all the information in X needed to infer θ:

P(θ | X) = P(θ | T(X))

For context compression, replace θ with future continuations Y:

P(Y | C) = P(Y | T(C))

So the dream object is a minimal sufficient state for the model’s continuation distribution. Not a transcript. Not a summary. A state.

This already has a name

The framing above is not new. It is the information bottleneck, introduced by Tishby, Pereira, and Bialek in 1999.

The bottleneck takes an input variable X and finds a compressed representation Z that keeps as much information as possible about a relevant variable Y, while throwing away the rest of X. Formally, you trade off two mutual informations:

minimize   I(X; Z)        (compress X)
subject to preserving I(Z; Y)   (stay relevant to Y)

Map X → C (the conversation), Z → the compressed state, Y → the future continuation, and the essay collapses into one line that is already twenty-five years old. My argmin_Z length(Z) subject to a KL bound is the bottleneck with a description-length rate term and KL as the distortion. My “minimal sufficient state Z*“ is the bottleneck solution in the lossless limit.

Two things the bottleneck gives for free, both of which matter here. First, the original IB paper frames itself explicitly as a generalized sufficient statistic, which is exactly the section above. Second, it is a rate-distortion problem in which the distortion measure is not chosen in advance but defined by relevance to Y. That second point is the hinge for everything that follows: gist tokens, soft prompts, compressed chain-of-thought, and KV compression are all, underneath, attempts to solve the same bottleneck with different parameterizations of Z. The labs are not circling a new idea. They are building practical solvers for an old one.

The literature: practical solvers for the bottleneck

1. Prompt compression: removing the redundant words

One line of work compresses the input prompt while preserving task performance. LLMLingua (Microsoft Research, with Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, Lili Qiu, and collaborators) uses a coarse-to-fine pipeline with budget control and token-level compression. LongLLMLingua extends it to long contexts, where position and density of key information matter as much as raw length.

The insight is that token value is not uniform:

importance(x_i) ≈ D_KL(P(. | C) || P(. | C \ x_i))

If deleting a token barely changes the output distribution, the token is compressible. The bigger the KL after deletion, the more load-bearing the token. That is the heart of prompt compression, and it is a direct, local estimate of the bottleneck’s relevance term.

2. Gist tokens: compiling prompts into model-readable artifacts

A more radical move is to stop pretending compressed context must stay human-readable. “Learning to Compress Prompts with Gist Tokens” (Jesse Mu, Xiang Lisa Li, Noah Goodman, Stanford) trains models to compress prompts into special tokens that can be cached and reused:

C → [GIST_1, ..., GIST_k],  k << length(C)
P(y | C) ≈ P(y | GIST_1, ..., GIST_k)

The instruction is not “summarize the prompt.” It is “compile it.” The output is meant for the model, not for a human reader.

3. AutoCompressors and soft prompts: latent memory slots

AutoCompressors (Alexis Chevalier, Alexander Wettig, and collaborators in Danqi Chen’s group at Princeton) push into latent compression. A long context becomes a small set of summary vectors, fed back as soft prompts:

C → z_1, ..., z_k

where each z_i is a vector, not a word, and the model generates from P(y | z_1, ..., z_k). The In-Context Autoencoder (Ge, Hu, Wang, Wang, Chen, Wei) is similar: compress context into memory slots the model can use later. Conversation history starts to look less like a transcript and more like a recurrent state:

state_t = update(state_{t-1}, message_t)
P(y_t | state_t)

This is the cleanest realization of the direction I care about. A thread becomes a handful of vectors that condition future behavior.

4. Dynamic soft-token allocation: capacity follows density

Compressing every chunk equally is obviously wrong. A paragraph holding the key constraint deserves more capacity than a thousand words of preamble. Split C into chunks and assign each a budget b_i, with Σ b_i = B, allocating where predictive information is densest.

DAST, Dynamic Allocation of Soft Tokens (Chen et al., Findings of ACL 2025), does exactly this, combining perplexity for local importance with attention for global relevance, without relying on an external model. Compression should follow information density, and DAST makes the allocation itself a learned function of it.

5. Compressed chain-of-thought: compressing the reasoning itself

Prompt compression attacks the context. But the reasoning trace is also compressible. Let the explicit trace be R = r_1, ..., r_n. Standard chain-of-thought gives the model P(y | C, R). Compressed chain-of-thought asks for a smaller Z_R with:

P(y | C, R) ≈ P(y | C, Z_R)

Compressed Chain-of-Thought proposes continuous “contemplation tokens,” dense stand-ins for explicit reasoning. Coconut, Chain of Continuous Thought (Shibo Hao, Sainbayar Sukhbaatar, Zhiting Hu, Jason Weston, Yuandong Tian), goes further and reasons in continuous latent space rather than committing each step to language.

The philosophical hinge is simple:

language ≠ thought

Language is one rendering of thought. The hidden state already exists inside the transformer. The question is whether reasoning can proceed through those states directly, turning:

thought → word → thought → word → answer

into:

thought → latent state → latent state → answer

6. KV-cache compression: preserving internal attention memory

There is a deeper layer: compress not the text, but the model’s internal cache. During generation, a transformer stores previous keys and values, and the next token depends on that cache:

P(x_{t+1} | x_≤t) = model(KV_≤t)

KV-cache compression asks whether a smaller cache KV' preserves the same logits:

minimize D_KL(P(. | KV) || P(. | KV'))
subject to memory(KV') << memory(KV)

This is the closest existing work to the strict dream. Instead of preserving the text, preserve the internal attention state the text created. For long-running agents this may matter more than prompt compression, because the model does not need every old token. It needs the consequences of those tokens.

7. Cartridges: training the compressed state offline

The most direct instantiation I have seen is Cartridges (Eyuboglu et al., Hazy Research, 2025). Instead of placing a whole corpus in context, they train a small KV cache offline, once, and load it at inference time. The cost amortizes across every future query against that corpus.

The detail that matters for this essay is how they train it. Naive next-token prediction on the corpus does not work; it underperforms in-context learning. What works is a recipe they call self-study: generate synthetic conversations about the corpus, then train the cartridge with a context-distillation objective, matching the behavior of the model that had the full context. On long-context benchmarks the trained cache matches in-context performance while using roughly 38x less memory and serving roughly 26x faster.

Read that finding against the thesis. Training the compressed state to reproduce the text fails. Training it to reproduce the behavior succeeds. That is the whole argument of this essay, recovered empirically: preserve the distribution, not the words. Cartridges is the bottleneck solved by gradient descent on the relevance term directly.

What OpenAI and Anthropic are doing publicly

The commercial labs are clearly building around this, though they do not use these words.

OpenAI’s docs talk about reasoning models, reasoning effort, reasoning tokens, and compaction for long-running interactions. The abstraction is “more reasoning budget buys more internal computation, but more context costs more,” so the system needs to preserve useful state without carrying the full transcript. OpenAI also supports prompt caching, which is adjacent but different: caching says “same prefix, reuse computation,” while compression says “different, shorter prefix, same behavior.” Caching avoids recomputing context. Compression replaces it.

Anthropic exposes extended thinking, summarized thinking, prompt caching, and context editing or compaction for long-running agents. Extended thinking gives the model more internal room. Summarized thinking changes what the user sees. Context editing addresses agents accumulating too much history.

Both companies are visibly moving from “chat transcript” toward “agent state.” But neither publicly claims the strict version: for arbitrary C, construct Z with P(. | C) = P(. | Z) and length(Z) << length(C). That remains a research frontier.

The labs to watch

Three clusters, by approach. Microsoft Research owns the production-grade prompt compression branch through LLMLingua: cut prompt length, hold task performance, reduce cost and latency now. Danqi Chen’s group at Princeton owns the latent-context branch through AutoCompressors, treating compressed context as model-native state rather than shorter text. The Stanford line through Jesse Mu, Xiang Lisa Li, and Noah Goodman’s gist tokens is the cleanest bridge from prompts to learned compressed representations.

I would add two more. The latent-reasoning cluster around Coconut (Hao, Sukhbaatar, Hu, Weston, Tian) is the one most directly about replacing verbal traces with continuous computation. And Chris Ré’s group at Stanford, via Cartridges, is the one turning the offline-trained compressed state into something that actually ships.

Why this matters

Today, long-context models create the illusion that the problem is solved. Just make the window bigger.

But bigger windows are not a theory of memory.

They are a larger desk.

A good cognitive system should not need to reread every note it has ever taken. It should maintain state: which facts are live, which constraints are binding, which uncertainties matter, which old tokens were merely scaffolding. The goal is not infinite context. The goal is state:

Z* = argmin_Z length(Z)
subject to D_KL(P(. | C) || P(. | Z)) ≤ ε

If ε = 0, this is exact behavioral losslessness. If ε > 0, it is approximate thought compression. Most practical systems will live in the approximate regime, and even that would be transformative.

The hard limits

There is no free lunch, but the usual statement of the limit is the wrong one.

The reflex is to invoke entropy: lossless compression is bounded by H(C), so you cannot shrink an arbitrary context below the information it contains. True, and irrelevant. H(C) is the entropy of the whole conversation, most of which the thesis already concedes is irrelevant to the future. The conversation is full of phrasing, politeness, and dead branches that move the continuation not at all.

The binding constraint is not H(C). It is the predictive information: how much of that entropy actually tells you about the future. Bialek, Nemenman, and Tishby named it precisely, as the mutual information between the past and the future of a sequence:

I_pred = I(C ; Y)

This is the floor that matters:

length(Z*) ≥ I(C ; Y),  not H(C)

And I(C; Y) can be vastly smaller than H(C). That gap is not a technicality. It is the entire reason the dream is plausible rather than crazy. You are not compressing the conversation. You are compressing the part of the conversation that predicts what comes next, and most conversations carry far less predictive information than they do raw text.

The harder question is what “same behavior” even means, because it has several non-equivalent definitions. Same next token? P(x_{t+1} | C) = P(x_{t+1} | Z). Same full answer? Same decision, argmax_y P(y | C) = argmax_y P(y | Z)? Same uncertainty, H(P(.|C)) = H(P(.|Z))? Same behavior across every possible future user turn, ∀u: P(y | C, u) = P(y | Z, u)?

Exact equality across all futures is almost certainly impossible in general. But task-relative sufficiency is reachable. For a coding agent, preserve repo state, failing tests, current hypothesis, plan, constraints. For a research agent, preserve claims, sources, uncertainties, open questions, contradictions. For a personal assistant, preserve preferences, commitments, deadlines, relationships. Not universal losslessness. Sufficiency relative to a future distribution. And that may be enough.

The conclusion

The next frontier in AI reasoning may not be longer thoughts. It may be denser ones.

We started with prompts, then chain-of-thought, then long context. The obvious consequence is now arriving: if every useful interaction becomes a giant transcript, intelligence becomes expensive to maintain. The future probably looks less like “put the whole conversation back into the prompt” and more like “maintain a compact, model-native state that preserves what matters.”

Call it context compaction, latent memory, gist tokens, soft prompts, compressed chain-of-thought, KV-cache compression, cartridges, or the information bottleneck. The phrase I like is perplexity-preserving thought compression.

Because the real question was never whether we can make the transcript shorter.

It is whether we can remove the words without changing the mind.

About SG

I run Dobby Ads, an AI Creative Agency. I tend to overthink. This is where that overthinking goes. Connect with me on LinkedIn.

After I published the Interruptive Thinking paper, I wanted to keep going.

The paper had landed. The ideas were out. But the conversation that produced it - the long, looping one with Claude over many days, where the actual thinking happened - had open threads I hadn’t followed yet. Things the conversation had pointed toward, and we hadn’t reached. Adjacent territory, the paper didn’t cover, but the conversation had been pressing into.

So I opened a fresh thread to continue.

It didn’t work.

I tried giving the new thread a chronological summary. First, we talked about this, then this came up, then this turn shifted the frame. The new thread read it patiently. Said something polite. Started shallow.

I tried pasting the entire previous chat. Tens of thousands of words. The new thread read it. Started shallow.

I knew the ideas. I had the paper to prove it. What I had lost was harder to name. It was the state I had been in by the end of the previous conversation. The state where the next thought was almost arriving. Anjali Singh, writing about AI and human cognition, points back to Dewey: reflective thinking requires enduring a state of perplexity, confusion, or doubt prompting inquiry, and a suspension of judgment during this period of inquiry. That endured perplexity is the state. It is what gets thinking to keep going. And it is the first thing the summary throws away.

Lev Tankelevitch, in his work on what he calls the metacognitive demands of generative AI, points out that working with these tools well requires constantly tracking your own thinking - your goals, your confidence, your shifting strategies. Most of that tracking is invisible even to yourself. It almost never makes it into a summary.

Summaries kept failing because summaries are the wrong shape for what I needed. A summary captures what was said. It cannot capture where I was inside it.

The turns are visible. The state is not. The state lives in the pressure points where I almost said something and pulled back. In the questions, I noted internally but did not articulate. In contradictions, we surfaced but intentionally left unresolved. In the direction we were heading when we stopped.

None of this survives the compression to bullets.

There is a second layer to this, harder to talk about. When I tried to resume, I was not just re-entering my own state. I was trying to re-create a shared zone that had existed between me and the previous Claude instance. That instance was gone.

It had not been thinking about this. It could read the transcript, but it could not have been warming to it. It started cold because it could only start cold. And cold partners produce cold conversations, no matter how much context they have.

If I had been resuming with a person - a friend I think with - they would walk in still partly warm. They would have been turning it over. They would arrive with a new angle or a lingering doubt. Some of the state would be preserved in them. We would re-warm faster because at least one of us never fully cooled.

With AI, no one really stays warm. Not the AI, because it has no continuity. Not me, because I have slept and worked on other things. The only bridge is the artifact. And the artifact is summary-shaped when what I needed was state-shaped.

What eventually worked was not a better summary.

I gathered every conversation I had had on the topic. Every fragment. Every adjacent thread. I made something that was not a record of what we had discussed, but a map of where we had pressed and not finished pressing. Open questions. Live tensions. Unresolved directions. Things I almost saw. I wrote them as questions, not conclusions. The artifact was not the conversation. It was the unresolved perplexity the conversation had left in me, written down before it cooled.

That worked. Not all at once. Not perfectly. But I could re-enter.

Which makes me think the problem with summaries is not that they are bad summaries. The problem is the genre itself. A summary’s job is to compress toward resolution. To say what something was. But thinking does not live in what was.

The summary is the artifact of a conversation that is over. To continue a conversation, you need the artifact of one that has not ended yet.

I do not have a name for that artifact. I am not sure anyone does. I know it would have to do two things at once. It would have to bring me back to the state I left - re-enterable, not just readable. And it would have to bring the AI to a starting state that is warm - not a transcript to be processed, but a live position that had not fully cooled.

Both. Past that, I am still where you are.

What I am sure of is that almost everyone working with AI right now is trying to do this with summaries. The summaries keep producing shallow re-entries, and we keep blaming ourselves or the AI for not getting back to where we were.

The summary is not where we lost the thinking. The summary is the place we never built the right thing in the first place.

About SG

I run Dobby Ads, an AI Creative Agency. I tend to overthink. This is where that overthinking goes. Connect with me on LinkedIn.

I have been talking with AI almost every day for a while now. Long threads. Hundreds of them. Some run for hours.

And every now and then, I scroll back through one and ask myself a simple question.

What did we even talk about?

It is harder to answer than it should be.

I have tried.

First I tried summary. Top, middle, bottom. The shape of an essay. But a summary is a destination, and most of the thread was the road. The interesting parts were the wrong turns. You cannot summarise a wrong turn without losing why it was interesting.

Then I tried thinking trail. Which is closer. A trail at least admits that thought moves. But a trail is a line. And what happens in these threads is not a line. I digress. I come back. A new idea shows up sideways. The trail loses the digressions or pretends they were the path all along.

Most recently I have been working with a concept I call line of thought. A line of thought is something that does not live inside one thread. It runs across many. It starts in a chat with one AI on a Tuesday, gets dropped, picks up again three weeks later in a different chat, finishes somewhere else entirely. The thread is just where it happens to be visible. One thread could have many.

I like this one better than the others. It admits the thing I keep noticing: that what matters is not contained by what I am calling a conversation.

But it is still not enough. I am still bothered.

Each name got something true. Summary caught that there are conclusions. Thinking trail caught that thought moves. Line of thought caught that thought outlives any one container.

The reason I am writing this is because I have started to suspect the problem is not that I have not found the right name. The problem is that what I am trying to name does not behave like the kind of thing names usually fit.

Let me try a different angle.

When two friends talk, what are they doing?

Not exchanging information. Most friend conversations are not informational. They wander. They double back. Someone says something in the wrong order. Someone laughs and the laugh changes the next ten minutes. The good parts are not retrievable. If you transcribed a great evening with a friend you would not be able to find what made it great. It would be in the timing of a pause, a glance you did not write down, the way one of you said something you both already knew.

A counsellor and a patient is a different animal. There is a destination, sort of. The patient is supposed to find something. The counsellor is helping but not leading. The leading happens in a particular asymmetric way, where the counsellor pulls the patient gently toward parts of themselves the patient has been avoiding. If you transcribed that, you would find pauses, redirections, questions that landed and questions that did not. Not the same animal as two friends.

Two people texting on WhatsApp is yet another thing. Mostly social. Maintenance. Keeping the line warm. The actual content is often beside the point. The thread is the relationship breathing.

And then there is what I do. Most days. Sometimes for hours.

I talk to AI.

Or at least, that is what I have been calling it.

When I am with AI, I am not exchanging the way I do with a friend. I am not being pulled the way I am by a counsellor. I am not maintaining anything. What I am doing, if I am honest, is something closer to writing in a journal. Except the journal writes back.

I enter the park. I look at one garden. The AI describes it. I move to another garden. The AI describes that one too. I stop and look at something. The AI stops with me. I turn back. The AI turns back.

If you remember Weebo from the old Robin Williams movie Flubber. The little yellow robot that hovered next to the professor while he worked. Played back fragments of what he had said earlier. Sometimes added something of her own. Half companion, half mirror. That is closer to what is happening here than calling it a conversation.

The AI is good at this. It is patient. It is thorough. It catches things I would not have caught alone. But the structure of the thing we are doing is asymmetric in a way most other conversations are not. The AI does not have its own gardens it wants to show me. It does not interrupt because it just thought of something. It does not redirect because it suspects I am avoiding the real question. Those things happen, occasionally, in flashes. But mostly, no. Mostly, it feels like I lead.

Which makes me suspect the word conversation has been smuggling in an assumption I never tested.

The other party has their own pull. Their own thread of thought running alongside mine. AI does not. So whatever this is, it is probably not what I have been calling it.

When I scroll back through one of these threads and ask what we talked about, I am asking the wrong question.

The right question is closer to: what did I think about while the AI walked with me.

The thread is not a record of two minds meeting. It is a record of one mind wandering, with a very good companion who described the gardens.

That is a different essence than two friends. Different essence than counsellor-patient. Different essence than WhatsApp banter.

Same word, conversation, pointing at very different animals.

I notice this and I do not have a vocabulary for it. The tools I use to store these threads do not have a vocabulary for it either. They store every thread the same way. As text. With timestamps. As if the preservation problem were the same in each case.

It is not the same problem. A friend conversation, you preserve maybe with a feeling. A counsellor session, you preserve with what you uncovered. A WhatsApp thread, you do not really preserve at all. You let it scroll.

That one I am supposed to preserve. That one was thinking. Real thinking. Mine. The AI helped but the thought was mine.

I have started to suspect that what I am calling a thread is not one thing but several. The thing inside any given thread is shaped by the kind of conversation it was. And the geometry of the contents is different from the geometry of the container that holds them.

I have hundreds of these now and I cannot find anything in them. Tomorrow I will open a new chat and start over as if the previous hundred never happened.

I will keep writing as I keep noticing.

About SG

I run Dobby Ads, an AI Creative Agency. I tend to overthink. This is where that overthinking goes. Connect with me on LinkedIn.