Compaction

A long chat or a multi-step agent loop keeps adding messages. At some point the transcript passes the model's context limit and the call fails. You want the conversation to keep working without hitting that wall.

withCompaction shrinks provider context before each model call. When the context passes maxTokens, a strategy rewrites what the model sees. The canonical transcript does not change. Add this ChatMiddleware to the middleware array of any chat() call.

Install#

pnpm add @tanstack/ai-compaction

Quick start#

The default strategy drops the oldest messages once the transcript passes maxTokens and keeps the recent ones.

import { chat, toServerSentEventsResponse } from "@tanstack/ai";
import { openaiText } from "@tanstack/ai-openai";
import { withCompaction } from "@tanstack/ai-compaction";

export async function POST(request: Request) {
  const { messages } = await request.json();

  const stream = chat({
    adapter: openaiText("gpt-5.5"),
    messages,
    middleware: [withCompaction({ maxTokens: 100_000 })],
  });

  return toServerSentEventsResponse(stream);
}

Pick a strategy#

Pass strategy to change how the history shrinks. Three are built in.

evictOldest#

Cheapest. Keeps the recent tail, drops the older head, and leaves a short marker in its place. This is the default, so you only name it to tune keepRecentTokens.

import { withCompaction, evictOldest } from "@tanstack/ai-compaction";

withCompaction({
  maxTokens: 100_000,
  strategy: evictOldest({ keepRecentTokens: 40_000 }),
});

summarizeOldest#

Keeps the gist of old turns instead of dropping them, at the cost of one summarization call. Pass a summarize callback. It gets the messages about to be dropped and returns the summary text. Wire it to summarize() or any model call.

import { chat, summarize, toServerSentEventsResponse } from "@tanstack/ai";
import { openaiText, openaiSummarize } from "@tanstack/ai-openai";
import { withCompaction, summarizeOldest } from "@tanstack/ai-compaction";
import type { ModelMessage } from "@tanstack/ai";

async function summarizeHistory(messages: Array<ModelMessage>): Promise<string> {
  const text = messages
    .map((m) => `${m.role}: ${typeof m.content === "string" ? m.content : ""}`)
    .join("\n");

  const { summary } = await summarize({
    adapter: openaiSummarize("gpt-5.5"),
    text,
  });
  return summary;
}

export async function POST(request: Request) {
  const { messages } = await request.json();

  const stream = chat({
    adapter: openaiText("gpt-5.5"),
    messages,
    middleware: [
      withCompaction({
        maxTokens: 100_000,
        strategy: summarizeOldest({ summarize: summarizeHistory }),
      }),
    ],
  });

  return toServerSentEventsResponse(stream);
}

clearToolResults#

Best for agent loops. Tool output (file reads, command output) is usually most of the tokens. This strategy replaces the content of old tool results with a stub and keeps every message and its tool-call pairing in place. The conversation shape does not change.

import { withCompaction, clearToolResults } from "@tanstack/ai-compaction";

withCompaction({
  maxTokens: 100_000,
  // Keep the 5 most recent tool results in full, stub the older ones.
  strategy: clearToolResults({ keepRecentToolResults: 5 }),
});

Write your own#

A strategy is a function. It gets the messages and the budget, and returns the rewritten messages, or null to change nothing. It runs only when the estimate is over maxTokens.

import { withCompaction } from "@tanstack/ai-compaction";
import type { CompactionStrategy } from "@tanstack/ai-compaction";

// Keep only the last message.
const keepLastOnly: CompactionStrategy = (messages) => {
  if (messages.length <= 1) return null;
  return messages.slice(-1);
};

withCompaction({
  maxTokens: 100_000,
  strategy: keepLastOnly,
  strategyKey: "keep-last-v1",
});

Set strategyKey when you combine a custom strategy with persistence. Change the key when the strategy can produce different output. This prevents an old checkpoint from using stale behavior.

Combine strategies#

composeStrategies runs several strategies in order and escalates: it stops as soon as the result is back under maxTokens. Put the cheap, targeted strategy first and a broad fallback last. Here it clears old tool output first, and only drops old messages if that was not enough.

import {
  withCompaction,
  composeStrategies,
  clearToolResults,
  evictOldest,
} from "@tanstack/ai-compaction";

withCompaction({
  maxTokens: 100_000,
  strategy: composeStrategies(clearToolResults(), evictOldest()),
});

Options#

withCompaction#

Strategy options#

The token count is a rough characters / 4 estimate. It is good enough to trigger on, not exact. Pass estimateTokens for provider-accurate counts.

What it keeps safe#

• The system prompt is never dropped. chat() keeps it separate from messages, so compaction only touches the conversation.
• Tool calls stay paired with their results. The built-in strategies never leave an orphaned tool result, so the request stays valid.
• It runs before every model call. Compaction skips init. It runs on beforeModel and structuredOutput. Each later call can compact again.
• The canonical transcript stays complete. Compaction writes provider-only context. Persistence and other middleware still read ctx.messages.

DevTools#

After a compaction, the chat stream includes three CUSTOM events in order: compaction:started, compaction:state, then compaction:ended. compaction:started is sent before the strategy runs, so a slow summarizeOldest call still shows up as started on the client. The state and ended events follow when the strategy returns. TanStack AI DevTools has a Compaction tab on the hook. Each compact shows:

• started, state, and ended rows
• when it ran, with token and message counts
• the maxTokens budget
• dropped messages
• the transcript sent to the model

The conversation timeline also keeps onCompactStart, onCompact, and onCompactEnd steps.

Open the AI plugin in the DevTools panel (the ts-react-chat example mounts it). Select the Compaction hook, then open the Compaction tab.

The /compaction route in examples/ts-react-chat uses a small maxTokens so this fires after a few turns. It also uses withPersistence (SQLite). Reload after a compact: the chat still shows every message. The banner is example UI, not part of useChat.

Compaction and persistence#

Compaction and server-side withPersistence use two message views:

• messages is the complete canonical transcript. Persistence saves this view.
• providerMessages is temporary model context. Compaction rewrites this view.

Middleware order does not change this split. Dropped, summarized, and stubbed content remains in the message store.

If the persistence adapter has a metadata store, compaction also saves a small checkpoint. The next request validates the canonical prefix, restores the last compacted result, and adds only new messages. A changed prefix or strategy key invalidates the checkpoint.

When a checkpoint is reused, a later summarizeOldest pass sees the previous summary plus new messages. Then it folds the old summary into the new one. Folding needs a metadata store and a strategy key.

The default strategy, standard evictOldest, summarizeOldest, clearToolResults, and safe compositions get a strategy key automatically. Set strategyKey for custom strategies, custom estimators, or custom marker functions. Change strategyKey when your summarize function can change. Without a metadata store or safe key, compaction stays stateless.

Next steps#

• Middleware: the full hook reference and how middleware composes
• Built-in Middleware: ready-made middleware that ships in @tanstack/ai