A conversation must remember enough for the next question. The model does not need every previous tool call and result in its next prompt.
An implementation-level look at LangGraph checkpoints, thread-scoped conversation history, transient message shaping and oversized tool outputs.
The engineering problem
An enterprise assistant may answer a question using a SQL tool, Power BI semantic model or document retriever. A follow-up must retain enough of that exchange to interpret “those customers” or “last month”. But replaying every tool request and full result into every subsequent model call makes the prompt progressively larger.
The design problem has two different lifetimes: durable conversation state, and the transient context assembled for one inference step. Managing one does not automatically manage the other.
Every completed tool exchange is sent again, even when only its final answer matters to the new question.
Use thread-scoped checkpoints for conversation continuity and prepare a smaller message list immediately before calling the model.
Two layers · different responsibilities
The LangGraph state records the working conversation, including message objects written by agent and tool nodes. A separate message-building step chooses the previous turns and active exchange used for the current model invocation. That distinction is intentional, not an automatic property of MongoDB.
A thread's checkpoints and intermediate writes persist runtime state. A checkpoint is not the prompt itself.
This view is assembled for inference. Selecting fewer messages does not itself remove older checkpoints.
This article uses context to mean the message/instruction payload for a model call. Part 4 uses business context to mean the semantic meaning of a company's data. Both are forms of context engineering, but solve different problems.
State persistence
Ask Titan invokes its LangGraph orchestrator with a conversation-specific thread identifier and compiles the graph with a MongoDB-backed asynchronous checkpointer. When a follow-up uses the same thread, LangGraph can load the existing state before the new user message is processed.
There is also a separate, application-facing conversation record used to present user and assistant messages. It should not be confused with LangGraph’s technical checkpoint and intermediate-write records. Both exist for different consumers.
Conversation lifecycle
The application assigns a conversation ID that is also used as the LangGraph thread boundary.
The graph processes messages and tool responses, then stores its technical state through the MongoDB-backed checkpointer.
When a follow-up arrives on the same thread, LangGraph can load the checkpointed state first. The model view for the next turn is then assembled separately.
User-facing conversation record used to show messages and answers.
Technical runtime state and intermediate writes used to continue the thread.
The application conversation history and LangGraph checkpoints are distinct persisted representations. They serve different runtime and UI concerns.
Prompt assembly
Ask Titan locates the most recent user message. Earlier, finished turns become a limited series of user-question and final-assistant-answer pairs. Intermediate tool calls and their responses are not replayed for those finished turns. The active turn retains its tool-call exchange so the model can continue reasoning with the tool response it just received.
Why did margin fall?
Call Power BI tool
Detailed result + metadata
Margin decline explained
Which customers drove it?
Call specialist tool
Current result
Context
shaping
Select messages, not delete state
Why did margin fall?
Margin decline explained
Earlier tool calls and results are not replayed in this model pass.
Which customers drove it?
Call specialist tool
Current result or short notice
State is retained. Model context is selected. Completed turns contribute their question and final answer. The active turn keeps the assistant tool call paired with its response so the model can continue the exchange.
One conversation · two turns
A follow-up continues the same LangGraph thread. Ask Titan recovers its checkpointed state and builds a new model view from selected previous turns and the current question.
If the agent calls a permitted specialist, the result returns for another model pass before the final answer is produced. The new answer becomes part of persisted state, while earlier tool exchanges do not need to be replayed to the model.
Large tool results
A SQL query or document retrieval can return more text than is useful for the next reasoning step. The Ask Titan orchestrator estimates the size of tool messages. If an individual result exceeds its prompt-view threshold, it substitutes a small notice in the model-bound message list rather than inserting that payload in full.
That is a prompt-view substitution. The original tool message is not rewritten by this particular shaping function. It also does not mean the full data is automatically summarized or that every result is always available through a separate retrieval API.
One tool result · two representations
Detailed rows · metadata · intermediate evidence
“This tool output was too large to include fully. Request a narrower or summarized view.”
The model receives the notice, not the omitted rows.
This mechanism estimates individual tool messages. It does not guarantee that the complete model context stays within a total token budget.
Operational decisions
Context shaping is useful, but it is not a substitute for lifecycle, security and quality controls. These distinctions matter when turning a working assistant into a production service.
| Concern | Current Ask Titan mechanism | Boundary / engineering consideration |
|---|---|---|
| Thread continuity | LangGraph graph state backed by MongoDB checkpoints. | A shared thread must still be accessed through application-level user/conversation authorization. |
| Model input size | Finished-turn selection and approximate per-tool-payload estimation. | These are not proof that every possible request fits a model’s total context window. |
| Memory retention | Completed turns are omitted from a prompt view when not selected. | Prompt omission is not deletion, retention control or a guarantee of erased checkpoint data. |
| Summaries | Finished turns retain question/final-answer pairs rather than LLM-generated summaries. | No general-purpose automatic conversation summarizer is claimed here. |
| Restart and replay | Checkpoints support loading previous graph state. | External side effects and idempotency require their own handling; checkpointing alone is insufficient. |
| Sensitive content | Application-facing conversation records and technical checkpoints are separate stores/representations. | Each requires an explicit access, encryption, logging and retention review; one does not secure the other automatically. |
The orchestrator records approximate counts for full state and the assembled model view, and collects reported model-usage metadata when available. Those measurements help identify context growth; approximate counting is not a contractual token limit or a performance benchmark.
Engineering takeaway
LangGraph and MongoDB preserve conversation continuity. Ask Titan then assembles a new message view for each model call: recent finished user/final-answer pairs, the active tool exchange and bounded representations of oversized tool outputs.
The architectural gain is not “infinite memory”. It is an explicit boundary between durable state and the information a model needs at this moment. That boundary can be tested, observed and evolved independently.
The design in one sentence
Store enough to continue.
Send only what helps answer.
Next in the series · Part 6 of 7
How Ask Titan retrieves relevant information from enterprise documents and brings it into the context used to generate an answer.
Read Part 6FAQ
Practical answers about checkpointing, thread continuity, prompt selection, tool messages and context growth.
State is the graph’s accumulated working data, persisted through checkpoints in a thread. LLM context is the selected messages and instructions supplied to one model call; Ask Titan constructs it from state without equating the two.
The orchestrator uses the conversation ID as a LangGraph thread identifier and an asynchronous MongoDB-backed checkpointer. Reusing the same thread allows the next invocation to continue from its stored graph state.
No. In finished turns, the prompt view keeps recent user messages and final assistant answers. The active turn retains its tool exchange, subject to oversized-output substitution.
The inspected orchestrator condenses finished turns by selecting question/final-answer pairs; it does not use a general LLM-generated conversation summary in the observed path.
No. Prompt-view selection does not itself delete messages or checkpoints. Retention and deletion need separate lifecycle controls.
The runtime estimates individual tool-message size. If it exceeds the prompt-view threshold, the model receives a short explanatory notice instead of the full payload. The shaping operation does not itself modify the original stored tool message.
No. Checkpointing supports persisted graph state and resumption. Idempotency and external side-effect handling require additional application-level design.
No. Here context management concerns which conversation messages reach the LLM. In Part 4, business-context engineering concerns company-specific definitions, metadata and meaning used by a Power BI specialist.
Stateful AI architecture
Ask Titan combines stateful orchestration, specialist capabilities and governed data access. Discuss how to turn a LangGraph prototype into an assistant your teams can continue using across real business questions.
Four separate engineering responsibilities
Technical references
Product-specific observations are grounded in the supplied Ask Titan repository. These primary sources explain LangGraph persistence, message selection and how transient model context differs from durable state. The pseudocode and diagrams are deliberately simplified.