Context Systems

Context systems are the technical and organizational systems that collect, structure, retrieve, verify, and route context so that AI agents, software workflows, and human collaborators can act with relevant source-grounded information. On this page, "Context Systems" is used as an umbrella term for several adjacent practices and research areas rather than as a settled external field name.

The topic includes retrieval-augmented generation, long-context model behavior, agent memory, context engineering, tool and data-source protocols, provenance, security, and coordination context. It does not refer to a single product category. It also should not be reduced to personal CRM, although personal relationship-memory systems can be one useful example of private context infrastructure.

Background

Early language-model applications often treated context as text placed into a prompt. As systems became more agentic, context became a larger systems problem: which sources should be available, how those sources are represented, which parts are retrieved, how source provenance is preserved, and how humans can verify that an agent is acting on the right information.

Retrieval-augmented generation is one foundational approach. The original RAG paper describes combining parametric model memory with non-parametric retrieval from an external corpus, which made source updating and evidence retrieval part of the generation loop. That framing is central to context systems, but it is narrower than the full topic.

Long-context models changed the design space but did not remove the need for context management. Research on long-context behavior shows that models can fail to use relevant information reliably when it appears in the middle of long inputs. This makes selection, ordering, summarization, and evaluation important even when a model can accept large amounts of text.

Core Components

A context system usually combines several components.

Ingestion

Ingestion brings source material into the system. Sources may include documents, repositories, chat logs, meeting summaries, task records, tickets, browser state, database rows, tool outputs, or manually curated notes. Ingestion is not only a data pipeline problem. A useful context system needs to know where information came from, how fresh it is, who is allowed to use it, and whether it is raw, summarized, inferred, or reviewed.

Representation

After ingestion, context must be represented in forms that can be searched, retrieved, inspected, and updated. Common representations include chunks, embeddings, summaries, knowledge graphs, memory records, source ledgers, task state, and structured workflow artifacts. Graph-based retrieval systems are relevant when relationships and provenance matter, because they can preserve connections among entities, claims, and sources.

Retrieval and Ranking

Retrieval selects candidate context from a larger store. Ranking, filtering, and query rewriting decide what is actually shown to the model or human. Retrieval can use keyword search, vector search, graph traversal, recency filters, permissions, or hybrid approaches. In a context system, retrieval should be evaluated not only by whether it returns plausible material, but whether it returns the right material for the task.

Memory and State

Agent memory extends context across turns, sessions, or tasks. MemGPT frames this as a form of virtual context management: the model has a limited immediate context window, while the system manages longer-term memory outside that window. In practical systems, memory may include user preferences, project state, prior decisions, source summaries, task plans, and review outcomes. These records need provenance and review boundaries because remembered context can shape future behavior.

Context Routing

Context routing decides which sources, tools, memories, or documents should be available at a particular point in a workflow. A coding agent may need repository files, issue history, build logs, and review comments. A personal CRM may need contacts, messages, meeting notes, and follow-up tasks. A governance workflow may need proposals, votes, policy docs, and prior decisions. In each case, the routing problem is partly technical and partly organizational.

Verification and Evaluation

Verification asks whether the selected context is accurate, current, permitted, and sufficient for the action being taken. Evaluation may include retrieval benchmarks, source checks, citation checks, human review, deterministic tests, or workflow checkpoints. The more agency a system has, the more important it becomes to verify both the context and the action that follows from it.

Retrieval-Augmented Generation

Retrieval-augmented generation is one of the best-established parts of the context-system landscape. RAG systems retrieve external information and supply it to a model so the model can answer with information beyond its trained parameters. RAG is useful because source material can be updated without retraining the model, and because retrieved documents can provide evidence for generated output.

RAG also exposes common context-system problems. Retrieved material may be stale, incomplete, irrelevant, duplicated, adversarial, or missing important provenance. A generated answer can cite material incorrectly or overstate what the source supports. For that reason, RAG should be treated as a component in a larger context system that includes source tracking, verification, permissions, and evaluation.

GraphRAG and related graph-based approaches extend this pattern by using relationships among entities, claims, or documents to guide retrieval and generation. These methods are especially relevant when the system must preserve relationships, dependencies, ownership, or provenance across a body of knowledge.

Long-Context Limits

Long context windows allow models to process more input at once, but larger windows do not automatically produce better context use. The "Lost in the Middle" study found that model performance can drop when relevant information appears in the middle of long contexts. This is a useful warning for context-system design: adding more material can make retrieval and reasoning worse if the system does not select, order, compress, or highlight the right evidence.

Long-context systems therefore still need context engineering. The system must decide what belongs in the window, what belongs in memory, what should be summarized, and what should remain available as a source link rather than injected directly into the prompt.

Agent Memory and State

Agent memory is the part of a context system that persists beyond a single model call. It may store facts, user preferences, task state, source summaries, unresolved questions, prior tool results, or decisions made during a workflow. Memory can make agents more useful, but it can also compound errors if inferred or stale information is treated as verified fact.

A good memory design separates raw source material from summaries, and summaries from inferred claims. It also gives humans a way to inspect, correct, archive, or delete remembered context. In coordination-heavy settings, memory should preserve why a decision was made, who reviewed it, and which sources supported it.

Tool and Data-source Protocols

Context systems often need controlled access to tools and external data. The Model Context Protocol is one current example. It defines a standard way for applications to connect language models to tools and data sources. That makes it relevant to context routing: the system can expose files, databases, APIs, or actions to a model through a structured interface rather than through ad hoc prompt text.

Protocols do not solve trust by themselves. They make connection patterns more explicit, but each system still needs permission boundaries, source metadata, audit logs, and review rules for risky actions.

Provenance, Privacy, and Security

Context systems can fail when they lose track of where information came from or who is allowed to use it. Provenance is the link between a claim and its source. Without provenance, a system cannot reliably distinguish source-backed facts from summaries, stale memory, user preference, or model inference.

Security risks include prompt injection, sensitive information disclosure, supply-chain risk, poisoning, and excessive agency. These risks are especially important when context systems connect model workflows to external tools, private data, or long-lived memory. OWASP's LLM application guidance and NIST's Generative AI risk profile both support treating security, privacy, provenance, and evaluation as part of the system design rather than as later add-ons.

Context poisoning is a particularly important pattern. If malicious or low-integrity source material enters the context pipeline, it may influence downstream retrieval, memory, or agent action. Defensive design can include source allowlists, permission checks, human review, freshness metadata, content scanning, and separation between untrusted input and executable instructions.

Coordination Context

Context systems are not only model-runtime infrastructure. They also support coordination among people. A team may need to preserve requests, decisions, review comments, meeting summaries, source artifacts, and open questions so work can move between humans and agents without losing the thread.

The Spencer Graham fireside session that anchored this page raised context and verification as hard problems for AI agents and coordination. In that discussion, context systems appeared alongside personal CRM, agentic coding workflows, UX verification, and concerns about work becoming more atomized. That session is useful as a source anchor, but the page itself should remain a neutral reference page rather than a recap.

Related Examples

A personal CRM is one example of a private context system. It may collect contacts, messages, meeting notes, reminders, relationship history, and inferred follow-up prompts. That example shows why privacy, consent, provenance, and review matter. It should not define the broader Context Systems page.

Agentic coding workflows are another example. A useful coding agent may need repository context, issue context, prior review notes, test output, deployment state, and human acceptance criteria. UX QA agents similarly need interface state, screenshots, expected behavior, and review loops. These examples show how context systems connect retrieval and memory with verification and coordination.

Open Questions

• Should "Context Systems" remain an umbrella Portal title, or should a more established term replace it after review?
• Should "Context Engineering" become a separate wiki page now, or remain a major section of this page?
• Which tools should be listed without turning the page into a vendor survey?
• How much should the page emphasize human coordination context compared with model-runtime architecture?
• What evidence threshold should be required before claiming that a specific context system is secure, complete, or production-ready?

Related Topics

• Context Engineering
• AI Memory
• Retrieval-Augmented Generation
• GraphRAG
• Model Context Protocol
• Prompt Engineering
• Knowledge Graphs
• Personal CRM
• Agentic Coding Workflows
• UX QA Agents
• AI and Open Source Security in the Agentic Coding Era
• Local-first Software
• Personal Knowledge Management

Further Reading

• June Cohort Fireside Chats (Spencer Graham). RaidGuild Portal. https://portal.raidguild.org/sessions/49
• Spencer on Personal CRM, Agentic Workflows, and the Future of Coordination. Prism Memory. https://prism-memory-production-002c.up.railway.app/artifacts/20260610_170144Z-discord-voice-8c99be34
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv / NeurIPS 2020. https://arxiv.org/abs/2005.11401
• Lost in the Middle: How Language Models Use Long Contexts. arXiv / TACL. https://arxiv.org/abs/2307.03172
• MemGPT: Towards LLMs as Operating Systems. arXiv. https://arxiv.org/abs/2310.08560
• Specification - Model Context Protocol. Model Context Protocol. https://modelcontextprotocol.io/specification/2025-06-18
• Introducing the Model Context Protocol. Anthropic. https://www.anthropic.com/news/model-context-protocol
• Effective context engineering for AI agents. Anthropic Engineering. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
• Context Engineering for Agents. LangChain. https://www.langchain.com/blog/context-engineering-for-agents
• Graph Retrieval-Augmented Generation: A Survey. arXiv. https://arxiv.org/abs/2408.08921
• OWASP Top 10 for Large Language Model Applications. OWASP. https://owasp.org/www-project-top-10-for-large-language-model-applications/
• Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST. https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence