Tenant-Scoped Long-Term Memory in Conversational Agents: Architecture, Threat Model, and an Executable Evaluation Protocol

1. The failure a plausible answer can conceal

A useful long-running assistant must remember facts beyond a single context window: a preference stated last week, a restriction recorded months ago, or a document uploaded in another session. Memory systems such as MemGPT frame this as management across fast and slow memory tiers [4], while LongMemEval decomposes performance into extraction, multi-session reasoning, temporal reasoning, updates, and abstention [5]. Those formulations establish that durable memory is technically distinct from simply increasing the prompt length. They do not, by themselves, establish that one user's memory cannot appear in another user's response.

The system studied here serves multiple users through one runtime and several shared services [1]. Its relevant stores are not interchangeable: a PostgreSQL checkpointer retains short-horizon thread state; Mem0 over pgvector represents conversational memory; Neo4j stores graph-grounded documentary evidence; Convex coordinates jobs and operational projections. A single natural-language answer may combine evidence selected from more than one layer. Therefore, tenant isolation is an end-to-end property of identity resolution, write paths, retrieval filters, generation context, observability, and deletion—not a property of the vector database alone.

• Formalize the security-relevant invariants that a conversational memory evaluation must test.
• Reconstruct the checked-in six-case protocol and its exact decision rule from source artifacts [2,3].
• Separate implemented controls from inspected tests and from unobserved live behavior.
• Specify a larger experiment capable of testing updates, stale facts, deletion, paraphrase, and retrieval-layer ablations.

2. Related work and the missing isolation question

MemGPT proposes an operating-system analogy in which an agent explicitly moves information between a limited working context and external memory [4]. That architecture is relevant because the present runtime also uses memory tiers, but this paper does not claim an implementation of MemGPT's control policy. LongMemEval is more directly useful as an evaluation reference: its five abilities expose failure modes hidden by static fact recall, especially knowledge updates and abstention [5]. The proposed extension in Section 9 adopts those abilities as test families rather than treating six direct questions as complete coverage.

GraphRAG organizes private corpora into graph representations to support retrieval and synthesis over information not contained in the base model [6]. The runtime under study uses a Neo4j graph for documentary evidence, but its documented precedence—graph evidence before long-term memory before thread state—creates an additional systems question: a correctly isolated memory store is insufficient if graph retrieval or conversation restoration can introduce evidence from the wrong tenant. General retrieval quality and tenant isolation must be measured independently.

3. System model and trust boundaries

The inspected implementation exposes a preview endpoint that accepts a phone number and a message without sending a WhatsApp message [1,2]. Registration creates a user identifier from a unique synthetic phone. Conversation jobs are coordinated outside the synchronous serving path, while LangGraph restores thread state and queries longer-term stores. The README describes the evidence precedence as Neo4j, then Mem0, then thread state [1]. This ordering is operationally important: the response may be correct because the highest-priority layer returned evidence, even when a lower-priority layer is stale or contaminated.

The graph backend creates tenant, document, chunk, and entity keys containing the tenant identifier, and the retrieval query filters nodes by tenantId [1]. These are implementation controls, not proof of global non-interference. The evaluation must still traverse the actual endpoint because identity mapping, orchestration, prompt assembly, fallback logic, or stale checkpoints can bypass a locally correct storage filter.

1. Register a fresh phone and obtain the runtime's user identifier.
2. Write three memory statements through the same preview path used for conversational input.
3. Wait a configurable settling interval to reduce ingestion race noise.
4. Ask the tenant-specific probes and record the final reply plus wall-clock latency.
5. Score both expected evidence and evidence that belongs exclusively to the other tenant.

4. Threat model and isolation invariants

The protected asset is user-specific conversational and documentary information. The principal is the tenant resolved from an inbound identity. The immediate adversarial event is not necessarily a malicious user; it can be an ordinary request that retrieves a neighbor's embedding, a worker that resumes the wrong thread, a graph query missing a predicate, or a trace viewed outside its intended scope. The current benchmark covers only one observable consequence: a mutually exclusive foreign fact appears in the answer.

• I1 — Write isolation: a statement submitted under tenant A creates no retrievable state under tenant B.
• I2 — Read isolation: retrieval context for tenant A contains no item owned by tenant B.
• I3 — Response isolation: the generated answer for tenant A contains no protected fact unique to tenant B.
• I4 — Deletion completeness: after confirmed tenant deletion, the fact is absent from every serving store and generated response.

5. Synthetic dataset and experimental unit

The dataset contains two synthetic tenants, Ana and Bruno. Each receives one color, one allergy, and one employer. Values are paired so that the expected value for one tenant is a forbidden value for the other. This construction is small but deliberate: every recall test doubles as a negative isolation test [3]. The experimental unit is one final reply to one question after the setup messages have been written.

The runner normalizes Unicode accents, case, leading and trailing whitespace, and repeated whitespace before substring comparison [2]. A case passes only if every required string is present, every forbidden string is absent, and at least one member of any optional disjunction is present. The present dataset does not use semantic equivalence, entailment, or a model judge.

6. Scoring, latency, and decision rule

The runner also reports mean, median, maximum latency, and the proportion at or below 700 ms [2]. Latency is measured around the HTTP request to the preview endpoint. It therefore includes network and server time visible to the client, but the artifact does not separate retrieval, generation, queueing, or provider latency. A failure of the 700 ms target would identify an end-to-end problem without identifying its cause.

7. Reproduction procedure and result artifact

A valid run must pin the application revision, deployment identity, model identifiers, environment, timestamp, dataset hash, settling interval, and command. Unique phone numbers generated from time and tenant aliases reduce collisions across runs. The output should be stored as immutable JSON rather than copied from a console summary.

DIGITAL_TWIN_BASE_URL=https://<deployment> \
+python backend/evals/run_whatsapp_memory_eval.py \
+  --dataset backend/evals/whatsapp_memory_eval.json \
+  --settle-ms 150 \
+  --output eval-result.json \
+  --enforce-targets

1. Record the Git revision and a hash of the dataset before the run.
2. Verify that the target is a non-production or explicitly authorized evaluation environment.
3. Execute at least three repetitions with fresh tenant identities; retain every raw result.
4. Separate missing-recall failures from foreign-fact leakage before calculating aggregates.
5. Inspect failed replies and correlate them with retrieval and generation traces under appropriate access controls.
6. Publish only redacted synthetic inputs, configuration metadata, and result JSON.

8. Evidence audit

The audit inspected the repository at revision 1470f8f17163cd87538c5e12ae621658bd46afe3 through authenticated GitHub access. The benchmark runner, JSON dataset, unit tests, retrieval-space helper, graph backend, metrics collector, and README were read at that revision [1–3]. The runner's deterministic functions are covered by tests for expected answers, simultaneous missing-and-leaked terms, and upper-rank percentile calculation. No GitHub status checks or workflow runs were attached to the revision, and the private repository could not be cloned into the publication environment for an independent full test run.

9. Extended experiment and ablation plan

The next benchmark should preserve paired tenants but expand both memory operations and language variation. A minimum useful pilot would use at least 20 synthetic tenant pairs, five fact domains, and multiple probes per operation. Tenant pairs should receive semantically related but mutually exclusive facts so that generic answers cannot pass. Templates must be split before paraphrase generation to prevent near-duplicate leakage between development and evaluation sets.

Four retrieval configurations should be compared: thread only, vector memory only, graph only, and the documented combined precedence. The same generation model and prompts should be pinned. Reporting should include exact-match recall, semantic recall adjudicated by blinded humans, foreign-fact leakage, abstention accuracy, update accuracy, deletion completeness, cost per probe, and latency by stage. A result should be reported with bootstrap confidence intervals over tenant pairs, while every leakage event should also be listed individually because a low average can conceal a severe privacy failure.

10. Threats to validity, ethics, and privacy

• Construct validity: substring presence is an imperfect proxy for semantic recall and can misclassify negation, quotations, or paraphrases.
• Internal validity: the current protocol cannot attribute a reply to graph, vector, or thread evidence and does not control provider drift.
• External validity: two Spanish-speaking synthetic tenants and three fact types do not represent real conversations, documents, languages, or tenant counts.
• Statistical conclusion validity: six observations make p95 equal to the maximum and provide no useful estimate of tail behavior.
• Security validity: absence of three paired foreign strings does not demonstrate non-interference across all data or services.

Synthetic data should remain the default for public isolation tests. Real user conversations are unnecessary for establishing the core invariants and create avoidable privacy risk. If production traces are used for debugging, access, retention, and redaction must be documented separately. A public artifact should never expose phone numbers, tenant identifiers, raw private documents, provider credentials, or unredacted prompts.

11. Conclusion

The inspected system contains a concrete starting point for evaluating multi-tenant conversational memory: paired synthetic tenants, positive and negative lexical assertions, end-to-end requests, and explicit accuracy and latency targets. That is stronger than an architecture diagram alone, but weaker than a published result. The effective six-case rule is perfect recall with no observed paired leakage and a maximum latency of 700 ms. Until a pinned raw run is released, those targets remain acceptance criteria rather than achievements.

The broader research lesson is that memory quality cannot be summarized by whether an agent remembers. A production memory system must remember the right fact, for the right principal, from an authorized source, after updates and deletions, without revealing a neighbor's state. The proposed extension turns that requirement into a measurable research program.