Recall, measured
How well ContextQ answers questions about long conversation histories, on the public LongMemEval benchmark. Small sample, stated plainly.
70.0% on 30 questions
| Ability | Count | Correct | Accuracy |
|---|---|---|---|
| information-extraction | 6 | 5 | 83.3% |
| multi-session-reasoning | 6 | 2 | 33.3% |
| temporal-reasoning | 6 | 4 | 66.7% |
| knowledge-updates | 6 | 4 | 66.7% |
| abstention | 6 | 6 | 100.0% |
| Aggregate | 30 | 21 | 70.0% |
Four caveats
-
The sample is small
n=30. One question moves the aggregate by 3.3 points, and each ability row is 6 questions.
-
Runs vary by about 2 questions
The answer and judge steps are not deterministic. Two later runs on the identical memory set scored 66.7% and 73.3% on the same 30 questions, so a single run carries about +/-2 questions of noise. Read the 70.0% headline with that band.
-
Not comparable to other vendors' numbers
This score is not directly comparable to the headline numbers Mem0, Zep or ByteRover publish: the answer and judge models, the question subset and reranking all differ. Vendors in this category say themselves that published numbers are not comparable.
-
Multi-session reasoning is the weakest area
2/6 (33.3%). Questions that need facts joined across several conversations are where ContextQ does worst today, and it is the area to improve first.
How the run worked
| Parameter | Value |
|---|---|
| Dataset | longmemeval_s_cleaned.json (MIT), the s variant. Each question carries its own haystack of about 48 sessions, most of them distractors. |
| Questions | 30 of 500, 6 per ability; 1,438 haystack sessions in total. |
| Ingest | Every session goes through ContextQ's real memory-extraction pipeline (POST /api/memory), with the session date passed along. |
| Retrieval | The real search pipeline (POST /api/search): PostgreSQL full-text, Elasticsearch and vector search, merged with RRF fusion. k=8. |
| Reranking | Off for the whole run, so all 30 questions were measured the same way. |
| Answer step | Gets the question's date and the date of the conversation each retrieved memory came from. |
| Models | claude-haiku-5-5 for extraction, answer and judge. |
| Grading | An LLM judge compares each answer with the gold answer. Abstention questions count as correct only when the answer also declines. |
| Run date | 2026-10-13 |
Artifacts
The dataset is LongMemEval by Wu et al. The per-question results are kept as eval/lme-s-30q.json and eval/lme-s-30q.md in the ContextQ repository, which is private; ask us if you want them.