Just-in-Time Memory for LLM Agents: The 16-Point Case

Just-in-time memory gave LLM agents a 16.2-point lift on ALFWorld.

And the system behind it, JitMem, earned that number by waiting. Just-in-time memory means an agent stores raw trajectories and defers all memory curation until read time, when the current task is actually known, instead of compressing every finished run into a summary on the spot. In results reported by AI Weekly, JitMem beat the strongest baseline by 16.2, 16.3. And 3.9 absolute success-rate points on ALFWorld, WebShop. And τ²-bench, and it outperformed no-memory agents along with both heuristic and learned write-time memory methods. The whole argument compresses to one line: stop deciding what matters before you know the question.

Write-Time Curation Guesses About a Future It Can't See

The authors' critique of standard memory pipelines is blunt. Compressing a finished trajectory into a reflection, workflow, or skill forces the system "to decide what is worth remembering before the future query is known, irreversibly discarding information and producing a query-independent summary that must serve many possible downstream tasks."

Every clause there is a problem description. "Before the future query is known" means the memory layer is writing for tasks that don't exist yet. "Irreversibly discarding" means whatever the summarizer drops is gone, not archived. "Query-independent" means one generic summary has to serve every possible future use.

So it serves each of them thinly.

I hit the identical failure with meeting notes.

If I compress a client call into three takeaways, I can answer the question the notes were written for and nothing else. Six weeks later someone asks what the client said about edge cases in their CRM sync. And the answer is not in the bullets. Raw transcripts are boring to store and unbeatable to query. A write-time summarizer optimizes for the average future question, and the average question never gets asked.

JitMem's Move: Store Raw, Curate Only When a Task Shows Up

JitMem flips the pipeline order. Per the coverage, it "keeps raw traces and only synthesizes a compact payload once a new task shows up." Nothing gets distilled until a real task arrives. And then curation happens with the task in hand.

That single reordering changes what memory is for. Write-time memory is a filing system that bets on categories in advance. Read-time memory is a research assistant that holds the full archive and pulls exactly what this task needs. Memory selection becomes dependent on the current task instead of dependent on a guess made weeks earlier.

The benchmark spread tells you where the payoff lands. The reported gains are 16.2 points on ALFWorld, 16.3 on WebShop, and 3.9 on τ²-bench. Two of three benchmarks saw large jumps and the third a modest one, all against the strongest available baseline. I would treat 3.9 as the honest floor for what task-aware retrieval buys you, not as an asterisk on the headline.

The Untrained Curator Is the Real Headline

The result I would tattoo on the wall of any small shop building agents is this one. The source notes that "even an untrained curator is already competitive with or surpasses these baselines, showing that task-adaptive read-time curation itself is a major source of the gain; training the curator further compounds the improvement."

Read that as a build decision. You can skip the expensive part, training a bespoke memory policy. And still match or beat write-time systems that have one. The architecture does most of the work. Training compounds on top of it. But the structural switch itself is where the major share of the gain lives.

The industry reflex runs the other way. Every memory vendor pitches a smarter learned layer, and the implied roadmap is more training, more tuning, more lock-in. JitMem's result suggests the boring choice, delaying curation until the task is known, beats cleverness applied at the wrong moment. When something works untrained, a solo operator can actually run it without a research budget.

Where This Fits in the Memory Stack You Already Have

Production agents have mostly settled into a familiar shape. Per a memory design guide, working memory is the context window plus whatever structured scratchpad the agent keeps in front of itself, while long-term records live in external stores such as extracted fact records, knowledge bases, and retrieval indexes. A survey of agent memory mechanisms sorts the field into five families: context-resident compression, retrieval-augmented stores, reflective self-improvement, hierarchical virtual context, and policy-learned management.

Most write-time memory products sit in the reflective self-improvement family: the agent reflects after a task and writes lessons forward. JitMem reads as an argument that this family compresses too early.

Keep the archive raw, keep it external.

And spend your intelligence budget at retrieval, where the task is already sitting in the prompt.

If you run agents for clients, three moves fall out of this research:

- Stop over-summarizing at write time. Keep raw transcripts, tool calls, and outputs in cheap storage before you distill anything. Storage is the cheap half of the trade. - Make retrieval task-aware. Whatever fetches memories should see the current task, not just a similarity score against a stale summary. - Test the dumb version first. Before paying for a trained memory policy, run an untrained read-time curator against your current setup. The reported results say the dumb version already competes.

Two honest caveats.

Every number here comes from a single writeup of the paper. So treat them as reported results rather than settled fact. And read-time curation moves work to query time. Because synthesizing a payload per task costs more per run than reusing a pre-digested summary. Whether that trade clears your latency budget is something the coverage does not quantify. So test it on your own workload before committing.

The takeaway for a small operation is still clean. Memory systems that summarize at write time are guessing; systems that curate at read time are answering.

Guess less, archive raw, and let the query do the work.

If you want a second pair of eyes on your agent's memory pipeline, that is exactly the kind of build my agency does.

So reach out and we will audit what your agents keep and what they throw away.