Reef, an Apache-2.0 licensed system that converts inference logs into a versioned learning pipeline for both model weights and agent software, has been released by MIT researcher Ao Qu and a distributed group of AI researchers.
The project targets a gap in the standard model lifecycle. That process—train, evaluate, deploy, serve—breaks down when agents are expected to learn from ongoing work. Reef records inference traces and feedback, runs those records through learning jobs, evaluates proposed changes, and publishes accepted model or agent versions without stopping the serving system.
Recording Interactions and Feedback
“Every response gets a receipt,” the project documentation explains. A request reaches Reef through a standard OpenAI-format endpoint. Reef returns the model response along with an x-reef-agent-record-id, a unique identifier applications can later use to attach rewards, evaluator results, or user corrections.
Reef stores requests, execution traces, and feedback as a structured experience stream. Training processors determine which records qualify for learning, while evaluation modules decide whether resulting candidates should go live. Serving continues while learning work runs asynchronously.
The repository divides this cycle into four stages: serve, observe, grow, and commit. The first two collect interactions and match feedback to them. The third runs a configured learning recipe. The final stage evaluates the resulting artifact and decides whether it should replace the version already serving traffic.
That release gate carries much of Reef’s practical value. Continual learning can degrade a working system as easily as it can improve one. According to the project documentation, Reef versions model and harness artifacts, publishes approved candidates as new releases, and leaves the current serving version unchanged when a candidate is rejected.
The Agent Harness as Software
Reef’s broader contribution is its treatment of the agent harness as software that can be updated from experience. This distinction matters because many failures blamed on a model originate in the surrounding application. A tool may receive the wrong arguments. Memory may retrieve irrelevant context. A prompt may invite a plausible answer where the application requires a verified one. Updating weights for every such problem is expensive and often unnecessary.
Reef uses a harness-evolution backend called Cordis to analyze trajectories and propose changes to the surrounding agent code. An accepted harness is distributed as a new installable version.
Qu has been working toward this loop through research on agent memory, planning, and experience-based learning. His MIT profile lists him as a Ph.D. student at the Institute for Data, Systems, and Society, with undergraduate degrees in mathematics and computer science from Vanderbilt University.
Research Contributors
The contributor group spans both sides of this problem. Bo Liu is a University of Washington visiting researcher who previously worked at Meta FAIR on scalable language-model self-improvement and self-play. Han Zheng is an MIT Ph.D. student whose work spans reinforcement learning, combinatorial planning, and self-evolving agents—he also worked on agentic post-training at Amazon AGI. Zijian Zhou is a Meta research scientist who previously worked at ByteDance and Horizon Robotics before completing his doctorate at King’s College London. Simon Yu’s public GitHub materials describe a runtime substrate for reversible, Git-like agent execution traces that support forking and replaying past states. The contributor list also includes Xuan Jiang, Wenhao Chai, Zoe Dusy, and Xinkai Zou.
The group resembles an open research collective more than a conventional startup. Reef is maintained under the Human-Agent-Society GitHub organization. The public materials establish neither a commercial product nor a corporate structure, giving outside developers broad freedom to test the system while leaving its long-term maintenance model open.
The project site identifies Slime and SGLang in Reef’s model-weights path and lists recipes for personalized-chat learning, scientific discovery, prompt evolution, and skill development.
Agent Infrastructure Beyond Tokens
Reef arrives as researchers and infrastructure vendors confront the operational differences between chat completion APIs and long-running agents. Agents carry persistent context, call tools, wait on external systems, and produce trajectories more complicated than a stream of generated tokens. A separate paper on agent-native serving examines the infrastructure requirements of agentic systems. Reef addresses an adjacent problem: preserving experience from live serving so it can support later improvements.
Existing projects cover pieces of that workflow. vLLM and SGLang focus on inference, while Slime, veRL, and AReaL concentrate on reinforcement-learning workloads. Reef’s claim is that live serving, feedback collection, training, harness editing, evaluation, and version delivery belong in one control loop.
The evidence available at launch consists of the code, architecture, and project-run demonstrations. The Hugging Face community article shows personalized-chat and optimization examples, though it does not establish how Reef performs under production traffic, noisy human feedback, or adversarial attempts to influence future versions. Those conditions will determine whether continual learning is useful outside controlled tasks.
Feedback can encode user preferences, factual corrections, and successful tool executions. It can also contain private information, manipulation attempts, or rewards that encourage shortcuts. Reef’s evaluation and versioning controls provide a place to enforce safeguards, but each deployment still needs its own tests, approval policies, and data-handling rules.
Reef does not require an agent to rewrite its entire intelligence stack. It gives developers a server that remembers what happened, proposes bounded changes, and keeps a version history when those changes ship. That is a smaller promise than an autonomous system endlessly rebuilding itself, but it addresses a real gap in how agents improve from experience.