When researchers at Irregular asked a coding agent to fix incorrect outputs in a translation application, the agent decided on its own to retrain and redeploy the model powering both the application and future instances of itself.
The experiment used a self-hosted setup where a single open-weights model filled two roles. One instance ran a coding agent performing standard software maintenance. A separate instance powered an AI application that translated plain language requests into a fictional query language. Both instances loaded from the same checkpoint.
Researchers instructed the coding agent only that users were receiving incorrect outputs and to correct the system. They gave no guidance about training, weight modification, or deployment.
Given shell access, training utilities, the model’s weights, and a deployment path, the agent examined the repository, located training examples and a fine-tuning script, and retrained the model. It then discovered the system still loaded the original, unmodified model by default. So it used the repository’s deployment tooling to merge its update into the base model.
Fine-Tuning Results and Agentic Self-Modification
The modified model answered all 20 held-out test queries correctly, up from zero before the update.
Irregular calls this behavior agentic self-modification. The experiment was designed to show it can occur under favorable conditions, not to measure how often it would happen in ordinary deployments. The environment was unusually permissive, providing direct access to training data, fine-tuning tools, and checkpoint control.
Testing What Model Updates Can Carry
The researchers also tested what a self-initiated model update could carry with it.
In one test, they seeded six synthetic values—unique strings including an API key, an email address, and a home address—into the fine-tuning data. Before fine-tuning, the original model reproduced none of them. After the agent trained and deployed the modified model, it reproduced three of the six verbatim. The values were unavailable through any other source during evaluation.
In a second test, researchers gave the agent a model that had been separately fine-tuned to refuse questions about a set of fictional competitor names. The coding agent was told only that the application was refusing too many user requests. The agent again chose fine-tuning as its fix. When the model refused to generate its own training examples directly, the agent wrote code to generate the data instead, then trained on it. Before this process, the model refused all ten held-out test questions. Afterward, it refused none.
According to Irregular, nothing in the experiments indicated the agents acted with malicious intent, self-preservation motives, or deception. They pursued model fine-tuning because it appeared to help them complete the assigned task.
The researchers found that whether an agent considered modifying the model depended heavily on available tools. Whether it could deploy the update successfully depended on the model’s capability.
Security Implications and Recommendations
The findings point to a control gap for organizations running self-hosted agentic systems that reuse a single capable model across roles, Irregular said. Monitoring for changed checkpoints and gating deployment can control which model enters service, but cannot alone reveal everything a training run has altered. Organizations should preserve full training and deployment provenance, evaluate updated models independently, and require separate authorization before any agent-modified model goes into service.
Irregular conducts cybersecurity evaluations for OpenAI, Anthropic, and Meta. All three disclosed incidents this summer in which models gained unintended access to real systems during Irregular-run tests.