Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Models

The Core Insight

Coding agents need to handle a specific problem that standard language model pretraining barely addresses: integrating tool outputs mid-reasoning. When an agent calls a tool (like running code or querying a database), it gets back a result and must continue from there. The authors noticed this mirrors a universal structure in code itself—function calls. A function receives arguments, returns a computed value, and the caller uses that result downstream. Since this pattern saturates real codebases, the authors propose exploiting it through targeted pretraining.

How Function-Aware Fill-in-the-Middle Works

Standard fill-in-the-middle training masks random code spans and trains models to predict them. The authors' variant is selective: instead of random masking, they use program dependency graphs (which track how data flows through code) to identify functions worth masking, then apply a "complexity-inferability" filter. This double criterion means they target functions that are complex enough to be non-trivial but whose outputs are inferable from context—mirroring the agent scenario where tool outputs are often predictable from what you asked the tool to do.

They mid-trained existing Qwen models (7B, 14B variants, and Qwen3-8B) on 2.6 billion tokens of decontaminated Python code from 968 GitHub repositories. This sits between pretraining and post-training in the pipeline—hence "mid-training."

Why This Matters

The results are consistent and substantial. On SWE-Bench-Verified (a benchmark for software engineering agents), mid-training gave +2.8 to +3.2 point improvements depending on model size. SWE-Bench-Lite saw larger gains: +3.7 to +5.4 points. Importantly, these gains persisted across two different post-training pipelines (R2E-Gym and SWE-Smith), suggesting the effect is robust.

But the deeper payoff emerges elsewhere. Agentic post-training typically degrades performance on non-agent tasks—the model optimizes for tool-use and loses general coding ability. Mid-training mitigated this erosion on LiveCodeBench (general coding), tau-bench and BFCL (tool-use without agents). This happened even though the mid-training corpus was Python-only, suggesting the function-call inductive bias generalizes.

The contribution is methodologically lean: no architectural changes, no new objectives beyond selective masking, and no external data beyond GitHub. It's a mid-training stage that makes agent-style reasoning more natural by pre-exposing models to the structural pattern they'll encounter. For practitioners building coding agents, this is a straightforward win—insert a 2.6B-token training phase and gain 3–5 points on standard benchmarks while protecting non-agent capabilities.