Teaching a Local Model to Review Code: Building the Evaluation Foundation

Training wheels before training

In the preamble, I laid out the hypothesis for this series: given enough training data from my own monorepo and work processes, could I use LoRA or QLoRA to make a small local LLM perform better than its baseline? At minimum, could it become a more useful adversarial reviewer by learning my preferred guidelines and accumulated lessons?

I spent the first several weeks without fine-tuning anything.

That was necessary because I first needed a stable definition of "better than baseline." Without one, any result after training would be open to interpretation. Many fine-tuning tutorials begin with the training script, but the useful work starts earlier: building an evidence corpus and an evaluation harness that can turn "did it improve?" into a measurable question.

This post covers how I built that foundation.

Turning workflow artifacts into a body of evidence

The raw material came from several months of sprint records, review notes, post-mortems, and workflow findings generated by my own tooling. There was plenty of text, but only a small portion of it was immediately suitable for training or evaluation.

The first structured pass produced 45 usable records. I converted each one into what I called a claim record: a structured row containing both the original text and the metadata needed to use it safely later.

Each record captured:

• The artifact being discussed, anchored precisely enough to be replayed.
• The expected verification, along with what was actually observed.
• The asserted outcome and the method used to check it.
• Whether the record could be used for training, including the role it was allowed to play.

The final field was especially important. When the baseline model failed a check, I kept the record but classified it as a negative control. These were examples that the tuned model should not reproduce. Treating baseline failures as good training examples would have taught the model to repeat the mistakes I was trying to remove.

Existing text is not automatically trustworthy

The corpus eventually grew from 45 records to several hundred. During that expansion, I evaluated externally sourced review text such as comments, feedback, and discussion for possible inclusion.

The acceptance rate was low. Of 262 candidate rows, only 12 were accepted. The rest were rejected or quarantined.

The main problem was not necessarily that the claims were false. Most could not be verified. A comment such as "this breaks on mobile" is only useful as training evidence when it can be tied to a specific artifact and independently confirmed. Without that connection, the model would be learning an unverified opinion. Once encoded into the model's weights, that opinion becomes difficult to identify and remove.

The final corpus contained roughly 560 records:

• 225 training records
• 31 development records
• 30 held-out test records
• 274 excluded or quarantined records

The excluded records remain part of the corpus accounting. They provide a provenance trail showing what the model was considered for, what it was taught, and what was deliberately withheld.

Choosing the baseline through a bake-off

The answer to "Which local model should I use?" changes frequently. Rather than choosing one from general benchmarks or current recommendations, I tested several candidates against fixtures based on actual review scenarios from my work.

The fixture set included a critical subset covering the failures I cared most about:

• Making claims without evidence
• Asserting authority the model did not have
• Approving changes that should have been blocked

Qwen3-8B, using the Q4_K_M quantization introduced in the preamble, passed all 15 fixtures, including every critical case. A Ministral 8B variant passed 12 of 15 and failed some of the critical fixtures.

That result established Qwen3-8B as the default baseline for this experiment.

This does not establish that Qwen3-8B is generally better than Ministral. It only shows that it performed better on the bounded review task I intended to test. For that decision, a small set of representative fixtures from my own work was more useful than a large public benchmark covering unrelated tasks.

Three possible sources of improvement

The evaluation design separated three factors that are often combined in reports of local-model improvement:

• Raw model capability: what the base model can do with a plain prompt.
• Prompt and harness lift: what the same model can do with a structured prompt pack and evaluation harness.
• Corpus and data quality: whether the measured change is genuine or caused by mislabeled, inconsistent, or unverifiable data.

These factors lead to different decisions.

When most of the improvement comes from the harness, the practical result may be to ship the harness and avoid training. When the corpus is unreliable, the output of a training run cannot be trusted. Fine-tuning should only receive credit for an improvement after prompt effects and data-quality problems have been measured separately.

The harness therefore needed to run the same fixtures and scoring process while allowing the model and prompt configuration to be swapped independently. Building that system required more work than an individual training run, but it made the later comparisons interpretable.

What a readiness flag actually means

Late in the corpus work, the pipeline reported that it was ready for a serious LoRA experiment.

That flag referred only to corpus integrity. The rows were checksummed, anchors had been validated, the dataset splits were clean, and excluded records were accounted for. It did not predict whether fine-tuning would improve the model.

"Ready to train" and "training is likely to work" are separate claims. The first describes the condition of the pipeline. The second is an experimental hypothesis.

I kept that distinction explicit in the project artifacts. It became important later, when the results did not match what I had hoped to see.

What this phase produced

By the end of this phase, I had:

• A corpus with provenance tracking and explicit exclusions
• Frozen training, development, and test splits
• A fixture harness covering critical failure modes
• A measured default baseline
• A scoring process that could distinguish model changes from prompt effects and data problems

I had not trained the model, and its capabilities had not changed. The project had nevertheless gained the measurement system required to evaluate everything that followed.

Part 2 — Owned compute as insurance covers the strategy and economics of the local lane. Part 3 — The results covers the final scorecard. Both depend on the work described here.

Whatever the outcome of a fine-tuning run, this evaluation foundation is what makes the result honest.