When the Fine-Tune Loses: An Honest Ending
Adapting a Local Model — Part 3
The finale of this series is a negative result: the QLoRA adapter performed worse than its unmodified base model on the task it was trained for. What the controls caught, what the adapter actually learned, and why the expensive part was producing a conclusion I could trust.
The result
The preamble stated the hypothesis this series was designed to test: given enough training data from my monorepo and work processes, could LoRA or QLoRA make a small local LLM perform better than its baseline? At minimum, could it become a more useful adversarial reviewer?
On the path I tested, the answer was no. While this was disappointing, I believe it is important to show the failures as well as the wins. Many times, ideas that fail can provide valuable learnings.
The experiment used QLoRA adaptation of Qwen3-8B, my own corpus, and a fixed training recipe. In the final controlled comparison, the adapted model performed worse than the unmodified base model on the bounded task it had been trained to handle.
This post describes how that result emerged, what the controls detected, and what the adapter appears to have learned.
The training run looked successful
The training pipeline from Part 1 — Building the evaluation spine supplied a checksum-frozen dataset to a QLoRA run on a high-end consumer GPU. The run completed in minutes. Based on integrated power readings, the direct compute cost of the successful training pass was approximately one cent.
The loss curves declined, and the held-out metrics for the training-shaped task appeared strong.
On the frozen held-out split, the adapter improved from 0 of 22 strict passes for the base model to 17 of 22. It also produced the expected source citations in every passing case.
Those results showed that the adapter had learned the response format represented in the training data. They did not establish that it had become better at the broader review task.
The behavioral evaluation was a different story
The evaluation system was designed to distinguish learning the training format from improving at the intended job.
The full scorecard compared five matched configurations using fixed prompts, deterministic decoding, a fixed seed, and one pass per configuration. It included the base and adapted models, both with and without the structured review harness.
On the behavioral fixture set, the adapted model performed substantially worse:
Configuration Fixture pass Critical-fixture pass
Base + review harness 11/15 7/11
Adapter + review harness 4/15 2/11
The behavioral fixtures represented the failures that mattered most for an adversarial reviewer. These included unsupported claims, assertions of authority the model did not possess, and approval of changes that should have remained blocked.
The adapter applied its learned citation and disclaimer format to cases where that format was inappropriate. Instead of adapting its response to the fixture, it often reproduced the same template.
The safety evaluation also found fabricated reference identifiers, invented status fields, and an unsupported "ready for merge" decision. Those failures disqualified the adapter from the intended role. A reviewer that invents supporting evidence can make an unsafe decision appear justified.
An encouraging result that did not count
During this phase, an early specialist experiment produced 11 of 11 outputs containing the intended content.
The result looked promising, but the experiment was invalidated before the baseline comparison ran. Without the matched baseline, there was no way to determine whether the adapter had caused the apparent improvement.
I recorded the output as unadjudicated evidence rather than treating it as a successful result. That distinction prevented an incomplete comparison from influencing the final conclusion.
The final controlled comparison
The project ended with a final controlled experiment using a frozen 30-case corpus. The corpus covered six artifact families with five edge variants for each family.
Before running either model, I fixed:
- The evaluation corpus
- The execution order
- One pass per arm
- No retries or output repairs
- The quality gates
- The required improvement threshold
For the adapter to pass, it needed to outperform the base model by at least 20 percentage points and six complete cases. The plan was hashed before execution so that the criteria could not be revised after the results were known.
The final result was:
Arm Strict contract compliance
Unmodified base model 30/30
Adapted model 10/30
The task required the model to return one bare JSON object.
The base model complied in all 30 cases. The adapter wrapped its response in a Markdown code fence in 20 cases. When that fence was removed, the content inside was correct in all 30 cases.
The adapter had preserved the required information but failed the output contract because it reproduced a formatting behavior learned during training. In this task, format compliance was the primary requirement rather than a cosmetic detail.
The adapted model also required:
- 1.77 times the wall-clock time
- 1.23 times the GPU energy
- 1.66 times the cost per artifact
It therefore delivered lower strict quality at a higher operating cost. Under the tested conditions, increasing usage volume would not produce a break-even point because the adapted model was worse on both dimensions.
The experiment ended at that gate. I did not run additional training attempts or change the base model, corpus, or success criteria after seeing the result.
What the adapter learned
The adapter learned a response pattern more reliably than it learned the task.
Training reinforced several surface behaviors:
- Adding citations
- Including boundary disclaimers
- Wrapping output in Markdown fences
- Following a repeated response structure
Those behaviors improved performance on examples that matched the training distribution. Outside that narrow distribution, the same pattern interfered with task-specific decisions.
The adapter sometimes applied the learned format even when the fixture required different reasoning or a stricter output contract. In the worst cases, it generated evidence-shaped content without real evidence behind it.
The result was not an absence of learning. The adapter clearly changed the model's behavior. The problem was that the change did not improve the intended job and degraded several important behaviors that the base model already handled better.
The cost of reaching a reliable conclusion
As discussed in Part 2 — Owned compute as insurance, the compute used for the training run was negligible. Corpus construction and evaluation were not.
Most of the project cost came from:
- Collecting and validating the evidence corpus
- Maintaining provenance and exclusions
- Building the evaluation harness
- Freezing the datasets and fixtures
- Defining the gates before execution
- Running matched comparisons
- Investigating failures without changing the criteria
The project spent very little on teaching the model. It spent far more determining whether the resulting model should be used.
GPU cost alone is therefore a poor estimate of the cost of a fine-tuning project. Training can be inexpensive while the work required to produce a trustworthy decision remains substantial.
What the result establishes
This experiment rejects a specific hypothesis.
Consumer-hardware QLoRA applied to Qwen3-8B, using this corpus and training recipe, did not produce a better adversarial reviewer or a useful specialist for the tested task.
The result does not establish that LoRA is ineffective in general. It does not cover other base models, corpora, training recipes, task types, or evaluation criteria. A future experiment would need to begin as a separate hypothesis with its own controls and success gates rather than as an attempt to recover this result.
The experiment also strengthened the case for the unmodified baseline. Across the series, the best-performing configuration was the base model combined with the structured review harness.
That suggests a practical order of operations:
- Establish the unmodified baseline.
- Improve the prompt and harness.
- Measure whether those changes already meet the task requirements.
- Fine-tune only when the remaining gap is explicit and measurable.
In this experiment, the simplest configuration was also the most accurate and least expensive.
Why I am publishing the negative result
Published fine-tuning results tend to overrepresent successful experiments. That makes it harder to estimate failure rates, recognize common regressions, or understand the amount of evaluation work required.
This project did not find an improvement, but the evaluation system identified a specific adaptation-induced regression. It showed where the model changed, where those changes helped, and where they damaged behavior the baseline already performed correctly.
The original hypothesis failed. The corpus, harness, controls, and evaluation process remain useful.
The local system is still running, but it is running the unmodified base model.
Final Thoughts
This post is not meant to discourage investigating this line of thinking. I am but a single engineer and amateur researcher. This experiment was tailored to my own data and requirements. I am merely posting the negative results for others to read. I however, will not be returning to my original hypothesis and will pursue other avenues of investigation.