Adapting a Local Model: Reasoning, Hypothesis, and a LoRA/QLoRA Primer

Why this post exists

After working with frontier models for several months and gathering hundreds of metrics a question started to form in my mind. That question was: "Given enough training data from my own monorepo and work processes could I augment an existing small local LLM via LoRA/QLoRA to allow it to perform better than baseline? If not, to at least act as a better adversarial reviewer based on my preferred guidelines and learnings." That hypothesis forms the foundation of this blog post series and my journey towards answering whether this question is true.

Part A — The reasoning and the hypothesis

Why adapt a local model at all

The motivation for this question arose out of the fact that frontier models are outside of consumer control and in the hands of ultra wealthy corporations with the means to pull access or restrict legitimate cybersecurity and other knowledge pursuits without input from the wider public. See Project Glasswing (https://www.anthropic.com/glasswing) and US Government Intervention (https://www.anthropic.com/news/fable-mythos-access) selectively restricting access, with control resting in the hands of the providers and their gatekeepers rather than the wider public. This forces uncomfortable questions around the democratization of access to frontier AI knowledge and safeguards against free access to knowledge rather than usage. This post is however, not a political or policy debate and I will not be going in-depth into my own beliefs on this matter. I will however, attempt to lay the foundations for distilling my 6 months of agentic research into an easy to digest series that might encourage almost everyone to give local models a shot. The good news is that on mid-level consumer hardware this testing can be carried out. The actual training will require beefier hardware but, still in the higher end gaming rig territory. Cloud training options and OpenRouter access will also be discussed. It should be noted that the specific methods I have used to acquire data are from my own workflows and from researching arXiv papers.

The hypothesis

Hypothesis

Given enough training data from my own monorepo and work processes could I augment an existing small local LLM via LoRA/QLoRA to allow it to perform better than baseline? If not, to at least act as a better adversarial reviewer based on my preferred guidelines and learnings.

What success and failure each look like

Success: A small local model IS able to punch above its weight and perform adversarial review and mid-level agentic tasks (not orchestration of other agents) satisfactorily with minimum hallucinations and false positives.

Failure: A small local model performs worse or becomes downright unusable. However, the failure itself can still be a valuable learning experience.

Part B — What LoRA and QLoRA actually are

Starting from a base model

It should be noted that I will be starting with Qwen3-8B-Q4_K_M.gguf. I DID NOT train a full model from scratch but will be applying LoRA/QLoRA techniques to augment the existing model weights with my own corpus from months of sprints, the tickets and post-mortems they generated, and my own workflow learnings. Also, the only time a frontier model will be used will be to set up the actual training, verification of results, and validation of the methodology. The idea, training data and hypothesis are my own.

LoRA, in plain language

There is a good article on Medium (https://medium.com/@dsh.2065/fine-tuning-llms-with-lora-and-qlora-from-confusion-to-kinda-working-results-89b348bcce71) that explains both LoRA and QLoRA and what they are a bit better. But in a nutshell they inject low-rank trainable matrices into specific layers of the target model without needing heavy hardware to fully retrain a model.

QLoRA — the same idea on modest hardware

This is the same idea but using quantization (smaller bases) so less powerful hardware can perform the low-rank adaptation.

Honest magnitude

For QLoRA I wouldn't go below 4-bits and a maximum of 8-bits. 8-bits (https://arxiv.org/abs/2305.14314) might be the sweet spot to gain a 50% smaller bit size without a substantial loss of quality. You can be more aggressive if your hardware demands require it, just be mindful that smaller means a quantifiable loss of quality.

What LoRA is NOT

LoRA is not a way to train a brand new model from scratch. It is also not a way to magically make a small local LLM outclass a frontier model. It is not going to happen. It's like thinking that because you built a model plane you are ready for building a 737; you don't have the money and datacenter for it.

Where the series goes from here

This preamble is the reference the rest of the series links back to. From here:

Part 1 — Building the evaluation spine. Assembling the review corpus and a baseline harness before any fine-tuning, so "did it get better?" is a measurable question rather than a vibe.

Part 2 — Owned compute as insurance. The strategic case for running a capable model on hardware you own, and the bar a local model actually has to clear to be worth it.

Part 3 — The results. The LoRA/QLoRA training run and a reproducible baseline-vs-tuned scorecard reporting the numbers honestly, whichever way the hypothesis falls.