Fine-tuning: adapting a model to your data with LoRA and QLoRA

Fine-tuning is the next step when prompts and RAG stop being enough: you adapt the model's weights to your data, and the model learns your format, your tone, and your rules. No data center required: with LoRA and QLoRA a 7B model can be adapted on a single consumer GPU, and with the right tool you start from a configuration file, not from scratch code.

When you actually need it (and when you don't)

First rule: don't fine-tune for memory. If you want the model to know your documents, the right answer is RAG: add a document index and the model consults it on every answer. Fine-tuning instead changes the model's baseline behavior: output format, language register, answer style, the conventions of your domain.

A good test: if you can explain what you want to the model with a prompt (or an example in the prompt), stop there. A prompt is free, reversible, and takes a minute to change. But if you've been repeating the same instruction in every request for months, fine-tuning saves you time on every conversation.

A third, often underestimated case: reducing context tokens. A model adapted to your output format replies with shorter messages, because it learned the schema once instead of re-reading it in every prompt.

How it works: LoRA and QLoRA in plain words

A 7B model has roughly 7 billion parameters. Retraining all of them on your data costs tens of GB of VRAM and hours of compute. LoRA (Low-Rank Adaptation), the technique described in the 2021 paper, works around this: the original weights stay frozen, and next to each key layer a small extra matrix (the adapter) is inserted — and that is what gets trained. The end result is a file of a few MB, not a model of tens of gigabytes.

The paper's numbers: with a rank-8 matrix the trainable parameter drops to as little as one thousandth of the full model, with no measurable quality loss. And the rank (r) is the main knob: r=8-16 for most cases, higher values only when your data demands it.

QLoRA (2023) pushes further: it quantizes the base model to 4 bits (NF4 format) before training, so a 65B model fits on a single 48 GB GPU. In practice, the line between "adaptable models" and "out of reach models" moved a full hardware generation.

The most important practical advantage: the adapter is a separate file. You can keep three adapters (formal tone, JSON output, short summaries) on the same base model and load whichever you need. Same base weights, three behaviors.

Preparing your data: 90% of the work

The tool matters, the data matters more. Three tips that make the difference:

Format. The most widely used and universally supported format is JSONL: one line per example, with conversation fields. For LLaMA-Factory the alpaca format (prompt, instruction, output) is what you'll use in 90% of cases.

Quantity. You don't need thousands of examples: for a well-defined output format, 100-500 quality examples beat 10,000 generated on the fly. Label quality (the answer you're teaching) matters more than quantity.

Consistency. If you want JSON output, every example must be valid JSON. If you mix three formats, the model learns the average — which is disorder.

A classic mistake: training on data the model can already do. If the model answers well without an adapter, you're spending time and VRAM for zero gain. Run a prompt test first and see where it actually fails.

Local tools for fine-tuning

LLaMA-Factory is the de facto standard for local fine-tuning: one framework for 100+ models (Qwen, Llama, Mistral, Gemma), methods from full SFT to LoRA/QLoRA and DPO, a built-in web interface (WebUI), and startup from a single YAML file. It's the most-used fine-tuning project in the community, with over 75,000 GitHub stars (repo).

A concrete example: to adapt Qwen on an RTX 4090 with QLoRA, the config file contains the base model, the dataset (your JSONL, registered in dataset_info.json), the LoRA method with rank and target layers, and the training parameters. Then one command, and the WebUI shows you the loss and the training curve.

Unsloth is the alternative that bets on speed: optimizations that cut memory usage and raise training speed (the project claims up to 2x speed and half the memory versus standard methods), with support for NVIDIA GPUs and Apple Silicon (repo). If your GPU is at its limit, it's often the shortest path.

MLX is the Apple Silicon route: Apple's machine learning framework for the chip, with the mlx-lm library that supports LLM training and fine-tuning directly on a Mac, with unified memory as an asset (repo). On M3/M4/M5 with 32-64 GB, a LoRA fine-tune of a 7B model is an evening, not a project.

How much hardware do you need?

Rule of thumb for a 7B model:

• QLoRA (4-bit): 8-12 GB VRAM. An RTX 3060 12GB or a Mac with 16-32 GB makes it.
• Full-precision LoRA: 16-24 GB. RTX 4090 or RTX 3090.
• 13-14B model in QLoRA: 16-24 GB.
• 70B model in QLoRA: 48 GB+ (RTX A100 48GB, or a 128+ GB Mac).

Training context weighs more than you'd think: an extra 2048 context tokens nearly doubles memory usage. Start at 1024 if your examples are short.

The practical limits

Be honest about what fine-tuning cannot do:

1. It doesn't add stable factual knowledge. If you teach a fact, the model "learns" it in a fragile way: it says it today, forgets or contradicts it within ten examples. For knowledge, RAG. For behavior, fine-tuning.
2. It forgets part of the base model. The phenomenon is called catastrophic forgetting: the more you train, the further the model drifts from general behavior. "Filler" data (generic examples mixed with yours) mitigates it.
3. You need a metric. Without a test set with correct answers, you can't tell whether you're improving or regressing. Hold out 50-100 examples you never train on, and compare answers before/after.
4. The base model changes, the adapter doesn't. If you move to a new model version, the adapter must be redone. Budget an hour of verification per upgrade.

In summary

• Prompt/RAG first; fine-tune when the repeated behavior doesn't hold.
• 100-500 quality examples, consistent format, careful labels.
• LLaMA-Factory to start, Unsloth when VRAM is tight, MLX on Mac.
• QLoRA is the entry point: 4-bit, an adapter of a few MB, one consumer GPU.
• Always measure before/after, and remember: knowledge goes in RAG, behavior goes in fine-tuning.

References

• LoRA: Low-Rank Adaptation of Large Language Models (arXiv 2106.09685) — the original paper describing the technique.
• QLoRA: Efficient Finetuning of Quantized LLMs (arXiv 2305.14314) — 4-bit quantization + LoRA, fine-tuning 65B on one GPU.
• LLaMA-Factory (GitHub) — unified fine-tuning framework, 100+ models.
• Unsloth (GitHub) — optimized fine-tuning, NVIDIA and Apple Silicon.
• MLX (GitHub) — Apple Silicon framework with mlx-lm for LLM fine-tuning.

Article generated with AI