Fine-Tuning Language Models: A Practical Guide from Dataset to Deployment

The Prompt Engineering Ceiling

You've engineered the perfect prompt. You've added few-shot examples, chain-of-thought reasoning, output format instructions. Your LLM produces beautiful JSON — 80% of the time. The other 20%, it wraps the JSON in markdown code fences, hallucinates extra fields, or drops the priority key entirely. You tighten the prompt. Now it works 90% of the time. You add more examples. 93%. You've hit the ceiling.

Meanwhile, each request burns through 2,000 tokens of prompt boilerplate that never changes. At 10,000 requests per month, those few-shot examples alone cost more than your fine-tuning budget for the entire year.

Fine-tuning is the escape hatch. Instead of telling the model what to do every time, you teach it once and let it remember. The result: consistent output formatting, shorter prompts, lower latency, and dramatically lower cost per request. In our LoRA post, we built the theory from scratch — low-rank matrices, parameter-efficient updates, why you only need to train 0.1% of the weights. This post is the practical bridge: we'll fine-tune a real model on a real task and measure exactly what we gain.

But first, the honest disclaimer: most tasks don't need fine-tuning. Prompt engineering handles roughly 70% of use cases. RAG handles another 20%. Fine-tuning is for the remaining 10% where you need rock-solid consistency, domain-specific behavior, or cost optimization at scale. Let's figure out which camp you're in.

When to Fine-Tune

Before you spend a single dollar on training data, run your task through this decision tree:

The tasks that benefit most from fine-tuning share common traits: structured extraction (turn messy text into consistent JSON), classification (route tickets, categorize content), tone and style (match a brand voice), and distillation (teach a cheap model to mimic an expensive one). In production, many teams combine all three approaches: fine-tune for format consistency, layer RAG for factual grounding, and use prompt engineering for task-specific instructions.

Here's the uncomfortable truth about fine-tuning: your model will never be better than your training data. Two hundred carefully curated examples will outperform ten thousand sloppy ones. Every example must be correct, consistent, and contain all the information the model needs — no expecting it to fill in gaps from "general knowledge."

We'll work through a concrete task: extracting structured data from customer support tickets. The input is a messy, human-written ticket. The output is a clean JSON object with standardized fields. This is exactly the kind of task where fine-tuning shines — the base model can do it, but inconsistently.

Training data uses the JSONL conversation format. Each line is a complete conversation with a system prompt, a user message (the ticket), and the assistant's response (the structured JSON):

{"messages": [
  {"role": "system", "content": "Extract structured data from support tickets. Return valid JSON with fields: category, priority, platform, feature, sentiment."},
  {"role": "user", "content": "Subject: App crashes on upload\nBody: Every time I try to upload a file larger than 10MB the iOS app crashes. iPhone 14 Pro, iOS 17.2. Been happening since the last update."},
  {"role": "assistant", "content": "{\"category\": \"bug\", \"priority\": \"high\", \"platform\": \"ios\", \"feature\": \"file_upload\", \"sentiment\": \"frustrated\"}"}
]}
{"messages": [
  {"role": "system", "content": "Extract structured data from support tickets. Return valid JSON with fields: category, priority, platform, feature, sentiment."},
  {"role": "user", "content": "Subject: Can't find dark mode\nBody: Is there a dark mode? I looked in settings but couldn't find it. Using the Android app."},
  {"role": "assistant", "content": "{\"category\": \"feature_request\", \"priority\": \"low\", \"platform\": \"android\", \"feature\": \"dark_mode\", \"sentiment\": \"neutral\"}"}
]}
{"messages": [
  {"role": "system", "content": "Extract structured data from support tickets. Return valid JSON with fields: category, priority, platform, feature, sentiment."},
  {"role": "user", "content": "Subject: Charged twice\nBody: I see two charges of $29.99 on my credit card statement for December. Order #98765. Please refund the duplicate."},
  {"role": "assistant", "content": "{\"category\": \"billing\", \"priority\": \"high\", \"platform\": \"unknown\", \"feature\": \"payments\", \"sentiment\": \"upset\"}"}
]}

A few rules that will save you hours of debugging:

• Keep the system prompt identical across all examples — it becomes the model's "job description"
• Use the exact same JSON schema in every assistant response — same fields, same casing, same value types
• Include edge cases: tickets with missing information (output "unknown"), ambiguous categories, multi-issue tickets
• No contradictions: if similar tickets get different labels, the model learns noise instead of signal

Before uploading, validate your data. This script catches the most common mistakes:

import json

def validate_training_data(filepath):
    """Validate JSONL training data before uploading."""
    errors, examples = [], []
    required_fields = {"category", "priority", "platform", "feature", "sentiment"}

    with open(filepath) as f:
        for i, line in enumerate(f, 1):
            try:
                obj = json.loads(line)
            except json.JSONDecodeError:
                errors.append(f"Line {i}: Invalid JSON")
                continue

            msgs = obj.get("messages", [])
            roles = [m["role"] for m in msgs]
            if roles[0] != "system" or "user" not in roles or "assistant" not in roles:
                errors.append(f"Line {i}: Missing system/user/assistant role")
                continue

            # Check assistant output is valid JSON with required fields
            assistant_msg = [m for m in msgs if m["role"] == "assistant"][0]
            try:
                output = json.loads(assistant_msg["content"])
                missing = required_fields - set(output.keys())
                if missing:
                    errors.append(f"Line {i}: Missing fields: {missing}")
            except json.JSONDecodeError:
                errors.append(f"Line {i}: Assistant response is not valid JSON")

            examples.append(obj)

    print(f"Total examples: {len(examples)}")
    print(f"Errors found: {len(errors)}")
    for e in errors[:10]:
        print(f"  {e}")
    return len(errors) == 0

validate_training_data("training_data.jsonl")

How many examples do you need? OpenAI recommends starting with 50-100 for simple tasks. The sweet spot for most extraction and classification work is 200-500. You'll see "a similar amount of improvement every time you double the number of training examples" — so 100 → 200 gives roughly the same lift as 200 → 400. Start small, measure, and add data only where the model fails.

Hand-writing 200 training examples is tedious. The bootstrap trick: write 20 diverse seed examples by hand, then use a powerful model (GPT-4o) to generate realistic variations. This is a form of knowledge distillation — you're transferring the large model's capabilities into training data for a smaller, cheaper model.

from openai import OpenAI
import json

client = OpenAI()

# Your 20 hand-written seed examples (abbreviated)
seed_tickets = [
    "Subject: App crashes on upload\nBody: File upload crashes on iOS...",
    "Subject: Billing error\nBody: Charged twice for subscription...",
    "Subject: Slow performance\nBody: Dashboard takes 30s to load...",
    # ... 17 more diverse examples
]

def generate_variations(seed, n=10):
    """Generate n realistic ticket variations from a seed example."""
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{
            "role": "system",
            "content": "Generate realistic customer support tickets. "
                       "Vary the writing style, detail level, and tone. "
                       "Return one ticket per line as: Subject: ...\nBody: ..."
        }, {
            "role": "user",
            "content": f"Generate {n} variations of this ticket:\n{seed}"
        }],
        temperature=0.9
    )
    return response.choices[0].message.content.strip().split("\n\n")

# Generate, then label each variation with GPT-4o
def label_ticket(ticket_text):
    """Use GPT-4o to label a ticket (the 'teacher' model)."""
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "Extract structured data from support tickets. Return ONLY valid JSON with fields: category, priority, platform, feature, sentiment."},
            {"role": "user", "content": ticket_text}
        ],
        response_format={"type": "json_object"}
    )
    return response.choices[0].message.content

# Build the dataset
training_data = []
for seed in seed_tickets:
    variations = generate_variations(seed, n=10)
    for ticket in variations:
        label = label_ticket(ticket)
        training_data.append({
            "messages": [
                {"role": "system", "content": "Extract structured data from support tickets. Return valid JSON with fields: category, priority, platform, feature, sentiment."},
                {"role": "user", "content": ticket},
                {"role": "assistant", "content": label}
            ]
        })

# Save as JSONL — always review a random sample before training!
with open("training_data.jsonl", "w") as f:
    for example in training_data:
        f.write(json.dumps(example) + "\n")
print(f"Generated {len(training_data)} training examples")

Critical step: manually review a random sample. Check 20-30 examples for correctness. If the teacher model (GPT-4o) mislabels tickets, those errors become part of your training signal. Fix or remove bad examples. Then split 80/20 into training and validation sets — never evaluate on data the model trained on.

Option A: OpenAI API (Fastest Path)

The OpenAI fine-tuning API is the simplest way to get started. Upload your JSONL file, create a job, wait 10-20 minutes, and you have a fine-tuned model you can query exactly like the base model. The cost is remarkably low: 200 examples at ~500 tokens each, trained for 4 epochs on GPT-4o-mini, costs roughly $1.20.

from openai import OpenAI
import time

client = OpenAI()

# Step 1: Upload training file
training_file = client.files.create(
    file=open("training_data.jsonl", "rb"),
    purpose="fine-tune"
)
print(f"File uploaded: {training_file.id}")

# Step 2: Create fine-tuning job
job = client.fine_tuning.jobs.create(
    training_file=training_file.id,
    model="gpt-4o-mini-2024-07-18",
    hyperparameters={
        "n_epochs": 3,                 # 2-4 for most tasks
        # "learning_rate_multiplier": 1.0  # usually auto is fine
    },
    suffix="ticket-classifier"         # your model gets a readable name
)
print(f"Job created: {job.id}")

# Step 3: Wait for completion
while True:
    status = client.fine_tuning.jobs.retrieve(job.id)
    print(f"Status: {status.status}")
    if status.status in ("succeeded", "failed"):
        break
    time.sleep(60)

# Step 4: Use your fine-tuned model
fine_tuned_model = status.fine_tuned_model
print(f"Model ready: {fine_tuned_model}")

response = client.chat.completions.create(
    model=fine_tuned_model,  # Use just like any other model
    messages=[
        {"role": "system", "content": "Extract structured data from support tickets. Return valid JSON with fields: category, priority, platform, feature, sentiment."},
        {"role": "user", "content": "Subject: Payment failed\nBody: Tried to upgrade to Pro plan but payment keeps getting declined. Using Visa. Tried 3 times."}
    ]
)
print(response.choices[0].message.content)
# {"category": "billing", "priority": "high", "platform": "unknown", "feature": "payments", "sentiment": "frustrated"}

A few things to watch during training: the training loss should decrease steadily. If it drops to near zero, you're likely overfitting — reduce epochs. The validation loss (if you uploaded a validation file) should decrease and then plateau. If it starts rising while training loss keeps dropping, stop — you've memorized the training data instead of learning the pattern.

Option B: Open-Source with LoRA (Full Control)

When you need data privacy, cost control at scale, or self-hosting, fine-tune an open-source model with LoRA. The LoRA post explains the math; here's the practical setup with Hugging Face PEFT:

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model
from trl import SFTTrainer, SFTConfig

# Load base model with 4-bit quantization (QLoRA)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    load_in_4bit=True,         # QLoRA: 4-bit quantization
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")

# LoRA configuration — these are the key hyperparameters
lora_config = LoraConfig(
    r=16,                      # rank: 16 is a good default
    lora_alpha=16,             # scaling factor: usually equal to rank
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 13.6M || all params: 8.03B || trainable%: 0.17%

# Training config
training_config = SFTConfig(
    output_dir="./ticket-classifier-lora",
    num_train_epochs=2,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,  # effective batch size = 16
    learning_rate=2e-4,             # the most critical hyperparameter
    warmup_steps=10,
    logging_steps=10,
    save_strategy="epoch",
)

# Train
trainer = SFTTrainer(
    model=model,
    args=training_config,
    train_dataset=train_dataset,     # your formatted dataset
    tokenizer=tokenizer,
)
trainer.train()

The single most important hyperparameter is learning rate. A February 2025 paper ("Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning") showed that with properly tuned learning rate, plain LoRA matches far more complex fine-tuning methods. Start with 2e-4 for supervised fine-tuning. If the model overfits, lower it.

Never trust training loss alone. The real question: does the fine-tuned model actually beat the base model on held-out examples you never trained on? Here's how to measure that properly (connecting to our evaluation post):

import json
from openai import OpenAI

client = OpenAI()

def evaluate_model(model_name, test_data, use_few_shot=False):
    """Run a model on test data and measure accuracy."""
    correct_schema, correct_fields, total = 0, 0, 0
    required_fields = {"category", "priority", "platform", "feature", "sentiment"}

    few_shot_prefix = []
    if use_few_shot:
        # Base model needs examples in the prompt; fine-tuned model doesn't
        few_shot_prefix = [
            {"role": "user", "content": "Subject: Login broken\nBody: Can't login on Chrome."},
            {"role": "assistant", "content": '{"category":"bug","priority":"high","platform":"web","feature":"auth","sentiment":"frustrated"}'},
        ]

    for example in test_data:
        user_msg = [m for m in example["messages"] if m["role"] == "user"][0]
        expected = json.loads([m for m in example["messages"] if m["role"] == "assistant"][0]["content"])

        response = client.chat.completions.create(
            model=model_name,
            messages=[
                {"role": "system", "content": "Extract structured data from support tickets. Return valid JSON with fields: category, priority, platform, feature, sentiment."},
                *few_shot_prefix,
                user_msg
            ]
        )

        try:
            output = json.loads(response.choices[0].message.content)
            if set(output.keys()) >= required_fields:
                correct_schema += 1
            matching = sum(1 for k in required_fields if output.get(k) == expected.get(k))
            correct_fields += matching / len(required_fields)
        except (json.JSONDecodeError, KeyError):
            pass  # Failed to produce valid JSON at all

        total += 1

    return {
        "schema_match": f"{correct_schema/total:.0%}",
        "field_accuracy": f"{correct_fields/total:.0%}",
        "total": total
    }

# Load held-out test set
with open("test_data.jsonl") as f:
    test_data = [json.loads(line) for line in f]

base_results = evaluate_model("gpt-4o-mini", test_data, use_few_shot=True)
ft_results = evaluate_model("ft:gpt-4o-mini:ticket-classifier:abc123", test_data)

print("Base model (few-shot):", base_results)
print("Fine-tuned model:    ", ft_results)

Here's what you'll typically see on a well-prepared extraction task:

The accuracy improvement is meaningful, but the cost reduction is dramatic. The fine-tuned model doesn't need few-shot examples in every prompt, which slashes input token count by 15x. At 10,000 requests per month, that's the difference between $10 and $0.80.

The Cost Equation

Pitfalls and How to Avoid Them

Overfitting. Your training loss hits 0.01 and you celebrate. Then you test on new data and accuracy is worse than the base model. Classic overfit: the model memorized your 200 examples instead of learning the pattern. Fix: use only 2-3 epochs, hold out 20% for validation, and stop when validation loss plateaus. With LoRA, you can also increase dropout to 0.05-0.1.

Catastrophic forgetting. Your fine-tuned model classifies tickets perfectly but can no longer hold a normal conversation. It forgot its general capabilities. This is less severe with LoRA (you're only modifying 0.1% of weights) but still happens. Fix: mix in some general instruction-following examples alongside your task-specific data — 10-20% of the dataset should be generic Q&A.

The "it already works" trap. Always benchmark the base model with good prompting before fine-tuning. If GPT-4o-mini with a well-crafted prompt already hits 95% accuracy on your task, fine-tuning to 97% probably isn't worth the ongoing maintenance cost of a custom model. Fine-tune for large leaps, not small gains.

Data contamination. You evaluate on data that looks suspiciously like your training data because it was generated by the same process. Always split before generation: create your validation tickets independently, not as variations of training seeds. Better yet, use real production data for evaluation.

Schema drift. Six months later, you add a new ticket category. Your fine-tuned model has never seen it and confidently misclassifies everything into the old categories. Unlike prompt engineering (where you just update the prompt), fine-tuned behavior requires retraining. Budget for periodic refresh cycles.

Where Fine-Tuning Fits in the Series

• LoRA from Scratch — the mathematical theory behind parameter-efficient fine-tuning. That post builds the low-rank adaptation from pure NumPy; this post applies it to a real task with real tools.
• Evaluating LLM Systems — the eval framework we used to compare base vs. fine-tuned models. Always build the eval harness before training.
• Structured Output from LLMs — four approaches to getting consistent JSON. Fine-tuning is the nuclear option when the other three aren't enough.
• Batch Processing LLMs — generating training data at scale. The bootstrap pipeline in Step 2 uses the same batch patterns.
• RAG from Scratch — the alternative to fine-tuning for knowledge-intensive tasks. The decision tree above helps you choose between them.
• Knowledge Distillation — using GPT-4o to generate training data for GPT-4o-mini is literally distillation: transferring a large model's capabilities into a smaller one through curated examples.

References & Further Reading

• OpenAI — Fine-Tuning Guide — official documentation with the latest supported models and pricing
• Hugging Face — PEFT Documentation — parameter-efficient fine-tuning library for open-source models
• Phil Schmid — How to Fine-Tune Open LLMs in 2025 — comprehensive walkthrough from a Hugging Face engineer
• Unsloth — 2x faster LoRA training with 70% less VRAM; the pragmatic choice for solo developers
• Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning (2025) — key finding that learning rate is the single most important hyperparameter
• Together.ai — Fine-Tuning Updates — cloud fine-tuning for open-source models without managing GPUs

Fine-tuning isn't magic and it isn't always the answer. But when your task hits the prompt engineering ceiling — when you need consistent formatting, domain-specific behavior, or dramatic cost reduction — fine-tuning turns a 90%-reliable system into one you can actually ship to production. Start with 200 examples. Measure everything. Let the numbers tell you whether it was worth it.