Fine-Tuning LLMs on User Preferences: Why the Weights Are the Wrong Layer

TL;DR.

• Fine-tuning encodes static, population-level patterns into weights; user preferences are dynamic and individual — that mismatch compounds silently over time.
• RLHF and DPO align models to aggregate human preference distributions, not per-user taste profiles. The two are not the same problem.
• Per-user LoRA adapters work in theory but fail in production: staleness, cold-start blindness, adapter-swap latency, and O(users) GPU memory overhead all compound before you hit 100k users.
• In 2026, the correct split is fine-tune for domain vocabulary and format (stable, shared), inject individual user context at inference time (dynamic, individual).
• If you only remember one thing: user preferences belong in a retrieval layer, not in model weights.

The conversation always starts the same way. An LLM is producing generic output. Users want something that feels tailored to them. Someone on the team says "we should fine-tune on our interaction data." The instinct is understandable — fine-tuning solved domain adaptation, so why not personalization? The answer is in what fine-tuning actually optimizes versus what individual personalization actually requires. They operate on different axes. Conflating them produces systems that are expensive to build, expensive to maintain, go stale within weeks, and are typically outperformed by a well-prompted baseline with a retrieval layer bolted on.

The goal here is a precise architecture breakdown: what fine-tuning does and doesn't do, where the per-user adapter approaches break down, and what the correct stack looks like when you want personalization that holds up past the demo.

What Does Fine-Tuning Actually Optimize?

Fine-tuning adjusts a model's weights to minimize loss on a new training dataset. The model learns statistical patterns from that corpus and encodes them into parameters. Once training is complete, those parameters are frozen until the next training run.

That property — frozen, population-level, static — is what makes fine-tuning powerful for the right problems. Domain adaptation is the canonical case. A base model doesn't know your internal code conventions, your medical terminology, your regulatory vocabulary, your product's specific format requirements. Fine-tuning on examples of those patterns updates the model's priors to reflect them. That knowledge is stable. It applies uniformly to all users of your system. It changes slowly. Fine-tuning is the correct tool.

User preferences satisfy exactly none of these conditions:

Stability. A user's content preferences shift with context, mood, professional role, life stage, and the composition of their feed. A researcher deep in a distributed systems reading phase may pivot to product strategy within a quarter. Their preferences on Monday are not their preferences on Friday if they've had an unusually hard week of reading.

Population applicability. Knowing that one user prefers long-form analytical writing with heavy citations tells you almost nothing about the next user. Individual preferences are not patterns across users — they are patterns within a user, over time. Fine-tuning learns cross-user patterns. Those are different things.

Training signal volume. A 7B-parameter model requires thousands of high-quality examples to shift its behavior measurably on a given task. Most users generate tens to hundreds of preference signals per month, not thousands. The signal-to-noise ratio is too low to train against reliably at the individual level.

Fine-tuning a model on user interaction data doesn't personalize the model to individual users. It adapts the model to the statistical average of your user base — which is almost certainly worse than what you had before, because it smooths over the individual variation you were trying to preserve.

Why RLHF and DPO Don't Solve This

Reinforcement Learning from Human Feedback (Christiano et al., 2017) and its more parameter-efficient successor Direct Preference Optimization (Rafailov et al., 2023) are the dominant paradigms for aligning LLMs to human preferences. InstructGPT (Ouyang et al., 2022) demonstrated that RLHF produces models that follow instructions more reliably, hedge appropriately on uncertain claims, and avoid outputs that human raters penalize. Every frontier model in production has gone through some version of this pipeline.

This is worth understanding precisely, because teams sometimes assume RLHF gives them a personalization-ready base. It does not.

RLHF and DPO both learn from pairwise preference comparisons — given two model outputs, which does a human rater prefer? The reward model and subsequent fine-tuning are optimized against the aggregated preference signal across raters. The resulting model is better aligned to what humans-in-aggregate prefer: clearer explanations, fewer hallucinations, less toxicity, better formatting. These are population-level improvements.

What RLHF gives you is a model that is easier to steer. That's genuinely valuable. But steering is still your job at inference time. The residual gap — between "what the average human prefers" and "what this specific user, on this platform, with their particular history, prefers right now" — is exactly what your personalization layer must close. RLHF narrows that gap from the general-alignment side. It does not close the individual-preference side. That's a retrieval problem, not a training problem.

The Per-User LoRA Argument — and Where It Breaks

The natural rebuttal: train one LoRA adapter per user. LoRA (Hu et al., 2021) learns low-rank weight deltas rather than full parameter updates, cutting per-adapter storage from gigabytes to tens of megabytes. A 7B model might carry 20–50 MB of LoRA weights per user adapter instead of a full 14 GB copy. Run per-user fine-tuning on interaction history, store the adapter, load it at inference time. Personalized output, no retrieval layer needed.

This architecture is technically coherent and fails at production scale for four compounding reasons.

1. Adapter staleness

LoRA adapters encode a snapshot of user preferences at training time. Preferences drift. A user who spent six weeks engaging with ML infrastructure content may pivot to founding-team dynamics after a role change. The adapter doesn't know. You're serving a stale personality model with increasing confidence as the underlying reality moves away from it.

The remediation — periodic retraining — turns into a continuous fine-tuning pipeline per user. You're now running O(users) training jobs on a cadence, with infrastructure complexity that dwarfs the original retrieval alternative you were trying to avoid.

2. Cold start is completely unaddressed

A new user has no interaction history. You have nothing to train a LoRA adapter on. The fallback is a bare base model serving zero personalization — which is exactly the problem you started with. Per-user fine-tuning approaches have no structural answer to cold start. They require data that doesn't exist yet, which means every new user gets a degraded experience until they've generated enough signal to train against. For consumer products, most user churn happens before that threshold.

3. Inference-time adapter loading overhead

Standard serving infrastructure assumes a static model graph. Dynamic per-request adapter loading — fetch the user's LoRA weights from storage, apply the delta to base weights, run the forward pass, flush — adds latency at every hop: storage round-trip, weight composition, batching disruption. Most high-throughput serving frameworks (vLLM, TensorRT-LLM) are built around continuous batching of requests against a shared model. Per-request adapter swaps break that assumption and force either architectural workarounds or serialized request processing.

Research systems like S-LoRA (Sheng et al., 2023) address this specifically, demonstrating that adapter-aware batching can bring per-adapter inference closer to baseline throughput. It's real work that's been done. It also adds a significant systems layer that most product teams don't have the engineering bandwidth to operate.

4. Memory overhead that grows with users

50 MB per adapter sounds manageable. At 100k users with any reasonable hot-adapter cache for p50 latency targets, you're holding multiple terabytes of adapter weights in warm storage with GPU-memory access patterns that don't compress cleanly across diverse users. At 1 million users, this is a storage and cache invalidation problem most teams are not equipped to operate. The math works against you before you hit meaningful scale.

The teams that have shipped per-user LoRA systems in production share a common profile: small user counts (sub-10k), high average revenue per user that justifies the infrastructure cost, and tolerance for personalization that's weeks stale. That describes specialized B2B tooling, not most consumer or mid-market products.

When Fine-Tuning on Preference Data Actually Earns Its Cost

This is not a blanket case against fine-tuning. There are two real scenarios where fine-tuning on preference-adjacent data is the right call.

Domain vocabulary and format adaptation. If your application serves a specific domain — clinical documentation, legal discovery, financial analysis, developer tooling, technical support — fine-tuning on domain-appropriate text improves fluency, reduces hallucination on domain-specific terminology, and aligns the model's default output format to what your users expect. This is stable, shared knowledge. Fine-tune once on a domain corpus, update it on a slow cadence, serve to everyone. The investment is justified because the knowledge is durable and applies uniformly.

Tone and voice calibration. If your product has a consistent voice — precise and terse, warm and conversational, formal and hedged — fine-tuning on examples of that voice is effective. This is a population-level property. It should apply uniformly to all users of your product, not differently per user. Fine-tuning is the correct lever.

The pattern: fine-tune what is shared and stable, retrieve what is individual and dynamic. These are distinct problems with distinct tools. The error is treating them as the same problem because both involve "preferences."

The Right Architecture: Retrieval Over Injection

Individual user preferences belong in an inference-time context layer, not in model weights. The implementation pattern is context injection: build a structured representation of the user's preferences, retrieve the relevant slice at query time, and inject it into the model's context window.

Why this works where fine-tuning doesn't:

It updates in real time. When a user generates a new signal — a skip, a dwell, a save, a search — you update the representation immediately. The next request reflects it. No training cycle, no staleness window, no pipeline to operate.

It handles cold start structurally. With a synthetic user clone seeded from contextual priors — device, geography, acquisition channel, onboarding signals — you can make reasonable personalization decisions before any behavioral data exists. The synthetic prior decays as real data accumulates. Day-zero personalization is a retrieval problem, not a training problem.

It is inspectable. A retrieved context bundle can generate an explanation: "Because you've engaged deeply with distributed systems content, we're prioritizing consensus algorithm coverage." That's a real explanation. A weight delta in a LoRA adapter is not inspectable in any user-facing sense.

It composes without retraining. You can combine signals across modalities, time horizons, and signal types — explicit ratings, implicit dwell, cross-product behavior — in the retrieval layer without touching the model. Adding a new signal type means updating your graph schema and retrieval query, not rerunning a training job.

The stack at ×marble: a personalization knowledge graph encodes user signals as typed, temporally-decayed edges — watched, skipped, rated, searched, spent-time-on, abandoned. At inference time, graph traversal surfaces a context bundle — the top-k most relevant preference signals for the current session — that gets injected into the model's prompt or used to rerank candidates from a retrieval index. The model is frozen. The personalization layer is live.

This borrows RAG mechanics but the index is user state, not documents. The retrieval query is the current session context, not the user's text. The output is a preference scaffold that steers generation, not source passages that ground factual claims. The distinction matters for how you design the retrieval query and how you inject the results.

Designing the Preference Representation

The retrieval layer is only as good as the preference representation it retrieves from. Several design decisions determine whether this works in practice.

Edge typing matters. A flat "interaction" edge loses critical signal. A user who searches for a topic is expressing different intent than a user who watches it, skips past it, or completes and then shares it. Your graph schema should preserve these distinctions. The retrieval layer can then weight edge types differently — completions and shares signal stronger preference than searches; skips signal active dispreference, not just absence of interest.

Temporal decay is not optional. A user's interaction from 14 months ago is a weaker preference signal than an interaction from last week. The representation should encode recency either through edge timestamps with decay functions applied at retrieval time, or through explicit recency-weighted scores maintained in the graph. Flat user profiles that average all historical interactions equally produce stale personalization even when the retrieval layer is fast.

Session context is a retrieval input, not a separate system. What a user is doing right now — what they've searched for, what they've engaged with in this session, how long they've been in the app — is the most informative preference signal you have. The retrieval query should be conditioned on session context, not just on historical profile. A user's historical preference for long-form content matters less if they opened the app at 6 AM from a mobile device — context suggests a short session and a different format preference.

The retrieval query must be fast. Preference graph traversal at inference time needs to be on the critical path. p50 < 10 ms is achievable with a well-indexed property graph database (Neo4j, TigerGraph, Kuzu) and a query designed against indexed edge properties. If your graph traversal is adding 50–100 ms to every request, you've created a new latency problem in the process of solving the personalization problem.

Measuring Personalization Quality Without a Training Signal

A practical concern for teams coming from a fine-tuning mindset: without a held-out test set to measure perplexity against, how do you know if your retrieval-based personalization is working?

The right metrics are behavioral, not statistical:

Engagement delta. Does injecting user context increase dwell time, completion rate, or interaction rate relative to a held-out control group receiving no context injection? A/B experimentation is the ground truth here. Run it continuously.

Ranking hit-rate. For recommendation use cases, does the user click or engage with items that rank higher when preference context is injected? Compare against a baseline that ranks without personalization — the delta is your personalization lift.

Preference alignment on logged pairs. Given pairs of items where the user has historically chosen one over the other, does your system rank the chosen item higher? This is measurable offline against existing interaction logs and requires no live experimentation to get a baseline read.

Explanation acceptance rate. If you surface preference-grounded explanations and users can override them ("less of this"), the override rate tells you where your preference representation is wrong. High override rate on a specific signal type means that signal type is generating wrong inferences. Fix the retrieval query or the edge weight, not the model.

What you do not need: perplexity on a held-out preference corpus. That metric is relevant for evaluating fine-tuning quality. For retrieval-based personalization, perplexity tells you nothing about whether users are getting output tailored to them.

Where to Start if You're Building This in 2026

The decision tree is simpler than it looks once you separate the two problems clearly.

Audit your current output for the source of the genericness. Is the model generic because it doesn't know your domain — wrong vocabulary, wrong format, wrong default tone? That's a fine-tuning problem. Is it generic because it doesn't know this specific user — applying the same voice, the same depth, the same content mix to everyone regardless of their signals? That's a retrieval problem. Most teams have both, in different proportions. Identify which is dominant before choosing a tool.

Build the preference representation before the injection layer. You need somewhere to store user signals — typed, timestamped, weighted. A graph database or a property graph over a document store works. The schema matters: edges need types and temporal metadata. Flat profile tables average away the signal variance you need. Design the schema before you write the retrieval query.

Instrument for implicit signals from day one. Explicit feedback — ratings, saves, shares — is high-value but low-volume. Most users never rate anything explicitly. Implicit signals — dwell time, scroll depth, skip rate, return visits to a topic, abandonment point — are noisier but abundant, and they're what your preference graph will actually run on at scale. Instrument for them now; adding instrumentation later means a cold start on signal accumulation.

Keep the model frozen as long as possible. Every fine-tuning run is a synchronization event: the model's behavior changes, your retrieval layer's injected context may need to change to match, your prompt templates may need to update, your evaluation suite needs to re-run. Fine-tuning is expensive to operate in terms of infrastructure and coordination overhead. Don't do it until you've exhausted the retrieval layer's headroom. For most teams, that headroom is substantial.

If you've done all of this and you still see a clear gap — the model's output format is wrong for your domain, its vocabulary doesn't match your users' expectations, its default tone is misaligned — then run a domain fine-tuning job on shared, stable properties. Not on individual user preferences. Never on individual user preferences.

The teams shipping best-in-class personalization in 2026 are not the ones with the most fine-tuned models. They're the ones with the richest, fastest, most current preference representations and the architecture discipline to inject them at the right layer. The model is a commodity. The preference layer is where the differentiation lives.