Sequence Models for Next-Item Prediction: SASRec, BERT4Rec, and Real-World Fit

TL;DR.

• Sequence models for next-item prediction treat user history as an ordered sequence, not a bag of preferences — they catch the session arc that static embeddings flatten.
• SASRec (Kang and McAuley, 2018) uses causal self-attention to predict the next item from everything that came before it, left to right.
• BERT4Rec (Sun et al., 2019) uses bidirectional attention and a masked-item Cloze task, similar to BERT for language.
• A RecSys 2023 paper showed SASRec actually beats BERT4Rec when both are trained with the same full-softmax cross-entropy loss — the original SASRec used negative sampling, which under-represented the architecture.
• Simpler sequence models (GRU4Rec, item-to-item k-NN, n-gram) often match Transformer performance on short sessions and ship faster — pick complexity to match your data, not the other way around.

The next click usually depends on the last five clicks, not the average click. Static user embeddings — the kind a vanilla matrix-factorization or two-tower model produces — collapse that ordering into a single point. Sequence models for next-item prediction keep the order intact and condition the prediction on it. This post walks through what sequence modeling captures that static doesn't, how SASRec and BERT4Rec differ in architecture and training, what the RecSys community discovered about benchmark comparisons in 2023, and the cases where a simpler model is the right call.

What sequence models capture that static doesn't

Consider a user whose interactions are [Sci-Fi Movie, Action Movie, Sci-Fi Book]. A static collaborative-filtering model averages these into a preference vector that says "this person likes action and sci-fi." A sequence model notices the user has shifted modality — from movies to books — within the sci-fi genre, and that the most recent action is a strong signal for what's next. The example is from a survey of sequential models in recommendation, and it generalises: order, recency, and context dependence are first-class signals that static embeddings smear out.

This matters most in three places:

• Session-based recommendation — short anonymous sessions where the user has no profile but a few clicks tell you a lot
• Next-item prediction — the canonical task where you predict the immediate next interaction given the prior n
• Intent shift detection — recognising when the user pivoted from one job-to-be-done to another inside the same session

Static models can be made sequence-aware with hacks (recency weighting, time-decay on co-occurrence counts, hand-crafted session features), but sequence models for next-item prediction handle this natively because order is part of the architecture.

SASRec: causal attention for next-item prediction

SASRec (Self-Attentive Sequential Recommendation) was introduced by Wang-Cheng Kang and Julian McAuley at ICDM 2018. The paper takes the Transformer encoder, applies a causal attention mask, and trains it to predict each item in a user's history from the items before it. The architecture is roughly:

1. Embed each item ID in the user's sequence.
2. Add a positional embedding.
3. Pass through stacked self-attention blocks with a lower-triangular mask, so position t can only attend to positions <= t.
4. At each step, score candidate items against the position's output to predict the next item.

The causal mask is the whole point. It enforces "predict the next item from past items only" at training time, which matches inference. Kang and McAuley reported SASRec outperformed Markov chain, CNN, and RNN baselines on sparse and dense datasets, and ran an order of magnitude faster than comparable CNN/RNN models per their published evaluation. SASRec became the default Transformer baseline for sequential recommendation almost overnight.

Two operational properties make SASRec friendly in production: training is parallelisable across positions, because there's no left-to-right unrolling, and inference is O(L) attention for a sequence of length L, which is fast enough for sub-100 ms serving on modern hardware as long as you cap L at something sane (50 to 200 items is typical).

BERT4Rec: bidirectional masked modeling

BERT4Rec came one year later, from Sun et al. at CIKM 2019. The paper argues that causal left-to-right modeling restricts the representation power of the historical sequence, and replaces causal attention with bidirectional attention plus a masked-item training objective borrowed from BERT. The training task is the Cloze task: randomly mask items in the user's sequence and predict the masked items from both their left and right context.

This is the same trick that lets BERT outperform GPT-style models on classification — bidirectional context strictly dominates unidirectional context for representation learning. The BERT4Rec authors reported gains on the order of 7 to 11 percent in HR@10, NDCG@10, and MRR over the strongest baselines across four datasets. The paper became one of the most-cited sequential recommendation works of the decade.

There's a subtlety, though. BERT4Rec is trained on masked-item prediction, but at inference time you want the next item — there's no future context to condition on. The standard workaround is to append a [mask] token at the end of the sequence and treat the bidirectional model as if the missing future is empty. This makes BERT4Rec less natural for strict next-item prediction than SASRec, and it leaks information at training time in subtle ways. Several follow-up papers, including the RecSys 2023 BERT4Rec replicability study, have flagged this.

SASRec vs BERT4Rec: what 2023 found

For five years the field treated BERT4Rec as the stronger model. A 2023 RecSys paper — "Turning Dross Into Gold Loss: Is BERT4Rec really better than SASRec?" — changed the consensus. The authors' code and results show that the original SASRec used a negative-sampling binary cross-entropy loss, while BERT4Rec used a full softmax over the item catalog. When you swap SASRec's loss for the BERT4Rec-style full-softmax cross-entropy, SASRec significantly outperforms BERT4Rec on quality and training speed.

The numbers from their replication on standard benchmarks:

• MovieLens-1M: SASRec+ reached NDCG@10 of 0.1821 vs BERT4Rec's 0.1537
• MovieLens-20M: SASRec+ reached NDCG@10 of 0.1833 vs BERT4Rec's 0.1703
• Similar margins on Amazon Beauty and Yelp

A separate systematic review of BERT4Rec, published in TOIS, ran 134 comparisons across the published literature and found BERT4Rec won 64 percent, SASRec won 24 percent, and they tied 12 percent — not the consistent dominance the original paper suggested.

The practical takeaway: the architecture choice between causal and bidirectional attention is less decisive than the choice of loss function and training regime. For most sequence models for next-item prediction, the full-softmax cross-entropy loss is worth its compute cost — it gives the model a calibrated probability distribution over the entire item catalog, which downstream re-ranking depends on. We have written more about how recommendation losses interact with retrieval quality in our piece on the path of a recommendation.

When simpler sequence models win

A repeated finding from session-based recommendation research is that simpler methods often match or beat Transformers, especially on short sessions and sparse catalogs. A widely-cited evaluation of session-based recommendation algorithms showed that nearest-neighbour and association-rule baselines are competitive with GRU4Rec and often outperform it in accuracy, while being trivial to deploy and update online.

The candidates worth keeping in your toolkit:

• Item-to-item k-NN with recency weighting — score next items by their co-occurrence with the last k items in the session. No training. Updates online. Often hits 80 percent of Transformer performance on short sessions for a fraction of the operational cost.
• GRU4Rec — the 2016 RNN that started session-based recommendation. Still competitive on long sessions where the gated recurrence acts as a soft attention over the session arc.
• N-gram models — fixed-context next-item probability tables. Embarrassingly easy to debug, easy to explain, and they work fine when the catalog is small.
• FPMC — factorised Markov chain that combines matrix factorisation with a first-order Markov chain. The historical baseline most sequential papers compare against.

The decision rule we use: start with a sparse k-NN or GRU4Rec baseline before reaching for SASRec. If the baseline gets you 80 percent of the way and your sessions are short, the marginal NDCG@10 you gain from a Transformer is rarely worth the GPU bill. If your sessions are long (more than 20 items typical) and your catalog is large, the Transformer earns its keep. Our broader take on this trade-off is in the post on why collaborative filtering is aging.

Production trade-offs that matter

Once you've picked a sequence model class, the production cost shows up in four places:

1. Training cost — SASRec and BERT4Rec on a 1M-user catalog take hours to a day on a single A100. With full-softmax loss, BERT4Rec is materially slower than SASRec per epoch — the RecSys 2023 paper found SASRec converges faster across all tested datasets.
2. Retraining cadence — sequence models drift fast because user behaviour drifts fast. We retrain ours on a fixed cadence (daily for hot domains, weekly for slower ones) rather than ad-hoc on alerting, because the failure mode of stale sequence models is silent regression on tail items.
3. Cold-start handling — pure sequence models predict from item IDs, which is useless on day one. You need a content-feature fallback (item metadata embeddings) or a knowledge-graph adjacency model to serve new items. We cover the patterns in the post on the cold-start problem and day-zero personalization.
4. Serving latency — Transformer inference at sequence length 50 with a 4-layer model is roughly 5 to 15 ms on a modern GPU. That fits comfortably inside a 100 ms request budget. Sequence length 500 with a 6-layer model is 60 to 150 ms — at that point you're spending most of your latency budget on the recommender alone.

The serving-side architecture matters as much as the model choice. Our reference architecture for real-time personalization walks through how to compose a sequence model with a candidate generator and a re-ranker without blowing the p95 latency budget.

How ×marble fits in

×marble runs sequence models for next-item prediction on top of a knowledge graph that holds the catalog, the user, and their interaction history as first-class entities. The sequence model gives us the immediate "what's next" signal; the graph gives us the structural priors that keep day-one users and tail items from cratering. We use this pattern across our products — daily AI briefings at vivo.timesmarble.com, the personalized YouTube product at video.timesmarble.com, and Spotify and Apple Music personalization at marblexmusic.com.

If you're building sequence-aware recommendations and would rather not assemble the training loop, the retraining pipeline, the cold-start fallback, and the serving stack yourself, you can book a call at timesmarble.com — that's what we built ×marble to be.

FAQ

What is a sequence model for next-item prediction?

A sequence model for next-item prediction is a recommender that takes the ordered history of a user's interactions and predicts the most likely next interaction. Unlike static collaborative-filtering models, it conditions on the order and recency of past actions — so two users with the same item set but different click orders get different predictions.

Is SASRec or BERT4Rec better?

A 2023 RecSys replicability paper showed SASRec significantly outperforms BERT4Rec when both are trained with the same full-softmax cross-entropy loss. The original BERT4Rec paper looked stronger partly because the SASRec reference implementation used negative sampling, which under-represented its architecture. The practical answer in 2026: start with SASRec plus full-softmax loss before reaching for BERT4Rec.

When should I use a Transformer-based sequential recommender instead of a simpler model?

Use a Transformer-based sequential model when your sessions are long (more than 20 items typical), your catalog is large (more than 100,000 items), and you have enough interaction data to train on. For short sessions or sparse data, item-to-item k-NN with recency weighting or GRU4Rec often match Transformer performance for a fraction of the operational cost.

How does BERT4Rec handle next-item prediction at inference if it's bidirectional?

BERT4Rec appends a [mask] token at the end of the sequence and asks the model to predict the masked position. This is a workable hack but technically mismatched with the training task, where masks appear anywhere in the sequence. It's one reason follow-up papers have argued the bidirectional advantage is smaller than it first appeared.

What's the difference between session-based and sequential recommendation?

Session-based recommendation predicts the next item inside an anonymous short session, usually without a user profile. Sequential recommendation typically uses the full known history of an identified user. The architectures overlap — GRU4Rec, SASRec, and BERT4Rec are used in both — but session-based emphasises recency and intent shift, while sequential emphasises long-term preference drift.

Further reading

• Why collaborative filtering is aging — how matrix-factorisation baselines compare to sequence models and graph models.
• Cold-start problem and day-zero personalization — what to do when a sequence model has nothing to condition on.
• Reference architecture for real-time personalization — composing a sequence model with candidate generation and re-ranking.
• Explainable recommendations and the path of a recommendation — why interpretable scoring matters even when the model is a Transformer.
• SASRec paper (Kang and McAuley, 2018) — the original arXiv preprint.
• BERT4Rec paper (Sun et al., 2019) — the bidirectional follow-up.
• BERT4Rec replicability study (Petrov and Macdonald, RecSys 2022) — the systematic 134-comparison review.