Prompt: Training and Serving Laplace 3B Linear Attention Model on AWS Trainium2

The article explains how i14 adapted its Laplace 3B model, which uses Gated Linear Attention instead of softmax, to run on AWS Trainium2. By rewriting the attention kernel in native PyTorch, optimizing Muon communication with all_to_all, and unrolling the backward pass, they achieved ~120k tokens/s training and 11.5k tokens/s inference with a fixed 29.5 MB state per sequence.

Copy This Prompt

I am building a linear attention model for [task] with a fixed context length of [N] tokens. I want to train it on AWS Trainium2 using native PyTorch. Please provide a step‑by‑step guide that includes: (1) setting chunk size and count, (2) rewriting the attention kernel without Triton, (3) configuring Muon optimizer communication, and (4) unrolling the backward pass. Include code snippets for each part.

Example input

Use the placeholders in the prompt with permitted, anonymized source data.

Expected output

Follow the output format requested in the prompt and verify the result against the source.

Tools

AWS Trainium2, PyTorch, TorchNeuron Native, vLLM, NeuronCore, MuonFSDP2