How does a neural network become Flappy Bird?

The model keeps the last 15 frames, each with the key that made it. At the start they are real frames. After 15 frames, every frame in memory is one the model drew itself.

You press TAP or nothing. That one bit goes in with the new frame. It is the only thing you control.

The new frame starts as random static. The transformer must turn this static into the next frame.

Each pass reads the noise, the 15 frames in memory and your key. It outputs a slope for every pixel, and we move the static along it. Two moves give a clean frame.

The frame goes to your screen. A third pass stores it in memory, and the oldest frame drops out. Then the loop starts again.

Each small picture is one real frame from a real game. The number above it is its frame number. The model reads all 15 frames.

The key is 0 for no tap and 1 for a tap. The key under a frame is the key that made that frame. Past keys are not dropped.

Use the two buttons above the picture. The 15 frames stay the same. Only your key changes. It is the key that will make the next frame.

The model draws the right-hand big picture itself, from the same 15 frames and the same noise. Press each button and watch the bird. With no tap it falls. With a tap it rises.

The oldest frame leaves. Memory holds 15 frames again, so the loop repeats. No game engine runs in this loop. The model reads pixels and writes pixels.

It is an action-conditioned, autoregressive video diffusion transformer, trained with rectified flow on raw pixels. The next pages open each word.

The data is a public Flappy Bird set (Dreamer4 format). Each frame is a 128 x 128 colour picture. Each frame also has the key the player pressed.

28 games never tap, 28 games always tap, and 115 games choose when to tap. The choosing games hold 98.8% of all frames. The longest game has 30,000 frames.

14 whole games are held out: the model never trains on them. Neighbour frames are near copies, so a split by frame would leak the test into training.

168 games end in a crash, on their last frame. 3 more games stop at a 30,000-frame time limit. So the data never shows what happens after the bird hits a pipe.

During training, all 300,153 frames stay on the GPU as bytes: 14.75 GB. A batch is cut out of that block with no disk reads.

The key pressed at frame t shows up in frame t + 1. So we store, for each frame, the key that produced it: the key of the frame before.

Nothing produced the first frame of a game. Its key is set to 0, so no key ever crosses from one game into the next.

One training example is 16 frames in a row from one game, with their 16 keys. A window never crosses into another game.

The model predicts all 16 frames of a window, each from the frames before it. Frame 0 has no earlier frame to read: it sees only itself and its key. Chapter F shows how one pass does all 16.

A training step takes 8 random windows. That is 8 x 16 = 128 next-frame predictions in each step.

This is a short stretch of one real game, frame by frame. The row under it shows the key pressed at each frame. A 1 is a tap. Most frames show 0.

The key pressed at frame 73 makes frame 74. So the key that made frame k is the key pressed at frame k minus 1. The second row is the first row moved right by one.

The first frame of a game has no frame before it. It gets key 0. Games 3 and 4 tap all the time, yet frame 0 of game 4 still gets 0. Without this rule, the last key of one game would leak into the next game.

A window starts at any frame where 16 frames in a row stay in one game. This game has 267 frames, so it gives 252 windows. Windows overlap, and each one is a different example.

One training step takes 8 windows. That is 8 times 16 = 128 frames to learn from. A pixel byte b becomes b / 127.5 minus 1, so every value lies in the range -1 to 1.

The frame is 128 pixels wide and 128 high. Each pixel holds 3 numbers from 0 to 255: red, green and blue. That is 49,152 numbers.

Lines every 8 pixels cut the frame into a 16 x 16 grid. Each square is a patch. The patches are numbered 0 to 255, row by row, like reading. That is 256 patches, not 255.

Colour shows what each patch holds. Sky, ground and grass repeat across the screen. Only 128 different patches exist. The other 128 are repeats of them.

Patch 132 (row 8, column 4) holds the bird. Enlarged, it is just 8 x 8 coloured squares. Hover a square to read its 3 numbers.

The 64 pixels are read row by row. The top row is 0 to 7, the next is 8 to 15, and so on to 63. Every patch uses this order.

The patch splits into three 8 x 8 grids: one of red values, one of green, one of blue. Each cell is one number from 0 to 255.

The red grid is read row by row: 64 numbers. Then all 64 green, then all 64 blue. Together that is one list of 192 numbers.

Each number b becomes b / 127.5 − 1. So 0 becomes −1, 255 becomes 1, and the middle becomes 0. Networks learn best with numbers near 0.

The strip shows the 192 values: red for positive, blue for negative. The bird's bright orange is the red block on the left.

The code does for all 256 squares of all frames what B1 and B2 did for one patch. It needs three lines. Each box shows the shape after one line.

The view line gives names to the axes: 128 rows become 16 square-rows of 8, and the same for columns. The permute line moves "which square" to the front. The reshape line flattens each square into 192 numbers.

The table has 256 rows and 192 columns. Row 132 is the bird patch from B2. It holds the same 192 numbers: 64 red, 64 green, 64 blue.

A batch has 8 windows of 16 frames, so N = 128 frames. The same three lines run on all of them. Each copy of the batch becomes [128, 256, 192].

Top strip: the bird patch's 192 numbers from B2. Middle: 192 learned weights. Bottom: each input times its own weight.

The line adds the products from left to right. It ends at 0.147. One more learned number, the bias 0.054, makes 0.201. That is output number 0.

The layer holds 768 rows of 192 weights: a table of 768 x 192. Each row gives one output number, the same way. So one patch gives 768 numbers.

The layer is made with one line: 192 numbers in, 768 out. It is Linear(192, 768), not (768, 192). PyTorch stores the weights as [out, in], so the table is 768 rows of 192. The bias holds 768 numbers.

Nothing is picked or thrown away. All 768 outputs, side by side, are the token of this patch. A token is one row of 768 numbers. Every patch uses the same table.

768 is the model's width, a size we chose when we built it. Every token in every block has 768 numbers. GPT-2 small and ViT-Base use the same width.

There is no codebook and no search for a nearest word. A VQ-VAE would keep 1 integer per patch. This layer keeps 768 numbers, so nothing is rounded away.

The table starts as random numbers, spread evenly between −0.072 and +0.072 (PyTorch default). The flow loss trains it with all 16 blocks. After training, row 0 runs from −0.084 to +0.080.

The grid compares the chosen patch's token with all 256 tokens of the frame. Red = alike. Sky matches sky, ground matches ground. Try the buttons.

The same weights run on all 256 squares. A table of 256 rows by 192 becomes 256 rows by 768. One row is now one token.

Squares 0 and 143 are both plain sky. Their pixels are identical, so the Linear gives identical rows. Nothing in a row says where its square sits.

The table pos_s has 256 rows of 768. Row i is added to square i. Now the two sky rows differ. This is the only place that says where in the picture.

pos_s starts as small random numbers and is trained like any weight. Paint one number of each row on the 16 by 16 grid. Some numbers grow from top to bottom. Others grow from left to right.

At play time the model holds 15 memory frames and draws one new frame. Each memory frame has the key that made it. Look at the real keys under the strip. Past keys are kept, not only the latest one.

The noise level t says how much static a frame has. Memory frames have t = 0.05, a pinch of static. The new frame starts as pure static, t = 1. The model must be told how much static each frame has.

Each frame gets one row of 768 numbers, called c. It is made from that frame's key and that frame's t. In play, one frame goes through per pass, so c is [B, 1, 1, 768]. In training, 16 clean and 16 noisy frames go in together, so c is [B, 32, 1, 768].

The table nn.Embedding(2, 768) has 2 rows of 768 learned numbers. Key 0 reads row 0. Key 1 reads row 1. That is all the key does here.

The value t is a decimal, so a table cannot list it. A Linear layer on t alone would only slide along a straight line. So the code spreads t into 256 waves with no weights. Then Linear(256, 768), SiLU and Linear(768, 768) give 768 numbers.

Add the key row and the noise row. Apply SiLU, which squashes large negative numbers to about 0. The result is c, one row of 768 numbers for this frame. The code is world.py:58-61.

The 256 token rows of the frame do not get c added. Instead c goes to every block. There a Linear called ada turns it into shift, scale and gate. The same c reaches all 16 blocks.

A block is one attention half plus one MLP half. The model stacks 16 blocks with the same code but different weights. Each block has its own Linear called ada. It turns the same c into six rows of 768.

The six rows are shift, scale and gate for the attention half, then the same three for the MLP half.

This is one real token row in block 1. The six numbers are the ones where the gate matters most. LayerNorm moves its 768 numbers to mean 0 and spread 1. Then each number is multiplied by 1 plus scale, and shift is added. Six numbers are shown.

Every token row of the frame is multiplied by the same scale and shifted by the same shift. The row is copied 256 times. So c steers the whole frame at once.

After attention, the result is multiplied by the gate and added to x. Where a gate is near 0, the token does not move. The six numbers shown have big gates on purpose. A typical gate1 in block 1 has size 0.009. The result comes from this one frame alone, with no past.

We built a fresh stack of 16 blocks and sent this frame through it. We tried random ada weights of size 0.02 and of size 0.1. The larger ones push the output far from the input. With zero weights it stays identical, by construction.

Lines 22 to 24 of block.py set the weights and bias of ada to zero. A new block adds nothing and passes x straight through. Training opens each gate only as far as it helps.

This is adaLN-Zero, from the DiT paper (Peebles and Xie, 2022).

The frame is a table of 256 rows. Each row has 768 numbers. The model reads frame 326 with half noise on it. We follow row 164, the square with the bird.

The layer qkv turns the 768 numbers into 2,304. That is three lists of 768 for this row: q, k and v.

Each list of 768 is cut into 12 pieces of 64. A piece is one head. The code only relabels the numbers.

Each head of q and k is rescaled to a steady length. Head 9 of q shrinks from 10.2 to 8.0. This happens before attention.

The row q asks a question. The row k says what a row offers. The match q·k says how much to read. The row v holds what is read.

All 256 rows of the frame make their own q, k and v at the same time. With 16 frames that is 4,096 rows in every head.

The reader has a q. Each row to read has a k and a v. We want to know how much to take from each row.

Multiply the matching numbers of q and k, then add. A big result means a good match. A negative result means none.

Each match goes through e to the power of it, then divides by the sum. The shares are positive and add to 1.

Multiply each v row by its share and add them up. The result is the reader's new row. The shares only weigh; v is the content.

Row 164 reads 15 frames of 256 rows, plus its own 256. Head 9 puts 17.6% on one square: the one just left of the bird, in frame 323.

Each head does the same steps on its own 64 numbers. The 8 is the square root of 64. It is not the cut into heads.

The 12 results sit side by side as 768 numbers. A Linear layer mixes them. The gate scales them, and the sum goes back into the row.

Take the k of the bird's square in each of 16 frames, before any rotation. Frames 312 and 318 give q·k 16.4 and 16.6, yet one is 14 frames back and one is 8. No distance is in there.

Rotate q by its position m and k by its position n. Press "+1 to both". The arrows turn, but q·k does not change. It depends on m − n.

Number i and number i+32 form pair i. Each pair is a 2-number arrow. Pair 0 turns fast. Pair 31 hardly turns. Move n to see it.

Frames in the window get positions 0 to 15. The real q sits at m = 15. With the same q and the same k, one distance always gives the same q·k, here 19.18.

The 4,096 rows are one flat list. A second list, pos, holds the frame number of each row. All 256 squares of a frame share it.

The code is the slide, for 32 pairs at once. angle is position times speed. The last line rotates every pair.

The block rotates q and k, not v. In training, both copies use positions 0 to 15. Playing keeps k rows un-rotated; page H3 shows why.

The second half of the block is an MLP. It works on each row alone. It widens 768 numbers to 3,072, bends them, and narrows them back.

Before GELU, 86% of the 3,072 numbers are negative. GELU turns them to about 0. Without the bend, two Linear layers equal one.

Each half computes a correction and adds it to the row. Row 164 enters block 8 at length 25.5 and leaves at 16.2. The shape stays the same.

The same c goes to all 16 blocks. Each block has its own ada layer, so its own shift, scale and gate. All gates start at 0.

A block is a stack of layers. The 16 blocks run one after another. Inside each block, 12 heads run side by side. That is 192 heads.

One block holds 10,628,480 learned numbers. The ada layer holds one third of them. The 16 blocks hold 98.6% of the model.

After block 15, one LayerNorm, a shift and scale from c, and one more Linear layer make the output. Each row is handled alone. This head is not an attention head.

The Linear layer gives 192 numbers per row: red, green and blue, 64 each. They fold back into three grids of 8 × 8.

Put all 256 squares together and the output has the shape of a frame. It is not a frame. It holds a slope for each pixel. Part E explains slopes.

The two head layers start at exactly 0. A brand-new model outputs 0 everywhere. Its loss is 1.28. Training moves it from that calm start.

The game is running. The model holds 15 memory frames and must draw the next frame for key 0, no tap. This pass works on a frame that is half real picture, half noise. We call it z, and its noise level is t = 0.5.

Out comes the slope: one number per pixel. It points from the real frame towards the noise. The slope is the direction from the real frame towards the noise. A later page shows it on one pixel.

Take z, then subtract half the slope: z − 0.5 · slope. z moves against the slope, away from the noise. Half of z was the real frame, so this one step is easy. Real play starts from pure noise.

Each chip is one stop, with the shape of the numbers there. Click a chip to jump to its page. We follow the bird's square, number 132, along this road.

Everything between the left picture and the right picture is done by learned numbers. The next pages open every stop and show the real values of this one pass.

Before this pass, each memory frame went through the same 16 blocks once. It had t = 0.05 and the key that made it. Here all 15 keys are 0, no tap. This pass only reads what they left behind.

A frame of 128 × 128 pixels is cut every 8 pixels into 16 × 16 squares. That is 256 squares. Each square becomes one row of 768 numbers.

Stack the 15 frames: 15 × 256 = 3,840 rows. Frame 1 holds rows 0 to 255, frame 2 holds rows 256 to 511, and so on.

In every block, each frame leaves two tables behind: k and v, 256 rows each. One shelf per block holds all 15 frames. The grid shows 16 shelves × 15 frames.

The new frame has its own 256 rows. Block n reads the 3,840 saved rows of shelf n plus these 256 new rows. That is 4,096 rows.

On the left are the 15 past frames, each with the key that made it. Each frame went through the same 16 blocks once, earlier. Every block kept that run's k and v rows: 15 × 256 = 3,840 rows per block.

The new frame z is half real, half noise. Its key and its noise level t make c, one row of 768 numbers. The squares of z become x, a table of 256 rows × 768.

The middle column is the only transformer. It is 16 blocks, one after another. Each block has an attention half and an MLP half, and x keeps the shape 256 × 768.

Blue arrows: block n's attention reads its own 256 rows plus the 3,840 saved rows of block n. Yellow arrows: c sets scale, shift and gate in every block. This is the only place the past and the key enter.

After block 16, one small layer turns each row of 768 into 192 pixel values. A reshape puts the 256 squares back into a 128 × 128 picture. There is no second transformer.

z minus half the model's output is the new frame. The new frame is then pushed into memory and the oldest one is dropped, so the loop repeats.

Each pixel value p, from 0 to 255, first becomes p / 127.5 − 1, a number from −1 to 1. The top-left red pixel of the bird's square is 78, so it becomes −0.388. In z it is mixed with noise and reads −0.299.

The 128 × 128 frame is cut every 8 pixels into 16 × 16 = 256 squares. The code calls a square a patch. Square 0 is top left and square 255 is bottom right.

Count 8 squares down and 4 across: 8 × 16 + 4 = 132. It covers pixel rows 64 to 71 and columns 32 to 39. Enlarged, it is 8 × 8 pixels.

Every pixel has a red, a green and a blue number. So the square holds 3 × 8 × 8 = 192 numbers. These are the real values of z, colour by colour.

The 192 numbers go into one line: the red rows first, then the green rows, then the blue rows. This line is the square's list. Red is positive and blue is negative.

All 256 squares do the same, so the frame becomes 256 lists of 192 numbers. The shape is [1, 256, 192]. This step has no weights: numbers only move.

The weight is a table of 768 rows and 192 columns. With the bias it holds 148,224 learned numbers. All 256 squares use this same table.

Multiply row 0 pairwise with the 192 numbers of the square, then add. That gives the 192 products below. Their sum plus bias 0 is output number 0.

Rows 1 to 767 do the same, each with its own 192 weights. The result is 768 numbers for this square. The model's own output 0 matches the sum we did by hand.

The model also learns one row of 768 per square place. That table is pos_s, and its row 132 is added to the square's list. Hover a column: the three numbers line up, and the bottom one is the sum.

The same steps run for all 256 squares at once. The result x has 256 rows, one per square, and 768 columns. This table enters block 1.

The noise level t = 0.5 becomes 128 cosines and 128 sines of different speeds. The first cosine is cos(500) = −0.884. This step uses no learned weights.

Two Linear layers with a SiLU between them turn the 256 numbers into 768. The first layer is 256 → 768. Then SiLU, then 768 → 768.

The table act holds one learned row per key. Key 0 uses act[0]. Row 1 is for a tap, so this frame does not use it.

Add the two rows, then apply SiLU, the curve on the right. It squashes big negative numbers to about 0 and lets positive numbers stay. The result is c.

The shape [1, 1, 1, 768] has a 1 where the 256 squares would be. Wherever c is used, that one row is copied to all 256 squares. The key and t reach every block through c.

One row per square, 768 numbers per row. B is games, F is frames in this pass, N is squares and W is numbers per square. Here B = 1 and F = 1.

The same formula runs on row 0, row 1, up to row 255, separately. LayerNorm, scale and shift, qkv and the MLP all work this way. No row looks at another row.

This is where the bird's square looks at its neighbours and at the past. The next pages open that step. First we need c, cut into six rows.

Block 1's own ada layer turns c into 4,608 numbers. They are cut into six rows of 768. None of the six depends on the square.

The same row serves every square. Below, the row 1 + sc1 is copied onto 256 rows. The row has size 1 where the squares go, so it is applied to every row. Every row is identical.

Each of the 256 rows gets its own mean and spread. Every number becomes (value − mean) / spread. After this, every row has mean 0 and spread 1.

Column j of every row is multiplied by the same number, (1 + sc1)[j]. The middle table shows that factor copied to all 256 rows. So its rows are identical.

Then column j gets the same sh1[j] added in every row. The result is y. Here j is only a column number, from 0 to 767.

Here is the bird's row as four strips. Hover a column to read the sum. The first strip times the second, plus the third, gives the fourth.

Each row of y, 768 numbers, is multiplied by a weight table of 768 × 2,304. So each row becomes 2,304 numbers. The table is now 256 × 2,304, still one row per square.

view changes no number. It names the columns: 0–767 are q, 768–1535 are k, and 1536–2303 are v. Inside each, columns 0–63 are head 1, 64–127 are head 2, and so on.

Take the q columns alone: a table of 256 × 768. Its columns form 12 groups of 64 numbers. Each group is one head.

permute moves the head number to the front. Now q is 12 separate tables of 256 × 64. Every head still holds all 256 squares, because rows are never split.

Each head runs the same attention on its own 64 columns. That is all "multi-head" means. The tables k and v are cut in the same way.

At this point q, k and v are only three tables. Attention comes next and multiplies q rows with k rows. If a q or k row is very long, its scores become huge.

These are the lengths of head 1's 256 q rows in block 1, before qn. They run from 8.14 to 16.24. Long rows give big scores.

qn divides each 64-number row by its root mean square, then multiplies by 64 learned weights. After it, all rows have about the same length: 6.85 to 6.97.

Softmax uses exp(score). Without the norm the scores run from −4.55 to 18.21, and exp(18.2) is 81 million times exp(0). A few squares would dominate.

A table row has no time of its own. So rope turns each row by its frame number. Memory rows get 0 to 14, the new frame gets 15.

Head h owns 64 of the 768 columns. It scores q against k on those columns only and divides by 8. Then it takes a weighted sum of the v columns.

The calls view and permute lay the 12 slices out as 12 tables. One call then computes all 12 heads at once. It matches the loop to 3.1e-06. One big head over 768 columns differs by 0.611.

Each card is one head: the columns it owns and the real range of its scores. Each head writes its 256 × 64 result next to the others.

8 is the square root of 64, the width of one head. Every head divides its scores by 8, so a dot product of 64 numbers does not grow too large. The cut into heads is made by the calls view and permute.

q is 256 × 64. The table k, turned on its side, is 64 × 4,096. Their product is 256 × 4,096 scores.

It holds the 3,840 stored rows of the 15 memory frames plus this frame's 256 rows. Entry (r, j) is row r of q times row j of k, divided by 8. So every square scores every square of every frame.

Take the exp of each score, then divide by the sum of those three. The three shares are positive and add up to 1. A bigger score gets a bigger share.

Softmax runs along each row of the 256 × 4,096 score table. Each row becomes 4,096 shares. The columns are grouped by frame: memory 1 to 15, then the new frame.

Row 132 holds 4,096 shares, one per square it may read. Put them back onto the 16 frames. Darker means a bigger share, and the numbers are the share of each frame.

Every row sums to 1 within 0.00001. The largest single share anywhere in head 1 is 0.09%. So head 1 spreads its attention widely.

Shares are only weights. The table v holds the content: 64 numbers for each token. The tables k and v come from different columns of qkv.

Three tokens, with v rows of two numbers. Each v row is multiplied by its share, and the results are added. That gives the new row o.

Block 9, head 12, in the t = 0.5 pass. The bird's row takes 69% from five tokens, all in memory. The two biggest are the square to its left, one and two frames ago.

Block 1, head 1, same pass. In the bird's row the largest share is 0.053%. So its row o is close to a plain average of almost all 4,096 tokens.

Row r of o is the 4,096 rows of v, each multiplied by its share in row r, then added. Every square makes its own mix.

Every head does this. The 12 tables of 256 × 64 are put back side by side: 256 × 768 again. The calls transpose and reshape only move numbers into that order.

The layer proj is a Linear(768, 768) applied to each row of o. It mixes all 768 columns, so the 12 heads can share what they read. The result is the table proj(o).

g1 is one row, the same for every square. Each column of proj(o) is multiplied by its g1 number. Most of g1 is near 0.

The product is the change this half makes. Only a few columns are big, because only a few g1 numbers are big.

The change is added to x. The result is x after the attention half. Most columns of x hardly move.

The MLP half starts like the attention half: LayerNorm, then scale and shift. This time it uses sc2 and sh2 from c. The result is the table y2.

A Linear layer turns each row of 768 numbers into 3,072 numbers. No row looks at another row. The table is now 256 × 3,072.

GELU works on each number alone. A negative number becomes about 0 and a positive number stays. In the bird's row, 780 of the 3,072 numbers stay above 0.01.

Another Linear layer turns each row of 3,072 back into 768 numbers. The table is 256 × 768 again. This is the MLP's output.

The output is multiplied by g2, one row for all squares. Then it is added to x. The result is x out of block 1.

Block 1 takes a table of 256 × 768 and gives back a table of the same shape. The numbers are different. Look at the yellow row: the bird's square has changed.

Read the chips from left to right. Each chip is the shape after one step. Attention is the only step where rows meet.

We wrote every step by hand outside the model. The result matches the model's own block 1 to the precision stored (difference 0.0). Shares times v match its attention output to 3.1e-06.

Block 2 repeats every step with its own weights. It uses the same c but its own saved k and v rows.

Each row is one block. The bars show how the bird's square splits its attention over the 15 memory frames and the new frame. Blue is memory and red is the new frame.

This map shows the share the bird's square gives to each square of the new frame. Darker means a bigger share. Block 1 puts 97% on memory, so its map is faint.

This map shows how much the attention half changed each square. Darker blue means a bigger change.

The same for the MLP half, in green. Compare the two columns block by block.

The strip is the bird's 768 numbers after the block. The text gives the length of its row going in and the size of both changes.

Attention changes the bird's square most in blocks 1 and 10. The MLP change is largest in blocks 14 to 16. The row length goes up and down: 8.75 into block 1, 41 into block 2, 48 into block 15.

Inside attention, v is the values table. The model's output is something else: a number for every pixel that we call the slope. From here on, only the slope is the output.

The input is z = (1 − t) · real + t · noise. At t = 0.5, z is half real frame and half noise. As t goes from 0 to 1, z slides on a straight line from the real frame to pure noise.

The slope is noise − real: the direction from the real frame towards the noise. If the model gives it, one subtraction returns the real frame at any t. That is z − t · slope = real. The training loss measures the miss on the slope.

The top-left red pixel of the bird's square. The real value is 78, which is −0.388. The noise is −0.209, so z = −0.299.

The model's slope is 0.199. The right answer, noise − real, is 0.179. The model is close.

z − 0.5 × 0.199 gives −0.398, which is pixel value 77. The real pixel is 78. The pixel from elsewhere lands exactly: real 32, drawn 32.

The table is still 256 × 768, one row per square. The head does the same three moves as the start of every block. Then it adds one last Linear layer.

Each row gets its own mean 0 and spread 1, as in every block. No row looks at another row.

The layer ada_out turns c into one row sc and one row sh. They are the same for all 256 squares. The table is multiplied by (1 + sc) and sh is added.

The last Linear layer turns each row of 768 into 192 numbers. Every row is still one square. Its 192 numbers are 64 red, 64 green and 64 blue.

The step unpatchify is the exact reverse of the cut at the start. Row r goes to row r // 16 and column r % 16 of the 16 × 16 grid. It has no weights.

Left: the model's slope table. Right: the right answer, noise − real, in the same layout. They look almost the same.

Unpacked into 3 colours × 8 × 8 pixels per square, the tables become pictures. The third picture is the squared error per pixel. Darker means more error.

The model's slope next to the right answer for the 8 × 8 red pixels of square 132. The numbers are close.

Every pixel of z minus half its slope. The slope is almost exactly noise − real. So what is left is almost exactly the real frame.

Left: the drawn red pixels of square 132. Right: the real ones. The first row is sky, with real value 78. The drawn row reads 77 77 77 79 79 79 78 78.

Over the whole frame, the drawn pixel is on average 0.199 steps of 255 away from the real one. Half of z was already the real frame, so this is easy. Real play uses two passes from pure noise.

Each chip is one stop with the shape of its numbers. Click a chip to jump back to its page. The bird's square, number 132, went through every one.

In real play the new frame starts as pure noise at t = 1. It takes a few passes, and each pass moves t closer to 0. Here two passes take it from t = 1 to t = 0.

One more pass at t = 0.05 stores the frame's k and v as memory. The oldest frame is dropped. The next pages follow one live frame from start to end.

These pages follow the code that runs when you play. Each past frame was run through the 16 blocks once. Every block kept that frame's k and v rows on its shelf.

When the new frame is drawn, every block's attention reads its shelf. That is 3,840 past rows next to the 256 new rows. It is the only place the past enters.

A frame takes two passes, from t = 1 to 0.5 to 0. Then one more run at t = 0.05 pushes it into memory. The oldest frame leaves.

The new frame starts as pure noise. After pass 1 it is half noise. After pass 2 it is the frame. Compare it with the real next frame.

This replica of the play code was checked against the real Streamer with the same random seed. The difference is 0.0. It used 2 passes and memory noise t = 0.05.

These are the last 15 frames the game showed, each with the key that made it. Here all keys are 0, no tap. The model does not keep these pictures.

A picture is used once, when it is pushed. What the model keeps is the k and v rows that the push leaves in every block. The pictures only show which frame each set of rows came from.

Each pixel goes from 0 … 255 to −1 … 1 by value / 127.5 − 1. This is the same rule as for any frame.

A push first mixes the picture with a little noise, t = 0.05. Both pictures below are real. The noise is never added to k or v, because they are computed afterwards.

In play, the past is the model's own drawings, which are slightly off. The model was trained on a slightly noisy past, so it can read an imperfect past. Errors can still pile up, and a later training stage works on that.

Without the shelf, each pass would run all 15 pictures again, plus the new frame. That is 4,096 rows through the 16 blocks, twice per new frame. With the shelf, only the new frame's 256 rows go in. Two draw passes and one push make 768 rows in all, 10.7 times fewer. The block still gets 256 rows and gives back 256 rows on every pass.

A saved frame's rows may read only older frames and themselves. They never read the new frame, so drawing it cannot change them. Running them again would give the same k and v. We checked this on 15 real frames of one game. The biggest difference between both ways was 0.00003, which is rounding.

The pushed frame is cut into 256 squares like any frame. It runs through all 16 blocks with its own key. In each block it makes q, k and v.

Its attention reads the frames already on the shelf, plus itself. Then the block appends this frame's 256 k rows and 256 v rows to its shelf. The k rows are stored before the time rotation.

The head runs during a push, but its output is not used. Only the k and v rows matter.

Block n's k and v for this frame come from what block n − 1 gave it. Block n − 1 already read the older frames' rows. So each frame's rows also carry some of the frames before it.

At the start of the game the 15 frames were pushed one after another. Every shelf grew by 256 rows each time, up to 3,840 rows.

Each block has its own shelf. A shelf holds a k table and a v table, 3,840 rows × 768 each. The code stores the 768 numbers as 12 heads × 64.

Each table shows 15 bands of 256 rows. Each band is one frame. The top band is the oldest frame, and the bottom band is the newest.

One shelf is 3,840 × 768 for k and the same for v. With 16 shelves, k has 47,185,920 numbers and v has 47,185,920 numbers.

The new frame begins as random numbers, one per pixel per colour. That is z, with shape 3 × 128 × 128 and noise level t = 1. It holds no picture.

Nothing in z tells the model where the pipes or the bird are. Whatever appears in the frame has to come from the shelves and the key.

z is cut into 256 squares of 192 numbers. Each goes through Linear(192 → 768) and gets its position row added. That gives x: 256 rows × 768.

Separately, the key row and the noise level t = 1 make c: one row of 768 for the whole frame. Below: the 256 features of t, the key row and c.

Block 9's ada layer turns c into six rows of 768. They are sh1, sc1, g1, sh2, sc2 and g2. The same six rows act on all 256 squares.

LayerNorm gives each of the 256 rows mean 0 and spread 1. Then column j is multiplied by (1 + sc1)[j], and sh1[j] is added.

Left: x entering block 9. Right: y, after LayerNorm, scale and shift. Both are 256 rows × 768 columns.

The layer qkv turns each row of y into q, k and v. Each has 256 rows, cut into 12 heads of 64. They come from the new frame only.

In head 1, the row lengths fall from 6.27 – 15.61 to 8.42 – 8.77. This keeps the scores in a safe range.

Shelf 9's 3,840 rows go on top of the new k and the new v. That makes K and V with 4,096 rows each. The red line marks where the shelf ends.

Rope turns every row by its frame. Memory frames get positions 0 to 14, and the new frame gets 15. The shelf keeps plain keys, and every pass numbers the frames again.

Each of the 256 new q rows is compared with all 4,096 K rows. Softmax turns each row of scores into shares that add up to 1. Darker means more.

These are the six largest shares in the bird's row of head 12. Five are memory squares, mostly the square left of the bird in the last frames. One is in the new frame.

The bird's whole row of head-12 shares, put back on the 16 frames. Red means more. The new frame's squares are drawn on the finished frame.

This is the percent of the bird's attention that each head puts on each frame. Heads 2, 3, 4, 5, 9 and 11 put 86 to 97% on the new frame. The other six read mostly memory, and head 12 puts 75% on it.

Each new row becomes the share-weighted sum of the 4,096 V rows. The 12 heads are put side by side, mixed by proj, multiplied by the gate g1 and added to x.

The MLP takes each row from 768 to 3,072 and back to 768. It is multiplied by g2 and added to x. Out comes the new x, still 256 × 768.

The numbers below are the first 8 of the bird's row at each stage. The change is mostly in other columns. Of these first 8, only column 5 moves much.

For each block: how the bird's square splits its attention between the 15 memory frames and the new frame. It is the average over 12 heads.

The last three columns give the size of the row going in, the attention change and the MLP change. They differ a lot from block to block.

These maps show where each block's attention half changed the 256 squares. Blue means a bigger change. The box marks the bird's square.

After block 16, x is still 256 × 768. The head gives each square 192 pixel values. LayerNorm, scale and shift from c, and Linear 768 → 192 do this.

The 256 × 192 values are put back into a 3 × 128 × 128 picture: the slope. It points from the real frame towards the noise.

The step goes from t = 1 to t = 0.5. Half of the slope is subtracted from z, so z moves against it. The frame is now half noise.

These are the first four red pixels of the bird's square, row 64. After pass 1 they are only half way to the real value, −0.388. The line below shows them after pass 2.

z, now half noise, goes through the whole stack again. The shelves are not changed. Only c differs, because t is now 0.5.

The second slope takes z the rest of the way: z + (0 − 0.5) × slope. The result is the frame at t = 0, near the real pixel values.

Compare how much of the bird's attention goes to memory in each pass. Compare the size of the changes too.

Two passes from pure noise give the drawn frame on the left. The real next frame is on the right.

The new frame starts as the same pure noise, with the same key, both times. Only the 15 past frames differ. The noise holds no picture.

The 15 past frames plus the noise go in. The model draws the frame on the right of the arrow. Next to it is the real next frame.

Same noise, same key. Only the past changes. The model now draws a different frame, with pipes and the bird in other places.

The error is 0.24 for past A and 0.23 for past B on the 0 to 255 scale. The noise holds no picture. So the picture came from the past frames and the key.

This test uses 4 passes and a clean past. The live player uses 2 passes and a past at t = 0.05. Its error is 0.73 on a different frame. We did not test which change matters.

It gets noise at t = 0.05. Then one run through the 16 blocks reads the 15 frames on the shelves plus itself. That is 4,096 rows.

Its k and v rows are joined to every shelf. The shelf is cut to its last 15 frames in the same step, so it never holds 16.

The oldest frame's 256 rows drop out. The shelves stay at 3,840 rows. The next key press starts again from pure noise.

The core is a DiT. It is a transformer that removes noise from an image cut into patches. It is told the noise level through adaLN-Zero.

A frame is 256 patches, each a row. The noise level and key set scale, shift and gate in every block. The frame is drawn from noise in steps.

The output is a slope, noise − frame, so z moves on a straight line. Each frame has its own noise level: memory at 0.05, the new frame from 1 to 0.

The new frame reads 15 past frames from saved k and v. Rope tells attention how far back a frame is. Normalising q and k keeps the scores safe.

A world model predicts what the world shows next, given what it showed before and the action taken. Here the 15 past frames and the key go in. The next frame comes out.

It needs observations in, an action in, and the next observation out. Here these are the last 15 frames, the key, and the next 128 × 128 frame.

The drawn frame is pushed back into memory. The oldest frame leaves. When you play, the game engine is not running at all.

There is no hand-written rule and no stored bird position or score variable. What it knows about the game is in its learned numbers and in the last 15 frames and their keys. Anything older is gone.

On the left is real frame 326. On the right is eps, one random number for every pixel and colour. The ring marks the bird pixel we follow.

The rule is mix(x, e, t) = (1 − t)·x + t·e. At t = 0.25 the picture is 75% frame and 25% noise. At t = 1 only noise is left.

Look at the red number of the ringed pixel. It is 0.694 in the frame and 1.228 in the noise. At t = 0.5 it is 0.961.

Plot the pixel's value against t. The five dots fall on one straight line. In training, every pixel moves on a straight line like this.

The same t is used for all 16,384 pixels and all three colours of the frame. Each frame gets its own t.

eps = torch.randn_like(x) draws a fresh random number for each pixel and colour. A frame has 49,152 of them. The picture shows a real eps for frame 326.

Most values sit near 0, but nothing stops a large one. About 32% of this eps lies outside −1 to 1. The largest is 4.1.

Real pixels are bytes turned into the range −1 to 1. Only the frame is limited. So the bird pixel's eps of 1.228 is an ordinary noise value.

Each frame in each training step gets new noise. Here is the same pixel in four different draws. It has a different number each time.

Play also starts from torch.randn noise. Training noise must be the same kind. Otherwise the model would learn to clean noise it never meets.

The horizontal axis is the noise amount t. The vertical axis is the pixel's red value. In y = m·x + c the axis is called x, but here x means the frame. So we rename the axis t.

At t = 0 the line is at the real frame value. So c = x = 0.694. Here c is the line's start, not the condition row c of the blocks.

Take the two ends: t = 0 gives 0.694 and t = 1 gives 1.228. Then (y_b − y_a) / (t_b − t_a) = (1.228 − 0.694) / (1 − 0) = 0.534. That is eps − x. Here v is this slope, not the value rows of attention.

Move t. The noisy value is noisy = (eps − x)·t + x, which is m·t + c. At t = 0.60 it is 0.694 + 0.60 × 0.534 is about 1.015. Walking back down the line is the reverse.

Here are twelve other pixels. Each has its own start x and its own end eps, so its own slope. The model must give one slope per pixel and colour.

The slope is eps − x, so it is mostly noise with the frame taken away. That is why a picture of v looks like noise.

The noisy frame 326 at t = 0.60 goes in. So do the lightly noised past frames, the key, and both noise levels. We draw only the noisy frame here.

The head gives 256 squares of 192 numbers. Then squares go back into a 128 × 128 × 3 picture. Each number is a slope v, not a colour.

The right answer for every number is eps − x. The two pictures match because training made them match. Each dot on the right is one real number.

Walk back down each line: x_hat = noisy − t·v. For the ringed pixel, 1.015 − 0.60 × 0.371 = 0.792. The real value is 0.694.

With the right slope, eps = noisy + (1 − t)·v gives 1.228. So the slope holds the noise. The model predicts the noise minus the frame.

Take the frame value 0.694. A noisy value of 1.015 at t = 0.60 needs v = 0.534. If the same noisy value arrived at t = 0.30, it would need v = 1.068. So t changes the right answer.

The mean miss against the real frame is 65.7 out of 255 for the noisy picture. After one step with v it is 0.16.

Add eps to the negated frame, and you get the answer. The noise is the big part, so the answer looks like noise. For the ringed pixel, 1.228 + (−0.694) = 0.534.

Look at the bottom of the answer. The dark band is the sandy ground, negated. The frame is in there. Mostly it is noise, and that is why the picture looks like noise.

Grey means zero. Magnified 3 times, the misses stay small. The ringed pixel misses by 0.371 − 0.534 = −0.163. The loss is 0.00004. Frame 326 is from a training game.

A new model outputs 0 everywhere. Its difference is the whole answer, and its loss is 1.29. The two pictures match only because training made them match.

For the ringed pixel the model guessed v = 0.371. The right slope is 0.534. The miss is 0.163. Squared, it is 0.0266.

Do this for all 49,152 numbers of the frame. Take the mean. A real step averages 6,291,456 numbers into one.

A brand-new model outputs 0 everywhere, so its miss is the whole slope. Its loss is 1.29. The trained model scores 0.00004 on this frame, which is from a training game.

loss.backward() gives each weight a gradient. Then opt.step() moves it against that gradient. One weight of a small model moved from 0 to −0.001. Adam's first step is about the learning rate, against the gradient's sign.

A small model trained on these frames fell from 1.29 to 0.35. On held-out frames, the stage 1 model scores about 0.0004.

Give the model the noisy image and let it output the clean image x_hat. The loss is a normal MSE: mean((x_hat − x)²). This is a real method.

The model outputs the slope v. We get the image with x_hat = noisy − t·v. The loss is mean((v − (eps − x))²). Both ways end in the same picture.

The miss in x is t times the miss in v. So the loss in x is t² times ours. On frame 326, t = 0.60 and the ratio is 0.36.

At t = 0.1 the x-loss counts a frame 0.01 times as much as the v-loss. Ours weighs lightly noised questions more.

Play walks along the line in steps: z = z + (t_next − t)·v. A step needs exactly the slope. It is also the standard rectified-flow recipe.

A 2025 paper (JiT) shows that predicting x also works for pixel models. Here it is a choice of bookkeeping, not a different idea.

There is no real frame to hide, so play starts from torch.randn. This picture sits at t = 1.0. Its mean miss against the real frame 326 is 94.8 out of 255.

The model sees the noise, the past 15 frames and the key. It outputs v1. The slope picture looks like noise, but it holds the way back.

Play steps by t_next − t = 0.5 − 1.0, so z = z + (t_next − t)·v. Half of the noise is removed. Nobody adds noise. The miss falls to 48.2.

The same model gets the half-clean picture and t = 0.5. It outputs v2. This is a new question, at a new noise level.

Step by 0 − 0.5 along v2. That gives the new frame. Its miss against the real frame is 0.7 out of 255.

Training never walks. It hides frame 326 at one random t, gets one slope, grades it against eps − x, and stops. Then it adjusts the weights.

The left picture is a real frame. The middle one is the model's own redraw of it. They look the same, but the right picture shows the small differences, brightened 20 times.

Suppose training only ever shows real frames as the past. Then play hands the model a past it has never seen. Each small error becomes the next frame's input. This sketch is an idea, not a measurement.

Each past frame gets noise t_c, drawn between 0 and 0.35. These are real pictures of one window. At 0.001 the frame is clean. At 0.311 it is clearly noisy.

Play always uses 0.05 for the past. Training saw every amount from 0 to 0.35, so 0.05 is familiar. Each dot is one real t_c from this window.

The model gets t_c with each past frame. So it can tell a clean past from a noisy one. The idea is that it then relies less on fine detail. This is not measured.

Take 16 frames in a row from one game: frames 319 to 334. Under each frame is the key that produced it. Frame 326 has key 1, a tap.

Each frame is noised a little, by its own t_c from 0 to 0.35. This is the past as play will see it. It is never perfectly clean.

Each frame is noised by t_n, a random number from 0 to 1. Noisy position 9 is built from real position 9 itself, not from position 8. The exact eps is kept.

The clean value uses t_c = 0.311. The noisy value uses t_n = 0.794. The noise of the clean copy is thrown away. The eps of the noisy copy is kept for the answer.

The two lines with mix are the same formula. They differ in the job. Clean is the past. Noisy is the question.

The call passes clean, t_c, noisy and t_n. The model needs t_n because the right slope changes with t. It needs t_c to know how far to trust the past.

The past frames get t_c = rand * 0.35. The blue band shows the range. The dots are the 16 real t_c values of the window on the last page.

When play pushes a frame, it uses 0.05. That sits inside the blue band. So the model has met this noise level in training.

The noisy copy gets t_n = sigmoid(randn). Most values are medium. Half of them fall between 0.34 and 0.66. Only 1.4% are below 0.1.

Play starts at pure noise and removes noise in two passes. Nobody adds noise at 0.5. The picture simply is at 0.5 after pass 1.

The 0.60 on earlier pages is one dot in the bottom left box. The range 0 to 0.35 belongs only to the past. The range 0 to 1 belongs only to the frame being drawn.

Each row is a frame that reads. Each column is a frame that may be read. The order is C0 to C15, then N0 to N15. Grey means blocked.

The blue triangle is the clean copy. It reads itself and the clean frames before it. This is exactly what a push does in play.

Each orange row reads the clean frames before it. It also reads its own noisy frame. This is exactly what a draw does in play.

Noisy 9 reads clean 0 to 8 and itself. It must not read clean 9. That is the same frame, only lightly noised, so it would give away the answer.

That is 26.6% of the grid. The rule is the function can_read in world.py:11-20. The other 73.4% of pairs get zero share.

Clean C0 to C15 and noisy N0 to N15 sit in one list. Each frame has 256 squares, so each frame is 256 rows of 768 numbers. One window is 32 × 256 = 8,192 rows.

All 8,192 rows go through the 16 blocks together. The grid from the last page decides who reads whom. The shape does not change.

The model uses x[:, T:], the last 16 frames. The clean half only served as memory. It gets no output and no grade. That is world.py:84-96.

The head gives 256 × 192 numbers per frame. They fold into a 3 × 128 × 128 picture of slopes. Each one is graded against eps − x.

Every noisy frame has its own t_n. So the 16 questions ask about 16 different noise levels in one pass. Play draws at two levels, 1.0 and 0.5.

In the real run one batch is 8 windows of 16 frames. The model guesses a slope for every pixel. The loss averages all the misses into one number.

The numbers on steps 1 to 4 come from a small demo model: 758,848 weights, one 4-frame window.

First zero_grad forgets the last step. Then loss.backward() gives every weight its own gradient. A gradient says which way the loss moves if that weight goes up.

Follow one weight of the demo model: out.weight[0,0]. Its gradient is +0.0000469, so raising it would raise the loss.

All gradients together have one length. If that length is above 1.0, they are all shrunk until it is exactly 1.0. In stage 1, 2 of the 296 logged values went above the line. We log every 100th step.

Each weight moves against its gradient. Our demo weight goes from +0.0000000 to −0.0009998. The demo learning rate was 0.001 with no warmup, so the first move is about that size. The real run uses 0.0002. AdamW uses betas 0.9 and 0.95 and weight decay 0.01.

The first gradients are junk, so the learning rate starts at zero. It climbs in a straight line to 0.0002 over 1,000 steps. Then it stays there.

Next window, next noise, same four moves. The demo model goes from 1.29 to 0.35 in 1,500 steps. The real model reaches 0.00055 at the last logged step, 29,500.

Under autocast the maths runs in bf16, which is faster on the GPU. And torch.compile wraps the loss once. Neither changes the idea above.

In stage 1 both copies start from the same random weights, so nothing comes from a finished model. The real numbers on this page come from a small demo model of 758,848 weights. The copy gets no gradients, so it never learns by itself.

The model takes its AdamW step. Then each shadow weight moves a small part of the way toward the model weight. The rule is ema = d * ema + (1 - d) * model.

The shadow was at the old value. The model weight has moved on. The shadow takes only 2.2% of the distance and stays close to where it was.

At step 1 the shadow moves 82% of the way, so it can leave the random start. In the 600-step demo, d only reaches 0.985. In the real run d reaches 0.9995 near step 18,000.

The grey line is the model weight. It wobbles because each step uses one random batch. The blue shadow follows a little behind and stays smooth.

At the end of training we save both copies. The EMA file is the one we load to play the game. The next page shows why.

Every AdamW step is pushed by one random batch. So the weights keep jumping around a good spot. Look at the second half of a small demo run: 3,000 steps of the 758,848-weight model. The grey model loss on 64 held-out windows zig-zags.

The blue shadow copy goes smoothly through the middle of the zig-zag. Here it beat the model at 24 of the 31 checks in this zoom. In this short run d is 0.997, so it averages about the last 334 steps. In the real run d is 0.9995, about 2,000 steps.

On the whole run the two lines look the same. But from step 50 to 500 the shadow lags behind a model that still improves fast.

After step 1,550 it is a little better. Its loss jumps about two thirds as much between checks.

On the small run the gain is tiny, 0.4082 against 0.4088. On our full stage-1 run it is clear. At step 28,000 the model scores 0.00074 and the shadow scores 0.00039, about half.

The loss we train on is the model's. The weights we keep, play and check are the shadow's. The shadow never gets gradients, so it costs one extra copy.

The small model has 2 blocks and width 128. That is 758,848 learned numbers. The real model has 172,518,848, so the small one is 0.44% of its size.

We train it on one window of 4 real frames from a training game, again and again. The loss falls fast from 1.29 and then stalls near 0.35.

We hide frame 326 behind noise at level 0.6. Then we ask the model for the clean frame. The left picture is the real frame, and the second is the noisy one it sees.

At step 0 it outputs zero, so we see the noisy frame back. At step 1,500 the scene shows, but it is still covered in noise. The miss is the average gap per pixel and colour, 33 on a 0 to 255 scale.

Our final stage-3 model answers with a clean frame. Its miss is 0.20 out of 255. Its loss on this question is 0.00013, against 0.35 for the small one.

We trained the full 172.5M model from random weights on one H100. It ran for 121 minutes at about 0.246 seconds per step. That gave 29,525 steps, using 56.1 GB of memory.

The loss starts at 1.28, the same as the small model. At the last logged step it is 0.00055, about 2,300 times lower. The line wobbles because every step uses a new batch.

Every 2,000 steps we score windows from games the model never trained on. These dots fall smoothly. They sit on or a little above the training loss, which is what we expect for games it never saw.

The dashed line is the shadow copy on the same windows. At step 28,000 the model scores 0.00074 and the shadow scores 0.00039.

For every new frame, push the 15 past frames through the 16 blocks again. Each frame is 256 rows. So that is 15 x 256 rows, plus 2 passes of 256 rows for the new frame.

A finished frame never changes. So we save its k and v once, on the shelf. Then each new frame costs 2 passes to draw and 1 pass to save: 3 x 256 rows.

A frame is 128 x 128 pixels. Squares of 8 pixels give 16 x 16 = 256 squares, so 256 rows. 15 frames give 15 x 256 = 3,840 rows.

Every block saves k and v for those rows, each 768 numbers wide. The shelf is the kept k and v of all 16 blocks. That is 16 x 3,840 x 768 for k, and the same again for v.

Each block's 256 new rows read 3,840 rows from that block's part of the shelf, plus their own 256. That is 4,096 rows, inside attention in every block. No past frame is ever added to the noise as pixels.

The shelf is the kept k and v of all 16 blocks. Draw it as 16 layers, one per block, and 15 columns, one per past frame. Each cell holds that block's k and v for the 256 rows of that frame. One block's part of the shelf has k and v of shape [B, 12, 3840, 64]. Blocks are counted from 1.

First we add 0.05 noise to the pixels of the frame. Then one pass runs through the 16 blocks. Each block writes its k and v into a new column, and the oldest column drops out.

The noise goes on the picture, never on k or v. The head output of this pass is thrown away.

We start from pure noise. Pass 1 goes from t = 1.0 to 0.5, and pass 2 from 0.5 to 0.0. Both passes read the shelf. The shelf does not change during a draw.

Pass 1 asks the model for a slope. We take half a step back along it: z minus 0.5 times v. Pass 2 does the same from half noise.

No new noise is added while drawing. The pixel miss falls from 94.8 to 48.0 to 0.72 out of 255. In stage 1 there is one pass and the model never draws. In stages 2 and 3 the EMA copy redraws two frames of each window.

The finished frame has not run through the blocks yet. So one more pass makes its k and v. Pass 2 only saw a half-noisy picture.

In training the past gets a random noise level from 0 to 0.35. Our 0.05 sits inside that range. Streaming costs 3 passes of 256 rows.

It draws the score digit as pixels, like the pipes. A separate pixel check ends the game. The bird counts as found with 3 orange pixels, and 3 missed frames in a row end the run. The play checks draw with 4 passes, these pages show 2.

The next part follows one real look-up through the shelf with real numbers.

The shelf holds the last 15 frames. The oldest is at position 0 and the newest at 14. The frame being drawn is at 15, as streamer.py:24 says.

Each frame has 256 squares, so each position is used for 256 keys. Click any frame to follow it.

Move the slider by one. The oldest frame leaves, and every other frame moves down by one position. Frame 318 goes from position 7 to 6, 5, 4 and so on.

Row B saves k already rotated, once, when the frame was saved. It was the frame in hand then, so it was rotated by 15. It is never rotated again.

Row A saves k plain and rotates it at each use. So m − n is always the true number of frames back.

Inside the block, block.py:35-36 would rotate only the 256 new keys. With pos=None the block skips it and hands k back plain at line 41.

Then attention, called at line 37, glues the shelf on and rotates all 4,096 keys with pos_all.

This is _pass for one frame, batch size 1. The yellow rows are the ones that matter for this trap. Every pass, 4,096 keys are rotated in each block.

We push 15 real frames into an empty shelf. Then we draw frame 313 with plain keys, and again with keys rotated at push. Both miss the real frame by about 0.7 out of 255.

The shelf was still filling, so the rotated keys carry 0 to 14. Those are the correct offsets for this first draw. The two designs are the same maths here, so one frame cannot tell them apart.

The bars show how much the frame being drawn looks at each saved frame. It is our stage-3 model on real frames, drawing frame 326 after 28 pushes. We look at block 9 of 16, counted from 1.

Plain keys: it looks most at itself and the newest frames, then less and less. We read this as how it sees the bird's speed. We did not test that.

The window has slid by now. Almost every saved key was rotated by 15 when it was saved, and the two oldest by 13 and 14. So they look about 0 frames back. The bars are flat, and old and new look the same.

Now each drawn frame goes back on the shelf, as in real play. Plain keys follow the real game. Rotated keys draw a bird that does not move. The window slides, and that is the whole trap.

The line is the bird's height in each frame. Plain keys stay on the real line. Rotated keys stay at row 59 to 60 for all 30 frames.

A frame is 128 × 128 pixels. We cut it into squares of 8 × 8 pixels, 16 across and 16 down. That gives 16 × 16 = 256 squares. Each square becomes one row of 768 numbers.

The memory holds the last 15 frames. Each frame is 256 rows, and the frames sit one after the other. So 15 × 256 = 3,840 rows. We call this the past table.

The 15 frames go through the 16 blocks in one batch. Live play pushes them one at a time. It makes the same v, and the same k before the turn by position. Each block keeps k and v, each 3,840 × 768, which is 2,949,120 numbers.

Every block keeps its own pair, so there are 16 shelves. For k alone, 16 × 3,840 × 768 = 47,185,920 numbers. The tables for v have the same size.

Add k and v: 47,185,920 + 47,185,920 = 94,371,840. Count it another way: 15 frames in 16 blocks make 240 saved pieces. Each piece is a k and a v of 256 × 768, so 480 tables.

The new frame is only 256 rows. Block n reads its own 3,840 saved rows and these 256 new rows. So 3,840 + 256 = 4,096 rows, each of 768 numbers.

These pictures read the 15 past frames in one batch. Each frame gets a little noise (t = 0.05) and is cut into 256 squares. 15 × 256 = 3,840 rows. Live play pushes one frame at a time and builds the same tables.

The 3,840 rows enter block 1. Block 1 makes q, k and v for every row, 3,840 rows each. It puts k and v on shelf 1. That is what "saved" means. These k pictures are turned by position. Live play stores k plain (page H3).

Block 1 also runs attention and its MLP on the past table, so the table changes. The changed table enters block 2. Block 2 makes its own k and v, with its own weights, for shelf 2.

Every block fills one shelf. So 16 shelves × 15 frames = 240 saved pieces, each 256 rows. Shelf n holds the k and v that block n made from the table entering block n. The table after block 16 is never used.

The new frame is a table of only 256 rows. Here it is the half-noisy frame (t = 0.5, key 0) that enters draw pass 2. Block 1 makes its own q, k and v. K is shelf 1's k with the new k below it, 4,096 rows, and V is built the same way. This stacking is the look-up.

Block 2 reads shelf 2, block 9 reads shelf 9, and so on. Each shelf was made by the same block in run 1. So the past enters the new frame 16 times, once in each block.

After block 16, the head turns the 256 rows into pixels: the new frame. Live play then pushes it at once, in one pass at t = 0.05. Its 256 rows join every shelf, and the oldest frame's 256 rows leave.

The 15 past frames make a tall blue table of 3,840 rows. The new frame makes a short red table of 256 rows. It is half noise (t = 0.5). Both are drawn to scale. Nothing from the past is added to its pixels.

These tables come from one batch read of the 15 frames. Live play pushes them one at a time. Block 9 kept k for matching and v for content, 3,840 rows each. Its k is turned by position here; play stores it plain (page H3).

The new table reaches block 9. Its 256 rows make q (the question of each square), k and v. The saved rows go on top of the new rows, so K and V have 4,096 rows. This stacking is the only place the past enters.

Take row 132 of q, the bird's square. Its 64 numbers are compared with all 4,096 rows of K, and each score is divided by 8. Softmax turns the scores into shares that add up to 1.

Each share belongs to one square of one frame. Drawn back on the pictures, the biggest sit on past frames 15 and 14. They are at square 131, next to the bird's square 132. Past frame 15 is the newest of the 15.

Each of the 4,096 v rows has 64 numbers. Multiply each row by its share, then add up all the rows. The five biggest parts are shown. The sum is the bird's new 64 numbers for this head.

All 12 heads do this. Their 12 × 64 results join into 768 numbers, pass through proj, and are scaled by the gate. The result is added to the bird's row, so the new table now carries the past. Block 10 repeats this with its own shelf.

After stage 1 the held-out loss is tiny. But the loss always reads a real past. In play the model reads its own drawings. So we let it play and measure.

We find the bird as the orange pixels and take their average row. The model draws 300 frames with the real keys. Each frame, we compare its bird row with the real bird row.

We pick 256 moments where a new pipe enters on the right. The model starts 4 frames early and draws 5 frames. We count green pixels in the yellow strip. A pass needs 30% of the real green. A short clip may not show a failure, so we test many moments. Stage 1 and stage 3 are two models of ours, explained later.

In a free dream an auto-pilot taps toward the gap, so no real key helps. The dream runs all 1,000 frames. The loss frame is the first time the bird is missing for 3 frames. A wall is a pipe with no gap of 12 sky rows.

A number alone means little. The real game is the target. By definition its bird miss is 0 and its pipe score is 100%. Survival is capped at 1,000, so more is better.

We saved stage 1 at steps 6,604, 14,169 and 29,525. Each bar is one checkpoint, with the first in grey. Green means better than the checkpoint before. Red means worse. Each bar is one run, so small changes may be noise.

Later checkpoints have fewer wall frames and a smaller bird miss. At 6,604 the bird misses by 2.02 px. At 29,525 it misses by 0.135 px. But the bird is found less and less.

These are real dreams of the final checkpoint. Green frames still show the bird. The dream runs on after the bird is lost, with pipes only. The loss never showed this.

The bird is found in only 6% of frames 101-200. A new pipe appears after 4 own frames in 84.4% of cases, and the real game gives 100%.

Each training window has 16 real frames. Each frame is asked from the real frames before it, lightly noised. It never reads a picture that it drew itself.

At the start the 15 past frames are real. But each frame it draws becomes part of the past. After 15 frames, the whole past is its own work.

Real frame 8 and the model's drawing look alike. The difference, times 8, shows only a few bright pixels. The mean miss is under one level out of 255.

The next frame is drawn from a slightly wrong past. Its own miss adds to the old one. So the bird height miss grows with every 100 frames of play.

This is one real window of 16 frames with its keys. Stage 1 uses it as it is. Stage 2 begins with the same window.

We pick s0 between 1 and 14. Here s0 is 8. The EMA reads frames before 8 and draws frame 8. Then it draws frame 9. It uses 4 passes and no gradient.

The clean copy is the past. Its frames 8 and 9 are now the EMA's drawings. The noisy copy and the targets stay real. So the model must still draw the real frame.

In loss.py the clean copy reads xc instead of x. In loop.py the EMA fills xc. The EMA only draws the past. The target stays the real frame, so nothing is copied from the EMA.

Stage 2 starts from the stage-1 EMA, not from random numbers. The learning rate halves and the warmup is shorter. The bird counts five times in the loss.

This frame has 16,384 pixels. The bird is only 14 orange pixels, under 0.1% of the frame. A model can miss the bird and still have a tiny loss.

We zoom into 28 by 28 pixels around the bird. Each square is one pixel with its real colour. The black squares pass the orange test from rollout.py.

The bird edge and the sky next to it matter too. So bird_mask grows the orange pixels by 3 pixels each way. The mask now covers 141 pixels.

Each miss is multiplied by 1 plus 4 times the mask. So a miss near the bird counts five times. The bird share of the loss rises from 0.86% to about 4%.

Grey is our stage 1. The next four bars are stage 2 after 2,581, 5,501, 11,348 and 17,181 steps. Green means better than stage 1. Red means worse.

Survival goes from 118 to 196 frames. Wall frames drop from 0.84% to 0.006%. The bird is found in 18% of frames 101-200, up from 6%. Bird miss counts only frames where both birds are found.

Survival is highest at step 11,348 with 273 frames. Walls are lowest at the end. With one run and 32 dreams, small gaps may be noise.

In these free dreams the bird stays for the frames shown. The four saved dreams last 949, 300, 895 and 41 frames. The loss would never tell us this.

New pipes appear 80% of the time, down from 84%. Stage 2 is lower at every start: by 1.1, 5.1 and 4.3 points. Stage 3 targets this dip.

A birth window is 16 real frames. Its last frame is the first frame with a new pipe at the right edge. We find 7,162 of them in the training games.

In frame 14 the right edge has no green. In frame 15 it has green. Check B picks its moments with the same rule.

Birth windows are 2.60% of all windows. Stage 3 uses them in 25% of the batches. So the model sees a pipe being born about 10 times more often.

In a birth window s0 is 13. The EMA redraws frames 13 and 14. Then the model must draw the real frame 15, with its new pipe, from that past.

The loop chooses a pool of windows with probability 0.25. In birth windows it sets s0 to 13. The loss and the bird weight stay as in stage 2.

These are the five numbers of stage 1. Every other cell is coloured against them. Green means better than stage 1. Red means worse.

Stage 2 cuts wall frames to 0.006% and lifts survival to 195.5. The bird miss and bird found also improve. Bird miss counts only frames where both birds are found. But new pipes drop from 84.4% to 80.1%.

New pipes appear 97.3% of the time. Survival is 201.5 frames, the best of the three final models. Stage 2 at step 11,348 reached 273. Against stage 2, walls rise to 0.12% and bird found falls to 16%.

These are stage 3 dreams. In dream 1 the bird is lost at frame 80. In dream 2 it stays past frame 500. The median over 32 dreams is 201.5 frames.

All three stages are public at huggingface.co/Prahaladha/flappy-world-model. The bird in long real-key runs is still lost too often.