QMIND Research · Lecture 5 of 7

Our research, and how diffusion works

Why we picked neural rendering of games, how a denoiser turns static into a frame, and exactly how our game gets into the network.

PART A · NOW, OUR PROJECT

Okay, that was cool. Now let's talk about our project.

Deck 2

One neuron learns

Deck 3

Transformers

Deck 4

The frontier at datacentre scale

So where, in all of that, can a student team do research that actually matters?

Why not LLMs

The LLM frontier is a capital race we can't enter

>$700B

planned 2026 capex: Alphabet, Amazon, Microsoft, Meta

~90%

of notable 2024 AI models came from industry

~5 mo

doubling time of training compute

So: pick a question where scale isn't the answer.

Sources: Sherwood News, Apr 30 2026 (2026 capex plans) · Stanford HAI, AI Index 2025

~1 min

This is a log scale: every tick is ten times the last.
Our project has $10,000 total: $5k cash, $5k AWS credits from Chatforce.
The four biggest spenders plan more than $700 billion of capex in 2026 (as of April 2026). That's about seventy million times our budget: almost eight orders of magnitude.
And it shows in who builds models: the AI Index 2025 says nearly 90% of notable 2024 models came from industry; training compute doubles about every five months.
Conclusion: don't fight on the axis where money wins. Choose a question where cleverness and exact measurement beat scale.

Intuition: LLM research isn't closed to students, but the saturated part of it is decided by compute, and the open parts are crowded with thousands of other groups.

If asked: "Isn't capex mostly datacentres, not research?" Yes, it's infrastructure, not one training run. The point is the slope: whoever owns the compute sets the pace on scale-driven questions. Academia still leads in highly cited papers, per the same AI Index, so ideas still matter.

The map

Where can a student team do real research?

01

Interpretability of small models

Reverse-engineer what a small network computes.

02

Efficient fine-tuning

Adapt pretrained models cheaply: LoRA, quantization.

03

RL for small domains

Agents for games, puzzles and control tasks.

04

Robotics, sim-to-real

Train in simulation, run on real hardware.

05

Scientific ML

Learned surrogates for physics, chemistry, biology.

06

Generative models of structured worlds

Neural rendering, world models.

~1 min

These are the areas I actually weighed. All six are real, active fields with good student papers.
Interpretability: cheap compute, deep questions (it's what our QMIND music-transformer proposal does). Hard part: "did we understand it?" has no exact grader.
Efficient fine-tuning: very practical, very crowded. Hard to be novel.
RL for small domains: great for learning, measurable rewards. Novelty is the hard part unless the environment is new.
Robotics: exciting, but hardware costs money and time, and evaluation in the real world is slow.
Scientific ML: high impact, but you need domain experts and their data.
Generative models for structured worlds: neural rendering, world models. This is where we landed.
Next slide: how they score.

Intuition: a fair map matters. We didn't pick diffusion because it's trendy; we picked it because it wins on the factors we can control.

Scoring on deck 1's ROI factors

Scored on our constraints, one area stands out

Scores are Matt's judgement on a 1–5 scale for a student team with $10k, not measurements.

~1.5 min

Five factors from deck 1's ROI pitch. Bigger dot = better for us.
Rows: the six areas, plus frontier LLM work as a reference row.
Near the frontier: will people care? Neural rendering sits right next to GameNGen and Genie.
Compute fits: 64×64 frames train on a single GPU. LLM pretraining scores 1.
Measurable: our skinner gives an exact answer for every frame. Interpretability scores low here: no answer key.
Open question: can we find something nobody knows? Fine-tuning is crowded.
Client fit: Chatforce wants exactly this (live re-theming) and funds it.
Totals. Neural rendering leads clearly; fine-tuning and small-domain RL tie for second.

Intuition: the winner isn't best at everything; it has no weak column. Every other area has a 1 or 2 somewhere.

If asked: "Isn't this rigged?" Partly subjective, yes, so argue the cells. Changing any single score by 1 doesn't change the winner; the gap is 6 points.

Second pass on deck 1

Why neural rendering of games won

Small compute. 64×64 frames: 12,288 numbers each.
Exact ground truth. The skinner renders the right answer for any combination.
A real open question. Where does compositional generalization break?
A paying client. Chatforce wants live re-theming.
Close to hot work. GameNGen, DIAMOND, Genie.

~1 min

Five reasons, each tied to a column of the matrix.
This is our actual resolution: 64×64. Every pixel you can see is a pixel the model predicts. Small images mean fast iterations: many runs per day, not one per month.
The killer feature: the skinner is a deterministic function, so for any frame and any theme we know the exact right image. Most generative-model research grades by eye or with fuzzy metrics.
The question isn't "can diffusion render a game?" (GameNGen showed it can). It's "where does it break on combinations it never saw?" Nobody can prompt an agent loop to that answer.
Watch the frame: same geometry, different themes. That's the client's product in miniature: re-theme a running game.
And it's adjacent to the hottest work in generative AI, which is next.

Intuition: we deliberately made the problem small enough to afford and exact enough to measure, without making it trivial.

If asked: "Isn't 64×64 a toy?" It's a controlled setting. Physics uses toy models on purpose. The question (compositional generalization) doesn't need megapixels, and the grid can grow once the small version works.

Close to hot work

We sit next to the hottest work in generative AI

Alonso et al. NeurIPS 2024 · Valevski et al. ICLR 2025 · Decart/Etched · DeepMind Genie 2, 3, Project Genie · Microsoft Muse (Nature 2025)

~1 min

Two years of neural game engines.
DIAMOND (May 2024, NeurIPS spotlight): a diffusion world model good enough to train RL agents inside; later a playable CS:GO model at about 10 fps on one RTX 3090. GameNGen (Aug 2024): Doom simulated by a fine-tuned Stable Diffusion 1.4, 20 fps on one TPU, 4 DDIM steps.
Oasis (Oct 2024): a playable Minecraft-like world at 20 fps. Genie 2 (Dec 2024): 3D worlds from a single image.
Muse/WHAM (Feb 2025, Nature): Microsoft's gameplay model trained on Bleeding Edge. Genie 3 (Aug 2025): real-time 720p at 24 fps, consistent for a few minutes.
Project Genie (Jan 2026): Genie 3 as a public prototype in Google Labs for AI Ultra subscribers.
Us: we take the rendering half of this story and grade it exactly. These systems learn gameplay and rendering together and are judged by eye or against recorded frames; we isolate rendering and have an answer key for combinations never seen.

Intuition: world models = next frame given past frames and actions. We fix the dynamics (the engine owns gameplay) so we can measure rendering generalization cleanly. World modelling is our stretch goal.

If asked: "Will Google just solve this?" They're solving scale. Nobody has published a clean map of where compositional generalization breaks, because their games have no answer key.

The research question

The question: where does generalization break?

Hold out combinations, never values: every theme and material is seen, some pairings never are.

H1 · Coverage

Held-out accuracy jumps at a threshold of grid coverage.

H2 · Factor type

Pixel-aligned channels compose better than a global theme code.

H3 · Interaction

Non-local effects (shadows, reflections) generalize worse.

H4 · Scale

More factors, worse; rarest combinations fail first.

~1 min

Every cell is a real render from our toy skinner: 5 themes across, 5 wall materials down.
This is the whole grid the skinner can produce. The real grid adds lighting as a third axis.
Hold out 20% of the cells. Each held-out cell's theme and material both appear elsewhere in training; only the pairing is new.
Then the model renders the held-out cells. Illustration only: some come out right (green), some leak the wrong palette (rose). Mapping which ones fail, and why, is the result.
H1: does accuracy rise smoothly with coverage or jump at a threshold?
H2: channels like material ID arrive per pixel; theme arrives as a global vector. We predict per-pixel factors compose more easily. You'll see why in Part D.
H3: shadows and reflections couple distant pixels; we predict they generalize worse.
H4: more factors make the grid sparser; failure should start with the rarest combinations. Deck 6 designs the splits and sweeps.

Intuition: a model that memorizes can't pass, because the test combinations never appear in training.

Make-or-break

Three moments where the project lives or dies

Moment 1

A denoised image that isn't static

Moment 2

A held-out combination rendered right

Moment 3

A curve showing where it breaks

If moment 2 fails, the failure becomes the finding: we explain why.

What you get out of it

Five streams, and what each one teaches you

Stream	You build	You learn
Engine & rendering	ray-caster, map generator, channel outputs	ray casting, procedural generation, determinism
Skinning & data	theme space, skinner, dataset generator	procedural textures, parameter spaces, data at scale
Model & training	U-Net, conditioning, samplers	diffusion from scratch, architecture, training dynamics
Evaluation & analysis	holdout splits, metrics, classifiers	experimental design, statistics, honest claims
Infrastructure & MLOps	AWS training jobs, storage, tracking	cloud GPUs, pipelines, reproducibility

Everyone owns a piece of one stack: build the world, make the data, train the model, design the test.

~1 min

Deck 1 listed the streams; here's what each one actually teaches you.
Engine: a real ray-caster (the Wolfenstein/Doom trick: one ray per screen column). Determinism matters: same seed, same frame, forever.
Skinning and data: themes are data, not code. Procedural patterns, a parameter space you can sample, and a generator that makes hundreds of thousands of labelled frames.
Model: everything in Parts B–D of this deck, written by you, not imported.
Evaluation: the scientific core. Splits, PSNR/LPIPS, per-factor classifiers, seen-vs-unseen gaps, error bars.
Infra: deck 7. AWS credits, data storage, experiment tracking.
The point of deck 1's "you own the whole stack": every stream feeds the others; you'll understand the full system, not one box.

Intuition: pick a stream for what you want to learn, not what you already know.

The paper story

The paper we're aiming for, in three sentences

Setup

Neural renderers like GameNGen work, but nobody can say where they fail, because real games have no answer key.

Method

We build a game whose answer key is exact, and train on only some combinations of theme, material and lighting.

Result

A map of where compositional generalization breaks, and evidence for why.

CUCAI paperWorkshop submissionPlayable demoA negative result is still a result.

PART B

How diffusion works

Destroy a frame with noise, a little at a time. Then train a network to undo one little step.

Diffusion

Generative modelling means learning a distribution

Training data: samples from an unknown .

Learn , or something that lets you sample from it.

Sample: new points that look like the data but aren't copies.

A 64×64 frame is one point in . Real frames sit on a thin, curved surface: the data manifold.

~1 min

A 2D toy: each dot is one "image". Real images have 12,288 dimensions, but the geometry carries over.
The data: we never see , only samples from it. Here the data lies on a spiral.
Learning means building a model of where the probability mass is (the glow).
Generating means drawing new points from that model: blue dots land on the spiral but in new places.
The jump to images: a random point in pixel space is static. Real frames are a vanishingly thin subset: the data manifold. The whole game is landing on it.

Intuition: a generative model doesn't store images; it learns the shape of the space they live in.

If asked: "Why not just learn directly?" The normalizing constant (integral over all of pixel space) is intractable. Diffusion sidesteps it: it learns the direction toward higher probability (the score), which doesn't need the constant. That's Intuition 1.

Diffusion · forward process

Forward: add a little noise, a thousand times

: fresh Gaussian noise every step. grows from to over steps.

Shrink the image slightly, add a little noise.

By it is indistinguishable from pure noise.

Nothing here is learned. It's a fixed recipe.

Schedule: Ho, Jain & Abbeel, "Denoising Diffusion Probabilistic Models", NeurIPS 2020 (linear , T = 1000).

~1 min

Our running image: one frame, pixels scaled to .
One step: scale down by , add noise with variance . The two coefficients' squares sum to 1, so the total variance stays about 1 ("variance-preserving").
Running the actual recursion, step by step, with real Gaussian noise. At you can still see everything.
At fine texture is gone but the corridor's layout survives. Notice what dies first: detail, not layout.
At the signal coefficient is about 0.006: pure static.
Key point: no parameters here. The forward process is a fixed corruption we can run on any training frame for free.

Intuition: each step is tiny, so each step is easy to undo approximately. That's the whole bet.

If asked: "Why shrink by instead of just adding noise?" Without shrinking, variance grows without bound ("variance-exploding", the Song & Ermon family, which also works). Shrinking keeps everything at unit scale, which networks like.

Diffusion · closed form

You can jump straight to any noise level

signal weight

noise weight

cosine schedule

Cosine schedule: Nichol & Dhariwal, "Improved Denoising Diffusion Probabilistic Models", ICML 2021.

~1 min

The recursion has a shortcut.
Because a sum of Gaussians is Gaussian, a thousand small noise additions collapse into one: is just a blend of the clean image and one fresh noise sample. is "how much signal is left".
The schedule: (violet) falls from 1 to almost 0. The blue curve is the weight on the image, the grey one the weight on the noise.
At : mostly signal. The frame on the right is computed with the closed form, no loop.
At : mostly noise, layout still faintly there.
At : essentially pure noise.
Cosine schedule: the linear schedule wipes images out too early; the last few hundred steps are wasted on pure noise. The cosine schedule destroys information more evenly. Schedules are a design choice we can sweep.

Intuition: training never runs the chain. It picks a random , jumps there in one line, and asks the network to undo it. That's why diffusion training is cheap per example.

If asked: "Why is the closed form exact?" Each step adds independent Gaussian noise; scaled sums of independent Gaussians are Gaussian, with variances adding. Unroll and the coefficients multiply into .

Diffusion · the key trick

The trick: undo the noise one small step at a time

forward

: fixed

reverse

: learned

one giant jump: not learnable

For small , each reverse step is almost Gaussian, so the network only has to predict its mean.

Sohl-Dickstein et al., "Deep Unsupervised Learning using Nonequilibrium Thermodynamics", ICML 2015 · Ho et al., NeurIPS 2020.

~1 min

Six snapshots of the same chain, clean on the left, noise on the right.
These are real closed-form samples at .
Forward arrows: the fixed corruption. Easy.
Reverse arrows: what we learn. Each one says "given this slightly noisier image, what did the slightly cleaner one look like?"
Why it's learnable: if a step adds only a little noise, the reverse step is also approximately Gaussian (a classic result going back to Feller; Sohl-Dickstein et al. 2015 built diffusion on it). A Gaussian is fixed by its mean, and the network predicts that mean.
The giant jump from noise to image in one step is a wildly non-Gaussian, multimodal distribution. A network trained with MSE on it would output the blurry average (Intuition 3).

Intuition: you can't sculpt a statue in one blow, but you can always remove a little marble.

If asked: "Is it really 1000 network calls to sample?" In the original DDPM, yes. DDIM and distillation cut that to tens or even one or a few. More on that soon.

Diffusion · the step function

The denoising step is one function

noisy frame

noise level

conditioning

predicted noise

clean guess

~1 min

Here's the one function everything rests on. Say it slowly: given a noisy image, the noise level, and the conditioning, predict the noise.
Input 1: the noisy frame .
Input 2: how noisy it is. The same blob means different things at (big structure unknown) and (just grain).
Input 3: conditioning. For us: depth, surface type, material, UV, light per pixel, plus the theme. Part D is all about this.
Output: an image-shaped guess of the noise that was added. Same shape as the input: 64×64×3.
Predicting the noise is equivalent to predicting the clean image: rearrange the closed form. Subtract the predicted noise, rescale, and you have . At that guess is blurry: it's the average of every frame consistent with what's visible (Intuition 3).

Intuition: the network is a denoiser that's been told how much noise to expect. Everything else (training, sampling, guidance) is a loop around this one call.

If asked: "Why predict noise rather than the image?" Both work; they're algebraically linked. Ho et al. found -prediction trained better: the target always has unit scale. Newer work also uses "v-prediction", a blend of the two. Our code can switch between them.

Diffusion · training

Training: add known noise, then predict it

1 sample a frame

with its channels

2 pick

at random, draw

3 jump:

4 predict

5 loss, backprop, update

Plain mean-squared error. The target is the noise we added, so we always know the right answer.

~1 min

The training loop has five moves. It's simpler than the MNIST loop in deck 2.
Grab a training frame and its structural channels from the dataset.
Pick a random noise level, uniformly from 1 to 1000, and a random noise image.
Jump straight to with the closed form. No chain.
Ask the network for its guess of .
MSE between the true noise and the guess. Backprop, Adam step. This is Ho et al.'s "simple" loss: it drops the weighting terms of the full variational bound and works better in practice.
Repeat a few hundred thousand times. The loss curve flattens but never hits zero: at high noise the noise is genuinely unpredictable from the image.

Intuition: self-supervised: the data labels itself. We made the noise, so we know the answer.

Common mistake: expecting the loss to go to zero, or comparing losses across noise levels. High- examples always have high loss; look at samples and held-out PSNR, not just the loss.

Diffusion · training

The whole training step fits on one screen

def train_step(model, x0, cond, theme):    B = x0.shape[0]    t = torch.randint(0, T, (B,))      # noise level    eps = torch.randn_like(x0)          # the answer    ab = alpha_bar[t].view(B, 1, 1, 1)    xt = ab.sqrt()*x0 + (1-ab).sqrt()*eps    inp = torch.cat([xt, cond], dim=1)  # 3+13=16    theme = drop(theme, p=0.1)       # for CFG    eps_hat = model(inp, t, theme)    loss = F.mse_loss(eps_hat, eps)    opt.zero_grad(); loss.backward()    opt.step()    return loss

Concatenate

Structural channels ride along with the noisy image. Part D.

Global inputs

and theme go in separately, as vectors.

Same as deck 2

Loss, backward, step. Only the data recipe is new.

~1 min

The previous slide as PyTorch. Lines 2–6 are steps 1–3: random , random noise, closed-form jump. Note it's per-example: each frame in the batch gets its own noise level.
The noisy RGB (3 channels) and 13 structural channels are stacked into one 16-channel tensor. This is our planned channel layout.
Theme and timestep go in as separate arguments: they're global, not per-pixel. Dropping the theme 10% of the time trains the unconditional mode needed for guidance (coming up).
The last four lines are exactly deck 2's loop. Deck 7 explains what backward() and opt.step() do under the hood.

Intuition: all the cleverness is in the data recipe (how is made) and the architecture. The optimization is ordinary.

If asked: "Why randint(0, T) and not 1..T?" Zero-indexed arrays: alpha_bar[0] corresponds to . Off-by-one bugs in the schedule are the classic diffusion bug. Write a unit test.

Diffusion · sampling

Sampling: start from noise, step back to a frame

Start: . For :

Remove a bit of the predicted noise, then add a little fresh noise .

Original DDPM: 1000 network calls per image.

~1 min

Now run the step function in a loop.
Start from pure noise. At each step, call the network, subtract a scaled amount of its predicted noise, then add a smaller amount of fresh noise (with ). This is DDPM's "ancestral" sampler.
Watch: early steps fix the big shapes (walls vs floor, light vs dark).
Middle steps: edges, the enemy, the doorway.
Last steps: texture and grain. Done. 1000 network calls in the original paper; we're showing 50 here.

Honesty note: this animation uses a stand-in denoiser (the true frame, blurred in proportion to the noise level) instead of a trained network, so you see the real sampler equations without waiting for training.

If asked: "Why add noise back in during sampling?" It corrects errors: if the network's previous guess was slightly off, fresh noise lets the next step re-decide. It's what makes DDPM a proper sampler of the learned distribution. DDIM drops it (next slide).

Diffusion · samplers

DDPM adds fresh noise each step; DDIM adds none

DDIM: same trained network, no fresh noise. Same start, same frame. Every time.

Skips steps safely: 10–50× faster.

For a game: same input, same frame, so less flicker.

Song, Meng & Ermon, "Denoising Diffusion Implicit Models", ICLR 2021.

~1 min

Same starting noise, two samplers, two runs each.
DDPM run A.
DDPM run B: same starting noise, but different fresh noise along the way.
The difference map (amplified 5×): the details differ. Fine for art, bad for a game, where it shows up as flicker frame to frame.
DDIM: predict , then re-noise it deterministically using the predicted noise itself. No . Same start gives exactly the same output.
DDIM uses the same trained network (no retraining) because the DDPM loss only depends on the marginals . That freedom lets you take big strides: 50 or fewer steps instead of 1000.
This is one of our three flicker fixes: fixed starting noise plus DDIM makes the renderer a deterministic function of its inputs. GameNGen used 4 DDIM steps.

Intuition: DDPM is a random walk downhill; DDIM follows the deterministic flow underneath that walk (an ODE).

If asked: "Why ever use DDPM then?" With a slightly wrong network, stochastic sampling can self-correct and sometimes gives better quality at many steps. DDIM's knob interpolates between the two.

Intuition 1

The predicted noise points toward the data

The score: an arrow toward higher probability.

At low noise the arrows hug the data; at high noise they only know roughly where it is.

Sampling = follow the arrows while turning the noise down.

Annealed Langevin dynamics: Song & Ermon, NeurIPS 2019 · score SDEs: Song et al., ICLR 2021.

~2 min

This is the most important intuition in the deck. Back to the 2D world.
The data: a spiral (green).
Noisy samples: blue dots scattered everywhere. That's what looks like at high .
The noise prediction, flipped and scaled, is the score: the gradient of log-density. It points from where you are toward where the data is more likely. Predicting "what noise was added" is predicting "which way is home".
The field depends on the noise level. High noise: arrows point roughly toward the middle of the mass (smooth, coarse). Low noise: arrows snap onto the nearest arm (sharp, local).
Annealed Langevin dynamics, running live: step along the arrows plus a little noise, starting at high noise and turning it down. The particles flow onto the spiral. This is a real simulation of Song & Ermon's sampler.

Intuition: the network learns a vector field over image space that points to the data manifold, at every blur level. Sampling is gradient ascent on log-probability, with noise to keep it honest.

Common mistake: thinking the network "draws" the image. It never does. It only says, at each point, which way is more image-like.

If asked: "Why does noise-prediction equal the score?" Tweedie's formula: for Gaussian corruption, . The MSE-optimal is exactly that conditional expectation.

Intuition 2

Intuition 2: coarse first, fine detail last

High noise decides layout; low noise decides texture. Diffusion is roughly autoregression over frequencies.

Dieleman, "Diffusion is spectral autoregression", blog post, 2024 · natural-image spectra fall off roughly as .

~1 min

Left: the network's clean guess at different noise levels. Right: why it looks like that, in frequency space.
Natural images have most of their power at low frequencies, falling roughly as (blue line). White noise has equal power at every frequency (violet line). At – the noise drowns almost everything: the best guess is a colour blob.
At – only the lowest frequencies stick out above the noise: the guess has the layout (walls, floor, the bright doorway) but no detail.
At – the noise floor has dropped below most of the spectrum: edges and bricks appear. The last steps only touch the highest frequencies: grain and texture.
So sampling builds the image coarse-to-fine, like an artist blocking in shapes before detail. For us: theme (palette) and layout are decided early; texture patterns late.

Intuition: adding white noise is a low-pass filter on what survives. Reversing it adds frequencies back from low to high.

If asked: "Does this matter for our project?" Yes: if a held-out combination fails, we can ask at what noise level it went wrong. Wrong palette means an early-step failure; wrong pattern means a late-step one. That's a diagnostic tool for "why".

Intuition 3

Intuition 3: one big jump lands on the average

The MSE-best single guess is the average of every plausible answer: a blur. Small steps commit one detail at a time.

~1 min

Why not train a network to go from noise to image in one shot?
Start from a few pure-noise points.
The best one-shot guess under squared error is the mean of all data points consistent with the noise. From pure noise, that's the mean of the whole dataset: the middle of the spiral, which is not on the spiral at all.
Take 30 small steps instead (DDIM with the exact optimal denoiser). Each step makes a small commitment, and the paths curve onto the spiral, each to a different place.
Same in pixels: the one-step guess from pure noise is the average frame, a blur; the many-step sample is crisp.
The general lesson: MSE regression to a multimodal target gives the mean of the modes. Diffusion avoids it by making each step's target nearly unimodal.

Intuition: ask "which of a million possible images?" and the safest answer is the average. Ask "which way, a little?" and there's a clear answer.

If asked: "But distilled models do 1–4 steps?" Yes: consistency models and distillation train a student to match the many-step teacher's endpoint, not the blurry average. GameNGen distilled to a single step at 50 fps with some quality loss.

Intuition 4

Intuition 4: when does it just memorize?

Kadkhodaie et al., ICLR 2024

Faces: up to ~100 training images, samples are copies. At , two models trained on disjoint data generate nearly identical new faces.

Carlini et al., 2023

Over a thousand training images extracted from large text-to-image models.

Held-out combinations are our memorization test.

Kadkhodaie, Guth, Simoncelli & Mallat, ICLR 2024 · Carlini et al., "Extracting Training Data from Diffusion Models", USENIX Security 2023.

~1 min

Uncomfortable fact: the perfect minimizer of our training loss on a finite dataset only reproduces training images. Its score points to the nearest training example.
Left: 8 training points, exact optimal score. Every sample collapses onto a training point. Pure memorization.
Right: the same sampler with a smooth model of the spiral, which is what a network learns when data is plentiful. Samples land anywhere along the curve, including where no training point exists. That's generalization.
Why networks generalize: they can't represent the jagged exact optimum, and their inductive biases (convolutions, smoothness) favour simple fields. Kadkhodaie et al.: with enough data, two networks trained on different images learn the same function. But it does happen: Carlini et al. pulled training images out of production models, especially duplicated ones.
For us, held-out combinations are this question in its sharpest form. A memorizer scores perfectly on seen cells and fails every unseen one. The seen-vs-unseen gap is one of our primary metrics.

Intuition: generalization in diffusion comes from the model being unable to fit the training set perfectly, in the right way.

If asked: "How much data is enough for us?" Unknown, and that's exactly the "data size" control sweep. Watch duplicates: identical frames make memorization much more likely.

Conditioning & guidance

Conditioning, and turning it up with guidance

One network, trained with dropped 10–20% of the time, gives both terms.

: sharper and more on-theme, less diverse.

Cost: two network calls per step.

Ho & Salimans, "Classifier-Free Diffusion Guidance", NeurIPS 2021 Workshop (arXiv 2022).

~2 min

Two classes now: class A (blue moon) and class B (violet moon).
Unconditional, : samples cover both moons.
Conditioned on A, : the score is computed from A only, so samples land on A. Conditioning just means "the arrows depend on ".
Classifier-free guidance: take the conditional arrow, subtract the unconditional one, and amplify the difference. You push away from "generic image" toward "specifically A". Train it by replacing with a null token some of the time.
: tighter on A, pushed away from B, less spread. That's the trade-off: fidelity up, diversity down.
Cost: two forward passes per step (batched together in practice).

Watch the notation: Ho & Salimans write . It's the same thing with shifted by one. Stable Diffusion's "guidance scale 7.5" uses the form on the slide.

For us: our structural channels already pin the layout down, so guidance matters mostly for theme adherence. Whether it helps held-out combinations or just sharpens seen ones is a cheap, interesting ablation.

Inside

· the U-Net

The U-Net: shrink to understand, grow back to place

Ronneberger, Fischer & Brox, "U-Net: Convolutional Networks for Biomedical Image Segmentation", MICCAI 2015 · sizes shown are our planned config.

~1 min

This is what actually is. U-Nets come from 2015 medical segmentation and became the standard denoiser in DDPM, Stable Diffusion and GameNGen.
Input: 64×64 pixels with 16 channels. A first convolution lifts it to 64 feature channels.
Encoder: each level halves the resolution and doubles the channels: 64→32→16 pixels, 64→128→256 channels. Watch the box on the input: the region one feature can "see" (its receptive field) grows from about 11 pixels to 33 to the whole frame.
Bottleneck, 16×16×256: every feature sees the entire frame, plus self-attention. This is where the network knows what the scene is: corridor, doorway, theme.
Decoder: upsample back, halving channels. It works out where things go at full resolution.
Skip connections: copy each encoder level's features straight across and concatenate them into the decoder. Without them, fine detail like a brick edge would have to squeeze through the 16×16 bottleneck and would come out blurry.
Output: 64×64×3, the predicted noise. Same size as the input image: denoisers must be image-to-image.

Intuition: the encoder zooms out to understand, the decoder zooms in to draw, and the skips hand over the fine print.

If asked: "Why not a transformer?" Diffusion Transformers (DiT) are now common at large scale. At 64×64 with a small budget, the U-Net's convolutional bias toward local structure is data-efficient, and it's what GameNGen used.

U-Net · building block

Every block: normalize, activate, convolve, add back

GroupNorm

Normalize each image's channels in groups (e.g. 32). Doesn't depend on batch size, so small batches are fine.

SiLU

: a smooth ReLU.

Residual

The block learns a correction to . Gradients flow straight through the skip, as in deck 3's residual stream.

~1 min

Every level of the U is a stack of these ResNet blocks. The DDPM U-Net (Ho et al.) used exactly this recipe: GroupNorm, a smooth activation, 3×3 convs, residuals.
The pipeline: norm, activation, conv, then again, with conditioning injected in the middle.
GroupNorm: BatchNorm averages across the batch, which breaks with small batches and makes train and test behave differently. GroupNorm averages within one image over groups of channels: same behaviour always.
SiLU (also called swish): times its sigmoid. Like ReLU but smooth, with a small negative dip. Smoothness tends to help diffusion training.
The amber arrow is where time and theme enter: per-channel scale and shift after the second norm. That's the AdaGN slide in Part D.
Residual: output = input + correction. At initialization the block is close to identity, so a deep stack trains stably. Same idea as deck 3's residual stream.

Intuition: each block nudges the features; conditioning decides which way to nudge.

If asked: "1×1 conv on the skip?" When the block changes the channel count (64→128), the skip path needs a 1×1 conv to match shapes. Otherwise it's the identity.

U-Net · the noise level

The timestep becomes a vector that reaches every block

~1 min

The network must know how noisy its input is. A single number is a poor input on its own: one neuron can't easily do different things at and .
So we expand into sines and cosines at many frequencies: the same sinusoidal trick the original 2017 transformer used for positions (deck 3's GPT learns its position vectors instead). Each row of the heatmap is one ; each column one frequency.
Pick : read off its row. Fast columns wiggle, slow columns barely move: a multi-scale code for "where on the schedule".
: a very different pattern. Nearby have similar vectors, so the code is smooth.
A small MLP (linear, SiLU, linear) turns it into a 256-dimensional embedding. The theme embedding is added here too.
Every ResBlock gets its own linear projection of that vector, into per-channel scale and shift. Every block knows the noise level.

Intuition: the timestep is a mode switch. At high noise the network acts like a layout sketcher; at low noise, like a texture polisher. The embedding tells each block which job it's doing.

If asked: "Why every block, not just the input?" Different blocks do different work at different noise levels; injecting everywhere lets each one adapt. It's cheap: about 80k parameters for the time MLP in our plan.

U-Net · attention

Self-attention at 16×16 lets distant pixels agree

Each of the 256 cells is a token: , , exactly as in deck 3.

A wall cell attends to the same wall across the frame: one pattern, one lighting.

Quadratic cost is why attention lives only at the lowest resolutions.

~1 min

Convolutions are local. Even with the U's downsampling, some things need direct global communication.
At the 16×16 level, treat each cell's 256-channel feature vector as a token. Queries, keys, values, softmax: deck 3's attention, with pixels instead of words.
The amber cell is on the left wall. Its strongest links go to other cells of that wall, even far away. That's how a brick pattern stays consistent across the frame, and how one light colours a whole room.
A floor cell attends mostly to other floor cells. (Weights here are computed from toy features, to illustrate.)
Cost: attention compares every pair of tokens. At 64×64 that's 4,096 tokens and about 16.8 million pairs per head; at 16×16 it's 256 tokens and 65,536 pairs, 256 times fewer.
So: convs everywhere, attention only where the map is small. DDPM used attention at 16×16; later work added 32 and 8.

Intuition: convs are good neighbours; attention is the group chat.

Link to hypothesis 3: shadows and reflections need exactly this long-range communication, which is why we suspect they generalize worse.

U-Net · our planned config

Our planned U-Net: about 16.5 million parameters

Level	Size	Channels	Blocks
Input	64×64	16 → 64	1 conv
Level 1	64×64	64	2 res
Level 2	32×32	128	2 res
Level 3	16×16	256	2 res + attn
Middle	16×16	256	res, attn, res
Decoder	mirror	256 → 64	3 res / level
Output	64×64	64 → 3	1 conv

≈17.5 GFLOPs per denoising step at 64×64.

MNIST net: 13,002DDPM CIFAR-10: 35.7MOurs: ~16.5MWidth 128: ~66M

Plan, not final: base width 64, multipliers (1, 2, 4), AdaGN conditioning; counts computed by us. DDPM figure: Ho et al. 2020.

~1 min

A concrete starting config, so the Model stream has numbers to argue with. Treat it as a plan.
Base width 64, channel multipliers 1, 2, 4, two ResBlocks per level on the way down, three on the way up (one extra to absorb each skip), attention at 16×16 and in the middle.
Where the parameters live: mostly in the decoder (the concatenated skips double its input channels) and the 256-channel levels. Conv parameters scale with channels squared, so the deepest levels dominate even though they're spatially tiny.
Compute is the reverse: the 64×64 levels have few parameters but many positions. ≈17.5 GFLOPs per network call, our own estimate.
For scale: the deck-2 MNIST network had 13,002 parameters; DDPM's CIFAR-10 U-Net 35.7M. Doubling the width to 128 roughly quadruples parameters, to about 66M. That's the "model size" control sweep.

Intuition: parameters live where channels are many; FLOPs live where pixels are many.

If asked: "Is 16M enough?" For 64×64 frames from a procedural renderer, very likely. If it isn't, that's a finding about task complexity, and the width sweep exists to check.

How our inputs get in

Every pixel arrives as a stack of 16 numbers

~1 min

Now the part specific to us. Each plane here is rendered live from the toy ray-caster: these are the real channel types the engine emits.
Noisy RGB, 3 channels: the thing being denoised.
Depth, 1 channel: distance to the surface along the view. Near is bright.
Surface type, 4 channels one-hot: wall, floor, ceiling, enemy. One-hot because categories have no order; "floor = 2" shouldn't be "between" wall and ceiling.
Material ID, 5 channels one-hot (for our 5 materials). Alternative: a learned embedding per material, which is cheaper when materials are many.
UV, 2 channels: where along the surface each pixel sits. This is what anchors texture to walls.
Light map, 1 channel: how lit each point is.
Stack them all: 3 + 1 + 4 + 5 + 2 + 1 = 16 channels, one tensor of shape 16×64×64, into the U-Net's first conv. Channel count is our plan and will change with the factor grid.

Intuition: the engine already knows the scene's geometry exactly. We hand the network that knowledge per pixel, so it only has to learn appearance.

If asked: "Isn't depth redundant with UV?" Partly. Redundancy is cheap, and depth also drives fog. Which channels matter is itself an ablation.

How our inputs get in

Concatenation works because the first conv sees it all

Each output mixes all 16 channels around pixel .

SR3: low-res image concatenated SD inpainting: 9 input channels GameNGen: past frames in latent channels ControlNet: depth and edge maps

~1 min

Why does stacking channels work at all?
A conv kernel spans all input channels. The first layer's 3×3 window at pixel sees the noisy colour and the depth, surface, material, UV and light at exactly that spot, and mixes them with learned weights.
Slide it across the image: the same weights everywhere. So "material 2 at this pixel" can directly shape "what colour belongs here", from the very first layer.
This is a well-trodden pattern for pixel-aligned conditioning. SR3 super-resolution concatenates the upsampled low-res image. Stable Diffusion's inpainting U-Net takes 9 input channels (4 latent, 4 masked-image latent, 1 mask). GameNGen concatenates past frames' latents. ControlNet injects spatial maps like depth or edges through a trainable encoder copy, a heavier version of the same idea.
Rule of thumb: if information is per pixel, give it per pixel. Global information goes elsewhere: next slide.

Intuition: concatenation puts the answer key's structure right next to the pixel that needs it. No long-range reasoning required.

Hypothesis 2 lives here: we predict factors given this way compose more easily than factors given as a global vector.

How our inputs get in

Global inputs turn per-channel dials

FiLM: Perez et al., AAAI 2018 · AdaGN: Dhariwal & Nichol, "Diffusion Models Beat GANs on Image Synthesis", NeurIPS 2021 · dial values illustrative.

~1 min

Theme has no "where": it applies to the whole frame. So it shouldn't be a channel; it should be a vector that changes how every block behaves.
The mechanism: after GroupNorm, multiply each channel by and add , both computed from the conditioning. This is FiLM (feature-wise linear modulation), called AdaGN in diffusion. Each dial is one channel's scale; each bar its shift.
Theme 1: the theme vector goes through a small MLP and sets the dials. A channel that detects "wall" might get turned up and shifted toward red.
Theme 2: same network, different dials. The channels' meanings stay put; their loudness changes.
Theme 3. On the right, the skinner's target for each theme: what the dials must produce.
The timestep uses the same path: 's embedding and the theme's embedding are summed, then projected into , for every block. Dhariwal & Nichol found this beat plain addition (FID 13.1 vs 15.1 in their ablation).

Intuition: the U-Net is a mixing desk; the theme is a preset that moves every fader at once.

If asked: "Why not cross-attention, like Stable Diffusion's text?" Text is a sequence of tokens; our theme is one short vector. FiLM is the natural, cheap fit. Cross-attention would be an interesting ablation.

Theme encoding · hypothesis 2

Theme as an ID can't generalize; as parameters, it might

~1 min

Two ways to hand the theme to the network. This choice is itself an experiment.
As an ID: a lookup table with one learned row per theme. Simple, and it works perfectly for themes seen in training.
A brand-new theme has no row. There's nothing to look up. An ID can never generalize to a new theme, by construction.
As a parameter vector: palette, pattern family, scale, finish, fog. An MLP maps that vector to the embedding. Training themes are points in that space.
A new theme is a new point, ideally inside the cloud of training themes. A smooth MLP gives it an embedding between its neighbours'. Whether the rendered frame is right is the open question.
That's the encoding sweep (pixel map vs ID vs parameter vector) and the leave-one-theme-out split. Deck 6 designs them.

Intuition: an ID is a name; a parameter vector is a description. You can only generalize from a description.

Subtle point: even the parameter vector can fail if the new theme lies outside the training cloud (extrapolation), or if the MLP learns to recognize the training themes instead of reading their parameters. Too few training themes makes the second failure likely.

Temporal stability

Three tricks keep frames from flickering

No autoregressive drift: the engine owns gameplay, so every frame is rendered from exact structure, not from our last guess.

Schematic: hallucinated detail exaggerated for visibility.

~1 min

A renderer that's right frame by frame can still look awful in motion. Three panels, camera moving.
Naive: fresh random noise every frame. The detail the model invents differs every frame: shimmering, boiling texture.
Fix the starting noise and use DDIM: the renderer becomes deterministic. But the noise is fixed to the screen, so the invented detail stays put while walls slide past: "swimming" texture.
Add UV conditioning: the model can tie detail to the surface's own coordinates, so it moves with the wall. Stable.
Unlike GameNGen, we never feed our own output back in, so errors don't compound over time. We measure flicker by warping the previous frame with the known camera motion and comparing.

Intuition: anything the model invents must be a function of the world, not of the screen or the dice.

If asked: "What about conditioning on the previous frame?" That's a stretch goal: temporal conditioning on the warped previous frame. It helps consistency but brings back some drift risk.

Inference

Few-step sampling makes a playable demo plausible

Cost per frame = steps × one U-Net call (≈17.5 GFLOPs).

At 20 fps with 4 steps: ≈1.4 TFLOP/s, a small fraction of one modern GPU's peak.

GameNGen: 4 DDIM steps, 20 fps on one TPU.

GameNGen figures: Valevski et al., 2024. FLOP counts are our estimates for the planned config; real frame time is a measurement we'll make.

~1 min

Can our renderer run live? Log scale, GFLOPs per frame.
Original DDPM: 1000 calls per frame, about 17,500 GFLOPs. Not real time.
DDIM with 50 steps: 875 GFLOPs.
10 steps, then 4 steps: 70 GFLOPs. Classifier-free guidance doubles each bar.
At 20 frames per second, 4 steps needs about 1.4 trillion FLOPs per second. Modern GPUs have tens to hundreds of TFLOP/s of peak half-precision throughput, so on paper there's headroom. Real kernels reach a fraction of peak, which is why frame time is a measured metric, not a promise.
GameNGen hit 20 fps with 4 DDIM steps at much higher resolution (and reported 50 fps after distilling to a single step). Our playable demo is a stretch goal, but the arithmetic says it's not crazy.

Intuition: sampling cost = steps × network size. Shrink either one.

Next: deck 6

Now: what do we train this on, and how do we prove it generalizes?

The model is the easy part to describe. The data and the tests are where the science is.