noise⇄rendered frame

QMIND Research × Chatforce · 2026–27

Diffusion-Model Rendering for Re-Themable Games

Why this is worth a year of your time, and what we are going to build.

~1 min

Welcome. Today is long (3–4 hours, seven decks), and by the end you will understand everything from a single neuron to the system we are building.
The little screen on the right is the whole project in one loop: a game frame dissolving into noise and a model pulling it back out. That is diffusion, and that is what we will train.
Open with the frame, not the logistics: "This deck is a pitch. I want to convince you this is the best use of your hours this year."
Set expectations: first 10 minutes are about why; then a non-technical picture of the project; technical depth comes later today.

Intuition: people commit to a project when they can see both the payoff and the shape of the work. This deck gives them both.

If asked: "Who is Chatforce?" Our client and funder: $10k total ($5k direct + $5k in AWS credits). Supervisor/client contact: Robert Ciborowsko.

Return on effort

Conceptual illustration · not data

The domain you pick sets your return more than the hours do

Same input. Very different output.

The difference is where you point it.

~1.5 min

You are going to spend hundreds of hours on something this year: a club project, a side project, a course you go deep on. That time is already committed. The only question is where it goes.
Two people, same ability, same hours. The blue blocks fill identically. Effort is not the variable here.
The payoff bars come out wildly different. Say clearly: this is a conceptual sketch, not measured data.
What differs is the domain. Same hours poured into a crowded field where the easy wins are gone, versus a young field where the frontier is close.
Land the thesis: picking the domain is a bigger lever than grinding harder. Most students optimise hours; few optimise where.

Intuition: effort is the input; the domain is the exchange rate.

Analogy: two engines burning the same fuel, one in a car on a flat road, one in a car stuck in sand. Same fuel, very different distance.

If asked: "Isn't talent the real variable?" It matters, but you can't choose it this week. You can choose the domain today.

Return on effort

Think of it as return on effort

D · Demand

Worth it now?

How much the skill is valued right now.

F · Frontier

How close?

Hours until you can do something new.

C · Compounding

Does it transfer?

Reused in the next thing you learn.

S · Scarcity

How rare?

How few people have the combination.

P · Proof

Can others see it?

Do you leave with evidence.

~1.5 min

Borrow the finance idea: return on investment. Here the investment is your hours and the return is everything you walk away with.
Demand: is the skill valued right now? Not "will it be cool in ten years", but today.
Frontier: how many hours before you can do something nobody has done? This is the one people ignore.
Compounding: does it make the next thing easier? Skills that transfer keep paying.
Scarcity: lots of people have one skill. The value is in rare combinations.
Proof: can someone outside your head verify it? A paper, a demo, a repo.
Why multiply and not add: if any factor is near zero, the whole thing collapses. A rare skill nobody wants is worth little; an in-demand skill you can't prove is hard to cash in.

Intuition: this is a mental model, not a formula with units. Its job is to make you ask five questions before committing a year.

If asked: "Where's effort in this?" Mostly fixed: it's your year. The lever is the numerator. (One exception: fun makes hours cheaper. Slide 10.)

Return on effort

Conceptual sketch · not data

Every domain has a curve, and you want the steep part

API wrapper: a quick win, then flat.

Mature field: the easy wins are taken.

Young field: a slow start, then steep.

Your return on the next hour is the slope:

~1.5 min

Plot effort against reward for one person in one domain. Not data, a picture of an idea.
The API-wrapper project: you get something on screen in a weekend, which feels great, then it's flat. Hour 50 teaches you about as much as hour 5.
A mature field: real rewards, but the easy wins were taken decades ago. You climb a lot before anything changes.
A young field where the frontier is close: honestly, the start is slow. The first stretch is learning, and it does not feel like progress. Then it bends upward fast.
The slope is your marginal return: what the next hour buys you. You want your year to land on the steep part. Today and the first month are the slow start; that's the price of admission.

Intuition: don't ask "how good is this field?", ask "how steep is it where I would be standing?"

If asked: "Won't ML saturate too?" Parts already have: training yet another chatbot from scratch is not a student game (deck 5 covers why we avoided LLMs). That's why we picked a narrow, open corner of it.

Factor F · Frontier

Conceptual illustration

In generative ML the frontier is months away, not a PhD away

Months, not a PhD.

Elsewhere: years of prerequisites before the first new result.

Why here: a young, mostly empirical field, open papers and tools, and questions small enough for a student budget.

~1.5 min

Two ladders. Each rung is a chunk of prerequisite knowledge. At the top: open questions, where you can find out something nobody knows yet.
Two people climb at the same pace.
On the short ladder, you are at the frontier after a few rungs. Point at the labels: today's decks literally are those rungs.
On the tall ladder, same effort and you are barely a third of the way. Think research-level pure math or theoretical physics: you typically need graduate training before you can contribute.
Why ML is different right now: the field is young, much of it is empirical (you run the experiment and see), papers and code are open, and a small, well-designed study fits on a student GPU budget.

Be honest: "reachable" is not "easy". ML has deep theory, and the hard questions are hard. The claim is narrower: well-posed empirical questions are within reach of motivated undergrads in months.

If asked: "Do undergrads really publish in ML?" Yes, especially at workshops and student venues like CUCAI, which is our target. The key is a sharp question, which is exactly what we have.

Factor C · Compounding

The same math shows up in every model

~1 min

Three roots: linear algebra, probability, optimization. Every model you have heard of is built from these.
They branch into the big model families: neural networks, transformers, diffusion, reinforcement learning.
And those branch into everything: LLMs, image and video generation, speech, robotics, protein design, recommendation, game and world models, weather forecasting.
This is compounding: an hour on the roots is reused in every branch. Learn what a gradient is once and you use it in every model forever.
Today walks exactly this tree: deck 2 is neural nets, deck 3 transformers, deck 4 RL, deck 5 diffusion. Our project sits on the diffusion branch but touches all of them.

Intuition: skills that compound make the next skill cheaper. Tool-specific skills (one framework's API) depreciate; math doesn't.

If asked: "Do I need to be strong at math already?" No. You need to be willing to get stronger. Today we build the intuition; the math follows.

Factor S · Scarcity

Conceptual illustration

Few people hold the whole stack, and that is the scarce part

Research: ask a sharp question, design the test.

Engineering: build systems that work.

Infrastructure: make it run on real machines.

Theory: know why it works.

Many people have one. Few have all four.

~1 min

Each dot is a person. Conceptual, not a survey.
Research: people who can turn a vague idea into a testable question.
Engineering: people who can build software that works and keeps working.
Infrastructure: people who can get it running on cloud GPUs, move data around, keep track of hundreds of runs.
Theory: people who know why the method works, so they can debug it when it doesn't.
The overlap is small. Most courses and side projects train one circle. This project forces all four, because you can't finish it otherwise.

Intuition: value comes from scarcity of the combination, not of any single skill.

Analogy: a full-stack engineer versus a front-end specialist; here it's "full-stack ML": from the data generator to the paper.

If asked: "Will I really touch all four if I'm in one stream?" You own one stream, but streams hand work to each other weekly and everyone reads the results. You'll see all of it up close.

Factor P · Proof

Leave with proof other people can read

“I took a course” is a claim. A paper, a demo and a repo are evidence.

~1 min

The last factor: can someone who has never met you verify what you can do?
A paper. Our targets are CUCAI and a workshop submission. A paper says you can ask a question and answer it rigorously.
A demo: something people can try. A game that re-themes is easy to show anyone in 30 seconds.
Code: a public repo with your commits in it. Engineers read this first.
Each artefact speaks to a different reader: recruiters skim the demo and repo, grad schools read the paper, future collaborators read all three.
The contrast: a line on a résumé is a claim. These are evidence.

Be honest: a paper is a target, not a guarantee. It depends on the work, and a clean negative result can still be a paper if the study is well designed.

If asked: "Will my name be on the paper?" Authorship follows real contribution; that's the norm, and we'll be explicit about it early.

Return on effort

Low return hides inside ML too

Project	Skills it trains	Answer known?	Stakes	You can show
Tutorial clone	one	yes, in the tutorial	none	a repo like everyone's
API wrapper	one (glue code)	yes	none	someone else's model
Kaggle fine-tune	one (tuning)	metric fixed in advance	a rank	a leaderboard number
This project	research, engineering, infra, theory	no: it's open	client, funding, a paper	paper, demo, code

~1 min

"ML" isn't automatically high-return. Plenty of ML projects are low on every factor.
The tutorial clone: you follow the steps, you get the answer the tutorial already had. One skill, no surprises.
The API wrapper: call someone else's model, add a UI. Useful glue skill, but the interesting part belongs to someone else.
The Kaggle fine-tune: real skill, but the question, data and metric are handed to you. You learn tuning, not how to pose a problem.
This project: four skills, an open answer, real stakes (a client, money, a publication target), and three artefacts to show.

Be fair: none of the first three are worthless. They're great warm-ups. The problem is they plateau fast: slide 4's flat curve.

If asked: "Isn't Kaggle respected?" Top competitors are excellent. But a typical fine-tune on a standard dataset is one skill with a known target, which is the point here.

Return on effort

This project scores on every factor

DDemand

3Foundations, not wrappers

6Real infrastructure

FFrontier

1A real open question

2Make-or-break moments

CCompounding

4How research actually works

5Serious engineering

SScarcity

7You own the whole stack

8Funded, with a client

PProof

9Something to show

effort

10It's a game: the hours go faster

~1.5 min

The ten reasons I pitched this project, sorted into the five factors.
Demand. (3) You'll understand noise schedules, the denoising objective, U-Nets, attention and conditioning, because you write every line. (6) Real infra: AWS, datasets, experiment tracking.
Frontier. (1) A real open question: you can't prompt an agent loop to the answer. (2) Make-or-break moments: the first denoised image that isn't static, the first held-out combination rendered right, the first curve showing where it breaks.
Compounding. (4) How research actually works: hypotheses, controls, baselines, ablations, negative results, a paper. (5) Serious engineering: a ray-caster, procedural maps and textures, a deterministic pipeline.
Scarcity. (7) You build the world, generate the data, train the model and design the test. (8) It's funded ($10k: $5k direct + $5k AWS credits) and there is a real client, Chatforce.
Proof. (9) A CUCAI paper, a workshop submission, a playable demo.
And the denominator: (10) it's a game. Fun doesn't reduce the hours, but it makes them cheaper to spend, and it's why you'll still show up in March.

Honest cost: it is hard and it's a full year. That is the price of every bar on this slide being full.

The payoff

What you'll be able to say in a year

“I trained a diffusion model from scratch.”

“I designed a controlled generalization study.”

“I ran it on cloud GPUs.”

“I co-authored a paper.”CUCAI + workshop target

“I shipped a demo you can play.”

The cost: it's hard, and it's a year. None of this is guaranteed. All of it is reachable.

~1 min

Make it concrete. Imagine an interview a year from now. What can you say, truthfully?
You trained a diffusion model from scratch: not a fine-tune, not an API call. You know what every line does.
You designed a controlled study: held-out combinations, sweeps, baselines. That's the scientific method applied to ML.
You ran it on cloud GPUs, on a real budget ($5k of AWS credits), with experiment tracking.
You co-authored a paper. Target: CUCAI plus a workshop.
You shipped a demo you can hand someone.
Say the cost out loud. It's hard, it's a year, and some weeks will be frustrating. I'd rather you join knowing that. The point of the ROI argument is that this cost buys more here than anywhere else you could spend it.

Intuition: each sentence is checkable. That's what makes it worth more than "I'm interested in ML".

If asked: "How many hours a week?" Be straight about your expectation for the team; don't undersell it.

PART B

The project in one picture

A tiny game engine, a painter, and a diffusion model that learns to take over the painting.

The question

Chatforce's dream is huge; our question is precise

Our research question

When a diffusion renderer is trained on only some combinations of scene factors, where does its ability to render the unseen ones break, and why?

64×64 frames3 factors, 4–5 values eachan exact answer key

Inspired by GameNGen: a diffusion model that simulates Doom.

GameNGen: Valevski et al., “Diffusion Models Are Real-Time Game Engines”, arXiv:2408.14837 (2024), ICLR 2025.

~1.5 min

Start with what the client wants, then what we will actually study.
Chatforce's north star: live, promptable re-theming of a running game. Type "frozen cathedral" and the game you're playing becomes one. That bundles many hard problems: real-time speed, text-to-theme, no flicker, any game, even learning the game itself.
A student team can't solve all of that in a year, and shouldn't try. We carve out one slice that is precise and answerable: rendering combinations the model never saw.
Read the question. The point is not "can a diffusion model render a simple game" (it obviously can). It's where generalization to new combinations fails as the task gets harder. Memorising can't pass, because the test combinations never appear in training.
Start small: 64×64 frames, three factors (theme, material, lighting) with 4–5 values each. And every test has an exact right answer, which almost no generative-model study gets.
Inspiration: Google's GameNGen (2024) showed a diffusion model can simulate Doom interactively. We take the rendering half of that idea and ask a sharper scientific question about it.

If asked: "Does this help the client?" Yes: their practical question is whether a re-theme holds up in rooms and lighting it wasn't trained on together. That's our question.

The system

Every frame flows through four stages

The engine runs the game: map, player, enemies. It never draws the final picture.

One ray per screen column: how far to the wall, what it hit, where on the wall.

Channels: pixel-aligned maps of what is where. Structure, no style.

The skinner paints a themed frame from the channels. Same input, same pixels, every time.

Our diffusion model learns to do the skinner's job, starting from noise.

The skinner stays on as the answer key for every test.

~2 min

Four stages: engine, channels, skinner, frame. This picture is the backbone of the whole project. The map on the left is a real map; the player is the dot.
The engine is a Doom-style ray-caster: for every screen column it shoots one ray and finds the first wall. Distance gives the wall's height on screen. Gameplay is deterministic, so any frame can be regenerated exactly.
From that it writes channels per pixel: depth, surface type (wall/floor/ceiling/enemy), material ID, texture coordinates (UV: where along the wall this pixel sits) and a light map. These are false-colour images: geometry with no style.
The skinner is a plain deterministic function: skin(channels, theme, lighting). It samples brick or tile patterns at the UV coordinates, so patterns stay glued to walls, then applies light and fog.
The twist: the diffusion model replaces the skinner. It gets the same channels plus a theme code, starts from pure noise, and denoises toward the frame. It must learn to paint.
The skinner doesn't go away. It can render any combination, including ones the model never saw, so it grades every test exactly instead of by eye.

Intuition: the engine owns what happens; the model only owns how it looks. That's the "neural renderer" framing.

If asked: "Why not let the model learn the gameplay too?" That's a world model (GameNGen-style) and it's future work; it would blur what we're measuring.

The system · themes

A theme is data, not code

Theme record · 1 of 4

Dungeon

patternbricks, flagstonesfinishmattefogbrown, dense

Theme record · 2 of 4

Frozen

patternice tiles, blocksfinishglossyfogpale blue, thick

Theme record · 3 of 4

Neon

patternpanels, gridfinishemissive trimfogblack, dense

Theme record · 4 of 4

Overgrown

patternmossy stonefinishmattefoggreen mist

Same channels, four themes

Adding a theme = adding a row

So we can hand-write themes or sample as many as we want.

~45 s

Same frame, same channels. A theme is a small record: palette, pattern family and parameters, surface finish, atmosphere (fog colour and density).
Frozen: different palette, ice tiles, a glossy finish that catches the light.
Neon: dark panels with glowing trim. Note the geometry never changed; only the paint did.
Overgrown: mossy stone and a green mist.
All four side by side. Because a theme is data, adding one is adding a row. We can hand-write them or sample them from the parameter space, so we can make as many as an experiment needs.

Intuition: patterns are sampled at the UV coordinates, so bricks stay glued to the wall as you move. That's why the channels matter.

If asked: "How does the model know which theme?" Either as an ID (a learned vector per theme) or as the parameter vector itself. That choice is an experiment: an ID can never generalise to a brand-new theme; a parameter vector might. Deck 5.

The question, as a picture

The research question is a grid with holes in it

Trained on: most of the pairings.

Can the model render the dark cells?

Every theme and every lighting was seen. Only the pairing is new.

And we know the exact right answer for every one.

Then sweep the coverage and find where it breaks. Deck 6 goes deep.

~1.5 min

Rows are themes, columns are lighting setups. Every cell is one combination. (The real grid adds materials as a third factor; two fit on a slide.)
Fill in what the model trains on: most cells.
A few cells stay dark: held out. The question: can it render those?
Key rule: hold out combinations, never values. Frozen appears in training (with other lights). Torches appear (with other themes). Only "Frozen under torches" is new. This is compositional generalization.
Because the skinner can render any cell, we have the exact answer for each hole, pixel for pixel. No arguing about whether it "looks right".
Then turn the dial: train on less and less of the grid and watch held-out quality. Hypothesis 1: it doesn't degrade smoothly; there's a threshold. Deck 6 covers the other sweeps.

Intuition: a model that memorises can't pass, because the test cells never appeared in training.

If asked: "What if it just works everywhere?" Then we make the task harder (more factors, shadows, reflections) until it breaks. The breaking point is the result.

The team

Five streams, and each of you owns one

Stream 1

Engine & rendering

The ray-caster, map generator and channel outputs.

Stream 2

Skinning & data

The theme space, the skinner, dataset generation.

Stream 3

Model & training

The U-Net, conditioning and sampler. Train it.

Stream 4

Evaluation & analysis

Holdout splits, metrics, the curves in the paper.

Stream 5

Infra & MLOps

AWS jobs, storage, experiment tracking.

~1 min

The pipeline from the last slides, split into ownable pieces.
Engine & rendering: the ray-caster, procedural maps, and every channel it outputs. Good if you like graphics and systems.
Skinning & data: design the theme parameter space, write the skinner, generate the datasets. Good if you like procedural art and pipelines.
Model & training: the U-Net, how channels and themes get injected, the sampler. Good if you want to live in PyTorch.
Evaluation & analysis: holdout splits, metrics, per-factor classifiers, the sweep curves. This is where the paper's figures come from.
Infrastructure & MLOps: AWS training jobs, data storage, experiment tracking. It runs underneath everything, which is why it spans the whole strip.
All five lit: one pipeline, five owners. A result in the paper needs every stream to have done its part.

Intuition: you own one stream, but streams hand work to each other every week. Nobody is siloed.

If asked: "When do we pick?" Think about it during today; deck 6 ends with what each stream needs, and we assign at the end of the day.

Today

Today: from a single neuron to our system

~1 min

Here's the day. We're in deck 1.
Decks 2–4: how modern AI works. Deck 2 builds a neural network from a single neuron (the MNIST digits example). Deck 3: transformers, the architecture behind LLMs. Deck 4: the frontier: serving, scale, GPUs, RL, test-time compute.
Decks 5–7: our project. Deck 5: diffusion and our model. Deck 6: dataset engineering and experiment design. Deck 7: PyTorch, the compute stack and MLOps.
Why start from a single neuron? Because everything in our project is built from these parts. The U-Net in deck 5 is neurons, convolutions and attention, arranged differently. If you understand deck 2 deeply, nothing later is magic.

Logistics: about 4h40 of material at note pace, so plan two breaks (after deck 3 and after deck 5) and say when they are. Deck 4 is the easiest place to trim if you run long.

If asked: "I already know neural nets, can I skip?" Stay: the 3Blue1Brown-style intuitions in deck 2 are exactly what you'll need to explain tomorrow.

Tomorrow

Tomorrow, you teach it back

Explaining it is how you find out what you don't understand yet.