Independent and unofficial. The Test Fixture is not affiliated with, endorsed by, or operated by any model developer or platform mentioned here. LTX-2.3 and related marks belong to their respective owners.
A test report is not a description of what happened. It is a set of instructions that lets someone else find out whether it happens again.
That distinction sounds academic until you try to reproduce a generated video from a post that says “LTX-2.3, 4090, default settings, seed 42.” Which default settings? Which runtime? Which quantization? Which sampler? Which ComfyUI build, and which custom nodes? Was the model file the same commit? Was the prompt identical, including punctuation? Did the machine have another process using VRAM? Did the run include a warm-up, or was the first render cold?
None of those questions are nitpicks. They are the difference between a result and a story.
This post is the protocol I use for The Test Fixture. It is not a universal standard, and it is not endorsed by anyone. It is a minimum viable reporting format for AI video tests: enough detail that a careful reader can attempt the same run, and enough honesty that a failed attempt is still informative.
The minimum viable test report
Every controlled run on this site will answer six questions:
What was the test question? One sentence. Not “testing LTX-2.3.” Something like: “Does changing only the seed move the subject’s position across ten runs?”
What was the configuration? Hardware, software, model version, parameters, and inputs.
What was observed? What the outputs actually did, including failures.
What is interpretation? What I think the observation means, clearly separated from the observation.
What is speculation? What I suspect but did not test. Labeled as speculation, not smuggled in as fact.
How can it be reproduced, and what would change the conclusion? Commands, files, checksums, and known limits.
If a report cannot answer these, it is a demo write-up. Demos are allowed. They just should not be called test reports.
The configuration block
The configuration is part of the result. I will publish it before the interesting clips, not in a footnote below them.
Minimum fields:
Hardware: GPU model, VRAM, CPU, system RAM, storage type if it affects loading.
OS and driver: OS build, GPU driver version.
Runtime: Python, CUDA, PyTorch, inference framework, ComfyUI or equivalent version.
Model: exact model identifier, version or commit, quantization, file checksum if available.
Custom nodes or extensions: names and versions. “A few custom nodes” is not a version.
Generation parameters: sampler, scheduler, CFG, steps, resolution, frame count, FPS, duration, motion settings, denoise settings.
Seed policy: fixed seed, seed range, or per-run seeds. If seeds were selected after looking at outputs, say so.
Test set: the prompts or input files, with a stable identifier for each case.
Environment notes: background load, thermal state if relevant, warm-up runs, number of repetitions.
I do not need a reader to own the same machine. I need them to know which differences matter.
The run block
A single successful run is an anecdote with a timestamp. For controlled runs, I will report:
Number of runs per condition.
Number of conditions.
Total samples generated.
Failure rate, with the failure categories used.
Render time distribution, not just the best run.
Whether outputs were cherry-picked, and by what rule.
Whether any runs were discarded, and why.
Discarding runs is not automatically cheating. Discarding runs without saying so is.
The observation block
Observation means what the outputs did. Not why. Not whether it is good. Not what it implies for production.
Examples of observations:
“Subject position shifted by more than one body width in 6 of 10 seeds.”
“Flicker appeared after frame 48 in 4 of 10 runs at 720p.”
“The same prompt produced a stable first frame in all runs, but identity drift appeared by frame 12 in 7 of 10.”
Examples of things that are not observations:
“The model struggles with motion.”
“This workflow is production-ready.”
“LTX-2.3 is better at consistency.”
Those may become interpretations or conclusions later. They are not observations.
The interpretation block
Interpretation is where I say what I think the observation means, and how confident I am. I will use plain confidence labels:
Strong: repeated across runs and conditions, with a plausible mechanism.
Moderate: repeated in this test set, but not yet separated from workflow effects.
Weak: seen once or twice, or confounded by another variable.
Speculative: a hypothesis I have not tested.
That is an observation, not yet a conclusion. The label matters more than the confidence.
The failure block
Failure cases are part of the result. Every controlled run will include a failure section, even if the failure is “no obvious failure in this test set, under these conditions.”
Failure categories I will use:
Temporal flicker.
Identity drift.
Object deformation.
Camera instability.
Prompt contradiction.
Render failure or crash.
Workflow-induced failure.
Irreproducible success.
The last one is the most important. An impressive clip that cannot be reproduced is not a capability result. It is a reproducibility failure, and it gets its own report.
The reproduction block
A report is not reproducible if the reader cannot find the inputs.
I will publish, where legally and technically possible:
Prompt text or prompt files, with stable IDs.
Seed lists.
Configuration manifest in JSON or CSV.
Checksums for model files and input assets.
Commands or workflow files, with versions.
Proxy videos for review, plus original metadata.
A note on what was not published, and why.
I will not distribute model files or installers without source, license, checksum, and legal-status documentation. I will not ask readers to run unknown binaries. I will not gate results behind an account, a login, or a “download portal.” This site does not request passwords, API keys, verification codes, wallet details, recovery codes, or remote access.
The correction block
Every article will show a publication date and an update log. If a result changes after a runtime update, a driver change, or a reader’s reproduction attempt, I will note it. I will not quietly delete a failed test. A correction is part of the record.
If I cannot reproduce my own result later, that goes in the log too. Especially then.
What this protocol does not solve
It does not make a test universally valid. It does not control for every hardware difference. It does not turn a small test set into a benchmark. It does not prove a model is good or bad. It does not replace official documentation, and it is not official documentation.
What it does is make the conditions visible, the failures publishable, and the conclusions falsifiable. That is the minimum.
A successful sample is evidence. It is not a warranty.
If it can’t be reproduced, it’s a story—not a result.