Independent and unofficial. The Test Fixture is not affiliated with, endorsed by, or operated by any model developer or platform mentioned here. LTX-2.3 and related marks belong to their respective owners.
A test report that says “LTX-2.3, ComfyUI, latest drivers” has not told you what was tested. It has told you what was installed on the day the author stopped paying attention.
Software versions are not metadata. They are experimental variables. Change the runtime, the driver, the custom node, or the quantization, and you have changed the test — even if the model file is byte-identical. This post is about why that matters, how version drift produces false conclusions, and what I record so that a version change is visible instead of silent.
Test question
When only software versions change within a recorded known-good range, does the output change — and if it does, how do I distinguish a version-induced change from a model-level result?
The working hypothesis: version changes can produce output differences that look like model behavior, and the only defense is recording versions as part of the configuration rather than as a footnote.
Why versions get demoted to footnotes
Three reasons, and all three are understandable.
The model file is the thing people care about. A test report is usually about the model, so the model gets the attention, and the software stack gets a line at the bottom. That is backwards for reproducibility: the model file is a constant in a given test, and the software stack is where the uncontrolled variation lives.
Version numbers are boring until they are not. Nobody wants to read a driver version in a test report. Then a result fails to reproduce, and the first question is always “what changed?” — and the answer is usually in the part nobody wanted to read.
“Latest” feels like a safe default. It is not a default. It is a moving target. A test run on “latest” is a test run on an unrecorded condition, and it cannot be compared to a run on a different “latest” without a version number to anchor it.
What counts as a version variable
On this bench, the version variables are:
OS build. Minor updates can change scheduler behavior and library availability.
GPU driver. Driver changes have moved results before. This is not hypothetical.
CUDA. Runtime and toolkit versions can change kernel behavior and memory handling.
PyTorch. Framework version changes can alter numerical behavior, even when the math is nominally the same.
Inference framework. ComfyUI or equivalent, with the exact version. Node behavior changes between releases.
Custom nodes and extensions. Each one is a versioned dependency. A workflow screenshot with three nodes cropped out is not a version record.
Model file. Exact identifier, commit, quantization, and checksum where available.
Quantization format. A quantized model is a different model file, not the same model at a lower setting. It gets its own version record.
That is a long list. It is the list because each item has, at some point in my logs, been the difference between a run that reproduced and a run that did not.
Observation: version drift in practice
I do not have a clean controlled experiment here, and I am not going to pretend I do. What I have is a pattern from my own logs, recorded as observation rather than as a result.
Pattern 1: driver and runtime changes moved fine detail, not composition. When I changed driver and runtime versions within a recorded known-good range and reran the same seed and prompt, gross composition and motion stayed consistent, while fine texture and frame-level detail shifted in some pairs. This is consistent with the general LTX-2.3 reproducibility pattern: file-level and output-level reproducibility are different properties, and a version change can move the second one without breaking the first.
Pattern 2: custom node updates produced workflow-induced failures. A node update changed the default behavior of a preprocessing step, which changed the input to the model. The model was not the cause; the pipeline was. This is the version-drift failure that is easiest to misattribute to the model, because the output changes and the model file did not.
Pattern 3: “latest” runs were not comparable to earlier runs. A run recorded as “latest” could not be compared to a run from two weeks earlier, because “latest” had changed and nobody wrote down what it was. This is a documentation failure, not a software failure, and it is the most common one in my own logs.
These are observations from a small set of runs. They are enough to justify recording versions as variables. They are not enough to claim a general rate.
How version drift produces false conclusions
Version drift is dangerous because it produces confident wrong answers in both directions.
False regression. A result looks worse after a change, and the report blames the model. The actual cause is a driver, runtime, or node update. The model did not regress; the environment did. I have caught this in my own logs, and it is the reason every test report now includes an environment snapshot.
False improvement. A result looks better after a change, and the report credits the model. The actual cause is a version change that happened to help. Same problem, opposite direction, and harder to catch because good news gets less scrutiny.
False stability. A result looks stable across two runs, and the report concludes the workflow is stable. But both runs used different “latest” versions, so the stability is an artifact of not recording the variable. This is the quietest failure of the three and the one that erodes trust slowly.
In all three cases, the fix is the same: record versions as part of the configuration, and treat a version change as a variable change that requires a note.
What I record, and how
A known-good environment snapshot per model version. Not a full disk image, but the version set that produced the baseline for a given model. When a result changes, the first check is whether the environment changed underneath it.
Version fields in every configuration manifest. OS, driver, CUDA, PyTorch, framework, custom nodes, model commit, quantization. These are fields in the manifest, not prose in the article.
A version-change log. When I update a version on purpose, it goes in the log with the date and the reason. When I update a version by accident — because a package manager did what package managers do — it also goes in the log, with a note that it was accidental. The accidental ones are often the most informative.
A rule against “latest” in test reports. “Latest” is allowed in a bench description as long as it is paired with the actual version at the time of writing. In a test report, it is not allowed, because it is not a version.
A check before attributing a result to the model. If the environment changed between two runs, the result is a version-change candidate until the environment is held constant. This check is cheap and it has saved me from publishing a model regression that was really a dependency update.
Failure cases
Failure category: workflow-induced failure (version drift). A custom node update changed preprocessing defaults and altered model input. Sample count: 1 incident in my logs. Reproducible: yes, by pinning the old node version. Attribution: workflow, not model.
Failure category: documentation failure (“latest”). Runs recorded as “latest” could not be compared across time. Sample count: multiple incidents, not individually counted. Reproducible: not applicable; this is a recording failure, not a run failure. Attribution: test design.
Failure category: false regression candidate. A result appeared worse after an environment update. After holding the environment constant, the difference was not attributable to the model. Sample count: 1 incident in my logs. Recorded because it is the exact failure this post is about.
These are incidents, not a controlled experiment. They are reported as such.
Reproduction notes
The version fields listed above are the fields in the configuration manifest. A sample manifest skeleton will be published with the Testing Methodology page. The specific versions for this bench are in the test bench post and in each test's manifest.
If you rerun a test and get a different result, the first question is not “did the model change?” It is “did my environment change?” Check the version fields before attributing anything to the model. That is the whole protocol in one sentence.
What this post does not show
It does not measure how often version drift changes results. It does not rank which version variable matters most; that depends on the model and the workflow, and I have not tested it systematically. It does not establish that version changes are always the cause of non-reproduction; sometimes the model or the seed is the cause. It does not replace the Reproducibility Protocol; it is the version-specific part of it.
The configuration is part of the result. A successful sample is evidence. It is not a warranty.
If it can’t be reproduced, it’s a story—not a result.
