I recently tackled what should not have been difficult: given exactly the same source, toolchain, and virtual filesystem, two clean Linux builders should produce byte-for-byte identical WebAssembly bundles.

The first time I ran two builders, the raw Wasm was identical but the final Wasm was not. Worse, on the same runner, starting two separate pack processes with the same raw Wasm still produced different results.

The main path I ultimately traced was not in the compiler or linker, but:

pack-time CLOCK_MONOTONIC
→ CPython 3.14 mimalloc weak-random seed
→ ChaCha20 and allocator internal state
→ Wizer snapshot guest linear memory
→ Core Wasm section 11 is different every time

The fix was applied only to the pack context: fixing the monotonic clock during packaging to now = 0. After that, exact equality returned for the same runner, across runners, and for the whole bundle.

This article records not just “fixing time is enough,” but how a many-megabyte Wasm mismatch was narrowed layer by layer into a causal chain that can be explained from source, rejected by experiments, and verified in CI.

Define what “reproducible” means first

The goal here is not “same behavior,” and not “roughly the same after removing timestamps,” but identical SHA-256 for the full bundle.

The build chain can be simplified into four layers:

Lock source and toolchain
        ↓
compile, static link
        ↓
raw.wasm
        ↓
wasi-vfs + Wizer pack
        ↓
final.wasm + manifest + SBOM + notices

If you only compare the final file, any layer can be blamed. So the first step was not changing linker flags, but preserving evidence at every critical boundary:

• pre-pack raw Wasm;
• frozen VFS manifest;
• wasi-vfs archive and linked object;
• pack CLI identity;
• output from first and second independent pack runs;
• digest for every file in the final bundle.

The decision rule also had to be fail-closed: only when raw Wasm and required pack inputs are all identical, while final Wasm is not, can the first drift be attributed to the pack stage.

First useful conclusion: the issue occurs in the pack stage

The initial two-builder experiment gave a clean boundary:

• raw Wasm is identical;
• VFS manifest is identical;
• source lock is identical;
• pack CLI is identical;
• final Wasm is not identical.

Then I made each builder start one more independent pack process for the same raw Wasm. Digests of the four packed Wasms were all different, and in all six pairwise comparisons, the first mismatch only appeared in Core Wasm section 11, which is the data section.

That rules out “the two GitHub runner environments were simply different.” A fresh wasi-vfs/Wizer process alone is enough to produce drift.

But section 11 only tells us “the snapshotted memory is different,” not who wrote those bytes.

Two real but incomplete directory issues

The 3-byte tail padding of Dirent

fd_readdir in Wasmtime-WASI Preview 1 copies a host #[repr(C)] Dirent representation into guest memory. The field layout is:

d_next    u64   bytes 0..8
d_ino     u64   bytes 8..16
d_namlen  u32   bytes 16..20
d_type    u8    byte 20
padding          bytes 21..24

The fields only occupy up to byte 20, but the C ABI size is 24, so there are 3 bytes of padding at the end. These paddings do not belong to any logical field, yet they are copied along with the full structure into the guest buffer and eventually end up in the Wizer snapshot.

I only zeroed the full dirent range [21, 24) and did not change inodes, names, types, cookies, or traversal order. The result was clear: the normal drift dropped from roughly 27–28 data segments, 379–438 bytes down to a fixed 6 segments, 104–106 bytes.

So padding is a real contributor, but not the only source.

Residue in the fd_readdir scratch buffer

wasi-vfs uses a reusable 4 KiB buffer to read directories. Even after full entry padding is handled, stale contents from truncated entries and the final returned range can still remain in the buffer.

Before each walk_dir returns normally, I applied a volatile zeroing over the buffer’s full live length. The new result dropped to a fixed 5 segments, 101–104 bytes, and the previous occasional section-size outlier disappeared.

This confirms another real contributor, but the remaining ~100 bytes clearly came from elsewhere.

Key clue: expand 32-byte k

Comparing the raw Wasm initial memory with packed Wasm, the remaining differences included one continuous high-entropy state. Nearby appeared this ASCII string:

expand 32-byte k

This is ChaCha state initialization constant material. Tracing the hash-locked CPython 3.14.0 source led to Objects/mimalloc/random.c: mimalloc uses ChaCha20 to manage its random context.

The core initialization path is:

static void mi_random_init_ex(mi_random_ctx_t* ctx, bool use_weak) {
  uint8_t key[32] = {0};
  if (use_weak || !_mi_prim_random_buf(key, sizeof(key))) {
    uintptr_t x = _mi_os_random_weak(0);
    // derive the key from x
    ctx->weak = true;
  }
  chacha_init(ctx, key, (uintptr_t)ctx);
}

The key behavior is in the mimalloc WASI prim implementation:

bool _mi_prim_random_buf(void* buf, size_t buf_len) {
  return false;
}

In this CPython WASI build, mimalloc always falls back to weak-random. The fallback then takes the seed via:

uintptr_t _mi_os_random_weak(uintptr_t extra_seed) {
  uintptr_t x = (uintptr_t)&_mi_os_random_weak ^ extra_seed;
  x ^= _mi_prim_clock_now();
  // shuffle x
  return x;
}

And WASI _mi_prim_clock_now() ultimately calls clock_gettime(CLOCK_MONOTONIC).

The causal chain closes here: each pack process sees a different monotonic time, mimalloc gets a different weak seed, which then produces different ChaCha and allocator state; Wizer snapshots that supposedly init-only memory state into the final Wasm.

Why “fixed random” did not work

When seeing high-entropy bytes, the first instinct is often random_get. I also tested replacing pack-only WasiCtxBuilder::secure_random with a deterministic generator.

To confirm the patch really worked, I also built a minimal Wasm whose init function calls random_get(0, 16) only once. Two consecutive packs with the patched CLI produced identical output.

But the same CLI, used on the real retained raw Wasm, left the same five segments of drift untouched.

This is not a contradiction. mimalloc’s WASI prim never calls random_get; it reports failure and then constructs a weak seed from clock and address. The minimal probe proved the secure RNG treatment was effective and that it is not on the real failure path.

This is an important point in the investigation: a change that seems reasonable is not enough; you need to prove the target code path actually consumes that input.

Fixing the clock at the right boundary

The final treatment did not globally change guest time APIs, and did not alter production runtime clock behavior. It only injected a fixed monotonic clock into the WASI context used by Wizer pack:

struct DeterministicPackMonotonicClock;

impl wasmtime_wasi::clocks::HostMonotonicClock
    for DeterministicPackMonotonicClock
{
    fn resolution(&self) -> u64 {
        1
    }

    fn now(&self) -> u64 {
        0
    }
}

let mut wasi = wasmtime_wasi::WasiCtxBuilder::new();
wasi.monotonic_clock(DeterministicPackMonotonicClock);

This boundary is important:

• build-time snapshot gets stable input;
• production runtime still uses real clock;
• no global random_get stub;
• no post-processing canonicalization on final Wasm;
• exact full-bundle equality gate remains unchanged.

In local Linux preflight, starting two separate CLI processes to pack the same retained raw Wasm and manifest-identical VFS produced byte-for-byte identical output.

Then, dual-builder CI passed for the first time.

Final evidence

In the final controlled CI run:

All four packed Wasms converged to the same SHA-256:

97539f2fa73e2cad5ea4c2b122e7a859ecdeb90b8f8505420f590ef3990b9446

Dual builder, per-side repeat-pack, and final exact comparison jobs all passed; downloaded artifact digests, bundle contracts, and local comparator replay were all consistent as well.

Which conclusions should not be overgeneralized

This result came from cumulative experiments. It supports these conclusions:

1. Dirent tail padding is a real drift contributor;
2. directory scratch-buffer residue is a smaller but real contributor;
3. after fixing these directory-representation issues, pack-time monotonic clock is the remaining necessary mechanism;
4. fixing the pack-context clock is enough to restore exact reproducibility in the current experimental sequence.

But it does not automatically prove that the current five research treatments form the minimal production patch set. deterministic HashMap hasher and avoiding unconsumed filestat metadata had already been shown individually insufficient; to move toward production repair, ablation should be done to remove those ineffective treatments and rerun the full gate.

It also does not imply that “all Wizer builds must freeze time.” The correct conclusion applies only to the validated chain here: CPython 3.14 mimalloc WASI weak seed during pack-time initialization, which is snapshotted into the final module.

What I kept from this investigation

1. Find the first stage where drift appears

Do not guess from the final binary. Keep raw, pack inputs, and repeat-pack evidence, and answer first: where do they diverge first.

2. Repeat packing on the same runner is more informative than cross-runner comparison

If independent processes on the same runner already differ, there is no need yet to chase kernel versions, CPU models, or runner images.

3. Structural differences are clues, not conclusions

“Only section 11 differs” can still contain directory padding, scratch residue, PRNG state, and allocator metadata. You still need to map it to guest memory and source.

4. Verify that treatment is truly effective by falsification

The secure RNG experiment is convincing not just because the real module still drifted, but because the minimal random_get probe proved the patched path actually worked.

5. Put fixes at the narrowest boundary where nondeterminism is introduced

Here the right boundary is a pack-only WASI context, not the production runtime and not byte rewriting of the final binary.

The hardest part of reproducible builds is usually not “there is a timestamp” alone, but time, randomness, uninitialized representation, and snapshot semantics combining at boundaries. The practical solution remains simple: freeze inputs, keep intermediate artifacts, change one hypothesis at a time, and let failures keep failing until the evidence chain closes.