Recovering a Partially Written Grid After a Worker Crash

A worker died eleven hours into a fourteen-hour grid build. Most of the output is on disk. The question is whether any of it can be trusted, and the honest answer at that moment is usually no — not because the data is wrong, but because nothing recorded which tiles finished.

Part of Async Execution for Large Grids: parallel execution makes partial failure the normal case rather than the exceptional one, and a build with no resume story converts every transient fault into a full restart.

Root Cause: Existence Is Not Completion

The natural resume check is “does the output file exist?” and it is wrong in a specific, damaging way: a worker killed mid-write leaves a file that exists, is readable, has plausible dimensions, and is missing its last rows. The resume skips it, the build reports success, and the defect surfaces months later as a band of nodata in a downstream product.

The problem compounds under parallelism. A crashed worker may have been partway through several tiles; the scheduler may have retried a task whose first attempt is still flushing; and two processes writing the same path concurrently produce a file that is neither attempt.

Three properties are needed, and they are cheap:

  • Atomic publication. A tile appears at its final path only when it is complete. Write to a temporary name and rename, because rename within a filesystem is atomic and a partially written temporary file is obviously garbage.
  • Independent completion evidence. A record, written after the rename, saying what was produced and what it hashes to. The file’s own existence cannot be that evidence, because the file is what is in question.
  • Deterministic task identity. Tile n must be the same work on the resume as it was on the original run, or the resumed output is a mixture of two different builds.
Four resume strategies against four partial-write cases A matrix with four resume strategies as rows and four failure cases as columns. Checking whether the output file exists accepts a truncated file, a file still being flushed, and a file left over from a run with different code — it only rejects a tile that is entirely absent. Checking the file size against an expected value rejects truncation but still accepts a file being flushed whose size has reached the target, and cannot see a code change at all. Checking a hash recorded before the write rejects truncation and stale content, but claims tiles that never reached their final path if the run died between recording and renaming. Writing the manifest entry after an atomic rename rejects all four, because the entry can only exist if the complete file already does. The note underneath makes the ordering explicit: rename first, then record, because the reverse produces a manifest that claims a tile which is not there — a worse failure than the one being fixed, since it is trusted. Rename first, then record — the reverse order is worse than no manifest resume strategy truncated still flushing stale code absent file exists accepts accepts accepts rejects file size matches rejects accepts accepts rejects hash recorded before write rejects rejects rejects claims it manifest entry after rename rejects rejects rejects rejects The third row's failure is the instructive one. Recording completion before the file is in place produces a manifest claiming a tile that does not exist, and because the manifest is the thing everything downstream trusts, that is worse than having no manifest at all — the resume skips work that was never done and reports success.
Four resume strategies against what each one accepts: only a post-rename manifest entry rejects every partial-write case.

Prerequisite Check: What Does the Current Build Leave Behind?

Before designing a resume, establish what evidence the existing build actually produces, because the answer is often “less than assumed”.

python
def audit_partial_output(tile_dir, expected_tiles) -> dict:
    """Classify what is on disk after a failed run."""
    present = {p.stem for p in tile_dir.glob("*.tif")}
    temp = {p.stem for p in tile_dir.glob("*.tmp*")}
    return {
        "expected": len(expected_tiles),
        "present": len(present),
        "temporaries_left": len(temp),      # a nonzero count means non-atomic writes
        "missing": sorted(set(expected_tiles) - present)[:20],
        "unverifiable": len(present),       # until there is a manifest, all of them
    }

The unverifiable field is deliberately blunt. Without a manifest, every present file is a candidate truncation, and the only sound recovery from that state is to rebuild everything — which is exactly the cost the resume was meant to avoid, and the argument for adding the manifest before the next long run rather than after the next failure.

Fix: Atomic Writes, a Manifest, and Deterministic Task Identity

1 — Publish atomically

python
import os
import tempfile


def write_tile(tile_id: str, array, out_dir, profile):
    """Write to a temporary name in the same directory, then rename into place."""
    fd, tmp = tempfile.mkstemp(dir=out_dir, prefix=f".{tile_id}.", suffix=".tmp")
    os.close(fd)
    try:
        save_raster(tmp, array, profile)
        os.replace(tmp, out_dir / f"{tile_id}.tif")   # atomic within one filesystem
    except BaseException:
        os.unlink(tmp)
        raise

Same directory, so the rename stays within one filesystem — a rename across filesystems is a copy followed by a delete, and is not atomic. The BaseException catch rather than Exception is deliberate: a worker killed by a signal that raises should still clean up its temporary.

2 — Record completion after the rename, with a hash

python
def record(manifest_path, tile_id: str, path, seed: int, code_version: str):
    """Append-only completion evidence, written after the file is in place."""
    entry = {
        "tile": tile_id,
        "sha256": sha256_of(path),
        "bytes": path.stat().st_size,
        "seed": seed,
        "code_version": code_version,
    }
    with open(manifest_path, "a") as fh:               # append is atomic for small lines
        fh.write(json.dumps(entry, sort_keys=True) + "\n")
        fh.flush()
        os.fsync(fh.fileno())

Order matters absolutely: rename first, then record. Recording first and crashing before the rename produces a manifest claiming a tile that is not there, which is a worse failure than the one being fixed because it is trusted.

3 — Derive each tile’s seed from its identity, not from a counter

python
def tile_seed(root_seed: int, tile_id: str) -> int:
    """Same tile, same seed, on the original run and on every resume."""
    return int.from_bytes(
        hashlib.sha256(f"{root_seed}:{tile_id}".encode()).digest()[:8], "big"
    )

Without this the resume is not a resume. A counter-based seed assigns different values when tasks run in a different order, so the recovered tiles come from a different realisation than their neighbours and the seams show — the same derivation-tree argument, with a failure mode that only appears after a crash.

The same crash under two write orders Two horizontal timelines share the same sequence of events. In both, three tiles complete normally, a fourth is midway through its write when the worker is killed, and a fifth never starts. The upper timeline records the manifest entry before the rename. At the moment of the crash the manifest already claims the fourth tile, but only a temporary file exists at that path, so the resume skips a tile that was never published and the build finishes reporting success with a hole in it. The lower timeline renames first and records afterwards. At the crash the fourth tile has neither a final path nor a manifest entry, so the resume re-runs it, and the only cost is repeating work that was genuinely incomplete. Beneath each timeline a summary states what the resume does and whether the finished build is correct. The note underneath generalises the rule to any publish-then-record pipeline. The crash is identical; the write order decides whether the build is correct record → rename tile 1 ok recorded tile 2 ok recorded tile 3 ok recorded tile 4 — killed mid-write recorded, not renamed tile 5 never ran — resume skips tile 4 — build completes with a hole rename → record tile 1 ok renamed + recorded tile 2 ok renamed + recorded tile 3 ok renamed + recorded tile 4 — killed mid-write neither — temp discarded tile 5 never ran — resume re-runs tile 4 — build is correct The rule generalises past grids: in any pipeline where one step publishes an artifact and another records that it exists, the record has to come second. Recording first makes the failure window produce a claim without a thing, and a claim without a thing is trusted by everything downstream.
A crash mid-build under the two write orders: what is on disk, what the manifest claims, and which tiles the resume re-runs.

Verification Step: Prove the Resume Is Equivalent to a Clean Run

python
def test_resume_matches_clean_run(tmp_path, tiles):
    clean = build_all(tmp_path / "clean", tiles, root_seed=7)
    partial = build_all(tmp_path / "resumed", tiles[:len(tiles) // 2], root_seed=7)
    resumed = build_all(tmp_path / "resumed", tiles, root_seed=7, resume=True)
    for tile in tiles:
        assert sha256_of(clean[tile]) == sha256_of(resumed[tile]), f"{tile} differs"


def test_truncated_tile_is_rebuilt(tmp_path, tiles):
    out = build_all(tmp_path, tiles, root_seed=7)
    truncate(out[tiles[3]], fraction=0.6)              # simulate a killed writer
    rebuilt = build_all(tmp_path, tiles, root_seed=7, resume=True)
    assert verify_against_manifest(rebuilt), "a truncated tile survived the resume"


def test_manifest_entry_implies_a_complete_file(manifest_path, tile_dir):
    for entry in read_manifest(manifest_path):
        path = tile_dir / f"{entry['tile']}.tif"
        assert path.exists(), f"manifest claims {entry['tile']} which is absent"
        assert sha256_of(path) == entry["sha256"], f"{entry['tile']} does not match its hash"


def test_no_temporaries_survive(tile_dir):
    assert not list(tile_dir.glob("*.tmp*")), "non-atomic write path"

The first test is the one that makes the feature trustworthy, and it is the one most often skipped because it needs a full small build run twice. It is worth the seconds: a resume that produces plausible output rather than identical output is a silent correctness bug with a long latency.

Expected build time under full restart versus tile-level resume The horizontal axis is the probability that any single task fails, running from zero to just under two percent. The vertical axis is expected total wall-clock time in hours for a build of nine hundred tasks that takes fourteen hours when nothing goes wrong. The full-restart curve begins at fourteen hours and rises steeply: with nine hundred tasks, even a one-in-a-thousand per-task failure rate means most attempts contain at least one failure, so the expected number of complete restarts climbs quickly and the curve leaves the top of the chart. The tile-level resume curve is almost flat across the whole range, because a failure costs one task's worth of repeated work rather than the entire build. A marker shows where the restart curve first exceeds double the clean build time. The note underneath gives the reading: the resume is not an optimisation for unreliable infrastructure, it is what makes a build of this size finish at all on reliable infrastructure. At 900 tasks, a one-in-a-thousand failure rate is not a rare event 0.0% 0.4% 0.8% 1.2% 1.6% 0 50 100 150 200 per-task failure probability expected build time (hours) full restart tile-level resume restart doubles at 0.10% 900 tasks, 14 hours clean. The restart curve is not about unreliable infrastructure: at this task count a per-task failure probability of one in a thousand already means most whole-build attempts contain a failure. The resume is what makes a build of this size finish at all, not an optimisation for bad hardware.
Expected total build time against per-task failure probability, for full restart versus tile-level resume.

Edge Cases & Gotchas

Object storage has no atomic rename. On S3-like stores a rename is a copy plus a delete. Publication by multipart upload completion is atomic and is the equivalent primitive; a manifest entry written after the completion call is still the evidence.

The manifest itself is a single point of contention. Appending from hundreds of workers to one file works on a POSIX filesystem for short lines and does not work on object storage. One small manifest file per tile, listed at resume time, is slower to enumerate and much harder to corrupt.

Code version in the manifest is not optional. A resume run with different code produces tiles that differ from their neighbours for reasons no hash check will surface. If the recorded code_version does not match, the correct behaviour is to refuse the resume rather than to warn.

Partial tiles inside a single output file. A build writing all tiles into one large file cannot use rename at all, and needs either a sidecar recording which windows are complete or a write-ahead approach. This is the strongest practical argument for tiled output on any build long enough to fail.

What the Manifest Is Worth Beyond Recovery

The manifest is introduced here to make a resume safe, and once it exists it turns out to be the artifact several other things were missing.

It is the completeness proof. A consumer asking “is this release whole?” currently has to enumerate files and trust that the expected set is known. A manifest listing every tile with its hash answers the question directly, and answers it the same way months later when the expected set has been forgotten.

It is the diff between two releases. Two manifests compared by hash immediately give the set of tiles that changed, which is both a useful release note and the cheapest possible input to incremental downstream processing. Without it, “what changed” requires reading every tile of both releases.

It records the code version at tile granularity. A build that spans a deployment — long builds do — produces tiles from two code versions, and the manifest is the only place that fact survives. This is the same information lineage tracking needs at the release level, recorded one level finer because the failure mode demanded it.

It makes corruption detectable rather than inferable. Storage bit-rot, an interrupted copy, a partial sync to another bucket: all of them produce a file whose hash no longer matches its recorded value, and none of them are visible any other way.

None of this justifies building a manifest on its own — but it is worth knowing that the cost is already paid once the resume exists, and that the format should be chosen with these uses in mind rather than as a minimal resume marker.