Caching GDAL and PROJ Layers in CI Containers

A spatial generation pipeline whose containers install GDAL and PROJ from scratch on every run pays a double tax: minutes of rebuild time on each job, and intermittent Cannot find proj.db or wrong-datum-shift failures whenever the PROJ data directory is not where the runtime expects it.

Part of CI/CD Integration for Spatial Data: that stage automates generation, evaluation, and release, and this page removes the single slowest and flakiest link in it — the geospatial native stack — by pinning GDAL and PROJ, baking their data layers into a cached image, and keying that cache so it is both fast and reproducible.

Root Cause: PROJ Data Resolution and Cold Rebuilds

PROJ does coordinate transformations by reading two kinds of data at runtime: proj.db, the SQLite catalogue of CRS and transformation definitions, and the datum-shift grids (NTv2 .tif/.gsb, GTX geoid models, and the horizontal/vertical grids that make transforms accurate to centimetres rather than metres). PROJ finds these by searching a data directory identified through an environment variable — PROJ_LIB on PROJ 8 and earlier, renamed to PROJ_DATA on PROJ 9 — falling back to a compiled-in path. GDAL, in turn, links against a specific PROJ and adds its own GDAL_DATA directory for driver metadata.

Two failure classes follow directly.

Path resolution failures. When a container installs pyproj from a binary wheel and installs system GDAL from the OS package manager, two independent copies of PROJ arrive with two independent data directories. The wheel bundles its own proj.db under the Python package; the system PROJ expects its data under /usr/share/proj. Whichever PROJ_DATA/PROJ_LIB the environment sets wins for the whole process, so one of the two libraries is now pointed at a data directory built for a different PROJ version. The symptom is a hard Cannot find proj.db when the variable points nowhere valid, or the subtler pj_obj_create: Cannot find proj.db and silent fallback to a low-accuracy transform when the version of proj.db does not match the linked library. These are exactly the CRS integrity problems the enforcing CRS contracts in a generation pipeline stage is meant to catch — but a contract check cannot repair a broken data directory, only fail on it.

Cold rebuilds. The datum grids are large and are fetched on demand by projsync or downloaded lazily from a CDN the first time a transform needs one. A CI job that starts from a bare image reinstalls the native stack and re-downloads grids on every run — minutes of wall-clock per job, plus a hard dependency on network egress that turns a CDN hiccup into a red build. Because the download is lazy and order-dependent, two runs can even end up with different grid sets, so a transform that was centimetre-accurate yesterday silently degrades today.

The fix treats the GDAL/PROJ native stack and its data grids as a pinned, content-addressed artifact: fix the versions, bake proj.db and the required grids into an image layer, set PROJ_DATA/GDAL_DATA explicitly, and cache the layer under a key hashed from the pinned versions and a grid manifest so a rebuild only happens when one of those inputs actually changes.

Content-addressed caching of the GDAL and PROJ layer in CI The CI build trigger feeds a cache-key node that hashes the pinned GDAL and PROJ versions together with a grid manifest. A decision node asks whether the cache key hits. On a hit, the flow restores the prebuilt GDAL, PROJ, and grid layer in seconds. On a miss, it installs the pinned GDAL and PROJ, runs projsync to fetch the datum grids over minutes, then saves the cache under the key. Both branches converge on the runtime container, which sets PROJ_DATA to a fixed directory so proj.db and the datum grids resolve deterministically and transforms stay centimetre-accurate. CI build trigger cache key = sha256( GDAL 3.8 + PROJ 9.3 + grid manifest ) cache hit? HIT · restore prebuilt layer GDAL + PROJ + datum grids, ready seconds, no network MISS · install pinned stack projsync fetches datum grids then save cache under key minutes, needs egress runtime: PROJ_DATA=/opt/proj proj.db + datum grids resolve deterministic, cm-accurate yes no
The cache key is content-addressed: identical pinned versions and grid manifest hit the cache and restore in seconds; any change misses and rebuilds once.

Minimal Reproducer: Provoke the PROJ_LIB Failure

Reproduce the resolution failure in a throwaway container so the fix can be verified against it. The mismatch appears the moment the data directory does not match the linked library.

bash
# A wheel-installed pyproj alongside system GDAL, with PROJ_DATA pointed at the
# wrong directory, is enough to break transform resolution.
docker run --rm python:3.11-slim bash -c '
  pip install --quiet pyproj==3.* &&
  apt-get update -qq && apt-get install -y -qq gdal-bin >/dev/null &&
  export PROJ_DATA=/usr/share/proj &&      # system PROJ dir, NOT the wheel-s
  python - <<PY
from pyproj import Transformer
# NAD83 -> UTM 10N needs a datum grid; resolution fails or silently degrades.
t = Transformer.from_crs("EPSG:4269", "EPSG:26910", always_xy=True)
print(t.transform(-122.4, 37.8))
PY'
# pyproj.exceptions.ProjError: Cannot find proj.db   (or a low-accuracy fallback)

bash
# Confirm which data dir each library actually resolves — the two often disagree.
python -c "import pyproj; print('pyproj:', pyproj.datadir.get_data_dir())"
projinfo --version        # system PROJ version + its data search path

Fix: Pin, Bake, and Key the GDAL/PROJ Layer

The fix has three parts: pin GDAL and PROJ to exact versions from a single source so no second copy sneaks in, bake proj.db and the required grids into a dedicated image layer with PROJ_DATA set explicitly, and key the CI cache on a hash of the pinned versions plus a grid manifest so the layer is content-addressed. Pinning the versions is the same discipline the scoping rules and data contracts stage applies to every dependency the pipeline promises to reproduce.

Step 1 — Multi-stage Dockerfile with an explicit PROJ data layer

dockerfile
# syntax=docker/dockerfile:1
# --- grid stage: fetch ONLY the datum grids the pipeline actually uses ---
FROM ghcr.io/osgeo/gdal:ubuntu-small-3.8.4 AS grids
# PROJ ships inside this pinned GDAL image; do not add a second copy.
ENV PROJ_DATA=/opt/proj
RUN mkdir -p /opt/proj && cp -r /usr/share/proj/* /opt/proj/
# projsync only the regions we transform, not the whole global set.
RUN projsync --target-dir /opt/proj \
      --file us_noaa_nadcon5 \
      --file ca_nrc_ntv2 \
      --file eur_nkg && \
    sha256sum /opt/proj/proj.db /opt/proj/*.tif > /opt/proj/GRID_MANIFEST.sha256

# --- runtime stage: copy the baked layer, set the env once ---
FROM ghcr.io/osgeo/gdal:ubuntu-small-3.8.4 AS runtime
ENV PROJ_DATA=/opt/proj \
    GDAL_DATA=/usr/share/gdal \
    PROJ_NETWORK=OFF                      # never lazily download at runtime
COPY --from=grids /opt/proj /opt/proj
# Install pyproj with NO bundled data, so it uses the system proj.db only.
RUN pip install --no-binary pyproj "pyproj==3.*"

PROJ_NETWORK=OFF is the load-bearing setting: it forbids the lazy CDN download entirely, so a missing grid fails loudly at build time instead of degrading silently at run time. Installing pyproj with --no-binary prevents the wheel from shipping its own proj.db, collapsing the two-copy problem to one.

Step 2 — Hashed cache key from versions and grid manifest

Compute the cache key from the exact inputs that determine the layer’s contents. When none of them change, the key is identical and the cache hits; when any changes, the key changes and the layer rebuilds exactly once.

yaml
# .github/workflows/build.yml — content-addressed layer cache
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      # Key = hash of the Dockerfile (pins GDAL/PROJ) + the resolved grid manifest.
      - name: Compute layer cache key
        id: key
        run: |
          echo "hash=$(cat Dockerfile grids/GRID_MANIFEST.sha256 \
            | sha256sum | cut -d' ' -f1)" >> "$GITHUB_OUTPUT"

      - name: Restore GDAL/PROJ layer cache
        uses: actions/cache@v4
        with:
          path: /tmp/proj-layer.tar
          key: gdalproj-$   # miss only on real change
          # No restore-keys: a partial/older layer must NOT be reused.

      - name: Build (buildx honours the cache mount)
        run: docker buildx build --tag gen:ci --load .

Omitting restore-keys is deliberate: a fuzzy prefix match would restore a stale grid layer under a new version pin, silently reintroducing the mismatch this whole exercise removes. A grid layer must be reused only on an exact key match. This mirrors the reproducibility contract in syncing synthetic data generation with GitHub Actions, where every cached artifact is content-addressed rather than time- or branch-addressed.

Verification Step: Assert Resolution and Manifest Integrity

Run a fast smoke test inside the built image so a broken data directory fails the build, not a downstream generation job.

python
# tests/test_proj_layer.py — runs inside the built container in CI
import hashlib, os, pathlib
from pyproj import Transformer, datadir

def test_proj_data_dir_is_the_baked_one():
    # pyproj must resolve the SAME dir the image baked, not a wheel copy.
    assert datadir.get_data_dir() == os.environ["PROJ_DATA"] == "/opt/proj"

def test_proj_db_present_and_network_off():
    assert pathlib.Path("/opt/proj/proj.db").is_file(), "proj.db missing"
    assert os.environ.get("PROJ_NETWORK") == "OFF", "lazy download not disabled"

def test_datum_grid_transform_is_high_accuracy():
    # A NAD83 -> UTM transform must use the baked grid, not a null-shift fallback.
    t = Transformer.from_crs("EPSG:4269", "EPSG:26910", always_xy=True)
    x, y = t.transform(-122.4, 37.8)
    assert 550_000 < x < 560_000 and 4_180_000 < y < 4_190_000

def test_grid_manifest_matches_baked_layer():
    # Recompute hashes; any drift means the cache key no longer describes the layer.
    manifest = pathlib.Path("/opt/proj/GRID_MANIFEST.sha256").read_text().split("\n")
    for line in filter(None, manifest):
        want, path = line.split()
        got = hashlib.sha256(pathlib.Path(path).read_bytes()).hexdigest()
        assert got == want, f"grid drift: {path}"

The manifest test is what keeps the cache honest: if the baked grids ever diverge from the manifest the key was computed over, the content-addressing guarantee is broken and the build must fail rather than promote a mislabelled layer.

Edge Cases & Gotchas

PROJ_LIB vs PROJ_DATA across a version bump. PROJ 9 renamed the data-directory variable from PROJ_LIB to PROJ_DATA but still honours the old name for compatibility. A base image that upgrades PROJ from 8 to 9 while your CI still exports only PROJ_LIB can appear to work until a tool reads PROJ_DATA first and finds it unset, resolving to the compiled-in fallback instead. Set both variables to the same directory during any migration window, and fold the PROJ version into the cache key so the bump forces a clean rebuild.

Missing regional grid, silent low-accuracy fallback. If a transform needs a datum grid you did not projsync (say an Australian GDA2020 grid absent from a US-only manifest), PROJ with PROJ_NETWORK=OFF falls back to a null or ballpark transformation that is off by a metre or more — no error, just wrong coordinates. Enumerate every source and target CRS the pipeline actually uses, derive the grid list from that, and assert high-accuracy output for a representative transform per region so a missing grid fails the manifest test.

Antimeridian and datum-boundary precision. Transforms near the ±180∘\pm180^\circ seam or at the edge of a grid’s coverage polygon can land just outside the grid extent and fall back to the coarser transform for that one point while neighbours use the fine grid, producing a discontinuity of centimetres to metres exactly at the boundary. Test coordinates a few cells inside each grid’s coverage rather than on its edge, and clamp or reproject seam-crossing geometry into a local CRS before transforming, as the CRS contract stage requires.

Frequently Asked Questions

Should I set PROJ_NETWORK to ON so missing grids download automatically?
Not in CI. Leaving network downloads on makes builds depend on CDN availability and lets two runs end up with different grid sets, which defeats reproducibility and can silently change transform accuracy between builds. Bake the exact grids you need at image-build time with projsync, set PROJ_NETWORK to OFF, and let a missing grid fail loudly. Reserve network fetching for interactive development, never for the pipeline of record.
Why hash a grid manifest into the cache key instead of just the version numbers?
Version pins alone do not capture which regional grids you chose to sync, and projsync content can change under a fixed PROJ version. Hashing a manifest of the actual proj.db and grid file checksums makes the cache key describe the layer's true contents, so the cache misses whenever the grids change even if the version string did not. That is the difference between a cache that is merely fast and one that is also correct.
Can I avoid the two-copies problem without building pyproj from source?
Yes. The goal is one PROJ and one data directory in the process. Either install pyproj against the system PROJ (the no-binary approach shown here) so it shares proj.db, or go the other way and rely entirely on the pyproj wheel's bundled data by pointing PROJ_DATA at the wheel's directory and not installing a separate system PROJ. What breaks is having both with conflicting data dirs; pick one authority and set the environment to match it.
How much build time does caching the layer actually save?
The saving is the cost of reinstalling the native stack plus every lazy grid download, which is typically the dominant term in a cold spatial CI job — often several minutes per run. On a cache hit the layer restores in seconds with no network calls, so the payoff scales with how often the pipeline runs. The versions and grids change rarely, so most runs hit the cache and skip the expensive path entirely.