Build cache size limit, eviction and a full disk

A self-hosted build cache cannot grow forever. FosterStack Cache has a size cap, and when it is reached it removes the entries used least recently. This page shows what that looks like in real 0.2.1 runs, and what happens when the disk itself fills up.

Setting the cap

Set FSCACHE_MAX_BYTES, in bytes. The default, 0, means no limit. Real servers use gigabytes; see server sizing for values you can copy.

How the size cap works

• In our run with a 3,500-byte cap, the stored size never went above the cap. Each time something new did not fit, the entry that had gone longest without being used was removed.
• Reading counts as using: a cache hit keeps an entry alive. In the last step of our run k1 went too, even though we had read it earlier, because our own checks after storing k4 read k1, k3 and k4 in that order, which made k1 the oldest again.
• An entry bigger than the whole cap is refused. We tried storing 4,000 bytes against the 3,500 cap. The server refused it with a 413 and stored nothing, and nothing else was evicted to make room. The other limit, FSCACHE_MAX_BODY_BYTES, is separate and defaults to 1 GiB per entry. See the status codes page.

What happens when the build cache disk fills up?

In our runs, a full disk did not break builds. The cache server refused new uploads and kept serving what it already held. Gradle still ended with BUILD SUCCESSFUL.

• New uploads failed with a 500. The server logged no space left on device each time. It counted them in the fscache_http_requests_total metric, as PUT requests with status 500.
• Reads kept working. Entries already stored came back with a 200. A build restored 4 of 4 cacheable tasks from the cache while the disk was full.
• A build that needed to store got one warning. Gradle printed Could not store entry, then turned the remote cache off for the rest of that build. The build still passed.
• The server stayed up. /healthz still answered ok, so it does not tell you the disk is full.
• It recovered once we freed space. No restart was needed. Uploads were accepted again until the space ran out.

What to do

Our advice, not a test result:

• Set a size cap below the disk size. The server then removes the least recently used entries instead of failing. In our runs a 4 MiB cap on an 8 MiB volume never failed an upload. A 20 MiB cap, bigger than the volume, did not help. See how the size cap works.
• Alert on failed uploads. Watch for fscache_http_requests_total with method="PUT" and status="500" going up. Your builds will not tell you. See what to alert on.
• To recover, free space on the volume. In our runs the running server noticed and needed no restart.
• Free space before you restart the server or start a new, empty data folder. On a full disk the server would not start. The log said open metadata store and no space left on device.

Watching it on a real cache

The same numbers are in /statusz as store_bytes, max_bytes, store_entries and evicted_entries, and in the Prometheus metrics (fscache_evicted_entries_total and fscache_store_bytes), except that the two store gauges read 0 after a restart until the next upload. A rising eviction count together with a hit rate that never climbs means the cap is too small for your build. See what to alert on.

How we checked

A deliberately tiny cap

FosterStack Cache 0.2.1. We used 3500 bytes, so that a few small entries fill it:

FSCACHE_MAX_BYTES=3500 ./fscache

We stored three 1,000-byte entries, read one of them, then stored two more. After each step we looked at /statusz, the status page, and tried to read every key back.

A real full disk

FosterStack Cache 0.2.1 on a Mac (arm64), no password, and no size cap unless a row says so. We made a real 8 MiB disk image with APFS, mounted it, and put the data folder on it. Gradle 9.8.0 and Java 21 built a four-module project with a new Gradle home each time, no reused daemon and the local build cache off. Each row is one run, and we did not repeat them.

We found no leftover temporary files in the data folder after the first failed uploads. On this small APFS volume, uploads started to fail while df still showed about 1 MiB free, so do not plan around the last megabyte.

Not tested: Docker or Kubernetes volumes, file systems other than APFS (the second run below adds a Linux RAM disk), Maven, a very large single upload, a full inode table, a disk quota, releases other than 0.2.1 and 0.2.2, and a disk that fills while many builds upload at once.

Try the tiny-cap run yourself

FSCACHE_ADDR=127.0.0.1:8080 FSCACHE_DATA_DIR=./data FSCACHE_MAX_BYTES=3500 ./fscache &
until curl -s localhost:8080/healthz >/dev/null; do sleep 0.2; done   # wait until it is up

put() { head -c 1000 /dev/zero | curl -s -o /dev/null -w "PUT $1 %{http_code}\n" -X PUT --data-binary @- localhost:8080/$1; }
get() { curl -s -o /dev/null -w "$1=%{http_code} " localhost:8080/$1; }
put k1; put k2; put k3
get k1; echo
put k4
for k in k1 k2 k3 k4; do get $k; done; echo
curl -s localhost:8080/statusz | grep -E 'store_bytes|evicted'

# the output of the middle line in my run:
# k1=200 k2=404 k3=200 k4=200

Related: how to clear a build cache and Gradle and Maven setup.

Second run: Linux, 0.2.2

The runs above were on a Mac with 0.2.1. We ran the tiny-cap table and the size-cap runs again on a GitHub-hosted Linux machine with FosterStack Cache 0.2.2. For the full-disk runs we used an 8 MiB RAM disk there (tmpfs, a small disk that lives in memory, so it can be filled safely).

• The 3,500-byte table gave the same rows, and the 4,000-byte entry got the same 413. A 4 MiB cap with forty 256 KiB uploads accepted all forty, removed 24 entries and held 4,194,304 bytes.
• On the full disk, uploads got 500 with the body internal error, reads kept working, a build still ended with BUILD SUCCESSFUL and printed one Could not store entry line, and uploads were accepted again once we freed space, with no restart.
• The counts differ by file system. On Linux, 25 uploads of 256 KiB were accepted before the first failure (21 on the Mac). A 20 MiB cap let 31 of forty uploads through (25 on the Mac).
• After we filled the volume until a write failed, the server would not start, even with its old data. The log said write shutdown marker and no space left on device. When we shrank a filler file to leave 16 KiB free, it started with its old data, and a new, empty data folder failed at open metadata store, as in the table.

The run and its log are public.