What happens to my build when the cache server is down?
Does a cache server that is down break your build? In our runs with FosterStack Cache as the server, no: Gradle and Maven both finished the build locally and printed a warning. The page also covers what happens if you run a second server.
The short answer
- Server not running, or gone since the last build: the build succeeded in Gradle and in Maven. Everything was built locally, and a warning was printed.
- Wrong password (401): the same. The build succeeded and warned.
- Read-only login that tries to store a result (403): in Gradle the build succeeded and warned. We did not run this one in Maven.
- An address that never answers: the build still succeeded. In our timed Gradle runs it first waited about 30 seconds, which took a 13-second build to 44 seconds. Maven also waited; we did not time it. If the address answered with “connection refused” (nothing was listening there), we saw no clear extra time.
The warnings look different. Gradle printed one warning and then a line saying the remote cache was disabled. Maven printed an [ERROR] line, and then a Java stack trace for each upload it could not make, even though the build succeeded.
What to do
Our advice, not a test result:
- Search your build logs for the warning. A build that quietly stopped using the cache looks like a slow build. In Gradle, look for
Could not load entryorCould not store entry. In Maven, look forError downloading cache itemandUnable to save to remote cache. On Maven 3.10.0 an ordinary first miss also logs the first line, so read it together with the second. - Check the cache address. A server that is down on a machine you can reach refuses the connection at once, and that showed no clear extra time in our timed runs (we tested this on the same machine). An address that drops packets makes the build wait for the timeout. Make sure the address in your build is the right one.
- Shorten the timeout on CI. Gradle reads two system properties,
org.gradle.internal.http.connectionTimeoutandorg.gradle.internal.http.socketTimeout, both 30000 milliseconds by default. Maven readsaether.connector.connectTimeout, 10000 milliseconds by default.
In one local run each, setting them to 2000 ended the wait sooner. We did not time it on the hosted machine, so we give no numbers. We set them with -D on the command line, for example -Daether.connector.connectTimeout=2000. To use the Gradle properties in gradle.properties, prefix them with systemProp.; we did not test that form.
If the cache is up but nothing is restored, see Gradle remote build cache not working or Maven build cache extension not restoring. For logins and the 401 and 403 errors, see Gradle remote build cache authentication.
Can I run two build cache servers?
FosterStack Cache 0.2.1 runs as one server. In our runs, a second server on the same data folder refused to start, and two servers on separate folders restored fewer results than one. If the host fails, builds still succeed without the cache, and a copy of the data folder gets a replacement running.
- Two servers on one data folder. The second one exited with code 1 after about 5 seconds, in each of three tries. The first kept serving.
- Two servers on separate folders. Each stores its own copy. In eight builds, one server restored 25 of 32 compile tasks from the cache. Two servers, with builds taking turns, restored 18 of 32, fewer than the single server's 25.
- A spare from a copy. A server started on a copy of a stopped server's folder had all 14 entries. Builds restored them.
- Gradle used one remote cache in our run. With two remote blocks in the settings, only the second one was used.
Our advice, not a test result:
- Run one server and keep a copy of its data folder. The steps are on back up and restore a build cache.
- If the host fails, start a server on another host from the copy. Then change the cache address in your build settings. Read the address from one property or environment variable, as on use a remote build cache in GitHub Actions, so there is one place to change.
- Do not put a load balancer in front of two servers. Each build lands on a server that may not hold what the last build stored.
- Do not start a second server on the same folder. It refuses to start, so it gains you nothing.
- A restart is short: see can I restart the cache server while builds run.
How we checked
When the server is down
| What went wrong | Gradle 9.8.0 | Maven 3.9.9 |
|---|---|---|
| Server never started | Built locally, build succeeded. One warning: Could not load entry ... from remote build cache: Connect to 127.0.0.1 ... failed: Connection refused | Built locally, build succeeded. [ERROR] Error downloading cache item, then a stack trace per failed upload (Connection refused) |
| Server filled, then stopped, then a new build | Same as above | Same as above |
| Server stopped just before the build's first cache lookup (one attempt) | Same as above: refused at the first lookup | Not tested |
| Wrong password (401) | Built locally, build succeeded. Could not load entry ... response status 401: Unauthorized | Built locally, build succeeded. [ERROR] Error downloading cache item, then a stack trace saying 401 on upload |
| Read-only login, build has something to store (403) | Build succeeded. Could not store entry ... response status 403: Forbidden | Not tested |
| Address that never answers | Built locally, build succeeded. Connect timed out | Built locally, build succeeded. Stack trace saying Connect timed out. Four attempts against the same dead address |
The timed part (Gradle only). We timed four setups on a GitHub-hosted machine, seven times each, and recorded the whole build command.
| Setup | Fastest | Median | Slowest | Build result |
|---|---|---|---|---|
| No remote cache | 13.21 s | 13.37 s | 13.91 s | succeeded, 7 of 7 |
| Cache on, server up and empty | 13.75 s | 14.28 s | 14.51 s | succeeded, 7 of 7 |
| Cache on, connection refused | 13.64 s | 13.83 s | 14.11 s | succeeded, 7 of 7 |
| Cache on, address that never answers | 43.46 s | 43.98 s | 44.22 s | succeeded, 7 of 7 |
At the median, an address that never answers added 30.61 s. That matches Gradle's 30-second default connect timeout, so each of those builds waited about one timeout. A “connection refused” answer was not clearly different from no cache at all: 13.83 s against 13.37 s at the median, and the ranges overlap.
Timed runs: a GitHub-hosted machine, Ubuntu 24 image 20261004.327.1, 4 vCPUs (AMD EPYC 9V74), 16 GB. The run and its log are public, and the script is bin/bench-server-down.sh. Gradle 9.8.0, Java 21.0.12.1, FosterStack Cache 0.2.1, a project of 4 modules with 150 generated classes each, compile tasks only.
How: the four setups ran in turn, seven times, after one dropped warm-up round. Every build used a fresh copy of the project, a new empty Gradle home and no daemon, so each time includes Gradle's start-up. The "never answers" address was 10.255.255.1, which gave no answer within 5 seconds before the run. We did not time Maven.
In every Gradle case the last line of the build said the remote cache was disabled during the build due to errors, and in the builds we printed the failure appeared once. In Maven the one-module project made four attempts: one download and three uploads.
What: Gradle 9.8.0; Maven 3.9.9 with Build Cache Extension 1.2.3 and all eight plugin versions pinned; Temurin 21.0.12.1; FosterStack Cache 0.2.1. Gradle used a four-module project and the compile tasks, with a new Gradle home and no daemon. Maven used a one-module project with two tests and mvn verify. Both on a Mac (arm64).
Limits: the behaviour table above comes from a Mac; the times come only from the hosted machine. We did not test a server that dies after serving some entries in the same build, a slow server that is still up, a multi-module Maven build, TLS or DNS errors, or other versions.
Two servers
FosterStack Cache 0.2.1 on a Mac (arm64), no password. Gradle 9.8.0 with Java 21.0.12.1. A four-module Java project, with the compile tasks only. Every build used a fresh copy of the project and a new Gradle home, with the local build cache off. We started each server ourselves and stopped it by its process ID. Each row is one run, and we report counts, not build times.
| Setup | What we did | What happened |
|---|---|---|
| Two servers, one data folder | Started the first and built. Started a second on the same folder, once with a build and then in three timed tries. | The second exited with code 1 every time. The three timed tries took about 5 seconds each. It logged open metadata store: metadata: open: timeout. The first answered /healthz and the next build restored 4 of 4. |
| Same, then a build aimed at the second server | One build pointed at the address of the second server, which had already exited. | The build succeeded, restored 0 of 4 and printed one Could not load entry warning. |
| One server, eight builds | Base, base again, then three edits, each built twice. | Restored 0, 4, 3, 4, 3, 4, 3 and 4 of 4 compile tasks, 25 of 32. The server logged 16 misses and held 16 entries. |
| Two servers, separate folders, builds taking turns | The same eight builds, the first of each pair to server A and the second to server B. | Restored 0, 0, 3, 3, 3, 3, 3 and 3, 18 of 32. The two servers logged 16 misses each, 32 in all, and held 16 entries each. |
| A spare from a copy | Built twice on A. Stopped A, copied its data folder, started B on the copy. | B showed 14 entries. Builds on B restored 4 of 4 for both earlier builds, and 3 of 4 for a new edit. B then held 15 entries. |
| Build aimed at the stopped server | One build pointed at A's address after A stopped. | The build succeeded, restored 0 of 4 and printed one Could not load entry warning. |
| Two remote blocks in one settings file | One block for a live server and one for an address with nothing listening, in both orders. | Gradle accepted both. With the dead address second, the build restored 0 of 4 and the remote cache was disabled for the rest of the build. With the live server second, it restored 4 of 4. |
The server counters count every request, including the compiled build scripts, so a build adds 13 hits or misses. The compile tasks are the four that the table counts. All builds ended with BUILD SUCCESSFUL.
What Gradle says: its build cache guide says it supports a local and a remote build cache that can be configured separately. FosterStack's own Kubernetes notes give the reason a shared folder fails: the metadata file is held under an exclusive lock by one process.
Not tested: Linux, Docker or Kubernetes, a real load balancer, a folder on a network disk, Maven, more than two servers, and copying the folder of a server that is still running.