Can a Failed or Cancelled Build Poison the Cache?
In our runs, a failed or stopped build left no half-built result in FosterStack Cache. Only tasks that finished were stored, and a job killed during an upload left nothing behind.
The short answer
- A failed compile stored nothing for the failing module. The modules that compiled were stored. The next build took them from the cache.
- A failed test was not stored. Running it again ran it again and it failed again. Only the passing result was stored, after we fixed it.
- Ctrl-C and a hard kill gave the same result. The two tasks that had finished were stored, and nothing else was. The next build took those two from the cache and ran the rest.
- A job killed during an upload left no partial entry. The next build stored the result and the build after that restored it, byte for byte.
- One kind of wrong result does get in. A task whose output depends on something Gradle does not track, such as the clock, is stored with whatever it produced the first time. That is a build-script problem, not a failed-build problem.
What to do
Our advice, not a test result:
- Do not clear the cache because a build failed. In our runs the failure stored nothing to clear.
- If a task keeps giving an old result, look for an input Gradle cannot see. A clock, an environment variable or a file outside the project are the usual ones. Declare that value as a task input, or turn off caching for that task.
- If you still suspect a bad entry, see how to clear or reset a self-hosted Gradle build cache. It clears everything.
- Limit who can write. Give developers the read-only login, as on Gradle build cache: let CI write, developers only read. Someone who writes bad results on purpose is a different question, on can someone poison our shared build cache.
- If the server is stopped, not the build, see can I restart the cache server while builds run.
How we checked
FosterStack Cache 0.2.1 on a Mac (arm64), no password, a fresh data folder for each case. Gradle 9.8.0 with Java 21, one fresh copy of the project for every build, no reused daemon, the local build cache off. All builds shared one Gradle home, filled first by a build with the cache off. We counted entries with the server's /statusz page and listed its data folder. Each row is one run; we did not repeat them.
| What went wrong | What the build did | Entries stored | The next builds |
|---|---|---|---|
Syntax error in module b. Three modules, compileJava --continue | a and c compiled. b failed. Exit code 1 | 2 (a and c) | Same build again: a and c from the cache, b failed again, still 2 entries. Code fixed: b ran, a and c from the cache, 3 entries. Fixed again in a new copy: all three from the cache |
| A JUnit test fails | compileJava and compileTestJava ran. The test failed. Exit code 1 | 2 (the two compile tasks) | Same build again: both compile tasks from the cache, the test ran and failed again, still 2 entries. Test fixed: compileTestJava and the test ran, 4 entries. Again in a new copy: the test from the cache |
Ctrl-C (SIGINT) to the Gradle process, 3 seconds after the second of four 8-second tasks had finished | Tasks 1 and 2 had finished. Gradle exited with code 130. No Gradle process was left 6 seconds later | 2 (tasks 1 and 2). No temporary file | Next build: tasks 1 and 2 from the cache, tasks 3 and 4 ran, all four outputs correct. The build after: all four from the cache |
SIGKILL to the whole build, at the same moment | Tasks 1 and 2 had finished. The build was killed, so it has no exit code. No Gradle process was left 6 seconds later | 2. No temporary file | Same as Ctrl-C |
SIGKILL to the whole build while it was uploading a 300 MiB result | The server held a 2.8 MiB temporary file when we killed it | 0. The temporary file was gone | Next build: the task ran and stored the result. The build after: from the cache, and the file was identical to the one the build before it made |
curl killed while uploading 40 MiB at 4 MiB a second | The server held a temporary file of about 16 MiB, then logged blobstore: write: unexpected EOF | 0. The temporary file was gone | Reading that key gave 404. The same upload left alone gave 201 and then 200 |
| A task writes the time into its output, and the time is not an input | First build: the task ran and wrote a timestamp | Not recorded | A new copy built a few seconds later took it from the cache with the same timestamp |
The three slow-task rows used four tasks that each sleep for 8 seconds, built one at a time. We sent the signal to the process, or process group, we had started, and nothing else. For the 300 MiB row we watched the server's data folder and sent the kill when a temporary file above 1 MiB appeared. A first try caught a small upload instead and is not counted.
Not tested: Maven, Kotlin or Android builds, a CI system's own cancel button, Windows, a second Ctrl-C, more than one build at a time, and a kill at any other moment than the ones in the table.