Is my build cache server healthy? What to alert on

Watch three things: the server answers, uploads succeed, and builds get hits. FosterStack Cache reports all three, but /healthz alone only says the process is running.

The short answer

• /healthz answers ok when the process is up. In our runs it kept answering ok after we made the data folder read-only and after we deleted it.
• A broken data folder shows up as uploads failing. Each upload we tried got a 500, and the server counted it in fscache_http_requests_total.
• A cache that is too small throws entries away before they are used. That shows as fscache_evicted_entries_total going up while fscache_cache_hits_total stays flat. With a 20,000-byte cap, three builds in a row got zero hits.
• A failing build is not your signal. In the broken-folder and wrong-login runs below, Gradle still ended with BUILD SUCCESSFUL. It printed a warning and stopped using the remote cache for that build.

What to alert on

Our advice, not a test result:

• The server is not answering. Probe /healthz from another machine with curl -sf --max-time 3. Alert when it fails. When we stopped the server, this command exited with 7.
• Uploads are failing. Alert when fscache_http_requests_total with method="PUT" and a 5xx status goes up. This is the signal for a data folder that cannot be written. We ran this rule by hand on saved /metrics output. It stayed quiet on healthy builds and fired on both broken-folder runs.
• The cache is full and churning. Alert when fscache_evicted_entries_total goes up while fscache_cache_hits_total does not. We ran this rule by hand too. It stayed quiet on healthy builds and fired on all three builds against the tiny cap. Then read server sizing.
• The server restarted. The process_start_time_seconds metric changed when we restarted. Alerting on a change is our suggestion and we did not run it as a rule.
• The hit rate is low. See how to measure the hit rate. A low rate is not always a server fault, so we would send this to a dashboard, not a pager.

What not to alert on

• /healthz on its own. In our runs it did not notice a broken data folder.
• fscache_store_entries or fscache_store_bytes at zero after a restart. Right after a restart both metrics read 0 until the first upload, even though the data is there. /statusz shows the real numbers.
• A count of 401 or 403 responses. The server does not count them. Refused logins did not appear in fscache_http_requests_total or in the server log. A wrong login shows up on the client: the build log says response status 401 and the build gets no hits from the server.
• Made-up health URLs. /readyz and /livez returned 404. The server counted each 404 as a cache miss.

Counters start again at 0 when the server restarts. Prometheus increase() handles the reset. Reading the raw value does not. The full list of metric names is in the monitoring section below.

Monitor a Gradle build cache with Prometheus and Grafana

The cache server exposes Prometheus metrics and a read-only status page. This section lists what is there, a scrape setting, a few queries, and where to get a ready-made Grafana dashboard.

Our advice, not a test result: the scrape setting and the queries are examples.

Metrics need no password

The /metrics and /healthz paths stay open even when Basic Auth is on, so a Prometheus scraper needs no credentials. The status page, /statusz, sits behind Basic Auth when it is enabled.

A scrape setting

scrape_configs:
  - job_name: fscache
    static_configs:
      - targets: ['cache.example.com:8080']

This is a plain example. Use your own address, and HTTPS if you have put the server behind a proxy.

The metrics

These are the names a running 0.2.1 server reports:

Queries worth having

# hit rate over the last 5 minutes
sum(rate(fscache_cache_hits_total[5m]))
  / clamp_min(sum(rate(fscache_cache_hits_total[5m])) + sum(rate(fscache_cache_misses_total[5m])), 0.0001)

# entries removed per second to stay under the cap
sum(rate(fscache_evicted_entries_total[5m]))

# 95th percentile request time
histogram_quantile(0.95, sum by (le, method) (rate(fscache_http_request_duration_seconds_bucket[5m])))

A ready-made Grafana dashboard

The repository includes a dashboard with eight panels: hit rate, cache size against the cap, entries, evictions, hits and misses, requests by status, throughput, and request latency. Download grafana-dashboard.json and import it in Grafana.

The status page

For a quick look without Prometheus, /statusz returns the numbers in one place, in a browser or as JSON. This is what a freshly started 0.2.1 server returned after one stored entry and two requests:

$ curl -s localhost:8080/statusz
{
  "version": "v0.2.1",
  "revision": "0bfc498ecfca3e38cb4a456acda44359a1620c5a",
  "fips140": false,
  "fips140_note": "off",
  "uptime_seconds": 2.118684875,
  "uptime": "2s",
  "store_bytes": 5,
  "max_bytes": 1073741824,
  "store_entries": 1,
  "cache_hits": 1,
  "cache_misses": 1,
  "hit_ratio": 0.5,
  "evicted_entries": 0,
  "auth_enabled": false
}

Related: Gradle remote build cache not working and Gradle build cache with Docker Compose.

How we checked

We ran real FosterStack Cache servers, one per case, and built a four-module Gradle project against each. After every step we asked /healthz and /statusz, read /metrics, and kept the build log.

The two rules: rule 1 is the increase of PUT requests with a 5xx status. Rule 2 is evictions up while hits are flat. We worked out the difference between two saved /metrics scrapes with a small script. We did not run a Prometheus server or an alerting tool, so how you write these rules in your alerting tool is our suggestion.

Where the figures come from: a build asks the server for 13 things (four compile tasks and nine compiled build scripts), so one build adds 13 hits or misses. fscache_evicted_entries_total counts entries, not builds.

Setup: FosterStack Cache 0.2.1 and Gradle 9.8.0 on a Mac (arm64), Temurin 21.0.12.1, all on one machine over plain HTTP on the loopback address. Every build used a fresh copy of the project and a new empty Gradle home, with Gradle's local cache off. We broke the data folder with chmod and rm as the same user that ran the server.

The metric names and the status page output in the monitoring section come from a running 0.2.1 server.

Limits: we did not fill a real disk, we only used the size cap. We did not test Docker or Kubernetes probes, HTTPS, Maven, a slow disk, or a very large cache. The first runs were on a Mac; the second set, in the note below, was on Linux. A disk that fills up may behave differently from a cap or a read-only folder. For the exit with permission denied in a container, see the startup errors page. For what a build does while the server is down, see server down.

Second run: Linux, 0.2.2

The runs above were on a Mac with 0.2.1. We ran the table again on a GitHub-hosted Linux machine with FosterStack Cache 0.2.2, Gradle 9.8.0 and Java 21, on a four-module project of our own.

• Every row held: what /healthz and /statusz answered, the probe's exit code 7, the counters after a restart, the read-only and deleted data folders (a 500 on the upload, counted as PUT 500, and a build that still ended with BUILD SUCCESSFUL), the 20,000-byte cap, the 413, and the three login rows.
• The numbers differ because the project differs. A build asked the server for 11 things (13 on the Mac), and the cap evicted 3, 11 and 11 entries over the three builds (6, 13 and 13 on the Mac).
• We checked the two alert rules on live /metrics values, not in a Prometheus server. The rule for failed uploads stayed quiet on the three healthy builds and fired on both broken-folder runs. The rule for evictions stayed quiet on the three healthy builds and fired on all three builds against the cap.

The run and its log are public.