My build cache server won't start. What does the error mean?
If FosterStack Cache exits right after you start it, read the last line of its log. In our runs every startup failure ended with one fscache: fatal line and exit code 1, and the line named the cause.
The short answer
- A rejected setting stops the server before it touches the data folder. The line names the variable and the value. Sizes must be whole numbers of bytes, so
10GBis refused. - A problem with the address or the data folder stops it a moment later. A busy port, a folder it cannot write to and a data path that is a file each gave a clear line. In a container we reproduced the folder case with a data volume that belonged to root: see permission denied on the data folder.
- A second server on the same data folder waits about 5 seconds, then exits. It ends with the same fatal line and code 1. The first one kept serving.
- Some mistakes do not stop the server. A misspelled setting name was ignored without a warning, and an emptied index file was accepted without a warning, so lost entries go unnoticed.
Find your error
The first column is what we saw in the log. The other two columns are our advice, not a test result. DIR stands for a folder on our test machine and PORT for the port we used.
| The last line of the log | What it means | What to do |
|---|---|---|
serve: listen tcp 127.0.0.1:PORT: bind: address already in use | Another program, or another copy of the server, already listens on that address. | Stop it, or give this server another FSCACHE_ADDR. |
serve: listen tcp: address VALUE: missing port in address, also invalid port and unknown port | FSCACHE_ADDR is not a host and a port. We tried garbage, 127.0.0.1, 18702, :99999, :-1 and 127.0.0.1:abc. | Write it as :8080 or 127.0.0.1:8080. |
write shutdown marker: open DIR/.unclean-shutdown: permission denied | The data folder exists, but the user running the server cannot write to it. | Give that user write access. In a container, see permission denied on the data folder below. |
write shutdown marker: mkdir DIR: permission denied, or mkdir data: permission denied | The server could not create the data folder. With no FSCACHE_DATA_DIR it uses ./data, in the folder you started it from. | Create the folder first, or set FSCACHE_DATA_DIR to a path you can write. |
check shutdown marker: stat DIR/.unclean-shutdown: not a directory | FSCACHE_DATA_DIR points at a file. | Point it at a folder. |
open metadata store: metadata: open: timeout, after about 5 seconds | Another server already holds this data folder. | Stop the other server. See can I run two build cache servers. |
open metadata store: metadata: open: invalid database | The index file meta.db in the data folder is damaged. We overwrote it with random bytes. | Restore the folder from a backup, or start with a new, empty folder. See back up and restore a build cache. |
open metadata store: metadata: open: open DIR/meta.db: permission denied | The server's user cannot read and write the index file. We had made it read-only. | Fix the file's owner or mode. |
FSCACHE_MAX_BYTES="10GB" is not a valid byte count (whole non-negative decimal number) | Sizes are whole numbers of bytes. A unit, a decimal point, an exponent, a space and letters after the number were all refused. | Write 10737418240 for 10 GiB, not 10GB. |
FSCACHE_MAX_BYTES="-1" is negative; a byte count cannot be | A negative size. FSCACHE_MAX_CONCURRENT_UPLOADS="-1" gets the same line. | Use a whole number, or 0 for no limit. |
FSCACHE_USERNAME and FSCACHE_PASSWORD must both be set or both be empty, and the same line for FSCACHE_RO_USERNAME and FSCACHE_RO_PASSWORD | Only one half of a login pair is set. | Set both halves, or neither. |
FSCACHE_RO_USERNAME and FSCACHE_RO_PASSWORD require FSCACHE_USERNAME and FSCACHE_PASSWORD | A read-only pair is set without the read-write pair. | Set the read-write pair as well. |
FSCACHE_RO_USERNAME must differ from FSCACHE_USERNAME | Both pairs use the same username. | Choose another read-only username. See change the build cache logins. |
FSCACHE_MAX_CONCURRENT_UPLOADS="abc" is not a valid byte count, and the same for FSCACHE_MAX_BODY_BYTES | The same rule applies. The message says "byte count" even for the upload limit. | Use a whole number. |
Build cache server exits with "permission denied" on its data
You start the container and it stops straight away. The log ends with permission denied. The cause we reproduced is who owns the data directory.
What you see
The container exits with status 1 within a few seconds, and its log says:
{"level":"ERROR","msg":"fscache: fatal","error":"write shutdown marker: open data/.unclean-shutdown: permission denied"}Why it happens
The server does not run as root. It runs as user 65532, and it keeps its files in a data folder inside /home/nonroot. If that folder already exists and belongs to someone else, the server cannot write to it and stops.
We reproduced it with a volume that already held data owned by root, which is what you get after restoring a backup as root:
/d 0:0 drwxr-xr-x (owner:group, mode)
/d/data 0:0 drwxr-xr-xThe fix
Give the whole tree to user 65532. The image has no shell, so use a throwaway container to do it:
docker run --rm -v fscache-data:/d ubuntu:24.04 chown -R 65532:65532 /dIn our runs, after that the same image started normally: /healthz answered ok and an upload returned 201.
What does not cause it
- A new, empty named volume. This is what our first-run steps create, and the volume then belongs to 65532. We also tried a new empty volume whose top folder we had set to root. The server still started, and afterwards the folder belonged to 65532, because Docker copies the image's ownership into an empty volume the first time it is used.
When the server starts, but not the way you meant
- A misspelled setting name was ignored. With
FSCACHE_MAXBYTES=1500the server started with no warning, and its start line said"max_bytes":0. Four uploads of 1,000 bytes were all stored. WithFSCACHE_MAX_BYTES=1500, one entry was left and 3 were evicted. - An emptied or deleted
meta.dbwas not an error. The server started and still served an old entry. But/statuszcounted only the entry we stored afterwards: 1 entry and 100 bytes, not 4 entries. - A bad address still left a data folder behind. The server created
meta.dbandblobsbefore it failed to listen. A rejected setting did not create any folder. - An HTTPS address does not work. The docs say the server speaks plain HTTP and has no certificate handling, so TLS goes in front of it.
curlwith anhttps://address to the server's own port failed with exit code 35, and the server logged nothing. See production build cache with Docker Compose, nginx and TLS.
What to do
Our advice, not a test result:
- Read the last line first. It names the variable, the value or the path.
- After every change, read the
fscache: startingline. It prints the address, the data folder, the size cap and whether logins are on. A misspelled name shows up there as a setting that did not change. - Write sizes in bytes.
10GBwill stop the server, which is better than a silent default. - Start the server as the user that owns the data folder. In a container, that means giving the data volume to the user the image runs as, 65532 in our runs. The command is in the permission-denied section above. If you restore a backup into the volume, the restore has to keep that ownership.
- Keep one server per data folder. A start on a full disk is on what happens when the disk fills up.
How we checked
FosterStack Cache 0.2.1 on a Mac (arm64). The permission-denied cases used the 0.2.1 container image. Each non-container case was one start with a clean environment that held only the address, the data folder and the setting under test, watched for 3 seconds with a check every quarter of a second. Every failed start exited with code 1. The non-container ones took under a second, except the shared folder. The container cases exited within a few seconds. A server we stopped with a normal stop exited with code 0.
| What we did | What the server did |
|---|---|
| Started a second server on an address in use | Exited with code 1, under a second |
Six invalid FSCACHE_ADDR values | All exited with code 1, under a second |
| Data path was a file; folder not writable; folder inside a read-only parent; no data setting in a read-only folder | All exited with code 1, under a second |
Size setting: -1, abc, 10GB, 10G, 1.5, 12abc, 1e6, " 100" | All exited with code 1, under a second. No data folder was created |
Size setting: 0 and 5000 | Started. The start line showed "max_bytes" as 0 and 5000 |
| One half of a login pair; read-only pair without the read-write pair; same usernames | All exited with code 1, under a second |
Upload limit: abc and -1; body limit: big | All exited with code 1, under a second |
Upload limit: 0 and 32 | Started |
| The 0.2.1 container image on a volume that already held data owned by root | Exited with status 1 within a few seconds. The log ended with permission denied |
The same volume after the chown command above | Started normally. /healthz answered ok and an upload returned 201 |
| A new, empty named volume, and one whose top folder we had set to root | Both started. Afterwards the folder belonged to 65532 |
| A second server on a data folder held by a running server | Exited with code 1 after about 5 seconds. The first answered /healthz and kept its entry |
Copies of a data folder with meta.db emptied, and deleted | Both started with no warning. An old entry was still served, and /statusz showed 1 entry |
A copy with meta.db overwritten with random bytes; a copy with meta.db read-only | Both exited with code 1, under a second |
FSCACHE_MAXBYTES and FSCACHE_MAX_BYTES, four 1,000-byte uploads each | 4 entries stored with the misspelled name. 1 entry and 3 evicted with the right one |
curl with https:// to the plain HTTP port | curl exit code 35. /healthz over http:// answered 200 |
The log is one JSON line per event. A start that got as far as listening printed fscache: starting first. A rejected setting or a data folder problem printed only the fatal line. We caused one mistake at a time, so we do not know which line shows first when two settings are wrong.
Not tested: container starts other than the permission-denied cases above, a bind mount of a host folder on Linux (the same rule should apply, as the folder has to be writable by 65532, but the container runs used Docker volumes, not a bind mount), Kubernetes without the fsGroup setting (the manifests on our Kubernetes page set fsGroup: 65532, and we ran them with it), running the container as root (it is not what we document), Windows, IPv6 addresses, a data folder on a network disk, and a Gradle build against an HTTPS address that the server does not speak.
The first run above was on a Mac with 0.2.1. We ran the page again on a GitHub-hosted Linux machine with FosterStack Cache 0.2.2 and the 0.2.2 container image. Each start failure in the table printed the line shown and exited with code 1, and the last line of the log was that line. A second server on a held data folder gave up after about 5 seconds. Each setting the page lists as accepted or ignored started the server, and its start line, entry counts and evictions matched the page. We did not check those logs for warnings this time. The container on a volume owned by root exited with status 1, and after the chown command it started. The run and its log are public.