Build cache behind a corporate proxy: Gradle and Maven

Many companies make builds reach the network through a proxy. This page shows how the remote build cache behaves behind one, in Gradle and in Maven: the settings, the ways it goes wrong, and how to tell a proxy problem from a cache problem.

In our runs Maven behaved differently from Gradle: it did not skip the proxy for loopback addresses. If you test on 127.0.0.1 with Maven, list it in nonProxyHosts: see the Maven section.

Checklist for Gradle

Our advice, not a test result:

• Ask for the cache's host and port to be allowed through the proxy (HTTPS).
• Put the proxy lines in gradle.properties, with the systemProp. prefix, and list internal hosts in nonProxyHosts.
• Use HTTPS to the cache. See a production setup with nginx and TLS.
• If the proxy re-signs TLS, Java will need your company's certificate authority.

Gradle build cache behind a corporate proxy

The settings

Gradle takes proxy settings from Java system properties. Put them in ~/.gradle/gradle.properties or the project's gradle.properties, with the systemProp. prefix:

systemProp.http.proxyHost=proxy.example.com
systemProp.http.proxyPort=3128
systemProp.https.proxyHost=proxy.example.com
systemProp.https.proxyPort=3128
systemProp.http.nonProxyHosts=cache.internal.example.com|localhost|127.*

Use the https pair for an https:// cache URL and the http pair for plain HTTP. The last line lists hosts that should skip the proxy.

What we saw

For the plain-HTTP rows we set isAllowInsecureProtocol = true, which Gradle requires for non-loopback http://. That is for the test rig only; use HTTPS for real. In our runs:

Three things that catch people

1. Testing against localhost skips the proxy. In our runs, with the cache at 127.0.0.1, Gradle never used the proxy even though it was configured. A setup that works on your laptop against a local cache proves nothing about the proxy path. Test against the real address.

2. A refusing proxy looks like a cache error. In our runs, when the proxy answered 403 to the tunnel, Gradle printed:

Could not load entry d4f2de693b561fd2c5d60b17a9232d6f from remote build cache: Loading entry from 'https://<cache-ip>:18443/d4f2de693b561fd2c5d60b17a9232d6f' response status 403: Forbidden

That is the same text you would get from the cache refusing a write with the read-only login (see the status codes). To tell them apart, take Gradle out of it and try the same route with curl:

$ curl -sS -x http://proxy.example.com:3128 https://cache.example.com/healthz
curl: (56) CONNECT tunnel failed, response 403      # the proxy said no
$ curl -sS --noproxy '*' https://cache.example.com/healthz
ok                                                   # the cache server answered

We substituted example hostnames in those commands; the two output lines are the real ones, from our test proxy refusing and from the cache answering directly. If the first one fails and the second works, ask whoever runs the proxy to allow the cache's host and port.

3. Over HTTPS the proxy only sees the address. A CONNECT tunnel shows the proxy the host and port and nothing inside, so the password and the cache keys are not visible to it. That also means any rule on the path cannot be applied. This is how the proxy log looked in our run: one CONNECT line per build.

Maven build cache extension behind a corporate proxy

Maven has its own proxy settings, and the Apache Build Cache Extension follows them.

The settings

In ~/.m2/settings.xml, next to the cache's password entry:

<settings>
  <servers>
    <server>
      <id>fosterstack-cache</id>
      <username>maven</username>
      <password>your-password-or-CI-secret</password>
    </server>
  </servers>
  <proxies>
    <proxy>
      <id>corp</id>
      <active>true</active>
      <protocol>http</protocol>
      <host>proxy.example.com</host>
      <port>3128</port>
      <nonProxyHosts>cache.internal.example.com|localhost</nonProxyHosts>
    </proxy>
  </proxies>
</settings>

The cache itself is set up as in the Maven setup. The server id must still match the remote's.

What went through the proxy

In our runs, with the proxy configured, all of the extension's cache traffic went through it: the lookups, the download that restores a build, and the uploads. This is what the proxy logged for two builds, the first filling the cache and the second restoring from it (the proxy also saw Maven's own traffic to repository.apache.org, which is not shown):

build 1 (stores)
GET http://<cache-ip>:18132/v1.1/demo/demo/<id>/buildinfo.xml
GET http://<cache-ip>:18132/v1.1/demo/demo/<id>/buildinfo.xml
PUT http://<cache-ip>:18132/v1.1/demo/demo/<id>/demo.jar
PUT http://<cache-ip>:18132/v1.1/demo/demo/<id>/buildinfo.xml
PUT http://<cache-ip>:18132/v1.1/demo/demo/<id>/build-cache-report.xml

build 2 (restores)
GET http://<cache-ip>:18132/v1.1/demo/demo/<id>/buildinfo.xml
GET http://<cache-ip>:18132/v1.1/demo/demo/<id>/buildinfo.xml
GET http://<cache-ip>:18132/v1.1/demo/demo/<id>/demo.jar
PUT http://<cache-ip>:18132/v1.1/demo/demo/<id>/build-cache-report.xml

The second build logged Found cached build, restoring demo:demo from cache.

Other cases

In our runs:

Things that catch people

• A local test can fail because of the proxy. If you try the cache on 127.0.0.1 with a proxy configured in settings.xml, in our runs Maven sent it to the proxy, which may not be able to reach your machine's loopback. List it in <nonProxyHosts>.
• A refusing proxy looks like a cache error. The 403 comes from the proxy, not from the cache. To check, run the same request from the build machine with and without the proxy; see the curl check in the Gradle section.
• Failure does not break the build. In our runs, a build behind a blocking proxy just stopped using the cache, so look for Error downloading cache item and for missing Saved to remote cache lines. On Maven 3.10.0 an ordinary first miss also logs the first line, so read it together with the missing second one.

How we checked

Gradle run. Gradle 9.8.0 and FosterStack Cache 0.2.1. We wrote a small proxy for the test: it forwards plain HTTP, tunnels HTTPS with CONNECT, can refuse a host, and logs every request. The cache was on a non-loopback address, once over plain HTTP and once over HTTPS through nginx with a self-signed certificate that Java was told to trust.

Maven run. Maven 3.10.0, extension 1.2.3 and FosterStack Cache 0.2.1. We wrote a small proxy for the test, which forwards plain HTTP, can refuse a host, and logs every request (we made it close each connection after one request so that none go unlogged). The cache was on a non-loopback address, over plain HTTP, with a password.

Not tested in Gradle: a real corporate proxy product, a proxy that asks for a password (407), a proxy that inspects and re-signs TLS, automatic proxy configuration files, and the HTTP_PROXY variables. Not tested in Maven: HTTPS through a proxy, a proxy that asks for a password, other Maven or extension versions, and any real proxy product.

Second run: Linux, 0.2.2

The runs above were on a Mac with 0.2.1. We ran both sections again on a GitHub-hosted Linux machine with FosterStack Cache 0.2.2, Gradle 9.8.0, Maven 3.10.0 and extension 1.2.3. The cache was on the runner's own non-loopback address.

The proxy was again a small one we wrote for the test: about 150 lines that forward plain HTTP, tunnel CONNECT, can refuse a host with a 403 and log every request. It is not a production proxy product. Real proxies may ask for a password (407) or inspect TLS, and we did not test those. These results show how the cache, Gradle and Maven behave with this test proxy. They do not show how any real corporate proxy behaves. For the HTTPS rows we used a small TLS server of our own, with a self-signed certificate, in front of the cache (the Mac run used nginx), and told Java to trust it.

On the Linux run:

• Gradle without proxy settings: nothing reached the proxy. With proxy settings and a plain-HTTP cache address, the requests went through the proxy.
• Gradle with the cache at 127.0.0.1, or its address in nonProxyHosts: nothing reached the proxy, and the second build was still FROM-CACHE.
• Gradle with an HTTPS cache through the proxy: one CONNECT line per build and no GET or PUT lines, so the proxy saw no request inside the tunnel.
• Gradle with a proxy that refuses the cache host: both builds succeeded without the cache, and each logged a warning ending in response status 403: Forbidden, like the one shown above. The first build then said the remote cache was disabled. The two curl commands printed curl: (56) CONNECT tunnel failed, response 403 and ok.
• One difference from the Mac run: with proxy settings, each Gradle build made two GET requests to the cache, not one, and the two builds made one PUT between them.
• Maven: the proxy log of the first and second builds matched the two lists above.
• Maven with the cache at 127.0.0.1 and the proxy configured: it still used the proxy (4 requests for the restoring build).
• Maven with the cache's address in nonProxyHosts: nothing reached the proxy, and the build restored from the cache.
• Maven with a proxy that refuses the cache host: nothing was stored or restored, and the log had Error downloading cache item with HTTP Status: 403.
• Maven with no proxy running: the lookup failed with Connect to 127.0.0.1:18131 ... Connection refused, the build succeeded and nothing was uploaded.

The run and its log are public.

Related: Gradle remote build cache not working, authentication and 401 errors and Maven build cache extension not restoring.