CI: registry.heptapod.net / S3 cache for OSUOSL runners

Once a runner already got an image, it keeps it locally, like any repetitive use of docker pull would do, but that's not enough.

Our OSUOSL x86_64 runners all share the same link. That makes 8 machines that will download the images from registry.heptapod.net, to which the bandwidth isn't great. In particular, the bandwidth to our registry is a few times less than to docker.io, which makes totally sense geographically.

This is especially problematic for the Heptapod image, because it is really heavy (1GB, compressed) and produced automatically by CI (potentially more frequently). In the same time frame, we went from 4 OSUOSL runners to 8.

Caching alternative registries is notoriously problematic if one wants to cache all interactions with the registry. But the bulk of the traffic being taken directly from the S3 backend means that we should be able to cache only that.

The plan would be:

• declare at least two runners to be also caching servers for their peers. We obviously don't want a SPOF.
• make a private Certificate Authority (CA)
• make all runners accept the private CA
• create certificate for the S3 service signed by the private CA
• setup local DNS servers to redirect S3 traffic to two machines (simple round-robin DNS works well) and forward all other zones to the datacenter-wide resolvers. I'm usually happy using unbound for that kind of purpose.
• configure the two caching servers as bi-layered HA NGINX reverse proxies.