Setup Nexus OSS as docker registry proxy

Jul 16, 2021 · 6 min read

I’ve been playing with the swarm deployment from my previous post too much, I’m completely sure because I was temporally banned from downloading images from hub.docker.io: banned from dockerhub

So I decided it was time to dive into some kind of self hosted docker registry, but I wanted to have access to dockerhub images transparently. The company I’ve been working for uses Sonatype Nexus OSS, so I decided to give it a try.

Depolying Nexus OSS

Nexus OSS was offered as a download from its own page, but also as a docker image from docker hub, so I thought it would be the easiest option. On dockerhub nexus image page there are several examples for launching the container. As I was thinking on having it running on my host computer, I wrote a compose file based on the examples:

version: "2"
services:
  nexus:
    image: sonatype/nexus3
    container_name: nexus
    volumes:
      - ./nexus-data:/nexus-data
    restart: on-failure 
    ports:
      - 8081:8081
      - 8082:8082

Once started I pointed my browser to http://mycomputer:8081 and the nexus interface was waiting for me.

nexus oss welcome screen

For signing in, I was requested the admin password, it was an autogenerated password which looked like an UUID, and it was stored on /nexus-data/admin.password inside the container and, in my case, the binded volume.

The first thing I got was a small config wizard (5 steps), the first and more important thing was changing the admin password. I chose mine and moved to another important question: allowing anonymous access.

At my first attempt I selected disabling anonymous access, but enabling it later was full of troubles, I even had to destroy the volume and deploy nexus again, event it was working perfectly with logged in users. So I recommend to enable anonymous access through the wizard.

The last question of the wizard was about telemetry, the choice is on you.

Setting up the registry (part I)

For setting the registry I followed Maarten Tijhof’s guide, except for the private registry, but I ended with and “insecure” registry.

But It was working pretty well, first for registered users, and after redeploying nexus, for unauthenticated too.

After pulling several images, I was able to see some data cached using the browser view. nexus oss repo browser

Adding SSL support

As configuring a SSL listener directly on nexus was a bit tedious, the official documentation and forums recommended using in front of nexus, and having nexus running using docker-compose, made setting up nginx a piece of cake.

The first step was creating a certificate using Easy-RSA. I downloaded it from github, untarred it on a folder and setup an alias to invoke it without specifying a path. The command sequence is simple:

# init everything, will create ./pki where it's invoked
easyrsa init-pki
# create a CA authority
easyrsa buid-ca
# create a request NOTE THE ALTERNATE NAME
easyrsa --subject-alt-name="IP:PUT.YOUR.IP.HERE" gen-req nexuscert nopass
# sign the request with the server profile
easyrsa sign-req server nexuscert

The subject alternate name was important to prevent issues with the certificate subject.

Then I copied the certificate and key files to a ./nginx/tls directory inside my nexus' project folder and also created (copied) an nginx.conf file from nexus doc.

events { }

http {

  server {
    listen 8081 ssl;
    server_name nexus.garmo.local;
    
    ssl_certificate /etc/nginx/tls/nexuscert.crt;
    ssl_certificate_key /etc/nginx/tls/nexuscert.key;
    
    access_log /var/log/nginx/data-access.log combined;

    client_max_body_size 1G;

    location / {
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto "https";
       proxy_pass http://nexus:8081;
    }
  }
}

And then I added the nginx service to my docker-compose file, moving the exposed port form nexus to nginx:

  nginx:
    image: nginx:latest
    container_name: nginx
    volumes:
      - ./nginx/tls:/etc/nginx/tls
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf
    ports:
      - 8081:8081
    restart: on-failure

This worked well for the web interface, but docker-pull was not working because the registry I created was on port 8082, so I added another location block on the nginx configuration for redirecting /v2 to the registry group http connector:

    location /v2 {
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto "https";
       proxy_pass http://nexus:8082;
    }

I had to trust my freshly created CA on my host system in order to be able of downloading the images, in Ubuntu this is made by:

sudo cp myfreshca.crt /usr/local/share/ca-certificates
sudo cp update-ca-certificates

Setting up the registry (part II)

Once I was able to securely download images, it was time to take advantage of the registry group feature provided by nexus.

In my docker swarm setup I was downloading images from docker.elastic.co so I made a proxy registry for the elastic registry. proxy for elastic

And added it to my repo group. That allowed me to use

docker pull myhost:8081/elasticsearch/elasticsearch:7.13.3

to download the images from docker.elastic.co transparently.

But that came with a risk of downloading elasticsearch images from dockerhub, so I put in place a routing rule.

First I created a blocking routing rule using /elasticsearch as matcher.

routing rule

Then I added the rule to my docker-hub proxy repository:

routing rule

After saving the changes, nexus stopped querying dockerhub for elasticsearch images. I could have done the same for kibana images, or logstash or whichever image I want to prevent downloading from dockerhub. As there was the elastic proxy repository in the same group, images blocked on one repository are requested to the rest.

Setting nexus as mirror on all swarm nodes

In order to automatically use nexus for pulling images from any node on my swarm, they were two required tasks:

Set the new certificate authority as trusted CA, because I was using a custom CA, not a commercial and already trusted one.
Set nexus as a registry mirror on each node.

For the first task I used the ansible provisioning mechanism on vagrant, copying the ca certificate and running the update-ca-trust script.

....
    - name: install anthrax ca
      copy:
        src: tls/myfreshca.crt
        dest: /etc/pki/ca-trust/source/anchors/myfreshca.crt
      notify: update ca trust
 
  handlers:
    - name: update ca trust
      command:
        cmd: update-ca-trust

And for the second, I added a pre_task on my playbook, passing the nexus address as a variable:

  pre_tasks:
    - name: setup docker registry
      copy:
        content: |
          {
            "registry-mirrors": ["https://{{ glstackdeploy_docker_registry_mirror }}"]
          }          
        dest: "/etc/docker/daemon.json"
      notify: reload docker
      when: "(glstackdeploy_docker_registry_mirror | default('')) != ''"
  handlers:
    - name: reload docker
      service:
        name: docker
        state: reloaded

Conclusion

This exercise helped me to understand how custom registries are configured on docker, and also to discover the nexus repository server, which I was aware of because the company I work for uses it, but never tried to configure it.

And it’s a really interesting product, because not only it allows you to set up a variety of repositories beside docker, it has amazing features like repository grouping which simplifies a lot the complexity of accessing several repositories.

The final docker-compose file and the vagrant configuration file are available at my github homelab repo.