Building container images from Jenkins using buildah (part 2)

· 5 min read

Last week I deployed a jenkins container and a buildah container on docker in order to create docker images without requiring access to a Docker daemon. All was set up, but no real builds were made. It was time to change that.

Modifications to my previous setup: adding gitlab

As I didn’t build anything last week, I wasn’t aware there were some missing things on my setup, so I improved it with a few changes:

First I added a gitlab container to my docker-compose, I could have used github or any other public repository, but I was thinking about testing hooks in the future, and I didn’t want to deal with setting NAT on my ISP’s router, so I put everything as close as I could.

The gitlab service was like:

  gitlab:
    container_name: gitlab
    image: gitlab/gitlab-ce:latest
    volumes:
      - "$PWD/gitlab/config:/etc/gitlab"
      - "$PWD/gitlab/logs:/var/log/gitlab"
      - "$PWD/gitlab/data:/var/opt/gitlab"
    ports:
      - "8180:80"
      - "8443:443"
      - "8122:22"

After bringing it up I logged into the the web interface, changed the password and uploaded my public ssh key, basic post-installation stuff.

Then I pushed a project I had previously created using a python:3.9 container and the gunicorn library, it was inside a virtualenv, that was dirty, but as it was for demonstration of the build process, it sufficed. The code of the gunicorn app was the Getting Started example from gunicorn’s web page

 def app(environ, start_response):
        data = b"Hello, World!\n"
        start_response("200 OK", [
            ("Content-Type", "text/plain"),
            ("Content-Length", str(len(data)))
        ])
        return iter([data])

I also created a simple Dockerfile to turn the app into a container:

FROM python:3.9
COPY app /app
WORKDIR /app
ENTRYPOINT ["/app/bin/python", "-m", "gunicorn", "-b", "0.0.0.0:9090", "app:app"]
EXPOSE 9090

Probably I could have used a more elaborated dockerfile, or have dug more into how to use python image, but that wasn’t the objective of this exercise. And also, in order to follow KISS methodology, I configured the project as public.

gitlab access public

Configuring the jenkins job

The next step was to configure a jenkins job. I created a Freestyle Project and selected Git as the source. Then I filled the URL field with my repository url. As I had set up the project as public on Gitlab, no credentials were needed. jenkins project sources

The next important section was the Build Steps. I decided to split the build in two steps, one for building the image, and another to push it to my local nexus. Please note that I used password authentication on the build step, this SHOULD NOT BE DONE THIS WAY, but it was late and the focus here was to build the image. A credential file should be used instead, mounted directly on the build container, and, as a secret is you are using swarm. jenkins project sources

After saving the job configuration, was time to do the real build. So I clicked Build Now and it failed, the good thing was the console option was accessible through jenkins UI.

00:28:49 [ssh-agent] Looking for ssh-agent implementation...
00:28:49 Could not find ssh-agent: IOException: Cannot run program "ssh-agent": error=2, No such file or directory
00:28:49 Check if ssh-agent is installed and in PATH

So I had to add the openssh-clients to my buildah image, but later I discovered I didn’t need the ssh-agent at all.

The second attempt failed too:

00:32:17 Caused by: java.io.IOException: Cannot run program "git" (in directory "/home/build/workspace/workspace/gunicorn"): error=2, No such file or directory
00:32:17 	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)

Git was also missing from my image, so I added it.

The third attempt was something more complicated:

00:33:31 + buildah bud -t juanjovlc2/gunitest:dev .
00:33:31 Error during unshare(CLONE_NEWUSER): Operation not permitted
00:33:31 time="2021-09-20T22:33:31Z" level=error msg="error parsing PID \"\": strconv.Atoi: parsing \"\": invalid syntax"
00:33:31 time="2021-09-20T22:33:31Z" level=error msg="(unable to determine exit status)"
00:33:31 Build step 'Execute shell' marked build as failure

Despite the hours, I was able to find an issue explaining the problem and I adapted the solution to my case editing the compose file, adding this lines to my buildah service:

    security_opt:
      - "seccomp=unconfined"
      - "apparmor=unconfined"

Trying directly from inside the container it also raised an error about not being able to mount the layers, it was fixed sharing the /dev/fuse device with the container:

    devices:
      - "/dev/fuse"

Time for the forth attempt and forth failure:

01:14:04 [gunicorn] $ /bin/sh -xe /tmp/jenkins5235393987736518159.sh
01:14:04 + buildah bud -t 192.168.123.100:8038/juanjovlc2/gunitest:dev .
01:14:04 /tmp/jenkins5235393987736518159.sh: 2: buildah: not found

This was easy to fix: full path to buildah. (Note it was already fixed on the screenshot above)

The fifth and sixth attemps were also failures, one because I forgot to hit Save after adding the full paths, so it was exactly the same as the fourth, and the other a typo on the port number to tag and push the image. But at least, at this point the building part was successful.

They say seven is a lucky number: jenkins project sources

Conclusions

This exercise meant a lot of new concepts for me, I had deployed jenkins and gitlab in the past, but never set up a build job. It has helped me to understand various concepts of jenkins, but I only scratched the surface, of jenkins and continuous integration, there is still a long journey for me.

Fixing buildah images

After applying the changes needed to build the images, the Dockerfile was this:

FROM quay.io/buildah/stable:latest
RUN dnf update -y \
    && dnf install -y openssh-server openssh-clients podman \
             java-1.8.0-openjdk-devel java-1.8.0-openjdk git\
    && dnf clean all \
    && install -d /home/build/.ssh --mode 700 --owner build --group build\
    && install -d /home/build/workspace --mode 700 --owner build --group build\
    && /usr/libexec/openssh/sshd-keygen ecdsa \
    && /usr/libexec/openssh/sshd-keygen rsa

EXPOSE 22
ENTRYPOINT ["/usr/sbin/sshd"]
CMD ["-D", "-e"]

IMPORTANT: the sshd keys are only there as failsafe to start the openssh service, they should be overwritten or mounted as volumes or secrets when running in production.

References

https://docs.gitlab.com/ee/install/docker.html#install-gitlab-using-docker-compose https://developers.redhat.com/blog/2019/08/14/best-practices-for-running-buildah-in-a-container#running_buildah_inside_a_container https://www.redhat.com/sysadmin/building-buildah https://github.com/containers/buildah/issues/2262#issuecomment-619427566