Adding Content

Writing

This is a blog. So first of all to add new content to it, I write stuff. For now the topics originate from the development and deployment of the blog itself.

I write all of it in markdown (files ending in .md), organized in a single folder per post. On build, these markdown files are automatically converterd to HTML and after deployment thats what your browser shows you. Your browser retrieves the content from a webserver, in my case that’s a nginx from inside a docker container (actually there’s another nginx in a docker container inbetween acting as a reverse proxy). As I described in the last post, the docker container is created from the default gatsby base image, which copies the built (optimized) contents of the blog into the docker container and configures a nginx to serve it.

For now this is a manual process, which means I usually write the posts in Visual Studio Code on my PC, commit them in my git repository and then push them to my personal git server. The same server also hosts the blog and some other things (to be explained in future posts) all separated in individual docker containers. I then

1. log onto my server via ssh ssh stephan@while-false.de
2. pull the latest version from said git repository git pull
3. build the blog npx gatsby build
4. build a new version of the docker container docker build -t while-false/blog
5. swap the (old) running version of the container with the newly built one. docker stop blog && docker rm blog && docker run --name blog while-false/blog

(Some of the commands actually need more parameters for production use, but are not relevant for the concept)

All steps on the server are handled from the terminal and typing the same commands over and over on each new post is rather unconfortable.

Automation - Planning

Whenever you do the same thing often, automation comes to mind. So the steps on the server are to be automated. I have some ideas for that.

The relevant event I want to react on is the change in the blog. I consider a change as relevant, when a new commit happens on the master branch of the git repository of the blog. Luckily, git has a builtin concept for reacting on events on the server, called “server-side hooks”. In my case the git server is an instance of gitea, so I looked up server side hooks in the gitea documentation. I quickly found the hook I needed:

So I can make git notify some other component of each change. Now I need something to listen to these notifications and then execute the update-steps automatically.

My first intention was to build that component myself. I know all the commands to execute (see above). I would just need some preparation:

• Have something listen on HTTP for the call of the git hook
• Then pull the latest commit from the git server (it should be included in the call of the hook)
• Oh wait, I need to authenticate the component first. Maybe with ssh keys?
• Then install the npm dependencies. But first make sure node.js is really installed
• Build the gatsby.js project. But first make sure gatsby.js is really installed
• Build the docker image. But first make sure docker is really installed
• Push the docker image to my private docker repository. Another authentication required
• On the host exchange the currently running docker container with the newly built one

Around that time of planning I decided this isn’t the way to go. At my job I rely heavily on Azure DevOps which conveniently covers all these tasks and requirements. But for my private free-time-projects I imposed the restriction on myself to run everything on my own server(s). But until now I only looked at the furthest cases on the automation spectrum: doing everything myself and have everything done by Microsoft in the cloud. I decided that the truth probably lies somewhere inbetween (as it does so often in life). I then looked at self-hosted CI/CD (“Continous Integration”/“Continous Delivery”) systems.

Drone caught my eye, as it has full docker support, is open source and can have multiple, distributed workers. Perfect, I finally get to use the “sandbox” VPS I rent which just accumulates virtual dust. After reading the documentation, the setup was fairly easy.

Automation - Setting up

First I spun up a docker container for drone, drone/drone on Docker-Hub. It has lots of required arguments so I will only describe some of the configuration:

• The gitea server I want to depend upon (other git servers can also be used)
• oAuth2 credentials (clientId and clientSecret) for an application registered on the gitea server
• A self-generated secret to use for connection of the workers
• Hostname and protocol (https) of the drone server itself

As mentioned above I also wanted to use another machine as a worker for drone. Luckily, theres also a docker image for that: drone/drone-runner-docker. The worker also has some (similar) parameters:

• The hostname and protocol of the drone server (see above)
• The shared secret to connect the worker to the ci server (see above)
• The hostname of the worker
• The parallelism of the worker

The worker then connected to the drone server. In the UI of the drone server I could then log in using my account from the gitea server (yay, oAuth!) and immedeatly see the repository for the blog. There are only a few options for the repository, because the build pipeline itself is defined as code in a file .drone.yml. For my case with the gatsby blog I use the following pipeline:

kind: pipeline
type: docker
name: Blog build and release

steps:
- name: build
  image: node:alpine
  commands:
    - npm install
    - npx gatsby build

- name: docker
  image: plugins/docker
  settings:
    username: ***
    password: ***
    repo: registry.while-false.de/blog
    tags:
      - 'latest'
      - '1.3.0'

For now I only require two steps:

1. install node.js dependencies and build the gatsby project
2. build the new docker image and push it to the registry

An additional benefit of the drone build is this beautiful badge, every project has nowadays, conveniently prepared as markdown:

Also, It has nice visualization of the build process with logs for debugging in case it’s needed:

Then, the last required step is to update the running container to the new version. The event on which to react would be the upload to the registry. There are some ways to handle this myself using webhooks, but as with the build trigger I decided to take a route a little more convenient: use Watchtower. I tried watchtower before and don’t feel comfortable blindly updating every container I run, so I configure it to just watch the one blog container and update that automatically. As I am the only one pushing updates of the image for the blog I can take precautions when I know something will break.

Conclusion

As a result, I now just happily type a new article in a new markdown file. As soon as I commit and push that (onto the master branch) the automation takes over:

1. The git server notifies drone of the new commit
2. drone spins up a new worker that:
3. pulls the latest version of the code
4. installs node dependencies
5. builds the gatsby project
6. builds the docker image
7. pushes the docker image to my repository
8. watchtower sees a new version of the image in the repository and updates the running container

It took me quite some hours to get everything working and being used to out-of-the-box CI/CD experiences like Azure DevOps I knew it had to work, but stumbled over the details quite a few times. It was a great opportunity to learn about all the connected parts and hopefully will help me in the future when I need to dive into the details of any pipeline.