I write this blog using Jekyll, which allows me to write posts in markdown and generate a static website from it.

For a few years now it had been running on App Services, but I wasn’t awed by the response time for what is just a static website (~2.5 seconds to load everything on the home page¹). I decided on switching it to a container with a bare Nginx server and the corresponding files, on Kubernetes.

This is a very simple example of a public-facing web application, and I thought it would make a good example to illustrate containerizing something and deploying it to Kubernetes.

My requirements are:

• Hosts static pages with the simplest webserver possible
• Get faster response times
• Use https, redirect http to https, and get a certificate from letsencrypt
• Redirect my other domains (and non www.) to www.feval.ca
• Deploy automatically when pushing to the repository.

The solution I’m using builds and runs a website in Docker, deploys it in Kubernetes using Azure Devops Pipelines. This is done with the following steps:

• Build a Docker image
• Deploy to Kubernetes
  • Create a deployment
  • Create a service
  • Create an ingress
  • Get TLS certificate(s) from Let’s Encrypt
  • Redirect other domains
• Automate build and deployment with Azure DevOps Pipelines

Here is an overview of the whole solution (it looks much more complex than it really is)

In this article I assume you have docker, and kubectl on your machine, and provisioned a managed Kubernetes cluster on Azure (AKS). All the commands are ran in Ubuntu using the Windows Subsystem for Linux.

Build the Docker image ⤓ ⮍

The build happens in Docker. It’s using a multi-staged Docker build. The first stage creates a container that is used only to build the website, the second one takes the result (static files) from the first, and copies it to a barebones Nginx. The first stage image is discarded, the second is the result of the build.

# First stage: build the website
FROM jekyll/builder:3.4.1 AS meeseeks
WORKDIR /blog
COPY _config.yml ./
COPY Gemfile ./
COPY Gemfile.lock ./
COPY favicon.ico ./
COPY index.html ./
COPY build ./build 
COPY _posts ./_posts
COPY _img ./_img
RUN bundle install
RUN bundle exec jekyll build

# Second stage: create the actual image from nginx,
# and shove the static files into the /blog directory
FROM nginx:1.15.8-alpine
WORKDIR /blog
COPY --from=meeseeks /blog/_site .
COPY default.conf /etc/nginx/conf.d/default.conf
EXPOSE 4000

This seems casual, but if you think of what’s effectively happening, this is the basic equivalent of a VM sprouting to build my stuff, then dying immediately after its purpose is done². 15y old gullible me’s brain just melted and spilled through his ears.

Notice that I’m using version tags to images (jekyll/builder:3.4.1 and nginx:1.15.8-alpine). This is something I usually advise, rather than using stable or latest (or even no) tags. This guarantees³ you’re starting from the same version, making the process deterministic.

The configuration for Nginx is the simplest you can think of, listening to port 4000, using files in /blog, using index.html as its default index.

server {
    listen       4000;
    server_name  localhost;
    root   /blog;

    location = / {}
    
    location / {
        index  index.html index.htm;
    }
}

Once the Dockerfile is written, after starting the docker daemon, I can test it using:

docker build . -t blog
docker create -p 4000:4000 --name blog blog
docker start blog

Then the website should show up on port 4000. We’ll automate the build later, for now, I’m pushing it manually to my registry. I’m using Azure Container Registry for that purpose, which is free (up to a point).

# First, add a tag to indicate I want to push that image to my registry
docker tag blog chfev12221.azurecr.io/blog
# Then push
docker login chfev12221.azurecr.io # Only the first time
docker push chfev12221.azurecr.io/blog

Deploy to Kubernetes ⤓ ⮍

The website sits in Kubernetes, we need to deploy and configure the components listed earlier:

• A deployment, which configures what application to run
• A service, which configures what endpoints to open
• An ingress, which configures how the application is exposed to the external world
• A certificate issuer, which gets TLS certificates from Let’s Encrypt, and a certificate.

1- A deployment deploys the app in pods ⤓ ⮍

A pod is a logical host in Kubernetes. See this as a disposable VM running the container(s) you specify. Pods are usually created by Deployments, which can be specified by a yaml file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: blog
  namespace: blog
  labels:
    app: blog
spec:
  replicas: 1
  selector:
    matchLabels:
      app: blog
  template:
    metadata:
      labels:
        app: blog
    spec:
      containers:
      - name: blog
        image: chfev12221.azurecr.io/blog

A few important things about that deployment:

1. The container image reference specifies the container(s) you want to run, and simply point to a container registry. Since there’s no tag on this image, it will just pull the latest version for now (this is not great for the reasons highlighted earlier, but we’ll fix that when we come to automating the build). ACR is a private registry, it requires authentication to access images. This is easily done through RBAC as described in this section of the documentation.
2. The number of replicas is set to 1, that means that there will be 1 pod created with this container. If I wanted some resiliency, I would create at least 2. But then I’d need 2 nodes, and I’m not rich, so 1 is OK for my small venture.
3. Pods are disposable. They get deleted by Kubernetes with (little to) no warning. Deployments are persistent. Always assume your pod will die on you.

If I deploy this pod with kubectl apply -f deployment.yml, I end up with one pod:

$ kubectl get pods,deployments
NAME                        READY   STATUS    RESTARTS   AGE
pod/blog-7dc5f84d56-5c2cd   1/1     Running   0          1d

NAME                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.extensions/blog   1         1         1            1           3d

I can have a look at it by forwarding ports from my local machine to the pod:

$ kubectl port-forward pod/blog-7dc5f84d56-5c2cd 4003:4000
Forwarding from 127.0.0.1:4003 -> 4000
Forwarding from [::1]:4003 -> 4000

The directing a browser to http://localhost:4003 shows me the website⁴.

2- A service makes it accessible ⤓ ⮍

A service describes how you allow access to the pods. There are multiple ways of doing so, and access can be private (you only allow other Kubernetes entities to touch it) or public (provision a public IP). In this case, we don’t want direct access to it since we need to handle HTTPS, so we are using a clusterIP, which provides an internal IP only.

kind: Service
apiVersion: v1
metadata:
  namespace: blog
  name: blog-service
spec:
  selector:
    app: blog
  ports:
  - protocol: TCP
    port: 80
    targetPort: 4000
  type: ClusterIP

This specifies that we need a listener on port 80 with an IP address that will forward requests to an app called blog on its port 4000.

$ kubectl get services
NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
blog-service         ClusterIP   10.0.254.6     <none>        80/TCP    3d

Notice that the service got assigned a private IP address, but no public one. I could very well just replace the type by type: LoadBalancer, and this would provide a public IP address to which I can redirect traffic, but I want https, and to do some redirections, so I keep the IP private and use an ingress instead.

3- An ingress creates a reverse proxy with rules to access internal services ⤓ ⮍

Next I define an ingress. Ingress is a reverse proxy, see this as your entry point for traffic coming from the internet. It requires a controller to be created in the cluster. I’m using Nginx ingress controller. The simplest way to install it is with Helm.

1. First, install helm locally, then install tiller (the remote portion of Helm deployed on your kubernetes cluster) by running helm init.
2. Then install the controller: helm install stable/nginx-ingress --namespace kube-system.

The ingress controller creates a service with a public ip address. We get it using the following command:

$ kubectl get service -l app=nginx-ingress --namespace kube-system
NAME                                         TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)                      AGE
winning-puma-nginx-ingress-controller        LoadBalancer   10.0.23.229   13.71.166.22   80:32722/TCP,443:32440/TCP   3d
winning-puma-nginx-ingress-default-backend   ClusterIP      10.0.101.79   <none>         80/TCP                       3d

The ingress resource is described as follows, and will be served by the ingress controller:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
  name: blog-ingress
  namespace: blog
spec:
  rules:
  - host: blog.feval.ca
    http:
      paths:
      - backend:
          serviceName: blog-service
          servicePort: 80
        path: /
  tls:
  - hosts:
    - blog.feval.ca
    secretName: tls-secret

This indicates to direct all traffic coming for blog.feval.ca to the blog-service we created earlier ; and to use the TLS certificate stored in the secret tls-secret.

At this stage I create a subdomain in my DNS provider that redirects to the ingress controller’s external IP, and I should be able to access the website using https://blog.feval.ca. It works, except that Firefox is complaining about my certificate… which should be expected since we didn’t take care of that.

4- Get a TLS certificate(s) from Let’s Encrypt ⤓ ⮍

Let’s Encrypt issues certificates by a mechanism of challenge-response. To make it work, you first need to direct the DNS to a location that is configured to handle that mechanism. For this purpose I use cert manager.

We install cert manager with Helm: helm install --name cert-manager --namespace kube-system stable/cert-manager.

From there I will get certificates from Let’s Encrypt. I provision two certificate providers: staging and production. You can make a limited number of request for “real” certificates from production for a given domain in let’s encrypt, so it’s always good to try in staging first, then switch to production when you get the setup to work. The issuers are configured like so:

apiVersion: certmanager.k8s.io/v1alpha1
kind: Issuer
metadata:
  name: letsencrypt-staging
  namespace: blog
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: acme@yourdomain.ca #Use your own email
    privateKeySecretRef:
      name: letsencrypt-staging
    http01: {}

---

apiVersion: certmanager.k8s.io/v1alpha1
kind: Issuer
metadata:
  name: letsencrypt-prod
  namespace: blog
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: acme@yourdomain.ca #replace with your own email
    privateKeySecretRef:
      name: letsencrypt-prod
    http01: {}

Then I request the certificate for my domain with another resource. This resource indicates which domains to use the certificate for, and which domains it’s being asked for. This allows to request a certificate for a *.something.com.

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  namespace: blog
  name: tls-secret
spec:
  secretName: tls-secret
  dnsNames:
  - blog.feval.ca
  acme:
    config:
    - http01:
        ingressClass: nginx
      domains:
      - blog.feval.ca
  issuerRef:
    name: letsencrypt-staging
    kind: Issuer

It may take a couple of minutes for the process to kick in and retrieve the certificates. After a while, hitting https://blog.feval.ca should give me a certificate issued by “Fake Issuer”. At this point I switch to the issuer to letsencrypt-prod, reapply the resource, and wait for a couple more minutes.

5- Redirect other domains to only one ⤓ ⮍

I could configure the ingress to route multiple domains to the blog-service, but I like uniformity, so I prefer doing redirects. To do so, I configure a new ingress thusly⁵:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/configuration-snippet: |
        return 301 https://www.feval.ca$request_uri;
  name: blog-ingress-redirect
  namespace: blog
spec:
  rules:
  - host: feval.ca
    http:
      paths:
      - backend:
          serviceName: blog-service
          servicePort: 80
        path: /
  - host: www.feval.fr
    http:
      paths:
      - backend:
          serviceName: blog-service
          servicePort: 80
        path: /
  - host: feval.fr
    http:
      paths:
      - backend:
          serviceName: blog-service
          servicePort: 80
        path: /
  tls:
  - hosts:
    - www.feval.fr
    - feval.fr
    - feval.ca
    secretName: tls-secret

This will return a 301 to my www.feval.ca with the request uri. Note that you need to get the corresponding certificates as well, add them to the list of step 4.

Automate with Azure DevOps Pipelines ⤓ ⮍

Continuous integration is setup in Azure DevOps Pipelines (which is free to use) and has 4 steps:

1. Build the image (using the Dockerfile from the beginning)
2. Push it to the container registry.
3. Update the Kubernetes deployment yaml file with the image tag (which happens to simply be the build number). For that purpose we replace the image in the deployment.yml with: image: charles.azurecr.io/blog:#{CONTAINER_TAG}#, and define a CONTAINER_TAG variable in the Azure DevOps pipeline build on the build with value $(Build.BuildId).
4. Store this file as a build artifact. See this as a physical link between the build and the image in the container.

This is what the configuration exported as yaml looks like:

resources:
- repo: self
queue:
  name: Hosted Ubuntu 1604
variables:
  searchUrl: 'https://cfeval.search.windows.net'
steps:
- task: Docker@0
  displayName: 'Build an image'
  inputs:
    azureSubscription: '...'
    azureContainerRegistry: '...'
    dockerFile: Dockerfile
    imageName: 'blog:$(Build.BuildId)'
- task: Docker@0
  displayName: 'Push an image'
  inputs:
    azureSubscription: '...'
    azureContainerRegistry: '...'
    action: 'Push an image'
    imageName: 'blog:$(Build.BuildId)'
- task: qetza.replacetokens.replacetokens-task.replacetokens@3
  displayName: 'Add container version in deploy.yml'
  inputs:
    rootDirectory: deploy
    targetFiles: '**/deploy.yml'
- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifact: deploy.yml'
  inputs:
    PathtoPublish: deploy/deploy.yml
    ArtifactName: deploy.yml

We set a trigger on the source repo, so that any push to master triggers the build.

The deployment step really just does a kubectl apply -f deploy.yml. Since the image tag is different, Kubernetes will update the deployment, deploy a new pod and destroy the old one. We setup a trigger on the build pipeline.

Conclusion

This is the response time on app services (no cache):

This is the response time on AKS (no cache):

And this is the average response time measured by App Insights, notice the drop on the 22nd when I deployed to AKS:

To be honest this is far from a scientific measurement, and it might just be that my app service is configured… well, is not configured at all. Or it might just be contextual.

This whole exercise was more of a geeky thing than anything else. For a static website, there are other alternatives that could work, probably even better. Jules posted a static stack based on Azure blob storage and Azure CDN that would do the job as well, if not probably better.

It demonstrates the steps to run a static website on Kubernetes within a container. There are quite a few steps to get there, but they are actually quite straight forward. It’s definitely more complex than running it on Azure App Services though, but it leaves me with a cluster that I can use for whatever else I want. And I have other plans for it!

Notes