Setting up a complete CI/CD pipeline with Jenkins, Nexus and Kubernetes

This is the first in a series of tutorials about setting up a secure production grade CI/CD pipeline, with Kublr cluster manager, Jenkins, Nexus, and Kubernetes using your cloud provider of choice or a colocation provider with bare metal servers. dd 

We know that one of the common goals of SRE and DevOps practitioners is to enable developers and QA engineers to do their best. We’ve developed a list of tools and practices that allow developers to iterate quickly, get instant feedback about their builds and failures, experiment and try new things within minutes, not days:

  • Self service pipelines for build and release process for the developers. Every commit results in a new automated build of a relevant service, and runs any needed tests.

 

  • Pipelines for QA engineers to run any type of testing, either automatic or a manual one, including those which replicate a full Stage or Production environment by creating a new Kubernetes cluster on demand, if the test requires a new cluster to run (it’s easy to launch or decommission a fully working Kuberentes cluster, using Kublr API).

 

  • Build and deployment pipelines are represented as code under version control. And are automatically tested when changes are introduced into the pipeline.

 

  • Ability to rollback the deployment easily, which can be automated based on a threshold set by measuring the performance of a new release in Prometheus, or watching the post-deploy error message rate in Kibana/Elasticsearch to roll back the deployment if a high error rate is detected (or “higher than usual”, if it was a hotfix release for a partially unstable production service which already had some errors or warnings in its log).

 

  • Local secure artifacts storage where we can keep all our Docker images, deployed in the Kubernetes cluster (it can be Nexus or Artifactory).

 

  • Role based access control for all users of the platform, to separate access to the production as well as temporary clusters that we may create during testing or development of complex features.

 CI/CD Pipeline w/ Jenkins, Nexus, Kubernetes

First, we will cover the initial setup and configuration of:

  • Kubernetes cluster for Jenkins, Nexus, and the Jenkins build workers.
  • Jenkins master installed into Kubernetes.
  • Integrating Jenkins master with Kubernetes as a platform to run the build workers. 
  • Docker image for Jenkins workers. We will spawn them inside the Kubernetes cluster on demand, when a build or deploy job is started on master.
  • Nexus artifact storage for our artifacts and Docker images. 

 

We have the option to store our Docker images and other artifacts securely in our local cluster, assuming we have enough storage capacity. During the build process, the artifacts will be pulled from Nexus securely, all within the cluster, then built and packaged into a production Docker image, which will be saved back to Nexus. And when we run our services, the needed images will be deployed to the cluster much quicker than if they were pulled from the Internet. The benefits are obviously speed and security, you own the images and artifacts, and everything takes place inside the local network (or the local virtual network of the cloud provider like VPC in AWS, or VNET in Azure). This limits the ability of intruders to intercept the downloaded package and substitute content, using a MITM attack. 

 

For this tutorial we will be explaining a static setup, with one Kubernetes cluster used for builds and deployment. You may have multiple Kubernetes clusters managed by Kublr for different environments, or even Kubernetes clusters spun up on demand for certain build/test tasks by the build pipelines, via the Kublr API, which is super convenient for full scale integration tests. Below is a diagram showing an advanced-use case of Kublr in the CI/CD cycle. Kublr is used as a central management component for all Kubernetes clusters, creating and terminating dynamic clusters with all relevant triggers of Helm deployments and monitoring metrics inspection takes place in the same pipeline where a new Kubernetes cluster is being created using a few lines of code (to call the Kublr API). 



Deploy Jenkins Master to Kubernetes

The tutorial prerequisites are basic knowledge of how Kubernetes works, a test cluster (you can use existing cluster or create a new one with a few clicks using Kublr), and a desire to automate all the things! Automation is the key to fast development iteration, based on instant feedback and unattended creation and termination of Dev/QA/Test environments. 

 

Before we begin writing Jenkins pipelines to demonstrate the dynamic workers for Docker image builds, let’s create a highly available Jenkins master in Kubernetes. Here is a quick diagram that represents a recommended setup, with a backup and disaster recovery plan:



The dynamic worker pods are created each time a Jenkins job runs a new build. We can run some of the builds inside Kubernetes nodes, and some on other worker servers and server pools like EC2 instances or static Linux, Mac, and Windows servers as usual.

 

A disaster recovery procedure for this setup looks like that: 

  • If a Jenkins master pod fails, it will be recreated by Kubernetes, and Jenkins will instantly get back to a working state. No actions needed, except an investigation of a pod failure.

 

  • If the physical node fails, and the volume data is lost, the pod will be recreated with an empty mounted volume on a new node, in this case the content from the external backup location has to be restored into the physical volume that is mounted in that newly spawned Jenkins pod. The copy can be done easily using “kubectl exec” from within the pod itself, just copy the full backup from S3 or another storage, into the mounted folder location (by default it’s /var/jenkins_home/ in the official Jenkins image), and execute a “reload configuration from disk” command from Jenkins “Global configuration” page. 



All of that is possible thanks to the publicly tested and fully featured Helm chart of Jenkins, so we will not reinvent the wheel. But we will configure it using an existing well designed and polished Helm chart here, from the stable repository. You do not need to download it, Helm CLI will do it when we run the install command. But before installation, it’s worth it to take a note of the most important configuration options for persistent storage, backup and recovery:

  • “persistence.enabled” if set to true, will use a PVC for the pod
  • “persistence.size” specify the PVC size like 10Gi, 50Gi, etc’
  • “persistence.existingClaim” you can use this to run the Jenkins pod on top of an existing preconfigured PVC. You’ll need to specify the PVC name in this case

Also, it is worth mentioning, that the following are important settings that you might want to tweak before installing the chart, they are listed in the “master” section of the readme:

  • “master.numExecutors” is zero by default which means any job will run on a slave pod. If for some reason (like custom legacy Groovy pipelines that use local files and can run only on master) you need the master to be able to execute particular jobs, set it to a number of desired executors on the master pod. This is not recommended.
  • “master.resources” you should tweak the values to a Guaranteed QoS type. Which means same numbers of CPU/Memory in both requests and limits, in order for this pod to not be amongst the first candidates for eviction if the cluster runs out of capacity. Also make sure to set it to a higher number if you plan to run a massive amount of jobs at the same time.
  • “master.slaveKubernetesNamespace” to make your slave pods start in a particular Namespace in the cluster, in order to not start them in the “default” namespace. It makes sense to separate the Dev/QA/Stage Kubernetes clusters, but if you run these environments as a part of the same cluster, then you might want to put the slaves in other namespaces.
  • “master.installPlugins” super useful but not forget to include the list of default ones that the chart wants to install, if you use a ‘–set’ flag to set values, and not the values.yml file from the repo, with ‘-f values.yml’ flag (we recommend to use the original file, just change some of the needed values in it). For example, current list of plugins installed by default looks like this:

    installPlugins:

  – kubernetes:1.18.1

  – workflow-job:2.33

  – workflow-aggregator:2.6

  – credentials-binding:1.19

  – git:3.11.0

 

You might want to try running the Helm install command from their readme examples, where a “–set someKey=something,someOtherKey=somethingElse” command format is used. If you do that, and override the “master.installPlugins” with only your plugins list, the Jenkins installation will have no Kubernetes plugin to launch the agent slaves in Kubernetes cluster, and you will need to install it manually.

  • “master.prometheus.enabled” will enable Prometheus monitoring, which is very useful to track our Jenkins workload, and know when to add resources, or which jobs consume the most

 

  • “master.scriptApproval” the list of scriptSecurity Groovy functions to whitelist (well known list for the Groovy pipeline developers)

 

The Jenkins agent section configures the options of the dynamic slaves that we will run in the cluster, the most interesting configuration settings are:

  • “agent.image” to specify your own builder image. If you want to create a custom builder with your tools included, use a “FROM jenkins/jnlp-slave” in your Dockerfile, to build upon the standard jnlp slave image, so Jenkins will seamlessly run your image as a slave pod.
  • “agent.customJenkinsLabels” a list of labels to add in Jenkins. It is the equivalent of creating a “Node” manually in the global configuration and assigning it a label, which can be used later in any jobs to specify on which executors we want to restrict the build to run.

Install and run the Helm command to work with the cluster. Check that you can see existing releases with “helm list” (in case we use the Kublr demo cluster, you will see a few default Helm releases installed) just to make sure Helm is configured properly: 



If it works, proceed with installing Jenkins with persistence enabled, download the values.yaml file, modify required sections like persistence, agent and master, then run this command:

 

helm install –name jenkins -f your_edited_values_file.yaml stable/jenkins

 

If it works, you should see something like this, with instructions about how to read the initial random password from the secret, written in the bottom:



Run the example command to get the password:

 

printf $(kubectl get secret –namespace default jenkins -o \

jsonpath=”{.data.jenkins-admin-password}” | base64 –decode);echo

 

Now run this, to get the URL for the LoadBalancer of Jenkins service:

 

export SERVICE_IP=$(kubectl get svc jenkins –namespace default –template \

“{{ range (index .status.loadBalancer.ingress 0) }}{{ . }}{{ end }}”)

 

echo http://$SERVICE_IP:8080/login

 

Then visit the URL and login with “admin” and the password you just received earlier.

 

When you first login, you will see that we don’t have any executors and nodes:

This is because the “nodes” are configured to be run on-demand in Kubernetes cluster, and you can find their static configuration in “Manage Jenkins” -> “Configure System” -> “Cloud” -> “Kubernetes” with settings as shown on the screenshot below. Or specify the “worker pod” settings in the build pipeline, which is a preferred and more flexible approach. But to better understand all the available options, here is a description of primary parameters of a Kubernetes worker pod template:




  • Name: template name, to differentiate between templates. 
  • Namespace: is the Kubernetes namespace in which the build will take place.
  • Labels: are the target “node label” for the build jobs to know where to run.
  • Containers: the pod settings like Docker image name, working directory (workspace), environment variables.

Pay attention to the additional label “mylabel” in the settings, we can run any job with this label set as an executor name, and the pod will be created for that build to run. You can customize the requests and limits of these pods in the “Advanced” section. When creating a template, you should calculate the limits carefully, because Kubernetes might terminate a running pod which exceeds the limits, so it is sometimes better to omit limits and allow the pods to consume as much resources as needed, during the build, unless you are trying to setup quotas between teams and enforce strict limitations on their cluster capacity consumption.

Most of the chances are, you are not going to use a static template defined once, because each development team has different needs, and build tools or dependencies. Some might use the Node.js ecosystem, test frameworks and tools, during the build, and others require C/C++ or Golang environment, and so on. Luckily the Jenkins Kubernetes plugin allows us to specify a custom worker pod template in the build pipeline itself, which is probably the most useful and powerful feature of the plugin. We will discuss this a bit more in depth in our next article about running and deploying the builds, but you can read more and explore the features at the official GitHub page here.

Create a demo build pipeline

Now we can try and run a test build pipeline. It will be a quick demo because we’ve provided an in-depth tutorial for production CI/CD process on Kubernetes in the next post. So at this stage we will just make sure our setup works, and then proceed to a Nexus repository configuration. Create a new pipeline job by clicking “New Item”, and then give it a name, select “Pipeline” type and click save.



Copy the demo pipeline code below, and paste into the “Pipeline script” part of the configuration:

 

pipeline {

    agent { label ‘mylabel’ } 

    stages {

        stage(‘demo build’) {

            steps {

                sh ‘env; python –version; java -version; sleep 30’

            }

        }

    }

}



Then click save:

 

And run the job. Wait for the executor pod to start, then you will see something similar in the beginning of the log (navigate inside the build and click “console” to see it):

 

Agent default-1cf45 is provisioned from template Kubernetes Pod Template

Agent specification [Kubernetes Pod Template] (jenkins-jenkins-slave mylabel): 

* [jnlp] jenkins/jnlp-slave:3.27-1(resourceRequestCpu: 200m, resourceRequestMemory: 256Mi, resourceLimitCpu: 500m, resourceLimitMemory: 256Mi)

 

Running on default-1cf45 in /home/jenkins/workspace/dummy-build

 

This means that a pod was successfully created and started. Then this simple pipeline will show all environment variables available inside the container when it runs, knowing what is available during the build, can help you to implement some flexible scripts inside the build pipeline like accessing the local Kubernetes API to find about the current state of pods and services, and so on. It will also display the default versions of Python and Java that exist inside this Docker image that we used as the slave base image. If you navigate to the homepage of Jenkins, you will see the ephemeral “build executor” (which is the pod) appeared in the list:

 

It will disappear as soon as the build is completed, but the logs of the build remain on Jenkins storage (remember the PVC we assigned for persistent storage during Helm install?). Our artifacts, when we decide to actually build them using this new job, need to be uploaded to some storage, or saved in our own Nexus repo that we will install in the next step. But for now this was enough to verify our Kubernetes plugin setup. The Helm chart configured most of it for us, but if you have an existing Jenkins installation (and most of the chances you do), then all you need is to install the “Kubernetes” plugin, and configure it with access to the cluster where the worker pods will run. We will do an in-depth blog post about build jobs and CI on Kubernetes, multibranch pipelines, GitHub triggers, and the feedback loop with notifications to Slack or Jira, about the build results. But for now, let’s install our local Nexus repository, to be able to store the Docker images and other build dependencies locally in the cluster.

 

Installing local Nexus repository on Kubernetes

You can find a Helm chart for Nexus as well. It’s well maintained and is as official as an open source Helm chart can get. It’s the best option out there for installing the repository in your cluster with minimal hassle and maximum flexibility. All configuration options are documented, and you can tweak your installation according to your requirements. An example basic setup is described below, but you might want to customize it with your TLS certificates, different storage types, and so on, depending on your use case (dev, QA, staging, production).

 

To prepare the Helm chart for deployment to your cluster, first download the default values file from GitHub repo here, and update the required parameters:

 

 

  • “persistence.storageSize” the volume size to create for Nexus deployment. Note that this is not where your artifacts will be stored, the volume will be used as a metadata storage so it doesn’t have to be huge, a 10-20Gb volume is good enough for most use cases. The first thing to do when configuring the Nexus after it is launched, is to add a new “Blob Store” definition, with the settings of S3 bucket, and make sure all our repositories will use that new Blob Store.

 

  • “persistence.enabled” is true by default, which is good, because it’s the primary goal of Nexus, to store information about our images and artifacts in a persistent manner.

 

  • “nexusBackup.enabled” enables the sidecar container which will backup the metadata of the repository. It supports only Google Cloud Storage as a target for backups, so we might not want to use it. It is fairly easy to setup a backup ourselves, without using this feature. Just make sure to backup the data folder of the StatefulSet using a script or a Jenkins job, into any secure location that your organization uses for backups.

 

  • “nexus.service.type” by default is “NodePort”, which is not good for our setup that uses an ingress, so change it to “ClusterIP” instead, as we do not want to expose the service on a host port. It will be accessible through the ingress controller.

 

  • “nexusProxy.env.nexusHttpHost” is the DNS address for accessing the UI after the Helm chart is deployed. Set is to something like “nexus.yourwebsite.com” or whatever you would like to use, in your DNS zone, for the address of Nexus web UI.

 

  • “nexusProxy.env.nexusDockerHost” is the address for pulling and pushing Docker images to the Docker registry of our Nexus. Set it to anything that you will use to access it, and create that DNS record pointing at your ingress controller (if you do not know your ingress load balancer address yet, and you’re following the tutorial using a new Kublr cluster, those steps will be explained below). For example “docker.yourwebsite.com”

 

  • “ingress.enabled” should be set to “true”, to enable the ingress rules. This will render the “nexusProxy.env.nexusHttpHost” and “nexusProxy.env.nexusDockerHost” into the ingress resource definition as the hosts to serve through a Kubernetes service.

 

  • “ingress.tls.secretName” has to be set to “false” if we want to use the ingress controller TLS termination, which usually has a wildcard “*.yourdomain.com” certificate, which covers all our subdomains created for the services. On a demo cluster, also set this to “false”, and the ingress will just use the load balancer default TLS, we’ll skip the certificate warning when navigating to the web UI.

 

When ready, run the install command specifying the official Helm chart and the newly edited file:

 

helm install –name nexus -f edited_values.yaml stable/sonatype-nexus

 

If everything works well, you should see an output similar to this:

 

If you haven’t created the new DNS records yet, and need to find the URL of the default ingress controller, for example in the Kublr cluster, run the following command to show the needed ingress address, to which you’ll be able to set your Alias or CNAME records:

 

kubectl get service kublr-feature-ingress-nginx-ingress-controller -n kube-system -o jsonpath='{.status.loadBalancer.ingress[*].hostname}’

 

This will display the load balancer endpoint, and you may go ahead and create the “docker.yourwebsite.com” and “nexus.yourwebsite.com” DNS records pointing at that load balancer address. After a few minutes when the DNS records propagate, you can navigate to the Nexus UI dashboard in the browser, to configure the Docker registry. But first, read the admin password from inside the pod (this is how it works, the Nexus container will have a random admin password generated at launch time, but the first step the UI dashboard will ask you to do after first login, is to change that password to a permanent one). The password is located at “/nexus-data/admin.password”, to display it, run:

 

kubectl exec nexus-sonatype-nexus-0 cat /nexus-data/admin.password

 

To find out the exact pod name, you can list all pods with “kubectl get pods”, but because it’s a StatefulSet and it names the pods with numbers starting from zero, the pod will probably be named “nexus-sonatype-nexus-0” as in “<helm release>-<helm chart>-<pod number>”.

 

When you navigate first time to the newly created “nexus.yourwebsite.com” page and login to Nexus web UI (top right corner), you’ll be presented with a quick setup wizard that will require you to change the temporary password, and whether or not to allow anonymous access to the artifacts.



After the initial setup is completed, you can browse the existing default repositories, and see which storage type they use. By default, they will point to the PV of the pod, which is not good. We should not upload artifacts to the pod physical volume (unless you mounted a large Ceph/GlusterFS/other filesystem as your physical volume in Kubernetes for that Helm chart, but if you did, you probably know what you’re doing and can skip the next section about default storage settings modification in Nexus). We are going to setup an example AWS S3 storage for artifacts and Docker images (basically for all of our repositories at once, they will all be configured to use that storage).

 

Navigate to the settings section, and browse the “Repositories” to see the default ones:

 

Browse into any of the listed default repositories to see which blob storage it uses. 



The storage is defined in the “Blob Stores” section of the config (see sidebar, right above the “Repositories”). 

 

Now, let’s create the S3 storage, which will be used by all repositories.

 

Navigate to “Blob Stores” and click the “Create blob store”. Then select S3 type, set a name, specify bucket name to store the files into, supply AWS access keys if the Kubernetes cluster nodes are not on AWS or do not have IAM Roles assigned (IAM role that allows access to that S3 bucket).



Pay attention to the capacity of our new Blob Store. I says “Unlimited”  Yay! 

 

Create a Docker repository using the new Blob Store:



Give it a name, and select the storage type. We named the Blob Store “primary” and that is what we see in the selection box. The “default” is the Kubernetes pod physical volume, which we should not use, and the “primary” is the newly created S3 store. Select the S3 store, and click “Create Repository”:



We can now login to the repository with “docker login docker.yourwebsite.com -u admin”/ If you used a self signed TLS certificate with your ingress, on a test Kublr cluster, the Docker daemon will not allow such login unless this registry address in whitelisted in “/etc/docker/daemon.json” on the client machine (please follow the instructions from here in case you use Windows or Mac https://docs.docker.com/registry/insecure/). After you set the whitelist as shown below, try to login to the registry. We used “Docker Desktop for Mac” on the client machine so here is an example of how to whitelist that custom registry in this Docker type.




Login with “docker login docker.yourwebsite.com -u admin” and enter your password that was set during the intro wizard of Nexus. Now try to pull a random upstream image like “nginx” or “ubuntu”, tag it as “docker.yourwebsite.com/some-test-name” and do a “docker push” for this new tag to push the image to a Nexus registry. As a result, the image will be uploaded to our Nexus registry, and stored in the S3 bucket. You can now browse the metadata and tags, including file-system layers of that image, in the Nexus UI:



In our example we can see the “kinda-new-image” and “nginx” images were successfully uploaded. Now we can use the image address “docker.yourwebsite.com/nginx” in our pods, to deploy from our local registry. There are numerous benefits to this approach, as described earlier, one of them is complete security of your dockerized software in transit – the images cannot be intercepted as they never get transferred through the public internet. The S3 storage will be accessed securely by your Kubernetes EC2 nodes, if a VPC endpoint to S3 is enabled. This feature allows AWS instances to access S3 buckets securely using internal VPC network which never leaves the boundaries of your AWS account and your specified VPC where the Kubernetes cluster resides.

 

This concludes the first blog in a series of posts dedicated to building a production grade CI/CD pipeline using open source tools. In the next article, we will work in depth with the Jenkins Kubernetes plugin, to setup an automatic build and deploy pipeline, which will use Docker images stored securely in Nexus. The pipeline will build our service images in Kubernetes using dynamic Jenkins worker pods and upload resulting image to Nexus. Then a second pipeline job which represents the “continuous delivery” will get triggered by the fact that a new image was uploaded to the Nexus registry, and will deploy or upgrade the Kubernetes service using that image. We hope you enjoyed the tutorial and will follow us to the next one!