Using Jenkins and Kubernetes for Continuous Integration and Delivery

Modern development requires Continuous Integration / Continuous Delivery (CI/CD) and its emphasis on building and running tests on every commit to ensure your development/test environment is always up-to-date. In this tutorial, we’ll show you how CI/CD works with Java services and a Docker/Kubernetes technology stack.

In our example, we’ll be deploying Jenkins in a Kubernetes cluster. We’re using Jenkins because it:

• Has a number of plugins
• Is open source
• Is simple to install and manage

Here’s an overview of how it works:

We’ll implement this scenario in three steps:

1. Prepare the application to be CI/CD ready.
2. Build a Docker Image with Jenkins, Maven, Docker, and Kubernetes Control.
3. Configure Jenkins.

Preparing a CI/CD-ready application

Whenever we use a containerization tool, we deploy the same Docker Image on different environments. Some application properties may need to be changed depending on the environment. Basically, there are two main types of properties:

We use Kubernetes to define the environment-specific property values in a deployment definition, such as:

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
    name: some_application
spec:
..... SKIPPED .....
    env:
    - name: SOME_ENDPOINT
      value: "10.222.11.1"

We can (and should!) use Kubernetes’ ConfigMaps to separate the environment properties’ deployment configuration and values.

In the application’s ENTRYPOINT script, the Docker file will add code to override the application properties with the values from the Linux environment provided by Kubernetes.

Here’s an example of a script used to override application properties in the Red Hat Fuse:

config:edit CONFIG-NAME
config:propset some_endpoint $SOME_ENDPOINT
config:update

Building a Docker Image containing Jenkins, Maven, Docker, and Kubernetes Control

Now that we’ve created a Docker file from the official Jenkins Docker Image (jenkins/jenkins:lts), we need to add Maven and some libraries.

Inside the Kubernetes cluster, you are basically in a Docker-inside-Docker situation. Install the Docker CE and pass-through /var/run/docker.sock from Kubernetes (so we share the same Docker Agent).

You must also install Kubectl to control the Kubernetes cluster. To do this, place the Kubernetes “config” file into the user home directory to have access to the cluster without any additional setup.

The resulting Docker file for our Jenkins will look like this:

MAINTAINER Evgeny Pishnyuk <maintainer-email@gmail.com>

EXPOSE 8080 50000

USER root

# Install prerequisites for Docker
RUN apt-get update && apt-get install -y sudo maven iptables libsystemd-journal0 init-system-helpers libapparmor1 libltdl7 libseccomp2 libdevmapper1.02.1 && rm -rf /var/lib/apt/lists/*

ENV DOCKER_VERSION=docker-ce_17.03.0~ce-0~ubuntu-trusty_amd64.deb
ENV KUBERNETES_VERSION=v1.6.6

# Set up Docker
RUN wget https://download.docker.com/linux/ubuntu/dists/trusty/pool/stable/amd64/$DOCKER_VERSION
RUN dpkg -i $DOCKER_VERSION

# Set up Kubernetes
RUN curl -LO https://storage.googleapis.com/kubernetes-release/release/$KUBERNETES_VERSION/bin/linux/amd64/kubectl
RUN chmod +x ./kubectl
RUN mv ./kubectl /usr/local/bin/kubectl

ENTRYPOINT ["/bin/tini", "--", "/usr/local/bin/jenkins.sh"]

Configuring Jenkins

To deploy Jenkins into the Kubernetes cluster, supply the deployment and service definitions:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
 name: jenkins-ci
spec:
 replicas: 1
 template:
  metadata:
   labels:
    name: jenkins-ci
  spec:
   imagePullSecrets:
    - name: regsecret
   containers:
   - name: jenkins-ci
     imagePullPolicy: Always
     image: some-docker-registry/jenkins-ci:latest
     ports:
     - containerPort: 8080
     - containerPort: 50000
     readinessProbe:
      tcpSocket:
        port: 8080
      initialDelaySeconds: 40
      periodSeconds: 20
     securityContext: 
       privileged: true 
     volumeMounts: 
       - mountPath: /var/run
         name: docker-sock 
       - mountPath: /var/jenkins_home
         name: jenkins-home 
   volumes: 
     - name: docker-sock
       hostPath: 
         path: /var/run
     - name: jenkins-home
       hostPath: 
         path: /var/jenkins_home

apiVersion: v1
kind: Service
metadata:
  name: jenkins-ci-lb
spec:
 type: LoadBalancer
 ports:
   - name: jenkins
     port: 8080
     targetPort: 8080
   - name: jenkins-agent
     port: 50000
     targetPort: 50000
 selector:
   name: jenkins-ci

You should configure two low-level items before beginning to work with Jenkins in Kubernetes:

1. Set parameter excludeClientIPFromCrumb=true in the file /var/jenkins_home/config.xml (This fixes an annoying “No valid crumb was included in the request” error.)
2. Create a “docker login” so Jenkins’ Docker can login and use the correct Docker Registry.

Now we can create the new Jenkins project:

Add build parameters:

Configure an access to the source code repository:

And, finally, add a build step:

cd $ROOT_JAVA

#Maven build
mvn clean install
rm -f $ROOT_DOCKER/install/*
#Copy artifacts for Docker
find . -regex '.*target/[^\/]*\.jar' -exec cp {} $ROOT_DOCKER/install \;
ls $ROOT_DOCKER/install

#Docker build and publish
DOCKER_IMAGE="some-repository/rhesb:$BUILD_NUMBER"
cd $ROOT_DOCKER
docker build -t rhesb .
docker tag rhesb $DOCKER_IMAGE
docker push $DOCKER_IMAGE

#Kubernetes redeploy
kubectl set image statefulset/rhesb rhesb=$DOCKER_IMAGE
kubectl get pod | grep 'rhesb' | cut -d " " -f1 - | awk '{ print $1; system("sleep 60") }' |  xargs -n1 kubectl delete pod --v=3

In this script, we build and test our artifacts with Maven, and then copy them to the Docker/install directory.

Next, we build a Docker Image and push it to the Docker Registry. We update the Docker Image version for the Kubernetes deployment and restart each node with a 60-second interval. (Right now we have no roll-out functionality for a Kubernetes StatefulSets.)

For complex-scenario projects, we suggest making several build steps, each invoking its own shell-script (e.g., for Maven build, Docker build, Kubernetes restart), in order to store build code as part of a project and reuse it later if needed.

Conclusion

Let’s check on the results of our efforts. In this example, we’ve set up git polling on every minute (cron expression ‘* * * * *’), and we have committed some code changes to this repository.

As we can see, a full redeployment takes about 5 minutes — generally consider a good response time. Looking at the above timeline, we can see that the Maven Build/Test phase took only 23 seconds. For a more complex scenario, this part will most likely significantly increase. If you’re interested in more CI/CD pipelines with Jenkin tutorial, check out this blog.