Building a CI/CD pipeline using Docker, Docker compose and Jenkins

Introduction

Let's build a CI/CD pipeline using complete opensource solution like Gitea, Jenkins, Docker-Compose to self host and fully automate the perfect end-to-end pipeline.

Set up a Dockerized Git Server

version: '3'
services:
  gitea:
    image: gitea/gitea:latest
    environment:
      - USER_UID=1000
      - USER_GID=1000
      - DB_TYPE=sqlite3
    volumes:
      - ./gitea/data:/data
    ports:
      - "3000:3000"  # Web interface
      - "2222:22"    # SSH
    restart: always

Run the Git Server

  docker-compose up -d

Let's setup our repository, Access http://localhost:3000, create an account, and initialize a repository.

Set up Jenkins as a CI/CD Server

Add jenkin's docker container that includes all the necessary plugins

......
jenkins:
  image: jenkins/jenkins:lts
  user: root
  ports:
    - "8080:8080"  # Web interface
    - "50000:50000"  # Agent communication
  volumes:
    - ./jenkins_home:/var/jenkins_home  # Persist data
    - /var/run/docker.sock:/var/run/docker.sock  # Access Docker
  restart: always

Stop the docker compose docker-compose down and then restart it docker-compose up -d

Setup Jenkins

• Go to http://localhost:8080, follow the setup instructions and install recommended plugins.
• Set up the Git plugin to connect with our Gitea repository( which we created above ).
• Install Docker and Docker Pipeline plugins for Docker based build environments.

Create Credentials

• In Jenkins dashboard, navigate to Manage Jenkins > Manage Credentials and add SSH credentials for connecting to your Gitea server.

Configure a Jenkins Pipeline for CI/CD

• In Jenkins Dashboard, create a new job and select Pipeline
• Write a Jenkinsfile, In our git repository, let's create a Jenkinsfile to define the CI/CD pipeline.

pipeline {
  agent { docker { image 'node:14' } }  // Example for a Node.js project

  stages {
    stage('Build') {
      steps {
        sh 'npm install'
        sh 'npm run build'
      }
    }

    stage('Test') {
      steps {
        sh 'npm test'
      }
    }

    stage('Deploy') {
      steps {
        script {
          def app = docker.build("myapp:latest")
          app.run("-d -p 80:80")
        }
      }
    }
  }

  post {
    always {
      archiveArtifacts artifacts: '**/target/*.jar', allowEmptyArchive: true
      junit 'reports/**/*.xml'
    }
  }
}

• Build Stage - Installs dependencies and build the project
• Test Stage - Runs the test
• Deploy Stage - Deploys the Dockerized app
• Post Actions - Archives build artifacts and test results for later inspection

Push the Jenkinsfile to Gitea Repository

Connect the Jenkins to the repository - In the Jenkins pipeline job, set the source repository to the Git URL.

Run the pipeline - Jenkins will detect changes and trigger the pipeline automatically

Add automated deployemnt to a staging environment

• Add Rollback Logic in Jenkinsfile - Define a rollback mechanism in case the deployemnt fails. This might involve restarting the last known good container.

...
post {
  failure {
    script {
      // Rollback to the previous image
      sh 'ssh user@staging-server "docker-compose -f docker-compose-rollback.yml up -d"'
    }
  }
}

-Send Notifications - Use the email-ext plugin or Slack plugin in Jenkins to notify team members of build and deployemnt status.

Setup Webhooks for Continuous Integration

• Configure Webhooks - Setup a webhook in Gitea to notify Jenkins of any push to the repository. Go to your repository in Gitea, navigate to Settings > Webhooks, and add a webhook that points to http://jenkins-server:8080/github-webhook/.
• Test the webhook - Push a commit to your repository to trigger the pipeline in Jenkins automatically

Monitor and Scale

• Use Jenkins Plugins - Install plugins for monitoring resource usage and job performance in Jenkins.
• Scale with Docker Swarm or Kubernetes - Once the pipeline works well, consider container orchestraion for load balancing and scaling.

Finally, what we have achieved so far

• A self hosted Git Server for version control and webhook triggers, authentication and authorization (preinstalled)
• Jenkins to run CI/CD jobs in Docker
• Staging server to run and test dockerized application after successful builds
• Notification System using Jenkins plugins for Slack or email alerts

Happy reading!