How To Create A CI CD Pipeline In GitLab Using API
Hello Everyone
Welcome to CloudAffaire and this is Debjeet.
In the last blog post, we have discussed how to add members to groups and projects in GitLab.
https://cloudaffaire.com/how-to-add-members-in-group-and-projects-in-gitlab-using-api/
In this blog post, we will discuss how to create a CI-CD Pipeline in GitLab. The demo is created using API as we are taking a programmatic approach to cover most of the demo. But we have also provided the console steps as well so that if you want to perform the demo from GitLab console, you can do the same.
What Is GitLab CI/CD?
GitLab CI/CD is a tool built into GitLab for software development through continuous methodologies like Continuous Integration (CI), Continuous Delivery (CD), Continuous Deployment (CD). Continuous Integration works by pushing small code chunks to your application’s codebase hosted in a Git repository, and, to every push, run a pipeline of scripts to build, test, and validate the code changes before merging them into the main branch. Continuous Delivery and Deployment consist of a step further CI, deploying your application to production at every push to the default branch of the repository.
These methodologies allow you to catch bugs and errors early in the development cycle, ensuring that all the code deployed to production complies with the code standards you established for your app. GitLab CI/CD is configured by a file called .gitlab-ci.yml placed at the repository’s root. The scripts set in this file are executed by the GitLab Runner.
Continuous Integration (CI):
Consider an application that has its code stored in a Git repository in GitLab. Developers push code changes every day, multiple times a day. For every push to the repository, you can create a set of scripts to build and test your application automatically, decreasing the chance of introducing errors to your app. This practice is known as Continuous Integration; for every change submitted to an application – even to development branches – it’s built and tested automatically and continuously, ensuring the introduced changes pass all tests, guidelines, and code compliance standards you established for your app.
Continuous Delivery (CD):
Continuous Delivery is a step beyond Continuous Integration. Your application is not only built and tested at every code change pushed to the codebase, but, as an additional step, it’s also deployed continuously, though the deployments are triggered manually. This method ensures the code is checked automatically but requires human intervention to manually and strategically trigger the deployment of the changes.
Continuous Deployment (CD):
Continuous Deployment is also a further step beyond Continuous Integration, similar to Continuous Delivery. The difference is that instead of deploying your application manually, you set it to be deployed automatically. It does not require human intervention at all to have your application deployed.
How GitLab CI/CD works?
To use GitLab CI/CD, all you need is an application codebase hosted in a Git repository, and for your build, test, and deployment scripts to be specified in a file called .gitlab-ci.yml, located in the root path of your repository. In this file, you can define the scripts you want to run, define include and cache dependencies, choose commands you want to run in sequence and those you want to run in parallel, define where you want to deploy your app, and specify whether you will want to run the scripts automatically or trigger any of them manually. Once you’re familiar with GitLab CI/CD you can add more advanced steps into the configuration file.
To add scripts to that file, you’ll need to organize them in a sequence that suits your application and are in accordance with the tests you wish to perform. To visualize the process, imagine that all the scripts you add to the configuration file are the same as the commands you run on a terminal on your computer. Once you’ve added your .gitlab-ci.yml configuration file to your repository, GitLab will detect it and run your scripts with the tool called GitLab Runner, which works similarly to your terminal. The scripts are grouped into jobs, and together they compose a pipeline.
GitLab CI/CD Components:
- Pipeline: Pipelines are the top-level component of continuous integration, delivery, and deployment. Pipelines comprise of jobs that define what to run. For example, code compilation or test runs and stages that define when and how to run. For example, tests run only after code compilation.
- .gitlab-ci.yml: GitLab CI/CD pipelines are configured using a YAML file called .gitlab-ci.yml within each project. The .gitlab-ci.yml file defines the structure and order of the pipelines and determines what to execute using GitLab Runner and what decisions to make when specific conditions are encountered.
- GitLab Runner: GitLab Runner is the open source project that is used to run your jobs and send the results back to GitLab. It is used in conjunction with GitLab CI, the open-source continuous integration service included with GitLab that coordinates the jobs.
- GitLab Package Registry: GitLab Packages allows organizations to utilize GitLab as a private repository for a variety of common package managers. Users are able to build and publish packages, which can be easily consumed as a dependency in downstream projects.
How To Create A CI CD Pipeline In GitLab Using API:
Step 1: Create a GitLab project to play with.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
######################################################## ## How To Create A CI/CD Pipeline In GitLab Using API ## ######################################################## ## Prerequisites: GitLab Access Token ## https://cloudaffaire.com/how-to-create-a-gitlab-project-using-api/ ## One CentOs system with internet access ## git, curl and jq package installed ## ------------------------------------ ## Create A GitLab Project To Play With ## ------------------------------------ ## Create a project in GitLab ## GitLab Console => Projects => Your Projects => New Project => PROJECT_ID=$(curl --silent --header "PRIVATE-TOKEN: -XPOST "https://gitlab.com/api/v4/projects?name=cloudaffairecicd&visibility=private&initialize_with_readme=true" | jq '.id') |
Step 2: Create a .gitlab-ci.yml file and upload to your GitLab project repository.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 |
## Get the ssh clone url for your project ## GitLab Console => Projects => Your Projects => GIT_SSH_CLONE_URL=$(curl --silent --header "PRIVATE-TOKEN: -XGET "https://gitlab.com/api/v4/projects/$PROJECT_ID" | jq -r '.ssh_url_to_repo') ## Clone your GitLab project repository git clone $GIT_SSH_CLONE_URL ## Create the .gitlab-ci.yml for your GitLab pipeline configuration cd cloudaffairecicd && vi .gitlab-ci.yml --------------------- stages: - build - test - deploy job 0: stage: .pre script: echo "make something useful before build stage" job 1: stage: build script: echo "make build dependencies" job 2: stage: build script: echo "make build artifacts" job 3: stage: test script: echo "make test" job 4: stage: deploy script: echo "make deploy" job 5: stage: .post script: echo "make something useful at the end of pipeline" --------------------- :wq ## Push the .gitlab-ci.yml file into GitLab git add . git commit -m "pipeline definition added" git push |
Step 3: Create, update and delete CI/CD pipeline in GitLab using API.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 |
## Get GitLab pipeline details for your project PIPELINE_ID=$(curl --silent --header "PRIVATE-TOKEN: -XGET "https://gitlab.com/api/v4/projects/$PROJECT_ID/pipelines" | jq '.[].id') && curl --silent --header "PRIVATE-TOKEN: -XGET "https://gitlab.com/api/v4/projects/$PROJECT_ID/pipelines/$PIPELINE_ID" | jq '.' ## A pipeline a autometicially created as soos and we uploaded the .gitlab-ci.yml file ## Get GitLab pipeline jobs details curl --silent --header "PRIVATE-TOKEN: -XGET "https://gitlab.com/api/v4/projects/$PROJECT_ID/pipelines/$PIPELINE_ID/jobs" | jq '.' ## Retry the entire pipeline (all jobs) curl --silent --header "PRIVATE-TOKEN: -XPOST "https://gitlab.com/api/v4/projects/$PROJECT_ID/pipelines/$PIPELINE_ID/retry" | jq '.' ## Retry a specific job (only works if you job step has manual action) JOB_ID=$(curl --silent --header "PRIVATE-TOKEN: -XGET "https://gitlab.com/api/v4/projects/$PROJECT_ID/pipelines/$PIPELINE_ID/jobs" | jq -r '.[0].id') && curl -v --silent --header "PRIVATE-TOKEN: "https://gitlab.com/api/v4/projects/$PROJECT_ID/jobs/$JOB_ID/play" | jq '.' ## Delete the pipeline curl --silent --header "PRIVATE-TOKEN: --request "DELETE" "https://gitlab.com/api/v4/projects/$PROJECT_ID/pipelines/$PIPELINE_ID" | jq '.' ## Create a new pipeline curl --silent --request POST --header "PRIVATE-TOKEN: "https://gitlab.com/api/v4/projects/$PROJECT_ID/pipeline?ref=master" | jq '.' ## Cancel a pipelines jobs PIPELINE_ID=$(curl --silent --header "PRIVATE-TOKEN: -XGET "https://gitlab.com/api/v4/projects/$PROJECT_ID/pipelines" | jq '.[].id') && curl --silent --request POST --header "PRIVATE-TOKEN: "https://gitlab.com/api/v4/projects/$PROJECT_ID/pipelines/$PIPELINE_ID/cancel" | jq '.' ## Delete your project curl --silent --header "PRIVATE-TOKEN: -XDELETE "https://gitlab.com/api/v4/projects/$PROJECT_ID" | jq '.' cd .. && rm -rf cloudaffairecicd |
Hope you enjoyed this article. In the next blog post, we will discuss CI/CD in GitLab.
To get more details on GitLab you can follow the below link.
https://docs.gitlab.com/ee/README.html
To Get more details on Git you can follow the below links.
https://cloudaffaire.com/category/devops/git/