.gitlab-ci.yml Part One – Basics Of Script Image Services Stages

Hello Everyone

Welcome to CloudAffaire and this is Debjeet.

In the last blog post, we have discussed how to create a CI/CD pipeline in GitLab using API.

https://cloudaffaire.com/how-to-create-a-ci-cd-pipeline-in-gitlab-using-api/

In this blog post, we will discuss some basic building blocks like script, image, services, and stages that control the behavior of a GitLab CI/CD pipeline jobs.

What Is GitLab .gitlab-ci.yml?

GitLab CI/CD pipelines are configured using a YAML file called .gitlab-ci.yml within each project. The .gitlab-ci.yml file defines the structure and order of the pipelines and determines what to execute using GitLab Runner and what decisions to make when specific conditions are encountered. For example, when a process succeeds or fails.

Pipelines:

Pipelines are the top-level component of continuous integration, delivery, and deployment. Pipelines comprise of Jobs and Stages.

Multiple jobs in the same stage are executed by Runners in parallel, if there are enough concurrent Runners. If all the jobs in a stage succeed, the pipeline moves on to the next stage but if any job gets failed, the next stage is not (usually) executed and the pipeline ends early.

Jobs:

Pipeline configuration begins with jobs. Jobs are the most fundamental element of a .gitlab-ci.yml file. Jobs are defined with constraints stating under what conditions they should be executed. You can give any name to the job and there is no limit on the number of jobs that can be defined in a single pipeline. Job has different configuration options. A job is defined as a list of parameters that define the job’s behavior. .gitlab-ci.yml supports lost of configuration parameters and in this blog post, we will cover some basic building blocks of a GitLab CI/CD pipeline jobs.

.gitlab-ci.yml Part One – Basics Of Script Image Services Stages:

Script:

Script is the only required keyword that a job needs. It’s a shell script which is executed by the Runner. If any of the script commands return an exit code different from zero, the job will fail and further commands will not be executed. This behavior can be avoided by storing the exit code in a variable.

before_script is used to define a command that should be run before each job, including deploy jobs, but after the restoration of any artifacts. This must be an array. Scripts specified in before_script are concatenated with any scripts specified in the main script, and executed together in a single shell.

after_script is used to define the command that will be run after each job, including failed ones. This must be an array. Scripts specified in after_script are executed in a new shell, separate from any before_script or script scripts. As a result, they have a current working directory set back to the default, have no access to changes done by scripts defined in before_script or script, have a separate timeout, which is hardcoded to 5 minutes and do not affect the job’s exit code. If the script section succeeds and the after_script times out or fails, the job will exit with code 0 (Job Succeeded).

Note:It’s possible to overwrite a globally defined before_script or after_script if you set it per-job.

#################################################################
## .gitlab-ci.yml Part One: Script | Image | Services | Stages ##
#################################################################
## Prerequisites: GitLab Access Token
## https://cloudaffaire.com/how-to-create-a-gitlab-project-using-api/
## One CentOs system with internet access
## git, curl and jq package installed

## ------------------------------------
## Create A GitLab Project To Play With
## ------------------------------------

## Create a project in GitLab
## GitLab Console => Projects => Your Projects => New Project => <YOUR_PROJECT_NAME>
PROJECT_ID=$(curl --silent --header "PRIVATE-TOKEN: <YOUR_ACCESS_TOKEN>" \
-XPOST "https://gitlab.com/api/v4/projects?name=cloudaffairecicd&visibility=private&initialize_with_readme=true" | jq '.id') 

## Get the ssh clone url for your project
## GitLab Console => Projects => Your Projects => <YOUR_PROJECT_NAME> => Clone => Clone with SSH
GIT_SSH_CLONE_URL=$(curl --silent --header "PRIVATE-TOKEN: <YOUR_ACCESS_TOKEN>" \
-XGET "https://gitlab.com/api/v4/projects/$PROJECT_ID" | jq -r '.ssh_url_to_repo')

## Clone your GitLab project repository
git clone $GIT_SSH_CLONE_URL

## ------
## Script
## ------

## Create the pipeline configuration file
cd cloudaffairecicd && vi .gitlab-ci.yml
---------------------
before_script:
  - echo "this will be executed before everything else"
mygitlabjob_one:
  script: echo "executed under job one"
mygitlabjob_two:
  script: 
    - echo "this will be executed 1st under job two"
    - echo "this will be executed 2nd under job two"
mygitlabjob_three:
  script: 
    - set +e
    - touch /path/does/not/exist/error.txt; exit_code=$?
    - if [ $exit_code -ne 0 ]; then echo "Previous command failed"; fi;
    - echo "but this will get executed"
    - set -e
mygitlabjob_four:
  before_script:
    - echo "this will get executed before script execution"
  script:
    - echo "the main script execution"
  after_script:
    - echo "this will get executed after script execution"
after_script:
  - echo "this will be executed after everything else"
----------------------
:wq

## Push the .gitlab-ci.yml file into GitLab
git add .
git commit -m "jobs with script parameter"
git push

#################################################################

## .gitlab-ci.yml Part One: Script | Image | Services | Stages ##

#################################################################

## Prerequisites: GitLab Access Token

## https://cloudaffaire.com/how-to-create-a-gitlab-project-using-api/

## One CentOs system with internet access

## git, curl and jq package installed

## ------------------------------------

## Create A GitLab Project To Play With

## ------------------------------------

## Create a project in GitLab

## GitLab Console => Projects => Your Projects => New Project =>

PROJECT_ID=$(curl --silent --header "PRIVATE-TOKEN: " \

-XPOST "https://gitlab.com/api/v4/projects?name=cloudaffairecicd&visibility=private&initialize_with_readme=true" | jq '.id')

## Get the ssh clone url for your project

## GitLab Console => Projects => Your Projects => => Clone => Clone with SSH

GIT_SSH_CLONE_URL=$(curl --silent --header "PRIVATE-TOKEN: " \

-XGET "https://gitlab.com/api/v4/projects/$PROJECT_ID" | jq -r '.ssh_url_to_repo')

## Clone your GitLab project repository

git clone $GIT_SSH_CLONE_URL

## ------

## Script

## ------

## Create the pipeline configuration file

cd cloudaffairecicd && vi .gitlab-ci.yml

---------------------

before_script:

- echo "this will be executed before everything else"

mygitlabjob_one:

script: echo "executed under job one"

mygitlabjob_two:

script:

- echo "this will be executed 1st under job two"

- echo "this will be executed 2nd under job two"

mygitlabjob_three:

script:

- set +e

- touch /path/does/not/exist/error.txt; exit_code=$?

- if [ $exit_code -ne 0 ]; then echo "Previous command failed"; fi;

- echo "but this will get executed"

- set -e

mygitlabjob_four:

before_script:

- echo "this will get executed before script execution"

script:

- echo "the main script execution"

after_script:

- echo "this will get executed after script execution"

after_script:

- echo "this will be executed after everything else"

----------------------

:wq

## Push the .gitlab-ci.yml file into GitLab

git add .

git commit -m "jobs with script parameter"

git push

Image:

Used to specify a Docker image to use for the job. The image keyword is the name of the Docker image the Docker executor will run to perform the CI tasks. By default, the executor will only pull images from Docker Hub, however, this can be configured in the gitlab-runner/config.toml by setting the Docker pull policy to allow using local images. You can simply define an image that will be used for all jobs during build time. It is also possible to define different images per job. You can also pass custom environment variables to fine tune your Docker images directly in the .gitlab-ci.yml file. Image supports below two parameters:

name: Full name of the image that should be used. It should contain the Registry part if needed.
entrypoint: Command or script that should be executed as the container’s entrypoint.

## ------
## Image
## ------

## Edit the pipeline configuration file
cat /dev/null > .gitlab-ci.yml && vi .gitlab-ci.yml
---------------------
image: 
  name: ubuntu:16.04
mygitlabjob_one:
  script: cat /etc/os-release
----------------------
:wq

## Push the .gitlab-ci.yml file into GitLab
git add .
git commit -m "jobs with image parameter"
git push

## ------

## Image

## ------

## Edit the pipeline configuration file

cat /dev/null > .gitlab-ci.yml && vi .gitlab-ci.yml

---------------------

image:

name: ubuntu:16.04

mygitlabjob_one:

script: cat /etc/os-release

----------------------

:wq

## Push the .gitlab-ci.yml file into GitLab

git add .

git commit -m "jobs with image parameter"

git push

Services:

Used to specify a service Docker image, linked to a base image specified in image. The services keyword defines just another Docker image that is run during your job and is linked to the Docker image that the image keyword defines. This allows you to access the service image during build time. The service image can run any application, but the most common use case is to run a database container, e.g., mysql. It’s easier and faster to use an existing image and run it as an additional container than install mysql every time the project is built. You can simply define an services that will be used for all jobs during build time. It is also possible to define different services per job. You can also pass custom environment variables to fine tune your Docker services directly in the .gitlab-ci.yml file. Services supports below parameters:

name: Full name of the image that should be used. It should contain the Registry part if needed.
entrypoint: Command or script that should be executed as the container’s entrypoint.
command: Command or script that should be used as the container’s command. It will be translated to arguments passed to Docker after the image’s name
alias: Additional alias that can be used to access the service from the job’s container.

## --------
## Services
## --------

## Edit the pipeline configuration file
cat /dev/null > .gitlab-ci.yml && vi .gitlab-ci.yml
---------------------
services:
  - name: ubuntu:16.04
    alias: myservice
mygitlabjob_one:
  script: cat /etc/os-release
----------------------
:wq

## Push the .gitlab-ci.yml file into GitLab
git add .
git commit -m "jobs with services parameter"
git push

## --------

## Services

## --------

## Edit the pipeline configuration file

cat /dev/null > .gitlab-ci.yml && vi .gitlab-ci.yml

---------------------

services:

- name: ubuntu:16.04

alias: myservice

mygitlabjob_one:

script: cat /etc/os-release

----------------------

:wq

## Push the .gitlab-ci.yml file into GitLab

git add .

git commit -m "jobs with services parameter"

git push

stages:

stages is used to define stages that can be used by jobs and is defined globally. The specification of stages allows for having flexible multi stage pipelines. The ordering of elements in stages defines the ordering of jobs’ execution. Jobs of the same stage are run in parallel and jobs of the next stage are run after the jobs from the previous stage complete successfully.

If no stages are defined in .gitlab-ci.yml, then the build, test and deploy are allowed to be used as job’s stage by default. If a job doesn’t specify a stage, the job is assigned the test stage. The following stages are available to every pipeline, .pre, which is guaranteed to always be the first stage in a pipeline and .post, which is guaranteed to always be the last stage in a pipeline.

Note: When using your own Runners, GitLab Runner runs only one job at a time by default (can be changed using concurrent flag)

## ------
## stages
## ------

## Edit the pipeline configuration file
cat /dev/null > .gitlab-ci.yml && vi .gitlab-ci.yml
---------------------
stages:
  - build
  - test
  - deploy

mygitlabjob_one:
  stage: .pre
  script: echo "make something useful before build stage"
mygitlabjob_two:
  stage: build
  script: echo "make build dependencies"
mygitlabjob_three:
  stage: build
  script: echo "make build artifacts"
mygitlabjob_four:
  stage: test
  script: echo "make test"
mygitlabjob_five:
  stage: deploy
  script: echo "make deploy"
mygitlabjob_six:
  stage: .post
  script: echo "make something useful at the end of pipeline"
----------------------
:wq

## Push the .gitlab-ci.yml file into GitLab
git add .
git commit -m "jobs with stages parameter"
git push

## Delete your project
curl --silent --header "PRIVATE-TOKEN: <YOUR_ACCESS_TOKEN>" \
-XDELETE "https://gitlab.com/api/v4/projects/$PROJECT_ID" | jq '.' 
cd .. && rm -rf cloudaffairecicd

## ------

## stages

## ------

## Edit the pipeline configuration file

cat /dev/null > .gitlab-ci.yml && vi .gitlab-ci.yml

---------------------

stages:

- build

- test

- deploy

mygitlabjob_one:

stage: .pre

script: echo "make something useful before build stage"

mygitlabjob_two:

stage: build

script: echo "make build dependencies"

mygitlabjob_three:

stage: build

script: echo "make build artifacts"

mygitlabjob_four:

stage: test

script: echo "make test"

mygitlabjob_five:

stage: deploy

script: echo "make deploy"

mygitlabjob_six:

stage: .post

script: echo "make something useful at the end of pipeline"

----------------------

:wq

## Push the .gitlab-ci.yml file into GitLab

git add .

git commit -m "jobs with stages parameter"

git push

## Delete your project

curl --silent --header "PRIVATE-TOKEN: " \

-XDELETE "https://gitlab.com/api/v4/projects/$PROJECT_ID" | jq '.'

cd .. && rm -rf cloudaffairecicd

Hope you enjoyed this article. In the next blog post, we will discuss other configuration options of .gitlab-ci.yml.