Azure Pipeline Part 3 – steps jobs stages
Hello Everyone
Welcome to CloudAffaire and this is Debjeet.
Today we are going to discuss steps, jobs and stages in Azure Pipelines. Azure pipeline consists of one or more stages that describe a CI/CD process. Stages are the major divisions in a pipeline. For example, your pipeline can have one stage each for build, test and deploy your application.
A stage consists of one or more jobs, which are units of work assignable to the same machine. You can arrange both stages and jobs into dependency graphs. For example, your build stage can have multiple jobs for each component of your application.
A job consists of one or more steps. Steps can be tasks, scripts, or references to external templates. Steps are the lowest performable actions in your pipeline. For example, you can have multiple steps to build a component of your application.
This hierarchy is reflected in the structure of a YAML file like:
- Pipeline
- Stage 1
- Job 1
- Step 1.1
- Step 1.2
- …
- Job 2
- Step 2.1
- Step 2.2
- …
- Job 1
- Stage 2
- …
- Stage 1
Azure Pipeline Part 3 – steps jobs stages
Prerequisites:
- One active Azure DevOps account
- Personal Access Token (PAT)
- A self-hosted agent registered to your Azure DevOps organization
Setup:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 |
## ----- ## Setup ## ----- ## Define variables ORGANIZATION=" USER=$(az account show | jq -r .user.name) PROJECT=" PAT=' B64_PAT=$(printf "%s"":$PAT" | base64) ## Test if authentication working (List projects in your organization) curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/_apis/projects?api-version=6.0 | jq . ## Create a new Azure DevOps project curl -s --request POST \ -u $USER:$PAT \ --header "Content-Type: application/json" \ --data '{ "name": "'"$PROJECT"'", "description": "Demo Project For Azure Pipeline", "capabilities": { "versioncontrol": { "sourceControlType": "Git" }, "processTemplate": { "templateTypeId": "6b724908-ef14-45cf-84f8-768b5384da45" } } }' \ https://dev.azure.com/$ORGANIZATION/_apis/projects?api-version=6.0 | jq . ## Get project default repository details REPOSITORY_ID=$(curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/git/repositories?api-version=6.0 | jq -r .value[0].id) && curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/git/repositories/$REPOSITORY_ID?api-version=6.0 | jq . ## Create a new Azure Pipeline using REST API curl -s --request POST \ -u $USER:$PAT \ --header "Content-Type: application/json" \ --data '{ "folder": null, "name": "mypipeline", "configuration": { "type": "yaml", "path": "/azure-pipelines.yml", "repository": { "id": "'"$REPOSITORY_ID"'", "name": "'"$PROJECT"'", "type": "azureReposGit" } } }' \ https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/pipelines?api-version=6.0-preview.1 | jq . ## Clone project default repository REPO_HTTPS_CLONE_URL=$(curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/git/repositories?api-version=6.0 | jq -r .value[0].webUrl ) && B64_PAT=$(printf "%s"":$PAT" | base64) && git -c http.extraHeader="Authorization: Basic ${B64_PAT}" \ clone $REPO_HTTPS_CLONE_URL cd $PROJECT |
Azure Pipeline – Steps:
A step is a linear sequence of operations that make up a job. Each step runs in its own process on an agent and has access to the pipeline workspace on a local hard drive. This behavior means environment variables aren’t preserved between steps but file system changes are. Azure Pipeline supports below types of steps –
- script: Using script, you can run a script using cmd.exe on Windows and Bash on other platforms.
- bash: Using bash, you can run a script or command in Bash on Windows, macOS, and Linux.
- pwsh: Using pwsh, you can run a script or command in PowerShell Core on Windows, macOS, and Linux.
- powershell: Using powershell, you can run a script or command in Windows PowerShell.
- task: Using task, you can perform various pre-configured steps in your pipeline.
- checkout: Using checkout, you can clone and use an external repository in your pipeline.
- templates: Using template, you can define a set of steps in one file and use it multiple times in another file.
Azure Pipeline – Steps syntax:
|
1 2 3 4 5 6 7 8 9 |
steps: - [ script | bash | pwsh | powershell | checkout | task | templateReference ] displayName: string # friendly name displayed in the UI name: string # identifier for this step (A-Z, a-z, 0-9, and underscore) condition: string continueOnError: boolean # 'true' if future steps should run even if this step fails; defaults to 'false' enabled: boolean # whether to run this step; defaults to 'true' timeoutInMinutes: number env: { string: string } # list of environment variables to add |
Azure Pipeline – Steps examples:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 |
## ------------------------------------- ## The simplest pipeline you can execute ## ------------------------------------- ## Create a new pipeline definition cat <<'EOF'> azure-pipelines.yml steps: - script: echo Hello World! EOF ## Push the changes to Azure Repo git add . git commit -m "initial commit" git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push ## ------------------------------------- ## Execute multi-line commands in a step ## ------------------------------------- ## Update pipeline definition cat <<'EOF'> azure-pipelines.yml pool: mypool steps: - script: | echo Hello World! echo Welcome to cloudaffaire EOF ## Push the changes to Azure Repo git add . git commit -m "update1" git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push ## ------------------------------------------------ ## Execute multiple types of steps in a single step ## ------------------------------------------------ ## Update pipeline definition cat <<'EOF'> azure-pipelines.yml pool: mypool steps: - script: | echo The script keyword is a shortcut for the command-line task. echo The task runs a script using cmd.exe on Windows and Bash on Linux. echo $SHELL displayName: script_step - bash: | echo The bash keyword is a shortcut for the shell script task. echo $SHELL displayName: bash_step - task: Bash@3 inputs: targetType: 'inline' script: | echo Tasks are the building blocks of a pipeline. echo $SHELL displayName: task_step EOF ## Push the changes to Azure Repo git add . git commit -m "update2" git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push |
Azure Pipelines – Jobs
You can organize your pipeline into jobs. Every pipeline has at least one job. A job is a series of steps that run sequentially as a unit. In other words, a job is the smallest unit of work that can be scheduled to run. Jobs can be of different types, depending on where they run.
- Agent pool jobs run on an agent in an agent pool.
- Server jobs run on the Azure DevOps Server.
- Container jobs run in a container on an agent in an agent pool.
Azure Pipelines – Jobs syntax:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
jobs: - job: string # name of the job (A-Z, a-z, 0-9, and underscore) displayName: string # friendly name to display in the UI dependsOn: string | [ string ] condition: string strategy: parallel: # parallel strategy; see the following "Parallel" topic matrix: # matrix strategy; see the following "Matrix" topic maxParallel: number # maximum number of matrix jobs to run simultaneously continueOnError: boolean # 'true' if future jobs should run even if this job fails; defaults to 'false' pool: pool # see the following "Pool" schema workspace: clean: outputs | resources | all # what to clean up before the job runs container: containerReference # container to run this job inside of timeoutInMinutes: number # how long to run the job before automatically cancelling cancelTimeoutInMinutes: number # how much time to give 'run always even if cancelled tasks' before killing them variables: # several syntaxes, see specific section steps: [ script | bash | pwsh | powershell | checkout | task | templateReference ] services: { string: string | container } # container resources to run as a service container uses: # Any resources (repos or pools) required by this job that are not already referenced repositories: [ string ] # Repository references to Azure Git repositories pools: [ string ] # Pool names, typically when using a matrix strategy for the job |
Azure Pipelines – Jobs examples:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 |
## --------------------------------------- ## Execute multiple jobs in Azure Pipeline ## --------------------------------------- ## Update pipeline definition cat <<'EOF'> azure-pipelines.yml pool: mypool jobs: - job: job1 displayName: JOB1 steps: - script: echo do something displayName: JOB1STEP - job: job2 displayName: JOB2 steps: - script: echo do something displayName: JOB2STEP EOF ## Push the changes to Azure Repo git add . git commit -m "update3" git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push ## -------------------------------------------- ## Execute next job even if previous job failed ## -------------------------------------------- ## Update pipeline definition cat <<'EOF'> azure-pipelines.yml pool: mypool jobs: - job: job1 displayName: JOB1 continueOnError: true steps: - script: someerror displayName: JOB1STEP - job: job2 displayName: JOB2 steps: - script: echo job2 will still get executed displayName: JOB2STEP EOF ## Push the changes to Azure Repo git add . git commit -m "update4" git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push ## ------------------------ ## Execute job sequentially ## ------------------------ ## Update pipeline definition cat <<'EOF'> azure-pipelines.yml pool: mypool jobs: - job: job1 dependsOn: job2 steps: - script: echo job1 - job: job2 dependsOn: job3 steps: - script: echo job2 - job: job3 steps: - script: echo job3 EOF ## Push the changes to Azure Repo git add . git commit -m "update5" git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push ## -------------------------------- ## Execute jobs based on conditions ## -------------------------------- ## Update pipeline definition cat <<'EOF'> azure-pipelines.yml pool: mypool jobs: - job: job1 steps: - script: someerror - job: job2 dependsOn: job1 condition: failed() steps: - script: echo this will run when job1 fails - job: job3 dependsOn: - job1 - job2 condition: succeeded('job2') steps: - script: echo this will run when job2 runs and succeeds EOF ## Push the changes to Azure Repo git add . git commit -m "update6" git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push ## ---------------------------------- ## Execute multiple instance of a job ## ---------------------------------- ## Update pipeline definition cat <<'EOF'> azure-pipelines.yml pool: mypool jobs: - job: myjob steps: - script: echo hello world strategy: parallel: 5 EOF ## Push the changes to Azure Repo git add . git commit -m "update7" git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push ## -------------------- ## Manage job workspace ## -------------------- ## Update pipeline definition cat <<'EOF'> azure-pipelines.yml pool: mypool jobs: - job: job1 steps: - script: echo hello job1 - job: job2 steps: - script: echo hello job2 - job: job3 workspace: clean: all steps: - script: echo hello job3 EOF ## Push the changes to Azure Repo git add . git commit -m "update8" git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push |
Azure Pipelines – Stages:
Stages are logical boundaries in your pipeline where you can pause the pipeline and perform various checks. Every pipeline has at least one stage even if you do not explicitly define it. You can also arrange stages into a dependency graph so that one stage runs before another one. Each stage can have one or more collection of related jobs. There is a limit of 256 jobs for a stage.
Azure Pipelines – Stages syntax:
|
1 2 3 4 5 6 7 |
stages: - stage: string # name of the stage (A-Z, a-z, 0-9, and underscore) displayName: string # friendly name to display in the UI dependsOn: string | [ string ] condition: string variables: # several syntaxes, see specific section jobs: [ job | templateReference] |
Azure Pipelines – Stages examples:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
## Update pipeline definition cat <<'EOF'> azure-pipelines.yml pool: mypool stages: - stage: Build jobs: - job: BuildJob steps: - script: echo Building the code! - stage: Test jobs: - job: TestJob steps: - script: echo Testing the code! - stage: Deploy jobs: - job: DeployJob steps: - script: echo Deploying the code! EOF ## Push the changes to Azure Repo git add . git commit -m "update9" git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push |
Clean up:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
## -------- ## Clean up ## -------- ## Delete the Azure DevOps project and local repository PROJECT_ID=$(curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/_apis/projects?api-version=6.0 \ | jq -r '.value[] | select(.name == "'"$PROJECT"'") | .id') && curl -s --request DELETE \ -u $USER:$PAT \ --header "Content-Type: application/json" \ https://dev.azure.com/$ORGANIZATION/_apis/projects/$PROJECT_ID?api-version=6.0 cd .. rm -rf $PROJECT |
Hope you have enjoyed this article. To know more about Azure DevOps, please refer below official documentation
https://docs.microsoft.com/en-us/azure/devops/?view=azure-devops