How to create and execute Azure Pipelines using REST API?
Hello Everyone
Welcome to CloudAffaire and this is Debjeet.
Today we are going to discuss how to create and execute Azure Pipelines using REST API. We will also discuss components and structure of Azure Pipelines.
Azure Pipeline Structure:
A pipeline is one or more stages that describe a CI/CD process. Stages are the major divisions in a pipeline. The stages “Build this app,” “Run these tests,” and “Deploy to preproduction” are good examples. A stage is one or more jobs, which are units of work assignable to the same machine. You can arrange both stages and jobs into dependency graphs. Examples include “Run this stage before that one” and “This job depends on the output of that job.” A job is a linear series of steps. Steps can be tasks, scripts, or references to external templates.
This hierarchy is reflected in the structure of a YAML file (azure-pipelines.yml) like:
- Pipeline
- Stage A
- Job 1
- Step 1.1
- Task 1.1.1
- Task 1.1.2
- …
- Step 1.2
- …
- Step 1.1
- Job 2
- …
- Job 1
- Stage B
- …
- Stage A
Components of Azure Pipeline:
Pipeline: A pipeline defines the continuous integration and deployment process for your app. It’s made up of one or more stages. It can be thought of as a workflow that defines how your test, build, and deployment steps are run.
Stage: A stage is a collection of related jobs. By default, stages run sequentially. Each stage starts only after the preceding stage is complete unless otherwise specified via the dependsOn property.
Job: A stage contains one or more jobs. Each job runs on an agent. A job represents an execution boundary of a set of steps. All of the steps run together on the same agent. Jobs are most useful when you want to run a series of steps in different environments. For example, you might want to build two configurations – x86 and x64. In this case, you have one stage and two jobs. One job would be for x86 and the other job would be for x64.
Step: A step is the smallest building block of a pipeline. For example, a pipeline might consist of build and test steps. A step can either be a script or a task. A task is simply a pre-created script offered as a convenience to you.
Task: A task is the building block for defining automation in a pipeline. A task is packaged script or procedure that has been abstracted with a set of inputs.
Script: A script runs code as a step in your pipeline using command line, PowerShell, or Bash. You can write cross-platform scripts for macOS, Linux, and Windows. Unlike a task, a script is custom code that is specific to your pipeline.
Trigger: A trigger is something that’s set up to tell the pipeline when to run. You can configure a pipeline to run upon a push to a repository, at scheduled times, or upon the completion of another build. All of these actions are known as triggers.
Pool: The pool keyword specifies which agent to use to execute your pipeline, stage, or jobs.
Variables: You can pass variables in your Azure Pipeline using variable block.
Image source: https://docs.microsoft.com/en-us/azure/devops/pipelines/get-started/media/key-concepts-overview.svg?view=azure-devops
I know its hard to graphs all these at once, particularly if you are just starting with Azure Pipeline. But don’t worry as I will be explaining each component of an Azure pipeline separately in the upcoming blogs. I have intentionally left out some components like resource, template, parameters, artifacts etc. as we are just getting started, once again I will cover those as well at a later stage once your basic understanding is clear.
Next, we are going to create and execute a new Azure pipeline using REST API.
How to create and execute Azure Pipelines using REST API?
Prerequisites:
- One active Azure DevOps account
- Personal Access Token (PAT)
- A self-hosted agent registered to your Azure DevOps organization
Step 1: Check if you can make API call to your Azure DevOps account.
|
1 2 3 4 5 6 7 8 9 |
## Define variables ORGANIZATION=" USER=$(az account show | jq -r .user.name) PROJECT=" PAT=' ## Test if authentication working (List projects in your organization) curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/_apis/projects?api-version=6.0 | jq . |
Step 2: Create a new Azure DevOps project.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
## Create a new Azure DevOps project curl -s --request POST \ -u $USER:$PAT \ --header "Content-Type: application/json" \ --data '{ "name": "'"$PROJECT"'", "description": "Demo Project For Azure Pipeline", "capabilities": { "versioncontrol": { "sourceControlType": "Git" }, "processTemplate": { "templateTypeId": "6b724908-ef14-45cf-84f8-768b5384da45" } } }' \ https://dev.azure.com/$ORGANIZATION/_apis/projects?api-version=6.0 | jq . |
Step 3: Clone the default Azure Repo created in the new Azure DevOps project.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
## Get project default repository details REPOSITORY_ID=$(curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/git/repositories?api-version=6.0 | jq -r .value[0].id) && curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/git/repositories/$REPOSITORY_ID?api-version=6.0 | jq . ## Clone project default repository REPO_HTTPS_CLONE_URL=$(curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/git/repositories?api-version=6.0 | jq -r .value[0].webUrl ) && B64_PAT=$(printf "%s"":$PAT" | base64) && git -c http.extraHeader="Authorization: Basic ${B64_PAT}" \ clone $REPO_HTTPS_CLONE_URL |
Step 4: Create the Azure Pipeline configuration file and push it to your Azure Repo.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
## Create your pipline definition cd $PROJECT cat < pool: mypool #or u can use a microsoft-hosted agent trigger: branches: include: - '*' stages: - stage: Build jobs: - job: BuildJob steps: - script: echo Building! - stage: Test jobs: - job: TestOnWindows steps: - script: echo Testing on Windows! - job: TestOnLinux steps: - script: echo Testing on Linux! - stage: Deploy jobs: - job: Deploy steps: - script: echo Deploying the code! EOF ## Push the changes to Azure Repo git add azure-pipelines.yml git commit -m "azure-pipelines.yml added" git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push |
Step 5: Create a new Azure Pipeline using REST API.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
## Create a new Azure Pipeline using REST API curl -s --request POST \ -u $USER:$PAT \ --header "Content-Type: application/json" \ --data '{ "folder": null, "name": "mypipeline", "configuration": { "type": "yaml", "path": "/azure-pipelines.yml", "repository": { "id": "'"$REPOSITORY_ID"'", "name": "'"$PROJECT"'", "type": "azureReposGit" } } }' \ https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/pipelines?api-version=6.0-preview.1 | jq . ## Get Azure Pipeline details PIPELINE_ID=$(curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/pipelines?api-version=6.0-preview.1 | jq .value[0].id) && curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/pipelines/$PIPELINE_ID?api-version=6.0-preview.1 | jq . |
Now we are ready to execute our Azure Pipeline. We can execute our Azure Pipeline in two ways, either by pushing any change to our Azure Repo (Since Trigger in the pipeline configuration file is set to *, any push to any branch in the Azure repo will result execution of your Azure Pipeline) or using REST API to trigger the pipeline manually. I will cover both 😊
Step 6: Update the Azure Pipeline configuration file and push the changes to your Azure Repo.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 |
## Update your pipline definition cat <<'EOF'> azure-pipelines.yml pool: mypool #or you can use microsoft-hosted agent trigger: branches: include: - '*' variables: MY_VAR: 'cloudaffaire' stages: - stage: Build jobs: - job: BuildJob steps: - script: echo Building! - stage: Test jobs: - job: TestOnWindows steps: - script: echo Testing on Windows! - job: TestOnLinux steps: - script: echo Testing on Linux! - stage: Deploy jobs: - job: Deploy steps: - script: echo Deploying the code! - script: echo $(MY_VAR) EOF ## Push the changes to Azure Repo git add azure-pipelines.yml git commit -m "azure-pipelines.yml updated" git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push ## Get Azure pipeline execution details PIPELINE_RUN_ID=$(curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/pipelines/$PIPELINE_ID/runs?api-version=6.0-preview.1 | jq .value[0].id) && curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/pipelines/$PIPELINE_ID/runs/$PIPELINE_RUN_ID?api-version=6.0-preview.1 | jq . |
As soon as the change is pushed to your Azure Repo, a new Azure Pipeline will get triggered.
Next, we will execute our Azure Pipeline manually using REST API.
Step 7: Execute an Azure Pipeline manually using REST API.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
## Execute Azure pipeline manually using REST API curl -s --request POST \ -u $USER:$PAT \ --header "Content-Type: application/json" \ --data '{ "resources": { "repositories": { "self": { "refName": "refs/heads/main" } } } }' \ https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/pipelines/$PIPELINE_ID/runs?api-version=6.0-preview.1 | jq . |
Step 8: Clean up.
|
1 2 3 4 5 6 7 8 |
## Clenup PROJECT_ID=$(curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/_apis/projects?api-version=6.0 \ | jq -r '.value[] | select(.name == "'"$PROJECT"'") | .id') && curl -s --request DELETE \ -u $USER:$PAT \ --header "Content-Type: application/json" \ https://dev.azure.com/$ORGANIZATION/_apis/projects/$PROJECT_ID?api-version=6.0 |
Hope you have enjoyed this article. To know more about Azure DevOps, please refer below official documentation
https://docs.microsoft.com/en-us/azure/devops/?view=azure-devops
hi, Can you please help in create and executing release as well using rest api. as of now with the above article we are able to create pipeline automatically. like wise need help in release and deploy applicaiton to AZURE