Azure Pipeline Part 1 – Pool Demands Capabilities Agent Variables
Hello Everyone
Welcome to CloudAffaire and this is Debjeet.
Today we are going to discuss pool, demands, capabilities, and agent variables in Azure Pipeline. When you create an Azure Pipeline, you need to define the place where the pipeline will get executed. Using pool, you can define in which server (agent) your pipeline will get executed.
What is Azure Pipelines agents?
To build your code or deploy your software using Azure Pipelines, you need at least one agent. As you add more code and people, you’ll eventually need more. When your pipeline runs, the system begins one or more jobs. An agent is computing infrastructure with installed agent software that runs one job at a time.
- Microsoft-hosted agents: If your pipelines are in Azure Pipelines, then you’ve got a convenient option to run your jobs using a Microsoft-hosted agent. With Microsoft-hosted agents, maintenance and upgrades are taken care of for you.
- Self-hosted agents: An agent that you set up and manage on your own to run jobs is a self-hosted agent. You can use self-hosted agents in Azure Pipelines or Team Foundation Server (TFS). Self-hosted agents give you more control to install dependent software needed for your builds and deployments. Also, machine-level caches and configuration persist from run to run, which can boost speed.
What is Azure Pipelines agent pools?
Instead of managing each agent individually, you organize agents into agent pools. When you configure an agent, it is registered with a single pool, and when you create a pipeline, you specify which pool the pipeline uses. When you run the pipeline, it runs on an agent from that pool that meets the demands of the pipeline.
Pool: The pool keyword specifies which pool to use for a job of the pipeline. You can specify a pool at the pipeline, stage, or job level. The pool specified at the lowest level of the hierarchy is used to run the job.
Demand: The demand keyword is used to check if your agent meets a certain capability. Demands allows you to execute your pipeline based on certain conditions (capability) that is met in your agent. For example, suppose you have two agents with same OS but with different Python version, you can create a custom capability for Python version and assign to your agent and use that capability with demand in your pipeline config file to execute your pipeline in one of the agents which meets the Python version required by your pipeline.
Capability: Certain features of the agent that you can use as condition with demand to execute your pipeline on a specific agent based on some condition.
|
1 2 3 4 5 6 |
## Syntax pool: name: string # name of the pool to run this job in demands: string | [ string ] # see the following "Demands" topic vmImage: string # name of the VM image you want to use; valid only in the Microsoft-hosted pool |
Azure Pipeline Part 1 – Pool Demands Capabilities Agent Variables
Prerequisites:
- One active Azure DevOps account
- Personal Access Token (PAT)
- A self-hosted agent registered to your Azure DevOps organization
Step 1: Setup a new Azure DevOps project for this demo.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 |
## ----- ## Setup ## ----- ## Define variables ORGANIZATION=" USER=$(az account show | jq -r .user.name) PROJECT=" PAT=' B64_PAT=$(printf "%s"":$PAT" | base64) ## Test if authentication working (List projects in your organization) curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/_apis/projects?api-version=6.0 | jq . ## Create a new Azure DevOps project curl -s --request POST \ -u $USER:$PAT \ --header "Content-Type: application/json" \ --data '{ "name": "'"$PROJECT"'", "description": "Demo Project For Azure Pipeline", "capabilities": { "versioncontrol": { "sourceControlType": "Git" }, "processTemplate": { "templateTypeId": "6b724908-ef14-45cf-84f8-768b5384da45" } } }' \ https://dev.azure.com/$ORGANIZATION/_apis/projects?api-version=6.0 | jq . ## Create a new Azure Pipeline using REST API curl -s --request POST \ -u $USER:$PAT \ --header "Content-Type: application/json" \ --data '{ "folder": null, "name": "mypipeline", "configuration": { "type": "yaml", "path": "/azure-pipelines.yml", "repository": { "id": "'"$REPOSITORY_ID"'", "name": "'"$PROJECT"'", "type": "azureReposGit" } } }' \ https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/pipelines?api-version=6.0-preview.1 | jq . ## Get project default repository details REPOSITORY_ID=$(curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/git/repositories?api-version=6.0 | jq -r .value[0].id) && curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/git/repositories/$REPOSITORY_ID?api-version=6.0 | jq . ## Clone project default repository REPO_HTTPS_CLONE_URL=$(curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/git/repositories?api-version=6.0 | jq -r .value[0].webUrl ) && B64_PAT=$(printf "%s"":$PAT" | base64) && git -c http.extraHeader="Authorization: Basic ${B64_PAT}" \ clone $REPO_HTTPS_CLONE_URL cd $PROJECT |
Step 2: Microsoft-Hosted agent.
If you do not define any pool configuration in your Azure pipeline config file, your pipeline will get executed in a Microsoft-hosted agent.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
## ----------------------- ## Microsoft-Hosted Agents ## ----------------------- ## Create a new pipeline definition cat <<'EOF'> azure-pipelines.yml steps: - script: echo Hello World! EOF ## Push the changes to Azure Repo git add azure-pipelines.yml git commit -m "pipeline updated" git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push |
Next, we will define pool configuration to use Microsoft-hosted agents.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
## Update pipeline definition cat <<'EOF'> azure-pipelines.yml pool: vmImage: 'windows-latest' stages: - stage: TestLinuxWindowsAgent jobs: - job: WindowsJob steps: - script: echo Testing on Windows! pool: vmImage: 'windows-latest' - job: LinuxJob steps: - script: echo Testing on Linux! pool: vmImage: 'ubuntu-latest' - stage: TestMacAgent pool: vmImage: 'macOS-latest' jobs: - job: MacJobOne steps: - script: echo Testing on MAC! - job: MacJobTwo steps: - script: echo Testing on MAC! EOF ## Push the changes to Azure Repo git add azure-pipelines.yml git commit -m "pipeline updated" git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push |
Observe: We have defined the pool keyword in pipeline, stage and job levels.
Get the latest Microsoft-hosted agents name using this link.
Next, we will use a self-hosted agent to explain demand and capabilities.
Step 3: Self-hosted agent.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
## ------------------ ## Self-Hosted Agents ## ------------------ ## Update pipeline definition cat <<'EOF'> azure-pipelines.yml pool: mypool steps: - script: echo Hello World! EOF ## Push the changes to Azure Repo git add azure-pipelines.yml git commit -m "pipeline updated" git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push |
Next, we will add a capability in our self-hosted agent and use this capability in our Azure pipeline using demand.
Step 4: Create a custom self-hosted agent capability
Log in to Azure DevOps portal and navigate to self-hosted agent capability configuration and click “Add a new capability”.
Provide a name and value for the new capability and click “Add”.
Step 5: Demand and Capability.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 |
## ---------------------- ## Demands & Capabilities ## ---------------------- ## Update pipeline definition cat <<'EOF'> azure-pipelines.yml stages: - stage: TestAgentDemand jobs: - job: Job1 steps: - script: echo Job1 will get executed as Agent OS is - $(Agent.OS) pool: name: mypool demands: - Agent.OS -equals Linux - job: Job2 steps: - script: echo Job2 will fail as mypool has Linux OS continueOnError: true # as this job will fail pool: name: mypool demands: - Agent.OS -equals Windows - stage: TestUserDefinedCapabilities jobs: - job: Job1 steps: - script: echo Job1 will get executed pool: name: mypool demands: - OSNAME # checks if capability exist - OSNAME -equals AmazonLinux2 # checks capability value EOF ## Push the changes to Azure Repo git add azure-pipelines.yml git commit -m "pipeline updated" git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push |
Next, we will explore different agent variables available in Azure Pipeline.
Step 6: Agent variables.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
## --------------- ## Agent variables ## --------------- ## Update pipeline definition cat <<'EOF'> azure-pipelines.yml pool: mypool steps: - bash: | echo The local path on the agent where all folders for a given build pipeline are created - $(Agent.BuildDirectory) echo A mapping from container resource names in YAML to their Docker IDs at runtime - $(Agent.ContainerMapping) echo The directory the agent is installed into - $(Agent.HomeDirectory) echo The ID of the agent - $(Agent.Id) echo The name of the running job - $(Agent.JobName) echo The name of the machine on which the agent is installed - $(Agent.MachineName) echo The name of the agent that is registered with the pool - $(Agent.Name) echo The operating system of the agent host - $(Agent.OS) echo The operating system processor architecture of the agent host - $(Agent.OSArchitecture) echo A temporary folder that is cleaned after each pipeline job - $(Agent.TempDirectory) echo Used by Node Tool Installer and Use Python Version to switch between multiple versions - $(Agent.ToolsDirectory) echo The working directory for this agent - $(Agent.WorkFolder) displayName: Agent Variables EOF ## Push the changes to Azure Repo git add azure-pipelines.yml git commit -m "pipeline updated" git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push |
Step 7: Clean up.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
## -------- ## Clean up ## -------- ## Delete the Azure DevOps project and local repository PROJECT_ID=$(curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/_apis/projects?api-version=6.0 \ | jq -r '.value[] | select(.name == "'"$PROJECT"'") | .id') && curl -s --request DELETE \ -u $USER:$PAT \ --header "Content-Type: application/json" \ https://dev.azure.com/$ORGANIZATION/_apis/projects/$PROJECT_ID?api-version=6.0 cd .. rm -rf $PROJECT |
Hope you have enjoyed this article. To know more about Azure DevOps, please refer below official documentation
https://docs.microsoft.com/en-us/azure/devops/?view=azure-devops