Azure Pipeline Part 7 – Variables

Hello Everyone

Welcome to CloudAffaire and this is Debjeet.

Today we are going to discuss how to declare and use variables in Azure pipelines with examples. Variables are used to pass and use certain information in your Azure pipelines. You can use your own user-defined variables and environment variables or system defined variables in Azure pipeline.

Next, we are going to explain different concepts related to variables with examples.

Azure Pipeline Part 7 – Variables

Prerequisites:

Setup:

## -----
## Setup
## -----

## Define variables
ORGANIZATION="<EXISTING_AZURE_DEVOPS_ORGANIZATION_NAME>"
USER=$(az account show | jq -r .user.name)
PROJECT="<NEW_AZURE_DEVOPS_PROJECT_NAME>"
PAT='<PERSONAL_ACCESS_TOKEN>'
B64_PAT=$(printf "%s"":$PAT" | base64)

## Test if authentication working (List projects in your organization)
curl -s -u $USER:$PAT \
https://dev.azure.com/$ORGANIZATION/_apis/projects?api-version=6.0 | jq .

## Create a new Azure DevOps project
curl -s --request POST \
-u $USER:$PAT \
--header "Content-Type: application/json" \
--data '{
  "name": "'"$PROJECT"'",
  "description": "Demo Project For Azure Pipeline",
  "capabilities": {
    "versioncontrol": {
      "sourceControlType": "Git"
    },
    "processTemplate": {
      "templateTypeId": "6b724908-ef14-45cf-84f8-768b5384da45"
    }
  }
}' \
https://dev.azure.com/$ORGANIZATION/_apis/projects?api-version=6.0 | jq .

## Get project default repository details
REPOSITORY_ID=$(curl -s -u $USER:$PAT \
https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/git/repositories?api-version=6.0 | jq -r .value[0].id) &&
curl -s -u $USER:$PAT \
https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/git/repositories/$REPOSITORY_ID?api-version=6.0 | jq .

## Create a new Azure Pipeline using REST API
curl -s --request POST \
-u $USER:$PAT \
--header "Content-Type: application/json" \
--data '{
  "folder": null,
  "name": "mypipeline",
  "configuration": {
    "type": "yaml",
    "path": "/azure-pipelines.yml",
    "repository": {
      "id": "'"$REPOSITORY_ID"'",
      "name": "'"$PROJECT"'",
      "type": "azureReposGit"
    }
  }
}' \
https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/pipelines?api-version=6.0-preview.1 | jq .

## Clone project default repository
REPO_HTTPS_CLONE_URL=$(curl -s -u $USER:$PAT \
https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/git/repositories?api-version=6.0 | jq -r .value[0].webUr
) &&
B64_PAT=$(printf "%s"":$PAT" | base64) &&
git -c http.extraHeader="Authorization: Basic ${B64_PAT}" \
clone $REPO_HTTPS_CLONE_URL
cd $PROJECT

## -----

## Setup

## -----

## Define variables

ORGANIZATION=""

USER=$(az account show | jq -r .user.name)

PROJECT=""

PAT=''

B64_PAT=$(printf "%s"":$PAT" | base64)

## Test if authentication working (List projects in your organization)

curl -s -u $USER:$PAT \

https://dev.azure.com/$ORGANIZATION/_apis/projects?api-version=6.0 | jq .

## Create a new Azure DevOps project

curl -s --request POST \

-u $USER:$PAT \

--header "Content-Type: application/json" \

--data '{

"name": "'"$PROJECT"'",

"description": "Demo Project For Azure Pipeline",

"capabilities": {

"versioncontrol": {

"sourceControlType": "Git"

"processTemplate": {

"templateTypeId": "6b724908-ef14-45cf-84f8-768b5384da45"

}

}' \

https://dev.azure.com/$ORGANIZATION/_apis/projects?api-version=6.0 | jq .

## Get project default repository details

REPOSITORY_ID=$(curl -s -u $USER:$PAT \

https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/git/repositories?api-version=6.0 | jq -r .value[0].id) &&

curl -s -u $USER:$PAT \

https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/git/repositories/$REPOSITORY_ID?api-version=6.0 | jq .

## Create a new Azure Pipeline using REST API

curl -s --request POST \

-u $USER:$PAT \

--header "Content-Type: application/json" \

--data '{

"folder": null,

"name": "mypipeline",

"configuration": {

"type": "yaml",

"path": "/azure-pipelines.yml",

"repository": {

"id": "'"$REPOSITORY_ID"'",

"name": "'"$PROJECT"'",

"type": "azureReposGit"

}

}' \

https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/pipelines?api-version=6.0-preview.1 | jq .

## Clone project default repository

REPO_HTTPS_CLONE_URL=$(curl -s -u $USER:$PAT \

https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/git/repositories?api-version=6.0 | jq -r .value[0].webUr

) &&

B64_PAT=$(printf "%s"":$PAT" | base64) &&

git -c http.extraHeader="Authorization: Basic ${B64_PAT}" \

clone $REPO_HTTPS_CLONE_URL

cd $PROJECT

Azure Pipeline – Variables:

Azure Pipeline – Variable syntax:

## Syntax:
variables:
- name: string  # name of a variable
  value: string # value of the variable
- group: string # name of a variable group

## Syntax:

variables:

- name: string # name of a variable

value: string # value of the variable

- group: string # name of a variable group

Azure Pipeline – Variable example:

User defined variables:

In YAML pipelines, you can set variables using a variable block. You can also specify variables outside of a YAML pipeline in the UI. When you set a variable in the UI, that variable can be encrypted and set as secret. Azure Pipelines supports three different ways to reference variables: macro, template expression, and runtime expression. Each syntax can be used for a different purpose and has some limitations.

Template expression variables (${{ variables.var }}) get processed at compile time, before runtime starts.
Macro syntax variables ($(var)) get processed during runtime before a task runs.
Runtime expressions ($[variables.var]) also get processed during runtime but were designed for use with conditions and expressions.

## ---------
## Variables
## ---------

## Create pipeline definition
cat <<'EOF'> azure-pipelines.yml
pool: mypool
variables:
- name: name
  value: debjeet
steps:
- script: echo "hello $(name)!"
EOF

## Push the changes to Azure Repo
git add .
git commit -m "update1"
git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push

## ------------------------------------------
## Different ways to retrive variables values
## ------------------------------------------

## Update pipeline definition
cat <<'EOF'> azure-pipelines.yml
pool: mypool
variables:
- name: name
  value: cloudaffaire
- name: isMain
  value: $[eq(variables['Build.SourceBranch'], 'refs/heads/master')]
stages:
- stage: mystage
  condition: eq(variables.isMain, 'true')
  jobs:
  - job: myjob
    steps:
    - script: |
        echo $(name)
        echo $NAME
        echo ${{ variables.name }}
EOF

## Push the changes to Azure Repo
git add .
git commit -m "update2"
git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push

## ---------

## Variables

## ---------

## Create pipeline definition

cat <<'EOF'> azure-pipelines.yml

pool: mypool

variables:

- name: name

value: debjeet

steps:

- script: echo "hello $(name)!"

EOF

## Push the changes to Azure Repo

git add .

git commit -m "update1"

git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push

## ------------------------------------------

## Different ways to retrive variables values

## ------------------------------------------

## Update pipeline definition

cat <<'EOF'> azure-pipelines.yml

pool: mypool

variables:

- name: name

value: cloudaffaire

- name: isMain

value: $[eq(variables['Build.SourceBranch'], 'refs/heads/master')]

stages:

- stage: mystage

condition: eq(variables.isMain, 'true')

jobs:

- job: myjob

steps:

- script: |

echo $(name)

echo $NAME

echo ${{ variables.name }}

EOF

## Push the changes to Azure Repo

git add .

git commit -m "update2"

git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push

Azure Pipeline Variable scope:

When you define the same variable in multiple places with the same name, the most locally scoped variable wins. So, a variable defined at the job level can override a variable set at the stage level. A variable defined at the stage level will override a variable set at the pipeline root level. A variable set in the pipeline root level will override a variable set in the Pipeline settings UI.

## --------------
## Variable scope
## --------------

## update pipeline definition
cat <<'EOF'> azure-pipelines.yml
pool: mypool
variables:
- name: name
  value: cloudaffaire
- name: pipeline_level
  value: scope_pipeline
stages:
- stage: one
  variables:
  - name: stage_level
    value: scope_stage
  jobs:
  - job: A
    steps:
    - script: |
        echo $(pipeline_level)
        echo $(stage_level)
  - job: B
    variables
    - name: job_level
      value: scope_job
    - name: name
      value: debjeet
    steps:
    - script: |
        echo $(pipeline_level)
        echo $(stage_level)
        echo $(job_level)
        echo $(name)
EOF

## Push the changes to Azure Repo
git add .
git commit -m "update3"
git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push

## --------------

## Variable scope

## --------------

## update pipeline definition

cat <<'EOF'> azure-pipelines.yml

pool: mypool

variables:

- name: name

value: cloudaffaire

- name: pipeline_level

value: scope_pipeline

stages:

- stage: one

variables:

- name: stage_level

value: scope_stage

jobs:

- job: A

steps:

- script: |

echo $(pipeline_level)

echo $(stage_level)

- job: B

variables

- name: job_level

value: scope_job

- name: name

value: debjeet

steps:

- script: |

echo $(pipeline_level)

echo $(stage_level)

echo $(job_level)

echo $(name)

EOF

## Push the changes to Azure Repo

git add .

git commit -m "update3"

git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push

Azure Pipelines – Variable Groups:

Variable groups store values and secrets that you might want to be passed into a YAML pipeline or make available across multiple pipelines. You can share and use variables groups in multiple pipelines in the same project.

Create a new variable group:

Step 1: Log in to your Azure DevOps organization > Select the project created during setup => Azure Pipelines => Library => Click “Variable group”.

Azure Pipeline Part 7 – Variables

Step 2: Provide a name and description to the variable group and click “Add” to add new variable name and value under this variable group. Finally click “Save”.

Azure Pipeline Part 7 – Variables

Now we can use this variable group in any pipeline inside this project.

Use variable group in Azure Pipeline:

## ---------------
## Variable Groups
## ---------------

## update pipeline definition
cat <<'EOF'> azure-pipelines.yml
pool: mypool
variables:
- group: myVarGroup
- name: password
  value: 'mypassword'
steps:
- script: |
    echo "UserName: $(username)"
    echo "Password: $(password)"
EOF

## Push the changes to Azure Repo
git add .
git commit -m "update4"
git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push

## ---------------

## Variable Groups

## ---------------

## update pipeline definition

cat <<'EOF'> azure-pipelines.yml

pool: mypool

variables:

- group: myVarGroup

- name: password

value: 'mypassword'

steps:

- script: |

echo "UserName: $(username)"

echo "Password: $(password)"

EOF

## Push the changes to Azure Repo

git add .

git commit -m "update4"

git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push

Note: We have added the password in plain text in our Azure pipeline config YAML file. If you want to store and use any sensitive data in your Azure Pipeline, you should use Azure Vault and reference that Azure Vault to your Variable group. Which is exactly what we are going to cover next.

Note: You might need to grant additional permission to your Azure Pipeline to able to use Variable group.

Azure Pipeline Part 7 – Variables

Use Secrets in Azure Pipeline using Azure Key Vault and Variable Groups:

Step 1: Create a new Azure Key Vault.

Provide a unique name and click “Review + create”, and once validation is done click “Create”.

Azure Pipeline Part 7 – Variables

Step 2: Get inside the Key vault and create a new secret.

Azure Pipeline Part 7 – Variables

Provide a name and value for your secret and click “Create”.

Azure Pipeline Part 7 – Variables

I have created two secrets namely username and password for this demo.

Azure Pipeline Part 7 – Variables

Step 3: Log in to your Azure DevOps account and navigate to the variable group under Azure pipeline library and click

Provide a name and select “link secrets from an Azure key vault as variables”, select your Azure subscription and Azure Key vault which has your secrets.

Azure Pipeline Part 7 – Variables

Note: You might need to authorize your Azure DevOps account to use your Azure subscription and key vault.

Click “Add” to add the secrets from Azure Key Vault to Azure Pipeline variable group and finally “Save” your variable group.

Azure Pipeline Part 7 – Variables

Now we are ready to use secrets from Azure Key Vault into our Azure DevOps Pipeline.

## -----------------------------------
## Variable Groups With Sencitive Data
## -----------------------------------

## update pipeline definition
cat <<'EOF'> azure-pipelines.yml
pool: mypool
variables:
- group: myNewVarGroup
steps:
- script: |
    echo "UserName: $(username)"
    echo "Password: $(password)"
EOF

## Push the changes to Azure Repo
git add .
git commit -m "update5"
git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push

## -----------------------------------

## Variable Groups With Sencitive Data

## -----------------------------------

## update pipeline definition

cat <<'EOF'> azure-pipelines.yml

pool: mypool

variables:

- group: myNewVarGroup

steps:

- script: |

echo "UserName: $(username)"

echo "Password: $(password)"

EOF

## Push the changes to Azure Repo

git add .

git commit -m "update5"

git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push

Note: You might need to grant additional permission to your Azure Pipeline to able to use Variable group and Azure subscription.

Environment Variables:

Environment variables are specific to the operating system you are using. They are injected into a pipeline in platform-specific ways. The format corresponds to how environment variables get formatted for your specific scripting platform.

On UNIX systems (macOS and Linux), environment variables have the format $NAME. On Windows, the format is %NAME% for batch and $env:NAME in PowerShell.

System and user-defined variables also get injected as environment variables for your platform. When variables are turned into environment variables, variable names become uppercase, and periods turn into underscores.

## ---------------------
## Environment Variables
## ---------------------

## update pipeline definition
cat <<'EOF'> azure-pipelines.yml
pool: mypool
variables:
- name: name
  value: debjeet
steps:
- script: |
    echo "User Defined Variable: $(name)"
    echo "User Defined Environment Variable: $WEBSITE"
    echo "All Available Environment Variables: "
    env
  env:
    WEBSITE: cloudaffaire
EOF

## Push the changes to Azure Repo
git add .
git commit -m "update6"
git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push

## ---------------------

## Environment Variables

## ---------------------

## update pipeline definition

cat <<'EOF'> azure-pipelines.yml

pool: mypool

variables:

- name: name

value: debjeet

steps:

- script: |

echo "User Defined Variable: $(name)"

echo "User Defined Environment Variable: $WEBSITE"

echo "All Available Environment Variables: "

env

env:

WEBSITE: cloudaffaire

EOF

## Push the changes to Azure Repo

git add .

git commit -m "update6"

git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push

System Variables:

In addition to user-defined variables, Azure Pipelines has system variables with predefined values. System variables are set with their current value when you run the pipeline. Some variables are set automatically. As a pipeline author or end user, you change the value of a system variable before the pipeline is run. System variables are read-only.

## ----------------
## System Variables
## ----------------

## update pipeline definition
cat <<'EOF'> azure-pipelines.yml
pool: mypool
steps:
  - bash: |
      echo Used as an environment variable in a script and as a parameter in a build task - $(System.AccessToken)
      echo The GUID of the TFS collection or Azure DevOps organization - $(System.CollectionId)
      echo The URI of the TFS collection or Azure DevOps organization - $(System.CollectionUri)
      echo The local path on the agent where your source code files are downloaded - $(System.DefaultWorkingDirectory)
      echo The ID of the build pipeline - $(System.DefinitionId)
      echo Set to build if the pipeline is a build - $(System.HostType)
      echo Set to 1 the first time this job is attempted, and increments every time the job is retried - $(System.JobAttempt)
      echo The human-readable name given to a job - $(System.JobDisplayName)
      echo A unique identifier for a single attempt of a single job - $(System.JobId)
      echo The name of the job, typically used for expressing dependencies and accessing output variables - $(System.JobName)
      echo Set to 1 the first time this phase is attempted, and increments every time the job is retried - $(System.PhaseAttempt)
      echo The human-readable name given to a phase - $(System.PhaseDisplayName)
      echo A string-based identifier for a job, typically used for expressing dependencies and accessing output variables - $(System.PhaseName)
      echo Set to 1 the first time this stage is attempted, and increments every time the job is retried - $(System.StageAttempt)
      echo The human-readable name given to a stage - $(System.StageDisplayName)
      echo A string-based identifier for a stage, typically used for expressing dependencies and accessing output variables - $(System.StageName)
      echo If the pull request is from a fork of the repository, this variable is set to True. Otherwise, it is set to False - $(System.PullRequest.IsFork)
      echo The ID of the pull request that caused this build. For example: 17 - $(System.PullRequest.PullRequestId)
      echo The number of the pull request that caused this build - $(System.PullRequest.PullRequestNumber)
      echo The branch that is being reviewed in a pull request - $(System.PullRequest.SourceBranch)
      echo he URL to the repo that contains the pull request - $(System.PullRequest.SourceRepositoryURI)
      echo The branch that is the target of a pull request - $(System.PullRequest.TargetBranch)
      echo The URI of the TFS collection or Azure DevOps organization - $(System.TeamFoundationCollectionUri)
      echo The name of the project that contains this build - $(System.TeamProject)
      echo The ID of the project that this build belongs to - $(System.TeamProjectId)
    displayName: Syatem Variables
EOF

## Push the changes to Azure Repo
git add .
git commit -m "update7"
git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push

## ----------------

## System Variables

## ----------------

## update pipeline definition

cat <<'EOF'> azure-pipelines.yml

pool: mypool

steps:

- bash: |

echo Used as an environment variable in a script and as a parameter in a build task - $(System.AccessToken)

echo The GUID of the TFS collection or Azure DevOps organization - $(System.CollectionId)

echo The URI of the TFS collection or Azure DevOps organization - $(System.CollectionUri)

echo The local path on the agent where your source code files are downloaded - $(System.DefaultWorkingDirectory)

echo The ID of the build pipeline - $(System.DefinitionId)

echo Set to build if the pipeline is a build - $(System.HostType)

echo Set to 1 the first time this job is attempted, and increments every time the job is retried - $(System.JobAttempt)

echo The human-readable name given to a job - $(System.JobDisplayName)

echo A unique identifier for a single attempt of a single job - $(System.JobId)

echo The name of the job, typically used for expressing dependencies and accessing output variables - $(System.JobName)

echo Set to 1 the first time this phase is attempted, and increments every time the job is retried - $(System.PhaseAttempt)

echo The human-readable name given to a phase - $(System.PhaseDisplayName)

echo A string-based identifier for a job, typically used for expressing dependencies and accessing output variables - $(System.PhaseName)

echo Set to 1 the first time this stage is attempted, and increments every time the job is retried - $(System.StageAttempt)

echo The human-readable name given to a stage - $(System.StageDisplayName)

echo A string-based identifier for a stage, typically used for expressing dependencies and accessing output variables - $(System.StageName)

echo If the pull request is from a fork of the repository, this variable is set to True. Otherwise, it is set to False - $(System.PullRequest.IsFork)

echo The ID of the pull request that caused this build. For example: 17 - $(System.PullRequest.PullRequestId)

echo The number of the pull request that caused this build - $(System.PullRequest.PullRequestNumber)

echo The branch that is being reviewed in a pull request - $(System.PullRequest.SourceBranch)

echo he URL to the repo that contains the pull request - $(System.PullRequest.SourceRepositoryURI)

echo The branch that is the target of a pull request - $(System.PullRequest.TargetBranch)

echo The URI of the TFS collection or Azure DevOps organization - $(System.TeamFoundationCollectionUri)

echo The name of the project that contains this build - $(System.TeamProject)

echo The ID of the project that this build belongs to - $(System.TeamProjectId)

displayName: Syatem Variables

EOF

## Push the changes to Azure Repo

git add .

git commit -m "update7"

git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push

You can get all types system variables in this link.

Clean up:

## --------
## Clean up
## --------

## Delete the Azure DevOps project and local repository
PROJECT_ID=$(curl -s -u $USER:$PAT \
https://dev.azure.com/$ORGANIZATION/_apis/projects?api-version=6.0 \
| jq -r '.value[] | select(.name == "'"$PROJECT"'") | .id') &&
curl -s --request DELETE \
-u $USER:$PAT \
--header "Content-Type: application/json" \
https://dev.azure.com/$ORGANIZATION/_apis/projects/$PROJECT_ID?api-version=6.0

cd ..
rm -rf $PROJECT

## --------

## Clean up

## --------

## Delete the Azure DevOps project and local repository

PROJECT_ID=$(curl -s -u $USER:$PAT \

https://dev.azure.com/$ORGANIZATION/_apis/projects?api-version=6.0 \

| jq -r '.value[] | select(.name == "'"$PROJECT"'") | .id') &&

curl -s --request DELETE \

-u $USER:$PAT \

--header "Content-Type: application/json" \

https://dev.azure.com/$ORGANIZATION/_apis/projects/$PROJECT_ID?api-version=6.0

cd ..

rm -rf $PROJECT

Hope you have enjoyed this article. To know more about Azure DevOps, please refer below official documentation

https://docs.microsoft.com/en-us/azure/devops/?view=azure-devops

Azure Pipeline Part 7 – Variables

Azure Pipeline Part 7 – Variables

Prerequisites:

Setup:

Azure Pipeline – Variables:

Azure Pipeline – Variable syntax:

Azure Pipeline – Variable example:

User defined variables:

Azure Pipeline Variable scope:

Azure Pipelines – Variable Groups:

Create a new variable group:

Use variable group in Azure Pipeline:

Use Secrets in Azure Pipeline using Azure Key Vault and Variable Groups:

Environment Variables:

System Variables:

Clean up:

Leave a Reply Cancel reply