Azure Pipeline Part 9 – Resources
Hello Everyone
Welcome to CloudAffaire and this is Debjeet.
Today we are going to discuss how to use Resources in Azure Pipeline with examples. A resource is any external service that is consumed as part of your pipeline. An example of a resource is another CI/CD pipeline that produces:
- Artifacts like Azure Pipelines or Jenkins.
- Code repositories like GitHub, Azure Repos, or Git.
- Container-image registries like Azure Container Registry or Docker hub.
Resources in YAML represent sources of pipelines, containers, repositories, and types.
Azure Pipeline Part 9 – Resources
Prerequisites:
- One active Azure DevOps account
- Personal Access Token (PAT)
- A self-hosted agent registered to your Azure DevOps organization
Setup:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 |
## ----- ## Setup ## ----- ## Define variables ORGANIZATION="<EXISTING_AZURE_DEVOPS_ORGANIZATION_NAME>" USER=$(az account show | jq -r .user.name) PROJECT="<NEW_AZURE_DEVOPS_PROJECT_NAME>" PAT='<PERSONAL_ACCESS_TOKEN>' B64_PAT=$(printf "%s"":$PAT" | base64) ## Test if authentication working (List projects in your organization) curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/_apis/projects?api-version=6.0 | jq . ## Create a new Azure DevOps project curl -s --request POST \ -u $USER:$PAT \ --header "Content-Type: application/json" \ --data '{ "name": "'"$PROJECT"'", "description": "Demo Project For Azure Pipeline", "capabilities": { "versioncontrol": { "sourceControlType": "Git" }, "processTemplate": { "templateTypeId": "6b724908-ef14-45cf-84f8-768b5384da45" } } }' \ https://dev.azure.com/$ORGANIZATION/_apis/projects?api-version=6.0 | jq . ## Get project default repository details REPOSITORY_ID=$(curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/git/repositories?api-version=6.0 | jq -r .value[0].id) && curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/git/repositories/$REPOSITORY_ID?api-version=6.0 | jq . ## Create a new Azure Pipeline using REST API curl -s --request POST \ -u $USER:$PAT \ --header "Content-Type: application/json" \ --data '{ "folder": null, "name": "mypipeline", "configuration": { "type": "yaml", "path": "/azure-pipelines.yml", "repository": { "id": "'"$REPOSITORY_ID"'", "name": "'"$PROJECT"'", "type": "azureReposGit" } } }' \ https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/pipelines?api-version=6.0-preview.1 | jq . ## Clone project default repository REPO_HTTPS_CLONE_URL=$(curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/git/repositories?api-version=6.0 | jq -r .value[0].webUrl ) && B64_PAT=$(printf "%s"":$PAT" | base64) && git -c http.extraHeader="Authorization: Basic ${B64_PAT}" \ clone $REPO_HTTPS_CLONE_URL cd $PROJECT |
Azure Pipeline – Pipeline Resource:
If you have an Azure pipeline that produces artifacts, your pipeline can consume the artifacts by using the pipeline keyword to define a pipeline resource. You can also enable pipeline-completion triggers.
Azure Pipeline – Pipeline Resource Syntax:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
## Pipeline Resource Syntax resources: pipelines: - pipeline: string # identifier for the resource used in pipeline resource variables project: string # project for the source; optional for current project source: string # name of the pipeline that produces an artifact version: string # the pipeline run number to pick the artifact, defaults to latest pipeline successful across all stages; Used only for manual or scheduled triggers branch: string # branch to pick the artifact, optional; defaults to all branches; Used only for manual or scheduled triggers tags: [ string ] # list of tags required on the pipeline to pickup default artifacts, optional; Used only for manual or scheduled triggers trigger: # triggers are not enabled by default unless you add trigger section to the resource branches: # branch conditions to filter the events, optional; Defaults to all branches. include: [ string ] # branches to consider the trigger events, optional; Defaults to all branches. exclude: [ string ] # branches to discard the trigger events, optional; Defaults to none. tags: [ string ] # list of tags to evaluate for trigger event, optional; 2020.1 and greater stages: [ string ] # list of stages to evaluate for trigger event, optional; 2020.1 and greater |
Azure Pipeline – Pipeline Resource Example:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 |
## ----------------- ## Pipeline Resource ## ----------------- ## Trigger one pipeline from another ## Define a new repository name REPOSITORY="debjeet" ## Create a new repository PROJECT_ID=$(curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/_apis/projects?api-version=6.0 \ | jq -r '.value[] | select(.name == "'"$PROJECT"'") | .id') && curl -s --request POST \ -u $USER:$PAT \ --header "Content-Type: application/json" \ --data '{ "name": "'"$REPOSITORY"'", "project": { "id": "'"$PROJECT_ID"'" } }' \ https://dev.azure.com/$ORGANIZATION/_apis/git/repositories?api-version=6.0 | jq . ## Get new Azure repo details REPOSITORY_ID=$(curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/git/repositories?api-version=6.0 \ | jq -r '.value[] | select(.name == "'"$REPOSITORY"'") | .id') && curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/git/repositories/$REPOSITORY_ID?api-version=6.0 | jq . ## Clone new Azure repo cd .. REPO_HTTPS_CLONE_URL=$(curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/git/repositories/$REPOSITORY_ID?api-version=6.0 \ | jq -r .webUrl) && B64_PAT=$(printf "%s"":$PAT" | base64) && git -c http.extraHeader="Authorization: Basic ${B64_PAT}" \ clone $REPO_HTTPS_CLONE_URL && cd $REPOSITORY ## Create a new Azure Pipeline using the new Azure repo curl -s --request POST \ -u $USER:$PAT \ --header "Content-Type: application/json" \ --data '{ "folder": null, "name": "mynewpipeline", "configuration": { "type": "yaml", "path": "/azure-pipelines.yml", "repository": { "id": "'"$REPOSITORY_ID"'", "name": "'"$PROJECT"'", "type": "azureReposGit" } } }' \ https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/pipelines?api-version=6.0-preview.1 | jq . ## Create a new pipeline definition in new Azure repo cat <<'EOF'> azure-pipelines.yml trigger: none resources: pipelines: - pipeline: cloudaffaire source: 'mypipeline' trigger: true pool: mypool steps: - script: echo Hello world EOF ## Push the changes to new Azure Repo git add . git commit -m "update" git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push ## Update and push in source pipeline (mypipeline) cd .. cd $PROJECT cat <<'EOF'> azure-pipelines.yml trigger: - master pool: mypool steps: - script: echo This pipeline will trigger mynewpipeline EOF ## Push the changes to Azure Repo git add . git commit -m "update" git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push |
Azure Pipeline – Repository Resource:
If your pipeline has templates in another repository, or if you want to use multi-repo checkout with a repository that requires a service connection, you must let the system know about that repository. The repository keyword lets you specify an external repository.
Azure Pipeline – Repository Resource Syntax:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
## Repository Resource Syntax resources: repositories: - repository: string # identifier (A-Z, a-z, 0-9, and underscore) type: enum # see the following "Type" topic name: string # repository name (format depends on `type`) ref: string # ref name to use; defaults to 'refs/heads/main' endpoint: string # name of the service connection to use (for types that aren't Azure Repos) trigger: # CI trigger for this repository, no CI trigger if skipped (only works for Azure Repos) branches: include: [ string ] # branch names which will trigger a build exclude: [ string ] # branch names which will not tags: include: [ string ] # tag names which will trigger a build exclude: [ string ] # tag names which will not paths: include: [ string ] # file paths which must match to trigger a build exclude: [ string ] # file paths which will not trigger a build |
Azure Pipeline – Repository Resource Example:
Create a service connection to GitHub as shown in this link.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
## Update pipeline definition cat <<'EOF'> azure-pipelines.yml pool: mypool resources: repositories: - repository: MyGitHubRepo type: github endpoint: MyGitHub name: CloudAffaire/sample_data ref: master steps: - checkout: MyGitHubRepo displayName: clone_repo - bash: ls -al displayName: list_files EOF ## Push the changes to Azure Repo git add . git commit -m "update2" git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push |
Azure Pipeline – Container Resource:
Container jobs let you isolate your tools and dependencies inside a container. The agent launches an instance of your specified container then runs steps inside it. The container keyword lets you specify your container images.
Azure Pipeline – Container Resource Syntax:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
## Container Resource Syntax resources: containers: - container: string # identifier (A-Z, a-z, 0-9, and underscore) image: string # container image name options: string # arguments to pass to container at startup endpoint: string # reference to a service connection for the private registry env: { string: string } # list of environment variables to add ports: [ string ] # ports to expose on the container volumes: [ string ] # volumes to mount on the container mapDockerSocket: bool # whether to map in the Docker daemon socket; defaults to true mountReadOnly: # volumes to mount read-only - all default to false externals: boolean # components required to talk to the agent tasks: boolean # tasks required by the job tools: boolean # installable tools like Python and Ruby work: boolean # the work directory |
Azure Pipeline – Container Resource Example:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
## ------------------ ## Container Resource ## ------------------ ## Update pipeline definition cat <<'EOF'> azure-pipelines.yml resources: containers: - container: u14 image: ubuntu:14.04 - container: u16 image: ubuntu:16.04 - container: u18 image: ubuntu:18.04 jobs: - job: RunInContainer pool: vmImage: 'ubuntu-18.04' strategy: matrix: ubuntu14: containerResource: u14 ubuntu16: containerResource: u16 ubuntu18: containerResource: u18 container: $[ variables['containerResource'] ] steps: - script: printenv EOF ## Push the changes to Azure Repo git add . git commit -m "update3" git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push |
Azure Pipeline – Package Resource:
You can consume NuGet and npm GitHub packages as a resource in YAML pipelines. When specifying package resources, set the package as NuGet or npm.
Azure Pipeline – Package Resource Syntax:
|
1 2 3 4 5 6 7 8 9 |
## Package Resource Synatx resources: packages: - package: myPackageAlias # alias for the package resource type: Npm # type of the package NuGet/npm connection: GitHubConnectionName # Github service connection with the PAT type name: nugetTest/nodeapp # <Repository>/<Name of the package> version: 1.0.1 # Version of the packge to consume; Optional; Defaults to latest trigger: true # To enable automated triggers (true/false); Optional; Defaults to no triggers |
Azure Pipeline – Package Resource Example:
Create a service connection to GitHub package as shown in this link.
You need to have a GitHub package, create one if don’t have using this guide.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
## ---------------- ## Package Resource ## ---------------- ## Update pipeline definition cat <<'EOF'> azure-pipelines.yml resources: packages: - package: MyPackage type: npm connection: MyGitHub name: CloudAffaire/MyPackage version: 1.0.0 trigger: true pool: vmImage: 'ubuntu-latest' steps: - getPackage: MyPackage EOF ## Push the changes to Azure Repo git add . git commit -m "update4" git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push |
Clean up:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
## -------- ## Clean up ## -------- ## Delete the Azure DevOps project and local repository PROJECT_ID=$(curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/_apis/projects?api-version=6.0 \ | jq -r '.value[] | select(.name == "'"$PROJECT"'") | .id') && curl -s --request DELETE \ -u $USER:$PAT \ --header "Content-Type: application/json" \ https://dev.azure.com/$ORGANIZATION/_apis/projects/$PROJECT_ID?api-version=6.0 cd .. rm -rf $PROJECT rm -rf $REPOSITORY |
Hope you have enjoyed this article. To know more about Azure DevOps, please refer below official documentation
https://docs.microsoft.com/en-us/azure/devops/?view=azure-devops