Azure Pipeline Part 5 – Checkout
Hello Everyone
Welcome to CloudAffaire and this is Debjeet.
Today we are going to discuss how to clone multiple git repositories in Azure Pipelines using checkout block. By default, the repository hosting your azure pipeline config YAML file gets cloned when you execute a pipeline. But your pipeline may require source code hosted in different git repositories. In such scenarios, you can use checkout block inside a job to clone other git repositories.
Note: In order to use checkout block in your Azure pipeline, you first need to define repository resource in the pipeline config YAML file.
Next, we are going to clone a GitHub repository in our Azure Pipeline using checkout block.
Azure Pipeline Part 5 – Checkout
Prerequisites:
- One active Azure DevOps account
- Personal Access Token (PAT)
- A self-hosted agent registered to your Azure DevOps organization
- GitHub Personal Access Token (PAT)
Setup:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 |
## ----- ## Setup ## ----- ## Define variables ORGANIZATION="<EXISTING_AZURE_DEVOPS_ORGANIZATION_NAME>" USER=$(az account show | jq -r .user.name) PROJECT="<NEW_AZURE_DEVOPS_PROJECT_NAME>" PAT='<PERSONAL_ACCESS_TOKEN>' B64_PAT=$(printf "%s"":$PAT" | base64) ## Test if authentication working (List projects in your organization) curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/_apis/projects?api-version=6.0 | jq . ## Create a new Azure DevOps project curl -s --request POST \ -u $USER:$PAT \ --header "Content-Type: application/json" \ --data '{ "name": "'"$PROJECT"'", "description": "Demo Project For Azure Pipeline", "capabilities": { "versioncontrol": { "sourceControlType": "Git" }, "processTemplate": { "templateTypeId": "6b724908-ef14-45cf-84f8-768b5384da45" } } }' \ https://dev.azure.com/$ORGANIZATION/_apis/projects?api-version=6.0 | jq . ## Get project default repository details REPOSITORY_ID=$(curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/git/repositories?api-version=6.0 | jq -r .value[0].id) && curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/git/repositories/$REPOSITORY_ID?api-version=6.0 | jq . ## Create a new Azure Pipeline using REST API curl -s --request POST \ -u $USER:$PAT \ --header "Content-Type: application/json" \ --data '{ "folder": null, "name": "mypipeline", "configuration": { "type": "yaml", "path": "/azure-pipelines.yml", "repository": { "id": "'"$REPOSITORY_ID"'", "name": "'"$PROJECT"'", "type": "azureReposGit" } } }' \ https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/pipelines?api-version=6.0-preview.1 | jq . ## Clone project default repository REPO_HTTPS_CLONE_URL=$(curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/$PROJECT/_apis/git/repositories?api-version=6.0 | jq -r .value[0].webUrl ) && B64_PAT=$(printf "%s"":$PAT" | base64) && git -c http.extraHeader="Authorization: Basic ${B64_PAT}" \ clone $REPO_HTTPS_CLONE_URL cd $PROJECT |
Azure Pipeline – Checkout:
By using multiple checkout steps in your pipeline, you can fetch and check out other repositories in addition to the one you use to store your YAML pipeline.
Azure Pipeline – Checkout syntax:
|
1 2 3 4 5 6 7 8 9 |
## Checkout - Syntax steps: - checkout: self | none | repository name # self represents the repo where the initial Pipelines YAML file was found clean: boolean # if true, run `execute git clean -ffdx && git reset --hard HEAD` before fetching fetchDepth: number # the depth of commits to ask Git to fetch; defaults to no limit lfs: boolean # whether to download Git-LFS files; defaults to false submodules: true | recursive # set to 'true' for a single level of submodules or 'recursive' to get submodules of submodules; defaults to not checking out submodules path: string # path to check out source code, relative to the agent's build directory (e.g. \_work\1); defaults to a directory called `s` persistCredentials: boolean # if 'true', leave the OAuth token in the Git config after the initial fetch; defaults to false |
Azure Pipeline – Repository Resource syntax:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
## Repository Resource - syntax resources: repositories: - repository: string # identifier (A-Z, a-z, 0-9, and underscore) type: enum # see the following "Type" topic name: string # repository name (format depends on `type`) ref: string # ref name to use; defaults to 'refs/heads/main' endpoint: string # name of the service connection to use (for types that aren't Azure Repos) trigger: # CI trigger for this repository, no CI trigger if skipped (only works for Azure Repos) branches: include: [ string ] # branch names which trigger a build exclude: [ string ] # branch names which won't tags: include: [ string ] # tag names which trigger a build exclude: [ string ] # tag names which won't paths: include: [ string ] # file paths which must match to trigger a build exclude: [ string ] # file paths which won't trigger a build |
Since we are using a GitHub repository, we can use service connection feature to define the GitHub endpoint in Azure DevOps project.
Create a service connection for GitHub:
Step 1: Log in to Azure DevOps portal and navigate to the project where your pipeline is hosted.
Step 2: Navigate to your Azure DevOps project setting page => Pipelines => Service connections and click “Create service connection”.
Step 3: Select “GitHub” and click “Next”.
Step 4: Provide personal access token of your GitHub account, service connection name and click “Verify and save”.
Note: The service connection name (in our case “MyGitHub”) will be used in the repository resource endpoint to refer this GitHub connection.
Now we are ready to create pipeline config YAML file and clone the GitHub repository using this service connection.
Step 5: Create the pipeline config YAML file and push the changes to Azure repos.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
## Create pipeline definition cat <<'EOF'> azure-pipelines.yml pool: mypool resources: repositories: - repository: MyGitHubRepo type: github endpoint: MyGitHub ## referenced the service connection name: CloudAffaire/sample_data ##GitHub user_name/repo_name ref: master ##branch or tag name steps: - checkout: MyGitHubRepo ## referenced the repository resource displayName: clone_repo - bash: ls -al displayName: list_files EOF ## Push the changes to Azure Repo git add . git commit -m "update" git -c http.extraHeader="Authorization: Basic ${B64_PAT}" push |
You might need to provide the access to your pipeline to use the GitHub service endpoint.
Observe, we have both the repository checked out in our pipeline.
Clean up:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
## -------- ## Clean up ## -------- ## Delete the Azure DevOps project and local repository PROJECT_ID=$(curl -s -u $USER:$PAT \ https://dev.azure.com/$ORGANIZATION/_apis/projects?api-version=6.0 \ | jq -r '.value[] | select(.name == "'"$PROJECT"'") | .id') && curl -s --request DELETE \ -u $USER:$PAT \ --header "Content-Type: application/json" \ https://dev.azure.com/$ORGANIZATION/_apis/projects/$PROJECT_ID?api-version=6.0 cd .. rm -rf $PROJECT |
Hope you have enjoyed this article. To know more about Azure DevOps, please refer below official documentation
https://docs.microsoft.com/en-us/azure/devops/?view=azure-devops