Azure DevOps Pipeline Tutorial — Part 2: How to make an Azure DevOps template
In part two of the Azure DevOps Pipeline Tutorial, Penny Xuran Qian, explains how to convert this pipeline into a reusable template and add more features. Read part one here and read more below.
By: Penny Xuran Qian Machine Learning Engineer at ABN AMRO
Source: ABN AMRO Developer news
In the first article, we covered basic terminology to start a simple CI pipeline in Azure DevOps using YAML definition. We are satisfied with it and now we want more people to be able to reuse our CI pipeline.
Surely the copy+paste can be a solution, but it is also error-prone and will add difficulty for maintenance. One solution from Azure DevOps is to make the reusable content and logic into a template.
Templates function in two ways: insert reusable content with a template or you can use a template to control what is allowed in a pipeline. The second way is more useful for increasing security, this article is mainly around the first way.
In Part 1, we created a simple one job one step CI pipeline named azure-pipeline.yml, to use it as a template, we can simply make a few changes to it:
nametriggerstages and stageThe new code should look like this, name it as ci_template.yml
# Repo: MySpace/TemplateRepo
# File: ci_template.yml
jobs:
- job:
displayName: yamllint_checks
pool:
vmImage: 'ubuntu-latest'
steps:
- script: |
python -m pip install --upgrade pip
pip install yamllint
yamllint -d "{extends: relaxed, rules: {line-length: {max: 200}, new-line-at-end-of-file: disable, new-lines: disable}}" .
displayName: 'YAML Lint Checks'
We made our CI pipeline into a template on jobslevel, the reference level should also be the same as the template level, in this case, jobs. It is flexible to make templates at other levels, depending on your needs.
Now we can already refer to this template in the pipeline definition, create another azure-pipeline2.yml , in this YAML file, we can use our template by simply using keyword: template: template_path, this should give the same results as the pipeline in blog Part 1.
# Repo: MySpace/UserRepo
# File: azure-pipeline2.yml
name: cicd_ci
trigger:
branches:
include:
- master
stages:
- stage: CI_Checks
jobs:
- template: ci_template.yml
azure-pipeline2 hosted by GitHub
The template paths should be relative to the file that does the including. The nested hierarchy works the same way as the file system. For example:
steps:
- template: dir1/fileB.yml
- template: dir1/dir2/fileC.yml
The templates can also be put in a repo and then referred to it from different repos. One template can be used in multiple pipelines. Let’s take our above example YAML file, which is located in repo1 , now we create another repo named repo2 , we create a pipeline YAML file azure-pipeline3.yml :
# Repo: MySpace/UserRepo
# File: azure-pipeline3.yml
name: cicd_ci
trigger:
branches:
include:
- master
resources:
repositories:
- repository: ci_template
type: git
name: MySpace/Repo1
ref: 'refs/heads/master'
stages:
- stage: CI_Checks
jobs:
- template: ci_template.yml@ci_template
azure-pipeline3 hosted by GitHub
A resource is anything used by a pipeline that lives outside the pipeline. To use templates in another repository, the system should know about that repository first.
git type refers to Azure Repos Git repos. There are other types supported for example githubtype .Use the resources specification to provide the location of the core/original repo. When you refer to the core repo, use @ and the name you gave it in resources. In our case, we used template: ci_template.yml@ci_template
Let’s extend our template further, add one more step for Pylint check (a code analysis for Python), we name it as pylint_checks in this step, we include three commands to install Pylint packages, set up the system PATH, and execute the Pylint command.
# Repo: MySpace/TemplateRepo
# File: ci_template.yml
parameters:
- name: pylint_threshold
type: string
default: 8.0
- name: lint_folder
type: string
default: "mockcase-on-test"
jobs:
- job:
displayName: yamllint_checks
pool:
vmImage: 'ubuntu-latest'
steps:
- script: |
python -m pip install --upgrade pip
pip install yamllint
yamllint -d "{extends: relaxed, rules: {line-length: {max: 200}, new-line-at-end-of-file: disable, new-lines: disable}}" .
displayName: 'YAML Lint Checks'
- job:
displayName: pylint_checks
pool:
vmImage: 'ubuntu-latest'
steps:
- script: |
pip install pylint-fail-under pylint-junit
export PATH="/home/pipeuser/.local/bin":$PATH
pylint-fail-under --fail_under ${{parameters.pylint_threshold}} --output-format=colorized ${{ parameters.lint_folder}}
displayName: 'Pylint fail under ${{ parameters.pylint_threshold }}'
ci_template2 hosted by GitHub
To enable users to pass values to the pipeline and have more control over the steps, we specified two runtime parameters in the above pipeline template, pylint_threshold and lint_folder. They are used to set the Pylint pass threshold and which folder to run the Pylint scan individually.
Parameters can be viewed as placeholders, they are only available at template parsing time, and they can’t be changed by a pipeline while it’s running. Pipeline parameters are expanded just before the pipeline runs so the values surrounded by ${{ }} are replaced with parameter values.
From the user upfront, the parameters’ values can be rewritten with key-value pairs, as shown in the example below:
# Repo: MySpace/UserRepo
# File: azure-pipeline3.yml
name: cicd_ci
variables:
- name: PylintThreshold
value: 8.0
- name: LintFolder
value: "folder_dir"
trigger:
branches:
include:
- master
resources:
repositories:
- repository: ci_template
type: git
name: MySpace/Repo1
ref: 'refs/heads/master'
stages:
- stage: CI_Checks
jobs:
- template: ci_template.yml@ci_template
parameters:
pylint_threshold: $(PylintThreshold)
lint_folder: $(LintFolder)
azure-pipeline4 hosted by GitHub
variables can be a convenient way to collect information. Unlike parameters that have data types such as numbers and strings, all variables are stored as strings and are mutable. This means if you define a parameter type as number and pass the value from a variable, the pipeline would fail to parse because of unmatched data type.
The variable syntax varies depending on the usage, there are three different ways to reference variables: macro, template expression, and runtime expression. Macro is used to provide input for a task.
The secret variables should be avoided being set in the YAML file, as operating systems often log commands for the processes that they run, and you wouldn’t want the log to include a secret that passed in as an input.
There are several ways to set secret variables in the pipeline:
Read more articles like this here!
Lees meer van dit soort artikelen hier!
