How to filter yaml data using yq?
With the increase use of API and event driven architectures with dynamic configurations mostly defined in YAML or JSON, its essential for a DevOps or Software engineer to know how to parse yaml data.
What is YAML?
YAML or Yet Another Markup Language is a human-readable data-serialization language. It is commonly used for configuration files and in applications where data is being stored or transmitted. YAML targets many of the same communications applications as Extensible Markup Language (XML) but has a minimal syntax.
YAML was first proposed by Clark Evans in 2001. Originally YAML was said to mean Yet Another Markup Language, because it was released in an era that saw a proliferation of markup languages for presentation and connectivity (HTML, XML, SGML, etc). Its initial name was intended as a tongue-in-cheek reference to the technology landscape, referencing its purpose as a markup language with the yet another construct, but it was then repurposed as YAML Ain’t Markup Language, a recursive acronym, to distinguish its purpose as data-oriented, rather than document markup.
In the cloud and DevOps space most of the configuration files are defined YAML formats. YAML is used in everywhere from your pipeline definition or configuration management suite like Ansible or containerization suites like Kubernetes.
Sample YAML data:
|
1 2 3 4 5 6 7 8 9 |
--- id: '0001' name: debjeet dob: 10 Oct 1986 weight_kg: 80 skill: cloud: - provider: aws proficiency: intermediate |
What is yq?
yq a lightweight and portable command-line YAML processor written by Mike Farah. yq uses jq like syntax but works with yaml files as well as json. It doesn’t yet support everything jq does – but it does support the most common operations and functions, and more is being added continuously. yq is written in go – so you can download a dependency free binary for your platform and you are good to go! If you prefer there are a variety of package managers that can be used as well as docker, all listed below.
How to install yq?
YQ is available in almost all modern operating systems and can be installed fairly easy in couple of commands. In my view tools like yq, jq, git,python should be included as part of the OS distribution.
Installing YQ in Windows OS:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
## ---------- ## Windows OS ## ---------- ## Check if Chocolety is already installed choco --version #should return an output ## Install Chocolety if not already installed ## Open a command propmt as admin and execute below command @"%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "[System.Net.ServicePointManager]::SecurityProtocol = 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))" && SET "PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin" ## Install yq chocolatey install yq |
Installing YQ in MAC OS:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
## ------ ## MAC OS ## ------ ## Check if Homebrew already installed brew --version #should return an output ## Install Homebrew if not already installed /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" ## Install yq brew install yq |
Installing YQ in Linux OS:
|
1 2 3 4 5 6 7 8 9 10 11 |
## -------- ## Linux OS ## -------- BINARY="yq_linux_amd64" VERSION="v4.14.1" wget https://github.com/mikefarah/yq/releases/download/\ ${VERSION}/${BINARY}.tar.gz -O - | tar xz && sudo mv ${BINARY} /usr/bin/yq ## Check or get the latest version using below link ## https://github.com/mikefarah/yq/releases/latest |
How to parse YAML data using YQ?
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 |
## Create a sample yaml data cat << EOF > data.yaml --- company: Cloudaffaire compcontacts: phone: "+91 97482 39853" email: contact@cloudaffaire.com employees: - id: 101 name: Debjeet emails: - debjeettoni@gmail.com - cloudaffaire@gmail.com - id: 102 name: Chandrima Phone: - "+91 80177 02983" skills: - cloud: aws database: mysql scripting: python EOF cat data.yaml | yq e '.company' - ## Returns Cloudaffaire cat data.yaml | yq e '.compcontacts.email' - ## Returns contact@cloudaffaire.com cat data.yaml | yq e '.employees[0]' - ## Returns ## id: 101 ## name: Debjeet ## emails: ## - debjeettoni@gmail.com ## - cloudaffaire@gmail.com cat data.yaml | yq e '.employees[0].emails[1]' - ## Returns cloudaffaire@gmail.com cat data.yaml | yq e '.employees[1].skills[0].cloud' - ## Returns aws cat data.yaml | yq e -j '.employees[0]' - ## Returns ## { ## "id": 101, ## "name": "Debjeet", ## "emails": [ ## "debjeettoni@gmail.com", ## "cloudaffaire@gmail.com" ## ] ## } |
How to add new elements in a YAML file.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 |
## Update the yaml file cat << EOF > data.yaml --- employees: - name: Chandrima details: - id: 101 - skills: - aws EOF ## Add a new element in yaml file cat data.yaml | yq e '.employees[0].details[1].skills' - ## Returns aws yq e '.employees[0].details[1].skills += "python"' -i data.yaml cat data.yaml | yq e '.employees[0].details[1].skills' - ## Returns aws python cat data.yaml | yq e '.employees[0].details' - ## Returns ## - id: 101 ## - skills: ## - aws ## - python yq e '.employees[0].details += {"sex": "female"}' -i data.yaml cat data.yaml | yq e '.employees[0].details' - ## Returns ## - id: 101 ## - skills: ## - aws ## - python ## - sex: female |
How to update an existing element in YAML file?
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
## Update an existing element in yaml file cat data.yaml | yq e '.employees[0].name' - ## Returns Chandrima yq e '.employees[0].name = "Debjeet"' -i data.yaml cat data.yaml | yq e '.employees[0].name' - ## Returns Debjeet ## Update multiple elements yq e ' .employees[0].details[2].sex = "male" | .employees[0].details[1].skills[1] = "gcp" ' -i data.yaml cat data.yaml | yq e '.employees[0]' - ## Returns ## name: Debjeet ## details: ## - id: 101 ## - skills: ## - aws ## - gcp ## - sex: male |
How to delete an existing element in a YAML file?
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
## Delete an existing element in yaml file cat data.yaml | yq e '.employees[0].details' - ## Returns ## - id: 101 ## - skills: ## - aws ## - gcp ## - sex: male yq e 'del(.employees[0].details[2])' -i data.yaml cat data.yaml | yq e '.employees[0].details' - ## Returns ## - id: 101 ## - skills: ## - aws ## - gcp yq e 'del(.employees[0].details[1].skills[1])' -i data.yaml cat data.yaml | yq e '.employees[0].details' - ## Returns ## - id: 101 ## - skills: ## - aws |
How to check if a key or value exists in a YAML file?
|
1 2 3 4 5 6 7 8 9 10 11 12 |
## Check if a key or value exists in yaml file cat data.yaml | yq e 'has("employees")' - cat data.yaml | yq e '.employees[0] | has("name")' - cat data.yaml | yq e '.employees[0] | has("salary")' - cat data.yaml | yq e '.employees[0] | (.name == "Debjeet")' - cat data.yaml | yq e '.employees[0] | (.name == "Chandrima")' - ## List all root level keys in your yaml document cat data.yaml | yq e 'keys' - cat data.yaml | yq e '.employees[0] | keys' - cat data.yaml | yq e '.employees[0].details[] | keys' - |
Hope you have enjoyed this article. To get more details on YQ, please refer below official documentation.
https://mikefarah.gitbook.io/yq/