You are currently viewing How to filter json data using jq?

How to filter json data using jq?

How to filter json data using jq?

With increase use of API and event driven architectures which mostly use JSON as payload, its essential for a DevOps or Software engineer to know how to filter json data.

What is JSON?

JSON or JavaScript Object Notation is a text-based standard used to store and transmit data objects between applications, clients and servers in the web. A JSON data object consist of key-value pairs and arrays and is written is human readable format. JSON was first conceptualized and introduced to the world by Douglas Crockford around the year 2000 and since then it has slowly become the standard for communication between web applications and event driven architectures and has replaced XML like markup languages due to its simplicity to use. In the cloud and DevOps space most of the application now uses JSON or YAML formats.

Sample JSON data:

What is jq?

JQ is a program developed to filter JSON data. You can consider jq as sed, awk, grep like program but designed specifically for filtering JSON data. JQ supports filters, conditions, operators and other language features. While creation and understanding of JSON data is very simple due to its human readable format, JSON data parsing is still a very complicated task to achieve if done manually. Instead, you can use programs like jq to parse JSON data in minutes.

How to install jq?

JQ is available in almost all modern operating systems and can be installed fairly easy in couple of commands. In my view tools like jq, git should be included as part of the OS distribution.

Installing JQ in Windows OS:

Installing JQ in MAC OS:

Installing JQ in Ubuntu or Debian OS:

Installing JQ in REDHAT or CentOS OS:

Installing JQ in Fedora OS:

Installing JQ in AWS EC2 instance with Amazon Linux OS:

Once you have installed JQ in your OS, validate the installation by running jq –version command.

How to parse JSON data using JQ?

Next, we will start with JSON data parsing and explore different concepts of JQ using some examples.

Basic filtering of JSON data using jq:

Identity: .

A . (dot) in jq represent the entire input without any filters. JQ command followed by a dot will output the input as it is with some formatting to make it pretty. You can use jq . command to format the output of a curl command.

Object Identifier-Index: .element,.element.sub-element

You can get the value of an JSON key using the . (dot) notation. Like path you have to give full key location in case the key is a sub-key of another key with dot notation. Let me explain with an example.

In the above example, since “name” is a top level element, we can get the value of “name” using .name but “cloud” is not a top level element but is under top level element “skills”, hence we cannot get the value of “cloud” element using .cloud instead we have to descend the path like .skills.cloud to get the value “aws”.

Array Index: .[<array_index>]

If you have an array in your json data, you can retrieve the data inside the array element using array index.

Comma: , and Pipe |

You can select multiple elements from your JSON data using comma and pipe in jq.

Array construction: []

You can create an array on the filtered output of jq using [].

Object construction {}

You can create a dictionary or hash from the filtered output in jq.

Recursive Descent: ..

You can recursively decent to the required element using two dot operators in jq.

Built-in operators :

JQ supports all arithmetic operators like addition, substation, multiplication and division.

Inbuilt functions:

JQ has a complete set of in-built function to add, remove, sort, update and filter JSON data. Lets explore some of them with examples.

Check if a key exists in your json data

Get all the elements of json data by element types

Sort json data by key or value

Get json element with maximum and minimum vales.

Remove duplicate data from JSON output

Check if a key has certain value in json

Hope you have enjoyed this article. To get more details on JQ, please refer below official documentation.

https://stedolan.github.io/jq/manual/