Notes on using jq (JSON Query Language)
Links
- website: https://jqlang.org/
- GITHUB: https://github.com/jqlang/jq
- Playground: https://github.com/jqlang/jq - use this to try JQ online
- Documentation: https://jqlang.org/manual/#pipe
- YouTube tutorial - https://www.youtube.com/watch?v=m9dhrq9iRHA
Basics
Get some data and format it.
JQ_DATA=$HOME/jq.data
curl 'https://api.github.com/repos/jqlang/jq/commits?per_page=5' > $JQ_DATA
cat $JQ_DATA | jq '[.[] | {message: .commit.message, name: .commit.committer.name}]' | head -30
In General the syntax is
jq ‘{ “key1”: <<jq filter>>, “key2”: <<jq filter>> }’
.[]- Loop through all of the elements in the incomming array.[2:4]- selecting the 3rd and 4th objects in an array of objects.[ .[2:4] ]- select the 3rd and 4th objects in an array of objects and return them as an array{
echo '[{"a":"a"}, {"b":"b"}, {"c":"c"}, {"d":"d"}, {"e":"e"}, {"f":"f"} ]' | jq '.[2,4]'
{
"c": "c"
}
{
"e": "e"
}
If you want them returned in a list, wrap the jq output in an array [] as shown next
echo '[{"a":"a"}, {"b":"b"}, {"c":"c"}, {"d":"d"}, {"e":"e"}, {"f":"f"} ]' | jq '[ .[2,4] ]'
[
{
"c": "c"
},
{
"e": "e"
}
]
Filtering
The function select() can be used to filter/select only items you are interested in.
echo '[{"a":"a"}, {"a":"b"}, {"a":"c"}, {"a":"b"} ]' | jq '[ .[] | select(.a == "b") ]'
[
{
"a": "b"
},
{
"a": "b"
}
]
Additional filtering can be done using a | pipe symbol
echo '[{"a":"a"}, {"a":"b"}, {"a":"c"}, {"a":"b"} ]' | jq '[ .[] | select(.a == "b") ] | unique'
[
{
"a": "b"
}
]
Understanding map and select
(Q) What does map do? (A) It’s another syntax for saying to return the result as an array of objects. The value is when you have multiple arrays being processed. The following two examples are equivalent but the first uses map() syntax and the second uses the array syntax ( [ .[] | stuff ] ).
cat << EOF > t.sh
#!/bin/bash
curl -s https://api.github.com/repos/jqlang/jq/issues'?'per_page=50 | \
jq 'map({ title, number, labels_count: .labels | length, title_length: .title | length, labels: .labels | sort_by(.name) }) | map(select(.title_length < 30))' > t1.a
curl -s 'https://api.github.com/repos/jqlang/jq/issues?per_page=50' | \
jq 'map(
{
title,
number,
labels_count: (.labels | length),
title_length: (.title | length),
labels: (.labels | sort_by(.name))
}
)
| map(select(.title_length < 30))' > t1.b
curl -s https://api.github.com/repos/jqlang/jq/issues'?'per_page=50 | \
jq '[ .[] | { title, number, labels_count: .labels | length, title_length: .title | length, labels: .labels | sort_by(.name) } | select(.title_length < 30) ]' > t2.a
curl -s 'https://api.github.com/repos/jqlang/jq/issues?per_page=50' | \
jq \
'[ .[]
| {
title,
number,
labels_count: (.labels | length),
title_length: (.title | length),
labels: (.labels | sort_by(.name))
}
| select(.title_length < 30)
]' > t2.b
wc t1.{a,b} t2.{a,b}
EOF
# To run this script use the bash command as follows
bash t.sh
130 256 2688 t1.a
130 256 2688 t1.b
130 256 2688 t2.a
130 256 2688 t2.b
520 1024 10752 total
The above shows you that there are several ways to write the same script using map and the array syntax.
If you get error
t.sh: line 2: : command not found
Then copy/paste screwed up.
IMPORTANT NOTE: When copying and pasting this examples, extra spaces were added which prevented the script from running. Change all “\ “ (notice space after the backslash) to just a slash.
You can use multiple items in a single select or chain them together.
jq '.[] | select(.age > 25 and .active == true)'
jq '.[] | select(.age > 25) | select(.active == true)'
Special characters
See URL: https://jqlang.org/manual/#object-identifier-index
If the key contains special characters or starts with a digit, you need to surround it with double quotes
like this: .”foo$”, or else .[“foo$”].
Transforming regular text into JSON
The jq command has options that allow you to take regular input and output it in JSON format and visa-versa.
Transform an array of items without a uniq key to an array that has a unique key
echo '[{"a":"a"}, {"b":"b"}, {"c":"c"} ]' | jq 'to_entries'
[
{
"key": 0,
"value": {
"a": "a"
}
},
{
"key": 1,
"value": {
"b": "b"
}
},
{
"key": 2,
"value": {
"c": "c"
}
}
]
To get the 2nd and 3rd entry (key:1 and 2), you can use map(select(.key == 1 or .key == 2)) | map(.value)
echo '[{"a":"a"}, {"b":"b"}, {"c":"c"} ]' | jq 'to_entries | map(select(.key == 1 or .key == 2)) | map(.value)'
If/then/else
Jq also supports if/then/else. See the jq playground URL: https://play.jqlang.org/?q=if%20.%20%3D%3D%200%20then%0A%20%20%22zero%22%0Aelif%20.%20%3D%3D%201%20then%0A%20%20%22one%22%0Aelse%0A%20%20%22many%22%0Aend&j=2
which does this
echo 2 | jq 'if . == 0 then
"zero"
elif . == 1 then
"one"
else
"many"
end'
The output in this example is many.