How to filter invalid jsonlines using jq
When using jsonlines data with jq, invalid lines can cause parsing errors, e.g.
e.g
jq: parse error: Expected separator between values at line 61755, column 1954
```plaintext {filename="e.g"}
To filter out invalid lines, you can use the following command:
```bash {filename="filter_invalid_jsonlines.sh"}
jq -R 'fromjson?'
```plaintext {filename="filter_invalid_jsonlines.sh"}
This will passthrough valid JSON lines and return `null` for invalid ones. You can then filter out the `null` values using `select`. The subsequent command needs to handle `null` values correctly, but it won't see any invalid lines.
### Example usage
```bash {filename="filter_and_process_jsonlines.sh"}
cat data.json | jq -R 'fromjson?' | jq -c '{name: .node?.name?}'
```plaintext {filename="filter_and_process_jsonlines.sh"}Check out similar posts by category:
JSON
If this post helped you, please consider buying me a coffee or donating via PayPal to support research & publishing of new posts on TechOverflow