"filter parsing" JSON

"filter parsing" is the name i use to refer to the practice of fallibly parsing some type T from (bytes/json/etc) to Option<T>, rather than Result<T>. sometimes, you just don't care what went wrong, and only care about the valid data.

when there's a parsing error, instead of bubbling the error, filter parsing swallows them (though may report them to logging/tracing) and returns None. here's an example:

dealing with irregular upstream data

we have an upstream data source that we query with batches of keys, and the struct we usually parse each item in the response array to is like this:

#[derive(Deserialize)]
pub struct ResponseElement {
    // other fields...
    pub decimals: u8
}

sometimes, the response includes a JSON object where decimals is out of range even for u16, which is a clearly unreasonable case - decimal places lose human meaning long before 255 of them, let alone 65,535. when we inspected the bad data, clearly other fields in the object looked whack as well, so we just want to throw these cases out.

crucially however, we don't want to throw away the other elements of the array that parse fine. if you parse a Vec<T> from a JSON array, using serde_json, and a single array member fails to parse, the whole parse throws an error.

so, we've been using the following lil' tool i made for this pattern in production at work, for at least a month.

FilterParsedJsonVec<T>

this struct wraps up the JSON parsing pattern, and makes it easy to get a Vec<T> of all the parseable values.

use std::collections::VecDeque;
use serde::{Deserialize, de::DeserializeOwned};

#[derive(Debug, Deserialize)]
#[serde(transparent)]
pub struct FilterParsedJsonVec<T: DeserializeOwned>(
    pub VecDeque<serde_json::Value>,
    #[serde(skip)] std::marker::PhantomData<T>,
);

it accomplishes that via a custom iterator that yields Option<T>:

impl<T: DeserializeOwned> Iterator for FilterParsedJsonVec<T> {
    type Item = Option<T>;

    fn next(&mut self) -> Option<Self::Item> {
        self.0.pop_front().map(|v| match serde_json::from_value(v) {
            Ok(val) => Some(val),
            Err(_) => None,
        })
    }
}

and this important bit of sweetener:

impl<T: DeserializeOwned> From<FilterParsedJsonVec<T>> for Vec<T> {
    fn from(val: FilterParsedJsonVec<T>) -> Self {
        val.into_iter().flatten().collect()
    }
}

using it

the idea is, you parse as FilterParsedJsonVec<T> first, because serde_json can't fail to parse a serde_json::Value for any valid JSON value. and then the iterator over it does the actual parsing to your T.

an example using integers for simplicity (in practice, this is meant to be used with more complex structs):

let json_arr: FilterParsedJsonVec<u16> = serde_json::from_str("[{}, 2, \"3\", 4]");
let valid_items: Vec<u16> = json_arr.into();
assert_eq!(valid_items, [2, 4]); 

that's the whole thing

it's really simple and it made dealing with weird parsing failures in incoming JSON arrays a breeze. being able to convert it to Vec<T> with just an .into() is a good devex in my experience.

the code for FilterParsedJsonVec<T> is available on my github.

it's not packaged as a crate yet, maybe if i mature it a bit i will do that.