deserializing stringified data

sometimes, upstream data you rely on might have fields with structured data that's been stringified, which is less than ideal.

in my experience, this usually occurs as a result of trying to glue disparate systems together: at some point in that process, someone decides to squeeze structured data through an interface that only supports strings, or that couldn't handle variations in the JSON otherwise, and then you, the downstream consumer, have to live with it.

For example, for some reason it's not uncommon to encounter Ethereum contract ABI json as a string.

doing it the low-friction way

say we have this incoming JSON:

{"data": "{\"key\": 1, \"enabled\": false}"}

as you can see, the data field's value is a stringified JSON object, which in its un-stringified form would be described by this Rust struct:

#[derive(Deserialize)]
struct Metadata {
    key: i32,
    enabled: bool,
}

the easiest way to work around this if it comes up in day to day development is just to accept a secondary parsing step, where String is the initial type of the stringified field, as shown in deser_two_step here:

#[derive(Deserialize)]
#[serde(bound(deserialize = "T: serde::de::Deserialize<'de>"))]
pub(crate) struct Outer<T> {
    data: T,
}

fn deser_two_step<T: DeserializeOwned>(json: &str) -> Result<Outer<T>, serde_json::Error> {
    let outer: Outer<String> = serde_json::from_str(json)?;
    let data = serde_json::from_str(&outer.data)?;
    Ok(Outer { data })
}

let json_str = r#"{"data": "{\"key\": 1, \"enabled\": false}"}"#;

let parsed: Outer<Metadata> = deser_two_step::<Metadata>(json_str).unwrap();
assert_eq!(parsed.data.key, 1);
assert_eq!(parsed.data.enabled, false);

this is a lot like the approach discussed in the previous filter parsing JSON post - parse first to something infallible, then do your secondary parse. it works OK, but you allocate & parse twice for that field - once to String, then to your T. It also is less than ideal in terms of developer experience.

we could do better!

custom deserializers

by implementing a custom serde deserializer, we can customize the deserialization of a given field. we can use this to move the parsing logic out of our consuming code!

first draft

what would be the simplest custom deserialize_with function we could implement that works for stringified JSON?

pub fn deser_stringified_json<'de, D, T>(deserializer: D) -> Result<T, D::Error>
where
    D: Deserializer<'de>,
    T: DeserializeOwned,
{
    let s: String = Deserialize::deserialize(deserializer)?;
    serde_json::from_str(&s).map_err(serde::de::Error::custom)
}

which can then be used like this, by annotating the data field:

let json = r#"{"data": "{\"key\": 1, \"enabled\": false}"}"#;

#[derive(Deserialize)]
struct Example<T: DeserializeOwned> {
    #[serde(deserialize_with = "deser_stringified_json")]
    data: T,
}

let parsed: Example<Metadata> = serde_json::from_str(json).unwrap();
assert_eq!(parsed.data.key, 1);
assert_eq!(parsed.data.enabled, false);

OK, that's pretty simple, and it works! check out this playground to run the code.

this has improved our developer experience, as now we don't have to directly deal with the stringified form in our consuming code, and the parsing is one step. great, right ?

but we still have one issue remaining, the same as we had with the naive approach of 2-step parsing: double allocation.

customizing the Visitor

I asked if there was a better way of doing this, and got a great answer: customize the entire deserializer!

how does that change our deser_stringified_json function?

pub fn deser_stringified_json<'de, D, T>(deserializer: D) -> Result<T, D::Error>
where
    D: Deserializer<'de>,
    T: DeserializeOwned,
{
    struct StringifiedJsonVisitor<T>(PhantomData<T>);

    impl<'de, T> Visitor<'de> for StringifiedJsonVisitor<T>
    where
        T: DeserializeOwned,
    {
        type Value = T;

        fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
            formatter.write_str("a string containing JSON data")
        }

        fn visit_str<E>(self, value: &str) -> Result<Self::Value, E>
        where
            E: serde::de::Error,
        {
            serde_json::from_str(value).map_err(E::custom)
        }

        fn visit_borrowed_str<E>(self, value: &'de str) -> Result<Self::Value, E>
        where
            E: serde::de::Error,
        {
            serde_json::from_str(value).map_err(E::custom)
        }
    }

    deserializer.deserialize_str(StringifiedJsonVisitor(PhantomData))
}

this implementation of serde's Visitor<'de> makes it so that JSON is parsed from the stringified form without any intermediate copying: it parses directly from a &str that represents some JSON field contents.

it's used exactly the same as the previous implementation, but now doesn't cause extra allocations. check out the playground to run the code.

so, mission accomplished ? almost!

once the pattern is established for one format, it's pretty trivial to make it work generically.

making it generic: deser-stringified

deser-stringified is a new crate that implements this utility in a way that's generic across data formats. initially, it supports JSON, YAML, and TOML. hopefully, it makes implementing the techniques discussed above not needed.

this is a personal itch i wanted to scratch, as i've run into this stringified parsing problem multiple times. please see the repo / crate for more information on it if you're interested.

caio!