Set Member Parsing
A fairly common scenario in writing web servers is the need to check that some parameter value in a user request is a member of a set - a list of datasets, a machine learning model name, etc. We do this because some operation further inside the server needs to operate with the assumption the values it's using won't cause bad behavior.
let's look at some ways of doing this, and try to arrive at a better one, following the principle "parse, don't validate". We'll use a real example from my work: there are unsigned 64-bit integer keys known as chain IDs, which uniquely identify a given blockchain. We use these as parameters in some HTTP routes, and we only support a subset of them, so we must return an error to the request if one we don't support is requested.
compiling code for this post is available on my GitHub.
easy mode: compile-time sets
If we know the full set of possible valid values at compile time, implementing this behavior efficiently is mostly obvious: use an enum. We'll also use the TryFromPrimitive derive from num_trait to provide us with an easy way to convert from a raw u64 value.
#[derive(Debug, TryFromPrimitive)]
#[repr(u64)]
enum SupportedChainIdEnum {
Ethereum = 1,
Base = 8543
}
this is simplest with integers, since we can use #[repr(u64)] to make sure our enum is represented as a u64, but it's possible with strings as well, using a crate like strum.
how should we use this enum in our route ? a common strategy might be:
we have an integer input in the query string, which we will validate as being in the enum at the beginning of the route
and implementing that looks like this. note: examples use poem_openapi's way of declaring route handlers.
here, the Query(chain_id): Query<u64> means that we expect a u64 argument in place of the :chain_id placeholder in the route path.
#[oai(path = "/:chain_id/supported", method = "get")]
async fn chain_supported(
&self,
Query(chain_id): Query<u64>,
) -> poem::Result<()> {
let id = SupportedChainIdEnum::try_from(chain_id)
.map_err(|_| poem::Error::from_string(
format!("unsupported chain ID: {}", chain_id),
StatusCode::BAD_REQUEST,
))?;
inner_handler(id) // do your actual route handler
}
this is analagous to how you would commonly do it if you are writing in a language like Typescript:
import type { Request, Response } from 'express';
enum SupportedChainId {
Ethereum = 1,
Base = 8543,
}
const validChainIds = new Set<number>(
Object.values(SupportedChainId).filter((v) => typeof v === 'number') as number[]
);
async function chainSupported(
req: Request<{}, any, any, { chain_id: string }>,
res: Response
): Promise<Response> {
const chainId = Number(req.query.chain_id);
if (!validChainIds.has(chainId)) {
return res.status(400).json({ error: `unsupported chain ID: ${chainId}` });
}
return await innerHandler(res, chainId);
}
this is pretty decent - our validation of the input parameter is just one line of boilerplate at the start of the route, we get a solid type out of it, hard to complain. however, it's annoying to write that line over and over if we have a lot of routes using it. can we improve this?
what if our enum could be understood directly in the route? it can be!
async fn chain_supported(
&self,
// if a request passes an unsupported u64, it's now an error parsing this enum,
// and the HTTP framework will return an appriate error to the requester
Query(chain_id): Query<SupportedChainId>,
) -> poem::Result<()> {
inner_handler(chain_id) // do your actual route handler
}
Now this is nice, right? It's impossible for any of our route handling code to get an invalid value for chain_id, bad requests get appropriate errors returned, and there's no validation boilerplate in the routes.
Unfortunately, there are a lot of sets we'd like to check membership of that are not possible to know at compile time, so this enum approach can't be extended to those scenarios. How can we make a type that will provide us the same safety and ergonomics, but work with runtime-only sets? We'll come back to this in a moment.
Using a stronger type directly in the route declaration is possible thanks to our http framework implementing support for parsing common data types by default. How does the framework know how to parse a given type from an http request anyway?
parameter parsing
All values parsed from an HTTP request must be parsed from a string representing some part of the request.
So, poem_openapi provides a trait ParseFromParameter with a single required method that looks exactly like you might expect: take in &str, and fallibly return Self.
pub trait ParseFromParameter: Sized + Type {
fn parse_from_parameter(value: &str) -> ParseResult<Self>;
}
ParseResult is a result type from poem that will return a 400 error to the requester when thrown.
we can implement this trait for our own types, the data types supported by default (like our u64 enum) aren't special.
could a custom type get us the goals i mentioned above for runtime sets? probably!
runtime sets
let's say that instead of our SupportedChainId enum, our server loads a file supported_chains.json at startup from a config file. this file provides similar information to our enum, with the following contents:
[1, 10, 100, 8543, 42161]
what does that setup step look like in Rust?
setup
we'll use a LazyLock, which was recently stabilized in Rust 1.80, to initialize a global value. we rely on serde_json to parse the json array into either a HashSet or a BTreeSet, depending on your use case. You can easily extend this approach to further use cases by using a Map instead of a Set.
use std::{collections::BTreeSet, path::Path};
// BTreeSet is faster than HashSet with small number of int keys
pub static SUPPORTED_CHAIN_IDS: LazyLock<BTreeSet<u64>> =
LazyLock::new(|| {
let path = Path::new("./chain_ids.json");
serde_json::from_reader(
File::open(path).expect("can't open ./chain_ids.json"),
)
.expect("failed to parse chain IDs file")
});
there's also the slightly more complex case, of a server that needs to re-load the configuration every so often without restarting itself. we're going to ignore this case, as multithreading synchronization practices aren't the point of this post.
now that we're doing with setup, we can return to our original problem.
in-route comparison
before get to writing the nice version of this, let's look at what the most obvious version might be, one that parallels the first implementation we did with an enum.
async fn chain_supported(
&self,
Query(chain_id): Query<u64>,
) -> poem::Result<()> {
match SUPPORTED_CHAIN_IDS.contains(&number) {
true => {
// call actual route handler here
inner_handler(SupportedChainId(number))
}
false => Err(poem::Error::from_string(
format!("unsupported chain ID: {}", chain_id),
StatusCode::BAD_REQUEST,
)),
}
}
the route declares the primitive type, then we validate the value. let's see how we could parse the value instead.
encoding support in a type
let's use everything we just discussed to solve our problem. first, our struct definition.
#[derive(
Serialize, Deserialize, PartialEq, Eq, poem_openapi::NewType,
)]
#[oai(from_parameter = false)]
#[repr(transparent)]
/// a currently supported chain ID
pub struct SupportedChainId(pub u64);
note: this type definition isn't strict about preventing the type from being constructed with arbitrary numbers by consuming code, so it's not %100 airtight. this sort of strictness isn't the point of the post, but it's something to think about if you use this sort of code in production.
the NewType derive tells poem to get the type/parsing implementation for SupportedChainId by looking at u64, which it is a newtype of.
the #[oai(from_parameter = false)] prevents the NewType derive from providing u64's implementation of ParseFromParameter, because customizing that impl is the key to this whole thing. let's do that now.
impl ParseFromParameter for SupportedChainId {
fn parse_from_parameter(value: &str) -> ParseResult<Self> {
// first, parse to plain u64, propagating any error
let number = value.parse::<u64>()
.map_err(|_| ParseError::custom("must be a uint64"))?;
// then, use the static set we setup earlier to check support
match SUPPORTED_CHAIN_IDS.contains(&number) {
// wrapping in our type encodes that we've checked the number
true => Ok(SupportedChainId(number)),
// represent lack of support as a parsing error
false => Err(ParseError::custom(
format!("unsupported chain ID: {}", number)
)),
}
}
}
you may notice that the parsing implementation, but also the errors that should be returned to the requester in case parsing goes wrong.
What does our route definition look like if we use this new type? Exactly the same as when using an enum - only the definition of SupportedChainId has changed.
async fn chain_supported(
&self,
Query(chain_id): Query<SupportedChainId>,
) -> poem::Result<()> {
inner_handler(chain_id) // do your actual route handler
}
now, we've arrived at our goals:
- our route code doesn't validate the ID: we parse it as part of a type, directly in the route definition
- it works with sets defined at runtime in the same way as at compile time
- it's efficient enough to use frequently
caio!
i intend to write more on the topic of strong types at serialization boundaries, since i think it's a very practical way to embrace "parse, don't validate". thanks for reading.