Schemas

std::schema describes the shape of a value once, and decodes it from anywhere. The description does not know where the bytes came from, which is the whole of the library and the reason it is worth a page.

Everything else in this section is reached through the generated std::schema API. This page is the model behind it.

The separation it is built on

A reader per source reads well, and none of it is reusable. A reader that both names a field and fetches it says why:

pub fn string(name: String) -> Validated<String, ConfigError> with { env: Env }

string(name) is not this field is text. It is go to the environment, fetch this variable, and give me the text or a reason — the shape and the reading are one function. A JSON body then needs its own vocabulary, and so does a CLI argument, and so does a database row, and none of them can share a description with the others.

A Schema<A> splits that in half. It describes an A; a [Raw] is whatever some source produced; decode is one function over the pair.

let decoded = Schema::decode(schema(), incoming);

There is no ! on that call: decode answers a Validated<Settings, Rejection> rather than raising, which is what lets it report every bad field at once. What a failure is below is how you read one.

Where incoming came from is the caller’s business. The same Schema<Settings> reads the environment, a request body and a test fixture.

A schema is two halves in one record

pub type Schema<A> = {
  shape: Shape,
  read: (List<Segment>, Raw) -> Validated<A, Rejection>,
};

Both halves are load-bearing, and neither would do on its own.

The closure is what makes schema.decode(value) an ordinary call, and what lets schemas be combined — optional(list(int())) is three closures wrapped around each other.

The shape is untyped, deliberately: it has no type parameter, which is what lets a record’s fields — all of different types — sit in one List and be walked. That is how a deployment can ask which keys a configuration needs without starting the program:

print("it needs: ${Shape::keys(schema().shape)}");

A record of two closures would have lost the shape. A bare description tree would have lost the decoder. the schema design note has the argument in full.

The vocabulary

Six primitives, the combinators that wrap another schema, and the assemblers. Every constructor is named after the type it answers.


`string()`	a `String`; a number is not text, and is refused
`int()`, `float()`	an `Int` or a `Float`, parsed from the token
`decimal()`	a `Decimal`, parsed from the token, or from text, because money travels as a string on most wires
`bool()`	a `Bool`
`any()`	the `Raw` as it arrived, for a field whose shape is somebody else’s business
`optional(s)`	an `Option<A>`; absent or `null` is `None`, present and wrong is still an error
`nullable(s)`	an `Option<A>` that must be present and may be `null`
`default(s, value)`	an `A`, with `value` when the field is absent; `null` is still an error
`list(s)`	a `List<A>`, indexing each element into the error path
`dict(s)`	a `Dict<String, A>`, for a record whose keys are data
`refine(s, must, holds)`	the same `A`, rejected unless `holds`; `must` is the rest of the sentence after `must be` — `"between 1 and 65535"` reads as port must be between 1 and 65535
`between`, `at_least`, `at_most`, `min_length`, `max_length`, `min_items`, `max_items`, `non_empty`, `one_of`	the same, with a rule that carries its bounds, so a rendered document can carry them too
`secret(s)`	a `Redacted<A>`, and a failure inside it does not quote what it saw
`key(wire, s)`	the same `A`, read under a key that is not the field’s name
`renamed(s, wire, field)`	a record schema with one key spelled differently on the wire
`Schema::map`, `Schema::try_map`	something made from the value, or something it may fail to become
`Schema::closed`	the same record, refusing a key it did not declare
`Schema::described`	the same schema, with a sentence for a document to carry
`Schema::cases`	a variant: a bare string for a payload-free case, an object tagged with `type` for the rest
`Schema::lazy`	a schema built when first read, so a type may mention itself
`struct({ .. })`	a record: a literal whose fields are schemas, typed by the compiler as the record they decode
`Schema::record` over `Fields`	the same, assembled by hand, for what neither `struct` nor a derive can say

They compose in the obvious way, which is most of the point:

between(int(), 1, 65535)

Reaching a schema through the type

A type that can be read from untrusted input implements Decode, whose one function is schema() -> Schema<Self>. std implements it for its own types — String, Int, Float, Bool, Decimal, Date, Time, DateTime, Option<A>, List<A>, Vector<A>, Dict<String, V>, Map<String, V>, Redacted<A> (as secret), Json and Raw — and a program implements it for a record or a variant with the constructors above:

impl Decode for Listen {
  fn schema() -> Schema<Listen> {
    struct({ host: string(), port: between(int(), 1, 65535) })
  }
}

Every schema that contains a Listen then reaches this one through the trait, which is the whole override story: write the impl for the type that needs something derivation cannot know, and composition does the rest. The schema is reached by name, Listen::schema(), or by the type the surrounding expression asks for:

let settings: Validated<Settings, Rejection> = decode(raw);

Encode is the other direction, encode(self) -> Raw, and Raw::to_json is the bridge out. It is a separate trait rather than a second half of the schema because a secret has no representation on the wire: Redacted implements Decode and not Encode, so a record holding one reads and does not write, and the build says so. Rejection implements Encode, so a list of problems is a response body a client can read: one object per problem, with its path and its message.

Two routes to text, and they differ in key order. json_text(value) – what Response::json sends – writes a derived record straight to text, its keys in the order the type declares them, so a { zebra, apple, mango } record encodes as {"zebra":..,"apple":..,"mango":..}. An Option field holding None is left out. encode(Raw::to_json(value.encode())) builds a Json value on the way, and a Json::Object holds a Map, so every object it prints is sorted by key. A Map encodes sorted by key on both routes, because a hash table has no order of its own that stays the same from one run to the next.

Encode has a second method, encode_json(self) -> String, which is the fast route. It has a default that goes through encode, so a hand-written impl writes only encode; derive(Encode) writes both. An impl that writes its own encode_json must produce the same value as its encode, with an empty string for a value that is absent.

Where a `Raw` comes from

Raw::of_json turns a parsed document into one, and Raw::to_json turns one back, which is the one bridge in each direction. Raw::of_map takes a query string, the path parameters or the headers; Raw::of_arguments_for(shape, args) takes the command line, so --port 8080 is a field named port, -c is one named c, and whatever is not a flag is collected under arguments. It asks the shape which flags are switches: a field that is a Bool never takes the word after it, so khq -c '.name' f.json reads c as true and the query as the first argument. Raw::of_arguments is the same reading without a shape, for a program that has none in hand.

A source that cannot label its values, which is every one of those but JSON, hands over Raw::Untyped, and every primitive reads it. A source that can label them is taken at its word: "port": "8080" in a JSON body is text, and int() refuses it with port should be a whole number, and is "8080", the way serde and encoding/json refuse it. Strictness is a fact about the source, recorded by the source.

Decode untrusted input puts them together into a program that runs.

What a failure is

decode answers a Validated, not a Result, so a record with four bad fields reports four. A person fixing a deployment wants the list, not one line of it per restart:

refused:
  listen.host is not set
  listen.port must be between 1 and 65535
  password is not set
  rate should be an exact decimal, and is "about seven percent"

Each line is one Rejection, and Rejection::describe is what wrote it. A rejection carries a path, so it can say listen.port or items[3].id; the path is held innermost-first and turned round only when a message is built.

A caller with nothing to do with a list wants a Result instead: decode_or_stop answers Result<A, List<Rejection>>, which ! carries straight out of the function. Validated::to_result converts one that was already decoded, and that is exactly what decode_or_stop does — it is the shape that differs, not the work. It is the caller that stops, not the decode: the Err still holds every rejection, because a schema’s read ends in a Validated and has no way to short-circuit.

A number keeps its text

Raw::Number holds the token, not a Float. A Float carries about fifteen significant digits, so 9007199254740993 comes back one short of itself and 10.10 is never exactly recoverable — and a schema that decoded a Decimal through one would rebuild, inside the library meant to prevent that class of thing, the exact bug it exists to prevent.

std::json keeps its numbers the same way, so the two trees agree about what a number is. They remain separate types: a Raw is what any source produced, and JSON is one source.

A secret does not appear in its own error

Quoting the bad value is most of what makes a decode error worth reading, and it is also the easiest imaginable way to put a password in a log. So Rejection’s found is a Redacted<String> unconditionally — not only inside a secret — and describe decides once whether to expose it. No later variant can forget.

public should be a whole number, and is "not a number"
token should be a whole number

The declaration is the schema

Effect writes the schema and recovers the type, because TypeScript erases types. Khora’s compiler holds every declaration, so the direction reverses: derive(Decode) writes the schema from the type.

derive(Show, Decode, Encode)
pub type Mode = | Local | Remote(url: String);

derive(Show, Decode)
pub type Settings = {
  pub listen: Listen,
  pub password: Redacted<String>,
  pub debug: Option<Bool>,
  pub rate: Decimal,
  pub tags: List<String>,
  pub mode: Mode,
};

Nothing describes the record a second time. Every customization is a type: Option<A> is optional, List<A> is a list, Redacted<A> is a secret whose failures never quote what they saw, Decimal is exact, a nested type is found through the trait — so Listen above may derive its schema or write one by hand with a refined port, and Settings picks up whichever it is. A variant derives to a bare string for a payload-free case and an object tagged with type for the rest; a newtype, type UserId = Int, is transparent; a type that mentions itself needs nothing written, because a derived schema is built when it is first read. The /// above the type and above each field is carried along as the description, so the JSON Schema rendered from Settings says about rate what the comment next to rate says.

Two things a derive refuses, at the derive line: a field whose type has no Decode, and a case whose payload has no field names, because the wire needs a key and a name the type did not declare is not the compiler’s to invent.

The record literal

The hand-written form is the record’s own literal with a schema where each value would go:

impl Decode for Listen {
  fn schema() -> Schema<Listen> {
    struct({ host: string(), port: between(int(), 1, 65535) })
  }
}

Which record it is comes from the type the expression is asked for — the declared return type here, an annotation or a parameter elsewhere — or from the labels alone when only one record has them, the way any record literal resolves. A field whose schema decodes the wrong type is reported at that schema; a field the record does not have is reported at the call.

struct is not a function that runs. Its argument is a record of schemas and its result a schema of the record they decode, and there is no type-level map from one to the other; a call to it is rewritten before it is typed into Schema::record over Fields, which is what a derived schema is too. So it takes a record literal and nothing else, there is no arity, and a record with a hand-written schema is picked up by every schema that contains it.

There is no field limit. struct is rewritten into Schema::record over Fields, and Fields::zip nests a tuple however many fields there are — a seven-field record checks, and so does a twelve-field one.

std::json parses and prints, and does not decode. parse turns text into a Json and Raw::of_json turns that into what a schema reads; Raw::to_json and encode are the way back. There is no second decoding vocabulary beside this one.