std::ai

Model inference as a capability, and tensors whose shapes are checked.

Two things that look unrelated and are the same argument. A program that calls a model is a program that reaches the network, spends money and returns something different every time, so LLMService is an effect and the with clause says which functions can do it. A program that multiplies two tensors is a program where a shape mismatch is discoverable before it runs, so Tensor carries its device, its shape and its element type in the type and matmul unifies the shared dimension.

What this is not

Not an implementation of anything. There is no HTTP here, no tokenizer, no BLAS. The ecosystem design note puts numeric kernels and vendor clients in packages for the reason it puts database drivers there: what belongs in std is the vocabulary two packages must agree on to exchange a value, and the effect declaration that lets a caller say what it needs without naming a provider.

extract is the shape of the whole module. A caller writes let report: Report = extract(prompt)! and LLMService never learns that reports exist – the handler answers complete: Prompt -> String, and the type’s own schema, which derive(Decode) writes, describes the answer to the model and turns that string into the type the call site asked for.

API at a glance

Types: Device · Scalar · Tuple · F32 · Tensor · Embedding · Message · Prompt · ModelError

Effects: LLMService

Methods: Tensor<D, Shape, T> · Prompt

Functions: matmul · extract · embed · cosine_similarity

Types

Device

pub type Device =
  | Cpu
  | Cuda(device_id: Int)
  | Metal
  | Npu;

Where a tensor’s memory lives.

Carried in the type, so a program that multiplies a CPU tensor by a GPU one is refused rather than copying silently. Provisional along with the rest of this module.

Scalar

pub type Scalar;

The kind of an element type: what may appear as a tensor’s T.

A kind rather than a type — nothing is a value of Scalar. It exists so that Tensor<D, Shape, T> can say what T is allowed to be.

Tuple

pub type Tuple;

The kind of a shape: what may appear as a tensor’s Shape.

(M, K) is a value of this kind, not a value of any type. Matching two shapes is ordinary unification, which is what makes a mismatched product a type error.

F32

pub type F32;

Thirty-two-bit floating point, as an element type.

Tensor

pub type Tensor<D: Device, Shape: Tuple, T: Scalar>;

A tensor, with its device, its shape and its element type in the type.

The shape is not a runtime field. Tensor<Cpu, (1536), F32> and Tensor<Cpu, (768), F32> are different types, so a dimension mismatch is discovered by the checker rather than asserted at the first multiply. That is the whole argument for this being in std while the kernels are not: the vocabulary two packages must agree on is here, and the arithmetic is somebody else’s.

Embedding

pub type Embedding<const Dim: Int, T: Scalar> = Tensor<Device::Cpu, (Dim), T>;

A one-dimensional CPU tensor of a known width.

The shape a vector database and an embedding model have to agree on, spelled once: Embedding<1536, F32>.

A type of its own, not another name for the tensor inside it. Khora has no transparent alias: type X = Y wraps, so an Embedding is built with Embedding(t) and opened with match e { Embedding(t) => t }. That is the right way round here — a tensor of the correct shape is not automatically an embedding, and handing a model’s output straight to a distance function without saying which is which is the mistake this module is meant to make hard.

Message

pub type Message = {
  role: String,
  content: String,
};

One turn of a conversation: who said it, and what they said.

role is text rather than a variant type deliberately. Providers disagree about the set — system, user, assistant, tool, developer — and a variant here would have to be widened by std every time one of them added a word.

Prompt

pub type Prompt = {
  system_instructions: Option<String>,
  messages: List<Message>,
  temperature: Float,
};

What is sent to a model: standing instructions, a conversation, and how much variation to allow.

Built by piping rather than by filling in a record, so a half-built prompt is never a value anybody holds. See the impl below.

ModelError

pub type ModelError =
  | ContextLengthExceeded(max_tokens: Int)
  | RateLimited(retry_after_ms: Int)
  | InferenceEngineFailure(msg: String)
  | SchemaExtractionError(problems: List<Rejection>);

Why a model call did not produce an answer.

Coarse on purpose, on the same reasoning std::db’s DbError gives: every provider numbers its failures differently, and a std that picked one would be wrong for the second. RateLimited carries the wait because a caller can act on it; the rest carry text.

SchemaExtractionError

| SchemaExtractionError(problems: List<Rejection>)

The model answered, and the answer was not the type asked for: every problem with it, the way a request body’s problems are reported, because a model is allowed to be wrong and the ordinary case is that it is wrong in several places.

Effects

LLMService

pub effect LLMService {
  complete: Prompt -> String raises ModelError,
  embed_raw: String -> List<Float> raises ModelError,
}

Inference is a capability, not a value: a function that classifies text declares with { ai: LLMService }, and the caller decides what serves it.

Two operations, and both are rank-1. A model takes a prompt and returns text, or takes text and returns numbers; that is the whole of what it does, and it is the whole of what belongs here. Asking for a typed answer is extract below — a generic function over this effect, because an operation is a field of a handler and a field has one type.

Keeping the effect this small is what makes a mock one line: return a string. An effect that asked a handler to fabricate whatever shape the caller wanted would make a test exercise the mock rather than the code under it.

complete

complete: Prompt -> String raises ModelError

embed_raw

embed_raw: String -> List raises ModelError

The raw vector, at whatever width the model produces. embed below is the shape-safe reading of it.

Methods

Tensor<D, Shape, T>

impl<D: Device, Shape: Tuple, T: Scalar> Tensor<D, Shape, T>

zeros

pub fn zeros(shape: Shape) -> Tensor<D, Shape, T>

A tensor of the given shape, every element zero.

The shape is passed as a value and read back as a type, so Tensor::zeros((1536)) has the shape Embedding<1536, F32> wraps — Embedding(Tensor::zeros((1536))) is the embedding. Provisional along with the rest of this module.

Prompt

impl Prompt

Built by piping: Prompt::new() |> Prompt::system(..) |> Prompt::user(..). Each returns a new prompt rather than mutating one, so a half-built prompt cannot be shared and then changed underneath its other holder.

new

pub fn new() -> Prompt

An empty prompt: no instructions, no messages, temperature one.

system

pub fn system(self, instructions: String) -> Prompt

Replaces the system instructions; the last one written wins.

user

pub fn user(self, message: String) -> Prompt

Adds a turn from the person asking.

assistant

pub fn assistant(self, message: String) -> Prompt

Adds a turn from the model, which is how a conversation is replayed.

saying

pub fn saying(self, role: String, content: String) -> Prompt

Adds a turn under any role. user and assistant are this with the two roles every provider agrees on.

at_temperature

pub fn at_temperature(self, temperature: Float) -> Prompt

Sets how much variation to allow. Zero is as close to deterministic as a provider offers, which is not a promise any of them makes.

Functions

matmul

pub fn matmul<D: Device, const M: Int, const K: Int, const N: Int, T: Scalar>(a: Tensor<D, (M, K), T>, b: Tensor<D, (K, N), T>) -> Tensor<D, (M, N), T>

The matrix product, with the shared dimension checked by the type system.

K appears in both arguments, so it has to unify: an (M, K) times a (J, N) does not compile, and no assertion runs to discover it. Declared rather than implemented — the ecosystem design note puts the kernel in a package.

extract

pub fn extract<A: Decode>(prompt: Prompt) -> A with { ai: LLMService } raises ModelError

A typed answer from the model.

The call every caller wants — let report = extract(prompt)! — and the reason LLMService does not have to know that reports exist. A is chosen by the caller, which is what makes this a function rather than an operation: a generic function is specialized at each call site, and a field of a handler cannot be.

The type’s schema is the whole of the contract. Its shape, rendered as a JSON Schema, is what the model is told to produce; its decoder is what reads the answer back, reporting every problem. A type that derives Decode needs nothing else to be extracted.

embed

pub fn embed<const Dim: Int>(text: String) -> Embedding<Dim, F32> with { ai: LLMService } raises ModelError

An embedding of a known width.

The width is the model’s, not the caller’s. Nobody can choose to get 768 numbers out of a model that returns 1536, so asking for a Dim is asking a question, and a Dim that does not match the answer is a ModelError. The old declaration made it a caller-chosen parameter, which said otherwise.

cosine_similarity

pub fn cosine_similarity<const Dim: Int>(a: Embedding<Dim, F32>, b: Embedding<Dim, F32>) -> Float

How alike two embeddings are, from -1 to 1.

Both must have the same width, and the type says so rather than a length check discovering it: comparing a 1536-wide embedding with a 768-wide one is refused by the checker. Declared rather than implemented, like matmul — the arithmetic belongs in a package.