Testing and benchmarks

test and bench are declarations, not a naming convention, and they live beside ordinary source. Tests run with khora test; benchmarks run with khora bench. The declaration syntax is in Declarations; this page is how they are used.

Write a test

A test has a string name and a block:

module pricing_test;

import std::core::{assert};

fn double(value: Int) -> Int {
  value * 2
}

test "double returns twice its input" {
  assert(double(21) == 42);
}

assert is a std::core function, not a keyword: the import line above is required. It is also legal only inside a test block — in application code the compiler answers error: `assert` is only allowed inside a `test` block; elsewhere, `raise` says the same thing and says where it goes, and it does so at khora check, so the editor shows it as you type rather than the build finding what the check passed.

Run a package’s tests from its root:

khora test .

Filter by a substring of the test name:

khora test . --filter double

Typed failures

Use the same catch or attempt that application code uses:

module users_test;

import std::core::{Result, assert, attempt};

pub type UserError =
  | NotFound(id: Int)
  | Unavailable(why: String);

fn load_user(id: Int) -> String raises UserError {
  if id == 999 { raise UserError::NotFound(999) } else { "ada" }
}

test "missing users are reported" {
  let result = attempt(fn () => load_user(999)!);

  match result {
    Result::Ok(_) => assert(false),
    Result::Err(UserError::NotFound(id)) => assert(id == 999),
    Result::Err(UserError::Unavailable(_)) => assert(false),
  }
}

attempt and Result are std::core names like assert, so all three are on the import line. UserError and load_user are declared here to keep the block runnable; in a real test they come from the module under test.

Failures is the model behind it.

Capabilities, supplied by the test

Effectful code takes controlled handlers instead of reaching real services:

import std::clock::{Clock};

const fixed_clock = handler for Clock {
  unix_seconds: fn () => 1700000000,
  unix_millis: fn () => 1700000000000,
  monotonic_millis: fn () => 5000,
  sleep: fn _millis => (),
};

test "session uses the supplied clock" {
  let session = create_session(user) with {
    clock: fixed_clock,
  };

  assert(session.created_at == 1700000000000);
}

The function’s with row tells the test exactly which outside authority has to be supplied — there is nothing to discover by reading the body. A named context can supply an ordinary application environment with one binding overridden for the test; see Capabilities.

sleep is one of the clock’s operations, so that four-line handler is also how a test drives a retry loop or a poller without waiting for it. There is no test runtime and no special mode.

Failure and cancellation paths

Khora’s strongest guarantees are about control flow that does not return normally, so a test for a resource, a transaction or concurrent work should cover successful completion, typed failure, cancellation where it applies, and the cleanup or rollback each of those paths owes. Cancellation-safe resources is that written out.

A trap ends the whole run

An assert that fails, or a raise that escapes, is a failing test: it is reported, the other tests still run, and the process exits 1. A trap is different — dividing by zero, indexing past the end, arithmetic overflow — and ends the process immediately, including the tests that had not run yet:

test a ... ok
khora: Int division by zero on fiber 8
note: re-run with KHORA_BACKTRACE=1 to see where

khora: this ended the test run. 3 test(s) had not been reported: `b traps`, `c`, `d`.
khora: a trap ends the process, so the rest of the run did not happen. `khora test --filter <name>` runs one at a time.

Tests run concurrently, so how many verdicts appear before the trap varies between runs — a test whose result had not been printed yet is listed as not reported, even if it would have passed. What does not vary is that the run says it ended early and names what it did not reach.

Trapping cannot be contained into a failing test: recovering would need an unwinder to run the reference-count operations between the trap and a landing point, and a fiber abandoned without them leaks everything it was holding.

So a trap in a test is a bug to fix rather than a result to read, and khora test --filter <name> is how to work on one test while another traps.

Write a benchmark

bench takes the same named-block shape:

bench "parse representative payload" {
  parse_payload(fixture)!;
}

khora bench .
khora bench . --filter payload

Keep setup that is not part of the measurement outside the operation being compared, where the benchmark’s design allows it.

Benchmarks build unoptimized unless told otherwise

They build unoptimized like everything else the toolchain compiles, because a language being brought up should give a readable crash by default. There is no --release flag here; khora test and khora bench read KHORA_PROFILE, because a flag on every subcommand is three ways to say one thing:

KHORA_PROFILE=release khora bench .

khora bench prints the profile it used above the results, so a number pasted into an issue carries that with it. A small integer loop differs by about a factor of two between the two profiles; code the optimizer can see through differs by much more.

And some workloads differ by nothing measurable. Code whose time goes on allocation, reference counting or I/O spends it in the runtime, which is compiled once and is the same either way. A release build buying nothing is information about where the time is going, not a sign the profile did not take. Performance is the methodology.

In CI

A useful baseline:

khora fmt . --check
khora check .
khora test .

A project with performance budgets can run selected khora bench workloads separately from correctness CI — with KHORA_PROFILE=release, or the budget is measured against a build nobody ships.