std::resilience

Running something again when the first time did not work.

Schedule says when and how many times; retry and repeat are the two ways to drive one. docs/vision.md names retry policies as a non-negotiable, and until Clock.sleep existed the checked-in version was { attempts: Int } – a counter with no clock, used by nothing.

Named for the family rather than for retry. A circuit breaker, a rate limiter, a bulkhead and a hedged request are all the same subject and none of them is a retry; a module called std::retry could not hold them and this one can.

The two things worth knowing before reading further

A Schedule is a description, not a stream of instants. It holds no clock, so it can be compared, printed, sent to another fiber, and written down in a test with no runtime at all. Elsewhere this type is a closure (attempt, elapsed) -> Option<delay>, which is smaller and cannot do any of that: sharing.md refuses a record of closures across a fiber, and a log line saying Exponential { base: 100, .. } is worth more than one saying nothing.

The schedule never sleeps; the driver does. next_after answers when the next attempt should start, as an offset from the moment the whole thing began, and the driver compares that to the clock and waits only for the difference. That is what makes a schedule pure, and it is what lets Union, Intersect and UpTo be real comparisons: the sooner of two delays is not the sooner of two instants once the sides have been running for different lengths of time.

It also lets the two honest readings of “again later” coexist without either being a special case. Spaced is a grid – attempts begin at 0, millis, 2 * millis and so on whatever the body cost, so a run that falls behind does not pile up delays. Exponential and Fibonacci are backoffs, measured from the failure that just happened, because “wait twice as long as last time” is a statement about the other end rather than about the calendar. Both are instants by the time the driver sees them.

API at a glance

Types: Schedule · Tried

Methods: Schedule

Functions: retry · retry_while · retry_counting · repeat

Types

Schedule

derive(Eq, Show)
pub type Schedule =
  | Times(n: Int)
  | Spaced(millis: Int)
  | Exponential(base: Int, factor_pct: Int, cap: Int)
  | Fibonacci(base: Int)
  | Jittered(inner: Schedule, low_pct: Int, high_pct: Int)
  | Union(left: Schedule, right: Schedule)
  | Intersect(left: Schedule, right: Schedule)
  | AndThen(first: Schedule, then: Schedule)
  | UpTo(inner: Schedule, millis: Int);

When to run something again, and when to stop.

Built up from small pieces rather than offered as a list of named policies, so that the combination somebody needs and nobody anticipated is spellable: AndThen(Times(3), Spaced(60000)) retries hard three times and then keeps trying once a minute, and no library had to think of that.

The two Ints in Exponential and Jittered are percentages rather than floats. A policy is a thing people read in a review and a config file, and factor_pct: 200 is unambiguous where 2.0 invites the question of what it multiplies – and Float in a value that derives Eq is a trap of its own.

Times

| Times(n: Int)

At most n attempts in total, with no delay between them. 1 is “do not retry”, and is what you get from once.

Spaced

| Spaced(millis: Int)

Every millis, without end.

Anchored to the start, so it does not drift. Attempts begin at 0, millis, 2 * millis and so on from the beginning, whether or not the one before ran long, which is what a caller who wrote “every thirty seconds” meant.

Exponential

| Exponential(base: Int, factor_pct: Int, cap: Int)

Wait base, then base * factor_pct / 100, and so on, never past cap.

The one almost every remote call wants. Exponential(100, 200, 30000) waits 100ms, then 200, then 400, then 800 … capped at thirty seconds.

Measured from the last failure, not from the beginning — the opposite of Spaced, and the two are the two honest readings of “again later”. A grid says when attempts happen and a backoff says how long to leave the other end alone, which is a statement about the failure rather than about the calendar. A backoff anchored to the start would shorten every wait by however long the failing call took, which is exactly backwards: the call that took thirty seconds to time out is the one to be most patient after.

Fibonacci

| Fibonacci(base: Int)

Waits of base, base, 2 * base, 3 * base, 5 * base …

Grows more gently than doubling, which suits a dependency that is briefly busy rather than actually down. Measured from the last failure, as Exponential is.

Jittered

| Jittered(inner: Schedule, low_pct: Int, high_pct: Int)

inner, with each delay multiplied by a percentage drawn from [low_pct, high_pct).

The one that stops a thundering herd. A thousand clients that all failed at the same instant and all back off by exactly 100ms retry at the same instant, which is the outage again. The draw comes from Random, so it is seedable and the row says so.

Union

| Union(left: Schedule, right: Schedule)

Runs while either side would: the sooner of the two next instants, and it stops only when both have.

Intersect

| Intersect(left: Schedule, right: Schedule)

Runs while both sides would: the later of the two next instants, and it stops as soon as either has.

How a bound is written. Intersect(Exponential(..), Times(5)) is “back off, but no more than five attempts”, and it is a real interval intersection rather than a maximum of delays because the instants are absolute.

AndThen

| AndThen(first: Schedule, then: Schedule)

first until it stops, then then from there on.

UpTo

| UpTo(inner: Schedule, millis: Int)

inner, but nothing after millis from the very beginning.

A wall-clock budget rather than an attempt count, which is usually what a person means by “give up after a minute”.

Tried

pub type Tried<A, E> = {
  outcome: Result<A, E>,
  attempts: Int,
};

How a retried call ended, and how many goes it took.

The count is the thing retry cannot tell you. repeat has always answered one, because it cannot fail and there is nothing else to report; retry answers the value and drops it, so a caller who wanted to log “succeeded on attempt 3” kept a Shared<Int> and took a lock on every attempt to find out. That is a mutex and a fiber-crossing allocation to count to three.

outcome

outcome: Result<A, E>

What happened in the end: the value, or the failure that stopped it.

attempts

attempts: Int

How many times body ran. One when it succeeded first go, and never zero – the first attempt is unconditional.

Methods

Schedule

impl Schedule

once

pub fn once() -> Schedule

Run once, and do not retry. The identity, and what you get by not asking.

times

pub fn times(n: Int) -> Schedule

Up to n attempts, as fast as they can be made.

backoff

pub fn backoff(base: Int, cap: Int) -> Schedule

Doubling from base, capped at cap, with the usual jitter.

The one to reach for, spelled out here so that the common case is not four constructors deep. 50%–100% of the computed delay is what most published guidance settles on: enough spread to break up a herd, not so much that a retry arrives before the thing it is retrying could have recovered.

next_after

pub fn next_after(self, attempt: Int, elapsed: Int) -> Option<Int> with { random: Random }

When the attempt after attempt should start, as milliseconds from the beginning, or None to stop.

attempt counts the ones already made, so the first call is with 1. elapsed is what the clock says has passed since the beginning, which only UpTo looks at – a budget is the one policy that cannot be decided from the count alone.

An absolute instant, not a delay. Every combinator below depends on it: the sooner of two delays is not the sooner of two instants once the two sides have been running for different lengths of time, and Spaced only stays drift-free because it answers attempt * millis rather than millis.

import std::clock::{Clock};
import std::random::{Random};
import std::resilience::{Schedule};

// Two attempts made; wait for the third, or say the plan is done.
fn hold(plan: Schedule, began: Int) -> Bool
  with { clock: Clock, random: Random }
{
  match Schedule::next_after(plan, 2, clock.monotonic_millis() - began) {
    Option::None => false,
    Option::Some(at) => {
      let now = clock.monotonic_millis() - began;
      if at > now { clock.sleep(at - now) } else { () };
      true
    }
  }
}

at is an instant, so only the part of it still ahead is waited for. Passing it to sleep unchanged waits from the beginning a second time.

None means the schedule is finished, which is how retry knows to stop. Exposed because a caller driving its own loop – a queue consumer deciding when to redeliver – needs the decision without the waiting.

Functions

retry

pub fn retry<A, E, 'ef>(schedule: Schedule, body: () -> A with 'ef raises E) -> A with { 'ef | clock: Clock, random: Random } raises E

Runs body until it succeeds or the schedule runs out.

The last failure is the one that comes back, because it is the one that stopped the retrying — an earlier error is a thing that was already recovered from, and reporting it would say the wrong thing about what happened.

The row is where the cost is written down: this needs a clock because it waits, and a source of randomness because a schedule may be jittered. A caller who wanted neither reaches for Schedule::times, and still declares both — which is honest, since the policy is a value and could have been a jittered one.

import std::clock::{Clock};
import std::random::{Random};
import std::resilience::{Schedule, retry};

fn fetch() -> String with { client: HttpClient } raises HttpError { .. }

fn fetch_hard() -> String
  with { client: HttpClient, clock: Clock, random: Random }
  raises HttpError
{
  retry(Schedule::Exponential(100, 200, 30000), fn () => fetch()!)!
}

clock and random are in the row, and both have to be. Waiting is something the caller grants – a retry that slept without saying so would have latency invisible in its type, and a test could not make it instant – and the jitter is drawn from random rather than invented, so a thousand clients that failed together do not come back together.

retry_while

pub fn retry_while<A, E, 'ef>(schedule: Schedule, again: (E) -> Bool, body: () -> A with 'ef raises E) -> A with { 'ef | clock: Clock, random: Random } raises E

retry, for a caller that can tell a retryable failure from a permanent one.

A predicate rather than a Schedule case, which is why Schedule can stay a plain ADT: “retry while the error looks like this” is a closure over the caller’s own error type, and putting one inside the description would make the description a record of closures. The two questions are separate anyway — when to try again is a policy somebody configures, whether this failure is worth trying again is knowledge about the error.

Retrying a 404 is the failure this exists to prevent, and it costs the caller one fn.

import std::resilience::{Schedule, retry_while};

fn load(id: Int) -> Row
  with { db: Db, clock: Clock, random: Random }
  raises DbError
{
  retry_while(Schedule::times(3), fn problem =>
    match problem {
      DbError::Disconnected(_) => true,
      DbError::Rejected(_) => false,
      DbError::RolledBack(_) => false,
    }, fn () => fetch(id)!)!
}

The predicate is the point: a dropped connection is worth another go and a rejected statement is not, and retrying the second one just fails again three times more slowly.

retry_counting

pub fn retry_counting<A, E, 'ef>(schedule: Schedule, again: (E) -> Bool, body: () -> A with 'ef raises E) -> Tried<A, E> with { 'ef | clock: Clock, random: Random }

retry_while, reporting how many attempts it took.

The loop is here and the other two are wrappers over it, so there is one place the schedule is consulted and one place the count is kept.

This one does not raise. The failure comes back inside the Tried rather than out of it, which is the only way the count survives a run that ended badly – and “it failed after six goes over ninety seconds” is exactly the line somebody wants in the log when it did. A caller who wants the raise wants retry_while, which is two lines further down and does the match.

The last failure is the one reported, for the reason retry gives: it is the one that stopped the retrying, and an earlier error is something that was already recovered from.

import std::resilience::{Schedule, Tried, retry_counting};

fn fetch_reporting() -> Tried<String, HttpError>
  with { client: HttpClient, clock: Clock, random: Random }
{
  retry_counting(Schedule::times(5), fn _e => true, fn () => fetch()!)
}

The count comes back with the answer, which is what a caller logs or turns into a metric. retry throws it away, which is right when nobody was going to look at it.

repeat

pub fn repeat<A, E, 'ef>(schedule: Schedule, body: () -> A with 'ef raises E) -> Int with { 'ef | clock: Clock, random: Random }

Runs body on the schedule until it fails or the schedule runs out, and answers how many runs succeeded.

The other half of retry, and the one a poller is written with: repeat(Schedule::Spaced(30000), fn () => reconcile()!) is “every thirty seconds until something breaks”. The count comes back so a caller can tell “the schedule ended” from “it never ran”.

import std::resilience::{Schedule, repeat};

fn poll_forever() -> Int
  with { clock: Clock, random: Random, client: HttpClient }
  raises HttpError
{
  repeat(Schedule::Spaced(30000), fn () => check_health()!)!
}

The mirror of retry: it runs the body again while it succeeds and stops at the first failure, answering how many times it ran. Spaced rather than Exponential because a poller wants a grid it does not drift off.