std::fs

The file system, over ISO C.

Two layers, and the split is the point.

Inside this module are the foreign declarations — fopen, fread and the rest — and the code that gets their conventions right: a null FILE * means the open failed, a short read means something went wrong, and a file has to be closed on the path where it failed as much as on the path where it did not. That code is not exported. This module is the trusted boundary, in the same way that Rust’s standard library is the place unsafe lives so that nothing above it needs to.

Outside are the two effects, and nothing reaches a file without one. That buys two things at once. It is a permission: a function that reads a file says so in its signature, all the way up to the main that decided to allow it. And it is a seam: a test installs its own handler and the code under test cannot tell, which is the thing a file system mock is usually a poor imitation of.

A free function here is one that does more than an operation. read_text, write_text and append_text are text where the operation is bytes, and copy needs both effects and a buffer; each of the four earns a name because there is a decision inside it. Everything else is called on the capability – writes.create_dir(path)!, reads.read_dir(path)! – and has no wrapper on purpose, because a wrapper would be the operation’s own argument list typed out a second time. The asymmetry is the rule rather than a gap in the list.

Two effects rather than one, FsRead and FsWrite, because [permissions.fs] already grants reading and writing separately and a single capability made the finer half unexpressible: a function allowed only to read still received the authority to delete. copy is the one helper here that needs both, and it says so.

The permissions design note for the manifest half.

The ffi design note for the boundary rules this obeys.

API at a glance

Types: IoError

Effects: FsRead · FsWrite

Methods: FsRead · FsWrite

Functions: read_text · write_text · append_text · chunk_size · fold_chunks · fold_lines · copy · join · parent · file_name · extension · stem

Types

IoError

derive(Show, Eq)
pub type IoError =
  | NotFound(path: String)
  | Failed(path: String)
  | Denied(path: String);

Why a file operation did not work.

Deliberately coarse. C reports failure through errno, whose values are not portable and whose meanings are not always the ones a caller wants to branch on; promising more detail than can be delivered everywhere would be promising something. What a caller can act on is “it was not there” against “it did not work”, and both carry the path so the message can say which file.

Coarse is not the same as arbitrary, and it was arbitrary. fopen returns null whether the file is absent, unreadable or a directory, and every one of those was reported as NotFound – so the one distinction this type does draw was not drawn. open_or_say_why reads the reason in the runtime, on the thread that set errno, and only a genuine ENOENT is NotFound now.

Denied

| Denied(path: String)

The manifest does not grant this path.

A third case rather than Failed, because the two want different things from whoever reads the message. Failed is a disk, a permission bit, a name that was there a moment ago; Denied is khora.toml, and the fix is a line in a file the reader owns. Folding it into Failed would send somebody to look at their file system for a decision their own manifest made.

Effects

FsRead

pub effect FsRead {
  read: (String) -> Array<U8> raises IoError,
  exists: (String) -> Bool raises IoError,
  size: (String) -> Int raises IoError,
  read_at: (String, Int, Int) -> Array<U8> raises IoError,
  read_dir: (String) -> List<String> raises IoError,
  is_dir: (String) -> Bool raises IoError,
}

Reading the file system.

Split from writing, because the manifest already splits them. [permissions.fs] has read and write as separate grants, and a capability that carried both made the finer half of that grant unexpressible: a program allowed only to read still received the authority to delete. Two effects say in a signature what the manifest says in a table.

Whole files, not handles. A handle would have to be an opaque type, which nothing but this module could construct — and an effect whose operations only the real implementation can satisfy is not a seam, it is a wrapper. Streaming will want one and can have its own effect when something needs it.

Every operation here is one a test double has to write, which is the standing argument against adding another.

read

read: (String) -> Array raises IoError

The whole file, as bytes.

exists

exists: (String) -> Bool raises IoError

Whether the path can be opened for reading.

Not “whether it exists”. ISO C has no stat, so the only portable question is whether fopen succeeds, and a file that is there but unreadable answers false. The name says exists because that is what callers mean; a caller that needs the distinction wants a permissions API rather than this one.

A path the manifest denies raises Denied rather than answering false. Answering false would make three different situations one word: a file that is not there, a file that is there and unreadable, and a file that is there, readable, and simply not granted. The third is the one somebody debugs for an hour, because nothing in a false points at the manifest – and it is exactly the confusion IoError::Denied exists to prevent everywhere else.

The cost is a ! on a probe, which is the same cost read has always had and for the same reason. What it buys is that the remaining false means one thing: the operating system would not open it.

size

size: (String) -> Int raises IoError

How many bytes the file holds.

read_at

read_at: (String, Int, Int) -> Array raises IoError

At most size bytes, starting at offset.

Fewer near the end, and none at or past it. This is how a file larger than memory is read, and the position is an argument rather than a handle on purpose: a handle would be an opaque type nothing but this module could construct, and an effect whose operations only the real implementation can satisfy is not a seam, it is a wrapper. An offset is an Int, so a double needs arithmetic and nothing else.

fold_chunks and fold_lines are what most callers want.

read_dir

read_dir: (String) -> List raises IoError

The names in a directory, without the directory.

Names, not paths: joining is the caller’s, and join is right here. . and .. are not included, which is the one place this differs from readdir and the difference every caller would have written itself.

The directory needs its own grant, and ./data/** is not one. The glob dialect is .gitignore’s, where data/** covers what is inside data and not data itself – so a manifest that grants

[permissions.fs]
read = ["./data/**"]

can read every file in the directory and cannot list it. Both entries are what a walk wants:

read = ["./data", "./data/**"]

This is the first thing anybody writes with FsRead and the first place the dialect’s one surprise shows up, which is why it is written here rather than only beside granted.

is_dir

is_dir: (String) -> Bool raises IoError

Whether the path is a directory.

Raises Denied for a path the manifest does not grant, the same as FsRead::exists and for the same reason – and this is where it matters most. ./data/** does not grant data itself, so a manifest granting only the glob makes is_dir("data") raise Denied rather than answer false about a directory whose every file the program can read.

FsWrite

pub effect FsWrite {
  write: (String, Array<U8>) -> () raises IoError,
  append: (String, Array<U8>) -> () raises IoError,
  remove: (String) -> () raises IoError,
  rename: (String, String) -> () raises IoError,
  create_dir: (String) -> () raises IoError,
  remove_dir: (String) -> () raises IoError,
}

Changing the file system.

The other half of [permissions.fs]. A function that takes this can alter what is on disk; one that takes only FsRead cannot, and its signature is the proof.

write

write: (String, Array) ->() raises IoError

Replaces the file’s contents, creating it when it is not there.

append

append: (String, Array) ->() raises IoError

Adds to the end, creating the file when it is not there.

remove

remove: (String) ->() raises IoError

Deletes a file. Not a directory — that is remove_dir.

rename

rename: (String, String) ->() raises IoError

Moves or renames. Across file systems C’s rename fails rather than copying, and that failure is reported rather than papered over.

create_dir

create_dir: (String) ->() raises IoError

Creates one directory. The parents are not created: a caller that wants that can walk the path, and one that did not ask should be told the parent is missing rather than have four directories appear.

remove_dir

remove_dir: (String) ->() raises IoError

Removes an empty directory.

Empty only, which is rmdir’s own rule and worth keeping: a recursive delete that removed the wrong tree is the one file system mistake with no undo.

Methods

FsRead

impl FsRead

real

pub fn real() -> FsRead

The real file system, reading.

FsWrite

impl FsWrite

real

pub fn real() -> FsWrite

The real file system, writing.

Functions

read_text

pub fn read_text(path: String) -> String with { reads: FsRead } raises IoError

The bytes of a file, as text.

A convenience over read, and the reason the effect traffics in bytes rather than strings: a file is bytes, and whether they are text is a question with a wrong answer. This one takes the optimistic view and is named for it.

import std::fs::{FsRead, read_text};

fn settings() -> String with { reads: FsRead } raises IoError {
  read_text("config.toml")!
}

pub fn main() -> Int {
  with { reads: FsRead::real() } {
    match attempt(fn () => settings()!) {
      Result::Ok(text) => print(text),
      Result::Err(_) => print("no config"),
    }
  };
  0
}

FsRead::real() reaches the actual filesystem and is checked against the [permissions.fs] your manifest grants. A test installs a handler of its own instead, which is the whole reason this is a capability.

write_text

pub fn write_text(path: String, text: String) ->() with { writes: FsWrite } raises IoError

Writes text to a file, replacing whatever was there.

import std::fs::{FsWrite, write_text};

fn save(report: String) -> () with { writes: FsWrite } raises IoError {
  write_text("report.txt", report)!
}

writes, not fs: the label is chosen by the function being called, and reading and writing are separate capabilities so that a function which only reads cannot quietly gain the ability to overwrite.

append_text

pub fn append_text(path: String, text: String) ->() with { writes: FsWrite } raises IoError

Adds text to the end of a file, creating it when it is not there.

import std::fs::{FsWrite, append_text};

fn note(line: String) -> () with { writes: FsWrite } raises IoError {
  append_text("audit.log", line + "\n")!
}

No newline is added, because a caller appending a fragment did not ask for one. Add it yourself, as above.

chunk_size

pub fn chunk_size() -> Int

How much of a file to read at a time.

64 KiB: large enough that the per-call open is amortised, small enough that a file of any size costs the same memory. Not configurable, because a caller who needs a different number can call read_at and have exactly the loop they want.

fold_chunks

pub fn fold_chunks<A>(path: String, start: A, step: (A, Array<U8>) -> A) -> A with { reads: FsRead } raises IoError

Folds over a file in chunks, without holding it in memory.

The whole point of read_at: a file larger than memory is read a piece at a time, and the accumulator is the only thing that grows.

let bytes = fold_chunks(path, 0, fn (n, chunk) => n + Array::length(chunk))!;

fold_lines

pub fn fold_lines<A>(path: String, start: A, step: (A, String) -> A) -> A with { reads: FsRead } raises IoError

Folds over a file’s lines, without holding it in memory.

Lines are split on \n and a trailing \r is dropped, so a file written on Windows reads the same as one written anywhere else. The last line is delivered whether or not the file ends with a newline – a file ending mid-line has a last line, and silently dropping it is the bug this note exists to prevent.

import std::fs::{FsRead, fold_lines};

fn longest(path: String) -> Int with { reads: FsRead } raises IoError {
  fold_lines(path, 0, fn (so_far, line) =>
    if String::length(line) > so_far { String::length(line) } else { so_far })!
}

The file is read a line at a time rather than into one string, so a log bigger than memory is still foldable. start is the answer for an empty file, which is why it is a value rather than an Option.

Bytes that are not UTF-8 raise IoError::Failed, the same as read_text and for the same reason: a file handed to something that folds over lines was supposed to hold text, and a tool pointed at a binary by accident should be able to say so and exit rather than abort. Converting without asking would trap, which no caller can catch.

step cannot fail, and that is a real limitation rather than a choice. Every List combinator takes (..) -> A with 'ef raises 'er; this does not. So a fold that wants to raise – a malformed line reported with its number, which is the usual reason to fold over lines at all – does not compile, and the error names the lambda rather than the missing row on the parameter it is checked against.

Carry the failure in the accumulator and raise after the fold, which is what this function does internally for bytes that are not text. The alternative people find first is read_text with List::fold, and it gives up the only reason to use this: the file no longer has to fit in memory.

Widening the row is the fix and is not small – both rows have to thread through fold_chunks and fold_from, and with { reads: FsRead } + 'ef does not keep reads in scope today.

copy

pub fn copy(from: String, to: String) ->() with { reads: FsRead, writes: FsWrite } raises IoError

Copies a file, replacing the destination.

Read then write, deliberately: it is not an operation of either effect because a handler that implemented it differently from its own read and write would be lying about itself, and a double would have to write a third thing that agreed with the other two.

This is the function that shows the split doing its job. It needs both capabilities and says so, where every other helper here needs one.

The whole file passes through memory, which is the same limit read has.

import std::fs::{FsRead, FsWrite, copy};

fn back_up() -> () with { reads: FsRead, writes: FsWrite } raises IoError {
  copy("ledger.db", "ledger.db.bak")!
}

Two labels in one row, and both are needed at the call: a function that declared only reads would not compile against this.

join

pub fn join(base: String, child: String) -> String

Joins two path pieces with a single separator.

Always /, on every platform including Windows. Reading accepts either – a base already ending in \ is not given a second separator – but what is written is /, because Windows takes it everywhere and one rule that works on all three targets beats three rules that each work on one.

An empty base is child, so folding over a list of segments needs no special first step.

parent

pub fn parent(path: String) -> Option<String>

Everything before the last separator, or None for a bare name.

file_name

pub fn file_name(path: String) -> String

Everything after the last separator, which is the whole path when there is none.

extension

pub fn extension(path: String) -> Option<String>

The extension of the last component, without its dot.

None for a name with no dot, and for a leading dot — .gitignore is a name rather than an extension, which is the rule every other language settled on the hard way.

stem

pub fn stem(path: String) -> String

The last component without its extension.