I’ve been pretty annoyed for the last few months with using sillyfmt from the command line on MacOS. Previously, I’d just run sillyfmt with no arguments, which reads from STDIN and attempts to format whatever gets passed in. I would then copy and paste interesting snippets into the open window.

Unfortunately, any large blocks of text would mysteriously truncate themselves and hang for a bit on MacOS. I had worked around this by piping the output from pbpaste (i.e. pbpaste | sillyfmt), but it was annoying and I tended to use the WASM version instead (in fact, this is a lot of why there’s a WASM version at all).

Today, I finally figured out what was going on: the trick here is the difference between canonical input and non-canonical input. On MacOS, the maximum line length of canonical input is set to 1024 bytes, and pasting into the terminal pastes the entire string on a single line. This meant that I was running into the canonical mode limits all the time.

Canonical input mode is a basic shell functionality to let you do simple text-editing functionality on the command line. For example, it implements the backspace and delete keys, and when you hit return, the whole line is made available to waiting programs to read. Since sillyfmt was reading directly from stdin:

silly_format(
  io::stdin().lock(),
  io::stdout().lock(),
  // ...
)

this meant that it defaulted to canonical input, and the corresponding limitations. Not that I knew what canonical input was, at first…

With some experimentation I realized that some programs were able to take long inputs – in fact, the shell was one of them – so there was definitely some setting somewhere to make this work. Eventually, I realized that there’s a termios function which lets you turn off canonical mode and handle the keypresses yourself. Being that I didn’t really want to implement an editor environment, though, I decided to import a readline implementation instead.

Of course, the readline implementation (rustyline) doesn’t look like a std::io::Read:

let mut rl = rustyline::Editor::<()>::new();

loop {
  let line = rl.readline("")?;
  // ...
}

so it made code sharing a little annoying. I resolved this by creating a new function silly_format_iter which takes an iterator over lines, and adding a wrapper over rustyline::Editor that exposes an iterator interface.

struct EditorIter {
    editor: Editor<()>,
}

impl Iterator for EditorIter {
    type Item = io::Result<String>;
    fn next(&mut self) -> Option<io::Result<String>> {
        match self.editor.readline("") {
            Ok(line) => Some(Ok(line)),
            Err(ReadlineError::Eof) | Err(ReadlineError::Interrupted) => None,
            Err(ReadlineError::Io(e)) => Some(Err(e)),
            Err(e) => {
                eprintln!("Unexpected err {:?}", e);
                Some(Err(io::Error::new(io::ErrorKind::Other, "unknown error")))
            }
        }
    }
}

As hacks go, not too bad – and now, sillyfmt works correctly when I copy and paste random blobs into it!

Amusingly, I didn’t figure this out early on in the development of sillyfmt because Linux’s canonical line length limit is higher (4K) and the paste mechanism pastes newlines more aggressively. But, it’s fixed now!