Introduction: Setting the stage

Years ago I explored writing system level tools in Groovy. The idea here was to create OS independent, easily deployable programs for use in systems level tasks.

Hint: for me this usually means “A binary I can easily install as part of some CI workflow but that also won’t mess up whatever the developers are doing in their codebase”.

If I can tell I’m on a Unix system then the most obvious tool to grab for is Bash: plain ol’ installed everywhere shell scripting. (If I can’t, then it’s probably Powershell, or Groovy, time!)

There’s a couple problems with grabbing Bash as the easiest tool:

1. The “this is now too big for Bash” upgrade path is rough. That 500 line shell script that probably should be something else? Putting it in something more maintainable means rewritting the whole thing
2. It heavily relies on external programs being installed (ever had the perfect shell script just not work on CI because jq isn’t installed?)
3. Large Bash scripts are hard to debug, maintain, etc

Instead we might grab Ruby, or Python, but what those have in ease of development they lose in easy of deployment: I want one file installs.

Enter Rash

Racket is an interesting Lisp that calls itself Language-Oriented Programming. This results in some neat experimental work: a text rendering language that hooks into Racket, for example. But what we’re most interested in is Rash: a shell scripting language that lets you call Racket from pipelines.

The ACM Paper on Rash is a good, but computer science, overview of the library.

But stand back, where going to do SCIENCE! (by experimenting in our terminal!)

A Rash Introduction

For quick and dirty experimentation I really like Dr Racket’s Package Manager: use it to find and install rash, if you’re following along.

First script: how many files in a directory

A simple script to count the number of files in a directory:

#lang rash

ls | wc -l

This is valid Rash! This is the thing I really like about Rash: I can take a Bash pipeline, and (probably) unmodified it’ll work in my Rash program!

Let’s upgrade it with some Racket:

#lang rash
(require racket/port)

ls |> port->lines |> length

Looks like we need to introduce some Rash concepts.

First, a Rash script is just a Lisp file. I can do Lisp things like import libraries, define functions, do computations, whatever.

Next, |>. This is the “basic object pipeline” pipeline operator. In small words: “Lisp stuff follows”

|> outputs a Unix pipe to the next function. port->lines takes that Unix port (pipe) results and translates it to a Racket list.

So we have a Racket list. The next function in the pipeline is the Racket list length function, and boom, number of files in the directory.

There’s some convenience pipeline operators too: echo "foobar" |> port->string |> print can be written as echo "foobar" |>> print: |>> automatically converts incoming port results to a Racket string.

Emacs’s eshell can do a similar thing: mix bash pipelines with ELisp functions. So the trick isn’t totally new to me.

Second script: print capitalized versions of every file name in a directory

#lang rash
(require racket/port)

ls |> port->lines |> map (lambda (x) (string-upcase x))

So very interesting bit of code here: we use Racket’s map to iterate the resulting array.

But wait, map takes two parameters: the proc and the list, but you provide only the proc!?

Rash provides a bunch of magic to auto expand the pipeline arguments and inserts the current pipeline argument at the end of the argument list you’re filling out.

We can make this explicit like so:

ls |> port->lines |> map (lambda (x) (string-upcase x)) _

What if the data has to be not at the end of the argument list? Move the _

ls |> port->lines |> map _ (lambda (x) (string-upcase x)) <— will error, but will error in a fun way that lets you know Racket really wants a procedure in that first parameter spot.

See How does the underscore work? for some more details.

Third script: no thank you, jq

#lang rash
(require json)
(require json-pointer) ;; from https://pkgs.racket-lang.org/package/json-pointer

echo '"{\"foo\": \"bar\", \"bar\": \"baz\"}" |>> string->jsexpr |> json-pointer-value "/foo"

returns bar.

That ' in front of the string is to prevent Rash from expanding those {} globs.

We use Jesse Alama’s json-pointer package and grab our value out.

Really after we’ve done string->jsexpr we have a Racket json object we could do anything with.

A Side Journey: A more Lisp redesign of capitalizing file names

If we revisit - almost - one of the first examples, let’s capitalize every folder name in the current directory. (We’ll implement code that’s equivalent to ls |>> string-upcase, which unlike our previous example, upcases the entire output string and not item by item)

Except this time we’ll use more Racket standard library provided paradigms:

#lang racket
(require racket/system)
(require racket/port)
(require threading) ;; from threading-lib (needs installation too)

( ~> 
  (with-output-to-string (lambda () (system "ls")))   
  string-upcase) ;; upcases the entire output

The ~> parameter gives us a similar looking syntax as | does in Bash: as I’m already thinking “output result pipes to next step in the pipeline” this maps to my mental model really well right now.

Calling Racket procedures in Rash pipelines

Given a long Rash script you may want to break the shell work into Racket procedures. This is, in fact, relatively easy - even if those procedures include Rash code themselves!

#lang rash

(define (my-function x)
  (rash #<<here-rash-delimiter
     echo $x

here-rash-delimiter
))

pwd |>> my-function

Our variable x is passed via the |>> operator, and we can access it - or any other Racket variables - using the same syntax for shell script variables!

Here we’re using a Racket here-doc to create the required code string: we could add multiple lines of Rash code inside this function.

Mixing modes

That (rash #<<here) pattern is kind of annoying. There’s a better way to do it!

In a Racket function you can use the #{} form to enter Rash mode again

(define (better-my-function x)
  #{
      echo $x
   }
)

So much better than our previous implementation!

If you are in Rash mode already you can enter Racket mode easily too:

echo (+ 1 1)

Here’s an example where we grab the output of a Rash command, shove it into a Racket variable, then use that in a Rash command

(define uptime #{uptime})
echo $uptime

We’re done coding! Time to package this for distribution!

For very simple scripts we can run Rash scripts by calling the Racket CLI

$ racket my_script.rkt, or using a #! line works great.

In instances where I can’t know I’ll have Racket pre-installed, Racket does have bundling and cross-compiling features.

raco exe ++lang rash my_file.rkt bundles everything up so it can be installed on another machine.

For me this created a 56MB bundle: I’m not sure if there’s something I can do to reduce the size, but that’s what I got!

Conclusion

We can go from simple Bash script to running it in Racket, iterating on the script easily and as we lean into the tools provided by Racket, and then we can distribute them.

Pretty neat!