Intro

A mid-2010s style MERN stack separates out backend and frontend concerns into separate deployable units (which we’ll call “projects” here), resulting in an architecture that looks something like this for a consumer web app with admin backend:

In this diagram we have four projects: “Consumer Microservice”, “Consumer Frontend”, “Admin Microservice” and “Admin Frontend”. The Consume Frontend project only talks to the Consumer Microservice, and the Admin Frontend only talks to the Admin Microservice.

The initial idea of this general architecture (seperate frontend and backend projects) is to be able to separate out responsibilities in a scaled up environment: keep your team of front-end developers separate from your team of backend developers. This may or may not pan out in practice, but this architecture is pretty much the standard even in very tiny startups today.

On a developer’s machine (and, increasingly, in production) all those projects run servers to (potentially) transpile code, reflect developer made changes, talk to downstream sources of record, etc.

This architecture means everything at the top of the graph depends on the other layers along its path to the bottom of the graph. If you’re working on the “Consumer Frontend” project you need to run the “Consumer Microservice” project too. Why? Because the “Consumer Frontend” project is making requests of the “Consumer Microservice” to do things like retrieve customer data, perform whatever customer thing you do, etc.

So in order to develop locally the frontend developers need whatever downstream microservice their project requests, in order to have a working environment at all!

In this example it’s worth pointing out here that we have two groups (or cliques) of things now: the consumer projects and the admin projects. The admin stuff doesn’t depend on the consumer stuff, and the opposite is true too. These seperations could be explicitly planned out from the beginning using (for example) domain driven design’s idea of bounded contexts or evolve naturally (the product list services don’t need to talk to user login flow - nobody planned it, but it just doesn’t happen).

A large company may have development environments running partially or totally in the cloud, or Docker Compose setups to give you some or all of the microservice herd on your machine, with clever mocking services to draw boundaries between cliques/bounded contexts. In small shops you don’t have any of that. But still, the problem remains…

The problem: How do I boot up my project’s dependencies automatically?

If I’m doing frontend work, I don’t want to care that I need to do to boot up the microservice with all the data: I just want to quickly make it go. Also, it would be nice to not have to spawn off two Terminal windows to do it.

It’s also not just booting up the backend microservice, it’s made worse by the fact that backend microserivces usually need extra stuff. Thinking back to my Rails days I would often need the database, sometimes a memory store, maybe a worker queue, and sometimes something else (an AWS mock?)

Now, in the past, I’ve put together complicated solutions using Docker Compose, but sometimes that’s an investment I can’t make: nobody will use it, or we have company provided laptops so it’s pointless… sometimes you need simplicity - the kind of simplicity you can later replace with complexity (ideally without major rework!) once you need it to be more complex.

Reintroducing Foreman

Foreman sprang out of Heroku: you feed it a Procfile, where each line has a name and the command to execute to do that thing. It the common scenario it executes every line, in parallel, until any process quits.

Example Procfile

web: npm run start

This launches one process, labelled “web”, and runs the “npm run start” command, running whatever Node server that runs.

A more complex Procfile:

web: npm run start
mongo: mongodb

Runs that same Node process AND ALSO a Mongodb database. Foreman runs both of those processes at the same time, exiting if either one of them quits.

Taking Foreman to the next level: Introducing the Foreman Foreman concept

Let’s say you have the following directory structure for your herd:

consumer_microservice/
consumer_frontend/
admin_microservice
admin_frontend/
user_mailer/

As a developer, if I go into the consumer_frontend folder and run that app it’ll probably crash soon after, expecting the consumer_microservice to also be running. But I forgot! Oh no!!

How can I prevent this problem?

The answer: Have Foreman call Foreman. (Say it three times in front of a mirror)

First, put Procfile in each project. Its first responsibility is to launch whatever server you need to run for each project. Its second responsibility is to launch whatever dependencies it might have.

So, now the Procfile for consumer_frontend:

web: npm run start
cm: sh -c 'cd ../consumer_microservice && foreman start'

The web label runs the frontend server (providing developer reloading or whatever). The cm label cds into the consumer_microservices folder and runs foreman on the Procfile it finds there.

In this case consumer_frontend only depends on consumer_microservice but there could be more!

The Procfile for consumer_microservice:

srv: gradle runLocal
database: mongodb
user_mailer: sh -c 'cd ../user_mailer/ && foreman start

With this setup consumer_microservice will launch itself! It has two dependencies: a mongo instead and another microservice (whatever the user_mailer service does!)!

We follow the trails of dependencies down the line until we run out of nodes in our dependency graph: not just the direct dependencies of consumer_frontend but the transitive dependencies down the graph.

So: Foreman calling Foreman. Whatever that Foreman down the line does it not any concern of the foreman at the top of the line.

Foreman Foreman Problems

This concept works with very small herds! Theoretically this approach could scale up to as many as you want. Practically I wouldn’t want to try this with more than maybe 6 projects? Because there’s one thing this approach doesn’t handle: dependencies declared more than once in the whole graph. Given a large enough herd of projects you’ll eventually hit a loop (where microservice B and C both try to launch on microservice A, and that’ll cause conflict trying to launch the same project with the same port setup twice). This technique does not handle that, so that would stop you.

I’d highly suggest graphing/analyzing your microservice dependencies first. Either use computers (as illustrated in that previous blog entry) to solve the problem or sit down with a pad of paper for an hour and draw it out. If you can do that and your graph contains no projects depended on by multiple projects then maybe this is a match for you!

In fact, this approach offers cheap code-as-documentation for your projects, and in a compact manner: I know X depends on Y because the Procfile launches it. They don’t have to retrace parts of the code that call out: it’s all right there in one location! Great for small projects!

Conclusion

I was pretty amazed when I was able to get Foreman to not just launch a project’s direct dependencies, but also its transitive dependencies, through a primitive form of recursion. here’s to good quality tools that let you extend them in unexpected ways!