How to quickly iterate on your private Node libraries

You’re fixing a bug and turns out the problem is in some private library of yours! You fix the bug in the library, but are you sure ? How do you test it in your application locally!??

First, a word about Node module resolution

In general Node module resolution traverses up the directory structure, looking at node_modules folders it finds in order to resolve references. Source.

When a library author packages their NPM module up for distribution the specified files and their package.json are packaged up into .tar.gz files by npm publish. Only module specific files are in this package: dependent libraries are declared by the library’s package.json and downloaded at install time.

To take a slight detour for those unfamiliar with Node, it has a relatively simple approach for installing libraries: in the simplest case just putting them in a node_modules folder in the current working directory. Files loading libraries involve traversal of that node_modules folder mentioned above.

Modern versions of NPM look for situations where dependencies can be shared. Modules are “hoisted” up the file system hierarchy, in such situations. For example, both my-library and Library B both depend on the same version of dep-lib then hoisting occurs, but if different versions are kept apart (in the node_modules folder created in that library’s folder).

Example folder structure:

app/
    node_modules/
	   dep-lib (v1)
	   my-library/
	     node_modules/
		   dep-lib/ (v2)
	   library-b/  <-- requires dep-lib v1
	   another-dependency/  <-- requires dep-lib v1

Files in my-library will resolve dep-lib dependencies (v2) because those are the closest node_modules folder.

Most of the time Node abstracts this whole “where exactly are the files on disk?” care from us, but sometimes it matters. npm link may be one of those times.

Building and testing your own library code

In the directory structure above we have ‘app’ which uses ‘my-library’.

We’ll call app a “consumer application”: meaning it uses my-library for some purpose.

My-library is an internally created library, and during creation you should test your library inside an app. How then can I develop, and test, changes to my-library?

NPM provides npm link, which creates a symlink from the package’s installation directory to where the library lives on disk. Library files aren’t moved and a series of symlinks means NPM’s module resolution code works without change, while your my-library source lives outside app/’s node_modules folder.

npm link and node_modules

This comes with a very subtle behavior. Since npm link declarations are just symlinks to the folder on disk that means the first node_modules folder it finds will be the node_modules from the library, not the application. These even include peer or dev dependencies.

Sometimes that matters, especially when using peer deps. (And sometimes when using Typescript!)

Away from the abstractions that hurt us, like the fox runs swiftly

The only time we can get away from this problem - of extra node_modules being there for maybe unwanted resolution - is when the library is published to a registry and pulled down: as mentioned previously it won’t contain node_modules, as those are resolved during install time.

Can we make a quick, local, iteration loop somehow?

Local builds of my-library

We want a local NPM registry, or something close enough to one. In fact, there’s multiple ways to refer to a package source in NPM: by semantic version or local paths.

The “local path” can be other, unexpected, things:

• a direct URL to download a .tgz file. (Hint: You could stand up a local web server and have it serve your package, ask me how I know…)
• a file reference to a .tgz file. Which will then be extracted into our consumer application’s node_modules folder

Let’s make a temp-artifacts folder and point NPM to it!

To do so I created the following NPM script, in my-library

"build:local-registry": "npm run build && npm pack && mv my-library-1.0.0.tgz temp-artifacts/",

This script builds the module, packs the module into a .tar.gz file (normally a step automatically called by npm publish) and shove it into a directory.

Using this build in a consumer application

During development, declare the dependency on my-library like so

"my-library": "file:../my-application/temp-artifacts/my-library-1.0.0.tgz",

Npm install should work!

Refreshing new versions is mildly annoying, as npm install has a cache that it’s pretty insistent about using. Npm uninstall seems to be the only way to force NPM to get the uncached version of the thing BUT that also removes that from package.json. Need to counteract that, so I wrote an NPM script:

"install:refresh-my-library": "npm uninstall my-library && npm install --save ../my-application/temp-artifacts/my-library-1.0.0.tgz"

So, what does the workflow look like?

$ cd my-library
$ emacs my-library
  .... waiting for Emacs to close, you're doing good typing....
$ npm run build:local-registry
$ cd ../my-application
$ npm run install-refresh-my-library
$ echo "look, new modifications are available!"

Of course, the embedded version number is a bit annoying to update.

Conclusion

Most of the time you don’t need this information! Sometimes you may really need it. I was working on a project recently where a module would look for copies of itself in the Node consumer-app, and if found the library would throw an exception. I’ve also seen the Typescript compiler get very confused when compiling against two different versions of the same library (the one found in my-library’s node_modules folder, and the version found by the rest of the program in the consumer applications node_modules` folder.

Alternatives

npx link also looks useful! The author found even more reasons to not trust npm link and npx link’s publish fixes the node_modules resolution problem in a very clever way (using hard links, which also avoids the hard link problem).