How to quickly iterate on your private Node libraries
You’re fixing a bug and turns out the problem is in some private library of yours! You fix the bug in the library, but are you sure ? How do you test it in your application locally!??
First, a word about Node module resolution
In general Node module resolution traverses up the directory structure, looking at
node_modules folders it finds in order to resolve references. Source.
When a library author packages their NPM module up for distribution the specified
files and their
package.json are packaged up into .tar.gz files by
npm publish. Only module specific files are in this package: dependent libraries are declared by the library’s
package.json and downloaded at install time.
To take a slight detour for those unfamiliar with Node, it has a relatively simple approach for installing libraries: in the simplest case just putting them in a
node_modulesfolder in the current working directory. Files loading libraries involve traversal of that
node_modulesfolder mentioned above.
Modern versions of NPM look for situations where dependencies can be shared. Modules are “hoisted” up the file system hierarchy, in such situations. For example, both my-library and Library B both depend on the same version of
dep-lib then hoisting occurs, but if different versions are kept apart (in the
node_modules folder created in that library’s folder).
Example folder structure:
app/ node_modules/ dep-lib (v1) my-library/ node_modules/ dep-lib/ (v2) library-b/ <-- requires dep-lib v1 another-dependency/ <-- requires dep-lib v1
my-librarywill resolve dep-lib dependencies (v2) because those are the closest
Most of the time Node abstracts this whole “where exactly are the files on disk?” care from us, but sometimes it matters.
npm link may be one of those times.
Building and testing your own library code
In the directory structure above we have ‘app’ which uses ‘my-library’.
app a “consumer application”: meaning it uses
my-library for some purpose.
My-library is an internally created library, and during creation you should test your library inside an app. How then can I develop, and test, changes to
NPM provides npm link, which creates a symlink from the package’s installation directory to where the library lives on disk. Library files aren’t moved and a series of symlinks means NPM’s module resolution code works without change, while your my-library source lives outside
npm link and
This comes with a very subtle behavior. Since npm link declarations are just symlinks to the folder on disk that means the first
node_modules folder it finds will be the
node_modules from the library, not the application. These even include peer or dev dependencies.
Sometimes that matters, especially when using peer deps. (And sometimes when using Typescript!)
Away from the abstractions that hurt us, like the fox runs swiftly
The only time we can get away from this problem - of extra
node_modules being there for maybe unwanted resolution - is when the library is published to a registry and pulled down: as mentioned previously it won’t contain
node_modules, as those are resolved during install time.
Can we make a quick, local, iteration loop somehow?
Local builds of my-library
We want a local NPM registry, or something close enough to one. In fact, there’s multiple ways to refer to a package source in NPM: by semantic version or local paths.
The “local path” can be other, unexpected, things:
- a direct URL to download a .tgz file. (Hint: You could stand up a local web server and have it serve your package, ask me how I know…)
- a file reference to a .tgz file. Which will then be extracted into our consumer application’s
Let’s make a
temp-artifacts folder and point NPM to it!
To do so I created the following NPM script, in
"build:local-registry": "npm run build && npm pack && mv my-library-1.0.0.tgz temp-artifacts/",
This script builds the module, packs the module into a .tar.gz file (normally a step automatically called by
npm publish) and shove it into a directory.
Using this build in a consumer application
During development, declare the dependency on my-library like so
Npm install should work!
Refreshing new versions is mildly annoying, as npm install has a cache that it’s pretty insistent about using. Npm uninstall seems to be the only way to force NPM to get the uncached version of the thing BUT that also removes that from package.json. Need to counteract that, so I wrote an NPM script:
"install:refresh-my-library": "npm uninstall my-library && npm install --save ../my-application/temp-artifacts/my-library-1.0.0.tgz"
So, what does the workflow look like?
$ cd my-library $ emacs my-library .... waiting for Emacs to close, you're doing good typing.... $ npm run build:local-registry $ cd ../my-application $ npm run install-refresh-my-library $ echo "look, new modifications are available!"
Of course, the embedded version number is a bit annoying to update.
Most of the time you don’t need this information! Sometimes you may really need it. I was working on a project recently where a module would look for copies of itself in the Node
consumer-app, and if found the library would throw an exception. I’ve also seen the Typescript compiler get very confused when compiling against two different versions of the same library (the one found in
node_modules folder, and the version found by the rest of the program in the consumer application
npx link also looks useful! The author found even more reasons to not trust
npm link and
publish fixes the
node_modules resolution problem in a very clever way (using hard links, which also avoids the hard link problem).