Tuesday, September 27, 2016

Magic of require extensions

Magic of babel register


Some development tools are like magic. Babel require hook, at least to me, is such a tool. Not only does it convert the flashy ES6 and ES7 to plain old ES5, which itself is quite awesome, it does so on the fly. You make one require call to babel-register and then require any js file with ES6/ES7 code and it just works!

I needed to do something like what it does at work. So to learn how it work I peeked into babel-register's source. It's very small. The the transpiling stuff is done by babel core so they were outside the source. I was after the stuff that does the require magic and within minutes I found what I was looking for.

Require extensions.


Require extensions allow you to define a function describing what to do when a file with a given extension is required. Can't get more flexible than that!

Here is a way to extend require to text files.

fs = require('fs');

// Now you can require '.txt' files
require.extensions['.txt'] = (m, filename) => {
  m.exports = fs.readFileSync(filename, 'utf8');
}

const text = require('./test.txt')
console.log(text); // Cool!

This is cool! But..

It's deprecated.


In fact its been deprecated for a while. Node versions as early as 0.10 documents it as deprecated. But it has survived close to 3 years and many versions to appear in node version 6 as well. Even the documentation admits that its unlikely to go away.
Since the Module system is locked, this feature will probably never go away.
But it also say,
However, it may have subtle bugs and complexities that are best left untouched.
I don't know about its internal bugs. But complexities might occur because it may compel developers to publish non-javascript packages for javascript projects. For example someone can write the entire source of their package using TypeScript. And only in the entry point to the package use require extensions to register a require extension for .ts. This handler can use a transform function to compile TypeScript to javascript dynamically.

I can see two reasons why this is bad.
  • Require extensions is global.
The .ts extension handler set by this package could be over-written by another package that use TypeScript. 
The second package could be using a compiler for a different version of TypeScript. Now that compiler will try to compile the first package's TypeScript sources and will break it!
  • Compilation unnecessarily takes time
When an application developer install the package written in TypeScript he/she will be compiling it every time their app is run. But the packages source won't change. So they will be compiling the same source files over and over again.
If the source is precompiled and published these problems does not occur.

Its not hard to understand that using require extensions this way should be avoided. But what about using it for development? Development is where we use babel-register. Development is where I needed it too.

My Use-case


Many projects run tests for front end code in node. Many projects use webpack to compile front end code from jsx and ES6/ES7 stuff to plain js. With webpack we can use special require calls that invoke webpack loaders. For this reason test tools like jest, enzyme and others need some special configuration to work with webpack. In kadirahq's storyshots project we provide an easy way for these projects to run snapshot tests. For more info read its introduction on Kadira voice. In storyshots we have a simple setup that use babel-register, and we needed it to work with webpack loaders.

So if we want a to run the front end code on node we have to run webpack on node. Tests are run often so should run fast. Running webpack and saving a file and then requiring it again takes time. It's common for a webpack build to take around 5 seconds. That is what we tried to avoid with require extensions.

We made a substitute for webpack loaders in a few lines using require extensions.

const loader = loaders[ext];

require.extensions[`.${ext}`] = (m, filepath) => {
  m.exports = loader(filepath);
};

The loader function mimics some loader for the file extension ext.

For example following mimics the url loader for jpgs.

loaders['jpg'] = filepath => filepath;

If we consider css content is not important for our tests, because say we only need to test if we add the correct css classes at correct places, we can ignore css with a loader function like following.

loaders['css'] = () => null;


Useful enough to take a measured risk


Transpilers are put to heavy use these days with lots of react and ES6 development going on. If dared to be used, require extensions could help to develop more magic-like tools like babel-register.