Developer Guide

Middleware

Middleware is custom Node.js code written by you or your users or sourced from NPM that executes right before their extension code is run. Developers already using middleware in Express applications will find this familiar. It allows you to augment the default extension execution logic supported by Extend without requiring any changes in the code of your users. It also allows for “pipelines” of middleware that execute one after the other.

Most common situations where middleware is useful include:

  1. Authorization. Middleware allows you to implement authorization and access control logic for calls to extension endpoints. For example, the Creating Extensions section recommends using a pattern of securing extension endpoints that builds on middleware.

  2. Logging. Middleware allows you to build a custom logging solution. For example, you can intercept and capture all calls to console.log made by extension code to filter or export it to an external system.

  3. Context enhancement. You can use middleware to provide extension code with additional data or functionality beyond what is provided by default. For example, you can pre-fetch data from a database or make a function that fetches data available for calling from the extension code.

Middleware for an extension is determined via the wt-middleware metadata property. The value can be one of three things:

  • A URL where the middleware is hosted. It is important to note that this URL should return the source of the middleware and not actually execute it. In other words, if you open the URL specified here in your browser you should see the JavaScript code for the middleware. This is different from a regular extension URL where opening that URL in your browser would actually execute the code versus returning it.
  • A NPM module where the middleware function will include it by running require('npm_module_name')().
  • An export from a NPM module in the form of: npm_module_name/exportName. The middleware function will be obtained by require('npm_module_name').exportName().

Another important facet of middleware is that the function you write must return a function. In other words, you’re going to write a function that will return the middleware that will then be used by your extensions.

Let’s look at a middleware example that adds a simple authentication check to an extension. Consider the code below:

'use strict';

const AUTH_SECRET_NAME = 'wt-auth-secret';

module.exports = createAuthenticationMiddleware;

function createAuthenticationMiddleware() {
    return function authMiddleware(req, res, next) {
        const ctx = req.webtaskContext;

        if (ctx.secrets && ctx.secrets[AUTH_SECRET_NAME]) {
            const match = (ctx.headers['authorization'] || '')
                .trim()
                .match(/^bearer\s+(.+[^\s-])\s*$/i);

            if (match && match[1] === ctx.secrets[AUTH_SECRET_NAME]) {
                return next();
            }

            const error = new Error('Unauthenticated extensibility point');
            error.statusCode = 401;

            return next(error);
        }

        return next();
    };
}

First make note of how the function createAuthenticationMiddleware is returning a function (authMiddleware). As described above, the code needs to return the middleware function that will be used.

The middleware looks for a secret named by the constant AUTH_SECRET_NAME. If it exists, then the value of that secret must be passed in via the Authorization header. So for example, an extension would add a secret named wt-auth-secret and give it a value like catsAreBetterThanDogs. When this middleware is applied to an extension, the caller will need to pass in an Authorization header that matches that value.

In order to use this middleware, it must be placed online and available over HTTPS. You can find this particular example on GitHub here: https://github.com/auth0/webtask-userspace/blob/master/packages/bearer-auth-middleware/index.js. The corresponding raw URL may be found here: https://raw.githubusercontent.com/auth0/webtask-userspace/master/packages/bearer-auth-middleware/index.js. This is the URL you would use as your middleware.

Then to associate middleware with the extension, you can use this command line:

wt create someCoolExtension.js --middleware https://raw.githubusercontent.com/auth0/webtask-userspace/master/packages/bearer-auth-middleware/index.js

Compilers as Middleware

While it may be useful to think of middleware in terms of Express, essentially “run this before my code”, you can build far more powerful applications by using middleware as a compiler. Compilers, written as middleware, allow you to completely transform how your extensions are written.

By using this feature, your extensions can be written as:

  • Node.js scripts with custom programming models (i.e., not one of the three standard programming models
  • Other programming languages such as T-SQL or C#
  • CoffeeScript, TypeScript
  • Pug, EJS
  • or anything else that can be transpiled to one of the three basic programming models

A compiler written in middleware is possible because the string literal of the extension task is actually passed to the middleware. Since you have access to the code, alternative compilers may be applied against it. Two arguments are available off the req.webtaskContext.compiler object in your middleware to help with this:

  • script: This is the raw string of the extension code. It may be regular JavaScript, or any other language you wish to compile.
  • nodeJsCompiler: This is a Node compiler that can be used to transform the script into an executable function.

Creating a compiler

Creating a compiler in middleware follows the same “rules” as a regular middleware script. You must publish the code either as an NPM module or as a URL.

Let’s consider a simple example of a compiler that simply transforms the webtask’s source code into upper case.

module.exports = function() {

    return function(req, res, next) {
        //get the source of the original webtask
        let code = req.webtaskContext.compiler.script.toUpperCase();
        // return the string
        return res.end(code);
    }
}

As the comments say, the function reads in the source code of the extension and then simply upper cases it. Now imagine this extension:

module.exports = function helloCompilers(name) {
    return `Hello, ${name}`;
}

If we create this extension via the CLI and assign the middleware, executing the task will return:

MODULE.EXPORTS = FUNCTION HELLOCOMPILERS(NAME) {
    RETURN `HELLO, ${NAME}`;
}

This is a pretty useless example, but take a look at the extension code used. It has one argument, name. This is not a “standard” Extension programming model but this is where compilers can be truly powerful. Let’s look at an updated version of the compiler that will make this extension work properly.

module.exports = function() {

	return function(req, res, next) {
		//get the source of the original webtask
		let code = req.webtaskContext.compiler.script;

		req.webtaskContext.compiler.nodejsCompiler(code, function(error, func) {
		  
			if(error) return next(error);

			//grab name from the query string
			let name = req.webtaskContext.query.name;

			//func is the compiled code of the extension, so we can pass name in
			let result = func(name);
			return res.end(result);
		  
		});
	  
	}
}

As before, the compiler grabs the source of the original extension. But this time it makes use of the passed in Node.js compiler to compile the source string into a function (func). When it has this, it can then find the name value in the query string and pass it directly to the function. This compiler example is a bit brittle, but it illustrates how compilers allow you to build completely unique extensions. This would allow your users to write extensions in simpler programming models. You could also provide additional data for their use and pass it up their code.

A Debugging Tip

Normally console messages and other debugging information is not visible from code run in a middleware script. In order to see that information, add the following meta definition:

wt-debug=wt-middleware

A Note about Middleware and Dependencies

When building middleware that requires an NPM dependency, you will most likely run into an issue when your extension is executed. If you deploy your middleware as a simple URL endpoint for the source, and if that code requires an NPM module, your extension will not correctly run unless you add that dependency to the extension itself. This “fixes” the issue but is less than ideal. It requires you to add things to your extension that are not directly related to it. If the middleware were updated in the future, then your extension could cease to work, or load NPM modules it no longer needs.

However - if you use middleware hosted on NPM, then this is not a concern. Any dependency will simply work as is. Because of this, it is generally recommended to use middleware from NPM versus hosting your code at a URL.

A Note about Middleware Hosting and Webtask.io

If you decide to host your middleware online, then you may run into an issue if you make use of the Gist from GitHub. Gists are easy to create and give you the ability to serve up the JavaScript source of your middleware, but every time you modify your code, the URL for the “raw” version of the code changes as well. This can be a bit problematic while developing. An alternative would be to use a Webtask that returns the source code of the middleware as a string. Here is an example based on the very first middleware shown above:

const middleware = `
'use strict';

const AUTH_SECRET_NAME = 'wt-auth-secret';

module.exports = createAuthenticationMiddleware;

function createAuthenticationMiddleware() {
    return function authMiddleware(req, res, next) {
        const ctx = req.webtaskContext;

        if (ctx.secrets && ctx.secrets[AUTH_SECRET_NAME]) {
            const match = (ctx.headers['authorization'] || '')
                .trim()
                .match(/^bearer\s+(.+[^\s-])\s*$/i);

            if (match && match[1] === ctx.secrets[AUTH_SECRET_NAME]) {
                return next();
            }

            const error = new Error('Unauthenticated extensibility point');
            error.statusCode = 401;

            return next(error);
        }

        return next();
    };
}
`;

module.exports = function(ctx, req, res) {
  res.writeHead(200, {'content-type': 'application/javascript'});
  res.end(middleware);
};

Basically the source code is contained within one string (in this case, a template literal) that is than returned by the webtask as plain text (although note the header used to specify that the text is JavaScript). This method allows you to modify your middleware at will without having to worry about the URL changing. The drawback is that this is a bit harder to write and debug.