Operations Guide

Enabling External Logging

Your Extend installation can be configured to export logging information to your own AWS Firehose service. This provides you with the capability to store and analyze logs generated by your users’ extensions in Amazon S3, Amazon ElasticSearch, Amazon Kinesis Analytics, or Amazon Redshift. The logging information contains all the output generated by your users’ extension code to standard output, as well as selected infrastructure events generated by Extend itself.

Extend uses bunyan to format the logs exported to AWS Firehose. Logs are a stream of individual JSON objects separated with a newline.

The fields contained in every JSON record vary, but some of the most important ones include:

  • time: server time of the event.

  • level: log level: 30 - info, 40 - warning, 50 - error.

  • msg: description of the event.

  • req_id, deployment_key: A unique request ID of the execution request within your Extend installation. If you are contacting support with an issue to investigate, this information will help zero-in on the transaction in question.

Customizing Behavior with Settings

Behavior of your Extend installation is controlled with a large number of installation-wide settings. The most commonly adjusted settings are listed below.

  • inactivity_timeout (default: 30 seconds): time after which a webtask container with no new incoming execution requests is subject to recycling. This setting also implies the longest lifetime of any single webtask execution you are guaranteed. You may want to increase this if your webtasks take longer than 30 seconds to run.

  • life_timeout (default: 20 minutes): maximum time after which any webtask container (active or not) is subject to recycling. You may want to decrese this timeout as a mitigation of unintended leaks in your users’ code or any of the modules they depend on.

  • max_code_length (default: 100KB): Maximum length of webtask code. You may want to increase this to accommodate larger webtasks.

  • quarantine_timeout (default: 5 seconds): Time span during which a container that was improperly terminated by the webtask code (e.g. due to uncaught exception) is put under quarantine. During this time, all requests targeting that container (e.g. requests for that tenant) will be rejected with HTTP 429. This timeout prevents badly written user code that executes frequently from exhausting system resources.

  • tripwire_timeout (default: 2 seconds): Maximum time user code is allowed to block the Node.js event loop before it is considered run-away and terminated. This prevents run-away user code from exhausting computing resources. You may want to increase this in specialized cases of CPU-intensive workloads. NOTE run-away code will put the container it is running in under quarantine when it is terminated.

  • cron_max_jobs_per_container (default: 10): Maximum number of CRON jobs that can be created in a single container.

There are many more configuration options that address advanced scenarios and specialized workloads.

Image customization

All extensions in an Extend installation execute in a uniform execution environment. The environment includes built-in support for any NPM module from the public NPM repository. In cases when you need to add modules from a private NPM repository, or custom native components, you can use the image customization feature.

Image customization allows you to add custom components on top of a base Docker image we provide, and use the new Docker image as the execution environment for your extensions. The process involves the following steps:

  1. Auth0 provides you with access to an AWS ECR repository from which you can obtain the base Docker image (currently based on Ubuntu 16.04).

  2. You customize the image by creating a derived Docker image with additional components and modules.

  3. You push the new image to another AWS ECR repository Auth0 provides you access to.

  4. Auth0 performs a set of validations on your new image to ensure integrity and puts it in production in your installation. This process does not involve any downtime.