Spencer Hawkins is a Software Automation Engineer by day, and the lead developer at MTGATracker the rest of the time. He graduated from UT Dallas with a B.S. in Computer Science and has a passion for games and building tools that people want to use. When he’s not working on MTGATracker, you might find him in a draft or sealed MTG event at a local game store in Seattle.

In this guest post, he covers a bit of his design philosophy, his serverless workflow and demonstrates how he does continuous deployment including test-driven development with webtask on Webtask.io.

Spencer’s success with this model has led him to become one of the first Extend Starter Plan customers and we are happy to have him.


MTGATracker Inspector

MTGATracker is a native electron application that helps users track their decks, collections, and game statistics in Magic: The Gathering Arena. It also includes a web-based application called Inspector which users can use to gain better insights into their various decks’ win/loss ratios and their overall play styles.

It’s ok if you don’t know any of those words because we’re not going to talk much about games in this post.

Instead, we’ll talk about how we used Webtask.io to create a serverless application that’s scaled out to hundreds of unique users, calling thousands of tasks per day (and growing).

We will also showcase a selection of tools that helped us accomplish all of this over the span of a few months before the game even left closed beta.

Finally, we’ll explore how to use Pytest and test-driven development on webtask projects to help you deploy changes fearlessly; even though they aren’t primarily Python!

Let’s Brag A Little

MTGATracker usage stats over its first month in action

MTGATracker usage stats over its first month in action. Graph generated April 5th, 2018

MTGATracker is still super new to the world, and even to us. We’ve been working on it for just a few months, and only put out our first release on March 22nd, 2018.

Since then, we still have little private celebrations each time we hit new made-up milestones:

Now, we’re well on our way towards 100k tracked games! We’ve gotten here without ever once worrying about server downtime for upgrades, or exceeding our server’s capacity.

So How Do We Do It?

Think less "serverless," and more "think less about the servers." - @Spencatro

(and say it 5 times fast)

If you’re not already in the loop, Serverless Architecture might sound too good to be true. Indeed, it is a bit of a misnomer. I promise though, “Serverless” isn’t just something you throw in front of a random buzzword and then pretend it’s a startup–I’m looking at you, blockchain.

Serverless is a phrase often used to encapsulate the idea of FaaS (Function as a Service), which is another fancy way of saying, “Take this chunk of code and just run it. Also make it so I can run it anywhere, anytime, as many times as I want. And also, do it right now.”

No more worrying about:

  • Provisioning servers for your worst-case spikes
  • Balancing loads across your clusters
  • Digging through mazes of some web service provider’s configuration pages

Of course, there is some machine somewhere running your code, but in this vein, the term “Serverless” is more about not having to think about that server your code runs on, rather than there not actually being one.

Serverless architectures are very popular for developing handlers for “Webhooks,” or small HTTP endpoints usually triggered from chat programs like Slack, Discord, or Gitter. And it’s excellent at it!

But the power of serverless is also so much greater than that of a chat toy. MTGATracker successfully uses a serverless architecture to power an application used by hundreds of unique visitors, calling tasks nearly thousands of times per day.

The Incredible Speed of Webtask.io Development

Writing your first “Hello World” webtask will take you minutes. No, this isn’t an exaggeration, check out this quick start.

MTGATracker uses Express-style webtasks; similar to the Express with MongoDB template. It is a great starting point if you’re not sure which to pick.

To drive home how fast webtask development can be though, I’ll start with the Express template which is just a little simpler.

Once created, we will see some code like this in the editor:

var express    = require('express');
var Webtask    = require('webtask-tools');
var bodyParser = require('body-parser');
var app = express();

app.use(bodyParser.json());

app.get('/', function (req, res) {
  res.sendStatus(200);
});

module.exports = Webtask.fromExpress(app);

This code will run right out of the box, seconds after created. In fact, if you look towards the bottom of the editor you will find a URL. Clicking it shows that the webtask is already published and available for requests!

https://wt-<your-container>-0.sandbox.auth0-extend.com/express-example

Now let’s prove how fast webtask really is for development. Adding a second endpoint to the webtask is as simple as adding the following code and clicking save:

app.get('/hello', function (req, res) {
  res.status(200).send({ hello: "Extend blog reader!" });
});

A loading bar shoots across the top of the screen, indicating not only that the webtask code saved; but also that again, in seconds, the new code is already deployed!

It can be accessed at the same URL, but ending with your new endpoint:

https://wt-<your-container>-0.sandbox.auth0-extend.com/express-example-blog/hello

Try getting this same, nearly instant development workflow out of any other server architecture!

Try getting this same, nearly instant development workflow out of any other server architecture! - @Spencatro

Enough Hello World

“Hello World” programs often make new technologies look better or simpler than they are. But even more mature webtask applications don’t take that much more code or setup! Let’s look at a real* endpoint from MTGATracker: on_demand.

* For clarity, the code in each file has been reduced in scope & modified for brevity but is otherwise genuine.

api/user-api.js

'use latest';

const express = require('express'),
      router = express.Router();
import { MongoClient, ObjectID } from 'mongodb';

// ... some endpoints omitted

router.get('/games/user', (req, res, next) => {
  console.log("/api/games/user/" + JSON.stringify(req.params))
  if (req.query.per_page) {
    let per_page = parseInt(req.query.per_page)
  } else {
    let per_page = 10;
  }
  const { page = 1 } = req.query;

  const { MONGO_URL, DATABASE } = req.webtaskContext.secrets;

  MongoClient.connect(MONGO_URL, (connectErr, client) => {
    const { user } = req.user;
    if (connectErr) return next(connectErr);
    let collection = client.db(DATABASE).collection(gameCollection)
    let cursor = collection.find({'players.name': user});
    cursor.count(null, null, (err, count) => {
      let numPages = Math.ceil(count / per_page);
      let docCursor = cursor.skip((page - 1) * per_page).limit(per_page);

      docCursor.toArray((cursorErr, docs) => {
        if (cursorErr) return next(cursorErr);
        res.status(200).send({
          totalPages: numPages,
          page: page,
          docs: docs
        });
        client.close()
      })
    })
  })
})

module.exports = router

mtga-tracker-game.js

'use latest';

import bodyParser from 'body-parser';
import express from 'express';
import Webtask from 'webtask-tools';
const ejwt = require('express-jwt');

const secrets = require('./secrets')
const userAPI = require('./api/user-api')
const { getCookieToken } = require('../util')

const server = express();
server.use(bodyParser.json());

let ejwt_wrapper = (req, res, next) => {
  return ejwt({ secret: req.webtaskContext.secrets.JWT_SECRET, getToken: getCookieToken })
    (req, res, next);
}

server.use('/api', ejwt_wrapper, userAPI)

// ... some routers omitted

module.exports = Webtask.fromExpress(server);

This code is slightly more complicated; we’re now also using webtask secrets, express middleware, an external database, and some javascript destructuring syntactic sugar. But the overall structure is still mostly similar, and the task of deploying is still just as fast and easy.

Webtask.io Is a Rich Environment

The Editor

The web based Editor comes with plenty of great features, including a built-in webtask runner, an npm module configuration tool, a secrets manager, and a real-time log stream viewer.

The Command Line Interface

Webtask is just as much of a joy to work with locally, thanks to tools like wt-cli. If you’re used to auto-reloading frontend tools, you’ll find yourself right at home with the webtask command-line interface.

Instead of writing code in the Editor, you can write your code locally, and (after doing some one-time login/setup), deploying to webtask happens in one quick command: wt create my-webtask.js.

By using the --watch option, webtask will watch your file(s) for changes and automatically deploy those changes as soon as you save them in your text editor.

It takes mere seconds to deploy new code!

Use Your Local Workflow

Using wt-cli also allows you to create a more sophisticated project structure. For example, you can use an npm-style package.json file to declare npm dependencies.

Or, if your project continues to grow, you can also spread your webtask across multiple javascript files, to be later bundled into a single webtask with the --bundle option.

Using wt serve, you can even serve your webtask locally for testing purposes.

Even Custom Domains!

Webtask is even so sophisticated that it will easily allow you to set up features that look and feel super-premium, for free. For example, for us, deploying webtasks to a custom domain was faster even than the time it took us to pick out our API’s new fancy domain. It’s really that easy!

Tons of Guidance

The Extend team and Auth0 have also created and published tons of useful additions to webtask, many in the form of npm modules.

  • Need a solution for generating JSON Web Tokens? Try node-jsonwebtoken.

  • How about pre-built express middleware that will verify those tokens? Express-jwt has you covered.

  • Maybe you’d even like to deploy a whole dedicated webtask that will help warn you if you ever accidentally commit your secrets to your public github repository? Give repo-supervisor a shot!

These tools and tons and tons of other npm modules are available within the webtask runtime. MTGATracker uses these tools & many more!

My Workflow is Test First

I can already hear you groaning but bear with me here. If you’re planning to distribute your application to thousands, or even hundreds of users, you should probably be testing it. Fear not! Writing tests can be easy, and rewarding!

I won’t spend too much time trying to convince you to do this, but know that there are few things in this world better than catching a critical bug with a test you wrote before writing the implementation; especially if you know it’s one you wouldn’t have caught before hitting production.

Testing your application can be easy and painless with the right tools, and give you the confidence to deploy as fast as webtask will let you. - @Spencatro

Since webtasks expose simple HTTPS API’s, testing your application can be easy and painless with the right tools, and give you the confidence to deploy as fast as webtask will let you.

Using the CLI, we can quickly spin up a local clone of the MTGATracker API:

wt serve . --hostname localhost --port 8080

Let’s look at a test for the same endpoint shown earlier. MTGATracker’s API is tested with pytest and the python requests module. Familiarity with python will help when writing pytest tests, but even python novices should feel pretty comfortable with the syntax we’ll be using.

import copy
import random
import requests
import string

ROOT_URL = "https://localhost:8080/mtga-tracker-game"

_game_shell = {
    "schemaver": 0,  # this will not be present on actual records
    "gameID": 0,
    "winner": "jenny",
    "opponent": "joe",
    "hero": "jenny",
}

def _random_string():
    return ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(10))

def post_random_game(hero=None, winner=None):
    game = copy.deepcopy(_game_shell)
    game["gameID"] = _random_string()
    if winner:
        game["winner"] = winner
    if hero:
        game["hero"] = hero
    post_url = ROOT_URL + "/anon-api/game"
    return requests.post(post_url, json=game).json()

def test_user_can_only_see_own_games(empty_game_collection):
    # empty_game_collection is a pytest fixture defined in
    # conftest.py that empties the staging database.
    # you can read more about fixtures here:
    #               https://pytest.org/latest/fixture.html
    post_random_game(winner="gemma")
    games = requests.get(ROOT_URL + "/api/games/user").json()
    for game in games["docs"]:
        assert game["hero"] == "gemma"

    post_random_game(hero="bobby", winner="bobby")
    post_random_game(hero="gemma", winner="gemma")
    post_random_game(hero="spencer", winner="spencer")
    post_random_game(hero="gemma", winner="gemma")
    post_random_game(hero="jane", winner="jane")
    post_random_game(hero="thomas", winner="thomas")

    games = requests.get(ROOT_URL + "/api/games/user").json()
    for game in games["docs"]:
        assert game["hero"] == "gemma"

Execution is simple:

python -m pytest . -v

The output will look similar to this:

============================= test session starts =============================
platform win32
  -- Python 3.6.4, pytest-3.5.0, py-1.5.3, pluggy-0.6.0
  -- C:\Users\Spencatro\PycharmProjects\mtga-tools\venv\Scripts\python.exe
cachedir: .pytest_cache
metadata: {'Python': '3.6.4', 'Platform': 'Windows-10-10.0.16299-SP0',
'Packages': {'pytest': '3.5.0', 'py': '1.5.3', 'pluggy': '0.6.0'},
'Plugins': {'metadata': '1.6.0', 'html': '1.16.1'}}
rootdir: C:\Users\....\webtasks, inifile:
plugins: metadata-1.6.0, html-1.16.1
collected 1 item / 0 deselected

test_staging_endpoints.py::test_user_can_only_see_own_games PASSED      [100%]

 generated html file: C:\Users\...\webtasks\pytest_report.html
=================== 1 passed, 0 deselected in 0.90 seconds ===================

If you build up a robust suite of tests, you will likely find yourself making complex, but perhaps necessary, refactors without fear. Your first application update may feel strange; like it shouldn’t be this easy. But if you’ve taken the time to test your application; relax! You deserve it!

Test Driven Development is simple: all you have to do is write the test before you write the code… and guess what? Your tests are now driving your development! Yeah, it’s that simple! If you are meticulous about this process and give your tests “human” enough names, you can even treat your tests as your spec. Two birds with one stone!

Anecdotally, while creating MTGATracker’s various API’s, I found more than one occasion where writing the test first ended up informing parts of the design I might not have thought of if I had just started writing the code. Putting the minimal effort up front to consider the bounds of your use cases & user stories can sometimes help you discover amazing things.

Automating the Whole Process

You can even take this concept of serving your webtask locally & running tests against it, and apply it within a continuous integration service, like Travis CI.

MTGATracker’s suite of post-commit tests has allowed us to make radical changes to our webtask project, like:

  • Reorganizing project file structure
  • Reorganizing our API
  • Migrating from an auth-less system to a state-of-the-art JWT system

All while deploying each new change either fearlessly within minutes of development or after resolving minor (or sometimes even major) regressions caught in testing.

What are you waiting for? Go give it a shot!

In this post we’ve covered how MTGATracker is using Webtask.io to back a multi-user application, serving nearly thousands of requests per day. We’ve shown some sample code to help you get started, and we’ve also demonstrated how you can leverage Test Driven Development to make application deployments easy and painless.

So, what are you waiting for? Go(Extend) and build something remarkable - and tell me about it in the comments!

From the Extend Team: This is our first guest post. If you are interested in writing a post about your experience working with Webtask.io or Extend, please reach out to us.