ToDo TDD

If you're a JavaScript developer you've probably seen TodoMVC but just in case you haven't I'll explain it to you quickly. TodoMVC helps you select an MV* framework by demonstrating how to develop a todo list app in a ton of them. It chooses a todo list app because that is complicated enough to demonstrate how the framework would look in a real world app, but simple enough to be understood by the reader.

What does this have to do with TDD? Well the number one complaint about learning TDD is that there are no "real world" examples. This is reflected when I teach and mentor others. The basics are understood easily, but then come the complaints:

"My code is complex!"

"I need to work with the database on everything!"

"You don't understand, our situation is unique!"

Ultimately this ends with, "I wish we could do this here."

All these complaints are true, but none of them are legitimate. Of course your code is complex, of course you use a database, and yes your situation is unique. If it wasn't - you wouldn't have a job! Nobody said TDD made coding easy. Yes your situation is unique and complicated, that's why you get paid the big bucks. That doesn't mean you have to throw every best practice into the garbage, and it definitely doesn't mean you can't use TDD.

What it does mean is that it's hard to learn TDD on toy examples and then turn around and use it in a Real World situation. That's why people complain about a lack of "Real World" examples. You don't just need the basics, you need examples you can use today.

As an author I can't work on your particular codebase1 but I can provide you with a real world codebase that looks like yours, and that's what TodoTDD is for. This is codebase that will have:

  • Tests that use the database.
  • Code that doesn't have tests.
  • Slow tests.
  • UI Code.
  • Probbly BUGS!

You'll write new code in this codebase. You'll be modifying the code to make it easier to work with, so you can write tests. You'll work in sections you don't completely understand. The idea being we want to simulate real life. This will be your own personal practice setup, where you'll be abe to do the right thing over and over again, so that you can do the right thing on the code you're getting paid for.

JavaScript?

First and foremost you should know I don't consider myself a JavaScript expert. I'm pretty good at it, but not great, and it's probably not my favorite language. So why JavaScript? Because this book isn't about languages. It's not a "TDD in C#/Java/Clojure/Elm/Brainfuck" book. It's a book on Real World TDD. Initially to address this I wrote examples in multiple different languages and this was a terrible idea. It was confusing for the reader, as they constantly switched context, and meant I probably wasn't going to finish this book before I retired. It was a problem.

In order to teach you how to solve the problems you have today, databases, slow code, legacy garbage and more I needed to simplify. I needed one app to setup, not many, that's why we're writing the Todo app. I needed a language that almost everybody knows at least a little of. That's why we're using JavaScript. If you've never written a line of code in JavaScript (really?) you should still be able to follow along to the examples. You also won't need to install Visual Studio or learn Java. One language, all the time.

Note this app uses ES5 JavaScript as most developers aren't up to speed on ES6 yet, and even those that are do not get to use those idioms at their real job.

Getting Started

Before you continue go to this URL:

https://github.com/paytonrules/TodoTDD

And follow the steps in the README. If you can start the app and run the tests you are ready to go.

Test Framework

The testing frmework we're using is Mocha. Mocha should be familiar to anybody who has used a BDD framework before like RSpec or Jasmine. The basics look like this (taken from the Mocha website):

var assert = require('assert');
describe('Array', function() {
  describe('#indexOf()', function () {
    it('should return -1 when the value is not present', function () {
      assert.equal(-1, [1,2,3].indexOf(5));
      assert.equal(-1, [1,2,3].indexOf(0));
    });
  });
});

For the documentation you can look at mochajs.org. Where applicable I'll try and explain what's going on as we code. Since this isn't a book on Mocha.js I'll leave some of the work of learning the API up to you. The expectation library is expect.js, so instead of seeing the asserts above your expectations will look like this:

var expect = require('expect.js');

expect(foo).to.be.ok();
expect(foo).to.be('string');
expect(foo).to.equal('bar');
expect(foo).to.eql({a: 'b'}); //eql uses == and works for objects
expect(foo).to.have.length(3);

Of course there's more, and this is taken from the expect.js documentation. It's typical when joining an organization that does some flavor of unit testing, be it TDD or not, you won't have your first choice of tools. There is an existing application, and you'll need to fit into that infrastructure. That's what we're looking to simulate here.

WebServer Framework

The TODO application runs on Node.js. Most Node.js apps use the Express framework and the example originally used that, but in order to reduce the amount you need to learn I've ripped that out in favor of a pure Node.js app. This decision allows us to focus on TDD and not on the Express.js framework and it's idioms.

Interestingly this also meant that all the choices for the initial structure were my own, and many of them are terrible. In fact I wrote much of this app by simply not refactoring, or refactoring sparingly, leaving a pretty messy application. I believe this approach better matches the real world, where teams have to rotate through the latest popular framework and frequently succumb software rot anyway. It didn't hurt that writing intentionally crummy code was fun.

So what do you need to know in order to use this app? You need to have Node.js installed, obviously, but beyond that you'll need to understand how handles requests asynchronously and how to test it. Keep that in mind as we'll be doing more with that in a moment.

Database Access

Database access (currently the database is sqlite) is done using Sequelize, an ORM for Node. I know I just argued against using frameworks like Express.js2 only to through a peculiar ORM at you. Well the app needed at least one strange third party API that's tough to test, and this is the one I picked.

It looks like this:

var Sequelize = require('sequelize');
var sequelize = new Sequelize('database', 'username', 'password');

var User = sequelize.define('user', {
  username: Sequelize.STRING,
  birthday: Sequelize.DATE
});

sequelize.sync().then(function() {
  return User.create({
    username: 'janedoe',
    birthday: new Date(1980, 6, 20)
  });
}).then(function(jane) {
  console.log(jane.get({
    plain: true
  }));
});

Yup - from the official documentation again. Sequelize is a little different from most ORMs that you've probably worked with, because Node places a premium on being asynchronous. If you're unfamiliar with promises, you can oversimplify them by recognizing that a then replaces a callback. So this:

sequelize.sync().then(function() {
  return User.create({
    username: 'janedoe',
    birthday: new Date(1980, 6, 20)
  });

Is roughly equivalent to:

var a = nil;
sequelize.sync(function() {
  a = User.create({
    username: 'janedoe',
    birthday: new Date(1980, 6, 20)
  });
});

The chief advantage of promises is that they can be be chained together avoiding the dreaded Pyramid of Doom. They can also be created and passed around, meaning you can create your own promises in a test and force an asynchronous operation to become synchronous.

But don't let me clumsily explain how promises work, let's use some tests.

What does an app do?

When approaching a new app you usually are handed reams of documentation, get a demo, have a bunch of people tell you personally what it does and attend a few meetings. And after all that you usually don't have the foggiest clue what it actually does. If you've read this far you probably have a similar feeling - you've got some tests, there's something to do with Node and Sequelize, and TODO is prominently involved. But would you feel comfortable writing code? Probably not.

So your first step should be getting yourself comfortable with the app, at least a little bit. There's several approaches people take. They might fiddle with the app and see changes in the UI. They might use an interpreter or REPL like the node command line to interact with the system. Both of those approaches, but as you might expect I like to try writing a test or two. You probably saw that coming.

Let's take a walk through the app. To start it you use npm start which maps to node server.js. That looks like this:

"use strict";

var app = require("./app");
app.start(8888);

Well that doesn't tell us much, other than it's time to look at app.js:

//... bunch of junk ...
        } else {
          var parts = request.url.split('/');
          var data = '';
          request.on('data', function(chunk) {
            data += chunk;
          });
          request.on('end', function() {
            var json, params;
            try {
              json = JSON.parse(data.toString());
            } catch(e) {
              params = querystring.parse(data.toString());
            }

            if (json && json._method === 'DELETE') {
              models.Task.destroy({where: {id: parts[4]}}).then(function() {
//... more junk ...

Yeah there's 100 lines of terrible in there, probably bugs, certainly confusing code. Where's the tests for app.js:

Well there is no app_test.js file, but that doesn't mean there are no tests. Look back at that server.js file and notice it starts a server at port 8888. One common feature you should look for on an app with poor unit tests is that the tests themselves will work in an end-to-end fashion. In other words the tests will be too damn big. Imagine you'd been ordered to do TDD in the middle of a project with minimal or no familiarity, and you weren't sure what to do. You'd probably write big old tests that went all the way through the app too.

So why don't we look for some tests that start the server and make HTTP requests. Let's do a search for 8888 and see if we get lucky:

server.js|4| app.start(8888);
test/routes/index_test.js|7| app.start(8888);
test/routes/index_test.js|16| http.get("http://localhost:8888", function(res) {
test/routes/tasks_test.js|9| app.start(8888);
test/routes/tasks_test.js|31| http.get('http://localhost:8888/users/' + user.id + '/tasks', function(res) {
test/routes/tasks_test.js|50| http.get('http://localhost:8888/users/' + user.id + '/tasks', function(res) {
test/routes/tasks_test.js|61| http.get('http://localhost:8888/users/' + user.id + '/tasks', function(res) {
test/routes/tasks_test.js|80| port: 8888,
test/routes/tasks_test.js|101| port: 8888,
test/routes/tasks_test.js|128| port: 8888,
test/routes/tasks_test.js|153| port: 8888,
test/routes/tasks_test.js|184| port: 8888,
test/routes/users_test.js|9| app.start(8888);
test/routes/users_test.js|30| http.get('http://localhost:8888/users/', function(res) {
test/routes/users_test.js|51| port: 8888,

Well now we have somewhere to write tests. Now of course it won't be this easy normally, because nobody wrote a book about your codebase, but the principle is the same. When you can't find tests directly on a piece of code, look for higher level tests. You can grep like we did here, or you can comment out the confusing code and re-run the tests. Then follow the failures.

Experimental Tests

The tests we're about to write are not intended to be kept. You might keep them if you find them useful, but these are just meant to teach you a little about the code you are about to change. These aren't the same as characterization tests although they are similar. Let's start by looking at the tests in index_test.js.

describe("Index page - routes right to users routes", function() {
  var http = require('http');
  var expect = require('expect.js');
  var app = require('../../app.js');

  before(function() {
    app.start(8888);
  });

  after(function() {
    app.stop();
  });

  describe("get /", function() {
    it ("redirects to users/", function(done) {
      http.get("http://localhost:8888", function(res) {
        expect(res.statusCode).to.equal(302);
        expect(res.headers['location']).to.equal('/users/');
        done();
      });
    });
  });
});

Now I'm gonna advocate something a little strange, we're gonna go ahead and rewrite the test from scratch. Well sort of. The test we're gonna write starts like this:

it ("does stuff", function() {
    app.start(8888);
});

That's a terrible description and you've repeated what's in the before block. You can probably predict what it does:

  3) Users page - CRUD for users "before all" hook:
     Uncaught Error: listen EADDRINUSE :::8888
      at Object.exports._errnoException (util.js:856:11)
      at exports._exceptionWithHostPort (util.js:879:20)
      at Server._listen2 (net.js:1237:14)
      at listen (net.js:1273:10)
      at Server.listen (net.js:1369:5)
      at Object.module.exports.start (app.js:93:12)
      at Context.<anonymous> (test/routes/users_test.js:9:9)

This is actually one of several errors you'll get when you make this change. EADDRINUSE::8888 happens because so many tests are starting a web server and connecting to it. This is a terrible idea and attempting to do all your TDD through a http requests is likely to make you abandon TDD entirely. It's that bad. Fortunately we won't be doing this for much longer.

So what have we learned from this test? App starts the server on port 8888. You might have guessed that, you might not, but now you know. Let's rip out that start and try another test:

it ("does stuff", function() {
    http.get("http://localhost:888");
});

This passes of course. Even with my typo - you don't see it? I put in 888 instead of 888. After fixing that the test still passes, but doesn't tell us much. How do we get a failing test? Well look in the test above the one you're writing and you'll see a callback is optionally passed to http.get:

it ("redirects to users/", function(done) {
    http.get("http://localhost:8888", function(res) {
        expect(res.statusCode).to.equal(302);

You can start to see what's happening here. If you know that 302 is the http code for a redirect then you can see that the root URL redirects to another page. What if we try to get a failing test through a 404.

it ("does stuff", function() {
    http.get("http://localhost:8888/thingthatwontexist", function(res) {
        expect(res.statusCode).to.equal(404);
    });
});

Now the errors get interesting. I got two different ones. This:

  1) Index page - routes right to users routes "after all" hook:
     Uncaught Error: read ECONNRESET
      at exports._errnoException (util.js:856:11)
      at TCP.onread (net.js:544:26)

Or if I run it again sometimes this:

  1) Tasks page - CRUD for tasks "before each" hook:
     Uncaught Error: expected 200 to equal 404
      at Assertion.assert (node_modules/expect.js/index.js:96:13)
      at Assertion.be.Assertion.equal (node_modules/expect.js/index.js:216:10)
      at ClientRequest.<anonymous> (test/routes/index_test.js:25:35)
      at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:427:21)
      at HTTPParser.parserOnHeadersComplete (_http_common.js:88:23)
      at Socket.socketOnData (_http_client.js:317:20)
      at readableAddChunk (_stream_readable.js:146:16)
      at Socket.Readable.push (_stream_readable.js:110:10)
      at TCP.onread (net.js:523:20)

That second error is really interesting, because if you look at the stack trace you'll see an error at (test/routes/index_test.js:25:35) but the description is 'Tasks page - CRUD for tasks "before each" hook'. So the test running is reporting one test failing when a test in a completely different file is the failure. That kind of bizzare error usually means one thing - asynchronous code. Sure enough that's the problem here. Let's modify the code to do this:

it ("does stuff", function(done) {
    http.get("http://localhost:8888/thingthatwontexist", function(res) {
        expect(res.statusCode).to.equal(404);
        done();
    });
});

Mocha's it functions take two parameters, a string and a function callback that executes the test. That callback takes an optional parameter - conventionally called done - which is itself a function. When that function is called the test is done. If the test you wrote does not have a done function then the test will run synchronously, and our expectations here will not be run.

I've walked you through each step here so let's take a moment to review what happened:

  • We started with a blank test, and found the app starts on port 8888.
  • Which means we learned this is a web app running on port 8888.
  • We've made a get request on the root...
  • Which means we know how to make requests now.
  • We've verified that going to the root page does a redirect.
  • We've had problems because the tests were asynchronous.
  • So we addressed that by learning how to write asynchronous tests in Mocha.

As of this point we've now learned a bit about how to write tests in this app, and how to access the app. After running this new test we get:

  1 failing

  1) Index page - routes right to users routes get / does stuff:
     Uncaught Error: expected 200 to equal 404
      at Assertion.assert (node_modules/expect.js/index.js:96:13)
      at Assertion.be.Assertion.equal (node_modules/expect.js/index.js:216:10)
      at ClientRequest.<anonymous> (test/routes/index_test.js:25:35)
      at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:427:21)
      at HTTPParser.parserOnHeadersComplete (_http_common.js:88:23)
      at Socket.socketOnData (_http_client.js:317:20)
      at readableAddChunk (_stream_readable.js:146:16)
      at Socket.Readable.push (_stream_readable.js:110:10)
      at TCP.onread (net.js:523:20)

Welp! That's a bug - our app can't handle a 404. Looks like we've got something to fix. At this point you have a choice. You can fix this bug the simplest way possible, but the mess that is app.js implies there may not be a simplest way to handle that. It looks like we're going to need to do some redesign in order to make the changes. We'll do that in the upcoming chapters.

1. Well I can provided you contact me through http://www.8thlight.com. ↩
2. For this book, not for life. ↩