At Salsify we write lots of Capybara integration tests of our single-page app. These tests are a key part of the infrastructure that allow us to move fast and refactor without break anything. Unfortunately as we added more and more tests we started experiencing sporadic hangs and database deadlocks while tearing down tests. After some investigation we figured out the problem was our JavaScript client making AJAX requests to our Rails server during the test teardown process. This post details our journey to stamp out these problems and get our tests running reliably again.  

A First Attempt

Our JavaScript code uses JQuery (often via Backbone.js) to make all AJAX requests. Fortunately JQuery keeps an undocumented counter of the number of active AJAX requests in the active variable. Our first attempt at fixing our test teardown problem was adding an after block for each feature test that waited for all active AJAX requests to complete:

We knew this approach wasn't perfect since there was nothing stopping our JavaScript code from making additional AJAX requests after the number of active AJAX requests dropped to zero but it worked well enough for a few months. Over time our application got more complex and we added more UI components each of which could make AJAX requests. As we added more of these components and more tests, problems during test teardown starting creeping up again. It was time for a more reliable solution. 

Bringing Out The Big Guns

The basic outline of our more complete solution was to do the following during test teardown:

1. Prevent the server from accepting any new requests
2. Prevent the client from making any new requests
3. Wait until all active requests complete

To implement this solution we first wrote a piece of Rack middleware that would take care of #1 and #3 (for an introduction to Rack middleware checkout this RailsCast):

The next hurdle was to figure out how to prevent the client from making any new AJAX requests. We contemplated adding a JQuery ajaxSend() handler to block any new AJAX request but instead settled on a simpler solution of just navigating to the about:blank page. This would prevent the client from making any new AJAX requests as well as requesting any other resources. With all these changes here's what our spec_helper looked liked:

We updated our test environment to insert our RackRequestBlocker at the beginning of the Rack middleware chain:

Finally we had to add our dependency on the atomic gem to our Gemfile:

That's it! Goodbye deadlocks and timeouts! Hello clean test runs!

Conclusion

I'm happy to report that our tests have been deadlock and timeout free since making these changes to our test infrastructure. Have you faced similar problems writing reliable feature tests? If so, drop us a comment. We'd love to hear about it.