At Salsify we run most of our Ruby on Rails based infrastructure in Heroku. We recently switched our Puma web workers over from Heroku 2X dynos to more powerful Heroku PX dynos and saw dramatic performance improvements: a 51% reduction in mean response time and a 59% reduction in 99th percentile response time. Based on this we did some experiments running our more memory/CPU intensive Delayed Jobs on PX dynos and saw a similarly encouraging 43% reduction in mean job execution time and 66% reduction in 99th percentile job execution time. This was great for a proof of concept, but only one of the eight cores on the PX dyno was being used. In order to take this to production in a cost-effective manner, we had to figure out how to utilize all of the cores on the dyno.

Background

Delayed Job has built-in support for running a pool of workers. This works great in some environments but unfortunately Heroku isn't one of them. Delayed Job daemonizes all the worker processes which doesn't work with the Heroku process model. We couldn't find anything that met our needs so we created the delayed_job_worker_pool gem.

Delayed Job Worker Pool

The delayed_job_worker_pool gem manages a pool of non-daemonized Delayed Job worker processes. It supports the following features:

• Forking multiple Delayed Job worker processes to leverage multiple cores given Ruby MRI's parallelism limitations
• Monitoring forked processes, restarting them as necessary
• Optionally preloading the application prior to forking workers to reduce startup time and memory consumption
• A simple, Puma-like configuration DSL with callbacks to customize worker startup

To get started with the gem add the following entry to your Gemfile:

Then create a configuration file in config/delayed_job_worker_pool.rb e.g.

Finally run the worker pool via the following command:

There are a few gotchas to be aware of when preloading the application:

• Sockets (e.g. for database connections) are not inherited by worker processes and need to be closed and reopened in the after_preload_app and on_worker_boot callbacks.
• Threads (e.g. for sending metrics to a metrics service) are not inherited by worker processes and need to be restarted in the on_worker_boot callback. The worker pool will log a warning if there any uninheritable threads running when a worker is forked.

Conclusion

At Salsify we've seen great performance improvements by moving our web and background workers to Heroku PX dynos. In order to fully leverage the larger dynos, we needed to make sure we were running multiple processes per machine. The Puma web server supports multiple processes (and multiple threads) out of the box. The delayed_job_worker_pool gem extends Delayed Job to support multiple worker processes in a Heroku friendly way.

Have you had similar experiences pooling background workers or using Heroku PX dynos? If so, we'd love to hear your thoughts!