Salsify is a single-page app (Backbone on the client, Rails + Postgres + Elastic Search on the backend) that requires very low latency responses to provide an interactive experience for browsing and editing product catalogs. In order to satisfy these latency requirements we rely heavily on Delayed Job for running time consuming tasks like importing data, exporting data, and applying bulk updates in background processes. We've found useful pattern for using Delayed Jobs callbacks and hooks together that I wanted to share.

(If you've never used Delayed Jobs, checkout this Rails Cast  for a brief introduction.)

As we wrote more and more jobs we found lots of cross cutting concerns that we really wanted to make reusable across jobs. These included things like:

• Error reporting to Airbrake
• Setting the security context for the job since our system is multi-tentant
• Updating additional database tables based on the success or a failure of a job

Fortunately Delayed Job has a rich plugin mechanism that can be used for adding lifecycle callbacks to handle requirements like these. In this blog post we explore how you can use Delayed Job's plugin mechanism to add global job lifecycle callbacks as well as job specific lifecycle callbacks.

Global Delayed Job Hooks

Delayed Job has a great plugin mechanism that allows users to add functionality to all jobs. For example, here's a plugin that we use to report all job errors to Airbrake:

To register this example plugin with Delayed Job you'll need to add the following to a config initializer:

Pretty easy huh?

Delayed Job has a rich set of before, after and around callbacks for running jobs, enqueuing jobs, job attempt errors and job failures. See the Delayed::Lifecycle doc for a full list of the available lifecycle events. These global lifecycle hooks are really powerful and go along way towards letting us DRY-up our job implementations. The only downside is the plugin applies to all job classes. What do you do if you want to factor out cross-cutting job concerns that only apply to certain job classes? Read on to find out how we solved this problem at Salsify.

Job Specific Hooks

Delayed Job supports callback methods on your job class that are called at convenient times in the lifecycle of a job:

The job parameter is optional for all of the callbacks except error. I've never had to use it in any callback definitions so I usually omit it.

The order of callback calls for a successful job execution is:

• before
• perform (to do the real work of the job)
• success
• after

The order of callback calls for a failed job execution is:

• before
• perform (to do the real work of the job)
• error
• after
• failure

Note that the failure callback is called after the after callback. This has bitten me in the past!

These callbacks are great but relying on standard Ruby method calls is problematic when factoring out callbacks into superclasses or mixins. Consider the following example:

Unfortunately this doesn't work. Running the job outputs the following:

Hello from SampleJob#error

What happened? ErrorHandlingJobMixin#error is never called because we overrode error in SampleJob but we forgot to call the super class version of the method. We could fix this with a call to super but that's error prone. It's just too easy to forget to call the super class version of the callback method (not to mention checking to see if a superclass version of the callback method exists before invoking it). At Salsify we wrote a Delayed Job plugin that exposes Rails style callback macros which automatically handles the chaining of job callbacks. This allows us to rewrite our example like this (and actually have it work as expected!):

Running this example we see the following output as expected:

Hello from ErrorHandlingJobMixin#after_job_attempt_error Hello from SampleJob#after_job_attempt_error

Notice the SampleJob callbacks are completely encapsulated from the ErrorHandlingJobMixin callbacks. There are no more error prone calls to super class method implementations. The example above was obviously a toy but here's a more realistic usage the automatically sets an organization's security context for a job execution:

By including the MultitenantJob module in a job class, the job automatically runs in an organization's security context. The job class doesn't have to worry about setting up and tearing down the security context. It's all handled automatically.

Our plugin currently exposes the following lifecycle hooks (although adding others would be easy):

• before_job_attempt
• after_job_attempt_error
• after_job_failure
• after_job_success
• after_job_attempt

The after_job_attempt callback is called after the after_job_attempt_error, after_job_failure and after_job_success callbacks. This is very useful for failure handlers that rely on context (e.g. a security context for the job) that is set in a before_job_attempt context and unset in an after_job_attempt callback. The plugin doesn't interfere with the standard Delayed Job callback methods so you're free to mix and match the two paradigms which is useful if you want to incrementally adopt the plugin (or you rely on gems that use the standard Delayed Job callback methods).

Job Hooks Plugin Implementation

So how did we implement the plugin? We used the hooks gem which allows us to define hooks and have the gem handle the details running the callbacks in the order they are inherited. First we declared a module that defines our hooks:

Then we wrote a plugin to convert from Delayed Job's global callbacks to our job class level callbacks:

Conclusion

Delayed Job is a powerful tool that we use heavily here at Salsify for running asynchronous tasks. The library provides a rich plugin model that we've leveraged to factor out cross cutting concerns at a global level as well as for specific types of jobs. Watch out for more posts on how we use Delayed Job in the future...