Queue Jobs with Kue

Ask developers about Redis and many will tell you it is a key/value store. However, it’s more than that; it’s also a mechanism for implementing job queues.

There are several advantages to encapsulating units of work as jobs:

• A job can be run in the background. In the context of a web application, you can handle the request without waiting for the job to complete.
  
  

• You can delay the execution of a job, setting it aside to run at a later date or time.
  
  

• A job can run in a different context to your “main” application, be it a process or thread, environment, or even a different server.
  
  

• If a job fails, you can configure your application to retry it multiple times.
  
  

• You can also prioritize jobs. Suppose you need to email a failure notice as a matter of urgency, while your application is sending out a bunch of newsletters. Set a high priority and it will “jump to the front of the queue”, so to speak.
  
  

• Many queue implementations also provide the ability to monitor jobs; and their status – whether they’ve succeeded or failed or are waiting to be re-tried, for example – and to monitor the progress of long-running jobs.

Now that we’ve looked at some of the advantages of jobs, let’s look at one of the ways you can implement it in your Node.js applications; using Kue.

Introducing Kue

Kue is a Node.js package that provides a level of abstraction over Redis queues and tools for monitoring your jobs.

At its most basic, Kue allows you to define a unit of code as a job, and behind the scenes, it will place the necessary information – what to run and the data to run it with – on a Redis queue. A worker is then responsible for actually running those jobs.

Pre-requisites

To use Kue, you’ll need Node.js and npm installed, and of course, it requires that you have Redis installed and configured. We will assume you have Redis installed on localhost (127.0.0.1) and listening on the default port – 6379 – but you can configure this appropriately.

Installation

Install Kue using npm:

npm install kue

Then you’ll need to require it:

var kue = require('kue');

Creating a Job

Before you create a job, you need to create a queue instance:

var queue = kue.createQueue();

There are also several additional options such as naming queues and overriding the default Redis connection settings; consult the documentation for more details.

Now let’s create a job by calling the create() method on your newly created queue.

The first argument identifies the type of job, and any additional data is passed as a hash via the second argument. The job must then be saved to the queue. Here’s an example:

queue.create('email', {
    title: 'Welcome to the site',
    to: '[email protected]',
    template: 'welcome-email'
}).save();

Setting the Priority

Queues are set to “normal” priority by default, which means that they’ll be handled in a standard FIFO (First In First Out) manner. As you’d expect, a higher priority means the job will jump up the queue, and the reverse is true for low-priority jobs.

Priorities are defined by number or name; a name is simply a constant that maps to a numeric value, as follows:

Do be aware that somewhat counterintuitively the lower the value, the higher the priority.

Setting the priority is easy; just call the priority() method on the job, before you save it. Either provide a numeric value:

queue.create(..).priority(-10).save();

…or, a string, referring to the table above:

queue.create(..).priority('high').save();

Retrying Failed Jobs

By default, if a job fails then it won’t be re-run. However, a job will be re-tried if you provide the maximum number of attempts as follows:

queue.create(..).attempts(5).save();

Putting it Altogether

Since the priority(), attempts(), and save() methods are all chainable, we can put it all together like this:

queue.create('email', {
    title: 'Welcome to the site',
    to: '[email protected]',
    template: 'welcome-email'
}).priority('high').attempts(5).save();

Running Jobs with a Worker

To run jobs that have been added to a queue, typically you’ll create a separate worker process. This should create a queue instance as before and call the process() method. The first argument should be the name of the job, and the second a function, which provides as arguments the job itself and a callback. For example:

var queue = kue.createQueue();

queue.process('email', function(job, done) {
    sendEmail(job.data, done);
});

You’ll notice from the example above that the data we provided when we added the job to the queue is available via the data property on the job object.

As is commonplace in a Node.js application, the done callback should be called with an Error as its first argument to indicate failure, or null otherwise. If you need to provide a result of the job – for example a status code, or the ID of a database entry – then you can provide it as an entirely optional second argument.

Additional Considerations

There are a couple of things you may wish to consider when creating a worker. To run across multiple processes, you may wish to consider using Cluster.

As you might have guessed, it’s important that the worker remains running at all times; for this, you might wish to consider either Forever or PM2. You might also find supervisord useful for ensuring the constant running of the process, even after a system restart.

Monitoring Jobs

In addition to providing a method for creating and processing jobs, Kue also offers several features for monitoring them.

The UI

Kue provides an optional, very simple user interface for monitoring jobs as an Express application.

You can either run it as a standalone application from the command line:

node_modules / kue / bin / kue - dashboard - p 3050 - r redis: //127.0.0.1:3000

Or from your Express application:

kue.app.listen(3000);

Alternatively, the Kue UI package provides a slicker, more comprehensive UI.

The JSON API

For even more flexibility in monitoring jobs, Kue also provides a JSON API.

Summary

Jobs are a great way of adding fault tolerance, decoupling your application’s architecture, and executing long-running processes in the background. Kue not only provides a high-level abstraction to Redis queues for just that but also a suite of tools for monitoring the status of all of your jobs.

You can read more on the project page and in the Kue documentation center.