#344 Queue Classic

In episode 342 we showed you how to set up a Rails application that used PostgreSQL as its database. One great thing about Postgres is that it can take on the responsibility for things that often require a background process such as managing a job queue. Below is a screenshot from a Rails application that uses Postgres and which sends out newsletters.

We can deliver a newsletter by clicking its “Deliver Newsletter” link. This takes a while to complete and until it does this Rails instance will be unable to process any other requests and the browser will appear to “stick” while is waits for a response. Because of this it’s a good idea to move long-running tasks into a background process whenever possible. When a newsletter is finally delivered we’ll see a message at the top of the page and its status will be shown in the list above.

Introducing Queue Classic

We’ll solve this problem by using the queue_classic gem. This gem makes it easy to take a long-running task and move it into a background process which is managed by Postgres. We’ll be using version 2 of this gem which is currently at the release candidate stage. The README has some instructions for setting up queue_classic in a Rails application and we’ll follow these. As ever the first step is to add the gem to the gemfile and then run bundle. As we’re installing a pre-release version we’ll need to specify the version number.

gem 'queue_classic', '2.0.0rc14'

Next we’ll load queue_classic’s Rake tasks by creating a queue_classic.rake file under /lib/tasks and pasting in these two lines from the README.

require "queue_classic"
require "queue_classic/tasks"

We’ll also need to create a queue_classic initializer. In here we specify the configuration, which is done through environment variables. This makes it fully compatible with Heroku. We just need to add a DATABASE_URL variable that contains the connection information for the database. As the database is on our local system we don’t need to specify a username and password here.

ENV["DATABASE_URL"] = "postgres://localhost/mailer_development"

Next we need a migration called add_queue_classic to set up the database.

$ rails g migration add_queue_classic
      invoke  active_record
      create    db/migrate/20120502000000_add_queue_classic.rb

In this migration we’ll add code to load the queue_classic functions.

class AddQueueClassic < ActiveRecord::Migration
  def up
    QC::Setup.create
  end

  def down
    QC::Setup.drop
  end
end

We’ll need to run rake db:migrate to run the migration. Next we’ll start up the background worker process by running this command.

$ rake qc:work

If all goes well we won’t see any output from this command. We can experiment with queue_classic now in the Rails console in a new terminal window. To add a job to the queue we call QC.enqueue and pass it the method we want to trigger and any arguments we want to pass to that method.

1.9.3p125 :001 > QC.enqueue "puts", "hello world"
lib=queue_classic action=insert_job elapsed=13
 => nil

In the tab that’s running rake qc:work we should now see “hello world”. When we call enqueue the arguments are serialized into JSON and passed to the queues table to be processed by the worker. The JSON serialization is quite picky so its important to keep the arguments that we pass in as simple as possible. It doesn’t even like symbols so if we pass in a hash like this an exception will be thrown.

1.9.3p125 :002 > QC.enqueue "puts", msg:"hello world"
lib=queue_classic level=error error="QC::OkJson::Error" message="Hash key is not a string: :msg" action=insert_job
QC::OkJson::Error: Hash key is not a string: :msg

If we want to pass in a hash we’ll need to use strings instead, like this:

1.9.3p125 :003 >   QC.enqueue "puts", "msg" => "hello world"

This will run correctly and print out the hash.

Delivering Newsletters With Queue Classic

We’ll modify our application now so that when we click a “deliver newsletter” link the long process of delivering a newsletter happens in the background. When a link is clicked it triggers the deliver action in the NewslettersController. This action fetches a newsletter, sleeps for ten seconds to simulate a long-running process then updates the newsletter’s delivered at time.

def deliver
  newsletter = Newsletter.find(params[:id])
  sleep 10 # simulate long newsletter delivery
  newsletter.update_attribute(:delivered_at, Time.zone.now)
  redirect_to newsletters_url, notice: "Delivered newsletter."
end

Whenever we have a long-running process in a controller it’s a good idea to move it into a class method in a model to simplify the interface as much as possible. We’ll create a deliver method in the Newsletter model and call this from the controller.

def deliver
  Newsletter.deliver(params[:id])
  redirect_to newsletters_url, notice: "Delivered newsletter."
end

Next we’ll write that method in the model.

class Newsletter < ActiveRecord::Base
  attr_accessible :delivered_at, :subject
  
  def self.deliver(id)
    newsletter = find(id)
    sleep 10 # simulate long newsletter delivery
    newsletter.update_attribute(:delivered_at, Time.zone.now)
  end
end

It’s easy now to move the delivery off into a background process and change the flash notice to say “Delivering newsletter.” instead of “Delivered newsletter.”.

def deliver
  QC.enqueue "Newsletter.deliver", params[:id]
  redirect_to newsletters_url, notice: "Delivering newsletter."
end

We can try this out now. If we click a “Deliver newsletter” link we’ll see the response straight away, but the newsletter won’t yet be delivered as it’s being processed in the background so it won’t show as delivered in the list.

If we wait ten seconds and reload the page we’ll see that the newsletter has been delivered.

Handling Failed Jobs

What happens when a job fails and an exception is raised while it’s being processed? We’ll throw an exception to see that the result is.

def self.deliver(id)
  newsletter = find(id)
  raise "Oops"
  sleep 10 # simulate long newsletter delivery
  newsletter.update_attribute(:delivered_at, Time.zone.now)
end

Now when we click “Deliver newsletter” we’ll get the message saying that the message is being delivered but the exception will be hit when the job’s processed in the background. If we look at the output from the worker process we’ll see the failure mentioned there. (We should log the output from this in a production application.) By default queue_classic doesn’t try to do anything such as automatically retry the job. If we look at the Worker class in the source code we’ll find a handle_failure method and this prints out the error output.

#override this method to do whatever you want
def handle_failure(job,e)
  puts "!"
  puts "! \t FAIL"
  puts "! \t \t #{job.inspect}"
  puts "! \t \t #{e.inspect}"
  puts "!"
end

If we want to handle failed jobs in a different way we need to override this method. An example of how we can do this by subclassing the Worker class is shown in the README. There’s a lot of other useful information in this README file such as an example of how to set up a worker executable instead of using the Rake task. The advantage of this is that we can control exactly what gets loaded so if we don’t need our entire Rails application loaded in for the worker we can skip it and save memory.

Another useful feature is the ability to listen to job notifications. This way if a job is added to the queue our worker can be notified instantly and start processing it immediately instead of waiting for the database polling. This might play a part in choosing queue_classic over another queuing solution.

The final question is: how does queue_classic stack up to the many other queuing solutions? It’s closest to Delayed Job, which is also backed by a database, but queue_classic takes better advantage of Postgres’s features and doesn’t require ActiveRecord so with it we can minimize the worker process. As for the other options, such as Resque, RabbitMQ and Beanstalk’d these require a separate daemon process to manage the queue. If you want to keep your server setup simple, queue_classic is a great option, although one thing that it is missing is a Web interface such as the one that Resque provides.