Home

Pyres is a Resque clone built in python. Each job in the queue corresponds to a python class that needs to define a perform method.

Let’s say I have a blog which needs to check comments for spam. When the comment is saved in the DB we create a job in the queue with comment data. Looks something like.

 class Comment(Model):
     name = Model.CharField()
     email = Model.EmailField()
     body = Model.TextField()
     spam = Model.BooleanField
     queue = "Spam"
#
     @staticmethod
     def perform(comment_id):
          // You ORM or SQL to get the comment data from the comment_id
          x = urllib.open("http://apikey.rest.akismet.com/1.1/comment-check?comment_author="+self.name.. so forth
          if x == "true":
               // save the comment spam field to true.
          else:
               // comment is fine.

You can convert your existing class to be compatible with Pyres. All you need to do is add a queue variable and define a perform method on the class.

To insert a job into the queue you need to do something like this:

from pyres import ResQ
r = Resq()
r.enqueue(Comment, 23)

This puts a job into the queue Spam. Now we need to fire off our workers. In the scripts folder there is an executable pyres_worker which is used to start a worker.

$ ./pyres_worker Spam

Just pass a comma seperated list of queues the worker should poll.

Tips and Tricks

Developing Locally

While developing locally, it’s uncommon to need to rely on a separate worker process that watches the queues. Instead, it’s better to run the job synchronously. You can do this with a simple superclass that your jobs will inherit from:

from pyres import ResQ
r = Resq()

class JobBase(object):
    @classmethod
    def enqueue(cls, *args):
        if app.debug:
            cls.perform(*args)
        else:
            r.enqueue(cls, *args)

class GenerateThumbnailsJob(JobBase):
    pass

From then on, you can enqueue jobs one way regardless if you’re in development, testing or production mode:

from jobs import GenerateThumbnailsJob
GenerateThumbnailsJob.enqueue(*args)