Ruan Bekker's Blog

From a Curious mind to Posts on Github

Using Python RQ for Task Queues in Python

This is a getting started on python-rq tutorial and I will demonstrate how to work with asynchronous tasks using python redis queue (python-rq).

What will we be doing

We want a client to submit 1000’s of tasks in a non-blocking asynchronous fashion, and then we will have workers which will consume these tasks from our redis queue, and process them at the rate of what our consumer can handle.

The nice thing about this is that, if our consumer is unavailable for processing the tasks will remain in the queue and once the consumer is ready to consume, the tasks will be executed. It’s also nice that its asynchronous, so the client don’t have to wait until the task has finished.

We will run a redis server using docker, which will be used to queue all our tasks, then we will go through the basics in python and python-rq such as:

  • Writing a Task
  • Enqueueing a Job
  • Getting information from our queue, listing jobs, job statuses
  • Running our workers to consume from the queue and action our tasks
  • Basic application which queues jobs to the queue, consumes and action them and monitors the queue

Redis Server

You will require docker for this next step, to start the redis server:

$ docker run --rm -itd --name redis -p 6379:6379 redis:alpine

Python RQ

Install python-rq:

$ pip install rq