# sfJobQueuePlugin



## Introduction

### What the **** ?

This plugins enables job queues into Symfony. It includes all the common job 
queues tasks (start, stop, scheduling through job election strategies, etc.), 
command line tasks, and a graphical interface for managing queues and jobs. 
Using a job queue can be useful when asynchronised server-side operations have 
to be performed (periodically grabbing a RSS feed, automatically sending 
emails, etc.) or in environments without a cron access.

The advantages compared to cron are:
  * a more acurate job-execution scheduling (cron's best precision is at the 
minute level)
  * error logging
  * jobs creation from a graphical interface
  * API to create jobs programmatically

### Requirements

  * a shell access
  * the right to run PHP from the command line
  * this plugin has not been tested with windows

### Glossary

A "job" is a task that has to be done. In this plugin, it is modeled as a call 
to a JobHandler, which understands the parameters associated to the job in 
order to perform a dedicated action.

A "job queue" is a group of jobs to be run, eventually several times, 
associated to a job election strategy. For the moment, one job queue can only 
run one job at a time.

The "job queue manager" is the process that is responsible for the launch of 
the job queues. It creates a PHP process for every queue the user has requested 
the start.

## Features

  * multi-queues support
  * one given queue can contain heterogeneous job types
  * scheduling (job election) strategy abstraction
  * support for reccuring jobs
  * CLI tasks
  * Queue and jobs management graphical module:
   * job-queues start/stop
   * job-queues statistics
   * job creation

### Job-queues admin panel

![sfJobQueuePlugin_admin_panel.png](http://trac.symfony-project.org/attachment/wiki/sfJobQueuePlugin/sfJobQueuePlugin_admin_panel.png?format=raw)


## Get it installed

  * go to your project's root

  * Install the plugin:
       
         ./symfony plugin-install http://plugins.symfony-project.com/sfJobQueuePlugin
       

  * rebuild the model:
       
         ./symfony propel-build-all
       

  * enable the "sfJob" and the "sfJobQueue" modules in your app's settings.yml:
       
          enabled_modules:   [sfJob, sfJobQueue](default,)
       

  * clear cache:
       
         ./symfony cc
       


## Usage

### Starting the job queues manager

The "queue manager" is the process that is responsible for the management of 
the job queues. When requested, it is able a given job queue. No job will be run
until you start the queues manager.

As this process must be able to create child processes, it has to be runned from 
the command line. This notably means that you must have a shell access to your 
server in order to be able to use this plugin.

Starting the queue manager can be done with the following Symfony task:
    
      $ ./symfony sfqueue-start-queuemanager YOUR_APP_NAME
    

You usually will want that the executions continues when you disconnect from 
your server. Therefore, you should rather run the following command:
    
     $ nohup ./symfony sfqueue-start-queuemanager YOUR_APP_NAME &
    

Doing so will forbid you to stop it without using the "kill" command. If this 
sounds like a problem to you, you will probably be interested in the 
[screen](http://www.gnu.org/software/screen) utility.

### Creating a queue

In order to create a job queue, you must first activate the administration 
module. You can then create a job queue by defining:
  * its name
  * its job election strategy (either FIFO or priority-based). Other job-election
 strategies may be developed in the future.

You usually won't need it, but it is also possible to create job queues 
dynamically, directly from your application's code:
    
    [php]
    <?php
    $queue = new sfJobQueue();
    $queue->setName('RSS grabbing queue');
    $queue->setSchedulerName('fifo');
    $queue->save();
    

By default, the job queue is stopped when created. In order it to get active, 
you need to start it.

### Running a job queue

Starting a job queue can eitherbe done from the graphical interface, or directly
in the code:
    
    [php]
    <?php
    $queue->setRequestedStatus(sfJobQueue::RUNNING);
    $queue->save();
    

A job queue can also be started with Symfony's command-line utility:
    
      ./symfony sfqueue-start-queue YOUR_APP_NAME QUEUE_NAME
    

So, for instance :

    
      ./symfony sfqueue-start-queue frontend 'RSS grabbing queue'
    

### Adding a job to the JobQueue

Adding a job to one job queue is rather simple. You only have to give the type 
of the job which has to be created, and set its execution parameters.
    
    [php]
    <?php
    $queue->addJob('mail', array('to' => 'xavier@lacot.org', 'topic' => 'Testing the MailingJobQueue :)'));
    

This means that you will have to create, somewhere in your project (why not in 
another plugin ;)) a JobHandler that supports the "mail" job type. For more 
details, see the paragraph "[a new job type](Creating)".

## Job parameters

A job queue can be tweaked using several parameters:
  * scheduler_name: the scheduler name defines the policy of job election. For instance, with a "fifo" scheduler, the oldest eligible job will be the one who will be processed next. With a "priority" scheduler, the eligible job with the highest priority will be the first out.
  * polling_delay: time between the end of the execution of one job, and the next job election

![sfJobQueuePlugin_job_election.png](http://trac.symfony-project.org/attachment/wiki/sfJobQueuePlugin/sfJobQueuePlugin_job_election.png?format=raw)

At the job level, it is also possible to set some general execution parameters:
  * max_tries:
  * retry_delay: retry delay, in seconds (minimal delay between two tries of the same job)
  * priority: priority : from 0 to 9 (lower to higher)
  * params: an array of parameters for the job execution
  * scheduled_at: date of scheduling after which the job can be runned

## Creating a new job type

The sfJobQueuePlugin has been designed so that creating a new job type should 
be the less painful possible for the developer. Actually, developping a new job 
type only requires one single `JobHandler` class to be written, that 
implements the interface 
[sfJobHandlerInterface](browser:plugins/sfJobQueuePlugin/lib/jobHandlers/sfJobHandlerInterface.class.php).
Nothing else.

The name of this class is important, as the plugins gets the name of the job 
type from it. For instance, if you create a "sfClearCacheJobHandler", the plugin 
graphical interface will automagically propose the job type "`ClearCache`" 
during the creation of new jobs.

Here is for instance a minimal `JobHandler`:

    
    [php]
    <?php
    class sfMailJobHandler extends sfJobHandler implements sfJobHandlerInterface
    {
      public function getParamFields()
      {
        return array('from', 'to', 'subject', 'message');
      }
    
      public function run($params)
      {
        if (mail($params['to'], 
                 $params['subject'], 
                 $params['message'], 
                 'From: '.$params['from']))
        {
          return sfJob::SUCCESS;
        }
        else
        {
          throw new Exception('There was an error while sending the mail.');
          return sfJob::ERROR;
        }
      }
    }
    

Several remarks about it:
### getParamFields()

This must return the name of the parameters expected when running the job. It 
is used by the graphical interface, in order to ease up the job creation. In 
the graphical interface, the fields will be displayed in the order of the 
elements in this array:

![sfJobQueuePlugin_job_creation_detail.png](http://trac.symfony-project.org/attachment/wiki/sfJobQueuePlugin/sfJobQueuePlugin_job_creation_detail.png?format=raw)

### run($params)

The method **`run()`** uses an array of parameters for completing the 
job. It must do a proper use of the return status, either `sfJob::SUCCESS` 
in case the job is successful, or `sfJob::ERROR` if there is an error. 
Remember that, if a PHP error is raised while running the job, its execution 
will be stopped and the job will be marked as failed. You'll then be able to 
track the error on the graphical interface:

![sfJobQueuePlugin_error_display.png](http://trac.symfony-project.org/attachment/wiki/sfJobQueuePlugin/sfJobQueuePlugin_error_display.png?format=raw)

The `run()` method may launch exceptions. In this case, the job will be 
marked as failed, except if it is a recuring job.

## API

sfJobQueue:
  * **`addJob($type = '', $options = null)`** - Creates a new job in the queue
  * **`getNbActiveJobs()`** - Returns the number of active jobs, ie. jobs that are neither failed, nor successful, nor cancelled
  * **`getNbActiveReadyJobs()`** - Returns the number of active jobs ready to be runned
  * **`getNbActiveRecuringJobs()`** - Returns the number of active recuring jobs
  * **`getNbActiveScheduledJobs()`** - Returns the number of active scheduled jobs (ie. scheduled in the future)
  * **`getNbActiveWaitingJobs()`** - Returns the number of active waiting jobs (ie. active, but not ready to be runned)
  * **`getNbCompletedCancelledJobs()`** - Returns the number of jobs that have been cancelled
  * **`getNbCompletedFailureJobs()`** - Returns the number of jobs completed on failure
  * **`getNbCompletedSuccessfulJobs()`** - Returns the number of successfully runned jobs
  * **`getNbCompletedJobs()`** - Returns the number of completed jobs
  * **`getStatusText()`** - Returns the status of the queue, as a text
  * **`isRunning()`** - Indicates whether the queue is running or not
  * **`run()`** - Runs the queue

sfJob:
  * **`getStatusText()`** - Returns the status of the job, as a text
  * **`run()`** - Runs the job

## Unit testing

To be done.

## Roadmap

  * job messages detail
  * have several jobs running at one time
  * unit-testing
  * job reporting hooks (measure jobs progress)
  * handle dependancy between jobs

Long term runners might want to see:
  * integration with http://yowl.googlecode.com/svn/trunk/test/test-yowl.html
  * exposition to Web services

## Bibliograhy

  * http://en.wikipedia.org/wiki/Queue_%28data_structure%29
  * http://en.wikipedia.org/wiki/FIFO
  * http://en.wikipedia.org/wiki/Priority_queue

## License and credits

This plugin is licensed under the MIT license and maintained by [Xavier Lacot](http://lacot.org/) 
<xavier@lacot.org>. External contributions and comments are welcome !

## Changelog

### version 0.2 - 2007-10-23

  * upgraded the graphical interface :
   * possibility to create jobs
   * support for recuring jobs
   * support for jobs scheduling
  * improved documentation
  * introduced job queue manager concept

### version 0.1 - 2007-09-17

Initial public release.