ajDoctrineLuceneablePlugin - 0.1.1

This plugins allows for easy lucene full text integration in doctrine and the admin generator

You are currently browsing
the website for symfony 1

Visit the Symfony2 website


« Back to the Plugins Home

Signin


Forgot your password?
Create an account

Tools

Stats

advanced search
Information Readme Releases Changelog Contribute
Show source

This is the documentation of the Doctrine Lucene Behaviour. This will provide you with full text search in the admin generator filters, and a convenient way to roll out your own powerful search engine.

Knowledge of lucene is not nescesairy, but it is recommended. The jobeet tutorial on searching is an excellent place to start.

Simple Tutorial: Easy admin generator integration

This tutorial will show you how to get fulltext doctrine query with just a few lines.

1. Install the plugin

symfony plugin:install ajDoctrineLucenablePlugin --stability=beta

2. Configure the plugin

By default, no configuration is required, but it's good to look into the lucene.yml file. To configure the plugin: copy the lucene.yml from the ajDoctrineLucenablePlugin/config/lucene.yml to your SF_ROOT/config. See the yml file for details on the configuration

2. Enable the lucene behaviour for your model

Project:
  columns:
        id:
          type: integer(4)
          primary: true
          autoincrement: true
        location:
          type: string(255)
          notblank: true
        body:
          type: clob
        name:
          type: string(255)
          notblank: true
  actAs:
        Luceneable:
          fields:
                - name: unstored
                - location: unstored
                - body: unstored

This is a key/value pair, in which the name corresponds with the field (actually the model method for the field, so it can be your custom getFullName() method, then you use the name 'full_name'), and the value corresponds with the Zend Lucene field type (for reference http://framework.zend.com/manual/en/zend.search.lucene.overview.html). This can either be keyword, unindexed, text or unstored. If you're unsure, you will probably want to use unindexed.

Since we 99% of the time only use the lucene index to retrieve document id's, the default value is UnStored. This saves the metadata, but not the data.

3. (re)build the model

symfony doctrine:build --model

4. Add the filter to the model

To create a searchable filter for a model, add this to the ProjectFormFilter.class.php:

class ProjectFormFilter extends BaseProjectFormFilter
{
  public function configure()
  {
        $this->widgetSchema['search'] = new sfWidgetFormFilterInput(array('with_empty' => false));
        $this->validatorSchema['search'] = new sfValidatorPass(array('required' => false));
        $this->useFields(array('search'));
  }
 
  public function addSearchColumnQuery($query,$field,$value)
  {
 
    $ids = LuceneSearch::find($value['text'])
      ->fuzzy()
      // search fuzzy?
      ->in($this->getModelName())
      ->setFilterQuery($query);
 
    return $query;
  }
}

5. You're done.

Example 2: Custom model (recommended)

Project:
  columns:
    id:
      type: integer(4)
      primary: true
      autoincrement: true
    location:
      type: string(255)
      notblank: true
    body:
      type: clob
    name:
      type: string(255)
      notblank: true
  actAs: [Luceneable]

In lib/models/doctrine/Project.class.php: Here we add to code that is being called when the model is inserted or updated. This example shows how to load and parse a HTML document to the lucene index. The updateLucene method should always return a Zend_Search_Lucene_Document.

public function updateLucene()
{
  $doc = Zend_Search_Lucene_Document_Html::loadHTML($this->getBody());
  $doc->addField(Zend_Search_Lucene_Field::Unstored('name', $this->getLocation(), 'utf-8'));
  $doc->addField(Zend_Search_Lucene_Field::Unstored('location', $this->getLocation(), 'utf-8'));
  return $doc;
}

You can override the parameters from schema.yml from the model by defining a public attribute $luceneSearchFields.

public $luceneSearchFields = array(
    'name' => 'unstored',
    'location' => 'text'
);

Filters for the Admin Generator

To create a searchable filter for a model (and the admin generator), add this to the lib/filters/doctrine/FormFilter.class.php:

class OrganisationFormFilter extends BaseOrganisationFormFilter
{
  public function configure()
  {
    $this->widgetSchema['search'] = new sfWidgetFormFilterInput(array('with_empty' => false));
    $this->validatorSchema['search'] = new sfValidatorPass(array('required' => false));
    $this->useFields(array('search'));
  }
 
  public function addSearchColumnQuery($query,$field,$value)
  {
    LuceneSearch::find($value['text'])
      ->fuzzy() // do we want to search fuzzy?
      ->in($this->getModelName())
      ->setFilterQuery($query);
    return $query;
  }
}

Adding the ->fuzzy() true will mean fuzzy searching (or in fact adding a ~ behind every keyword in the search string.) This means fuzzy searching. This is great for newbees and noobs, but not so great if you wish to use the cool Zend Lucene Query Language.

Example 3. Creating your own search engine

The trick in creating a simple search engine is that for each model you have to render a custom partial for that model.

in search/actions/actions.class.php:

public function executeSearch(sfWebRequest $request)
{
    $this->getResponse()->setTitle('Search results for: ' . $request->getParameter('query'));
    $this->searchquery = $request->getParameter('query');
    $query = LuceneSearch::find($query)->fuzzy()->in(array('News','Member','Project','Event','Content','Organisation'));
    $this->results = $query->getRecords();
    $this->hits = $query->getHits();
}

in search/templates/searchSuccess.php:

<?php $count = count($hits);
if ($count == 0) {
    echo "No results found for '<strong>" . $searchquery . "</strong>':";
} elseif ($count == 1)
{
    echo "1 result found for '<strong>" . $searchquery . "</strong>':";
} else {
    echo sprintf("%d results found",$count) . " for '<strong>" . $searchquery . "</strong>':";
}
 
?>
 
<?php
      foreach ($hits as $hit)
      {
        echo get_partial($hit->model, array(
            'obj' => $results[$hit->model][$hit->pk],
            'pk' => $hit)
        );
      }
?>

in the get_partial() line lies the magic, here you can load for each model a seperate template. My template for news looks like this:

in templates/_News.php

<?php use_helper('Date'); ?>
<?php $data = $obj;
if ($data->is_published):
?>
<div class="contentItem clickableItem">
        <div class="itemDescription">
                <p>
                        <h1 class="searchCatagory">News</h1><br />
                        <?php echo format_date($data['created_at'], 'EEE dd MMMM yyyy') ?>:
                        <h1><?php echo $data['title'] ?></h1>
                        <?php
                        $abstract = substr($data['abstract'], 0, 180);
                        echo nl2br($abstract) . '... <br />' . link_to('> READ MORE','news/show?id='. $data['id'],array('class'=>'follow'));
           ?>
                </p>
        </div>
        <div class="itemImage"><img src="/uploads/news/s/<?php echo $data['image']; ?>"></div>
        <div class="borderDiv"> </div>
</div>
<?php endif; ?>

Likewise, you can create such a template for each model that you wish to include in your search engine. (in this case for: 'News','Member','Project','Event','Content','Organisation')

Searching

There is a convenient class to search models:

$query = LuceneSearch::find($string)
->in('model1','model2','model3');

Results

This method will return the Pk's (primary keys, per model)

$hits = $query->getHits();
// get an array of lucene hits
 
$ids = $query->getPks();
// get an array of lucene hits in the form:
 
array (
'Model' => array(1,2,3,45,5,6,7),
'Model2' => array(5,2,1,4)
)

Connection with the model:

$ids = LuceneSearch::find($value['text'])
  //->fuzzy()
  ->in($this->getModelName())
  ->getPksForModel();
 
$query = Doctrine_Query::create()
->select('m.*')
->from('myModel')
->andWhereIn('id',$ids);

This will return you the data from the database.

Tasks

This plugin provides a task to (re)index your models, when data has been modified in the database by other means that doctrine.

lucene:reindex

usage

symfony lucene:reindex

This will use the models defined in lucene.yml (see example in the ajDoctrineLucenablePlugin/config) folder for documentation

symfony lucene:reindex Model1 Model2

This will update and optimize Model1 and Model2.

lucene:optimize

usage

symfony lucene:optimize

This will optimize the index file.

Using your own zend framework

By default, the plugin will use the packaged Zend Framework. To disable this behaviour:

in: lucene.yml

use_packaged_lucene: false

Then add in: ProjectConfiguration.class.php:

public function setup()
{
    // ... load plugins here ... //
    // Here add the listener for the event lucenable.autoload, and connect it to your own
    // ZF autloading method
    $this->dispatcher->connect('luceneable.autoload','ProjectConfiguration::registerZend');
}
 
static protected $zendLoaded = false;
 
static public function registerZend()
{
    if (self::$zendLoaded)
    {
      return;
    }
 
    set_include_path(sfConfig::get('sf_lib_dir').'/vendor'.PATH_SEPARATOR.get_include_path());
    require_once sfConfig::get('sf_lib_dir').'/vendor/Zend/Loader/Autoloader.php';
    Zend_Loader_Autoloader::getInstance();
    self::$zendLoaded = true;
}

Download Zend library from: http://framework.zend.com/download/current/

To use a minimal zend library, download the entire zip, and from the ZendFramwork-1.x.x/library/Zend copy only these to your /lib/vendor/Zend folder:

Zend/Loader/*
Zend/Search/*
Zend/Exception.php
Zend/Loader.php

Extra Documentation

Things you should know

  • lucenable requires you to have a primary key defined, and to call it 'id' (The admin generator will also become funky if you don't do so.)
  • The index folder should be writable
  • It's useful to optimize the index every now and then. (symfony lucene:optimize)
  • This has been tested on small records. For datasets of >100000 perhaps a more industrial solution as java based lucene or solr is recommended.

Classes

The plugin consists of 5 classes * LuceneHandler (Global class, to handle the autoloading and singleton class for lucene) * LuceneRecord (Callback methods, for the lucenable listener) * LuceneSearch (to ease the searching of indexes) * Luceneable (doctrine Behvaiour) * LuceneableListener (Doctrine_Record_Listener)

TODO

  • make it compatible with i18n (can be accomplished by a method that adds a new record in each language, and by adding a mandatory language field to each record).
  • automagickly correct the encoding type
  • create a better task to update it
  • write tests (performance tests, etc.)
  • Evaluate the performance and document the ways to optimize the search results (hand picking a lot of results is not always cool.)