# Introduction

sfPropelZSLSearchPlugin integrates symfony, Propel, and Zend Search Lucene together to instantly add a search engine to your application.  Documents are either created off Propel models or HTML output from actions.  The plugin ships with a fully functional search interface that you can easily add to your application.  The goal of this plugin is to be a flexible search engine that can integrate with other plugins, such as sfSimpleCMSPlugin or sfSimpleBlogPlugin.

# Features

  * Instant search engine by just editing a few yaml files
  * i18n ready with unlimited cultures
  * Stop words and short words out-of-box
  * Keyword highlighting from Lucene and other engines, such as Google and Yahoo!
  * Completely integrates with symfony, Propel, and Zend Search Lucene

# Plugin Status

This plugin is still a alpha release.  Use at your own risk!  I encourage you to use it, but if you wake up and find your hard drive deleted, then you know why.  A beta will not come until after symfony 1.1 ships.  This documentation is still much a work in progress.  It assumes that you have knowledge of symfony, Propel, and basic plugin tasks.  If you find a bug or have a feature suggestion, please let me know!

# Installation

  * Install the plugin:
    
    $ symfony plugin-install http://plugins.symfony-project.com/sfPropelZSLSearchPlugin
    
  * Initialize configuration files (Don't do this if you are upgrading):
    
    $ symfony pzsl-init myapp
    
  * Clear the cache:
    
    $ symfony cc
    
  * Configure the search engine per the instructions below

# Indexing

There are currently two supported ways to add documents to the search engine:

  1. Directly through the models, or
  2. Output from actions

Whenever possible, it is recommended to use the first option.  The action output is often error-prone.

The entire search engine is configured by search.yml files, located in your project's config directory, your applications' config directories, and your modules' config directories, which is where you set up the various indexing options.

## Indexing Models

Setting up the plugin to index your models is easy.  First, you must define the models you want to index in your project's search.yml file.  The example below shows a search.yml for a blog:

    
    models:
      BlogPost:
        fields:
          title:
            boost: 1.5
            type: text
          content: unstored
          author: unindexed
          friendlytitle: text
          description: shortdescription
        title: friendlytitle
        description: shortdescription
      BlogComment:
        fields:
          title: keyword
          content: unstored
        title: title
    
    index:
      name: MyIndex
      encoding: UTF-8
      cultures: [fr_FR](en_US,)
    

See [the Zend_Search_Lucene documentation](http://framework.zend.com/manual/en/zend.search.lucene.html) for more about the field types.

  * If you look carefully at the above example, you will notice that the "title" for "!BlogPost" is set to the "friendlytitle" field.  This means that when this model is returned, the title for the hit will be the result returned from ->getFriendlyTitle() on the model.  If you do not specify a title field, the system will guess the title. The same applies for the description.  Note: The field you specify in "title" and "description" must also be presents in the fields list.
  * By specifying the boost, you tell the system that this field is more important than all the others.

Next, you need to tell the engine where to send the user should the model be returned.  You do this by creating your application's config/search.yml and defining a route for the model:

    
    models:
      BlogPost:
        route: blog/showPost?slug={$slug}
      BlogComment:
        route: blog/showComment?id={$id}
    

In routes, {$xxxx} is a token and will be replaced by the appropriate field value.  So, {$slug} will be the value returned by ->getSlug() on the model.

Finally, you must register the models with the plugin so that they will automatically update the index.  You can do this by opening up the model's file and putting
    
    sfPZSLBehavior::getInitializer()->setupModel('MyModel');
    
after the class declaration.  So, for a blog, you would open project/lib/model/BlogPost.php and append the above, replacing "!MyModel" with "!BlogPost". (Note: this is the same thing as a Propel behavior.)

You're now set.  If you want to index your already existing data, simply run
    
    $ symfony pzsl-rebuild myappp
    
on the command line, replacing "myapp" with your application.

## Indexing Actions

Note: This is not recommended for dynamic data because the actions are only updated when you rebuild the index.  Also, currently, it is not possible to index more than one application (if anyone has a nice way of doing this, let me know).

To setup an action to be indexed, you must create a file in the module's config directory named search.yml.  Inside this file, you define the actions you want indexed:

    
    privacy:
    tos:
      security:
        authenticated: true
        credentials: [admin]
    disclaimer:
      params:
        advanced: true
      layout: true
    

As you can see, it is possible to define request parameters, manipulate authentication, and toggle decorating the response.  By default, the response is not decorated, the user is not authenticated without any credentials, and there aren't any request parameters.

After you have saved this file, you can rebuild the index by:

    
    $ symfony pzsl-rebuild myapp
    

# Searching the Index

The system ships with a very basic search interface that you can use as a search engine on your public application.  To enable the engine, open the application's settings.yml file and in the enabled_modules section, add 'sfPropelZSLSearch'.  It will look like:

    
    all:
      .settings:
        enabled_modules: [sfPropelZSLSearch](default,)
    

You can now access the search engine in your project: ` http://localhost/myproject/web/sfPropelZSLSearch `.  You can define your own routes in the routing.yml file.

If you want to add a search box to your navigation for instance, simply make a call to the following component:
    
    <?php include_component('sfPropelZSLSearch', 'publicControls') ?>
    

## Customizing the Interface

You can customize the search results by overriding various templates and actions to tailor the look and feel to your application.  To do this, first create the folder ` yourapp/modules/sfPropelZSLSearch `.   To override the templates, then create a templates directory: ` yourapp/modules/sfPropelZSLSearch/templates `.  To see the templates that you can override, browse to the files in ` plugins/sfPropelZSLSearchPlugin/modules/sfPropelZSLSearch/templates `.

Often, when writing a search engine, you need to display a different result template for each model.  For instance, a blog post should show differently than a forum post.  You can easily customize your results by changing the "partial" value in your application's search.yml file.   For example:
    
    models:
      BlogPost:
        route: blog/showPost?slug={$slug}
        partial: blog/searchResult
      ForumPost:
        route: forum/showThread?id={$id}
        partial: forum/searchResult
    

The partial that you specify is given a $result object that you can use to build that result.  The API for this object is pretty simple:

  * ` $result->getInternalTitle() ` returns the title of the search result.
  * ` $result->getInternalRoute() ` returns the route to the search result.
  * ` $result->getScore() ` returns the score / ranking of the search result.
  * ` $result->getXXX() ` returns the XXX field.

In addition to the $result object, it is also given a $query string, which was what the user searched for.  This is useful for highlighting the results.

# Highlighting Pages

The plugin has an optional highlighter than will attempt to highlight keywords from searches.  The highlighter will hook into this search engine and also attempts to hook into external search engines, such as Google and Yahoo!.

To enable this feature, open the application's config/filters.yml file and add the highlight filter before the cache filter:
    
    rendering: ~
    web_debug: ~
    security:  ~
    
    # generally, you will want to insert your own filters here
    
    highlight:
      class: sfPZSLHighlightFilter
    
    cache:     ~
    common:    ~
    flash:     ~
    execution: ~
    

By default, the highlighter will also attempt to display a notice to the user that automatic highlighting occured.  The filter will search the result document for ` <!--[HIGHLIGHTER_NOTICE]--> ` and replace it with an i18n-ready notice (note: this is case sensitive).

To highlight a keyword, it must meet the following criteria:
  * must be X/HTML response content type
  * response must not be headers only
  * must not be an ajax request
  * be inside the <body> tag
  * be outside of <textarea> tags
  * be between html tags and not in them
  * not have any other alphabet character on either side of it

To efficiently achieve this, the highlighter parser assumes that the content is well formed X/HTML.  If it is not, unexpected highlighting will occur.

The highlighter is also highly configurable.  The following filter listing shows the default configuration settings and briefly explains them:

    
    highlight:
      class: sfPZSLHighlightFilter
      param:
        check_referer: on # if true, results from Google, Yahoo, etc will be highlighted.
        highlight_qs: sf_highlight # the querystring to check for highlighted results
        highlight_strings: [class="highlight hcolor1">%s</strong>](<strong) # how to highlight terms.  %s is replaced with the term
        notice_tag: "<!--[HIGHLIGHTER_NOTICE]-->" # this is replaced with the notice (removed if highlighting does not occur)
        notice_string: > # the notice string for regular highlighting.  %keywords% is replaced with the keywords.  i18n ready.
          <div>The following keywords were automatically highlighted: %keywords%</div>
        notice_referer_string: > # the notice string for referer highlighting.  %keywords% is replaced with the keywords, %from% with where they are from,.  i18n ready
          <div>Welcome from <strong>%from%</strong>!  The following keywords were automatically highlighted: %keywords%</div>
    

If you need to configure it more, it is possible to extend the highlighting class.  Refer to the API documentation for this.

# i18n

The plugin ships i18n ready and can handle any number of cultures.  The system hooks into symfony's i18n management and automatically detects internationalized tables from your schema.  Internally, each culture is a separate index to improve performance.  As a result, updates can be expensive because the plugin has to update n indexes each time you save a Propel object, where is n is the number of cultures you support.  But, this is the nature of the game.

By default, the plugin has i18n enabled.  If you do not want to use i18n, then you do not need to do anything.  But, if you do want to use i18n, then the only thing you must do is define the cultures that you have enabled.  To do this, open your project's search.yml file. Under the "index" category, define the "cultures" array:

    
    index:
      name: MyIndex
      cultures: [fr_FR, es_ES](en_US,)
    

The generic search interface that ships with symfony is also i18n-ready.  All of the templates use the appropriate helpers.  All you must do is define the phrases for each language you support.  Refer to the symfony documentation for details on how to do this.

# Integrating sfPZSL with another plugin

It is possible to integrate sfPZSL with other plugins.  To add support to your Propel models, you must append the following:
    
    if (class_exists('sfPZSL', true))
    {
      sfPZSLBehavior::getInitializer()->setupModel('MyModel');
    }
    

The conditional lets your plugin function should the user not have this plugin installed.

Then, you must configure sfPZSL with your plugin.  In project/plugins/sfMyPlugin/config/search.yml, you can define the settings for your models.  You can also create a search.yml file in your modules file.  But, be warned that these files can be overloaded by the user.

# Command Line Reference

The plugin ships with a handful of command line utilities for managing your index.  They are listed below:

  * ` $ symfony pzsl-about [` provides information about the plugin and the index.  [application](application]) is optitonal.
  * ` $ symfony pzsl-init application ` initializes the search configuration files.
  * ` $ symfony pzsl-optimize application ` optimizes the index for all cultures.
  * ` $ symfony pzsl-optimize-culture application culture ` optimizes the index for the given culture.
  * ` $ symfony pzsl-rebuild application ` rebuilds the index for all cultures.
  * ` $ symfony pzsl-rebuild-culture application culture ` rebuilds the index for the given culture.

# Common Pitfals

  * The webserver daemon and you need read and write access to the index.  The index is stored in your project's data/index folder, so make sure that you can write to all the files and folders in that directory.  The plugin should automatically chmod the files to 0777 for the index, but if you get a permissions error, then you need to manually apply the permission changes.
  * You may end up with duplicate documents if you use ` propel-load-data `.  To fix this, simply run ` $ symfony pzsl-rebuild myapp `.
  * The index cannot exceed 2GB on 32bit systems or 4GB on 64bit systems.  This is a Lucene limitation.
  * You have to remember to define a route for each model, otherwise an exception will be thrown asking you to define a route.
  * You should optimize the index from time to time.  To determine if you need to optimize, run ` symfony pzsl-about myapp ` and to begin the optimization, run ` symfony pzsl-optimize myapp `.

# To Do

  * Indexing for non-text documents
  * Make the plugin more extensible & configurable
  * Unit tests
  * Find a better way to paginate
  * Indexes across applications
  * Improve the search interface (ajax?)
  * Improve documentation
  * [Doctrine port](http://www.symfony-project.com/forum/index.php/t/8403/)

# Contribute

Contributions and criticism are always welcome! :-)