= Introduction = sfLucenePlugin integrates symfony and Zend Search Lucene to instantly add a search engine to your application. The plugin will auto-detect your ORM layer, but currently only supports Propel. (A Doctrine port is coming.) = Requirements = * symfony 1.0.x * Propel = Main Features = * Configured all by YAML files * Complete integration with symfony * i18n ready * Keyword highlighting * Stop words, short words * Index optimization [[Image(sfLucenePlugin-search-results.jpg, 120)]] [[Image(sfLucenePlugin-highlighting.jpg, 120)]] [[Image(sfLucenePlugin-advanced-search.jpg, 120)]] [[Image(sfLucenePlugin-cli.jpg, 120)]] = Installation = * Install the plugin: {{{ symfony plugin-install http://plugins.symfony-project.com/sfLucenePlugin }}} * Initialize configuration files: {{{ symfony lucene-init myapp }}} * Clear the cache {{{ symfony cc }}} * Configure sfLucene per the instructions below. = Configuring Lucene = The entire plugin is configured by search.yml files placed throughout your application. You must be careful that you are aware of what search.yml file you are working in because each one has a different purpose. As you will later learn, the project level search.yml file controls the entire engine while a module's search.yml defines indexing parameters. Open your project's search.yml file, located in {{{ myproject/config/search.yml }}}. If you followed the installation instructions above, you will see at the bottom: {{{ index: name: MyIndex encoding: UTF-8 }}} Enter a name for the index. This is used internally by the plugin and does not matter much. If you require a different encoding to be used, enter it. Note, however, that UTF-8 is generally the best charset to store your indexes in. If you require i18n support, you must define the cultures that you support under index. Use the following syntax: {{{ index: cultures: [en_US, fr_FR] }}} By default, the plugin will not index or search on common words, such as "the" and "a". Further, it ignores single characters. If you require different behavior, you can define them like so: {{{ index: stop_words: [the, an, it] short_words: 2 }}} = Indexing = sfLucene currently supports two ways to add information to the index: 1. Through the ORM layer 2. Through symfony actions Through the ORM layer is the recommended method to add information to the index. The plugin can keep the index synchronized if you use the ORM layer. Through symfony actions is intended only for static content, such as the privacy policy. == ORM layer method == Open your project's search.yml file and you will find a model declaration towards the top. This is where you put the models you wish to index. For each model, you define the fields you want to index and other parameters. The syntax is: {{{ models: BlogPost: fields: id: unindexed title: boost: 1.5 type: text content: unstored description: text BlogComment: fields: id: unindexed summary: text message: text description: message title: summary }}} In the above example, two models are set to index: BlogPost and BlogComment. In BlogPost, the fields title, content, and description are stored, but the title fields holds the most weight with a boost factor of 1.5. When search results are displayed, the system intelligently guesses which field should be displayed as the result "title" and which field is the result "description." However, to be explicit, you can specify a description and title field, as in BlogComment. Note that the fields do not have to exist in your database. As long as it has a getter, you can use it in your index. See [http://framework.zend.com/manual/en/zend.search.lucene.html the Zend_Search_Lucene documentation] for more about the field types. Next, you must tell your application where to route the model when it is returned. You do this by opening your application's config/search.yml file and defining a route: {{{ models: BlogPost: route: blog/showPost?id=%id% BlogComment: route: blog/showComment?id=%id% }}} In routes, %xxx% is a token and will be replaced by the appropriate field value. So, %id% will be the value returned by the ->getId() method. Warning: You must also define the field in the project's search.yml to be indexed or unexpected results will occur! Finally, you must register the model with the system. If you are using Propel, you must use Propel's behaviors. === Propel === You can do this by opening up the model's file and putting {{{ sfLucenePropelBehavior::getInitializer()->setupModel('MyModel'); }}} after the class declaration. So, for a blog, you would open project/lib/model/BlogPost.php and append the above, replacing "!MyModel" with "!BlogPost". === Doctrine === [coming soon] == symfony actions method == To setup an action to be indexed, you must create a file in the module's config directory named search.yml. Inside this file, you define the actions you want indexed: {{{ privacy: tos: security: authenticated: true credentials: [admin] disclaimer: params: advanced: true layout: true }}} As you can see, it is possible to define request parameters, manipulate authentication, and toggle decorating the response. By default, the response is not decorated, the user is not authenticated without any credentials, and there aren't any request parameters. = Building the Index = After you have defined the indexing parameters, you must build the initial index. You do this on the command line: {{{ $ symfony lucene-rebuild myapp }}} replacing myapp with the name of your application you want to rebuild. This will build the index for all cultures. = Searching = sfLucene ships with a basic search interface that you can use in your application. Like the rest of the plugin, it is i18n ready and all you must do is define the translation phrases. To enable the interface, open your application's settings.yml file and add "sfLucene" to the enabled_modules section: {{{ all: .settings: enabled_modules: [default, sfLucene] }}} You are free to define your own routes in the routing.yml file. == Customizing the Interface == As every application is different, it is easy to customize the search interface to fit the look and feel of your site. Doing this is easy as all you must do is overload the templates and actions. To get started, simply run the following on the command line: {{{ $ symfony lucene-init-module myapp }}} If you look in myapp's module folder. you will see a new sfLucene module. Use this to customize your interface. Often, when writing a search engine, you need to display a different result template for each model. For instance, a blog post should show differently than a forum post. You can easily customize your results by changing the "partial" value in your application's search.yml file. For example: {{{ models: BlogPost: route: blog/showPost?slug=%slug% partial: blog/searchResult ForumPost: route: forum/showThread?id=%id% partial: forum/searchResult }}} The partial that you specify is given a $result object that you can use to build that result. The API for this object is pretty simple: * {{{ $result->getInternalTitle() }}} returns the title of the search result. * {{{ $result->getInternalRoute() }}} returns the route to the search result. * {{{ $result->getScore() }}} returns the score / ranking of the search result. * {{{ $result->getXXX() }}} returns the XXX field. In addition to the $result object, it is also given a $query string, which was what the user searched for. This is useful for highlighting the results. = Highlighting Pages = The plugin has an optional highlighter than will attempt to highlight keywords from searches. The highlighter will hook into this search engine and also attempts to hook into external search engines, such as Google and Yahoo!. To enable this feature, open the application's config/filters.yml file and add the highlight filter before the cache filter: {{{ rendering: ~ web_debug: ~ security: ~ # generally, you will want to insert your own filters here highlight: class: sfLuceneHighlightFilter cache: ~ common: ~ flash: ~ execution: ~ }}} By default, the highlighter will also attempt to display a notice to the user that automatic highlighting occured. The filter will search the result document for {{{ <!--[HIGHLIGHTER_NOTICE]--> }}} and replace it with an i18n-ready notice (note: this is case sensitive). To highlight a keyword, it must meet the following criteria: * must be X/HTML response content type * response must not be headers only * must not be an ajax request * be inside the <body> tag * be outside of <textarea> tags * be between html tags and not in them * not have any other alphabet character on either side of it To efficiently achieve this, the highlighter parser assumes that the content is well formed X/HTML. If it is not, unexpected highlighting will occur. The highlighter is also highly configurable. The following filter listing shows the default configuration settings and briefly explains them: {{{ highlight: class: sfLuceneHighlightFilter param: check_referer: on # if true, results from Google, Yahoo, etc will be highlighted. highlight_qs: sf_highlight # the querystring to check for highlighted results highlight_strings: [<strong class="highlight hcolor1">%s</strong>] # how to highlight terms. %s is replaced with the term notice_tag: "<!--[HIGHLIGHTER_NOTICE]-->" # this is replaced with the notice (removed if highlighting does not occur) notice_string: > # the notice string for regular highlighting. %keywords% is replaced with the keywords. i18n ready. <div>The following keywords were automatically highlighted: %keywords%</div> notice_referer_string: > # the notice string for referer highlighting. %keywords% is replaced with the keywords, %from% with where they are from,. i18n ready <div>Welcome from <strong>%from%</strong>! The following keywords were automatically highlighted: %keywords%</div> }}} If you need to configure it more, it is possible to extend the highlighting class. Refer to the API documentation for this. = Categories = Starting with 0.0.5 Alpha, each document in the index can be tied to one or more categories. It is then possible to limit your search results to that category in the provided interface. To enable this, you must define a "category" key to your models or actions. For instance, an example model: {{{ models: Blog: fields: title: text post: text category: text category: [%category%, Blog] }}} The "Blog" model above will be placed both into the blog category and the string returned by ->getCategory() on the model. After you rebuild your model, a category drop down will automatically appear on the search interface. = Using the search Criteria API = sfLucene ships with a basic criteria API for easily constructing queries. The API is ideal for most uses, but if you require more advanced functionality, you should Zend's API. This section will just document the most common ways to interface with the API: * You can either use {{{ $c = new sfLuceneCriteria; }}} or {{{ $c = sfLuceneCriteria::newInstance() }}}. The latter is ideal for method chaining. * To add a search criteria, use the ->add() method. The first argument takes either a Zend API object, a string, or another instance of sfLuceneCriteria. The second argument determines how Lucene should handle this criteria. If you give true (default) to the second argument, then document *must* match that criteria. If you give null, then the document *may* match. If you give false, then the document *may not* match. For example, the following: {{{ $c = sfLuceneCriteria::newInstance()->add('symfony plugins')->add('cakephp', false); }}} will return documents that contain "symfony plugins" but not "cakephp". * If you need to match a field, then you can use ->addField(). ->addField() takes 4 arguments, but only the first one is required. The first one is either a string or an array of values to search under. The second argument is the field name to search under, but if the field is null, then it searches under all fields. The third argument is boolean indicating whether it must match all of the values given. The final argument is how Lucene should handle it (same as above). * Use ->addAscendingSortBy() and ->addDescendingSortBy() to sort. Beware that these will drastically slow down your application. = Integrating sfLucene with another plugin = It is possible to integrate sfLucene with other plugins. To add support to your Propel models, you must append the following: {{{ if (class_exists('sfLucene', true)) { sfLucenePropelBehavior::getInitializer()->setupModel('MyModel'); } }}} The conditional lets your plugin function should the user not have this plugin installed. Then, you must configure sfLucene with your plugin. In project/plugins/sfMyPlugin/config/search.yml, you can define the settings for your models. You can also create a search.yml file in your modules file. But, be warned that these files can be overloaded by the user. = Command Line Reference = The plugin ships with a handful of command line utilities for managing your index. They are listed below: * {{{ $ symfony lucene-about [application] }}} provides information about the plugin and the index. [application] is optitonal. * {{{ $ symfony lucene-init application }}} initializes the search configuration files. * {{{ $ symfony lucene-init-module application }}} initializes a base module for you to customize. * {{{ $ symfony lucene-optimize application }}} optimizes the index for all cultures. * {{{ $ symfony lucene-optimize-culture application culture }}} optimizes the index for the given culture. * {{{ $ symfony lucene-rebuild application }}} rebuilds the index for all cultures. * {{{ $ symfony lucene-rebuild-culture application culture }}} rebuilds the index for the given culture. = Changelog = == Version 0.0.5 Alpha == * Made titles in interface i18n-ready. * Fixed E_STRICT errors in indexers * Search query is now auto-populated in interface. * Fixed type-o in README * Fixed bug in duplicate indexes in i18n models * Added category support * Added a handful of unit tests * Added sfLuceneCriteria API for easily constructing queries == Version 0.0.4 Alpha == * Added exception suite and updated throws. * Removed required dependency on Propel * Made room for Doctrine port * Changed name to sfLucene * Added access keys to the search interface * Made all paths use DIRECTORY_SEPARATOR constant. * Started writing unit tests. * Added "Basic" button to advanced search page * BC: Changed routing tokens to %XXX% -- you must manually update (fixes forum topic 8403). == Version 0.0.3 Alpha == * Added [remove highlighting] button to sfPZSLHighlightFilter * Added integration support for plugins. * BC: Removed many poltergeist anti-patterns * BC: Changed sfPZSL::getInstance() method signature & removed sfPZSL::getCultureInstance() * BC: sfPZSLModelIndexer separated to allow for other ORM layers * Added better automatic i18n detection * Made browser a little bit more robust == Version 0.0.2 Alpha == * Full i18n support. * Fixed index permissions bug between Apache and user. * Fixed explicit title and description bug. * Revamped & improved CLI interface with more feedback to user. * Added boost support and expanded field syntax. * Added highlighting to titles in search interface. * Revamped config handlers. * Improved indexing performance 3-fold. * Added generic advanced search page * Added highlighting filter with hooks into Google, Yahoo, MSN, and Ask. * sfPZSL goes into a reduced state when there are syntax errors in the query. * Added stop word filter support * Added short word filter support == Version 0.0.1 "Proof of Concept" == * First public release = To Do = * Rich media indexing support (PDF, images, MP3) * Improve search interface with ajax * More unit tests! = Discussion Items = * Indexes across applications? = Contribute = All contributions are welcome. The project lives in symfony's SVN repository in plugins/sfLucenePlugin. Feel free to commit your suggestions anytime.