Chapter 19. Zend_Search

Table of Contents

19.1. Overview
19.1.1. Introduction
19.1.2. Document and Field Objects
19.1.3. Understanding Field Types
19.2. Building Indexes
19.2.1. Creating a New Index
19.2.2. Updating Index
19.2.3. Updating Documents
19.2.4. Index optimization
19.3. Searching an Index
19.3.1. Building Queries
19.3.2. Search Results
19.3.3. Results Scoring
19.4. Query Language
19.4.1. Terms
19.4.2. Fields
19.4.3. Term Modifiers
19.4.4. Proximity Searches
19.4.5. Boosting a Term
19.4.6. Boolean Operators
19.4.7. Grouping
19.4.8. Field Grouping
19.4.9. Escaping Special Characters
19.5. Query Types
19.5.1. Term Query
19.5.2. Multi-Term Query
19.5.3. Phrase Query
19.6. Character set.
19.6.1. UTF-8 and single-byte character sets support.
19.7. Extensibility
19.7.1. Text Analysis
19.7.2. Scoring Algorithms
19.7.3. Storage Containers
19.8. Interoperating with Java Lucene
19.8.1. File Formats
19.8.2. Index Directory
19.8.3. Java Source Code
19.8.4. Using LuceneIndexCreation.jar

19.1. Overview

19.1.1. Introduction

Zend_Search_Lucene is a general purpose text search engine written entirely in PHP 5. Since it stores its index on the filesystem and does not require a database server, it can add search capabilities to almost any PHP-driven website. Zend_Search_Lucene supports the following features:

  • Ranked searching - best results returned first

  • Many powerful query types: phrase queries, wildcard queries, proximity queries, range queries and more [6]

  • Search by specific field (e.g., title, author, contents)

Zend_Search_Lucene was derived from the Apache Lucene project. For more information on Lucene, visit http://lucene.apache.org/java/docs/.

19.1.2. Document and Field Objects

Zend_Search_Lucene operates with documents as atomic subjects for indexing. A document is divided into named fields, and fields have content that can be searched.

A document is represented by the Zend_Search_Lucene_Document object, and this object contains Zend_Search_Lucene_Field objects that represent the fields.

It is important to note that any kind of information can be added to the index. Application-specific information or metadata can be stored in the document fields, and later retrieved with the document during search.

It is the responsibility of your application to control the indexer. This means that data can be indexed from any source that is accessible by your application. For example, this could be the filesystem, a database, an HTML form, etc.

Zend_Search_Lucene_Field class provides several static methods to create fields with different characteristics:

<?php
$doc = new Zend_Search_Lucene_Document();

// Field is not tokenized, but is indexed and stored within the index.
// Stored fields can be retrived from the index.
$doc->addField(Zend_Search_Lucene_Field::Keyword('doctype',
                                                 'autogenerated'));

// Field is not tokenized nor indexed, but is stored in the index.
$doc->addField(Zend_Search_Lucene_Field::UnIndexed('created',
                                                   time()));

// Binary String valued Field that is not tokenized nor indexed,
// but is stored in the index.
$doc->addField(Zend_Search_Lucene_Field::Binary('icon',
                                                $iconData));

// Field is tokenized and indexed, and is stored in the index.
$doc->addField(Zend_Search_Lucene_Field::Text('annotation',
                                              'Document annotation text'));

// Field is tokenized and indexed, but that is not stored in the index.
$doc->addField(Zend_Search_Lucene_Field::UnStored('contents',
                                                  'My document content'));

?>

You could give names for fields by your own choice. A "contents" field name is used to search by default. It's good idea to place main document data into this field with this name.

19.1.3. Understanding Field Types

  • Keyword fields are stored and indexed, meaning that they can be searched as well as displayed in search results. They are not split up into separate words by tokenization. Enumerated database fields usually translate well to Keyword fields in Zend_Search_Lucene.

  • UnIndexed fields are not searchable, but they are returned with search hits. Database timestamps, primary keys, file system paths, and other external identifiers are good candidates for UnIndexed fields.

  • Binary fields are not tokenized or indexed, but are stored for retrieval with search hits. They can be used to store any data encoded as a binary string, such as an image icon.

  • Text fields are stored, indexed, and tokenized. Text fields are appropriate for storing information like subjects and titles that need to be searchable as well as returned with search results.

  • UnStored fields are tokenized and indexed, but not stored in the index. Large amounts of text are best indexed using this type of field. Storing data creates a larger index on disk, so if you need to search but not redisplay the data, use an UnStored field. UnStored fields are practical when using a Zend_Search_Lucene index in combination with a relational database. You can index large data fields with UnStored fields for searching, and retrieve them from your relational database by using a separate fields as an identifier.

    Table 19.1. Zend_Search_Lucene_Field Types

    Field Type Stored Indexed Tokenized Binary
    Keyword Yes Yes No No
    UnIndexed Yes No No No
    Binary Yes No No Yes
    Text Yes Yes Yes No
    UnStored No Yes Yes No


[6] Only term, multi term and phrase queries are supported at this time.