Lucene in Action

...Before we get into the gory details of what lurks inside an analyzer, let's look at how an analyzer is used within Lucene. Analysis occurs at two spots: during indexing and when using QueryParser ... square brackets to make the separations apparent. During indexing, the tokens extracted during analysis are the terms indexed. And, most important, the terms indexed are the terms that are searchable ... of the analysis process visible to the end user. Terms pulled from the original text are indexed and are matched...

...An important point about analyzers is that they're used internally for fields flagged to be tokenized. Documents such as HTML, Microsoft Word, XML, and others, contain meta-data such as author, title, last modified date, and poten- tially much more. When you're indexing rich documents, this meta-data should be separated and indexed as separate fields. Analyzers are used to analyze a spe- cific ... options for indexing them; it also discusses parsing various document types in detail....

...Lucene does not have a standard object for searching multiple indexes with dif- ferent queries. Because jGuru needs to search the foreign database versus its internal search databases with slightly different query terms, I made a subclass of Lucene's MultiSearcher, JGuruMultiSearcher (shown ... and ScoreDoc as well as the Searcher interface. Listing 10.1 Searching multiple indexes ... on each index and merges the results like this JGuruMultiSearcher. protected TopDocs search(Query query...

... was the fast generation of extracts for a given search result hit. We currently index the docu- ment ... is that this method is not really efficient when indexing long documents: the index becomes very large...

... the search criteria. Rebuilding the search index involved taking the search facility offline ... the customer's patience. Scalability--The tool must scale well both in terms of the amount of data indexed as well as with the site's load during peak traffic. Robustness--The index must be frequently...

... 17 Authors' note: Index optimization is covered in section 2.8. from Doug's Nutch efforts. The Nutch ... is incorporated yields nifty effects. Indexing hexadecimal RGB values and providing external indexing...

... on your architecture. Supposedly, if the indexes reside on different physical disks and you're able ... CPU, single physical disk, and multiple indexes, performance with MultiSearcher was slightly better ... . An exam- ple, using ParallelMultiSearcher remotely, is shown in listing 5.9. Searching multiple indexes remotely Lucene includes remote index searching capability through Remote Method Invocation (RMI ... remote (and/or local) indexes, and each server could search only a single index. In order...

...A major hurdle (unrelated to Lucene) remains when you're dealing with various languages: handling text encoding. The StandardAnalyzer it still the best built-in general-purpose analyzer, even accounting for CJK characters; however, the Sandbox CJKAnalyzer seems better suited for Asian language analysis. When you're indexing documents in multiple languages into a single index, using a per-Document analyzer is appropriate. You may also want to add a field to documents indicating their language...

...Because of the nature of PyLucene ("compiler and SWIG gymnastics"), its indexes are compatible with those of Lucene....

...Searching is the process of looking up words in an index to find documents where they appear. The quality of a search is typically described using precision and recall metrics. Recall measures how well the search system finds relevant docu- ments, whereas precision measures how well the system filters out the irrelevant documents. However, you must consider a number of other factors when think- ing about searching. We already mentioned speed and the ability to quickly search large quantities... [Full sample chapter]

...Not all Documents and Fields are created equal--or at least you can make sure that's the case by selectively boosting Documents or Fields. Imagine you have to write an application that indexes ... Lucene to consider it more or less important with respect to other Documents in the index. The API ... is a company employee. b When we index messages sent by the company's employees, we set their boost ... . Imagine that another requirement for the email-indexing application is to consider the subject Field...

...Suppose you're tasked with adding search to an application. You've tackled get- ting the data indexed, but now it's time to expose the full-text searching to the end users. It's hard to imagine ... . Table 3.1 Lucene's primary searching API Class Purpose IndexSearcher Gateway to searching an index ... is returned from IndexSearcher's search method. When you're querying a Lucene index, an ordered ... that will be pre- sented to the user. For large indexes, it wouldn't even be possible to collect all matching... [Full sample chapter]

... left by the words removed? Suppose you index "one is not enough". The tokens emitted from StopAnalyzer ... for words removed, so the result is exactly as if you indexed "one enough". If you were to use ... analyzes phrases, and each of these reduces to "one enough" and matches the terms indexed. There is a "hole ... , only the tokens emitted from the analyzer (or indexed as Field.Keyword) are available for searching....

...It's easy to index a keyword using Field.Keyword, which is a single token added to a field that bypasses tokenization and is indexed exactly as is as a single term. It's also straightforward to query ... indexing. There is nothing special about keyword fields once they're indexed; they're just terms. Let's see the issue exposed with a straightforward test case that indexes a doc- ument with a keyword ... ); here assertEquals(1, hits.length()); Document found } as expected } So far, so good--we've indexed...

...dotLucene is compatible with Lucene at the index level. That is to say, an index created by Lucene can be read by dotLucene and vice versa. Of course, as Lucene evolves, indexes between versions of Lucene itself may not be portable, so this compatibility is currently limited to Lucene version 1.4....

... the documents and index them. Indexing is carried out with the Apache Lucene search engine. The Lucene indexer itself buffers documents in a RAMDirectory, using a separate thread to merge them periodically with an on-disk index. Figure 10.4 Alias-i Tracker architecture The next two stages...

... TheServer- Side built an infrastructure that allows us to index and search our different content using Lucene. We will chat about our high-level infrastructure, how we index and search, as well as how ... ? We were using ht://Dig and having it crawl our site, building the index as it went along ... me to build an index just like ht://Dig was doing. At the time, LARM was in the Lucene Sandbox...

...A stream of tokens is the fundamental output of the analysis process. During indexing, fields designated for tokenization are processed with the specified ana- lyzer, and each token is written to the index as a term. This distinction between tokens and terms may seem confusing at first. Let's see ... into terms After text is analyzed during indexing, each token is posted to the index as a term ... to the index. Start and end offset as well as token type are discarded--these are only used during...

...According to Plucene's author, indexes created by Lucene 1.3 and Plucene 1.19 are compatible. A Java application that uses Lucene 1.3 will be able to read and digest an index created by Plucene 1.19 and vice versa. As is the case for other ports with compatible indexes, indexes between versions of Lucene itself may not be portable as Lucene evolves, so this compatibility is restricted to Lucene version 1.3....

... to terms with the following method: public static Directory index(String[] terms) { Directory ... are TF/IDF weighting of the n-gram vectors, indexing of terms by n-grams, and cosine computation ... names were then indexed using 2-grams, 3-grams, and 4-grams. Then each of the names was used ... to read the strings from a file, index them in mem- ory, optimize the index, and then parse...