12/29/2022 0 Comments Use apache lucene for indexingSo if you need to score based on custom scoring factors or you need to access per-document values FieldCache provides very efficient access to them. However, there are special cases where other datastructures are used in FieldCache but those are out of scope in this post. Univerting a field to FieldCacheįieldCache serves very well for its purpose since accessing a value is basically doing a constant time array look. When the FieldCache is loaded Lucene iterates all terms in a field, parses the terms values and fills the arrays slots based on the document IDs associated with the term. For simplicity think of an array indexed by Lucene’s internal documents ID. Internally we call this process un-inverting the field since the inverted index is a value to document mapping and FieldCache is a document to value datastructure. Lucene provides a RAM resident FieldCache built from the inverted index once the FieldCache for a specific field is requested the first time or during index reopen. So what if you need custom information to score or filter documents? Stored fields are designed for bulk read, meaning the perform best if you load all their data while during document retrieval we need more fine grained data. So far so good! However, the retrieval process is essentially limited to the information available in the inverted index like term & document frequency, boosts and normalization factors. Once the matching documents have been scored stored fields are loaded for the top N documents for display purposes. In a typical usecase the inverted index is used to retrieve & score documents matching one or more terms. From day one Apache Lucene provided a solid inverted index datastructure and the ability to store the text and binary chunks in stored field.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |