Comments on: Palantir: search with a twist (part one: memory efficiency) http://blog.palantirtech.com/2009/08/13/palantir-search-with-a-twist-part-one-memory-efficiency/ Articles from the Engineering Group at Palantir Technologies Tue, 24 Jan 2012 09:51:18 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: Katherine http://blog.palantirtech.com/2009/08/13/palantir-search-with-a-twist-part-one-memory-efficiency/comment-page-1/#comment-240 Katherine Thu, 01 Oct 2009 23:20:52 +0000 http://blog.palantirtech.com/?p=1088#comment-240 Mark, Sorry if the blog post didn't make it clear - we think Lucene is some of the best third-party searching software out there, which is why we use it as the base index/search engine, and we're not trying to discourage people from using it. However, no generic library is going to fit our exact use case, and we're outlining some places where we've run into that. The post also elided over some technical details for the sake of brevity - apologies if you feel that amounted to a misrepresentation of Lucene. About loading results: Lucene does use a PriorityQueue for the results, so our description would be substantially different from what actually happens if we didn't want access to all the matching results. Unfortunately, our use case requires that users be able to see all matching results if they want to - in order to get n results from out-of-the-box Lucene, even if you plan to break them into k pages of size n/k, you have to size the PriorityQueue to hold n results. This means that you're effectively loading all matching results - supporting paging over chunks of results is not built in. About RangeQueries: We initially started using Lucene in 2005, so some of our custom code has been replicated in later versions. Some of it hasn't - for instance, we use the same range-query logic for both ranges and for broad wildcards, although only numeric ranges were discussed in the post. The latter does not have support that I know of as of Lucene 2.4 (the version we're currently using, and the most current release as of the time of the original post). However, the 2.9 release that came out last week looks like it may have added similar functionality, using index-size-based cutoffs instead of the memory-size-based ones we use (I haven't traced through all the new source code yet, but that's what I've seen so far). And, since Lucene is a generic platform, some custom code has been replicated in a more generic version. For example, Payloads, which were introduced after we started using Lucene, are similar to but not as optimized for our use case as how we do security enforcement. Mark,
Sorry if the blog post didn’t make it clear – we think Lucene is some of the best third-party searching software out there, which is why we use it as the base index/search engine, and we’re not trying to discourage people from using it. However, no generic library is going to fit our exact use case, and we’re outlining some places where we’ve run into that. The post also elided over some technical details for the sake of brevity – apologies if you feel that amounted to a misrepresentation of Lucene.

About loading results: Lucene does use a PriorityQueue for the results, so our description would be substantially different from what actually happens if we didn’t want access to all the matching results. Unfortunately, our use case requires that users be able to see all matching results if they want to – in order to get n results from out-of-the-box Lucene, even if you plan to break them into k pages of size n/k, you have to size the PriorityQueue to hold n results. This means that you’re effectively loading all matching results – supporting paging over chunks of results is not built in.

About RangeQueries: We initially started using Lucene in 2005, so some of our custom code has been replicated in later versions. Some of it hasn’t – for instance, we use the same range-query logic for both ranges and for broad wildcards, although only numeric ranges were discussed in the post. The latter does not have support that I know of as of Lucene 2.4 (the version we’re currently using, and the most current release as of the time of the original post).
However, the 2.9 release that came out last week looks like it may have added similar functionality, using index-size-based cutoffs instead of the memory-size-based ones we use (I haven’t traced through all the new source code yet, but that’s what I’ve seen so far).
And, since Lucene is a generic platform, some custom code has been replicated in a more generic version. For example, Payloads, which were introduced after we started using Lucene, are similar to but not as optimized for our use case as how we do security enforcement.

]]> By: Mark Harwood http://blog.palantirtech.com/2009/08/13/palantir-search-with-a-twist-part-one-memory-efficiency/comment-page-1/#comment-239 Mark Harwood Thu, 01 Oct 2009 16:26:09 +0000 http://blog.palantirtech.com/?p=1088#comment-239 Some derogatory and misleading information here about Lucene.   >>Vanilla Lucene uses the following algorithm for accumulating search results: Load all matching results.   No it doesn't and never has. PriorityQueues are used everywhere.     >>RangeQuery… has one very nasty property   That is why the Javadocs tell you not to use it and the QueryParser hasn't supported it as the default for quite some time. The alternative approach you describe is implemented in ConstantScoreRangeQuery which is the newer default in the query parser.   Are you working on very old versions of Lucene? If so please target your comments at something vaguely recent or take the trouble to read the current documentation more closely before spreading bad advice. Some derogatory and misleading information here about Lucene.
 
>>Vanilla Lucene uses the following algorithm for accumulating search results: Load all matching results.
 
No it doesn’t and never has. PriorityQueues are used everywhere.
 
 
>>RangeQuery… has one very nasty property
 
That is why the Javadocs tell you not to use it and the QueryParser hasn’t supported it as the default for quite some time.
The alternative approach you describe is implemented in ConstantScoreRangeQuery which is the newer default in the query parser.
 
Are you working on very old versions of Lucene? If so please target your comments at something vaguely recent or take the trouble to read the current documentation more closely before spreading bad advice.

]]>