Technical details

Trove Boolean Logic

This article has been written to assist users in understanding the behaviour of Trove's handling of boolean operators ('AND', 'OR' and 'NOT') in compiling search results. The article will become progressively more technical as it continues, since many of our users have expressed interest in these sorts of details. Feel free to continue reading until the level of detail exceeds your geek threshold and/or interest levels.

Some simple advice

  1. Consider whether you even need Boolean operators first. Trove's 'Basic Search' functionality is fairly smart and will try hard to find relevant results for you with just basic search terms. The default operator is 'AND', so simply typing additional search terms will narrow your results set further.
  2. If you want to perform more complicated searches without building your own Boolean queries, you can use the 'Advanced Search' screen. Trove will combine the search criteria into a Boolean search on your behalf.
  3. Of course there are still a lot of scenarios where users will want to provide their own Boolean logic, which Trove most definitely supports. But please keep in mind the #1 Golden Rule of Boolean searching in Trove: Always use parentheses to wrap boolean search clauses!

Details on this are covered later, but at a basic level this means that users are discouraged from searching for terms like this:

term1 AND term2 OR term3

And should instead use:

(term1 AND term2) OR term3

If readers are familiar with normal Boolean logic (particularly operator precedence) you could be forgiven for thinking those two statements are the same. However they are not the same in Trove, and you will get very different search results by mistake.

It really is a golden rule

Things will start to get more technical from here on, so feel free to heed just the advice given above and know that you can happily search Trove in any way you desire.

To confuse matters further, the next logical assumption about how Trove would interpret that result is probably this:

term1 AND (term2 OR term3)

But this would unfortunately be incorrect as well. Trove would really interpret the query as something like:

term1

And 'something like' is very apt here, if somewhat confusing, because in reality the way Trove handles your query can't really be expressed in terms of Boolean logic.

Why do you do this, Trove? 

The discrepancy originates in the underlying search technologies upon which Trove is built; Lucene and Solr, which are largely the same in this regard, but we are focusing on Solr. At face value Solr understands Boolean searches, but in practice it is translating your Boolean operators into another form. This translation does not always go as expected however.

So how do they differ? Well Boolean operators are a way of logically combining two search terms (or clauses); those to the left and right of the operator. It makes no sense see a Boolean operator in isolation beside an individual search term, because each term contributes to the requirements of the search in a way that cannot be disconnected from the terms around it. This makes perfect sense in Boolean terms because the end-goal of evaluating a Boolean statement is to rule each record either in or out of the result set.  It is a binary proposition in that a record either matches or does not with no ambiguity in between.

Solr however works with the concepts of 'MUST', 'MUST_NOT' and 'SHOULD' because it is instead trying to compile a list of search results that it can sort by how relevant they are compared to what you are looking for. By implication, a record with low relevancy may be one that matches only some of your search terms but not others. Search terms falling into the 'SHOULD' category are considerable optional terms that do not need to be present, and these are the particularly problematic ones, since this is not an idea that Boolean logic can completely encompass. The gap between the two approaches is often the cause of confusion. The 'OR' operator and 'SHOULD' terms have a somewhat similar purpose, but they are not completely compatible because 'OR' operators relate to 2 search terms, but 'SHOULD' applies to a single term at a time.

We have been careful above not to talk about 'MUST', 'MUST_NOT' and 'SHOULD' as operators themselves, since they are not. Solr calls these 'Occur' flags, but that doesn't really matter. To flag a search term as 'MUST' occur in Solr you can place the '+' operator in front the term, or use the '-' operator to flag it as 'MUST_NOT'. There is however no operator for 'SHOULD' occur, and Solr instead considers the absence of an operator to mean 'SHOULD'.

Umm... what?

The above description is very dry and academic, so it is understandable if the nature of the problem does not immediately sink in. Here is an example to demonstrate the difference. Say you want to generate a Solr search similar to this with a single mandatory term and two optional terms:

+term1 term2 term3

Every record in the result set must have 'term1' at the very minimum, and the presence of either 'term2' or 'term3' will give a record a relevancy boost that brings it closer to the top of the results list. If both of the optional terms are found the record would receive a second boost elevating its position once again.

Unfortunately there is no equivalent to this search in Boolean terms because there is no operator that we can use between the first two terms that would respect our intention to keep a mandatory term on the left and an optional term on the right. Here is a few attempts that all fail (Boolean on the left, Solr translation on the right):

term1 AND (term2 OR term3)  =>  +term1 +(term2 term3)  : 1 of the two optionals is required

(term1 AND term2) OR term3  =>  (+term1 +term2) term3  : 1 of the optionals is now a mandatory

term1 OR (term2 OR term3)  =>  term1 (term2 term3)  : the mandatory term is no longer mandatory

Some of those examples are obviously silly, but the key point it that there is no way to represent this search in Boolean logic (ignoring Solr's version of Boolean logic) that is true to normal Boolean operator precedence.

Solr Operators

This is presumably why Solr has a different process for interpreting operators (source):

  • Queries are parsed left to right
  • NOT sets the Occurs flag of the clause to it’s right to MUST_NOT
  • AND will change the Occurs flag of the clause to it’s left to MUST unless it has already been set to MUST_NOT
  • AND sets the Occurs flag of the clause to it’s right to MUST
  • If the default operator of the query parser has been set to “And”: OR will change the Occurs flag of the clause to it’s left to SHOULD unless it has already been set to MUST_NOT

The last point here is the most significant, because Trove (like many discovery systems in the Library sector) uses 'AND' as the default operator. Evaluating those points in order against our original, ambiguous, example search (the one we forgot to use parentheses on)...

term1 AND term2 OR term3

... gives us this (evaluating the final 'OR' operator has flipped 'term2' to a 'SHOULD' occur term):

+term1 term2 term3

Look familiar? It is the example we said can't be represented by Boolean logic, but using Solr's 'sort of' Boolean logic we can get what we want. Whether it is intentional or not, this lets advanced users remove the parentheses to generate searches that Boolean logic would never allow you to perform.

Phrase Searching

When you are performing phrase searches in the Trove newspapers or gazettes zones there are a few considerations that will be of interest to you.

Do you need a phrase?

The 'Basic Search' feature will automatically perform a phrase search on your behalf for most queries. For example, if you search for this (notice no quotes):

sydney harbour bridge

Trove will perform this search for you against Solr:

+(+sydney +harbour +bridge) "sydney harbour bridge"~5

The specifics of this search will make more sense if you have read our help article on how Boolean logic works in Solr, but broadly speaking, this search requires that all of our search terms are present in the document and the most relevant articles in your result set will have the search terms very close together. The closest together they can get (and the most relevant) would be anything that appears as an exact phrase.

So consider trying a 'Basic Search' first, you might find some articles you would have missed with a simple phrase search.

What's that squiggly thing?

In the example above you will notice '~5' following the phrase.Trove allows you to search for phrases with a little bit of imprecision to find things that almost match what you asked for; this is known as 'phrase slop'. Perhaps you want to search for "Charles Dickens" but the article records the name as "Dickens, Charles"? A 'phrase slop' value of ~2 would allow this match to occur.

The 'Basic Search' phrase search will always apply a phrase slop of ~5, since it is not trying to be particularly precise about its phrase match. But it is important to note here that exact phrases are still considered more relevant when ordering your search results; each 'point' of slop that must be applied to get an article to match your search will lower its relevancy until the slop exceeds the limit you have requested and the article is ignored.

I'll do my own phrase searching, thanks!

There are plenty of scenarios where you will still want to provide your own phrase search directly. You should keep in mind in these cases that Trove will provide a default phrase slop of ~1 in order to account for many of the oddities found in newspapers and gazettes data and peoples names. In most cases this is helpful, but if you are still finding too many articles you don't want in your search results you can be explicit about it using ~0 to exclude all slop. Eg.:

"sydney harbour bridge"~0