The most common problem reported in Spotlight search is failure to find target file(s). In this series of articles, I’ll examine what can go wrong, and how you can make local searches more successful using features in macOS. I’m well aware that there are other utilities for searching, some relying on Spotlight, others working independently, but here I’ll confine myself as much as possible to what’s provided in macOS.

Successful searches using Spotlight have four essential requirements:

• The target file(s) must have had their metadata added to Spotlight’s volume indexes.
• Those metadata must be accessible to the tool used to perform the search.
• The search queries used must be capable of finding the target(s).
• Spotlight’s search must work correctly.

Check indexes and access

The first of those can be checked using the mdimport command tool in Terminal, using the command
mdimport -t -d3 [filepath]
where [filepath] is the path of the file. You can vary the digit used in the -d3 option from 1-3, with larger numbers delivering more detail. For -d3 you’ll first be given the file type and mdimporter used, following which all the data extracted is given according to its Spotlight attributes:
Imported '/Users/hoakley/Documents/SpotTestA.rtf' of type 'public.rtf' with plugIn /System/Library/Spotlight/RichText.mdimporter.
37 attributes returned
followed by a long list.

If the file hasn’t been indexed, this article works through the steps you can take to rectify that. Note that recent experience is that using mdutil -E / to erase and force rebuild of indexes on the Data volume may not work as expected, and you should either perform this in System Settings, or using the command mdutil -E /System/Volumes/Data

Which tool?

Global Spotlight is accessed through the magnifying glass icon in the right of the menu bar, or using the default key combination of Command-space. This includes website content, and isn’t ideal when you’re searching for local files. If you want to use this as an easy gateway to local search, enter the text you want to search for and scroll down to use the command Search in Finder, which opens a Finder Find window for the results of that search query. Alternatively, you can click on Show More in the Documents section of the search results.

Local Spotlight can also be opened by the Find command at the foot of the Finder’s File menu, and takes you straight to its search box, at the right of the toolbar in that Finder window.

This window offers a choice of two search scopes at the upper left. The first covers all the accessible contents of that Mac, and the second is the folder open in that window when it was converted to a Find window. To set the scope for a local Spotlight search, start from a normal Finder window with the target folder open, then use the Find command on that.

Searching

Typing text into the search box at the right of the toolbar then performs live or incremental search for both filenames and content at the same time, or you can select one of them in the menu.

Text entered into the search box can simply be the start of a word in the target, or can be a basic search query such as name:"target"*cdw. I will explain those in a later article about search queries.

Instead of, or in addition to, entering text in the search box, you can set further search criteria in search bars below that.

In this case, the file Name is required to contain the string entered in the text box. Add more of these search bars for additional criteria to narrow your search.

Search attributes

In search bars, the first popup sets the metadata attribute to use in the query. For example, both Name and Filename refer to the name of the file, although Name is given in its localised form. Many more options are available by selecting the Other… item at the foot of the attribute menu. You can either set those as a one-off, or add them to the menu of attributes if you’re likely to use them again. These roughly correspond to the metadata attributes as in formal Spotlight search queries used elsewhere, although their names are different.

The second popup sets the operator to be used. While they may appear self-explanatory, two merit fuller explanation as they may work differently from how you might expect:

• matches finds the start of words found by breaking the whole string at word boundaries (see below). Thus it will find danger and danger-es in one-danger-est, but won’t find anger there.
• contains is applied without reference to word boundaries, and simply finds the given string, regardless of what characters occur before or after that. Thus it will find anger in dangerous. It has one unusual behaviour, though: it ignores hyphens, so dangeres will be found in the string danger-est.

Word boundaries

These are crucial in search queries run from the search box in a Find window, and the matches operator used in a search bar below. Although the search box claims to use the contains operator, it actually behaves as the matches operator does in a search bar.

In many languages word roots and meaning appear at the start of words, with declensions and conjugations at the end. If you want to find words related to harvest, like harvester, harvesting and harvested, then you’re going to enter a search query using harvest rather than vest. Like other search engines designed for live or incremental search, Spotlight is fastest when searching for the start of words. It therefore divides compound words often used for filenames into component words. It does so using rules for word boundaries laid down in the International Components for Unicode.

In practice, word boundaries include a space, the underscore _, hyphen – and changes of case used in CamelCase. Spotlight treats each of the following examples as three words:
one target two
one_target_two
one-target-two
OneTargetTwo
Languages other than English may allow other word boundaries, but those are the most common.

The rules recognise that hyphens are difficult, and Spotlight makes them even trickier as it can ignore them altogether when searching for an arbitrary string without word boundaries, and will then happily find netargett in one-target-two! Spotlight also struggles with multiple hyphens mixed with underscores. For example, it may not be able to find danger in the file name a_a-b-c-e-danger_z.txt when using matches, but should work as expected when using contains instead.

Other tools in macOS

In-app search or Core Spotlight relies on search features provided by the app, for example that in Mail. Although these use Spotlight’s indexes and its query language, their function is different from Global or Local Spotlight, and implemented through a distinct Core Spotlight API.

The primary command tool for performing Spotlight search is mdfind, which uses formal query strings and predicates. I’ll tackle those in a future article.

Recommendations for successful search

• To perform local search of files indexed by Spotlight, use a Find window in the Finder.
• Limit search scope by folder when possible, by opening the folder in a Finder window then converting it into a Find window using the menu command.
• For the quickest results, type the start of a distinctive word in the file name or content into the search box.
• For the most inclusive results, leave the search box empty and construct a series of search bars to limit the list of hits.
• Limit the type of files when possible, using the Kind metadata attribute in the first popup menu.
• For file names with a limited number of hyphens and underscores as word boundaries, use Name matches with the start of a word included in the localised file name.
• For file names with more word boundaries, use Name contains with a distinctive target string.

I’m very grateful to Aldous and to Thomas Tempelmann for their unstinting help in understanding word boundaries and their importance, and for solving the mystery of Cleta.

How everything grows over time. Twenty years ago a hard disk of 100 GB was often ample, now twenty times that can be insufficient, and some have even larger media libraries. Finding files from among tens of thousands used to be straightforward, but now we’re working with millions we’re often struggling. Last week’s discussions of Spotlight search and its alternatives highlighted how important search strategies have become.

Strategy

Perhaps the most common strategy we use to search quickly and effectively is to apply a series of properties or attributes narrowing from the general to the specific: a dog, a small dog, a small grey-and-white dog, a small grey-and-white Havanese dog. In just a few adjectives we have narrowed the field to a description applying to a small number of domestic pets.

This strategy has two essential requirements: the target of your search must be included in the list of items being searched, and each of the attributes or criteria you apply in succession must include the search target. The first is obvious and critical to Spotlight’s success, and the second is the basis of how attributes are chosen. If the dog’s colour had been specified as red, then that search would have failed.

One of many skills in successful searching is judging how exclusive each criterion should be, and being more inclusive to ensure none of the criteria might inadvertently exclude the target.

Although you can combine attributes in this way when searching using the general Spotlight search window accessed through the menu bar, that’s a global search including websites and everything searchable from Wikipedia to Photos albums and Messages. When looking for a file, searching in the Finder immediately narrows the scope, and saves you wading through many irrelevant results. You can then add a search bar for each criterion, perhaps specifying that you’re looking for an image in your ~/Documents folder, each time reducing the number of hits until your choice becomes sufficiently limited.

Incremental search

Spotlight offers another technique that has become popular in search engines as their performance has improved, in what’s known as live or incremental search. As you type letters into one of its search boxes, it shows results as it gets them. This isn’t much use when entering common combinations of letters, but as they become more specific this can save time and accommodate any uncertainty you might have over spelling or the rest of the word. I use this frequently in MarsEdit when looking for old articles I have written: for example, typing wrestl will find wrestler, wrestlers, wrestling, wrestled, etc.

This works well with most languages including English, where roots and meanings are concentrated in the first parts of words, and declension and conjugation are usually found in their endings. Not all languages work like that, though, and this may not perform as well in Georgian or even German due to their morphology.

Predicates

For those who prefer to use the command line, mdfind can use predicates to express combinations of attributes, but those aren’t readily used in the same incremental way to narrow results down interactively. Another situation where predicates often come into play is when searching log entries and using the log show command, and that brings me on to LogUI, my other concern last week.

Searching the log

Let’s say you want to discover all the information RunningBoard gathers about an app, something you know is written in a log entry by the com.apple.runningboard subsystem shortly after that app starts its launch sequence. While you could search for all entries for that subsystem in the minute or so around the time you launched the app, there are likely to be thousands of hits.

To narrow down that search you have several options, including:

• launch the app at a known time, and set that as the Start time, with a Period of just a couple of seconds;
• set a one-off predicate to subsystem == "com.apple.launchservices" OR subsystem == "com.apple.runningboard";
• search subsystems for com.apple.launchservices to identify the time that LaunchServices announces the app will be launched through RunningBoard;
• search messages for constructed job description, RunningBoard’s log entry giving the details you’re looking for.

Those are ordered in increasing specificity, reducing numbers of hits, and increasing requirement for prior knowledge. That’s a general association, in that the more prior knowledge you have, the more specific you can make your criteria, and the fewer irrelevant hits you will see. As with Spotlight search, the more of these criteria you apply, the greater your chance of success, provided they all match the entry you’re looking for.

LogUI

To make LogUI more amenable to incremental search strategies, two additional features are needed. Instead of only exporting whole log extracts to Rich Text, the app needs to save and read formatted extracts. It also needs the ability to eliminate entries that don’t meet search criteria. Together those will enable use of a predicate to save an extract of reduced size, then application of search criteria, maybe saving an even smaller extract.

One way to combine multiple searches is to use multiple search bars, in a similar way to the Finder’s Find window. However, that tends to become overcomplicated, and I suspect is relatively little-used. If you do need a series of search criteria, then you also need different ways of combining them, including OR as well as AND, and that becomes a GUI predicate editor. I have yet to see any successful GUI predicate editor.

Next week, in the days prior to WWDC, I’m going to be focussing on search strategies for Spotlight, before turning to LogUI to implement these changes. This is an ideal time to let me know what you’d like to see, and how LogUI can support more successful search.

Searching for a file with a distinctive word in its name should be straightforward, but here I show some weird problems that could catch you out. I’m very grateful to Sam for drawing my attention to this, and welcome all and any rational explanations of what’s going on.

In some accounts of ancient Greek mythology, Cleta (Κλήτα) was one of the two Charites or Graces, alongside Phaenna. Her name apparently means renowned, and is still occasionally used as a first name today. It’s not the sort of word that should give Spotlight any cause for concern, and should prove easy to find.

Demonstration

To see the problems it can cause, create a folder somewhere accessible, in ~/Documents perhaps, and create half a dozen files with the names shown below.

Now open a new Finder window, and set it to Find mode using that command at the foot of the File menu. Then type into its search box the letters cleta

Only four of the files in that folder are found, excluding the first two, despite the fact that all their names clearly contain the search term.

Now clear the search box, and in the search criterion below, set it to find Name contains cleta, which you might have thought would be the same as the previous search.

Now all six files are found successfully.

You can try other variations of the file name to see which can be found using the search box, and which remain hidden. For example,
1995z_spectacletable_01.txt
also appears susceptible to this problem, suggesting that other examples might have the form
[digits]_[chars]cleta[chars]_[digits].[extension]

Separators

There are some other oddities at work as well, that you can see in the four file names that haven’t yet played hide and seek. So far I’ve been using Spotlight to find file names that simply contain the characters cleta. Now extend that to cletapainting

While you would expect the second of those to appear, Spotlight has elided the hyphen embedded in the first, as if it wasn’t there. Although Spotlight doesn’t provide a simple way to search for discrete words in file names, that’s a feature readily accessible in several third-party search utilities, including Find Any File and HoudahSpot. If you use Spotlight much, both of those are essentials, and you may wish to add Alfred as well.

As expected, Find Any File has no problems in finding all six test files when looking for names containing cleta

Set it to find names containing the word cleta, though, and it recognises spaces, hyphens and underscore _ characters as word separators, but doesn’t oblige with CamelCase, whether or not you capitalise its initial character.

Conclusions

• Avoid using the characters cleta in file names, as they can confuse Spotlight.
• Leave the search box in the Finder’s Find window empty and construct your search in the lower search bars instead.
• Spotlight can overlook hyphens in file names, but does treat them as word separators.
• Searching for words in file names can treat spaces, hyphens and underscore _ characters as word separators, but can’t cope with CamelCase.
• Spotlight’s rules are largely unwritten. Apple’s brief account is here, and doesn’t even mention the name Cleta.

My thanks again to Sam for providing me with the example of cleta that made this possible if apparently highly improbable.

Postscript

For those who think this all works as they expect, try the following file name:

1995z_star-post-office-cleta-hunt-portrait_01.txt