Local Spotlight search problems appear common, and are all too often tackled blindly by forcing a volume’s indexes to be rebuilt. Although that can sometimes resolve the problem, without knowing what its cause is, it can just waste time and effort. Indeed, in some cases rebuilding the indexes can worsen the problem, at least temporarily. This article explains how you can use SpotTest 1.1 to perform systematic testing and arrive at a diagnosis before hazarding a guess at what the treatment should be.

1. Spotlight settings

Before opening SpotTest, check that Spotlight settings are in order, and don’t exclude the volume or folder you’re trying to search, or the document type you’re looking for. Then open Activity Monitor and watch its CPU % listing to verify that Spotlight isn’t currently in the process of reindexing or performing other maintenance to its indexes. If it is, delay testing until those have completed. Searching using Spotlight while it’s actively working on its indexes will give odd results, or none at all.

If you’re going to use SpotTest on any location with privacy control, such as your Home Documents folder, or an external disk, consider adding the app to the Full Disk Access list in Privacy & Security settings, before opening it.

2. Home folder test

Even if your interest is in a different volume, you should perform a basic test of a new test folder in your current Home folder, to establish a baseline and confirm that Spotlight is working there.

Open SpotTest and set it to defaults, with a Term of cattle, a Scope of [Home], and both Search Keywords and Search EXIF ticked.

Click the Create Tests tool (the leftmost of the tools) to create the folder of test files at the top level of your Home folder. Make a note of the time you do this to the second.

About 10-15 seconds after that, click either the Run NSMetadata test or Run mdfind test tool. You should then see a list of files found, including those in the test folder ~/0_SpotTestFiles, including A, B, C, D, E, F, G, K, L, M

If you don’t see those listed, open Mints and use the Log Window button in its Spotlight section to obtain a log extract from the time the test files were created, or use LogUI to do the same. You’ll then need to look at that log extract to see if there are clues as to why indexing didn’t take place in the period.

Leave the test folder where it is, and anything from 1 hour to 2 days later, repeat the search using either or both of those tools. Once additional indexing has been undertaken:

• NSMetadata should now find A, B, C, D, E, F, G, I, K, L, M but not H
• mdfind should now find A, B, C, D, E, F, G, H, I, K, L, M.

I is found by the Live Text process, and H by Visual Look Up, the latter only being found by the mdfind search.

These tests have demonstrated:

• mdworker and mds indexing of files supported by system mdimporters;
• delayed background mediaanalysisd image analysis and mds indexing of Live Text and Visual Look Up content.

To match test files with their importers, click the Check importers tool. Note that file L doesn’t use a plugin, and N uses a plugin but can’t be found because the search term is inside an image in the PDF document, which currently isn’t recoverable content.

3. Custom mdimporter

Many apps come with their own custom mdimporter that provides Spotlight indexing support for document types not supported by macOS. In the past, these were normally installed in /Library/Spotlight, but more recent apps typically keep them inside the app bundle in Library/Spotlight. These can be tested easily.

Create and save one of those custom document types, so that it contains the word cattle in a way that should be searchable by Spotlight. Copy that document to the ~/0_SpotTestFiles folder, wait 10-15 seconds, then repeat the test search. You may well notice that NSMetadata search doesn’t find your custom test document, but mdfind does. This is because of the difference in the search criteria they use.

You should also click the Check importers tool to check that the correct mdimporter was recognised and used for the custom document type.

4. Volume test

If Spotlight works correctly with the test folder in your Home folder, you may wish to progress to testing a different volume or location. Having created its test folder in ~/0_SpotTestFiles, copy that to the other location. Before you try to change the Scope of the search, click on the button to list available volumes, then select the volume containing the copied test folder in the Scope menu.

When you perform the two types of search on that volume, the same rules should apply as to which will be found. Note though that finding files I and H can take much longer, or they may not appear at all.

5. Search term test

When you’re confident that a search term of cattle can be found reliably, you may wish to extend your testing to other terms. Take care when choosing custom terms, as you want to be confident that they should be found, but not in such numbers that the results are overwhelming. You will also need to create your own test files containing the custom term.

Diagnosis

SpotTest can thus provide key information on:

• delay or absence of find following creation of test files. If no indexing activity is seen in the log, that suggests indexing failure. If the test files are indexed promptly, it suggests search failure;
• delay or absence in finding files H and I, indicating an indexing failure;
• failure of a custom mdimporter to index a custom document type;
• failure to index another volume.

Those should fit in with the overall scheme used by Spotlight, as shown below.

Happy hunting!

As promised, this new version of my Spotlight indexing and search utility SpotTest extends its reach beyond the user’s Home folder, and can now test and search any regular volume that’s connected to your Mac and mounted in /Volumes.

By default, its searches remain restricted to the user’s Home folder, where SpotTest’s folder of crafted test files is installed. That applies whether you opt to use the search using its NSMetadataQuery tool, or the much faster option of the mdfind tool instead. If you want to search another mounted volume, click on the button for the app to check which volumes are available, then select one from its new Scope menu items. Volumes listed there exclude Time Machine backups and any hidden volumes whose names start with a dot, which will in any case be excluded from Spotlight indexing as they’re hidden.

This new version also fixes a weird bug that you’re unlikely to encounter in the previous version, but in rare circumstances could be infuriating. When searching using the NSMetadataQuery tool, if you had two windows open both with results from that tool, both would be updated with the same search results, and the time taken in them could rise to the absurd. This occurred because both windows were being updated with the data returned from the most recent search, as the NSMetadataQuery is shared in the app’s MainActor. After some fraught debugging, windows in this version ignore any search result updates initiated by other windows. I hope!

Volumes set in the Scope menu only affect search scope. Test folders are created in and removed from the user’s Home folder, and mdimporters are checked there as well. If you want to investigate indexing and search performance on other volumes, then you should manually create your own test folders as necessary. One quick and simple approach is to create a standard test folder in the Home folder, and copy that onto the volume(s) you want to test. A little later this week I’ll illustrate this in an article explaining how to get the best out of SpotTest and how it can help diagnose Spotlight problems.

I have taken the opportunity to improve SpotTest’s reporting of errors, such as trying to remove a test folder that doesn’t exist. I have also thoroughly revised the Help book, and added a page about search scopes.

SpotTest version 1.1 for macOS 14.6 and later, including Tahoe, is now available from here: spottest11
from Downloads above, and from its Product Page.

Enjoy!

There are some topics that invariably generate comments from those who have either abandoned a major feature in macOS, or are struggling with it. Some of the most persistent are problems with Spotlight, particularly with its local search of files on your Mac. To help grapple with those, four years ago I added some Spotlight tests to Mints that can be used to work out where those problems are occurring. I’m delighted now to offer an extension to those in a whole new app, perhaps predictably named SpotTest.

Spotlight is so substantial, almost silent in the log, and impenetrable that the best approach to diagnosing its problems is to test it out in a controlled way. Mints has been doing that by creating a folder of files containing an unusual word, then searching for that. Although that’s still useful for a quick test, we need something more focused and flexible, and that’s what SpotTest aims to deliver.

Following deep dives into how Spotlight indexes and searches metadata and contents of files, and how it can search text extracted from images and the results of image analysis, I’ve realised that different test files are required, together with alternative means of search. For example, the standard approach used in compiled apps, with NSMetadataQuery, is incapable of finding content tags obtained using Visual Look Up, which only appear when using the mdfind command. SpotTest takes these into account.

There are now 15 carefully crafted test files, of which one cannot currently be found, no matter what method of search you try.

A perfect 13/15 result from NSMetadataQuery is only possible after waiting a day or more for background mediaanalysisd processing to recognise and extract the text in file I, a PNG image. The other 12 here should all be found when running this test a few seconds after the test files have been created. They rely on a range of mdimporter modules bundled in macOS, apart from file L, an XML property list.

Another of SpotTest’s tools will list the mdimporters used for each of the test files.

Run the search using the mdfind command within SpotTest and, once mediaanalysisd has done its image recognition, you should get a perfect 14/15.

The only current limitation of SpotTest version 1.0 is that it can only run tests on the Data volume that your Mac started up from, using a folder at the top level of your Home folder. A future version will let you test other volumes as well. Its Help book runs to nine pages: please read them, as its test might seem deceptively simple but provide a lot of useful information about how Spotlight local search is functioning. Coupled with log extracts using LogUI it should shine light in the darkness.

SpotTest 1.0, which requires macOS 14.6 or later, is now available from here: spottest10
and from its new place in its Product Page.

I wish you successful searching.

Spotlight has been able to find text extracted from images using Live Text, and the names of objects recognised using Visual Look Up, for some years now. This article considers how you can and cannot search for those. Although this might seem obvious, it’s more confusing than it appears and could mislead you into thinking that Spotlight indexing or search isn’t working.

As detailed in yesterday’s account of Live Text, text recognition in images uses a lexicon to match words rather than proceeding in single characters. Terms assigned to recognised objects are also words. Thus, when searching for either type you should use words as much as possible to increase the chances of success.

Global Spotlight

Type a word like cattle into Spotlight’s search window and you can expect to see a full selection of documents and images containing the term or cattle objects. Those include images containing the word, and images containing objects identified as cattle, but don’t include images in PDF files, as they’re not analysed by mediaanalysisd, so don’t undergo character or object recognition in the same way that regular images like JPEGs do.

The Finder’s Find

Open a new window in the Finder and turn it into a Spotlight search using the Finder’s File menu Find command. In its search box at the top right, type in a word like cattle and in the popup menu select the lower option, Content contains. Press Return and the search box will now display ANY cattle. Then set the Kind to Image in the toolbar above search results, to narrow results down to image files. You should then see a full listing of image files that either contain the word cattle, or contain objects identified by image analysis as being cattle. Note how many of those appear.

Reconfigure the search so the search box is empty, and there are two rows of search settings: the first can remain the same as Kind is Image, but set the second to Contents contains cattle. Those images containing objects identified as cattle will now vanish, leaving images containing the word cattle still listed.

To understand the difference between these, you can save those two Find windows and read the underlying terms used for each search. The search that returned text obtained by both Live Text and Visual Look Up used
(((** = "cattle*"cdw)) && (_kMDItemGroupId = 13))
while the one excluding Visual Look Up used
(((kMDItemTextContent = "cattle*"cdw)) && (_kMDItemGroupId = 13))
instead. We can transfer those to the mdfind command tool to explore further.

mdfind

To use those as search predicates with mdfind we’ll translate them into more general form,
mdfind "(** == 'cattle*'cdw) && (kMDItemContentTypeTree == 'public.image'cd)"
should return both Live Text and Visual Look Up, while
mdfind "(kMDItemTextContent == "cattle*"cdw) && (kMDItemContentTypeTree == 'public.image'cd)"
only returns Live Text results.

The term (** == 'cattle*'cdw) has a special meaning because of its wild card **, and will return any match found in the metadata and contents of files. kMDItemTextContent is similar, but confined to text content, which doesn’t include the names of objects recognised in an image, and Apple doesn’t reveal whether there’s an equivalent that does.

Code search

Although apps can call mdfind to perform searches, they normally use NSMetadataQuery with an NSPredicate instead. That isn’t allowed to use a predicate like
** ==[cdw] "cattle"
so can’t search for objects identified using Visual Look Up. When it uses the substitute of
kMDItemTextContent ==[cdw] "cattle"
it also fails to find text obtained using Live Text. So the only way to search for recovered text in a compiled app is to call mdfind.

Timing

Searching for text obtained using Live Text, or object labels obtained using Visual Look Up, depends entirely on those being added to the Spotlight indexes on the volume. Observations of the log demonstrate just how quickly normal indexing by mdworker processes takes place. Here’s an example for a screenshot:
06.148292 com.apple.screencapture Write screenshot to temporary location
06.151242 [0x6000009bc4b0] activating connection: mach=true listener=false peer=false name=com.apple.metadata.mds
06.162302 com.apple.screencapture Moving screenshot to final location
06.169565 user/501/com.apple.mdworker.shared.1E000000-0600-0000-0000-000000000000 internal event: WILL_SPAWN, code = 0
06.198868 com.apple.DiskArbitration.diskarbitrationd mdworker_shared [7266]:118071 -> diskarbitrationd [380]
06.226997 kernel Sandbox apply: mdworker_shared[7266]

In less than 0.1 second an mdworker process has been launched and granted the sandbox it uses to generate metadata and content for Spotlight’s indexes. Unfortunately, that doesn’t include any Live Text or Visual Look Up content, which are generated separately by mediaanalysisd later. It’s hard to estimate how much later, although you shouldn’t expect to find such recovered text for several hours or days, depending on the opportunities for mediaanalysisd to perform background image analysis for this purpose.

Summary

• Words recovered from images (not those in PDF files, though) and objects recognised in them can be found by Spotlight search. For best results, choose words rather than letter fragments for search terms.
• Global Spotlight gives full access to both types.
• The Finder’s Find gives full access to both only when the search term is entered in the search box as Content contains, otherwise it may exclude objects recognised.
• mdfind can give full access to both types only when using a wildcard term such as (** == 'cattle*'cdw).
• NSMetadataQuery currently appears unable to access either type. Call mdfind instead.
• The delay before either type is added to the volume Spotlight indexes can be hours or days.

According to scientific tradition, we first observe then experiment. If you proceed to the latter before you understand how a system behaves, then you’re likely to labour under misapprehensions and your trials can become tribulations. Only when a system is thoroughly opaque and mysterious can we risk attempting both together.

That’s the case for Spotlight, which despite its name does everything but shine any light on its mechanisms. It presents itself in several guises, as a combination of web and local search (), as local search using terms limited in their logical operators (Finder’s Find), as full-blown predicate-based local search (mdfind), as in-app file search (Core Spotlight), and the coder’s NSMetadataQuery and predicates. It relies on indexes scattered across hundreds of binary files, and runs multiple processes, while writing next to nothing in the log.

Last week’s code-doodling has been devoted to turning the Spotlight features in Mints into a separate app, SpotTest, so I can extend them to allow testing of different volumes, and search for text that has been derived from images. Those are proving thorny because of Spotlight’s unpredictable behaviour across different Macs running Sequoia.

Every week I search for screenshots to illustrate another article on Mac history. When using my old iMac Pro where most of them are stored, Spotlight will find many images containing search terms from the text shown within them, even from ancient QuickDraw PICT images, demonstrating that text is being recovered using Live Text’s optical character recognition. When I try to repeat this using test images on an Apple silicon Mac, Spotlight seems unable to recognise any such recovered text.

Image analysis on Macs has a stormy history. In a well-intentioned gaffe four years ago, Apple shocked us when it declared it was intending to check our images for CSAM content. Although it eventually dropped that idea, there have been rumours ever since about our Macs secretly looking through our images and reporting back to Apple. It didn’t help that at the same time Apple announced Live Text as one of the new features of macOS Monterey, and brought further image analysis in Visual Look Up.

Although I looked at this in detail, it’s hard to prove a negative, and every so often I’m confronted by someone who remains convinced that Apple is monitoring the images on their Mac. I was thus dragged back to reconsider it in macOS Sonoma. What I didn’t consider at that time was how text derived from Live Text and image analysis found its way into Spotlight’s indexes, which forms part of my quest in SpotTest.

This doesn’t of course apply to images in PDF documents. When I looked at those, I concluded: “If you have PDF documents that have been assembled from scans or other images without undergoing any form of text recognition, then macOS currently can’t index any text that you may still be able to extract using Live Text. If you want to make the text content of a PDF document searchable, then you must ensure that it contains its own text content.” I reiterated that in a later overview.

My old images aren’t PDFs but QuickDraw PICTs, TIFFs, PNGs and JPEGs, many from more than 20 years ago. When the circumstances are right, macOS quietly runs Live Text over them and stores any text it recovers in Spotlight’s indexes. It also analyses each image for recognisable objects, and adds those too. These happen more slowly than regular content indexing by mdworker, some considerable time after the image has been created, and have nothing whatsoever to do with our viewing those images in QuickLook or the Finder, or even using Live Text or Visual Look Up ourselves.

There are deeper problems to come. Among them is discovering the results of image recognition as can be revealed in the command line using a search such as
mdfind "(** == 'cattle*'cdw) && (kMDItemContentTypeTree == 'public.image'cd)"
to discover all images that have been recognised as containing cattle. There’s no equivalent of the first part of that when calling NSMetadataQuery from Swift code, and a predicate of
kMDItemTextContent CONTAINS[cd] \"cattle\"
will only discover text recovered by Live Text, not the names of objects recognised within an image.

What started as a quick doodle is now bogged down in the quirks of Spotlight, which defies the scientific method. Perhaps it’s time for a little sorcery.

With additional information about Spotlight’s volume indexes, this article looks in more detail at how those indexes are updated when new files are created, and how local search is reflected in log entries. To do this I use the Spotlight test feature in my free utility Mints, which is designed to assess its performance.

Mints can create a test folder and populate it with 9 test files, copied from inside the app bundle using File Manager’s copyItem function. Provided the app and that newly created folder in ~/MintsSpotlightTest4syzFiles are in the same volume, this will almost certainly occur by creating APFS clone files. The files are:

• SpotTestA.rtf, RTF, relying on /System/Library/Spotlight/RichText.mdimporter
• SpotTestB.pdf, PDF, relying on /System/Library/Spotlight/PDF.mdimporter
• SpotTestC.txt, plain text, relying on /System/Library/Spotlight/RichText.mdimporter
• SpotTestD.html, HTML, relying on /System/Library/Spotlight/RichText.mdimporter
• SpotTestE.docx, Microsoft Word docx, replying on /System/Library/Spotlight/Office.mdimporter
• SpotTestEXIF.jpg, JPEG image not containing the target string in the image, but in the Make field of its EXIF IDF0 metadata, and relying on /System/Library/Spotlight/Image.mdimporter
• SpotTestF.numbers, Numbers spreadsheet containing the target string in one of its cells, and relying on /System/Library/Spotlight/iWork.mdimporter
• SpotTestG.pages, Pages document, relying on /System/Library/Spotlight/iWork.mdimporter
• SpotTestMD.txt, plain text not containing the target string in its contents, but in an extended attribute of type com.apple.metadata:kMDItemKeywords attached when the file is copied.

The search target they each contain is a string starting with the characters syzygy999 so most likely to be unique to the test files and Mints’ bundle.

This example test was performed on a Mac mini M4 Pro running macOS 15.6 from its internal SSD, with the Home folder located in the boot Data volume in that storage.

Indexing

Folder creation and file copying should take place soon after clicking on the Create Test button, in this example test in just over 0.3 seconds. The first mdworker process responding to this is spawned about 0.9 seconds after file copying starts, and further mdworker processes are added to it, making a total of 11. These are uncorked by launchd in about 0.03-0.05 seconds and are active for a period of about 0.2 seconds before they appear to have finished creating new posting lists to be added to the local indexes.

Shortly after those are complete, spotlightknowledge is active briefly, then 0.2 seconds after file copying mediaanalysisd receives a text processing request and initialises CGPDFService. Just over 0.1 second later CoreSceneUnderstanding is invoked from mediaanalysisd. For the following 5 seconds, a series of MADTextEmbedding (mediaanalysisd text embedding) tasks are performed. Finally, content obtained by mdworker and MADTextEmbedding is compressed by mds_stores, finishing about 6.7 seconds after the new files were created.

These steps are summarised in the diagram above, where those in blue update metadata indexes, and those in yellow and green update content indexes. It’s most likely that each file to be added to the indexes has its own individual mdworker process that works with a separate mdimporter, although in this case at least 11 mdworker processes were spawned for 9 files to be indexed.

Text extraction from files by mediaanalysisd involves a series of steps, and will be covered in more detail separately. In this test, local Spotlight indexes should have been updated within 7-8 seconds of file creation, so waiting at least 10 seconds after clicking the Create Test button should be sufficient to enable the test files to be found by search.

Searching

When the Check Search button in Mints is clicked, an NSMetadataQuery is performed using a search predicate over the search scope of NSMetadataQueryUserHomeScope, the Home folder. This uses the search predicate (kMDItemTextContent CONTAINS[cd] "syzygy999") OR (kMDItemKeywords CONTAINS[cd] "syzygy999") OR (kMDItemAcquisitionMake CONTAINS[cd] "syzygy999") to find all items containing any of:

• kMDItemTextContent CONTAINS[cd] "syzygy999" looks for the target string in a text representation of the content of the document, as in most of the test files used.
• kMDItemKeywords CONTAINS[cd] "syzygy999" looks for keywords in an extended attribute of type com.apple.metadata:kMDItemKeywords as attached to the SpotTestMD.txt file.
• kMDItemAcquisitionMake CONTAINS[cd] "syzygy999" looks for the Make field of an EXIF IDF0 as embedded in the SpotTestEXIF.jpg file.

Each of those is both case- and diacritic-insensitive, although the test cases all use lower case without any diacritics.

Remarkably little is recorded in the log for these test searches. com.apple.metadata starts the query within 0.001 second of clicking the Check Search button, and is interrupted by TCC approving the search to proceed. About 0.03 seconds after the start of the query, a series of entries from mds_stores reports query node counts of 4, and results are fetched and sent as replies between 0.09 and 19.5 seconds after the start of the query. However, Mints typically reports a considerably shorter search time of around 4 seconds.

Better testing

The test currently included in Mints appears useful, in that it assesses the whole of Spotlight local search from file creation to completion of a test search. However, in its current form it doesn’t test LiveText or other in-image text extraction, and only applies to the volume containing the Home folder. Log extracts obtained in its Log Window are basic and don’t capture much of the detail used for this article. There seems scope for a dedicated Spotlight test app that addresses these and more.

Summary

• Spotlight local indexing from the creation of nine new files reliant on macOS bundled mdimporters is initiated promptly, and should be completed within 10 seconds of their creation. It appears to follow the scheme shown in the diagram above.
• Text extraction from images by mediaanalysisd (‘MAD’) is significantly slower than extraction of metadata and other text content.
• NSMetadataQuery search should also be relatively quick when the number of hits is small, but leaves little information in the log.

One thing we humans are good at is searching. It’s a task we engage in from a few moments after birth until the time we slip away in death, we search everything around us. Locating and identifying that bird of prey wheeling high, finding the house keys, and that book we mislaid some time last week, meeting the perfect partner, discovering the right job, choosing the best education, looking through a Where’s Wally? or Where’s Waldo? book, and so on. Searching has transformed some into explorers like Christopher Columbus, and was the purpose of the chivalric quest. It’s what researchers in every field do, and thanks to Douglas Adams can be answered by the number 42.

Last week my searching took two new turns.

Spotlight

The first was more of a meta-search, in trying to discover more about the internals of Spotlight. Following the example of Maynard Handley, who has used them so successfully in understanding how M-series CPUs work, I looked through patents that have been awarded to Apple for the work of its search engineers. Yesterday’s slightly fuller history of Spotlight search is one result, and there are more to come in the future as I digest those patents concerned with performing search.

There’s a tinge of irony here, as many of my searches have been conducted using Google Patents, alongside Google Scholar one of the remaining search engines that doesn’t yet use AI and attempt to provide its own answers.

Logs

The other marks a new phase in my quest to get more information from the Unified log. Looking back to my first comment here, I realise how wildly over-optimistic I was when I wrote that it “should make my life a lot easier”, and that “a new version of Console will provide improved features to help us wade through logs.” Nine years later, I look wistfully at what remains of Console and realise how wrong I was on both counts.

When RunningBoard arrived in macOS Catalina, I soon noticed how “its log entries are profuse, detailed, and largely uncensored for privacy.” Since then it has proved garrulous to the point where its apparently ceaseless log chatter is a distraction, and can overwhelm attempts to read other log entries. I suspect it has contributed significantly to those advanced Mac users who now refuse to even try to make sense of the log.

One answer might be to tweak log preferences to shut out this noise, but given the purpose of RunningBoard in monitoring the life cycle of apps, why not try to use the information it provides? To do that, it’s first necessary to understand RunningBoard’s idiosyncratic language of assertions, and its protocols under which they’re acquired. The only way to do that without documentation is by observation: catalogue over 30 of those assertions for an interesting example like Apple’s Developer app, and see what they reveal.

By far the most informative entries from RunningBoard are those announcing that it’s acquiring an assertion, such as
Acquiring assertion targeting [app<application.developer.apple.wwdc-Release.9312198.9312203(501)>:2946] from originator [osservice<com.apple.uikitsystemapp(501)>:748] with description <RBSAssertionDescriptor| "com.apple.frontboard.after-life.subordinate" ID:424-748-2228 target:2946 attributes:[
<RBSDomainAttribute| domain:"com.apple.frontboard" name:"AfterLife-Subordinate" sourceEnvironment:"(null)">
]>

In a log often censored to the point of being unintelligible, this contains frank and explicit detail. The app is identified clearly, with the user ID of 501 and process ID of 2946. The originator is similarly identified as com.apple.uikitsystemapp with its PID of 748, which is confirmed in the middle digits in the Assertion ID. This is explicitly related to FrontBoard and an attribute named AfterLife-Subordinate. There’s not a single <private> to blight this entry, although further knowledge is needed to decode it fully.

Normally to get such information from a running process would require its source code to be instrumented with calls to write log entries, many of which would be lost to <private>, yet RunningBoard seems happy, for the moment, to provide that information freely. You can see what I mean by applying the predicate
subsystem == "com.apple.runningboard" AND message CONTAINS "Acquiring assertion t"
in LogUI, to obtain a running commentary on active apps and processes. Once you’ve identified a relevant assertion, you can focus attention on other log entries immediately prior to that. I will be following this up in the coming week, with fuller instructions and some demonstrations.

Although neither patents nor assertions have the significance of the number 42, in their own ways they show how the art and science of search aren’t dead yet, nor have they succumbed to AI.

Since writing A brief history of local search, I have come across numerous patents awarded to Apple and its engineers for the innovations that have led to Spotlight. This more detailed account of the origins and history of Spotlight uses those primary sources to reconstruct as much as I can at present.

1990

ON Technology, Inc. released On Location, the first local search utility for Macs, a Desk Accessory anticipating many of the features to come in Spotlight 15 years later. This indexed text found in the data fork of files, using format-specific importer modules to access those written by Microsoft Word, WordPerfect, MacWrite and other apps of the day. Those files and their indexed contents were then fully searchable. This required System Software 6.0 or later, and a Mac with a hard disk and at least 1 MB of RAM. It was developed by Roy Groth, Rob Tsuk, Nancy Benovich, Paul Moody and Bill Woods.

1991

Version 2 of On Location was released. ON Technology was later acquired by Network Corporation, then by Symantec in 2003.

1994

AppleSearch was released, and bundled in Workgroup Servers. This was based on a client-server system running over AppleShare networks. September’s release of System Software 7.5 introduced a local app Find File, written by Bill Monk.

1998

Sherlock was released in Mac OS 8.5. This adopted a similar architecture to AppleSearch, using a local service that maintained indexes of file metadata and content, and a client app that passed queries to it. This included remote search of the web through plug-ins working with web search engines, as they became available.

Early patent applications were filed by Apple’s leading engineers who were working on Sherlock, including US Patent 6,466,901 B1 filed 30 November 1998 by Wayne Loofbourrow and David Cásseres, for a Multi-language document search and retrieval system.

1999

Sherlock 2 was released in Mac OS 9.0. This apparently inspired developers at Karelia Software to produce Watson, ‘envisioned as Sherlock’s “companion” application, focusing on Web “services” rather than being a “search” tool like Sherlock.’

2000

On 5 January, Yan Arrouye and Keith Mortensen filed what became Apple’s US Patent 6,847,959 B1 for a Universal Interface for Retrieval of Information in a Computer System. This describes the use of multiple plug-in modules for different kinds of search, in the way that was already being used in Sherlock. Drawings show that it was intended to be opened using an item on the right of the menu bar, there titled [GO-TO] rather than using the magnifying glass icon of Sherlock or Spotlight. This opened a search dialog resembling a prototype for Spotlight, and appears to have included ‘live’ search conducted as letters were typed in.

2001

Karelia Software released Watson.

2002

Mac OS X Jaguar brought Sherlock 3, which many considered had an uncanny resemblance to Watson. That resulted in acrimonious debate.

2005

In preparation for the first Intel Macs, Mac OS X 10.4 Tiger, released in April 2005, introduced Spotlight as a replacement for Sherlock, which never ran on Intel Macs.

Initially, the Spotlight menu command dropped down a search panel as shown here, rather than opening a window as it does now.

2006

On 4 August, John M Hörnkvist and others filed what became US Patent 7,783,589 B2 for Inverted Index Processing, for Apple. This was one of a series of related patents concerning Spotlight indexing. Just a week later, on 11 August, Matthew G Sachs and Jonathan A Sagotsky filed what became US Patent 7,698,328 B2 for User-Directed search refinement.

A Finder search window, precursor to the modern Find window, is shown in the lower left of this screenshot taken from Tiger in 2006.

2007

Spotlight was improved in Mac OS 10.5 Leopard, in October. This extended its query language, and brought support for networked Macs that were using file sharing.

This shows a rather grander Finder search window from Mac OS X 10.5 Leopard in 2009.

2014

Search attributes available for use in the search window are shown here in OS X 10.9 Mavericks, in 2014.

In OS X 10.10 Yosemite, released in October, web and local search were merged into ‘global’ Spotlight, the search window that opens using the Spotlight icon at the right end of the menu bar, accompanied by Spotlight Suggestions.

2015

John M Hörnkvist and Gaurav Kapoor filed what was to become US Patent 10,885,039 B2 for Machine learning based search improvement, which appears to have been the foundation for Spotlight Suggestions, in turn becoming Siri Suggestions in macOS Sierra. Those were accompanied by remote data collection designed to preserve the relative anonymity of the user.

This shows a search in Global Spotlight in macOS 10.12 Sierra, in 2017.

c 2019

Apple acquired Laserlike, Inc, whose technology (and further patents) has most probably been used to enhance Siri Suggestions. Laserlike had already filed for patents on query pattern matching in 2018.

I’m sure there’s a great deal more detail to add to this outline, and welcome any additional information, please.

4 August 2025: I’m very grateful to Joel for providing me with info and links for On Location, which I have incorporated above.

This week’s batch of app updates includes two for working with metadata, a fun obfuscator of text, and my in-house performance test for storage. In each case, they have been given a new app icon that should display well in all versions of macOS from Big Sur to Tahoe. Their windows have been overhauled to accommodate Tahoe’s larger controls, and they have been rebuilt.

Spotlight and metadata

Metamer gives access to 16 of the most useful types of metadata that can be saved as extended attributes to any file, and others that are text-based if you wish. It has a built-in scratchpad you can use to assemble groups of keywords, for instance. It thus gives access to a wide range of metadata that you can use in Spotlight search.

Metamer version 1.6 is now available from here: metamer16
from its Product Page, and via its auto-update mechanism.

To accompany that is Spotcord, which scans folders to build vocabularies of keyword metadata (kMDItemKeywords in Spotlight’s terms), subjects and other specified types. Although it can look for those Spotlight derives from image analysis, experience shows that few are likely to be accessible outside Spotlight itself. This is the first release version of Spotcord, which had lingered in beta far too long.

Spotcord version 1.0 is now available from here: spotcord10
and from its Product Page. It doesn’t use the auto-update mechanism, though.

Text obfuscation

Dystextia is a bit of fun using Unicode lookalike characters to obfuscate Roman text. For example, it will convert this line into
Dуstехtіа іs а bіt оf fun usіng Unісоdе lооkаlіkе сhаrасtеrs tо оbfusсаtе Rоmаn tехt.
or even
Dу𝚜𝚝ех𝚝іа і𝚜 а bі𝚝 о𝚏 𝚏𝚞𝚗 𝚞𝚜і𝚗ɡ U𝚗ісоⅾе ⅼоо𝚔аⅼі𝚔е с𝚑а𝚛ас𝚝е𝚛𝚜 𝚝о оb𝚏𝚞𝚜са𝚝е Rо𝚖а𝚗 𝚝ех𝚝․
if you prefer. This is easy to reverse using AI, but throws find and spellchecking out of the window.

Dystextia 1.9 is now available from here: dystextia19
from its Product Page, and via its auto-update mechanism.

Storage performance testing

In contrast, today’s last update is the app I use to measure read and write speeds of storage, from internal SSDs to external hard disks and NAS systems. This offers a flexible range of test methods, all based on the same API calls used by apps to ensure they represent real-world performance. This comes with a detailed Help book that explains how testing and data analysis are performed.

Stibium version 1.2 is now available from here: stibium12
from its Product Page, and via its auto-update mechanism.

Coming next

My next batch of updates concludes the straightforward ones, and will bring Dintch, Fintch and Cormorant.

I currently don’t intend updating any of my command tools like blowhole, as they appear to continue working fine in Tahoe.

I have started work on updates to Unhidden, SilentKnight/Skint, and the Viable family of virtualisers. Each needs more work before they will work properly with Tahoe, in the case of SilentKnight/Skint a complete new version designed for future macOS.

I currently don’t intend updating ArchiChect, Taccy, Signet, Scrub, LockRattler or the command tool silnite for Tahoe, as they’ve now been superseded or outdated. If you want one of those reinstated, please let me know.

Searching the Internet, more recently its web servers, has proceeded in four main phases. Initially, humans built structured directories of sites they considered worth visiting. When those couldn’t keep pace with the Internet’s growth, commercial search engines were developed, and their search results were ranked. Around 2000, Google’s PageRank algorithm became dominant for ranking pages by their popularity. Then from late 2024 that is being progressively replaced with AI-generated summaries. Each of these has been reflected in the tools provided by Mac OS.

Directories

In the earliest years of the Internet, when the first web servers started to appear, and files were downloaded using anonymous FTP, users compiled their own lists by hand. Some curated directories were made public, including one maintained by Tim Berners-Lee at CERN, and another at NCSA. Individuals started using Gopher, a client to discover the contents of servers using the service of the same name. The next step was the development of tools to catalogue Gopher and other servers, such as Veronica and Jughead, but it wasn’t until 1993 that the first search engine, W3Catalog, and a bot, the World Wide Web Wanderer, started to transform Internet search.

Berners-Lee’s directory grew into the World Wide Web Virtual Library, and still exists, although it was last updated several years ago, most is now hosted elsewhere, and some is broken. The most famous directory was originally launched in 1994 and was then known as Jerry and David’s Guide to the World Wide Web, later becoming Yahoo! Directory. This offered paid submission and entry subscriptions, and was closed down at the end of 2014.

The favourite of many (including me) was launched as GnuHoo in 1998, and later that year, when it been acquired by Netscape, became the Open Directory Project, then DMOZ, seen here in the Camino browser in 2004. Although owned by AOL, it was maintained by a volunteer community that grew rapidly to hold around 100,000 links maintained by about 4,500 volunteers, and exceeded a million links by the new millennium. DMOZ closed in 2017 when AOL lost interest, but went on as Curlie using the same hierarchy.

Sherlock was first released in Mac OS 8.5 in 1998. As access to the web grew, this came to encompass remote search through plug-ins that worked with new web search engines.

Those were expanded in Sherlock 2, part of Mac OS 9.0 from 1999 and shown above, and version 3 that came in Mac OS X 10.2 Jaguar in 2002.

Indexing and ranking

Human editors couldn’t keep pace with the growth of the web, and demand grew for searching of indexes. This posed the problem of how to rank pages, and development of a series of ranking algorithms, some of which were patented. The first to use links (‘hyperlinks’) was Robin Li’s RankDex, patented in 1996, two years before Sergey Brin and Larry Page’s PageRank that brought their success in Google.

Ranking search results wasn’t new. In the late twentieth century, sciences started measuring the ‘impact’ of published papers by counting their citations in other papers, and university departments and scientific journals laid claim to their greatness by quoting citation and impact indexes. Early search ranking used features such as the frequency of occurrence of the words in the search term, which proved too crude and was manipulated by those trying to promote pages for gain. The obvious replacement was incoming links from other sites, which also quickly became abused and misused.

Research into networks was limited before 1998, when Jon Kleinberg and the two founders of Google entered the field. As with citation indexes before, they envisaged link-based ranking as a measure of popularity, and popularity as a good way of determining the order in which search results should be presented. They also recognised some of the dangers, and the need to weight incoming links to a page according to the total number of such links made by each linking site. Oddly, Kleinberg’s prior work wasn’t incorporated into a search engine until 2001, by which time Brin and Page were powering Google to dominance, and in June 2000 provided the default search engine for Yahoo!

This is Yahoo! Search seen in Firefox in 2007, by which time it was using its own indexing and search engine.

PageRank and algorithms

Google grew prodigiously, and became rich because of its sales of advertising across the web, a business dependent on promotion of its clients, something that could be achieved by adjusting its PageRank algorithm.

Although it’s hard to find now, at one time Google’s Advanced Search was widely used, as it gives more extensive control. Here it’s seen in Safari of 2011.

Google Scholar gives access to published research in a wide range of fields, and was introduced in late 2004. Here it’s seen in use in 2011, listing work that’s recently become topical again. Scholar doesn’t use the same PageRank-based algorithm for ranking its results, but does give substantial weight to citation counts.

When Apple replaced Sherlock with Spotlight in Mac OS X 10.4 Tiger in April 2005, web search defaulted to newly-arrived Safari and Google’s search engine. Its major redesign, in OS X 10.10 Yosemite in 2014, merged web and local search into Global Spotlight, the search window that opens from the Spotlight icon at the right end of the menu bar. That in turn brought Spotlight Suggestions, which became Siri Suggestions in macOS Sierra.

This shows a search in Global Spotlight in macOS 10.12 Sierra, in 2017.

Apple has never explained how Siri Suggestions works, although it appears to use machine learning and includes partial results from web search probably using Google. It offers a taste of what is to come in the future of Internet search.

Summarising

Google started the transition to using Artificial Intelligence in 2024, and that September introduced Audio Overview to provide spoken summaries of documents. This year has brought full AI overviews, in which multiple pages are summarised succinctly, and presented alongside links to the pages used to produce them. Although some can be useful, many are vague and waffly, and some blatantly spurious.

We’ve come a long way from Tim Berners-Lee’s curated directories, and PageRank in particular has transformed the web and more besides.

References

Wikipedia:
Gopher
Web directory
Search engine
Google Scholar

Amy N Langville and Carl D Meyer (2006) Google’s PageRank and Beyond: the Science of Search Engine Rankings, Princeton UP. ISBN 978 0 691 12202 1.

There are several well-known methods for excluding items from Spotlight search. This article details one that, as far as I can tell, has remained undocumented for the last 18 years since it was added in Mac OS X 10.5 Leopard, when Spotlight was only two years old, and can still catch you out by hiding files from local Spotlight search. This was discovered by Matt Godden, who has given his account here.

Well-known exclusions

There have been three general methods of excluding folders from Spotlight’s indexing and search, although only two of them still work reliably:

• making the folder invisible to the Finder by prefixing a dot ‘.’ to its name;
• appending the extension .noindex to the folder name (this earlier worked with .no_index instead);
• putting an empty file named .metadata_never_index inside the folder; that no longer works in recent macOS.

Additionally, System Settings offers Spotlight Privacy settings in two sections. Search results won’t normally prevent indexing of those items, but does block them from appearing in search results. Spotlight’s indexing exclusion list is accessed from the Spotlight Privacy… button, where you can add items that you don’t want indexed at all.

Extended attribute

Matt Godden investigated repeated failure of Spotlight search to find some images in his large media library, and discovered that the extended attribute (xattr) named com.apple.metadata:kMDItemSupportFileType was responsible. Images that weren’t returned in a search of that library all had that xattr attached, and when that was removed, those images were found reliably.

According to Apple’s documentation, that xattr was available in Mac OS X 10.5 and has since been deprecated. No further information is given about its function or effect, nor does it appear in an older list of Spotlight metadata attribute keys.

Search of previous mentions of this xattr reveal that it has been found with either of two values, iPhotoPreservedOriginal as described for Matt’s images, and MDSystemFile used with several apps that have proved equally inaccessible to Spotlight search. Images that have this xattr attached appear to have originated in old iPhotos libraries, which may have been migrated to Photos libraries. Searches for files with this xattr suggest that even old collections of images seldom have the xattr present, in my case on only 9 files out of over 800,000 checked, and the MDSystemFile variant wasn’t found in over 100,000 application files.

The mere presence of this xattr is sufficient to exclude a file from Spotlight search: setting its value to the arbitrary word any, for example, was as effective as setting it to either iPhotoPreservedOriginal or MDSystemFile.

Strangely, the method used to search is important: files with the com.apple.metadata:kMDItemSupportFileType xattr can’t be found when using Local Spotlight search in a Find window, but can be found by Mints using a standard search predicate with NSMetadataQuery.

Detection and removal

The simplest way to detect whether your Mac has files with the com.apple.metadata:kMDItemSupportFileType xattr is to use the Crawler tool in my free xattred, with Full Disk Access. Open its window using the Open Crawler… command in the Window menu, paste the xattr name into the Xattr type box. Click on the Scan button and select the volume or folder to check. xattred then crawls the full directory tree within that and reports all files with that xattr.

The xattr can then be removed by dragging the file onto one of xattred’s main windows, selecting the xattr, and clicking on the Cut button. That change will be effective immediately, and the file made available through Spotlight search within a few seconds.

If you have more than a handful of files with the xattr, use xattred’s Stripper to remove them all. Paste the xattr name into the Xattr type box. Click on the Strip button and select the volume or folder to process.

Recommendations

• If your Mac is likely to search old images that might have the com.apple.metadata:kMDItemSupportFileType xattr attached, search for and remove all such xattrs to ensure those files aren’t excluded from search.
• Whether this behaviour is intentional or not, it’s clearly an undesirable legacy, has been deprecated for many years, and should be removed from Spotlight search.

I’m extremely grateful to Matt Godden for his painstaking research and keeping me informed.

The Swiss Army knife has fallen victim to unintended consequences. Once the dream of every schoolboy and pocketed by anyone who went out into the countryside, my small collection of Swiss Army knives and multi-tools now remains indoors and unused. This is the result of strict laws on the carriage of knives in the UK; although not deemed illegal, since 1988 carrying them in a public place has put you at risk of being stopped and searched, and one friend was subjected to that for carrying a mere paint-scraper.

Swiss Army knives have another more sinister danger, that they’re used in preference to dedicated tools. Over the last week or two as I’ve been digging deeper into Spotlight, I can’t help but think how it has turned into the Swiss Army knife of search tools, by compromising its powers for the sake of versatility.

At present, I know of four different Spotlights:

• Global Spotlight, incorporating local, web, and some in-app search, accessed through the Spotlight tool in the menu bar;
• Local Spotlight, restricted to searching files in local and some network storage, typically through a Find window in the Finder;
• Core Spotlight, providing search features within an app, commonly in the contents of an app’s database;
• Third-Party Local Spotlight, a more limited local search available to third-party apps.

Of those, it’s Global Spotlight that I find most concerning, as it’s the frontline search tool for many if not most who use Macs, and the most flawed of the four. It’s not even the fault of Spotlight, whose 20th birthday we should have celebrated just over a month ago. No, this flaw goes right back to Sherlock, first released in Mac OS 8.5 in 1998.

At that time, few Macs had more than 5 GB of hard disk storage, and local search typically dealt with tens of thousands of files. That was also the first year that Google published its index, estimating that there were about 25 million web pages in all. Apple didn’t have its own web browser to offer, but made Microsoft’s Internet Explorer the default until Safari was released five years later. Merging local and web search into a single app seemed a good idea, and that’s the dangerous precedent set by Sherlock 27 years ago.

The result today only conflates and confuses.

In the days of Sherlock, web search was more a journey of discovery, where most search engines ranked pages naïvely according to the number of times the search term appeared on that page. That only changed with the arrival of Google’s patented PageRank algorithm at the end of the twentieth century, and placement of ads didn’t start in earnest until the start of the new millennium, by which time Safari was established as the standard browser in Mac OS X.

Local search was and remains a completely different discipline, with no concept of ranking. As local storage increased relentlessly in capacity, file metadata and contents became increasingly important to its success. Internally local searches have been specified by a logical language of predicates that are directly accessible to remarkably few users, and most of us have come to expect Spotlight’s indexing to handle metadata for us.

The end result challenges the user with negotiating web search engines and dodging their ads using one language, confounded by the behaviour of Siri Suggestions, and hazarding a wild guess as to what might come up in the metadata and content of files. More often than not, we end up with a potpourri that fails on all counts.

As an example, I entered the terms manet painting civil war into Spotlight’s Global Search box and was rewarded with a link to Manet’s painting of The Battle of the Kearsarge and the Alabama from 1864, as I’d hope. But entered into the search box of a Find window, those found anything but, from Plutarch’s Lives to a medical review on Type 2 diabetes. In MarsEdit’s Core Spotlight, though, they found every article I have written for this blog that featured the painting.

To get anything useful from local Spotlight, I had to know one of the ships was the USS Kearsarge, and that unusual word immediately found an image of the painting, but no useful content referring to it. Had I opted to search for the word Alabama instead, I would have been offered 94 hits, ranging from linguistics to the Mueller report into Russian interference in the 2016 US Presidential election. Adding the requirement that the file was an image narrowed the results down to the single image.

Conversely, entering Kearsarge into Global Spotlight offered a neighbourhood in North Conway, New Hampshire, in Maps, information about three different US warships from Siri Knowledge, Wikipedia’s comprehensive disambiguation page, a list of five US warships of that name, and three copies of the image of Manet’s painting without any further information about them.

Spotlight is also set to change with the inevitable addition of AI. Already suggestions are tailored using machine learning, but as far as I’m aware local Spotlight doesn’t yet use any form of AI-enhanced search. Words entered into search boxes and bars aren’t subject to autocorrection, and although Global Spotlight may suggest alternative searches using similar words, if you enter acotyle Spotlight doesn’t dismiss it as a mistake for acolyte. It remains to be seen whether and when local Spotlight switches from Boolean binaries to fuzziness and probability, but at least that will be more akin to the ranking of web pages, and we’ll no longer need to be bilingual.

For the time being, we’re left with a Swiss Army knife, ideal for finding where Apple has hidden Keychain Access, but disappointing when you don’t know exactly what you’re looking for.

Spotlight, the current search feature in macOS, does far more than find locally stored files, but in this brief history I focus on that function, and how it has evolved as Macs have come to keep increasingly large numbers of files.

Until early Macs had enough storage to make this worthwhile, there seemed little need. Although in 1994 there were precious few Macs with hard disks as large as 1 GB, networks could provide considerably more. That year Apple offered its first product in AppleSearch, based on a client-server system running over AppleShare networks, and in its Workgroup Servers in particular. This was a pioneering product that was soon accompanied by a local app, Find File, written by Bill Monk and introduced in System 7.5 that September.

Sherlock

The next step was to implement a similar architecture to AppleSearch on each Mac, with a service that maintained indexes of file metadata and contents, and a client that passed queries to it. This became Sherlock, first released in Mac OS 8.5 in 1998. As access to the web grew, this came to encompass remote search through plug-ins that worked with web search engines.

Those were expanded in Sherlock 2, part of Mac OS 9.0 from 1999 and shown above, and version 3 that came in Mac OS X 10.2 Jaguar in 2002. The latter brought one of the more unseemly conflicts in Apple’s history, when developers at Karelia claimed Sherlock 3 had plagiarised its own product, Watson, which in turn had been modelled on Sherlock. Apple denied that, but the phrase being Sherlocked has passed into the language as a result.

Spotlight

Sherlock remained popular with the introduction of Mac OS X, but was never ported to run native on Intel processors. Instead, Apple replaced it with Spotlight in Mac OS X 10.4 Tiger, in April 2005.

Initially, the Spotlight menu command dropped down a search panel as shown here, rather than opening a window as it does now.

A Finder search window, precursor to the modern Find window, is shown in the lower left of this screenshot taken from Tiger in 2006.

Spotlight was improved again in Mac OS 10.5 Leopard, in 2007. This extended its query language, and brought support for networked Macs that were using file sharing.

This shows a rather grander Finder search window from Mac OS X 10.5 Leopard in 2009.

Search attributes available for use in the search window are shown here in OS X 10.9 Mavericks, in 2014.

Spotlight’s last major redesign came in OS X 10.10 Yosemite, in 2014, when web and local search were merged into Global Spotlight, the search window that opens using the Spotlight icon at the right end of the menu bar. With Global Spotlight came Spotlight (then Siri from macOS Sierra) Suggestions, and they have been accompanied by remote data collection designed to preserve the relative anonymity of the user.

This Finder window in OS X 10.10 Yosemite, in 2015, shows a more complex search in progress.

This shows a search in Global Spotlight in macOS 10.12 Sierra, in 2017.

Local Search in the Finder’s Find window can now use a wide variety of attributes, some of which are shown here, in macOS 10.13 High Sierra, in 2018. Below are search bars for several different classes of metadata.

Over the years, Spotlight’s features have become more divided, in part to safeguard privacy, and to deliver similar features from databases. Core Spotlight now provides search features within apps such as Mail and Notes, where local searches lack access.

Spotlight’s indexes are located at the root level of each indexed volume, in the hidden .Spotlight-V100 folder. Those are maintained by mdworker processes relying on mdimporter plugins to provide tailored access for different file types. If an mdimporter fails to provide content data on some or all of the file types it supports, those are missing from that volume’s indexes, and Spotlight search will be unsuccessful. This happened most probably in macOS Catalina 10.15.6, breaking the indexing of content from Rich Text files. That wasn’t fixed until macOS Big Sur 11.3 in April 2021.

Over the last few years, macOS has gained the ability to perform optical character recognition using Live Text, and to analyse and classify images. Text and metadata retrieved by the various services responsible are now included in Spotlight’s indexes. From macOS 13 Ventura in 2022, those services can take prolonged periods working through images and file types like PDF that include images they can process to generate additional content and metadata for indexing.

Those with large collections of eligible files have noticed sustained workloads as a result. Fortunately for those with Apple silicon Macs, those services, like Spotlight’s indexing, run almost exclusively on their Mac’s E cores, so have little or no effect on its ability to run apps. For those with Intel processors, though, this may continue to be troubling.

In less than 30 years, searching Macs has progressed from the basic Find File to Spotlight finding search terms in text recognised in photos, in almost complete silence. Even Spotlight’s 20th birthday passed just over a month ago, on 29 April, without so much as an acknowledgment of its impact.

Although the great majority use GUI search tools provided in the Global Spotlight menu, Finder Find windows, and in-app Core Spotlight, macOS also provides access using query languages. This article takes a brief tour of those available in macOS. As with previous coverage, this doesn’t include third-party utilities such as HoudahSpot that provide their own interface to Spotlight searching, nor alternative search methods.

Search boxes

Search boxes provided by both Global Spotlight and Local Spotlight can accept a simple form of query language, for example
name:"target"*cdw
which also works with filename:, and performs the equivalent of the Matches operator in a Find window. These use English terms for the attributes to be used, like name and filename, including some of those listed here for Core Spotlight. However, limited information is available and this doesn’t appear to be extensive enough to use at scale. Operators available are also limited within those listed for Core Spotlight.

Modifiers available in current macOS include

• c for case-insensitivity,
• d to ignore diacritics such as accents,
• w to match on word boundaries, as marked by space, underscore _, hyphen – and changes of case used in CamelCase.

The asterisk * can be used as a wildcard to match substrings, and the backslash \ acts as an escape character, for example \" meaning a ” literal. In theory, simple predicates can be combined using && as AND, and || as OR.

In practice, getting these to work is tricky, and rarely worth the effort of trying.

Raw queries

One of the least-used attributes available in search bars in the Find window enables the use of what are termed raw queries. Confusingly, these use different names for attributes, such as kMDItemDisplayName instead of name. Otherwise these are more reliable than those used in search boxes. For example, when searching for the string target,
kMDItemDisplayName = "*target*"
is the equivalent of Contains, and
kMDItemDisplayName = "target*"w
appears functionally identical to Matches.

These appear to be an option of last resort, and need documentation.

mdfind

This command tool provides the most complete access to the central Spotlight Query Language, which defies abbreviation to SQL. In addition, it also supports a direct form that searches for matching file names only, using
mdfind -name "target"
to find the word target in filenames.

Unfortunately, although Spotlight Query Strings are predicates, they aren’t the same as NSPredicates used elsewhere within macOS. One of the most obvious differences is that Spotlight’s modifiers are appended to the value, not the operator, as they are when using search predicates in the log command, for example.

Query strings take the general form
attribute operator value[modifiers]
where

• attribute is a kMD… name defined as a metadata attribute key,
• operator can take a formal version such as ==, or may be abbreviated to just =, which appear to be identical in effect,
• value can be a string containing wildcards or escapes, or those detailed for special cases such as numbers and dates,
• modifiers include those given above.

Simple examples are
mdfind "kMDItemDisplayName = '*target*'"
or
mdfind "kMDItemDisplayName == '*target*'"

Apple’s current list of common metadata attribute keys is given here. Otherwise, documentation is old if not ancient, and there are obvious differences from current Spotlight and mdfind, such as the expanded list of modifiers.

File metadata queries are explained here, with an apparently duplicated version here. File metadata attributes are documented here, and a general account of predicates is here.

Saved Search Queries

On the face of it, one way to become familiar with and to develop query strings for use in mdfind might be to set them up in a Find window, save that and use the query string within it for your own. Although this can be helpful, queries in saved search files often use attributes not accessible to mdfind. For example, entering the string target in the search box setting a search bar to Kind is Image All is represented by the query
(((** = "target*"cdw)) && (_kMDItemGroupId = 13))
where the second attribute is an internal form of kMDItemKind.

However, this quickly runs into difficulties, as values of _kMDItemGroupId don’t appear to be documented, and substituting that with an alternative such as
kMDItemKind = "public.image"
fails silently.

Conclusions

• Spotlight query strings take several forms, none of them well-documented.
• Queries provided in Saved Search are of limited use, and are only likely to confuse.
• For occasional use, they are usually frustrating.
• For frequent use, third-party alternatives are more consistent and much better documented.

The ability to save Spotlight searches is perhaps its most underused feature. This article explains how Saved Search and Smart Folders work, and how you can use them to your advantage.

Save a search

Open a new Finder window and turn it into a Find window using the Find command at the foot of the File menu. Leave its search box blank, and set up one or more search bars with criteria that find some files of interest.

Then click on the Save button at the upper right, just below the search box. In the save dialog, set the saved location to an accessible folder such as ~/Documents, leave the Add To Sidebar checkbox empty, and click Save.

Close that Find window, and select your Saved Search ‘folder’ in the Finder. That will display the same files you just found in that search, as if it was a normal folder. Select any of those files, though, and you’ll see that they’re not really there, and their paths are for the original file, as if they were symbolic links, perhaps.

You can now move that Saved Search around, even copy it to another Mac, and wherever it goes it performs the same search and shows the results. Open Get Info on the Saved Search item and it’s described as a Saved Search Query, and the predicate used internally by Spotlight for that search is shown as its Query.

Edit a search

Now double-click on the Saved Search item to open it in its own window, and click on the Action tool (the circle containing an ellipsis …) and select the Show Search Criteria item in the menu. This restores your original Find window, complete with all its original settings, search bars, and their contents.

You can now change that search, and it will update to show the new results. To save your modified search, click on the Save button at the upper right, and the Saved Search will be duly updated.

Saved Search and Smart Folders

You can do exactly the same starting from the Finder’s New Smart Folder command in its File menu, as that creates a new Find window with identical features. The end results are the same, a file of type com.apple.finder.smart-folder. Only, as you’ll have gathered by now, this isn’t a real folder at all, just a property list that the Finder handles in an unconventional way.

Open the .savedSearch file using a text editor (you can make that easier by changing its extension to .plist if you wish), and you’ll see that it doesn’t even list the files ‘contained’ by this ‘folder’, all it gives is the search predicate used by Spotlight to find the items that are shown as being inside it.

That predicate, also shown as the Query in the Get Info dialog, is saved in the property list against its RawQuery key. Much of the rest of the property list is devoted to details enabling the Finder to reconstruct the Find window should you should decide to Show Search Criteria. But nowhere does it list any of the items found in the search.

This ensures that Saved Searches and Smart Folders take almost no space, just a few KB for that property list, and unless they’re open and displaying the search results, they don’t take any memory either.

They also don’t behave like regular folders. You can’t add items to them except by changing the search criteria used. You can only copy items from them, and can’t remove anything either. If you duplicate a Saved Search then you simply get another copy of the property list, not the items found by it. Their contents are also dynamic: create another item that meets their search criteria, and that will automatically be added to their contents, hence the alternative name of Smart Folder.

They’re also versatile and can have several roles:

• to return to a search still in progress;
• to set defaults for future Find windows, laid out ready for fresh searches;
• as records of important searches you might wish to repeat;
• as templates for types of search you’re likely to repeat.

They’re also the only place that I’m aware of that provides a bridge between searches made using the GUI in the Finder, and the terms and format of predicates used internally and by mdfind. This is important, as Saved Searches are themselves of no use in the command line, but their Query string can be used directly, as I’ll demonstrate in a later article.

Key points

• Saved Search = Smart Folder;
• they aren’t folders at all, but reconstruct and replay the search in a Find window;
• change the search by opening the Saved Search and using Show Search Criteria in the Action menu;
• their Query gives the search predicate used.

The most common problem reported in Spotlight search is failure to find target file(s). In this series of articles, I’ll examine what can go wrong, and how you can make local searches more successful using features in macOS. I’m well aware that there are other utilities for searching, some relying on Spotlight, others working independently, but here I’ll confine myself as much as possible to what’s provided in macOS.

Successful searches using Spotlight have four essential requirements:

• The target file(s) must have had their metadata added to Spotlight’s volume indexes.
• Those metadata must be accessible to the tool used to perform the search.
• The search queries used must be capable of finding the target(s).
• Spotlight’s search must work correctly.

Check indexes and access

The first of those can be checked using the mdimport command tool in Terminal, using the command
mdimport -t -d3 [filepath]
where [filepath] is the path of the file. You can vary the digit used in the -d3 option from 1-3, with larger numbers delivering more detail. For -d3 you’ll first be given the file type and mdimporter used, following which all the data extracted is given according to its Spotlight attributes:
Imported '/Users/hoakley/Documents/SpotTestA.rtf' of type 'public.rtf' with plugIn /System/Library/Spotlight/RichText.mdimporter.
37 attributes returned
followed by a long list.

If the file hasn’t been indexed, this article works through the steps you can take to rectify that. Note that recent experience is that using mdutil -E / to erase and force rebuild of indexes on the Data volume may not work as expected, and you should either perform this in System Settings, or using the command mdutil -E /System/Volumes/Data

Which tool?

Global Spotlight is accessed through the magnifying glass icon in the right of the menu bar, or using the default key combination of Command-space. This includes website content, and isn’t ideal when you’re searching for local files. If you want to use this as an easy gateway to local search, enter the text you want to search for and scroll down to use the command Search in Finder, which opens a Finder Find window for the results of that search query. Alternatively, you can click on Show More in the Documents section of the search results.

Local Spotlight can also be opened by the Find command at the foot of the Finder’s File menu, and takes you straight to its search box, at the right of the toolbar in that Finder window.

This window offers a choice of two search scopes at the upper left. The first covers all the accessible contents of that Mac, and the second is the folder open in that window when it was converted to a Find window. To set the scope for a local Spotlight search, start from a normal Finder window with the target folder open, then use the Find command on that.

Searching

Typing text into the search box at the right of the toolbar then performs live or incremental search for both filenames and content at the same time, or you can select one of them in the menu.

Text entered into the search box can simply be the start of a word in the target, or can be a basic search query such as name:"target"*cdw. I will explain those in a later article about search queries.

Instead of, or in addition to, entering text in the search box, you can set further search criteria in search bars below that.

In this case, the file Name is required to contain the string entered in the text box. Add more of these search bars for additional criteria to narrow your search.

Search attributes

In search bars, the first popup sets the metadata attribute to use in the query. For example, both Name and Filename refer to the name of the file, although Name is given in its localised form. Many more options are available by selecting the Other… item at the foot of the attribute menu. You can either set those as a one-off, or add them to the menu of attributes if you’re likely to use them again. These roughly correspond to the metadata attributes as in formal Spotlight search queries used elsewhere, although their names are different.

The second popup sets the operator to be used. While they may appear self-explanatory, two merit fuller explanation as they may work differently from how you might expect:

• matches finds the start of words found by breaking the whole string at word boundaries (see below). Thus it will find danger and danger-es in one-danger-est, but won’t find anger there.
• contains is applied without reference to word boundaries, and simply finds the given string, regardless of what characters occur before or after that. Thus it will find anger in dangerous. It has one unusual behaviour, though: it ignores hyphens, so dangeres will be found in the string danger-est.

Word boundaries

These are crucial in search queries run from the search box in a Find window, and the matches operator used in a search bar below. Although the search box claims to use the contains operator, it actually behaves as the matches operator does in a search bar.

In many languages word roots and meaning appear at the start of words, with declensions and conjugations at the end. If you want to find words related to harvest, like harvester, harvesting and harvested, then you’re going to enter a search query using harvest rather than vest. Like other search engines designed for live or incremental search, Spotlight is fastest when searching for the start of words. It therefore divides compound words often used for filenames into component words. It does so using rules for word boundaries laid down in the International Components for Unicode.

In practice, word boundaries include a space, the underscore _, hyphen – and changes of case used in CamelCase. Spotlight treats each of the following examples as three words:
one target two
one_target_two
one-target-two
OneTargetTwo
Languages other than English may allow other word boundaries, but those are the most common.

The rules recognise that hyphens are difficult, and Spotlight makes them even trickier as it can ignore them altogether when searching for an arbitrary string without word boundaries, and will then happily find netargett in one-target-two! Spotlight also struggles with multiple hyphens mixed with underscores. For example, it may not be able to find danger in the file name a_a-b-c-e-danger_z.txt when using matches, but should work as expected when using contains instead.

Other tools in macOS

In-app search or Core Spotlight relies on search features provided by the app, for example that in Mail. Although these use Spotlight’s indexes and its query language, their function is different from Global or Local Spotlight, and implemented through a distinct Core Spotlight API.

The primary command tool for performing Spotlight search is mdfind, which uses formal query strings and predicates. I’ll tackle those in a future article.

Recommendations for successful search

• To perform local search of files indexed by Spotlight, use a Find window in the Finder.
• Limit search scope by folder when possible, by opening the folder in a Finder window then converting it into a Find window using the menu command.
• For the quickest results, type the start of a distinctive word in the file name or content into the search box.
• For the most inclusive results, leave the search box empty and construct a series of search bars to limit the list of hits.
• Limit the type of files when possible, using the Kind metadata attribute in the first popup menu.
• For file names with a limited number of hyphens and underscores as word boundaries, use Name matches with the start of a word included in the localised file name.
• For file names with more word boundaries, use Name contains with a distinctive target string.

I’m very grateful to Aldous and to Thomas Tempelmann for their unstinting help in understanding word boundaries and their importance, and for solving the mystery of Cleta.