Over the last few weeks, as I’ve been digging deeper into Spotlight local search, what seemed at first to be fairly straightforward has become far more complex. This article draws together some lessons that I have learned.

Apple’s search patents have been abandoned

Having tracked down a batch of Apple’s patents relating to search technologies that are likely to have been used in Spotlight, I was surprised to see those have been abandoned. For example, US Patent 2011/0113052 A1, Query result iteration for multiple queries, filed by John Hörnkvist on 14 January 2011, was abandoned five years later. I therefore have no insights to offer based on Apple’s extant patents.

Search is determined by index structure

Spotlight indexes metadata separately from contents, and both types of index point to files, apparently through their paths and names, rather than their inodes. You can demonstrate this using test file H in SpotTest. Once Spotlight has indexed objects in images discovered by mediaanalysisd, moving that file to a different folder breaks that association immediately, and the same applies to file I whose text is recognised by Live Text.

Extracted text, that recovered using optical character recognition (Live Text), and object labels obtained using image classification (Visual Look Up) are all treated as content rather than metadata. Thus you can search for content, but you can’t obtain a list of objects that have been indexed from images, any more than you can obtain Spotlight’s lexicon of words extracted as text.

Language

Spotlight’s indexes are multilingual, as demonstrated by one of Apple’s earliest patents for search technology. Extracted text can thus contain words in several languages, but isn’t translated. Object labels are likely to be in the primary language set at the time, for example using the German word weide instead of the English cattle, if German was set when mediaanalysisd extracted object types from that image. You can verify this in SpotTest using test file H and custom search terms.

If you change your Mac’s primary language frequently, this could make it very hard to search for objects recognised in images.

Search method makes a difference

The Finder’s Find feature can be effective, but has a limited syntax lacking OR and NOT unless you resort to using Raw Query predicates (available from the first popup menu).* This means it can’t be used to construct a search for text containing the word cattle OR cow. This has a beneficial side-effect, in that each term used should reduce the number of hits, but it’s a significant constraint.

The Finder does support some search types not available in other methods such as mdfind. Of the image-related types, John reports that kMDItemPhotosSceneClassificationLabels can be used in the Finder’s Find and will return files with matching objects that have been identified, but that doesn’t work in mdfind, either in Terminal or when called by an app. Other promising candidates that have proved unsuccessful include:

• kMDItemPhotosSceneClassificationIdentifiers
• kMDItemPhotosSceneClassificationMediaTypes
• kMDItemPhotosSceneClassificationSynonyms
• kMDItemPhotosSceneClassificationTypes.

One huge advantage of mdfind is that it can perform a general search for content using wildcards, in the form
(** == '[searchTerm]*'cdw)

Using NSMetadataQuery from compiled code is probably not worth the effort. Not only does it use predicates of different form from mdfind, but it’s unable to make use of wildcards in the same way that mdfind can, a lesson again demonstrated in SpotTest. mdfind can also be significantly quicker.

For example, you might use the form
mdfind "(kMDItemKeywords == '[searchTerm]*'cdw)"
in Terminal, or from within a compiled app. The equivalent predicate for NSMetadataQuery would read
(kMDItemKeywords CONTAINS[cdw] \"cattle\")

Another caution when using NSMetadataQuery is that apps appear to have their own single NSMetadataQuery instance on their main thread. That can lead to new queries leaking into the results from previous queries.

Key points

• Spotlight indexes metadata separately from contents.
• Text recovered from images, and objects recognised in images, appear to be indexed as contents. As a result you can’t obtain a lexicon of object types.
• Indexed data appear to be associated with the file’s path, and will be lost if a file is moved within the same volume.
• Text contents aren’t translated for indexing, so need to be searched for in their original language.
• Object types obtained from images appear to be indexed using terms from the primary language at the time they are indexed. If the primary language is changed, that will make it harder to search for images by contents.
• The Finder’s Find is constrained in its logic, and doesn’t support OR or NOT, although using Raw Queries can work around that.
• mdfind is most powerful, including wildcard search for content.
• NSMetadataQuery called from code uses a different predicate format and has limitations.

I’m very grateful to Jürgen for drawing my attention to the effects of language, and to John for reporting his discovery of kMDItemPhotosSceneClassificationLabels.

• I’m grateful to Jozef (see his comment below) for reminding me that there is a way to construct OR and NOT operations in the Finder’s Find, although it’s not obvious, and even when you know about it you may well find it difficult to use. I will explain more about this in another article next week.

Spotlight has been able to find text extracted from images using Live Text, and the names of objects recognised using Visual Look Up, for some years now. This article considers how you can and cannot search for those. Although this might seem obvious, it’s more confusing than it appears and could mislead you into thinking that Spotlight indexing or search isn’t working.

As detailed in yesterday’s account of Live Text, text recognition in images uses a lexicon to match words rather than proceeding in single characters. Terms assigned to recognised objects are also words. Thus, when searching for either type you should use words as much as possible to increase the chances of success.

Global Spotlight

Type a word like cattle into Spotlight’s search window and you can expect to see a full selection of documents and images containing the term or cattle objects. Those include images containing the word, and images containing objects identified as cattle, but don’t include images in PDF files, as they’re not analysed by mediaanalysisd, so don’t undergo character or object recognition in the same way that regular images like JPEGs do.

The Finder’s Find

Open a new window in the Finder and turn it into a Spotlight search using the Finder’s File menu Find command. In its search box at the top right, type in a word like cattle and in the popup menu select the lower option, Content contains. Press Return and the search box will now display ANY cattle. Then set the Kind to Image in the toolbar above search results, to narrow results down to image files. You should then see a full listing of image files that either contain the word cattle, or contain objects identified by image analysis as being cattle. Note how many of those appear.

Reconfigure the search so the search box is empty, and there are two rows of search settings: the first can remain the same as Kind is Image, but set the second to Contents contains cattle. Those images containing objects identified as cattle will now vanish, leaving images containing the word cattle still listed.

To understand the difference between these, you can save those two Find windows and read the underlying terms used for each search. The search that returned text obtained by both Live Text and Visual Look Up used
(((** = "cattle*"cdw)) && (_kMDItemGroupId = 13))
while the one excluding Visual Look Up used
(((kMDItemTextContent = "cattle*"cdw)) && (_kMDItemGroupId = 13))
instead. We can transfer those to the mdfind command tool to explore further.

mdfind

To use those as search predicates with mdfind we’ll translate them into more general form,
mdfind "(** == 'cattle*'cdw) && (kMDItemContentTypeTree == 'public.image'cd)"
should return both Live Text and Visual Look Up, while
mdfind "(kMDItemTextContent == "cattle*"cdw) && (kMDItemContentTypeTree == 'public.image'cd)"
only returns Live Text results.

The term (** == 'cattle*'cdw) has a special meaning because of its wild card **, and will return any match found in the metadata and contents of files. kMDItemTextContent is similar, but confined to text content, which doesn’t include the names of objects recognised in an image, and Apple doesn’t reveal whether there’s an equivalent that does.

Code search

Although apps can call mdfind to perform searches, they normally use NSMetadataQuery with an NSPredicate instead. That isn’t allowed to use a predicate like
** ==[cdw] "cattle"
so can’t search for objects identified using Visual Look Up. When it uses the substitute of
kMDItemTextContent ==[cdw] "cattle"
it also fails to find text obtained using Live Text. So the only way to search for recovered text in a compiled app is to call mdfind.

Timing

Searching for text obtained using Live Text, or object labels obtained using Visual Look Up, depends entirely on those being added to the Spotlight indexes on the volume. Observations of the log demonstrate just how quickly normal indexing by mdworker processes takes place. Here’s an example for a screenshot:
06.148292 com.apple.screencapture Write screenshot to temporary location
06.151242 [0x6000009bc4b0] activating connection: mach=true listener=false peer=false name=com.apple.metadata.mds
06.162302 com.apple.screencapture Moving screenshot to final location
06.169565 user/501/com.apple.mdworker.shared.1E000000-0600-0000-0000-000000000000 internal event: WILL_SPAWN, code = 0
06.198868 com.apple.DiskArbitration.diskarbitrationd mdworker_shared [7266]:118071 -> diskarbitrationd [380]
06.226997 kernel Sandbox apply: mdworker_shared[7266]

In less than 0.1 second an mdworker process has been launched and granted the sandbox it uses to generate metadata and content for Spotlight’s indexes. Unfortunately, that doesn’t include any Live Text or Visual Look Up content, which are generated separately by mediaanalysisd later. It’s hard to estimate how much later, although you shouldn’t expect to find such recovered text for several hours or days, depending on the opportunities for mediaanalysisd to perform background image analysis for this purpose.

Summary

• Words recovered from images (not those in PDF files, though) and objects recognised in them can be found by Spotlight search. For best results, choose words rather than letter fragments for search terms.
• Global Spotlight gives full access to both types.
• The Finder’s Find gives full access to both only when the search term is entered in the search box as Content contains, otherwise it may exclude objects recognised.
• mdfind can give full access to both types only when using a wildcard term such as (** == 'cattle*'cdw).
• NSMetadataQuery currently appears unable to access either type. Call mdfind instead.
• The delay before either type is added to the volume Spotlight indexes can be hours or days.

According to scientific tradition, we first observe then experiment. If you proceed to the latter before you understand how a system behaves, then you’re likely to labour under misapprehensions and your trials can become tribulations. Only when a system is thoroughly opaque and mysterious can we risk attempting both together.

That’s the case for Spotlight, which despite its name does everything but shine any light on its mechanisms. It presents itself in several guises, as a combination of web and local search (), as local search using terms limited in their logical operators (Finder’s Find), as full-blown predicate-based local search (mdfind), as in-app file search (Core Spotlight), and the coder’s NSMetadataQuery and predicates. It relies on indexes scattered across hundreds of binary files, and runs multiple processes, while writing next to nothing in the log.

Last week’s code-doodling has been devoted to turning the Spotlight features in Mints into a separate app, SpotTest, so I can extend them to allow testing of different volumes, and search for text that has been derived from images. Those are proving thorny because of Spotlight’s unpredictable behaviour across different Macs running Sequoia.

Every week I search for screenshots to illustrate another article on Mac history. When using my old iMac Pro where most of them are stored, Spotlight will find many images containing search terms from the text shown within them, even from ancient QuickDraw PICT images, demonstrating that text is being recovered using Live Text’s optical character recognition. When I try to repeat this using test images on an Apple silicon Mac, Spotlight seems unable to recognise any such recovered text.

Image analysis on Macs has a stormy history. In a well-intentioned gaffe four years ago, Apple shocked us when it declared it was intending to check our images for CSAM content. Although it eventually dropped that idea, there have been rumours ever since about our Macs secretly looking through our images and reporting back to Apple. It didn’t help that at the same time Apple announced Live Text as one of the new features of macOS Monterey, and brought further image analysis in Visual Look Up.

Although I looked at this in detail, it’s hard to prove a negative, and every so often I’m confronted by someone who remains convinced that Apple is monitoring the images on their Mac. I was thus dragged back to reconsider it in macOS Sonoma. What I didn’t consider at that time was how text derived from Live Text and image analysis found its way into Spotlight’s indexes, which forms part of my quest in SpotTest.

This doesn’t of course apply to images in PDF documents. When I looked at those, I concluded: “If you have PDF documents that have been assembled from scans or other images without undergoing any form of text recognition, then macOS currently can’t index any text that you may still be able to extract using Live Text. If you want to make the text content of a PDF document searchable, then you must ensure that it contains its own text content.” I reiterated that in a later overview.

My old images aren’t PDFs but QuickDraw PICTs, TIFFs, PNGs and JPEGs, many from more than 20 years ago. When the circumstances are right, macOS quietly runs Live Text over them and stores any text it recovers in Spotlight’s indexes. It also analyses each image for recognisable objects, and adds those too. These happen more slowly than regular content indexing by mdworker, some considerable time after the image has been created, and have nothing whatsoever to do with our viewing those images in QuickLook or the Finder, or even using Live Text or Visual Look Up ourselves.

There are deeper problems to come. Among them is discovering the results of image recognition as can be revealed in the command line using a search such as
mdfind "(** == 'cattle*'cdw) && (kMDItemContentTypeTree == 'public.image'cd)"
to discover all images that have been recognised as containing cattle. There’s no equivalent of the first part of that when calling NSMetadataQuery from Swift code, and a predicate of
kMDItemTextContent CONTAINS[cd] \"cattle\"
will only discover text recovered by Live Text, not the names of objects recognised within an image.

What started as a quick doodle is now bogged down in the quirks of Spotlight, which defies the scientific method. Perhaps it’s time for a little sorcery.