You probably don’t use Spotlight to search files likely to be in one of your Mac’s Library folders, but if you do you may have noticed that it hardly ever finds anything there. I’m grateful to Scott for pointing out that some folders inside those Library folders, like Preferences, can’t be searched at all. This article attempts to discover what’s going on.

How Spotlight finds

When files are created or changed, provided they’re in a location that it indexes, Spotlight’s mdworker processes index data about those files, including metadata about each file itself, such as its name, and about its contents, enabling you to search for terms found in the file’s data. If a Spotlight search is unable subsequently to find known contents of a file, then one of the following has occurred:

• The contents of that file weren’t correctly indexed, most commonly because that file is in a location excluded from indexing, as set in Search Privacy… in Spotlight settings.
• Although indexes for that volume contain data correct for the file, Spotlight is withholding information about it from search results, most commonly because that category of file (or folder) shouldn’t provide search results, according to Search results settings in Spotlight settings.

I’ll refer to these two causes as unindexed and withheld respectively.

Does Spotlight reindex files that have been moved?

Before trying to establish which folders Spotlight won’t return search results for, I needed to be able to distinguish between those two causes. One way to achieve that might be to ensure that test files were first indexed at a location that is reliably indexed, for which no search results are withheld, then move the test files to a different location. That relies on Spotlight retaining indexed data when files are moved, rather than reindexing them.

I therefore assembled a folder of test files, based on those created and used by Mints, with the addition of a property list containing the search term syzygy999. These were first placed at the top of the user’s Home folder, time allowed for Spotlight to add them to the volume’s indexes, and that was confirmed by performing Mints’ test search, which found them all. That folder was then moved to ~/Documents, with Mints given Full Disk Access to ensure search wouldn’t be blocked by TCC.

Inspection of the log for 30 seconds following the change of location demonstrated that no reindexing occurred, although Spotlight was informed of the changed paths for the files. Thus, any search failures are expected to result from results being withheld, rather than the files being unindexed.

Which folders don’t return expected search results?

I then moved the test folder from ~/Documents to other folders and repeated Mints’ test search for each. Results were:

• ~/Library and /Library – search only found the term in a plain text file, where it was attached as an extended attribute and not in the text data.
• ~/Library/Trial – only found in plain text file with the search term in xattr.
• ~/Library/Preferences – no searches returned successfully.
• ~/Library/Application Support – all test files were found successfully, including the property list.

Is this specific to the active boot volume group?

All those test were performed on the active Data volume. To test whether this extends to other (inactive) boot volume groups, I connected an external bootable SSD with Sonoma installed and repeated the tests. As Mints only searches the current Data volume, I used the Find feature in the Finder. Results were exactly the same for that Data volume’s single user.

Where is this behaviour controlled?

Within each volume’s Spotlight index folder is VolumeConfiguration.plist, whose name suggests it might contain Spotlight configuration for that volume. Comparison of that file from a regular APFS volume with that from the Data volume of a boot volume group failed to suggest any value that might be responsible for this behaviour, though.

That indicates it’s determined in another property list, or coded into Spotlight.

If you have purchased the Spotlight master tool HoudahSpot, which I strongly recommend, then this may seem familiar, as it’s mentioned in its Help file, suggesting that it’s not new to Sequoia.

Conclusions

• Spotlight in Sequoia (at least) withholds the results of searches in Library folders and their contents, except for those in /Library/Application Support.
• There’s no way apparent to change that behaviour.

Postscript on dealing with prolonged indexing

At the end of my testing, I tried to unmount the external SSD containing the bootable Sonoma installation, only to be told that it was still in use. This was the inevitable reindexing that has caused similar problems to many others. Rather than forcing ejection in the Finder, I left this to run, in the hope that it would complete reasonably swiftly on a Mac mini M4 Pro. I gave up after over three hours, forced the external Data volume to be unmounted, and disconnected the SSD.

Intensive reindexing activity continued to occupy all the E cores, as if the volume was still present. This was occurring in the /var directory rather than in any volume indexes, and was only terminated by restarting the Mac. At no time did Spotlight indicate that indexing was in progress, nor provide any indication of progress to completion. If this isn’t a bug in Sequoia, then it’s a serious flaw in Spotlight that needs to be addressed. I will see if I can repeat this, and then file a bug report if I can.

When you search but Spotlight doesn’t find what you’re looking for, you might wonder whether that’s because the file containing it hasn’t been indexed correctly. This article suggests some methods you can use to investigate this.

Spotlight relies on hidden indexes it maintains at the top level of each volume, in the .Spotlight-V100 folder. When an app or process creates a new file or changes the contents of an existing file, that’s recorded in the volume’s FSEvents database. If that file is in a location that isn’t excluded from Spotlight’s index, an mdworker process should then start adding its contents to the volume indexes. To do this, it first checks what type of file it is, by its UTI. Spotlight then looks up the correct mdimporter for that type, and uses that to generate data that Spotlight adds to that volume’s indexes as a set of attributes.

Recent versions of macOS can extract additional information from certain types of files such as images and PDFs. This includes text recognised within images using Live Text, and object recognition and classification, performed in the background by other services such as photoanalysisd. Those are slower and take significantly more processing, and are normally run after traditional mdimporter extraction.

mdimporter plugins

For many file types, these are provided as part of macOS and stored in the system, in /System/Library/Spotlight on the SSV. Importers for third-party apps may be in /Library/Spotlight or ~/Library/Spotlight, or in the /Library/Spotlight folder inside the app bundle. To discover all mdimporter plugins currently installed, use the command
mdimport -L
although that doesn’t inform you which file types they support.

To get more information, identify a file of the correct document type, normally with that type’s standard filename extension, and use mdimport to discover which mdimporter is used for that type, and what it can extract from that specific file, in a command of the form
mdimport -t -d3 [filepath]
where [filepath] is the path of that file. You can vary the digit used in the -d3 option from 1-3, with larger numbers delivering more detail. For -d3 you’ll first be given the file type and mdimporter used, following which all the data extracted is given according to its Spotlight attributes:
Imported '/Users/hoakley/Documents/SpotTestA.rtf' of type 'public.rtf' with plugIn /System/Library/Spotlight/RichText.mdimporter.
37 attributes returned
followed by a long list.

If you’re still in the dark, the nuclear option is to perform a diagnostic dump similar to sysdiagnose but concentrating on Spotlight, using the command
mddiagnose -f [path]
to write the diagnostic output to the specified path. Further details are given in man mddiagnose.

Mints

My free utility Mints has a whole section devoted to testing and observing Spotlight. This includes creation of a folder of nine test files in the user’s Home folder, carrying out a test search that should find those test files, custom log analysis, and cleaning up test files afterwards. This is all fully documented in the Spotlight check and test section of its Help book.

To start the test, click on the Create Test button at a known clock time, such as 12:30:00. Wait at least ten seconds, say until 12:30:10, then click the Check Search button. Give your Mac a good 20 seconds to complete that, then open the Log Window and view entries from 12:30:00 for a period of 30 seconds, to cover both the indexing of the test files and the search. Once you’re happy that all is well, you can delete the test files.

Test files include:

• RTF rich text
• PDF with embedded text
• plain UTF-8 text
• HTML
• Microsoft Word DOCX
• JPEG image with the search term in its EXIF metadata
• Numbers spreadsheet
• Pages document
• plain UTF-8 text with the search term in a kMDItemKeywords extended attribute.

You can also customise the test files with other formats, provided that they contain the search text syzygy999 that Mints will look for.

Exclusions

Perhaps the most common reason for Spotlight not indexing a file’s contents is that it has been excluded from doing so. This is always worth checking before making the assumption the problem is in Spotlight.

There have been three general methods of excluding folders from indexing and search, although only two of them still work reliably:

• appending the extension .noindex to the folder name (this previously worked using .no_index instead);
• making the folder invisible to the Finder by prefixing a dot ‘.’ to its name;
• putting an empty file named .metadata_never_index inside the folder; that no longer works in recent macOS.

System Settings offers Spotlight Privacy settings in two sections. Items listed under Search results won’t normally prevent indexing of those items, but will block them from appearing in search results. Spotlight’s indexing exclusion list is accessed from the Search Privacy… button, where listed items won’t be indexed at all.

Summary of useful commands

• mdimport -L lists known Spotlight mdimporter plugins.
• mdimport -t -d3 [filepath] reports the mdimporter used for a file, and data extracted from that file.
• mddiagnose -f [path] runs full Spotlight diagnostics.

If you encounter problems with QuickLook not creating Thumbnails or Previews properly, one of the first steps is to discover which code is responsible for generating those for QuickLook. Prior to macOS Sequoia, the standard way to do that was using the command tool qlmanage, among whose options is -m, to list all the qlgenerators available on your Mac. If you’ve tried that in Sequoia, you’ll surely have noticed that no longer works.

qlmanage

Since Catalina, Apple has been encouraging developers to switch away from qlgenerators to app extensions to create custom Thumbnails and Previews for QuickLook, and Sequoia is the first version of macOS that can’t use third-party qlgenerators. I have noticed some document types that only a few weeks ago in Sonoma still used custom thumbnails and full previews, but now can’t do so, although others continue to work normally.

These are controlled in the Quick Look item in Login Items & Extensions in General settings.

That should list all third-party app extensions providing this service, and enabling the right one(s) could fix some of those problems. But it turns out this list isn’t complete, and doesn’t in any case tell you which app extension handles which file type. For those, you’d normally turn to qlmanage, but its -m option can only see the qlgenerators in macOS, and no third-party app extensions at all. In fact, qlmanage is now of little help for anything related to QuickLook. I’ve gone back through Sonoma and Ventura, and qlmanage there is no different: although it does list third-party qlgenerators, none of those provided in app extensions appear in its list.

QuickLook app extensions

As far as I can discover, Apple doesn’t provide any equivalent of qlmanage that can report on QuickLook app extensions. The closest it comes is in the pluginkit tool, that can list all app extensions known to macOS. With a bit of tweaking, its -m option can reveal which of those use the QuickLook SDKs for Thumbnails or Previews.

Armed with the appex bundle path from pluginkit, you can then inspect the Info.plist in each, where there’s an array of QLSupportedContentTypes giving the UTIs of all file types supported by that appex. Although I’m sure someone could implement that in a shell script, this seemed an ideal task for my free utility Mints.

Mints and QuickLook

Version 1.20 of my free utility Mints is now available from here: mints120
from Downloads above, from its Product Page, and via its auto-update mechanism.

This adds a twenty-fifth button to the app’s control window, named QuickLook, at the bottom left. Click on that and Mints will open a new window and fill it with information about all the qlgenerators and QuickLook appexes your Mac knows about.

For qlgenerators, you’re given the file UTI, the path to the qlgenerator file, and (when available) its version number, e.g.
com.adobe.pdf /System/Library/QuickLook/PDF.qlgenerator (1002.2.3)

App extensions are divided into two, the first are those providing Previews, and the second those for Thumbnails, e.g.
com.apple.applescript.text /Applications/PreviewCode.app/Contents/PlugIns/Code Previewer.appex

This is an appex provided in one of Black Pyramid Software’s superb Preview series, in PreviewBundle 2 from the App Store (highly recommended).

You will see a few entries like Safari’s
[none] /System/Volumes/Preboot/Cryptexes/App/System/Applications/Safari.app/Contents/PlugIns/SafariQuickLookPreview.appex
with an appex that doesn’t have a list of file types in QLSupportedContentTypes.

Checking UTIs

It’s easy to guess which UTIs represent many file types, but some are a bit more cryptic. For those, copy and paste the UTI into the UTI field of my free UTIutility and it will give you clues as to its identity, including file extensions.

Unfortunately, some of the system qlgenerators support generic UTIs such as
public.audio /System/Library/QuickLook/Audio.qlgenerator (1002.2.3)
public.image /System/Library/QuickLook/Image.qlgenerator (1002.2.3)
public.movie /System/Library/QuickLook/Movie.qlgenerator (1002.2.3)
which clearly cover broad ranges of more specific file types, but don’t provide any more specific information.

How to identify QuickLook extensions

• List installed QuickLook extensions using Mints’ QuickLook button.
• Identify the file’s UTI using UTIutility.
• Locate the UTI in the list of extensions.
• If no match is found, check UTIs listed in UTIutility as Conforms.
• Check Quick Look item in Login Items & Extensions in General settings, to ensure that extension is enabled.

Next up for Mints is a feature to explore app extensions. I may be a little longer on that one.