Local Spotlight search problems appear common, and are all too often tackled blindly by forcing a volume’s indexes to be rebuilt. Although that can sometimes resolve the problem, without knowing what its cause is, it can just waste time and effort. Indeed, in some cases rebuilding the indexes can worsen the problem, at least temporarily. This article explains how you can use SpotTest 1.1 to perform systematic testing and arrive at a diagnosis before hazarding a guess at what the treatment should be.

1. Spotlight settings

Before opening SpotTest, check that Spotlight settings are in order, and don’t exclude the volume or folder you’re trying to search, or the document type you’re looking for. Then open Activity Monitor and watch its CPU % listing to verify that Spotlight isn’t currently in the process of reindexing or performing other maintenance to its indexes. If it is, delay testing until those have completed. Searching using Spotlight while it’s actively working on its indexes will give odd results, or none at all.

If you’re going to use SpotTest on any location with privacy control, such as your Home Documents folder, or an external disk, consider adding the app to the Full Disk Access list in Privacy & Security settings, before opening it.

2. Home folder test

Even if your interest is in a different volume, you should perform a basic test of a new test folder in your current Home folder, to establish a baseline and confirm that Spotlight is working there.

Open SpotTest and set it to defaults, with a Term of cattle, a Scope of [Home], and both Search Keywords and Search EXIF ticked.

Click the Create Tests tool (the leftmost of the tools) to create the folder of test files at the top level of your Home folder. Make a note of the time you do this to the second.

About 10-15 seconds after that, click either the Run NSMetadata test or Run mdfind test tool. You should then see a list of files found, including those in the test folder ~/0_SpotTestFiles, including A, B, C, D, E, F, G, K, L, M

If you don’t see those listed, open Mints and use the Log Window button in its Spotlight section to obtain a log extract from the time the test files were created, or use LogUI to do the same. You’ll then need to look at that log extract to see if there are clues as to why indexing didn’t take place in the period.

Leave the test folder where it is, and anything from 1 hour to 2 days later, repeat the search using either or both of those tools. Once additional indexing has been undertaken:

• NSMetadata should now find A, B, C, D, E, F, G, I, K, L, M but not H
• mdfind should now find A, B, C, D, E, F, G, H, I, K, L, M.

I is found by the Live Text process, and H by Visual Look Up, the latter only being found by the mdfind search.

These tests have demonstrated:

• mdworker and mds indexing of files supported by system mdimporters;
• delayed background mediaanalysisd image analysis and mds indexing of Live Text and Visual Look Up content.

To match test files with their importers, click the Check importers tool. Note that file L doesn’t use a plugin, and N uses a plugin but can’t be found because the search term is inside an image in the PDF document, which currently isn’t recoverable content.

3. Custom mdimporter

Many apps come with their own custom mdimporter that provides Spotlight indexing support for document types not supported by macOS. In the past, these were normally installed in /Library/Spotlight, but more recent apps typically keep them inside the app bundle in Library/Spotlight. These can be tested easily.

Create and save one of those custom document types, so that it contains the word cattle in a way that should be searchable by Spotlight. Copy that document to the ~/0_SpotTestFiles folder, wait 10-15 seconds, then repeat the test search. You may well notice that NSMetadata search doesn’t find your custom test document, but mdfind does. This is because of the difference in the search criteria they use.

You should also click the Check importers tool to check that the correct mdimporter was recognised and used for the custom document type.

4. Volume test

If Spotlight works correctly with the test folder in your Home folder, you may wish to progress to testing a different volume or location. Having created its test folder in ~/0_SpotTestFiles, copy that to the other location. Before you try to change the Scope of the search, click on the button to list available volumes, then select the volume containing the copied test folder in the Scope menu.

When you perform the two types of search on that volume, the same rules should apply as to which will be found. Note though that finding files I and H can take much longer, or they may not appear at all.

5. Search term test

When you’re confident that a search term of cattle can be found reliably, you may wish to extend your testing to other terms. Take care when choosing custom terms, as you want to be confident that they should be found, but not in such numbers that the results are overwhelming. You will also need to create your own test files containing the custom term.

Diagnosis

SpotTest can thus provide key information on:

• delay or absence of find following creation of test files. If no indexing activity is seen in the log, that suggests indexing failure. If the test files are indexed promptly, it suggests search failure;
• delay or absence in finding files H and I, indicating an indexing failure;
• failure of a custom mdimporter to index a custom document type;
• failure to index another volume.

Those should fit in with the overall scheme used by Spotlight, as shown below.

Happy hunting!

As promised, this new version of my Spotlight indexing and search utility SpotTest extends its reach beyond the user’s Home folder, and can now test and search any regular volume that’s connected to your Mac and mounted in /Volumes.

By default, its searches remain restricted to the user’s Home folder, where SpotTest’s folder of crafted test files is installed. That applies whether you opt to use the search using its NSMetadataQuery tool, or the much faster option of the mdfind tool instead. If you want to search another mounted volume, click on the button for the app to check which volumes are available, then select one from its new Scope menu items. Volumes listed there exclude Time Machine backups and any hidden volumes whose names start with a dot, which will in any case be excluded from Spotlight indexing as they’re hidden.

This new version also fixes a weird bug that you’re unlikely to encounter in the previous version, but in rare circumstances could be infuriating. When searching using the NSMetadataQuery tool, if you had two windows open both with results from that tool, both would be updated with the same search results, and the time taken in them could rise to the absurd. This occurred because both windows were being updated with the data returned from the most recent search, as the NSMetadataQuery is shared in the app’s MainActor. After some fraught debugging, windows in this version ignore any search result updates initiated by other windows. I hope!

Volumes set in the Scope menu only affect search scope. Test folders are created in and removed from the user’s Home folder, and mdimporters are checked there as well. If you want to investigate indexing and search performance on other volumes, then you should manually create your own test folders as necessary. One quick and simple approach is to create a standard test folder in the Home folder, and copy that onto the volume(s) you want to test. A little later this week I’ll illustrate this in an article explaining how to get the best out of SpotTest and how it can help diagnose Spotlight problems.

I have taken the opportunity to improve SpotTest’s reporting of errors, such as trying to remove a test folder that doesn’t exist. I have also thoroughly revised the Help book, and added a page about search scopes.

SpotTest version 1.1 for macOS 14.6 and later, including Tahoe, is now available from here: spottest11
from Downloads above, and from its Product Page.

Enjoy!

There are some topics that invariably generate comments from those who have either abandoned a major feature in macOS, or are struggling with it. Some of the most persistent are problems with Spotlight, particularly with its local search of files on your Mac. To help grapple with those, four years ago I added some Spotlight tests to Mints that can be used to work out where those problems are occurring. I’m delighted now to offer an extension to those in a whole new app, perhaps predictably named SpotTest.

Spotlight is so substantial, almost silent in the log, and impenetrable that the best approach to diagnosing its problems is to test it out in a controlled way. Mints has been doing that by creating a folder of files containing an unusual word, then searching for that. Although that’s still useful for a quick test, we need something more focused and flexible, and that’s what SpotTest aims to deliver.

Following deep dives into how Spotlight indexes and searches metadata and contents of files, and how it can search text extracted from images and the results of image analysis, I’ve realised that different test files are required, together with alternative means of search. For example, the standard approach used in compiled apps, with NSMetadataQuery, is incapable of finding content tags obtained using Visual Look Up, which only appear when using the mdfind command. SpotTest takes these into account.

There are now 15 carefully crafted test files, of which one cannot currently be found, no matter what method of search you try.

A perfect 13/15 result from NSMetadataQuery is only possible after waiting a day or more for background mediaanalysisd processing to recognise and extract the text in file I, a PNG image. The other 12 here should all be found when running this test a few seconds after the test files have been created. They rely on a range of mdimporter modules bundled in macOS, apart from file L, an XML property list.

Another of SpotTest’s tools will list the mdimporters used for each of the test files.

Run the search using the mdfind command within SpotTest and, once mediaanalysisd has done its image recognition, you should get a perfect 14/15.

The only current limitation of SpotTest version 1.0 is that it can only run tests on the Data volume that your Mac started up from, using a folder at the top level of your Home folder. A future version will let you test other volumes as well. Its Help book runs to nine pages: please read them, as its test might seem deceptively simple but provide a lot of useful information about how Spotlight local search is functioning. Coupled with log extracts using LogUI it should shine light in the darkness.

SpotTest 1.0, which requires macOS 14.6 or later, is now available from here: spottest10
and from its new place in its Product Page.

I wish you successful searching.