Files, documents and everything else we store on our Macs consist of data. For an image, those are the pixels that have to be displayed for that image, for an illustrated book it’s the laid-out pages of text and pictures.

Associated with each of those is additional information about what’s in the data, such as the datestamp of its creation both as data and as that file, details of its creator, and about how it was created, such as the camera used. Those are data about its data, thus metadata.

Until 1984 and the first Mac, it was almost universal that most metadata was contained in the same file as the data, although some, such as a file’s datestamps, were stored separately in that file’s record in the file system, in its attributes. The Mac tried to change that by introducing a second fork to files, their resource fork, intended to contain metadata. Unfortunately, while that became standard on Macs, dominant operating systems like MS-DOS didn’t change, and continued to embed data and metadata together in flat files.

A lot has changed over the nearly 42 years since the first Mac, and now macOS has multiple sources of metadata for its files.

File attributes

APFS file records contain an extensive set of attributes, including

• time of creation
• time of modification of data
• time of attribute modification
• time of access
• file name
• owner, group and permissions.

These are largely common to other modern file systems.

Extended attributes

Mac OS X brought the extension of classic resource forks to other metadata objects, as extended attributes (xattr), named using a reverse-URL scheme, such as com.apple.FinderInfo containing metadata for the Finder. In this scheme, the traditional resource fork becomes a xattr of type com.apple.ResourceFork. Many of these are now used by macOS for security and privacy protection, but the user can add xattrs containing copyright information, names of creators, an arbitrary description, a text headline, and others. Anyone can define their own type of xattr, and some apps make good use of them for storing metadata.

Their main disadvantages are:

• Xattrs rarely transfer to other platforms, making most Mac-only.
• They’re commonly stripped when transferred even between Macs, or when shared in iCloud Drive. Apple has a system of tags to determine which xattrs should be stripped and which retained, but those aren’t as widely used as they deserve.
• Most xattrs aren’t shown by the Finder, either in Preview panes or in Get Info dialogs.

For largely historical reasons, even Apple doesn’t take fullest advantage of xattrs. For example, Finder Comments, which are shown in Get Info, are primarily stored in a folder’s hidden .DS_Store file and only secondarily in a com.apple.metadata:kMDItemFinderComment xattr.

Embedded metadata

Because so few file systems use extended attributes or their equivalents, most file-specific metadata is now embedded in file data. For some file formats, such as those widely used by word processors and spreadsheets, this is relatively straightforward, particularly when using XML-based formats.

It becomes more complicated and less reliable when used with images, as in Exchangeable Image File Format, EXIF. Although usually treated as a metadata standard, in fact EXIF encompasses both data and metadata formats embedded in a single file for convenience.

EXIF metadata can include camera settings such as aperture and shutter speed, image metrics such as colourspace, date and time of creation, location and copyright information, and a thumbnail version of the image (which is arguably data rather than metadata). How the metadata is embedded with data is determined by the format of the image data. For JPEG data, EXIF metadata is stored in an Application Segment of the image, but for TIFF data there’s a sub-image file directory that can spread the metadata anywhere within the data.

The danger is that apps that edit data can inadvertently damage or remove the EXIF metadata, and that’s all too common among image editors that may need to rewrite the whole of the data when saving an edited image. Fortunately macOS relies on its own QuickLook thumbnails rather than embedded EXIF thumbnails, as some image editors don’t update the latter reliably.

There are more subtle disadvantages to embedding metadata with data. In Apple’s preferred model, there are separate datestamps in file attributes for saved changes to data and extended attributes, allowing them to be distinguished.

Summary

In macOS, metadata can be stored

• in the file system as attributes,
• as extended attributes,
• embedded with the file data.

It’s hardly surprising how often it goes missing, or is overlooked.

Spotlight local search problems are common, and are all too often tackled blindly by forcing a volume’s indexes to be rebuilt. Although that can sometimes resolve the problem, without knowing its cause, it can just waste time and effort. In some cases rebuilding the indexes can worsen the problem, at least temporarily. This article explains how to use SpotTest and other tools to perform systematic testing and arrive at a diagnosis before hazarding a guess at its treatment.

1. Setting up

Before going any further, check that Spotlight settings are in order, and don’t exclude the volume or folder you’re trying to search, or the document type you’re looking for. In Spotlight settings,

• check all Results from System are turned on, particularly Files and Folders,
• click on Search Privacy… and remove any locations you want to include in search.

Then open Activity Monitor and watch its CPU % listing to verify that Spotlight isn’t currently in the process of reindexing or performing other maintenance to its indexes. If it is, delay testing until those have completed. Searching using Spotlight while it’s actively working on its indexes will give odd results, or none at all.

If you’re going to use SpotTest on any location with privacy control, such as your Home Documents folder or an external disk, add the app to the Full Disk Access list in Privacy & Security settings, before opening it.

2. Home folder test

Even if your interest is in a different volume, perform a basic test of a new test folder in your current Home folder, to establish a baseline and confirm that Spotlight is working there.

Open SpotTest and set it to defaults, with a Term of cattle, a Scope of [Home], and both Search Keywords and Search EXIF ticked.

Click the Create Tests tool (the leftmost of the tools) to create the folder of test files at the top level of your Home folder. Make a note of the time to the second that you do this.

About 10 seconds after that, click either the Run NSMetadata test or Run mdfind test tool. You should then see a list of files found, including those in the test folder ~/0_SpotTestFiles, including A, B, C, D, E, F, G, K, L, M.

If you don’t see those listed, open Mints and use the Log Window button in its Spotlight section to obtain a log extract from the time the test files were created, or use LogUI to do the same. You’ll then need to look at that log extract to see if there are clues as to why indexing didn’t take place in that period.

Leave the test folder where it is, and anything from 1 hour to 5 days later, repeat the search using either or both of those tools. Once additional indexing has been undertaken:

• NSMetadata should now find A, B, C, D, E, F, G, I, K, L, M but not H
• mdfind should now find A, B, C, D, E, F, G, H, I, K, L, M.

I is found using Live Text, and H by Visual Look Up, the latter only being found by the mdfind search.

These tests have demonstrated:

• mdworker and mds indexing of files supported by system mdimporters;
• delayed background mediaanalysisd image analysis and mds indexing of Live Text and Visual Look Up content.

To match test files with their importers, click the Check importers tool. Note that file L doesn’t use a plugin, and N uses a plugin but can’t be found because the search term is inside an image in the PDF document, which currently isn’t recoverable content.

3. Metadata indexing

If that test isn’t fully successful, or you’re uncertain whether indexing is complete, inspect the metadata of the test files. Open a Finder window on the contents of ~/0_SpotTestFiles, set it to Column View, and widen the window to provide a suitable Preview pane within it. Select each of three files in turn and confirm their metadata are shown correctly. To inspect all available metadata, click on any Show More text.

SpotTestC.pdf will have different datestamps, but the seven fields of metadata below those should be identical to those shown above.

SpotTestK.jpg will also have different datestamps, but the five fields below should be identical to those above.

SpotTestM.txt should include a final line of Keywords one, cattle, three.

You can also check all indexed metadata for those files in Terminal using the commands
mdls ~/0_SpotTestFiles/SpotTestC.pdf
mdls ~/0_SpotTestFiles/SpotTestK.jpg
mdls ~/0_SpotTestFiles/SpotTestM.txt

Missing metadata on those three files demonstrates that the test folder hasn’t been indexed correctly. You can try restarting your Mac, leaving it a few minutes to update its Spotlight indexes, then repeating the tests using SpotTest.

4. Custom mdimporter

Many apps come with their own custom mdimporter that provides Spotlight indexing support for document types not supported by macOS. In the past, these were normally installed in /Library/Spotlight, but more recent apps typically keep them inside the app bundle in Library/Spotlight. These can be tested easily.

Create and save one of those custom document types, so that it contains the word cattle in a way that should be searchable by Spotlight. Copy that document to the ~/0_SpotTestFiles folder, wait about 10 seconds, then repeat the test search. You may well notice that NSMetadata search doesn’t find your custom test document, but mdfind does, because of the difference in the search criteria they use.

You should also click the Check importers tool to check that the correct mdimporter was recognised and used for the custom document type.

5. Volume test

If Spotlight works correctly with the test folder in your Home folder, you may wish to progress to testing a different volume or location. Having created its test folder in ~/0_SpotTestFiles, copy that to the other location. Before you change the Scope of the search, click on the button to list available volumes, then select the volume containing the copied test folder in the Scope menu.

When you perform the two types of search on that volume, the same rules should apply as to which will be found. Note though that finding files I and H can take much longer, or they may not appear at all.

6. Search term test

When you’re confident that a search term of cattle can be found reliably, you may wish to extend your testing to other terms. Take care when choosing custom terms, as you want to be confident that they should be found, but not in such numbers that the results are overwhelming. You will also need to create your own test files containing the custom term.

Diagnosis

SpotTest can thus provide key information on:

• Delay or absence of find following creation of test files. If no indexing activity is seen in the log, that indicates indexing failure. If the test files are indexed promptly, it indicates search failure.
• Delay or absence in finding files H and I, indicating an indexing failure.
• Failure of a custom mdimporter to index a custom document type.
• Failure to index another volume.

Those should fit in with the overall scheme used by Spotlight, as shown below.

Happy hunting!

The Finder can display more information about files than their size and datestamps, and for some types of file can extend to a lot of useful metadata. These are shown in the Preview pane containing the file’s QuickLook thumbnail, in the Get Info dialog, and some can be added to the columns shown in List View. This article explains where those come from, and how you can customise what the Finder displays.

Metadata collection

Within a second or two of a new file being created, or an existing file being saved, Spotlight’s indexing services analyse that file and extract both metadata and, where possible, content to be added to that volume’s indexes. Metadata that is common to most or all files, including datestamps, and the contents of any extended attributes, that might include titles and keywords, is indexed separately from that extracted from a file’s contents.

In between those are metadata embedded in file data. They’re specific to certain types of file, for example EXIF metadata that is commonly included in images, so are extracted by the specialist mdimporter for that file type, then incorporated into the indexes on that volume.

Finder display

When you select an item in a Finder window showing the Preview pane (typically in Column View), two chains of processes are started. One calls on QuickLook to return the thumbnail to be displayed in the upper section of the Preview pane, the other starts a metadata query at a high Quality of Service (25, userInitiated), which is passed to SpotlightServer, and access to that data is checked by TCC. Once approved, the metadata is returned from Spotlight to the Finder to populate the Information section below the thumbnail.

Information displayed in the Preview pane depends on that available in the indexes, the type of file, whether the list is set to show more or less, and the Finder’s settings in its Show Preview Options command in the View menu. That displayed in a Get Info dialog undergoes similar processing, to populate its More Info section in particular, although those don’t appear to come with any options.

Preview Options

To a degree, the user determines which fields are displayed in the Information shown in the Preview pane, although Apple doesn’t mention the key setting involved. Select the file, ensure the blue text to the right of Information is set to Show Less, then open its Preview Options using the Finder’s View menu.

Here are my current Preview Options for all Image files, which only include a single item from EXIF metadata, the Content Creator (which is duplicated in the list of options). While that window is open, those are the only items shown in the Preview pane.

When that Preview Options window is closed, the Finder immediately reverts to its comprehensive list, including many of those in the EXIF metadata, until you click on Show Less.

It’s only when the Preview pane is showing less information that your Preview Options are applied, and they’re now used the same for all types of Image.

These are the extensive Preview Options for this CorelDRAW document with a cdr extension, although here they’re claimed to be for a file archive because of a clash in extensions. This list is derived from the mdimporter provided, and correct for CorelDRAW files. Unfortunately, this window is too tall to be accommodated on the display, and doesn’t scroll.

When set to show more information, all non-empty fields appear in the list.

With less information showing, the list conforms to that set in its Preview Options.

To confirm the list of metadata, we can usually inspect what Spotlight should have indexed from that file.

Discovering metadata

In Terminal there are two ways to list the metadata for a file. The first is to interrogate Spotlight with the command
mdls filename
which should list all attributes with their values, except indexed content such as text tokens.

The other method using mdimport does something subtly different. Enter the command
mdimport -t -d2 filename
for a file with the path and name filename, and you’ll either see a long list of all its Spotlight metadata, or the command will crash. Although it’s easy to mistake this for the metadata stored in Spotlight’s indexes, it’s actually what should be stored there when that file is processed by the mdimporter named in the output. Its occasional crashes are a mystery, though, as it used to be reliable up to and including macOS Sonoma.

If there are metadata missing from mdls and mdimport‘s output and not shown in the Preview Pane when listing more information, you can only presume that they’re missing from Spotlight’s indexes, so won’t be discoverable in a Spotlight search.

Conclusions

• The Finder populates the information in its Preview pane from the file’s metadata in that volume’s Spotlight indexes.
• When showing more information, the list should include all non-empty metadata appropriate to that type of file.
• The Finder’s View Options customise what’s shown for all files of that type when there’s less information being shown.
• Use mdls to check those against metadata stored in Spotlight’s indexes, and mdimport is also helpful, if it doesn’t crash.
• If metadata are missing from the Preview pane, mdls and mdimport, they’re likely to be missing from Spotlight’s indexes as well, and are unlikely to be discovered by Spotlight search.

Here’s a bonus riddle for this weekend: what’s so invisible you can never see it in the Finder, is in many of the folders in your Home folder, and can break your backups? The answer is a .DS_Store file, officially a Desktop Services Store. Although they might appear more ancient, they originated in Mac OS X when its Finder was being rewritten from scratch in 1999.

It had been intended that Desktop Services would eventually gain a public API, but somewhere along the line Apple decided to keep it private, and their format and function have never been officially documented. Its name starts with a dot/stop/period to make it invisible in the Finder, and since macOS Sierra it has been made invisible even when the Finder reveals other invisible files. Currently the best way to see it is in Terminal, where the -a option to ls should include .DS_Store files.

They can be confused with another annoying but more useful hidden file: shadow files whose names start with ._ that are used to carry extended attribute data as part of the AppleDouble file format used on some FAT file systems. They too are invisible in the Finder even when hidden files are supposed to be displayed, but are associated with individual files rather than folders.

Function

The Finder will normally create a .DS_Store file in a folder that you have write access to, when some change is made to it in the Finder, such as creating or copying a file into that folder.

.DS_Store files contain a folder’s custom attributes, data like icon positions, and in more recent versions of macOS custom settings for the display of file metadata.

Among the most important of their contents for some users are Finder or Spotlight Comments, which are normally displayed in the Comments section of the Get Info dialog for a file. Those comments may also be duplicated in the com.apple.metadata:kMDItemFinderComment extended attribute (xattr) of that file, but that’s a secondary copy that can fall out of sync with what’s stored in the .DS_Store file, and the Finder ignores the xattr anyway. The reliance of Finder Comments on invisible .DS_Store files can lead to their unreliability compared with other forms of metadata.

Problems

You’re more likely to come across .DS_Store files when they make a nuisance of themselves by tripping something up. Send a folder from your Mac to a Windows or Linux system, for example, and it’s likely to confuse the recipient with that mysterious extra file that you can’t see at all. Send a folder to another Mac by AirDrop, and any .DS_Store file inside it will also accompany its visible contents. That in turn can cause problems with some backup utilities if it results in an older .DS_Store file being found in a folder that has already been backed up with a newer one.

Recent versions of macOS should no longer write .DS_Store files to computers connected to them over a network. If you want to stop them from being exposed in network volumes of older systems, use the command
defaults write com.apple.desktopservices DSDontWriteNetworkStores -bool true
to disable that. One place .DS_Store files can prove particularly troublesome is in Git repositories. Mikey @0xmachos has provided a simple solution for eradicating them.

At one stage Apple even recommended that they should be explicitly excluded from servers used for network backups or other storage. They can trip up revision control systems, baffle those who open archives created on a Mac, stop folder copying, and confound folder comparison. The simple solution to these, as with so many other problems with .DS_Stores, is to open the folder containing that hidden file, move some of its contents about to force it to be refreshed, and move on.

In the past, .DS_Store files have been suspected of leaking data, and were involved in at least one security vulnerability. Thankfully they now seem as puzzling and opaque to the developers of malware as they are to other users, but I’m sure that one day, someone else will try to do bad things with them again.

Removal

You can recursively delete .DS_Store files from a hierarchy using the command
find . -name .DS_Store -delete
and Ross Tulloch’s BlueHarvest can automatically remove them.