Just over six months ago a series of weird bugs came to light in Spotlight indexing. The first report was that plain text files beginning with the characters LG are never indexed, so their contents can never be found by Spotlight search. The mystery deepened when the same was discovered for text files beginning with the characters NPA or Draw. It was appropriately Drew who worked out the common factor behind this apparently bizarre connected behaviour: all three files are identified as not being text by the old Unix utility file(1), used to recognise file types by ‘sniffing’ their contents.
You can verify that by creating a plain text file with any of those three sets of characters at its start, then running the command file on that file. In the case of one beginning with Draw, file will identify it as RISC OS Draw file data, even though the file has an extension of txt or text and a UTI of public.plain-text. At that the RichText mdimporter, which analyses all text-based files for metadata to enter into Spotlight’s indexes, throws its hands up in horror and refuses to index the file’s contents. Change those opening characters in that file, perhaps by adding a leading space, and all of a sudden the mdimporter works as expected.
Following our collaborative effort here, particularly Drew’s insight, we realised this bug has been silently blocking the indexing of seemingly random text files for the last three years or more. What remained unanswered at the time was what that mdimporter was doing running file(1) on files whose UTI made it clear that they were in plain text, not some long-forgotten binary vector graphics format from 1989. I believe I now have an answer, thanks to my recent work on QuickLook’s qlgenerators.
QuickLook’s generators take advantage of the hierarchical structure of UTIs. Rather than accepting the most specific UTIs such as public.jpeg, Image.qlgenerator works with all files whose UTI conforms to the generic UTI of public.image, and then undertakes its own format detection. This enables it to generate correct thumbnails and previews of HEIC images that have been given the incorrect extension of jpg, for instance.
Similarly, a Swift source-code file with the extension of swift and the UTI of public.swift-source is handled by the Text.qlgenerator because public.swift-source conforms to public.plain-text, the UTI required for use of that generator.
What if Spotlight’s mdimporters were to work the same?
We know the built-in RichText.mdimporter is used to extract metadata for a wide range of files containing text, which all conform to the generic UTI of public.text. It then classifies them on the basis of their contents to work out what to index. What if that’s performed using file(1), so rejecting perfectly valid text files as ancient binary vector graphics files, and so on?
We can’t get the same direct evidence from the log that I obtained for QuickLook, as Spotlight is far less informative in its log entries. We can get clues from looking at output from mdimport and mdls, though. While a non-deviant text file contains a metadata attribute extracted by its importer as kMDItemTextContent containing the text in the file’s data, that’s missing from a text file starting with any of the three known triggers. In turn that’s associated with the attribute _kMDItemPrimaryTextEmbedding containing ‘vec_data’ listed by mdls, which is also missing for the deviant files.
There is hope that a third party might be able to undercut RichText.mdimporter by providing a bug-free importer for public.plain-text, but that relies on the built-in importer targeting public.text rather than public.plain-text. The best solution would be for Apple to fix the identification of text files instead of relying on file(1), which dates from 1973. Given that these deviant files work perfectly with QuickLook’s generator, it appears Apple has already solved this problem there. So I suspect this bug in RichText.mdimporter will never be fixed in Sequoia or Tahoe.
With the first beta-release of macOS 27 just a couple of weeks away, this leaves those using the last Intel Macs stuck with Spotlight indexing that will never work on some text files, assuming that at some point in the not too distant future this bug is finally fixed in an Arm-only macOS. This is all sadly familiar from the loss of 32-bit support in the transition from Mojave to Catalina, when little if any effort was devoted to making Mojave as free of bugs as possible before it was abandoned in the rush forward to 64-bit.
It would have been far better to be able to look back in fondness with macOS that worked better, than looking back in anger at what never got fixed.
One last thing to remember is that, when Apple does fix this bug, you’ll have to force Spotlight indexes to be rebuilt on each of your Mac’s volumes to ensure that the contents of these files are incorporated. We learned that last time there was a serious bug in the same importer, which failed to index the contents of RTF files.
In case you haven’t got the message from the last few weeks looking at Spotlight and QuickLook, UTIs (Uniform Type Identifiers) are important, but not always as critical as they could be. To understand how macOS copes with misleading UTIs, I have a little demonstration you can try in the privacy of your own Mac.
All you need for this is an image with some Exif metadata. Those taken by an iPhone are particularly suitable, as they usually contain rich Exif information about which model took that image, focal length, aperture, exposure time and more. In my case, the image is in HEIC format. If you have my apps UTIutility and SpotTest, you can also explore UTIs more thoroughly, and inspect the metadata from images that gets indexed by Spotlight, but those are optional extras.
A file with the extension HEIC or heic is assumed to have the UTI public.heic, which conforms to public.heif-standard, and that in turn conforms to public.image, the parent of most image formats in macOS. The Help book for UTIutility shows these in a dense diagram.
Select that image in the Finder’s Column view to inspect its public metadata. While the image is selected, open Show Preview Options in the View menu and enable all the metadata listed there to be shown in previews.
You should then have a good preview pane with lots of metadata below it.
Next open a new Finder window and set it to Find. Using the search criterion popup menu, enable the Device model attribute, or another your image has metadata for, and search for that attribute, here iPhone XR, and you should see your image among the hits.
If you have SpotTest to hand, drop your image on its Drop Window. Being an image, it will crash mdimport, so the information you’ll see will be the metadata fields from Spotlight’s indexes, which should include the Device model as kMDItemAcquisitionModel.
So far, everything has worked as expected, but we’re now going to throw a spanner in the works by changing the extension on that image from HEIC to jpg, which changes the image’s UTI to public.jpeg, although that still conforms to public.image.
Its basic thumbnail icon now changes to a generic JPEG icon, so we’ve managed to confuse the basic thumbnailing scheme in QuickLook. But it’s still shown in the preview pane correctly, with all its metadata intact.
This is because that image has its larger thumbnails and previews generated by the qlgenerator for public.image, and that goes out of its way to parse the file data correctly, and recognise this is really a HEIC not a JPEG. If you’ve left the Finder Find window open, you’ll see that continues to find the image as if nothing had happened, as Spotlight also imports metadata using a common mdimporter for public.image, rather than relying on the more specific UTIs of public.jpeg or public.heic.
Finally, change the file’s extension to text, and you’ll see a preview of its text content, and it vanishes from the Find window too. That’s because text files are handled by their UTI of public.text, which includes public.rtf and others. Those don’t check the file data to ensure they’re not images, so the file is now being handled by the wrong qlgenerator and mdimporter, and won’t make any sense. As public.text formats don’t support Exif data, that isn’t extracted either, as you can see in SpotTest.
Change the extension back to heic, and you’ll see how quickly the right qlgenerator and mdimporter correct its thumbnails, previews, and search discovery, thanks to UTIs.
With an understanding of how QuickLook provides thumbnails and previews, you can be systematic when tackling its problems, although thankfully those are infrequent if not rare.
Generic icon
By far the most common problem with QuickLook thumbnailing is when a file’s specific thumbnail isn’t shown, but a generic icon for that type of file appears instead. This has been particularly common since the release of macOS Sequoia, as that ended support for older third-party generators in qlgenerators. To be able to extend the range of types supported by thumbnailing, third-party generators must now be supplied as appexes stored in an app bundle’s PlugIns folder or similar.
To pin this down, you’ll first need to discover the UTI of the files whose icons can no longer be turned into specific thumbnails. One easy way to do that is in my free UTIutility. Type in the file’s extension, press Return and the app will tell you that file’s UTI and those it conforms to.
You next need to discover which generator handles those UTIs. The official way to do that is using the command qlmanage -m
but that now only lists qlgenerators supplied in macOS, as qlgenerators. To see listings of others as well, open my free Mints and click on its QuickLook button.
For qlgenerators, you’re given the file UTI, the path to the qlgenerator file, and (when available) its version number, e.g. com.adobe.pdf/System/Library/QuickLook/PDF.qlgenerator (1002.2.3)
App extensions are divided into two, first those providing Previews, and second those for Thumbnails, e.g. com.apple.applescript.text/Applications/PreviewCode.app/Contents/PlugIns/Code Previewer.appex
If no generator handles the file’s UTI, ascend the list of UTIs it conforms with to discover which generator should attempt to. If you think an old qlgenerator is the problem now, contact the app’s developer and ask whether they intend supporting macOS with an appex replacement.
Occasionally you may come across an extension conflict, in which the same extension is used for another UTI, resulting in the wrong generator trying to create a thumbnail.
The nuclear option for any QuickLook problem is to reset its caches, using the command qlmanage -r
Although its effects might be to slow thumbnail generation for a while, it’s unlikely to prove any more damaging.
Digging deeper
If you have to go any deeper than that, you’re going to need to capture good log extracts to enable diagnosis. As far as Icon Services, QuickLook and related features are concerned, it’s essential to disable log privacy before going any further, or you’ll be driven crazy by all those messages gutted and rendered meaningless by censorship.
Even then, log entries refer to key items such as files and folders using references that may appear opaque. Some abbreviate file names and directories as ‘B{14}1.jpg’ for BeltedGalloways1.jpg, and ‘t{5}s’ for testims, as well as referring to them by hex numbers like 0xBBDBEFDB0. Another common habit in log entries is to refer to files by their inode number, either as an ino, or in a full URL such as file:///.file/id=6571367.243284. The use of UUIDs is also common, for example as uuid:0AD8986E-6325-4FF1-92FD-9FD3C15D57EA.
Example thumbnail generation
This was initiated by a mouse click, following which a thumbnail isn’t immediately available from cache.
This leads to the file’s UTI type being looked up in the dictionary of those known to be handled by bundled qlgenerators. These are the log entries most important to those hunting generator problems.
01.017971 QuickLookSupport com.apple.quicklook | No exact match found in type dictionary 0xc352a7ae0 for 'public.jpeg' #UTI
01.018010 QuickLookSupport com.apple.quicklook | Getting 5 for 'icon flavor' from UTI 'public.image' #UTI
01.018012 QuickLookSupport com.apple.quicklook | Getting 5 for 'icon flavor' from UTI 'public.jpeg' #UTI
01.018765 QuickLookThumbnailingDaemon com.apple.quicklook | Generating thumbnail for <QLThumbnailItem: 0xc351c36c0> (size (16.0, 16.0)) with badge type 1 with extension <QLThumbnailExtension: 0xc354ec540>
That thumbnail is then entered into the store and its index.
Once in the Thumbnail Cache, loading should be very quick, typically around 0.0001 seconds from start to the thumbnail being set as displayable.
Example missing generator
This shows some of the salient log entries made when a file type doesn’t have a generator available, in this case for an IconComposer icon file.
This is most obvious from UTI dictionary lookup
00.690352 QuickLookThumbnailingDaemon com.apple.quicklook | About to generate a thumbnail locally from URL: file:///Users/howardoakley/Documents/mints%20icon/Mints.icon/
00.690897 QuickLookSupport com.apple.quicklook | No exact match found in type dictionary 0xb933cfce0 for 'com.apple.iconcomposer.icon' #UTI
00.690957 QuickLookSupport com.apple.quicklook | No exact match found in type dictionary 0xb933cfce0 for 'com.apple.package' #UTI
00.691040 QuickLookSupport com.apple.quicklook | No exact match found in type dictionary 0xb933cfce0 for 'public.directory' #UTI
00.691092 QuickLookSupport com.apple.quicklook | No exact match found in type dictionary 0xb933cfce0 for 'public.item' #UTI
00.691098 QuickLookSupport com.apple.quicklook | Caching NSNULL as icon flavor for 'public.item' #UTI
00.691100 QuickLookSupport com.apple.quicklook | Caching NSNULL as icon flavor for 'public.directory' #UTI
00.691101 QuickLookSupport com.apple.quicklook | Caching NSNULL as icon flavor for 'com.apple.package' #UTI
00.691102 QuickLookSupport com.apple.quicklook | Caching NSNULL as icon flavor for 'com.apple.iconcomposer.icon' #UTI
00.691797 QuickLookThumbnailingDaemon com.apple.quicklook | Generating thumbnail for <QLThumbnailItem: 0xb92e10500> (size (16.0, 16.0)) with badge type 1 with extension <QLThumbnailExtension: 0xb934000e0>
The resulting error refers to the missing generator, not the file whose thumbnail is being generated.
00.769110 error QuickLookThumbnailing com.apple.quicklook | Generation error for request <QLFileThumbnailRequest:0x8cb0a8140 maximumSize=(16.00, 16.00) minimumSize=(0.00,0.00) scale=2.0 item=<QLThumbnailItem: 0x8cb014dc0>> : Error Domain=NSCocoaErrorDomain Code=260 "The file couldn’t be opened because it doesn’t exist."
The solution is to generate a placeholder icon as the ‘most representative’.
When that is accessed later for use in the Preview pane, that’s made clear.
02.773702 Finder com.apple.AppKit | ISImage reported a placeholder, image rep is providing a placeholder image for <ISBundleIcon 0xbd0165d10> Bundle URL: file:///System/Library/CoreServices/Finder.app/ type: (null) tag: (null) tag class: (null) digest:96BABFB5-FBB5-0662-25FA-99CA1A147F30
Once the generator has been made available, in this case by running the IconComposer app for the first time, it can be used to generate a thumbnail and preview. Note this requires the appex to be launched first.
02.409348 QuickLookSupport com.apple.quicklook | No exact match found in type dictionary 0xbd01fcf00 for 'com.apple.iconcomposer.icon' #UTI
02.409406 QuickLookSupport com.apple.quicklook | Getting <QLGenerator Package.qlgenerator> for 'Generator' from UTI 'com.apple.package' #UTI
02.409408 QuickLookSupport com.apple.quicklook | Getting <QLGenerator Package.qlgenerator> for 'Generator' from UTI 'com.apple.iconcomposer.icon' #UTI
02.676198 ExtensionFoundation com.apple.extensionkit | Extension `/Applications/Icon Composer.app/Contents/PlugIns/Icon Composer QuickLook Preview.appex/Contents/MacOS/Icon Composer QuickLook Preview` of type: `` launched.
To see almost all these log entries if you’re using LogUI, fetch all the entries for the period in question, then use its Search feature to display only those with a subsystem of com.apple.quicklook.
Login
