Reading view

There are new articles available, click to refresh the page.

Turn a sparse bundle into a TARDIS: an odd bug in Sequoia

The most distinguishing feature of Doctor Who’s TARDIS is that it’s much smaller on the outside, only capable of accommodating a couple of people at a squeeze, but huge on the inside. Allow me to introduce you to a sparse bundle that’s similar, in that its bundle size is only 14 MB, but the size of its internal band files comes to a total of over 90 GB. Here’s how to make yourself one.

Method

To do this, you need an Intel or Apple silicon Mac running Sequoia 15.0 or 15.0.1, and an external SSD. A hard disk might do, but I haven’t tried it.

Connect and format the external SSD in APFS, using Disk Utility. Then create an empty sparse file with the following specifications:

  • Size: 10 GB or more
  • Format: APFS
  • Encryption: none
  • Partitions: Single partition – GUID Partition Map
  • Image Format: sparse bundle disk image.

Save that to the external SSD at its top (root) level. As Disk Utility leaves your new sparse bundle mounted, unmount it, wait a few seconds, then double-click it to mount it again. Copy one or more large files to the mounted sparse bundle. With a bundle size of 10 GB, something around 5-6 GB should be ideal, but the sky’s the limit if you created a large sparse bundle to begin with.

Once that’s complete, unmount the sparse bundle, then select it on your external SSD and press Command-I to see the Get Info dialog.

Although the size will vary slightly, in my case, with a nearly full 125 GB sparse bundle, the Finder reports its size as 13.8 MB, or 14.8 MB on disk.

diskimbug1

So where did all that file vanish to? Has APFS somehow made it into a sparse file? Sadly not: view the contents of the sparse bundle, select its band folder containing all the band files with your data, and its size will be much bigger than that of the bundle. In my case, that’s 92.7 GB in more than 10,000 band files.

diskimbug2

There’s your TARDIS: a 13.8 MB sparse bundle containing over 90 GB of band files.

Breaking the illusion

Once you’ve created your TARDIS sparse bundle, you can see its magic still works when you connect that external SSD to another Mac. But there’s one thing that will break the magic spell: create a folder in the same volume, and move the sparse bundle inside that. If you then mount it and add a file, you’ll see it suddenly expand to its full bundle size. Similarly, if you repeat the whole trick, but this time save the sparse bundle inside a folder, then it won’t work. This is why you can only do this on an external SSD, as you can’t write anything to the top level of your boot disk.

I don’t know whether this works with other bundles, such as VMs for Apple silicon Mac virtualisation, but it might be interesting to try it.

Act now, don’t wait

While this remarkable bug is present in macOS Sequoia 15.0 and 15.0.1, I’m afraid its days are numbered. If you want to experience the TARDIS sparse bundle, you’ve only got another week or two, as it appears to be fixed in the current beta of 15.1. For once, I’ll be a little sad to see a bug fixed, although when I discovered it, it couldn’t have been more infuriating.

Last Week on My Mac: the Finder is growing less consistent

Over the last forty years, one of the essential virtues of the Mac’s interface has been its consistency. From the outset, Apple has ensured the Finder behaves according to a small set of rules that are readily learned, often without conscious effort. These form a grammar grounded on those of languages like English, of subject-verb-object. Select a file, do something with it, and it changes state.

Drag a file from its current folder and drop it onto another, and it’s moved or copied from one to the other. The outcome depends on context, where performing that action within the same storage volume results in the file being moved, but between folders in different volumes it’s copied instead. Folders play two roles: as enclosing elements, and as objects in their own right. Drag a folder from one place to another, and you expect its entire contents to go with it, but apply a tag to a folder, and only that folder gains the tag.

iCloud Drive has never quite conformed to the same grammar. Change its mode by enabling the Optimise Mac Storage setting and it stops behaving so consistently. Files that appear to be present turn out to lack data content, and have to be marked as evicted exceptions. Folder commands to control download and eviction depart further from established norms. Select all, and if there are more than ten items selected, Remove Downloads is no longer available in the contextual menu. Can the Finder no longer cope with certain actions when they’re applied simultaneously to more items than we have fingers?

There is a workaround, though: select just the enclosing folder, and Remove Downloads on that. If you then want to download individual items within that evicted folder, you do so by selecting them and using Download Now, which oddly doesn’t seem constrained to the same limit of ten. You can thus have a hybrid folder, with some items downloaded, others evicted, and the folder itself continues to display the icon indicating its contents remain evicted.

Now consider Sequoia’s new pinning behaviour. Pin a folder and its contents are shown as being pinned, as you’d expect. Select one or two of those individual files, though, and there’s no option offered to unpin them. The only way to do that is to unpin the whole folder, then pin individual items within that folder. But like the Remove Downloads command, that too won’t work for more than ten items at once. Move a new item inside a pinned folder, and it automatically becomes pinned, and can’t be individually unpinned as long as its enclosing folder remains pinned.

If you have a folder of 100 files, and want to pin all of them bar one, this becomes laborious. If you pin the folder, you can’t unpin that one file, so you must leave the folder unpinned and instead pin the individual files inside it, no more than ten files at time.

Against the Finder’s consistent model, eviction and downloading are well-behaved, as you can

  1. select the folder, and apply the action to it,
  2. select the exception(s), then apply the inverse action to them,

requiring just two selections and two actions.

On the other hand, pinning behaves inconsistently, as you must:

  1. select a group of no more than ten files, apply the action to them,
  2. repeat that as many times as necessary, until only the files you want pinned have been pinned.

Consistency is often taken as a mark of reliability. iCloud Drive comes from a position of lower reliability, and inconsistencies in its human interface only serve to reduce the user’s trust.

Explaining this idiosyncratic behaviour is more important still. Instead of being designed for the human, its interface has been determined by its engineering, in a step back to computers that preceded the Mac. This reflects an API that is every bit as awkward as its human interface. In common with other properties of files in iCloud Drive, developers are prevented from discovering how iCloud Drive handles them. For example, Apple makes checking whether a file is evicted as difficult as possible, as there’s no property or attribute that records that.

Pinning is similarly convoluted, and currently undocumented. When pinned individually, files are given a distinctive extended attribute, but when pinned by virtue of any of their enclosing folders being pinned, that xattr is attached not to the files, but to the folder. To check whether any given file in iCloud Drive is pinned, an app therefore has first to check whether that file has the xattr, then to traverse its path upward and look for the xattr on every folder in that path.

Thus the human interface for pinning is determined not by the Finder’s consistent grammar, but by the implementation of this feature in iCloud Drive’s virtual file system. The sooner consistency is restored, the better for the Finder and its users.

A brief history of the Finder Alias

It wasn’t until System 7 in 1991 that the Mac gained any feature that let the user create links to files or folders. Without a command shell, Unix traditions of symbolic and hard links weren’t available. Apple belatedly added what it termed an alias, created using the Make Alias command in the Finder’s File menu. This made a document-like object bearing the name of its original with alias appended, displayed in italics to distinguish it from the original. To help users locate the original file for an alias, the Finder’s Get Info dialog gained a button to Find Original, later moved to the File and contextual menus.

Later versions of classic Mac OS added refinements to this transformative feature, including the selection of a new target for a broken alias, creation by drag-and-drop of a file or folder with the Command and Option keys held, and a distinctive arrow badge to both the item’s icon and the pointer during drag-creation.

Internally, the alias was a small file of less than 5 KB size containing opaque data with more information than just the path to the original. Aliases were designed to support free movement within the same file system, at the time an HFS volume (or partition, as they’re the same in HFS and HFS+).

Aliases transferred across to Mac OS X, where the more adventurous could open Terminal and create both symbolic and hard links instead. It has remained a sign of the lasting schism between the GUI and Unix internals of Mac OS X that there are no standard command tools supporting Finder aliases, and there’s no way to create or maintain symbolic or hard links in the Finder.

Something happened to aliases when they transitioned into Mac OS X, and their size started to rise steadily until they reached 1-5 MB in El Capitan.

aliastiger104

Back in Mac OS X 10.4 Tiger they were still fairly small, this one at 48 KB. Note the Select New Original… button in the Get Info dialog.

Aliases were all very good for the user, but OS X needed something similar that could be stored in property lists and other files, so the alias format was rejigged into more generalised form as the bookmark, introduced in OS X 10.6 in 2009. In early 2012 with 10.7.3, bookmarks could be security-scoped with permission on a per-app or per-document basis, to enable their use with the app sandbox. By OS X 10.9 Mavericks in 2013, bookmarks were in widespread use throughout the system.

aliaselcap1011

In El Capitan (2015), bookmark size often reached 1 MB, while the Select New Original… button remained the same in the Get Info dialog.

macOS Sierra brought a major revision to both aliases and bookmarks, reducing their size substantially, but bringing other problems. As late as 10.12.1 they were still having problems resolving some links. The end result was worth the struggle, though, as they have since become more robust than ever, with a typical size of only 1 KB, making them almost as efficient as symbolic links.

Although aliases and bookmarks are still intended to be treated as opaque, their contents have become more accessible. Among the useful values each contains are:

    • _NSURLPathKey gives the full path to the file,
    • _NSURLFileIDKey gives the inode number of the file,
    • _NSURLBookmarkURLStringKey gives the file’s full URL,
    • NSURLCreationDateKey gives the file’s creation timestamp,
    • NSURLIsRegularFileKey indicates whether it’s a normal file,
    • NSURLIsPackageKey indicates whether it’s a package rather than a file,
    • _NSURLBookmarkSecurityScopeCryptoKeyKey is used if this is a Security-Scoped Bookmark, most used with sandboxed apps.

I even have a couple of utilities that work with them. Among the many features of Precize is the ability to generate bookmarks, and to analyse and resolve them. For those who want to bridge the divide between the GUI and command line, alisma is a tool that can both create aliases and resolve them to paths. Finally, Alifix helps you refresh and identify broken aliases.

❌