Since Catalina and Big Sur, macOS has started up not from a single volume, but a whole boot volume group. Among those are the System and Data volumes, intertwined by their firmlinks, a paired Recovery volume, and hidden volumes for virtual memory swap space and preboot firmware. To help macOS know which is which, each of those has a role assigned. This article explores how that works, how you can hand-craft your own Time Machine backup volume, and wonders what a Sidecar backup is.

Volume roles

Tucked away in the superblock of each APFS volume is an unsigned 16-bit integer setting that volume’s roles, chosen from 18 values ranging from None to Prelogin. Although I’m sure I’ve seen these disclosed in Disk Utility in the past, at the moment it appears the only way to read a volume’s set roles is in the command line, using the diskutil command tool, which can also create roles for a new volume, and change them for existing volumes.

The volume superblock of those that are part of a boot volume group also contains a UUID identifying that group.

Boot disk structures on Intel and Apple silicon Macs differ, as shown in the diagrams below.

That on the internal storage of Intel Macs consists of two partitions, of which only one is an APFS container.

Apple silicon Macs have three APFS containers, with their own volume groups.

According to Apple’s ageing APFS Reference, now over four years since its last update, roles found in Macs include:

• System (S), for a bootable system,
• Data (D), for mutable system components and mutable data,
• Preboot (B), for boot loader ‘firmware’,
• Recovery (R), for a Recovery system,
• VM (V), for virtual memory swap space,
• Update (E), whose purpose isn’t clear,
• XART (X), for hardware security on Apple silicon,
• Hardware (H), for firmware data in iOS, but also present on Apple silicon,
• Backup (T), for Time Machine backup stores.

There are also some that may not be encountered on Macs:

• Enterprise (Y), for enterprise-managed data in iOS,
• Installer (I), for install logs etc.,
• Sidecar (C), for Time Machine.

Finally, there are three that don’t currently have a documented character code:

• User, for Home directories,
• Prelogin, for system data used before login,
• Baseband, for radio firmware in iOS.

Using volume roles

You can view, set and change volume roles in the diskutil command tool, using its apfs command set. Although not listed now by Disk Utility, the command
diskutil apfs list [containerReference]
displays role information about every volume in the container with the given reference. Omit that option and you’ll get information for containers on all mounted disks. Passing a container reference of disk9, for example, might reveal that volume disk9s1 has a Backup role, when it’s the current Time Machine backup store.

This is potentially useful information when you’re trying to understand some of the complex structures that can occur within containers. If you follow Apple’s advice when creating multiple boot volume groups, you’ll install two or more versions of macOS within the same container. If anything goes wrong with that, then it’s essential to be able to identify which are within each boot volume group, something that should be shown clearly by diskutil apfs list.

When adding a new APFS volume to an existing container using the addVolume command, you can pass an option -role to set its role using the single characters given in the lists above, such as T for a Time Machine backup store. If that option is omitted, then no role is assigned as a default.

You can change the role of an existing APFS volume using the changeVolumeRole (or chrole) verb
diskutil apfs chrole [volumeDevice] [role]
for example,
diskutil apfs chrole disk9s1 T
to set disk9s1 to a Time Machine backup role.

This enables you to investigate how volume roles work.

Investigating backup roles

There are enormous problems in trying to perform surgery on boot volume groups, as you’re unable to pair System and Data volumes with firmlinks, or set the volume group UUID in each volume’s superblock. But there are two roles that merit further investigation, Backup and Sidecar, both apparently for use with Time Machine backup stores. I have seen it suggested that older Time Machine backups are stored on volumes with a Backup role, while newer backups are on those with Sidecar roles. So I created two test volumes, both using case-sensitive APFS, as Time Machine likes.

The Finder displayed the Backup volume using its distinctive icon for Time Machine backup stores, and Time Machine appeared happy to add it as a store, although it would need to change the volume’s permissions to set the User to read-only access.

The Sidecar volume vanished from the Finder’s normal list of mounted volumes, although it remained accessible in /Volumes, which it was shown with the regular volume icon, not that for Time Machine backup stores. That’s very different behaviour from current Time Machine backup stores. There’s another problem with the explanation given for these two roles: older Time Machine backups are made to HFS+ not APFS volumes, which don’t have volume roles at all.

Apple’s other use for the name Sidecar refers to the use of an iPad as a secondary display for a Mac, and doesn’t involve APFS volumes at all. So I’m left wondering whether Sidecar volumes are the unused remains of an old now-abandoned backup project, or the promise of something in the future.

Key point

Discover and investigate APFS volume roles using the diskutil apfs list command, passing the reference to a container, e.g. disk9, if you wish to be more specific.

Have you ever wondered whether someone else has changed your Mac’s storage? Or which version of macOS formatted each of its volumes? As all good forensic investigators know, APFS keeps detailed records of the formatting and modification of each volume. This article explains how you can read and interpret them. As in the tale of Goldilocks and the Three Bears, you may be able to tell who has been eating your APFS porridge.

Information available

Each APFS volume stores details of its history in the volume superblock apfs_superblock_t. Those include information on how that volume was created in apfs_formatted_by, and up to the last 8 times the volume has been modified, in apfs_modified_by.

Although you’ll need a forensic disk analysis tool to get full details, some of that data is easy to access. Select a volume in the Finder, and Get Info will give a time and date that volume was last formatted.

Run First Aid in Disk Utility on that volume’s container, and there’s even more information given about each volume within the container, including those you can’t see. If you’d rather not run a full check and repair, then you should see the same information in Terminal by using
diskutil verifyVolume disk10
where disk10 is the device name for the container. If you prefer you can use fsck_apfs directly, but verifyVolume should use that command’s options most efficiently.

One lingering problem you may encounter in Disk Utility is that it still fails frequently because it can’t unmount volumes for checking. If you encounter that error when trying to run First Aid on a container, try manually unmounting each volume within that container. If all else fails, diskutil verifyVolume appears to be better at handling the problem.

Workthrough

As shown above, when run on one of my external SSDs, information about two APFS volumes was returned, itself something of a surprise. The volume I expected gave
The volume ThunderBay3 was formatted by diskmanagementd (1412.81.1) and last modified by apfs_kext (2313.1.2).
and the surprise, which isn’t mounted, thus effectively hidden, gave
The volume Update was formatted by com.apple.Mobile (1677.50.1) and last modified by apfs_kext (1677.141.2).
The Finder’s Get Info dialog for ThunderBay3 gave a volume creation date of 11 February 2020, and last modification of 20 December 2020.

Taking the visible volume ThunderBay3 first, APFS says that it was formatted by its own formatting tool, diskmanagementd, in APFS version 1412.81.1, which came in macOS 10.15 Catalina (see the Appendix below). A look through details of versions released pins that down to 10.15.3, released on 28 January 2020, which tallies with the creation date from the Finder. Its last modification was performed by a general APFS function, in APFS version 2313.1.2, which is that current for macOS 15.0 and 15.0.1.

The hidden Update volume has had quite a different history, as it was created in APFS version 1677.50.1 in Big Sur, to be more precise in macOS 11.0.1 released on 12 November 2021. That wasn’t a conventional volume creation either, and was performed by com.apple.Mobile, part of the Big Sur installer. It was last modified using APFS version 1677.141.2, which came in macOS 11.6 on 13 September 2021. Since then it appears to have been left unmounted and unused.

The history of that container therefore reads:

• ThunderBay3 created by the user on 11 February 2020 in macOS 10.15.3
• Update created by a macOS installer after 12 November 2021 in macOS 11.0.1
• Update last mounted after 13 September 2021 in macOS 11.6
• ThunderBay3 currently in use.

Conclusions

The hidden Update volume contains a restore log apparently left behind after the 11.5.2 update, together with some empty folders. These demonstrate that it was a temporary volume created by Big Sur’s new macOS installer, but never cleaned up afterwards, and left abandoned for the last three years. As Big Sur was the first version of macOS to use Apple’s new installer that created a Signed System Volume, this is likely to be present on other external disks that were mounted when any version of Big Sur was installed. Although it takes little space, it’s a surprising omission that no subsequent installer has seen fit to clean this up by deleting the volume.

Otherwise, information about the visible and mounted volume appears consistent, and confirms what I recall of its history. No one has been eating this bear’s APFS porridge.

Appendix: APFS and macOS version details

APFS major version numbers change with major version of macOS:

• APFS version 0.3 or 249.x.x in macOS 10.12
• 748.x.x in 10.13
• 945.x.x in 10.14
• 1412.x.x in 10.15
• 1677.x.x in macOS 11
• 1933.x.x in 12.0-12.2.1
• 1934.x.x 12.3 and later
• 2142.x.x in 13
• 2235.x.x in 14.0-14.3.1
• 2236.x.x in 14.4 and later
• 2313.x.x in 15.

Minor version numbers increment according to the minor version of macOS, and patch numbers wander without pattern. Those can be checked by looking at the changes given for each macOS update listed on this page.

If you’re going to use disk images of any type, then getting the right tool for the job is essential. This article considers the leading candidates:

• Disk Utility, bundled with macOS
• DropDMG, $24.99 from C-Command, or from the App Store
• Spundle, free from its Product Page here
• hdiutil, the command tool bundled with macOS.

Although I’m sure there are a great many others, IMHO those should be at the top of your list.

Disk Utility

Create a new disk image using the New Image command in its File menu and there’s a basic range of choices on offer.

This dialog has a longstanding bug, where it can reset the size you’ve entered if you change another setting, which can help you make mistakes. Otherwise, it gives limited access to some of the many options available, sufficient for the occasional and not too demanding user. Further options are available in its Images menu, including verification, adding checksums, conversions between types, and resizing. Notable by its absence is the ability to change the password of an encrypted disk image, which is unhelpfully deferred to the command line.

Documentation in Disk Utility’s Help book is also scant, and insufficient to serve as a reference. As Apple doesn’t provide any further technical information, apart from that in man hdiutil, you may find yourself searching websites such as this.

DropDMG

Since its release over 22 years ago, this has been the first choice for many who need to work with disk images, and is without doubt the best for those who distribute software in disk images. It has grown into the most comprehensive and capable utility for working with any type of disk image, and is backed up by a superb 123-page manual that goes a long way to filling the gap left by Apple. That manual is well-maintained, and contains links to recent technical articles and further information.

DropDMG’s options for creating a new disk image far exceed those in Disk Utility. Particularly helpful are the compatible version hints shown on various options, to remind you of which file systems are available in different macOS versions, and which types of disk image container are supported. DropDMG will even convert old NDIF disk images last used in Mac OS 9 to more modern formats. It will also change the password of an encrypted disk image from a menu command.

For those who need to work with standard configurations, perhaps for software distribution, it lets you save and reuse them with ease. Those can include signing with developer certificates, product licences, background images, custom volume icons, and more. Whichever type of disk image you want to create or maintain, DropDMG should be your first choice.

Spundle

There are a few options for sparse bundles that even DropDMG doesn’t expose, such as control over band size, the ability to resize a sparse bundle, and to change its band size. If you want access to those, Spundle is a useful adjunct.

Note that, unlike DropDMG, Spundle only works with sparse bundles.

hdiutil

Although this remains the definitive command tool that offers types of disk image and features you didn’t even know existed, it’s fiendishly complex to use, with a sprawling and overloaded grammar. Its man page runs to more than 11,000 words, but appears never to have been rewritten into any cohesive account of disk images, or command options. For example, change information is given in two sections, Compatibility and What’s New. Changes made in Catalina and later appear at the end of the Compatibility section, then the final What’s New section reverses time order and goes back from Catalina to Mac OS X 10.5.

I only recommend hdiutil for those who need to work with disk images in shell scripts, or for those few features that aren’t available in DropDMG or, for sparse bundles, in Spundle. It’s a command tool of last resort.

Previous article

Introduction

A disk image is a file or a bundle containing what could otherwise be the contents of a disk. It’s a common way to store and move complete file systems in a neat package, for items that need to be separated from the physical storage provided by a drive. macOS uses disk images for some tasks of great importance, including:

• Recovery and Hardware Diagnostics systems,
• additions to macOS such as Safari, its supporting frameworks, and dyld caches, in cryptexes,
• networked storage for Time Machine backups, in sparse bundles,
• lightweight virtual machines on Apple silicon Macs.

You could use them to store encrypted data on unencrypted volumes, and they’re often used for delivering Apple and third-party software.

Disk images are poorly documented for both users and developers, and have changed significantly over the last few years. Articles in this series explain how to choose between different types of disk image, how to create and use them, and what to do when they go wrong.

Containers and file systems

Disk images consist of two distinct components: the file or bundle itself functioning as a container, and the file system contained inside it. When referred to in this context, disk image containers are completely unrelated to the sandbox containers found in ~/Library/Containers.

This distinction is important in several respects, although it isn’t apparent when you use disk images in the Finder. Preparing a disk image for access involves two separate functions: attaching its container, and mounting any file systems found inside it. When that’s performed by the Finder, perhaps by double-clicking the disk image, those two actions appear fused into one. Similarly, removing the disk image requires all its mounted file systems to be unmounted first, then the container is detached.

One feature that’s widely confused is the encrypted disk image. This involves encryption of the whole container, rather than using an encrypted file system within it. Now that Disk Utility no longer supports the creation of encrypted HFS+ volumes, one remaining alternative is to use an encrypted disk image containing an HFS+ volume.

If you want an analogy for disk images, attaching the container is like connecting an external disk, and once that has been performed, the file systems contained by that disk have to be mounted before you can access their contents.

Types

There are many different types of disk image in use, of which the two this series is most concerned with are plain UDIF read-write disk images (UDRW), and sparse bundles (UDSB). Others you may encounter include:

• plain UDIF read-only (UDRO),
• various compressed versions of UDRW,
• CD/DVD master for export (UDTO),
• sparse disk image (UDSP), a single file rather than a bundle.

Those specify the container format; within each, there’s the usual choice of file systems, although throughout these articles it will normally be assumed that APFS will be used unless otherwise specified.

The word sparse in sparse bundle and sparse disk image doesn’t refer to APFS sparse files, but to the fact that those types of disk image can grow and diminish in size, and normally try to occupy the minimum amount of disk space. This is an unfortunate name collision.

Structure

With the exception of sparse bundles, all disk images are contained within a single file of opaque structure.

Sparse bundles consist of a single bundle folder containing:

• bands, a folder containing the contents of the disk image in band files
• info.plist and its backup copy info.bckup, containing settings including band size
• lock, an empty file
• mapped, a folder containing small data files to match all of the band files except the first
• token, an empty file.

Container size

Until this changed in Monterey (or thereabouts), non-sparse disk images had fixed container sizes. Create a UDIF read-write disk image (UDRW) of 10 GB, and the space occupied by it on disk was approximately 10 GB, whether it was empty or full. Although it remains undocumented, when stored on APFS volumes, UDRW disks can now take advantage of APFS sparse file format, and will normally only occupy the disk space required for the contents of their file system.

This is only true once the disk image has been mounted for the first time after it has been created, mounted and unmounted. To see this, create a test read-write disk image (for example, using Disk Utility) of 10 GB size. Then unmount it, and use the Finder’s Get Info command to inspect its size on disk. That will be 10 GB. Then mount the disk image again, pause a couple of seconds, unmount it, and Get Info will show its size on disk is now much smaller.

As I’ll explain in detail in a later article, this is because each time that disk image is mounted, if its internal file system is HFS+ or APFS, its contents will be trimmed, and saved to disk in sparse file format, which omits all its empty space. This only applies to read-write disk images when they’re stored in APFS file systems; copy them to HFS+ and they’ll explode to full size, as HFS+ doesn’t support the sparse file format.

Considering just the two leading types, empty sizes for a 100 GB disk image are:

• a sparse bundle is 35.4 MB empty, 53.3 GB when containing about 51 GB files, stored across 6,359 band files.
• a read-write disk image (UDRW) shrinks to 333.8 MB once stored as an empty sparse file, 53.6 GB when containing about 51 GB files, in a single file container.

Performance

Some types of disk image perform poorly, and can be very slow to write to. Recent versions of macOS have brought improvements, although some options such as encryption can still impair performance significantly. For the two leading types, when their container is stored on the internal SSD of an Apple silicon Mac, with native read and write speeds of around 6-7 GB/s:

• an initially empty unencrypted sparse bundle reads at 5.1 GB/s, and writes at 4.8 GB/s.
• an initially empty unencrypted read-write disk image (UDRW) reads at 5.3 GB/s, and writes at only 1 GB/s.

Tests were performed using my utility Stibium, across a range of 160 files of 2 MB to 2 GB size in randomised order, with macOS 15.0.1.

Key points

• Disk images consist of a file or bundle containing one or more file systems; the container and its contents are distinct.
• To access the contents of a disk image, the container is first attached, then the file system(s) within it are mounted. In the Finder, those two processes appear as a single action.
• Encrypted disk images encrypt the container, and don’t necessarily contain encrypted file systems.
• Disk images come in many different types, and can contain different file systems.
• Sparse bundles have a file and folder structure inside their bundle folder, with their data saved in band files; all other disk images are single files.
• Sparse bundles grow and shrink according to the size of files stored within them.
• In recent macOS, and on APFS, read-write disk images (UDRW) are stored in APFS sparse file format, enabling them to grow and shrink as well.
• In recent macOS, unencrypted sparse bundles have read and write performance close to that of the disk they’re stored on. Read-write disk images read at similar speeds, but write more slowly, at about 20% of read speed.