APFS has two special file types designed to economise on storage space: clone and sparse files. Clone files are two or more distinct files within the same volume whose data is shared; sparse files save space by skipping empty data and only storing data containing information. This article explores how they behave in use, with particular emphasis on Time Machine backups and iCloud Drive. The latter also involves a third type of special file, dataless files.

Clone files

In contrast to hard-linked files, clone files are two or more distinct files within the same file system (volume) whose file extents are identical, so share the same data, as shown below. They’re created by variants of normal file copying, including duplicating in the Finder (and drag-copying within the same volume), and the cp -c command.

Instead of duplicating everything, only the inode and its attributes (blue and pink) are duplicated, together with their file extent information. You can verify this by inspecting the numbers of those inodes, as they’re different, and information in the attributes such as the file’s name will also be different. There’s a flag in the file’s attributes to indicate that cloning has taken place. At first, the two cloned files share the same data blocks and extended attributes, but as the two files are changed by editing, they start to drift apart and become uncloned.

Clone files are becoming more popular thanks to the Hyperspace app, which deduplicates files within the same volume by replacing copies with clones.

Because they can only exist within the same file system, clone files are fragile. Any copy or move to another file system is invariably accompanied by the copying of their full data, and their economy of storage can only remain as long as they stay within the same volume.

Backups

One notable exception to this same-volume rule is in Time Machine backups. As clone files are preserved in local snapshots, when Time Machine constructs a backup as a snapshot in the backup storage volume, shared file extents are retained, so preserving clones. This is reflected in the size of the backup snapshot, and in the report written to the log. For example, when backing up three distinct files and ten clones of one of those, that report included:
14 Total Items in Backup (l: 16 GB p: 11.02 GB)
3 Files Copied (l: 6 GB p: 1.02 GB)
1 Directories Copied (l: Zero KB p: Zero KB)
10 Files Cloned (l: 10 GB p: 10 GB)

Backups made by other utilities are unlikely to reproduce this behaviour, though, as they can’t synthesise snapshots in the way that Time Machine does. To preserve clone files in their backups, they’d have to identify clones in the source and explicitly perform cloning in their backup store. Although Carbon Copy Cloner claims that “in some cases CCC may clone a file on the destination prior to updating its contents”, it doesn’t appear to attempt to preserve clone files in the backups it makes. I’m not aware of any third-party utility that does.

Unfortunately, Time Machine appears unable to restore directly from backup snapshots in the backup store, and performs Finder copies when restoring. That saves each of those clone files as a completely separate file, without any sharing of data. As a result, the space occupied on disk for a restored volume can be substantially greater than the original or its backup. Extensive use of clone files could thus cause problems when restoring from backups.

Of course, rolling a volume back to a local snapshot, such as one made during Time Machine backups, preserves all clone files within that volume.

iCloud Drive

Clone files created within the same volume as local iCloud Drive storage on the Data volume, or cloned when within a folder in iCloud Drive, remain within the same file system and clones are therefore preserved, and when the file is moved to other folders in the same volume.

However, clone files are treated as simple copies as far as iCloud Drive’s remote storage is concerned. While a pair of cloned 5 GB files only use a total of 5 GB local storage, they require a full 10 GB of your iCloud allocation, indicating that their cloud storage is separate and not common to both. Although the effects of eviction (removing local data) and materialisation (restoring local data from cloud storage) are difficult to observe directly, they appear to lose the benefits of cloning.

When the local copy of a file also stored remotely in the cloud is evicted, its data is removed from local storage, rendering it dataless, as shown below.

When that file is to be used locally again, its data has to be downloaded from the cloud service, and the local dataless file is materialised by adding its data back. As far as I can tell, that doesn’t result in the reconstruction of the shared file extents, so changes cloned files into normal copies with different file extents. You would then need to use Hyperdrive to restore them as clone files. Other Macs sharing the same iCloud Drive also see them as full copies rather than clones.

These behaviours could also catch the user by surprise.

Sparse files

Unlike clone files, the structure of sparse files in APFS is conventional, as shown below.

They achieve their economy in storage by only including file extents containing non-null data, and thus aren’t dependent on remaining within the same file system (volume), making them more robust. Their primary requirement is that they’re created and maintained using specific file system operations, and are only copied or moved to other APFS file systems.

Backups

When backed up by Time Machine to another APFS volume, sparse files are preserved reliably, and are also restored as sparse files. That isn’t likely to hold, though, if the file is transferred using a network file system such as SMB, as all network transfers currently appear to explode sparse files to full size prior to transfer. Because of the way in which they have to be created, only the app maintaining that file could restore its sparse format. In the case of disk images, this should normally occur the next time they’re mounted in the Finder and Trimmed by APFS.

iCloud Drive

Assessing what happens with sparse files in iCloud Drive is considerably simpler than with clone files. As long as they remain downloaded to local storage, they are preserved, and can be moved in and out of iCloud Drive storage without exploding in size. However, they too are stored in full when in iCloud storage, requiring their full size in your iCloud allocation, and the eviction-materialisation cycle explodes them to full size, and their sparse file flag is removed.

The only way to return a former sparse file to its original economical format is then to open and save it using the app that creates and maintains it. In the case of disk images, this should occur when they’re next mounted and Trimmed.

Conclusions

Clone files:

• are only preserved when moved within the same file system (volume);
• are preserved and restored from local snapshots;
• are preserved in Time Machine backups, but aren’t restored from them;
• aren’t preserved in other backups;
• could result in a restored volume being substantially larger than its original;
• occupy their full space in your iCloud allocation;
• are only preserved in iCloud Drive when they aren’t evicted from local storage;
• can be regenerated using Hyperdrive.

Sparse files:

• are only preserved when copied or moved directly between APFS volumes;
• aren’t preserved when copied or moved over network connections, or using SMB;
• aren’t preserved when copied or moved to different file systems, including HFS+;
• are preserved in and restored from local Time Machine backups;
• should be preserved in and restored from other local backups;
• occupy their full space in your iCloud allocation;
• are only preserved in iCloud Drive when they aren’t evicted from local storage;
• can only be regenerated by the app that creates and maintains them.

Both clone and sparse files can result in substantial savings in storage space. However, because that’s fragile, their greatest value is in minimising erase-write cycles in SSDs, hence slowing their ageing.

References

Apple’s APFS Reference (PDF), last revised 22 June 2020.
Dataless files are explained here.
How sparse files work
Files and clones
Special file types, including dateless files

I’ve long been critical of some of the best-selling utilities for the Mac, that set out to perform deduplication of files by detecting which appear to be identical, and removing ‘spare’ copies. This is because APFS introduced clone files, and in the right circumstances those take up no space in storage, as their data is common and not duplicated at all. As it’s practically difficult to tell whether two files are clones, any utility or command tool that claims to save space by removing duplicates can’t tell you the truth, and in most cases won’t save as much space as it claims.

Claims made by those utilities are often exaggerated. This is because they calculate how much space they think they’ve saved by adding the sizes of the potential duplicates they have deleted. That’s not correct when a clone file is deleted, as that doesn’t actually free any space at all, even though the clone file has exactly the same nominal size as the original.

Benefitting from clone files

I’m delighted to see the eminent John Siracusa turn this on its head and finally make better use of clone files in his app Hyperspace, available from the App Store. Instead of deleting clones, his app can replace duplicate copies with clones, and so achieve real space savings. This comes down to simple arithmetic:

• if you have two copies (not clones) of a file in the same APFS volume, the total size they take on disk is twice the size of one of them;
• if you have two clones (not copies) of a file in the same APFS volume, the total size they take on disk is only the size of one of them, as its clone takes no additional space at all.

Hyperspace thus checks all the files in a selected folder, identifies which are identical copies, and (where suitable) will replace those copies (except an original) with clones, so saving real storage space.

I also think it has the most user-friendly payment scheme: download Hyperspace free of charge and check your Mac with it. If it doesn’t find sufficient savings, and you decide not to use it to replace any duplicates with clones, then it costs you nothing. If you want to reclaim that space, then you can opt to pay according to the amount of space it saves, by subscription, or with a one-time payment. On that basis, I unhesitatingly recommend everyone to download it from the App Store, and at least check their Home folder to see if it’s worth paying to reclaim space. You have absolutely nothing to lose.

In my case, perhaps because I tend to clone files using the Finder’s Duplicate command, the savings that it offered were of little benefit, but your Home folder could be different and release 100 GB or more.

Sparse files

The other space-saving special file type in APFS is the sparse file. Although it can bring great savings in storage space, that’s largely up to the app(s) that create and maintain the file, rather than the user. Devising an app that could go round converting plain to sparse files is harder, and risks incompatibility with those apps that access those files.

Fitting 285 GB into 16.5 GB

As a demonstration of how effective APFS special files are in saving disk space, I built myself a 100 GB partition (APFS Container) on an SSD and tried to fill it with clone and sparse files until I got bored.

At this stage, the 100 GB partition contains:

• One 16.5 GB IPSW image file, with nine clones of it, created using the Duplicate command.
• Eleven 10 GB sparse files and one clone, created using my app Sparsity.

Add those file sizes together and they come to 285 GB, yet the 100 GB partition only has 16.5 GB stored on it, and still has over 83 GB free. No compression is involved here, of course.

As the saying goes, there ain’t such as thing as a free lunch, and that free space could vanish quickly depending on what happens to those files. The worst case is for an app not to recognise sparse files, and write one to disk in plain format, so swallowing 10 GB at once. Editing the cloned files would be a more gradual way of their growing in size. Only changed data would then need to be saved, so free disk space would steadily fall as more changes were made to the clone.

Clone and sparse files are by no means unique to APFS, but they can be impressive, and above all they’re effective at reducing excess erase-write cycles that age SSDs, whatever you do with the storage they free.

I’m very grateful to Duncan for drawing my attention to Hyperspace, and to John Siracusa for an outstanding app.

Most modern Macs start up from their internal SSD, and have a single bootable system installed. Although its structure has changed considerably over recent years, and differs between Intel and Apple silicon Macs, they are generally reliable, and knowledge of their structure and function is seldom required.

Those who need to start their Mac up from two or more versions of macOS, or who do so from an external disk, may be able to get by without deeper understanding, but the moment there’s a problem can become confused, and end up having to install macOS from scratch.

Intel T2 and Apple silicon

Both types of Mac are designed to support Secure Boot, but because of the reliance of Intel Macs on UEFI firmware, they can only boot securely from their internal SSD. Enabling an Intel T2 Mac to start up from an external disk thus requires its boot security to be reduced by changing that in Startup Security Utility, in Recovery mode. Booting from an external system is then straightforward, and doesn’t require additional security measures as used in Apple silicon.

Apple silicon Macs have been designed from the outset to support Secure Boot when starting up from both internal and external disks, and don’t require any reduction in their boot security to be able to start up from multiple systems on external SSDs. It’s a common misunderstanding that trying to change Boot Security in Startup Security Utility can help solve Apple silicon boot problems, but if anything it only complicates them. Almost the only good reason for reducing boot security of an Apple silicon bootable system is when third-party kernel extensions are required. Otherwise don’t tamper with Startup Security Utility, as it will only confuse, as we’ll see later.

For an Apple silicon Mac to boot from a macOS system, that must have a LocalPolicy created and saved to the internal SSD. LocalPolicy is normally created during macOS installation, or when intending to start up from a suitably installed system. Problems with LocalPolicy have been common when using external disks, and are covered in these articles:

• Checking bootable systems using bputil on Apple silicon
• Booting macOS on Apple silicon: LocalPolicy
• Booting macOS on Apple silicon: Multiple boot disks
• Apple’s Platform Security Guide here and here.

Boot disk architecture

Another significant difference between Intel and Apple silicon Macs is the architecture of their internal SSDs: Apple silicon has two additional partitions (APFS containers), and lacks the traditional EFI partition. However, there are no such differences in the structure of their bootable external disks, and its relevance here is limited, and mainly affects Big Sur as I explain below.

macOS 10.13-10.15

Bootable disks in versions of macOS between High Sierra and Mojave (above) are based on their single boot volume, supplemented by hidden Preboot, Recovery and VM volumes. macOS 10.15 Catalina (below) divided that boot volume into a Boot Volume Group, consisting of paired System and Data volumes, in addition to the same three hidden volumes.

Basic support for volume roles was introduced in the first release version of APFS in macOS 10.13, and was extended to cover further roles in 10.15. Thus versions of APFS prior to 1412.x.x don’t understand volume roles used by subsequent systems. From Catalina onwards you can use the command
diskutil apfs listVolumeGroups
to see paired System and Data volumes, for example
+-- Container disk5 5BA1AAC8-3AD4-4594-AF01-7C0AA75CABAD
| |
| +-> Volume Group 68627CBB-D774-444E-97C6-F9511B5030F3
| =================================================
| APFS Volume Disk (Role): disk5s1 (Data)
| Name: Macintosh HD - Data
| Volume UUID: 68627CBB-D774-444E-97C6-F9511B5030F3
| Capacity Consumed: 580769214464 B (580.8 GB)
| -------------------------------------------------
| APFS Volume Disk (Role): disk5s5 (System)
| Name: Macintosh HD
| Volume UUID: 8B9FF440-75C8-4F7F-B09E-9222D44A2276
| Capacity Consumed: 11239186432 B (11.2 GB)

Note that each volume group has its own UUID, which is normally the same as that of the Data volume in the pair. Identification of volumes, containers and other structures in APFS is dependent on these UUIDs, which are an essential part of the GUID Partition Table scheme used to partition disks.

Catalina’s System volume is mounted read-only, but macOS is booted from that volume rather than the Signed System Volume snapshot introduced in Big Sur.

macOS 11

Although the structure of Big Sur internal SSDs in Apple silicon Macs has two additional partitions (APFS containers), Apple_APFS_ISC containing three volumes, and Apple_APFS_Recovery containing two volumes, bootable external disks have the same structure as the internal SSD in Intel Macs.

One important functional difference, which remains relevant to Big Sur boot disks, is that Apple silicon Macs don’t use the paired Recovery volume as their primary Recovery system: booting an Apple silicon Mac running Big Sur into Recovery should instead use the Recovery system installed in their internal SSD, in the Apple_APFS_Recovery partition. In subsequent versions of macOS, that’s used instead for secondary or Fallback Recovery. Thus Big Sur can be a problem when it comes to Recovery, and for this reason is best avoided on Apple silicon Macs. If it’s essential to install a copy of Big Sur, then be prepared for problems with Recovery mode.

macOS 11 also established the architecture for dual-boot partitions, with two or more Boot Volume Groups.

Although the Boot Volume Group has only ever referred to paired System and Data volumes, within each partition/container the group also requires three or four additional volumes, for Preboot, Recovery, VM and (apparently in Big Sur only) Update. How those work with multiple Boot Volume Groups is explained below. Once way to avoid the inevitable complexities is to install each Boot Volume Group into a separate partition/container, which provides each with its own suite of Preboot, Recovery and VM volumes.

As the Signed System Volume (SSV) was introduced in macOS 11, versions of APFS prior to 1677.x.x shouldn’t be expected to understand SSVs.

macOS 12-15

The next significant architectural changes in Apple silicon Macs were the introduction of paired Recovery volumes in macOS 12 Monterey, and of cryptexes in macOS 13 Ventura.

In macOS 11 and 12, Safari, its supporting components, and some other parts of macOS are installed to the Data volume for ease of maintenance. Apple replaced that with separate secure disk images termed cryptexes, loaded from the Preboot volume during the boot process and grafted into the root file system. As a single Preboot volume can be shared by more than one Boot Volume Group, bundled cryptexes must be provided to the correct group, and I suspect that’s accomplished by associating them with their Volume Group UUID. If that becomes confused in any way, cryptexes could be incorrectly associated or missing altogether, something that’s almost certainly fatal to the boot process.

Above is the current layout of a partition/container containing a single bootable system for a Mac running Sequoia. Device names given are illustrative, although their numbering is typical of that seen for external bootable volumes. In this instance, the base name for the Boot Volume Group is External, and the External System volume is unmounted as usual.

Below is a typical partition/container containing two bootable systems named ExternalA and ExternalB. This has two Boot Volume Groups, each consisting of a System volume with its SSV, and a Data volume, to which are added their three common volumes, Preboot, Recovery and VM.

Management

Creation

The most reliable tools for creating and working with Boot Volume Groups and other system components are those in macOS installer apps, and even they can have their moments and bugs. It’s usually possible to ‘clone’ groups using the asr command tool, a feature that’s offered in some third-party utilities including Carbon Copy Cloner and SuperDuper. However, Apple has made it clear that given a choice between supporting asr and addressing boot security, it gives the latter absolute priority. asr has suffered some serious bugs since the SSV was introduced in Big Sur, and shouldn’t be relied on.

asr is careful to ensure that, when cloning Boot Volume Groups, UUIDs are changed so that it doesn’t end up with volumes or groups with the same UUID. That may not be the case when using some other tools such as dd for duplication. If you prefer a simple and stress-free life, it’s better to put your trust in a macOS installer rather than try crafting your own Boot Volume Groups.

Apple’s own tools such as Disk Utility now try to steer the user away from making mistakes, such as deleting just one of a Boot Volume Group, leaving the other orphaned with its firmlinks broken.

Firmware

Firmware is another source of confusion. That installed in the Mac, using its internal storage, should always reflect the firmware that has been included in the most recent version of macOS installed or updated on that Mac, whether that was installed to the internal SSD or to an external disk. The only exception to this is when installing or updating macOS in a Virtual Machine, which can’t affect the host’s firmware.

What may appear more puzzling are the versions reported in System Information: that given as System Firmware should be the iBoot version for the main Mac, and the most recent. The OS Loader, though, varies with the Boot Volume Group, and in older macOS is likely to be earlier than the iBoot version of the main Mac.

Recovery

Recovery system versions are even more complex. When everything works as it should, the version installed in any paired Recovery volume should be the newest of the Boot Volume Groups that it’s paired with, in the same partition/container. If a Mac running 15.3 from its internal SSD has a bootable external disk with a single container with two Boot Volume Groups for 13.7 and 14.7, then the Recovery system for the internal SSD should be 15.3, while that shared by the two systems on the external disk should be 14.7. It’s likely that Fallback Recovery on the internal SSD will be from an earlier version of macOS 15, but that’s less predictable, and there’s no separate Fallback Recovery in external systems.

Switching between systems and their Boot Volume Groups is normally performed using Startup Disk in General settings, or its equivalent in System Preferences. Those record the chosen Boot Volume Group in NVRAM, from where it’s used during the next boot.

That setting is used to determine which Recovery system is used if the next boot is performed with the Power button pressed to enter Recovery mode. That in turn determines what can be performed in Recovery. This is most critical for determining how Startup Security Utility works, as that can only change boot security settings for the chosen Boot Volume Group saved in NVRAM.

For example, in the dual-boot container shown above, selecting ExternalB as the Startup Disk, shutting down, then starting up into Recovery mode should enable Startup Security Utility to control boot security for ExternalB but not ExternalA, nor the system installed on the internal SSD. This can be used to discover which Recovery system is running: open Startup Security Utility and the only boot system that can be changed there is the current default boot system.

APFS

Although APFS should be backward compatible, making it relatively safe to make changes to an older version of APFS from a newer system, forward compatibility is more limited. Using older versions of Disk Utility or tools like fsck on newer versions of APFS risks errors, failure and at worst damage. The Appendix at the end of this article summarises version numbering in APFS and major changes to beware of. Although not easy to discover the version of APFS used to create or modify any given volume, one way is to use
fsck_apfs -n -S [device]
giving the volume’s device name. The report should then be prefaced by a statement such as
The volume [volume name] was formatted by diskmanagementd (1412.141.1) and last modified by apfs_kext (945.250.134).
telling you that volume was created by macOS 10.15, and was last changed by macOS 10.14.

Troubleshooting

The first and fundamental step in trying to diagnose the cause of problems with multiple Boot Volume Groups or bootable external disks is to examine their container and volume structure using two diskutil commands:
diskutil apfs list
lists all APFS volumes by container and gives key information about each, including role and UUID, and
diskutil apfs listVolumeGroups
lists all recognised Boot Volume Groups, which you can tally against the first. Pipe these to text files so you can study and refer back to them.

Those should enable you to verify that the structure is as intended, and to establish the relationships between systems and paired Recovery volumes. From there you should be able to focus on the Boot Volume Group that’s misbehaving, ensuring that its Data volume has been backed up. If necessary, that can be used as a migration source if that group needs to be deleted and reinstalled.

Tips & tricks

• The host Mac must be capable of running that version of macOS.
• macOS installers are the most reliable means of creating and installing Boot Volume Groups.
• During macOS installation, an external disk must be connected to a port other than the DFU port, as listed here and in Mactracker.
• When installing an older major version of macOS, perform this from an external bootable HFS+ volume as detailed by Apple.
• Use an HFS+J partition on an external SSD rather than a USB ‘thumb’ drive.
• Boot from an installer volume through Recovery mode.
• When using a laptop Mac, run it from mains power throughout macOS installation.
• Apple silicon Macs boot from external disks in Full Security, and reducing that doesn’t solve any problems.
• Each container with one or more Boot Volume Groups contains one set of Preboot, Recovery and VM volumes shared between them.
• If you’re unsure which Recovery system you’re using, open Startup Security Utility, as the only group it can change settings for is the one it’s booted into.
• Big Sur doesn’t support paired Recovery on Apple silicon, and that can cause problems.
• The version of Recovery installed in any paired Recovery volume should be that of the newest of the Boot Volume Groups that it’s paired with.
• Use fsck_apfs -n -S [device] to discover the APFS version.
• Use diskutil apfs list and diskutil apfs listVolumeGroups to discover volume structure.
• Try using a Virtual Machine instead, if you don’t need to be able to run software from the App Store.

Appendix: APFS version numbers

APFS major version numbers change with major version of macOS:

• macOS 10.12 has APFS version 0.3 or 249.x.x, which shouldn’t be used at all.
• 10.13 has 748.x.x, which doesn’t support Fusion Drives, but has basic support for volume roles.
• 10.14 has 945.x.x, the first version to support Fusion Drives.
• 10.15 has 1412.x.x, the first version to support the multi-volume boot group, and introduces extended support for volume roles, including Data, Backup and Prelogin.
• 11 has 1677.x.x, the first version to support the SSV, and Apple silicon. On M1 Macs, it doesn’t support the paired Recovery volume.
• 12 has 1933.x.x until 12.2, thereafter 1934.x.x, which support the paired Recovery volume on Apple silicon.
• 13 has 2142.x.x, and is probably the first to support trimming of UDRW disk images and their storage as sparse files.
• 14 has 2235.x.x, until 14.3, thereafter 2236.x.x.
• 15 has 2313.x.x until 15.2, thereafter 2317.x.x until 15.4, thereafter 2332.x.x.

Sometimes Macs behave in ways that you don’t forget. It was nearly six years ago, in 2019 when my iMac Pro was still running macOS Mojave, that it took over half an hour to start up in Safe mode. As became obvious, that was because it checked the integrity of every snapshot, including its long series of Time Machine backups. That behaviour stopped with the release of Catalina later that year, and revisiting the experience last week was interesting, when I examined what Safe mode does now in Sequoia.

Safe mode fsck

Checking disk integrity has long been a feature claimed of Safe mode, although Apple has only recently clarified that its checks are “similar to the more comprehensive check performed by the First Aid feature of Disk Utility,” unlike those in the days of Mojave.

Because these are run when starting up into Safe mode, macOS can’t report them directly to the user. There are only two places for their results to be recorded: in the fsck_apfs log at /var/log/fsck_apfs.log (and its error log at /var/log/fsck_apfs_error.log), and in the Unified log, neither of which are likely to be visited by users. It doesn’t seem possible that checks could be run before entries are made in the Unified log, and if they were their results would be inaccessible, and of no value to the user.

If you do look there, you might be surprised to discover that the checks run by fsck_apfs are ‘quick checks’, described as a “check whether the device is `clean’. If device is an APFS volume, fsck_apfs will quickly check the APFS container and the specified APFS volume. If device is an APFS container, fsck_apfs will quickly check the APFS container and all the APFS volumes in it. By default, no repairs are attempted during a quick check.” Thankfully they also no longer include Time Machine snapshots or backups.

For users, the most important of volumes to be checked is the Data volume in the current boot volume group. Yet careful examination of both log records reveals that such checks fail, returning error 65, because the device is mounted with write access. Although they could be performed using fsck_apfs‘s live verification, no attempt is made to do that. From the active boot volume group, only the Recovery and Update volumes are checked, together with all accessible external volumes.

Normal user fsck

Before you think that’s better than making no checks at all, the sequence of fsck_apfs checks run at the start of Safe mode appears to be identical to those made during startup into normal user mode, and reports the same failures. Thus, in this respect Safe mode is no different from normal, despite Apple listing this as a feature of Safe mode.

It’s worth reading the fsck_apfs man page to see the many options available in this command, which is the basis for all checks and repairs made to APFS volumes and containers. There are two in particular that you may prefer to invoke rather than using Disk Utility’s single option to both check and repair. My favourites that make it worth opening Terminal instead of Disk Utility are -n to prevent it from attempting to make any repairs, and -S to skip snapshots.

Given that fsck_apfs appears unable to repair any errors it finds in snapshots and admits to that in its man page where it states that “no repairs can be made”, it puzzles me that without the -S option it will still check all snapshots it comes across when running its checks. Perhaps it’s time to alter that default behaviour and make checking snapshots the explicit option.

First Aid

Disk Utility’s First Aid is descended from a succession of utilities starting with the Disk First Aid app in Classic Mac OS, which separated disk Verify and Repair features.

Mac OS X merged the two Classic apps Drive Setup and Disk First Aid, but kept them as distinct tools within its new Disk Utility app and retained separate Verify and Repair features.

Those two features were still present in Mountain Lion or Mavericks, shown here, but soon afterwards only Repair was available.

Now that APFS is mature and has proven itself more than a worthy successor to HFS+, the need to both check and repair its containers and volumes should be abating. This would be a good time to return to separate controls, allowing users to check APFS in the expectation that there are no errors to be found. This is more important now because of the absence of any alternatives.

Monopoly

Disk Utility is in a unique position. Apple has chosen to lock third-parties out of developing competing disk repair utilities, in particular those that can be used in Recovery mode. Users can’t opt for apps with more or better features, putting the onus on Apple to provide a comprehensive solution that meets users’ wishes, not its idea of essential requirements.

Conclusions

• Apple needs to be explicit about what disk checks are run in Safe mode, and where users can inspect their results.
• fsck_apfs is flexible and powerful, but has strange defaults that merit reconsideration.
• Disk Utility should offer more options beyond its single ‘check and repair’, in particular to check only and with snapshots excluded. Those are most important in Recovery mode, where there can be no alternative.

If you want a quiet life, just format each external disk in APFS (with or without encryption), and cruise along with plenty of free space on it. For those who need to do different, either getting best performance from a hard disk, or coping with less free space on an SSD, here are some tips that might help.

File system basics

A file system like APFS provides a logical structure for each disk. At the top level it’s divided into one or more partitions that in APFS also serve as containers for its volumes. Partitions are of fixed size, although you can always repartition a disk, a process that macOS will try to perform without losing any of the data in its existing partitions. That isn’t always possible, though: if your 1 TB disk already contains 750 GB, then repartitioning it into two containers of 500 GB each will inevitably lose at least 250 GB of existing data.

All APFS volumes within any given container share the same disk space, and by default each can expand to fill that. However, volumes can also have size limits imposed on them when they’re created. Those can reserve a minimum size for that volume, or limit it to a maximum quota size.

How that logical structure is implemented in terms of physical disk space depends on the storage medium used.

Faster hard disks

Hard disks store data in circular tracks of magnetic material. To store each file requires multiple sectors of those tracks, each of which can contain 512 or 4096 bytes. As the length (circumference) of the tracks is greater as you move away from the centre of the platter towards its edge, but the disk spins at a constant number of revolutions per minute (its angular velocity is constant), it takes a shorter time for sectors at the periphery of the disk to pass under the heads than for those closer to the centre of the platter. The result is that read and write performance also varies according to where files are stored on the disk: they’re faster the further they are from the centre.

This graph shows how read and write speeds change in a typical compact external 2 TB hard disk as data is stored towards the centre of the disk. At the left, the outer third of the disk delivers in excess of 130 MB/s, while the inner third at the right delivers less than 100 MB/s.

You can use this to your advantage. Although you don’t control exactly where file data is stored on a hard disk, you can influence that. Disks normally fill with data from the periphery inwards, so files written first to an otherwise empty disk will normally be written and read faster.

You can help that on a more permanent basis by dividing the disk into two or more partitions (APFS containers), as the first will normally be allocated space on the disk nearest the periphery, so read and write faster than later partitions added nearer the centre. Adding a second container or partition of 20% of the total capacity of the disk won’t cost you much space, but it will ensure that performance doesn’t drop off to little more than half that achieved in the most peripheral 10%.

Reserving free space on SSDs

The firmware in SSDs knows nothing of its logical structure, and for key features manages the whole storage as a single unit. Wear levelling ensures that individual blocks of memory have similar numbers of erase-write cycles, so they age evenly. Consumer SSDs that use dynamic SLC write caches allocate those from across the whole storage, and aren’t confined to partitions. You can thus manage free space to keep sufficient dynamic cache available at a disk level.

One approach is to partition the SSD to reserve a whole container, with its fixed size, to support the needs of the dynamic cache. An alternative is to use volume reserve and quota sizes for the same purpose, within a single container. For example, in a 1 TB SSD with a 100 GB SLC write cache you could either:

• with a single volume, set its quota to 900 GB, or
• add an empty volume with its reserve size set to 100 GB.

Which of these you choose comes down to personal preference, although on boot volume groups you won’t be able to set a quota for its Data volume, and the most practical solution for a boot disk is to add an empty volume with a specified reserve size.

To do this when creating a new volume, click on the Size Options… button and set the quota or reserve.

Summary

• Partition hard disks so that you only use the fastest 80% or so of the disk.
• To reserve space in an SSD for dynamic caching, you can add a second APFS container.
• A simpler and more flexible way to reserve space on SSDs is setting a quota size for a single volume, or adding an empty volume with a reserve size.
• Size options can currently only be set when creating a volume.

It must be almost two years since I last demonstrated some magic tricks involving available and purgeable disk space. At that time, the amount of space involved was a mere 83.71 GB. Today I’m going to show you how I converted 228.16 GB of purgeable space into used space, recovering a lot of my files in the process.

Prior to my iMac Pro’s forced update to Sequoia 15.3.1, described here yesterday, its internal SSD had around 150-160 GB free, with no purgeable space at all. Immediately before installing that update, SoftwareUpdate reported that there was 160.57 GB available. When I had coaxed it back into life, now running 15.3.1, the foot of each Finder window told me there was now “393.72 GB available”. Imagine my surprise/shock/horror that about 240 GB of what had been on that SSD before it was updated had now vanished.

Recalling my previous experience, I selected Macintosh HD in the Finder, and opened the Get Info dialog. That confirmed the situation, stating

• Available 393.72 GB (228.16 GB purgeable)
• Used: 828,672,419,328 bytes (829.67 GB on disk)

A little arithmetic reveals that of the 393.72 GB “available”, only 165.56 GB was actually free at the time, the rest being “purgeable”. Together the truly free and that used “on disk” amounted to 995.23 GB. Adding the 16.16 GB used by other volumes, my Mac’s internal SSD had grown in capacity to 1.011 TB, which made that slightly traumatic update worthwhile after all.

Sadly, Disk Utility wasn’t so impressed. The figures it gave were very different indeed:

• Available: 165.56 GB (none purgeable)
• Used: 818.52 GB + 16.16 GB on other volumes = 834.68 GB
• One snapshot of 7.16 GB

for a total disk size of exactly 1 TB. The figures my own Mints gave were in accord with those from Disk Utility.

Although I much preferred the Finder’s figure of nearly 400 GB of “available” space, I realised that could only come at the cost of purging all that 228 GB of “purgeable” space. As that seemed to include many of my files, I thought it was time to work this week’s magic trick. I therefore restarted the Mac, and all of a sudden purgeable space had vanished, leaving me with only about 165 GB of free space after all.

To remind you of what I found nearly two years ago, after updating to macOS 13.3.1, the Finder found 83.71 GB “purgeable”, and my SSD had then grown to 1.08 TB in size.

That’s two major versions of macOS and almost two years apart, and the Finder still can’t come up with correct figures.