Time Machine (TM) has evolved to be a good general-purpose backup utility that makes best use of APFS backup storage. However, it does have some quirks, and offers limited controls, that can make it tricky to use with more complex setups. Over the last few weeks I’ve had several questions from those trying to use TM in more demanding circumstances. This article explains how you can design volume layout and backup exclusions for the most efficient backups in such cases.

How TM backs up

To decide how to solve these problems, it’s essential to understand how TM makes an automatic backup. In other articles here I have provided full details, so here I’ll outline the major steps and how they link to efficiency.

At the start of each automatic backup, TM checks to see if it’s rotating backups across more than one backup store. This is an unusual but potentially invaluable feature that can be used when you make backups in multiple locations, or want added redundancy with two or more backup stores.

Having selected the backup destination, it removes any local snapshots from the volumes to be backed up that were made more than 24 hours ago. It then creates a fresh snapshot on each of those volumes. I’ll consider these later.

Current versions of TM normally don’t use those local snapshots to work out what needs to be backed up from each volume, but (after the initial full backup) should rely on that volume’s record of changes to its file system, FSEvents. These observe two lists of exclusions: those fixed by TM and macOS, including the hidden version database on each volume and recognised temporary files, and those set by the user in TM settings. Among the latter should be any very large bundles and folders containing huge numbers of small files, such as the Xcode app, as they will back up exceedingly slowly even to fast local backup storage, and can tie up a network backup for many hours. It’s faster to reinstall Xcode rather than restore it from a backup.

Current TM backups are highly efficient, as TM can copy just the blocks that have changed; older versions of TM backing up to HFS+ could only copy whole files. However, that can be impaired by apps that rewrite the whole of each large file when saving. Because the backup is being made to APFS, TM ensures that any sparse files are preserved, and handles clone files as efficiently as possible.

Once the backup has been written, TM then maintains old backups, to retain:

• hourly backups for the last 24 hours, to accompany hourly local snapshots,
• daily backups over the previous month,
• weekly backups stretching back to the start of the current backup series.

These are summarised in the diagram below.

Local snapshots

TM makes two types of snapshot: on each volume it’s set to back up, it makes a local snapshot immediately before each backup, then deletes that after 24 hours; on the backup storage, it turns each backup into a snapshot from which you can restore backed up files, and those are retained as stated above.

APFS snapshots, including TM local snapshots, include the whole of a volume, without any exceptions or exclusions, which can have surprising effects. For example, a TM exclusion list might block backing up of large virtual machine files resulting in typical backups only requiring 1-2 GB of backup storage, but because those VMs change a lot, each local snapshot could require 25 GB or more of space on the volume being backed up. One way to assess this is to check through each volume’s TM exclusion list and assess whether items being excluded are likely to change much. If they are, then they should be moved to a separate volume that isn’t backed up by TM, thus won’t have hourly snapshots.

Some workflows and apps generate very large working files that you may not want to clutter up either TM backups or local snapshots. Many apps designed to work with such large files provide options to relocate the folders used to store static libraries and working files. If necessary, create a new volume that’s excluded completely from TM backups to ensure those libraries and working files aren’t included in snapshots or backups.

TM can’t run multiple backup configurations with different sets of exclusions, though. If you need to do that, for instance to make a single nightly backup of working files, then do so using a third-party utility in addition to your hourly TM backups.

This can make a huge difference to free space on volumes being backed up, as the size of each snapshot can be multiplied by 24 as TM will try to retain each hourly snapshot for the last 24 hours.

Macs that aren’t able to make backups every hour can also accrue large snapshots, as they may retain older snapshots, that will only grow larger over time as that volume changes from the time that snapshot was made.

While snapshots are a useful feature of TM, the user has no control over them, and can’t shorten their period of retention or turn them off altogether. Third-party backup utilities like Carbon Copy Cloner can, and may be more suitable when local snapshots can’t be managed more efficiently.

iCloud Drive

Like all backup utilities, TM can only back up files that are in iCloud Drive when they’re downloaded to local storage. Although some third-party utilities can work through your iCloud Drive files downloading them automatically as needed, TM can’t do that, and will only back up files that are downloaded at the time that it makes a backup.

There are two ways to ensure files stored in iCloud Drive will be backed up: either turn Optimise Mac Storage off (in Sonoma and later), or download the files you want backed up and ‘pin’ them to ensure they can’t be removed from local storage (in Sequoia). You can pin individual files or whole folders and their entire contents by selecting the item, Control-click for the contextual menu, and selecting the Keep Downloaded menu command.

Key points

• Rotate through 2 or more backup stores to handle different locations, or for redundancy.
• Back up APFS volumes to APFS backup storage.
• Exclude all non-essential files, and bundles containing large numbers of small files, such as Xcode.
• Watch for apps that make whole-file changes, thus increasing snapshot and backup size.
• Store large files on volumes not being backed up to minimise local snapshot size.
• If you need multiple backup settings, use a third-party utility in addition to TM.
• To ensure iCloud Drive files are backed up, either turn off Optimise Mac Storage (Sonoma and later), or pin essential files (Sequoia).

Further reading

Time Machine in Sonoma: strengths and weaknesses
Time Machine in Sonoma: how to work around its weaknesses
Understand and check Time Machine backups to APFS
Excluding folders and files from Time Machine, Spotlight, and iCloud Drive

APFS has many of the features of modern file systems that make most efficient use of space, and others that aren’t found in more traditional file systems like HFS+. While those should all work well when used locally with other APFS volumes, they can prove incompatible with other file systems, and may have untoward side effects. This article looks at those features that you could encounter problems with, and how best to work around them.

Sparse files

Most modern file systems store files that contain significant amounts of blank or absent data in a special sparse file format. In APFS, this is implemented by only allocating file extents to those blocks that do contain data. In many cases, this can save significant disk space. Initially, sparse files were unusual in APFS, but over time they have become increasingly common, and can now be found in some types of disk image, virtual machine storage, and in some databases. Note that disk image types whose name includes the word sparse, sparse bundles and sparse disk images, don’t use sparse file format at all, but are the victims of an unfortunate name collision.

Neither macOS nor APFS can simply convert regular files to sparse format; for a file to be written in sparse format, the code writing it must explicitly skip the empty space. As a result, sparse files are prone to explode to full size when they’re copied or moved unless that’s between two local volumes both using APFS. Examples of where you should expect them to explode include:

• copy or move to HFS+, as that has no sparse file format
• copy between Macs using AirDrop or file sharing
• back up to network storage, although local Time Machine backups to APFS should preserve them
• copy or move to any other local file system other than APFS, even if that file system has its own support for sparse files.

In each of those cases, sparse files explode in size as they’re copied from the source volume. If you have a 100 GB sparse file that only takes 20 GB of local disk space, when it’s copied over the network, or to a local HFS+ volume, the full 100 GB has to be transferred.

One potential workaround is to compress the sparse file before transferring it. All good compression algorithms will work efficiently on the blank space in the file, so when compressed its size could be as small as the original sparse file. However, when it’s decompressed, even on another APFS volume, it will explode to full size. For disk images, that can be corrected by mounting them, as APFS will then trim their contents, and the disk image should be saved back into sparse format.

Clone files

These are two distinct files that share common data, normally the result of duplicating the original within the same APFS volume. Those two cloned files then only require the storage of the whole file, plus those data blocks that differ between them. This only works within the same volume, and the moment that either of the clones is moved or copied to any other volume, it assumes full size, as it can no longer share data with the other clone.

However, most other file systems don’t support file cloning in this way. When you duplicate a file in an HFS+ volume, there’s no shared data between them, and the two require twice the amount of space as one does.

Snapshots

Snapshots consist of a complete copy of the volume at an instant in time, so require a copy of the file system metadata for that volume, and retain copies of storage blocks for each file as they are changed subsequent to the snapshot being made, so you could roll back that volume to its state at the moment of the snapshot.

Although Time Machine backups contain snapshots of the volume they back up, snapshots can’t normally be copied to another disk or volume. Some have been able to make a complete copy of a disk including its snapshots using the dd command tool, but that should be considered experimental. In all other circumstances, snapshots stay where they were made, but you can always copy from an existing snapshot to reconstitute a volume.

Directory hard links

These aren’t available in APFS, but are supported in HFS+, where they’re used extensively in Time Machine backups. They work like regular hard links, but act on directories rather than individual files. They can’t be copied in any way to an APFS volume, but can be used to reconstitute a volume.

Extended attributes

These are additional metadata associated with files and folders, and are fully supported in both APFS and HFS+. However, many are treated as being ephemeral, and may not be preserved during copying or other actions. The system of flags used to determine which are preserved is detailed in this article.

Several other file systems also support extended attributes, but copying between them is unlikely to transfer them between file systems. In some cases, extended attributes are preserved using AppleDouble format, in which each file can have a hidden shadow with its name prefixed with ._ (dot-underscore) characters. These are most often seen in FAT and ExFAT volumes, but are prone to confuse users of other computers.

Key points

• APFS sparse files are only preserved when copying or moving files between local APFS volumes. In other circumstances they explode to full size.
• Good compression methods can keep a sparse file to a similar size, but decompression explodes them to full size. Disk images may then be restored as sparse files after they have been mounted again from APFS.
• Clone files aren’t preserved when copied or moved to any other volume.
• Snapshots can’t be copied at all, although they can be reconstituted as volumes.
• APFS doesn’t support the directory hard links used in Time Machine backups to HFS+, but a backup can be reconstituted as a volume.
• Extended attributes are preserved between APFS and HFS+, but not with other file systems, except in the shadow files of AppleDouble format, seen in FAT and ExFAT volumes.