Time Machine (TM) has evolved to be a good general-purpose backup utility that makes best use of APFS backup storage. However, it does have some quirks, and offers limited controls, that can make it tricky to use with more complex setups. Over the last few weeks I’ve had several questions from those trying to use TM in more demanding circumstances. This article explains how you can design volume layout and backup exclusions for the most efficient backups in such cases.

How TM backs up

To decide how to solve these problems, it’s essential to understand how TM makes an automatic backup. In other articles here I have provided full details, so here I’ll outline the major steps and how they link to efficiency.

At the start of each automatic backup, TM checks to see if it’s rotating backups across more than one backup store. This is an unusual but potentially invaluable feature that can be used when you make backups in multiple locations, or want added redundancy with two or more backup stores.

Having selected the backup destination, it removes any local snapshots from the volumes to be backed up that were made more than 24 hours ago. It then creates a fresh snapshot on each of those volumes. I’ll consider these later.

Current versions of TM normally don’t use those local snapshots to work out what needs to be backed up from each volume, but (after the initial full backup) should rely on that volume’s record of changes to its file system, FSEvents. These observe two lists of exclusions: those fixed by TM and macOS, including the hidden version database on each volume and recognised temporary files, and those set by the user in TM settings. Among the latter should be any very large bundles and folders containing huge numbers of small files, such as the Xcode app, as they will back up exceedingly slowly even to fast local backup storage, and can tie up a network backup for many hours. It’s faster to reinstall Xcode rather than restore it from a backup.

Current TM backups are highly efficient, as TM can copy just the blocks that have changed; older versions of TM backing up to HFS+ could only copy whole files. However, that can be impaired by apps that rewrite the whole of each large file when saving. Because the backup is being made to APFS, TM ensures that any sparse files are preserved, and handles clone files as efficiently as possible.

Once the backup has been written, TM then maintains old backups, to retain:

• hourly backups for the last 24 hours, to accompany hourly local snapshots,
• daily backups over the previous month,
• weekly backups stretching back to the start of the current backup series.

These are summarised in the diagram below.

Local snapshots

TM makes two types of snapshot: on each volume it’s set to back up, it makes a local snapshot immediately before each backup, then deletes that after 24 hours; on the backup storage, it turns each backup into a snapshot from which you can restore backed up files, and those are retained as stated above.

APFS snapshots, including TM local snapshots, include the whole of a volume, without any exceptions or exclusions, which can have surprising effects. For example, a TM exclusion list might block backing up of large virtual machine files resulting in typical backups only requiring 1-2 GB of backup storage, but because those VMs change a lot, each local snapshot could require 25 GB or more of space on the volume being backed up. One way to assess this is to check through each volume’s TM exclusion list and assess whether items being excluded are likely to change much. If they are, then they should be moved to a separate volume that isn’t backed up by TM, thus won’t have hourly snapshots.

Some workflows and apps generate very large working files that you may not want to clutter up either TM backups or local snapshots. Many apps designed to work with such large files provide options to relocate the folders used to store static libraries and working files. If necessary, create a new volume that’s excluded completely from TM backups to ensure those libraries and working files aren’t included in snapshots or backups.

TM can’t run multiple backup configurations with different sets of exclusions, though. If you need to do that, for instance to make a single nightly backup of working files, then do so using a third-party utility in addition to your hourly TM backups.

This can make a huge difference to free space on volumes being backed up, as the size of each snapshot can be multiplied by 24 as TM will try to retain each hourly snapshot for the last 24 hours.

Macs that aren’t able to make backups every hour can also accrue large snapshots, as they may retain older snapshots, that will only grow larger over time as that volume changes from the time that snapshot was made.

While snapshots are a useful feature of TM, the user has no control over them, and can’t shorten their period of retention or turn them off altogether. Third-party backup utilities like Carbon Copy Cloner can, and may be more suitable when local snapshots can’t be managed more efficiently.

iCloud Drive

Like all backup utilities, TM can only back up files that are in iCloud Drive when they’re downloaded to local storage. Although some third-party utilities can work through your iCloud Drive files downloading them automatically as needed, TM can’t do that, and will only back up files that are downloaded at the time that it makes a backup.

There are two ways to ensure files stored in iCloud Drive will be backed up: either turn Optimise Mac Storage off (in Sonoma and later), or download the files you want backed up and ‘pin’ them to ensure they can’t be removed from local storage (in Sequoia). You can pin individual files or whole folders and their entire contents by selecting the item, Control-click for the contextual menu, and selecting the Keep Downloaded menu command.

Key points

• Rotate through 2 or more backup stores to handle different locations, or for redundancy.
• Back up APFS volumes to APFS backup storage.
• Exclude all non-essential files, and bundles containing large numbers of small files, such as Xcode.
• Watch for apps that make whole-file changes, thus increasing snapshot and backup size.
• Store large files on volumes not being backed up to minimise local snapshot size.
• If you need multiple backup settings, use a third-party utility in addition to TM.
• To ensure iCloud Drive files are backed up, either turn off Optimise Mac Storage (Sonoma and later), or pin essential files (Sequoia).

Further reading

Time Machine in Sonoma: strengths and weaknesses
Time Machine in Sonoma: how to work around its weaknesses
Understand and check Time Machine backups to APFS
Excluding folders and files from Time Machine, Spotlight, and iCloud Drive

Over the last forty years, one of the essential virtues of the Mac’s interface has been its consistency. From the outset, Apple has ensured the Finder behaves according to a small set of rules that are readily learned, often without conscious effort. These form a grammar grounded on those of languages like English, of subject-verb-object. Select a file, do something with it, and it changes state.

Drag a file from its current folder and drop it onto another, and it’s moved or copied from one to the other. The outcome depends on context, where performing that action within the same storage volume results in the file being moved, but between folders in different volumes it’s copied instead. Folders play two roles: as enclosing elements, and as objects in their own right. Drag a folder from one place to another, and you expect its entire contents to go with it, but apply a tag to a folder, and only that folder gains the tag.

iCloud Drive has never quite conformed to the same grammar. Change its mode by enabling the Optimise Mac Storage setting and it stops behaving so consistently. Files that appear to be present turn out to lack data content, and have to be marked as evicted exceptions. Folder commands to control download and eviction depart further from established norms. Select all, and if there are more than ten items selected, Remove Downloads is no longer available in the contextual menu. Can the Finder no longer cope with certain actions when they’re applied simultaneously to more items than we have fingers?

There is a workaround, though: select just the enclosing folder, and Remove Downloads on that. If you then want to download individual items within that evicted folder, you do so by selecting them and using Download Now, which oddly doesn’t seem constrained to the same limit of ten. You can thus have a hybrid folder, with some items downloaded, others evicted, and the folder itself continues to display the icon indicating its contents remain evicted.

Now consider Sequoia’s new pinning behaviour. Pin a folder and its contents are shown as being pinned, as you’d expect. Select one or two of those individual files, though, and there’s no option offered to unpin them. The only way to do that is to unpin the whole folder, then pin individual items within that folder. But like the Remove Downloads command, that too won’t work for more than ten items at once. Move a new item inside a pinned folder, and it automatically becomes pinned, and can’t be individually unpinned as long as its enclosing folder remains pinned.

If you have a folder of 100 files, and want to pin all of them bar one, this becomes laborious. If you pin the folder, you can’t unpin that one file, so you must leave the folder unpinned and instead pin the individual files inside it, no more than ten files at time.

Against the Finder’s consistent model, eviction and downloading are well-behaved, as you can

1. select the folder, and apply the action to it,
2. select the exception(s), then apply the inverse action to them,

requiring just two selections and two actions.

On the other hand, pinning behaves inconsistently, as you must:

1. select a group of no more than ten files, apply the action to them,
2. repeat that as many times as necessary, until only the files you want pinned have been pinned.

Consistency is often taken as a mark of reliability. iCloud Drive comes from a position of lower reliability, and inconsistencies in its human interface only serve to reduce the user’s trust.

Explaining this idiosyncratic behaviour is more important still. Instead of being designed for the human, its interface has been determined by its engineering, in a step back to computers that preceded the Mac. This reflects an API that is every bit as awkward as its human interface. In common with other properties of files in iCloud Drive, developers are prevented from discovering how iCloud Drive handles them. For example, Apple makes checking whether a file is evicted as difficult as possible, as there’s no property or attribute that records that.

Pinning is similarly convoluted, and currently undocumented. When pinned individually, files are given a distinctive extended attribute, but when pinned by virtue of any of their enclosing folders being pinned, that xattr is attached not to the files, but to the folder. To check whether any given file in iCloud Drive is pinned, an app therefore has first to check whether that file has the xattr, then to traverse its path upward and look for the xattr on every folder in that path.

Thus the human interface for pinning is determined not by the Finder’s consistent grammar, but by the implementation of this feature in iCloud Drive’s virtual file system. The sooner consistency is restored, the better for the Finder and its users.

As I hinted on Monday, I’ve been updating my free iCloud Drive utility Cirrus to work with pinning files and folders in macOS Sequoia. It’s only when delving into the depths of iCloud Drive and pinning that you realise how bizarre it is, and how its interface can be intensely frustrating. This article starts with a brief practical demonstration of the oddities of this alien interface before trying to explain it, and ends with the new version of Cirrus.

Pinning frustration

To perform this brief demonstration, all you need is iCloud Drive with Optimise Mac Storage enabled, running in macOS Sequoia. Although it’s entirely non-destructive, you might like to try this using scrap files that you can trash in frustration at the end.

Create a new folder in iCloud Drive, give it a suitable name, and copy 12-18 files into it. Wait until they have all synced up to iCloud, then select just one of them and open the Finder’s contextual menu using Control-click, noting the new command Keep Downloaded that would pin that file to ensure it isn’t evicted from local storage.

Now press Command-A to select all those files in that folder, and open the contextual menu again. Notice that both the Keep Downloaded and Remove Download commands are now absent from the menu, and the only option offered is Download Now, even though although all those files are still downloaded.

Now select just a few of those files, ten or less, and the contextual menu offers Keep Downloaded and Remove Download commands again.

Rather than pinning these files piecemeal, pin their parent folder instead, using the contextual menu. A second or so after the folder shows the pinned icon, it’s shown against all the files inside that folder. So pinning a folder is simpler than trying to pin the files inside it, at least until you want to unpin any of them: select one of the files inside the pinned folder, open the contextual menu, and you’ll see there’s nothing there about Keep Downloaded, just another command to Show Downloaded Folder instead. Use that, and the Finder will then select the pinned folder for you (note, if you try this in Files in iOS, it doesn’t always work!).

The lesson is that, before you can unpin any of the files in that folder, you have to unpin the folder, which in turn unpins every file (and folder) within that folder, leaving you to manually pin the files you do want to pin, in batches of no more than ten at a time.

If you want further fun, with the parent folder still pinned, create a new folder inside that, and copy a file into that enclosed folder. You’ll see that, whatever you might want, everything that goes into that pinned folder is also pinned now, but can only be unpinned by unpinning everything inside the folder.

Pinning files and folders

The reason for this bizarre and annoying interface is the way that pinning is implemented.

When you pin files individually or in groups of up to ten, each file gains its own pinning extended attribute, of com.apple.fileprovider.pinned. But when you pin a folder, only that folder gains the extended attribute, none of the files or folders within it. The whole folder and the paths within it are designated as being pinned. And, as far as I can tell from the absence of any better information in Apple’s missing documentation, there’s no single method to determine whether a file in iCloud Drive is pinned.

Instead, you have to both

• look for the extended attribute attached to the file, and
• check all the folders in its path to determine if any of them has the extended attribute, which would then pin everything in their path.

Apple doesn’t document any file or URL attribute that can be used to determine whether a file or folder is pinned.

Cirrus 1.15

I had hoped that this new version of Cirrus, my free utility for exploring, working with and testing iCloud Drive, would be able to identify individual files and folders that are pinned in Sequoia. Because of the lack of any attribute to help this, I have temporarily abandoned my attempts.

What Cirrus now does, though, is performs deep scans of folders in iCloud Drive and reports which files and folders are pinned, giving individual file sizes, and a total file size and number of pinned files. This enables you to check where all your pinned files and folders are, and the pinning burden of iCloud Drive.

If you use pinning much, this is surely essential housekeeping.

Cirrus version 1.15 requires Big Sur 11.5, but its new feature requires macOS 15 Sequoia anyway, and is available from here: cirrus115
from Downloads above, from its Product Page, and through its auto-update mechanism. I hope it makes your pinning more painless.

The biggest changes to iCloud and iCloud Drive came in macOS Sonoma, when the latter adopted the new FileProvider framework to manage its syncing. If you’re intending upgrading from Ventura or an earlier version of macOS to Sequoia, then you should read this explanation and this discussion.

iCloud services for macOS fall into two broad categories: databases such as Contacts synced by CloudKit, and files stored in iCloud Drive, as managed in the Finder. Local copies of files stored in iCloud Drive can be kept downloaded or materialised, so that the whole of their data is stored locally, or their download can be removed, or evicted, so that their data isn’t stored on your Mac but only in iCloud, making the local file dataless.

FileProvider offers iCloud Drive users a choice of two schemes for syncing those files stored remotely:

• replicated, selected by turning Optimise Mac Storage off, in which FileProvider syncs between local and remote storage to ensure that their contents remain identical, and local files can’t be evicted to make them dataless;
• nonreplicated, when Optimise Mac Storage is turned on, which allows the user and macOS to remove local data from files, rendering them dataless by their eviction.

Although those options had been available long before Sonoma, their implementation had been flawed, allowing some to set iCloud Drive to operate in replicated mode, but still to evict files from local storage. Rather than using the current system of dataless files, older implementations used hidden stub files instead. Those issues were addressed by the adoption of FileProvider in Sonoma.

One curiosity here is that, according to Apple’s documentation, the public FileProvider API for macOS only offers replicated mode, but nonreplicated FileProvider is confined to iOS and iPadOS. In practice, iCloud Drive functions differently, in iOS and iPadOS only offering nonreplicated mode, while macOS offers the choice between either, according to the Optimise Mac Storage setting. I’ve been unable to find any explanation, but suspect this is the result of private APIs.

Pinning downloaded files

The most obvious change introduced in Sequoia is the ability to pin individual files to remain downloaded in local storage when operating in nonreplicated mode. I have already explained how this is used, and how it works in detail. Further testing since then has revealed some strange behaviours you need be aware of.

You can use the Finder’s contextual menu to pin multiple files and folders at once. When you pin a folder, its entire contents, including those of folders nested within it, are individually pinned, as is the folder itself. However, if you select more than 10 items in the Finder, the pinning option isn’t available in the menu. This appears to be an unintended oversight in this initial implementation, and should be rectified in a future macOS update.

If you pin an item that is currently evicted, then it will be automatically downloaded before being pinned, as you’d expect.

Pinning mechanism

Files and folders that have been pinned are distinguished by their com.apple.fileprovider.pinned extended attribute. Apple doesn’t currently document any file attribute or other means for telling whether a file is pinned. Using a utility such as xattred, or the xattr command tool, it’s simple to demonstrate that merely pasting or cutting that extended attribute from files sets their pinning status.

Pinning is performed by FileProvider, as an FPPinOperation, and iCloudDriveFileProvider then records the change in the cloud service’s file system FSTree to reflect the file’s pinning status, which is synced up to iCloud. Unpinning follows a similar sequence in an FPUnpinOperation. Those are reflected in log entries, making pinning and unpinning easy to observe there.

Pinning pains

The introduction of FileProvider in Sonoma highlighted how many Macs now have so much stored in iCloud Drive that they’re unable to download all files stored there, as they have insufficient free space on their boot disk to accommodate them all. For those Macs that are unable to operate iCloud Drive in replicated mode because of insufficient local storage capacity, pinning can be dangerous unless managed carefully.

Consider a Mac with 1 TB of files in iCloud Drive, operating in nonreplicated mode with Optimise Mac Storage turned on, and only 100 GB of free space on its internal SSD; most of those files in iCloud Drive are likely to have been evicted. If many of those are pinned, they will be downloaded and local free space will fall until it may be insufficient to even update macOS.

Although the effect of pinning files is of course reflected in measurements of free space provided by the Finder and Disk Utility, keeping track of pinned files and the space they occupy is currently not possible. Pinning is only shown for individual files and folders in Finder views, and isn’t reflected in the Get Info dialog in any way, or anywhere else. The user thus has no idea of the total number or size of files currently pinned, nor of their location, without inspecting every folder in iCloud Drive.

It appears that the only way that a utility could provide this information is by exhaustive checking of every file in iCloud Drive for the presence of the extended attribute. I hope to address that shortly.

iCloud caches on external storage

FileProvider in Sequoia brings a new feature allowing a cloud service to cache files on external storage, rather than having to keep them on the Data volume in the boot volume group. This could give users the ability to reduce the space required by FileProvider on their Mac’s internal SSD. At present, this doesn’t appear to be offered as an option for iCloud Drive, but may be intended as a future enhancement.

Third party behaviours

I am very grateful to Bader for drawing my attention to the behaviour of some third-party apps that may explain extended syncing with iCloud for some users. This may affect WhatsApp, which offers its own option of backing up to iCloud. If that’s enabled, then each time the app connects to its backup it may download the entire backup from iCloud to reconcile its data. When that backup reaches significant size, this results in a large initial sync taking place. The only way to prevent that is to disable that backup feature. Other apps with similar behaviours may also result in large syncs occurring. This can occur in any version of macOS, and is specific to apps with this feature.

Further reading

iCloud Drive in Sonoma: FileProvider and eviction
iCloud Drive in Sonoma: Copying a file
iCloud Drive in Sonoma: Mechanisms, throttling and system limits
iCloud Drive in Sonoma: Clones, sparse files, versions and more
iCloud Drive in Sonoma: Optimise Mac Storage or not?

One of the unannounced features in macOS Sequoia is, for many who use iCloud Drive, one of its most important, as this upgrade introduces the ability to pin files and folders to ensure they remain downloaded and don’t get evicted. This article explains how to use this feature, exactly what it does, and how it works.

Pin an item in iCloud Drive

Assuming that your Mac is connected to iCloud Drive and you have enabled Optimise Mac Storage, so that files in iCloud Drive can be evicted from local storage, you can use pinning to ensure that designated files aren’t evicted, and remain downloaded and synced with iCloud Drive. This is an important advance for anyone who needs certain files to be kept in local storage, so remaining accessible even when connection to iCloud isn’t possible, such as when you’re travelling.

Select the file or folder in the Finder and Control-click to bring up the contextual menu. You’ll see a new item there, Keep Downloaded. Select that.

That adds a new icon to the pinned files, with a white downward arrow on a circular grey background. This indicates that file is to be kept downloaded or pinned locally.

To unpin an item, select it and bring up the contextual menu. The Keep Downloaded item will be ticked. Select that command, and that toggles the setting off, unpinning the item so it can be evicted from local storage once again.

Pinning only works if your Mac’s iCloud Drive has Optimise Mac Storage enabled, which puts it into non-replicated FileProvider mode, and allows you and macOS to Remove Download or evict files. Its effect is to prevent that from occurring. Pinned files will remain downloaded to your Mac all the time, just as if iCloud Drive were in replicated FileProvider mode, with Optimise Mac Storage disabled.

A file’s pinning setting travels with it if you move that file within the same volume, and may be preserved when it’s moved away, for example by transferring the file by AirDrop. It’s preserved when you make changes to that file, but not when it’s copied, and pinning settings are specific to a local Mac, and not transferred to other Macs or devices connected to the same iCloud Drive.

How it works

Pinning is set by macOS attaching an extended attribute (xattr) named com.apple.fileprovider.pinned to each file that’s pinned. That normally contains a single byte, the character 1 as the single byte 0x31, although changing that to 0 works just as well, so it appears it’s the xattr that’s important, rather than its contents.

When a com.apple.fileprovider.pinned extended attribute is attached to a file, it ceases being eligible for eviction, and is kept downloaded, just as if it were in a replicated FileProvider. Unpinning the file strips the xattr.

Xattr flags

I have recently summarised the features of extended attributes, and explained the system of flags they use to determine how those are copied. In addition to adding the com.apple.fileprovider.pinned xattr to those encountered in Sequoia, Apple has changed the xattr flag system to cope with this new xattr. These flags are appended to the xattr name, after a # character, and pinning xattrs are actually named com.apple.fileprovider.pinned#PX to determine when they should be preserved during file copy operations. The X flag is new to Sequoia, and can be used for other xattrs.

This brings the table of available xattr flags to six. Flags can be upper or lower case letters C, N, P, S, B or X, and invariably follow the # separator, which is presumably otherwise forbidden from use in a xattr’s name. Upper case sets (enables) that property, while lower case clears (disables) that property. The properties are now (as of 15.0):

• C: XATTR_FLAG_CONTENT_DEPENDENT, which ties the flag and the file contents, so the xattr is rewritten when the file data changes. This is normally used for checksums and hashes, text encoding, and position information. The xattr is preserved for copy and share, but not in a safe save.
• P: XATTR_FLAG_NO_EXPORT, which doesn’t export or share the xattr, but normally preserves it during copying.
• N: XATTR_FLAG_NEVER_PRESERVE, which ensures the xattr is never copied, even when copying the file.
• S: XATTR_FLAG_SYNCABLE, which ensures the xattr is preserved during syncing with services such as iCloud Drive. Default behaviour is for xattrs to be stripped during syncing, to minimise the amount of data to be transferred, but this will override that.
• B: XATTR_FLAG_ONLY_BACKUP, which keeps the xattr only in backups, including Time Machine.
• X: XATTR_FLAG_ONLY_SAVING, which keeps the xattr only when saving and in backups, including Time Machine (introduced in macOS 15.0).

These must operate within another general restriction of xattrs: their name cannot exceed a maximum of 127 UTF-8 characters.

This new flag doesn’t change the macOS default flags set for different types of xattr, which remain as:

• com.apple.quarantine – PCS
• com.apple.TextEncoding – CS
• com.apple.metadata:kMDItemCollaborationIdentifier – B
• com.apple.metadata:kMDItemIsShared – B
• com.apple.metadata:kMDItemSharedItemCurrentUserRole – B
• com.apple.metadata:kMDItemOwnerName – B
• com.apple.metadata:kMDItemFavoriteRank – B
• com.apple.metadata:* (except those above) – PS
• com.apple.security.* – S
• com.apple.ResourceFork – PCS
• com.apple.FinderInfo – PCS
• com.apple.root.installed – PC

(* here represents the wild card.)

Until Apple releases the source code for Sequoia’s open source components, the table of copy intents remains unchanged at

• XATTR_OPERATION_INTENT_COPY – a simple copy, preserves xattrs that don’t have flag N or B
• XATTR_OPERATION_INTENT_SAVE – save, where the content may be changing, preserves xattrs that don’t have flag C or N or B
• XATTR_OPERATION_INTENT_SHARE – share or export, preserves xattrs that don’t have flag P or N or B
• XATTR_OPERATION_INTENT_SYNC – sync to a service such as iCloud Drive, preserves xattrs if they have flag S, or have neither N nor B
• XATTR_OPERATION_INTENT_BACKUP – back up, e.g. using Time Machine, preserves xattrs that don’t have flag N

as documented in man xattr_name_with_flags. Once I have been able to check the new source code, I will confirm any changes to the intents table.

Summary

• If you have Optimise Mac Storage turned on for iCloud Drive, you can now pin files so that they aren’t evicted from local storage.
• To pin a file or folder, use the Finder’s contextual menu (Control-click) and select the Keep Downloaded item.
• To unpin a file, toggle the Keep Downloaded item off (unticked).
• Pinning works by adding an extended attribute to each pinned file.
• A sixth xattr flag has been added, X, to keep that xattr only when saving and in backups.