APFS has many of the features of modern file systems that make most efficient use of space, and others that aren’t found in more traditional file systems like HFS+. While those should all work well when used locally with other APFS volumes, they can prove incompatible with other file systems, and may have untoward side effects. This article looks at those features that you could encounter problems with, and how best to work around them.

Sparse files

Most modern file systems store files that contain significant amounts of blank or absent data in a special sparse file format. In APFS, this is implemented by only allocating file extents to those blocks that do contain data. In many cases, this can save significant disk space. Initially, sparse files were unusual in APFS, but over time they have become increasingly common, and can now be found in some types of disk image, virtual machine storage, and in some databases. Note that disk image types whose name includes the word sparse, sparse bundles and sparse disk images, don’t use sparse file format at all, but are the victims of an unfortunate name collision.

Neither macOS nor APFS can simply convert regular files to sparse format; for a file to be written in sparse format, the code writing it must explicitly skip the empty space. As a result, sparse files are prone to explode to full size when they’re copied or moved unless that’s between two local volumes both using APFS. Examples of where you should expect them to explode include:

• copy or move to HFS+, as that has no sparse file format
• copy between Macs using AirDrop or file sharing
• back up to network storage, although local Time Machine backups to APFS should preserve them
• copy or move to any other local file system other than APFS, even if that file system has its own support for sparse files.

In each of those cases, sparse files explode in size as they’re copied from the source volume. If you have a 100 GB sparse file that only takes 20 GB of local disk space, when it’s copied over the network, or to a local HFS+ volume, the full 100 GB has to be transferred.

One potential workaround is to compress the sparse file before transferring it. All good compression algorithms will work efficiently on the blank space in the file, so when compressed its size could be as small as the original sparse file. However, when it’s decompressed, even on another APFS volume, it will explode to full size. For disk images, that can be corrected by mounting them, as APFS will then trim their contents, and the disk image should be saved back into sparse format.

Clone files

These are two distinct files that share common data, normally the result of duplicating the original within the same APFS volume. Those two cloned files then only require the storage of the whole file, plus those data blocks that differ between them. This only works within the same volume, and the moment that either of the clones is moved or copied to any other volume, it assumes full size, as it can no longer share data with the other clone.

However, most other file systems don’t support file cloning in this way. When you duplicate a file in an HFS+ volume, there’s no shared data between them, and the two require twice the amount of space as one does.

Snapshots

Snapshots consist of a complete copy of the volume at an instant in time, so require a copy of the file system metadata for that volume, and retain copies of storage blocks for each file as they are changed subsequent to the snapshot being made, so you could roll back that volume to its state at the moment of the snapshot.

Although Time Machine backups contain snapshots of the volume they back up, snapshots can’t normally be copied to another disk or volume. Some have been able to make a complete copy of a disk including its snapshots using the dd command tool, but that should be considered experimental. In all other circumstances, snapshots stay where they were made, but you can always copy from an existing snapshot to reconstitute a volume.

Directory hard links

These aren’t available in APFS, but are supported in HFS+, where they’re used extensively in Time Machine backups. They work like regular hard links, but act on directories rather than individual files. They can’t be copied in any way to an APFS volume, but can be used to reconstitute a volume.

Extended attributes

These are additional metadata associated with files and folders, and are fully supported in both APFS and HFS+. However, many are treated as being ephemeral, and may not be preserved during copying or other actions. The system of flags used to determine which are preserved is detailed in this article.

Several other file systems also support extended attributes, but copying between them is unlikely to transfer them between file systems. In some cases, extended attributes are preserved using AppleDouble format, in which each file can have a hidden shadow with its name prefixed with ._ (dot-underscore) characters. These are most often seen in FAT and ExFAT volumes, but are prone to confuse users of other computers.

Key points

• APFS sparse files are only preserved when copying or moving files between local APFS volumes. In other circumstances they explode to full size.
• Good compression methods can keep a sparse file to a similar size, but decompression explodes them to full size. Disk images may then be restored as sparse files after they have been mounted again from APFS.
• Clone files aren’t preserved when copied or moved to any other volume.
• Snapshots can’t be copied at all, although they can be reconstituted as volumes.
• APFS doesn’t support the directory hard links used in Time Machine backups to HFS+, but a backup can be reconstituted as a volume.
• Extended attributes are preserved between APFS and HFS+, but not with other file systems, except in the shadow files of AppleDouble format, seen in FAT and ExFAT volumes.

One of the unannounced features in macOS Sequoia is, for many who use iCloud Drive, one of its most important, as this upgrade introduces the ability to pin files and folders to ensure they remain downloaded and don’t get evicted. This article explains how to use this feature, exactly what it does, and how it works.

Pin an item in iCloud Drive

Assuming that your Mac is connected to iCloud Drive and you have enabled Optimise Mac Storage, so that files in iCloud Drive can be evicted from local storage, you can use pinning to ensure that designated files aren’t evicted, and remain downloaded and synced with iCloud Drive. This is an important advance for anyone who needs certain files to be kept in local storage, so remaining accessible even when connection to iCloud isn’t possible, such as when you’re travelling.

Select the file or folder in the Finder and Control-click to bring up the contextual menu. You’ll see a new item there, Keep Downloaded. Select that.

That adds a new icon to the pinned files, with a white downward arrow on a circular grey background. This indicates that file is to be kept downloaded or pinned locally.

To unpin an item, select it and bring up the contextual menu. The Keep Downloaded item will be ticked. Select that command, and that toggles the setting off, unpinning the item so it can be evicted from local storage once again.

Pinning only works if your Mac’s iCloud Drive has Optimise Mac Storage enabled, which puts it into non-replicated FileProvider mode, and allows you and macOS to Remove Download or evict files. Its effect is to prevent that from occurring. Pinned files will remain downloaded to your Mac all the time, just as if iCloud Drive were in replicated FileProvider mode, with Optimise Mac Storage disabled.

A file’s pinning setting travels with it if you move that file within the same volume, and may be preserved when it’s moved away, for example by transferring the file by AirDrop. It’s preserved when you make changes to that file, but not when it’s copied, and pinning settings are specific to a local Mac, and not transferred to other Macs or devices connected to the same iCloud Drive.

How it works

Pinning is set by macOS attaching an extended attribute (xattr) named com.apple.fileprovider.pinned to each file that’s pinned. That normally contains a single byte, the character 1 as the single byte 0x31, although changing that to 0 works just as well, so it appears it’s the xattr that’s important, rather than its contents.

When a com.apple.fileprovider.pinned extended attribute is attached to a file, it ceases being eligible for eviction, and is kept downloaded, just as if it were in a replicated FileProvider. Unpinning the file strips the xattr.

Xattr flags

I have recently summarised the features of extended attributes, and explained the system of flags they use to determine how those are copied. In addition to adding the com.apple.fileprovider.pinned xattr to those encountered in Sequoia, Apple has changed the xattr flag system to cope with this new xattr. These flags are appended to the xattr name, after a # character, and pinning xattrs are actually named com.apple.fileprovider.pinned#PX to determine when they should be preserved during file copy operations. The X flag is new to Sequoia, and can be used for other xattrs.

This brings the table of available xattr flags to six. Flags can be upper or lower case letters C, N, P, S, B or X, and invariably follow the # separator, which is presumably otherwise forbidden from use in a xattr’s name. Upper case sets (enables) that property, while lower case clears (disables) that property. The properties are now (as of 15.0):

• C: XATTR_FLAG_CONTENT_DEPENDENT, which ties the flag and the file contents, so the xattr is rewritten when the file data changes. This is normally used for checksums and hashes, text encoding, and position information. The xattr is preserved for copy and share, but not in a safe save.
• P: XATTR_FLAG_NO_EXPORT, which doesn’t export or share the xattr, but normally preserves it during copying.
• N: XATTR_FLAG_NEVER_PRESERVE, which ensures the xattr is never copied, even when copying the file.
• S: XATTR_FLAG_SYNCABLE, which ensures the xattr is preserved during syncing with services such as iCloud Drive. Default behaviour is for xattrs to be stripped during syncing, to minimise the amount of data to be transferred, but this will override that.
• B: XATTR_FLAG_ONLY_BACKUP, which keeps the xattr only in backups, including Time Machine.
• X: XATTR_FLAG_ONLY_SAVING, which keeps the xattr only when saving and in backups, including Time Machine (introduced in macOS 15.0).

These must operate within another general restriction of xattrs: their name cannot exceed a maximum of 127 UTF-8 characters.

This new flag doesn’t change the macOS default flags set for different types of xattr, which remain as:

• com.apple.quarantine – PCS
• com.apple.TextEncoding – CS
• com.apple.metadata:kMDItemCollaborationIdentifier – B
• com.apple.metadata:kMDItemIsShared – B
• com.apple.metadata:kMDItemSharedItemCurrentUserRole – B
• com.apple.metadata:kMDItemOwnerName – B
• com.apple.metadata:kMDItemFavoriteRank – B
• com.apple.metadata:* (except those above) – PS
• com.apple.security.* – S
• com.apple.ResourceFork – PCS
• com.apple.FinderInfo – PCS
• com.apple.root.installed – PC

(* here represents the wild card.)

Until Apple releases the source code for Sequoia’s open source components, the table of copy intents remains unchanged at

• XATTR_OPERATION_INTENT_COPY – a simple copy, preserves xattrs that don’t have flag N or B
• XATTR_OPERATION_INTENT_SAVE – save, where the content may be changing, preserves xattrs that don’t have flag C or N or B
• XATTR_OPERATION_INTENT_SHARE – share or export, preserves xattrs that don’t have flag P or N or B
• XATTR_OPERATION_INTENT_SYNC – sync to a service such as iCloud Drive, preserves xattrs if they have flag S, or have neither N nor B
• XATTR_OPERATION_INTENT_BACKUP – back up, e.g. using Time Machine, preserves xattrs that don’t have flag N

as documented in man xattr_name_with_flags. Once I have been able to check the new source code, I will confirm any changes to the intents table.

Summary

• If you have Optimise Mac Storage turned on for iCloud Drive, you can now pin files so that they aren’t evicted from local storage.
• To pin a file or folder, use the Finder’s contextual menu (Control-click) and select the Keep Downloaded item.
• To unpin a file, toggle the Keep Downloaded item off (unticked).
• Pinning works by adding an extended attribute to each pinned file.
• A sixth xattr flag has been added, X, to keep that xattr only when saving and in backups.

In the previous article, I outlined what extended attributes do, and how they work in macOS. I also started to explain how some are considered ephemeral, while others are persistent. This article continues from there, by documenting how macOS decides what to do with them when a file containing xattrs is copied.

Although Apple does now explain a little about this in the context of the FileProvider framework and syncing with cloud services, the only useful documentation is provided in man xattr_name_with_flags, and two source code files that are part of the open source copyfile component.

In 2013, as part of its enhancements for iCloud in particular, Apple added support for flags on xattrs to indicate how those xattrs should be handled when the file is copied in various ways. Rather than change the file system, Apple opted for what’s perhaps best seen as an elegant kludge: appending characters to the end of the xattr’s name.

If you work with xattrs, you’ve probably already seen this in those whose name ends with a hash # then one or more characters: that’s actually the flags, not part of the name, what Apple refers to as a ‘property list’. To avoid confusion I won’t use that term here, but refer to them as xattr flags. A common example of this is com.apple.lastuseddate#PS, which is seen quite widely. In recent years, Apple has added one flag, B, and there’s another to come with Sequoia.

Xattr flags

Flags can be upper or lower case letters C, N, P, S or B, and invariably follow the # separator, which is presumably otherwise forbidden from use in a xattr’s name. Upper case sets or enables that property, while lower case clears or disables that property. There are currently (macOS 14.6.1) five properties:

• C: XATTR_FLAG_CONTENT_DEPENDENT, which ties the flag and the file contents, so the xattr is rewritten when the file data changes. This is normally used for checksums and hashes, text encoding, and position information. The xattr is preserved for copy and share, but not in a safe save.
• P: XATTR_FLAG_NO_EXPORT, which doesn’t export or share the xattr, but normally preserves it during copying.
• N: XATTR_FLAG_NEVER_PRESERVE, which ensures the xattr is never copied, even when copying the file.
• S: XATTR_FLAG_SYNCABLE, which ensures the xattr is preserved during syncing with services such as iCloud Drive. Default behaviour is for xattrs to be stripped during syncing, to minimise the amount of data to be transferred, but this will override that.
• B: XATTR_FLAG_ONLY_BACKUP, which keeps the xattr only in backups, including Time Machine (added recently).

These operate within another general restriction of xattrs: their name cannot exceed a maximum of 127 UTF-8 characters.

Defaults

macOS provides a standard ‘whitelist’ of default flag settings for different types of xattr. These aren’t contained in a configuration file, but are baked into the xattr flag code, where as of macOS 14.6.1 the following default flags are set for different types of xattr (* here represents the wild card):

• com.apple.quarantine – PCS
• com.apple.TextEncoding – CS
• com.apple.metadata:kMDItemCollaborationIdentifier – B
• com.apple.metadata:kMDItemIsShared – B
• com.apple.metadata:kMDItemSharedItemCurrentUserRole – B
• com.apple.metadata:kMDItemOwnerName – B
• com.apple.metadata:kMDItemFavoriteRank – B
• com.apple.metadata:* (except those above) – PS
• com.apple.security.* – S
• com.apple.ResourceFork – PCS
• com.apple.FinderInfo – PCS
• com.apple.root.installed – PC

Copy intents

Also contained in the source code is a table of intents, that explains how different types of copy are affected by different combinations of xattr flag. Currently, those are:

• XATTR_OPERATION_INTENT_COPY – a simple copy, preserves xattrs that don’t have flag N or B
• XATTR_OPERATION_INTENT_SAVE – save, where the content may be changing, preserves xattrs that don’t have flag C or N or B
• XATTR_OPERATION_INTENT_SHARE – share or export, preserves xattrs that don’t have flag P or N or B
• XATTR_OPERATION_INTENT_SYNC – sync to a service such as iCloud Drive, preserves xattrs if they have flag S, or have neither N nor B
• XATTR_OPERATION_INTENT_BACKUP – back up, e.g. using Time Machine, preserves xattrs that don’t have flag N

Use

If you want a xattr preserved when it passes through iCloud, you therefore need to give it a name ending in the xattr flag S, such as co.eclecticlight.MyTest#S. Sure enough, when xattrs with that flag are passed through iCloud Drive, those xattrs are preserved even if the default rule would treat them differently. Similarly, to have a xattr that is stripped even when you just make a local copy of that file, append #N to its name.

There’s a further limit imposed on xattrs synced by FileProvider, including those for iCloud Drive, that strips all individual xattrs that are larger than a certain size. Apple gives that as “about 32KiB total for each item”, and my measurements performed in the recent past put that at about 32,650 bytes, slightly less than 32,767.

In itself, this information is valuable if you ever use any metadata stored in xattrs. It’s used in my intergrity-checking utilities Dintch, Fintch and cintch to ensure the xattr containing a file’s hash isn’t stripped by passage through iCloud Drive, for instance. On Tuesday morning next week, once Sequoia has been released, I’ll explain how Apple has extended this system to achieve something that many have been wishing for.

One of the innovative features in classic Mac OS was its use of resource forks, allowing structured metadata to be attached to any file. When Mac OS X merged that with the more traditional Unix approach adopted by NeXTSTEP, those were nearly lost. Classic Mac apps were restructured from storing most of their components, including their executable code, in their resource fork, when Mac OS X flattened those into an app bundle consisting of a hierarchy of separate files in folders, without any resources.

For the first four years of Mac OS X resource forks were reluctantly tolerated, until the solution came in 10.4 with the introduction of extended attributes, including one to contain what had previously been stored in the resource fork, which became an extended attribute or xattr with the name com.apple.ResourceFork.

All files in HFS+ and APFS (and other file systems) contain a fairly standard set of metadata known as attributes, information about a file such as its name, datestamps and permissions. Xattrs are extensions to those that contain almost any other type of metadata, the first notable xattr coming in Mac OS X 10.5, named com.apple.quarantine. That contains quarantine information for apps and other files downloaded from the internet, in a format so ancient that the quarantine flag is stored not in binary but as text.

The quarantine xattr provides a good demonstration of some of the valuable properties of xattrs: it can be attached to any file (or folder) without changing its data, and isn’t included when calculating CDHashes for code signatures. It can thus be added safely without any danger of altering the app or its code, although it does change the way in which macOS handles the code, by triggering security checks used to verify it isn’t malicious. Once those have been run, the flag inside the quarantine xattr can be changed to indicate it has been checked successfully.

Far from being a passing phase, or dying out as some had expected, xattrs have flourished since those early days. This has happened largely unseen by the user: few change anything revealed in the Finder’s Get Info dialog, although they’re used to store some forms of visible metadata such as Finder tags, and the URL used to download items from the internet. Editing xattrs is normally performed silently: you’re not made aware of changes in the quarantine xattr, and in most cases the only way to manage xattrs is to use the xattr command tool, or one of very few apps like xattred that can edit and manage them.

Examples

Among the well-known and important xattrs you can encounter are:

• com.apple.quarantine the quarantine xattr, containing a quarantine flag
• com.apple.rootless marks items individually protected by System Integrity Protection (SIP)
• com.apple.provenance contains data about the origin of apps that have been quarantined
• com.apple.metadata:kMDItemCopyright records copyright info
• com.apple.metadata:kMDItemWhereFroms the origin of downloaded file as a URL
• com.apple.metadata:_kMDItemUserTags Finder tags
• com.apple.TextEncoding reveals text file encoding
• com.apple.ResourceFork a classic Mac resource fork

Storage

In APFS and HFS+, xattrs aren’t stored with file data, nor with a file or folder’s normal attributes.

For smaller extended attributes up to 3,804 bytes, their data is stored with the xattr in the file system metadata. Larger extended attributes are stored as data streams, with separate records, but still separately from the file data. Apple doesn’t give a limit on the maximum size of xattrs, but they can certainly exceed 200 KB, and each file and folder can have an effectively unlimited number of them.

Persistence

Most file systems to which macOS can write either handle xattrs natively (HFS+, APFS), or macOS uses a scheme to preserve them, as in the hidden files written to FAT and ExFAT volumes. NFS is an important exception, and files copied to NFS will have all their xattrs stripped. Neither are extended attributes unique to Macs: most file systems used by Linux support them, and even Windows can at a push.

Because xattrs contain a wide range of metadata, some are treated as being ephemeral, others as persistent. Moving files with xattrs around within the same volume shouldn’t affect their xattrs, as that takes place within the same file system. Copying files to another volume, even if both use APFS, may leave some xattrs behind if they’re considered to be ephemeral.

The most complex situation is when a file with xattrs is moved to iCloud Drive. The Mac that originated that file is likely to retain most if not all of its xattrs, because the local copy remains within the same volume and file system. However, not all xattrs are copied up to iCloud storage, so other Macs accessing that file may only see a small selection of them. The rules for which xattrs are to be preserved during file copying, including in iCloud Drive, are baked into macOS, and the subject of the next article.