In the days before Mac OS X, Apple didn’t provide a serious backup utility, and by the time we were starting to move up from Classic Mac OS the standard choice was normally Dantz Development’s Retrospect, first released in 1989 and still available today in version 19.

Time Machine wasn’t the first utility in Mac OS to back up local storage. In 2004, Apple’s first cloud subscription service .Mac included a Backup app that backed up local files to iDisk in the cloud, something that still isn’t supported today with iCloud.

In the following years, AirPort Wi-Fi systems flourished, and Apple decided to launch a consumer NAS incorporating an AirPort Extreme Base Station with a 500 GB or 1 TB hard disk. Software to support that was dubbed Time Machine, and was released in Mac OS X 10.5 Leopard on 26 October 2007, 17 years ago. The first Time Capsule was announced in January 2008, and shipped a month later.

Time Machine’s pane in System Preferences changed little until Ventura’s System Settings replaced it.

The application’s restore interface featured a single Finder-like window, much like today’s. Internally, Time Machine scheduled its backups using a system timer and launchd, making backups every hour regardless of what else the Mac might have been doing at the time.

The initial version of Time Machine was both praised and slated. Unlike Mike Bombich’s rival Carbon Copy Cloner, it couldn’t create bootable backups, and there were problems with FileVault encryption, which at that time could only encrypt Home folders, rather than whole volumes. Despite those, its introduction transformed the way that many used their Macs, and made it more usual for users to have backups.

From its release, Time Machine was dependent on features of the HFS+ file system to create its Finder illusion. Every hour the backup service examined the record of changes made to the file system since the last backup was made, using its FSEvents database. It thus worked out what had changed and needed to be copied into the backup. During the backup phase itself, it only copied across those files that had been created or changed since the last backup was made.

It did this by using hard links in the backup, and Apple added a new feature to its HFS+ file system to support this, directory hard links. Where an entire folder had remained unchanged since the last backup, Time Machine simply created a hard link to the existing folder in that backup. Where an existing file had been changed, though, the new file was written to the backup inside a changed folder, which in turn could contain hard links to its unchanged contents.

This preserved the illusion that each backup consisted of the complete contents of the source, while only requiring the copying of changed files, and creation of a great many hard links to files and folders. It was also completely dependent on the backup volume using the HFS+ file system, to support those directory hard links.

Without directory hard links, backups would quickly have become overwhelmed by hard links to files. If you had a million files and folders on the backup source volume, every hourly backup would have had to create a total of a million copied files or hard links. Directory hard links thus enabled the efficiency needed for this novel scheme to work.

Apple later introduced what it termed Mobile Time Machine, intended for notebooks that could be away from their normal backup destination for some time. In around 10,000 lines of code, Mac OS X came to create something like a primitive snapshot, only on HFS+.

When macOS introduced its new DAS-CTS scheduling and dispatch system for background activities, in (about) Sierra, Time Machine’s backups were added to that. That proved unfortunate at the time because of a bug in that system, which failed on Macs left running continuously for several days, when backups could become infrequent and irregular.

When Apple released the first version of APFS on Mac OS X in High Sierra, its new snapshot feature was immediately incorporated into Time Machine to replace the earlier Mobile variant. Initially, APFS snapshots were also used instead of the FSEvents database to determine what should be backed up. Since then, making each backup of an APFS volume has involved creating a snapshot that’s stored locally on the APFS volume being backed up. In High Sierra and Mojave, the structure of backups themselves didn’t change, so they still required an HFS+ volume and relied on directory hard links.

Catalina introduced a more complicated scheme to replace snapshots as the usual means for determining what to back up. This was presumably because computing a snapshot delta had proved slow. As the backup destination remained in HFS+ format that could’t use snapshots, it continued to rely on directory hard links.

Big Sur and its successors with Signed System Volumes (SSVs) retained the option to continue backing up to HFS+ volumes, but added the ability to back up APFS volumes to APFS backup storage at last.

When backing up to APFS, Time Machine reverses the design used in High Sierra: instead of using snapshots to determine what needed to be backed up before creating a backup using traditional hard links, most of the time Time Machine determines what has changed using the original method with FSEvents, then creates each backup as a synthetic snapshot on the backup store. Unlike earlier versions, in Big Sur and later Time Machine can’t back up the System volume.

Once Time Machine has made a detailed assessment of the items to be backed up, it forecasts the total size to be copied. The local snapshot is copied to an .inprogress folder on the backup volume, and backup copying proceeds. Where possible, only changed blocks of files are copied, rather than having to copy the whole of every file’s data, an option termed delta-copying that can result in significant savings. Old backups are removed both according to age, and to maintain sufficient free space on the backup volume, in what Time Machine refers to as age-based and space-based thinning.

Data copied to assemble the backup on the backup volume is formed into a synthetic snapshot used to present the contents of that backup both in the Time Machine app and the Finder. Those snapshots are presented in /Volumes/.timemachine/ although they’re still stored on the backup volume.

Although modern Time Machine backups to APFS are both quicker and more space-efficient, the structure of backup storage poses problems. Copying backup stores on HFS+ was never easy, but there are currently no tools that can transfer those on APFS to another disk.

Behind the familiar interfaces of its app and settings, Time Machine has come a long way over the last few years, from building an illusion using huge numbers of hard links to creating synthetic snapshots.

The most difficult question when setting up a backup scheme is determining how much storage you need for all those backups. This article suggests how you can estimate how big that backup storage needs to be, assuming that you’re using incremental backup, like Time Machine, which keeps a sequence of older versions of each file in a series of multiple backups.

Basic formula

At its simplest, you need to estimate the size of the initial backup, and the average size of new backups each month for the period you intend using that storage. Then work out
initial backup size + (average monthly backups × number of months taking new backups)

Let’s say my Mac has a 1 TB internal SSD, with around 500 GB to be backed up, and each month 50 GB will be added to that. If I want to use that as my backup storage for 2 years, then it will need a capacity of
500 + (24 x 50) = 1700 GB
so a 2 TB drive should do fine for slightly longer.

In practice, Time Machine and other backup utilities are more frugal with the space they require, as they automatically ‘thin’ or remove some older backups. Time Machine’s policy is to keep:

• hourly backups for the previous 24 hours, in addition to hourly local snapshots on each volume being backed up,
• daily backups over the last month,
• weekly backups back to the start of that backup series.

Initial backup size

When backing up data-only volumes, like media libraries, almost everything on them is likely to be backed up initially. Significant exceptions include hidden folders containing Spotlight indexes and version databases; the former can be substantial, and can be measured by showing hidden files in the Finder, selecting the .Spotlight-V100 folder, and inspecting its size in the Get Info dialog. The latter can only be measured in Terminal, and can be safely ignored. Another hidden occupier of free space are snapshots, whose size can be estimated in Disk Utility.

Estimating backup size from a boot volume group, such as that containing your Home folder, is far harder. Time Machine and other backup utilities will only back up the Data volume in any case, and it usually holds many files and folders that aren’t backed up, in addition to its Spotlight indexes and version database. You may be able to gain an estimate from the Storage item in General settings, although that tends to lump a great deal into its System Data, some of which will be backed up, and some won’t.

If you use iCloud Drive much, particularly if you keep your Desktop and Documents folders in iCloud Drive and Optimise Mac Storage is enabled so that many files are evicted from local storage, you’ll need to consider how that might change if many of those items are downloaded, and can then be backed up. Another consideration is whether your backups will preserve sparse files at their reduced size, or expand them to full size. Again, that’s difficult to estimate, although Time Machine will normally retain their small size when backing them up to APFS.

Incremental backup rate

If you’re already backing up a volume, and it’s likely to see similar use in the future, use records of recent backups to estimate their daily or weekly total size. For Time Machine, you can do that using my free utility T2M2 (The Time Machine Mechanic). This provides:

• current free space in backup storage, which you can track over a week or so;
• actual free space and thinning size each time an old backup is thinned;
• total file size copied in each backup.

Save those records each day for a week, and you’ll get a very good idea as to how much additional space is required for the period you’re intending to keep those backups in that store.

Full backup storage

Another decision to make is what you’re going to do when that storage is full. If you’re using Time Machine, those backups are stored as snapshots, so at present can’t be copied across to another disk to retain in your archives. The only options are to reformat the volume for reuse, or to retain it as an archive. If you’re going to reuse that disk, consider whether it will still be large enough for use when it comes to the end of its backup series.

Summary

• Estimate initial backup size from the total used by all volumes to be backed up.
• Your boot volume group won’t require as much backup space as it uses, and Storage settings may help you guess how much to reduce that.
• Estimate the rate of growth using current backups, where available, from reports in T2M2.
• Decide how long you want to back up to that storage, thus how much your backups will grow over time.
• Then Total space required = initial backup size + (monthly increment × number of months in use).

Most backup utilities now make snapshots of volumes they’re backing up, and Time Machine goes further by using snapshots in its backup process, and creating backups as snapshots. How then should we include snapshots in our backup plans? Could we rely on snapshots rather than conventional backups?

APFS snapshots

Each snapshot contains a complete set of the file system metadata for that volume at the time the snapshot was made, and all the extents required to reinstate that volume. Although tied to that volume, the snapshot is stored alongside the current volume metadata, in the same APFS container, in the same disk partition.

Extents list the storage blocks containing the data that composes every file that existed within that volume at the time the snapshot was made. Many of those will be the same as in the current volume, but the remainder will refer to data that has been deleted since the snapshot was made, but is being retained to enable the volume to be rolled back to its previous state. Those old extents can only be removed when that snapshot is deleted, which thus frees all those storage blocks at the same time. This is illustrated in the diagram.

This shows the same file in a snapshot and the current volume. Extents for the data of the earlier version of that file contained in the snapshot are shown at the top, and consist of blocks EA, EB, EC and ED. After that snapshot was made, the file was edited and then consists of the blocks shown at the bottom, EA, FB, FC and ED.

Thus, the extents listed for that file in the snapshot consist of two blocks, EA and ED, that are currently in use and included in its extents now, and two blocks, EB and EC, that were deleted after the snapshot was made. However, as those are referenced in the snapshot’s extents, those storage blocks are retained to enable the snapshot to restore the volume’s previous state. When that snapshot is deleted, blocks EB and EC will then be returned to the pool of free blocks for erasure and re-use.

This demonstrates how snapshots depend on their own file system metadata, and a combination of currently used storage blocks and those awaiting return for re-use.

Rolling back to a snapshot

The most popular reason for wanting to roll back to a snapshot is to recover from problems following a macOS update. As the System volume isn’t included among backup snapshots, and the process of rolling back would be so complex that it’s probably not possible, there’s no point in considering the use of snapshots for that purpose.

In other circumstances, where part or all of a user volume is to be restored from a snapshot, once that snapshot has been mounted, the procedure is the same as for restoring from a backup. What is different is that restoring a whole volume from a snapshot is a one-way trip, and there is no undo. This is because snapshots subsequent to that used to restore from will be removed, and you won’t then be able to ‘roll forward’ to a later snapshot. That contrasts with a normal backup, where items remain available from any other backup that is retained in the backup store.

Snapshot robustness

Long before the introduction of APFS, Time Machine created its own form of snapshots when a Mac was away from its normal backup store. Those enabled anyone travelling without an external drive to have limited ability to restore lost or damaged files despite their lack of conventional backups. Experience showed this to be useful in the event of human error, such as accidental deletion of files, but of limited or no use when there were file system errors.

Because snapshots share the same container as the current volume, and share many file extents with them, they are prone to common errors. In particular, common file extents make it more likely that faults occurring in extents and data storage will affect them both. This is particularly important as one of the most common file system errors that corrupts data in files occurs when extents for two separate files overlap. A snapshot is thus more vulnerable than a backup on a different disk, or even one in a different container on the same physical store.

Snapshots are whole-volume

Snapshots do have one specific advantage over backups when it comes to their coverage. As they include the whole file system metadata for the volume, no items present in that volume are excluded from its snapshots. If you want to restore an item that has been excluded from backups made of any volume, you can therefore do that from its latest snapshot, if that item was present in the volume at the time that was made.

The only disadvantage to this is that snapshots can be disproportionately large compared to volume backups. If large and ever-changing files such as Virtual Machines are excluded from backing up, but are in a volume for which snapshots are made, those snapshots may appear excessively large. The solution is to move those files to a separate volume, for which snapshots aren’t made. This behaviour can also raise privacy concerns over data that aren’t encrypted on disk, as they too will be accessible in that volume’s snapshots even when excluded from its backups.

Conclusions

• Because snapshots contain metadata and data common to the current version of a volume, and share the same container, they are prone to common errors.
• Restoring a snapshot can’t be undone, and removes later snapshots.
• Snapshots are a useful addition to conventional backups made to separate storage, but can’t replace them.
• When conventional backups aren’t available, snapshots can be invaluable.
• Snapshots can be used to restore items that were excluded from backups.
• Snapshots can’t be used to roll back from a bad macOS update.

Reference

APFS snapshots in detail

macOS Sonoma brought major internal changes to iCloud Drive, and coupled with the opacity of Time Machine, you may now be wondering whether it still backs up what you have stored in iCloud Drive, and if so, where to find those backups. This article explains how this now works, and where those backups can be found.

What does Time Machine back up?

By default, Time Machine backs up almost everything on volumes it’s set to back up. It doesn’t, of course, try to back anything up from the Signed System Volume (SSV) containing macOS itself, and there are a few folders it automatically excludes, such as the hidden folder at the root of each volume containing the versions database (as that can’t be restored in any case). But unless you add iCloud Drive to its exclusion list from your Home Library folder, Time Machine will back up all the files and folders in iCloud Drive as long as they’re stored locally at the time it makes that backup.

Dataless files

It’s that last condition that’s the most important consideration. If your Mac has Optimise Mac Storage turned off, then iCloud Drive is run as a Replicating FileProvider, where everything stored in iCloud Drive is also stored locally. In that case, all those items will be backed up by Time Machine just as they would be if kept in purely local storage.

If Optimise Mac Storage is turned on, iCloud Drive behaves as a Non-Replicating FileProvider, and some or many of the files you store in iCloud Drive may be evicted from local storage, and are then no longer available for backup. This is explained in the diagram.

Files in local storage consist of attributes, metadata (extended attributes) and data. Move them to iCloud Drive and their data is then copied up to iCloud. When a file is evicted to save local storage space, the local file’s data is removed, making that copy of the file dataless. In previous versions of macOS, that was accomplished using a special stub file, but in Sonoma it simply loses its data, and the distinctive iCloud icon is displayed next to the file in the Finder.

Dataless files can’t be backed up by Time Machine, as without their data there’s nothing to back up. If you want that file to be backed up, then it has to be downloaded or materialised by downloading its data from iCloud. Some third-party backup utilities can manage that automatically for you, but Time Machine doesn’t. If you want to ensure that a file in iCloud Drive is backed up when Optimise Mac Storage is turned on you’ll thus have to download it manually if it has been evicted to iCloud.

Restoring from backup

The simplest way to restore a file or folder in iCloud Drive from a Time Machine backup is through the Time Machine app. Before opening that app, select iCloud Drive in the sidebar of the front Finder window, then the app will take you back through its backups of iCloud Drive.

The main drawback of that is that items you restore will replace any existing items. You may also prefer to use the Finder’s or a command line interface to your backups, both of which allow you to copy restored items where you prefer. The difficulty here is that macOS deliberately hides the path to iCloud Drive: in your active Home folder, they’re located in ~/Library/iCloud Drive, but that path doesn’t work in Time Machine backups.

To access them in a backup, follow the Finder path
[backupname]/[backup].backup/Data/Users/[username]/Library/Mobile Documents/com~apple~CloudDocs/

From Terminal, you’ll have more of a struggle, but need to look for
/Volumes/.timemachine/[UUID]/[backup].backup/[backup].backup/Data/Users/[username]/Library/Mobile\ Documents/com\~apple\~CloudDocs/
where the path component [backup].backup is repeated. If you locate an item first in the Finder, you can always drag and drop that into the command line for it to automatically paste the correct path.

Summary

• Unless you manually exclude iCloud Drive from Time Machine backups, the contents of iCloud Drive will be backed up provided that items haven’t been evicted from local storage.
• If Optimise Mac Storage is turned off, the whole of iCloud Drive will be backed up by Time Machine, as eviction doesn’t occur.
• If Optimise Mac Storage is turned on, only those files and folders that are stored locally will be backed up by Time Machine. To ensure an item is backed up, manually download it before the backup is made.
• Restoring iCloud Drive items from a backup is simplest in the Time Machine app, but also available in the Finder, from a path equivalent to ~/Library/Mobile Documents/com~apple~CloudDocs/ in that backup.