Normal view

There are new articles available, click to refresh the page.
Before yesterdayMain stream

Understand and check Time Machine backups to APFS

By: hoakley
3 October 2024 at 14:30

Over the last 17 years, Time Machine has backed up many millions of Macs, ranging from the last Power Macs to the latest M3 models, every hour. It has changed greatly over that time. This article uses my free utility T2M2 to explain how Time Machine now makes automatic backups to APFS storage, and how to check that. This account is based primarily on what happens in macOS Sonoma and Sequoia, when they’re backing up automatically every hour to local storage.

T2M2 offers three features to see what’s happening with Time Machine and backups:

  • Check Time Machine button, to analyse backups made over the last few hours.
  • Speed button, to view progress reports in the log during a long backup.
  • Browse log button, to show filtered log extracts in fullest detail.

These are each detailed in T2M2’s Help book. Here I’ll concentrate on the first and last, and defer speed checks to that documentation.

Browse log for detail

In practice, you’re most likely to view T2M2’s analysis using the Check Time Machine button first, but here I’ll walk through the greater detail available in log extracts, to build understanding of the sequence of events in each automatic backup.

t2m2rept1

To help you see which subsystems are involved in each stage, T2M2 displays their entries in different colours, and you can show or hide each of those.

Automatic backups are called off by the DAS-CTS dispatching mechanism, whose entries are shown in red (DAS) and blue (CTS). They schedule backups so they don’t occur when the Mac is heavily loaded with other tasks, and call a chain of services to start the backup itself. Dispatching is now reliable over long time periods, but can be delayed or become irregular in some situations. Inspecting those DAS-CTS entries usually reveals the cause.

From there on, most informative log entries are made by Time Machine itself.

Preparations are to:

  • Find each destination backup storage, decide whether any rotation scheme applies, so determine the destination for this backup.
  • Check write performance to the backup destination.
  • Find the machine store on the destination.
  • Determine which local snapshots should be removed, and delete them.
  • Create local snapshot(s) as ‘stable’ snapshot(s), and mount them.
  • Mount previous local snapshot(s) as ‘reference’ snapshot(s).
  • Determine how to compute what needs to be backed up from each source. This should normally use FSEvents to build the EventDatabase.
  • Scan the volumes to be backed up to determine what needs to be backed up.
  • Estimate the total size of the new backup to be created.

Once backing up starts, entries cover:

  • Copying designated items to the destination.
  • Posting periodic progress reports during longer backups.

When that’s complete, closing stages are to:

  • Report details of the backup just completed.
  • Set local snapshots ready for the next backup, with the ‘stable’ snapshot(s) marked as ‘reference’, and unmount local snapshots.
  • Delete working folder used during the previous backup as ‘incomplete’.
  • Create the destination backup snapshot.
  • Delete any old backups due for removal.
  • Report backup success or other outcome.

They’re summarised in this diagram. Although derived from Sonoma, Sequoia brings no substantial change.

tmbackup14a

Or its PDF tear-out version: tmbackup14b

Check Time Machine summaries

t2m2rept2

T2M2 analyses all those log entries to produce a summary of how Time Machine has performed over the last few hours. That is broken down into sections as follows.

Backup destination. This is given with the free space currently available on that volume, followed by the results of write speed measurements made before each backup starts.
Backing up 1 volumes to ThunderBay2 (/dev/disk10s1,TMBackupOptions(rawValue: 257)): /Volumes/ThunderBay2
Current free space on backup volumes:
✅ /Volumes/ThunderBay2 = 1.67 TB
Destination IO performance measured:
Wrote 1 50 MB file at 286.74 MB/s to "/Volumes/ThunderBay2" in 0.174 seconds
Concurrently wrote 500 4 KB files at 23.17 MB/s to "/Volumes/ThunderBay2" in 0.088 seconds

Backup summary. This should be self-evident.
Started 4 auto backup cycles, and 0 manual backups;
completed 4 volume backups successfully,
last backup completed successfully 53.4 minutes ago,
Times taken for each auto backup were 0.3, 0.3, 0.2, 0.2 minutes,
intervals between the start of each auto backup were 61.4, 60.2, 60.3 minutes.

Backups created and deleted (or ‘thinned’).
Created 4 new backups
Thinned:
Thinning 1 backups using age-based thinning, expected free space: 1.67 TB actual free space: 1.67 TB trigger 50 GB thin 83.33 GB dates: (
"2024-10-01-043437")

Local snapshots created and deleted.
Created 4 new snapshots, and deleted 4 old snapshots.

How the items to be backed up were determined. This shows which methods Time Machine used to work out which items needed to backed up, and should normally be FSEvents, once a first full backup has been made.
Of 4 volume backups:
0 were full first backups,
0 were deep scans,
4 used FSEvents,
0 used snapshot diffs,
0 used consistency scans,
0 used cached events.

Backup results. For each completed backup, Time Machine’s report on how many items were added, and their size.
Finished copying from volume "External1"
1 Total Items Added (l: Zero KB p: Zero KB)
3 Total Items Propagated (shallow) (l: Zero KB p: Zero KB)
403274 Total Items Propagated (recursive) (l: 224.93 GB p: 219.31 GB)
403275 Total Items in Backup (l: 224.93 GB p: 219.31 GB)
1 Directories Copied (l: Zero KB p: Zero KB)
2 Files Move Skipped (l: Zero KB p: Zero KB) | 2 items propagated (l: 6 KB p: 12 KB)
1 Directories Move Skipped (l: Zero KB p: Zero KB) | 403269 items propagated (l: 224.93 GB p: 219.31 GB)

Error messages.
✅ No error messages found.

iCloud Drive and pinning

Backing up the contents of iCloud Drive is a longstanding problem for Time Machine, if Optimise Mac Storage is enabled. Those files in iCloud Drive that are stored locally should be backed up, but any that have been evicted from local storage could only be included if they were to be downloaded prior to the backup starting.

In Sonoma and earlier, the only way to ensure that an evicted file in iCloud Drive is included in a Time Machine backup is to manually download it. Sequoia now lets you ‘pin’ individual files and whole folders so that they aren’t evicted, so will always be included in Time Machine backups. This is explained here.

Discrepancies and glitches

T2M2 tries to make sense from log entries that don’t always behave as expected. As backups can be complex, there are situations when T2M2 may report something that doesn’t quite add up. This most commonly occurs when there have been manual backups, or a third-party app has been used to control the scheduling and dispatch of backups. If you do see figures that don’t appear quite right, don’t assume that there’s something wrong with Time Machine or your backups. Generally, these glitches disappear from later automatic backups, so you might like to leave it for a few hours before checking Time Machine again to see if the problem has persisted.

Further reading

What performance should you get from different types of storage?
Where should you back up to?
How big a backup store do you need?
How Time Machine backs up iCloud Drive in Sonoma
Time Machine backing up different file systems
Excluding folders and files from Time Machine, Spotlight, and iCloud Drive
Snapshots aren’t backups
Time Machine in Sonoma: strengths and weaknesses
Time Machine in Sonoma: how to work around its weaknesses
Time Machine in Sonoma: Rotating backups and NAS
A brief history of Time Machine

What should you do when an update goes wrong?

By: hoakley
20 September 2024 at 14:30

Even the smallest of updates to macOS or its security components can leave your Mac in a mess. Once those feelings of panic are subsiding, what should you do next?

Boot loop

There’s one emergency situation that happens on rare occasions after a failed firmware update: a boot loop, in which the Mac tries to start up, hits a kernel panic early, so tries to restart again, and continues in that loop. If that happens, press and hold the Power button until it shuts down, and refer to this article.

Restart, Safe mode

Otherwise the first thing to try is simply restarting your Mac normally. If that proves a problem, or things are still awry, try Safe mode. On an Apple silicon Mac, shut it down, wait ten seconds or so, then start it up in Recovery, select the disk to use for Safe mode, hold the Shift key, and click Continue in Safe Mode.

recovery02

On an Intel Mac, hold the Shift key during startup.

Once running, leave your Mac for a couple of minutes, then restart in normal mode, with your fingers tightly crossed.

Sometimes Safe mode works fine, but as soon as you return to normal mode everything goes wrong again. That’s a good indicator that something you have installed is at fault, rather than the macOS update, although it’s normally a combination of both working against one another. You now need to hunt down the third-party software that has become upset by the update, and either update or remove it.

If Safe mode either doesn’t help, or you can’t even enter it, then your Mac’s problems could be from the macOS update itself or something third-party, and teasing them apart isn’t going to be easy. This is when you should, for the first time, ask yourself whether you want to return to your previous version of macOS without the update, or try to fix what you’ve got. You can change your mind later, but this is a key question that determines what you do next.

Reinstall

Until Big Sur, the next step for those wanting to stay with the update was to install it differently, most commonly using the Combo update, because that contained all the changed components since the first release of that version of macOS, but was smaller and simpler than a full installer. Because of the way that macOS is now installed into a Signed System Volume (SSV), actually a snapshot, this is no longer available, and the only alternative beyond the update that brought your Mac’s problems is a full installer.

Before going any further, bear in mind that macOS booted from an SSV is very different to that of the past. Macs with T2 and especially Apple silicon chips verify the contents of the SSV down to the last bit, and they’re checked against what Apple considers to be a ‘perfect install’. So the incomplete or broken updates of the past simply can’t happen with an SSV, which is guaranteed to be perfect as Apple intended. Installing another identical copy of that is therefore most unlikely to change anything.

Simplest is to start up in Recovery mode and re-install the current version of macOS, which should by now be the version you’re trying to run. When you do that, ensure you install to your current Volume Group so that the existing Data volume is connected up with the fresh System volume. Because that may not always work, before starting this journey, ensure you’ve got a full copy of your Data volume. Carbon Copy Cloner is an excellent tool for doing that if you don’t already have a full Time Machine backup.

There are variations too, although these bring greater risk that your current Data volume will get trashed or ignored. You can download the latest installer app from the App Store, or in Terminal, which gives you a more precise choice that’s essential should you want to downgrade. The following command lists available macOS installers:
softwareupdate --list-full-installers

Currently, that list includes for Intel Macs:

  • Sequoia 15.0,
  • Sonoma 14.7, 14.6.1, 14.6, 14.4.1,
  • Ventura 13.7, 13.6.9, 13.6.8, 13.6.6,
  • Monterey 12.7.6, 12.7.4,
  • Big Sur 11.7.10,
  • Catalina 10.15.7, 10.15.6,
  • Mojave 10.14.6,
  • High Sierra 10.13.6.

You can then use a command like
sudo softwareupdate --fetch-full-installer --full-installer-version 11.7.7
to download the installer of your choice.

For an even wider choice, visit a site such as Mr. Macintosh.

This article explores other options in detail.

If you install macOS afresh with your existing Data volume and the problems persist, then it’s most likely they result not from any error in the system update, but in third-party software. You can then rip out all third-party extensions and anything from your apps that persists, until you find the offender.

Revert to previous

Returning to a previous version of macOS isn’t an easy option after you’ve performed a macOS update. There’s no secret Roll Back button, and you’ll have to perform a fresh install of that older version of macOS, ensuring that hitches up with your existing Data volume. If the latter part doesn’t work, be prepared to migrate from a backup into a fresh Data volume.

Intel Macs have one significant limitation here: firmware. While you can roll the system back, you can’t return an Intel Mac’s firmware to a previous version. So if the problems you’re encountering are firmware-related, your Intel Mac is out of luck.

Apple silicon Macs can readily be reverted to older firmware, by putting them into DFU mode and restoring an older version of macOS complete with its firmware, using Apple Configurator 2. You’ll need another Mac to run that free app (from the App Store), and the first time you do this is daunting, but it’s a valuable feature that can recover from apparent disaster.

Apple Support

If your Mac suffers any significant problems after a macOS update, don’t be afraid to contact Apple Support. Sometimes updates have serious failings with specific models, and only Apple Support is likely to discover this, and offer a way forward. Otherwise, I wish you success diagnosing and fixing your problems.

Summary

  • If in a boot loop, press and hold the Power button to shut down.
  • Otherwise restart the Mac.
  • If necessary, start up in Safe mode, wait a couple of minutes, and restart in normal mode.
  • Consider reinstalling that version of macOS.
  • Reverting to a previous version is a slow and hard process.
  • Consider contacting Apple Support.

What to do when your Mac can’t update

By: hoakley
19 September 2024 at 14:30

macOS and its smaller security updates are widely announced, here and across many other sites supporting Apple products. What should you do, though, when you know updates have been released by Apple but your Mac can’t find them, or when it tries to install them and fails?

Before Sequoia, almost all software updates are normally fetched by Software Update in System Settings/Preferences, or alternatively at the command line by softwareupdate, also used by my free SilentKnight. They work through the softwareupdated service that should be running in the background. If you run a local Content Caching server, then softwareupdated should automatically connect to that and ask it for the update; otherwise, it tries to connect to Apple’s software update servers over the internet. Although this chain is usually reliable, it has several points of weakness.

Sequoia brings greater complexity, in that one its most important security data updates for XProtect is intended to be delivered over a CloudKit connection with iCloud, although those updates can still arrive from Software Update as well. However, when delivered through softwareupdated the XProtect bundle is installed but not ‘activated’ for XProtect’s use. To do that, open Terminal and enter the command
sudo xprotect update
then authenticate with your admin password when prompted.

Update not found

You open Software Update or SilentKnight, and are told that your Mac is up to date, although it’s still running the older version of macOS, or hasn’t installed a smaller security update.

The most likely reasons for this include:

  • Apple’s software update servers are in heavy demand, and are temporarily refusing new connections. As Apple tends to release a lot of updates at once, this isn’t uncommon, particularly in the autumn/fall with the new versions of macOS and others. The only solution is to try again later, although sometimes you can kickstart the process by running SilentKnight or softwareupdate. Apple provides a page showing the status of its many internet services, where these are listed as macOS Software Update, but transient problems due to load seldom get reported there.
  • Your Mac, or its Content Caching server if you’re running one, can’t connect to Apple’s servers because of a network fault. Again the only solution is to try again later, in the hope that the fault has been fixed.
  • softwareupdated or your Content Caching server aren’t working properly. This is normally rectified by restarting that Mac and trying again once it’s up and running. In some cases, it can require that the client Mac is started up in Safe mode before the update becomes available.

If an update has only just been announced, then the software update servers that your Mac connects to may not be offering that update yet. Availability around the world isn’t instant, and often you’ll find that an Apple silicon Mac can find an update and install it readily, while an Intel Mac on the same network may be unable to discover the same update for another hour or more.

Note that, unless an update is listed as being available, you can’t force it by trying to install the update using its label, either in softwareupdate or SilentKnight.

For updates to XProtect in Sequoia, try opening Terminal and entering the command
sudo xprotect check
then authenticating with your admin password when prompted. This should force XProtect management to look for an update. If it finds one, then entering
sudo xprotect update
should download it from iCloud and install it. Note that this command is only available in Sequoia. For further information, man xprotect tells you as much as Apple lets you know.

Update fails to install

This is easiest to detect when you use SilentKnight, which will report the update is available, then when you try to install it, you’ll see an error message in the scrolling text window reporting that installation of the update failed, and the component being updated won’t change to the new version number.

If the Software Update pane shows an error, that should provide similar information. Otherwise, to download and install waiting updates you can type
softwareupdate -ia --include-config-data (or in El Capitan sudo softwareupdate -ia)
in Terminal, to see the same messages shown by SilentKnight, as that’s also the tool it uses to obtain waiting updates. If you know your way around the Unified Log, you should discover parallel entries there.

By far the most common cause for failure to install updates like this is that something has gone wrong with softwareupdate or softwareupdated, best corrected by restarting your Mac and trying again. If it still doesn’t work, start up in Safe mode and try from there. One of the primary purposes of Safe mode is to resolve problems with updates and updating, whether they’re full macOS updates or small security data updates like XProtect.

If you’re not running a local Content Caching server and still can’t get the update to install, all you can do is wait an hour or two and try again.

Content Caching problems

If you’re running a local Content Caching server, then the problem could now rest with the copy of the update stored in its cache. When the local server downloaded the update from Apple’s software update servers, it may have become damaged. Once that damaged copy has been put into your local server’s cache, that’s the update that it will serve to all your local Macs when they connect to it to obtain the update.

What can make this worse is that, even if you do manage to get the Mac running the Content Caching server to update successfully, that doesn’t mean that it will replace the damaged copy in its cache, which may continue to deliver that same damaged version to all the Macs that try connecting to it.

To confirm this, you can inspect the log, as I’ve described here.

The most immediate solution, which should allow all your local systems to update correctly, is to turn the Content Caching service off in Sharing, shut down the Content Caching server, or isolate that server from the rest of the network. Then update all your other systems, which should download fresh copies of the update directly from Apple’s servers. Once that’s done, you can bring the server back up in Safe mode and try updating it there.

For a period of over six months in 2022-23, updates for XProtect and XProtect Remediator obtained through Content Caching servers frequently failed to install correctly. In that time, the simplest solution was to disable the server before trying to download and install those updates, and to enable it again once all updates had been completed. It’s still not clear where that problem occurred, but it has since been fixed and updates should be reliable now.

I don’t know any way to remove individual updates from the Content Caching server. Apple’s command tool for its maintenance, AssetCacheManagerUtil, only knows how to flush whole caches, using
sudo AssetCacheManagerUtil [flushCache|flushPersonalCache|flushSharedCache]
where the commands set the cache to be flushed:

  • flushCache flushes the entire content cache.
  • flushPersonalCache flushes all personal (iCloud) content.
  • flushSharedCache flushes all shared (non-iCloud) content.

Flushing a large cache may not be what you want to do. So long as there’s no storage problem and the update affected was most probably supplied broken, there shouldn’t be any harm in leaving it where it is.

In Sequoia, XProtect’s new updates delivered from iCloud are likely to bypass Content Caching servers altogether, although Apple hasn’t clarified that yet.

Nothing helps

If you’ve worked your way through to the end here but still haven’t solved the problem, contact Apple Support, who can escalate it to someone who can hopefully do something about the problem.

Further reading

Repeated installations of the same updates
How security data updates should work

❌
❌