Apple has just released an update to XProtect for all supported versions of macOS, bringing it to version 5295. As usual, Apple doesn’t release information about what security issues this update might add or change.
This version adds a single new rule for MACOS.SOMA.BYTE.SEQUENCE.B.
You can check whether this update has been installed by opening System Information via About This Mac, and selecting the Installations item under Software.
A full listing of security data file versions is given by SilentKnight, LockRattler and SystHist for El Capitan to Sequoia available from their product page. If your Mac hasn’t yet installed this update, you can force it using SilentKnight, LockRattler, or at the command line.
If you want to install this as a named update in SilentKnight, its label is XProtectPlistConfigData_10_15-5295.
Sequoia systems only
This update has also been released for Sequoia via iCloud. If you want to check that manually, use the Terminal command sudo xprotect check
then enter your admin password. If that returns version 5295 but your Mac still reports an older version is installed, you can force the update using sudo xprotect update
I have updated the reference pages here which are accessed directly from LockRattler 4.2 and later using its Check blog button.
I hope that you enjoyed Saturday’s Mac Riddles, episode 303. Here are my solutions to them.
1: Harvard or Yale cipher brought emoji and Ugaritic.
Click for a solution
Unicode
Harvard or Yale (a university or uni) cipher (a code) brought emoji and Ugaritic (two of its supported character sets).
2: US standard for 128 Roman characters now over 60.
Click for a solution
ASCII
US standard (initially ASA X3.4-1963, later an ANSI standard) for 128 Roman characters (originally consisted of only 128 characters including a basic Roman alphabet) now over 60 (first published in 1963, it turns 62 years old this year).
3: Two plus 128 more came in 1989, gained a euro in 1998, and still supported.
Click for a solution
MacRoman
Two plus 128 more (it consists of the 128 characters in ASCII, plus 128 more including punctuation, symbols and diacritics) came in 1989 (first appeared in System 6.0.4 in that year), gained a euro in 1998 (the only change made since introduction), and still supported (it is, although now encoded in UTF-8).
The common factor
Click for a solution
They are text encodings that have been used in Mac OS.
For many years, most types of disk image were inefficient in their use of storage space, as they occupied their full size on disk. Until recently, when you created a 5 GB read-write UDIF disk image, one of the most popular, it invariably took up 5 GB in storage, even when empty. This also applied to the raw disk images used by Virtual Machines: give a VM 100 GB, and that’s just what it took on disk. With the introduction of sparse files in APFS, this has changed, and many disk images now only take the space they need. I’m not sure exactly when this change occurred, as Apple still doesn’t appear to have documented it, but it seems to have changed with macOS Monterey.
This is easiest to see with a plain read-write disk image, created using DropDMG or Disk Utility.
Disk image
Here’s one I made earlier, a whole 350 GB in size. When it’s created, it’s automatically attached and mounted at full size. For the sake of example, I then copied a large IPSW to it, so it wasn’t entirely empty.
Unmount it and Get Info on the disk image and you’ll see it does still take up a full 350 GB on disk. Mount it again, though, and APFS works its magic. You can see this in LogUI, or the custom log extract provided by Mints.
When unmounted again it has shrunk down to take little more than the size of the IPSW file in it, at just over 17 GB. That’s less than 5% of its nominal size, without using any compression.
It’s worth looking through entries in the log made by APFS for the mount process. First, APFS checks whether the data store for the disk image is already sparse: 01.470 container_backingstore_is_sparse:1652: Image url file:///Volumes/LaCie2tb/350gbudif.dmg Image path /Volumes/LaCie2tb/350gbudif.dmg
01.470 container_backingstore_is_sparse:1659: Image /Volumes/LaCie2tb/350gbudif.dmg is a flat file, do not consider as sparse
It then sets it to sparse, ready for sparsification: 01.475 handle_apfs_set_backingstore:6207: disk9s1 Set backing store as sparse
01.475 handle_apfs_set_backingstore:6240: disk9 Backing storage is a raw file
Space Manager performs an initial scan for free blocks without any Trimming: 01.479 spaceman_scan_free_blocks:4136: disk9 scan took 0.004272 s (no trims)
01.479 spaceman_fxc_print_stats:477: disk9 dev 0 smfree 81258479/85398014 table 4/452 blocks 81258479 32766:20314619:79974226 100.00% range 35869:85362145 99.95% scans 1
Space Manager then scans and Trims free storage blocks, taking just over 0.7 second to complete: 02.196 spaceman_scan_free_blocks:4106: disk9 scan took 0.717433 s, trims took 0.715705 s
02.196 spaceman_scan_free_blocks:4110: disk9 81258479 blocks free in 25 extents, avg 3250339.16
02.196 spaceman_scan_free_blocks:4119: disk9 81258479 blocks trimmed in 25 extents (28628 us/trim, 34 trims/s)
02.196 spaceman_scan_free_blocks:4122: disk9 trim distribution 1:0 2+:0 4+:0 16+:0 64+:0 256+:25
VM
What happens with an Apple silicon VM is a bit more complicated, and harder to observe. This time the virtualisation app should create the disk image inside the VM bundle as a sparse file to begin with, then copy into that what’s needed for the VM, so skipping the first mount stage and Trimming during the second mount.
The result is the same, though, with a 350 GB VM taking just 22 GB on disk. Inspect that disk image using my free utility Precize, and you’ll see that economy confirmed, and the Sparse File flag set.
Conclusions
For plain read-write disk images and those inside VMs to be sparse files:
they must contain a suitable raw disk image, such as UDIF read-write;
the host file system must be APFS, as HFS+ doesn’t support sparse files;
for normal disk images, they must be stored on an SSD that supports Trimming;
there must be sufficient free space in the disk image;
the guest file system can be APFS, either plain or encrypted, or HFS+J;
for normal disk images, they must have been mounted at least once since first being created.
Apple has just released an update to XProtect for all supported versions of macOS, bringing it to version 5293. As usual, Apple doesn’t release information about what security issues this update might add or change.
This version adds a single new rule for MACOS.SOMA.J.
You can check whether this update has been installed by opening System Information via About This Mac, and selecting the Installations item under Software.
A full listing of security data file versions is given by SilentKnight, LockRattler and SystHist for El Capitan to Sequoia available from their product page. If your Mac hasn’t yet installed this update, you can force it using SilentKnight, LockRattler, or at the command line.
If you want to install this as a named update in SilentKnight, its label is XProtectPlistConfigData_10_15-5293.
Sequoia systems only
This update has also been released for Sequoia via iCloud. If you want to check that manually, use the Terminal command sudo xprotect check
then enter your admin password. If that returns version 5293 but your Mac still reports an older version is installed, you can force the update using sudo xprotect update
I have updated the reference pages here which are accessed directly from LockRattler 4.2 and later using its Check blog button.
I hope that you enjoyed Saturday’s Mac Riddles, episode 302. Here are my solutions to them.
1: Shortened characters into the most common extension, formerly ASCII.
Click for a solution
txt
Shortened characters (text, shortened) into the most common extension (it is), formerly ASCII (it used to be).
2: Medical practitioner at the end of word files until gaining a cross in 2002.
Click for a solution
doc
Medical practitioner (a doc) at the end of word files (the extension for Word native format) until gaining a cross in 2002 (progressively replaced by the newer docx from 2002 onwards).
3: At the end of real estate inventory, most commonly for Info and preferences.
Click for a solution
plist
At the end (a filename extension) of real estate (property) inventory (list), most commonly for Info (Info.plist in bundles) and preferences (also usually property lists).
Apple has just released an update to XProtect for all supported versions of macOS, bringing it to version 5292. As usual, Apple doesn’t release information about what security issues this update might add or change.
This version removes the macos_toydrop_b rule for MACOS.ADLOAD, and amends the rules for MACOS.ADLOAD.I, MACOS.BUNDLORE.MDPLST and MACOS.ADLOAD.IN.
You can check whether this update has been installed by opening System Information via About This Mac, and selecting the Installations item under Software.
A full listing of security data file versions is given by SilentKnight, LockRattler and SystHist for El Capitan to Sequoia available from their product page. If your Mac hasn’t yet installed this update, you can force it using SilentKnight, LockRattler, or at the command line.
If you want to install this as a named update in SilentKnight, its label is XProtectPlistConfigData_10_15-5292.
Sequoia systems only
This update has now been released for Sequoia via iCloud. If you want to check that manually, use the Terminal command sudo xprotect check
then enter your admin password. If that returns version 5292 but your Mac still reports an older version is installed, you can force the update using sudo xprotect update
I have updated the reference pages here which are accessed directly from LockRattler 4.2 and later using its Check blog button.
Most modern Macs start up from their internal SSD, and have a single bootable system installed. Although its structure has changed considerably over recent years, and differs between Intel and Apple silicon Macs, they are generally reliable, and knowledge of their structure and function is seldom required.
Those who need to start their Mac up from two or more versions of macOS, or who do so from an external disk, may be able to get by without deeper understanding, but the moment there’s a problem can become confused, and end up having to install macOS from scratch.
Intel T2 and Apple silicon
Both types of Mac are designed to support Secure Boot, but because of the reliance of Intel Macs on UEFI firmware, they can only boot securely from their internal SSD. Enabling an Intel T2 Mac to start up from an external disk thus requires its boot security to be reduced by changing that in Startup Security Utility, in Recovery mode. Booting from an external system is then straightforward, and doesn’t require additional security measures as used in Apple silicon.
Apple silicon Macs have been designed from the outset to support Secure Boot when starting up from both internal and external disks, and don’t require any reduction in their boot security to be able to start up from multiple systems on external SSDs. It’s a common misunderstanding that trying to change Boot Security in Startup Security Utility can help solve Apple silicon boot problems, but if anything it only complicates them. Almost the only good reason for reducing boot security of an Apple silicon bootable system is when third-party kernel extensions are required. Otherwise don’t tamper with Startup Security Utility, as it will only confuse, as we’ll see later.
For an Apple silicon Mac to boot from a macOS system, that must have a LocalPolicy created and saved to the internal SSD. LocalPolicy is normally created during macOS installation, or when intending to start up from a suitably installed system. Problems with LocalPolicy have been common when using external disks, and are covered in these articles:
Another significant difference between Intel and Apple silicon Macs is the architecture of their internal SSDs: Apple silicon has two additional partitions (APFS containers), and lacks the traditional EFI partition. However, there are no such differences in the structure of their bootable external disks, and its relevance here is limited, and mainly affects Big Sur as I explain below.
macOS 10.13-10.15
Bootable disks in versions of macOS between High Sierra and Mojave (above) are based on their single boot volume, supplemented by hidden Preboot, Recovery and VM volumes. macOS 10.15 Catalina (below) divided that boot volume into a Boot Volume Group, consisting of paired System and Data volumes, in addition to the same three hidden volumes.
Basic support for volume roles was introduced in the first release version of APFS in macOS 10.13, and was extended to cover further roles in 10.15. Thus versions of APFS prior to 1412.x.x don’t understand volume roles used by subsequent systems. From Catalina onwards you can use the command diskutil apfs listVolumeGroups
to see paired System and Data volumes, for example +-- Container disk5 5BA1AAC8-3AD4-4594-AF01-7C0AA75CABAD
| |
| +-> Volume Group 68627CBB-D774-444E-97C6-F9511B5030F3
| =================================================
| APFS Volume Disk (Role): disk5s1 (Data)
| Name: Macintosh HD - Data
| Volume UUID: 68627CBB-D774-444E-97C6-F9511B5030F3
| Capacity Consumed: 580769214464 B (580.8 GB)
| -------------------------------------------------
| APFS Volume Disk (Role): disk5s5 (System)
| Name: Macintosh HD
| Volume UUID: 8B9FF440-75C8-4F7F-B09E-9222D44A2276
| Capacity Consumed: 11239186432 B (11.2 GB)
Note that each volume group has its own UUID, which is normally the same as that of the Data volume in the pair. Identification of volumes, containers and other structures in APFS is dependent on these UUIDs, which are an essential part of the GUID Partition Table scheme used to partition disks.
Catalina’s System volume is mounted read-only, but macOS is booted from that volume rather than the Signed System Volume snapshot introduced in Big Sur.
macOS 11
Although the structure of Big Sur internal SSDs in Apple silicon Macs has two additional partitions (APFS containers), Apple_APFS_ISC containing three volumes, and Apple_APFS_Recovery containing two volumes, bootable external disks have the same structure as the internal SSD in Intel Macs.
One important functional difference, which remains relevant to Big Sur boot disks, is that Apple silicon Macs don’t use the paired Recovery volume as their primary Recovery system: booting an Apple silicon Mac running Big Sur into Recovery should instead use the Recovery system installed in their internal SSD, in the Apple_APFS_Recovery partition. In subsequent versions of macOS, that’s used instead for secondary or Fallback Recovery. Thus Big Sur can be a problem when it comes to Recovery, and for this reason is best avoided on Apple silicon Macs. If it’s essential to install a copy of Big Sur, then be prepared for problems with Recovery mode.
macOS 11 also established the architecture for dual-boot partitions, with two or more Boot Volume Groups.
Although the Boot Volume Group has only ever referred to paired System and Data volumes, within each partition/container the group also requires three or four additional volumes, for Preboot, Recovery, VM and (apparently in Big Sur only) Update. How those work with multiple Boot Volume Groups is explained below. Once way to avoid the inevitable complexities is to install each Boot Volume Group into a separate partition/container, which provides each with its own suite of Preboot, Recovery and VM volumes.
As the Signed System Volume (SSV) was introduced in macOS 11, versions of APFS prior to 1677.x.x shouldn’t be expected to understand SSVs.
macOS 12-15
The next significant architectural changes in Apple silicon Macs were the introduction of paired Recovery volumes in macOS 12 Monterey, and of cryptexes in macOS 13 Ventura.
In macOS 11 and 12, Safari, its supporting components, and some other parts of macOS are installed to the Data volume for ease of maintenance. Apple replaced that with separate secure disk images termed cryptexes, loaded from the Preboot volume during the boot process and grafted into the root file system. As a single Preboot volume can be shared by more than one Boot Volume Group, bundled cryptexes must be provided to the correct group, and I suspect that’s accomplished by associating them with their Volume Group UUID. If that becomes confused in any way, cryptexes could be incorrectly associated or missing altogether, something that’s almost certainly fatal to the boot process.
Above is the current layout of a partition/container containing a single bootable system for a Mac running Sequoia. Device names given are illustrative, although their numbering is typical of that seen for external bootable volumes. In this instance, the base name for the Boot Volume Group is External, and the External System volume is unmounted as usual.
Below is a typical partition/container containing two bootable systems named ExternalA and ExternalB. This has two Boot Volume Groups, each consisting of a System volume with its SSV, and a Data volume, to which are added their three common volumes, Preboot, Recovery and VM.
Management
Creation
The most reliable tools for creating and working with Boot Volume Groups and other system components are those in macOS installer apps, and even they can have their moments and bugs. It’s usually possible to ‘clone’ groups using the asr command tool, a feature that’s offered in some third-party utilities including Carbon Copy Cloner and SuperDuper. However, Apple has made it clear that given a choice between supporting asr and addressing boot security, it gives the latter absolute priority. asr has suffered some serious bugs since the SSV was introduced in Big Sur, and shouldn’t be relied on.
asr is careful to ensure that, when cloning Boot Volume Groups, UUIDs are changed so that it doesn’t end up with volumes or groups with the same UUID. That may not be the case when using some other tools such as dd for duplication. If you prefer a simple and stress-free life, it’s better to put your trust in a macOS installer rather than try crafting your own Boot Volume Groups.
Apple’s own tools such as Disk Utility now try to steer the user away from making mistakes, such as deleting just one of a Boot Volume Group, leaving the other orphaned with its firmlinks broken.
Firmware
Firmware is another source of confusion. That installed in the Mac, using its internal storage, should always reflect the firmware that has been included in the most recent version of macOS installed or updated on that Mac, whether that was installed to the internal SSD or to an external disk. The only exception to this is when installing or updating macOS in a Virtual Machine, which can’t affect the host’s firmware.
What may appear more puzzling are the versions reported in System Information: that given as System Firmware should be the iBoot version for the main Mac, and the most recent. The OS Loader, though, varies with the Boot Volume Group, and in older macOS is likely to be earlier than the iBoot version of the main Mac.
Recovery
Recovery system versions are even more complex. When everything works as it should, the version installed in any paired Recovery volume should be the newest of the Boot Volume Groups that it’s paired with, in the same partition/container. If a Mac running 15.3 from its internal SSD has a bootable external disk with a single container with two Boot Volume Groups for 13.7 and 14.7, then the Recovery system for the internal SSD should be 15.3, while that shared by the two systems on the external disk should be 14.7. It’s likely that Fallback Recovery on the internal SSD will be from an earlier version of macOS 15, but that’s less predictable, and there’s no separate Fallback Recovery in external systems.
Switching between systems and their Boot Volume Groups is normally performed using Startup Disk in General settings, or its equivalent in System Preferences. Those record the chosen Boot Volume Group in NVRAM, from where it’s used during the next boot.
That setting is used to determine which Recovery system is used if the next boot is performed with the Power button pressed to enter Recovery mode. That in turn determines what can be performed in Recovery. This is most critical for determining how Startup Security Utility works, as that can only change boot security settings for the chosen Boot Volume Group saved in NVRAM.
For example, in the dual-boot container shown above, selecting ExternalB as the Startup Disk, shutting down, then starting up into Recovery mode should enable Startup Security Utility to control boot security for ExternalB but not ExternalA, nor the system installed on the internal SSD. This can be used to discover which Recovery system is running: open Startup Security Utility and the only boot system that can be changed there is the current default boot system.
APFS
Although APFS should be backward compatible, making it relatively safe to make changes to an older version of APFS from a newer system, forward compatibility is more limited. Using older versions of Disk Utility or tools like fsck on newer versions of APFS risks errors, failure and at worst damage. The Appendix at the end of this article summarises version numbering in APFS and major changes to beware of. Although not easy to discover the version of APFS used to create or modify any given volume, one way is to use fsck_apfs -n -S [device]
giving the volume’s device name. The report should then be prefaced by a statement such as The volume [volume name] was formatted by diskmanagementd (1412.141.1) and last modified by apfs_kext (945.250.134).
telling you that volume was created by macOS 10.15, and was last changed by macOS 10.14.
Troubleshooting
The first and fundamental step in trying to diagnose the cause of problems with multiple Boot Volume Groups or bootable external disks is to examine their container and volume structure using two diskutil commands: diskutil apfs list
lists all APFS volumes by container and gives key information about each, including role and UUID, and diskutil apfs listVolumeGroups
lists all recognised Boot Volume Groups, which you can tally against the first. Pipe these to text files so you can study and refer back to them.
Those should enable you to verify that the structure is as intended, and to establish the relationships between systems and paired Recovery volumes. From there you should be able to focus on the Boot Volume Group that’s misbehaving, ensuring that its Data volume has been backed up. If necessary, that can be used as a migration source if that group needs to be deleted and reinstalled.
Tips & tricks
The host Mac must be capable of running that version of macOS.
macOS installers are the most reliable means of creating and installing Boot Volume Groups.
During macOS installation, an external disk must be connected to a port other than the DFU port, as listed here and in Mactracker.
When installing an older major version of macOS, perform this from an external bootable HFS+ volume as detailed by Apple.
Use an HFS+J partition on an external SSD rather than a USB ‘thumb’ drive.
Boot from an installer volume through Recovery mode.
When using a laptop Mac, run it from mains power throughout macOS installation.
Apple silicon Macs boot from external disks in Full Security, and reducing that doesn’t solve any problems.
Each container with one or more Boot Volume Groups contains one set of Preboot, Recovery and VM volumes shared between them.
If you’re unsure which Recovery system you’re using, open Startup Security Utility, as the only group it can change settings for is the one it’s booted into.
Big Sur doesn’t support paired Recovery on Apple silicon, and that can cause problems.
The version of Recovery installed in any paired Recovery volume should be that of the newest of the Boot Volume Groups that it’s paired with.
Use fsck_apfs -n -S [device] to discover the APFS version.
Use diskutil apfs list and diskutil apfs listVolumeGroups to discover volume structure.
Try using a Virtual Machine instead, if you don’t need to be able to run software from the App Store.
Appendix: APFS version numbers
APFS major version numbers change with major version of macOS:
macOS 10.12 has APFS version 0.3 or 249.x.x, which shouldn’t be used at all.
10.13 has 748.x.x, which doesn’t support Fusion Drives, but has basic support for volume roles.
10.14 has 945.x.x, the first version to support Fusion Drives.
10.15 has 1412.x.x, the first version to support the multi-volume boot group, and introduces extended support for volume roles, including Data, Backup and Prelogin.
11 has 1677.x.x, the first version to support the SSV, and Apple silicon. On M1 Macs, it doesn’t support the paired Recovery volume.
12 has 1933.x.x until 12.2, thereafter 1934.x.x, which support the paired Recovery volume on Apple silicon.
13 has 2142.x.x, and is probably the first to support trimming of UDRW disk images and their storage as sparse files.
14 has 2235.x.x, until 14.3, thereafter 2236.x.x.
15 has 2313.x.x until 15.2, thereafter 2317.x.x until 15.4, thereafter 2332.x.x.
One of the magic tricks that characterised the Mac was its association between documents and their apps. No longer did a user have to type in both the name of the app and the document they wanted it to edit. All they needed to do was double-click the document, and it magically opened in the right app.
In Classic Mac OS, that was accomplished by hidden Desktop databases and type and creator codes. For example, a text document might have the type TEXT and a creator code of ttxt. When you double-clicked on that, the Finder looked up which app had the creator code ttxt, which turned out to be the SimpleText editor, and opened that document using that app.
Although those ancient type and creator codes still live on today in modern macOS, they no longer fulfil that role. Instead, each file has what used to be a Uniform Type Indicator (UTI), now wrapped into a UTType, such as public.plain-text, normally determined by the extension to its name, .txt or .text. When you double-click on a file, LaunchServices looks up that UTType in its registry, discovers which app is set as the default to open documents of that type, then launches that app with an AppleEvent to open the document you picked.
Recognising that we often want to open a document using a different app rather than the default, the Finder’s contextual menu offers a list of suitable apps in its Open With command. That list is built and maintained by LaunchServices, and has changed in recent versions of macOS. Whereas those lists used to consist of apps installed in the traditional Application folders, LaunchServices now scours every accessible volume and folder using Spotlight’s indexes to build the biggest lists possible. If you happen to have an old copy of an app tucked away in a dusty corner, LaunchServices will find it and proudly display it alongside those in everyday use, like a game dog triumphantly presenting not one dead pheasant but every one from miles around.
For those with lean systems, this gives them the flexibility to open a large text document using BBEdit rather than TextEdit, or to select which image editor to use for a JPEG. But for those of us with lots of apps lurking in storage, the result is absurd and almost unusable. It’s bad enough working through the 33 apps that LaunchServices lists as PNG editors, but being offered 70 text editors is beyond a joke.
Unfortunately, there’s no lasting way to block unwanted apps from being added to the list LaunchServices builds for this Open With feature. You can gain temporary relief by excluding them from Spotlight search, but should you ever open the folder they’re in using the Finder, those are all added back. This also afflicts apps in folders shared with a Virtual Machine, where the list includes App Store apps that can’t even be run from within that VM.
There are, of course, alternatives. I could drag and drop the document from its Finder window towards the top of my 27-inch display to the app’s icon in the Dock at the foot, which is marginally less awkward than negotiating my way through that list of 70 apps.
But there are better solutions: why not empower me to determine which of those 70 apps should be offered in the Open With list? This is such a radical idea that it used to be possible with the lsregister command that has become progressively impotent, as LaunchServices has cast its net further in quest of more apps to flood me with. Or maybe use a little machine learning to include only those text editors I use most frequently to open documents? Apple could even brand that LaunchServices Intelligence, although that’s a little overstated.
I can’t help but think of what those magicians from forty years ago would have done, but I’m certain they wouldn’t have offered me that list of 70 apps to choose from.
Apple has just released an update to XProtect for all supported versions of macOS, bringing it to version 5291. As usual, Apple doesn’t release information about what security issues this update might add or change.
This version amends the Yara rule for MACOS.PIRRIT.OBF.DROPPER, but doesn’t add any new rules.
You can check whether this update has been installed by opening System Information via About This Mac, and selecting the Installations item under Software.
A full listing of security data file versions is given by SilentKnight, LockRattler and SystHist for El Capitan to Sequoia available from their product page. If your Mac hasn’t yet installed this update, you can force it using SilentKnight, LockRattler, or at the command line.
If you want to install this as a named update in SilentKnight, its label is XProtectPlistConfigData_10_15-5291.
Sequoia systems only
This update has also been released for Sequoia via iCloud. If you want to check that manually, use the Terminal command sudo xprotect check
then enter your admin password. If that returns version 5291 but your Mac still reports an older version is installed, you can force the update using sudo xprotect update
I have updated the reference pages here which are accessed directly from LockRattler 4.2 and later using its Check blog button.
I hope that you enjoyed Saturday’s Mac Riddles, episode 300. Here are my solutions to them.
1: The first chips with six-packs celebrated Halloween.
Click for a solution
M3
The first chips (Apple silicon SoCs) with six-packs (the M3 family is the first to support six-core clusters) celebrated Halloween (they were announced at Apple’s ‘Scary Fast’ event on 30 October 2023).
2: First with FireWire and almost see-through in its two-tone case.
Click for a solution
Power Macintosh G3 (Blue and White)
First with FireWire (it was the first Mac to come standard with FireWire ports) and almost see-through in its two-tone case (it has a distinctive translucent blue and white case).
3: It brought Exposé, Fast User Switching and Xcode.
Click for a solution
Mac OS X 10.3 Panther
It brought Exposé, Fast User Switching and Xcode (all three were new features in 10.3, released on 24 October 2003).
The common factor
Click for a solution
The number 3; in binary 11, which looks like the number 2 in Roman numerals.
Sometimes Macs behave in ways that you don’t forget. It was nearly six years ago, in 2019 when my iMac Pro was still running macOS Mojave, that it took over half an hour to start up in Safe mode. As became obvious, that was because it checked the integrity of every snapshot, including its long series of Time Machine backups. That behaviour stopped with the release of Catalina later that year, and revisiting the experience last week was interesting, when I examined what Safe mode does now in Sequoia.
Safe mode fsck
Checking disk integrity has long been a feature claimed of Safe mode, although Apple has only recently clarified that its checks are “similar to the more comprehensive check performed by the First Aid feature of Disk Utility,” unlike those in the days of Mojave.
Because these are run when starting up into Safe mode, macOS can’t report them directly to the user. There are only two places for their results to be recorded: in the fsck_apfs log at /var/log/fsck_apfs.log (and its error log at /var/log/fsck_apfs_error.log), and in the Unified log, neither of which are likely to be visited by users. It doesn’t seem possible that checks could be run before entries are made in the Unified log, and if they were their results would be inaccessible, and of no value to the user.
If you do look there, you might be surprised to discover that the checks run by fsck_apfs are ‘quick checks’, described as a “check whether the device is `clean’. If device is an APFS volume, fsck_apfs will quickly check the APFS container and the specified APFS volume. If device is an APFS container, fsck_apfs will quickly check the APFS container and all the APFS volumes in it. By default, no repairs are attempted during a quick check.” Thankfully they also no longer include Time Machine snapshots or backups.
For users, the most important of volumes to be checked is the Data volume in the current boot volume group. Yet careful examination of both log records reveals that such checks fail, returning error 65, because the device is mounted with write access. Although they could be performed using fsck_apfs‘s live verification, no attempt is made to do that. From the active boot volume group, only the Recovery and Update volumes are checked, together with all accessible external volumes.
Normal user fsck
Before you think that’s better than making no checks at all, the sequence of fsck_apfs checks run at the start of Safe mode appears to be identical to those made during startup into normal user mode, and reports the same failures. Thus, in this respect Safe mode is no different from normal, despite Apple listing this as a feature of Safe mode.
It’s worth reading the fsck_apfs man page to see the many options available in this command, which is the basis for all checks and repairs made to APFS volumes and containers. There are two in particular that you may prefer to invoke rather than using Disk Utility’s single option to both check and repair. My favourites that make it worth opening Terminal instead of Disk Utility are -n to prevent it from attempting to make any repairs, and -S to skip snapshots.
Given that fsck_apfs appears unable to repair any errors it finds in snapshots and admits to that in its man page where it states that “no repairs can be made”, it puzzles me that without the -S option it will still check all snapshots it comes across when running its checks. Perhaps it’s time to alter that default behaviour and make checking snapshots the explicit option.
First Aid
Disk Utility’s First Aid is descended from a succession of utilities starting with the Disk First Aid app in Classic Mac OS, which separated disk Verify and Repair features.
Mac OS X merged the two Classic apps Drive Setup and Disk First Aid, but kept them as distinct tools within its new Disk Utility app and retained separate Verify and Repair features.
Those two features were still present in Mountain Lion or Mavericks, shown here, but soon afterwards only Repair was available.
Now that APFS is mature and has proven itself more than a worthy successor to HFS+, the need to both check and repair its containers and volumes should be abating. This would be a good time to return to separate controls, allowing users to check APFS in the expectation that there are no errors to be found. This is more important now because of the absence of any alternatives.
Monopoly
Disk Utility is in a unique position. Apple has chosen to lock third-parties out of developing competing disk repair utilities, in particular those that can be used in Recovery mode. Users can’t opt for apps with more or better features, putting the onus on Apple to provide a comprehensive solution that meets users’ wishes, not its idea of essential requirements.
Conclusions
Apple needs to be explicit about what disk checks are run in Safe mode, and where users can inspect their results.
fsck_apfs is flexible and powerful, but has strange defaults that merit reconsideration.
Disk Utility should offer more options beyond its single ‘check and repair’, in particular to check only and with snapshots excluded. Those are most important in Recovery mode, where there can be no alternative.
Here are this weekend’s Mac riddles to entertain you through family time, shopping and recreation.
1: The first chips with six-packs celebrated Halloween.
2: First with FireWire and almost see-through in its two-tone case.
3: It brought Exposé, Fast User Switching and Xcode.
To help you cross-check your solutions, or confuse you further, there’s a common factor between them. To celebrate this anniversary edition, that’s a number which can be expressed in a way that a Roman might read as being one less than it really is.
I’ll post my solutions first thing on Monday morning.
Please don’t post your solutions as comments here: it spoils it for others.
With Mac OS X came a new tool for installing and updating the system, quite different from what had been used in Classic Mac OS. The Mac OS X Installer app uses packages (.pkg), and metapackages (.mpkg) containing multiple packages to be installed together. Apple thus provided installations and updates as metapackages that could either be downloaded from its update servers using Software Update, or from the update’s web page. The same method was used until Big Sur, when updates changed again.
A system update in Mac OS X 10.1.3 in March 2002 was installed by the Installer app following authentication.
This update required just 148 MB of disk space, and could readily be accommodated by any of these volumes of 11.5 GB capacity.
Most updates were concluded by a period ‘optimising system performance’, as determined by the post-install scripts in the package.
Here, Software Update is delivering Security Update 2004-05-03 to Mac OS X 10.3.3 Panther. Some system components like QuickTime were still supplied and installed separately, but the great majority of Mac OS X was integrated into a whole, and there were no options to install separate components.
Over this period, the system and user data shared a single boot volume, and updating the system mainly involved replacement of updated files. Installer packages contained those replacements together with scripts that were run to update the contents of the boot volume. For much of this, firmware updates were still supplied and installed separately, although later they were integrated into macOS updates.
Until the introduction of System Integrity Protection (SIP) in El Capitan, the only protection provided to system files and folders was in their permissions. Incomplete or incorrect installations and updates were therefore not uncommon, as despite its name, SIP didn’t have any means of verifying the integrity of system files. A procedure was introduced to verify directory structures and permissions against those listed in the Bill of Materials (BoM) for macOS updates, by repairing permissions, but that was still unable to verify the integrity of the files themselves.
Installer metapackages are highly portable, and were commonly downloaded to be installed on multiple Macs. To keep updates as small as possible, they were provided in two forms: a Delta update converted only the previous release, while a Combo update contained everything required to update the last major version and all intermediate minor versions in a single step.
The High Sierra 10.13 upgrade in September 2017 brought greatest change, with its inclusion of Apple’s new file system APFS. Macs that didn’t have a Fusion Drive had their system volume converted to APFS during the upgrade, although it was another year before the same happened to Fusion Drives.
Updates didn’t always work out right for everyone. This was a common problem with High Sierra Supplemental Update of 29 November 2017, for example.
This all changed with the first version of macOS to boot from a signed snapshot, Big Sur, in November 2020, to support the improved Secure Boot of Apple silicon Macs. This abandoned the use of Installer packages, relying instead on an Update Brain integrated into each installer app and downloaded update.
From then onwards, Apple has provided several different presentations of macOS installers and updates:
Updates are only delivered through Software Update or its command tool equivalent softwareupdate, and have to be downloaded from Apple’s servers, or delivered through a local Content Caching Server.
Full macOS installer apps are offered in the App Store then delivered through Software Update.
Full macOS installer packages are available through softwareupdate or direct from Apple’s servers, and are named InstallAssistant. When installed, these create a full installer app.
Full macOS installers are offered in Recovery, from where they’re downloaded from Apple’s servers.
Full IPSW images containing firmware, Recovery and macOS partitions can be installed to restore Apple silicon Macs in DFU mode, using Apple Configurator 2 or later versions of the Finder. Those effectively reset that Mac to factory condition with that version of macOS pre-installed.
There are further complications to this. For instance, an older macOS Installer app can’t be run in a newer major version of macOS. The workaround for that is to create a bootable installer volume, and boot from that to run its older installer.
macOS updates are supplied compressed, and require up to 30 minutes preparation before they can be installed.
There are now no optional installations, as every copy of any given version of macOS is identical within its Signed System Volume (SSV).
The size of these new updates is considerably greater than those of older Installer packages, particularly in Big Sur, as engineering optimisations were being performed.
This chart shows cumulative sizes of updates to macOS on Intel Macs from 10.14 Mojave, with its traditional single boot volume, through to macOS Sonoma 14.6. Each point represents the cumulative sum of all updates to that major version of macOS required to reach that minor version. Thus the point for 14.2 is the total of update sizes for 14.1, 14.1.1, 14.1.2 and 14.2. Sizes used aren’t those reported by Software Update, but those of the download itself, as reported in articles here, indexed on this page. Lines shown are best fits by linear regression.
Update sizes rose markedly from Mojave, with its single boot volume, to Catalina, with its boot volume group, and again to a peak in Big Sur, with the SSV. They fell again as Monterey introduced greater efficiency, and Ventura and Sonoma have been almost identical, and smaller than Mojave.
Apple silicon Macs started with the huge updates of Big Sur, which were even larger than those for Intel models, and benefitted from the improved efficiency of Monterey and Ventura. Unlike Intel Macs, though, Sonoma has seen further reduction in update sizes, although in each update they remain significantly larger than those for Intel models.
macOS Ventura in 2022 experimented with Rapid Security Responses (RSRs), much smaller updates intended to provide urgent security fixes to Safari and some of its supporting components. These take advantage of cryptexes, cryptographically verified disk images stored on the hidden Preboot volume. Updating cryptexes alone is far quicker too, as the SSV is left untouched. Unfortunately, the second RSR resulted in serious problems with Safari so had to be replaced three days later. The last RSR was released on 12 July 2023, and they appear to have been abandoned since.
Upgrading to the first release of the next major version of macOS had required downloading its full installer app from the App Store. Apple broke from this in macOS Ventura in October 2022, when that new macOS was installed as an update instead. Although this reduced its size and installation time required, it caught many users on the hop, as Apple provided no warning of the change. This approach has since become standard with both Sonoma and Sequoia.
Installing and updating the Mac’s operating system has probably changed more over the last 41 years than any other feature.
Login
