One of the unannounced features in macOS Sequoia is, for many who use iCloud Drive, one of its most important, as this upgrade introduces the ability to pin files and folders to ensure they remain downloaded and don’t get evicted. This article explains how to use this feature, exactly what it does, and how it works.

Pin an item in iCloud Drive

Assuming that your Mac is connected to iCloud Drive and you have enabled Optimise Mac Storage, so that files in iCloud Drive can be evicted from local storage, you can use pinning to ensure that designated files aren’t evicted, and remain downloaded and synced with iCloud Drive. This is an important advance for anyone who needs certain files to be kept in local storage, so remaining accessible even when connection to iCloud isn’t possible, such as when you’re travelling.

Select the file or folder in the Finder and Control-click to bring up the contextual menu. You’ll see a new item there, Keep Downloaded. Select that.

That adds a new icon to the pinned files, with a white downward arrow on a circular grey background. This indicates that file is to be kept downloaded or pinned locally.

To unpin an item, select it and bring up the contextual menu. The Keep Downloaded item will be ticked. Select that command, and that toggles the setting off, unpinning the item so it can be evicted from local storage once again.

Pinning only works if your Mac’s iCloud Drive has Optimise Mac Storage enabled, which puts it into non-replicated FileProvider mode, and allows you and macOS to Remove Download or evict files. Its effect is to prevent that from occurring. Pinned files will remain downloaded to your Mac all the time, just as if iCloud Drive were in replicated FileProvider mode, with Optimise Mac Storage disabled.

A file’s pinning setting travels with it if you move that file within the same volume, and may be preserved when it’s moved away, for example by transferring the file by AirDrop. It’s preserved when you make changes to that file, but not when it’s copied, and pinning settings are specific to a local Mac, and not transferred to other Macs or devices connected to the same iCloud Drive.

How it works

Pinning is set by macOS attaching an extended attribute (xattr) named com.apple.fileprovider.pinned to each file that’s pinned. That normally contains a single byte, the character 1 as the single byte 0x31, although changing that to 0 works just as well, so it appears it’s the xattr that’s important, rather than its contents.

When a com.apple.fileprovider.pinned extended attribute is attached to a file, it ceases being eligible for eviction, and is kept downloaded, just as if it were in a replicated FileProvider. Unpinning the file strips the xattr.

Xattr flags

I have recently summarised the features of extended attributes, and explained the system of flags they use to determine how those are copied. In addition to adding the com.apple.fileprovider.pinned xattr to those encountered in Sequoia, Apple has changed the xattr flag system to cope with this new xattr. These flags are appended to the xattr name, after a # character, and pinning xattrs are actually named com.apple.fileprovider.pinned#PX to determine when they should be preserved during file copy operations. The X flag is new to Sequoia, and can be used for other xattrs.

This brings the table of available xattr flags to six. Flags can be upper or lower case letters C, N, P, S, B or X, and invariably follow the # separator, which is presumably otherwise forbidden from use in a xattr’s name. Upper case sets (enables) that property, while lower case clears (disables) that property. The properties are now (as of 15.0):

• C: XATTR_FLAG_CONTENT_DEPENDENT, which ties the flag and the file contents, so the xattr is rewritten when the file data changes. This is normally used for checksums and hashes, text encoding, and position information. The xattr is preserved for copy and share, but not in a safe save.
• P: XATTR_FLAG_NO_EXPORT, which doesn’t export or share the xattr, but normally preserves it during copying.
• N: XATTR_FLAG_NEVER_PRESERVE, which ensures the xattr is never copied, even when copying the file.
• S: XATTR_FLAG_SYNCABLE, which ensures the xattr is preserved during syncing with services such as iCloud Drive. Default behaviour is for xattrs to be stripped during syncing, to minimise the amount of data to be transferred, but this will override that.
• B: XATTR_FLAG_ONLY_BACKUP, which keeps the xattr only in backups, including Time Machine.
• X: XATTR_FLAG_ONLY_SAVING, which keeps the xattr only when saving and in backups, including Time Machine (introduced in macOS 15.0).

These must operate within another general restriction of xattrs: their name cannot exceed a maximum of 127 UTF-8 characters.

This new flag doesn’t change the macOS default flags set for different types of xattr, which remain as:

• com.apple.quarantine – PCS
• com.apple.TextEncoding – CS
• com.apple.metadata:kMDItemCollaborationIdentifier – B
• com.apple.metadata:kMDItemIsShared – B
• com.apple.metadata:kMDItemSharedItemCurrentUserRole – B
• com.apple.metadata:kMDItemOwnerName – B
• com.apple.metadata:kMDItemFavoriteRank – B
• com.apple.metadata:* (except those above) – PS
• com.apple.security.* – S
• com.apple.ResourceFork – PCS
• com.apple.FinderInfo – PCS
• com.apple.root.installed – PC

(* here represents the wild card.)

Until Apple releases the source code for Sequoia’s open source components, the table of copy intents remains unchanged at

• XATTR_OPERATION_INTENT_COPY – a simple copy, preserves xattrs that don’t have flag N or B
• XATTR_OPERATION_INTENT_SAVE – save, where the content may be changing, preserves xattrs that don’t have flag C or N or B
• XATTR_OPERATION_INTENT_SHARE – share or export, preserves xattrs that don’t have flag P or N or B
• XATTR_OPERATION_INTENT_SYNC – sync to a service such as iCloud Drive, preserves xattrs if they have flag S, or have neither N nor B
• XATTR_OPERATION_INTENT_BACKUP – back up, e.g. using Time Machine, preserves xattrs that don’t have flag N

as documented in man xattr_name_with_flags. Once I have been able to check the new source code, I will confirm any changes to the intents table.

Summary

• If you have Optimise Mac Storage turned on for iCloud Drive, you can now pin files so that they aren’t evicted from local storage.
• To pin a file or folder, use the Finder’s contextual menu (Control-click) and select the Keep Downloaded item.
• To unpin a file, toggle the Keep Downloaded item off (unticked).
• Pinning works by adding an extended attribute to each pinned file.
• A sixth xattr flag has been added, X, to keep that xattr only when saving and in backups.

Apple has just released an update to XProtect for all versions of macOS from El Capitan or so, bringing it to version 5273.

Apple doesn’t release information about what security issues this update might add or change. This adds Yara definitions for MACOS.DOLITTLE.CT, MACOS.SHEEPSWAP.CT and MACOS.SOMA.CT using a new format of rule, with each rule given a UUID and listing SHA256 hashes of file size.

You can check whether this update has been installed by opening System Information via About This Mac, and selecting the Installations item under Software.

A full listing of security data file versions is given by SilentKnight, LockRattler and SystHist for El Capitan to Sequoia available from their product page. If your Mac hasn’t yet installed this update, you can force it using SilentKnight, LockRattler, or at the command line.

If you want to install this as a named update in SilentKnight, its label is XProtectPlistConfigData_10_15-5273.

If you’ve upgraded to Sequoia and are still stuck at a version number of 0 or 5272, you can either leave macOS to catch up with this in its own good time, or you can force an update by typing into Terminal
sudo xprotect update
then entering your admin password.

I have updated the reference pages here which are accessed directly from LockRattler 4.2 and later using its Check blog button.

I maintain lists of the current versions of security data files for Sonoma on this page, Ventura on this page, Monterey on this page, Big Sur on this page, Catalina on this page, Mojave on this page, High Sierra on this page, Sierra on this page, and El Capitan on this page.

As promised last week, Apple has released the upgrade to macOS 15.0 Sequoia, together with security updates to bring Sonoma to version 14.7, and Ventura to 13.7. There should also be Safari updates to accompany the latter two.

The Sequoia update is around 6.6 GB for Apple silicon Macs, and 14.7 is around 1.6 GB. For Intel Macs, 15.0 is around 4.9 GB as an ‘update’, and 14.7 is around 860 MB.

Security release notes for Sequoia list around 77 vulnerabilities addressed, including two in the kernel, none of which Apple is aware may have been exploited in the wild. Release notes list 36 vulnerabilities addressed in Sonoma 14.7 here, and there are 30 listed for Ventura 13.7 here.

iBoot firmware is updated to version 11881.1.1, Intel T2 firmware to version 2069.0.0.0.0 (iBridge 22.16.10353.0.0,0), and Safari to 18.0 (20619.1.26.31.6).

After completing the upgrade to 15.0, you are likely to see that the installed XProtect version is 0, in other words that there is no XProtect data. You can leave your Mac to automatically download the required data from iCloud, or manually force it using the command
sudo xprotect update
then entering your admin password. That will normally ‘activate’ the XProtect data previously installed, and set the version to 5272, although that will then need to be updated to 5273 separately. Don’t be surprised if you end up repeating the trip to Terminal to get this to work.

Last updated 1900 GMT 16 September 2024.

Scientists from the United States, China and other nations called for an international authority to oversee artificial intelligence.

I hope that you enjoyed Saturday’s Mac Riddles, episode 273. Here are my solutions to them.

1: Resign, stop and almost quite is final.

Click for a solution

Quit

Resign (quit a job), stop (quit) and almost quite (it’s quite without the last letter) is final (it’s the last command).

2: Almost all round opens a window first.

Click for a solution

About

Almost (about a number) all round (about a location) opens a window (it opens the About window) first (it’s the first command).

3: Past preferences when celestial bodies sink below the horizon.

Click for a solution

Settings

Past preferences (what it used to be called) when celestial bodies sink below the horizon (when the sun/moon/planets/stars set in the sky).

The common factor

Click for a solution

They are all standard commands in the app menu.

I look forward to your putting alternative cases.

Later today, Apple is expected to release macOS Sequoia 15.0. For those interested in planning their immediate or delayed upgrade, these are my forecast dates for its minor versions over the coming year. Like all the best weather forecasts, this is most accurate for the next 5 days, and those for further into the future are likely to be decreasingly reliable.

Minor version release dates for Sonoma have been broadly similar to those of others since Big Sur:

• 14.0 – 26 September,
• 14.1 – 25 October,
• 14.2 – 11 December,
• 14.3 – 22 January,
• 14.4 – 07 March,
• 14.5 – 13 May,
• 14.6 – 29 July,
• 14.7 – 16 September.

Ventura differed mostly because it had a later start date to its cycle, in October, resulting in the delay of 13.1 until December. Subsequent versions thus trailed Sonoma by one, for example with 13.5 on 24 July, against 14.6 on 29 July. Although Apple is believed to have some flexibility in the release dates for minor updates, the timetable for the cycle appears to be fixed well in advance, and is probably already at least pencilled in for Sequoia.

Most minor updates bring new versions of firmware, the kernel and key kernel extensions such as APFS. In between those may be patch updates to fix serious bugs or security vulnerabilities that can’t wait for the next minor version, such as 14.3.1 on 8 February, two weeks after 14.3 and a month before 14.4.

According to Apple’s release notes, the current release candidate for 15.0 has no significant bugs that remain unfixed, and we hope that remains the case.

15.1: October 2024

Apple has already announced that this first ‘minor’ update will bring its AI features, including most significantly Writing Tools. Although those have been in beta-testing for almost as long as 15.0, in terms of changes, the step from 15.0 will in many ways be greater than that from 14.6 to 15.0. However, that only applies to Apple silicon Macs that support AI.

For all Macs, this is likely to bring fixes for some more substantial bugs, although because of the short interval between 15.0 and 15.1, few are likely to be addressed until 15.2.

This update is likely to coincide with new Mac products launched at an as-yet unannounced Mac event in October, where Apple is expected to promote its new M4 Macs as being ‘made for AI’, much in the way that it did last week with the iPhone 16 range.

15.2: December 2024

Turnaround time fixing even straightforward high priority bugs makes it likely that most in 15.0 will be addressed not in 15.1 but 15.2, before Christmas. This will also catch the first fixes and any additional enhancements required by AI, so may well be one of the more substantial updates this cycle. The aim is to give engineering teams a chance to catch up with the vacation without leaving too much to await their return in the New Year.

15.3: January 2025

This update is largely constrained by the effects of the Christmas vacation, but should enable most issues arising in 15.0 and 15.1 to be fixed, leaving Sequoia running sweetly.

15.4: March 2025

This is the major mid-cycle update, that is most likely to contain new and enhanced features, often making it the largest update of the cycle. Apple also seems to use this to introduce initial versions of new features intended to become fully functional before the end of the cycle. One example of this was XProtect Remediator, released on 14 March 2022 in Monterey 12.3, but not really functional until June that year.

Unfortunately, these enhancements can also cause problems, and this update in March has a track record of sporadic more serious bugs, including the occasional kernel panic.

15.5: May 2025

A month or so before the first beta-release of the next major version of macOS, this normally aims to fix as many remaining bugs as possible, and progress any enhancements introduced in the previous update. If you’ve reported a bug before April, then if it’s going to be fixed in this cycle, this is the most likely time; any new bugs reported after this update are most likely to be carried over to the next major release.

15.6: July 2025

This really is the last chance for fixes and feature-tweaks before the next major version is released in September. If all is working out well, this should be the most stable and bug-free release, although in some years late changes have turned this update into a nightmare, and Sonoma required a patch update in early August to address those.

When best to upgrade?

If third-party software, hardware and other compatibility requirements don’t apply, there’s no way to predict which is the best version to choose as an upgrade from previous macOS. Every version contains bugs, some of them may be serious, others may be infuriating and intrude into your workflows. But those aren’t predictable. If you’re unsure, wait a few days after a minor update, or even 15.0, check around with others, and decide then. If you’re really cautious and have an Apple silicon Mac, I suggest you might like to consider upgrading a week or two after the release of 15.1, by which time most of any major issues with 15.0 and AI should have come to the surface.

For myself, I already have my designated beta-testing Mac, a MacBook Pro M3 Pro, running 15.1 beta, and my other three Macs (iMac Pro, Mac Studio M1 Max and MacBook Pro 16-inch 2019) will all be running 15.0 by midnight tonight, I hope. I’ll let you know how I get on.

It’s strange to think that, as we’re wondering whether and when to upgrade to Sequoia, Apple’s engineering teams are already at work on macOS 16. While they’re thinking out what we’ll chew over next summer, you may well be asking if you should upgrade to 15.0 next week, wait for the AI features coming in 15.1 next month, or leave your decision until 2025?

For those with Macs and iPhones that can both be upgraded, iPhone Mirroring is probably the most obviously attractive new feature. It completes the integration of Continuity, and could transform your workflows. Fortunately for such a key feature, it should work with all supported Macs, not just Apple silicon models. There’s one small and temporary disappointment, though, as drag and drop between Mac and iPhone isn’t expected in 15.0, but in an update “later this year”.

The new Passwords app should spare you from wanting to pay for a third-party password manager. This is much more than just shelling out the existing Passwords feature from Safari and System Settings, and at last gives full control over passkeys and other shared secrets in your Keychain in iCloud.

Although some see Sequoia’s new dislike for apps that aren’t notarized (or from the App Store) as an unnecessary burden, for most of us this will raise the bar against running malware and increase our margin of safety. It has been some time since any malicious software has been successfully notarized, and most of the current epidemic of stealers aren’t even signed with a Developer certificate. Instead, they usually prompt the user to open them using the existing Finder bypass, something that no longer works in Sequoia without explicitly and individually giving permission to that app in Privacy & Security settings.

It will be interesting to see how malware developers respond to this challenge, as trying to give the user detailed instructions as to how they can be run without being blocked by Gatekeeper should now arouse the suspicion of even the most careless and inattentive.

While we’re on the subject of security, remember that Sequoia is now the only version of macOS that gets full security updates over the coming year. While Sonoma and Ventura will still get some, if you want the lot then you’ll need to upgrade. Monterey, of course, now gets none at all. This gets more brutal when considering other bugs that aren’t relevant to security: those will only be fixed in Sequoia, not even in Sonoma.

For those who virtualise macOS on Apple silicon, support for Apple ID gives VMs access to iCloud Drive at last, although it stops short of enabling the App Store or its apps, so isn’t as useful as it should have been. There are two important restrictions to this:

• Apple ID can only be used in a Sequoia guest running on a Sequoia host, and
• the Sequoia VM has to be built from a Sequoia IPSW file, and can’t be upgraded from a Sonoma or earlier VM.

As long as your Mac stays with Sonoma, you won’t be able to use Apple ID in any of its VMs, including Sequoia. This still leaves us with the paradox that Apple wants us to buy and run apps from its App Store, but VMs are the one place where you can’t use them.

Among the less prominent improvements that have caught my attention are a timed messaging feature of Send Later in Messages, and a batch of improvements in Freeform. If you’ve come to like that relatively new app, you should find Sequoia worth the effort. I’ve also been impressed to see one of the oldest bugs remaining in the Finder has finally been addressed in macOS 15. I’ll be putting the bunting out in celebration after I’ve upgraded on Monday.

As with Sonoma, some of the most important new features haven’t been documented even for developers. Among those are changes to XProtect in terms of its updating and management, and speculation as to how that might affect its function. As I have explained, XProtect’s detection rules have grown enormously over the last few months, and it’s likely that Apple intends improving how XProtect can apply its Yara rules, and making their updating more efficient.

Finally, Sequoia is almost certainly going to be delivered as if it were an update, and won’t download its installer app unless you’re upgrading from a significantly older version of macOS, just as has happened in all recent macOS upgrades. Remember that upgrading macOS these days comes with a one-way ticket: changing your mind afterwards will cost you a lot of time and messing about to step back to Sonoma. However, accidental upgrades shouldn’t be feared. For instance, if you inadvertently click the Install all updates button in SilentKnight and want to reverse that for a macOS update, let the download complete, shut down, start up in Safe mode, wait a minute, then restart in normal mode.

Whatever you choose tomorrow, I hope it works well for you. And in case you’re wondering, if you’ve got an Apple silicon Mac, you’re going to love 15.1.

Here are this weekend’s Mac riddles to entertain you through family time, shopping and recreation.

1: Resign, stop and almost quite is final.

2: Almost all round opens a window first.

3: Past preferences when celestial bodies sink below the horizon.

To help you cross-check your solutions, or confuse you further, there’s a common factor between them.

I’ll post my solutions first thing on Monday morning.

Please don’t post your solutions as comments here: it spoils it for others.

We all know about the Desktop Publishing revolution that the first Macs and their PostScript LaserWriter printers brought in the late 1980s, but many have now forgotten the Desktop Video revolution that followed in the next decade. At its heart was support for multimedia in Apple’s QuickTime.

QuickTime isn’t a single piece of software, or even an API in Classic Mac OS, but a whole architecture to support almost any media format you could conceive of. It defines container and file formats for multiple media types, forming the basis for the MPEG-4 standard, extensible encoding and decoding of a wide variety of media using Codecs, and more.

QuickTime development was initially led by Apple’s Bruce Leak, who first demonstrated it at the Worldwide Developers Conference in May 1991 before its release as a separate set of components for System 6 and 7 in December that year. Initially it came with just three Codecs, supporting animated cartoons, regular video and 8-bit still images. Cinepak video and text tracks were added in QuickTime 1.5 the following year, when high-end Macs were capable of playing 320 x 240 video at 30 frames/s, which was groundbreaking at the time.

By the mid-1990s QuickTime was starting to flourish. Hardware support included Apple’s new PowerPC Macs in 1994, and MIDI devices, PCs running Windows, MIPS and SGI workstations. QuickTime VR (for Virtual Reality) allowed the user to navigate the virtual space within panoramic images. QuickTime media were being licensed and distributed on CD-ROMs, innovative games such as Myst depended on it, and the QuickTime project brought in revenue to Apple at a time that it was most needed.

That period also brought conflict. Apple had contracted San Francisco Canyon Company to port QuickTime to Windows, but Intel also hired them to develop a competing product, Video for Windows. Source code developed for Apple ended up in Intel’s product, resulting in a lawsuit in 1994, finally settled three years later.

QuickTime was enhanced through the late 1990s, with version 5 the first to support Mac OS X, and just over a year later, in 2002, that was replaced by version 6. The following year, QuickTime 6.2 only supported Mac OS X, with a slightly older version for Windows.

QuickTime was one of the more used parts of what was then named System Prefs, here seen setting the MIME types to be handled by the QuickTime Plug-in, in 2002.

For most Mac users, bundled QuickTime Player was the standard way to play most types of video, as seen here in 2002.

Apple built apps like iMovie on the strengths of QuickTime. First released in 1999, iMovie is seen here in 2002.

QuickTime version 7 was both the first and last to use the QuickTime Kit (QTKit) Framework in Cocoa.

Apple’s flagship movie editing suite Final Cut Pro started as KeyGrip by Macromedia, but was first released by Apple in 1999; this ‘HD’ was actually version 4.5 in 2004.

Streaming movies in those days (here 2005) had to cope with a range of relatively low transfer rates, down to 56 Kb/s over a fast dial-up connection with a modem.

Users had to pay a small fee to upgrade QuickTime Player to the Pro version, unlocking more features including extensive transcoding options, here in 2007.

Mac OS X Server included a QuickTime Streaming Server, and a separate app, QuickTime Broadcaster (seen here in 2007), could be used to deliver real-time audio and video over a network.

QuickTime X for Mac OS X 10.6 Snow Leopard in 2009 marked the start of its slow decline, with the removal of support for some media formats, most noticeably MIDI. Internally, it had been converted to a Cocoa framework, AVFoundation, with modern 64-bit audio and video Codecs. This anticipated discontinuation of all support for 32-bit code in macOS Catalina. The impact on Codecs that were never ported to 64-bit is still felt today. While QuickTime is still alive in the AVFoundation framework, it’s very different now from its heyday in the opening years of this century.

By 2011, QuickTime Player was a shadow of its former self, and a far cry from its earlier Pro version.

Its pane in System Preferences, here in Panther of 2015, didn’t reflect the inner changes.

This is iMovie in 2011.

Further reading

Wikipedia, good on version details
AppleInsider, long and detailed account by Prince McLean in 2007
Computer History Museum, good background from Hansen Hsu, with a link to YouTube video from three of the creators of QuickTime.

In the previous article, I outlined what extended attributes do, and how they work in macOS. I also started to explain how some are considered ephemeral, while others are persistent. This article continues from there, by documenting how macOS decides what to do with them when a file containing xattrs is copied.

Although Apple does now explain a little about this in the context of the FileProvider framework and syncing with cloud services, the only useful documentation is provided in man xattr_name_with_flags, and two source code files that are part of the open source copyfile component.

In 2013, as part of its enhancements for iCloud in particular, Apple added support for flags on xattrs to indicate how those xattrs should be handled when the file is copied in various ways. Rather than change the file system, Apple opted for what’s perhaps best seen as an elegant kludge: appending characters to the end of the xattr’s name.

If you work with xattrs, you’ve probably already seen this in those whose name ends with a hash # then one or more characters: that’s actually the flags, not part of the name, what Apple refers to as a ‘property list’. To avoid confusion I won’t use that term here, but refer to them as xattr flags. A common example of this is com.apple.lastuseddate#PS, which is seen quite widely. In recent years, Apple has added one flag, B, and there’s another to come with Sequoia.

Xattr flags

Flags can be upper or lower case letters C, N, P, S or B, and invariably follow the # separator, which is presumably otherwise forbidden from use in a xattr’s name. Upper case sets or enables that property, while lower case clears or disables that property. There are currently (macOS 14.6.1) five properties:

• C: XATTR_FLAG_CONTENT_DEPENDENT, which ties the flag and the file contents, so the xattr is rewritten when the file data changes. This is normally used for checksums and hashes, text encoding, and position information. The xattr is preserved for copy and share, but not in a safe save.
• P: XATTR_FLAG_NO_EXPORT, which doesn’t export or share the xattr, but normally preserves it during copying.
• N: XATTR_FLAG_NEVER_PRESERVE, which ensures the xattr is never copied, even when copying the file.
• S: XATTR_FLAG_SYNCABLE, which ensures the xattr is preserved during syncing with services such as iCloud Drive. Default behaviour is for xattrs to be stripped during syncing, to minimise the amount of data to be transferred, but this will override that.
• B: XATTR_FLAG_ONLY_BACKUP, which keeps the xattr only in backups, including Time Machine (added recently).

These operate within another general restriction of xattrs: their name cannot exceed a maximum of 127 UTF-8 characters.

Defaults

macOS provides a standard ‘whitelist’ of default flag settings for different types of xattr. These aren’t contained in a configuration file, but are baked into the xattr flag code, where as of macOS 14.6.1 the following default flags are set for different types of xattr (* here represents the wild card):

• com.apple.quarantine – PCS
• com.apple.TextEncoding – CS
• com.apple.metadata:kMDItemCollaborationIdentifier – B
• com.apple.metadata:kMDItemIsShared – B
• com.apple.metadata:kMDItemSharedItemCurrentUserRole – B
• com.apple.metadata:kMDItemOwnerName – B
• com.apple.metadata:kMDItemFavoriteRank – B
• com.apple.metadata:* (except those above) – PS
• com.apple.security.* – S
• com.apple.ResourceFork – PCS
• com.apple.FinderInfo – PCS
• com.apple.root.installed – PC

Copy intents

Also contained in the source code is a table of intents, that explains how different types of copy are affected by different combinations of xattr flag. Currently, those are:

• XATTR_OPERATION_INTENT_COPY – a simple copy, preserves xattrs that don’t have flag N or B
• XATTR_OPERATION_INTENT_SAVE – save, where the content may be changing, preserves xattrs that don’t have flag C or N or B
• XATTR_OPERATION_INTENT_SHARE – share or export, preserves xattrs that don’t have flag P or N or B
• XATTR_OPERATION_INTENT_SYNC – sync to a service such as iCloud Drive, preserves xattrs if they have flag S, or have neither N nor B
• XATTR_OPERATION_INTENT_BACKUP – back up, e.g. using Time Machine, preserves xattrs that don’t have flag N

Use

If you want a xattr preserved when it passes through iCloud, you therefore need to give it a name ending in the xattr flag S, such as co.eclecticlight.MyTest#S. Sure enough, when xattrs with that flag are passed through iCloud Drive, those xattrs are preserved even if the default rule would treat them differently. Similarly, to have a xattr that is stripped even when you just make a local copy of that file, append #N to its name.

There’s a further limit imposed on xattrs synced by FileProvider, including those for iCloud Drive, that strips all individual xattrs that are larger than a certain size. Apple gives that as “about 32KiB total for each item”, and my measurements performed in the recent past put that at about 32,650 bytes, slightly less than 32,767.

In itself, this information is valuable if you ever use any metadata stored in xattrs. It’s used in my intergrity-checking utilities Dintch, Fintch and cintch to ensure the xattr containing a file’s hash isn’t stripped by passage through iCloud Drive, for instance. On Tuesday morning next week, once Sequoia has been released, I’ll explain how Apple has extended this system to achieve something that many have been wishing for.

One of the innovative features in classic Mac OS was its use of resource forks, allowing structured metadata to be attached to any file. When Mac OS X merged that with the more traditional Unix approach adopted by NeXTSTEP, those were nearly lost. Classic Mac apps were restructured from storing most of their components, including their executable code, in their resource fork, when Mac OS X flattened those into an app bundle consisting of a hierarchy of separate files in folders, without any resources.

For the first four years of Mac OS X resource forks were reluctantly tolerated, until the solution came in 10.4 with the introduction of extended attributes, including one to contain what had previously been stored in the resource fork, which became an extended attribute or xattr with the name com.apple.ResourceFork.

All files in HFS+ and APFS (and other file systems) contain a fairly standard set of metadata known as attributes, information about a file such as its name, datestamps and permissions. Xattrs are extensions to those that contain almost any other type of metadata, the first notable xattr coming in Mac OS X 10.5, named com.apple.quarantine. That contains quarantine information for apps and other files downloaded from the internet, in a format so ancient that the quarantine flag is stored not in binary but as text.

The quarantine xattr provides a good demonstration of some of the valuable properties of xattrs: it can be attached to any file (or folder) without changing its data, and isn’t included when calculating CDHashes for code signatures. It can thus be added safely without any danger of altering the app or its code, although it does change the way in which macOS handles the code, by triggering security checks used to verify it isn’t malicious. Once those have been run, the flag inside the quarantine xattr can be changed to indicate it has been checked successfully.

Far from being a passing phase, or dying out as some had expected, xattrs have flourished since those early days. This has happened largely unseen by the user: few change anything revealed in the Finder’s Get Info dialog, although they’re used to store some forms of visible metadata such as Finder tags, and the URL used to download items from the internet. Editing xattrs is normally performed silently: you’re not made aware of changes in the quarantine xattr, and in most cases the only way to manage xattrs is to use the xattr command tool, or one of very few apps like xattred that can edit and manage them.

Examples

Among the well-known and important xattrs you can encounter are:

• com.apple.quarantine the quarantine xattr, containing a quarantine flag
• com.apple.rootless marks items individually protected by System Integrity Protection (SIP)
• com.apple.provenance contains data about the origin of apps that have been quarantined
• com.apple.metadata:kMDItemCopyright records copyright info
• com.apple.metadata:kMDItemWhereFroms the origin of downloaded file as a URL
• com.apple.metadata:_kMDItemUserTags Finder tags
• com.apple.TextEncoding reveals text file encoding
• com.apple.ResourceFork a classic Mac resource fork

Storage

In APFS and HFS+, xattrs aren’t stored with file data, nor with a file or folder’s normal attributes.

For smaller extended attributes up to 3,804 bytes, their data is stored with the xattr in the file system metadata. Larger extended attributes are stored as data streams, with separate records, but still separately from the file data. Apple doesn’t give a limit on the maximum size of xattrs, but they can certainly exceed 200 KB, and each file and folder can have an effectively unlimited number of them.

Persistence

Most file systems to which macOS can write either handle xattrs natively (HFS+, APFS), or macOS uses a scheme to preserve them, as in the hidden files written to FAT and ExFAT volumes. NFS is an important exception, and files copied to NFS will have all their xattrs stripped. Neither are extended attributes unique to Macs: most file systems used by Linux support them, and even Windows can at a push.

Because xattrs contain a wide range of metadata, some are treated as being ephemeral, others as persistent. Moving files with xattrs around within the same volume shouldn’t affect their xattrs, as that takes place within the same file system. Copying files to another volume, even if both use APFS, may leave some xattrs behind if they’re considered to be ephemeral.

The most complex situation is when a file with xattrs is moved to iCloud Drive. The Mac that originated that file is likely to retain most if not all of its xattrs, because the local copy remains within the same volume and file system. However, not all xattrs are copied up to iCloud storage, so other Macs accessing that file may only see a small selection of them. The rules for which xattrs are to be preserved during file copying, including in iCloud Drive, are baked into macOS, and the subject of the next article.

A substantial number of Republican voters are losing faith in science.

Apple will release macOS 15.0 Sequoia on 16 September, that’s next Monday, alongside iOS and iPadOS 18.0, and upgrades and updates for lesser mortals. Among the latter are Sonoma 14.7 and Ventura 13.7, as I’ll explain later. Sequoia introduces two important changes to security data checked and updated by SilentKnight, for which I have built and notarized another new version of that app, 2.11, which is essential for anyone intending to upgrade to Sequoia, and worthwhile for all running Catalina or later.

What’s coming next week

Apple has just provided release candidates for the following three new versions of macOS:

• Sequoia 15.0, its first full release,
• Sonoma 14.7, its first security-only update,
• Ventura 13.7, the first of its security-only updates for its final year of support.

There’s not expected to be any update to Monterey 12.7.6, which is no longer supported, even with security updates.

The minor version numbers of Sonoma and Ventura will then be the same, the first time this has happened. In previous release cycles, the start of the first year of security-only updates has been with x.6, as it was with Ventura, and proceeded through the year with versions x.6.1, x.6.2, and so on. Over the coming year, we can expect 14.7.1 and 13.7.1, then 14.7.2 and 13.7.2, continuing until Ventura reaches the end of its third and final year of support in a year’s time.

Sequoia 15.1, the first release with AI support, is now expected in October, and continues in beta-testing, alongside AI-enhanced versions of iOS and iPadOS in versions 18.1.

TCC in Sequoia

The TCC database in /Library/Apple/Library/Bundles/TCC_Compatibility.bundle was introduced in Mojave (when it had a different location, of course), and has been updated with each new major version of macOS since. That has now vanished, and I can find no trace of it, nor any apparent substitute. If you run SilentKnight 2.10 in Sequoia, that will be reported as an error, so version 2.11 addresses that by omitting that result both from its display box and the text report below.

XProtect in Sequoia

Since it was first introduced many moons and versions of macOS ago, there has been a bundle named XProtect.bundle in CoreServices, most recently in the path /Library/Apple/System/Library/CoreServices/XProtect.bundle, that has provided data for XProtect scans of executable code and other security services. That bundle has been updated frequently in downloads labelled XProtectPlistConfigData. Although that can still be present in Sequoia, XProtect now uses a completely different source for its data, that is normally updated through iCloud’s CloudKit rather than Software Update.

The result is that your Mac can have an up-to-date XProtect.bundle in the normal location, but XProtect itself may not be up-to-date at all. For example, in fresh installs of Sequoia, XProtect.bundle is usually absent, and the new tool to check its version may report a number of 0.

SilentKnight versions 2.10 and 2.11 have been updated to cope with this major change, which Apple has apparently not seen fit to document (yet). They check the correct current version using a new command tool, and report that version number faithfully. At present, though, SilentKnight isn’t able to update this new form of XProtect. You can either leave macOS to do that itself in its own time, or you can run a command in Terminal to force the update immediately:
sudo xprotect update
following which you’ll need to authenticate with your admin user password.

I intend to address this more completely in SilentKnight version 3, but for the time being this is fully documented in SilentKnight’s Help book and Help Reference, in these latest versions.

SilentKnight, Skint, SystHist, LockRattler

SilentKnight version 2.11 is strongly recommended for anyone intending to update to Sequoia this year, and, as it also fixes a bug in reporting Studio Display firmware in VMs, is worthwhile for those remaining with Sonoma for longer. It’s available from here: silentknight211
from Downloads above, on its Product Page, and through its auto-update mechanism.

Thankfully, as Skint doesn’t check TCC, the current version 1.08 remains fully compatible with Sequoia. The current release of SystHist, 1.20, works well with Sequoia too, and usefully distinguishes between the two different types of XProtect update, XProtectPlistConfigData delivered through Software Update, and XProtectCloudKitUpdate the new one obtained through iCloud instead.

I don’t intend to update LockRattler for the time being. It won’t report the true version of XProtect, but does report that it can’t find TCC or the GKE data. Otherwise it should continue to function as expected in Sequoia.

More to come in Sequoia 15.0

These changes to XProtect are but one of the significant changes that Apple hasn’t yet mentioned. Once 15.0 has been released, I’ll be delighted to provide fuller details of others.

Summary

• On Monday 16 September, Apple will release macOS 15.0, and security updates 14.7 and 13.7.
• Monterey is no longer supported.
• Download and install SilentKnight 2.11 if you’re intending to upgrade to Sequoia this year.
• Skint and SystHist remain fully compatible with Sequoia.
• Watch here for further news on Sequoia once it has been released next week.
• Sequoia 15.1 with AI will be released next month (October).

External storage is invariably sold with ‘up-to’ performance figures. In practice, you’ll seldom realise anything like some write or read speeds claimed. And when it comes to prolonged tasks like that first full Time Machine backup, no matter how fast you thought that drive would be, it always takes longer than expected.

Over the last few years I have tested and reviewed many examples of different types of external storage, from basic USB 3 hard drives, to the latest USB4 SSD enclosures, and NAS packed with fast SSDs. This article draws on all those test results to give you a better idea of what to expect when they’re being used with your Mac.

Results quoted here are typical for those tests performed mostly using a Mac Studio M1 Max, but unless otherwise indicated should be similar for recent Intel models. They’re summarised in this table.

Write speeds are given for:

• the single 50 MB write test performed by Time Machine before each backup;
• 500 multiple concurrent writes of 4 KB each, performed in those same Time Machine tests;
• calculated net write speed over a first full backup to APFS of at least 400 GB;
• general write speed measurement using my app Stibium, which gives broadly similar results to other leading benchmarking apps.

General read speeds are also obtained using Stibium, and similar to other apps. All speeds are given as MB/s for consistency.

Before looking at individual types of storage, one obvious and important result is the effect of throttling by macOS on Time Machine backup performance. Considering Time Machine’s own tests, writing a single 50 MB file is performed consistently at around 200-225 MB/s to local storage of whatever type, and multiple concurrent writes of 4 KB files reach around 20-23 MB/s regardless of local storage type. Those hold good even when you back up to a fast Thunderbolt 3 SSD, and backing up to a NAS is little quicker unless it’s over 2.5GbE to an NVMe SSD. Local transfer speeds only differ more substantially in general tests, when they aren’t throttled as they are in Time Machine.

Hard disks

When writing to or reading from a local hard disk, performance varies substantially according to which sectors on the hard disk are being accessed. This is a well-known phenomenon, and the result of geometry, as sectors are faster at the periphery of the disk’s platter, and slower in the inner part. Ranges given here take that into account: the lower figure is for inner sectors, and the higher for outer ones. Some users compensate for this effect, and only ever use the outer half of a disk’s sectors to obtain better performance, but that reduces their available capacity, and effectively doubles their cost per TB.

SSDs

SATA SSDs may be cheapest, but they’re also slowest, and with Macs they generally don’t enjoy Trim or SMART health indicator support. Of the two, Trim support is usually the more important, as without that, they can accumulate blocks waiting to be erased and returned for further use, and as a result their write (but not read) speed can fall as low as 100 MB/s. Unless used for largely static storage, this is a significant risk.

NVMe SSDs deliver twice the performance of SATA models, and generally enjoy Trim but not SMART indicator support. This makes them far better suited to general use, as their write speeds should be sustained from new throughout their working life.

USB 3.2 Gen 2, Thunderbolt 3, USB4

Translating commonly quoted transfer speeds for these three protocols into real-world speeds turns out to be complex. In practice, these are what you can expect to see:

• USB 3.2 Gen 2 at 10 Gb/s is slightly less than 1 GB/s
• Thunderbolt 3 at 32 Gb/s is up to 3 GB/s
• USB4 at 40 Gb/s is up to 3.4 GB/s.

All recent models of Mac, both Intel and Apple silicon, should realise full performance over USB 3.2 Gen 2 and Thunderbolt 3, but support for USB4 is limited to Apple silicon. Unless a drive or enclosure specifically includes Thunderbolt 3 as a fallback, when connected to an Intel Mac, you should expect it to fall back to USB 3.2 Gen 2 at just under 1 GB/s, less than a third of the speed of USB4.

NAS

Although I haven’t made any systematic comparison between AFP and SMB network protocols, I can see no consistent difference in their performance, when used with the latest versions of macOS and NAS software. The latter, though, can be critical: older versions of NAS software can perform poorly when used over SMB with recent macOS. Keeping your NAS software up to date is important.

Throttling of Time Machine backup writing isn’t supposed to occur when backing up over a network, and there is some evidence here to support that, with significantly better results for 50 MB test files. However, those are only apparent when using NVMe SSDs in the NAS, with a wired Ethernet 2.5GbE connection to provide sufficient bandwidth.

Check TM performance

Provided that your Mac is running a recent version of macOS and backing up to APFS, it’s simple to read the two write performance tests that occur at the start of each Time Machine backup using my free T2M2. Alternatively, you can also read them using the Time Machine custom log extract in Mints. In T2M2 they should look something like:
Destination IO performance measured:
Wrote 1 50 MB file at 238.02 MB/s to "/Volumes/ThunderBay2" in 0.210 seconds
Concurrently wrote 500 4 KB files at 35.58 MB/s to "/Volumes/ThunderBay2" in 0.058 seconds

Check general performance

Although there are other apps that will do this, I developed Stibium for this purpose. Follow the ‘gold standard’ procedure detailed in its Help Reference to obtain the most accurate and reproducible results. Stibium can test any storage you can access in the Finder, including all local devices and networked systems such as NAS.

Further reading

Which external drives have Trim and SMART support?
How to evaluate an external SSD
You can read my reviews in MacFormat and MacLife magazines, available in the App Store.

I hope that you enjoyed Saturday’s Mac Riddles, episode 272. Here are my solutions to them.

1: Used by courting birds with a haven for video and audio to replace the others.

Click for a solution

DisplayPort

Used by courting birds (a courtship display) with a haven (a port) for video and audio (it can carry both) to replace the others (intended to replace the answers to 2 and 3, as well as VGA and others).

2: 506 Romans can handle analogue and digital to display.

Click for a solution

DVI

506 Romans (DVI in Roman numerals = 506) can handle analogue and digital (has both DVI-I for analogue support, and DVI-D digital-only) to display (it’s for video output to displays and TVs).

3: With CTA-861, 19 pins and five connectors, it’ll carry all your media, even HDCP.

Click for a solution

HDMI

With CTA-861 (the standards it uses for video and more), 19 pins (in its connectors) and five connectors (it supports five different connectors now), it’ll carry all your media (it will), even HDCP (a form of DRM for use over HDMI).

The common factor

Click for a solution

They are all video interfaces that have been supported by Macs.

I look forward to your putting alternative cases.

Apple silicon Macs are the first Apple computers to feature fully secure boot processes as one of their fundamental design goals. Although similar in some ways to those in Apple’s devices and in Intel Macs with T2 chips, there are substantial differences. Unlike T2 Macs, secure Boot in Apple silicon is maintained even when starting up from external storage, a feature completely unsupported by devices. Non-standard configurations are also available to give greater flexibility with reduced security modes, which are absent from devices. This article explains how to manage Secure Boot on Apple silicon Macs.

Full Security

By default, Apple silicon Macs start up in Full Security, the mode recommended by Apple for normal use. That requires:

• Following pre-boot stages, a Signed System Volume (SSV) is loaded from its snapshot, with its seal intact and signature verified.
• The only kernel extensions loaded during boot are those supplied by Apple in macOS. No third-party kernel extensions can be loaded.
• System Integrity Protection (SIP) is fully enabled.

Because they run outside the privileged space of the kernel and its extensions, third-party system extensions and their relatives can be loaded and used fully.

Booting in Full Security applies full control and verification of all components from the Boot ROM, through the Low-Level Bootloader (LLB) and iBoot ‘firmware’, to the kernel and its extensions in the SSV. If any part of that sequence fails to verify, the boot process is halted and, depending on where the failure has occured, the Mac is either put into DFU or Recovery mode for the user to address the problem. This not only ensures robust security throughout, but it also guards against potential conflicts such as those arising with third-party kernel extensions, and unintentional corruption of any component.

Changing security

Control over boot security is applied using Startup Security Utility when booted in paired Recovery (except in Big Sur, as explained below). Access this by starting the Mac up into normal paired Recovery with a single long hold of the Power button. When the first screen has loaded, click the Options button, then from the Utilities menu select the Startup Security Utility command.

You will then be prompted to select the boot volume group whose security policy will be set. Note that in macOS 12.0.1 and later this must be the same as that for the paired Recovery being used.

By default that will already be in Full Security. Set the new policy as explained below, and click the OK button. To apply that policy, select the Restart command to exit Recovery and start up in normal user mode.

Startup Security Utility changes the LocalPolicy settings for that boot volume group; those are stored in the iSCPreboot volume on the hidden Apple_APFS_ISC container on the internal SSD of that Mac. Full details of LocalPolicy are given in Apple’s Platform Security Guide, and explained here.

Big Sur is a notable exception to the requirement to change security settings in the paired Recovery volume: it doesn’t have a paired Recovery volume, but boots into Recovery from a hidden container on its internal SSD, from where Startup Security Utility controls security settings for all mounted boot volume groups.

Fallback Recovery (frOS) is unable to change LocalPolicy, thus Startup Security Utility is unavailable when booted in frOS, its most distinctive feature.

Although the bputil command tool gives access to security settings, its use to modify them is hazardous, and shouldn’t be attempted unless you know what you’re doing and are prepared to have to restore that Mac in DFU mode. It is, though, sometimes useful as an aid to solving problems with LocalPolicy settings, as explained here.

Reduced Security

When reducing security settings, the only other option available is Reduced Security, which forms the gateway to Permissive Security as well. There are two other common reasons for selecting Reduced Security, though: to enable the loading of third-party kernel extensions, and to run older versions of macOS.

To enable the loading of third-party kernel extensions, Reduced Security must be set, and the upper checkbox ticked. Once this has been applied by a restart, installation and loading of the kernel extension can proceed using its installer and Privacy & Security settings. That doesn’t enable loading of the kext on demand, as in the past, as it has to be merged into a signed Auxiliary Kernel Collection, from where it can loaded during startup.

Although Apple states that Reduced Security is required to run older versions of macOS, that doesn’t appear to be required at present. Reduced Security differs from Full Security in that LLB and iBoot trust ‘global’ signatures that are bundled with macOS rather than using personalised signatures for iBoot and beyond. For many, this may appear to be an insignificant reduction compared with Full Security, although it does add the risks of loading third-party kexts when used for that purpose.

Permissive Security

Startup Security Utility doesn’t offer Permissive Security as an option until Reduced Security has been selected and an action has been taken to downgrade that to Permissive Security. The most likely reason for doing this is to partially or completely disable SIP using csrutil, and once that has been performed this third setting is shown in Startup Security Utility.

Just as with Reduced Security, this offers two further options in checkboxes, including that to enable the loading of third-party kernel extensions.

The main reason for using Permissive Security is to reduce SIP settings, and I have recently provided a guide to the controls available in csrutil. The normal sequence for changing those is to start up in paired Recovery, open Startup Security Utility, set that to Reduced Security and click on the OK button. Following that, open Terminal, run the appropriate csrutil command, then restart the Mac, which will automatically be in Permissive Security.

The security effects of Permissive Security are determined largely by changes made to SIP and other controls. Although signatures are still validated throughout the chain of boot stages, ‘global’ signatures are trusted for iBoot and beyond, as are boot objects signed locally by the Secure Enclave. At its most extensive, this allows the use of a fully untrusted kernel, such as a debug or custom version. The penalty is that some decryption keys are no longer available, and those restrict features available in macOS, such as Apple Pay, which is normally disabled.

Reversing Permissive Security requires fully enabling SIP using csrutil, undoing any other relevant security settings, then using Startup Security Utility to set a higher level of security. In some cases, that may entail installing a fresh macOS with its SSV. Apple also notes that, on Apple silicon Macs, SIP settings aren’t stored in NVRAM but in the LocalPolicy.

External Boot Volumes

With the notable exception of Big Sur, boot security settings are saved into the LocalPolicy for the boot volume group containing that paired Recovery system, although all LocalPolicy settings are saved to the internal SSD, never to an external disk. This relies on the concept of ownership, as explained here. This can bring its own problems, as explored here.

Summary

• Unless there’s a good reason, leave boot volume groups in Apple silicon Macs in Full Security.
• If your Mac needs to load a third-party kernel extension, run it in Reduced Security with those kernel extensions enabled.
• Partially or completely disabling SIP requires Permissive Security, which brings significant penalties, and may require more extensive work to undo.
• Use Startup Security Utility rather than bputil.

Reference

Apple’s Platform Security Guide

If you’ve become blasé with the tail end of the summer’s sport, Olympics and Paralympics, events next week should make compulsive viewing. Apple starts with its regular September launch of iPhones, and a day later we’ll be enthralled by the first TV debate between the two main contenders in the US presidential election. My money is on the iPhone 16 to win.

With those new iPhones comes the next major version of iOS, and hot on its heels macOS Sequoia 15.0. Without its AI features, that might seem the least exciting announcement of the week, but it prepares the ground for the next batch of Macs to be announced most probably in October, for shipping the following month. All commentators seem agreed that they will come not with M3 chips, but will be the first Macs to use the M4 family.

By now, different M-series chips are becoming blurry, so I’ll try to draw distinctions between them, and suggest why Apple has rushed through the M3 as if it might have been better-named the M2.5.

M1, November 2020

Skipping silently over Apple’s Developer Transition Kit from the summer of 2020, Apple silicon Macs started at a leisurely pace, as the four members of the M1 family rolled out over nearly 18 months. Their CPU cores ranged from the base version with 4 P and 4 E cores, 4P+4E in short, up to the impressive Ultra at 16P+4E. There was little separation, though, between the Pro and Max versions, which both came with 8P+2E, and only really differed in their GPUs. Thus, in the M1 there only two fundamentally different CPU core configurations, 4P+4E and 8P+2E, each with up to four cores in a cluster. Both core types used Arm’s instruction set architecture (ISA) from 2018, designated ARMv8.5-A.

M2, July 2022

Some of us assumed that first cycle would prove the model for its successors, but we were wrong and getting wronger as time progresses. After a break of just four months, Apple leaped into the M2 cycle, which it shortened to just a year from the 4P+4E M2 base version in July 2022. This cycle Apple bumped the number of E cores in the higher versions, taking the Ultra to 16P+8E, but still leaving little distance between the Pro and Max versions, both with 8P+4E, and two fundamental core configurations and clusters of four.

What might have appeared at the time to be small change, an increment in the ISA to ARMv8.6-A from 2019, brought enhancements in matrix maths, support for bfloat16 numbers to help with AI, and additional virtualisation capability. Although the M2 series appeared evolutionary in performance, it has also proved more capable.

M3, November 2023

Introduction of the M3 came again after a brief break of around four months, in November 2023. This time there was no phased release, gradually building up through Pro and Max to Ultra. The lesser three all came together, and put more distance between Pro and Max versions. The base M3 stuck with the proven combination of 4P+4E, the Pro scaled up to 6P+6E, then the Max nearly doubled that at 12P+4E, making three fundamental core configurations. That was over nine months ago, and there’s still no sign of any doubled-up Ultra version. To match the M3 Pro and Max core counts, they now form clusters of up to six cores, a significant advance on the two previous families.

While the M3 kept to Arm’s ARMv8.6-A ISA, Apple redesigned the GPU to support Dynamic Caching, Mesh Shading, and hardware-accelerated ray tracing. There’s one oddity, though: as this is the same ISA as the M2, with its enhanced support for virtualisation, Apple has stated that nested virtualisation coming in Sequoia requires not an M2 chip, but the M3, with its identical ISA. I have yet to see an explanation for that requirement.

M4, May 2024

In the absence of any M3 Ultra, Apple caught us off-guard when in May of this year (only six months after the M3) it launched new iPad Pros with what we must assume is the base M4 version of 4P+6E, and the next Arm ISA of ARMv9.2-A from 2020. That enhances vector and matrix maths, and brings support for 1 GHz timers, enabling nanosecond time resolution. In addition to the GPU features added in the M3, those in the M4 now also have hardware accelerated AV1 decoding.

There have, of course, been many other changes between M1 and M4, including higher operating frequencies for CPU cores, growth in the capability of the neural engine (ANE), and better support for external displays in the base version.

M1 to M4

Assuming that we’re soon to be treated to a range of Macs running members of the M4 family, the pace of change is accelerating. Landmarks along the route so far include:

• Increased E cores and ARMv8.6-A ISA in the M2.
• Three core configurations, clusters of six cores, and extended GPU hardware support in the M3.
• Increased E cores, ARMv9.2-A ISA and extra GPU hardware support in the M4.

For those who have whetted their appetite for Apple silicon with an M1 or M2, the leap up to M4 should prove exhilarating. Bring on the M4 Mac Studio, please, no matter who ends up as President.

Postscript and non-apologia

A few have commented to me, either below or by email, that they find this article somehow “offensive” for its references to the US Presidential Election, specifically for “belittling” it. If you have reached the end of this article and feel that to be true, then once you have finished reading this postscript, please go back and read its first paragraph again, slowly and without imagining words or meanings that simply aren’t there.

The introductory paragraph is what is known in American (but not, yet, British) English as a lede, and introduction. It establishes that there are two important events taking place early this coming week: Apple’s Event, which is the link to the main body of the article, and the televised debate between “the two main contenders in the US presidential election”. I hope we all agree that those are both important events.

I deliberately refer to the latter as enthralling, a word whose implications appear not be understood by many. A thrall is literally a slave, someone in bondage, and the word has ambiguous connotations of both being fascinating and attention-holding (its more common usage today), and enslaving mentally or morally. I simply cannot think of a more appropriate word to describe that hugely important debate, and cannot understand how anyone could construe that as in any way belittling.

The sentence at the end of that lede, “my money is on the iPhone 16 to win” is a reference to the unpredictability of elections (don’t forget that we in the UK had our own General Election just two months ago), and for those who have deeper insight into events a reference to gambling. As any American knows, betting in the US on the outcome of the Presidential Election is illegal, although betting on events such as the date of the next general election, or its outcome, remains perfectly legal in the UK. However, there were several scandals in the UK concerning people close to the former Prime Minister who placed successful bets on that date, and the hidden reference here is to those scandalous events still under investigation.

The final para in the article then refers back to the lede, a common technique when rounding off articles. In no way does it dismiss the importance of the election, but is an expressed wish that whatever disaster might or might not take place, there are many of us looking forward to Apple’s future M4 Macs, in my case specifically a new Mac Studio M4. Yes, it is a little light-hearted, after all this article is more of an editorial, as I’m sure you’re aware.

If you still find this offensive, then I’m sorry, but you really do need to read what is written, and not what you wish to imagine it’s saying.

The weird, mostly bad science of nominative determinism.

Here are this weekend’s Mac riddles to entertain you through family time, shopping and recreation.

1: Used by courting birds with a haven for video and audio to replace the others.

2: 506 Romans can handle analogue and digital to display.

3: With CTA-861, 19 pins and five connectors, it’ll carry all your media, even HDCP.

To help you cross-check your solutions, or confuse you further, there’s a common factor between them.

I’ll post my solutions first thing on Monday morning.

Please don’t post your solutions as comments here: it spoils it for others.

In the days before Mac OS X, Apple didn’t provide a serious backup utility, and by the time we were starting to move up from Classic Mac OS the standard choice was normally Dantz Development’s Retrospect, first released in 1989 and still available today in version 19.

Time Machine wasn’t the first utility in Mac OS to back up local storage. In 2004, Apple’s first cloud subscription service .Mac included a Backup app that backed up local files to iDisk in the cloud, something that still isn’t supported today with iCloud.

In the following years, AirPort Wi-Fi systems flourished, and Apple decided to launch a consumer NAS incorporating an AirPort Extreme Base Station with a 500 GB or 1 TB hard disk. Software to support that was dubbed Time Machine, and was released in Mac OS X 10.5 Leopard on 26 October 2007, 17 years ago. The first Time Capsule was announced in January 2008, and shipped a month later.

Time Machine’s pane in System Preferences changed little until Ventura’s System Settings replaced it.

The application’s restore interface featured a single Finder-like window, much like today’s. Internally, Time Machine scheduled its backups using a system timer and launchd, making backups every hour regardless of what else the Mac might have been doing at the time.

The initial version of Time Machine was both praised and slated. Unlike Mike Bombich’s rival Carbon Copy Cloner, it couldn’t create bootable backups, and there were problems with FileVault encryption, which at that time could only encrypt Home folders, rather than whole volumes. Despite those, its introduction transformed the way that many used their Macs, and made it more usual for users to have backups.

From its release, Time Machine was dependent on features of the HFS+ file system to create its Finder illusion. Every hour the backup service examined the record of changes made to the file system since the last backup was made, using its FSEvents database. It thus worked out what had changed and needed to be copied into the backup. During the backup phase itself, it only copied across those files that had been created or changed since the last backup was made.

It did this by using hard links in the backup, and Apple added a new feature to its HFS+ file system to support this, directory hard links. Where an entire folder had remained unchanged since the last backup, Time Machine simply created a hard link to the existing folder in that backup. Where an existing file had been changed, though, the new file was written to the backup inside a changed folder, which in turn could contain hard links to its unchanged contents.

This preserved the illusion that each backup consisted of the complete contents of the source, while only requiring the copying of changed files, and creation of a great many hard links to files and folders. It was also completely dependent on the backup volume using the HFS+ file system, to support those directory hard links.

Without directory hard links, backups would quickly have become overwhelmed by hard links to files. If you had a million files and folders on the backup source volume, every hourly backup would have had to create a total of a million copied files or hard links. Directory hard links thus enabled the efficiency needed for this novel scheme to work.

Apple later introduced what it termed Mobile Time Machine, intended for notebooks that could be away from their normal backup destination for some time. In around 10,000 lines of code, Mac OS X came to create something like a primitive snapshot, only on HFS+.

When macOS introduced its new DAS-CTS scheduling and dispatch system for background activities, in (about) Sierra, Time Machine’s backups were added to that. That proved unfortunate at the time because of a bug in that system, which failed on Macs left running continuously for several days, when backups could become infrequent and irregular.

When Apple released the first version of APFS on Mac OS X in High Sierra, its new snapshot feature was immediately incorporated into Time Machine to replace the earlier Mobile variant. Initially, APFS snapshots were also used instead of the FSEvents database to determine what should be backed up. Since then, making each backup of an APFS volume has involved creating a snapshot that’s stored locally on the APFS volume being backed up. In High Sierra and Mojave, the structure of backups themselves didn’t change, so they still required an HFS+ volume and relied on directory hard links.

Catalina introduced a more complicated scheme to replace snapshots as the usual means for determining what to back up. This was presumably because computing a snapshot delta had proved slow. As the backup destination remained in HFS+ format that could’t use snapshots, it continued to rely on directory hard links.

Big Sur and its successors with Signed System Volumes (SSVs) retained the option to continue backing up to HFS+ volumes, but added the ability to back up APFS volumes to APFS backup storage at last.

When backing up to APFS, Time Machine reverses the design used in High Sierra: instead of using snapshots to determine what needed to be backed up before creating a backup using traditional hard links, most of the time Time Machine determines what has changed using the original method with FSEvents, then creates each backup as a synthetic snapshot on the backup store. Unlike earlier versions, in Big Sur and later Time Machine can’t back up the System volume.

Once Time Machine has made a detailed assessment of the items to be backed up, it forecasts the total size to be copied. The local snapshot is copied to an .inprogress folder on the backup volume, and backup copying proceeds. Where possible, only changed blocks of files are copied, rather than having to copy the whole of every file’s data, an option termed delta-copying that can result in significant savings. Old backups are removed both according to age, and to maintain sufficient free space on the backup volume, in what Time Machine refers to as age-based and space-based thinning.

Data copied to assemble the backup on the backup volume is formed into a synthetic snapshot used to present the contents of that backup both in the Time Machine app and the Finder. Those snapshots are presented in /Volumes/.timemachine/ although they’re still stored on the backup volume.

Although modern Time Machine backups to APFS are both quicker and more space-efficient, the structure of backup storage poses problems. Copying backup stores on HFS+ was never easy, but there are currently no tools that can transfer those on APFS to another disk.

Behind the familiar interfaces of its app and settings, Time Machine has come a long way over the last few years, from building an illusion using huge numbers of hard links to creating synthetic snapshots.

Every autumn/fall, the current version of macOS changes, and with it there are changes great and small that can affect the apps we run. If you use any of the free apps that I provide here, now is the time to check that you’re running the correct version to support both your current macOS, and any that you might aspire to in the coming months.

SilentKnight

Although most of my apps have auto-update mechanisms that inform you when their updates are available, there are some notable pitfalls that can lull you into a sense of false security. Most importantly, SilentKnight was upgraded to version 2 two years ago to ensure its compatibility with Catalina and later. Every few days I come across someone who is still using version 1 with a newer release of macOS and seeing incorrect results. If you use SilentKnight in any version of macOS from Catalina onwards, then please ensure that it’s updated to the current version 2.10:
SilentKnight 2.10 (Universal App for Catalina to Sequoia)

This is particularly important if you intend upgrading to Sequoia, because of the changes it brings in how XProtect is updated. If you’re still running 2.9 or earlier, then SilentKnight will give you incorrect versions for XProtect, and at worst could report a version of 0 (zero) as it might not be able to find XProtect at all.

Skint and SystHist

For the same reason, Skint should be updated to version 1.08:
Skint 1.08 (Universal App for Monterey, Ventura, Sonoma and Sequoia only)

SystHist lists full system and security update installation history, a task that invariably requires an annual update to cope with the quirks of the new version of macOS. If you’re aiming for Sequoia at some stage, ensure that you have updated it to version 1.20:
SystHist 1.20 (Universal App for High Sierra, Mojave, Catalina, Big Sur, Monterey, Ventura, Sonoma and Sequoia)

Writing Tools

Although Apple isn’t intending to release any of its new AI features in the initial version of Sequoia, 15.0, but is delaying them for 15.1, you might like to prepare for that by updating my rich text editor and PDF viewer in advance. Their latest versions should prove fully compatible with Writing Tools when they’re released.

DelightEd is a Rich Text (RTF) editor with special Dark Mode features and support for interlinear text, and version 2.3 should work fully with Writing Tools:
DelightEd 2.3 (Universal App for High Sierra, Mojave, Catalina, Big Sur, Monterey, Ventura, Sonoma and Sequoia)

Podofyllin is a lightweight PDF viewer (without any editing capability, so it can’t alter original PDF files) and shows source code and more. Version 1.3 should work fully with Writing Tools:
Podofyllin 1.3 (Universal App for High Sierra, Mojave, Catalina, Big Sur, Monterey, Ventura, Sonoma and Sequoia)

XProCheck, Nalaprop, Precize

Other recent updates you might have missed include the following.

XProCheck to check on XProtect Remediator scans completed and reported in the log:
XProCheck 1.6 (Universal App for Catalina, Big Sur, Monterey, Ventura, Sonoma and Sequoia)

Nalaprop for multilingual natural language parsing, now compatible with Writing Tools:
Nalaprop 1.3 (Universal App for Mojave, Catalina, Big Sur, Monterey, Ventura, Sonoma and Sequoia)

Precize, which looks deep into files, bundles and folders to show their full size including extended attributes, provides macOS Bookmarks and volfs paths as enduring file references, and detailed information contained in Bookmarks and Aliases:
Precize 1.15 (Universal App for High Sierra, Mojave, Catalina, Big Sur, Monterey, Ventura, Sonoma and Sequoia)

Key points

• For Catalina or later, particularly Sequoia, use SilentKnight 2.10.
• For Sequoia in particular, use Skint 1.08.
• For Sequoia in particular, use SystHist 1.20.
• Older versions of those apps will give incorrect results when run in more recent versions of macOS.

With macOS Sequoia fast approaching from the horizon comes the question as to how to upgrade and update, whether to Sequoia or one of its recent predecessors. If you’re happy to go with what Software Update offers, then that’s usually simplest and most efficient. This article considers what you should do if you want something different, from updating to any previous version, to using a single installer to update several different Macs.

Procedures given here should work with all versions of macOS from Monterey onwards. They may work too with Big Sur, but its installers weren’t always as reliable, so you should there be well-prepared to have to migrate from a backup in case the installation creates a fresh, empty Data volume instead of firmlinking up to your existing one.

Which installer?

As Apple discontinued standalone updater packages when it introduced Big Sur, the choice now is between downloading the full Installer app, and performing the process in Recovery mode. The latter severely limits your choice to what it’s prepared to offer, so you’re almost certainly going to need to obtain the full Installer for the version of macOS you want. Rather than use the Installer app provided in the App Store, download the Installer package from the links given by Mr. Macintosh. Those provide a package that’s easier to store and move around, unlike the Installer app itself. It will typically be a little over 13.5 GB, and works on both Intel and Apple silicon Macs.

Standard procedure

As with any update or upgrade, first ensure you have a full recent backup before starting. If anything does go wrong during the procedure you’ll then be able to perform a fresh install and migrate from that backup.

Unless you want to install everything afresh and migrate from your backup, don’t try erasing either your System or Data volume. You’d have to do that in Recovery mode anyway, limiting your options as to which version of macOS you can install unless you create a bootable installer first.

Double-click the installer package to launch it in the Installer utility. The default is to save the Installer app to your current Applications folder, which should work fine as long as you remember to delete it once you’ve finished. Once complete, launch that Installer app and follow its instructions.

When macOS restarts at the end of the process, check the version now running, confirm that your Data volume has survived intact, and run SilentKnight to ensure that all security data files are up-to-date.

Recovery

Intel Macs have a slight advantage when it comes to installing macOS in Recovery mode, as depending on the keys held during startup, you should be able to coax a choice of versions out of an Intel system. Unless you simply want to install or update to the current version, though, you’ll probably want to avoid doing so in Recovery.

There’s another good reason for not using Recovery, in that delivery of installers to Macs running in Recovery can be painfully slow, and you may well be in for a longer wait than if you downloaded the Installer direct.

However, if you want to erase the current boot volume group on your Mac’s internal storage so you can install a fresh copy of macOS and restore the contents of its Data volume from backups, Recovery is normally the best place to do that. Apple works through the process for Intel Macs, and Apple silicon models. The key step is to select the Macintosh HD boot volume group and click on the Erase tool to perform Erase Volume Group.

When the SSV was first introduced in Big Sur, there were many problems resulting from erasing just one volume in the boot volume group. If that happened to be the System volume, when macOS was installed it created a new firmlinked Data volume, leaving the existing Data volume as an orphan. That was usually done in a misguided attempt to have a fresh install of the System volume and SSV while keeping the existing contents of the Data volume, but doesn’t do that. Every installation of the SSV in any given version of macOS since Big Sur is identical, so it isn’t necessary to erase it, but simply to install or update macOS.

Bootable installer disk

Another traditional way to install macOS is using a bootable installer disk, normally a USB ‘thumb’ drive, although you can also create a small HFS+ volume for the purpose on an external SSD. Apple provides detailed instructions for doing this using a range of versions of macOS.

In many cases, installing a version of macOS older than the one that’s currently running requires this, as old Installers usually fail to run in newer macOS. Unfortunately, on Apple silicon Macs, this isn’t the powerful tool that it once was, as the Mac doesn’t boot fully from the external disk, and as a result it has no role in dealing with problems with internal storage.

Virtual Machines on Apple silicon

Installer apps and Recovery installs both work fine in virtual machines running on Apple silicon hosts. However, there’s one special circumstance you need to beware of. One of the major new features in virtualisation in Sequoia is support for iCloud and some other services dependent on Apple ID. If you want to use those, then the VM must be created new in Sequoia, using a Sequoia IPSW image. You can’t update or upgrade an existing VM from a previous version of macOS and use iCloud services in it.

Summary

• If you can, use Software Update to update or upgrade macOS, as it minimises download size and is simplest.
• If you want to perform a different update, or run one installer on several Macs, download and use the appropriate Installer package.
• If you want to erase the existing system including all your data, use Recovery mode to erase the whole volume group, then install macOS and migrate from your backup.
• Never erase only your Mac’s System volume, as that will orphan its current Data volume.
• If you want to downgrade to an older version of macOS, you’ll probably need to do so from a bootable installer disk.
• If you want a VM to use iCloud, then create a fresh VM using a Sequoia IPSW, as an upgraded VM can’t access iCloud.

My previous explanation of how recent versions of macOS merge their System and Data volumes into what appears to be a single volume, omitted a third component, including Safari. Look in the System/Applications folder where all the bundled apps are stored on the SSV, and there’s no Safari to be seen, yet it appears in the top-level Applications folder. This article explains how that now works using cryptexes, and how they differ between Intel and Apple silicon Macs.

Finding Safari

As the modern boot volume group evolved through Catalina to Big Sur, Safari and its supporting frameworks were stored in the Data volume. That stopped with the arrival of Ventura, and they’re now stored in the third components that complete the modern boot volume group. You can see when files are stored on a different volume using my free app Precize to reveal their full paths. Use that to examine three apps from the merged Applications folder, and you’ll understand what I mean:

• Chess.app has a path of /System/Applications/Chess.app demonstrating that it’s one of the apps bundled in the SSV, where almost all of the System folder is stored.
• Cirrus.app, like any other app you have installed, has a path of /Applications/Cirrus.app, making it clear that it’s stored on the writable Data volume.
• Safari.app has the weird path of /System/Volumes/Preboot/Cryptexes/App/System/Applications/Safari.app that demands further explanation.

Note that the Finder’s Get Info dialogs aren’t as truthful, and don’t tell the full story.

Their volfs paths are also worth noting. On my Intel Mac, they are:

• Chess.app is at /.vol/16777240/1152921500311883667; because all macOS 14.6.1 SSVs are identical, your Chess.app should have the same inode number too.
• Cirrus.app is at /.vol/16777240/461665725
• Safari.app is at /.vol/16777238/993517

The first two follow a familiar pattern you’ll see throughout the System and Data volumes: their volume ID 16777240 is common to both, and that assigned to the merged volumes, but their inode numbers are wildly different. Huge numbers like 1152921500311883667 come from the SSV, while smaller ones like 461665725 are from the Data volume. Then there’s a slightly lower volume ID of 16777238 and a small inode number of 993517 for Safari, demonstrating that it’s somewhere altogether different: that’s a cryptex, a cryptographically protected disk image with an interesting history.

Why a cryptex?

When the modern boot volume group was being designed and developed, it took into account Safari’s special needs by making it the only bundled app to be stored in the Data volume. This enables it to be updated without having to go through the whole process of building a new SSV, allowing Apple to deliver urgent security patches to Safari and its underlying WebKit and other frameworks. There could also have been political considerations in separating Apple’s bundled browser from the other apps included in macOS.

This changed in Ventura in the autumn/fall of 2022, when Apple applied technology it had originally developed for its customised iPhone, the Security Research Device, dubbed the cryptex, a name formed as a portmanteau for CRYPTographically sealed EXtension. This offers two advantages:

• Safari, its supporting frameworks, and other components of macOS that Apple prefers not to build into the SSV, can be delivered in cryptexes. As I’ll explain later, this also enables tailoring of macOS to platform.
• Some urgent security patches could be delivered in cryptexes, making them faster to release and simpler to install in a Rapid Security Response (RSR).

Since then, RSRs seem to have had their day, and appear to have fallen from favour. But, as a means of delivering Safari and other more changeable components of macOS, cryptexes have proved their worth.

How a cryptex works

Although a cryptex is at heart a read-only disk image that is mounted during the boot process, it has two properties of particular importance:

• Its contents are cryptographically verified, in much the same way that the contents of the SSV are, using hashes of its entire contents.
• Its internal file system is grafted into the root file system when it’s mounted, rather than being mounted as a separate volume.

Mounting a cryptex starts with validation of the payload and its manifest. It then undergoes a sequence of processes similar to the mounting of an APFS volume, with a checkpoint search to establish stable checkpoint indices, and a check to discover whether there’s anything to recover, which seems unlikely. The graft is then performed in a series of opaque steps, with root hash authentication and validation. The object ID is found, and the graft completed.

Once this has been completed for each of the standard cryptexes and any installed RSRs, the contents of those are effectively part of the system, as a hybrid of the SSV and cryptexes. In the case of the Safari app, this process effectively places it in the main Applications folder, even though the original app is actually located in the System/Applications folder of the App cryptex in /System/Volumes/Preboot/Cryptexes.

As with the current boot System and Data volumes, grafted cryptexes aren’t unmounted or ungrafted until shutdown.

There are currently three main cryptexes in use, App containing Safari, its frameworks and other supporting files, and OS, with a range of other system items including additional frameworks, and several large dyld shared caches. You’ll also see an Incoming cryptex in /System/Volumes/Preboot/Cryptexes. As they’re outside the SSV, new and replacement cryptexes are installed without rebuilding the SSV, and in some cases don’t even need a soft restart of macOS.

Architecture-specific cryptexes

In addition to providing Safari and its related components, cryptexes also provide useful economy in shared caches, and explain why macOS updates for Apple silicon Macs are invariably larger than those for Intel models.

While the contents of the SSV appear to be identical on both Intel and Apple silicon, thus have a single signature, the two architectures differ in their cryptexes. Those for Apple silicon Macs contain dyld shared caches for both architectures, and a set of aot shared caches, presumably to support Rosetta 2, and amounting to 5.24 GB in total size; those for Intel Macs only contain Intel dyld shared caches of 1.68 GB total size.

Given their sizes, that’s a valuable efficiency both for updates and in storage required, and is the major reason for updates for Apple silicon Macs always being larger than those for Intel. Thankfully, because those shared caches are supplied compressed, the difference in update sizes is much smaller than the 3.56 GB difference when they’re decompressed and installed.

Apple has just released an update to XProtect Remediator security software for Catalina or later, bringing it to version 145. The previous version was 142.

Apple doesn’t release information about what security issues this update might add or change. There are no changes in the number or names of its scanning modules, and Bastion rules also remain unchanged.

You can check whether this update has been installed by opening System Information via About This Mac, and selecting the Installations item under Software.

A full listing of security data file versions is given by SilentKnight, LockRattler and SystHist for El Capitan to Sonoma available from their product page. If your Mac has not yet installed these updates, you can force them using SilentKnight, LockRattler, or at the command line.

If you want to install this as a named update in SilentKnight, its label is XProtectPayloads_10_15-145.

I have updated the reference pages here which are accessed directly from LockRattler 4.2 and later using its Check blog button.

I maintain lists of the current versions of security data files for Sonoma on this page, Ventura on this page, Monterey on this page, Big Sur on this page, Catalina on this page, Mojave on this page, High Sierra on this page, Sierra on this page, and El Capitan on this page.

Over a series of three articles last week, I pored over thousands of log entries to examine how macOS Sonoma 14.6.1 checks applications it’s launching, under normal full security settings, with reduced security, and for known malware. This article draws together my conclusions from those tests run in virtual machines on an Apple silicon Mac.

Layered security

Like other security functions in macOS, app launch security is built in layers, including checks of

• code-signing certificates (multiple times);
• CDHashes, including their consistency, and against Apple’s database for notarized apps, and their revocation;
• quarantine extended attributes, which normally trigger a user consent dialog, and may result in app translocation;
• previous launch, in the LaunchServices database;
• matches with Yara rules in XProtect’s data;
• user consent to a first launch prompt dialog;
• launch and other constraints.

Additional data may also be collected and stored in the provenance database that first appeared in Ventura.

Not all checks are performed on every launch of an app. At a minimum, for a notarized app that has been run only recently, these might consist of only local checks against CDHashes and with the app’s existing entry in the LaunchServices database. Checks are also modified by reducing security settings:

• Disabling Gatekeeper checks doesn’t stop those checks from taking place, but apparently ignores some results, notably those obtained by XProtect. It doesn’t affect checks of CDHashes against Apple’s database.
• Disabling SIP has more pervasive effects in largely disabling the com.apple.syspolicy sub-system, affecting several layers, although checks of CDHashes against Apple’s database are unaffected.

com.apple.syspolicy

In full security conditions, one sub-system dominates log entries concerning app launch security, com.apple.syspolicy. This is clearest in Gatekeeper and XProtect checks. Although the log entries that follow may appear bewildering, they are the best illustration of this point.

When launching a notarized app that hasn’t previously been run on that Mac and has a quarantine xattr, Gatekeeper and XProtect scans are reported in the following sequence of entries:
com.apple.syspolicy.exec GK process assessment: <private> <-- (<private>, <private>)
com.apple.syspolicy.exec Gatekeeper assessment rooted at: <private>
com.apple.syspolicy.exec Skipping TCC check due to process: 692, 0, 692
com.apple.syspolicy.exec queueing up scan for code: PST: (vuid: 7C5C43BF-A338-4228-B61E-5038F1D93EDB), (objid: 62947), (team: (null)), (id: (null)), (bundle_id: (null))
com.apple.syspolicy.exec starting work for scan for code: PST: (vuid: 7C5C43BF-A338-4228-B61E-5038F1D93EDB), (objid: 62947), (team: (null)), (id: (null)), (bundle_id: (null))
com.apple.syspolicy.exec allowUI is YES, creating codeEval object: PST: (vuid: 7C5C43BF-A338-4228-B61E-5038F1D93EDB), (objid: 62947), (team: (null)), (id: (null)), (bundle_id: (null))
com.apple.syspolicy.exec Adding default exception for team: <private>
com.apple.syspolicy.exec Registered app bundle for protection: PST: (vuid: 7C5C43BF-A338-4228-B61E-5038F1D93EDB), (objid: 62947), (team: QWY4LRW926), (id: (null)), (bundle_id: (null))
com.apple.syspolicy.exec GK performScan: PST: (vuid: 7C5C43BF-A338-4228-B61E-5038F1D93EDB), (objid: 62947), (team: QWY4LRW926), (id: (null)), (bundle_id: (null))
com.apple.xprotect XProtectScan beginAnalysisWithResultsHandler continueOnError is set to 0
com.apple.xprotect XPAssessment performAnalysisOnFileImpl continueOnError set to 0
com.apple.xprotect Xprotect is performing a direct malware and dylib scan: <private>

Those checks later complete in entries such as:
com.apple.syspolicy.exec GK Xprotect results: PST: (vuid: 7C5C43BF-A338-4228-B61E-5038F1D93EDB), (objid: 62947), (team: QWY4LRW926), (id: (null)), (bundle_id: (null)), XPScan: 0,-7676743164328624005,2024-08-26 08:19:01 +0000,(null)
com.apple.syspolicy.exec GK scan complete: PST: (vuid: 7C5C43BF-A338-4228-B61E-5038F1D93EDB), (objid: 62947), (team: QWY4LRW926), (id: (null)), (bundle_id: (null)), 4, 4, 0
com.apple.syspolicy.exec scan finished, waking up any waiters: PST: (vuid: 7C5C43BF-A338-4228-B61E-5038F1D93EDB), (objid: 62947), (team: QWY4LRW926), (id: co.eclecticlight.SystHist), (bundle_id: co.eclecticlight.SystHist)
com.apple.syspolicy.exec App gets first launch prompt because responsibility: <private>, <private>
com.apple.syspolicy.exec GK evaluateScanResult: 0, PST: (vuid: 7C5C43BF-A338-4228-B61E-5038F1D93EDB), (objid: 62947), (team: QWY4LRW926), (id: co.eclecticlight.SystHist), (bundle_id: co.eclecticlight.SystHist), 1, 0, 1, 0, 4, 4, 0
com.apple.syspolicy.exec GK eval - was allowed: 1, show prompt: 1
com.apple.syspolicy.exec Skipping TCC check due to process: 692, 0, 692
com.apple.syspolicy Found console users: <private>
com.apple.syspolicy.exec Prompt shown (5, 0), waiting for response: PST: (vuid: 7C5C43BF-A338-4228-B61E-5038F1D93EDB), (objid: 62947), (team: QWY4LRW926), (id: co.eclecticlight.SystHist), (bundle_id: co.eclecticlight.SystHist)

When SIP has been disabled, there are precious few entries from com.apple.syspolicy or com.apple.syspolicy.exec. Instead, XProtect appears to be left to its own devices, and doesn’t fare well:
com.apple.xprotect XPAssessment performAnalysisOnFileImpl continueOnError set to 0
com.apple.xprotect XprotectService Calling SecAssessmentCreate with URL <private>, context <private>
XprotectService SecTrustEvaluateIfNecessary
com.apple.xprotect XprotectService Bundle is not apple signed
com.apple.xprotect XprotectService Bundle size result: 18388222 (YES)
com.apple.xprotect XprotectService Always scan: YES
com.apple.xprotect XprotectService Starting malware scan for: <private>
kernel XprotectService [697] crossed memory high watermark (15 MB); EXC_RESOURCE
kernel Full corpse enqueued for XprotectService
com.apple.xnu memorystatus kernel kernel EXC_RESOURCE -> XprotectService[697] exceeded mem limit: ActiveSoft 15 MB (non-fatal)
ReportCrash event condition bump 0 -> 1
ReportCrash post-exception thread qos drop 21 -> 17
ReportCrash PID 697 exceeded the memory high watermark; Invoking ReportMemoryException with corpse.

There are no other entries referring to Gatekeeper or those checks. The effects of disabling SIP appear extensive and pervasive throughout several of the layers of app launch security.

CDHashes are central

With the adoption of notarization, apps run in macOS should now fall into one of five categories:

• signed by Apple, either its own apps or those delivered through its App Store;
• notarized by Apple, with its CDHashes added to Apple’s database;
• signed (either with a Developer certificate, or ad hoc) locally, and not distributed over the internet, with its own unique CDHashes;
• unwanted or malicious, with revoked CDHashes,
• unrecognised, and potentially malicious.

These emphasise the importance of the online ‘notarization’ checks of CDHashes performed in all circumstances where macOS doesn’t have previous records of saved CDHashes for that code. Their primary purpose isn’t to validate notarization, but to identify code as known good, known bad, or unknown. When Apple’s security engineers identify new malware, its CDHashes can quickly be added to the database as being revoked, so ensuring that all subsequent checks of the same CDHash will be classified as revoked, for malicious code. This is a rapid response that should have no false positives, in which benign code is mistakenly identified as being malicious.

Typically, the checking sequence is reported in the log with:
com.apple.syspolicy looking up ticket: <private>, 2, 1
com.apple.syspolicy cloudkit record fetch: <private>, <private>
com.apple.syspolicy cloudkit request cache info: <private>, max-age=300
com.apple.syspolicy CKTicketStore network reachability: 1, Mon Aug 26 09:15:45 2024
com.apple.syspolicy Inserting ticket: <private>
com.apple.syspolicy completing lookup: <private>, 0
[and so on with further lookups]
and those are among the only entries from com.apple.syspolicy seen when SIP is disabled.

When full security is enabled, those are completed with
com.apple.syspolicy.exec GK evaluateScanResult: 0, PST: (vuid: 7C5C43BF-A338-4228-B61E-5038F1D93EDB), (objid: 62947), (team: QWY4LRW926), (id: co.eclecticlight.SystHist), (bundle_id: co.eclecticlight.SystHist), 1, 0, 1, 0, 4, 4, 0
But when SIP is disabled, those don’t appear, and seem to be substituted by application of Security rule 11 instead.

The downside of CDHash checks is that their false negative rate can be alarmingly high. Change a single bit in the code being hashed, and the hash will amplify that change, and is completely different. Hence the importance of notarization to establish which CDHashes definitely aren’t from malicious code.

One threat to this system occurs when a user mistakenly blocks their Mac from connecting to Apple’s database using CloudKit, for example using a misconfigured software firewall. Without a suitable vulnerability, malicious software shouldn’t be able to use this approach to block a payload from being checked.

I don’t know whether any third-party security products use a similar checking mechanism with their own local or remote CDHash databases, but this appears to be a great advantage to the protection built into macOS.

Performance

Two of the checks performed with full security enabled are dependent on the size of the app being checked. Fully validating an app’s CDHashes against those in its signature or notarization ticket should benefit from hardware acceleration, particularly on Apple silicon, and can be tackled hierarchically. It appears unlikely to result in significant delays to launching an app.

XProtect scans are more likely to be responsible for observable delays in app launch times, though. With the recent growth in the number of Yara rules, and their length, scans performed after an app’s first launch are the most probable cause of large and complex app bundles requiring several seconds before the app can be run.

Summary

I have updated the flow chart I first proposed as a result of observations made of app launches in Sonoma 14.4.1:

This is also available as a tear-out PDF here: launchsonomaapp2

I welcome any evidence that will refine and improve that, please.

Previous articles

Launching apps in Sonoma 14.6.1: Full security
Launching apps in Sonoma 14.6.1: Reduced security
Launching apps in Sonoma 14.6.1: Known malware
How does Sonoma check an app before launch? (Sonoma 14.4.1)

I hope that you enjoyed Saturday’s Mac Riddles, episode 271. Here are my solutions to them.

1: Table of chapters as the top-level directory.

Click for a solution

Contents

Table of chapters (contents of book) as the top-level directory (it’s the top-level folder in a macOS app bundle).

2: Real estate inventory, personal possessions or XML.

Click for a solution

Property list

Real estate (property) inventory (list), personal possessions (a property list) or XML (how it’s coded in the Info.plist file).

3: Reserves that could be human, such as strings and images.

Click for a solution

Resources

Reserves (resources) that could be human (human resources), such as strings and images (what goes inside the Resources folder in an app).

The common factor

Click for a solution

They are all key parts of a macOS app bundle.

I look forward to your putting alternative cases.

Perhaps you just tried to save a document, only to be told you don’t have sufficient permissions to do so, or attempted to make another change to what’s on your Mac’s internal storage, with similar results. You then select the Macintosh HD disk in the Finder and Get Info. No wonder that didn’t work, as you only have read-only access to that disk. But if you unlock it and try to make any changes to permissions, you see

What’s going on?

Between macOS Mojave, with its single system volume, and Big Sur, the structure of the Mac system or boot volume has changed, with Catalina as an intermediate. Instead of Macintosh HD (or whatever you might have renamed it to) being one volume on your boot disk, it’s now two intertwined and joined together. What you see now as Macintosh HD isn’t even a regular APFS volume, but a read-only snapshot containing the current macOS. No wonder you can’t change it.

Root

Select the boot disk Macintosh HD in the Finder, and it appears to have four visible folders, Applications, Library, System and Users, just like it always did. Press Command-Shift-. to reveal hidden folders and all the usual suspects like bin, opt and usr are still where they should be. That’s the root of the combined System and Data volumes, and what’s shown there is a combination of folders on both volumes, with the top level or root on the Sealed System Volume (SSV).

The contents of those folders are also the result of both volumes being merged together using what Apple terms firmlinks:

• Applications contains apps installed in your own Applications folder on the Data volume, and those bundled in macOS on the SSV. You can see just the latter in the path System/Applications, where they appear to be duplicated, but aren’t really.
• Library comes only from the Data volume, and all its contents are on that volume. But inside it, in the path Library/Apple/System/Library are some components that should appear in the main System/Library.
• System comes only from the SSV, although it has some contents merged into it using firmlinks, such as those folders in Library.
• Users also comes only from the Data volume, and includes all Home folders for users.

So while the root of Macintosh HD might be in the SSV, much of its contents are on the Data volume, and can be written to, even though the root is a read-only snapshot, thanks to those firmlinks.

Data volume

There are two places that mounted volumes are listed in the Finder: the hidden top-level folder Volumes, where Macintosh HD is just a link back to the root complete with its merged volumes, and in System/Volumes, where what’s shown as Macintosh HD is in fact not the merged volumes, but only the Data volume. You can confirm that by looking at what’s in System/Volumes/Macintosh HD/System, where you only see the parts of the System folder that are stored on the Data volume, and not those stored on the SSV.

What is more confusing there is that System/Volumes/Macintosh HD/Applications is the same merged folder containing both user and bundled apps as in the top-level Applications folder. That’s an artefact resulting from the way that its firmlink works.

But if you open the Get Info dialog on System/Volumes/Macintosh HD, you’ll see the same as with the root Macintosh HD disk, information about the root and not the Data volume.

Mounted in System/Volumes are several other volumes like VM and Preboot, and (depending on whether this is an Intel or Apple silicon Mac) folders such as Recovery and xarts, that you really don’t want to mess with.

Permissions problems

Tackling problems that appear to be the result of incorrect permissions is best done at the lowest folder level. If you’re trying to save a document to the Documents folder inside your Home folder, select that and Get Info on it. Chances are that you are the owner and have both Read & Write permissions as you should. In that case, the problem most likely rests with privacy protection as in Privacy & Security settings. You then suffer Catch-22, as you can only effect changes to those by closing and opening the app, and as you can’t save your document before closing the app, you’re at risk of losing its contents. You may have better luck trying a different folder, creating a new one inside your Home folder, or using the Save As… command instead (which may be revealed by holding the Option key when opening the File menu).

Full layout

In case you’re wondering exactly which folders are merged into the hybrid Macintosh HD ‘volume’, those are shown below in increasing levels of detail, starting with the broad layout.

Then to a simplified version of the full layout.

Finally, in complete detail.

Happy navigating!

If XProtect Remediator came of age in macOS Ventura, then it has been XProtect’s turn in Sonoma. Starting from version 2171 with 216 rules in under 3,000 lines in its Yara definitions, it emerged a year later in version 5272 with 347 rules in over 13,000 lines, although mercifully not after 3,100 versions.

I had always assumed that those Yara rules were compiled straightaway into something more tractable for checking executable code, but it seems that each time XProtect performs one of its ‘direct malware and dylib scans’, it first looks for a non-existent Yara file, then uses the rules in the XProtect.bundle, as it reports in the log:
com.apple.xprotect Xprotect is performing a direct malware and dylib scan: <private>
com.apple.xprotect Rule path is not accessible: /Library/Apple/System/Library/CoreServices/XProtect.bundle/Contents/Resources/XProtect2.yara
com.apple.xprotect Using XProtect rules location: /Library/Apple/System/Library/CoreServices/XProtect.bundle/Contents/Resources/XProtect.yara

Apparently, to cope with this explosive growth, and potentially support more frequent tweaks to its growing horde of Yara rules, macOS Sequoia is changing the way that XProtect’s data is updated and managed. A chance find by @L0Psec revealed how this has moved beyond those updates delivered by softwareupdate, and a new command tool xprotect handles this separately in CloudKit.

Last week’s update to XProtect’s Yara file was an experience those beta-testing Sequoia 15.0 or 15.1 must have found profoundly confusing, and I quickly became aware of reports that were changing by the minute.

When XProtect 5272 was first made available through softwareupdate, Sonoma and earlier systems found and installed it as usual, as did some running Sequoia betas. That updated the visible XProtect.bundle in CoreServices, but didn’t update XProtect according to its new xprotect command tool, which still reported the local version of XProtect as 5271. Without knowing how XProtect has changed, the user would most likely see this as a bug.

A little later, I saw reports of Sequoia installations apparently updating spontaneously via CloudKit, using its new mechanism, which did change the version reported by xprotect version.

At this stage, I had a 15.0 virtual machine that had updated ‘correctly’ via CloudKit, and its host 15.1 system that had updated its bundle via softwareupdate, but still wasn’t apparently running the new version afterwards. Those of us who didn’t experience a spontaneous CloudKit update were left in limbo. I had originally changed the version databases used by SilentKnight and Skint to show a correct version of 5272 for Sequoia, and hurriedly had to revert that to 5271 before I became inundated with complaints from those whose Macs hadn’t been able to update.

It then occurred to me to try using the xprotect command to force a CloudKit update on my 15.1 system. I first entered
sudo xprotect check
only to be told that the version available was still 5271. But when I ran
sudo xprotect update
a miracle happened, with the response
Update succeeded: Activated update LocalUpdate[5272]

That command had convinced macOS to ‘activate’ the updated bundle in /Library/Apple/System/Library/CoreServices rather than waiting for it to become available from CloudKit, a feature not mentioned in its man page or usage info. I returned to my version databases to change them a third time, back to 5272.

Previous XProtect updates such as 5271 that were obtained through CloudKit are now identified by SystHist as XProtectCloudKitUpdate, while those obtained by softwareupdate and activated using the xprotect command appear as standard XProtectPlistConfigData, as they do in Sonoma and earlier.

With the release of Sequoia due later this month, the xprotect command tool and XProtect’s new CloudKit updates have already encountered troubled water. If Apple stays true to form and doesn’t mention a word about this change, or its effect on XProtect updates, many of the millions of new Sequoia users could end up falling behind. But as we’re not supposed to know what the latest version is, nor which is currently active on our Macs without taking to Terminal’s command line, maybe most won’t be allowed to notice.

I’d like to think that Apple will explain these changes to users, document its new command tool properly, and ensure that users know the current version of XProtect data, and can check whether their Mac is up to date without having to resort to Terminal or third-party products, perhaps in System Information. Will I be disappointed?

Here are this weekend’s Mac riddles to entertain you through family time, shopping and recreation.

1: Table of chapters as the top-level directory.

2: Real estate inventory, personal possessions or XML.

3: Reserves that could be human, such as strings and images.

To help you cross-check your solutions, or confuse you further, there’s a common factor between them.

I’ll post my solutions first thing on Monday morning.

Please don’t post your solutions as comments here: it spoils it for others.