Normal view

There are new articles available, click to refresh the page.
Before yesterdayMain stream

Updating macOS with an Installer and in Recovery

By: hoakley
5 September 2024 at 14:30

With macOS Sequoia fast approaching from the horizon comes the question as to how to upgrade and update, whether to Sequoia or one of its recent predecessors. If you’re happy to go with what Software Update offers, then that’s usually simplest and most efficient. This article considers what you should do if you want something different, from updating to any previous version, to using a single installer to update several different Macs.

Procedures given here should work with all versions of macOS from Monterey onwards. They may work too with Big Sur, but its installers weren’t always as reliable, so you should there be well-prepared to have to migrate from a backup in case the installation creates a fresh, empty Data volume instead of firmlinking up to your existing one.

Which installer?

As Apple discontinued standalone updater packages when it introduced Big Sur, the choice now is between downloading the full Installer app, and performing the process in Recovery mode. The latter severely limits your choice to what it’s prepared to offer, so you’re almost certainly going to need to obtain the full Installer for the version of macOS you want. Rather than use the Installer app provided in the App Store, download the Installer package from the links given by Mr. Macintosh. Those provide a package that’s easier to store and move around, unlike the Installer app itself. It will typically be a little over 13.5 GB, and works on both Intel and Apple silicon Macs.

Standard procedure

As with any update or upgrade, first ensure you have a full recent backup before starting. If anything does go wrong during the procedure you’ll then be able to perform a fresh install and migrate from that backup.

Unless you want to install everything afresh and migrate from your backup, don’t try erasing either your System or Data volume. You’d have to do that in Recovery mode anyway, limiting your options as to which version of macOS you can install unless you create a bootable installer first.

Double-click the installer package to launch it in the Installer utility. The default is to save the Installer app to your current Applications folder, which should work fine as long as you remember to delete it once you’ve finished. Once complete, launch that Installer app and follow its instructions.

sininstall2

When macOS restarts at the end of the process, check the version now running, confirm that your Data volume has survived intact, and run SilentKnight to ensure that all security data files are up-to-date.

Recovery

Intel Macs have a slight advantage when it comes to installing macOS in Recovery mode, as depending on the keys held during startup, you should be able to coax a choice of versions out of an Intel system. Unless you simply want to install or update to the current version, though, you’ll probably want to avoid doing so in Recovery.

sininstall3

There’s another good reason for not using Recovery, in that delivery of installers to Macs running in Recovery can be painfully slow, and you may well be in for a longer wait than if you downloaded the Installer direct.

However, if you want to erase the current boot volume group on your Mac’s internal storage so you can install a fresh copy of macOS and restore the contents of its Data volume from backups, Recovery is normally the best place to do that. Apple works through the process for Intel Macs, and Apple silicon models. The key step is to select the Macintosh HD boot volume group and click on the Erase tool to perform Erase Volume Group.

When the SSV was first introduced in Big Sur, there were many problems resulting from erasing just one volume in the boot volume group. If that happened to be the System volume, when macOS was installed it created a new firmlinked Data volume, leaving the existing Data volume as an orphan. That was usually done in a misguided attempt to have a fresh install of the System volume and SSV while keeping the existing contents of the Data volume, but doesn’t do that. Every installation of the SSV in any given version of macOS since Big Sur is identical, so it isn’t necessary to erase it, but simply to install or update macOS.

Bootable installer disk

Another traditional way to install macOS is using a bootable installer disk, normally a USB ‘thumb’ drive, although you can also create a small HFS+ volume for the purpose on an external SSD. Apple provides detailed instructions for doing this using a range of versions of macOS.

In many cases, installing a version of macOS older than the one that’s currently running requires this, as old Installers usually fail to run in newer macOS. Unfortunately, on Apple silicon Macs, this isn’t the powerful tool that it once was, as the Mac doesn’t boot fully from the external disk, and as a result it has no role in dealing with problems with internal storage.

Virtual Machines on Apple silicon

Installer apps and Recovery installs both work fine in virtual machines running on Apple silicon hosts. However, there’s one special circumstance you need to beware of. One of the major new features in virtualisation in Sequoia is support for iCloud and some other services dependent on Apple ID. If you want to use those, then the VM must be created new in Sequoia, using a Sequoia IPSW image. You can’t update or upgrade an existing VM from a previous version of macOS and use iCloud services in it.

Summary

  • If you can, use Software Update to update or upgrade macOS, as it minimises download size and is simplest.
  • If you want to perform a different update, or run one installer on several Macs, download and use the appropriate Installer package.
  • If you want to erase the existing system including all your data, use Recovery mode to erase the whole volume group, then install macOS and migrate from your backup.
  • Never erase only your Mac’s System volume, as that will orphan its current Data volume.
  • If you want to downgrade to an older version of macOS, you’ll probably need to do so from a bootable installer disk.
  • If you want a VM to use iCloud, then create a fresh VM using a Sequoia IPSW, as an upgraded VM can’t access iCloud.

Copy speeds of large and sparse files

By: hoakley
7 August 2024 at 14:30

I have recently seen reports of very low speeds when copying large files such as virtual machines, in some cases extending to more than a day, even when they should have been sparse files, so requiring less time than would be expected for their full size. This article teases out some tests and checks that you can use to investigate such unexpectedly poor performance.

Expected performance

Time taken to copy or duplicate files varies greatly in APFS. Copies and duplicates made within the same volume should, when performed correctly, be cloned, so should happen in the twinkling of an eye, and without any penalties for size. This is regardless of whether the original is a sparse file, or a reasonably sized bundle or folder, whose contents should normally be cloned too. If cloning doesn’t occur, then the method used to copy or duplicate should be suspected. Apple explains how this is accomplished using the Foundation API of FileManager, using a copyItem() method. This is also expected behaviour for the Finder’s Duplicate command.

Copying a file to a different volume, whether it’s in the same container, or even on a different disk, should proceed as expected, according to the full size of the file, unless the original is a sparse file and both source and destination use the APFS file system. When an appropriate method is used to perform the copy between APFS volumes, sparse file format should be preserved. This results in distinctive behaviour in the Finder: at first, its progress dialog reflects the full (non-sparse) size of the file to be copied, and the bar proceeds at the speed expected for that size. When the bar reaches a point equivalent to the actual (sparse) size of the file being copied, it suddenly shoots to 100% completion.

Copying a sparse file to a file system other than APFS will always result in it expanding to its full, non-sparse size, and the whole of that size will then be transferred during copying. There is no option to explode to full size on the destination, nor to convert format on the fly.

External SSDs

When copying very large files, external disk performance can depart substantially from that measured using relatively small transfer sizes. While some SSDs will achieve close to their benchmark write speed, others will slow greatly. Factors that can determine that include:

  • a full SLC cache,
  • failure to Trim,
  • small write caches/buffers in the SSD,
  • thermal throttling.

Many SSDs are designed to use fast single-level cell (SLC) write caching to deliver impressive benchmarks and perform well in everyday use. When very large files are written to them, they can exceed the capacity of the SLC cache, and write speed then collapses to less than a quarter of that seen in their benchmark performance. The only solution is to use a different SSD with a larger SLC cache.

Trimming is also an insidious problem, as macOS by default will only Trim HFS+ and APFS volumes when they’re mounted if the disk they’re stored on has an NVMe interface, and won’t Trim volumes on SSDs with a SATA interface. The trimforce command may be able to force Trimming on SATA disks, although that isn’t clear, and its man page is forbidding.

Trimming ensures that storage blocks no longer required by the file system are reported to the SSD firmware so they can be marked as unused, erased and returned for use. If Trimming isn’t performed by APFS at the time of mounting, those storage blocks are normally reclaimed during housekeeping performed by the SSD firmware, but that may be delayed or unreliable. If those blocks aren’t released, write speed will fall noticeably, and in the worst case blocks will need to be erased during writing.

For best performance, SATA SSDs should be avoided, and NVMe used instead. NVMe is standard for USB 3.2 Gen 2 10 Gb/s, USB4 and Thunderbolt SSDs, which should all Trim correctly by default.

Disk images

Since Monterey, disk images with internal APFS (or HFS+) file systems have benefitted from an ingenious combination of Trimming and sparse file format when stored on APFS volumes. This can result in great savings in disk space used by disk images, provided that they’re handled as sparse files throughout.

When a standard read-write (UDIF) disk image is first created, it occupies its full size in storage. When that disk image is mounted, APFS performs its usual Trim, which in the case of a disk image gathers all free space into contiguous storage. The disk image is then written out to storage in sparse file format, which normally requires far less than its full non-sparse size.

This behaviour can save GB of disk space in virtual machines, but like other sparse files, is dependent on the file remaining on APFS file systems, otherwise it will explode to its full non-sparse size. Any app that attempts to copy the disk image will also need to use the correct calls to preserve its format and avoid explosion.

Tools

clonesparse3

Precize reports whether a file is a clone, is sparse, and provides other useful information including full sizes and inode numbers.

Sparsity can create test sparse files of any size, and can scan for them in folders.
Mints can inspect APFS log entries to verify Trimming on mounting, as detailed here.
Stibium is my own storage performance benchmark tool that is far more flexible than others. Performing its Gold Standard test is detailed here and in its Help book.

Securing virtual machines on Apple silicon

By: hoakley
18 July 2024 at 14:30

Using a virtual machine (VM) might seem an ideal platform for private computing. With all its data locked away in a VM that can’t otherwise be accessed from your Mac, it’s thoroughly safe from Spotlight indexing, access to old versions of documents, QuickLook previews, and everything else on your Mac’s Data volume that could give unintended access to its contents. But how secure could you make it? Prior to Sequoia, only as secure as the volume it’s stored on, it seems, which doesn’t make it so private after all.

In addition to Sequoia VMs on Apple silicon Macs being able to use services such as iCloud using Apple ID, they now appear able to support full-strength FileVault when Apple ID is activated. This contrasts with FileVault supported by previous macOS guests, which appears comparable to that provided by Intel Macs without T2 chips, or on external disks of any Mac, in that the Secure Enclave isn’t involved in protecting their encryption keys, as explained in Apple’s Platform Security Guide. Thus an attacker who has access to an older VM could copy that and attempt to gain access by brute force.

Apple hasn’t explained what is required for Sequoia VMs to support Apple ID, but it involves configuring “an identity for the VM that it derives from security information in the host’s Secure Enclave”. Support requires changes in macOS, both in the host and the guest, as its use requires the combination of Sequoia developer beta 3 or later running on both. It’s also significant that it can’t be used on VMs that have been upgraded from earlier versions of macOS, although thankfully it doesn’t require that of the host. This appears to be because Apple ID support requires structural change in the VM that can’t be achieved by a macOS updated inside it, and can only be performed when creating a new VM from scratch.

The best that a VM has been able to offer before Sequoia is relative privacy, but little more protection than already available on the host’s internal SSD. That assumes you store your VMs on the internal Data volume, which isn’t good practice in terms of snapshots and backups, as those will be significantly larger as a result. Storing VMs externally benefits from encrypted APFS, but that’s not as robust as full-strength FileVault.

If you want to set up a private VM using lightweight virtualisation on Apple silicon:

  • Upgrade the host to macOS Sequoia.
  • Build a new VM using macOS Sequoia.
  • Create the VM on the host’s Data volume on its internal SSD, with FileVault enabled.
  • Add that item to Time Machine’s list of exclusions from backups, only backing it up when necessary to an encrypted APFS volume. Bear in mind, though, that the VM will still appear in local snapshots.
  • During VM configuration, sign in with your Apple ID and enable FileVault for the VM.
  • For the VM’s primary admin account, use a different name and a different and robust password.
  • In the VM, disable all unnecessary iCloud and iCloud Drive access, and don’t enable network file sharing.

Remember that access to shared folders is only available from inside the VM. It’s currently not possible for the host or processes running on the host to access the contents of the VM unless enabled by network file sharing or through iCloud.

Sequoia VMs can cause kernel panics

By: hoakley
17 July 2024 at 23:01

If you are beta-testing macOS 15 Sequoia in a lightweight virtual machine on an Apple silicon Mac, beware that it can cause the host to suffer a kernel panic. Although I haven’t tested this with other virtualisers, my understanding is that the danger applies to them all, and not just my own Viable and Vimy.

Sequoia developer beta 2, both versions of developer beta 3, and I expect the first public beta, are already hungry for memory. In Sonoma and earlier VMs, if you give the guest 16 GB of memory, it’s likely to use considerably less than that. Those betas of Sequoia will probably use a little more than is allocated to them. But that will double if you restart the VM, and if your host Mac has insufficient memory for twice that VM’s original allocation, it’s likely to suffer a kernel panic with the VM still open.

Previously, in Sonoma and earlier, restarting the VM results in the VM Service for the virtualiser (as shown in Activity Monitor) yielding almost all its memory when restarting, and that rising during booting of the VM. In Sequoia, memory is fully retained during the restart, and then rises until it reaches twice that allocated to the VM, or your Mac panics if it runs out of physical memory first.

I’m very grateful to Joe for reporting this. He has sent a Feedback report to Apple, and I hope this is fixed in the next beta release. In the meantime, don’t restart any Sequoia VM unless your Mac has more than twice the free memory allocated to the VM.

I’d be grateful if those using other virtualisers could confirm whether those are also affected. I suspect they are, as the virtualiser doesn’t normally handle restarting of the VM, but leaves that to the host and guest macOS.

Last Week on My Mac: Did Apple forget its own App Store?

By: hoakley
14 July 2024 at 15:00

I was sorely tempted to pre-order an Apple Vision Pro, but it wasn’t the cost that was the decider. When I checked, I realised that Apple has locked in its most exciting new technology to running only what’s provided through its App Store. Not that I don’t buy through Apple’s App Stores, but if there’s one thing that stultifies innovation, it’s a bureaucracy that obsesses with its rules.

It took Steve Jobs a while to accept that iPhones needed third-party apps, and Apple launched its iTunes App Store for iOS in 2008, just over a year after the first iPhone had been released. Early in 2011, the Mac followed suit, but as an addition to well-established direct distribution. At first it provided a convenient central platform for Apple’s own products.

appstore1

Although promoted for its curation, security and trustworthiness, over the last 13 years each has been profoundly undermined. You don’t have to spend long looking in the App Store app to appreciate that it’s as well curated as a painting exhibition requiring all frames to be gilded and more than two inches wide, let alone the prevalence of scam apps on iOS App Stores.

appstore2

Its track record of security nearly came to grief in 2015, when hundreds of apps on the China store were discovered to have been victims of a supply-chain attack by XcodeGhost. Just a couple of months later the macOS App Store suffered major problems with its security certificates, causing most of its apps to be unusable and erroneously reported as damaged.

appstore173

Nevertheless it has continued to attract important apps from major developers, as shown below in 2015, when it was far more navigable.

Apple's Mac App Store now offers many quite expensive - and highly reputable - apps.

Despite being mired in controversy since they were unleashed, Apple’s App Stores have prospered, both for Apple and for the precious few developers who achieve success on them. The one growth area that they have so far missed out on has been virtual machines running on Apple silicon Macs, which have been unable to access the macOS App Store, or to run the great majority of apps purchased from it.

Shortly after Apple released lightweight virtualisation for Apple silicon Macs in 2022, those who had started to experiment with them discovered what appeared to be a major blind spot in their design: as they didn’t support signing in with an Apple ID, they could neither access iCloud services, nor run third-party apps supplied through the App Store. Obvious though this shortcoming was to users, it apparently hadn’t occurred to Apple, who hadn’t even started to build in support for Apple ID.

This was completed in time to be included among the new features announced for macOS Sequoia last month, when Apple promised that it “supports access to iCloud accounts and resources when running macOS in a virtual machine (VM) on Apple silicon”. With issues of virtualising what was needed from the host’s Secure Enclave apparently solved, some of us had come to expect that would include App Store access, which is also controlled by Apple ID. It’s now clear that Apple didn’t intend to include its App Store as a “related application”, which was implicitly excluded.

However little you might love the App Store, support in macOS VMs is essential if they are to be of any general use. VMs that can’t run all App Store apps as part of the benefits of signing in with an Apple ID are so stunted as to be of little use. Would it be that difficult to implement, now that those VMs can be signed in to all the other services that depend on an Apple ID? Did Apple really forget its own App Store when deciding what apps should be allowed to run in a VM?

If you consider this to be a showstopper for virtualising macOS on Apple silicon Macs, then please make it clear to Apple through Feedback.

Sequoia, virtualisation and Apple ID

By: hoakley
12 July 2024 at 14:30

The third developer beta of macOS 15 Sequoia finally brings support for Apple ID in macOS virtual machines (VM). As this is likely to form the first public beta-release next week, here’s a short guide to how to install a Sequoia VM, and what you can do with it. I’m delighted to report that my own free virtualisation apps Viable and Vimy already support Sequoia VMs on Sonoma 14.5 and Sequoia hosts, and I expect that will be true of other virtualisers for Apple silicon Macs.

Installing Sequoia as a VM

When running Sequoia developer beta 3, or the first public beta, download an IPSW image from Apple’s beta support site, or via Mr Macintosh’s compilation. Ensure that you download developer beta 3 or public beta 1 or later, depending on which programme you’ve joined. Then install that IPSW using Viable in the normal way, as detailed here.

If you’re virtualising Sequoia on a Sonoma 14.5 host, you may need to install additional software before installing the Sequoia IPSW using Viable. One way to discover that is to proceed normally using the IPSW you’ve just downloaded. You’ll then be prompted to install a software update.

sequoiavm1

At present, this will fail, but I expect that Apple will provide that additional software for the public beta.

sequoiavm2

If it doesn’t, and you’re unsuccessful in installing the additional software, trash that VM (but not the IPSW inside it), install and run the latest beta-release of Xcode 16 from Apple’s beta support site. Once that has been run, you should be able to install Sequoia without any problems.

First run

Open the VM using Viable, and work through its configuration as normal.

If the VM is hosted on Sequoia developer beta 3 or later, you should be able to enter your Apple ID and password, and opt for FileVault on its Data volume during that initial configuration. If it’s hosted on any older version of macOS, then you shouldn’t try entering your Apple ID and password, as that will fail. This is because the minimum requirements for Apple ID support in a VM are:

  • the host running Sequoia developer beta 3 or later, and
  • the VM running Sequoia developer beta 3 or later.

If your Mac and VM meet those, the VM should then trigger normal 2FA confirmation over iCloud, and then activate iCloud, iCloud Drive, and support for related applications such as passwords, calendar and file sharing via iCloud.

App Store support

In Apple’s release notes for Sequoia developer beta 3, it states that the following issue has been resolved: “Users will not be able to sign-in to iCloud and related applications”. Apple has previously stated that Sequoia “supports access to iCloud accounts and resources when running macOS in a virtual machine (VM) on Apple silicon”. However, that currently doesn’t include access to the App Store or use of apps purchased from it.

At present, “access to iCloud accounts and resources” does include:

  • iCloud Drive
  • Keychain in iCloud, fully supported in Passwords.app, including passkeys
  • syncing shared iCloud databases such as calendars and address book
  • shared Photos using iCloud
  • third-party apps sharing data using CloudKit.

It doesn’t include connecting to the App Store, and as a result apps obtained from the App Store that check the current user is entitled to run them will fail to open. There appears to be no workaround for this, although some apps including several of Apple’s will run because they don’t appear to perform those user checks. In those cases, copying the app from the host enables you to run the app in that VM, but that doesn’t apply to the great majority of paid-for App Store apps.

If you’re disappointed that Apple still hasn’t opened access to its own App Store in VMs, please request this feature using Feedback.

Nesting virtualisation

Apple has also announced that Sequoia will support nesting on models with M3 chips, where you can run a macOS VM inside a macOS VM. Although Viable is no longer blocked from running in a VM, this feature doesn’t appear to work yet, at least not using Viable on an M3 Pro.

Downloads

Viable version 1.0.12 (beta 12) and Vimy 0.7 (beta 4) are available from their Product Page, and appear fully compatible with Sequoia, although they don’t yet support the suspend/resume feature for closing VMs. I believe that ViableS 1.0.12 is also compatible.

❌
❌