Classic Mac OS didn’t have an operating system kernel as such, so on paper at least it couldn’t suffer kernel panics. Instead, the system simply crashed or froze, and the only way out was to force the Mac to restart or shut down. In those days before the HFS+ file system gained journalling (only added to OS X 10.2.2 in 2002), that was usually followed by checking and repairing the startup disk, as the file system there was often damaged in the process.

If you were really unlucky, when your Mac tried to start up again it would display a flashing question mark indicating that the system was too damaged to be run. That’s when you reached for the startup disk or CD-ROM, or a special disk you had ready to recover your Mac and restore the system. Recovery disks were almost essential, as app crashes often brought the whole system down with them.

Mac OS X brought a big change to that, with its kernel architecture to ensure that crashing apps were less likely to cause the whole system to collapse, although early versions of the kernel and its extensions often proved less stable that we’d hoped. In the first couple of versions of Mac OS X, handling of panics was fairly basic, with lines of text displayed, and the Mac left unresponsive for the user to force it to restart.

With Mac OS X 10.2 Jaguar came the well-known panic display, in which a transparent grey curtain dropped down the display to reveal a multi-language message instructing the user to restart the Mac.

From 10.2 in 2002 through 10.7 in 2011, minor variations were made to that panic display, including additional translations of its text. Because of the practical difficulties of getting a real screenshot of this event, that shown above is a slight cheat created by overlaying the display image stored in the system on a normal screenshot, but it’s a faithful reproduction of what we saw then.

This all changed in OS X Mountain Lion, in 2012, when the panic display disappeared and the user was informed of the panic only after their Mac had restarted automatically. Apple took to referring to them not as kernel panics, but in the euphemism unexpected restarts, and then removed the multi-language message altogether.

Instead, what was intended to happen was, shortly after the user had logged back into the automatically restarted Mac, a panic alert would appear, containing the panic log. Users were encouraged to send that to Apple, and most assumed that an engineer there would review it, and perhaps even respond to it if appropriate. In reality, panic logs were automatically and anonymously processed to provide Apple with overall statistics, and it seems extremely unlikely that any human read any of them individually unless they accompanied a bug report.

Kernel panics tended to break out in small epidemics confined to specific models. My own period of purgatory lasted six months, between the release of OS X 10.11.4 on 21 March 2016 and the upgrade to Sierra on 20 September that autumn/fall. Throughout that period, many newer MacBook Pros and iMacs suffered frequent kernel panics, in my iMac’s case every 2-5 days. Those may well have resulted from bugs in their graphics drivers, but were never fixed in El Capitan, as far as I’m aware.

Despite their frequency in the past, Apple has been remarkably reluctant to admit to them in its documentation. That for users avoids all mention of the term, merely referring to them as unexpected restarts “because of a problem”. Even that for developers buries them away in an ancient Tech Note from 14 August 2008, and still contains details for PowerPC models.

Maybe we’ve just been imagining them all those years.

Reference

Wikipedia

The CrowdStrike catastrophe has drawn attention to differences between modern macOS and Windows, and how kernel extensions (kexts) are being replaced in macOS by System Extensions. This article summarises what is changing, and how much progress macOS has made as of Sonoma 14.5.

Kernel space

The XNU kernel at the heart of macOS consists of central systems derived from Mach, BSD and I/O Kit. This runs at the highest level of privilege, in kernel space, often referred to as Ring 0 on Intel systems, although that term isn’t used for Arm CPUs as they have Exception Levels with EL0 as the least-privileged or user space. The kernel provides support for a great many services to interface with hardware and higher-level functions such as network protocols and file systems, that are almost entirely delivered as kexts running in Intel Ring 1.

Currently, in macOS Sonoma there are well over 650 kexts included in the System volume, three third-party kexts including SoftRAID that are in /Library/Extensions, and three that are firmlinked from outside the Signed System Volume (SSV), so they can be updated without creating a whole new sealed system snapshot.

Third-party code that needs direct access to kernel space has in the past used Kernel Programming Interfaces (KPIs) only accessible from kexts. These have included:

• I/O Kit drivers, including USB
• PCI and Thunderbolt
• Serial port drivers
• Audio drivers
• Network drivers, for network filters
• Storage drivers
• File systems.

Long and sometimes bitter experience of third-party kexts has demonstrated how their bugs and incompatibilities have resulted in kernel panics, where the kernel has no choice but to force a restart. Those risk losing data, and (in older file systems including HFS+) in leaving the file system in an unstable state, requiring repair. While recent models of Mac will restart rather than just shut down following a panic, the user needs to log back in and clean up before the Mac can be used again. When this happens at scale, as occurred with CrowdStrike, it has major impact.

User space extensions

Instead of delivering access to these features in kernel space using KPIs, macOS is transitioning to access in user space, where third-party code is delivered in System Extensions, running in Intel Ring 3 (Arm EL0). This transition involves Apple providing new and modified system kexts with their replacement interfaces for the System Extension to plug into.

One particularly relevant example is support for Endpoint Security, requiring monitoring of system events to discover potentially malicious activity. In the past this was handled by third-party kexts relying on KPIs, but should now be performed by a client System Extension that registers with Endpoint Security to receive notifications of different types of event, such as processes executing and file systems being mounted. These are documented at length, and are constrained to those events that Apple chooses to expose. If a security software developer wants access to other events, then they have to ask Apple to add them to Endpoint Security.

Deprecation and substitutes

Although Apple has been making discouraging noises about kexts for years, until it provided fully functional alternatives, developers had nowhere else to go and had to stay with kexts and KPIs. Many of those were officially deprecated in Catalina, and in Big Sur kexts using those no longer load by default. Among those affected are:

• Various KPIs now available in Endpoint Security
• Networking KPIs now available in Network Extensions
• IOHID input devices
• USB drivers now available in USBDriverKit and USBSerialDriverKit
• PCI drivers now available in PCIDriverKit and NetworkingDriverKit.

Network Extensions include changing Wi-Fi configuration, a Hotspot Helper to integrate with hotspots, VPN using built-in and custom protocols, network relaying, DNS configuration, and content filters.

Some replacements have been slower to arrive, and the following are among those made obligatory in Monterey 12.3 and later:

• Audio using AudioDriverKit
• Bluetooth using CoreBluetooth
• SCSI drivers now available in SCSIControllerDriverKit.

One of the last significant features to be transitioned to System Extensions is file system support, coming to user space in Sequoia as FSKit. Although it does now have a little documentation, that reveals that this initial release only supports simple file systems with one volume in each container; support for more sophisticated file systems is coming, but I don’t think Apple has announced when.

Penalties

Intel Macs have no restrictions or security limitations when using kexts, but Secure Boot in Apple silicon Macs won’t load any third-party kexts when run at Full Security. This makes the use of kexts on M-series Macs awkward to say the least. Apple has detailed the process of downgrading security and permitting loading of a kext, involving:

1. Start up in Recovery mode and select Options.
2. Open Startup Security Utility.
3. Downgrade boot security to Reduced Security.
4. Enable the loading of third-party kexts.
5. Restart in normal mode, and proceed to install the app with its kext installation steps, involving authorising the new kext in Privacy & Security settings.
6. Once the kext has been fully installed, and built into the Auxiliary Kernel Collection, restart.

Problems

Just as there are no clear-cut classes of app, kexts don’t all fall into neat groups. Although now widely implemented in some areas, such as Endpoint Security and Network Extensions, in others System Extensions still don’t fully replace KPIs. One good example is SoftRAID, a widely used driver supporting software RAID, which has had to be incorporated into macOS distributions as it still can’t be implemented in user space.

Kits to replace KPIs are also immature. Although I have been evaluating and reviewing products such as Little Snitch and various security suites using Network Extensions and Endpoint Security without encountering any problems, developers complain that support in macOS remains unstable and has vulnerabilities. When there’s a bug in that support, it may result in a kernel panic, causing the problem that this is intended to address.

macOS hasn’t completely replaced kexts yet, but is well on the way to achieving that. The benefits to stability and security are already being realised: I like to have a steady supply of panic logs to use in articles here, but I think the last I experienced was over three years ago, back in Big Sur, the result of a bug in iBoot firmware that has long since been fixed. Maybe I’ll have to start deliberately panicking VMs next. I’ll leave you with what used to be a common sight, lest we forget.

If you are beta-testing macOS 15 Sequoia in a lightweight virtual machine on an Apple silicon Mac, beware that it can cause the host to suffer a kernel panic. Although I haven’t tested this with other virtualisers, my understanding is that the danger applies to them all, and not just my own Viable and Vimy.

Sequoia developer beta 2, both versions of developer beta 3, and I expect the first public beta, are already hungry for memory. In Sonoma and earlier VMs, if you give the guest 16 GB of memory, it’s likely to use considerably less than that. Those betas of Sequoia will probably use a little more than is allocated to them. But that will double if you restart the VM, and if your host Mac has insufficient memory for twice that VM’s original allocation, it’s likely to suffer a kernel panic with the VM still open.

Previously, in Sonoma and earlier, restarting the VM results in the VM Service for the virtualiser (as shown in Activity Monitor) yielding almost all its memory when restarting, and that rising during booting of the VM. In Sequoia, memory is fully retained during the restart, and then rises until it reaches twice that allocated to the VM, or your Mac panics if it runs out of physical memory first.

I’m very grateful to Joe for reporting this. He has sent a Feedback report to Apple, and I hope this is fixed in the next beta release. In the meantime, don’t restart any Sequoia VM unless your Mac has more than twice the free memory allocated to the VM.

I’d be grateful if those using other virtualisers could confirm whether those are also affected. I suspect they are, as the virtualiser doesn’t normally handle restarting of the VM, but leaves that to the host and guest macOS.

One of the few things Apple got badly wrong in the original Macintosh forty years ago was memory. It came with just 128 KB of RAM, barely sufficient for demonstration purposes, and by the autumn/fall of 1984 had been replaced by the ‘Fat Mac’ with 512 KB costing an extra $700. Apple proved quick to demonstrate that memory for its new Mac products was never going to come cheap.

Until the release of System 7 in May 1991, Macs couldn’t use virtual memory when running Mac OS, and memory management was primitive to say the least. Even in System 8.6 eight years later, virtual memory remained optional. Some apps required it, while others couldn’t run when it was enabled. Most users stuck with only enabling it when their software needed it, and made do with the limitations of physical memory of 384 MB or even less. The maximum my Blue & White Power Mac G3/350 could accommodate was just 1 GB. As apps were far more conservative in their memory requirements, this worked better than you might expect.

My Power Mac G3 worked well with Mac OS taking its lion’s share of just over 50 MB, my mail client with less than 7 MB, and the whole of Microsoft Word in under 20 MB. But apps could and did run out of memory, when they would simply quit with an error alert.

Memory leaks still plagued Mac OS 8, and many users had to resort to utilities like R Fronabarger’s freeware Memory Mapper to track free memory and try to understand what was going on.

One memory problem never fixed in Mac OS 8.x occurred in many apps including Web browsers, Microsoft Office 98, and others. Using these led to a progressive reduction in the amount of contiguous free memory, until eventually the whole Mac crashed. This appeared worst in Macs with most physical memory, and although some patches were produced by third-parties, none was a complete solution. The only workaround was to keep an eye on memory, and restart the Mac before crashing became likely.

In those days, you had to set the amount of memory to be allocated to each app in the Finder’s Get Info dialog. Getting this right was usually a matter of trial and error.

Mac OS X was completely different, with virtual memory a permanent feature, and greatly improved management by the kernel. But memory leaks continued, and we learned the pain brought by those in Mach zone memory, memory blocks allocated for use by the kernel and its extensions. That happened as recently as macOS Catalina 10.15.6, resulting in kernel panics. Those were preceded by the kernel complaining of zone_map_exhaustion in the log:
03:21:09.447981+0100 kernel zone_map_exhaustion: Zone map size 12240662528, capacity 12884901888 [jetsam limit 95%]
03:21:09.448533+0100 kernel zone_map_exhaustion: Largest zone kalloc.48, size 6544393440
03:21:09.449437+0100 kernel kernel zone_map_exhaustion: Nothing to do for the largest zone [kalloc.48]. Waking up memorystatus thread.
Memory leaks, fortunately not affecting Mach zones this time, also troubled macOS 12.0.1.

With the advent of Apple silicon Macs came the greatest change in memory management and use since the release of Mac OS X twenty years earlier: instead of having separate physical memory for devices like GPUs, M-series chips use Unified Memory, one pool for use by CPU cores, GPU, and much else apart from the Secure Enclave.

Although it’s now over 23 years since Mac OS X was first fully released, take a look in the App Store at the products claiming to ‘clean up’ or somehow manage memory, as if we were still in the days of Classic Mac OS. If anyone can explain to me how those apps aren’t making fraudulent claims, I’d be very grateful, as I stopped trying to manage my Mac’s memory when I switched from Mac OS 9 to Mac OS X, and haven’t looked back.