System logs seem to have been introduced with Mac OS X in 2000-2001, and I don’t recall any equivalent in Classic Mac OS, although individual apps such as databases often kept their own logs.
2000-2016 text logs
As Mac OS X presented itself as a derivative of Unix, it brought with it bells and whistles such as support for code to write to system-level logs including system.log, console.log and dozens of other more specialist destinations, and its own log browser in the Console app.
As is traditional, log entries contained unstructured plain text to which a datestamp and other data were added to expand each into a line of text in log files that were rotated daily. As entries were relatively infrequent, many users learned to read the log and to use it to diagnose problems.
The Console app gave ready access to all standard logs as well as app-specific ones, such as this for mail processes such as sendmail, and crash reports. These two screenshots are from Mac OS X 10.0 Cheetah in April 2001.
By Mac OS X 10.4 Tiger in 2005, Console had acquired some basic tools and a sidebar to select from the many logs. Because they were plain text, those for previous days were compressed and stored in archives until they were removed during routine housekeeping. This excerpt shows entries in the system log over a restart that took over 2 minutes from the last entry to the start of the boot process.
There has also been the rare substitute for Console: this is LogMaster from Bright Light Software, shareware for $14.50 in 2006 until it was abandoned.
Although much in Console remained the same until 2016, at some stage Apple structured log entries into fields, as shown here in Mac OS X 10.6 Snow Leopard. Log entries were still infrequent, with this excerpt covering a period of almost 20 seconds.
This is another restart, here in OS X 10.10 Yosemite in April 2015. This time, the period recorded for that restart has fallen to 39 seconds. System shutdown is marked by the shutdown process and SHUTDOWN_TIME, and startup begins with BOOT_TIME.
2016 Unified log
With macOS Sierra in 2016, that was all swept away and replaced by the Unified log. There had been warning signs that change was coming: in May of that year, I complained that the log consisted of a torrent of messages like 17/05/2016 21:04:40.175 storeassetd[531]: multibyte ASN1 identifiers are not supported.
or 17/05/2016 20:55:15.298 WindowServer[233]: _CGXRemoveWindowFromWindowMovementGroup: window 0x91 is not attached to window 0x92
Even when running a fairly clean installation of El Capitan, All Messages clocked up around 4000 entries every 8 or 9 hours. At its worst, the log could fill those 4000 message slots in a minute or two. Little did we realise how busy our logs were about to become.
Apple declared the goals of its new log system at WWDC in June 2016:
a single efficient logging mechanism for user and kernel mode;
to maximise information collection with minimum observer effect;
the compression of log data;
a managed log message lifecycle;
as much logging on as much of the time as possible;
for privacy to be designed into the logging system;
a common system across macOS, iOS, watchOS, tvOS;
all legacy APIs (NSLog, asl_log_message, syslog, etc.) to be redirected into the new unified log;
to emphasise debugging of macOS and apps, not providing any facilities for system administration or audit;
to link to the sysdiagnose tool for gathering information for bug reports etc.
To achieve this, a log entry is made using a new call that’s handled by the logd daemon and compressed into a buffer. From there it’s either retained in memory if ephemeral, or written out to a file.
There are two main groups of files that store log entries: those kept in /var/db/diagnostics/Persist/ in the form of tracev3 files containing regular log entries, and further tracev3 files in /var/db/diagnostics/Special/ containing additional shorter-life entries. Additional and lengthier log data can be stored in files named by UUID in /var/db/uuidtext/, and there’s also scope for high-volume collection.
tracev3 files use a proprietary compressed binary format that remains undocumented to this day, but has been partially reversed. Apple doesn’t provide direct access to their contents, only through closed-source utilities such as the log command tool. Where users want a more portable format, Apple recommends conversion to a logarchive package, although that’s also undocumented and only directly accessible using log and Console. Apple has in recent years given limited access to the active log for third-party apps, but that lacks many of the useful features of its own log command.
Privacy
Privacy features have caused problems from the start. Log messages containing potentially sensitive information have that censored by <private>. Like so many good ideas, this had unintended consequences as many log entries only contain the dreaded <private>, and in some cases meaningful content is lost altogether.
Ironically, the most embarrassing security problem in the Unified log occurred in early versions of High Sierra, when encryption passwords were leaked apparently as a result of incorrect string formatting. Apple subsequently added an entry field to make explicit the formatting used for that entry.
Until the release of Catalina, there was an undocumented switch to turn privacy protection off, through an option to the log config command. When you needed to view all those censored messages, you could turn protection off, perform the test, and the log then contained all the information you required. That changed in Catalina 10.15. In order to bypass this privacy protection, you had to run your Mac in a special diagnostic mode intended for use exclusively by Apple engineers. Apple later relented and allowed this to be controlled through a profile, although because entries already made don’t contain the censored data, it can’t be applied retrospectively.
When first released, access to the new Unified log wasn’t restricted to admin users, but that changed in macOS 10.12.4, when that restriction was applied, and normal users found they were no longer able to browse the log at all.
Problems
The biggest disappointment, though, has been the Console log browser, which has made only limited use of log entry structure, displays just a small selection of entry fields, provides little aid for the high volume of entries, and worst of all gives no access at all to recent entries in the live log. Apple’s decision to restrict Console to browse the live stream of entries and logarchives has rendered it useless for many of the most compelling reasons for the app, but it has ensured that Console and the log provide no “facilities for system administration or audit”. It has also deterred third-party developers from writing to the log, and made it the exclusive preserve of Apple’s engineers, which perhaps was the original intention.
Since its first release in macOS Sierra, the log has flourished if not grown like an invasive weed. The number of fields available has increased from 16 to over 25, many of them added to support Signposts, introduced in late High Sierra and Mojave. Those are used extensively in macOS primarily to measure performance.
As the log now rolls its tracev3 files to maintain a maximum total file size, rising rates of entries by macOS have limited the period covered by retained entries. What in the early days was sufficient for up to 20 days of entries may now last little longer than a few hours. This also ensures that Console has more limited usefulness, and it struggles to cope with logarchives of any size.
Collection and retention of entries from different subsystems is set in logging profiles, XML property lists stored in /System/Library/Preferences/Logging (in the System volume, so read only) and /Library/Preferences/Logging, which the user controls. You can create your own custom profiles, or modify them on the fly using the log command, although this appears unusual even among the few left who can and do still browse the log.
What used to be a primary tool in diagnosing problems has been abducted without replacement. At least it keeps those pesky system administrators and auditors away.
I hope that you enjoyed Saturday’s Mac Riddles, episode 286. Here are my solutions to them.
1: Keeps an eye on exercise for checking 2, 3, memory and more.
Click for a solution
Activity Monitor
Keeps an eye on (monitor) exercise (activity) for checking 2, 3, memory and more (what Activity Monitor does).
2: Champion’s reward in disarray for the heart of your Mac, shown in 1.
Click for a solution
CPU
Champion’s reward (a cup) in disarray (rearranged to cpu) for the heart of your Mac (its CPU), shown in 1 (it is).
3: Endless high status in hiding place for treasure can be shown in 1.
Click for a solution
Cache
Endless high status (cachet without the last letter) in hiding place for treasure (a cache) can be shown in 1 (when Content Caching service is enabled, it’s one of the tabs or panes in Activity Monitor).
In the days of Classic Macs, most of our concerns centred on memory rather than CPU, disk or network performance. Tools for managing memory flourished, and I don’t recall any utility provided in Mac OS that looked more broadly. That changed when Mac OS X arrived, and we started taking an interest in Process Viewer and CPU Monitor.
CPU Monitor, seen here in 2001 with its two floating chart views, was a first step in development. I suspect this was taken on my QuickSilver Power Mac G4 with its dual processors, hence the pair of CPU charts.
Process Viewer is a more obvious ancestor of the modern Activity Monitor, with its list of processes in a column view. Note the surprisingly few system processes running at the time: only 34, compared with today’s list of several hundred.
Activity Monitor integrated those features into its new app in Mac OS X 10.3 Panther in 2003, and is shown here in 10.4 Tiger from a couple of years later, where it’s surprisingly similar to the current app. There are still only 72 processes running, though, and most have less than 5 threads.
Here’s Activity Monitor with its CPU History window on an 8-core Mac Pro, running Mac OS X 10.7 Lion in 2011. The number of processes is still only growing slowly, and had reached 89.
Xcode’s Instruments added an Activity Monitor template for developers wanting to monitor further details as they tested and debugged their code. At this stage it was limited to presenting the same fundamental measurements as were available in Activity Monitor, but these have since flourished into much greater depth.
Two years later, in 2013 for OS X 10.9 Mavericks, Activity Monitor underwent revision to add a new tab for Energy. Unlike other panes, the ‘Energy Impact’ is given in arbitrary units rather than calculating Joules from power use over time. It was at about this time that the app also added a Cache pane analysing the performance of the Content Caching service, when enabled on the Mac.
Until 2020, Activity Monitor had dealt with CPUs with uniform cores. Above are the eight physical and eight Hyper-threaded cores in an 8-core Intel Xeon W in an iMac Pro from 2017, running a heavy load of over 700% CPU. With the first Apple silicon Macs, it had to display CPU use for two different types of core. Note how, by 2020, the total number of processes has shot up to 458.
This is an example from one of the first base M1 Macs, with 4 Efficiency and 4 Performance cores displayed neatly in its CPU History window. Although Activity Monitor doesn’t take core frequency into account when measuring % CPU, and can’t display cluster frequencies, it remains one of the essential tools for everyone, whichever age and architecture their Mac.
I hope that you enjoyed Saturday’s Mac Riddles, episode 285. Here are my solutions to them.
1: This Scot with a volume was a notebook closed in 2019.
Click for a solution
MacBook
This Scot (Mac) with a volume (book) was a notebook closed in 2019 (it was a notebook Mac discontinued in 2019).
2: Ledger for baptismal water manages typefaces.
Click for a solution
Font Book
Ledger (book) for baptismal water (a font) manages typefaces (what this app does).
3: Root -1 with bound paper as a companion to the iMac with a handle.
Click for a solution
iBook
Root -1 (in maths, i) with bound paper (a book) as a companion to the iMac (it was) with a handle (it’s one of the few computers that features a handle).
While memory and its management have been important in the history of all computers, they were nearly the downfall of the first Macintosh, the original 128K named because it came with just 128 KB of RAM. That proved barely sufficient for demonstration purposes, and by the autumn/fall of 1984 had been replaced by the ‘Fat Mac’ with 512 KB costing an extra $700. Apple proved quick to demonstrate that memory for its new Mac products was never going to come cheap, when it came at all.
It took another couple of years for the Mac Plus, the first that came with memory slots to increase its standard memory from 1 to 4 MB, and from then pretty well every Mac had slots to accommodate expansion. By the IIx of 1988, those slots could accommodate a maximum of 128 MB, a thousand times that in the 128K Mac.
In early 1989, Connectix introduced Virtual, the first implementation of virtual memory for System 6. Two years later, in May 1991, Apple provided its own implementation in System 7, but its use remained optional even in System 8.6 eight years later. Some apps required it, while others couldn’t run when it was enabled. Most users stuck with only enabling it when their software needed it, and made do with the limitations of physical memory of 384 MB or less.
The maximum my Blue & White Power Mac G3/350 could accommodate was just 1 GB. As apps were far more conservative in their memory requirements, this worked better than you might expect.
My Power Mac G3 worked well with Mac OS taking its lion’s share of just over 50 MB, my mail client with less than 7 MB, and the whole of Microsoft Word in under 20 MB. But apps could and did run out of memory, when they would simply quit with an error alert, something we grew familiar with.
Memory leaks still plagued Mac OS 8, and many users had to resort to utilities like R Fronabarger’s freeware Memory Mapper to track free memory and try to understand what was going on.
One memory problem never fixed in Mac OS 8.x occurred in many apps including Web browsers, Microsoft Office 98, and others. Using these led to a progressive reduction in the amount of contiguous free memory, until eventually the whole Mac crashed. This appeared worst in Macs with most physical memory, and although some patches were produced by third-parties, none was a complete solution. The only workaround was to keep an eye on memory, and restart the Mac before a crash occurred.
In those days, you had to set the amount of memory to be allocated to each app in the Finder’s Get Info dialog. Getting this right was usually a matter of trial and error.
Although Classic Mac OS had such a struggle managing memory, the first Mac to support proper virtual memory had been the Macintosh II in 1987. That required it to be fitted with Motorola’s 68851 paged MMU chip, an option needed to run Apple’s A/UX port of Unix. That chip was no longer needed in Motorola 68030 and 68040 processors, as its functions were then integrated into the CPU.
Mac OS X was completely different, with virtual memory a permanent feature, and greatly improved management by the kernel. But memory leaks continued, and we learned the pain brought by those in Mach zone memory, memory blocks allocated for use by the kernel and its extensions. That happened as recently as macOS Catalina 10.15.6, when they caused kernel panics. Memory leaks, fortunately not affecting Mach zones this time, also troubled macOS 12.0.1.
Physical memory continued to grow in size, by the last of the Power Macs reaching 8 or even 16 GB in high-end models. Intel models offered even more, and by early 2009 8-core Mac Pro models could accommodate up to 128 GB, although Apple officially claimed a mere 32 GB. The original MacBook Air of 2008 was the first to ship with fixed memory, 2 GB that couldn’t be upgraded, and that became more general among models released from 2015 onwards.
With the advent of Apple silicon Macs came the greatest change in memory management and use since the release of Mac OS X twenty years earlier: instead of having separate physical memory for devices like GPUs, M-series chips use Unified Memory, one pool for use by CPU cores, GPU, and much else apart from the Secure Enclave. Unfortunately, that has also brought RAM to be integrated into the M-series chip carrier, even in those fitted to the Mac Pro.
Macs have thus returned to one of the problems of the original 128K of forty years ago, and once again their memory can’t be upgraded.
Macs have used four different architectures for their Central Processing Units over the last 40 years. From their launch by Steve Jobs on 24 January 1984, for the first decade they used Motorola 68K CPUs, then switched to PowerPCs designed by an alliance of Apple, IBM and Motorola, which were used for 12 years. After 14 years being built around Intel processors from 2006, Macs most recently changed a third time to use Apple’s own Arm-based chips.
Over those 40 years, continuous improvements in capabilities and performance of CPUs have transformed Mac OS and the apps it supports.
Motorola 68K
CPUs execute instructions in synchrony with a clock whose frequency determines the rate of instruction execution. The Motorola 68000 processor in the original Mac 128K ambled along at a clock speed of just 8 MHz. The last 68K models featuring 68040 CPUs had raised that to 33 MHz, and added specialist Memory Management Units (MMUs) and floating point units. The latter first appeared as 68881 and 68882 maths co-processors, but were later integrated into the 68040.
MMUs were particularly important for the implementation of virtual memory. When the Macintosh II was introduced in 1987, it was the first Mac that could be fitted with Motorola’s optional 68851 paged MMU, required for it to run Apple’s A/UX port of Unix with virtual memory support. Strangely, Apple’s own MMU fitted in the standard Mac II didn’t support virtual memory. Its 68020 CPU was also the first in Macs to use 32 bits rather than the 16 of the original 68000.
AIM PowerPC
When introduced in 1994, the first Power Macs came with PowerPC 601 or 601+ CPUs running at frequencies up to 110 MHz, nearly 14 times faster than the original Mac 128K. Just over a decade later, the last Power Mac G5 raised that to dual two-core CPUs at 2,500 MHz, more than 20 times the clock frequency, and the previous model had offered dual 2,700 MHz CPUs.
PowerPCs had their origins in IBM’s high-end POWER architecture based on a reduced instruction set (RISC) intended to be run at higher frequencies. Initially, these CPUs used a 32-bit design, but progressed to 64 bits. Not only do they have integrated floating point units that were extended for Apple, but later models include AltiVec vector processing for single-precision floating point and integer operations.
High CPU frequencies bring higher power consumption and heat output. A dual-core G5 with a PowerPC 970MP CPU used a maximum of 100 W at 2,000 MHz, and some of the last G5 Macs used liquid cooling to cope with the heat generated at higher frequencies. Those didn’t prove long-lived, with coolant leaks a common and fatal failing.
Intel x86
In early 2006 Apple started releasing its new range of Macs using Intel CPUs. With the exception of a base model of Mac mini, those came with 2-core Core Duo processors running at up to 2 GHz, and were soon followed by the first Mac Pros featuring two 64-bit 2-core Xeon 5100 CPUs (Woodcrest) at up to 3 GHz. By the following year, the first 8-core Mac Pro was available.
Earlier increases in CPU frequency gradually petered out. The last Intel Mac Pro was available with cores running at 2.5-3.5 GHz, boosted to a maximum of up to 4.4 GHz. Instead, high-end models offered as many as 28 cores and drew power up to 900 W. More typical of late desktop Macs were Intel Core i9 CPUs with 6-8 cores at similar frequencies. Adding more processor cores has been an effective way to run more code at the same time. Tasks are divided into threads that can run relatively independently of one another. Those threads can then be distributed across several CPU cores.
Rising power consumption and heat output were becoming even more of a problem in MacBook Pro models.
Apple Arm
Well before Apple had joined IBM and Motorola in the AIM alliance, it had co-founded the company based in Cambridge, England, that was to become ARM (for Acorn RISC Machines). Its RISC processor was used in Apple’s Newton MessagePad of 1993, and in 2010 Apple released its first iPhone and iPad designed around a single-core 32-bit Arm CPU running at a cool and economical 1 GHz.
From before macOS Mojave in 2018, Apple was preparing for its next migration, to its own integrated Systems on a Chip (SoC), starting with the M1 in 2020. The first iPhone to incorporate two CPU core types was the iPhone 7 of 2016, in its A10 Fusion SoC. Rather than simply adding more cores, Apple had adopted the Arm big.LITTLE architecture, in which background threads are run on slower, more efficient CPU cores, and higher priority user processes run on faster, more performant cores.
The first two families of M1 and M2 chips have cores grouped in clusters of no more than 4, but Apple increased cluster size to 6 in the M3 and M4. While the M1 family consists of two designs, one for the base variant, and the second for both Pro and Max, and doubled in the Ultra, M3 and M4 families have distinct designs for their Pro and Max variants. For the M4, this offers a full range from 8 to 16 cores in total, with an anticipated Ultra extending to 32. In addition to CPU cores with built-in vector processing (NEON), these chips incorporate specialist co-processors such as a neural engine and a proprietary matrix co-processor, AMX.
Performance (‘big’) cores have increased in maximum frequency, from 3.2 GHz in the M1 to 4.5 GHz in the M4 four years later.
Trends
The period 1984-2007 was dominated by increasing CPU frequency, as demonstrated in the two charts below.
This chart uses a conventional linear Y axis to demonstrate that frequency rose rapidly during the decade from 1997. As the form of this curve is S-shaped, the chart below shows the same data with a logarithmic Y axis.
Since about 2007, Macs haven’t seen substantial frequency increases. Many factors limit the maximum frequency that a processor can run at, including its physical dimensions, but among the most significant in practical terms are its power requirements and heat output, hence its need for cooling. Thus, the period 2005-2017 became dominated by increasing core count.
This chart shows how the number of processors and cores inside Macs didn’t start rising until around 2005, just as frequencies were topping out. Thus, many of the CPU performance improvements from 2007 onwards have been the result of providing more cores. But there’s a practical limit as to how many of those cores will get used, which is where processing more data becomes important, as it has from 1998 onwards.
It’s remarkable how much of Mac OS has survived if not flourished over those 40 years that our Macs have gone from a pedestrian Motorola 68000 processor to the 12 performance cores capable of 4.5 GHz in an M4 Max chip.
I hope that you enjoyed Saturday’s Mac Riddles, episode 282. Here are my solutions to them.
1: Scrambled coach or underwater vessel came with the first iMac.
Click for a solution
USB
Scrambled coach (‘bus’ rearranged) or underwater vessel (‘sub’ rearranged) came with the first iMac (it was first introduced with the original iMac in 1998).
2: Burning telegraph was best with DV cameras but couldn’t compete with 1.
Click for a solution
FireWire
Burning (fire) telegraph (wire) was best with DV cameras (it was standard with most of them) but couldn’t compete with 1 (Macs abandoned it in favour of USB and Thunderbolt).
3: Could sound seedy or sexy, so it was terminated by 2000.
Click for a solution
SCSI
Could sound seedy (it’s pronounced the same as the word ‘scuzzy’) or sexy (it had originally been intended to be pronounced like that), so it was terminated (one of the bugbears with SCSI is using its special bus terminators) by 2000 (Apple discontinued support in 1999).
The common factor
Click for a solution
In their day, they were each high-speed external interfaces for Macs.
Like most computers, Apple’s Macs have had five main types of port to allow them to be connected to external devices: low speed intended commonly for keyboards and other input devices, high speed often for connection to external storage, display, network, and audio. This brief history concentrates on the first of those.
The original 128K Macintosh launched in 1984 came with a good range of ports, including two RS-422 serial ports and a DB-19 supporting external floppy disk drives. It had separate mouse and keyboard ports, DE-8 and RJ-11 respectively. In 1986, Apple’s IIGS brought the first implementation of a new type of port, Apple Desktop Bus or ADB, that was introduced to the Mac SE and II the following year, replacing the original mouse and keyboard ports.
ADB
At the time, ADB was unique to Apple’s computers, although it was later adopted by NeXT. This uses a 4-pin mini-DIN connector to hook up a daisy-chain of peripherals. The theoretical maximum speed of ADB is 125 Kb/s, although around 62 Kb/s was closer to that experienced in practice. Among these ADB devices came dongles, used to enforce software copy-protection. Devices on an ADB chain each have an address, defaulting to $2 for keyboards, and $3 for mice.
This is ADB Parser, analysing its traffic. Note both Keyboard and Ext. Keyboard initially shared the original address of 02, and the Mouse has 03.
Perhaps the greatest shortcoming of ADB was that it wasn’t intended to be hot-swappable, and the Mac was supposed to be shut down before any changes were made to its connected ADB devices. Although many of us ignored that, it was at our peril. Occasionally, hot-swapping ADB devices fried a fuse that was soldered into the motherboard; although there was a workaround, for most that required a trip to the local authorised Apple dealer and a replacement motherboard, at considerable expense.
ADB and serial ports underwent the transition from Motorola 68K processors to the PowerPC in 1994, and it was another four years before their replacement came, in USB 1.
USB
The first iMac, released in 1998, was also the first Mac without an ADB port, and came with two USB ports for its mouse, keyboard and other peripherals. The first version of USB had only been introduced in 1996, and brought with it new USB-A connectors, although those were initially complicated by a different USB-B format on printers.
The move to USB opened up many new possibilities for external devices including storage, and hubs quickly became needed to support the profusion. Performance was also greatly improved over ADB, leaping from 62 Kb/s to 12 Mb/s at what was appropriately known as Full Speed. That improved again when USB 2.0 (released in 2000) appeared in the 15-inch iMac, iBook and PowerBook G4, and Power Mac G5 of 2003. These reached 480 Mb/s in what was justifiably called High Speed.
Apple System Profiler here shows five USB devices, each with its own driver, connected to a hub. These include a third-party mouse, and a USB-to-serial adaptor.
Third-party mice and other input devices thrived with USB, although driver support was variable. This is a Microsoft IntelliPoint mouse that was popular.
USB was far more complex than ADB, as seen here in ADB Prober. This is a selection of the data in a basic powered USB hub. One of the lasting mysteries was why some peripherals worked more reliably when connected to powered hubs, while others didn’t work at all unless connected direct to a port on the Mac.
USB continued to develop, and with it a profusion of different plug and socket formats. Several different Mini and Micro versions came to fill our drawers and boxes with their cables and adaptors. By the time of USB 3.0 and its 5 Gb/s SuperSpeed and USB-C format, we were all hoping for a reprieve and the return of simplicity. Those hopes were dashed with USB 3.1 in 2013, and its Gen 1 and Gen 2 with their confusion of different terms.
Bluetooth
The last transition for input devices was to do away with cables (almost) altogether, and in 2017 Apple dropped its last models of wired mice and keyboards. At first their wireless replacements required USB-A ports for their charging cables, and have most recently switched to USB-C. Bluetooth had its origins in 1999-2001 in headsets and mobile phones, and by the time our input devices had gone wireless they were achieving data rates upwards of 1 Mb/s over a short range. Few of us now rely on ports and cables to support our keyboards, mice and trackpads.
I hope that you enjoyed Saturday’s Mac Riddles, episode 281. Here are my solutions to them.
1: Second major version in 20 years came from the south near Carmel.
Click for a solution
Big Sur
Second major version in 20 years (it was the first increment to the Mac OS version number since Mac OS X 10.0) came from the south (Spanish ‘sur’) near Carmel (Big Sur is a section of rugged coast near Carmel, California).
2: Last stand for 32 is the desert with the valley of death.
Click for a solution
Mojave
Last stand for 32 (it’s the last version of macOS to support 32-bit code) is the desert (it’s one of the major deserts in North America) with the valley of death (it’s where Death Valley is located).
3: Fastest and first of ten sounds like a fraudster.
Click for a solution
Cheetah
Fastest (it’s the fastest land animal) and first of ten (first version of Mac OS X) sounds like a fraudster (a cheater).
The common factor
Click for a solution
They were each prominent versions of Mac OS X and later.
For the first 15 years of Classic Mac OS, right up to Mac OS 9 in 1999, Macs remained fundamentally single-user, and privacy wasn’t an issue of much concern. In those halcyon years of desktop publishing and HyperCard, users were more excited by opening information up than keeping it private, and the internet was in its infancy. It was Mac OS 9 that first integrated multiple user accounts and started to secure information using keychains.
Mac OS X brought the first full multi-user operating system to the Mac, but as internet connections became increasingly common and lasting, little attention was paid to privacy. By 2011, the Privacy tab in Security & Privacy, then in System Preferences, contained just three items: Location Services, Contacts, and Diagnostics & Usage. While privacy features developed elsewhere, for the sake of simplicity I’ll here focus on that pane in System Preferences, and its successor in System Settings.
Four years later, in OS X 10.10 Yosemite (2015) and still in 10.12 Sierra (2017), those three items had grown to eight, with the addition of Calendars, Reminders, Accessibility, and two social media platforms, Twitter and Facebook.
Then at WWDC in 2018, Apple revealed its new privacy architecture, putting it at the forefront in macOS 10.14 Mojave, and eventually reversing the order to Privacy & Security.
Mojave protected information in the following 15 categories:
Location Services,
Contacts (address books),
Calendars,
Reminders,
Photos (Photos libraries),
Mail,
Messages,
Safari browsing history,
HTTP cookies,
Call history (iOS),
Time Machine backups,
iTunes backups,
camera input,
audio input through the built-in microphone,
automation (AppleScript and others).
Its new protection system was dubbed TCC, for Transparency Consent and Control, and has since become prominent in the nightmares of developers, those who support Macs and many who use them. At its worst, it crashes apps that don’t comply with its rules, as shown in the diagram below for macOS 10.14.
Various classes of protected data are shown at the left, those in red being covered explicitly in Privacy controls. The first step was to determine whether the app trying to access protected data was signed by Apple: if it was, access was determined by private controls, and sometimes regular controls as well.
Apps developed by third parties were checked to see whether they already had access to that particular class of protected data according to Privacy settings. If they had, access was then granted without further dialogs. Note that the effect of adding an app to the Full Disk Access list was to give it access to all protected data, but not services or hardware, without any further consent being sought.
If they hadn’t already been given access, the next check was to see which version of the SDK they were built against. If they were built against 10.13 or earlier, then Mojave didn’t expect them to have support such as usage information, so it should have displayed a dialog inviting consent to the requested access. That would normally only contain the standard text information.
If consent was given, then that app was added to the appropriate class in Privacy settings; if it was declined, then it was denied access, but wasn’t put on any blacklist, so consent could still be given on another occasion.
If the app was built against the 10.14 SDK, then stricter rules were applied. It was then required to have a usage statement for the class of data it was trying to access, where that was in the class-specific list at the top, or a protected device or service. If the app didn’t provide the appropriate usage statement, TCC considered the request to access protected data was unintended, and crashed the app as an ‘unexpected quit’.
If the app did contain a usage statement appropriate for that class of protected data, then Mojave displayed the consent dialog, this time containing the text from that usage statement as well. If consent was then given, the app would be added to the list in Privacy.
Since Mojave, TCC has been a fertile source of vulnerabilities for third-party researchers to discover, and the malicious to exploit. Three were reported shortly after the initial release of 10.14. Two, discovered by Patrick Wardle and Jeff Johnson, weren’t disclosed, to allow Apple to address them, and the third, in ssh, wasn’t so much a vulnerability as a feature that could be exploited.
Each successive major version of macOS has added further to that list from Mojave. Catalina (10.15, 2019) added new locations that required user intent or consent to access, including:
~/Desktop, widely used for active documents
~/Documents, main document storage
~/Downloads, the default location for downloaded files
iCloud Drive, now widely used for shared working documents
third-party cloud storage, if used
removable volumes
network volumes.
This is one of the late Security & Privacy panes from macOS Catalina in 2019.
By the time that macOS 13 Ventura was released in 2022, its shiny new Privacy & Security section in System Settings listed 20 categories. Some, like Full Disk Access and Files and Folders, overlapped, while others like Accessibility appeared to have been misnamed. Controls provided varied between different categories, and many users dreaded having to tinker with them.
At this rate of growth, Privacy will soon have its own app alongside System Settings.
I hope that you enjoyed Saturday’s Mac Riddles, episode 280. Here are my solutions to them.
1: Third prime should run at twice three or four, and four times two.
Click for a solution
(Thunderbolt) 5
Third prime (5) should run at twice three or four (Thunderbolt 5 should deliver 80 Gb/s speed, twice that of TB3 or TB4), and four times two (and four times that of TB2).
2: The third of XV brought AI for some.
Click for a solution
(macOS) 15.1
The third (version of macOS 15, which is shipping in last week’s new M4 Macs) of XV (macOS 15) brought AI for some (it did).
3: If E > P and E + P = GPU what does E equal?
Click for a solution
6
If E > P (6 > 4) and E + P = GPU (Macs with the full base M4 chip have 10-core GPUs) what does E equal? (6, the number of E cores in the full base M4 chip.)
The common factor
Click for a solution
They are properties of the new M4 Macs announced last week.
One of the novel features in the original Finder in Classic Mac OS was the use of distinctive icons for different types of document in an extensible scheme.
Every file had its type and creator codes, each consisting of four single-byte characters. The Desktop databases contained indexes to those, to enable the Finder to display the appropriate icon for a text document of type TEXT created by an app with the creator code of ttxt, SimpleText, for instance.
Apps provided a custom icon in their Resource fork for each type of document they supported. Periodically, those Desktop databases became broken, and documents lost their custom icons. The solution was to rebuild those Desktop databases from the data in each app’s Resources, a procedure that every Mac user became only too familiar with.
At some stage, perhaps in System 6 of 1988, or System 7 of 1991, document icons such as images could be displayed as miniatures or thumbnails instead. This was accomplished by apps creating that file’s thumbnail and saving it as an ICN# resource in the file’s Resource fork. Amazingly, this still works in Sequoia, where I pasted a prepared Resource fork into a Zip file to give it an inappropriate thumbnail.
The raw Resource fork is shown below in xattred as a com.apple.ResourceFork extended attribute.
Initially, Mac OS X continued a similar system, including custom thumbnails, until Apple introduced Quick Look in Mac OS X 10.5 Leopard, in 2007. This came with built-in support for a wide range of common document types, extending to QuickTime media including audio and video. One curious omission at first was that animated GIFs weren’t supported as animations until OS X 10.7.
Display of Thumbnails used the QuickLook framework documented here. This enabled third-parties to extend coverage to their own document types using QuickLook generators with the extension .qlgenerator. Initially, they were installed into /Library/QuickLook from each app bundle.
Normally, when QuickLook generated a Thumbnail or Preview, that was stored in its cache database kept in NSTemporaryDirectory in the path C/com.apple.QuickLook.thumbnailcache/. Those could give revealing insights into images and other documents accessed recently, and Wojciech Regula and Patrick Wardle discovered that, in High Sierra and earlier, it was easy for malicious software to examine that cache. Apple addressed that in macOS 10.14 Mojave by making the cache completely inaccessible.
In-memory caching of Thumbnails has also proved controversial in more recent versions of macOS. To deliver smooth scrolling of Thumbnails in the Finder’s Gallery views in particular, the Finder has taken to caching them in memory for up to two days, sometimes using several GB in the process. That can readily be mistaken for a memory leak, until those cached Thumbnails are finally flushed.
I described how QuickLook Thumbnails worked in early 2019, in the days before the SSV.
When you select a document in the Finder, a dialog, or somewhere else where you expect its icon to be shown, the Finder passes details of the document path and its type (UTI) to IconServices, to fetch the appropriate icon. This calls on its main service, iconservicesd in /System/Library/CoreServices, to check its icon cache.
Although the main icon store is locked away in /Library/Caches/com.apple.iconservices.store, there’s additional data in a folder on a path based on /private/var/folders/…/C/com.apple.iconservices, where … is an unreadable alphanumeric name. For icons used in the Dock, their cache is at /private/var/folders/…/C/com.apple.dock.iconcache. If the icon should be replaced by a QuickLook Thumbnail, such as in a Finder column view, QuickLook is asked to provide that thumbnail. That in turn may be cached in its protected cache at /private/var/folders/…/C/com.apple.QuickLook.thumbnailcache.
QuickLook then relies on there being an appropriate qlgenerator to create a thumbnail of that document type; if the qlgenerator is flawed or can’t cope with the document’s contents, that could easily fall over. For example, if you renamed a text file with a .jpeg extension so that macOS considered it was a JPEG image, the bundled qlgenerator might have simply resulted in the display of a busy spinner, rather than resolving to a generic JPEG document icon. IconServices should then deliver the appropriate icon back to the Finder to display it.
In macOS 10.15 Catalina (2019), Apple started replacing this system with a new framework named QuickLook Thumbnailing, documented here. That replaces qlgenerators with QuickLook preview extensions, in particular Thumbnail Extensions, as explained to developers at WWDC in 2019.
macOS 15.0 Sequoia has finally removed support for qlgenerators. That has resulted in the unfortunate loss of custom Thumbnails and Previews for document types of third-party apps that are still reliant on qlgenerators, and haven’t yet got round to providing equivalent app extensions. It’s almost as if the Desktop databases need to be rebuilt again.
I hope that you enjoyed Saturday’s Mac Riddles, episode 279. Here are my solutions to them.
1: The first year it goes from London to Leeds with the First Eleven on its arm.
Click for a solution
2020
The first year (2020, the year Apple silicon Macs were released) it goes from London to Leeds (the M1 motorway in England) with the First Eleven (launched with macOS 11.0 Big Sur installed) on its arm (they use Arm CPUs).
2: When intel brought the fifth big cat in a solo or duo.
Click for a solution
2006
When intel (Intel) brought the fifth big cat (they came with OS X Tiger 10.4.4) in a solo or duo (they had Intel Core Solo or Duo processors in 2006).
3: When 6100-8100 came from the aim of seven.
Click for a solution
1994
When 6100-8100 (first PowerPC models were Power Mac 6100, 7100 and 8100 of 1994) came from the aim (the processors were developed by the AIM Alliance of Apple, IBM and Motorola) of seven (they shipped with System 7.1.2).
The common factor
Click for a solution
They are the years in which Apple released the first Macs in each of its new architectures.
Firmware, software that’s intimately involved with hardware at a low level, has changed radically with each of the different processor architectures used in Macs.
Classic Macs based on Motorola 68K processors come with their own Macintosh ROM. That changed after the first PowerPC models of 1994, and the Power Macintosh 9500 from 1995 supports Apple’s version of Open Firmware. That had originated as OpenBoot in Sun Microsystems’ SPARC-based computers, and is based on the language Forth. Macs with Open Firmware can be booted into an interactive interface that makes it relatively straightforward to support and bring up new hardware. It’s also a security nightmare.
Firmware version numbering was elaborate, with a ROM revision, here $77D.45F6, a Boot ROM version of $0004.25f1, and a Mac OS ROM file version of 8.4, for this Power Mac G4 running Mac OS 9.2.1 in 2002. Apple supplied separate Mac OS ROM file updates as needed.
EFI
In 1998, Intel started work on the original Extensible Firmware Interface (EFI) as its intended replacement for the BIOS in PCs. By the time Apple was beginning its transition from PowerPCs in 2006, EFI was changing into Unified EFI (UEFI), and has since progressed as far as version 2.10 in 2022.
Once an Intel Mac has cleared its initial self-test routines (POST), and key custom chips like the SMC are running, EFI firmware is loaded next. The purpose of the EFI phase and the boot loader boot.efi is to augment the basic facilities provided by BootROM to the point where the macOS kernel can be loaded with its extensions. Key to this is providing access to the Mac’s hardware through the device tree, IODeviceTree, listing and relating all the devices in that Mac. This is built by boot.efi and passed to the kernel when it loads, and forms the basis for IOKit within macOS.
Model-specific boot.efi software also provides ongoing and additional support for boot services, including memory management, basic functions for timers and events, and for hardware access. It supports basic console protocols for input and output, and access to storage systems. Runtime services extend these to give access to variables stored in the NVRAM, and to GUIDs/UUIDs used for key variables in the EFI phase and later. Most importantly, boot.efi looks for startup key commands, originally named snag keys by Apple, such as Command-R to run in Recovery mode, Command-S and -V for Single User and Verbose modes, and Shift for Safe mode.
When Apple introduced Boot Camp in 2006, it made changes to boot.efi to support booting from operating systems other than macOS. This essentially provides a suite of drivers supporting Mac hardware in terms of a Windows hardware platform, engaged when the Mac is to be booted in that operating system rather than macOS.
Firmware security
In March 2015, two security researchers from LegbaCore, Xeno Kovah and Corey Kallenberg, demonstrated proof-of-concept attacks on the BIOS of several computers including Dell, HP, and other PCs that could have been used to implant malicious code. Later that year, Kovah and Trammell Hudson turned their attention to Macs, demonstrating a firmware worm named Thunderstrike 2.
For the first nine years of Intel Macs, Apple had provided EFI firmware updates separately from updates to OS X. That year, Apple changed the way that it supplied firmware, delivering it only as part of system upgrades and updates. Although older separate firmware updates are still available, those were the last.
Then in 2017, Rich Smith and Pepijn Bruienne of Duo Labs discovered that many Macs were running outdated firmware. Their concern was less about potential bugs and other problems, and more about the security risk posed. Apple had already been busy, hiring Xeno Kovah and Corey Kallenberg who started work there in November 2015, and Nikolaj Schlej, another firmware security researcher, who joined them the following August. They developed a new tool eficheck, released in High Sierra on 25 September 2017. Each week until it was dropped from Sonoma, eficheck checked current firmware against a local database of versions known to be ‘good’, and with the user’s permission sent a report to Apple in the event that it found discrepancies.
Back in late 2017, this iMac17,1 was reported as running Boot ROM version IM171.0105.B26.
T2 firmware
In 2016, the year before Smith and Bruienne’s report, Apple introduced first the T1 chip, then hot on its heels the T2 the following year. With two separate CPUs in each T2 Mac, there are two separate sets of firmware, one EFI and the other known as iBridge or BridgeOS. Following the established pattern, both are only updated by macOS installers and updaters.
After standard power-on self-test and SMC initialisation, the T2 sub-system establishes the level of Secure Boot in force, and, if that’s Full or Medium Security, boot.efi is checked before being loaded, providing security throughout the boot process.
Apple silicon Macs
The introduction of Macs using the M1 family of chips in 2020 brought complete change in firmware to support Secure Boot, and moves away from UEFI completely. The aim of boot security in Apple silicon Macs is to provide a verified chain of trust through each step in the boot process to the loading of macOS, that can’t be exploited by malicious components. This consists of four main stages:
The Boot ROM in the hardware.
The Low-Level Bootloader, LLB, or first stage.
iBoot, or second stage.
The macOS kernel, which loads all its required kernel extensions.
One of many changes made from UEFI is that startup key combinations have been replaced by the Power button to engage Recovery and other special startup modes, which has both improved security of Recovery mode and made its features more accessible. Instead of the user having to memorise a list of different key combinations required to access different features, all are now integrated within a single environment.
Apple silicon Macs are the first Macs whose firmware can be both upgraded and downgraded by restoring them from IPSW image files when the Mac has been put into DFU mode. For the time being, at least, all Apple silicon Macs run a unified firmware version tied not to the chip or model, but to the macOS version, and only delivered in IPSW files and macOS updates.
I hope that you enjoyed Saturday’s Mac Riddles, episode 278. Here are my solutions to them.
1: Platform executive arranges props and lighting for window groups.
Click for a solution
Stage Manager
Platform (a stage) executive (a manager) arranges props and lighting (what a stage manager does) for window groups (it manages window groups in macOS).
2: Blank characters open in parks for multiple desktops.
Click for a solution
Spaces
Blank characters (spaces) open in parks (open spaces) for multiple desktops (what it provides in macOS).
3: Regulate operational flight to exposé them all.
Click for a solution
Mission Control
Regulate (control) operational flight (a mission) to exposé them all (it does for all apps what Exposé did for single apps, in displaying all open windows).
The common factor
Click for a solution
They are all tools for advanced window management.
I hope that you enjoyed Saturday’s Mac Riddles, episode 277. Here are my solutions to them.
1: Volatile anaesthetic to catch fish over coaxial cable.
Click for a solution
Ethernet
Volatile anaesthetic (ether) to catch fish (net) over coaxial cable (used for most Ethernet connections, although twisted pair and fibre-optic are also used).
2: Flight terminal for wireless in cards and base stations.
Click for a solution
AirPort
Flight terminal (an airport) for wireless (it’s wireless networking) in cards and base stations (first available in base stations and cards from 1999).
3: Neighbourhood chat between a twisted pair came with the LaserWriter.
Click for a solution
LocalTalk
Neighbourhood (local) chat (talk) between a twisted pair (it used twisted-pair cables) came with the LaserWriter (released with and supported by Apple’s LaserWriter printer in 1985).
The common factor
Click for a solution
They are all physical network systems that have been supported by Macs.
Login
