近日发现了一个非常刁钻的可能引起基于 Emscripten 编译的 WASM 程序内存泄漏的问题。Emscripten 工具链提供了 Fetch 功能模块，这个模块允许我们调用浏览器的 fetch 接口来进行网络访问。

一个使用 fetch 接口的简单例子是：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

#include <stdio.h>
#include <string.h>
#include <emscripten/fetch.h>

void downloadSucceeded(emscripten_fetch_t *fetch) {
  printf("Finished downloading %llu bytes from URL %s.\n", fetch->numBytes, fetch->url);
  // The data is now available at fetch->data[0] through fetch->data[fetch->numBytes-1];
  emscripten_fetch_close(fetch); // Free data associated with the fetch.
}

void downloadFailed(emscripten_fetch_t *fetch) {
  printf("Downloading %s failed, HTTP failure status code: %d.\n", fetch->url, fetch->status);
  emscripten_fetch_close(fetch); // Also free data on failure.
}

int main() {
  emscripten_fetch_attr_t attr;
  emscripten_fetch_attr_init(&attr);
  strcpy(attr.requestMethod, "GET");
  attr.attributes = EMSCRIPTEN_FETCH_LOAD_TO_MEMORY;
  attr.onsuccess = downloadSucceeded;
  attr.onerror = downloadFailed;
  emscripten_fetch(&attr, "myfile.dat");
}

Fetch API 提供了一些比较高阶的功能，一种一个比较重要的功能是，他可以将下载的内容缓存到 IndexDB 中，这个缓存机制能够突破浏览器自身的缓存大小的限制（一般超过 50MB 的文件浏览器的自动缓存机制会拒绝缓存）。但是这个缓存机制会导致内存泄漏。

1 泄漏产生的过程

在开头的例子中，我们需要再 onerror 和 onsuccess 回调中调用 emscripten_fetch_close 接口来关闭 fetch 指针代表的请求。在关闭过程中，fetch 使用的数据缓存区将会被回收。这个过程如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

EMSCRIPTEN_RESULT emscripten_fetch_close(emscripten_fetch_t* fetch) {
  if (!fetch) {
    return EMSCRIPTEN_RESULT_SUCCESS; // Closing null pointer is ok, same as with free().
  }

  // This function frees the fetch pointer so that it is invalid to access it anymore.
  // Use a few key fields as an integrity check that we are being passed a good pointer to a valid
  // fetch structure, which has not been yet closed. (double close is an error)
  if (fetch->id == 0 || fetch->readyState > STATE_MAX) {
    return EMSCRIPTEN_RESULT_INVALID_PARAM;
  }

  // This fetch is aborted. Call the error handler if the fetch was still in progress and was
  // canceled in flight.
  if (fetch->readyState != STATE_DONE && fetch->__attributes.onerror) {
    fetch->status = (unsigned short)-1;
    strcpy(fetch->statusText, "aborted with emscripten_fetch_close()");
    fetch->__attributes.onerror(fetch);
  }

  fetch_free(fetch);
  return EMSCRIPTEN_RESULT_SUCCESS;
}

可以看到，回收并非总会发生， emscripten_fetch_close 函数会对 fetch 的部分状态进行检查，如果检查失败，则会返回一个 EMSCRIPTEN_RESULT_INVALID_PARAM 的错误码，并且不会执行后续的清理过程(`fetch_free)。被检查的两属性中，fetch->id 是我们需要关注的对象。fetch->id 这个属性作为 fetch 的唯一标识符，是用来建立起 C++ 端的请求对象和 JS 端的请求对象的映射的。id 的值在 JS 端分配。查看源码中的 Fetch.js 文件，

1
2
3
4
5
6
7
8
9
10
11
12

function fetchXHR(fetch, onsuccess, onerror, onprogress, onreadystatechange) {
  // ...

  var id = Fetch.xhrs.allocate(xhr);
  #if FETCH_DEBUG
    dbg(`fetch: id=${id}`);
  #endif
    {{{ makeSetValue('fetch', C_STRUCTS.emscripten_fetch_t.id, 'id', 'u32') }}};

  // ...

}

这是唯一的一处 id 复制。这段代码位于 fetchXHR 函数中，这意味着只有发起了 XHR 请求时，id 才会被分配。那么，如果缓存存在呢？这时不会调用 fetchXHR 函数（而是调用 fetchLoadCachedData 函数）。这意味着回调函数中我们试图调用 emscripten_fetch_close 函数来关闭请求并回收资源时，这个回收过程无法进行，这导致了内存泄漏。

2 怎么解决这个问题

要解决这个问题我们只需要强行让 fetch->id == 0 的检查无法通过即可，我们可以在 emscripten_fetch_close 调用前，强行设置 fetch->id 为一个非零值。那么什么值合适呢？如果我们取值和已有的请求的 id 相同，那么 emscripten_fetch_close 可能将那个请求关闭。研究 id 分配的过程（即 Fetch.xhrs.allocate 的实现）

1
2
3
4
5
6

// libcore.js
allocate(handle) {
  var id = this.freelist.pop() || this.allocated.length;
  this.allocated[id] = handle;
  return id;
}

可以看到，id 是顺序分配的，且使用过的 id 会被回收使用（freelist）。因此我们可以设置一个较大的值，只要同一时间最大的并发请求数量不超过这个值，那就是安全的。我一般选择设置为 0xffff。 那么，正确的关闭请求的方式是：

1
2
3
4

if (fetch->id == 0) {
  fetch->id = 0xffff;
}
emscripten_fetch_close(fetch);

Until last November, lightweight virtualisation of macOS on Apple silicon Macs had behaved uniformly across M-series families. Although I have heard of one report of problems moving VMs between Macs, those were built with custom kernels. In ordinary experience, VMs running on M1, M2 and M3 chips seemed not to care about the host’s hardware, and most of the time just worked, and updated correctly. There was one unfortunate glitch with shared folders that were lost in macOS 14.2 and 14.2.1, but otherwise VMs largely worked as expected.

Then last November disaster struck those of us who had just started using our new M4 Macs: they couldn’t virtualise any version of macOS before Ventura 13.4. Running a macOS VM for any version before that on an M4 Mac resulted in a black screen, and the VM failed to boot. That was fixed swiftly in macOS 15.2, and we no longer had to keep an older Apple silicon Mac around to be able to run those older versions of macOS in VMs.

Like many who virtualise macOS on Apple silicon, I keep a library of VMs with different versions so I can readily run tests on my apps and other issues. This is one of the great advantages of virtualisation, provided that you don’t rely on being able to run most apps from the App Store. When Apple releases new versions of macOS, once I’ve updated my Mac hosts, I turn to updating VMs. I’m normally cautious when doing this, to avoid trashing the original version. I duplicate the most recent, open it and run Software Update. When I’m happy that has worked correctly, I trash the original and rename the updated VM with its new version number.

That worked fine with Ventura 13.7.4 updating to 13.7.5, and Sonoma going to 14.7.5, but Sequoia 15.3.2 failed with a kernel panic, as I’ve detailed. When several of you kindly pointed out that M1, M2 and M3 Macs had no such problem, I confirmed on my M3 Pro that this is confined to hosts with an M4 family chip.

I have since tried updating my 15.3.2 VM to 15.4.1 on the M4 Pro, a surprisingly large update of over 6 GB, and that continues to result in a kernel panic and failure. I have also tried updating from 15.1 to 15.4.1 with an extraordinarily large download of more than 15 GB, only to see a repeat of the same kernel panic, with an almost identical panic log.

The macOS 15.4 update was particularly large, and some Apple silicon Macs were unable to install it successfully, most commonly on external bootable disks. From your reports, the 15.4.1 update seems to have fixed those problems with real rather than virtualised macOS. However, it hasn’t done anything to solve problems with VMs.

If you have an existing VM running any version of Sequoia prior to 15.4, then you’re unlikely to be successful updating that to 15.4 or later using an M4 host.

In contrast, upgrading a VM currently running Sonoma 14.7.5 completed briskly and without error. To my great surprise, that only requires a download of 8.7 GB, a little over half the size of the update from 15.1 to 15.4.1, which seems to be the wrong way round. The snag with upgrading from a previous major version of macOS to 15.x is that VM will never be able to use one of the most attractive features of Sequoia, iCloud Drive. If you want support for that, you’ll have to build a fresh VM using a Sequoia IPSW image file.

So for the time being, M4 hosts have a barrier between 15.3.2 and 15.4 that they can’t cross with an update. If you want a VM running 15.4 or later, then you’ll have to build a new one, or update 15.4 or later.

I don’t know and probably wouldn’t understand what changed in the 15.4 update, but it has certainly upset a lot of apple carts and VMs. And if you’d like a little homework, can you please explain:

• Update 15.1 to 15.4.1, download 15 GB, failure.
• Upgrade 14.7.5 to 15.4.1, download 8.7 GB, success.

Lambing dates remain one of life’s great mysteries. Here in the UK, farmers in the north usually lamb earliest, often only just after Christmas when it’s usually bitter cold and snowy up there. Down here in the balmy south, lambs are born three months or more later, typically in April, when they’re often struggling to keep cool in the sunshine. Last week we saw the first of this year’s lambs, and Apple’s Spring OS fest, including Sequoia 15.4.

Size

That update was large, but that isn’t exactly unusual:

• 7 March 2024, Sonoma 14.4 was 3.6 GB (Apple silicon) with 64 vulnerabilities fixed, “the most substantial update of this cycle so far”;
• 27 March 2023, Ventura 13.3 was 4.5 GB with 49 vulnerabilities fixed, being “substantial, and brings many improvements and fixes”;
• 14 March 2022, Monterey 12.3 was 5.3 GB with 45 vulnerabilities fixed, being “very substantial, introducing major new features like Universal Control and Spatial Audio, changing several bundled apps, and fixing many bugs”;
• 26 April 2021, Big Sur 11.3 was 6.62 GB with over 50 vulnerabilities fixed, “the largest update to macOS since Mojave, and quite possibly the largest ever”.

(Figures and quotations from links here.)

Although the 15.4 update wasn’t quite as large as 11.3, at 6.2 GB for Apple silicon, it has comfortably surpassed it in the number of vulnerabilities fixed, 131 in all, and came close to the size of the 15.0 upgrade at 6.6 GB. What’s most disappointing is that, while the first release of Sequoia merited long and detailed accounts of much of what had changed, for 15.4 there’s precious little information beyond its lengthy security release notes.

A stroll through the version numbers of its bundled apps and /System/Library confirms the extent of changes. There was no point in my trying to compile an article listing them, as it might have been briefer to report what hasn’t changed. What’s more to the point is what’s new in 15.4, what are its Spring lambs?

Novelties

Among the new kernel extensions is the first version of AppleProcessorTrace, and there’s a brace to support hardware in Apple silicon chips including a T6020 and T8103 for PCIe, and a T6032. Those appear to be for M2 Pro, M1 and M3 Ultra chips, respectively. There are two new public frameworks, one named CLLogEntry that is presumably for Core Location log entries, the other tantalisingly named SecurityUI. Neither seems to align to anything in Apple’s developer documentation, so might be preparing the ground for what we’ll hear about in early June at WWDC, when the lambs have grown a bit.

I keep a track of the total number of bundles in several of the folders in /System/Library. Since the release of Sequoia 15.0, that containing Private Frameworks has grown from 4,255 to 4,398. Because of their layout, this total overestimates the real change in numbers, and that probably represents a true growth of around 70 Private Frameworks in Sequoia so far.

These Private Frameworks contain code features used privately by Apple’s apps, but not exposed to third-party developers. Although much is of little or no use or advantage, they also contain much that supports changing features in macOS. Using Private Frameworks is a sure way to madness, and something explicitly forbidden in the App Store, but, like the unaffordable car or boat we like to gloat at, there’s no harm in wondering what they will bring in the future.

The list of new Private Frameworks in Sequoia 15.4 is long, and includes: AUSettings, Bosporus, ComputationalGraph, CoreAudioOrchestration, CryptexKit, CryptexServer, DailyBriefing, DeepVideoProcessingCore, Dyld, ExclaveFDRDecode, FPFS, FindMyPairing, various GameServices, GenerativePlaygroundUI, MCCFoundation, MLIR_ML, MobileAssetExclaveServices, Morpheus, MorpheusExtensions, an OnDeviceStorage group, OpenAPIRuntimeInternal, OpenAPIURLSessionInternal, PIRGeoProtos, RapidResourceDelivery, SecureVoiceTriggerAssets, SecurityUICore, and VideoEffect.

While many of those names can inform speculation about what we’re about to see in macOS 16, three merit a little more decoding.

Cryptexes are secure disk images loaded during boot that currently deliver Safari and its supporting components, and the dynamic libraries for all those frameworks, public and private. Accessing them from user-level code isn’t something you’d expect to happen, so those two Private Frameworks, CryptexKit and CryptexServer, hint at further expansion in their use and support.

Bosporus

The Bosporus Strait in Turkey connects the Black Sea to the Sea of Marmara, thence through the Dardanelles to the eastern Mediterranean. It’s a busy thoroughfare formerly used heavily by ships carrying grain and other bulk cargoes from Ukraine and Russia.

View of Constantinople and the Bosphorus (1856) is one of many views that Ivan Aivazovsky made of this great city, which he visited on many occasions. The artist kept his studio in Crimea, on the opposite (northern) shore of the Black Sea.

Morpheus

Morpheus is the god of dreams, whose name is the source of the word morphine. Although usually distinct from Hypnos, god of sleep, he’s sometimes associated with Nyx, goddess of the night, most famously in reference to a passage from Virgil’s Aeneid, painted below by Evelyn De Morgan.

She pairs Nyx with Morpheus in her Night and Sleep, from 1878. The further figure is a young woman wearing long red robes, her eyes closed, clutching a large brown cloak with her right hand, and most likely Nyx. Her left arm is intertwined with a young man’s right arm. He also has his eyes closed, and is most probably Morpheus. He clutches a large bunch of poppies to his chest with his left arm, while his right scatters them, so they fall to the ground below.

Virgil’s lines in Book 4, line 486 read:
hinc mihi Massylae gentis monstrata sacerdos,
Hesperidum templi custos, epulasque draconi
quae dabat et sacros servabat in arbore ramos,
spargens umida mella soporiferumque papaver.
haec se carminibus promittit solvere mentes
quas velit, ast aliis duras immittere curas…

Translated (at Perseus at Tufts University), this reads:
From thence is come
a witch, a priestess, a Numidian crone,
who guards the shrine of the Hesperides
and feeds the dragon; she protects the fruit
of that enchanting tree, and scatters there
her slumb’rous poppies mixed with honey-dew.
Her spells and magic promise to set free
what hearts she will, or visit cruel woes
on men afar.

Spargens umida mella soporiferumque papaver, one of Virgil’s greatest lines, is conventionally translated as “scattering moist honey and sleep-inducing poppy”, and describes well the effects of the opiate drugs derived from opium poppies, including morphine.

I look forward to watching the lambs grow up through the coming summer, and learning about those lambs that came with Sequoia 15.4 at WWDC.

Over the last nine years, few of my articles here have been about XProtect, other than those announcing its updates. Until September 2024 and the release of macOS 15 Sequoia. This is now the tenth article I have written about the problems brought by XProtect updates in Sequoia over those six months, when there have been just 13 updates. The result of the last, on 4 March, was that for two days afterwards, many Macs running Sequoia were still using its data from 26 February rather than that in the new version 5289.

This not only affects XProtect, but the other front-line tool in macOS to detect and remove malicious software, XProtect Remediator (XPR). Earlier this year, I reported that at least 17 of the 24 scanning modules in XPR now use Yara definitions provided by XProtect’s data. All those Macs still running the superseded version of XProtect would also have had XPR scans run using that old version of the Yara rules.

XPR is a recent addition to these tools, introduced just three years ago, but XProtect goes way back before Yosemite in 2014. Although there have been occasional brief glitches in delivery of its updates, they have almost invariably completed quickly and reliably, leaving very few Macs stuck with an outdated version 24 hours after an update.

I have now come to dread XProtect updates because of the problems we encounter, and the latest update to 5289 was a good example. There’s a flurry of comments and emails from those whose Macs had failed to complete the update, previously a rare exception. For XProtect 5287 on 5 February, for example, there were 33, including my responses. For version 2184 exactly a year earlier there’s not one comment about that XProtect update.

Sole documentation provided about XProtect’s updates in Sequoia is the man file for its command tool, xprotect, which refers only to updates provided via iCloud, and doesn’t explain how those delivered via the traditional mechanism in softwareupdate might be involved. Yet we know there is a relation: the latest update has still not been supplied via iCloud, not even four days later, but relied instead on XProtectUpdateService working with an update obtained via softwareupdate. Previously that could be invoked using the xprotect update command, but that no longer works, leaving users with two versions of XProtect data, of which the copy used by XProtect and XPR is the older.

Late last year, when xprotect update appeared to be working as expected, I decided that my app SilentKnight would need to use that command in order to download and install updates. As that requires elevated privileges, I have been looking at how to implement a privileged helper app to perform that. With the latest update, that approach would have failed until the version in iCloud had been brought up to date. Instead we’re now reduced to restarting our Macs and hoping that, some time in the next day or two, they might update.

There’s a further problem emerging with the updates of 4 March. Many users have noticed subsequent XPR scans being terminated before completion. Although in most cases that fault appears to go away in later scans, in some Macs it prematurely terminates every set of XPR scans, leaving several of its scanning modules unused.

For example, this iMac Pro has failed to scan using ten of its 24 modules. This occurs because XPR apparently runs a timer, and when a round of scans is deemed to be taking too long, that timer fires and brings XPR to an abrupt halt. Indications are this is most likely when there are many Time Machine backups accessible; as those are all immutable snapshots and haven’t changed since they were made months ago, this is strange behaviour, and hadn’t occurred prior to the updates of 4 March.

Six months ago, if anyone had told me that macOS security protection in Sequoia was going to become less reliable, I wouldn’t have believed them. The truth is that, for many, it now has. As things stand in 15.3.1, a Mac is now more likely to be using an out of date version of XProtect’s detection rules, and for XPR scans to detect and remove malware. And there’s nothing you can do about that until Apple returns to using an update mechanism that’s both timely and reliable. Is that really too much to expect of this front-line security protection?

Selected previous articles:

What is happening with XProtect updates?
XProtect tormentor
How XProtect has changed in macOS Sequoia
A simple guide to how XProtect installs and updates in Sequoia
XProtect has changed again in macOS Sequoia 15.2
What happened with XProtect?
What has happened to XProtect in Sequoia?