“Some see in his critique of the Mao era parallels to today: the arbitrary rule of an aging leader, harsh treatment of dissent, and government programs that encourage people to inform on one another.” My profile of the octogenarian essayist Gao Ertai, who lived for years in the deserts of western China and now resides in Las Vegas.
With a working demo of what I hope will form the basis of a third-generation log browser, it was time to turn that into something useful. My next step with LogUI was to equip it with controls to browse excerpts of any section of the active log on your Mac. In particular, this means setting a time in the past, a period over which to collect and display log entries in its SwiftUI List view, and writing them out to a Rich Text file.
This uses a SwiftUI DatePicker to set the date and time of the start of the excerpt, which I’ll consider in detail on Sunday, adds an option to display just a selection of the fields, buttons to get the log extract and to save it to RTF, and moves all the other settings apart from the predicate into the top of the window. It’s already starting to look more like Ulbow.
Time and memory
As I realised in my initial explorations of the OSLog API in macOS, there are only three ways of specifying the starting point of any log extract: relative to the last boot, relative to the latest log entry (thus ‘now’), or at an arbitrary time point between the start and end of the log. There’s no way to specify an end point, and when working forward through the log, that defaults to the latest log entry, which could be days later. Thus, to collect entries between two arbitrary points, you can only specify the start, and stop collecting entries once their timestamp is more recent than the end point. As I pointed out in my initial assessment of using OSLog, you also need to keep a running total of entries, lest they overwhelm the app by the million.
Once I had developed LogUI to collect entries for set time periods after an arbitrary starting point in the past, I used that to estimate the amount of memory required for sample collections of all log entries over a period of one second, for which I also set a maximum of 2,000 entries. When collected in the very recent past, just a few seconds before the latest log entry, that amounted to an average of 30 MB, but from an hour earlier it rose alarmingly to 176 MB.
Using a predicate to restrict the entries collected was highly effective: in one test, using a predicate to include only entries from XProtect Remediator as an example, a 24 hour extract required only 40 MB. Although that might appear impressive in comparison with the results above, my current XProCheck, relying on log show rather than OSLog, uses just over 35 MB total memory to display four full days of log extracts.
With the existing OSLog API, it thus appears that only two use cases work well:
all log entries for periods of a few seconds in the very recent past, or
a narrow group of log entries, filtered using a suitable predicate, for periods up to many hours.
Even imposing a tight limit on the total number of log entries to be displayed, capturing all log entries just a few minutes into the past results in excessive memory use. In this respect, my original method using the log show command is far superior, as that imposes no memory overhead for extracts obtained from any time in the past, and doesn’t require narrow predicate filtering to control the memory it requires.
Memory not released
Memory limitations of using OSLog are compounded by the observation that the memory that it uses isn’t freed after use. Indeed, it isn’t even released when all the app’s windows are closed. This functions as a memory leak, and my more capable test app quickly consumed more than 1 GB of memory, after starting at only 16.9 MB before obtaining any log extracts.
To assess this further, I used the Memory Leaks feature in Xcode Instruments. Although that surprisingly failed to warn of any memory leak, the chart below shows a typical result.
For this test, I opened four windows and left them without log extracts. I set each to capture and display a different log extract of 1.0 seconds duration, or a maximum of 500 entries, from about an hour earlier. I then fetched the log extract for each window in quick succession. Each extract resulted in memory allocation by OSLog of more than 210 MB, rising in steps until the app was using just over 1 GB in total.
I then closed each window in turn, releasing just a few MB each time, to a final memory use of just under 1 GB without any open windows. Leaving the app in that state for an hour didn’t result in any further reduction in its memory use, which was only released when the app was quit.
I therefore believe that memory used by the logging system in OSLog isn’t currently being released after use, and will shortly be reporting this to Apple in Feedback.
Conclusions
OSLog needs an additional means of setting a time window for log collections.
Currently, OSLog only supports a narrow range of use cases.
The log show command remains far superior to OSLog in its capabilities and resource use.
The logging system fails to release memory correctly, resulting in a memory leak that severely limits its usefulness.
Nearly five years after its introduction, OSLog is still not suitable as a replacement for log show.
As a result, I have put this project on hold pending solutions to these problems. Ulbow already performs better than LogUI ever could.
If you are beta-testing macOS 15 Sequoia in a lightweight virtual machine on an Apple silicon Mac, beware that it can cause the host to suffer a kernel panic. Although I haven’t tested this with other virtualisers, my understanding is that the danger applies to them all, and not just my own Viable and Vimy.
Sequoia developer beta 2, both versions of developer beta 3, and I expect the first public beta, are already hungry for memory. In Sonoma and earlier VMs, if you give the guest 16 GB of memory, it’s likely to use considerably less than that. Those betas of Sequoia will probably use a little more than is allocated to them. But that will double if you restart the VM, and if your host Mac has insufficient memory for twice that VM’s original allocation, it’s likely to suffer a kernel panic with the VM still open.
Previously, in Sonoma and earlier, restarting the VM results in the VM Service for the virtualiser (as shown in Activity Monitor) yielding almost all its memory when restarting, and that rising during booting of the VM. In Sequoia, memory is fully retained during the restart, and then rises until it reaches twice that allocated to the VM, or your Mac panics if it runs out of physical memory first.
I’m very grateful to Joe for reporting this. He has sent a Feedback report to Apple, and I hope this is fixed in the next beta release. In the meantime, don’t restart any Sequoia VM unless your Mac has more than twice the free memory allocated to the VM.
I’d be grateful if those using other virtualisers could confirm whether those are also affected. I suspect they are, as the virtualiser doesn’t normally handle restarting of the VM, but leaves that to the host and guest macOS.
One of the few things Apple got badly wrong in the original Macintosh forty years ago was memory. It came with just 128 KB of RAM, barely sufficient for demonstration purposes, and by the autumn/fall of 1984 had been replaced by the ‘Fat Mac’ with 512 KB costing an extra $700. Apple proved quick to demonstrate that memory for its new Mac products was never going to come cheap.
Until the release of System 7 in May 1991, Macs couldn’t use virtual memory when running Mac OS, and memory management was primitive to say the least. Even in System 8.6 eight years later, virtual memory remained optional. Some apps required it, while others couldn’t run when it was enabled. Most users stuck with only enabling it when their software needed it, and made do with the limitations of physical memory of 384 MB or even less. The maximum my Blue & White Power Mac G3/350 could accommodate was just 1 GB. As apps were far more conservative in their memory requirements, this worked better than you might expect.
My Power Mac G3 worked well with Mac OS taking its lion’s share of just over 50 MB, my mail client with less than 7 MB, and the whole of Microsoft Word in under 20 MB. But apps could and did run out of memory, when they would simply quit with an error alert.
Memory leaks still plagued Mac OS 8, and many users had to resort to utilities like R Fronabarger’s freeware Memory Mapper to track free memory and try to understand what was going on.
One memory problem never fixed in Mac OS 8.x occurred in many apps including Web browsers, Microsoft Office 98, and others. Using these led to a progressive reduction in the amount of contiguous free memory, until eventually the whole Mac crashed. This appeared worst in Macs with most physical memory, and although some patches were produced by third-parties, none was a complete solution. The only workaround was to keep an eye on memory, and restart the Mac before crashing became likely.
In those days, you had to set the amount of memory to be allocated to each app in the Finder’s Get Info dialog. Getting this right was usually a matter of trial and error.
Mac OS X was completely different, with virtual memory a permanent feature, and greatly improved management by the kernel. But memory leaks continued, and we learned the pain brought by those in Mach zone memory, memory blocks allocated for use by the kernel and its extensions. That happened as recently as macOS Catalina 10.15.6, resulting in kernel panics. Those were preceded by the kernel complaining of zone_map_exhaustion in the log: 03:21:09.447981+0100 kernel zone_map_exhaustion: Zone map size 12240662528, capacity 12884901888 [jetsam limit 95%]
03:21:09.448533+0100 kernel zone_map_exhaustion: Largest zone kalloc.48, size 6544393440
03:21:09.449437+0100 kernel kernel zone_map_exhaustion: Nothing to do for the largest zone [kalloc.48]. Waking up memorystatus thread.
Memory leaks, fortunately not affecting Mach zones this time, also troubled macOS 12.0.1.
With the advent of Apple silicon Macs came the greatest change in memory management and use since the release of Mac OS X twenty years earlier: instead of having separate physical memory for devices like GPUs, M-series chips use Unified Memory, one pool for use by CPU cores, GPU, and much else apart from the Secure Enclave.
Although it’s now over 23 years since Mac OS X was first fully released, take a look in the App Store at the products claiming to ‘clean up’ or somehow manage memory, as if we were still in the days of Classic Mac OS. If anyone can explain to me how those apps aren’t making fraudulent claims, I’d be very grateful, as I stopped trying to manage my Mac’s memory when I switched from Mac OS 9 to Mac OS X, and haven’t looked back.
After months of work, on Dec. 13 we launched the China Unofficial Archives, a repository of hundreds of underground periodicals, books, and movies.
The site is a project that I began to think about when I was working on Sparks, my book on counter-history in China. One key point is that the digital revolution has made it easier than ever for people inside an authoritarian state like China’s to share information by basic technologies, such as PDFs or digital films. And yet much of the information sharing is ad hoc. A person may get an email with a PDF book attached on, say, the Cultural Revolution, but not realize what else has been published or filmed on this topic. Or they might like the author but might not have an easy way to access that person’s works. An online archive, I thought, could help fill this gap.
After incorporating the archive as a non-profit over the summer, I secured funding from a charity, hired a web designer and had invaluable help from people such as the independent journalist Jiang Xue (who also features in Sparks).
We “launched” via an online event sponsored by Westminister University’s China Centre. I was joined by Gerda Wielander, who has done much research on history and state narratives in China, and Shao Jiang, a London-based scholar who advises the archive. The event will eventually be posted to YouTube.
As I mention in the note, this is a work in progress. We’ve already received very useful feedback on how to improve it and are also in the middle of uploading hundreds of new movies and other files. We currently have about 850 items in the archive but need to double it–and fill in many holes (also outlined in my note)–before it will really take shape.
“With firm but never dogmatic moral conviction, Johnson pays tribute to the writers, the scholars, the poets, and the filmmakers who found the courage to challenge Communist Party propaganda. These dissenters looked beyond the official lies about the past and the present, and decided to document the truth about forbidden topics, including Mao Zedong’s campaigns to massacre putative class enemies. They often paid for their candor with long prison terms, torture, or death. Their conclusions—presented in homemade videos, mimeographed sheets, and underground journals—didn’t reach a wide audience when they appeared. And yet, as Johnson makes clear in his superb, stylishly written book, the value of their legacy is incalculable.”
The Economist
“A Pulitzer-prize-winning journalist describes the valiant e!orts of China’s “underground historians”, a motley and persistent group of academics, artists, film-makers and journalists attempting to correct the sanitised official record and provide truthful accounts of history. A rare insight into the extraordinary risks that some Chinese take to illuminate the darkest corners of communism.”
The FT
“‘Who controls the past, controls the future,’ wrote George Orwell. This is a fascinating and important story of dissident historians in China, who are challenging the Communist party’s authorised version of history.”
The New Statesman
“In a year of unrelentingly bleak news, I’ve chosen Sparks (Allen Lane), Ian Johnson’s evocative study of China’s underground historians documents both the relentless crackdown on civil society and intellectual freedom under Xi Jinping, and the quiet courage of those who refuse to be crushed. Drawing their inspiration from earlier acts of resistance that appeared hopeless in their own times, these independent scholars, filmmakers, and journalists have come to view their work as time capsules, determined to preserve an accurate record of the country’s past for future generations. ‘They want future Chinese to know,’ Johnson writes, ‘that in the 2020s, when things had never been darker…. Not everyone had given in.'”
The Tablet
“Controlling the interpretation of what has happened in China since the Communist revolution is an integral part of President Xi Jinping’s ever-tightening grip on his vast country. In Sparks: China’s Underground Historians and Their Battle for the Future (Allen Lane, £25; Tablet price £22.50), Ian Johnson charts the brave attempts of individuals, the ‘underground’ historians, to challenge the party line and reveal the brutal arbitrariness which has marked its rule from the beginning.”
I’ll start out on the launch date, Sept. 26, at McNally Jackson in New York City, followed the next day with a talk at the Harvard Bookstore in Cambridge, MA. Then back to NY for a Council on Foreign Relations talk, down to Washington DC for talks at Georgetown and Politics & Prose, followed by a trip to the U.K., and then the West Coast of the United States.
Next year: the Association for Asian Studies meeting in Seattle (where I’ve organized a panel on counter-history in China), Stanford, SMU in Dallas, and more… For some of the details, please visit my Sparks-tour page here.
A documentary filmmaker who spent years uncovering a Mao-era death camp; an independent journalist who gave voice to the millions who suffered through draconian Covid lockdowns; a samizdat magazine publisher who dodges the secret police: these are some of the people who make up Sparks: China’s Underground Historians and their Battle for the Future, a vital account of how some of China’s most important writers, filmmakers, and artists have overcome crackdowns and censorship to challenge the Chinese Communist Party on its most sacred ground–its monopoly on history.
Why history? The past is a battleground in many countries, but in China it is crucial to political power. In traditional China, dynasties rewrote history to justify their rule by proving that their predecessors were unworthy. Marxism gave this a modern gloss, describing history as an unstoppable force heading toward Communism’s triumph. The Communist Party builds on these ideas to whitewash its misdeeds and justify its continued hold on power. Indeed, one of Xi Jinping’s signature policies is the control of history, which he equates with the party’s very survival.
But in recent years, critical thinkers from across China have begun to challenge this state-led disremembering. Using digital technologies to bypass China’s ubiquitous surveillance state, their samizdat journals, underground films, and guerilla media posts document a persistent pattern of disasters: from famines and purges of years past to ethnic clashes and virus outbreaks of the present.
Based on ten years of on-the-ground investigations and interviews, Sparks challenges stereotypes of a China where the state has quashed all free thought, revealing instead a land engaged in of one of humanity’s great struggles of memory against forgetting–a battle that will shape the China that emerges in the mid-21st century.
Advance Praise
For more than three decades, Ian Johnson has conducted some of the most important grassroots research of any foreign journalist in China. With Sparks, he turns his attention to history—not the sanctioned, censored, and selective history promoted by the Communist Party, but the independent histories that are being written and filmed by brave individuals across the country. This book is a powerful reminder of how China’s future depends on who controls the past.
—Peter Hessler, MacArthur grantee, National Book Award Winning author of Rivertown, Oracle Bones, and Strange Stones.
An indelible feat of reporting and an urgent read, Ian Johnson’s Sparks is alive with the voices of the countless Chinese who fiercely, improbably, refuse to let their histories be forgotten. It’s a privilege to read books like these.
—Te-Ping Chen, author of Land of Big Numbers, and Wall Street Journal national correspondent.
China’s most famous modern writer Lu Xun predicted that “as long as there shall be stones, the seeds of fire will not die.” In Sparks Ian Johnson introduces us to a new generation of unofficial historians — modern-day “seeds of fire.” Their work will survive the Xi Jinping era, both to shed light on the past and to illuminate China’s better future.
— Geremie R. Barmé, editor, China Heritage.
Ian Johnson’s Sparks was a revelation: this historian from overseas spent years penetrating the world of underground Chinese historians, becoming in his own right a recorder of pioneers such as Hu Jie, Ai Xiaoming, and Jiang Xue, who use text and video to record China’s lost history.
—Liao Yiwu, author of The Corpse Walker, God is Red, For a Song and a Hundred Songs, and recipient of the Peace Prize of the German Book Trade.
Sparks tells the stories of underground historians who are determined to write down China’s hidden histories of famines, political campaigns, massacres, and virus outbreaks. These stories show why Xi Jinping wants to control history–because memories like these are sparks of light in a heavy darkness.
—Li Yuan, New York Times columnist and host of the Bumingbai podcast.
In the long years of Chinese people’s pursuit of justice and equality, preserving historical truth has always been a fierce but invisible battle. As Ian Johnson’s Sparks shows, today’s fighters for the truth are backed by vast armies—the seen and unseen, the living and the dead—who together are prying open the lies on which totalitarianism is built.
—Cui Weiping, Beijing Film Academy professor, translator of Vaclav Havel into Chinese.
Ian Johnson has presented a powerful narrative of how the human spirit has survived the cruel repression of Maoist totalitarianism and is still doing the same against Xi Jinping’s determined efforts to impose a new form of digital totalitarianism. In telling the individual stories of Chinese citizens who choose to defend freedom and dignity, Johnson has also provided a powerful illustration of how Xi’s repressive regime works. A must read for anyone interested in the Chinese and China.
—Steve Tsang, historian of Hong Kong, director of the China Institute at the School of Oriental and African Studies, London.
This is a necessary book charged with historical urgency. The sparks, left by the eponymous underground magazine suppressed in the 1950s, are preserved here and ready to burst into a firestorm.”
—Ha Jin, author of the National Book Award-winning novel Waiting
This compelling and highly enjoyable book will greatly enhance the general reader’s understanding of the subtle counter-currents of resistance at work in Chinese society below the smooth surface of control and compliance. In fifteen chapters and a conclusion, the author provides a comprehensive and detailed picture of what he calls “underground history” and its practitioners in mainland China—amateur or one might say guerilla historians who devote considerable efforts to reconstructing the past through independent inquiry, bypassing and challenging state-condoned narratives of the past.
—Sebastian Veg, author of Minjian: The Rise of China’s Grassroots Intellectuals, professor of history at the School for Advanced Studies in the Social Sciences, Paris.
Publication Rights and Editions
This book will come out in the United States and Canada via Oxford University Press, and via Penguin (Allen Lane) for other English-language rights areas, including the U.K., Australia, New Zealand and Asia.
Foreign rights have also been sold for Korea, Japan, and Taiwan.
Some notes on the covers
The US and Canada cover (at the top of this post) was made by Yang Kim, a Brooklyn-based book designer who works for Crown (Random House). She used a collage of images inside a torch, which she took from an image used in a 1960 student journal, Spark, which was the inspiration for my book title. Thanks Yang for such a brilliant job!
The designers for Allen Lane in London opted to use an Ai Weiwei papercut called “River Crabs.” Ai uses the traditional art form of paper cutting and combines it with topical issues, such as pollution, protests, and the state’s demolition of private property. River Crabs are a form of Internet slang for censorship and protests against it.
Photos
The book contains more than thirty photos. Some of them are historical, such as images of the students who founded the original journal. Many of these people ended up in labor camps for years and some were executed. Thanks to several Chinese historians, such as Song Yongyi and the documentary filmmaker Hu Jie for sending me these valuable historical prints. These images survived the maelstrom and thanks to digital technologies are now part of China’s collective memory–a key theme of this book.
Four students from Lanzhou University exiled to the city of Tianshui, standing in front of the party offices. They would publish the journal Spark, which has become a touchstone in the battle for China's history.
The first page of the 1960 journal Spark, founded by Lanzhou University students exiled to western China during the Anti-Rightist Campaign. They witnessed the Great Famine and wrote trenchant essays on China's political system that still echo today.
Others photos in the book were taken by the Singaporean artist and former Magnum photographer Sim Chi Yin, who accompanied me on some of the interviews. Chi Yin did beautiful landscapes that caught the theme of repressed and recovered memories that lie at the heart of this book.
Widow's Bridge, Daoxian, where many were beaten and tossed into the river in August 1967.
Chi Yin also took portraits of key people involved in the piece, especially the journalist Jiang Xue and the documentary filmmaker Ai Xiaoming.
Chinese writer Jiang Xue, whose essays were among the most popular accounts of China's draconian zero-Covid policy.
Chinese underground filmmaker and feminist scholar Ai Xiaoming in her home in Wuhan.
The book also contains reproductions of artworks that try to counter the “tyranny of the archive”–that reality is more than state-controlled archives can ever show us.
The artist and filmmaker Hu Jie's depiction of the poet Lin Zhao, forced to wear a "monkey mask" to prevent her from speaking in the days before her execution. Hu's work is an effort to fill in the archives' voids, allowing us to feel the past more viscerally than is possible with words.
Maps
Once again I was fortunate enough to work with the mapmaker Angela Hessler, who put together the beautiful map that you can see below, which reflects a key theme in the book–the landscape of memory. The logo of the magazine Spark is reproduced in the lower left-hand corner, while the logo of the contemporary journal Remembrance is in the lower right. The little torches indicate key locations mentioned in the book. Thanks Angela!
Purchase
Last but not least….the book is available now for preorder from OUP, Penguin, Amazon, Barnes, and any indy bookstore that you frequent.
I’d appreciate any pre-orders as it helps improve how the book is marketed, both in the bookstore and online. And afterwards, any reviews or feedback to the bookseller would be great–it helps keep the book in stock and in print.
Login
