Reading view

There are new articles available, click to refresh the page.

The Real Country: Idyll

Ask anyone who has lived in the country and they’ll recall its idyllic moments. To end this series, I celebrate a few of those in paintings from the turn of the nineteenth to twentieth centuries.

kuznetsovcelebration
Mykola Kuznetsov (1850-1929), In Celebration (1879-81), oil on canvas, 55 x 98 cm, Tretyakov Gallery Государственная Третьяковская галерея, Moscow, Russia. Wikimedia Commons.

This Ukrainian farm labourer is caught relaxing for a moment in the sun and flowers of early summer in Mykola Kuznetsov’s early In Celebration (1879-81).

segantiniploughing
Giovanni Segantini (1858–1899), Ploughing (1890), oil on canvas, 117.6 x 227 cm, Neue Pinakothek, Munich, Germany. Image © Ad Meskens, via Wikimedia Commons.

The pair of ploughmen in Giovanni Segantini’s Ploughing may not have had time to study the fine mountain views near the Alpine village of Savognin, but they and the other labourers in the right distance are enjoying the fine weather.

segantinihighnoonalps
Giovanni Segantini (1858–1899), High Noon in the Alps (1892), oil on canvas, 86 x 80 cm, Ohara Museum of Art 大原美術館, Kurashiki, Japan. Wikimedia Commons.

A couple of years later, Segantini’s High Noon in the Alps (1892) catches this shepherdess enjoying a brief break in her work, in the intense summer sunshine of the high plateau.

dagnanbouveretinforest
Pascal Dagnan-Bouveret (1852–1929), In the Forest (1893), oil on canvas, dimensions not known, Musée des beaux-arts de Nancy, Nancy, France. Image by G.Garitan, via Wikimedia Commons.

Pascal Dagnan-Bouveret shows that even itinerant workers could sit together and eat to the music of a violin when living In the Forest in 1893. Behind them are two oxen, and the forest that’s currently their home.

axentowiczfolkdance
Teodor Axentowicz (1859–1938), Kołomyjka, Oberek Taniec ludowy przed domem (Oberek Folk Dance in Front of a House) (1895), oil on canvas, 85 x 112.5 cm, Muzeum Narodowe w Warszawie, Warsaw, Poland. Wikimedia Commons.

In Poland, Teodor Axentowicz’s painting of Oberek Folk Dance in Front of a House from 1895 shows locals dancing the second most popular Polish folk dance after the polka. Most of these dancers are barefoot.

stottpeacefulrest
Edward Stott (1855–1918), Peaceful Rest (c 1902), oil on canvas, 60.5 x 81 cm, Private collection. Wikimedia Commons.

Edward Stott’s shepherd has stolen a moment of Peaceful Rest as his small flock drinks from a pond. He’s lighting a clay tobacco pipe, with his crook resting on his leg. Most of the painting uses a limited palette, with three splashes of colour standing out: the man’s face lit by the flame, the watchful sheepdog behind him, and a blue object protruding from the man’s jacket pocket.

bastienlepagelovevillage
Jules Bastien-Lepage (1848–1884), Love in the Village (1882), oil on canvas, 194 × 180 cm, Pushkin Museum of Fine Arts Музей изобразительных искусств им. А.С. Пушкина, Moscow, Russia. Wikimedia Commons.

Jules Bastien-Lepage’s Love in the Village shows a young couple on either side of a tumbledown fence, chatting intimately among the vegetable patches.

Idyllic moments indeed, but what happens in the country is often a far cry from the town.

astrupfjosfrieri
Nikolai Astrup (1880–1928), Fjøsfrieri (Early Courting) (1904), oil, dimensions not known, Private collection. Wikimedia Commons.

Nikolai Astrup’s humorous painting of Early Courting from 1904 shows a young couple at the far left engaged in ‘clothed courting’ in the unromantic surroundings of a cowshed. He has a bottle of drink in his pocket; whether that’s to give him courage or to weaken the resistance of his girlfriend is unclear.

The couple have sought the privacy of the cowshed, out of everyone’s way, but the boyfriend appears unaware that they’re being watched by someone up in the roof. From the apparent direction of gaze of the girlfriend and the blush on her cheeks, she has just noticed the peeping tom or watchful relative. The setting is enhanced by the sunlight pouring through the far window, illuminating two rows of the back-ends of cows. The wood floor between the cows appears to be decorated with small sketches, but those are actually piles of cow dung. Courting in the country must have been a sensorily rich experience.

Are sparse bundles faster than disk images?

Following tests over two years ago, I recommended the use of sparse bundles (UDSB) rather than plain UDIF read-write (UDRW) disk images, because of their faster and more consistent performance. Since then, read-write disk images have adopted a sparse file format, making them more space-efficient. However, their write performance has remained substantially lower than sparse bundles, and the latter remain the preferred format.

More recent performance tests have confirmed this in Sequoia, as shown in the table below.

xferdiskimage24

Most recently, tests of file compression showed relatively poor performance within a sparse bundle, with overall compression speed only 57% that of the SSD hosting the bundle despite apparently good write speeds. This article explores why, and whether that should alter choice of disk images.

Consistency in write speed

As previous articles note, inconsistent performance troubles disk images. Because of their complex bundle structure and storage across thousands of band files, it’s important to consider whether this poor performance is merely a reflection of that inconsistency.

sparsebundlewritesp0

The original Stibium write speed measurements, separated out from the collation shown in that recent article, show more dispersion about the regression line than other tests. As those are based on a limited number of file sizes, a total of 130 write speed measurements were performed using file sizes ranging from 4.4 MB to 3.0 GB, 50 of them being randomly chosen.

sparsebundlewritesp1

Those confirm that some individual write speeds are unexpectedly low, and dispersion isn’t as low as hoped for, but don’t make inconsistency a plausible explanation.

Band size

Unlike other types of disk image, data saved in sparse bundles is contained in many smaller files termed bands, and the maximum size of those band files can be set when a sparse bundle is created. By default, band size is set at 8.4 MB and there are only two ways to use a custom size, either when creating the sparse bundle using the command hdiutil create or in my free utility Spundle. Although those allow you to set a preferred band size, the size actually used may differ slightly from that.

Band size could have significant effects on the performance and stability of sparse bundles, as it determines how many band files are used to store data. Using a size that results in more than about 100,000 band files has in the past (with HFS+ at least) made sparse bundles prone to failure.

To assess the effect of different band sizes on compression rate, I therefore repeated exactly the same test file compression in sparse bundles with 12 band sizes ranging from 2.0 to 1020 MB, as measured rather than the value set.

sparsebundlebandstime

In the graph above, time to complete the compression task is shown for each of the band sizes tested, and shows a minimum time (fastest compression) at a band size of 15.4 MB, of about 8.3 seconds, significantly lower than the 11.08 seconds at the default of 8.4 MB. Band sizes between 2.0-8.4 MB showed little differences, but increasing band size to 33.8 MB and greater resulted in much slower compression.

sparsebundlebandsrate

When expressed as compression rates, peak performance at a band size of 15.4 MB is even clearer, and approaches 1.9 GB/s, more than twice that of a read-write disk image, but still far below rates in excess of 2.6 GB/s found for the USB4 and internal SSDs. With band sizes above 33.8 MB, compression rates fall below those of a read-write disk image.

Another way of looking at optimum band size is to convert that into the number of band files required. Fastest rates were observed here in sparse bundles that would have about 6,000-7,000 bands in total when the sparse bundle is full, and in this case with 2,000-2,300 band files in use. Performance fell when the maximum number of band files fell below 3,000, or those in use fell below 1,000.

Streaming

Compression tests impose different demands on storage from conventional read or write benchmarks because file data to be compressed is streamed by reading data from storage, and writing compressed data out to the same storage device. Although source and destination are separate files, allowing simultaneous reading and writing, if those files are stored in common band files, it’s likely that access has to be limited and can’t be simultaneous.

As observed here, that’s likely to result in performance substantially below that expected from single-mode transfer tests. Unfortunately, it’s also extremely difficult to measure attempts to simultaneously read and write from different files in the same storage medium.

Conclusions

  • Sparse bundles (UDSB) remain generally faster in use than read-write (UDRW) disk images.
  • Sparse bundle performance is sensitive to band size.
  • Default 8.4 MB bands aren’t fastest in all circumstances.
  • When sparse bundle performance is critical, it may be worth optimising band file size instead of using the default.
  • Tasks that stream data from and back to the same storage are likely to run more slowly in a sparse bundle than on its host storage.

Previous articles

Dismal write performance of Disk Images
Which disk image format?
Disk Images: Performance

Tune for Performance: Do you really need a big internal SSD?

The most common economy that many make when specifying their next Mac is to opt for just 512 GB internal storage, and save the $/€/£600 or so it would cost to increase that to 2 TB. After all, if you can buy a 2 TB Thunderbolt 5 SSD for around two-thirds of the price, why pay more? And who needs the blistering speed of that expensive internal storage when your apps work perfectly well with a far cheaper SSD? This article considers whether that choice matters in terms of performance.

To look at this, I’m going to use the same compression task that I used in this previous article, my app Cormorant relying on the AppleArchive framework in macOS to do all the heavy lifting. As results are going to differ considerably when using other apps and other tasks, I’d like to make it clear that this can’t reach general conclusions that apply to every task and every app: your mileage will vary. My purpose here is to show how you can work out whether using a slower disk will affect your app’s performance.

Methods

In that previous article, I concluded that compression speed was unlikely to be determined by disk performance, as even with all 10 P cores running compression threads, that rate only reached 2.28 GB/s, around the same as write speed to a good Thunderbolt 3 SSD. For today’s tests, I therefore set Cormorant to use the default number of threads (all 14 on my M4 Pro) so it would run as fast as the CPU cores would allow when compressing my standard 15.517 GB IPSW test file.

I’m fortunate to have a range of SSDs to test, and here use the 2 TB internal SSD of my Mac mini M4 Pro, an external USB4 enclosure (OWC Express 1M2) with a 2 TB Samsung 990 Pro SSD, and an external 2 TB Thunderbolt 3 SSD (OWC Envoy Pro SX). As few are likely to have access to such a range, I included two disk images stored on the internal SSD, a 100 GB sparse bundle, and a 50 GB read-write disk image, to see if they could be used to model external storage. All tests used unencrypted APFS file systems.

The first step with each was to measure its write speed using Stibium. Unlike more popular benchmarking apps for the Mac, Stibium measures the speed across a wide range of file sizes, from 2 MB to 2 GB, providing more insight into performance. After those measurements, those test files were removed, the large test file copied to the volume, and compressed by Cormorant at high QoS with the default number of threads.

Disk write speeds

compressionbydisk1

Results in each test followed a familiar pattern, with rapidly increasing write speeds to a peak at a file size of about 200 MB, then a steady rate or slow decline up to the 2 GB tested. The read-write disk image was a bit more erratic, though, with high write speeds at 800 and 1000 MB.

At 2 GB file size, write speeds were:

  • 7.69 GB/s for the internal SSD
  • 7.35 GB/s for the sparse bundle
  • 3.61 GB/s for the USB4 SSD
  • 2.26 GB/s for the Thunderbolt 3 SSD
  • 1.35 GB/s for the disk image.

Those are in accord with my many previous measurements of write speeds for those types of storage.

Compression rates

Times to compress the 15.517 GB test file ranked differently:

  • 5.57 s for the internal SSD
  • 5.84 s for the USB4 SSD
  • 9.67 s for the sparse bundle
  • 10.49 s for the Thunderbolt 3 SSD
  • 16.87 s for the disk image.

When converted to compression rates, sparse bundle results are even more obviously an outlier, as shown in the chart below.

compressionbydisk2

There’s a roughly linear relationship between measured write speed and compression rate in the disk image, Thunderbolt 3 SSD, and USB4 SSD, and little difference between the latter and the internal SSD. These suggest that disk write speed becomes the rate-limiting factor for compression when write speed falls below about 3 GB/s, but above that faster disks make little difference to compression rate.

Poor performance of the sparse bundle was a surprise, given how close its write speeds are to those of the host internal SSD. This is probably the result of compression writing a single very large file across its 14 threads; as the sparse bundle stores file data on a large number of band files, their overhead appears to have got the better of it. I will return to look at this in more detail in the near future, as sparse bundles have become popular largely because of their perceived superior performance.

The difference in compression rates between USB4 and Thunderbolt 3 SSDs is also surprisingly large. Of course, a Mac with fewer cores to run compression threads might not show any significant difference: a base M3 chip with 4 P and 4 E cores is unlikely to achieve a compression rate much in excess of 1.5 GB/s on its internal SSD because of its limited cores, so the restricted write speed of a Thunderbolt 3 SSD may not there become the rate-limiting factor.

Conclusions

  • The rate-limiting step in task performance will change according to multiple factors, including the effective use of multiple threads on multiple cores, and disk performance.
  • There’s no simple model you can apply to assess the effects of disk performance, and tests using disk images can be misleading.
  • You can’t predict whether a task will be disk-bound from disk benchmark performance.
  • Even expensive high-performance external SSDs can result in noticeably poor task performance. Maybe that money would be better spent on a larger internal SSD after all.

The Real Country: People

During the nineteenth century, paintings depicting ‘real’ life of ordinary people became increasingly popular, first in what has become known as social realism, pioneered by Jean-François Millet, then Naturalism, championed by Jules Bastien-Lepage until his untimely death in 1884. Among other themes, these put the case for the rural poor, and the desperate poverty that those living in the country had to endure.

Look carefully at many of their paintings, though, and the underlying stories aren’t as simple. One of the most evident problems is that many of those campaigning paintings used models who had been carefully posed. Look at their hands, feet and hair and you’ll often see someone who appears remarkably clean and kempt with no evidence of prolonged and arduous manual labour, even clothes that lack the dirt and mud so typical of those who work and walk on unpaved tracks and ploughed fields.

At the same time, photographic portraits of the poor became popular among some who sought to advance the art of photography. There are some notorious examples of early photographers who were caught posing carefully selected models in deliberately misleading circumstances, demonstrating how the camera can be made to lie.

My small selection of paintings of country people from the nineteenth century is an attempt to show some that appear most faithful records that weren’t intended to support political views or attract praise at a Salon.

bretonyounggirlknitting
Jules Breton (1827–1906), Young Girl Knitting (2) (1860), oil on canvas mounted on panel, 36 × 30 cm, Private collection. Wikimedia Commons.

Jules Breton had been born and brought up in the rural village of Courrières, and returned there to paint intimate portraits of those who continued to live there, including this Young Girl Knitting, seen in 1860. Many of these intimate works were sold to private collectors and have never been seen at exhibition.

bretonmotherfeedingbaby
Jules Breton (1827–1906), Mother Feeding her Baby (1863), oil on canvas, 55.2 x 45.1 cm, Private collection. Wikimedia Commons.

His portrait of this Mother Feeding her Baby from 1863 shows her wearing clogs, and clothing that has seen better days. She is feeding a very young baby in front of a frugal fire in what can only be her normal domestic conditions.

milletnormanmilkmaid
Jean-François Millet (1814–1875), A Norman Milkmaid at Gréville (1871), oil on cardboard, 80 × 55.6 cm, Los Angeles County Museum of Art, Los Angeles, CA. Wikimedia Commons.

Jean-François Millet had been born in the village of Gruchy, and was the first child of a farming family. Although his portrait of A Norman Milkmaid at Gréville from 1871 lacks fine detail to reinforce its authenticity, she’s clearly grubby and wasn’t painted in the studio.

liebermannpreservemakers
Max Liebermann (1847–1935), The Preserve Makers (1879), oil on mahogany wood, 49 × 65.3 cm, Museum der bildenden Künste, Leipzig, Germany. Wikimedia Commons.

Max Liebermann’s The Preserve Makers from 1879 shows a shed full of country women preparing foodstuffs for bottling and canning; the latter gradually came into use after 1810, but didn’t become popular until the First World War. This shows well the light factory work that was introduced to country areas in the late nineteenth and early twentieth centuries, and the women who provided its labour force.

dagnanbouveretaccident
Pascal Dagnan-Bouveret (1852–1929), An Accident (1879), oil on canvas, 90.7 x 130.8 cm, Walters Art Museum, Baltimore, MD. Wikimedia Commons.

Pascal Dagnan-Bouveret’s An Accident from 1879 is an exception among these, in that it was not only submitted to the Salon of 1880, but won the artist a first-class medal. It was also supposedly painted from memory, showing an incident that the artist witnessed with a medical friend who was similarly called to assist with an injury.

Despite that, it shows a country doctor cleaning and bandaging the injured hand of a boy, as the rest of the extended family looks on. Conditions appear primitive: the small bowl of water is heavily blood-stained, and the cloth by it looks filthy. It’s also rich in detail that appears authentic, from the boy’s shoe compared with that of the doctor, to the bald man standing in front of a treasured grandfather clock in the right background.

blepagenothingdoing
Jules Bastien-Lepage (1848–1884), Pas Mèche (Nothing Doing) (1882), oil on canvas, 132.1 x 89.5 cm, Scottish National Gallery, Edinburgh, Scotland. Wikimedia Commons.

Of Bastien-Lepage’s many portraits of country people, his Nothing Doing from 1882 appears the most convincing. From his unlaced mud-caked boots to his filthy and frayed waistcoat, this young agricultural worker looks the part. Bastien had spent his childhood in the village of Damvillers, although he was the son of a local artist. He frequently returned to his home village to paint local characters, such as this boy.

bretonsongoflark
Jules Breton (1827–1906), Song of the Lark (1884), oil on canvas, 110.6 × 85.8 cm, The Art Institute of Chicago, Chicago, IL. Wikimedia Commons.

The young woman in Breton’s Song of the Lark (1884) has detailed features that appear true to life, in her bare feet with grubby and battered toenails, and hands that have seen hard work with the hook she holds.

carpentierwashingturnips
Évariste Carpentier (1845–1922), The Turnip Washer (1890), oil on canvas, 130 x 195 cm, Musée des beaux-arts de Liège, Liège, Belgium. Wikimedia Commons.

Évariste Carpentier had been born into a farming family near the small town of Kuurne in Belgium. The Turnip Washer from 1890 is among the last of his thoroughly Naturalist paintings. Alongside the farmyard birds, two figures are busy washing piles of turnips in a small and dirty pond.

lhermittefarmworkerssupper
Léon Augustin Lhermitte (1844–1925), The Farmworkers’ Supper (1913), pastel on paper, dimensions and location not known. Wikimedia Commons.

Throughout the career of Léon Augustin Lhermitte he painted the working lives of farmworkers and country people. In his pastel of The Farmworkers’ Supper from 1913, he shows those who have been working outdoors during the long day enjoying a meal at its end.

Together these paintings build a dispassionate image of a countryside that might have been lacking in worldly goods, but was hardly starved and pestilent. And, as I’ll show in the next and concluding article, at times rural life could still be idyllic.

The Real Country: Potatoes

One important staple crop has been largely forgotten from both agricultural and rural history: the humble potato. First imported from South America to Europe in the latter half of the sixteenth century, for the first couple of centuries it was considered exotic eating. It quietly became increasingly important during the eighteenth century, and is now widely credited as the food that enabled the population boom in much of Europe in the nineteenth century.

For several years from 1845, a fungus-like organism late blight caused widespread crop failure in the poorer parts of Ireland, Scotland and elsewhere, causing the Great Irish Famine, with at least a million dying from starvation.

The potato is unusual for storing large amounts of starch in its tuber. Although starch is only about one fifth of the potato by weight, the remainder being water, it came be the staple food for much of the working population, in both country and city. Overton has calculated using estimated crop yields from the early nineteenth century that an acre of potatoes would have provided about 2.5 times as many calories as an acre of wheat.

Like wheat, growing potatoes is a process of amplification. Tubers from the previous crop are sown in the ground, and multiply to yield many times the weight originally sown. Production is thus not dependent on conventional seed, but on seed potatoes with a more limited life. If a whole crop is destroyed by blight, not only is there nothing to eat that year, but the seed potatoes for the following year are also lost. Blight remains in the soil, and once affected that land has to be used for crops other than potatoes.

However, potatoes had other advantages in a war-torn and taxed Europe. While troops rampaging in foreign countryside would often raid or burn fields of grain, damaging or stealing potatoes proved more resistant to invaders. They were also likely to escape taxes or charges levied on other arable crops.

During the eighteenth century, labourers fortunate enough to have a little land they could farm for themselves started to grow potatoes, to supplement their limited diet. Small potato patches sprung up in the countryside, and around towns. Production at scale remained unusual until the following century, when some English counties including Lancashire and Middlesex devoted significant areas to supply the growing cities. That in turn depended on transport such as canals and then railways to deliver sacks of potatoes to urban consumers.

vangoghpotatoes
Vincent van Gogh (1853–1890), Still Life with Potatoes (September 1885), oil on canvas, 47 x 57 cm, Museum Boijmans Van Beuningen, Rotterdam, The Netherlands. Wikimedia Commons.

When Vincent van Gogh was living in Borinage in Belgium, he was among labourers whose staple food was potatoes, in one of the areas that had grown them for longer than most. His Still Life with Potatoes, painted in September 1885, is his tribute to the humble vegetable.

Potato production was painted extensively by social realists including Jean-François Millet, in the middle of the nineteenth century.

milletpotatoplanters
Jean-François Millet (1814–1875), Potato Planters (c 1861), oil on canvas, 82.5 x 101.3 cm, Museum of Fine Arts Boston, Boston, MA. Wikimedia Commons.

Millet shows this back-breaking work in his Potato Planters from about 1861.

milletpotatoharvestwalters
Jean-François Millet (1814–1875), The Potato Harvest (1855), oil on canvas, 54 x 65.2 cm, Walters Art Museum, Baltimore, MD. Wikimedia Commons.

The Potato Harvest from 1855 is another more substantial work developed from Millet’s drawings. In the foreground, a man and woman are working together to fill sacks with the harvested potatoes, to be loaded onto the wagon behind them. The other four work as a team to lift the potatoes using forks and transfer them into wicker baskets, a gruelling task. Although the fields in the left distance are lit by sunshine, the dark scud-clouds of a heavy shower fill the sky at the right. Being poor, none of the workers has any wet-weather gear. The soil is also poor, full of stones, and yields would have been low despite the full sacks shown at the right.

milletangelus
Jean-François Millet (1814–1875), L’Angélus (The Angelus) (1857-59), oil on canvas, 55 x 66 cm, Musée d’Orsay, Paris. Wikimedia Commons.

Millet’s most famous single work, The Angelus, was completed around 1857-59. This had been commissioned by the American collector Thomas Gold Appleton, as Prayer for the Potato Crop, but underwent modification before Millet renamed it. At some stage, it’s thought to have included a child’s coffin, but that was overpainted.

It shows a couple, praying the Angelus devotion normally said at six o’clock in the evening, over the potatoes they have been harvesting. It’s dusk, and as the last light of the day fades in the sky, the bell in the distant church is ringing to mark the end of work, and the start of the evening. Next to the man is the fork he has been using to lift potatoes from the poor, stony soil; his wife has been collecting them in a wicker basket, now resting at her feet. Behind them is a basic wheelbarrow with a couple of sacks of potatoes on it, ready to be taken home.

blepageoctober
Jules Bastien-Lepage (1848–1884), October: Potato Gatherers (1878), oil on canvas, 180.7 x 196 cm, National Gallery of Victoria, Melbourne. Wikimedia Commons.

Twenty years later, Jules Bastien-Lepage painted what’s now sometimes known as October or Potato Gatherers (1878), but was originally shown as October: Potato Gatherers.

vangoghpotatoharvest
Vincent van Gogh (1853–1890), Woman Lifting Potatoes (1885), oil on canvas, dimensions not known, Van Gogh Museum, Amsterdam. Wikimedia Commons.

Vincent van Gogh painted Woman Lifting Potatoes (1885) and similar scenes during his time living with his parents in Nuenen, in North Brabant, the Netherlands.

molnarpotatoharvest
János Pentelei Molnár (1878–1924), The Potato Harvest (1901), oil on canvas, 79 x 109 cm, Private collection. Wikimedia Commons.

János Pentelei Molnár’s The Potato Harvest from 1901 takes this theme into the early years of the twentieth century, in Hungary.

ringmandiggingpotatoes
Laurits Andersen Ring (1854–1933), A Man Digging Potatoes (1901), oil on canvas, 86 x 67.5 cm, Statsministeriet, Copenhagen, Denmark. Wikimedia Commons.

Laurits Andersen Ring’s A Man Digging Potatoes from 1901 is pure social realism, as this smallholder uses his spade to lift the potatoes that are his staple diet. I believe that the plants shown here have suffered from potato blight, which also swept Europe periodically causing more famine and death.

fredericthreesisters
Léon Frédéric (1856–1940), The Three Sisters (1896), oil on canvas, dimensions not known, Private collection. Wikimedia Commons.

Although the girls shown in Léon Frédéric’s Three Sisters from 1896 are clean and well dressed, they’re sat together peeling their staple diet of potatoes in a plain and barren farmhouse. A glimpse of the shoes of the girl at the left confirms that they’re neither destitute nor affluent.

vangoghpotatoeaters
Vincent van Gogh (1853–1890), The Potato Eaters (1885), oil on canvas, 82 × 114 cm, Van Gogh Museum, Amsterdam, The Netherlands. Wikimedia Commons.

Vincent van Gogh’s The Potato Eaters (1885) is another revealing insight into the lives of poor labourers in Nuenen, who are about to feast on a large dish of potatoes under the light of an oil lamp.

friantfrugalmeal
Émile Friant (1863–1932), The Frugal Meal (1894), oil, dimensions and location not known. Wikimedia Commons.

Émile Friant’s The Frugal Meal (1894) continues the social theme, as a poor family with four daughters sits down to a meal consisting of a bowl piled high with potatoes, and nothing else. More worryingly, the pot on the floor at the left is empty.

Further reading

Christopher Shepherd’s brilliant essay on the history of the potato in The Oxford Handbook of Agricultural History (2024), ed. Jeannie Whayne, Oxford UP, ISBN 978 0 19 092416 4.

Why you need to make archives, and how to

We back up to ensure that we can recover files, whole volumes, our complete Mac if needed. When that crucial document you were working on earlier has vanished, or becomes damaged, or disaster strikes a disk, backups are essential. But how do you preserve all those documents that used to come on paper, records, correspondence and certificates? How will you or your successors be able to retrieve them in ten or thirty years time? This brief article considers how you should archive them safely, which isn’t the same as backing them up.

By archiving, I mean putting precious files somewhere they can be retrieved in at least ten years time. They may include financial, business, employment and personal records, as well as all finished work that you want to record for posterity. For most, they’ll also include a careful selection of still images, movies, and the more important documents you might create, such as books, theses and papers. They’re what you and the law want you to keep in perpetuity, and to be able to retrieve even after you’re gone.

To see how this can be achieved, I consider: the storage medium to be used, file formats that will be retrievable, how to index them for access, physical storage conditions, and the checks of their integrity that are needed.

Storage medium

While backups are most likely to be kept on hard disks or SSDs, neither of those is in the least suitable for archives, as they have relatively short lifetimes and are too sensitive to storage conditions. Instead, you need a removable medium, today probably Blu-ray disks intended for archival use, such as M-DISC.

For those with copious archives of importance beyond their family, Sony used to offer Optical Disk Archive systems, but those products were discontinued last year and don’t appear to have a suitable replacement. This illustrates one of the problems with planning for the more distant future: today’s technology can all too easily become orphaned.

Businesses are increasingly turning to cloud services to store their archives, but for the great majority of us the recurring cost makes this impractical. In any case, best practice should be to use cloud services as a supplement to a physical archive. iCloud is more affordable for the storage of most important documents, but requires a Legacy Contact to be appointed.

File formats

While it’s fine to archive documents in their original format, as you do in your backups, it’s also important to extract their contents into more permanent formats. Among those most likely to prove durable for the next 50-100 years are:

  • UTF-8 (and formerly ASCII) for text files,
  • JPEG and PNG for still images,
  • audio, video and rich media using one of the widely-used compression standards and file formats,
  • XML-based open document standards,
  • CSV for data,
  • PDF provided that it complies with one of the archival standards PDF/A-1 to /A-4.

You may find it worthwhile tarring together large collections of smaller files, but don’t use an unusual compression or ‘archive’ format, which might prove inaccessible in the future.

Indexing and access

For larger collections, even when structured carefully, a thorough list of contents in UTF-8 text format is essential. While there are index and search tools that could help, in this respect too archives are different from backups. If you’re going to be gathering TB of files, look at some of the commercial solutions. Although some are free to use, like the long-established Greenstone, they aren’t intended for casual users and might prove demanding.

Physical storage conditions

Never print on the disk itself, which can result in its degradation, and keep paper records alongside disks in the same container, but not inside the cases themselves, where they could damage them.

Archive optical disks should be stored in cases with centre hub security, not in sleeves. They must be kept in a cool, dry and dark container, in which there is no mould or fungus. They also need to be protected from physical threats such as flood and fire. Firesafes are popular furniture for this, but you must then ensure that their combination or keys are readily available and not separated from the safe.

There used to be a vogue for commercial data repositories, often underground storage sites that had been repurposed. Not only were those expensive, but many failed to take the care that they promised, and plenty went bankrupt and put their contents at risk. If you can arrange it, store one copy with you, and another at a friend’s or relative’s at least a few miles away.

Integrity checks

If you’re serious about maintaining your archives, some form of integrity checking, such as that provided by my free utilities Dintch, Fintch and cintch, is essential. Check a sample on each disk once a year, to ensure that none has started to deteriorate. If you do detect errors, that’s the time to burn a replacement before the original is lost to decay.

Conclusion

Backups are for recovery, while archives are for posterity. Start building your archives now, and keep them safe for the future.

Further reading

How to burn a Blu-ray disc in Monterey
Wikipedia point of entry

Postscript

Some of you are reporting widespread claims that some Blu-ray burners no longer work in Sequoia. I have therefore repeated the process that I described in Monterey, using exactly the same Pioneer burner connected to a Mac Studio M1 Max running macOS 15.1. I’m delighted to report that it still works perfectly, and I see no reason that any other recent Pioneer optical drive should prove incompatible. All you need to do is follow the instructions.

Happy archiving!

The Real Country: Hay

In the more northerly latitudes, grass that’s essential for cattle to graze grows little during the winter months. Farmers keeping cattle therefore have to provide alternative feed for their livestock for several months each year. This can include root crops such as brassica varieties including turnips and swedes (also known as rutabaga), but the most widespread is cut and dried grass as hay.

Where climate and day-length are suitable, as in much of England and France, dedicated hay meadows can provide two harvests each year. Left ungrazed through the winter, the first is normally ready to mow in the late Spring, and when there’s sufficient rainfall during the early summer, a second hay harvest can be obtained before the weather deteriorates in the early autumn. The mowing of hay has also been known as math, and mowing a second time is thus the aftermath or lattermath.

The essential requirement for hay is that it’s dried thoroughly, or it will rot over time and become unusable as fodder. In the centuries before mechanisation during the nineteenth century, this process was described as: first mow the grass, “scatter it about, gather it in windrows, cock it overnight, scatter it about, windrow it, cock it, and so on to the stack and stack it”. (Fussell) Those steps are shown well in paintings.

bruegelhayharvest
Pieter Bruegel the Elder (c 1525–1569), The Hay Harvest (1565), oil on panel, 114 x 158 cm, Lobkowicz Palace, Prague, Czechia. Wikimedia Commons.

The companion to Pieter Bruegel the Elder’s painting of the grain harvest, The Hay Harvest from 1565 shows all stages in progress. In the left foreground a man is beating the blade on his scythe to sharpen it ready for mowing. Three women are striding towards him with the rakes they use to scatter and gather the mown hay. Behind them, in the valley, others are gathering the hay into small stacks or cocks, where it continues to dry before being loaded onto the hay wagon to be taken back to the farm.

At the right are wicker baskets containing other crops, including what appear to be peas or beans, together with a red fruit.

hodlermower1898
Ferdinand Hodler (1853–1918), The Mower (c 1898), oil on canvas, 71.5 × 114 cm, Private collection. Wikimedia Commons.

Ferdinand Hodler’s marvellous Mower from about 1898 is seen sharpening the blade on his heavy scythe using a whetstone, as the sun rises behind and to the left.

blepagehaymaking
Jules Bastien-Lepage (1848–1884), Les Foins (Haymakers) (1877), oil on canvas, 160 x 195 cm, Musée d’Orsay, Paris. Wikimedia Commons.

The couple in Jules Bastien-Lepage’s Haymakers from 1877 are enjoying a short break from their labours, with the mown hay behind them still scattered to dry, before it can be raked into cocks.

Capitole Toulouse - Salle Henri-Martin - L'été par Henri Martin
Henri-Jean Guillaume Martin (1860–1943), Summer, or Mowers (1903), oil on canvas, dimensions not known, Capitole de Toulouse, Toulouse, France. Image by Didier Descouens, via Wikimedia Commons.

Henri-Jean Martin painted Summer, or Mowers in 1903, as mechanisation was spreading across Europe. Several small clusters of men are mowing the hay in this meadow with their scythes, as three young women are dancing in a ring on the bed of flowers, and another sits nursing an infant.

moretfenaisonbretagne
Henry Moret (1856–1913), Haymaking in Brittany (1906), oil on canvas, dimensions not known, Musée des beaux-arts de Vannes, Vannes, France. Wikimedia Commons.

Henry Moret’s Haymaking in Brittany from 1906 shows a smaller team busy mowing and raking on steeper ground.

pissarrohaymaking
Camille Pissarro (1830–1903), Haymaking, Éragny (1887), oil on canvas, 50 x 66 cm, Van Gogh Museum, Amsterdam, The Netherlands. Wikimedia Commons.

In Camille Pissarro’s Divisionist painting of Haymaking, Éragny from the summer of 1887, a team of women are raking the cocks into haystacks.

pymonenkohaymaking
Mykola Pymonenko (1862–1912), Haymaking (date not known), oil on canvas, dimensions not known, Fine Arts Museum Kharkiv Харківський художній музей, Kharkiv, Ukraine. Wikimedia Commons.

Women in this hay meadow in Ukraine are raking in the harvest to be transported by a hay wain drawn by a pair of oxen, as painted in Mykola Pymonenko’s undated Haymaking.

millethaystacksautumn
Jean-François Millet (1814–1875), Haystacks: Autumn (c 1874), oil on canvas, 85.1 x 110.2 cm, The Metropolitan Museum of Art, New York, NY. Wikimedia Commons.

In Jean-François Millet’s Haystacks: Autumn from about 1874, the harvest has been gathered, and three huge haystacks dominate the canvas. At the foot of one of them, a shepherd leans on his staff, resting from his labours as his flock grazes on the stubble.

Surplus hay was also a good cash crop for those who could get it transported to towns and cities. Along the east coast of England, barges were filled with hay then taken to London for sale. Much of the land in the county of Middlesex, to the west of London, was devoted to producing hay to feed horses in the city.

bevanhaycarts
Robert Bevan (1865–1925), Hay Carts, Cumberland Market (1915), oil on canvas, 47.9 x 61 cm, Yale Center for British Art, New Haven, CT. Wikimedia Commons.

Robert Bevan’s painting of Hay Carts, Cumberland Market from 1915 is a view of London’s last hay market, near to the artist’s studio. By this time, the bales shown were made by mechanical baling machines and brought to London by barge.

In the next article in this series, I’ll look at a novel crop that soon became the staple food for many, the potato.

Interiors by design: The Dutch Golden Age

Painting in the Dutch Golden Age underwent remarkable evolution. In the fifty years between the 1620s and the French invasion of the Dutch Republic in 1672, established genres grew novel sub-genres, with artists specialising in each. These included ‘genre’ scenes of everyday life, with artists devoted to painting taverns, women working, festivities, markets, or domestic interiors. The latter appear to have been among the first such depictions in modern art.

During the 1650s, interiors started to become distinct from other scenes of everyday life, as the significance of their figures diminished, although few if any dispensed with them altogether.

Gerard ter Borch (1617–1681) specialised in domestic interiors, some containing open-ended narratives to encourage the viewer to speculate on their resolution. Two centuries later, those were to be become popular again, particularly in Britain, where they were known as problem pictures and featured in the press.

terborchmessenger
Gerard ter Borch (1617–1681), The Messenger (Unwelcome News) (1653), oil on panel, 66.7 x 59.5 cm, Koninklijk Kabinet van Schilderijen Mauritshuis, The Hague, The Netherlands. Wikimedia Commons.

Ter Borch’s The Messenger, usually know as Unwelcome News, from 1653, develops his favourite theme of the arrival of a message. The young man at the left is still booted and spurred from riding to deliver a message to this couple. Slung over his shoulder is a trumpet, to announce his arrival and assert his importance. The recipient wears a shiny breastplate and riding boots, and is taken aback at the news the messenger brings. His wife leans on her husband’s thigh, her face serious.

The scene is the front room of a house in the Golden Age. Behind them is a traditional bed typical of living areas at the time, with some of their possessions resting on a table between the couple and their bed. Hanging up on a bedpost is the husband’s sword, and behind them are a gun and powder horn. Is this letter news of his recall to military service, perhaps? Will he soon have to ride away from his wife, leaving her alone to bring up their family?

Although the three figures take the limelight, ter Borch picks out the mundane details of the room behind them instead of letting them fade into darkness.

terborch3figuresconversing
Gerard ter Borch (1617–1681), Three Figures Conversing in an Interior (Paternal Admonition) (c 1653-55), oil on canvas, 71 x 73 cm, Rijksmuseum Amsterdam, Amsterdam, The Netherlands. Wikimedia Commons.

Three Figures Conversing in an Interior is another of ter Borch’s narrative genre works, and more popularly known as Paternal Admonition (c 1653-55). Standing with her back to us, wearing a plush going-out dress, is the daughter. To her left is a table, on which there’s a small reading stand with books, almost certainly including a Bible.

Her parents are young, and they too are fashionably dressed. Her mother appears to be drinking from a glass, but her father is at the very least cautioning his daughter, if not giving her a thorough dressing-down. He wears a sword at his side. Behind them is a large bed, and to the right the family dog looks on from the darkness.

terborchwomanwriting
Gerard ter Borch (1617–1681), Woman Writing a Letter (c 1655), oil on panel, 39 x 29.5 cm, Koninklijk Kabinet van Schilderijen Mauritshuis, The Hague, The Netherlands. Wikimedia Commons.

Ter Borch’s half-sister Gesina appears to have been his model for Woman Writing a Letter (c 1655), which makes obvious his connection with Vermeer. Move this woman to a desk lit through windows at the left, light her surroundings, and you have a painting similar to Vermeer’s interiors. This shows a heavy decorated table cover pushed back to make room for the quill, ink-pot, and letter. Behind the woman is her bed, surrounded by heavy drapery, and at the lower right is the brilliant red flash of the seat.

terborchletter
Gerard ter Borch (1617–1681), The Letter (c 1660-62), oil on canvas, 79 x 68 cm, The Royal Collection, London. Wikimedia Commons.

The Letter from about 1660-62 returns to ter Borch’s favourite theme of the reading and writing of letters. Two young women are working together, apparently replying to the letter being read by the woman on the right. A boy, perhaps their younger brother, has just brought in a tray bearing an ornate pitcher of drink. In front, a small dog is curled up asleep on a stool. Above them is an unlit chandelier suspended from a hanging ceiling.

metsuwasherwoman
Gabriël Metsu (1629–1667), Washerwoman (c 1650), oil on panel, 23.9 × 21 cm, Muzeum Narodowe w Warszawie, Warsaw, Poland. Wikimedia Commons.

Other specialists in genre painting like Gabriël Metsu also ventured towards interiors. His Washerwoman from about 1650 looks authentic and almost socially realist: the young woman is a servant, dressed in her working clothes, in the dark and dingy lower levels of the house. She looks tired, her eyes staring blankly at the viewer. She’s surrounded by the gear she uses, including a rope and pitcher to the right, and an earthenware bowl on display below it. The mantlepiece in the background features a blue and white plate.

dehoochwomandrinkingtwomen
Pieter de Hooch (1629–after 1684), A Woman Drinking with Two Men (c 1658), oil on canvas, 73.7 x 64.6 cm, National Gallery, London. Wikimedia Commons.

It’s easy to mistake Pieter de Hooch’s A Woman Drinking with Two Men from about 1658 for a Vermeer, and like the latter he decorates the far wall with a contemporary map. The Eighty Years’ War had not long ended, and the Dutch Republic was flourishing. Discarded objects are scattered on its black-and-white tiled floor. There’s a large and empty fireplace, and above it hangs a religious painting.

It was Jan Vermeer whose few surviving works explored interiors the most.

vermeermilkmaid
Johannes Vermeer (1632–1675), The Milkmaid (c 1660), oil on canvas, 45.5 x 41 cm, The Rijksmuseum Amsterdam, Amsterdam, The Netherlands. Wikimedia Commons.

In his Milkmaid from about 1660, a woman servant is pouring milk from a jug, beside a tabletop with bread. In the left foreground the bread and pots rest on a folded Dutch octagonal table, covered with a mid-blue cloth. A wicker basket of bread is nearest the viewer, broken and smaller pieces of different types of bread behind and towards the woman, in the centre. Behind the bread is a dark blue studded mug with pewter lid, and just in front of the woman (to the right of the mug) a brown earthenware ‘Dutch oven’ pot into which the milk is being poured. An ultramarine blue cloth (matching the woman’s apron) rests at the edge of the table. There are many other intriguing details in this interior.

vermeeryoungwomanwaterpitcher
Johannes Vermeer (1632–1675), A Young Woman with a Water Pitcher (c 1662-5), oil on canvas, 45.7 x 40.6 cm, The Metropolitan Museum of Art, New York, NY. Wikimedia Commons.

Vermeer’s Young Woman with a Water Pitcher from slightly later is another fine example, this time with more obvious control of focus effects for which his paintings are renowned. Details in this interior include the ornate tablecloth, a small lockable chest on the right of the table, the map hanging behind, and the window she is holding with her right hand.

Interiors then vanished from painting until their rebirth in the nineteenth century.

Disk Images: Bands, Compaction and Space Efficiency

Of the two most dependable types of disk image, read-write (UDRW) disk images are the simpler. The only options you can set are its internal file system, whether the container is encrypted, and its maximum capacity. From there on, macOS will automatically Trim the disk image when it’s mounted, and adjust the size it takes on disk accordingly.

Sparse bundles can be more complex to configure in the first place, and may need routine maintenance to ensure their size on disk doesn’t grow steadily with use. This article considers how significant those complications are.

Band size

Most if not all storage divides data into blocks; in the case of SSDs, the default block size in APFS is 4,096 bytes, and that’s effectively the unit of storage in the single file of a read-write disk image.

Its equivalent in a sparse bundle is the maximum size that each of its band files can reach, normally set to 8.4 MB in macOS Sequoia and its predecessors, and equivalent to slightly more than 2,000 SSD blocks. Smaller band sizes are more efficient in the space required to store the contents of a sparse bundle, but require more band files for the same quantity of data stored. Experience with older versions of macOS suggests that it’s not a good idea for the total number of band files to exceed 100,000, but I’m not aware of any recent evidence to support that.

When creating very large sparse bundles, of the order of TB, macOS now may adjust the band size to limit their number. Some users report that those sparse bundles perform better as a result, for instance when being used to store backups on a network. There don’t appear to be any advantages to using band sizes less than 8.4 MB.

Neither DropDMG nor Disk Utility currently appear to allow any control over band size; Spundle and hdiutil not only let you specify custom band sizes when creating sparse bundles, but also allow you to change band size by copying an existing sparse bundle into a new one, which could be useful if you were changing the maximum size of a sparse bundle.

Compaction

Each time a writeable APFS or HFS+ file system is mounted in a disk image, including both sparse bundle and read-write types, storage blocks that are no longer in use should be returned for reuse by the process of Trimming. This is automatic, and in the case of single-file read-write disk images, is the only maintenance that occurs.

Because sparse bundles add another layer of band files over SSD storage, they too require periodic maintenance in the process of compaction. To compact a sparse bundle, macOS scans the band files and removes those that are no longer being used by the file system in the sparse bundle. This only applies to sparse bundles with APFS or HFS+ file systems, though, and isn’t guaranteed to free up any space. One small catch is that, by default, compaction won’t take place on a notebook running on battery power; hdiutil and Spundle provide an option to override that.

Space efficiency in use

I’ve been unable to find any objective comparison between space efficiency of modern read-write disk images and sparse bundles, so have performed my own on a USB4 external SSD connected to my Mac Studio M1 Max. I created two test cases of 125 GB images on a freshly-formatted APFS volume, one a sparse bundle with the default 8.4 MB band size, the other a read-write disk image (UDRW). In macOS Sequoia 15.0.1, the initial size they took on disk was 14.8 MB for the sparse bundle, and 336 MB for the disk image.

Using my utility Stibium, I then wrote and deleted a series of files in each, as follows:

  1. 160 files of 2 MB to 2 GB written totalling 53 GB
  2. 40 largest of those, sizes 600 MB to 2 GB, were then deleted, leaving 120
  3. another 160 files written totalling 53 GB
  4. 40 largest of those deleted, leaving a total of 240
  5. another 160 files written totalling 53 GB
  6. 40 largest of those deleted, leaving a total of 360
  7. another 160 files written totalling 53 GB
  8. 40 largest of those deleted, leaving a total of 480
  9. another 160 files written totalling 53 GB, leaving a total of 640 files.

At that point, the sparse bundle had 104.08 GB stored in 12,408 band files, and the read-write disk image had 91.63 GB stored in its single file. I then deleted all files and folders in each of the images, and unmounted and remounted each image several times to ensure Trimming had completed. The sparse bundle then had 1.12 GB in 135 band files, and offered 124.66 GB of free space when mounted; the read-write disk image was slightly smaller at 937.9 MB on disk and offered exactly the same amount of free space when mounted.

Compacting the empty sparse bundle was reported as reclaiming 52.3 MB, and reduced the disk space taken by the band files slightly to 1.06 GB while retaining the same number of bands, and still offering 124.66 GB of free space when mounted.

In this case, following writing 800 files totalling over 250 GB, and deleting a total of 160 of them, compaction made a remarkably small difference to the free space returned following removal of all files. There is also very little difference in the space efficiency of sparse bundles and modern read-write disk images.

One consistent difference observed throughout was in write speeds: they remained constant at 3.2 GB/s for the sparse bundle, and 1.0 GB/s for the read-write disk image, as reported here previously.

Conclusions

  • Sparse bundles are more complicated than read-write disk images (UDRW), with band size to be set, and compaction to be performed.
  • Default band size appears to work well, and manually setting band size should seldom be necessary.
  • Both types appear highly efficient in their use of disk space, with only small differences between them.
  • Although it might be important to compact sparse bundles in some cases, the amount of free space returned by compaction is unlikely to be significant in many circumstances.

Previous articles

Introduction
Tools
How read-write disk images have gone sparse
Performance

Disk Images: Performance

Over the last few years, the performance of disk images has been keenly debated. In some cases, writing to a disk image proceeds at a snail’s pace, but this has appeared unpredictable. Over two years ago, I reported sometimes dismal write performance to disk images, summarised in the table below.

diskimages13

This article presents new results from tests performed using macOS 15.0.1 Sequoia, which should give a clearer picture of what performance to expect now.

Methods

Previous work highlighted discrepancies depending on how tests were performed, whether on freshly made disk images, or on those that had already been mounted and written to. The following protocol was used throughout these new tests:

  1. A 100 GB APFS disk image was created using Disk Utility, which automatically mounts the disk image on completion.
  2. A single folder was created on the mounted disk image, then it was unmounted.
  3. After a few seconds, the disk image was mounted again by double-clicking it in the Finder, and was left mounted for at least 10 seconds before performing any tests. That should ensure read-write disk images are converted into sparse file format, and allows time for Trimming.
  4. My utility Stibium then wrote 160 test files ranging in size from 2 MB to 2 GB in randomised order, a total of just over 53 GB, to the test folder.
  5. Stibium then read those files back, in the same randomised order.
  6. The disk image was then unmounted, its size checked, and it was trashed.

All tests were performed on a Mac Studio M1 Max, using its 2 TB internal SSD, and an external Samsung 980 Pro 2 TB SSD in an OWC Express 1M2 enclosure running in USB4 mode.

Results

These are summarised in the table below.

xferdiskimage24

Read speeds for sparse bundles and read-write disk images were high, whether the container was encrypted or not. On the internal SSD, encryption resulted in read speeds about 1 GB/s lower than those for unencrypted tests, but differences for the external SSD were small and not consistent.

Write speeds were only high for sparse bundles, where they showed similar effects with encryption. Read-write disk images showed write speeds consistently about 1 GB/s, whether on the internal or external SSD, and regardless of encryption.

When unencrypted, read and write speeds for sparse (disk) images were also slower. Although faster than read-write disk images when writing to the internal SSD, read speed was around 2.2 GB/s for both. Results for encrypted sparse images were by far the worst of all the tests performed, and ranged between 0.08 to 0.5 GB/s.

Surprisingly good results were obtained from a new-style virtual machine with FileVault enabled in its disk image. Although previous tests had found read and write speeds of 4.4 and 0.7 GB/s respectively, the Sequoia VM achieved 5.9 and 4.5 GB/s.

Which disk image?

On grounds of performance only, the fastest and most consistent type of disk image is a sparse bundle (UDSB). Although on fast internal SSDs there is a reduction in read and write speeds of about 1 GB/s when encrypted using 256-bit AES, no such reduction should be seen on fast external SSDs.

On read speed alone, a read-write disk image is slightly faster than a sparse bundle, but its write speed is limited to 1 GB/s. For disk images that are more frequently read from than written to, read-write disk images should be almost as performant as sparse bundles.

However, sparse (disk) images delivered weakest performance, being particularly slow when encrypted. Compared with previous results from 2022, unencrypted write performance has improved, from 0.9 to 2.0 GB/s, but their use still can’t be recommended.

Performance range

It’s hard to explain how three different types of disk image can differ so widely in their performance. Using the same container encryption, write speeds ranged from 0.08 to 3.2 GB/s, for a sparse image and sparse bundle, on an external SSD with a native write speed of 3.2 GB/s. It’s almost as if sparse images are being deprecated by neglect.

Currently, excellent performance is also delivered by FileVault images used by Apple’s lightweight virtualisation on Apple silicon. The contrast is great between its 4.5 GB/s write speed and that of an encrypted sparse image at 0.1 GB/s, a factor of 45 when running on identical hardware.

Recommendations

  • For general use, sparse bundles (UDSB) are to be preferred for their consistently good read and write performance, whether encrypted or unencrypted.
  • When good write speed is less important, read-write disk images (UDRW) can be used, although their write performance is comparable to that of USB 3.2 Gen 2 at 10 Gb/s and no faster.
  • Sparse (disk) images (UDSP) are to be avoided, particularly when encrypted, as they’re likely to give disappointing performance.
  • Encrypting UDSB or UDRW disk images adds little if any performance overhead, and should be used whenever needed.

Previous articles

Introduction
Tools
How read-write disk images have gone sparse

Disk Images: How read-write disk images have gone sparse

Until about three years ago, most types of disk image had fixed size. When you created a read-write disk image (UDRW) of 10 GB, it occupied the same 10 GB on disk whether it was full or empty. The only two that could grow and shrink in size were sparse bundles and sparse disk images, with the former generally preferred for its better resizing and performance.

This changed silently in Monterey, since when read-write disk images have been automatically resized by macOS, and saved in APFS sparse file format. As this remains undocumented, this article explains how this works, and how and where you can use it to your advantage.

Sparse

In this context, the word sparse refers to two very different properties:

  • Sparse bundles and sparse (disk) images are types of disk image that can change in size, and can be stored on a wide range of file systems and storage, including on NAS and other networked storage.
  • Sparse files are stored in a highly efficient file format available in modern file systems, where only the data in a file is stored, and empty space within that file doesn’t waste storage space. It’s available in APFS, but not in HFS+.

Requirements

For read-write disk images to be sparse, the following are required:

  • The disk image must be saved to, and remain on, an APFS volume.
  • The file system within the disk image can be either APFS or HFS+, but not FAT or ExFAT.
  • The disk image must be created and unmounted first. For that initial mount, the disk image isn’t a sparse file, so occupies its full size on disk.
  • Whenever that disk image is mounted again, and has sufficient free space within its set limit, it will be saved in sparse file format, and occupy less than its full size on disk.

Demonstration

  1. Create a new read-write disk image of at least 1 GB size with an internal file system of APFS on an APFS volume, using DropDMG, Disk Utility, or an alternative.
  2. Once it has been created and mounted, unmount it in the Finder.
  3. Select the disk image in the Finder and open the Get Info dialog for it. Confirm that its size on disk is the same as that set.
  4. (Optional) Open the disk image using Precize, and confirm that it’s not a sparse file.
  5. Double-click the disk image to mount it in the Finder, and wait at least 10 seconds before unmounting it.
  6. Select the disk image in the Finder and open the Get Info dialog for it. Confirm that its size on disk is now significantly less than that set.
  7. (Optional) Open the disk image using Precize, and confirm that it has now become a sparse file.

How it works

When you create the disk image, macOS creates and attaches its container, and creates and mounts the file system within that. This is then saved to disk as a regular file occupying the full size of the disk image, plus the overhead incurred by the disk image container itself. No sparse files are involved at this stage.

When that disk image is mounted next, its container is attached through diskarbitrationd, then its file system is mounted. If that’s APFS (or HFS+), it undergoes Trimming, as with other mounts. That coalesces free storage blocks within the image to form one contiguous free space. The disk image is then saved in APFS sparse file format, skipping that contiguous free space. When the file system has been unmounted and the container detached, the space used to store the disk image has shrunk to the space actually used within the disk image, plus the container overhead. Unless the disk image is almost full, the amount of space required to store it on disk will be smaller than the full size of the disk image.

This is summarised in the diagram below.

SparseDiskImage1

The size of read-write disk images is therefore variable depending on the contents, the effectiveness of Trimming in coalescing free space, and the efficiency of APFS sparse file format.

Conversion to sparse file

When mounting an APFS file system in a read-write disk image, APFS tests whether the container backing store is a sparse format, or a flat file. In the case of a newly created read-write disk image that hasn’t yet been converted into a sparse file, that’s detected prior to Spaceman (the APFS Space Manager) scanning for free blocks within its file system. When free blocks are found, APFS sets the type of backing store to sparse, gathers the sparse bytes and ‘punches a hole’ in the disk image’s file extents to convert the container file into sparse format. That appears in the log as:
handle_apfs_set_backingstore:6172: disk5s1 Set backing store as sparse
handle_apfs_set_backingstore:6205: disk5 Backing storage is a raw file
_punch_hole_cb:37665: disk3s5 Accumulated 4294967296 sparse bytes for inode 30473932 in transaction 3246918, pausing hole punching

where disk5s1 is the disk image’s mounted volume, and disk3s5 is the volume in which the disk image container is stored.

Trimming for efficient use of space

That conversion to sparse format is normally only performed once, but from then on, each time that disk image is mounted it’s recognised as having a sparse backup store, and Spaceman performs a Trim to coalesce free blocks and optimise on-disk storage requirements. For an empty read-write disk image of 2,390,202 blocks of 4,096 bytes each, as created in a 10 GB disk image, log entries are:
spaceman_scan_free_blocks:4106: disk5 scan took 0.000722 s, trims took 0.000643 s
spaceman_scan_free_blocks:4110: disk5 2382929 blocks free in 7 extents, avg 340418.42
spaceman_scan_free_blocks:4119: disk5 2382929 blocks trimmed in 7 extents (91 us/trim, 10886 trims/s)
spaceman_scan_free_blocks:4122: disk5 trim distribution 1:0 2+:0 4+:0 16+:0 64+:0 256+:7

accounting for a total of 9.8 GB.

Changes made to the contents of the disk image lead to a gradual reduction in Trim yield. For example, after adding files to the disk image and deleting them, instead of yielding the full 9.8 GB, only 2,319,074 blocks remain free, yielding a total of 9.5 GB.

For comparison, initial Trimming on a matching empty sparse bundle yields the same 9.8 GB. After file copying and deletion, and compaction of the sparse bundle, Trimming performs slightly better, yielding 2,382,929 free blocks for a total of 9.6 GB. Note that Trimming of sparse bundles is performed by APFS Spaceman separately from management of bands in backing storage, which isn’t a function of the file system.

Size efficiency

Although read-write disk images stored as sparse files are efficient in their use of disk space, they’re still not as compact as sparse bundles. For an empty 10 GB image, the read-write type requires 240 MB on disk, but a sparse bundle only needs 13.9 MB. After light use storing files, then deleting the whole contents, a 10 GB read-write disk image grows to occupy 501 MB, but following compaction a sparse bundle only takes 150 MB. That difference may not remain consistent over more prolonged use, though, and ultimately compacting sparse bundles may cease freeing any space at all.

It’s also important to remember that sparse bundles need to be compacted periodically, if any of their contents are deleted, or they may not reduce in size after deletions. Read-write disk images can’t be compacted, and reclaim disk space automatically.

Benefits and penalties

Read-write disk images saved as sparse files are different from sparse bundles in many ways. Like any sparse file, the disk image still has the same nominal size as its full size, and differs in the space taken on disk. Sparse bundles should normally only have band files sufficient to accommodate their current size, so their nominal size remains similar to the space they take on disk. The result is that, while read-write disk images in sparse file format will help increase free disk space, their major benefit is in reducing ‘wear’ in SSDs by not wasting erase-write cycles storing empty data.

Unlike the band file structure in sparse bundles, which can be stored on almost any disk, APFS sparse files have to be treated carefully if they are to remain compact. Moving them to another file system or over a network is likely to result in their being exploded to full size, and I have explained those limitations recently.

Both read-write disk images and sparse bundles deliver good read performance, but write performance is significantly impaired in read-write disk images but not sparse files. Encryption of disk images and sparse bundles also has significant effects on their performance, and in some cases write performance is badly affected by encryption. I have previously documented their performance in macOS Monterey, and will be updating those figures for Sequoia shortly.

Summary

  • Read-write disk images saved on APFS storage in Monterey and later are no longer of fixed size, and should use significantly less disk space unless full, provided the disk image has been unmounted at least once since creation, and the file system in the disk image is APFS or HFS+.
  • This is because macOS now saves the disk image in sparse file format.
  • To retain the disk image in sparse file format, it needs to remain in APFS volumes, and normal precautions are required to maintain its efficient use of space. The disk image can’t be manually compacted, in the way that sparse bundles require to be.
  • The main benefit of this strategy is to minimise erase-write cycles, so reducing ‘wear’ on SSDs.
  • Sparse bundles are still more efficient in their use of disk space, and have higher write speeds, but read-write disk images are now closer in their efficiency and performance.

Previous articles

Introduction
Tools

Disk Images: Tools

If you’re going to use disk images of any type, then getting the right tool for the job is essential. This article considers the leading candidates:

  • Disk Utility, bundled with macOS
  • DropDMG, $24.99 from C-Command, or from the App Store
  • Spundle, free from its Product Page here
  • hdiutil, the command tool bundled with macOS.

Although I’m sure there are a great many others, IMHO those should be at the top of your list.

Disk Utility

Create a new disk image using the New Image command in its File menu and there’s a basic range of choices on offer.

dmgdiskutil

This dialog has a longstanding bug, where it can reset the size you’ve entered if you change another setting, which can help you make mistakes. Otherwise, it gives limited access to some of the many options available, sufficient for the occasional and not too demanding user. Further options are available in its Images menu, including verification, adding checksums, conversions between types, and resizing. Notable by its absence is the ability to change the password of an encrypted disk image, which is unhelpfully deferred to the command line.

Documentation in Disk Utility’s Help book is also scant, and insufficient to serve as a reference. As Apple doesn’t provide any further technical information, apart from that in man hdiutil, you may find yourself searching websites such as this.

DropDMG

Since its release over 22 years ago, this has been the first choice for many who need to work with disk images, and is without doubt the best for those who distribute software in disk images. It has grown into the most comprehensive and capable utility for working with any type of disk image, and is backed up by a superb 123-page manual that goes a long way to filling the gap left by Apple. That manual is well-maintained, and contains links to recent technical articles and further information.

dmgdropdmg

DropDMG’s options for creating a new disk image far exceed those in Disk Utility. Particularly helpful are the compatible version hints shown on various options, to remind you of which file systems are available in different macOS versions, and which types of disk image container are supported. DropDMG will even convert old NDIF disk images last used in Mac OS 9 to more modern formats. It will also change the password of an encrypted disk image from a menu command.

For those who need to work with standard configurations, perhaps for software distribution, it lets you save and reuse them with ease. Those can include signing with developer certificates, product licences, background images, custom volume icons, and more. Whichever type of disk image you want to create or maintain, DropDMG should be your first choice.

Spundle

There are a few options for sparse bundles that even DropDMG doesn’t expose, such as control over band size, the ability to resize a sparse bundle, and to change its band size. If you want access to those, Spundle is a useful adjunct.

dmgspundle

Note that, unlike DropDMG, Spundle only works with sparse bundles.

hdiutil

Although this remains the definitive command tool that offers types of disk image and features you didn’t even know existed, it’s fiendishly complex to use, with a sprawling and overloaded grammar. Its man page runs to more than 11,000 words, but appears never to have been rewritten into any cohesive account of disk images, or command options. For example, change information is given in two sections, Compatibility and What’s New. Changes made in Catalina and later appear at the end of the Compatibility section, then the final What’s New section reverses time order and goes back from Catalina to Mac OS X 10.5.

I only recommend hdiutil for those who need to work with disk images in shell scripts, or for those few features that aren’t available in DropDMG or, for sparse bundles, in Spundle. It’s a command tool of last resort.

Previous article

Introduction

Disk Images: Introduction

A disk image is a file or a bundle containing what could otherwise be the contents of a disk. It’s a common way to store and move complete file systems in a neat package, for items that need to be separated from the physical storage provided by a drive. macOS uses disk images for some tasks of great importance, including:

  • Recovery and Hardware Diagnostics systems,
  • additions to macOS such as Safari, its supporting frameworks, and dyld caches, in cryptexes,
  • networked storage for Time Machine backups, in sparse bundles,
  • lightweight virtual machines on Apple silicon Macs.

You could use them to store encrypted data on unencrypted volumes, and they’re often used for delivering Apple and third-party software.

Disk images are poorly documented for both users and developers, and have changed significantly over the last few years. Articles in this series explain how to choose between different types of disk image, how to create and use them, and what to do when they go wrong.

Containers and file systems

Disk images consist of two distinct components: the file or bundle itself functioning as a container, and the file system contained inside it. When referred to in this context, disk image containers are completely unrelated to the sandbox containers found in ~/Library/Containers.

This distinction is important in several respects, although it isn’t apparent when you use disk images in the Finder. Preparing a disk image for access involves two separate functions: attaching its container, and mounting any file systems found inside it. When that’s performed by the Finder, perhaps by double-clicking the disk image, those two actions appear fused into one. Similarly, removing the disk image requires all its mounted file systems to be unmounted first, then the container is detached.

One feature that’s widely confused is the encrypted disk image. This involves encryption of the whole container, rather than using an encrypted file system within it. Now that Disk Utility no longer supports the creation of encrypted HFS+ volumes, one remaining alternative is to use an encrypted disk image containing an HFS+ volume.

If you want an analogy for disk images, attaching the container is like connecting an external disk, and once that has been performed, the file systems contained by that disk have to be mounted before you can access their contents.

Types

There are many different types of disk image in use, of which the two this series is most concerned with are plain UDIF read-write disk images (UDRW), and sparse bundles (UDSB). Others you may encounter include:

  • plain UDIF read-only (UDRO),
  • various compressed versions of UDRW,
  • CD/DVD master for export (UDTO),
  • sparse disk image (UDSP), a single file rather than a bundle.

Those specify the container format; within each, there’s the usual choice of file systems, although throughout these articles it will normally be assumed that APFS will be used unless otherwise specified.

The word sparse in sparse bundle and sparse disk image doesn’t refer to APFS sparse files, but to the fact that those types of disk image can grow and diminish in size, and normally try to occupy the minimum amount of disk space. This is an unfortunate name collision.

Structure

With the exception of sparse bundles, all disk images are contained within a single file of opaque structure.

Sparse bundles consist of a single bundle folder containing:

  • bands, a folder containing the contents of the disk image in band files
  • info.plist and its backup copy info.bckup, containing settings including band size
  • lock, an empty file
  • mapped, a folder containing small data files to match all of the band files except the first
  • token, an empty file.

Container size

Until this changed in Monterey (or thereabouts), non-sparse disk images had fixed container sizes. Create a UDIF read-write disk image (UDRW) of 10 GB, and the space occupied by it on disk was approximately 10 GB, whether it was empty or full. Although it remains undocumented, when stored on APFS volumes, UDRW disks can now take advantage of APFS sparse file format, and will normally only occupy the disk space required for the contents of their file system.

This is only true once the disk image has been mounted for the first time after it has been created, mounted and unmounted. To see this, create a test read-write disk image (for example, using Disk Utility) of 10 GB size. Then unmount it, and use the Finder’s Get Info command to inspect its size on disk. That will be 10 GB. Then mount the disk image again, pause a couple of seconds, unmount it, and Get Info will show its size on disk is now much smaller.

As I’ll explain in detail in a later article, this is because each time that disk image is mounted, if its internal file system is HFS+ or APFS, its contents will be trimmed, and saved to disk in sparse file format, which omits all its empty space. This only applies to read-write disk images when they’re stored in APFS file systems; copy them to HFS+ and they’ll explode to full size, as HFS+ doesn’t support the sparse file format.

Considering just the two leading types, empty sizes for a 100 GB disk image are:

  • a sparse bundle is 35.4 MB empty, 53.3 GB when containing about 51 GB files, stored across 6,359 band files.
  • a read-write disk image (UDRW) shrinks to 333.8 MB once stored as an empty sparse file, 53.6 GB when containing about 51 GB files, in a single file container.

Performance

Some types of disk image perform poorly, and can be very slow to write to. Recent versions of macOS have brought improvements, although some options such as encryption can still impair performance significantly. For the two leading types, when their container is stored on the internal SSD of an Apple silicon Mac, with native read and write speeds of around 6-7 GB/s:

  • an initially empty unencrypted sparse bundle reads at 5.1 GB/s, and writes at 4.8 GB/s.
  • an initially empty unencrypted read-write disk image (UDRW) reads at 5.3 GB/s, and writes at only 1 GB/s.

Tests were performed using my utility Stibium, across a range of 160 files of 2 MB to 2 GB size in randomised order, with macOS 15.0.1.

Key points

  • Disk images consist of a file or bundle containing one or more file systems; the container and its contents are distinct.
  • To access the contents of a disk image, the container is first attached, then the file system(s) within it are mounted. In the Finder, those two processes appear as a single action.
  • Encrypted disk images encrypt the container, and don’t necessarily contain encrypted file systems.
  • Disk images come in many different types, and can contain different file systems.
  • Sparse bundles have a file and folder structure inside their bundle folder, with their data saved in band files; all other disk images are single files.
  • Sparse bundles grow and shrink according to the size of files stored within them.
  • In recent macOS, and on APFS, read-write disk images (UDRW) are stored in APFS sparse file format, enabling them to grow and shrink as well.
  • In recent macOS, unencrypted sparse bundles have read and write performance close to that of the disk they’re stored on. Read-write disk images read at similar speeds, but write more slowly, at about 20% of read speed.

很久以前

(Abstract)《Long, Long Ago》是 1883 年英国作曲家创作的歌曲。目前经常有纯音乐用于儿童钢琴曲或婴儿催眠曲,也有英文唱词。然而上世纪 90 年代初它有中文唱词,目前已“失传”。

主宾谓

之前聊到,日文、藏文的语序结构,和我们习惯的中文、英文不同,是谓语动词放在句子最后的「主语-宾语-谓语」的形式。

  • 中文、英文,是「主-谓-宾」。譬如:我-是-学生。我-想-你。
  • 日文、藏文,是「主-宾-谓」。类似于:我-学生-是。我-你-想。

:(吐槽)所以人们常说的,日本人懂礼貌,会听人把话说完。其实是因为这样的结构,需要认真听到最后一个词,才知道整个句子要说「是」或「不是」啊。

:对于需要使用不同敬语的日本人,也方便他们先把宾语对象列出来,再根据其身份,决定用什么样的敬语去修饰动词。


另一个 blog 有时候写得少的原因,大概是在「文章是在写给谁?」这方面,无意识地发生了混乱。

除去一部分

  • 技术贴
  • 分享有趣的经历或见闻
  • 对自己状态的描述、分析、展示

的篇目;其它很多文章,应该是(有意识或无意识地)有一个,潜在的写作对象的。他可能是

  • 现实中特定的人,可能是情感相关,也可能只是隔空喊话。当然,对方未必会来看;
  • 一个虚幻的,用来倾诉的对象;
  • 想要吐槽的某些现象,所代表的人群;
  • 预计会来看这个 blog 的读者们,不是特定的人,但有某种同温层特质;
  • 也可能,这个对象还是我自己。

于是,经常写到一半,突然意识到这个对象的存在,然后陷入「我这样写,有什么意义吗」的沮丧,也就不写了。

又或者,吐槽吐到一半,突然意识到,我所吐槽的特质,其实和来看 blog 的人,并不相关。于是反而担心,会不会让读者们对号入座产生误解,或者觉得我这个对空掰扯道理的样子很爹味儿之类的。

——就像在「主-宾-谓」的句子里,谓语写一半了,才意识到,那个预设的宾语的存在。

❌