Normal view

There are new articles available, click to refresh the page.
Before yesterdayMain stream

The Real Country: Potatoes

By: hoakley
6 November 2024 at 20:30

One important staple crop has been largely forgotten from both agricultural and rural history: the humble potato. First imported from South America to Europe in the latter half of the sixteenth century, for the first couple of centuries it was considered exotic eating. It quietly became increasingly important during the eighteenth century, and is now widely credited as the food that enabled the population boom in much of Europe in the nineteenth century.

For several years from 1845, a fungus-like organism late blight caused widespread crop failure in the poorer parts of Ireland, Scotland and elsewhere, causing the Great Irish Famine, with at least a million dying from starvation.

The potato is unusual for storing large amounts of starch in its tuber. Although starch is only about one fifth of the potato by weight, the remainder being water, it came be the staple food for much of the working population, in both country and city. Overton has calculated using estimated crop yields from the early nineteenth century that an acre of potatoes would have provided about 2.5 times as many calories as an acre of wheat.

Like wheat, growing potatoes is a process of amplification. Tubers from the previous crop are sown in the ground, and multiply to yield many times the weight originally sown. Production is thus not dependent on conventional seed, but on seed potatoes with a more limited life. If a whole crop is destroyed by blight, not only is there nothing to eat that year, but the seed potatoes for the following year are also lost. Blight remains in the soil, and once affected that land has to be used for crops other than potatoes.

However, potatoes had other advantages in a war-torn and taxed Europe. While troops rampaging in foreign countryside would often raid or burn fields of grain, damaging or stealing potatoes proved more resistant to invaders. They were also likely to escape taxes or charges levied on other arable crops.

During the eighteenth century, labourers fortunate enough to have a little land they could farm for themselves started to grow potatoes, to supplement their limited diet. Small potato patches sprung up in the countryside, and around towns. Production at scale remained unusual until the following century, when some English counties including Lancashire and Middlesex devoted significant areas to supply the growing cities. That in turn depended on transport such as canals and then railways to deliver sacks of potatoes to urban consumers.

vangoghpotatoes
Vincent van Gogh (1853–1890), Still Life with Potatoes (September 1885), oil on canvas, 47 x 57 cm, Museum Boijmans Van Beuningen, Rotterdam, The Netherlands. Wikimedia Commons.

When Vincent van Gogh was living in Borinage in Belgium, he was among labourers whose staple food was potatoes, in one of the areas that had grown them for longer than most. His Still Life with Potatoes, painted in September 1885, is his tribute to the humble vegetable.

Potato production was painted extensively by social realists including Jean-François Millet, in the middle of the nineteenth century.

milletpotatoplanters
Jean-François Millet (1814–1875), Potato Planters (c 1861), oil on canvas, 82.5 x 101.3 cm, Museum of Fine Arts Boston, Boston, MA. Wikimedia Commons.

Millet shows this back-breaking work in his Potato Planters from about 1861.

milletpotatoharvestwalters
Jean-François Millet (1814–1875), The Potato Harvest (1855), oil on canvas, 54 x 65.2 cm, Walters Art Museum, Baltimore, MD. Wikimedia Commons.

The Potato Harvest from 1855 is another more substantial work developed from Millet’s drawings. In the foreground, a man and woman are working together to fill sacks with the harvested potatoes, to be loaded onto the wagon behind them. The other four work as a team to lift the potatoes using forks and transfer them into wicker baskets, a gruelling task. Although the fields in the left distance are lit by sunshine, the dark scud-clouds of a heavy shower fill the sky at the right. Being poor, none of the workers has any wet-weather gear. The soil is also poor, full of stones, and yields would have been low despite the full sacks shown at the right.

milletangelus
Jean-François Millet (1814–1875), L’Angélus (The Angelus) (1857-59), oil on canvas, 55 x 66 cm, Musée d’Orsay, Paris. Wikimedia Commons.

Millet’s most famous single work, The Angelus, was completed around 1857-59. This had been commissioned by the American collector Thomas Gold Appleton, as Prayer for the Potato Crop, but underwent modification before Millet renamed it. At some stage, it’s thought to have included a child’s coffin, but that was overpainted.

It shows a couple, praying the Angelus devotion normally said at six o’clock in the evening, over the potatoes they have been harvesting. It’s dusk, and as the last light of the day fades in the sky, the bell in the distant church is ringing to mark the end of work, and the start of the evening. Next to the man is the fork he has been using to lift potatoes from the poor, stony soil; his wife has been collecting them in a wicker basket, now resting at her feet. Behind them is a basic wheelbarrow with a couple of sacks of potatoes on it, ready to be taken home.

blepageoctober
Jules Bastien-Lepage (1848–1884), October: Potato Gatherers (1878), oil on canvas, 180.7 x 196 cm, National Gallery of Victoria, Melbourne. Wikimedia Commons.

Twenty years later, Jules Bastien-Lepage painted what’s now sometimes known as October or Potato Gatherers (1878), but was originally shown as October: Potato Gatherers.

vangoghpotatoharvest
Vincent van Gogh (1853–1890), Woman Lifting Potatoes (1885), oil on canvas, dimensions not known, Van Gogh Museum, Amsterdam. Wikimedia Commons.

Vincent van Gogh painted Woman Lifting Potatoes (1885) and similar scenes during his time living with his parents in Nuenen, in North Brabant, the Netherlands.

molnarpotatoharvest
János Pentelei Molnár (1878–1924), The Potato Harvest (1901), oil on canvas, 79 x 109 cm, Private collection. Wikimedia Commons.

János Pentelei Molnár’s The Potato Harvest from 1901 takes this theme into the early years of the twentieth century, in Hungary.

ringmandiggingpotatoes
Laurits Andersen Ring (1854–1933), A Man Digging Potatoes (1901), oil on canvas, 86 x 67.5 cm, Statsministeriet, Copenhagen, Denmark. Wikimedia Commons.

Laurits Andersen Ring’s A Man Digging Potatoes from 1901 is pure social realism, as this smallholder uses his spade to lift the potatoes that are his staple diet. I believe that the plants shown here have suffered from potato blight, which also swept Europe periodically causing more famine and death.

fredericthreesisters
Léon Frédéric (1856–1940), The Three Sisters (1896), oil on canvas, dimensions not known, Private collection. Wikimedia Commons.

Although the girls shown in Léon Frédéric’s Three Sisters from 1896 are clean and well dressed, they’re sat together peeling their staple diet of potatoes in a plain and barren farmhouse. A glimpse of the shoes of the girl at the left confirms that they’re neither destitute nor affluent.

vangoghpotatoeaters
Vincent van Gogh (1853–1890), The Potato Eaters (1885), oil on canvas, 82 × 114 cm, Van Gogh Museum, Amsterdam, The Netherlands. Wikimedia Commons.

Vincent van Gogh’s The Potato Eaters (1885) is another revealing insight into the lives of poor labourers in Nuenen, who are about to feast on a large dish of potatoes under the light of an oil lamp.

friantfrugalmeal
Émile Friant (1863–1932), The Frugal Meal (1894), oil, dimensions and location not known. Wikimedia Commons.

Émile Friant’s The Frugal Meal (1894) continues the social theme, as a poor family with four daughters sits down to a meal consisting of a bowl piled high with potatoes, and nothing else. More worryingly, the pot on the floor at the left is empty.

Further reading

Christopher Shepherd’s brilliant essay on the history of the potato in The Oxford Handbook of Agricultural History (2024), ed. Jeannie Whayne, Oxford UP, ISBN 978 0 19 092416 4.

Why you need to make archives, and how to

By: hoakley
1 November 2024 at 15:30

We back up to ensure that we can recover files, whole volumes, our complete Mac if needed. When that crucial document you were working on earlier has vanished, or becomes damaged, or disaster strikes a disk, backups are essential. But how do you preserve all those documents that used to come on paper, records, correspondence and certificates? How will you or your successors be able to retrieve them in ten or thirty years time? This brief article considers how you should archive them safely, which isn’t the same as backing them up.

By archiving, I mean putting precious files somewhere they can be retrieved in at least ten years time. They may include financial, business, employment and personal records, as well as all finished work that you want to record for posterity. For most, they’ll also include a careful selection of still images, movies, and the more important documents you might create, such as books, theses and papers. They’re what you and the law want you to keep in perpetuity, and to be able to retrieve even after you’re gone.

To see how this can be achieved, I consider: the storage medium to be used, file formats that will be retrievable, how to index them for access, physical storage conditions, and the checks of their integrity that are needed.

Storage medium

While backups are most likely to be kept on hard disks or SSDs, neither of those is in the least suitable for archives, as they have relatively short lifetimes and are too sensitive to storage conditions. Instead, you need a removable medium, today probably Blu-ray disks intended for archival use, such as M-DISC.

For those with copious archives of importance beyond their family, Sony used to offer Optical Disk Archive systems, but those products were discontinued last year and don’t appear to have a suitable replacement. This illustrates one of the problems with planning for the more distant future: today’s technology can all too easily become orphaned.

Businesses are increasingly turning to cloud services to store their archives, but for the great majority of us the recurring cost makes this impractical. In any case, best practice should be to use cloud services as a supplement to a physical archive. iCloud is more affordable for the storage of most important documents, but requires a Legacy Contact to be appointed.

File formats

While it’s fine to archive documents in their original format, as you do in your backups, it’s also important to extract their contents into more permanent formats. Among those most likely to prove durable for the next 50-100 years are:

  • UTF-8 (and formerly ASCII) for text files,
  • JPEG and PNG for still images,
  • audio, video and rich media using one of the widely-used compression standards and file formats,
  • XML-based open document standards,
  • CSV for data,
  • PDF provided that it complies with one of the archival standards PDF/A-1 to /A-4.

You may find it worthwhile tarring together large collections of smaller files, but don’t use an unusual compression or ‘archive’ format, which might prove inaccessible in the future.

Indexing and access

For larger collections, even when structured carefully, a thorough list of contents in UTF-8 text format is essential. While there are index and search tools that could help, in this respect too archives are different from backups. If you’re going to be gathering TB of files, look at some of the commercial solutions. Although some are free to use, like the long-established Greenstone, they aren’t intended for casual users and might prove demanding.

Physical storage conditions

Never print on the disk itself, which can result in its degradation, and keep paper records alongside disks in the same container, but not inside the cases themselves, where they could damage them.

Archive optical disks should be stored in cases with centre hub security, not in sleeves. They must be kept in a cool, dry and dark container, in which there is no mould or fungus. They also need to be protected from physical threats such as flood and fire. Firesafes are popular furniture for this, but you must then ensure that their combination or keys are readily available and not separated from the safe.

There used to be a vogue for commercial data repositories, often underground storage sites that had been repurposed. Not only were those expensive, but many failed to take the care that they promised, and plenty went bankrupt and put their contents at risk. If you can arrange it, store one copy with you, and another at a friend’s or relative’s at least a few miles away.

Integrity checks

If you’re serious about maintaining your archives, some form of integrity checking, such as that provided by my free utilities Dintch, Fintch and cintch, is essential. Check a sample on each disk once a year, to ensure that none has started to deteriorate. If you do detect errors, that’s the time to burn a replacement before the original is lost to decay.

Conclusion

Backups are for recovery, while archives are for posterity. Start building your archives now, and keep them safe for the future.

Further reading

How to burn a Blu-ray disc in Monterey
Wikipedia point of entry

Postscript

Some of you are reporting widespread claims that some Blu-ray burners no longer work in Sequoia. I have therefore repeated the process that I described in Monterey, using exactly the same Pioneer burner connected to a Mac Studio M1 Max running macOS 15.1. I’m delighted to report that it still works perfectly, and I see no reason that any other recent Pioneer optical drive should prove incompatible. All you need to do is follow the instructions.

Happy archiving!

The Real Country: Hay

By: hoakley
30 October 2024 at 20:30

In the more northerly latitudes, grass that’s essential for cattle to graze grows little during the winter months. Farmers keeping cattle therefore have to provide alternative feed for their livestock for several months each year. This can include root crops such as brassica varieties including turnips and swedes (also known as rutabaga), but the most widespread is cut and dried grass as hay.

Where climate and day-length are suitable, as in much of England and France, dedicated hay meadows can provide two harvests each year. Left ungrazed through the winter, the first is normally ready to mow in the late Spring, and when there’s sufficient rainfall during the early summer, a second hay harvest can be obtained before the weather deteriorates in the early autumn. The mowing of hay has also been known as math, and mowing a second time is thus the aftermath or lattermath.

The essential requirement for hay is that it’s dried thoroughly, or it will rot over time and become unusable as fodder. In the centuries before mechanisation during the nineteenth century, this process was described as: first mow the grass, “scatter it about, gather it in windrows, cock it overnight, scatter it about, windrow it, cock it, and so on to the stack and stack it”. (Fussell) Those steps are shown well in paintings.

bruegelhayharvest
Pieter Bruegel the Elder (c 1525–1569), The Hay Harvest (1565), oil on panel, 114 x 158 cm, Lobkowicz Palace, Prague, Czechia. Wikimedia Commons.

The companion to Pieter Bruegel the Elder’s painting of the grain harvest, The Hay Harvest from 1565 shows all stages in progress. In the left foreground a man is beating the blade on his scythe to sharpen it ready for mowing. Three women are striding towards him with the rakes they use to scatter and gather the mown hay. Behind them, in the valley, others are gathering the hay into small stacks or cocks, where it continues to dry before being loaded onto the hay wagon to be taken back to the farm.

At the right are wicker baskets containing other crops, including what appear to be peas or beans, together with a red fruit.

hodlermower1898
Ferdinand Hodler (1853–1918), The Mower (c 1898), oil on canvas, 71.5 × 114 cm, Private collection. Wikimedia Commons.

Ferdinand Hodler’s marvellous Mower from about 1898 is seen sharpening the blade on his heavy scythe using a whetstone, as the sun rises behind and to the left.

blepagehaymaking
Jules Bastien-Lepage (1848–1884), Les Foins (Haymakers) (1877), oil on canvas, 160 x 195 cm, Musée d’Orsay, Paris. Wikimedia Commons.

The couple in Jules Bastien-Lepage’s Haymakers from 1877 are enjoying a short break from their labours, with the mown hay behind them still scattered to dry, before it can be raked into cocks.

Capitole Toulouse - Salle Henri-Martin - L'été par Henri Martin
Henri-Jean Guillaume Martin (1860–1943), Summer, or Mowers (1903), oil on canvas, dimensions not known, Capitole de Toulouse, Toulouse, France. Image by Didier Descouens, via Wikimedia Commons.

Henri-Jean Martin painted Summer, or Mowers in 1903, as mechanisation was spreading across Europe. Several small clusters of men are mowing the hay in this meadow with their scythes, as three young women are dancing in a ring on the bed of flowers, and another sits nursing an infant.

moretfenaisonbretagne
Henry Moret (1856–1913), Haymaking in Brittany (1906), oil on canvas, dimensions not known, Musée des beaux-arts de Vannes, Vannes, France. Wikimedia Commons.

Henry Moret’s Haymaking in Brittany from 1906 shows a smaller team busy mowing and raking on steeper ground.

pissarrohaymaking
Camille Pissarro (1830–1903), Haymaking, Éragny (1887), oil on canvas, 50 x 66 cm, Van Gogh Museum, Amsterdam, The Netherlands. Wikimedia Commons.

In Camille Pissarro’s Divisionist painting of Haymaking, Éragny from the summer of 1887, a team of women are raking the cocks into haystacks.

pymonenkohaymaking
Mykola Pymonenko (1862–1912), Haymaking (date not known), oil on canvas, dimensions not known, Fine Arts Museum Kharkiv Харківський художній музей, Kharkiv, Ukraine. Wikimedia Commons.

Women in this hay meadow in Ukraine are raking in the harvest to be transported by a hay wain drawn by a pair of oxen, as painted in Mykola Pymonenko’s undated Haymaking.

millethaystacksautumn
Jean-François Millet (1814–1875), Haystacks: Autumn (c 1874), oil on canvas, 85.1 x 110.2 cm, The Metropolitan Museum of Art, New York, NY. Wikimedia Commons.

In Jean-François Millet’s Haystacks: Autumn from about 1874, the harvest has been gathered, and three huge haystacks dominate the canvas. At the foot of one of them, a shepherd leans on his staff, resting from his labours as his flock grazes on the stubble.

Surplus hay was also a good cash crop for those who could get it transported to towns and cities. Along the east coast of England, barges were filled with hay then taken to London for sale. Much of the land in the county of Middlesex, to the west of London, was devoted to producing hay to feed horses in the city.

bevanhaycarts
Robert Bevan (1865–1925), Hay Carts, Cumberland Market (1915), oil on canvas, 47.9 x 61 cm, Yale Center for British Art, New Haven, CT. Wikimedia Commons.

Robert Bevan’s painting of Hay Carts, Cumberland Market from 1915 is a view of London’s last hay market, near to the artist’s studio. By this time, the bales shown were made by mechanical baling machines and brought to London by barge.

In the next article in this series, I’ll look at a novel crop that soon became the staple food for many, the potato.

Interiors by design: The Dutch Golden Age

By: hoakley
25 October 2024 at 19:30

Painting in the Dutch Golden Age underwent remarkable evolution. In the fifty years between the 1620s and the French invasion of the Dutch Republic in 1672, established genres grew novel sub-genres, with artists specialising in each. These included ‘genre’ scenes of everyday life, with artists devoted to painting taverns, women working, festivities, markets, or domestic interiors. The latter appear to have been among the first such depictions in modern art.

During the 1650s, interiors started to become distinct from other scenes of everyday life, as the significance of their figures diminished, although few if any dispensed with them altogether.

Gerard ter Borch (1617–1681) specialised in domestic interiors, some containing open-ended narratives to encourage the viewer to speculate on their resolution. Two centuries later, those were to be become popular again, particularly in Britain, where they were known as problem pictures and featured in the press.

terborchmessenger
Gerard ter Borch (1617–1681), The Messenger (Unwelcome News) (1653), oil on panel, 66.7 x 59.5 cm, Koninklijk Kabinet van Schilderijen Mauritshuis, The Hague, The Netherlands. Wikimedia Commons.

Ter Borch’s The Messenger, usually know as Unwelcome News, from 1653, develops his favourite theme of the arrival of a message. The young man at the left is still booted and spurred from riding to deliver a message to this couple. Slung over his shoulder is a trumpet, to announce his arrival and assert his importance. The recipient wears a shiny breastplate and riding boots, and is taken aback at the news the messenger brings. His wife leans on her husband’s thigh, her face serious.

The scene is the front room of a house in the Golden Age. Behind them is a traditional bed typical of living areas at the time, with some of their possessions resting on a table between the couple and their bed. Hanging up on a bedpost is the husband’s sword, and behind them are a gun and powder horn. Is this letter news of his recall to military service, perhaps? Will he soon have to ride away from his wife, leaving her alone to bring up their family?

Although the three figures take the limelight, ter Borch picks out the mundane details of the room behind them instead of letting them fade into darkness.

terborch3figuresconversing
Gerard ter Borch (1617–1681), Three Figures Conversing in an Interior (Paternal Admonition) (c 1653-55), oil on canvas, 71 x 73 cm, Rijksmuseum Amsterdam, Amsterdam, The Netherlands. Wikimedia Commons.

Three Figures Conversing in an Interior is another of ter Borch’s narrative genre works, and more popularly known as Paternal Admonition (c 1653-55). Standing with her back to us, wearing a plush going-out dress, is the daughter. To her left is a table, on which there’s a small reading stand with books, almost certainly including a Bible.

Her parents are young, and they too are fashionably dressed. Her mother appears to be drinking from a glass, but her father is at the very least cautioning his daughter, if not giving her a thorough dressing-down. He wears a sword at his side. Behind them is a large bed, and to the right the family dog looks on from the darkness.

terborchwomanwriting
Gerard ter Borch (1617–1681), Woman Writing a Letter (c 1655), oil on panel, 39 x 29.5 cm, Koninklijk Kabinet van Schilderijen Mauritshuis, The Hague, The Netherlands. Wikimedia Commons.

Ter Borch’s half-sister Gesina appears to have been his model for Woman Writing a Letter (c 1655), which makes obvious his connection with Vermeer. Move this woman to a desk lit through windows at the left, light her surroundings, and you have a painting similar to Vermeer’s interiors. This shows a heavy decorated table cover pushed back to make room for the quill, ink-pot, and letter. Behind the woman is her bed, surrounded by heavy drapery, and at the lower right is the brilliant red flash of the seat.

terborchletter
Gerard ter Borch (1617–1681), The Letter (c 1660-62), oil on canvas, 79 x 68 cm, The Royal Collection, London. Wikimedia Commons.

The Letter from about 1660-62 returns to ter Borch’s favourite theme of the reading and writing of letters. Two young women are working together, apparently replying to the letter being read by the woman on the right. A boy, perhaps their younger brother, has just brought in a tray bearing an ornate pitcher of drink. In front, a small dog is curled up asleep on a stool. Above them is an unlit chandelier suspended from a hanging ceiling.

metsuwasherwoman
Gabriël Metsu (1629–1667), Washerwoman (c 1650), oil on panel, 23.9 × 21 cm, Muzeum Narodowe w Warszawie, Warsaw, Poland. Wikimedia Commons.

Other specialists in genre painting like Gabriël Metsu also ventured towards interiors. His Washerwoman from about 1650 looks authentic and almost socially realist: the young woman is a servant, dressed in her working clothes, in the dark and dingy lower levels of the house. She looks tired, her eyes staring blankly at the viewer. She’s surrounded by the gear she uses, including a rope and pitcher to the right, and an earthenware bowl on display below it. The mantlepiece in the background features a blue and white plate.

dehoochwomandrinkingtwomen
Pieter de Hooch (1629–after 1684), A Woman Drinking with Two Men (c 1658), oil on canvas, 73.7 x 64.6 cm, National Gallery, London. Wikimedia Commons.

It’s easy to mistake Pieter de Hooch’s A Woman Drinking with Two Men from about 1658 for a Vermeer, and like the latter he decorates the far wall with a contemporary map. The Eighty Years’ War had not long ended, and the Dutch Republic was flourishing. Discarded objects are scattered on its black-and-white tiled floor. There’s a large and empty fireplace, and above it hangs a religious painting.

It was Jan Vermeer whose few surviving works explored interiors the most.

vermeermilkmaid
Johannes Vermeer (1632–1675), The Milkmaid (c 1660), oil on canvas, 45.5 x 41 cm, The Rijksmuseum Amsterdam, Amsterdam, The Netherlands. Wikimedia Commons.

In his Milkmaid from about 1660, a woman servant is pouring milk from a jug, beside a tabletop with bread. In the left foreground the bread and pots rest on a folded Dutch octagonal table, covered with a mid-blue cloth. A wicker basket of bread is nearest the viewer, broken and smaller pieces of different types of bread behind and towards the woman, in the centre. Behind the bread is a dark blue studded mug with pewter lid, and just in front of the woman (to the right of the mug) a brown earthenware ‘Dutch oven’ pot into which the milk is being poured. An ultramarine blue cloth (matching the woman’s apron) rests at the edge of the table. There are many other intriguing details in this interior.

vermeeryoungwomanwaterpitcher
Johannes Vermeer (1632–1675), A Young Woman with a Water Pitcher (c 1662-5), oil on canvas, 45.7 x 40.6 cm, The Metropolitan Museum of Art, New York, NY. Wikimedia Commons.

Vermeer’s Young Woman with a Water Pitcher from slightly later is another fine example, this time with more obvious control of focus effects for which his paintings are renowned. Details in this interior include the ornate tablecloth, a small lockable chest on the right of the table, the map hanging behind, and the window she is holding with her right hand.

Interiors then vanished from painting until their rebirth in the nineteenth century.

Disk Images: Bands, Compaction and Space Efficiency

By: hoakley
21 October 2024 at 14:30

Of the two most dependable types of disk image, read-write (UDRW) disk images are the simpler. The only options you can set are its internal file system, whether the container is encrypted, and its maximum capacity. From there on, macOS will automatically Trim the disk image when it’s mounted, and adjust the size it takes on disk accordingly.

Sparse bundles can be more complex to configure in the first place, and may need routine maintenance to ensure their size on disk doesn’t grow steadily with use. This article considers how significant those complications are.

Band size

Most if not all storage divides data into blocks; in the case of SSDs, the default block size in APFS is 4,096 bytes, and that’s effectively the unit of storage in the single file of a read-write disk image.

Its equivalent in a sparse bundle is the maximum size that each of its band files can reach, normally set to 8.4 MB in macOS Sequoia and its predecessors, and equivalent to slightly more than 2,000 SSD blocks. Smaller band sizes are more efficient in the space required to store the contents of a sparse bundle, but require more band files for the same quantity of data stored. Experience with older versions of macOS suggests that it’s not a good idea for the total number of band files to exceed 100,000, but I’m not aware of any recent evidence to support that.

When creating very large sparse bundles, of the order of TB, macOS now may adjust the band size to limit their number. Some users report that those sparse bundles perform better as a result, for instance when being used to store backups on a network. There don’t appear to be any advantages to using band sizes less than 8.4 MB.

Neither DropDMG nor Disk Utility currently appear to allow any control over band size; Spundle and hdiutil not only let you specify custom band sizes when creating sparse bundles, but also allow you to change band size by copying an existing sparse bundle into a new one, which could be useful if you were changing the maximum size of a sparse bundle.

Compaction

Each time a writeable APFS or HFS+ file system is mounted in a disk image, including both sparse bundle and read-write types, storage blocks that are no longer in use should be returned for reuse by the process of Trimming. This is automatic, and in the case of single-file read-write disk images, is the only maintenance that occurs.

Because sparse bundles add another layer of band files over SSD storage, they too require periodic maintenance in the process of compaction. To compact a sparse bundle, macOS scans the band files and removes those that are no longer being used by the file system in the sparse bundle. This only applies to sparse bundles with APFS or HFS+ file systems, though, and isn’t guaranteed to free up any space. One small catch is that, by default, compaction won’t take place on a notebook running on battery power; hdiutil and Spundle provide an option to override that.

Space efficiency in use

I’ve been unable to find any objective comparison between space efficiency of modern read-write disk images and sparse bundles, so have performed my own on a USB4 external SSD connected to my Mac Studio M1 Max. I created two test cases of 125 GB images on a freshly-formatted APFS volume, one a sparse bundle with the default 8.4 MB band size, the other a read-write disk image (UDRW). In macOS Sequoia 15.0.1, the initial size they took on disk was 14.8 MB for the sparse bundle, and 336 MB for the disk image.

Using my utility Stibium, I then wrote and deleted a series of files in each, as follows:

  1. 160 files of 2 MB to 2 GB written totalling 53 GB
  2. 40 largest of those, sizes 600 MB to 2 GB, were then deleted, leaving 120
  3. another 160 files written totalling 53 GB
  4. 40 largest of those deleted, leaving a total of 240
  5. another 160 files written totalling 53 GB
  6. 40 largest of those deleted, leaving a total of 360
  7. another 160 files written totalling 53 GB
  8. 40 largest of those deleted, leaving a total of 480
  9. another 160 files written totalling 53 GB, leaving a total of 640 files.

At that point, the sparse bundle had 104.08 GB stored in 12,408 band files, and the read-write disk image had 91.63 GB stored in its single file. I then deleted all files and folders in each of the images, and unmounted and remounted each image several times to ensure Trimming had completed. The sparse bundle then had 1.12 GB in 135 band files, and offered 124.66 GB of free space when mounted; the read-write disk image was slightly smaller at 937.9 MB on disk and offered exactly the same amount of free space when mounted.

Compacting the empty sparse bundle was reported as reclaiming 52.3 MB, and reduced the disk space taken by the band files slightly to 1.06 GB while retaining the same number of bands, and still offering 124.66 GB of free space when mounted.

In this case, following writing 800 files totalling over 250 GB, and deleting a total of 160 of them, compaction made a remarkably small difference to the free space returned following removal of all files. There is also very little difference in the space efficiency of sparse bundles and modern read-write disk images.

One consistent difference observed throughout was in write speeds: they remained constant at 3.2 GB/s for the sparse bundle, and 1.0 GB/s for the read-write disk image, as reported here previously.

Conclusions

  • Sparse bundles are more complicated than read-write disk images (UDRW), with band size to be set, and compaction to be performed.
  • Default band size appears to work well, and manually setting band size should seldom be necessary.
  • Both types appear highly efficient in their use of disk space, with only small differences between them.
  • Although it might be important to compact sparse bundles in some cases, the amount of free space returned by compaction is unlikely to be significant in many circumstances.

Previous articles

Introduction
Tools
How read-write disk images have gone sparse
Performance

Disk Images: Performance

By: hoakley
16 October 2024 at 14:30

Over the last few years, the performance of disk images has been keenly debated. In some cases, writing to a disk image proceeds at a snail’s pace, but this has appeared unpredictable. Over two years ago, I reported sometimes dismal write performance to disk images, summarised in the table below.

diskimages13

This article presents new results from tests performed using macOS 15.0.1 Sequoia, which should give a clearer picture of what performance to expect now.

Methods

Previous work highlighted discrepancies depending on how tests were performed, whether on freshly made disk images, or on those that had already been mounted and written to. The following protocol was used throughout these new tests:

  1. A 100 GB APFS disk image was created using Disk Utility, which automatically mounts the disk image on completion.
  2. A single folder was created on the mounted disk image, then it was unmounted.
  3. After a few seconds, the disk image was mounted again by double-clicking it in the Finder, and was left mounted for at least 10 seconds before performing any tests. That should ensure read-write disk images are converted into sparse file format, and allows time for Trimming.
  4. My utility Stibium then wrote 160 test files ranging in size from 2 MB to 2 GB in randomised order, a total of just over 53 GB, to the test folder.
  5. Stibium then read those files back, in the same randomised order.
  6. The disk image was then unmounted, its size checked, and it was trashed.

All tests were performed on a Mac Studio M1 Max, using its 2 TB internal SSD, and an external Samsung 980 Pro 2 TB SSD in an OWC Express 1M2 enclosure running in USB4 mode.

Results

These are summarised in the table below.

xferdiskimage24

Read speeds for sparse bundles and read-write disk images were high, whether the container was encrypted or not. On the internal SSD, encryption resulted in read speeds about 1 GB/s lower than those for unencrypted tests, but differences for the external SSD were small and not consistent.

Write speeds were only high for sparse bundles, where they showed similar effects with encryption. Read-write disk images showed write speeds consistently about 1 GB/s, whether on the internal or external SSD, and regardless of encryption.

When unencrypted, read and write speeds for sparse (disk) images were also slower. Although faster than read-write disk images when writing to the internal SSD, read speed was around 2.2 GB/s for both. Results for encrypted sparse images were by far the worst of all the tests performed, and ranged between 0.08 to 0.5 GB/s.

Surprisingly good results were obtained from a new-style virtual machine with FileVault enabled in its disk image. Although previous tests had found read and write speeds of 4.4 and 0.7 GB/s respectively, the Sequoia VM achieved 5.9 and 4.5 GB/s.

Which disk image?

On grounds of performance only, the fastest and most consistent type of disk image is a sparse bundle (UDSB). Although on fast internal SSDs there is a reduction in read and write speeds of about 1 GB/s when encrypted using 256-bit AES, no such reduction should be seen on fast external SSDs.

On read speed alone, a read-write disk image is slightly faster than a sparse bundle, but its write speed is limited to 1 GB/s. For disk images that are more frequently read from than written to, read-write disk images should be almost as performant as sparse bundles.

However, sparse (disk) images delivered weakest performance, being particularly slow when encrypted. Compared with previous results from 2022, unencrypted write performance has improved, from 0.9 to 2.0 GB/s, but their use still can’t be recommended.

Performance range

It’s hard to explain how three different types of disk image can differ so widely in their performance. Using the same container encryption, write speeds ranged from 0.08 to 3.2 GB/s, for a sparse image and sparse bundle, on an external SSD with a native write speed of 3.2 GB/s. It’s almost as if sparse images are being deprecated by neglect.

Currently, excellent performance is also delivered by FileVault images used by Apple’s lightweight virtualisation on Apple silicon. The contrast is great between its 4.5 GB/s write speed and that of an encrypted sparse image at 0.1 GB/s, a factor of 45 when running on identical hardware.

Recommendations

  • For general use, sparse bundles (UDSB) are to be preferred for their consistently good read and write performance, whether encrypted or unencrypted.
  • When good write speed is less important, read-write disk images (UDRW) can be used, although their write performance is comparable to that of USB 3.2 Gen 2 at 10 Gb/s and no faster.
  • Sparse (disk) images (UDSP) are to be avoided, particularly when encrypted, as they’re likely to give disappointing performance.
  • Encrypting UDSB or UDRW disk images adds little if any performance overhead, and should be used whenever needed.

Previous articles

Introduction
Tools
How read-write disk images have gone sparse

Disk Images: How read-write disk images have gone sparse

By: hoakley
14 October 2024 at 14:30

Until about three years ago, most types of disk image had fixed size. When you created a read-write disk image (UDRW) of 10 GB, it occupied the same 10 GB on disk whether it was full or empty. The only two that could grow and shrink in size were sparse bundles and sparse disk images, with the former generally preferred for its better resizing and performance.

This changed silently in Monterey, since when read-write disk images have been automatically resized by macOS, and saved in APFS sparse file format. As this remains undocumented, this article explains how this works, and how and where you can use it to your advantage.

Sparse

In this context, the word sparse refers to two very different properties:

  • Sparse bundles and sparse (disk) images are types of disk image that can change in size, and can be stored on a wide range of file systems and storage, including on NAS and other networked storage.
  • Sparse files are stored in a highly efficient file format available in modern file systems, where only the data in a file is stored, and empty space within that file doesn’t waste storage space. It’s available in APFS, but not in HFS+.

Requirements

For read-write disk images to be sparse, the following are required:

  • The disk image must be saved to, and remain on, an APFS volume.
  • The file system within the disk image can be either APFS or HFS+, but not FAT or ExFAT.
  • The disk image must be created and unmounted first. For that initial mount, the disk image isn’t a sparse file, so occupies its full size on disk.
  • Whenever that disk image is mounted again, and has sufficient free space within its set limit, it will be saved in sparse file format, and occupy less than its full size on disk.

Demonstration

  1. Create a new read-write disk image of at least 1 GB size with an internal file system of APFS on an APFS volume, using DropDMG, Disk Utility, or an alternative.
  2. Once it has been created and mounted, unmount it in the Finder.
  3. Select the disk image in the Finder and open the Get Info dialog for it. Confirm that its size on disk is the same as that set.
  4. (Optional) Open the disk image using Precize, and confirm that it’s not a sparse file.
  5. Double-click the disk image to mount it in the Finder, and wait at least 10 seconds before unmounting it.
  6. Select the disk image in the Finder and open the Get Info dialog for it. Confirm that its size on disk is now significantly less than that set.
  7. (Optional) Open the disk image using Precize, and confirm that it has now become a sparse file.

How it works

When you create the disk image, macOS creates and attaches its container, and creates and mounts the file system within that. This is then saved to disk as a regular file occupying the full size of the disk image, plus the overhead incurred by the disk image container itself. No sparse files are involved at this stage.

When that disk image is mounted next, its container is attached through diskarbitrationd, then its file system is mounted. If that’s APFS (or HFS+), it undergoes Trimming, as with other mounts. That coalesces free storage blocks within the image to form one contiguous free space. The disk image is then saved in APFS sparse file format, skipping that contiguous free space. When the file system has been unmounted and the container detached, the space used to store the disk image has shrunk to the space actually used within the disk image, plus the container overhead. Unless the disk image is almost full, the amount of space required to store it on disk will be smaller than the full size of the disk image.

This is summarised in the diagram below.

SparseDiskImage1

The size of read-write disk images is therefore variable depending on the contents, the effectiveness of Trimming in coalescing free space, and the efficiency of APFS sparse file format.

Conversion to sparse file

When mounting an APFS file system in a read-write disk image, APFS tests whether the container backing store is a sparse format, or a flat file. In the case of a newly created read-write disk image that hasn’t yet been converted into a sparse file, that’s detected prior to Spaceman (the APFS Space Manager) scanning for free blocks within its file system. When free blocks are found, APFS sets the type of backing store to sparse, gathers the sparse bytes and ‘punches a hole’ in the disk image’s file extents to convert the container file into sparse format. That appears in the log as:
handle_apfs_set_backingstore:6172: disk5s1 Set backing store as sparse
handle_apfs_set_backingstore:6205: disk5 Backing storage is a raw file
_punch_hole_cb:37665: disk3s5 Accumulated 4294967296 sparse bytes for inode 30473932 in transaction 3246918, pausing hole punching

where disk5s1 is the disk image’s mounted volume, and disk3s5 is the volume in which the disk image container is stored.

Trimming for efficient use of space

That conversion to sparse format is normally only performed once, but from then on, each time that disk image is mounted it’s recognised as having a sparse backup store, and Spaceman performs a Trim to coalesce free blocks and optimise on-disk storage requirements. For an empty read-write disk image of 2,390,202 blocks of 4,096 bytes each, as created in a 10 GB disk image, log entries are:
spaceman_scan_free_blocks:4106: disk5 scan took 0.000722 s, trims took 0.000643 s
spaceman_scan_free_blocks:4110: disk5 2382929 blocks free in 7 extents, avg 340418.42
spaceman_scan_free_blocks:4119: disk5 2382929 blocks trimmed in 7 extents (91 us/trim, 10886 trims/s)
spaceman_scan_free_blocks:4122: disk5 trim distribution 1:0 2+:0 4+:0 16+:0 64+:0 256+:7

accounting for a total of 9.8 GB.

Changes made to the contents of the disk image lead to a gradual reduction in Trim yield. For example, after adding files to the disk image and deleting them, instead of yielding the full 9.8 GB, only 2,319,074 blocks remain free, yielding a total of 9.5 GB.

For comparison, initial Trimming on a matching empty sparse bundle yields the same 9.8 GB. After file copying and deletion, and compaction of the sparse bundle, Trimming performs slightly better, yielding 2,382,929 free blocks for a total of 9.6 GB. Note that Trimming of sparse bundles is performed by APFS Spaceman separately from management of bands in backing storage, which isn’t a function of the file system.

Size efficiency

Although read-write disk images stored as sparse files are efficient in their use of disk space, they’re still not as compact as sparse bundles. For an empty 10 GB image, the read-write type requires 240 MB on disk, but a sparse bundle only needs 13.9 MB. After light use storing files, then deleting the whole contents, a 10 GB read-write disk image grows to occupy 501 MB, but following compaction a sparse bundle only takes 150 MB. That difference may not remain consistent over more prolonged use, though, and ultimately compacting sparse bundles may cease freeing any space at all.

It’s also important to remember that sparse bundles need to be compacted periodically, if any of their contents are deleted, or they may not reduce in size after deletions. Read-write disk images can’t be compacted, and reclaim disk space automatically.

Benefits and penalties

Read-write disk images saved as sparse files are different from sparse bundles in many ways. Like any sparse file, the disk image still has the same nominal size as its full size, and differs in the space taken on disk. Sparse bundles should normally only have band files sufficient to accommodate their current size, so their nominal size remains similar to the space they take on disk. The result is that, while read-write disk images in sparse file format will help increase free disk space, their major benefit is in reducing ‘wear’ in SSDs by not wasting erase-write cycles storing empty data.

Unlike the band file structure in sparse bundles, which can be stored on almost any disk, APFS sparse files have to be treated carefully if they are to remain compact. Moving them to another file system or over a network is likely to result in their being exploded to full size, and I have explained those limitations recently.

Both read-write disk images and sparse bundles deliver good read performance, but write performance is significantly impaired in read-write disk images but not sparse files. Encryption of disk images and sparse bundles also has significant effects on their performance, and in some cases write performance is badly affected by encryption. I have previously documented their performance in macOS Monterey, and will be updating those figures for Sequoia shortly.

Summary

  • Read-write disk images saved on APFS storage in Monterey and later are no longer of fixed size, and should use significantly less disk space unless full, provided the disk image has been unmounted at least once since creation, and the file system in the disk image is APFS or HFS+.
  • This is because macOS now saves the disk image in sparse file format.
  • To retain the disk image in sparse file format, it needs to remain in APFS volumes, and normal precautions are required to maintain its efficient use of space. The disk image can’t be manually compacted, in the way that sparse bundles require to be.
  • The main benefit of this strategy is to minimise erase-write cycles, so reducing ‘wear’ on SSDs.
  • Sparse bundles are still more efficient in their use of disk space, and have higher write speeds, but read-write disk images are now closer in their efficiency and performance.

Previous articles

Introduction
Tools

Disk Images: Tools

By: hoakley
9 October 2024 at 14:30

If you’re going to use disk images of any type, then getting the right tool for the job is essential. This article considers the leading candidates:

  • Disk Utility, bundled with macOS
  • DropDMG, $24.99 from C-Command, or from the App Store
  • Spundle, free from its Product Page here
  • hdiutil, the command tool bundled with macOS.

Although I’m sure there are a great many others, IMHO those should be at the top of your list.

Disk Utility

Create a new disk image using the New Image command in its File menu and there’s a basic range of choices on offer.

dmgdiskutil

This dialog has a longstanding bug, where it can reset the size you’ve entered if you change another setting, which can help you make mistakes. Otherwise, it gives limited access to some of the many options available, sufficient for the occasional and not too demanding user. Further options are available in its Images menu, including verification, adding checksums, conversions between types, and resizing. Notable by its absence is the ability to change the password of an encrypted disk image, which is unhelpfully deferred to the command line.

Documentation in Disk Utility’s Help book is also scant, and insufficient to serve as a reference. As Apple doesn’t provide any further technical information, apart from that in man hdiutil, you may find yourself searching websites such as this.

DropDMG

Since its release over 22 years ago, this has been the first choice for many who need to work with disk images, and is without doubt the best for those who distribute software in disk images. It has grown into the most comprehensive and capable utility for working with any type of disk image, and is backed up by a superb 123-page manual that goes a long way to filling the gap left by Apple. That manual is well-maintained, and contains links to recent technical articles and further information.

dmgdropdmg

DropDMG’s options for creating a new disk image far exceed those in Disk Utility. Particularly helpful are the compatible version hints shown on various options, to remind you of which file systems are available in different macOS versions, and which types of disk image container are supported. DropDMG will even convert old NDIF disk images last used in Mac OS 9 to more modern formats. It will also change the password of an encrypted disk image from a menu command.

For those who need to work with standard configurations, perhaps for software distribution, it lets you save and reuse them with ease. Those can include signing with developer certificates, product licences, background images, custom volume icons, and more. Whichever type of disk image you want to create or maintain, DropDMG should be your first choice.

Spundle

There are a few options for sparse bundles that even DropDMG doesn’t expose, such as control over band size, the ability to resize a sparse bundle, and to change its band size. If you want access to those, Spundle is a useful adjunct.

dmgspundle

Note that, unlike DropDMG, Spundle only works with sparse bundles.

hdiutil

Although this remains the definitive command tool that offers types of disk image and features you didn’t even know existed, it’s fiendishly complex to use, with a sprawling and overloaded grammar. Its man page runs to more than 11,000 words, but appears never to have been rewritten into any cohesive account of disk images, or command options. For example, change information is given in two sections, Compatibility and What’s New. Changes made in Catalina and later appear at the end of the Compatibility section, then the final What’s New section reverses time order and goes back from Catalina to Mac OS X 10.5.

I only recommend hdiutil for those who need to work with disk images in shell scripts, or for those few features that aren’t available in DropDMG or, for sparse bundles, in Spundle. It’s a command tool of last resort.

Previous article

Introduction

Disk Images: Introduction

By: hoakley
7 October 2024 at 14:30

A disk image is a file or a bundle containing what could otherwise be the contents of a disk. It’s a common way to store and move complete file systems in a neat package, for items that need to be separated from the physical storage provided by a drive. macOS uses disk images for some tasks of great importance, including:

  • Recovery and Hardware Diagnostics systems,
  • additions to macOS such as Safari, its supporting frameworks, and dyld caches, in cryptexes,
  • networked storage for Time Machine backups, in sparse bundles,
  • lightweight virtual machines on Apple silicon Macs.

You could use them to store encrypted data on unencrypted volumes, and they’re often used for delivering Apple and third-party software.

Disk images are poorly documented for both users and developers, and have changed significantly over the last few years. Articles in this series explain how to choose between different types of disk image, how to create and use them, and what to do when they go wrong.

Containers and file systems

Disk images consist of two distinct components: the file or bundle itself functioning as a container, and the file system contained inside it. When referred to in this context, disk image containers are completely unrelated to the sandbox containers found in ~/Library/Containers.

This distinction is important in several respects, although it isn’t apparent when you use disk images in the Finder. Preparing a disk image for access involves two separate functions: attaching its container, and mounting any file systems found inside it. When that’s performed by the Finder, perhaps by double-clicking the disk image, those two actions appear fused into one. Similarly, removing the disk image requires all its mounted file systems to be unmounted first, then the container is detached.

One feature that’s widely confused is the encrypted disk image. This involves encryption of the whole container, rather than using an encrypted file system within it. Now that Disk Utility no longer supports the creation of encrypted HFS+ volumes, one remaining alternative is to use an encrypted disk image containing an HFS+ volume.

If you want an analogy for disk images, attaching the container is like connecting an external disk, and once that has been performed, the file systems contained by that disk have to be mounted before you can access their contents.

Types

There are many different types of disk image in use, of which the two this series is most concerned with are plain UDIF read-write disk images (UDRW), and sparse bundles (UDSB). Others you may encounter include:

  • plain UDIF read-only (UDRO),
  • various compressed versions of UDRW,
  • CD/DVD master for export (UDTO),
  • sparse disk image (UDSP), a single file rather than a bundle.

Those specify the container format; within each, there’s the usual choice of file systems, although throughout these articles it will normally be assumed that APFS will be used unless otherwise specified.

The word sparse in sparse bundle and sparse disk image doesn’t refer to APFS sparse files, but to the fact that those types of disk image can grow and diminish in size, and normally try to occupy the minimum amount of disk space. This is an unfortunate name collision.

Structure

With the exception of sparse bundles, all disk images are contained within a single file of opaque structure.

Sparse bundles consist of a single bundle folder containing:

  • bands, a folder containing the contents of the disk image in band files
  • info.plist and its backup copy info.bckup, containing settings including band size
  • lock, an empty file
  • mapped, a folder containing small data files to match all of the band files except the first
  • token, an empty file.

Container size

Until this changed in Monterey (or thereabouts), non-sparse disk images had fixed container sizes. Create a UDIF read-write disk image (UDRW) of 10 GB, and the space occupied by it on disk was approximately 10 GB, whether it was empty or full. Although it remains undocumented, when stored on APFS volumes, UDRW disks can now take advantage of APFS sparse file format, and will normally only occupy the disk space required for the contents of their file system.

This is only true once the disk image has been mounted for the first time after it has been created, mounted and unmounted. To see this, create a test read-write disk image (for example, using Disk Utility) of 10 GB size. Then unmount it, and use the Finder’s Get Info command to inspect its size on disk. That will be 10 GB. Then mount the disk image again, pause a couple of seconds, unmount it, and Get Info will show its size on disk is now much smaller.

As I’ll explain in detail in a later article, this is because each time that disk image is mounted, if its internal file system is HFS+ or APFS, its contents will be trimmed, and saved to disk in sparse file format, which omits all its empty space. This only applies to read-write disk images when they’re stored in APFS file systems; copy them to HFS+ and they’ll explode to full size, as HFS+ doesn’t support the sparse file format.

Considering just the two leading types, empty sizes for a 100 GB disk image are:

  • a sparse bundle is 35.4 MB empty, 53.3 GB when containing about 51 GB files, stored across 6,359 band files.
  • a read-write disk image (UDRW) shrinks to 333.8 MB once stored as an empty sparse file, 53.6 GB when containing about 51 GB files, in a single file container.

Performance

Some types of disk image perform poorly, and can be very slow to write to. Recent versions of macOS have brought improvements, although some options such as encryption can still impair performance significantly. For the two leading types, when their container is stored on the internal SSD of an Apple silicon Mac, with native read and write speeds of around 6-7 GB/s:

  • an initially empty unencrypted sparse bundle reads at 5.1 GB/s, and writes at 4.8 GB/s.
  • an initially empty unencrypted read-write disk image (UDRW) reads at 5.3 GB/s, and writes at only 1 GB/s.

Tests were performed using my utility Stibium, across a range of 160 files of 2 MB to 2 GB size in randomised order, with macOS 15.0.1.

Key points

  • Disk images consist of a file or bundle containing one or more file systems; the container and its contents are distinct.
  • To access the contents of a disk image, the container is first attached, then the file system(s) within it are mounted. In the Finder, those two processes appear as a single action.
  • Encrypted disk images encrypt the container, and don’t necessarily contain encrypted file systems.
  • Disk images come in many different types, and can contain different file systems.
  • Sparse bundles have a file and folder structure inside their bundle folder, with their data saved in band files; all other disk images are single files.
  • Sparse bundles grow and shrink according to the size of files stored within them.
  • In recent macOS, and on APFS, read-write disk images (UDRW) are stored in APFS sparse file format, enabling them to grow and shrink as well.
  • In recent macOS, unencrypted sparse bundles have read and write performance close to that of the disk they’re stored on. Read-write disk images read at similar speeds, but write more slowly, at about 20% of read speed.

What performance should you get from different types of storage?

By: hoakley
10 September 2024 at 14:30

External storage is invariably sold with ‘up-to’ performance figures. In practice, you’ll seldom realise anything like some write or read speeds claimed. And when it comes to prolonged tasks like that first full Time Machine backup, no matter how fast you thought that drive would be, it always takes longer than expected.

Over the last few years I have tested and reviewed many examples of different types of external storage, from basic USB 3 hard drives, to the latest USB4 SSD enclosures, and NAS packed with fast SSDs. This article draws on all those test results to give you a better idea of what to expect when they’re being used with your Mac.

Results quoted here are typical for those tests performed mostly using a Mac Studio M1 Max, but unless otherwise indicated should be similar for recent Intel models. They’re summarised in this table.

storage1

Write speeds are given for:

  • the single 50 MB write test performed by Time Machine before each backup;
  • 500 multiple concurrent writes of 4 KB each, performed in those same Time Machine tests;
  • calculated net write speed over a first full backup to APFS of at least 400 GB;
  • general write speed measurement using my app Stibium, which gives broadly similar results to other leading benchmarking apps.

General read speeds are also obtained using Stibium, and similar to other apps. All speeds are given as MB/s for consistency.

Before looking at individual types of storage, one obvious and important result is the effect of throttling by macOS on Time Machine backup performance. Considering Time Machine’s own tests, writing a single 50 MB file is performed consistently at around 200-225 MB/s to local storage of whatever type, and multiple concurrent writes of 4 KB files reach around 20-23 MB/s regardless of local storage type. Those hold good even when you back up to a fast Thunderbolt 3 SSD, and backing up to a NAS is little quicker unless it’s over 2.5GbE to an NVMe SSD. Local transfer speeds only differ more substantially in general tests, when they aren’t throttled as they are in Time Machine.

Hard disks

When writing to or reading from a local hard disk, performance varies substantially according to which sectors on the hard disk are being accessed. This is a well-known phenomenon, and the result of geometry, as sectors are faster at the periphery of the disk’s platter, and slower in the inner part. Ranges given here take that into account: the lower figure is for inner sectors, and the higher for outer ones. Some users compensate for this effect, and only ever use the outer half of a disk’s sectors to obtain better performance, but that reduces their available capacity, and effectively doubles their cost per TB.

SSDs

SATA SSDs may be cheapest, but they’re also slowest, and with Macs they generally don’t enjoy Trim or SMART health indicator support. Of the two, Trim support is usually the more important, as without that, they can accumulate blocks waiting to be erased and returned for further use, and as a result their write (but not read) speed can fall as low as 100 MB/s. Unless used for largely static storage, this is a significant risk.

NVMe SSDs deliver twice the performance of SATA models, and generally enjoy Trim but not SMART indicator support. This makes them far better suited to general use, as their write speeds should be sustained from new throughout their working life.

USB 3.2 Gen 2, Thunderbolt 3, USB4

Translating commonly quoted transfer speeds for these three protocols into real-world speeds turns out to be complex. In practice, these are what you can expect to see:

  • USB 3.2 Gen 2 at 10 Gb/s is slightly less than 1 GB/s
  • Thunderbolt 3 at 32 Gb/s is up to 3 GB/s
  • USB4 at 40 Gb/s is up to 3.4 GB/s.

All recent models of Mac, both Intel and Apple silicon, should realise full performance over USB 3.2 Gen 2 and Thunderbolt 3, but support for USB4 is limited to Apple silicon. Unless a drive or enclosure specifically includes Thunderbolt 3 as a fallback, when connected to an Intel Mac, you should expect it to fall back to USB 3.2 Gen 2 at just under 1 GB/s, less than a third of the speed of USB4.

NAS

Although I haven’t made any systematic comparison between AFP and SMB network protocols, I can see no consistent difference in their performance, when used with the latest versions of macOS and NAS software. The latter, though, can be critical: older versions of NAS software can perform poorly when used over SMB with recent macOS. Keeping your NAS software up to date is important.

Throttling of Time Machine backup writing isn’t supposed to occur when backing up over a network, and there is some evidence here to support that, with significantly better results for 50 MB test files. However, those are only apparent when using NVMe SSDs in the NAS, with a wired Ethernet 2.5GbE connection to provide sufficient bandwidth.

Check TM performance

Provided that your Mac is running a recent version of macOS and backing up to APFS, it’s simple to read the two write performance tests that occur at the start of each Time Machine backup using my free T2M2. Alternatively, you can also read them using the Time Machine custom log extract in Mints. In T2M2 they should look something like:
Destination IO performance measured:
Wrote 1 50 MB file at 238.02 MB/s to "/Volumes/ThunderBay2" in 0.210 seconds
Concurrently wrote 500 4 KB files at 35.58 MB/s to "/Volumes/ThunderBay2" in 0.058 seconds

Check general performance

Although there are other apps that will do this, I developed Stibium for this purpose. Follow the ‘gold standard’ procedure detailed in its Help Reference to obtain the most accurate and reproducible results. Stibium can test any storage you can access in the Finder, including all local devices and networked systems such as NAS.

Further reading

Which external drives have Trim and SMART support?
How to evaluate an external SSD
You can read my reviews in MacFormat and MacLife magazines, available in the App Store.

Where has Safari gone, and why are macOS updates larger for Apple silicon?

By: hoakley
4 September 2024 at 14:30

My previous explanation of how recent versions of macOS merge their System and Data volumes into what appears to be a single volume, omitted a third component, including Safari. Look in the System/Applications folder where all the bundled apps are stored on the SSV, and there’s no Safari to be seen, yet it appears in the top-level Applications folder. This article explains how that now works using cryptexes, and how they differ between Intel and Apple silicon Macs.

Finding Safari

As the modern boot volume group evolved through Catalina to Big Sur, Safari and its supporting frameworks were stored in the Data volume. That stopped with the arrival of Ventura, and they’re now stored in the third components that complete the modern boot volume group. You can see when files are stored on a different volume using my free app Precize to reveal their full paths. Use that to examine three apps from the merged Applications folder, and you’ll understand what I mean:

  • Chess.app has a path of /System/Applications/Chess.app demonstrating that it’s one of the apps bundled in the SSV, where almost all of the System folder is stored.
  • Cirrus.app, like any other app you have installed, has a path of /Applications/Cirrus.app, making it clear that it’s stored on the writable Data volume.
  • Safari.app has the weird path of /System/Volumes/Preboot/Cryptexes/App/System/Applications/Safari.app that demands further explanation.

Note that the Finder’s Get Info dialogs aren’t as truthful, and don’t tell the full story.

Their volfs paths are also worth noting. On my Intel Mac, they are:

  • Chess.app is at /.vol/16777240/1152921500311883667; because all macOS 14.6.1 SSVs are identical, your Chess.app should have the same inode number too.
  • Cirrus.app is at /.vol/16777240/461665725
  • Safari.app is at /.vol/16777238/993517

The first two follow a familiar pattern you’ll see throughout the System and Data volumes: their volume ID 16777240 is common to both, and that assigned to the merged volumes, but their inode numbers are wildly different. Huge numbers like 1152921500311883667 come from the SSV, while smaller ones like 461665725 are from the Data volume. Then there’s a slightly lower volume ID of 16777238 and a small inode number of 993517 for Safari, demonstrating that it’s somewhere altogether different: that’s a cryptex, a cryptographically protected disk image with an interesting history.

Why a cryptex?

When the modern boot volume group was being designed and developed, it took into account Safari’s special needs by making it the only bundled app to be stored in the Data volume. This enables it to be updated without having to go through the whole process of building a new SSV, allowing Apple to deliver urgent security patches to Safari and its underlying WebKit and other frameworks. There could also have been political considerations in separating Apple’s bundled browser from the other apps included in macOS.

This changed in Ventura in the autumn/fall of 2022, when Apple applied technology it had originally developed for its customised iPhone, the Security Research Device, dubbed the cryptex, a name formed as a portmanteau for CRYPTographically sealed EXtension. This offers two advantages:

  • Safari, its supporting frameworks, and other components of macOS that Apple prefers not to build into the SSV, can be delivered in cryptexes. As I’ll explain later, this also enables tailoring of macOS to platform.
  • Some urgent security patches could be delivered in cryptexes, making them faster to release and simpler to install in a Rapid Security Response (RSR).

Since then, RSRs seem to have had their day, and appear to have fallen from favour. But, as a means of delivering Safari and other more changeable components of macOS, cryptexes have proved their worth.

How a cryptex works

Although a cryptex is at heart a read-only disk image that is mounted during the boot process, it has two properties of particular importance:

  • Its contents are cryptographically verified, in much the same way that the contents of the SSV are, using hashes of its entire contents.
  • Its internal file system is grafted into the root file system when it’s mounted, rather than being mounted as a separate volume.

APFSCryptexMount1

Mounting a cryptex starts with validation of the payload and its manifest. It then undergoes a sequence of processes similar to the mounting of an APFS volume, with a checkpoint search to establish stable checkpoint indices, and a check to discover whether there’s anything to recover, which seems unlikely. The graft is then performed in a series of opaque steps, with root hash authentication and validation. The object ID is found, and the graft completed.

Once this has been completed for each of the standard cryptexes and any installed RSRs, the contents of those are effectively part of the system, as a hybrid of the SSV and cryptexes. In the case of the Safari app, this process effectively places it in the main Applications folder, even though the original app is actually located in the System/Applications folder of the App cryptex in /System/Volumes/Preboot/Cryptexes.

As with the current boot System and Data volumes, grafted cryptexes aren’t unmounted or ungrafted until shutdown.

There are currently three main cryptexes in use, App containing Safari, its frameworks and other supporting files, and OS, with a range of other system items including additional frameworks, and several large dyld shared caches. You’ll also see an Incoming cryptex in /System/Volumes/Preboot/Cryptexes. As they’re outside the SSV, new and replacement cryptexes are installed without rebuilding the SSV, and in some cases don’t even need a soft restart of macOS.

Architecture-specific cryptexes

In addition to providing Safari and its related components, cryptexes also provide useful economy in shared caches, and explain why macOS updates for Apple silicon Macs are invariably larger than those for Intel models.

While the contents of the SSV appear to be identical on both Intel and Apple silicon, thus have a single signature, the two architectures differ in their cryptexes. Those for Apple silicon Macs contain dyld shared caches for both architectures, and a set of aot shared caches, presumably to support Rosetta 2, and amounting to 5.24 GB in total size; those for Intel Macs only contain Intel dyld shared caches of 1.68 GB total size.

Given their sizes, that’s a valuable efficiency both for updates and in storage required, and is the major reason for updates for Apple silicon Macs always being larger than those for Intel. Thankfully, because those shared caches are supplied compressed, the difference in update sizes is much smaller than the 3.56 GB difference when they’re decompressed and installed.

主宾谓

By: fivestone
11 September 2023 at 23:48

之前聊到,日文、藏文的语序结构,和我们习惯的中文、英文不同,是谓语动词放在句子最后的「主语-宾语-谓语」的形式。

  • 中文、英文,是「主-谓-宾」。譬如:我-是-学生。我-想-你。
  • 日文、藏文,是「主-宾-谓」。类似于:我-学生-是。我-你-想。

:(吐槽)所以人们常说的,日本人懂礼貌,会听人把话说完。其实是因为这样的结构,需要认真听到最后一个词,才知道整个句子要说「是」或「不是」啊。

:对于需要使用不同敬语的日本人,也方便他们先把宾语对象列出来,再根据其身份,决定用什么样的敬语去修饰动词。


另一个 blog 有时候写得少的原因,大概是在「文章是在写给谁?」这方面,无意识地发生了混乱。

除去一部分

  • 技术贴
  • 分享有趣的经历或见闻
  • 对自己状态的描述、分析、展示

的篇目;其它很多文章,应该是(有意识或无意识地)有一个,潜在的写作对象的。他可能是

  • 现实中特定的人,可能是情感相关,也可能只是隔空喊话。当然,对方未必会来看;
  • 一个虚幻的,用来倾诉的对象;
  • 想要吐槽的某些现象,所代表的人群;
  • 预计会来看这个 blog 的读者们,不是特定的人,但有某种同温层特质;
  • 也可能,这个对象还是我自己。

于是,经常写到一半,突然意识到这个对象的存在,然后陷入「我这样写,有什么意义吗」的沮丧,也就不写了。

又或者,吐槽吐到一半,突然意识到,我所吐槽的特质,其实和来看 blog 的人,并不相关。于是反而担心,会不会让读者们对号入座产生误解,或者觉得我这个对空掰扯道理的样子很爹味儿之类的。

——就像在「主-宾-谓」的句子里,谓语写一半了,才意识到,那个预设的宾语的存在。

❌
❌