Normal view

There are new articles available, click to refresh the page.
Yesterday — 18 September 2024Main stream

Book Review: ‘Connie: A Memoir,’ by Connie Chung

17 September 2024 at 17:00
In a frank and entertaining new memoir, the TV newscaster recounts how sexism, and Dan Rather, sidelined her groundbreaking career.

© CBS, via Getty Images

Connie Chung at the “CBS Evening News” anchor desk. She shared the top job with Dan Rather for two years.
Before yesterdayMain stream

Why Are Museums So Afraid of Hans Haacke?

16 September 2024 at 17:02
As cultural institutions face an existential crisis over who funds them and how, the 88-year-old artist Hans Haacke is still making curators and collectors clutch their pearls.

Lured by a Promising Job, He Was Forced to Scam People

10 September 2024 at 23:49
A Ugandan man named Jalil Muyeke traveled to Thailand for a promising job opportunity. He ended up being trafficked into a cybercrime operation.

What’s the time Mr. APFS? With a new version of Precize

By: hoakley
12 August 2024 at 14:30

It’s the middle of August, the time when, if you’re not an Apple engineer or third-party developer, you’re most likely to be on holiday. To help you enjoy that, today I’m going to engage you in a little game I call What’s the time, Mr. APFS?

To play along with me, all you need is a copy of the latest version of Precize, version 1.15, available from here: precize115
from Downloads above, from its Product Page, or via its auto-update mechanism.

Precize update

Thanks to some questions from applice, who should really be spending more time watching the Olympics than worrying about file systems, this new version of Precize comes with significant enhancements in its datestamp reporting. First, it now gives datestamps with decimal seconds, for those who want to be truly precise, and it adds two new time fields, for Attribute Modification, and Access, as shown in the screenshot below.

precize92

Once you’ve downloaded the Zip archive, unZip it where it is, and move the Precize app from its folder into one of your Applications folders (that ensures it doesn’t undergo app translocation). Keep the Zip archive it came in, as we’ll use that in the game, whose rules are simple: all you have to do is roughly guess the times in each of Precize’s datestamp fields, for different files and bundles.

What are the times?

APFS stores four datestamps for each file and directory in its volumes, and the four fields shown in this version of Precize correspond to those four datestamps. There is a fifth that the Finder can display, the date and time that an item was added to its current location. That’s not derived from the file’s attributes, but from directories, and isn’t shown in Precize.

The four datestamps are:

  • Created, taken from the create_time value, the time that this item was created, given as the clock time corresponding to an unsigned 64-bit integer giving the number of nanoseconds since January 1, 1970 at 00:00 UTC, disregarding leap seconds.
  • Modified, taken from the mod_time value, the time that this item was last modified, likewise.
  • Attr Mod, taken from the change_time value, the time that the item’s attributes were last modified, likewise. Those attributes include permissions, for example, and changes made to extended attributes should also update this value.
  • Accessed, taken from the access_time value, the time that this item was last accessed, likewise.

By default, APFS is relaxed over how it records the last of these. Some other filesystems are strict, and change the access_time value every time a file is read. Instead, APFS only updates this if its value is prior to (earlier than) the mod_time value. You can change this behaviour by setting an APFS volume’s APFS_FEATURE_STRICTATIME flag, for instance when mounting it by using a command like
mount -u -o strictatime

However, that’s not the default for APFS volumes. For example, inspect your Mac’s Data volume using the command
mount | grep '/System/Volumes/Data '
and you should see something like
/dev/disk6s1 on /System/Volumes/Data (apfs, local, journaled, nobrowse, root data)
without any mention of the strictatime flag. List all mounted volumes using
mount
and you’re unlikely to see any using strictatime.

File datestamps

Open the new version of Precise, so it’s poised waiting for you to drop a file onto its icon in the Dock. Then take a screenshot, and drop that file onto Precise’s icon. What do you expect the four datestamps to be?

This is an easy one for starters: the four times should be very close, and almost match the time given in the screenshot’s name. The first two should be Created and Modified, the latter perhaps a thousandth of a second behind, reflecting the time it takes from creation of the screenshot file to completing the save. Next comes Accessed, when macOS opened the file to perform its animation of the screenshot sliding away. Last of all should be Attr Mod, recording when all its attributes such as permissions were set in its destination folder.

Now copy that file to another volume. What do you expect those four datestamps to be there?

Created and Modified should remain the same, as those are preserved when the file is moved around in local file systems. But Attr Mod and Accessed should record the time that file arrived on the second volume, reflecting its new attributes and its access for the copy.

Now try a couple of harder questions. Drop the Zip archive that contained Precize onto the running app’s icon. What do you expect those four datestamps to be?

All four datestamps should record times close to when you downloaded that Zip file from here. Just as with the screenshot, Created leads, closely followed by Modified, with Accessed and Attr Mod trailing. Note that they’re all four very recent, subsequent to its download to your Mac.

Unzip that Precize Zip archive again and drop the Precize app within it onto the running app’s icon. What do you expect those four datestamps to be?

Now the Created and Modified values are taken from that app when it left my Mac here, but Attr Mod should be more recent, and Accessed is likely to be older unless you have accessed the app bundle since then.

What’s the time, Mr. APFS?

All the best games involve learning. I hope these have shown you how those four datestamps are set and changed. In summary:

  • Software like the Finder, when copying between local volumes, also copies most of a file’s attributes, including Created and Modified datestamps for consistency, even though it creates a new file in the destination. Attr Mod and Accessed are linked to the copy event, though.
  • Software like browsers, when copying from a remote server to a local volume, don’t preserve the file’s attributes, so all four datestamps correspond to the local arrival and creation of the download.
  • Archives such as Zip files preserve most of a file’s attributes, and reconstitute its Created and Modified values to those saved when the archive was created.

As a final exercise, using Precize, determine whether copying files by AirDrop between Macs behaves conservatively like copying between local volumes, or resets datestamps as downloading from the internet does. Why do you think that Apple designed it to work that way?

Have fun!

Copy speeds of large and sparse files

By: hoakley
7 August 2024 at 14:30

I have recently seen reports of very low speeds when copying large files such as virtual machines, in some cases extending to more than a day, even when they should have been sparse files, so requiring less time than would be expected for their full size. This article teases out some tests and checks that you can use to investigate such unexpectedly poor performance.

Expected performance

Time taken to copy or duplicate files varies greatly in APFS. Copies and duplicates made within the same volume should, when performed correctly, be cloned, so should happen in the twinkling of an eye, and without any penalties for size. This is regardless of whether the original is a sparse file, or a reasonably sized bundle or folder, whose contents should normally be cloned too. If cloning doesn’t occur, then the method used to copy or duplicate should be suspected. Apple explains how this is accomplished using the Foundation API of FileManager, using a copyItem() method. This is also expected behaviour for the Finder’s Duplicate command.

Copying a file to a different volume, whether it’s in the same container, or even on a different disk, should proceed as expected, according to the full size of the file, unless the original is a sparse file and both source and destination use the APFS file system. When an appropriate method is used to perform the copy between APFS volumes, sparse file format should be preserved. This results in distinctive behaviour in the Finder: at first, its progress dialog reflects the full (non-sparse) size of the file to be copied, and the bar proceeds at the speed expected for that size. When the bar reaches a point equivalent to the actual (sparse) size of the file being copied, it suddenly shoots to 100% completion.

Copying a sparse file to a file system other than APFS will always result in it expanding to its full, non-sparse size, and the whole of that size will then be transferred during copying. There is no option to explode to full size on the destination, nor to convert format on the fly.

External SSDs

When copying very large files, external disk performance can depart substantially from that measured using relatively small transfer sizes. While some SSDs will achieve close to their benchmark write speed, others will slow greatly. Factors that can determine that include:

  • a full SLC cache,
  • failure to Trim,
  • small write caches/buffers in the SSD,
  • thermal throttling.

Many SSDs are designed to use fast single-level cell (SLC) write caching to deliver impressive benchmarks and perform well in everyday use. When very large files are written to them, they can exceed the capacity of the SLC cache, and write speed then collapses to less than a quarter of that seen in their benchmark performance. The only solution is to use a different SSD with a larger SLC cache.

Trimming is also an insidious problem, as macOS by default will only Trim HFS+ and APFS volumes when they’re mounted if the disk they’re stored on has an NVMe interface, and won’t Trim volumes on SSDs with a SATA interface. The trimforce command may be able to force Trimming on SATA disks, although that isn’t clear, and its man page is forbidding.

Trimming ensures that storage blocks no longer required by the file system are reported to the SSD firmware so they can be marked as unused, erased and returned for use. If Trimming isn’t performed by APFS at the time of mounting, those storage blocks are normally reclaimed during housekeeping performed by the SSD firmware, but that may be delayed or unreliable. If those blocks aren’t released, write speed will fall noticeably, and in the worst case blocks will need to be erased during writing.

For best performance, SATA SSDs should be avoided, and NVMe used instead. NVMe is standard for USB 3.2 Gen 2 10 Gb/s, USB4 and Thunderbolt SSDs, which should all Trim correctly by default.

Disk images

Since Monterey, disk images with internal APFS (or HFS+) file systems have benefitted from an ingenious combination of Trimming and sparse file format when stored on APFS volumes. This can result in great savings in disk space used by disk images, provided that they’re handled as sparse files throughout.

When a standard read-write (UDIF) disk image is first created, it occupies its full size in storage. When that disk image is mounted, APFS performs its usual Trim, which in the case of a disk image gathers all free space into contiguous storage. The disk image is then written out to storage in sparse file format, which normally requires far less than its full non-sparse size.

This behaviour can save GB of disk space in virtual machines, but like other sparse files, is dependent on the file remaining on APFS file systems, otherwise it will explode to its full non-sparse size. Any app that attempts to copy the disk image will also need to use the correct calls to preserve its format and avoid explosion.

Tools

clonesparse3

Precize reports whether a file is a clone, is sparse, and provides other useful information including full sizes and inode numbers.

Sparsity can create test sparse files of any size, and can scan for them in folders.
Mints can inspect APFS log entries to verify Trimming on mounting, as detailed here.
Stibium is my own storage performance benchmark tool that is far more flexible than others. Performing its Gold Standard test is detailed here and in its Help book.

How Time Machine backs up iCloud Drive in Sonoma

By: hoakley
5 July 2024 at 14:30

macOS Sonoma brought major internal changes to iCloud Drive, and coupled with the opacity of Time Machine, you may now be wondering whether it still backs up what you have stored in iCloud Drive, and if so, where to find those backups. This article explains how this now works, and where those backups can be found.

What does Time Machine back up?

By default, Time Machine backs up almost everything on volumes it’s set to back up. It doesn’t, of course, try to back anything up from the Signed System Volume (SSV) containing macOS itself, and there are a few folders it automatically excludes, such as the hidden folder at the root of each volume containing the versions database (as that can’t be restored in any case). But unless you add iCloud Drive to its exclusion list from your Home Library folder, Time Machine will back up all the files and folders in iCloud Drive as long as they’re stored locally at the time it makes that backup.

Dataless files

It’s that last condition that’s the most important consideration. If your Mac has Optimise Mac Storage turned off, then iCloud Drive is run as a Replicating FileProvider, where everything stored in iCloud Drive is also stored locally. In that case, all those items will be backed up by Time Machine just as they would be if kept in purely local storage.

If Optimise Mac Storage is turned on, iCloud Drive behaves as a Non-Replicating FileProvider, and some or many of the files you store in iCloud Drive may be evicted from local storage, and are then no longer available for backup. This is explained in the diagram.

iCloudDriveFileSummary1

Files in local storage consist of attributes, metadata (extended attributes) and data. Move them to iCloud Drive and their data is then copied up to iCloud. When a file is evicted to save local storage space, the local file’s data is removed, making that copy of the file dataless. In previous versions of macOS, that was accomplished using a special stub file, but in Sonoma it simply loses its data, and the distinctive iCloud icon is displayed next to the file in the Finder.

icloudnos1

Dataless files can’t be backed up by Time Machine, as without their data there’s nothing to back up. If you want that file to be backed up, then it has to be downloaded or materialised by downloading its data from iCloud. Some third-party backup utilities can manage that automatically for you, but Time Machine doesn’t. If you want to ensure that a file in iCloud Drive is backed up when Optimise Mac Storage is turned on you’ll thus have to download it manually if it has been evicted to iCloud.

Restoring from backup

The simplest way to restore a file or folder in iCloud Drive from a Time Machine backup is through the Time Machine app. Before opening that app, select iCloud Drive in the sidebar of the front Finder window, then the app will take you back through its backups of iCloud Drive.

tmicloud

The main drawback of that is that items you restore will replace any existing items. You may also prefer to use the Finder’s or a command line interface to your backups, both of which allow you to copy restored items where you prefer. The difficulty here is that macOS deliberately hides the path to iCloud Drive: in your active Home folder, they’re located in ~/Library/iCloud Drive, but that path doesn’t work in Time Machine backups.

To access them in a backup, follow the Finder path
[backupname]/[backup].backup/Data/Users/[username]/Library/Mobile Documents/com~apple~CloudDocs/

From Terminal, you’ll have more of a struggle, but need to look for
/Volumes/.timemachine/[UUID]/[backup].backup/[backup].backup/Data/Users/[username]/Library/Mobile\ Documents/com\~apple\~CloudDocs/
where the path component [backup].backup is repeated. If you locate an item first in the Finder, you can always drag and drop that into the command line for it to automatically paste the correct path.

Summary

  • Unless you manually exclude iCloud Drive from Time Machine backups, the contents of iCloud Drive will be backed up provided that items haven’t been evicted from local storage.
  • If Optimise Mac Storage is turned off, the whole of iCloud Drive will be backed up by Time Machine, as eviction doesn’t occur.
  • If Optimise Mac Storage is turned on, only those files and folders that are stored locally will be backed up by Time Machine. To ensure an item is backed up, manually download it before the backup is made.
  • Restoring iCloud Drive items from a backup is simplest in the Time Machine app, but also available in the Finder, from a path equivalent to ~/Library/Mobile Documents/com~apple~CloudDocs/ in that backup.

❌
❌