Disk images, files that contain the contents of a physical storage medium, go back long before the first Mac. Among other tasks, they were originally used to contain representations of floppy disks for replication in manufacture.

Today disk images are at the heart of macOS, and widely used by third-parties. They’re an essential part of macOS installers, home to Recovery mode, and the basis for cryptexes. They’ve been used to burn and replicate optical disks, to archive disk contents, extensively for network backups, and for the distribution of software.

Classic Mac OS

In Classic Mac OS there were two utilities that worked with different formats: Disk Copy used replicas later in DC42 format, after Disk Copy version 4.2, while compressed formats known as DART were handled by the Disk Archive/Retrieval Tool, hence their name.

Mac OS 9 brought Disk Copy 6.0 with added support for the New Disk Image Format (NDIF), which supported resource forks, and ended with its last release version 6.3.3. This also supported read-only Rdxx formats.

By this time, variants of formats had become complex. Here, Disk Copy is configured to create a read-only compressed .img file containing the contents of a standard 1.4 MB floppy disk. In the upper window, it has completed validating the checksum on a self-mounting .smi disk image that’s part of a DiskSet. These could also be signed, using certificates issued not by Apple but by DigiSign.

Here’s Disk Copy saving an image of a hard disk using a similar read-only compressed format, this time to accommodate 1.5 GB.

Mac OS X

The release of Mac OS X 10.1 Puma in 2001 brought Apple’s new Universal Disk Image Format (UDIF), used in DMG disk images, which only had a single fork as its resource fork was embedded in the data fork. Although pre-release versions of Disk Copy 6.4 and 6.5 were available with UDIF support for Mac OS 9, neither was ever released, leaving Classic Mac OS without access to UDIF images. Its support for compression options in Apple Data Compression (ADC) unified the two disk image types, and extended support for images larger than a floppy disk. This new format enabled disk images to represent whole storage devices, complete with a partition map and disk-based drivers.

Tools provided in Mac OS X for working with disk images include Disk Utility and the command tool hdiutil.

On 21 January 2002, the first version of DropDMG, a third-party substitute for creating disk images, was released by C-Command Software. This quickly enabled developers to create disk images with artwork, licences and other features that weren’t accessible from the tools bundled in Mac OS X. DropDMG has flourished over the last 23 years, and remains popular today.

DropDMG’s options for creating a new disk image far exceed those in Disk Utility. Particularly helpful are the compatible version hints shown on various options, to remind you of which file systems are available in different macOS versions, and which types of disk image container are supported. DropDMG will even convert old NDIF disk images last used in Mac OS 9 to more modern formats. It will also change the password of an encrypted disk image from a menu command.

In Mac OS X 10.2 (2002), UDIF and most other supported formats were served from a kernel extension without requiring a helper process. The following year, 10.3 Panther started using a faceless utility DiskImageMounter to mount disk images. Apple then dropped support for embedded resource forks in disk images in Mac OS X 10.4.7, and newly created disk images became less compatible with older Mac OS versions.

Sparse bundles

Until Mac OS X 10.5 Leopard in 2007, all disk images had used single-file formats, although some could be segmented across file sets. Leopard introduced the sparse bundle with its folder of smaller band files containing data. These enabled the image to grow and shrink in size, and became popular means of storing mountable Mac file systems on servers using different file systems.

This is another third-party tool that improved access to disk images from the GUI, DMG Packager, seen in 2009. Unlike DropDMG, this appears to have vanished without trace.

In 2011, with the release of Mac OS X 10.7 Lion, Apple removed more support for old disk image formats. DiskImageMounter no longer opened NDIF .img, .smi self-mounting, .dc42 and .dart compressed formats, although the hdiutil command tool still retained some access to them.

Disk Utility, seen here in 2011, has provided basic access to many disk image formats, but these are only a small selection of options available in the hdiutil command tool, or in DropDMG.

This shows the complex set of options available when creating a new disk image in Disk Utility in OS X 10.10 Yosemite, before the advent of APFS.

Support for compression was enhanced in OS X 10.11 El Capitan with the addition of lzfse in a new ULFO format, and macOS 10.15 Catalina added lzma in ULMO. In both cases, these new formats aren’t accessible in older versions of macOS.

APFS support

The arrival of a pre-release version of the new APFS file system in macOS 10.12 Sierra brought its support in disk images, although only for experimental purposes, and Apple cautioned users to ensure their contents were well backed up.

In addition to adding the more efficient ULMO compressed format, macOS 10.15 Catalina is the last to support many Classic Mac OS disk image formats, including those from DiskCopy42, DART and NDIF from Disk Copy 6.x. Support for AppleSingle and MacBinary encodings, and dual-fork file support, were also removed in macOS 11.0 Big Sur in 2020.

This ‘warning’ alert from 2020 illustrates one of the longstanding issues with disk images. Although integrity checking of disk images using checksums has been valuable, when an error is found there’s no possibility of repair or recovery as the image can’t be ‘attached’, so its file system can’t be mounted.

macOS 12 Monterey in 2021 brought multiple deprecations of older formats, including UDBZ using bzip2 compression, segmented UDIF images, and embedded resources. It’s also thought to be the first version of macOS in which UDIF read/write images (UDRW) have been stored in APFS sparse file format, although Apple has nowhere mentioned that. This has transformed what had previously been space-inefficient disk images that retained empty storage into a format that can prove almost as efficient as sparse bundles. This results from the Trim on mounting HFS+ and APFS file systems within the image freeing unused space, enabling that to be saved in the sparse file format.

Disk images have never been glamorous, but have remained at the heart of every Mac.

References

man hdiutil
Introduction
Tools
How read-write disk images have gone sparse
Performance
Bands, Compaction and Space Efficiency

Appendix: Disk image formats

Supported

• UDRW – UDIF read/write
• UDRO – UDIF read-only
• UDCO – UDIF ADC-compressed
• UDZO – UDIF zlib-compressed
• ULFO – UDIF lzfse-compressed (OS X 10.11)
• ULMO – UDIF lzma-compressed (macOS 10.15)
• UDTO – DVD/CD-R master for export
• UDSP – sparse image, grows with content
• UDSB – sparse bundle, grows with content, bundle-backed, Mac OS X 10.5
• UFBI – UDIF entire image with MD5 checksum.

Unsupported

• DC42 – Disk Copy 4.2 (Classic)
• DART – compressed, for Disk Archive/Retrieval Tool (Classic)
• Rdxx – read-only Disk Copy 6.0 formats
• NDIF – Disk Copy 6.0, including IMG and self-mounting SMI
• IDME – ‘Internet enabled’, on downloading post-processed to automatically copy visible contents into a folder, then move the image to the Trash. Now deemed highly insecure.
• UDBZ – UDIF bzip2-compressed image (deprecated).

Sometimes Macs behave in ways that you don’t forget. It was nearly six years ago, in 2019 when my iMac Pro was still running macOS Mojave, that it took over half an hour to start up in Safe mode. As became obvious, that was because it checked the integrity of every snapshot, including its long series of Time Machine backups. That behaviour stopped with the release of Catalina later that year, and revisiting the experience last week was interesting, when I examined what Safe mode does now in Sequoia.

Safe mode fsck

Checking disk integrity has long been a feature claimed of Safe mode, although Apple has only recently clarified that its checks are “similar to the more comprehensive check performed by the First Aid feature of Disk Utility,” unlike those in the days of Mojave.

Because these are run when starting up into Safe mode, macOS can’t report them directly to the user. There are only two places for their results to be recorded: in the fsck_apfs log at /var/log/fsck_apfs.log (and its error log at /var/log/fsck_apfs_error.log), and in the Unified log, neither of which are likely to be visited by users. It doesn’t seem possible that checks could be run before entries are made in the Unified log, and if they were their results would be inaccessible, and of no value to the user.

If you do look there, you might be surprised to discover that the checks run by fsck_apfs are ‘quick checks’, described as a “check whether the device is `clean’. If device is an APFS volume, fsck_apfs will quickly check the APFS container and the specified APFS volume. If device is an APFS container, fsck_apfs will quickly check the APFS container and all the APFS volumes in it. By default, no repairs are attempted during a quick check.” Thankfully they also no longer include Time Machine snapshots or backups.

For users, the most important of volumes to be checked is the Data volume in the current boot volume group. Yet careful examination of both log records reveals that such checks fail, returning error 65, because the device is mounted with write access. Although they could be performed using fsck_apfs‘s live verification, no attempt is made to do that. From the active boot volume group, only the Recovery and Update volumes are checked, together with all accessible external volumes.

Normal user fsck

Before you think that’s better than making no checks at all, the sequence of fsck_apfs checks run at the start of Safe mode appears to be identical to those made during startup into normal user mode, and reports the same failures. Thus, in this respect Safe mode is no different from normal, despite Apple listing this as a feature of Safe mode.

It’s worth reading the fsck_apfs man page to see the many options available in this command, which is the basis for all checks and repairs made to APFS volumes and containers. There are two in particular that you may prefer to invoke rather than using Disk Utility’s single option to both check and repair. My favourites that make it worth opening Terminal instead of Disk Utility are -n to prevent it from attempting to make any repairs, and -S to skip snapshots.

Given that fsck_apfs appears unable to repair any errors it finds in snapshots and admits to that in its man page where it states that “no repairs can be made”, it puzzles me that without the -S option it will still check all snapshots it comes across when running its checks. Perhaps it’s time to alter that default behaviour and make checking snapshots the explicit option.

First Aid

Disk Utility’s First Aid is descended from a succession of utilities starting with the Disk First Aid app in Classic Mac OS, which separated disk Verify and Repair features.

Mac OS X merged the two Classic apps Drive Setup and Disk First Aid, but kept them as distinct tools within its new Disk Utility app and retained separate Verify and Repair features.

Those two features were still present in Mountain Lion or Mavericks, shown here, but soon afterwards only Repair was available.

Now that APFS is mature and has proven itself more than a worthy successor to HFS+, the need to both check and repair its containers and volumes should be abating. This would be a good time to return to separate controls, allowing users to check APFS in the expectation that there are no errors to be found. This is more important now because of the absence of any alternatives.

Monopoly

Disk Utility is in a unique position. Apple has chosen to lock third-parties out of developing competing disk repair utilities, in particular those that can be used in Recovery mode. Users can’t opt for apps with more or better features, putting the onus on Apple to provide a comprehensive solution that meets users’ wishes, not its idea of essential requirements.

Conclusions

• Apple needs to be explicit about what disk checks are run in Safe mode, and where users can inspect their results.
• fsck_apfs is flexible and powerful, but has strange defaults that merit reconsideration.
• Disk Utility should offer more options beyond its single ‘check and repair’, in particular to check only and with snapshots excluded. Those are most important in Recovery mode, where there can be no alternative.

It must be almost two years since I last demonstrated some magic tricks involving available and purgeable disk space. At that time, the amount of space involved was a mere 83.71 GB. Today I’m going to show you how I converted 228.16 GB of purgeable space into used space, recovering a lot of my files in the process.

Prior to my iMac Pro’s forced update to Sequoia 15.3.1, described here yesterday, its internal SSD had around 150-160 GB free, with no purgeable space at all. Immediately before installing that update, SoftwareUpdate reported that there was 160.57 GB available. When I had coaxed it back into life, now running 15.3.1, the foot of each Finder window told me there was now “393.72 GB available”. Imagine my surprise/shock/horror that about 240 GB of what had been on that SSD before it was updated had now vanished.

Recalling my previous experience, I selected Macintosh HD in the Finder, and opened the Get Info dialog. That confirmed the situation, stating

• Available 393.72 GB (228.16 GB purgeable)
• Used: 828,672,419,328 bytes (829.67 GB on disk)

A little arithmetic reveals that of the 393.72 GB “available”, only 165.56 GB was actually free at the time, the rest being “purgeable”. Together the truly free and that used “on disk” amounted to 995.23 GB. Adding the 16.16 GB used by other volumes, my Mac’s internal SSD had grown in capacity to 1.011 TB, which made that slightly traumatic update worthwhile after all.

Sadly, Disk Utility wasn’t so impressed. The figures it gave were very different indeed:

• Available: 165.56 GB (none purgeable)
• Used: 818.52 GB + 16.16 GB on other volumes = 834.68 GB
• One snapshot of 7.16 GB

for a total disk size of exactly 1 TB. The figures my own Mints gave were in accord with those from Disk Utility.

Although I much preferred the Finder’s figure of nearly 400 GB of “available” space, I realised that could only come at the cost of purging all that 228 GB of “purgeable” space. As that seemed to include many of my files, I thought it was time to work this week’s magic trick. I therefore restarted the Mac, and all of a sudden purgeable space had vanished, leaving me with only about 165 GB of free space after all.

To remind you of what I found nearly two years ago, after updating to macOS 13.3.1, the Finder found 83.71 GB “purgeable”, and my SSD had then grown to 1.08 TB in size.

That’s two major versions of macOS and almost two years apart, and the Finder still can’t come up with correct figures.

If you have an Apple device, you’ll be familiar with the idea behind Storage Settings in System Settings > General. What you might not be prepared for is what you’ll see there. Like its equivalent in the General section of the Settings app, the bar chart at the top often shows that much of your Mac’s storage is filled with System Data, and it neither breaks that down into anything more meaningful, nor does it offer a tool to do anything about it.

Storage Settings on macOS has a troubled history. It used to be part of About This Mac, and often took so long to complete its bar chart that folk gave up waiting. In some cases, it even crashed in the process. More often than not, the totals it gave for used and free space on the startup disk were substantially different from those reported by the Finder or Disk Utility.

In recent versions of macOS, Storage Settings has improved steadily. It does now complete its analysis in a reasonable period, and in most cases its figures aren’t too different from Disk Utility’s. But System Data remains a puzzle that confuses rather than enlightens.

What is System Data?

From the information given by Apple, it’s most likely that this isn’t really what we’d consider to be data used or required by macOS, but the often substantial difference between how much space is used, and how much can be attributed to other categories, like Books and Music. It’s not a real category, but an etcetera, a ragbag including all sorts of files and other data.

Calculating some of the other categories seems difficult enough. For example, my Music folder contains over 100 GB of individual audio files and my Music library, but Storage Settings recognises less than 50 GB of that in its Music category. The remaining 50 GB has to be accounted for elsewhere, and that’s likely to be buried in System Data. With Photos and TV it’s the other way around, with almost all my libraries stored on an external SSD, but Storage Settings still claims I have 36 GB in TV and doesn’t even mention Photos.

So the categories it does list don’t always match with what we know is on the disk. When it adds all those together and takes that total away from the amount of disk space used, that difference is little more than a guesstimate, and unlikely to contain much data required by macOS. No wonder Storage Settings can’t suggest any way to reduce that size.

Dynamic storage

Unlike more traditional file systems, APFS is dynamic in its use of storage space, and macOS now uses aggressive caching policies. One of those came to light when we thought the Finder had a large memory leak, only to be told that, in the right circumstances, it can retain GB of Quick Look thumbnails in memory, so that scrolling through them can be smooth. When there’s sufficient free disk space, macOS can maintain large caches without using any swap space on disk.

APFS features like snapshots can retain data we presumed had been deleted, but has to be kept until the snapshot referencing it has been deleted up to 24 hours later. That space has to be accounted for somewhere, and in many cases that too goes into the System Data category, although it has actually been created by Time Machine, which oddly doesn’t have its own category.

Although not given in the Storage bar chart, both the Finder and Disk Utility should state the amount of purgeable space in use. This could be freed when necessary, but that’s determined by macOS rather than the user. I have previously looked at how purgeable some features like snapshots are, and in practice you shouldn’t rely on their automatic removal when you think your Mac needs more free space.

What is Storage Settings good for?

Like its equivalent on iPhones and iPads, some of the tools it provides are of great value in housekeeping. Click on the ⓘ Info button for each of them to get further information and for actions you can take. Categories with the more useful tools include:

• Applications, shows apps by size, and which are Intel-only, and duplicates;
• Developer, if you have Xcode installed, can clean up build files and device support;
• Documents, lists by size and type, 32-bit apps;
• Messages, lists larger attachments and lets you delete them;
• Music Creation, helps you remove sound libraries;
• macOS, tells you how much space is used by Apple Intelligence.

Although Music might seem promising, it’s only interested in music video files you can remove.

How to check free space

Don’t trust Storage Settings or the Finder to provide an accurate estimate of free space available on disk. Instead, open Disk Utility (with Show All Devices selected in its view) and in the list at the left in its main window, pick the volume you’re interested in, then go up to its container in the list and select that. Free space shown there is the most accurate estimate of that available to all volumes within that container, as in APFS volumes within the same container all share the same free space.

Select one of those volumes, and you’ll see free space given as a higher value, with a figure for the space that’s purgeable in brackets. That represents the maximum space that could be freed if all purgeable contents were to be deleted.

For example, free space shown here for a container is 619.72 GB, which is the space available without any purging. One volume within that container is given as having 864.5 GB available, with 244.78 GB purgeable:
864.5 = 619.72 + 244.78 GB

Figures given by the Finder are only refreshed periodically, while Disk Utility should recalculate them whenever you select a volume or container, so should always be up to date.

Summary

• System Data in Storage Settings isn’t just files and data for macOS, but everything not listed in another category, and even that is only approximate, a rough estimate.
• Storage Settings has useful tools for managing the contents of your startup disk.
• For an accurate estimate of free space, use Disk Utility, and the free space given there for the volume’s container.
• Don’t expect macOS to free up any purgeable space for you.

It always happens when you’re in a rush to eject an external disk, or when you’re trying to run First Aid in Disk Utility: you’re told that a volume or disk can’t be unmounted or ejected because “one or more programs may be using it” or similar excuses. Not that macOS gives you the slightest idea as to which programs are. This article looks at what you can do next.

Whatever you do, don’t just pull the cable of an external disk: not only will your Mac complain, but you could end up damaging the contents of its files, or even the file system on that volume.

Force eject

This is often offered as an option in the dialog, and if it’s available it’s worth trying if you don’t have the time to carry out a more orderly unmounting. macOS should then identify the processes that are accessing the disk, and terminate their access. That can take time, and seldom appears successful even if you allow a minute or two for it to complete. However, when it does work, it’s likely to be the simplest solution.

Try again

In Disk Utility, this is usually the best option, and if necessary you can try several times. You should also double-check that you’re trying the correct volume: if it’s one of the current boot volume group, System or Data, then you’re better off running First Aid in Recovery mode anyway.

Getting nowhere

When all else fails, the next step is to identify what’s using files on that volume or disk, so you can decide whether to force quit that process in Activity Monitor. Don’t do that blindly, as you could end up killing processes that your Mac does need to run.

To discover which files are open on any volume, use the command
sudo lsof /Volumes/myVol
where myVol is the name of the volume. If you’re unsure how to enter a volume name containing a space, locate it in the Finder’s listing for your Mac, and drag and drop that into Terminal. Once you’ve entered that, type your admin user password at the prompt, and you’ll see a list with entries like
mds 367 root 33r DIR 1,28 192 2 /Volumes/External2
mds 367 root 35u REG 1,28 0 87 /Volumes/External2/.Spotlight-V100/Store-V2/3DD5246F-9AEA-4F0E-9A53-AA63783C3C70/journalExclusion
which are the files and directories open on that volume. This needs to be run using sudo, as otherwise you won’t see any files that are opened by processes running as root, which are most often the culprits. Some recommend using grep, but that shouldn’t be necessary, as lsof is capable of its own filtering.

The information given about each open file contains, from the left:

• an abbreviated name of the command associated, here mds, the Spotlight metadata server;
• the open mode, as the single character following two digits, e.g. 33r is opened for read access only, while 35u is opened for read and write access;
• the type, DIR meaning directory, and REG meaning a regular file;
• the full path to the file or directory.

If you’d rather use an app, then my personal favourite is Sloth from here. Although it’s not notarized, it does everything that I’d want in terms of matching lsof or fuser‘s features. Most importantly, if you click its padlock at the lower right and authenticate, it will show all processes running as root.

If your Mac is running Sequoia, getting Sloth through its security checks is a bit of pain (and intentionally so). You can do this the ‘official way’ by downloading the app, unZipping it, and moving it into your Applications folder. When you try to run it, that will be denied and you’ll end up having to give it your individual authorisation in Privacy & Security settings. Alternatively, you can run the risk and cheat by stripping the com.apple.quarantine extended attribute from its Zip file first, so it doesn’t undergo full first run checks. Although that may be simpler, it also forces you to place complete trust in the app you’ve downloaded.

Once you know which processes are accessing files on that volume, you can decide whether to open the listing in one of Activity Monitor’s views, such as CPU or Disk, select that process, and click on the Stop tool to kill it.