MacInTouch Amazon link...

APFS / file systems

Channels
Apple, Security, Other, Products
I have a small business with a dozen iMacs, circa 2014-2017, all booting off external T5's, some formatted as APFS with Disk Utility, some auto-formatted to APFS via Mojave installer, all running Mojave. All work fine.
I've had no success booting from my T5 formatted as APFS, even after re-installing Mojave on it. So I erased the drive, formatted it as HFS+, and tried again, still with no luck. The boot process hangs about half-way through consistently. I thought it was the drive, so I tried booting from another APFS-formatted T5, and that didn't work either. So I'm wondering if it's my iMac. I'll try both drives with my MacBook Pro next.
 


I've had no success booting from my T5 formatted as APFS, even after re-installing Mojave on it. So I erased the drive, formatted it as HFS+, and tried again, still with no luck. The boot process hangs about half-way through consistently. I thought it was the drive, so I tried booting from another APFS-formatted T5, and that didn't work either. So I'm wondering if it's my iMac. I'll try both drives with my MacBook Pro next.
Did you install directly or try cloning the install from another disk? I'd try cloning it first from a boot disk that's known to work.
 


Did you install directly or try cloning the install from another disk? I'd try cloning it first from a boot disk that's known to work.
I tried both approaches. Interestingly, I can successfully boot from both T5s (one APFS, the other HFS+) on my MacBook Pro, so there's something weird about the iMac. I'm going to try them on my other iMac at home too.
 


I tried both approaches. Interestingly, I can successfully boot from both T5s (one APFS, the other HFS+) on my MacBook Pro, so there's something weird about the iMac. I'm going to try them on my other iMac at home too.
I'm beginning to think it's more due to the Mac involved, as well, having had a 13" MacBook Pro not boot repeatedly and the 15" boot just fine with this one T5.
 


Ric Ford

MacInTouch
I'm beginning to think it's more due to the Mac involved, as well, having had a 13" MacBook Pro not boot repeatedly and the 15" boot just fine with this one T5.
These results beg the question of Mac firmware versions and whether they’re up to date, or differ, etc.
 


These results beg the question of Mac firmware versions and whether they’re up to date, or differ, etc.
Yes, I suspect that this (or some other factor associated with my 2014 iMac) is at fault. My home iMac, which is a 2017 model, boots from an identical APFS-formatted T5 just fine.
 


It's unclear to me, though, if High Sierra's transitional and problematic APFS implementation has been updated to a more robust and current APFS version somehow (via stealth firmware updates, perhaps?), or if decent APFS support requires an update to macOS Mojave or Catalina.
I've been avoiding High Sierra, mostly because of its half-baked APFS implementation. If Apple managed to port a stable APFS to it, then I would consider upgrading my MacBook Air.
I have a Mac Mini Server (Mid 2011) running fully updated High Sierra with no obvious APFS issues for a couple of years. It is run by two SSDs in an OWC ThunderBay 4 Mini (one is the boot disk, the other a clone of the boot disk as a backup.

I've noticed that in Mojave and Catalina, when you run First Aid in Disk Utility on a APFS formatted SSD, you get an output line saying how the volume was formatted and the latest modification. Since I can easily remove the SSDs running the Mini, I popped them out and ran them through Disk Utility on my 2012 Mac Book Pro running Mojave. Here are the results:

The volume azaharserverSSD was formatted by hfs.convert (748.77.8) and last modified by apfs.kext (945.275.8)
The volume azaharserverSSD.PortA was formatted by newfs.apfs (748.77.10) and last modified by apfs.kext (945.275.8).
Since I have High Sierra, Mojave, and Catalina volumes on my boot SSD, I also tested those:
High Sierra on Mac Book Pro
The volume Untitledy was formatted by newfs.apfs (748.77.11) and last modified by apfs_kext (945.275.8).
Mojave on Mac Book Pro
The volume UntitledMojave was formatted by newfs_apfs (748.77.8) and last modified by apfs.kext (945.275.8).
Catalina on Mac Book Pro
The volume UntitledCatalina was formatted by diskmanagementd (1412.61.1) and last modified by apfs.kext (945.275.8)
Notice that all of results of fully updated systems were formatted by a variety of means but have all been modified by apfs.kext (945.275.8).

I don't know how complete the modification was for each system, but the final APFS systems have all been updated by the same kext. With all the lack of transparency with APFS, while the 5 volumes might not be exact duplicates, all have been updated.
 


... I popped them out and ran them through Disk Utility on my 2012 Mac Book Pro running Mojave. Here are the results:
... but the final APFS systems have all been updated by the same kext.
I could be under-informed but I would expect the "last modified" version to match the booted macOS's kext version. (By default macOS mounts drives as read/write. If a drive is writeable, macOS will update Spotlight's files et al when the drive is mounted.)
 


I could be under-informed but I would expect the "last modified" version to match the booted macOS's kext version. (By default macOS mounts drives as read/write. If a drive is writeable, macOS will update Spotlight's files et al when the drive is mounted.)
You may very well be right, but being bull-headed, I tried a different tack. This time I partitioned an SSD with Catalina into 4 APFS containers. Then using fsck_apfs on Catalina I checked and got the following result:
The volume Untitled3 was formatted by diskmanagementd (1412.61.1) and last modified by apfs_kext (1412.61.1)
Then, booting from High Sierra, I installed High Sierra on one of the new containers. Again using fsck_apfs from Catalina, I got:
The volume Untitled3 was formatted by diskmanagementd (1412.61.1) and last modified by apfs_kext (1412.61.1).
So the indication is that the High Sierra install does not completely reformat the APFS volume, but it also does not indicate whether it modifies it during the installation.

Unfortunately, there doesn't appear to be a working APFS tool that will tell you what the APFS version is in High Sierra. A suggested tool, gpt, doesn't seem to work in High Sierra. Howard Oakley has a series of posts regarding the status of APFS tools. Here's one:
Eclectic Light Co. said:
Why isn’t APFS fully supported yet in macOS?
... Tracking build numbers of APFS revealed that it remained very much a work in progress until its release nearly a year ago in Mojave, when it finally did achieve all its major objectives. It now copes well with a full range of Unicode names, hard disks and Fusion Drives, although many users remain unconvinced that it’s the file system of choice on those media.
Support tools for working with APFS have been considerably less impressive, though, and access through developer interfaces or documentation essentially non-existent.
 


Ric Ford

MacInTouch
In other file system news, here's an update on FUSE software, needed for such critical things as SMART support of external drives...
The Register said:
FUSE for macOS: Why a popular open source library became closed source and commercially licensed
In May this year, users of popular open source project FUSE for macOS noticed the source code for the latest update was missing. The project had become closed source and was no longer free for commercial use. But as The Reg discovered when we had a talk with its maintainer, there was a very good reason for that – and it's not a good look for the many companies that used it.

"... the number of applications relying on FUSE for macOS has grown rapidly. Today FUSE for macOS is used by many big internet companies, among them cloud storage and infrastructure-as-a-service providers and even popular social networks."

The problem? Fleischer was doing most of the development work for free. In May 2016, he asked for support. "In order to take FUSE for OS X to the next level, I would like to spend more time working on it, but this won't be possible without being compensated for my work," he explained at the time.

"My post did not produce the hoped-for results," Fleischer told us. "Then I altered my strategy and tried contacting companies directly, that are using FUSE, and asked for their support.

"I tried to advocate that having one project, that everyone contributes to, instead of many closed source forks without pushing fixes and improvements upstream, would benefit everybody. I was able to convince one or two companies to support my work financially. However, most companies were reluctant to support the project that their product depends on because it is available for free. This in itself is kind of ironic....
 


OWC has switched some of their Mercury Extreme Pro SSD controllers from Silicon Motion to Phison with their newer black drives. Phison has only 11 smartmon attributes that are reported, compared to 30 with Silicon Motion. I've never heard of Phison. Opinions?

These big SSDs are primarily for updating data like documents, graphics, video, etc. Would APFS be more reliable or wear out the drive less in Mojave, than if formatted MacOS Extended? How about with High Sierra? I'm guessing that APFS is better with Mojave, but maybe MacOS Extended is better for High Sierra, assuming APFS is an older version there.

Of course, there's a current lack of utilities for repairing APFS as a factor, but it sounds like DiskWarrior is working on it, according to comments the author made on an Eclectic Light thread a while back. Any advice is appreciated, thanks.
I've decided I'll keep an old iMac on High Sierra in case there is a firmware update this year, then use the DOS Dude patcher to go to Mojave in the fall. I'll keep the internal SSD on APFS to simplify that transition, and hope that APFS is reliable.

On a new iMac with Mojave, I have an external data-only SSD. I'm still debating APFS Encrypted vs. HFS Encrypted because of APFS reliability and SSD wear questions. Does anyone have advice please?
 


Ric Ford

MacInTouch
I found some fascinating file systems (and history) in this list at Wikipedia and thought others might appreciate the information, too:
Wikipedia said:
List of file systems
The following lists identify, characterize, and link to more thorough information on computer file systems. Many older operating systems support only their one "native" file system, which does not bear any name apart from the name of the operating system itself.
 


Ric Ford

MacInTouch
Here's some good, interesting information about file systems (and the Mac):
Ars Technica said:
The exFAT filesystem is coming to Linux—Paragon software’s not happy about it
... There are several problems with using MacOS' iffy NTFS support as a case against open source software. The first is that NTFS support doesn't seem to be a real priority for Apple in the first place. MacOS Classic had no NTFS support at all. The NTFS support present after Mac OS X 10.3 "Panther" was, effectively, a freebie—it was already there in the FreeBSD-derived VFS (Virtual File System) and network stack.

Another problem with this comparison is that NTFS is a full-featured, fully modern filesystem with no missing parts. By contrast, exFAT—the filesystem whose Linux kernel implementation Paragon is throwing FUD at—is an extremely bare-bones, lightweight filesystem designed for use in embedded devices.

The final nail in this particular coffin is that the open source NTFS implementation used by MacOS isn't Microsoft-sanctioned. It's a clean-room reverse-engineered workaround of a proprietary filesystem. Worse, it's an implementation made at a time when Microsoft actively wanted to close the open source community out—and it's not even the modern version.

As Paragon notes, NTFS-3G is the modern open source implementation of NTFS. NTFS-3G, which is dual-licensed proprietary/GPL, does not suffer from potential write-corruption issues—and it's available on MacOS, as well as on Linux.

Mac users who don't need the highest performance can install a FUSE implementation of NTFS-3G for free using Homebrew, while those desiring native or near-native performance can purchase a lifetime license directly from Tuxera. Each $15 license includes perpetual free upgrades and installation on up to three personal computers.
 


... So the indication is that the High Sierra install does not completely reformat the APFS volume, but it also does not indicate whether it modifies it during the installation. Unfortunately, there doesn't appear to be a working APFS tool that will tell you what the APFS version is in High Sierra. A suggested tool, gpt, doesn't seem to work in High Sierra. Howard Oakley has a series of posts regarding the status of APFS tools. ...
I did a similar thing after updating to Catalina 15.4 onto a new 2TB SSD and installing High Sierra on a new volume in the APFS container. This time I got a different result testing with fsck_apfs:

The volume HighSierraTest was formatted by diskmanagementd (1412.101.1) and last modified by apfs_kext (1412.101.1).

Now the High Sierra volume is originally formatted with the latest apfs_kext. Then I partitioned the disk to add an additional APFS container and installed High Sierra in the new container. The apfs_kext remained 1412.101.1

I don't know if this solves problems people have reported the APFS in High Sierra, but it is a different result. High Sierra may not be able to use all the APFS upgrades, but at least it seems to run ok in the latest APFS version.
 


Ric Ford

MacInTouch
Howard Oakley attempts to explain how Apple's new APFS file system works and some of its issues.
Eclectic Light Co. said:
Copy, move and clone files in APFS, a primer
They’re some of the commonest actions in the Finder: copying and moving files around within and between volumes. This primer explains how they work on APFS volumes in Catalina, where they aren’t as simple as they used to be in HFS+.
 



The presentation linked elsewhere says that one difference is, in Mojave, the SSD part of the fusion drive is just a cache. In High Sierra and earlier, the SSD drive [storage] was in addition to the hard drive. File blocks were either on the SSD or on the hard drive, but not duplicated. Fusion would move the data between the drives, keeping hot blocks (via some unknown algorithm) on the SSD. Writes would land on the SSD first, and then the blocks would migrate to the hard drive. This means that your total amount of storage was the sum of the two drives.

In Mojave, it doesn't work like this at all. Instead, the Fusion drive now works like a hard drive with a large cache: the SSD caches data that is also on the hard drive. Your total storage is now just the size of the hard drive. I don't recall Apple mentioning that installing Mojave would reduce the Fusion size by 128 GB.
I'm not positive which "presentation linked elsewhere" I was referring to, but perhaps it was "Busting 7 myths (lies) about Mojave", linked by James R Cutler, which in item #2 says:
Howard Oakley said:
Busting 7 myths (lies) about Mojave
APFS doesn’t use CoreStorage, but according to Apple’s latest documentation, the SSD component in the Fusion Drive is used as a cache for its companion hard disk. For the purposes of reading files, the SSD and HD contain the same file data (Apple doesn’t explain which of the files on the HD are so cached on the SSD, though). For writing files, the Mac writes to the SSD and space is allocated on the HD to receive that data in slower time.
I can find various sources that say that, with APFS Fusion drives, the file metadata is stored on the SSD and that, because APFS has a higher-level view of the data than CoreStorage, it can better optimize the distribution of files between the SSD and hard disk drive. But I can't find another source that says that the SSD and hard drive contain the same files.

My guess is that an APFS Fusion drive is not using the SSD as a cache for data that's also on the hard drive and that you do not lose storage space. In fact, my reported space after converting my Fusion drive to APFS is slightly higher: 3.12 TB vs. 3.11. It definitely is still reporting the space as the sum of the 3TB hard disk drive and the 121GB SSD.
 


Ric Ford

MacInTouch
I've been considering sparse disk images for remote backup, but Howard Oakley says there are bug problems when they get large (and there may be other issues for these uses, as well):
Eclectic Light Co. said:
How to make Time Machine backups to an APFS disk
... Sparse bundles store their data in many files, termed bands, in the same folder. Experience with shared storage is that, if the number of bands exceeds about 100,000, then the sparse bundle is likely to malfunction.
 


Using SSDs to store small files and/or metadata is an ongoing trend in the storage world. The ZFS ecosystem is transitioning to the possibility of making pools consisting of mixed storage systems to make the overall pool more responsive.

For example, as of FreeNAS 12 or ZFS on Linux 0.8, you have the option to use special VDEVs (consisting of SSDs) to host the metadata, small files, or de-dup tables to make the pool more responsive.

Unlike APFS, ZFS gives you the option to mirror the SSDs inside the special VDEV [multiple times], since if the special VDEV fails, all the data in the pool will be gone. Storing the data across 2+ mirrored SSDs will hopefully make a pool failure very unlikely.

L2ARC is a way to add a no-risk read cache to a ZFS system. It is not persistent between reboots but can store files and metadata alike. I use it to speed up browsing by setting it to metadata-only mode. The speed improvements can be astounding as the cache heats up. For example, Carbon Copy Cloner went from 1.5 hours to sync a 1.4TB iTunes folder to 5 minutes within 3 trials.
 


Amazon disclaimer:
As an Amazon Associate I earn from qualifying purchases.

Latest posts