MacInTouch Amazon link...

APFS / file systems

Channels
Apple, Security, Other, Products
Very interesting. I wonder what is going to happen in macOS 10.15, when the 32-bit APIs (including all of Carbon, where the Resource Manager lives) get deleted. They're going to need to do something about text clippings.

I can imagine a few possibilities:
  • Make a Cocoa port of the Resource Manager (or at least a basic subset of it), so the necessary APIs will remain available in the 64-bit world. This would be my preference, but it also seems unlikely. I can only see them doing this if there are a lot of different standard file types that all depend on resources.
  • Migrate the parts of the resource manager needed for text clippings into the text clipping code itself, so the text clipping library will open the resource fork as an opaque block of data and manually read/write the resources contained within - a hack, but an acceptable one if there aren't any other standard file types that require resources.
  • Change the format of text clipping files to something that doesn't use resources, converting all existing clippings as a part of the OS upgrade - probably the best for most users but will really tick off people who run multiple macOS versions and share clipping files among the systems. (Side question: does anyone actually do this?)
  • Change the format of text clipping files and don't auto-convert existing clippings, but keep an importer module that can read the old format for the next few OS releases, dropping support in the future. Maybe offer to upgrade clippings as they are used as well.
  • Change the format of text clipping files, don't convert anything, and let old clipping files break (the fastest, easiest and cheapest option, even though it is the most user-unfriendly).
I wonder if there will be a third-party (open source?) project to create a resource manager for 64-bit code. I assume the resource fork organization is well documented and it's not likely to change, so there's no technical reason why it couldn't be done. If the developers consist entirely of people who have never seen Apple's source code, working only from public information, there shouldn't be a copyright problem.
According to Howard Oakley's xattred, a .textclipping file resource fork appears to be stored as an extended attribute, at least in Mojave. The ls command in Terminal corroborates this:

ls -al@ Desktop/28*
-rw-r--r--@ 1 xz4gb8 staff 1081 May 1 10:47 Desktop/28 https-::eclecticlight.co.textClipping
com.apple.FinderInfo 32
com.apple.ResourceFork 1362
com.apple.lastuseddate#PS 16


It is also fascinating the BBEdit displays only the clipped text while Text Edit displays complex text content and Hex Fiend shows yet another view of the complex content.
 


Ric Ford

MacInTouch
According to Howard Oakley's xattred, a .textclipping file resource fork appears to be stored as an extended attribute, at least in Mojave.
This is very weird stuff - resource forks that aren't resource forks but are sort of virtual resource forks and are stored... where? Howard Oakley talks more about this whole mysterious, mostly undocumented mess:
Eclectic Light Co. said:
Documenting the hidden

... As I have pointed out before, extended attributes had a glorious, and very public, past in the ‘resources’ used in classic releases of Mac OS. In early versions of Mac OS X it was touch and go whether they would survive at all, but somehow the dark forces who wanted a pure Unix didn’t rule the day, and xattrs have flourished like mushrooms growing in a cavern.

Until this week, I had no idea of their prevalence in Sierra and High Sierra. I had assumed that traditional ‘resource forks’, now extended attributes of type com.apple.ResourceFork, had largely fallen into disuse. But they are still being employed quite widely, and more than half of all the files in my Home folder have at least one extended attribute attached to them.

... Overall, of the 1 TB of data stored on the Fusion Drive in my main working iMac, over 1.1 million of the 4.7 million files contain at least one xattr. Yet not one commonly-used app lets me even know of their existence, let alone about the information contained within them.

An even greater concern to me is that there appears to be no accessible resource which lists their types, or divulges what is contained within xattrs, or what they are used for. Many of the more than 150 different types of xattr that I have found have been established by Apple, which does provide a few small glimpses into some. Even the commonplace com.apple.quarantine and com.apple.TextEncoding appear to be completely undocumented.
There's a related command listed in Apple's "man" pages (e.g. man xattr in Terminal.app):
NAME
xattr -- display and manipulate extended attributes

DESCRIPTION
The xattr command can be used to display, modify or remove the extended
attributes of one or more files, including directories and symbolic
links. Extended attributes are arbitrary metadata stored with a file,
but separate from the filesystem attributes (such as modification time or
file size). The metadata is often a null-terminated UTF-8 string, but
can also be arbitrary binary data. ...
 


Eclectic Light Co. said:
…not one commonly-used app lets me even know of [extended attributes'] existence, let alone about the information contained within them.
The Finder can show one extended attribute. Get Info on a file downloaded with Safari (and later Firefox and at least one Firefox fork). Under "More Info" you can see "Where from," and two URLs: the URL of the downloaded file and the URL of the page from which it was downloaded. These are stored (usually in a plist) in an attribute called com.apple.metadata:kMDItemWhereFroms.
 


This is very weird stuff - resource forks that aren't resource forks but are sort of virtual resource forks and are stored... where?
I believe it is the other way around.

HFS+ has some odd types of legacy data: Finder info and Resource Forks, that really are stored in the file system. But now this info is also mapped into fake extended attributes: com.apple.FinderInfo and com.apple.ResourceFork. This means that commands, users, and applications can use the modern extended attribute techniques for reading this data.

(On APFS maybe the Finder Info really is stored in the extended attribute. You'd need a tool that can look at the raw file system data to know for sure.)
 


It is also fascinating the BBEdit displays only the clipped text while Text Edit displays complex text content and Hex Fiend shows yet another view of the complex content.
If you look at the contents of a textclipping with the DeRez command (should be installed as a part of Xcode), you will see that it contains many different representations of the clipped text, each as a different resource type (which appears to match the types that are used in the clipboard API).

There is TEXT (plain 8-bit text), UTF-8 text, Unicode 16-bit text, HTML representation, and various other text+formatting representations.

I assume that dropping a textclipping on an application reads all this data into the clipboard and then sends a paste event to the app, in which case it will work the way the clipboard normally works - the app will query for the list of available formats and pick one (usually the most robust representation it supports, unless the user requests otherwise) and paste that data (possibly after some conversion) where you dropped it.
 


This is very weird stuff - resource forks that aren't resource forks but are sort of virtual resource forks and are stored... where? Howard Oakley talks more about this whole mysterious, mostly undocumented mess:
As we all know, HFS Standard was introduced in the 80's. HFS+ came to exist later, and was augmented by Apple with additional btrees, and more deeply embedded changes to support extended attributes, OS interoperability, etc. But the basics of the file system remained the same. So HFS and all of its versions evolved over three decades to support the ever-changing tech world.

With the introduction of APFS, Apple engineers integrated all of the previous "add-ons" directly into the brand-new design. All "normal" data for files and folders, metadata, attributes, security, allocation, quotas, reference counts, etc., generally all reside in one place, or at least a reference to them. (Snapshots are not part of normal data and they have their own world in which to reside.)

APFS is complex, and this is all a bit of an over-simplification, though my point is that all of the centralization of file and folder information will make such information easier to locate and view, especially as existing tools are improved and new tools are introduced. We are still in the dawn of APFS as consumers. Obviously, Apple engineers have been working on it for a long time to get it to this point.
 


What still worries me personally about APFS is that we still have no 3rd-party repair tools and I have been informed by one of the developers that this is still due to lack of proper documentation from Apple, so it appears that it's still undergoing development within Apple. I still don't want to be a beta-tester by adopting Mojave or further systems yet.
 


What still worries me personally about APFS is that we still have no 3rd party repair tools and I have been informed by one of the developers that this is still due to lack of proper documentation from Apple, so it appears that it's still undergoing development within Apple. I still don't want to be a beta-tester by adopting Mojave or further systems yet.
If I were to impute beta status to APFS in macOS, I would perhaps describe APFS in macOS 10.12 Sierra that way. But macOS 10.14 Mojave would be described as an early production release at worst. I understand that APFS had been in production use on millions of [iOS] devices prior to Sierra. Ongoing development does not implicitly mean a product is still in beta or not suitable for use.

Speaking of Mojave in particular and as a regular participant in macOS beta testing, problems I have encountered have been primarily in user interface testing and event scheduling but not in any file systems. I have found no problems attributed to APFS while updating any system for myself or my clients. I won't say no problems were found - for example, the CanoScan LiDE 600F flatbed scanner no longer works in Mojave with any software I have yet found, and the removal of Back To My Mac has required workarounds, which are by no means trivial.

Third-party repair tools for APFS should not be a barrier to Mojave acceptance. Time Machine, along with other third-party tools, such as Carbon Copy Cloner, can provide recovery from data loss due to either hardware, software, or human errors. Using any computing device (macOS, iOS, Windows) without backups renders moot any worry regarding failures, include any from APFS.
 


Rusty Little's comments about this being "the dawn of APFS" do not provide any warm fuzzies about Apple's efforts. Am I the only one who sees APFS as betaware and Mac users as beta-testers? Perhaps the well-discussed workaround of cloning the boot drive to an HFS+ volume is an option when I upgrade to Mojave in September.
 


Rusty Little's comments about this being "the dawn of APFS" do not provide any warm fuzzies about Apple's efforts. Am I the only one who sees APFS as betaware and Mac users as beta-testers? Perhaps the well-discussed workaround of cloning the boot drive to an HFS+ volume is an option when I upgrade to Mojave in September.
I referred to it being "the dawn" for consumers and users of APFS, not the dawn of Apple development.

We are working on the APFS-savvy version of our software. (I interrupted my APFS work momentarily to write this reply), as I am sure other developers are doing, as well. So, in the coming months, and even years, more and more APFS software, tools, etc. will become available to consumers. The ability to peek inside the file system, see stats, and understand how it all works will be easier as more software becomes available.

I would not say APFS is betaware. I would only say there is a much more complete set of tools and utilities for consumers to use with HFS+ than APFS, simply due to time.
 


Ric Ford

MacInTouch
Third-party repair tools for APFS should not be a barrier to Mojave acceptance. Time Machine, along with other third-party tools, such as Carbon Copy Cloner, can provide recovery from data loss due to either hardware, software, or human errors.
That's true to a limited extent, but unnoticed data corruption can easily propagate to backups, destroying good data.

That said, I don't personally know of any data corruption issues with APFS, apart from destructive Thunderbolt bugs that may be unrelated to the file system. APFS has some extra features to help ensure data integrity and security, although it also has more complexity and instability that could potentially contribute to problems. (And neither HFS+ nor APFS has anything like the data integrity of ZFS.)
 


Speaking of Mojave in particular and as a regular participant in macOS beta testing, problems I have encountered have been primarily in user interface testing and event scheduling but not in any file systems. I have found no problems attributed to APFS while updating any system for myself or my clients.
APFS has a bug that keeps Quicken 2007 from completing backups.
 



How did you determine the bug was in APFS and not Quicken 2007? It could be a design difference between HFS+ and APFS. The postings I have found did not elaborate on this, only that the combination did not work.
I'm not sure, but I think the issue is that APFS doesn't support the Snow Leopard HFS+ file-level compression feature.

Rumor is that it was dropped because on modern large-capacity SSD drives, it is faster to store and read the uncompressed data than to compress and decompress.

But that means that any application that was using this HFS+ feature would break in Mojave, even on a spinning disk drive where compression would be an improvement.

In my opinion, any failure to maintain HFS+ compatibility that breaks ordinary applications is a bug in the new file system implementation.
 


How did you determine the bug was in APFS and not Quicken 2007? It could be a design difference between HFS+ and APFS. The postings I have found did not elaborate on this, only that the combination did not work.
I can see where Michael is coming from:
  • works under non-APFS
  • doesn't work under APFS
Difference: APFS

Obviously, there may be additional differences (different OS versions too?) that might also come into play, but from a user perspective ("it worked before this change, now it doesn't"), it's reasonable to assume the difference(s) to be the likely cause of the problem.
 


Perhaps this is a related issue: the wonderful free app called EasyFind is horribly slow with APFS. This was tested on a 2016 MacBook Pro with a 2TB SSD. It could be that Mojave impacts the speed of EasyFind — some sort of compatibility problem? — but all my previous experiences (on non-APFS systems) with EasyFind are positive. It has proven to dash through hundreds of thousands of files in just a few seconds, at least on HFS+ formatted drives. I gave up waiting for the app to finish its disk-wide search on the APFS laptop after a few minutes of watching it crawl.
 


What still worries me personally about APFS is that we still have no 3rd-party repair tools
I won't comment on APFS, since my two main Macs remain on HFS+, but I will say that one thing that Apple seems to have gotten right in the last fews years is HFS+.

I've always maintained a fairly complete stable of 3rd-party disk repair utilities, dating all the way back to classic Mac floppy repair tools, and they've generally gotten a fair amount of use. However, since at least the mid-Sierra releases, I haven't had much call to reach for them.

I can only remember needing to run a repair utility on one of my own Macs once in the last couple of years, whereas I used to need repair tools on my own machines at least once every month or two. I still keep my utility licenses current, but it does seem like Apple is doing something right with HFS+. (Note: I don't have hard statistics to prove that, only a clear change in my personal experiences.)

My guess is that the demands of supporting gigantic, complicated Time Machine volumes forced Apple to invest real resources into HFS+ filesystem management.
 


I referred to it being "the dawn" for consumers and users of APFS, not the dawn of Apple development.
We are working on the APFS-savvy version of our software. (I interrupted my APFS work momentarily to write this reply), as I am sure other developers are doing, as well. So, in the coming months, and even years, more and more APFS software, tools, etc. will become available to consumers. The ability to peek inside the file system, see stats, and understand how it all works will be easier as more software becomes available.
I would not say APFS is betaware. I would only say there is a much more complete set of tools and utilities for consumers to use with HFS+ than APFS, simply due to time.
Rusty, I do not mean to diss your efforts and those of the cadre of developers who are trying to get something shipped that is safe, effective, and understandable to users. My point is that we (users) are 19 months into APFS and the products are...? Again, not blaming the developers, but I do blame Apple for not having APFS fully fleshed out for you by the time High Sierra shipped. If you think I'm expecting too much, let's think back to the introduction of the original Macintosh. Important apps were ready to go on Day 1.

I'll wait for DiskWarrior for APFS. Then I'll know that a responsible developer has mastered APFS and maybe, just maybe, APFS will be ready for prime time (in my opinion).
 


Perhaps this is a related issue: the wonderful free app called EasyFind is horribly slow with APFS. This was tested on a 2016 MacBook Pro with a 2TB SSD. It could be that Mojave impacts the speed of EasyFind — some sort of compatibility problem? — but all my previous experiences (on non-APFS systems) with EasyFind are positive. It has proven to dash through hundreds of thousands of files in just a few seconds, at least on HFS+ formatted drives. I gave up waiting for the app to finish its disk-wide search on the APFS laptop after a few minutes of watching it crawl.
Unfortunately the slowdown is related to APFS. The author of Find Any File, Thomas Tempelmann, has detailed info about the slowdowns on blog posts:
 


I recently ran Disk Utility's First Aid routine on an external flash drive used for backup, and this warning was displayed:
Apple Disk Utility said:
Verifying allocated space.
warning: Overallocation Detected on Main device: (1 52935086+1) bitmap address (815c)
The volume /dev/disk2s2 appears to be OK.
Can Rusty (of Alsoft) or someone else let me know what this means and whether I need to do something about it?
 


Rusty, I do not mean to diss your efforts and those of the cadre of developers who are trying to get something shipped that is safe, effective, and understandable to users. My point is that we (users) are 19 months into APFS and the products are...?
19 months after HFS was introduced where were the mature file system products? 19 months after HFS+? (The detailed docs on those didn't show up immediately upon release either.) Part of this is expectations built on not having to deal with the situation for decade(s).
Again, not blaming the developers, but I do blame Apple for not having APFS fully fleshed out for you by the time High Sierra shipped. If you think I'm expecting too much, let's think back to the introduction of the original Macintosh. Important apps were ready to go on Day 1.
The important "file system fix" utility in the context of APFS would be fsck_apfs. I have yet to see any detailed examination that it is leaving tons of "low hanging fruit" for 3rd-party systems to 'fix' that it does not. APFS has substantially more resilient metadata protections, 'back ups', and 'dirty' write failure protections....

So if fsck_apfs is skimming off all the low-hanging fruit repairs, the remainder that is left for the rest is a smaller and more difficult set. It is a substantively different scope and problem.

Also, if APFS holds onto less stale/dirty metadata, and the storage can commit data quicker, the mangled metadata error rate will probably go down. As the corner cases get narrower (and likely more complicated), the problem scope isn't the same.
 


Unfortunately the slowdown is related to APFS. The author of Find Any File, Thomas Tempelmann, has detailed info about the slowdowns on blog posts:
APFS isn't the principal cause. The second article above points to these applications using a Carbon API call to get most of the work done. Apple deprecated Carbon more than several years ago. The first article eventually [notes] that there is a BSD/Posix interface that is being actively integrated with APFS and also notes that attribute access got better from macOS 10.12 -> 10.13.
 


APFS isn't the principal cause. The second article above points to these applications using a Carbon API call to get most of the work done. Apple deprecated Carbon more than several years ago. The first article eventually [notes] that there is a BSD/Posix interface that is being actively integrated with APFS and also notes that attribute access got better from macOS 10.12 -> 10.13.
I have not used Find Any File for a while, I switched to HoudahSpot. I can confirm this that HoudahSpot is super fast at finding things. So, it does not seem to have any problem with APFS.
 


Ric Ford

MacInTouch
I have not used Find Any File for a while, I switched to HoudahSpot. I can confirm this that HoudahSpot is super fast at finding things. So, it does not seem to have any problem with APFS.
I looked but couldn't quickly find the answer: Is HoudahSpot dependent on Spotlight for its searching? (Find Any File is not.)
 



I looked but couldn't quickly find the answer: Is HoudahSpot dependent on Spotlight for its searching? (Find Any File is not.)
HoudahSpot 5 User Guide says,
3. HoudahSpot and the Spotlight Index

HoudahSpot builds upon Spotlight, which comes preinstalled with macOS. With HoudahSpot, there is no need to build an additional index.

The Spotlight index gets its information from Spotlight importer plug-ins. For standard file formats, these plug-ins are provided by Apple. Third-party developers need to provide their own Spotlight plug-ins if they want their proprietary file formats to be read by Spotlight. Developers are free to decide what information to share with Spotlight and how to label it….
 


Perhaps this is a related issue: the wonderful free app called EasyFind is horribly slow with APFS. This was tested on a 2016 MacBook Pro with a 2TB SSD. It could be that Mojave impacts the speed of EasyFind — some sort of compatibility problem? — but all my previous experiences (on non-APFS systems) with EasyFind are positive. It has proven to dash through hundreds of thousands of files in just a few seconds, at least on HFS+ formatted drives. I gave up waiting for the app to finish its disk-wide search on the APFS laptop after a few minutes of watching it crawl.
I asked DEVONtechnologies about this. It is slated for upgrading to improve APFS compatibility, but is behind some other things they are working on. That's understandable considering it's freeware. [I think] it still is the best search utility, even if you have to wait.
 


I recently ran Disk Utility's First Aid routine on an external flash drive used for backup, and this warning was displayed:
Apple Disk Utility said:
Verifying allocated space.
warning: Overallocation Detected on Main device: (1 52935086+1) bitmap address (815c)
The volume /dev/disk2s2 appears to be OK.
APFS uses sparse files and lazy allocation. This means that if you allocate a large file, it doesn't actually allocate the entire file at once.

If the sum of your maximum file sizes exceeded the capacity of the volume, that would be an overallocation situation. (Whether that's what this particular error actually means is unknown.)

Perhaps there's enough space on your main drive, but not when you copy to the external flash drive?

As an aside, fun fact: some cheap flash drives have fake capacity. The drive will report it has 64 GB, for example, but actually only has a 4 GB NAND memory chip. So it appears to work at first, but then bad things happen once you exceed the actual capacity.
 


Amazon disclaimer:
As an Amazon Associate I earn from qualifying purchases.

Latest posts