Ugh, what a mess.Howard Oakley (via the Michael Tsai Blog) said:iCloud Drive Breaks the macOS Command Line
Apple’s current engineering solution breaks consistency of file names and paths. When a file has been evicted from local storage, and only exists in full in iCloud storage, the local stub file uses the previous name prefixed with a stop/period, and gains the extension of .icloud. When that file is downloaded to local storage – something which can be triggered by all sorts of events – the leading stop/period and the extension are stripped.
Many commands and scripts can safely ignore files which the user has placed in their iCloud Drive. But the moment that a user enables Desktop & Document Folders to be stored in iCloud, with Optimize Mac Storage enabled, file names in ~/Documents are affected, and commands and scripts will fail when run on one of the most important and active directories on most macOS systems.
Apple’s own most robust tool for locating files, the Finder alias, is broken by iCloud.
Inevitably, all hard and symbolic links made to evicted files are also broken by their eviction.
Faced with the problems posed by iCloud, a lot of commands, shell scripts and other scripting becomes inordinately complex, and in some cases impossible. Apple needs to continue to evolve the iCloud interface, making it consistent with the fundamental needs of commands and shell scripts. If it doesn’t, but continues to converge with iOS, it will undermine macOS itself.
Now that it's been mentioned, I'm not too surprised, but I would have hoped that Apple could implement "Optimize Mac Storage" in a way that doesn't break the command line.
It's definitely possible to do this. I'm thinking that it would be easiest to do it using some kind of network file system like the old Andrew File System. For those who don't know, AFS is/was a network file system designed for wide-area/slow connections. The file system would be mounted, much like any other remote volume, but it would cache the files locally for faster access to commonly-accessed files.
Instead of Apple's existing mechanism (where it replaces offloaded files with a special kind of iCloud alias that causes the file to be re-downloaded when accessed via the Mac-style (but apparently not UNIX-style) APIs), they could have implemented this by replacing your Documents folder with a mount-point for an AFS-like file system, which can download files from iCloud as needed and purge them at later times. You wouldn't be able to hard-link to these files (because it's a different volume), but you could have symlinks to this volume and scripts would continue working because all of the filenames would remain valid (triggering downloads as necessary to access the content).
If anyone from Apple is reading this, please forward this message to whoever is responsible for this feature. There is a better way to do what you're trying to do and Apple should migrate to it as soon as it is practical to do so.
Last edited by a moderator: