2.24.22 UPDATE: We've been listening to your feedback, and we've made some design changes. We're releasing a new version that addresses the most common themes and makes it easier to achieve the previous experience. Please read the latest blog post for more details.
2.15.22 UPDATE: We're actively reviewing feedback and are aware of the difficulties some users are experiencing with the recent update. We working as quickly as possible to resolve these issues. We will share an update soon.
Additional information can be found in the FAQ updated on 2.1.22.
- Team OneDrive
In June 2021, we announced several important updates for OneDrive on macOS, including an update to our Files On-Demand experience. This new experience is better integrated with macOS and will also help enable new features like Known Folder Move for macOS.
Today, we are excited to share that we have begun rolling out the new Files On-Demand experience to all our customers using macOS 12.1 or later. We also want to share some additional details about how the new Files On-Demand experience works, what changes you can expect, and when you can expect them.
We know that many of you are supporting organizations with lots of Macs that run OneDrive, and the more information we can provide you, the better you can serve your users.
Why we are building a new experience
In 2018, we shipped the first version of Files On-Demand for macOS. Since then, we’ve rolled it out for everyone using macOS 10.14 (Mojave) or later.
We are building a new experience for several reasons. One of the most important is that the new technology stack (based on Apple’s File Provider platform) is much better integrated with the operating system compared to the first version. This means a better user experience, better application compatibility, and better reliability. This technology stack also enables us to offer new features that we couldn’t offer before, like Known Folder Move, along with lots of other little improvements that you probably won’t notice right away!
Because the new experience is more integrated with macOS, it will have long-term support from Apple. The first version of Files On-Demand is built on several pieces of technology that are now deprecated. Moving to the new platform enables us to support this feature for years to come.
Supported macOS versions
For users who are not part of our Insiders program, the new Files On-Demand experience requires macOS 12.1 or later. Users on our Insiders program can continue to use macOS 12.0, but we strongly encourage them to update to the latest version.
macOS 12.2 will be the last version that supports the classic Files On-Demand experience. For macOS 12.3 or later, this means:
- Files On-Demand will default to on for all users and cannot be disabled.
- Devices will migrate automatically to the new Files On-Demand as soon as they receive a macOS update. You cannot delay this update without also delaying an update to macOS.
- Both our Standalone and App Store versions of OneDrive will have the same behavior.
- Users running a developer or beta version of macOS will have the same experience as a release version of macOS.
If you are concerned about application compatibility with this change, we strongly suggest that you install macOS 12.3 or later and test your workload. Depending on the results of that test, you may want to delay updating your Macs.
File system requirements
The new Files On-Demand experience requires a volume that is formatted with APFS. HFS+ volumes are not supported.
We are ending support for HFS+ after macOS 12.3 or later. A very small minority of customers are syncing OneDrive on HFS+ volumes today. As we roll out the new Files On-Demand, these users will first experience a warning in the OneDrive activity center telling them to upgrade to APFS. Once the new Files On-Demand experience is fully rolled out, OneDrive will not launch until the volume is upgraded to APFS.
You can use Disk Utility (built in to macOS) to determine which volumes are running APFS, and upgrade any HFS+ volumes to APFS.
Sync root location
When users set up OneDrive, they must choose a location where files will sync. This is called a sync root. Historically, we’ve allowed users to choose any location on any fixed volume mounted on their Mac.
With the new Files On-Demand experience, the sync root is always located within users’ home directory, in a path such as:
~/Library/CloudStorage/OneDrive-Personal
As part of the upgrade, the sync root will be moved to this location. This location cannot be moved or changed and is controlled by macOS.
This path is a little cumbersome for users to use, so they can access this directory in two other ways:
- Under Locations on the Finder’s left navigation pane.
- Via a symlink at the original location the user picked when setting up OneDrive. For example, if the user chose to sync OneDrive at ~/OneDrive, then a symlink will be created from here to ~/Library/CloudStorage/OneDrive-Personal.
Cache path
To support the new experience, OneDrive maintains a cache path in a hidden location. This path contains a replica of the file tree that the user is syncing. Most of the files in this location are usually dataless and don’t consume disk space, but occasionally files here can have data, such as if a user pins a file or if a change is being transferred to or from the cloud.
OneDrive tries to maintain as little data here as possible, and instead prefers to keep data in the sync root. As such, file data is not generally kept in both locations unless a file is marked as “Always Available on This Device.” In that case, the file’s data will sometimes be retained in both the sync root and the cache, but the files will be linked using a clone, so they do not occupy any additional space.
Using another volume
Sometimes, users choose a path on another volume to set up OneDrive. A typical use case for this happens when a user has a small internal drive on their Mac, but also has a larger external drive attached.
This configuration is still supported in the new Files On-Demand experience if an external drive is selected during the first-run experience. A few things change as a result:
- The sync root remains in ~/Library/CloudStorage, on the user’s home volume. As noted above, this path cannot be moved from this location.
- The cache path is on the volume that was selected during the first-run experience. This is located in a hidden folder that’s a sibling of the location that was chosen.
- This folder begins with the name “.ODContainer”.
- A symlink is created from the chosen location to ~/Library/CloudStorage.
For example, if the user selects /Volumes/BigDrive/OneDrive for their OneDrive path:
- The sync root will remain in ~/Library/CloudStorage/OneDrive-Personal
- The cache path will be set up at /Volumes/BigDrive/.ODContainer-OneDrive
- A symlink will link from /Volumes/BigDrive/OneDrive to ~/Library/CloudStorage/OneDrive-Personal
Because the cache path is located on an external drive in this scenario, any pinned content will be stored there and not on the main drive.
The cache path folder is hidden by default. Users should not modify this folder or its contents.
User consent
For OneDrive to complete setup with the new File Provider platform, the user must consent to allow OneDrive to sync. This experience is like the experience of allowing an application access to the Documents folder or the user’s Contacts.
Consent is not required in the following cases:
- If the user previously opted-in to use the Finder Sync extension. This is set by default for the Standalone build, and the vast majority of our App Store users have opted-in as well.
- If the OneDrive app was deployed and managed through an MDM tool. MDM-managed applications are considered to have implied consent by the administrator.
If consent is required, the user will be prompted to provide it during the first run experience when setting up OneDrive for the first time.
The user can withdraw consent from the System Preferences -> Extensions preference pane. If consent is withdrawn, OneDrive will display an error dialog, an error in the Activity Center, and an error icon, until the user provides consent again. OneDrive cannot run without consent.
Always Keep on This Device
A standard feature of Files On-Demand on all our platforms is the ability to mark files as “Always Keep on This Device.” Internally, we call this operation “pinning.”
When a file is pinned, it is downloaded to disk and is always available offline, even if there is no network connection. The presence of the check mark icon indicates that a file is in this state. Folders can also be pinned, which means that all files and folders underneath the folder will inherit the state, and new files added to that folder will also inherit the state.
Pinning a file on the new Files On-Demand platform means that its contents will be downloaded into the OneDrive cache. Because is the file is in the OneDrive cache, it can always be provided to the sync root whenever it is needed, even if the machine is offline or the OneDrive app isn’t running. The presence of the gray check mark indicates a file that is in this state.
You may notice that pinned files sometimes have an icon next to them that indicates they aren’t downloaded. This icon just means that the file isn’t in the sync root. If a file has the gray check icon, it is still always available because OneDrive has the file in its cache and can always provide it.
Free Up Space
When you no longer need a file on your Mac, you can use the “Free Up Space” option to immediately evict its data. When you do this, data is evicted from both the sync root and OneDrive’s cache, ensuring it occupies no bytes on disk. However, it is still available in the cloud.
Disk space usage
Files that are kept in the sync root do not count against disk space usage, unless they are marked as “Always Keep on This Device.”
For example, imagine the sync root contains five files, each 20mb in size, for a total of 100mb. These files are fully in-sync with the cloud. Now imagine another application asks about the amount of free space on the drive. These five files do not count against the disk space used, so the size reported to the application will be 100mb larger than you might expect.
The reason for this behavior is that in low disk space situations, these files can be automatically evicted from the disk to make room for more data. For instance, imagine an application wants to write 50mb of data to disk, but there is no more disk space. However, because the five files in the sync root can be evicted as needed, the write can safely complete. To do that, three of these five files will be evicted to make 60mb of space, and so the 50mb write completes.
This behavior has several implications:
- Files that have data in the sync root can be evicted at any time.
- The system will automatically clean up files as disk space runs low.
- Mark files as “Always Available on This Device” if you do not want them to be evicted.
We know that some organizations have scripts or something similar to automatically free up space for OneDrive content, usually at login or on a set schedule. Because macOS will automatically free space from OneDrive files as needed, such scripts are no longer necessary, unless you want to prevent users from keeping content in the “Always Keep on This Device” state.
File system feature support
The new Files On-Demand experience supports some existing features of APFS that were previously poorly supported by OneDrive. These include:
- File tags
- Last used date
- File system flags
- Extended attributes
- Type and creator code
- Symlinks
Note that changes to these properties do not sync to or from the cloud, but OneDrive will preserve them on the local file system. Previously, they might only have been preserved for a short while but overwritten by a change from the cloud.
Symlinks have special support in the new experience. They are preserved as a symlink in the sync root but do not sync to the cloud as a symlink, as the OneDrive cloud does not support symlinks. Instead, the symlink will sync to the cloud as a plain text file with the symlink target as its contents. Previously, OneDrive ignored symlinks.
Packages
OneDrive now supports syncing packages, or files that appear as a single file but are actually a directory with many files and folders underneath them. Some applications exclusively create packages. Additionally, most Mac applications are stored on disk as a package.
Traditionally, the problem with syncing packages has been that packages often contain file states that don’t sync well in the cloud. For example, some packages contain internal symlinks, extended attributes, or other file system quirks that can result in a corrupt package if these are not synced correctly. The OneDrive app itself is an example of such a package – previously, if you saved the OneDrive app in OneDrive and attempted to open it on another Mac, it would be corrupt.
With the new experience, packages are now synced as a single file with a hidden .pkgf extension appended automatically. For instance, if you create a file named “Foo.app” in your OneDrive, it will sync to the cloud as “Foo.app.pkgf”. OneDrive automatically strips the .pkgf extension on compatible Macs, and the file will appear as a valid package on all compatible Macs.
Note that Macs not running the new Files On-Demand experience cannot read files in the .pkgf format.
Unlink, unmount, and reset
When you unlink your Mac or unmount a syncing location, OneDrive will preserve the non-dataless contents of your sync root. This works by removing the symlink to ~/Library/CloudStorage, creating a new folder in its place, and moving the files in your sync root that are not dataless to that location. Files in the OneDrive cache path are removed.
OneDrive also ships with a reset script included in the application bundle. This script behaves in a similar way, except that the non-dataless files are always moved into a folder in the user’s home directory named something like “OneDrive (Archive).” Files in the cache path are removed, except if the cache path is located on a volume other than the home volume.
Over time, we expect to improve this experience.
Learning more
To learn more about OneDrive,
- You can stay up-to-date on all things via the OneDrive Blog and the OneDrive release notes.
- Check out the new and updated OneDrive documentation.
- Take advantage of end-user training resources on our Office support center.
Thanks for reading!
Jack Nichols
Principal Software Architect - OneDrive
Update 2/1/2022
Hi everyone - Jack from Microsoft here. Just to quickly introduce myself, I'm the author of the original blog post, and also the architect for the OneDrive sync client. I'm the engineer who led the teams that designed and built Files On-Demand for Windows, macOS, and now the new macOS experience, so I have the most context about how Files On-Demand works and the trade-offs involved in building something like this.
The entire OneDrive team has been reading your comments, concerns and feedback, and we really appreciate everyone taking the time to write them. The community clearly has a lot of passion for OneDrive and how it works. I've spent many hours in the last week or two reading comments here and elsewhere, to understand how we can improve our macOS experience further.
Although we can't respond to all of you directly, there are a couple of themes and frequently asked questions that I wanted to answer to help provide some more clarity.
Why are all my files redownloading with this update? Why are my always-available files displaying a "not downloaded" icon?
Let me first set you at ease: your files aren't actually redownloading. What you are seeing is a bit of an optical illusion.
When your OneDrive instance is upgraded to the new Files On-Demand, macOS creates a new folder for your OneDrive files and we move your old folder into our cache location. We do it this way for many reasons, but two of the most important are that we can preserve your settings around which files are always available, and we can prevent the sync client from performing a costly reindex of all of your content.
As your files are brought into our cache, we tell the macOS File Provider platform about them. That causes the operating system to create the files in the new OneDrive folder that you will actually use. As part of telling the File Provider platform about your files, we include metadata about them, so that the operating system knows how big they are, what icons to show, and so forth.
Unfortunately, the current implementation of File Provider does not allow us to tell the operating system that we already have the file's contents available – so they appear to be online-only, even though their contents are safe in our cache, ready for the first time you access them. The best that we can do is tell the system to show the always available icon (the checkmark), but we can't tell the system to hide the "not downloaded" icon. The "not downloaded" icon is shown automatically by the File Provider system when the file is dataless in the sync root, and there's no current way for OneDrive to override this. Please know that we are actively investigating ways to address this, as we understand that it is a top source (if not the top source) of user confusion with this update.
The key thing to remember here is that if you double-click the files that we already have in our cache (files that you pinned when you selected “Always Keep On This Device” and anything you had downloaded before we did the upgrade), they will be retrieved and opened as expected, without any network traffic. This will work even if OneDrive isn't running, is paused, and so forth.
Why were my Finder favorite folders removed?
During the upgrade to the File Provider platform, OneDrive removes these favorites as they no longer point to a valid location. Most users will have a "OneDrive" favorite that will be removed in this manner, but a few users have dragged other folders of interest to this sidebar, which will also be removed if they were pointing at a OneDrive folder.
After the upgrade, if you want these favorite folders back, you will need to add them again by dragging your favorite folders to the Finder's sidebar.
How can I make it so that all my files are synced on my Mac and made available for offline access?
If you want all files synced on your device, you should pin the OneDrive folder. The easiest way to do this is to browse to your OneDrive in the Finder, change the view to Icons view, and then right-click the blank space between icons. Then, select Always Keep on This Device.. We're actively looking at ways to make this easier to configure on both macOS and Windows.
Is there a technical reason that explains why Files On-Demand must always be enabled?
OneDrive has taken a dependency on Apple's File Provider platform as part of this update, as we believe it is the right long-term path forward for the product. Files On-Demand functionality is a core part of Apple's File Provider platform, but File Provider offers a lot more than that, too. I'll touch on a few of those things here below.
- For instance, the little icons you see next to your files in OneDrive for macOS are now handled by the File Provider platform. This seems like a small thing, but it has a big impact. Before File Provider, we used something called a Finder Sync extension to show these icons in the Finder, but the Finder Sync extension was one of the top sources of problems on the macOS sync client. For example, the icons sometimes mysteriously disappeared, or performance problems affected the system. Because we eliminated the Finder Sync extension, we also eliminated an entire class of problems as a result.
- This has also improved the reliability of OneDrive running on macOS. As part of our normal sync process, the sync client occasionally runs checks to ensure everything is syncing correctly. The results of those checks are reported to us as telemetry which we use to help ensure there aren't emerging bugs, and most of the time, we find and fix bugs before anyone notices them. We've been very closely monitoring this telemetry as the new macOS Files On-Demand experience has rolled out, and we’ve noticed is that reliability is significantly better than what we had before. This translates to a much better sync experience for you.
- Finally, it is important to note that beginning in macOS 12.3, File Provider is the only Files On-Demand solution that is supported on macOS. Our prior solution is no longer supported.
Files On-Demand has been available on Windows since 2017, and on macOS since 2018. In that time, we've progressed from the feature being opt-in only to being on by default for all users and have closely monitored how many users turn off Files On-Demand. Only a very small number of users disable Files On-Demand on both platforms, and there are two main reasons for that.
- Application compatibility: When Files On-Demand first shipped on Windows, some applications didn't work well with the way we stored files, or with anti-virus or other security software that was installed. Over time, we've fixed most of these problems. On macOS, we took a similarly cautious approach, but the application compatibility landscape is quite different and, in some ways, less complex. Still, there were a few cases where, due to the technology stack we were using on macOS, it made sense for certain users to disable Files On-Demand to preserve compatibility. With the File Provider platform, these problems have gone away, so application compatibility issues on macOS should be much less likely to occur. If you find something different with your setup, please reach out to your support contact so we can diagnose the issue.
- Locally available files: We know that keeping all content locally on the device is an important scenario for a small set of users. The best way to do this is to select Always Keep on This Device from the right-click menu to mark content as “pinned”.
Note that this applies to folders too; if you pin a folder, all of the content that's currently in it and new content that is added to it will be kept on the device.
Why is it sometimes slow to browse folders in my OneDrive?
To save space and system resources, the File Provider platform doesn't actually create the files OneDrive is managing until the first time you need them. The first time you open a OneDrive folder, macOS will create them on-demand. This can sometimes take a moment.
To avoid this delay, you can force the system to pre-create all of these files and folders for you without downloading your content. To do this, open a Terminal window and type "ls -alR ~/OneDrive" (or the path to your OneDrive). This will ensure all of your files and folders are created, but not downloaded, before you browse.
Can OneDrive be stored on an external drive? How does pinning a file work when I use an external drive? Are there multiple copies of my data?
I've seen several threads on this topic but let me clarify with an emphatic yes: external drives are fully supported without any difference in the end-user experience.
That said, external drive support as it exists today is implemented differently than it was in the past because of how File Provider works. Very few users are running this configuration, but for them, it's an important scenario because often their content won't fit on the home drive. File Provider doesn't support creating the sync root on any drive except for the home drive. So, we had to find a way to support external drives within these constraints.
When you choose a path to sync your OneDrive, we use that path to derive where we put your OneDrive cache path. If that path is on an external drive, we'll put the cache path there. We wanted to honor this preference because the cache path is where your pinned content is stored, as I'll explain below.
When your cache path is placed on an external drive, OneDrive tries to minimize the number of copies of your data it makes, and in most cases, only one copy will exist, usually in the sync root. If your home drive runs into disk pressure, the operating system will evict (dataless) files from the sync root, but they can always be obtained again from the cloud if needed. In some cases a file might exist in both places for a short time, but over time we will ship fixes that will optimize this further.
Pinned files on an external drive have behavior quirks that are worth understanding. If you pin a file, it will download to the cache path only, and will show both the checkbox and "not downloaded" state icons. This is because the file is dataless in the sync root but exists as a full file in the cache path. However, if you pin a file and also double-click it to open it, we will bring it into your sync root, so there are two copies, one in each location. Note that files brought into your sync root in this manner can still be evicted by macOS when it encounters disk pressure, but when this happens, only the file in the sync root is evicted. We still keep the data in your cache path, so you can always get to the file's content, even if you are offline.
The table below depicts how this works when you set up sync on external drives:
|
User action |
File in sync root |
File in cache path |
|
Default state |
Dataless |
Dataless |
|
Right click -> "Always Keep on This Device" |
Dataless |
Has data |
|
Double click the file |
Has data |
Has data |
|
macOS runs into disk pressure |
Dataless |
Has data |
|
Right click -> "Free Up Space" |
Dataless |
Dataless |
Essentially, the table depicts the guarantee that OneDrive makes about pinned files, namely that as long as we have a pinned file, we'll always keep the data available to you locally. The only time we don't have that data is either in the default state, or if you tell us to free up space for the file.
How does disk space usage work in the sync root?
In the original blog post, I mentioned that files with data in the sync root do not count against your disk usage. Some people took this to mean that these files occupy zero bytes on the disk, but what actually happens is that these files don't count against your used disk space. That is, if an application asks, "How much space is free on this disk?" that answer will exclude these files.
There are a handful of special cases where these rules don't apply:
- Pinned files, if your cache path is on your home drive. In this situation, the file in the cache path and the file in the sync root are Apple File System (APFS)clones of one another, and although there are two files, they share the same space on disk until one changes. File Provider won't evict files that have a clone, and such files will count against used disk space.
- If your cache path is on an external drive, there is no clone, so pinned files can be evicted from your sync root and don't count against used space on your home drive.
- OneDrive designates certain file types as non-evictable, and therefore these files count against used space. The most important of these file types are shortcuts to OneNote files, which only occupy a handful of bytes. This matches the behavior on Windows as well.
The system logic to decide what files count against used disk space and what files do not is provided by the File Provider platform. If you find behavior that works differently than I've described here, please reach out to support, or to Apple.
Will this work with local file indexing (e.g. Apple's Spotlight)?
Yes. Spotlight indexes everything that is in your sync root, but note that Spotlight will not fetch (or hydrate) files that are dataless. If you are looking for something in Spotlight that could only be read from the full file (such as image EXIF data), only fully hydrated files will be indexed.
Spotlight will not index our cache folder.
Why is my AutoSave not working after this upgrade?
We have been made aware of users experiencing issues with AutoSave when using the Store version of the OneDrive app. We are actively working to resolve this in the next few days. In the interim, if you want to get unblocked, you can move to the Standalone build.
To move to the Standalone build from the Store version, you can unlink your account, uninstall the App Store version, and reinstall the Standalone version from this link: https://go.microsoft.com/fwlink/?linkid=823060.
We will keep you updated about the fix on this thread.
Update 2/3: The fix for AutoSave for the App Store version was released in the Store today. It is fixed with 22.002.0201.0005.
If you still have more queries feel free to reach out to the team directly on this discussion thread. Files On-Demand for macOS QA - Microsoft Tech Community
Thank you for your constant support and partnership!
Jack