When working on our application in Helpling, at some point in the past we migrated from Vagrant to Docker to take advantage of the containerized approach and make updating parts of our stack easier. Recently I decided to research different alternatives to synchronizing the files between a host Mac machine and the containers to improve the performance and CPU use of our development environment.
osxfs
This is the old default way of mounting the local directory inside a Docker container on Mac.
Due to the operating system differences, Docker for Mac cannot run its apps and containers the same way as it does so on Linux. It works this problem around by running a tiny Linux kernel in a virtual machine and then running all the containers there instead. This means we cannot natively mount the host filesystem in the container since it’s a completely separate operating system. To work around this, Docker for Mac developers came up with osxfs.
Unfortunately, this solution is insufficient for anything other than very very simple applications. The moment you run a mature Rails application, this approach completely falls apart – it not only uses quite a lot of CPU, but it’s also painfully slow. We’re talking several minutes to even launch the app and then several seconds for every request – completely unusable!
gRPC FUSE
Knowing the problems of the osxfs approach, Docker developers tried to find an alternative. After a brief trial run with mutagen (more on that later) they decided to instead roll their own solution and gRPC-FUSE was born.
Unfortunately, it turned out the cure was worse than the sickness. gRPC-FUSE is still painfully slow slow but also unreliable, using lots of CPU, randomly failing and keeps getting broken between even patch versions of D4M. Not only that but the main reason it was created was because of reliability and CPU issues of the previous solutions and not to drastically improve the I/O performance so it doesn’t help us much with larger applications either.
NFS
Another common alternative that works without bringing any external tools is using NFS. It’s supported by default by Docker and requires just a bit of extra setup to work.
Usually, it’s also pretty reliable but unfortunately has one big drawback – the disk access time for the app running in the container is still slower than native which causes both the boot time and every request to take more time. As such, it’s fine for moderately sized projects or apps that do not require much I/O but it definitely was not enough for Rails.
The other problem is that it does not propagate any filesystem events, meaning you need to rely on polling to detect changes within your application (spring, Rails autoloader, etc.) which is always slower and requires more CPU.
Mutagen
Like I’ve mentioned before, Docker for Mac developers tried using Mutagen for a while. Mutagen is a fast file synchronization utility that was inspired by Unison and partially Syncthing and tries to require as little configuration as possible. You can either set up synchronization sessions yourself or use their docker-compose integration.
The way it works is that creates a Docker volume and then listens on file changes on the host machine and whenever it detects any file changes, it synchronizes the two. This allows us to get as close to native as possible read-write I/O for the app running inside the container by sacrificing the hard drive space (we store the files twice, once on the host and once on the external server).
Unfortunately, Mutagen is not a silver bullet either. To use docker-compose integration we need to dig into beta versions (beta3 at the time of writing this post) that are not production-ready and are still buggy. Sometimes conflicts get resolved in an unexpected manner, sometimes mutagen also messes up when doing heavy git operations (like rebase) and can in extreme cases ruin your local code copy.
There are also several unresolved issues related to random high CPU usage. I was testing this solution for months before giving up because sometimes my laptop was getting so hot I was not able to hold it on my lap anymore, not to mention other apps kept being affected, slowing my entire environment to a crawl. I still believe mutagen will in the future be the best solution for syncing but it’s simply not there yet.
docker-sync
The last and also very popular solution is docker-sync. It’s a tool that was built back when osxfs was the only option and it was aimed at solving the problem in the same fashion as later Mutagen.
There are several strategies for docker-sync:
- unison – uses Unison for watching file changes and syncs them back to the volume in a similar fashion to how mutagen does it
- native_osx – tries to merge the best of unison and osxfs by mounting the volume as osxfs container and then copying files from there to regular Docker volume with unison; unfortunately, it’s no longer reliable due to Docker for Mac bugs related to switching to gRPC-FUSE
- rsync – super simple strategy that is unusable for anything other than testing – it’s unidirectional only and never deletes any files
Since native_osx is currently unreliable and rsync is not very useful, the best strategy to use is Unison. Unfortunately, it’s not without its pros and cons. The biggest issue is that unison, by default, does not include a way to monitor filesystem changes on Mac (a tool called unison-fsmonitor). There is an implementation of that written in Python called unox that uses Python’s watchdog library but regrettably, it’s pretty slow – for our codebase sometimes it takes up to 5 seconds between the file changes until it’s synced back to the container.
There is an alternative though – we can use another unison-fsmonitor implementation that was written in Rust and as a result is much faster and uses fewer CPU resources than unox. Starting with 0.7.0 version docker-sync will no longer complain if this implementation is installed so you should definitely try it out and see if it also improves your developer experience.
As you can see there is no single good solution when it comes to Docker and Mac, only trade-offs. This is the unfortunate side effect of not being able to run Docker in the same native containerized fashion like on Linux and it does not seem that we’ll ever nail it perfectly (it’s a similar problem as with WSL). Hopefully, this research will help you pick the right choice for your particular set of requirements. Good luck and happy coding!
