Unison, syncing macOS with Ubuntu and the slowness of docker on macOS
Unison
Unison has been my trusty file sync mechanism to sync my source files between my various computers. Its basically a bi-directional rsync. The setup is a SSH account in the cloud as the central repository and all my machines do a sync with it.
This works perfectly for me, maybe because I am only on one machine at any given time, so I am only synching with myself. And I am only syncing manually, at the start of my session and then at the end and rarely inbetween.
Keeping the syncer in sync
Now it’s always been the case that in order for unison to work, that all unison
executable must have the same version. I have been
using apt
on the Ubuntu side and homebrew
on the macOS side and I could
always get it to work. But not this time!
OCaml
Unison is maybe - snide remark - the only useful application ever written in OCaml. But apparently binaries compiled with a certain OCaml compiler version may not be compatible with those of a subsequent ot prior version. (And no the last major version change of OCaml was 2011…)
So if you have two Unison 2.84.4 binaries, they may still be incompatible, if built with different OCaml compilers. “Jesus, what a clusterfuck” to quote Burn after Reading.
And they are indeed incompatible, as I experienced.
I could build the OCaml compiler myself (which one though ?) and then build
Unison with it. But that sounds like a lot of work. Wouldn’t it be better to
somehow just use the Ubuntu unison
in macOS ? How ? With Docker of course!
Maybe a brillant idea, maybe not.
Docker
A Dockerfile is quickly made, that just “wraps” a unix command into a docker.
It’s important that the FROM ubuntu:focal
matches the ubuntu version of my
unison host, otherwise there’s nothing special about it:
Dockerfile
:
FROM ubuntu:focal
RUN DEBIAN_FRONTEND=noninteractive \
apt-get update \
&& apt-get -y install unison
ENTRYPOINT ["/usr/bin/unison"]
CMD ["-version"]
I build this with
build-unison-docker
:
#! /bin/sh
docker build -t unison "$@" .
And I run i with
unison-docker
:
#! /bin/sh
UNISONLOCALHOSTNAME="${UNISONLOCALHOSTNAME:-`hostname -s`}"
docker run -e UNISONLOCALHOSTNAME="${UNISONLOCALHOSTNAME}" \
-v /Volumes:/Volumes:delegated \
-v ~/.ssh/docker:/root/.ssh \
-v ~/Library/Application\ Support/Unison:/root/.unison:delegated \
unison "$@"
Here are some points of interest. First set the UNISONLOCALHOSTNAME
to a
stable name, that doesn’t change. If your hostname is meaningful (not
“localhost”), the default I made should be fine.
The exposed volumes: All my interesting stuff is either in /Volumes/Users/
or
/Volumes/Source
, so I am just 1:1 exposing this to the docker. The docker
volumes are adorned with :delegated
in the hope that it speeds up the
operation a little, but I don’t notice any difference for unison. Then unison
has to save state between syncs. I use the same location
~/Library/Application\ Support/Unison
as a native unison would.
Unison needs ssh to connect to the host. But the entries for the
.ssh/config
are slightly different, so I could not just mount my ~/.ssh
to
/root/.ssh
. I had to create something special. And here’s what it looks
like:
.ssh/docker/config
:
Host unisonhost
HostName whatever.domain.de
StrictHostKeyChecking no
IdentitiesOnly yes
User unison
IdentityFile ~/.ssh/id_rsa_unison_docker
The main problem, this config fixes, is the default StrictHostKeyChecking yes
.
That will throw off the container the first time it is run as there is noone
there to press ‘y’ (verify the host identity). After the first run
you may change this back to yes
.
Problems
Docker containers are slow in macOS. A linux docker starts in a matter of milliseconds. But on macOS it’s a matter of seconds. For a one time sync operation that may not matter too much though.
The docker filesystem is EXT4, the macOS filesystem is AFPS. The unison in docker sees a mounted AFPS filesystem, but must assume it’s EXT4. And so it does and it does not consider AFPS metadata. This is not a problem for me, but it could be a problem for everbody else.
Syncing used to take only a few seconds, if there were little or no changes.
With the docker idea it takes minutes. For an initial scan it took something
like an hour.
I ran into Connection to unisonhost closed by remote host.
continually (and
still do).
I tried to remedy the initial syncing effort by syncing my folders piece by piece like so:
for i in *
do
./unison-docker /Volumes/Source/ ssh://unisonhost//home/nat \
-path "src/$i" \
-contactquietly -batch -prefer newer
done
This worked but it was just very, very slow. Think hours! That was due to the incredible slowness of everything. It is clearly impractical on a daily basis. After this was done, I could get the whole sync to finish without errors. But I wonder what would happen, if there are a lot of changes.
To illustrate just how slow this is on my system (Mac mini (2018) 10.14.6). Here is a fight between a native ‘find’:
time find /Volumes/Source/srcO/mulle-objc -name NSObject.h -print
...
real 0m0.322s
user 0m0.038s
sys 0m0.271s
and a dockerized ‘find’:
Dockerfile
:
FROM ubuntu:focal
ENTRYPOINT ["/bin/find"]
CMD ["-version"]
unison-find
:
#! /bin/sh
docker run -v /Volumes:/Volumes find:cached "$@"
time ./unison-find /Volumes/Source/srcO/mulle-objc -name NSObject.h -print
...
real 1m25.464s
user 0m0.071s
sys 0m0.039s
I tried out several docker modes, but the results, within a small margin of
error, appear to be the same. Notice that a native find
is more than a
hundred times faster.
Mode | Time |
---|---|
Native | 0m00.3 |
Docker Default | 0m54.0 |
Docker Delegated | 0m56.0 |
Docker Cached | 0m53.0 |
Solution
Forget unison packages, forget homebrew, apt and docker on macOS. I build a unison for macOS and Ubuntu myself with my own picked OCaml version. Copy it to everywhere manually. This takes less time than a sync operation using unison inside docker :)
Postscript
Aha! Newer versions of unison now also show the compiler they are built with. Guess why :)
Unison 2.51.3 (OCaml 4.10.2)
Post a comment
All comments are held for moderation; basic HTML formatting accepted.