Speed up file copies on OSX

[mikeparsons]

I did this on my machines a while ago but I was just chatting with a flame guy at the pub who was envious that I was working while we drank (just copying a LOT of files for tomorrows job) what most impressed him was the speed, and the way all the ladies present were so excited by my remote Unix skills. It is an under appreciated fact that women both young and old love a man who can handle a command prompt...

So now I'm back at the office waiting for the next disk to arrive (yay for bangkok traffic) so I can go back home I thought I'd do as asked and see if it really is any use to anyone but a bored old flame guy.

To start with my iPhone tool of choice for this stuff is Prompt from Panic software (makers of my favourite FTP client Transmit)

http://panic.com/prompt/

I'd been copying files via rlogin for a while since I saw Charlie Ellis doing it to his Resolve when we worked on a feature together a few years ago. I then saw somewhere on the net tips on how to speed up rSync on Mountain Lion. It makes me happy to know that after these tweaks I am copying as fast as possible but don't bother reading any further unless you do as much file copying as I do as the following may be very boring.

On OSX the version of rsync is still stuck at 2.x - even Mavericks ships with 2.6.9 as apparently Apple don't like the new GPL3 licenses won't ship with updated versions - this same issue means bash, gcc etc are all outdated versions in the latest and greatest Mac os companred to the rest of unixland (and also why samba is no longer included) this is a problem in our world as when you're transferring a lot of files (for example a movie's worth of DPX files) version 2 creates a complete file list before it starts copying, leaving you staring at the 'building file list' statement for hours before your copy even begins.

Luckily rsync version 3 has been out a while and does things differently using incremental file lists and so copying starts and therefore ends sooner. So I updated to rsync version 3. Obvious duh.

At the end of this is how I did it back then but first here's a short cut I've since discovered to do the same thing with MUCH less effort!

Carbon Copy Cloner;

http://www.bombich.com

uses its own later version of rsync to do its business. So if you do your copying using CCC (and your really should) you will benefit from fully up to date rsync behaviour. 

'But I want to use the command line to copy in the pub like the cool kids' I hear you say, well here's the sneaky bit - if you go to 

/Applications/Carbon Copy Cloner.app/Contents/MacOS/ccchelper.app/Contents/MacOS/

and copy the rsync executable there then paste it to

/usr/bin

you get to enjoy command line rsync 3 goodness with close to zero effort 🙂 

Well thats the short cut, BUT if you want to do it properly then its time to roll your own rsync. Your choices are to download the binaries from samba and compile them each time or to use a nice scripted command line trick. Which is what follows...

First off you will need Xcode installed but that's no drama as its free;

https://itunes.apple.com/en/app/xcode/id497799835

Note you WILL need to be running a minimum of 10.8.4 to get the current version of Xcode (5.0.2) but the compile tools are pretty much the same in Xcode 4.6 or 5.0 so basically any recent-ish build of Xcode will be fine. In fact I didn't install Xcode on this machine just the command line tools but Xcode is free and easy so just grab that and who knows you might start building your own apps...

Next we install Homebrew to do the installing of rsync. Homebrew is a Unix style package manager for osx.

http://brew.sh/

If you use git or Maven you know all you need do is type a few things in a terminal and bam - new software, saving that whole go to the web, download, expand, install nonsense. It's pretty useful and I've been using Homebrew on Mac and Yum on Centos for a while.

Install Homebrew by opening Terminal and typing;

ruby -e "$(curl -fsSL https://raw.github.com/mxcl/homebrew/go)"

It's a pretty good install script so it tells you what its going to do and gives you a chance to run away if you decide it all sounds too scary.

Once that ends you need to run brew doctor by typing

brew doctor

in the same terminal window. 

It will moan and complain about a bunch of things but the one you need to care about is 'You must: brew install git' which adds the git hub tools to let us access the apps we want. To do this in the same terminal type;

brew install git

(Duh)

Once thats done you can then install the new improved version of rsync, again in your by now beloved terminal type;

brew install https://raw.github.com/Homebrew/homebrew-dupes/master/rsync.rb

The nice repository over at github will send you the latest version of rsync and install it. Its currently sitting at 3.1 see

https://rsync.samba.org/

for whats new. 

3.1 has done some nice things as far as using it on a small screen is concerned (which is good for me on my iphone) such as abbreviating outputs eg xfer is now xfr and to-check is now to-chk all of which leaves more space on the tiny screen in my pocket to see things I actually want to know about as well as automatically breaking numbers up into 3 digit blocks which helps my ageing brain as I find 1,230,000 files easier to think about than 1230000 for some reason.

One last thing, every OSX update will of course replace the version of rsync back to the Apple licensed bad old one. This is because /usr/bin takes precedence over the /usr/local/bin that our nice new version of rsync lives in. The way around this is to edit the paths file.

(If you don't know how to edit protected system files grab my back burner guide of last year from the sticky at the top of the list as it has some info on editing system files using text wrangler.)

To do this go to /private/etc/ and open the 'paths' file in text wrangler.

Move the last line /usr/local/bin to the top ahead of /usr/bin

/usr/local/bin and the paths file don't get overwritten by software update and so OSX will keep looking at our version of rsync rather than Apple's smelly old one. Pretty cool eh?

Save 'paths' file, and then reboot your mac.

Now the fun begins. Do another rsync transfer of your metric tonne of DPX files and now instead of looking at an hour long message of 'building file list' all you see is 'sending incremental file list' and copying begins pretty much straight away.

But it gets better. 

As I said in another post I am actually way too lazy to type when sat at the computer so I tend to use aRsync 

http://sourceforge.net/projects/arrsync/

for a lot of my one off copying needs. 

As aRsync is just a GUI to the underlying rsync commands you don't need to wait for an updated front end to access the wonderful incremental world of Rsync 3. It just uses the version of rSync that you have. So the very next aRsync copy you do will be faster.

Alternatively if its an ongoing job with more bits arriving on the same drives day after day I tend to use Carbon copy cloner instead, but as I said at the very start of this now waaaay too long post it already uses the new version of rsync anyway and so if you use CCC you've wasted a good 10 minutes reading all this rubbish.

The main benefit of this hackery is to get faster command line transfers but the other good news is that the command line rsync 3 also has a updated progress indicator so as well as the all but useless file by file progress updates you can use -info=progress2 instead to show total transfer progress, which is invaluable to me as the only time I run rsync from the command line is where we started this post,  sitting in the pub using rlogin to move files around from my iphone. Now with -info=progress2 I can finally get an idea of just how much drinking time I have left before I have to go back to work.

If that all seems way too much trouble (and it almost was for me too until I got a feature film of frames to move and sat watching it build a file list for so long I thought it had crashed), bear in mind that on TV commercial volumes of files I am seeing around 15 - 30 minutes improvement of transfers. Whilst on the feature it was much more significant and probably shaved a good 2 hours off the total copying time -  20 minutes of which I have now spent typing all this, so I'm now only up about an hour and a half I guess 😞

Anyway, hope it helps but if nothing else I can come back here when I forget what to do! Happy copying.

Mike

Lastly I was told to drop in some info on command line rsync options but there's not much point as its all over the web but 

http://www.comentum.com/rsync.html 

is as good a list of options as any. 

I generally only use;

rsync -arvX source destination

-a is for archive which preserves permissions, times, links and devices

-r is for recursive so it copies all enclosed directories and subdirectories

-v is for verbose so it tells you what its doing

-X is for extended attributes (used to be -E but that's now for executability, confusing much?) such as spotlight comments and tags

-info=progress2 to see total transfer progress rather than the useless file by file

other handy arguments used once a blue moon are

m which ignores any empty directories

z which compresses file data during the transfer which is handy if copying over a network (by which I mean copying from a network drive to you mac drive not initiating the transfer over the network.