Notice: register_sidebar was called incorrectly. No id was set in the arguments array for the "Sidebar 1" sidebar. Defaulting to "sidebar-1". Manually set the id to "sidebar-1" to silence this notice and keep existing sidebar content. Please see Debugging in WordPress for more information. (This message was added in version 4.2.0.) in /usr/share/wordpress/wp-includes/functions.php on line 4139 Tech Notes » Blog Archive » rsync on Windows

rsync on Windows

howto Add comments

A friend asked me about rsync, and I remembered i hadn’t posted my success story; in short, it works, you don’t need to install all of cygwin, but there are weaknesses in the filesystem exposed occasionally.

in AutoFS Mounted FTP via FUSE, I discussed setting up an FTP client as a filesystem to do a client-side pull of FTP content.  That turned out quite vulnerable to our IT guy truncating a connection randomly through the firewall: the data stream didn’t seem to handle timeouts, and any process waiting on a data buffer tends to hang.  Pulling data from a filesystem is a kernel-level thing, so a process cannot really abort a hanging FUSE request.  Bad news.

I later found the simplified route, and installed it on the FTP server:

  1. download the ZIP file (cygwin-rsyncd-2.6.8_0.zip)
  2. unpack the zip file
  3. use rsync.exe, which uses a cygwin.dll
  4. that’s it

The rsync.exe is a cygwin binary, and acts very similar to its UNIX-y cousins; instead of installing a huge Cygwin stack, the developer has bundled just the necessary parts of Cygwin into the smaller single DLL, and provides it for rsync’s dependency.  There’s a few other files in there, but as an rsync client, I didn’t need them.

The server has a fairly standard rsync server config, with an rsyncd.conf identifying shared directories as filesystems, and such.  In general, if you have a working rsyncd, you can connect to it from this rsync.exe

Finally, my command line, as an example (my rsync.exe and the DLL are in C:Program Filesrsync, hand-installed), is as follows. Note that I’ve broken the line up with back-slashes to show functional groups, but when you run it, you should have it all on one line, without backslashes except the rsync.exe path.

“C:Program Filesrsyncrsync.exe”  

–exclude=some-skipped-file’ –exclude=*a_wildcard_pattern* –exclude=backup*z  

–exclude=backup*.zip –exclude=*.mov –exclude=*.wmv –exclude=.*  

–delete –delete-excluded –chmod=ugo=rX ‘  

-avr    /cygdrive/f/path/to/users/library/   server.example.com::library/

The server has a rsyncd.conf config that says:

[ftp-services]
path = /shared/docs/library
uid = libraryowner
gid = docs-ro
comment = Library to share to all remote staff
write only = true
read only = false
list = false
exclude = some-skipped-file’ *a_wildcard_pattern* backup*z
incoming chmod = u=rwX

You’ll notice a few things that are unusual here:

  1. I use “-avr” in my rsync command.  “r” should be redundant with “-a”.  try it without
  2. the source and destination paths end in a slash.  I would recommend the same convention.  Be consistent
  3. I used to have /library/* as my source, but on a push, deleted directories as direct children of /library/ are gone, no longer found by the “*” wildcard, so there deletion does not sync
  4. I have a bunch of –exclude options, some of which are duplicate on the rsyncd.conf entry.  Try removing some once you have yours working.
  5. ditto for the chmod — they should act similarly, and setting it on the server sets a consistency and avoids users forgetting
  6. my uploads are write-only — I use a different share label in rsyncd.conf for uploads than I do downloads so that those who have the keys to read can’t butcher my stuff.
  7. for those crazy A:, C: drive letters, you can use the cygwin special path /cygdrive/a/ or /cygdrive/c/ to map to them without playing the game of escape slashes: “was that 21 backslashes for 10 laters of scripting, or 22?”
If your transfer isn’t working (i.e. transfers nothing, or transfers it all every time):
  1. try adding “–stats” to get a list of what rsync is doing
  2. try adding “–max-size=20k” to avoid pushing HUGE files while diagnosing issues
  3. if the datestamping is seriously butchered on your windows boxes, try “-c” option to use checksums, understanding it’ll take (much) longer to sync
  4. if you habitually rename files, and rsync faithfully re-pushes the file every time, “-y” or “–fuzzy” tells rsync’s receiver to check around for similar files to use for missing files

Good Luck!

Leave a Reply

WP Theme & Icons by N.Design Studio
Entries RSS Comments RSS Log in