Tech Notes

I’ve been having a problem on an FTP server that may well be resolved using FUSE, curl, and autofs, plus some blind rsync:

When I synch content across the Atlantic to another server, it’s always truncated despite my best efforts. It seems that if there’s too large of latency, the windows-based FTP service chokes and dies, and the uploaded content merely stops. FTP is a bit of a dumb protocol, it doesn’t much realize that the upload was finished due to abort rather than an orderly finish unless the command channel reflects this — if the ftp-data socket closes, it’s done.

I’ve asked for rsync service to avoid this (and to make a more gentle sync than recursive wget) to no avail — the people discussing the request don’t ask me questions, and seem to refuse on assumption.

• FTP mounted as a filesystem allows a rsync
• Rsync allows a sync with retry and continues a bit more gracefully
• FUSE can mount a filesystem in userspace
• CurlFTPfs can mount a FTP server via FUSE
• AutoFS allows the filesystem to unmount when idle, isolating us from server restarts (it IS windows after all)

in /etc/auto.master, I added:
# --- a separate direct-map automount (large space is a tab)
/ftpfs /etc/auto.ftpfs
#

I created /etc/auto.ftpfs:
# from USA server: (large spaces are tabs) (area01 is all one line, no space between password and @exam...)
area01 -fstype=fuse,allow_other,ro curlftpfs#username01:PassWord01 @ftp01.example.com/
area02 -fstype=fuse,allow_other,ro curlftpfs#username02:PassWord02 @ftp02.example.com/


I added this to an RPM that requires fuse-curlftpfs, and made it service autofs reload in install (postin and postun)

Now, I can do the following: (still testing):

rsync /ftpfs/area01/* /my/backup/dir/area01/
rsync /ftpfs/area02/* /my/backup/dir/area02/

… and this command will implicitly automount the filesystems in turn, allowing the rsync to sync content. Of course, it’s more efficient if the FTP server simply activated the rsync service, but it’s Windows, and rsync is only a decade old, so Windows won’t have it yet (and the administrator fears both the Cygwin dependency issues and the simplified route)

The rsync might do horrible things with the FTP server, I don’t know. If it does, maybe that will open up a cooperative dialogue (Kidding! Forcing someone’s hand is never a good way to start!).