Rsync Over FTP, read-write, on a Mac or BSD Client

howto Comments Off on Rsync Over FTP, read-write, on a Mac or BSD Client

Recently I was discussing with someone the need to simplify the sync of a folder into an FTP server. The goal is that at set intervals, any change in a local folder is pushed to a remote folder: changes changed, new files created, removed files removed. Similar to yesterday’s article, except this is a read-write access to the FTP server, allowing either direction of sync.

This is how to do it using curlftpfs and rsync.

The general idea is to use the curlftpfs almost like FUSE-mounting a remote resource, then using rsync on that mounted filesystem. Like yesterday, we use mktemp to work relatively portably in a temp folder.

The exult would be something like this:

  • local folder: ${HOME}/contrib
  • remote server:
  • remote folder: ./Scott/ABC
  • remote user: scott
  • remote pass: tiger
# create a temporary/random mountpoint
TEMPFILE=$(mktemp -d -t ftprsync)

# mount the remote space
sudo curlftpfs -o rw ${TEMPFILE}

# sync
rsync -avr --delete-after ${TEMPFILE}/* ${HOME}/contrib

#shut it down
sudo umount ${TEMPFILE}
rm -fr ${TEMPFILE}

Similar to the method used with mount_ftp, curlftpfs will use a URL-encoded user/pass. Additionally, curlftpfs will allow -o options to provide username and password.

The issue with this method is that obtaining the curlftpfs for MacOSX is problematic: it’s based on glib-2.0, and glib itself has a bit of a dependency to build. As well, it uses pkg-config which is still not a standard part of Apple Xcode CLI tools, but if you have glib-2.0 built, then you’ve already got pkg-config working (glib is heavily dependent)

In my own experience, curlftpfs is quite function, and plays very well with the automount config; the problem is that on very long distance connections, it seems to choke quite often during the sync. I would recommend using this method for low-lartency connections to legacy servers, and for longer-hauls, work on getting rsync able to connect directly without using FUSE to indirectly map to FTP.

…so the task right now is to build curlftpfs, or find a functioning binary.

Rsync Over FTP, on a Mac or BSD Client

howto, Uncategorized Comments Off on Rsync Over FTP, on a Mac or BSD Client

Recently I was discussing with someone the need to simplify the sync of a folder into an FTP server. The goal is that at set intervals, any change in a remote server is pulled to q local folder: changes changed, new files created, removed files removed. This is that kind of thing that should be easier, but it’s mixing an old technology (rsync) with a very, very old technology (FTP).

This is how to do it using mount_ftp and rsync.

The general idea is to use the mount_ftp almost like FUSE-mounting a remote resource, then using rsync on that mounted filesystem. If we wrap it around mktemp to work relatively portably in a temp folder, we’d have something like this:

  • local folder: ${HOME}/contrib
  • remote server:
  • remote folder: ./Scott/ABC
  • remote user: scott
  • remote pass: tiger
# create a temporary/random mountpoint
TEMPFILE=$(mktemp -d -t ftprsync)

# mount the remote space; no output, but the return code matches errno.h values
sudo mount_ftp -o rw ${TEMPFILE} ; echo $?

# sadly, despite best efforts (and "-o rw"), this is only a read-only, so only good
# for syncing FTP content out to the local system

# sync
rsync -avr --delete-after ${TEMPFILE}/* ${HOME}/contrib

#shut it down
sudo umount ${TEMPFILE}
rm -fr ${TEMPFILE}

How to Convert Nicknames to JSON for VirtualWisdom4

fcfae No Comments »

VirtualWisdom4 enables a deeper insight to the metrics behind performance and available of large-scale data networks, but has a few challenges in the initial setup. One we see fairly often in the deployment side is the fact that few customers know what JSON is. Sure, a developer will say “JSON, yeah I got that”, but VirtualWisdom4 users are not all developers. Heck, our field staff don’t interpret JSON, don’t recognize when a “{” is where a “[” should be.

VirtualWisdom4 collects data as soon as it gets access, but that data is not aggregated into upper-level entities such as Hosts and Storage Arrays until those entities exist. In essence, VW4 collects immediately, but the data is of limited benefit until those entities are created, so creating those entities is critical to return-on-investment. The sooner we get to visualizing data, the sooner we can begin trouble-killing.

Often, it’s easier just to convert from a common format that our customers understand, or can produce from other tools, into JSON.

Read the rest of this entry »

Finding and Running Java

howto 1 Comment »

As a reference, I wanted to leave a few notes as to how to find a java runtime.

If you’re on a linux system (RPM-based or Debian-based including Ubuntu flavours), BSD (including OpenBSD, NetBSD, or the MacOSX fork), or even USL Unix (including AIX flavours and the other 5999 licensees), chances are typing “java” at a prompt does something. “java -version” should spit out a version.

If you’re on Windows, then you might already have Oracle’s Java package installed; look down the various “Program Files” directories for “Oracle” or “Java”, or if search is working today, try searching for “java.exe”. Additionally, BNA and DCNM (and their variants: HPNA, CMCNE, etc) run on Java, so you might find a java.exe in a Cisco\Network Manager\jre\bin\ directory.

Either way, “java.exe -version” or “java -version” should spit out a version number.

If you don’t have a Java interpreter, then you’ll need to install one. That changes form platform to platform, but again Linux variants have it easiest, typically typing “yum install java” or “zipper install java”, or “apt-get install java”. … or some variant of that. Be aware, you might be looking for a java-1.6.0-openjdk (or for the SuSE Experience, where underscores are preferred over consistency, java-1_6_0-openjdk).

In windows, search where you normally search. If you need to go to Oracle’s website, I assume you have web access.

In either case, if you’re unable to install Java, then the type of guide I tend to write might be a bit challenging. you may need some help from your colleagues in IT.

Also, very important, javaw is for windowed apps; we do not want javaw nor javaw.exe unless we’re running a windowed application.

Automount on MacOSX

howto No Comments »

As a quick reminder, when trying to use automount -hosts on a Mac (ie auto-mounts NFS shares found by name on the local subnet), ensure that you:

  1. edit client:/etc/autofs.conf: ensure “resvport” is in AUTOMOUNTD_MNTOPTS (mine says: AUTOMOUNTD_MNTOPTS=nosuid,nodev,resvport)
  2. defaults write DSDontWriteNetworkStores true to avoid writing .DS_Store in network shares, which MacOSX keeps open (hence “hot”, hence avoiding automount timeout)

Works quite well now with my Synology, for which I have enabled “ root_quash” for the shares in question:

$ showmount -e ds211.local
Exports list on ds211.local:
/volume1/Archive *
/volume1/music *

$ mount
ds211.local:/volume1/Scan on /net/ds211.local/volume1/Scan (nfs, nodev, nosuid, automounted, nobrowse)

.. the next step is to set all Scanned content to write to this new pathname to avoid mount issues down-the-road

Embedded Roving Devices

howto No Comments »

After a failed attempt to use a Dream Cheeky OIC to discipline a dog’s affection for a sofa, I see that the AC13 roving tank is upgraded. shows a teardown of the older AC13, has a more recent teardown but seems unaware of the previous.

AC13 Spytank
FlashROM S29GL032N S29GL032N
CPU Winbond W90N745
(a 128-bit chip)
128 pin TQFP – no discernable labeling or markings
Linux 2.4.20 unknown
wifi RT2870 RT3070
Audio Driver unknown GPY0030B

The real question is whether I could get away with purchasing this newer variant in an attempt to begin milling a proper embedded OS for it (rather than the serial fire-and-forget what’s-inittab-for crap most often seen)

Projects. Hmm. Need a workshop first…

Quick Reminder: Activate MacOSX 10.8 Remote Display from SSH

howto No Comments »

I got nailed by this, so this blog post is a quick reminder to myself that I hope helps at least one other person out there:

  1. Freshly cold-booted MacMini
  2. 10.8 installed
  3. I’d really like to get into the to get other stuff configured.  Yeah.  That would be outstanding.

The path to the solution is:

  1. ssh username@newserver.local
  2. /System/Library/CoreServices/RemoteManagement/ -activate -configure -access -on -restart -agent -privs -all
  3. RESTART your client-side Screen Sharing to see the fixed remote Macmini.  Yeah, silly as that sounds

Of course, if you have a new freshly unwrapped new-car-smell macmini, then you need to connect using username “root” and password equal to the serial number.  If “root” failed, try a blank username.

Issues with VMs’ Lost Performance

redirection No Comments »

In tuning a product that is a write-intensive database tool, we’ve found that performance on a VM can be wildly inconsistent. The hardest thing to explain is why we see a 30% performance drop and slow commits when moving from physical servers to the same-size physical server holding one VM generated form that server: literally the same system, but with a VM layer.

Shoot, 37 Signals had this performance impact:

Of course, layering in a VM lets us VMotion the process, if we weren’t using a lot of local disk (since the local disk doesn’t move with it). NAS? Bwa ha ha ha, please no, corporations with I/O response requirements use SAN. Sure, SAN is expensive, but it shares the disk nicely unless you need to use the tool to monitor the SAN — SAN congestion slows the tool you need to fix it 🙂

Layering a VM tends to be satisfying a corporate requirement of “everything on VM”.

When considering Tuning the MySQL InnoDB Engine, keep in mind that if it’s on a VM, you can’t guarantee accuracy of the numbers you’re using to tune.

Grabbing the rui.crt for vCenter Programmatically

howto No Comments »

The rui.crt for vCenter can be grabbed (assuming your server is by:

openssl s_client -connect < /dev/null   |  
sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p'   > rui-

So what’s up with this?  When installing my company’s product, we always go through this goose-chase to find a rui.crt.  Seriously, it’s a pain, we need to dig through a bunch of directories, and in the end, sometimes we’re still scratching our heads.

Our customers just want something they can cut-n-paste.  Of course, the line above only works on non-windows, but I have reduced concern for those trying to fly with wings of lead: there’s a limit as to what you can do with windows.  Accept it.

HUGE Kudos to “Bruno” on StackExchange whose LDAP Certificate response was the key win for this.

rsync on Windows

howto No Comments »

A friend asked me about rsync, and I remembered i hadn’t posted my success story; in short, it works, you don’t need to install all of cygwin, but there are weaknesses in the filesystem exposed occasionally.

in AutoFS Mounted FTP via FUSE, I discussed setting up an FTP client as a filesystem to do a client-side pull of FTP content.  That turned out quite vulnerable to our IT guy truncating a connection randomly through the firewall: the data stream didn’t seem to handle timeouts, and any process waiting on a data buffer tends to hang.  Pulling data from a filesystem is a kernel-level thing, so a process cannot really abort a hanging FUSE request.  Bad news.

I later found the simplified route, and installed it on the FTP server:

  1. download the ZIP file (
  2. unpack the zip file
  3. use rsync.exe, which uses a cygwin.dll
  4. that’s it

The rsync.exe is a cygwin binary, and acts very similar to its UNIX-y cousins; instead of installing a huge Cygwin stack, the developer has bundled just the necessary parts of Cygwin into the smaller single DLL, and provides it for rsync’s dependency.  There’s a few other files in there, but as an rsync client, I didn’t need them.

The server has a fairly standard rsync server config, with an rsyncd.conf identifying shared directories as filesystems, and such.  In general, if you have a working rsyncd, you can connect to it from this rsync.exe

Finally, my command line, as an example (my rsync.exe and the DLL are in C:Program Filesrsync, hand-installed), is as follows. Note that I’ve broken the line up with back-slashes to show functional groups, but when you run it, you should have it all on one line, without backslashes except the rsync.exe path.

“C:Program Filesrsyncrsync.exe”  

–exclude=some-skipped-file’ –exclude=*a_wildcard_pattern* –exclude=backup*z  

–exclude=backup*.zip –exclude=*.mov –exclude=*.wmv –exclude=.*  

–delete –delete-excluded –chmod=ugo=rX ‘  

-avr    /cygdrive/f/path/to/users/library/

The server has a rsyncd.conf config that says:

path = /shared/docs/library
uid = libraryowner
gid = docs-ro
comment = Library to share to all remote staff
write only = true
read only = false
list = false
exclude = some-skipped-file’ *a_wildcard_pattern* backup*z
incoming chmod = u=rwX

You’ll notice a few things that are unusual here:

  1. I use “-avr” in my rsync command.  “r” should be redundant with “-a”.  try it without
  2. the source and destination paths end in a slash.  I would recommend the same convention.  Be consistent
  3. I used to have /library/* as my source, but on a push, deleted directories as direct children of /library/ are gone, no longer found by the “*” wildcard, so there deletion does not sync
  4. I have a bunch of –exclude options, some of which are duplicate on the rsyncd.conf entry.  Try removing some once you have yours working.
  5. ditto for the chmod — they should act similarly, and setting it on the server sets a consistency and avoids users forgetting
  6. my uploads are write-only — I use a different share label in rsyncd.conf for uploads than I do downloads so that those who have the keys to read can’t butcher my stuff.
  7. for those crazy A:, C: drive letters, you can use the cygwin special path /cygdrive/a/ or /cygdrive/c/ to map to them without playing the game of escape slashes: “was that 21 backslashes for 10 laters of scripting, or 22?”
If your transfer isn’t working (i.e. transfers nothing, or transfers it all every time):
  1. try adding “–stats” to get a list of what rsync is doing
  2. try adding “–max-size=20k” to avoid pushing HUGE files while diagnosing issues
  3. if the datestamping is seriously butchered on your windows boxes, try “-c” option to use checksums, understanding it’ll take (much) longer to sync
  4. if you habitually rename files, and rsync faithfully re-pushes the file every time, “-y” or “–fuzzy” tells rsync’s receiver to check around for similar files to use for missing files

Good Luck!

WP Theme & Icons by N.Design Studio
Entries RSS Comments RSS Log in