Rsync Over FTP, read-write, on a Mac or BSD Client

howto Comments Off on Rsync Over FTP, read-write, on a Mac or BSD Client

Recently I was discussing with someone the need to simplify the sync of a folder into an FTP server. The goal is that at set intervals, any change in a local folder is pushed to a remote folder: changes changed, new files created, removed files removed. Similar to yesterday’s article, except this is a read-write access to the FTP server, allowing either direction of sync.

This is how to do it using curlftpfs and rsync.

The general idea is to use the curlftpfs almost like FUSE-mounting a remote resource, then using rsync on that mounted filesystem. Like yesterday, we use mktemp to work relatively portably in a temp folder.

The exult would be something like this:

  • local folder: ${HOME}/contrib
  • remote server: ftp.example.com
  • remote folder: ./Scott/ABC
  • remote user: scott
  • remote pass: tiger
# create a temporary/random mountpoint
TEMPFILE=$(mktemp -d -t ftprsync)

# mount the remote space
sudo curlftpfs -o rw ftp://scott:tiger@ftp.myserver.com/Scott/ABC ${TEMPFILE}

# sync
rsync -avr --delete-after ${TEMPFILE}/* ${HOME}/contrib

#shut it down
sudo umount ${TEMPFILE}
rm -fr ${TEMPFILE}

Similar to the method used with mount_ftp, curlftpfs will use a URL-encoded user/pass. Additionally, curlftpfs will allow -o options to provide username and password.

The issue with this method is that obtaining the curlftpfs for MacOSX is problematic: it’s based on glib-2.0, and glib itself has a bit of a dependency to build. As well, it uses pkg-config which is still not a standard part of Apple Xcode CLI tools, but if you have glib-2.0 built, then you’ve already got pkg-config working (glib is heavily dependent)

In my own experience, curlftpfs is quite function, and plays very well with the automount config; the problem is that on very long distance connections, it seems to choke quite often during the sync. I would recommend using this method for low-lartency connections to legacy servers, and for longer-hauls, work on getting rsync able to connect directly without using FUSE to indirectly map to FTP.

…so the task right now is to build curlftpfs, or find a functioning binary.

Rsync Over FTP, on a Mac or BSD Client

howto, Uncategorized Comments Off on Rsync Over FTP, on a Mac or BSD Client

Recently I was discussing with someone the need to simplify the sync of a folder into an FTP server. The goal is that at set intervals, any change in a remote server is pulled to q local folder: changes changed, new files created, removed files removed. This is that kind of thing that should be easier, but it’s mixing an old technology (rsync) with a very, very old technology (FTP).

This is how to do it using mount_ftp and rsync.

The general idea is to use the mount_ftp almost like FUSE-mounting a remote resource, then using rsync on that mounted filesystem. If we wrap it around mktemp to work relatively portably in a temp folder, we’d have something like this:

  • local folder: ${HOME}/contrib
  • remote server: ftp.example.com
  • remote folder: ./Scott/ABC
  • remote user: scott
  • remote pass: tiger
# create a temporary/random mountpoint
TEMPFILE=$(mktemp -d -t ftprsync)

# mount the remote space; no output, but the return code matches errno.h values
sudo mount_ftp -o rw ftp://scott:tiger@ftp.example.com/Scott/ABC ${TEMPFILE} ; echo $?

# sadly, despite best efforts (and "-o rw"), this is only a read-only, so only good
# for syncing FTP content out to the local system

# sync
rsync -avr --delete-after ${TEMPFILE}/* ${HOME}/contrib

#shut it down
sudo umount ${TEMPFILE}
rm -fr ${TEMPFILE}

Java getOutputStream() surprises

Uncategorized Comments Off on Java getOutputStream() surprises

As a not to my future self, apparently you need to open the connection before setting doOutput:

URLConnection connection = url.openConnection();
connection.setDoOutput(true);

return connection.getOutputStream();

It’s a good thing that’s poorly documented and non-obvious, and that it fails in misleading ways.

Scheduling Cyclic Jobs in MacOSX

How to Comments Off on Scheduling Cyclic Jobs in MacOSX

Many of us UNIX old-timers are quite accustomed to cronjobs, but MacOSX has a centralized “LaunchDaemon” called launchd — to leverage it to run cronjobs gives an OS-specific, perhaps OS-preferred, method of doing so.

The TL;DR:

<?xml version=”1.0″ encoding=”UTF-8″?>

<!DOCTYPE plist PUBLIC “-//Apple//DTD PLIST 1.0//EN” “http://www.apple.com/DTDs/PropertyList-1.0.dtd”>

<plist version=”1.0″>

<dict>

<key>Label</key><string>com.example.rsync.SyncTheRepos</string>

<key>Program</key><string>/usr/bin/rsync</string>

<key>ProgramArguments</key>

<array>

<string>-avr</string>

<string>–delete-after</string>

<string>rsync.example.com::repos</string>

<string>~/Documents/Repos</string>

</array>

<key>EnableGlobbing</key><true/>

<key>StartCalendarInterval</key>

<dict>

<key>Hour</key><integer>3</integer>

<key>Minute</key><integer>14</integer>

</dict>

<key>ProcessType</key><string>Background</string>

</dict>

</plist>

In general, there is a lot of flexibility in setting up a launchd plist — the config info on https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man5/launchd.plist.5.html plus the various examples on the internet should help, but I generally take this example and re-use it.

Once this plist is saved as a local text file in ~/Library/LaunchAgents/ (for example, I’ll save mine as ~/Library/LaunchAgents/rsync-repos.plist), I activate it using:

launchd load -w ~/Library/LaunchAgents/rsync-repos.plist

If I want to disable the job, I use:

launchd unload -w ~/Library/LaunchAgents/rsync-repos.plist

The files are not modified in either case.

How to Convert Nicknames to JSON for VirtualWisdom4

fcfae No Comments »

VirtualWisdom4 enables a deeper insight to the metrics behind performance and available of large-scale data networks, but has a few challenges in the initial setup. One we see fairly often in the deployment side is the fact that few customers know what JSON is. Sure, a developer will say “JSON, yeah I got that”, but VirtualWisdom4 users are not all developers. Heck, our field staff don’t interpret JSON, don’t recognize when a “{” is where a “[” should be.

VirtualWisdom4 collects data as soon as it gets access, but that data is not aggregated into upper-level entities such as Hosts and Storage Arrays until those entities exist. In essence, VW4 collects immediately, but the data is of limited benefit until those entities are created, so creating those entities is critical to return-on-investment. The sooner we get to visualizing data, the sooner we can begin trouble-killing.

Often, it’s easier just to convert from a common format that our customers understand, or can produce from other tools, into JSON.

Read the rest of this entry »

Git-Backed Icinga

How to No Comments »

I want to make git commits to configure my icinga2 monitor: commit to a “devel” tree for config validation, commit to a “prod” tree to validate-and-activate a config.

[wp_graphviz simple=”yes”] devel -> { devel prod } -> check -> { prod devel “live\nicinga” } [/wp_graphviz]

 

Felipe Contreras has slowly converted me to using git.  I fight with git, but it seems the way forward based on critical mass of user-base, and only takes 7 more steps to do each thing, so my fingers get used to the additional commands.

 

I’m looking at a few tasks to get this done:

  1. init the repository
  2. load the example config shipped with icinga2 to pre-seed the “master” branch
  3. create “prod” (from “master”) and “devel” branches
  4. pre-commit hooks (both branches): run a “/etc/init.d/icinga2 configcheck” on the content
  5. post-commit hook on “prod”: export to running directory and “/etc/init.d/icinga2 configcheck”

Let’s see how this goes…

Read the rest of this entry »

Blog Engine Moved Again! Hello world!

Uncategorized No Comments »

The underlying blog engine for the blog stack was moved again… and since I used an import to sanitize the data, some things get restarted.  This isn’t as fatal as past moves, but we do get another “Hello World!” post 🙂

10 PRINT 'Hello World!'
20 GOTO 10

DCNM CIMXML CQL Notes

self-notes No Comments »

A while ago — ages, I’m certain — I refused to do a CIM_XML client for DCNM to be able to write name/pwwn pairs. It had nothing to do with us: it was a CISCO thing to a CISCO service, and didn’t even touch our application.

Then I refused again.

Then I refused again.

Finally, I figured I’d give it a look, but I cautioned heavily that there are a number of risks, and we needed a way to test.

So here we are, weeks later, lacking a method of testing. Because we’re not Cisco, and we’ve only purchased a little bit of Cisco product, Cisco isn’t delivering demo licenses — so we cannot test the CISCO thing talking to the CISCO service because we’re not Cisco. If I know how this is going to go: the PM is going to give up and invalidate many late nights of investigation and work behind “oh well, it was a cheap hack that took no time, right?” BTW: software is hard.

Memo to my future self (beside “just say ‘no'”), these tabs are where I got my information and where debugging notes can be reaped:

  • http://www.cisco.com/c/en/us/td/docs/switches/datacenter/mds9000/sw/nx-os/smi_s/programming/guide/smi_s/MOF.pdf
  • http://www.cisco.com/c/en/us/td/docs/switches/datacenter/mds9000/sw/6_x/programming/guides/smi-s/smi_s/smis_client.html#wp1157277
  • http://www.cisco.com/c/en/us/td/docs/switches/datacenter/mds9000/sw/5_2/programming/guides/smi-s/smi_s/profiles.html
  • http://sblim.sourceforge.net/cim-client2-doc/
  • http://sblim.sourceforge.net/cim-client2-v2-doc/javax/cim/CIMElement.html#getName()
  • http://sblim.sourceforge.net/cim-client2-v2-doc/org/sblim/cimclient/discovery/WBEMServiceAdvertisement.html
  • http://cvs.opengroup.org/cgi-bin/viewcvs.cgi/pegasus/src/Clients/cimcli/

In the Brocade space:

  • http://www-01.ibm.com/support/docview.wss?uid=nas7f530c1afd86f53d8862577c40012b9de
  • Really cool Brocade breakouts: https://github.com/dell-asm/dell-brocade/blob/master/lib/puppet/brocademodel.rb
  • It’s better because it’s in ruby: https://github.com/dell-asm/dell-brocade/blob/master/lib/puppet/type/brocade_alias.rb

In the LMI Space:

  • https://fedorahosted.org/openlmi/wiki/CimProviderHowto

… and a (nonworking for me) client that gives really cool toMof breakouts:

  • http://ter2011asic.googlecode.com/svn-history/r87/trunk/src/parser/wbem/ClientWbem.java

Finding and Running Java

howto 1 Comment »

As a reference, I wanted to leave a few notes as to how to find a java runtime.

If you’re on a linux system (RPM-based or Debian-based including Ubuntu flavours), BSD (including OpenBSD, NetBSD, or the MacOSX fork), or even USL Unix (including AIX flavours and the other 5999 licensees), chances are typing “java” at a prompt does something. “java -version” should spit out a version.

If you’re on Windows, then you might already have Oracle’s Java package installed; look down the various “Program Files” directories for “Oracle” or “Java”, or if search is working today, try searching for “java.exe”. Additionally, BNA and DCNM (and their variants: HPNA, CMCNE, etc) run on Java, so you might find a java.exe in a Cisco\Network Manager\jre\bin\ directory.

Either way, “java.exe -version” or “java -version” should spit out a version number.

If you don’t have a Java interpreter, then you’ll need to install one. That changes form platform to platform, but again Linux variants have it easiest, typically typing “yum install java” or “zipper install java”, or “apt-get install java”. … or some variant of that. Be aware, you might be looking for a java-1.6.0-openjdk (or for the SuSE Experience, where underscores are preferred over consistency, java-1_6_0-openjdk).

In windows, search where you normally search. If you need to go to Oracle’s website, I assume you have web access.

In either case, if you’re unable to install Java, then the type of guide I tend to write might be a bit challenging. you may need some help from your colleagues in IT.

Also, very important, javaw is for windowed apps; we do not want javaw nor javaw.exe unless we’re running a windowed application.

Kernel: sandbox: xcscredd(141) deny file-read-metadata /Users

Uncategorized No Comments »

I keep getting this message on MacOSX 10.9.1 repeatedly:

10:42:47 Kernel: sandbox: xcscredd(141) deny file-read-metadata /Users
10:42:47 Kernel: sandbox: xcscredd(141) deny file-read-metadata /Users
10:42:47 Kernel: sandbox: xcscredd(141) deny file-read-metadata /Users
10:42:48 Kernel: sandbox: xcscredd(141) deny file-read-metadata /Users
10:42:48 Kernel: sandbox: xcscredd(141) deny file-read-metadata /Users

I found Leland Wallace‘s post on Apple’s self-help Support site, and tried the following:

*** /Applications/Server.app/Contents/ServerRoot/System/Library/Sandbox/Profiles/xcscredd.sb	2013-11-11 11:59:19.000000000 -0800
--- /Applications/Server.app/Contents/ServerRoot/System/Library/Sandbox/Profiles/xcscredd.sb	2014-02-11 11:03:50.000000000 -0800
***************
*** 62,68 ****

  ;; suppress spurious failures due to keychain notifications from xcscredhandler
  (deny file-read*
!        (subpath "/Library/Server/Xcode/Credentials/Data")   ;; we have no buisness looking in here
         (with no-log)
          )

--- 62,69 ----

  ;; suppress spurious failures due to keychain notifications from xcscredhandler
  (deny file-read*
!        (subpath "/Library/Server/Xcode/Credentials/Data")   ;; we have no business looking in here
!        (subpath "/Users")                                   ;; https://discussions.apple.com/message/23570150#23570150
         (with no-log)
          )

Let’s see if that silences a bunch of unnecessary log-noise.

WP Theme & Icons by N.Design Studio
Entries RSS Comments RSS Log in