rsync on Windows

howto No Comments »

A friend asked me about rsync, and I remembered i hadn’t posted my success story; in short, it works, you don’t need to install all of cygwin, but there are weaknesses in the filesystem exposed occasionally.

in AutoFS Mounted FTP via FUSE, I discussed setting up an FTP client as a filesystem to do a client-side pull of FTP content.  That turned out quite vulnerable to our IT guy truncating a connection randomly through the firewall: the data stream didn’t seem to handle timeouts, and any process waiting on a data buffer tends to hang.  Pulling data from a filesystem is a kernel-level thing, so a process cannot really abort a hanging FUSE request.  Bad news.

I later found the simplified route, and installed it on the FTP server:

  1. download the ZIP file (
  2. unpack the zip file
  3. use rsync.exe, which uses a cygwin.dll
  4. that’s it

The rsync.exe is a cygwin binary, and acts very similar to its UNIX-y cousins; instead of installing a huge Cygwin stack, the developer has bundled just the necessary parts of Cygwin into the smaller single DLL, and provides it for rsync’s dependency.  There’s a few other files in there, but as an rsync client, I didn’t need them.

The server has a fairly standard rsync server config, with an rsyncd.conf identifying shared directories as filesystems, and such.  In general, if you have a working rsyncd, you can connect to it from this rsync.exe

Finally, my command line, as an example (my rsync.exe and the DLL are in C:Program Filesrsync, hand-installed), is as follows. Note that I’ve broken the line up with back-slashes to show functional groups, but when you run it, you should have it all on one line, without backslashes except the rsync.exe path.

“C:Program Filesrsyncrsync.exe”  

–exclude=some-skipped-file’ –exclude=*a_wildcard_pattern* –exclude=backup*z  

–exclude=backup*.zip –exclude=*.mov –exclude=*.wmv –exclude=.*  

–delete –delete-excluded –chmod=ugo=rX ‘  

-avr    /cygdrive/f/path/to/users/library/

The server has a rsyncd.conf config that says:

path = /shared/docs/library
uid = libraryowner
gid = docs-ro
comment = Library to share to all remote staff
write only = true
read only = false
list = false
exclude = some-skipped-file’ *a_wildcard_pattern* backup*z
incoming chmod = u=rwX

You’ll notice a few things that are unusual here:

  1. I use “-avr” in my rsync command.  “r” should be redundant with “-a”.  try it without
  2. the source and destination paths end in a slash.  I would recommend the same convention.  Be consistent
  3. I used to have /library/* as my source, but on a push, deleted directories as direct children of /library/ are gone, no longer found by the “*” wildcard, so there deletion does not sync
  4. I have a bunch of –exclude options, some of which are duplicate on the rsyncd.conf entry.  Try removing some once you have yours working.
  5. ditto for the chmod — they should act similarly, and setting it on the server sets a consistency and avoids users forgetting
  6. my uploads are write-only — I use a different share label in rsyncd.conf for uploads than I do downloads so that those who have the keys to read can’t butcher my stuff.
  7. for those crazy A:, C: drive letters, you can use the cygwin special path /cygdrive/a/ or /cygdrive/c/ to map to them without playing the game of escape slashes: “was that 21 backslashes for 10 laters of scripting, or 22?”
If your transfer isn’t working (i.e. transfers nothing, or transfers it all every time):
  1. try adding “–stats” to get a list of what rsync is doing
  2. try adding “–max-size=20k” to avoid pushing HUGE files while diagnosing issues
  3. if the datestamping is seriously butchered on your windows boxes, try “-c” option to use checksums, understanding it’ll take (much) longer to sync
  4. if you habitually rename files, and rsync faithfully re-pushes the file every time, “-y” or “–fuzzy” tells rsync’s receiver to check around for similar files to use for missing files

Good Luck!

Connect to VMs Without Installing CoRD

redirection No Comments »

When I have to connect to Windows systems, except for the known issues with the en_CA language, CoRD is a great Remote Desktop.  If I could connect using MacOSX-bog-Standard stuff, then clearly this en_CA issue wouldn’t happen… and will show you how to do that. Read the rest of this entry »

Inaccuracy may Inhibit

Uncategorized No Comments »

Consider the effect of inaccuracy on Engineers when posing a task or asking a question.  It may reveal why you’re not getting what you expect.

Read the rest of this entry »

Kindle Fire VPN

redirection No Comments »

A friend asked how to configure a VPN on Kindle Fire; the short answer is that it’s difficult, resisted by the manufacturer, and can go away in a heartbeat.

Read the rest of this entry »

Users Only Know Possible Passwords

system config No Comments »

Users tend not to know the exact credentials for ServerXY, but rather the possibilities; should we configure for this in software?

Providing configuration options for what a user actually knows versus what would be proper and correct might be faster for the user, and allow for incremental addition of resources, but may offend users or appear confusing.

Read the rest of this entry »

Why SOPA and PIPA are incapable of helping, while still dangerous

Uncategorized No Comments »

SOPA and PIPA are two attempted bills in the US that seek to stop online piracy.  These two acts are incapable of actually stopping piracy, yet remain fully capable of an “internet death-penalty” against innocent sites.

Consider arriving in a strange, new city during the era of prohibition.  In order to find stores, markets, and local associates, you pick up two telephone guides.  Remember the White Pages and Yellow Pages?  Consider those as your resources, except that your phonebooks update instantly.

You’re able to look up your friend Mike, who has invited you to dinner.  His address is in the white pages, telling you “Mike Smith: 123 Main St”, and you know how to get there.  Mike lives above a tailoring shop, and has a market nearby, which you’re able to find to buy some beef enroute to dinner.

What about a bottle of wine?  That would be nice, but it’s prohibition, everything with any alcohol is illegal.

On the way to Mike’s, beef in-hand (a couple of excellent steaks), you notice that the tailor shop below him is closed, yet many people keep entering and leaving.  That’s odd.  Mike is happy to see you when you get to his door, and he makes a great dinner; he suggests doing so again next week.

Next week, Pork’s on the menu, but you have to look up the market again.  You wrote down the name (Joey’s Eastside Butcher) and your whitepages tell you where it is.  Mike tells you on the phone to be careful, police have detected that the Tailor shop is a Speak-Easy, and is illegal.  As a lawful citizen, despite that you’d like a drink occasionally, you tend to stay away from that sort of place anyhow.

Off to the butcher’s, but wait… where’s Mike’s place?  The whitepages no longer list his building at all.  From your always-updated whitepages, Mike’s entire building is gone.  You ask a friend, they have one that doesn’t update as quickly, and it shows that last week, Mike’s address was 123 Main St.  yeah, that’s right.  Another friend has a version of whitepages from Germany, and although in a different language, it does show the local city, and it agrees: Mike’s address is still 123 Main St.  Good thing you wrote down Mike’s phone number itself rather than his name, or you’d lose all contact with Mike.  You should get yourself a German phonebook, or use one that doesn’t delete entries, just adds them.

Arriving at Mike’s building, you notice that the Speak-Easy is doing a brisk trade.  It hasn’t been shut down!  The police haven’t lifted a finger, just “hid” it by removing Mike’s building from the whitepages.  Everyone who goes there often knows exactly where it is; if they haven’t, they’ve written down the phone number already, or have it on speed-dial.

This is how these new bills work: they don’t stop crime, they simply allow it to become unlisted quickly, without any recourse, any due-process.  They just make it harder to find, quickly.  The internet whitepages is called “DNS”, and is controlled by many different countries.  DNS updates take up to 72 hours to occur, and even after that time, the services are still open.  Anyone using these services knows where to find them without having to look them up, but if they need to, they can use alternative listings to find the same address.  The piracy isn’t even affected, it continues unabated, but common people are affected: new arrivals at a website or an interest group, and those who have to find their websites when they move (which happens about as frequently as people change homes)

Worse, with some allusion to Brazil, and perhaps to some implementations of photo-radar speed-traps, there’s no double-checking of errors, no due-process.  Due to an error, not only is a building, and all it’s businesses gone, while they are barely making it through the recession.  I’m sure the people who think this is a good idea are not the small businesses and private citizens who can vanish immediately due to error.  … and we know the Department of Homeland Security makes no mistakes.

Would you give the “Internet Kill Switch” to the TSA?

Criminals will simply go to other countries’ DNS, or just use the numbers, leaving only the lawful to be the victims of bad legislation and the zealots who support it.

Crime should be stopped by actually stopping crime, not by making it and its innocent neighbours into unlisted addresses.

Naming Consistency: Make a Script

Best Practices No Comments »

If you want a consistent, easy-to-use repository, use a script to build directories and copy in your content, which gives consistency as a side-effect of making things easier to move around.

Anyone who has tried to download software form a repository can tell when the ownership changes hands: the directory structure changes in subtle ways.  There’s a dot in the path now, there wasn’t before, or capitalization changes.  This isn’t a problem until you try to use the repository in an automated fashion: scripting and tools.  Suddenly, a change from “V” to “v” requires an entirely new case, as if it’s a whole new repository on a different server.

NOTE: if the files are moved around manually, and the owner of those hands doing the moving is a bit flakey or random, then this sort of speed-wobble might as well count as changing the ownership, only more frequently (every release)

People will have a problem with this, but they’ll never tell you just as they’ll never tell you that your shoes don’t match your belt… but unlike fashion faux-pas, inconsistency with directories actually impacts others.

Don’t be a flake.  Be consistent.  A script helps you do that.

Additionally, if the script is the final part of the build process, it reduces the manual steps to a build.  I would recommend either right before or right after running your self-tests.

Cloaked Twittering in Dangerous Places

dataflow No Comments »

Recently I’ve been bothered by the concepts such as censorship, Arab Spring, crowd discussions.

Many of the censorship tools in-use are just tools built by engineers who are not political, just building tools. Just doing their jobs. Often the desire to fulfill a challenging objective can blind the engineer to the possible uses — or the engineer simply doesn’t care (i.e. has bigger issues to care about than some foreign country’s citizens’ free speech).

I have an Idea I’d call Qloak (compression of “Quacking” and “Cloaked”, and “Quacking” based on what Chinese guys call gossip) that would allow:

    • twitter posts to get through firewalls and most paywall wifi APs
    • foursquare checkins to also get through
    • ability to check whether an app needs to self-destruct, flushing history

A lot of this technology is the way I used to configure the “ext” system as a phonebook at a past employer; as well, acting as the head of a TOR or a VPN connection may consistently get through.

I don’t judge Eqypt, or Libya, or China, but I worry over the limiting/chilling/hushing aspect of some engineering talent mis-applied.

I would prefer that more people are in the conversation.

Countries, Companies, people who claim to support freedom of speech should act to support it. Build a TOR gateway. Support free opinions in other countries. Listen to everyone, even the Gay, the Religious Fanatic, the Type-B personality, the Nature Fanatic, the Raging Republican — whatever grouping you put people into, those people will be oppressed in other countries. Listen to them, however much you may disagree.

Yes, if I built an App for this, I would give away free signups to anyone at an email with a domain such as .cn, .ly, etc.

WP Theme & Icons by N.Design Studio
Entries RSS Comments RSS Log in