Tech on C&P

My company’s flagship product collects in-depth content via SNMP. Outside of random port resets and some bit-shifting, switches usually give us their information accurately matching the port metrics they show in a portstatshow.

snmpget -v1 -c {community} {switch IP} 1.3.6.1.3.94.4.5.1.33.{portIndex – 1}

for example: port 91, switch 192.168.1.1, community BatM4n:

snmpget -v1 -c BatM4n 192.168.1.1 1.3.6.1.3.94.4.5.1.33.90

Successive checks of this value can show the deltas that VW is reporting.

I’m using the following:

• http://community.brocade.com/docs/DOC-1733 for MIB download
• FE_RFC2837.mib defined a mismatch as fcFeMIB.fcFeMIBObjects.fcFeError.fcFxPortErrorTable.fcFxPortErrorEntry.fcFxPortLinkResetIns
• FA.mib defines the corrected value as FA.experimental.fcmgmt.statSet.connUnitPortStatTable.connUnitPortStatEntry.RxLinkReset.{port}
• ~.TxLinkReset.{port} is available as 1.3.6.1.3.94.4.5.1.34.{portIndex – 1}
• http://www.filibeto.org/sun/lib/nonsun/brocade/manuals/6.2.0/53_1001156_01_MIB_v620.pdf

Meanwhile, OIDView is a great tool if you’re not used to using “vi” as your MIB browser. In this case, it would have helped me find the values (rather then “/” to search), but I’d still need to choose the correct MIB to start with.

My sister plays WoW, and had some latency issues. Rather than go to a higher-speed WAN connection (hey, Wifi-B works OK for most people, but not when you’re raiding) she drilled a bunch of holes in her floors and went direct-wired.

Not to nock direct-attached LAN connections: it’s faster overall, and your latency/jitter in the environment cannot be influenced by a steel stovepipe, or driving a car between your PC and your router. unfortunately, it may have the effect of switching the upstream bottleneck (of data blocks or ACKs stuck behind them) to the router.

Since Wifi bandwidth already exceeds Broadband bandwidth, your speed won’t go up by doing this, but latency improves (insert the first dweeb quoting Linus Torvalds on a “because Linus Sez So! Linus 3:16!” quote here)

Latency can also be a factor of buffering in terms of number of sliding windows, window size, etc. In cases of raw video, you can get better performance (ie less jitter) at the cost of a few dropped frames if you reduce your buffering, for example.

I would take notes on performance (which might be a subjective “feels better” or “feels sluggish”) and then twist a few knobs, as follows. DO NOT change more than one at a time, lest those changes be misattributed to the wrong change.

• reduce net.inet.tcp.sendspace
  • sudo sysctl -w net.inet.tcp.sendspace=250000
  • make sure that kern.ipc.maxsockbuf = (net.inet.tcp.recvspace + net.inet.tcp.sendspace)
  • net.inet.tcp.sockthreshold may need to be set lower (0 to disable) so that sendspace/recvspace are respected earlier on
• reduce net.inet.tcp.mssdflt to 1500 – (20 * wrapper) —
  • in most cases, this is 1480, because 20 bytes overhead for a PPPoE link
  • it’s OK to reduce that further without a huge drop in performance
  • further drop because of WiFi? Not logical, but it does protect your stream in the event of unseen X-over-Y tunneling
  • 1440 is OK on a local LAN, even a gigabit; if all LAN members permit jumboframes (9k), use 8940
• I’m not sure there’s benefit to increasing net.inet.tcp.win_scale_factor above 3 (for gigabit ethernet) because the bottleneck at the router and cablemodem/DSL will only be exacerbated. The congestion should be caught at the desktop to avoid filling the queue at the cable/DSL for outbound traffic.

These reductions are an attempt at reducing queuing at various hops that can reduce the effectiveness of TCP’s congestion algorithm.

If I get other ideas, I’ll add them here.

As always, Netalyzr is a good first-flinch when checking out an unknown network, even if you think you’ve used that net for months.

I was analyzing a situation at a local company, and found that large buffers on low-latency connections are counter-productive.

Consider the following:

• Big server has 12x HBAs to Fibrechannel SAN. This could be anything: teamed NICs, for example
• big writes (for example, 12 MByte) load-balanced (each HBA takes 1MB buffers to send)
• all 12 buffers need to arrive at once for a two-stage commit

So the problem is that in Fibrechannel, this is breoken up into2112-byte frames (think like an MTU of 1480 or 1500 in Ethernet). The smallest atomic chunk is 2112, so the megabyte is actually 497 frames. If any of those frames is discarded or corrupted, the session is retransmitted.

The important impact of the third item above is that it’s actually 12MByte in a single transfer, only “shotgunned” by loadbalancing, but all must arrive, or none must. This means that (12 x 497) frames must all arrive, or all need to be retransmitted (as a result of the host-side multiplexing — SAN faithfully sends the other content perfectly fine)

So with 5964 frames, you only need a 1/10000 failure rate to cause every second transmission to fail. At 1/100000, 1 in 20 fails.

In the multiplexing application’s recovery phase, it needs to wait 30 seconds for a failure in some cases: even though FibreChannel immediately aborts in 496 of 497 failure cases, the multiplexing doesn’t get alerted until its own timeout has expired. It seems that this might be created for slower connections, such as across IPIP, FCoE, DWDM, or similar slower-than-it-seems connections with larger latencies.

That means that a system processing 51MBit/sec, or 6.4 MByte/sec, can buffer a sequence of 193MByte before a retransmission is required. If that happens 1/20, then you’ll only get (Poisson distribution, I know) 228MByte between failures which gives you roughly 44% efficiency.

Part-and-parcel with that, a failure will show a huge delay (30 sec) in response time while the failure is getting detected during which no transaction can hit the storage; when it finally gets freed up, the backlog of nearly 30sec needs to be retransmitted. The failure cases may occur more frequently when data ramps up (such as link-level congestion exhausting buffers). That means that during the busiest times, the failures will occur more frequently, and can cause neighbouring systems sharing resources to similarly be impacted in a flip-flop action like trading around the “Old Maid” in a childrens’ card game.

So what happens when you reduce the session size? What about 8k pages, which give you 4 frames per session? Similar to cut/join FTP uploads to reduce the retransmission cost, more of those 8k pages arrive, and although a 30-sec timeout is still a 30-sec timeout, the in-flight retransmission is only (4×12) 40 frames. Less than 1% of the big buffer cost, similar to the difference is size of each buffer. Efficiency drops, since a 3.1% inefficiency (8448/8192) replaces a 0.104% inefficiency (1049664 / 1048576 (both *1024)) but the overall throughput in a sub-optimal situation should be much higher due to reduce retransmission.

Reducing the timeout in the multiplexer application should reduce the retransmission cost so long as the timeout is not too high that successful transactions are failed. Considering Fibre’s fast response time (typically 3ms, rarely exceeding 12ms during spike situations so long as no single server has too much queue depth to rob the SAN of its buffer resource)

It would be great to command a Skype status message from my meetings.

I get meetings through the day; sometimes, I’m the guy presenting a gotomeeting (which is a great tool), during which times I have to hide the constant stream from the Skype window. The bad part of that is that I cannot see chatter from coworkers… well, that’s not the bad part. The bad part is that when they send me messages, they assume I see them.

How can I remind them that I cannot see the messages? (should I care?)

One idea is to link the showing of my screen to a status that says “I’m ignoring Skype right now”; for which I looked at the Skype API. There’s scant detail, and the easiest (lightest way) seems to be an Applescript remote-executor. An rexec() proxy?

1. wakes-on-LAN using a UDP to a known port, from which the service starts, and timesout 5 minutes after idle? (this was a standard feature of inetd-run UDP ONC/RPC)
2. reads UDP-based command with PKI to confirm permissions (ie “here, take this cookie”)
3. sends the request to Skype (whitelist of accepted commands?) and sends the response back. The response might be too big for UDP out-of-order delivery

The net effect is a few days’ work (OK, a few hours, but try to get that in a single calendar day) I could tell my coworkers that I really am ignoring them. …or I can let them figure it out, because I work with really smart people (they tend to be smart about non-details, no sarcasm here, they really are very smart)

How to “ping” a system using SNMP… but why? We all know that Ping is an ICMP protocol-based tool, and that firewalls treat different protocols, well, differently.

Clearly, if an ICMP round-trip or timed decay works, then there is some routing, but that doesn’t prove, nor disprove, that SNMP gets through.

The most direct method would be a basic “hi, what’s your name?”:

snmpget -t 5 -r 2 -v 1 -c public 192.168.0.1 1.3.6.1.2.1.1.1.0

In essence: “timeout 5s, retry 2″, “version 1, authentication: community: public”, that’s to be expected, and will flex in different protocols.

The IP address is obvious too; the OID is the “what’s your name?” part:
$ snmpget -t 5 -r 2 -v 1 -c public 192.168.0.1 1.3.6.1.2.1.1.1.0
SNMPv2-MIB::sysDescr.0 = STRING: Linux UberHugeDiskNAS 2.6.32.12 #1594 Fri Feb 25 19:01:31 CST 2011 ppc


The equivalent in sapwalk2.exe?

sapwalk2.exe -v v1 -c public -i 192.168.0.1 -s 1.3.6.1.2.1.1.1 -n 1
#sapwalk: ver 2.7
#Copyright (c) 1994-2006 SIMPLESOFT Inc.
#Address=192.168.0.1, StartOid=1.3.6.1.2.1.1.1
#TimeOut=90000, MaxRetries=3, CompareFlag=0, DebugFlag=0
1.3.6.1.2.1.1.1.0 , OctetString , Linux UberHugeDiskNAS 2.6.32.12 #1594 Fri Feb 25 19:01:31 CST 2011 ppc
#ERROR: Walk terminated as max variable count [1] exceeded.


(too easy!)

After thrashing with sed, awk, and various other attempts to cleanly edit XML, I kept thinking “why don’t I use xpathset”?

xpathset is a tool based on an example in xmlsoft — I’ve got a copy on my cnp-tools project — but it runs on Linux, and my employer uses Windows for all supported installs of its Java-based product. That seems a non-happener.

Sometime last night, I remembered that we are a Java company, and I can compile and share Java code without incurring support issues, additional compiler toolchains, licenses, etc while still leveraging the underlying strength of my coworkers where needed.

From 7pm to 9pm I built XPathSet.java using XPathTool.java to effect behavior similar to xpathset; it took another 4 hours to clean up and document, but it’s available if you would like to re-use it. Although I didn’t open up xpathset while I was working (xpathset based on an xmlsoft example, hence MIT license), I’m certain I reused the same concepts I used in that tool, so I should license as MIT.

• Download: ftp://ftp.chickenandporn.com/kludge/xpath.jar or http://r.chickenandporn.com/xpath.jar
• License: MIT
• example: java -classpath xpath.jar nu.old.xpath.XPathSet -i input.xml -r textfile.txt -f "//ScanTask[@name='scanExample1']/@file" -o output.xml

In this example, we are (and it is sequential!):

1. load input.xml
2. registering “textfile.txt” as the replacement value
3. searching for XPath “//ScanTask[@name='scanExample1']/@file” and replacing all matches for “textfile.txt”
4. writing the result to output.xml

The example I did this for is a filename fix in an Axeda EDD_TEXT.xml file. It is possible to read and write the same file, effecting an in-place edit — with Axeda, there may be a timing issue: I’ve found that the file cannot be written on occasion.

The side-effect of the underlying javax.xml.transform technology used is that the attributes are alphabetized on the way in or out of the DOM, so don’t be too surprised if your attributes are re-ordered. Also, indentation may change.

Recently, we have a strange situation that certain critical users cannot log into an FTP server. Of course, Icinga is helping me to check this going forward:

First, define a service check:

define service{
use bidaily-service
host_name ftp.example.com
service_description FTP Login ftp.example.com-scott
check_command check_ncftpls!'ftp://scott:tiger@ftp.example.com/'
notifications_enabled 0
}


Next, catch that odd case when the script itself is missing (in past, payload of Nagios packages has added/dropped parts that I need)

define servicedependency{
dependent_host_name ftp.example.com
dependent_service_description FTP Login ftp.example.com-scott
host_name localhost
service_description Runnable check_ncftpls
execution_failure_criteria w,c,u
notification_failure_criteria w,c,u
}


Finally, the script itself:

#!/bin/bash

echo "${NCFTPLS} failed"
exit 2


Now, I could’ve/should’ve used the hostname in the check itself, but I was more interested in just getting it there. I will probably clean it up someday, make it more reusable, but there it is.

Note that I did not establish a dependency on the ncftpls -bearing package itself in my RPM hierarchy simply because it’s perfectly fine for the “runnable” to fail, and the script itself will never thereafter hit the FTP server until the script it safely runnable. Sure, it’s listed as a failure, but it’s a choice against a huge dependency that typically brings in 100 packages of inconsistent perl and such (hey, “just hit cpan”, they’ll do that in datacenters, sure)

I’m updating my LDAP patch for Nagios based on the most-recent release; I’m also doing it as a git repos so that it’s reusable in a more independent way.

First, there are a few non-LDAP-specific changes needed:
1) commit 06d6ca4e7dfc44b1f93dcd836625ec20a1bbc3f1 — use true/false rather than only 0/1 for booleans
2) commit b37f9f5cbc8cc93796ec68d7f7359634eca56ed3 — propagates EPOCH and BROKER build flags through specfile

Next, there are LDAP-specific changes:
1) commit 561f2521aac88244694dcd0ea264acaa3c6796a2 — read in the LDAP-based config as described in http://wiki.nagios.org/index.php/LDAP-Configured_Nagios

This is all available in git://git.chickenandporn.com/nagios.git

I haven’t ported over my test-harness, so it’s fairly unknown code right now. I’m using it, but shifting back to Icinga.

I was creating a dropbox for photo-import, and I found that I could not select iPhoto’s “Auto Import” folder for sharing.

Instead, I found that “creating an alias” (ie a softlink) gave me the solution:

1. control-click or right-click the “Music” folder, “Show Package Contents” to see inside
2. Navigate to the “Auto Import” directory
3. Right-click Auto Import, “Make Alias”, and drag the “Auto Import Alias” to your desktop or home directory
4. System Properties, “Sharing”, check the “File Sharing” to activate sharing
5. Click the “+” under the list of shared folders to add a folder, and navigate to your “Auto Import Alias” — but don’t click OK
6. Clicking a second time on “Auto Import Alias” will turn the filename in the top of the browser to “Auto Import” — click OK now
7. On the newly-added “Auto Import” item, select “Everyone” in the “Users” column, and make sure “Write Only (dropbox)” is selected in the third column to make sure no one can read your photos before they’re imported

What you’ve done is “follow symlinks” — follow the Alias to its origin — much like a spawning salmon seeks the streamhead where it hatched (yeah, a sex analogy, but it’s not a sex blog!)

Effectively, the “alias” or “softlink” or “symlink” has allowed you to access a folder that MacOSX probably doesn’t intend you to… software guys would say I was exposing a protected/private member function (“exposing”, “member function” — I swear it’s not a sex blog).

Now, you can sit on a remote system and drag-n-drop photos to the import folder. iPhoto will not import then unless it’s running; if it’s not running, then it’ll import on next startup.

NOTE: allowing anyone to drag-n-drop import files to your photos on a portable laptop might be a risky idea. “seriously, officer, that childporn is not mine”.