Fuck you, git

Uncategorized No Comments »

Try number 2 for git.

I’m not a genius, but for fucks sake, why did my one-step action become 26?

git rebase, git merge, git commit, git add (a file that’s already added), git commit (again), git rebase, git merge, git git git git gitigitigitigititititit

this replaces:

svn update

I seriously don’t yet see a compelling reason to change.

Fuck you, git. seriously. fuck you.

Expanding the Reach of Real-time Monitoring

Best Practices, Real-Time Monitoring, SAN, SNW, VirtualWisdom No Comments »

It’s been a busy and exciting few months at Virtual Instruments. At SNW Fall last month, we introduced the new high-density VirtualWisdom SAN Performance Probe. By doubling the density and supporting up to 16 Fiber Channel links per unit, the ProbeFC8-HD enables customers to monitor more of their infrastructure for less. In fact, customers can expect to reduce the cost of real-time monitoring by 25 percent and lower power consumption by 40 percent.

We also announced enhanced support for FCoE. With FCoE-specific enhancements to the current SAN Availability Probe module we’re able to deliver improved monitoring of top-of-rack FCoE switches, extending visibility into infrastructure performance, health and utilization across converged network environments.

We had the chance to meet with a number of customers, press and analysts at SNW to share our news. Check out the news and learn more about our VirtualWisdom platform, courtesy of W. Curtis Preston, Truebit.TV.

How to Re-Use VirtualWisdom Data in 3rd-Party Tools

BAT, How to, samplecode, Scheduler, VirtualWisdom No Comments »

Everyone I’ve personally met who has witnessed the detail of VirtualWisdom metrics tends to be first amazed, then relates it to LAN and ethernet tools, then questions why we haven’t seen this before. The next question in very large organizations is “how can we re-use this data in our [insert home-grown tool here] ?”

Incorporating VirtualWisdom into an organization has various points of “friction”: training on a new tool, understanding the metrics, collection of data to help VirtualWisdom correlate, and beginning to use it internally. As a Virtual Instruments Field Application Engineer (AE or FAE), I tend to see the initial friction (collection of data, such as nicknames, or grouping Business-Units as UDCs. The less common friction is “OK, we love VirtualWisdom, but our expansive storage team want to exploit the metrics in our custom home-grown planning tools”.

Converting VirtualWisdom into basic data-collector ignores the reporting, recording, and alerting capabilities it offers; re-using its data in multiple entities of a corporation is an expansion on VirtualWisdom’s utility, and I’m more than happy to help a customer do that. The more we can help our customers make an informed decision — including leveraging more data in their own tools — the more we can help our customers “free their data” and improve the performance and reliability of a complex data-storage environment.

My entries here on the Virtual Instruments Bast Practices blog tend to be of a how-to nature; in this article, I’d like to show how the opensource tool “MailDropQueue” can help push VirtualWisdom data into your home-grown toolset.

 

There was a time that customers tried to find our reports by digging through the “exports” directory after a new report was produced — because the most recent report.csv.zip is the correct one, right? This ran into problems when users generated reports just before scheduled reports, and when the scheduled “searcher” went searching, the wrong report would be found. Additionally, some reports took a long time, and would not always be finished by the time the customer’s scripts went searching. Customers typically knew what script they could run to consume the data and push it to their own systems, but the issues in finding that file (moreso due to a lack of shell on Windows) caused this solution to become over-complex.

Replication at the database level gives us the same problem: the data is in a schema, and it’s difficult to make sense of without the reporting engine.

A while ago, VirtualWisdom gained the ability to serialize colliding reports: if a user asks for a report at the same time the system or another user is generating a report, the requests get serialized. This allows VirtualWisdom to avoid deadlock/livelock situations at the risk of the delay we’re used to at printers: your two-page TPS Reports are waiting behind a 4000 page print of the History of the World, Part 1A. The benefit of a consistently-responsive VirtualWisdom platform are well worth this benefit. Unfortunately, the API that many users ask for poses this same risk: adding a parallel load onto VirtualWisdom that needs an immediate response, adding delay in both responses and risking concurrency delays at the underlying datastore.

The asynchronous approach — wherein VirtualWisdom can generate data to share through its reporting engine — is more cooperative to VirtualWisdom’s responsiveness, but returns us to the issue of “how do I find that report on the filesystem?  The correct report?”

MailDropQueue is a tool in the traditional nature of UNIX: small things that do specific jobs. UNIX was flush with small tools such as sed, awk, lpr, wc, nohup, nice, cut, etc that could be streamed to achieve complex tasks. In a similar way, MailDropQueue receives an email, strips off the attachment, and for messages matching certain criteria, executes actions for each.

It’s possible for VirtualWisdom to generate “the right data” (blue section, above), send it to MailDropQueue (red portion, above), and have MailDropQueue execute the action on that attachment (green part above).  In our example, let’s consider where a customer knows what they want to do with a CSV file; suppose they have a script such as:

@echo off
call DATABASE-IMPORT.BAT TheData.CSV

The actual magic in this script isn’t as important as the fact that we can indeed trigger it for every attachment we see to a certain destination. Now all we need is to make a destination trigger this script (ie the green portion of the diagram above):

 <?xml version='1.0' ?> <actions> <trigger name="all"> <condition type="true"/> <action>IMPORT</action> </trigger> <script id="IMPORT" name="import" script="DATABASE-IMPORT.BAT" parameters="$attachmentname"/> </actions> 

From the above, the “condition type=true” stands out, but it is possible to constrain this once we know it works, such as to trigger that specific script only when the recipient email matches “ftp@example.com”:

 <condition type="equal"> <recipient/> <value>ftp@example.com</value> </condition> 

Also, it’s not so obvious, but the result of every received email that matches the condition (“true”) is to run the script with the attachment as the first parameter. This means that if an email arrives with an attachment “performance.csv.zip”, MailDropQueue would Runtime.exec("DATABASE-IMPORT.BAT performance.csv.zip").

For reference, I’m running this on a host called fakemta.example.com, compiled to use the default port (8463) as:

java -jar maildropqueue.jar -c maildropqueue.xml

Where maildropqueue.jar is compiled by defaults (./configure && make) from a “git clone”, and maildropqueue.xml contains the configuration above. There’s a downloadable

Finally, We need to configure VirtualWisdom to generate and send this data attached to an email; this is a fairly simple problem for any VirtualWisdom administrator to do.  Following is a walk-thru up to confirming that the content is being generated and sent to the MailDropQueue; the composition of the report and the handler script “IMPORT-DATABASE.BAT” is too environmentally-specific to cover in this article.

  1. Create the report (outside the scope of this article) — confirm that it produces output. The following snapshot uses our internal demo-database, not actual customer data:

    Capacity / Performance Statistical Report

  2. Create a Schedule to regularly generate and send it:
    1. In the Report Generation Configuration, check that you have the hourly summary if so desired:
    2. Check that all probes are used, but you don’t need to keep the report very long:
    3. Confirm that you have file-format set to CSV unless your handler script can dismantle XLS, or you intend to publis a PDF:
    4. Choose to send email; this is the key part. The message can include anything as subject and body, but you must check “E-mail report as attachment”:
    5. …and finally: you may not yet have the distribution list set up, consider the following example. Note that the port number is 8025, and server is localhost because in this test, I’m running the MailDropQueue on the same server. The sender and recipient don’t matter unless you later determine which actions to run bases on triggers matching sender or recipient:
    6. Check that your MailDropQueue is running on the same port (this is an example running MailDropQueue using VirtualWisdom’s enclosed Java and the config example above; the two “non-body MIME skipped:” messages are from clicking “Send Test E-Mail” twice):
  3. Finally, run your MailDropQueue. The skip used above is shown here (except that running it requires removing the “-V”, highlighted), as well as the config, and an output of “java -jar maildropqueue.jar -V” to show how MailDropQueue parsed the configfile:
  4. Clicking “Run Now” on the Scheduled Action for the report generation shows end-to-end that VirtualWisdom can generate a report, send it to MailDropQueue, and cause a script to be triggered on reception. Of course, if the script configured into MailDropQueue hasn’t been written, a Java error will result, as shown:
  5. Now the only things left to do are:
    1. Write the report so that the correct data is sent as tables and Statistical Summary reports (only one View per section)
    2. Write the IMPORT-DATABASE.BAT so that it reacts correctly to the zipped archive of CSV files

Quick Reminder: Activate MacOSX 10.8 Remote Display from SSH

howto No Comments »

I got nailed by this, so this blog post is a quick reminder to myself that I hope helps at least one other person out there:

  1. Freshly cold-booted MacMini
  2. 10.8 installed
  3. I’d really like to get into the Server.app to get other stuff configured.  Yeah.  That would be outstanding.

The path to the solution is:

  1. ssh username@newserver.local
  2. /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -activate -configure -access -on -restart -agent -privs -all
  3. RESTART your client-side Screen Sharing to see the fixed remote Macmini.  Yeah, silly as that sounds

Of course, if you have a new freshly unwrapped new-car-smell macmini, then you need to connect using username “root” and password equal to the serial number.  If “root” failed, try a blank username.

What’s Hotter Than SSDs in the Storage Market? Knowing When and How to Use Them.

Best Practices, Infrastructure Performance Management, solid state disks No Comments »

According to searchstorage.com, there are over 400 companies designing and marketing solid state disks today.  Wow.  And according to the folks at the InfoPro, a service of 451 Research, which provides insight based on one-on-one, in-depth interviews with a proprietary network of the world’s largest buyers and users of IT, auto-tiering is perhaps the hottest storage technology today, as FC drives are being replaced with SSD and SAS blends (This figure came from their latest TIP Insight report.)  So I have to ask, are our performance problems over?  OK, it’s a rhetorical question.  But I have to laugh when someone points to IOPS or MB/s as their performance benchmark.  We’ve got plenty of examples where users were getting great utilization numbers, but their application owners were complaining of performance problems.  For example, in the time-correlated dashboard below, the top graph shows R/W MB/s and the middle graph shows application latency.  You probably can’t see the detail in this blog, but latency skyrockets from less than 25ms to over 150ms. Yet, MB/s doesn’t seem to be affected.

So, now that we know we have a problem, the solution is to apply SSD, right?  No.  Under normal circumstances, there is no performance problem and SSDs won’t make any difference.  The bottom graph shows a metric called Pending Exchanges.  In this case, there’s a very close correlation to how the HBA queue depths are set and the effect on application latency.  In this case, the customer adjusted the queue depth settings and the problem was remediated.

Maximizing application performance is one of IT’s most important tasks. The problem is that accurately identifying the root causes of poor performance is difficult, especially when the wrong metrics are used. This often leads to the assumption that storage subsystems are the bottleneck resulting in expensive upgrades, which still may not solve the problem. Through the use of a real-time, systems-level, Infrastructure Performance Management platform and metrics like transaction latency, IT can identify the real causes of application performance issues and fix those problems.  Sometimes, the best solution is to re-tier your application, applying SSD technologies.  But the best practice is to use the right tools to find out where these make sense, and when they don’t.

If you’d like to learn more about how to find the true cause of performance bottlenecks, call us, or call one of our resellers, and we’ll put you together with a performance specialist who can help you figure out when SSDs can help, and when they can’t.  Or check out this new report by Storage Switzerland, on “How do you know if you really need SSD”.

 

 

Merging UDCs in VirtualWisdom to Join Manual and Generated UDCs

BAT, How to, UDC, VirtualWisdom, xmllint, xsltproc No Comments »

UDCs — User-Defined Context — can be very useful for showing the actual use, or membership of a device on the SAN, or for assigning a priority for alerting and thresholds. Often, these are hand-generated, but we do have methods of creating them from other content.

One customer has both: a UDC generated/converted from other content, and some manually-assigned content. Merging these would help him to assign filters and alerts as a single group, but the effort to merge it was looking excessive.

As you may recall, my content on Virtual Instruments Best Practices blog tend to be the how-to variety, and in this article, I’d like to share how to merge two UDCs programmatically, which can then be scripted in any automated collection scripts or tools you’re already using.

Merge, Then Cleanup

The general process we use for this is to merge the content first (using xmllint), then clean it up (using xsltproc) so that it’s back to sane, predictable UDC that is ready for routinely-scheduled import:

UDCs merged using xmllint and xsltproc

Notice in this image that the only things that change are in the upper box (“UDC Files”), which can be either manually-edited or autonomously-generated by filter or transform. As well, the result is a standard UDC from which we can generate filters or otherwise edit using XML tools.

As you can see, the tools used here are fairly standard; the only real development are the smaller scripts for each tool. UDCs are simply XML, and as such, quite easy to manipulate using standard XML tools.

Let’s break this down into the multiple steps.

Concatenation

The easiest way I found to concatenate two XML files was to use XInclude with an XPointer statement:

<?xml version="1.0"?>
<list xmlns:xi="http://www.w3.org/2001/XInclude">
    <xi:include href="File1.udc" xpointer="xpointer(//list/*)"/>
    <xi:include href="File2.udc" xpointer="xpointer(//list/*)"/>
</list>

In parts, this is really duplicates of the following:

<xi:include
href="File1.udc"
xpointer="xpointer(//list/*)"
/>

If you’ve written any sourcecode, you’d recognize an #include statement, or an import com.example.*; this is really no different: the document referenced by the “href” (File1.udc) replaces this xi:include statement. The second part, an xpointer="...", further clarifies the import by indicating that only a part of the document we include should come in — in this case, child elements of the “list” element. If you look at a UDC File, you’ll see that “list” is the root node; if that statement makes very little sense, then think of this as “we’re including all the stuff inside the outermost container, but not the container itself”. And hey, look again at the full file above: we specify a <list> and </list> around the inclusions. Coincidence? Not at all; this is a method of avoiding having two outermost root nodes, which cannot be further altered using XML because XML can only have one outermost root node.

…and it’s easier this way: don’t filter out what you can avoid including in the first case. It’s possible that there’s a better set of inclusion elements here, but this works well enough.

If we had three UDCs to merge, you can see that it would merely require another xi:include statement.

To act on this file, we execute xmllint using the “-xinclude” parameter (normal hyphen, only one “-”, not two) as follows. Note that xmllint is available on most non-Windows systems, and should be easily acquired using Microsoft Services for UNIX for a Windows system.

xmllint.exe -xinclude concatenate-UDC.xml > Merged.udc

for Windows, or for non-windows:

xmllint -xinclude concatenate-UDC.xml > Merged.udc

(using “Merged.udc” as a temporary file)

We now have a UDC file with only one outermost “list” element or root element, but it has a few problems:

  1. Every new UDC starts with Evaluation Order of 1; this is reflected in the UDC, and has to be fixed
  2. Only one default item should be given: we choose the one from the first file
  3. We only copy the first file’s definition of the UDC (Metric, set, etc) so the user needs to avoid doing this on two UDCs of different metric/set (illogical UDCs will result)

The first two issues can be fixed in the next step.

Clean the Concatenated Result

XSLT, or XSL Transformations, uses XSL (Extensible Stylesheet Language) to transform XML into a different XML, or even into a simpler form such as straight text or ambiguous markups such as CSV. In general, XSLT can map XML data from one schema to another, convert data from one schema to another, or simply extract elements of data into a text stream.

In our case, we’re using it to remove the redundant parts that will cause VirtualWisdom’s UDC parser to reject the document. There is currently no schema definition, so we have to make best efforts to make the resulting UDC look like one exported from VirtualWisdom.

The XSLT is a bit complex to post here, but it should be available by clicking on the marked-up filename below. Note that like xmllint, xsltproc is widely available as Linux and UNIX packages, or via the Microsoft Services for UNIX (currently a re-packaged Cygwin environment).

We execute the cleanup XSLT as follows:

xsltproc.exe concatenate-UDC.xsl Merged.udc > VirtualWisdomDataUDCImportCombined.udc

for Windows, or for non-windows:

xsltproc concatenate-UDC.xsl Merged.udc > VirtualWisdomDataUDCImportCombined.udc

Note here that the file we use is similarly-named, but end in “xsl”, not “xml”. Also, we write the file directly into the UDCImport directory of a VirtualWisdomData folder, which is where an import schedule would look for it.

This resulting file can be directly imported; an example import schedule is at the bottom of the Use UDCs to Collect Devices by Name Pattern article presented on May 1st, 2012. As well, because the UDC is in a standard form, it can be used to Quickly Create Filters for use in Dashboards, Reports, and Alarms.

Evolving SANs tend to have evolving naming schemes and assignment methods, so there will often be many different systems of identifiers that can be joined to work with different Business Units, customers, or functional groups; different such groups tend to cause different sources of information to be polled, and different formats to result. I hope this process help you to reduce the manually copying of data attributes which is so prone to human error and scheduling delays.

I hope this helps you to “set it and forget it” on more sources of data. Accurate data drives decisions: how can you methodically fix what you cannot measure and make sense of?

Edit 2012-08-23: concatenate-UDC.xsl was misspelled on the webserver (concatenate-UDCs.xsl), four downloads failed due to my error. Sorry, it should be easier to download now.

Hidden NullPointerException in MimeUtility.java:226

troubleshooting No Comments »

Quick reminder to myself:

Exception in thread "main" java.lang.NullPointerException
at javax.mail.internet.MimeUtility.getEncoding(MimeUtility.java:226)
at javax.mail.internet.MimeUtility.getEncoding(MimeUtility.java:299)
at javax.mail.internet.MimeBodyPart.updateHeaders(MimeBodyPart.java:1375)
at javax.mail.internet.MimeBodyPart.updateHeaders(MimeBodyPart.java:1021)
at javax.mail.internet.MimeMultipart.updateHeaders(MimeMultipart.java:419)
at javax.mail.internet.MimeBodyPart.updateHeaders(MimeBodyPart.java:1354)
at javax.mail.internet.MimeMessage.updateHeaders(MimeMessage.java:2107)
at javax.mail.internet.MimeMessage.saveChanges(MimeMessage.java:2075)
at javax.mail.Transport.send(Transport.java:123)
at SendEmail.main(SendEmail.java:79)

was caused in code similar to (this example: http://java.sun.com/developer/onlineTraining/JavaMail/contents.html )

// Part two is attachment
messageBodyPart = new MimeBodyPart();
DataSource source = new FileDataSource(filename);
messageBodyPart.setDataHandler(new DataHandler(source));
messageBodyPart.setFileName(filename);
multipart.addBodyPart(messageBodyPart);

// Put parts in message
message.setContent(multipart);

// Send the message
Transport.send(message);

In this case, if the “filename” cannot be read, it silently works, but later causes a failure perhaps based on a null-load of the content.

Using VirtualWisdom to Reclaim Unused Disk Space

Best Practices, LUN, SAN performance storage i/o bottleneck, VirtualWisdom No Comments »

I was talking with an independent contractor a few days ago and she mentioned that more than a few customers justify buying storage management tools by using them to find unused disk space. It’s pretty common to find allocated but unused space that often amounts to tens or even hundreds of thousands of dollars’ worth of space. Though VirtualWisdom isn’t thought of as a storage capacity monitor, by watching for I/O activity, you can easily find opportunities to reclaim unused LUNs.

Below is a step-by-step process with screen grabs to illustrate exactly how this is done.

Start your VirtualWisdom Views client, which is the administration interface for VirtualWisdom. The Views client allows you to configure VirtualWisdom, create reports, set alarms and monitor the data collected by VirtualWisdom. Since we are looking for I/O traffic to a LUN, use the SAN Performance Probe to monitor the frames/sec metric in the SCSI metric set; the screen shot is below.

Then select the LUN and storage fields.

Sort the data first by Frames/Sec then Storage and LUN, via the Data Groupings tab.

Then, in the “data views” tab, select the Summary Table to list out each LUN, and a Trend chart to show the peak data for each period. The Trend chart is important because the Summary Table will show the average for a period. It’s important to note that such a small value over a long period could average out to zero. The Trend chart will let us spot these values.

Go to the reports tab in the views client. Set a period, say 30 days, and generate the report. For our small test lab, you can see that our tool found one LUN with zero activity in 30 days. With the SAN Performance Probe it’s easy to inspect the LUN and figure out why it hasn’t had any traffic.

You can use the same report with different selection criteria to look for underutilized LUNs. It’s easy, quick, and the ROI can be substantial. For a short video of this VirtualWisdom use case take a look below:

A Simple Strategy for Reducing Your Reactive Tickets

Best Practices, datacenter migration, VirtualWisdom No Comments »

One of Virtual Instruments’ missions is to reduce the big, messy outages that happen in our customers’ datacenters. For many sectors across the industry, we work to determine ways to catch problems early to avoid outages. Numerous competitors have analytics engines and all kinds of software to detect problems before they become outages, but they are not all equal.

A few weeks ago, I was reviewing a customer’s successful datacenter migration.  Their strategy for success when using VirtualWisdom is to tag all the tickets that result from our solution. The IT staff is told to do the VirtualWisdom tickets first, and the customer found that over time, the number of reactive tickets decreased. What was happening was that VirtualWisdom found all the small problems that might otherwise get overlooked, such as: increases in physical errors that precede the failure of a SFP fiber module, the increase of traffic on a link that creeps over the threshold where it won’t successfully fail over to its backup link, and the misconfiguration of a path. VirtualWisdom quietly and diligently finds these problems, and the customer found if they fixed them, bigger problems were avoided.

I wanted to share this policy with my blog readers because it makes a lot of sense and is simple to implement. If anyone else tries this strategy, let me know how well it works for you.

Using VirtualWisdom to De-risk a Migration / Consolidation Project

Best Practices, SAN performance storage i/o bottleneck, VirtualWisdom No Comments »

I wanted to share a real-life example of how VirtualWisdom can be used to de-risk your migration and consolidation projects:

Recently, one of our customers used VirtualWisdom to help successfully migrate a datacenter, and at the same time, consolidate two mission-critical, Oracle-based applications from two older-generation storage systems to one new storage system.

Pre-migration analysis for the new data center ensured that it was “production” ready.  VirtualWisdom was used to identify naming issues with the zone configuration, an incorrectly configured server from a multipathing perspective, queue depth configuration issues, physical layer problems, and miscellaneous performance concerns.  It’s worth noting that the physical layer issues concerned two links that were found to be borderline within specification at 4Gb, and several other ports that were found to be outside of specification at 8Gb, and were addressed before the migration occurred.  We highly recommend paying particular attention to physical layers issues when migrating to 8Gb SANs, as what worked fine at 4Gb may not work so well at 8Gb.

Before the move, the applications were benchmarked to help increase performance.  During the spin-up of the new site, which occurred on a weekend when traffic was low, VirtualWisdom reported an intermittent latency issue. The latency issue occurred for only a second or two every minute.  The  vendor performance tool that the customer was using could not detect the issue because it was averaging the latency metric and was not granular enough to pick up the anomaly.  The issue was serious enough that the team had to fix it by Monday or they forecasted an outage.  The fall-back plan was to re-deploy on the older storage arrays.  VirtualWisdom, which aggregates metrics to the one-second level, found a process that lasted one second, which was causing the problem.  Once the offending process was identified and remediated, the problem disappeared.  The new site went fully live; the Oracle-based applications functioned as predicted, and VirtualWisdom was able to confirm that the infrastructure performance of the new site, with the consolidated array, met its SLAs.

For more information on how VirtualWisdom can be used to de-risk your migration and consolidation projects, check out this tech brief on private cloud de-risking:  http://www.virtualinstruments.com/files/pdfs/tech-brief-private-cloud.pdf, or this blog on private cloud migration best practices:  http://www.virtualinstruments.com/sanbestpractices/best-practices/three-steps-to-de-risking-migration-to-the-private-cloud/ or this whitepaper on datacenter consolidation best practices at: http://www.virtualinstruments.com/files/pdfs/WP_Storage-Consolidation-Best-Practices.pdf.  If you would like to talk with the customer in this story to learn more, contact your Virtual Instruments account team and they can arrange it for you.

WP Theme & Icons by N.Design Studio
Entries RSS Comments RSS Log in