Notice: register_sidebar was called incorrectly. No id was set in the arguments array for the "Sidebar 1" sidebar. Defaulting to "sidebar-1". Manually set the id to "sidebar-1" to silence this notice and keep existing sidebar content. Please see Debugging in WordPress for more information. (This message was added in version 4.2.0.) in /usr/share/wordpress/wp-includes/functions.php on line 4139 Tech Notes » UDC

Merging UDCs in VirtualWisdom to Join Manual and Generated UDCs

BAT, How to, UDC, VirtualWisdom, xmllint, xsltproc No Comments »

UDCs — User-Defined Context — can be very useful for showing the actual use, or membership of a device on the SAN, or for assigning a priority for alerting and thresholds. Often, these are hand-generated, but we do have methods of creating them from other content.

One customer has both: a UDC generated/converted from other content, and some manually-assigned content. Merging these would help him to assign filters and alerts as a single group, but the effort to merge it was looking excessive.

As you may recall, my content on Virtual Instruments Best Practices blog tend to be the how-to variety, and in this article, I’d like to share how to merge two UDCs programmatically, which can then be scripted in any automated collection scripts or tools you’re already using.

Merge, Then Cleanup

The general process we use for this is to merge the content first (using xmllint), then clean it up (using xsltproc) so that it’s back to sane, predictable UDC that is ready for routinely-scheduled import:

UDCs merged using xmllint and xsltproc

Notice in this image that the only things that change are in the upper box (“UDC Files”), which can be either manually-edited or autonomously-generated by filter or transform. As well, the result is a standard UDC from which we can generate filters or otherwise edit using XML tools.

As you can see, the tools used here are fairly standard; the only real development are the smaller scripts for each tool. UDCs are simply XML, and as such, quite easy to manipulate using standard XML tools.

Let’s break this down into the multiple steps.

Concatenation

The easiest way I found to concatenate two XML files was to use XInclude with an XPointer statement:

<?xml version="1.0"?>
<list xmlns:xi="http://www.w3.org/2001/XInclude">
    <xi:include href="File1.udc" xpointer="xpointer(//list/*)"/>
    <xi:include href="File2.udc" xpointer="xpointer(//list/*)"/>
</list>

In parts, this is really duplicates of the following:

<xi:include
href="File1.udc"
xpointer="xpointer(//list/*)"
/>

If you’ve written any sourcecode, you’d recognize an #include statement, or an import com.example.*; this is really no different: the document referenced by the “href” (File1.udc) replaces this xi:include statement. The second part, an xpointer="...", further clarifies the import by indicating that only a part of the document we include should come in — in this case, child elements of the “list” element. If you look at a UDC File, you’ll see that “list” is the root node; if that statement makes very little sense, then think of this as “we’re including all the stuff inside the outermost container, but not the container itself”. And hey, look again at the full file above: we specify a <list> and </list> around the inclusions. Coincidence? Not at all; this is a method of avoiding having two outermost root nodes, which cannot be further altered using XML because XML can only have one outermost root node.

…and it’s easier this way: don’t filter out what you can avoid including in the first case. It’s possible that there’s a better set of inclusion elements here, but this works well enough.

If we had three UDCs to merge, you can see that it would merely require another xi:include statement.

To act on this file, we execute xmllint using the “-xinclude” parameter (normal hyphen, only one “-”, not two) as follows. Note that xmllint is available on most non-Windows systems, and should be easily acquired using Microsoft Services for UNIX for a Windows system.

xmllint.exe -xinclude concatenate-UDC.xml > Merged.udc

for Windows, or for non-windows:

xmllint -xinclude concatenate-UDC.xml > Merged.udc

(using “Merged.udc” as a temporary file)

We now have a UDC file with only one outermost “list” element or root element, but it has a few problems:

  1. Every new UDC starts with Evaluation Order of 1; this is reflected in the UDC, and has to be fixed
  2. Only one default item should be given: we choose the one from the first file
  3. We only copy the first file’s definition of the UDC (Metric, set, etc) so the user needs to avoid doing this on two UDCs of different metric/set (illogical UDCs will result)

The first two issues can be fixed in the next step.

Clean the Concatenated Result

XSLT, or XSL Transformations, uses XSL (Extensible Stylesheet Language) to transform XML into a different XML, or even into a simpler form such as straight text or ambiguous markups such as CSV. In general, XSLT can map XML data from one schema to another, convert data from one schema to another, or simply extract elements of data into a text stream.

In our case, we’re using it to remove the redundant parts that will cause VirtualWisdom’s UDC parser to reject the document. There is currently no schema definition, so we have to make best efforts to make the resulting UDC look like one exported from VirtualWisdom.

The XSLT is a bit complex to post here, but it should be available by clicking on the marked-up filename below. Note that like xmllint, xsltproc is widely available as Linux and UNIX packages, or via the Microsoft Services for UNIX (currently a re-packaged Cygwin environment).

We execute the cleanup XSLT as follows:

xsltproc.exe concatenate-UDC.xsl Merged.udc > VirtualWisdomDataUDCImportCombined.udc

for Windows, or for non-windows:

xsltproc concatenate-UDC.xsl Merged.udc > VirtualWisdomDataUDCImportCombined.udc

Note here that the file we use is similarly-named, but end in “xsl”, not “xml”. Also, we write the file directly into the UDCImport directory of a VirtualWisdomData folder, which is where an import schedule would look for it.

This resulting file can be directly imported; an example import schedule is at the bottom of the Use UDCs to Collect Devices by Name Pattern article presented on May 1st, 2012. As well, because the UDC is in a standard form, it can be used to Quickly Create Filters for use in Dashboards, Reports, and Alarms.

Evolving SANs tend to have evolving naming schemes and assignment methods, so there will often be many different systems of identifiers that can be joined to work with different Business Units, customers, or functional groups; different such groups tend to cause different sources of information to be polled, and different formats to result. I hope this process help you to reduce the manually copying of data attributes which is so prone to human error and scheduling delays.

I hope this helps you to “set it and forget it” on more sources of data. Accurate data drives decisions: how can you methodically fix what you cannot measure and make sense of?

Edit 2012-08-23: concatenate-UDC.xsl was misspelled on the webserver (concatenate-UDCs.xsl), four downloads failed due to my error. Sorry, it should be easier to download now.

Quickly Create Filters for VirtualWisdom UDC Values

Filters, How to, UDC, xsltproc No Comments »

The UDC capability in VirtualWisdom enables quite a powerful ability to group fabric entities based on a number of parameters, but creating the filters to use a large UDCs can be a bit cumbersome. UDC is VirtualWisdom’s User-Defined Context, allowing a virtual metric value to be defined within summaries, calculated based on powerful expressions.

Typically, UDCs are used to separate and group entities such as:

  • Physical Datacenter to filter physical-layer alerts (such as CRCs) to the correct ticket queue for inspection
  • Business Unit (BU) UDCs to filter performance alerts (such as response-time) against Business-Unit -specific thresholds (i.e. Oracle requires 12ms response time, but the NFS filer accepts 20ms)
  • Port/Blade/ASIC calculations
  • Grouping a SuperDome’s ports or an Array’s ports for filtered reports

As well, UDCs are used for “what-if” calculations: What if the SCSI traffic from a certain HBA was zoned to a different storage port, which it overload the Queue and link speed? What-if UDCs are an extremely powerful tool to prove capacity based on historical use, but somewhat out-of-scope for this article.

My content in Virtual Instruments’ SAN Best Practices tend to be of the how-to nature; in this article, I’d like to share a simple method of creating all the “X = Y” filters for a specific UDC programmatically, which can reduce the time-to-value in new installs or changing environments. When linked with other generation how-to articles (such as nickname collection, or generating UDC by transform), this can further reduce the effort of managing a very large SAN.

Process Overview

For this process, our workflow will look like the following:

As you can see, the starting file “UDCExport.udc” can be either exported from the VirtualWisdom Portal itself, or can be generated by other means. The file is converted using xsltproc using a “program” or “script” UDC2Filter.xsl, resulting in Filters.xml which can be imported manually to VirtualWisdom.

Overview

UDC Files in VirtualWisdom are a specific schema of XML file; as such, standard easily-available license-free tools such as xpathget, xmllint, or xsltproc can be used to interrogate, validate, or convert the starting XML to a different format, even generating CSV or simple text in the process.

XSLT is the XML Stylesheet Translations; XSL is a Stylesheet for XML, similar to CSS describing the stype of a free-form HTML page. In essence, XSL can be considered an CSS in XML, but rather than markup content — such as type facing and style for large printed content — XSL can also transform and convert content. XSLT is the act of using XSL markup in a standalone processor (xsltproc) to create content based on XML content. In many cases, this is XML generating XML, but can be used to write TSV, CSV, JSON, etc.

VirtualWisdom Filters are exported as another schema of XML file, and can be similarly manipulated by standard XML tools. Even though this XML is a text-based format, trying to edit it with a text editor can be prone to human-error. We can read XML for debugging (xmllint -format), but as the size of the content gets larger, to use it as thought XML is an opaque binary format, which again leads us to the free tool “XSLT”.

In our case, a specific XSLT file is used to manipulate a UDC definition into a list of Filter definitions: UDC2Filter.xsl guides the conversion of UDC Values to Filters which match them.

Running the Script

xsltproc is available on most non-Windows platforms as an installable RPM, SSO, .deb, .pkg, or similar pre-packaged open source project; on Windows, it can be installed per SageHill’s Instructions; a file xsltproc.zip is easily obtained from any VI FAE to accelerate your install process.

Running it is quite simple:

xsltproc.exe -o Filters.xml UDC2Filter.xsl UDCExport.udc

There’s no output: all generated content goes directly to the output filter file.

Complete Example

In order to show how the full process, in case I’ve left out some details or some details seem implied, this is a full example based on data in our demo databases (which we use for demos and training):

Given the following UDC:

UDC that we start with for our example UDC2Filter

We export this UDC to Application_SW.udc, run the XSL Transform as follows:

xsltproc.exe -o Application_SW_Filters.xml UDC2Filter.xsl Application_SW.udc

The result we get in Application_SW_Filters.xml looks like this:

Clearly this example is only a few filters, no big deal. The benefit comes in when there are more than a half-dozen to build (recently, a 212-value UDC was tested). As well, if the UDC is edited (perhaps based on automated processes) then the administrator must go through and check that every value has a filter.

Unfortunately, there is no schedule-action for Filter import.

Use UDCs to Collect Devices by Name Pattern in VirtualWisdom

awk, How to, Nickname, UDC No Comments »

Virtually all SAN devices that are zoned for traffic have names (in fact, if you have nicknames/aliases in your zone files, then you can directly convert zone info to nicknames). VirtualWisdom’s filtering capabilities allow you to restrict a Dashboard, Report, or Alarm Policy Ruleset to a specific datacenter or business unit, but often creating those UDCs can be cumbersome.

A recent customer created UDCs with 192 values across three metric sets, allowing him to group data by specific servers, storage, and virtualizers automatically; this “how-to” is intended to show how you can do the same.

Process Overview

For this process, we need only a set of nicknames; the simpler old-format nickname file looks like:

500604825D2E2144,"DMX1911_FA3AA"
2100001B329FE31D,"Billing44_HBA0"
10000000C741ABCD,"4241_7b1"

(Notice: WWN first, no spaces, optional quotes for safety)

Generate UDC Values from Nickname Pattern

Our flow for this process or pipeline looks like the following diagram:
Flow of a UDC generated by pattern from Nicknames

The tool we use here is “awk”, or “awk.exe”, or “gawk.exe”; in Solaris, look for “nawk”. It’s on virtually every non-Windows system, Microsoft has a version in its tools for UNIX, or Google may help you find a copy. As well, UnxUtils has a version.

Awk is an interpreter, so needs a script or program, and for that, we use TransformUDC.awk which takes the following parameters:

parameter Meaning
COL What column in the CSV input is the Nickname? (default: 1)
NAMEFCX_LINK Name of the ProbeFCX::Link UDC
NAMEFCX_SCSI Name of the ProbeFCX::SCSI UDC
NAMEFCX_SCSIINIT Name of the ProbeFCX::SCSI UDC, matching Initiators only
NAMEFCX_SCSITARG Name of the ProbeFCX::SCSI UDC, matching Targets only
NAMESW Name of the ProbeSW::Link UDC (default: Transformed_UDC)
TRANSFORM Transform (basically the ‘s/x/y/g’ in a “sed -e ‘s/x/y/g’” command) (default: remove last two _sect_sect: DC_Serv1_fcs0_SW12P121 –> DC_Serv1)
UDCDEFAULT Default value for UDCs (default: Unknown)

A simple command such as the following will generate our results:

gawk.exe -f TransformUDC.awk Nicknames.csv

In our nickname file, the older format (first and second variant up to VW-3.1) gave us the nickname as the second parameter, so let’s tell the script that the nickname is in column #2:

gawk.exe -v COL=2 -f TransformUDC.awk Nicknames.csv

The problem is: how do we want the UDC values defined? If you’ve used “sed” or “awk” before, there’s a basic replacement term that looks like s/dog/cat/g or gsub("dog","cat",$0) … this part really depends on your nickname format, but looking above, we have nicknames that look like:

500604825D2E2144,"DMX1911_FA3AA"
500604825D2E2145,"DMX1911_FA4AA"
500604825D2E7744,"DMX1927_FA3AA"
500604825D2E7745,"DMX1927_FA4AA"
500604825D2E7747,"DMX1927_FA6AA"
500604825D2E7748,"DMX1927_FA7AA"
500604825D2E774C,"DMX1927_FA13AA"
500604825D2E774D,"DMX1927_FA14AA"
2100001B329FE35D,"Billing43_HBA0"
2100001B329FE35E,"Billing43_HBA1"
2100001B329FE31D,"Billing44_HBA0"
2100001B329FE31E,"Billing44_HBA1"
10000000C741ABCD,"4241_7b1"

We see how, in this example, chopping off everything after the “_” gives names such as “Billing43″ and “DMX1927″. In see and awk, we would write: s/_.*$//g so we’ll use that as our transform. How can we test this?

Trim Quotation Marks

We could trim off the quotation marks around the second field using this: (“,” as field-separator, convert (“) to (), an empty replacement)

gawk -F, '{gsub(""","",$2); print; }' Nicknames.csv

… unfortunately, we need to use quotation marks for the script, and then we need a bunch of “” escape sequences, so it looks much more complex running it:

gawk.exe -F, "{gsub("\"","",$2); print; }" Nicknames.csv

… which looks like:

awk Transforms: Trim Quotations

Truncate All After “_”

Based on the example above, we can now test whether our transform (“s/._*$//g”, or gsub(“._*$”,””,…) ) gives us the results we want, such as (notice: “print $2″, so we’ll only see the second field):

gawk.exe -F, "{gsub("\"","",$2); gsub("_.*$","",$2); print $2; }" Nicknames.csv

… which looks like:
awk Transforms: Trim Quotations, Trim Nickname

This means our transform works, so let’s use it in the script:

gawk.exe -v TRANSFORM="s/_.*$//g" -v COL=2 -f TransformUDC.awk Nicknames.csv

Unfortunately, “4241″ is not worthwhile to us because it’s only one matching name, so let’s trim that one off by saying “minimum of 2 matching names per UDC value”:

gawk.exe -v MIN=2 -v TRANSFORM="s/_.*$//g" -v COL=2 -f TransformUDC.awk Nicknames.csv

Finally, What do we want to call the UDC? The tool always generates a ProbeSW::Link UDC, and if unnamed, defaults to “Transformed_UDC”. The Name of the UDC is limited to 32 characters, and values themselves to 24 characters; the name of the UDC becomes the name of the “metric” or context that we are generating. Suppose while working at XYZ Cheese and Dairy Distributors, we want a UDC called xyz-SW-BizUnit (we need to use “_” rather than “-”):

gawk.exe -v NAMESW="xyz_SW_BizUnit" -v MIN=2 -v TRANSFORM="s/_.*$//g" -v COL=2 -f TransformUDC.awk Nicknames.csv

Let’s run this with a redirection (“>”) to store the results to a file:

gawk.exe -v NAMESW="xyz_SW_BizUnit" -v MIN=2 -v TRANSFORM="s/_.*$//g" -v COL=2 -f TransformUDC.awk Nicknames.csv > VirtualWisdomDataUDCImportxyz-UDCs.udc

Running this as a command in cmd.exe is relatively quiet because this is a non-interactive command. It tends to look like the following:

gawk -f TransformUDC Nicknames.csv to xyz-UDC.udc

Importing this example, we see the following (you’ll note: the default has also been set using “-v UDCDEFAULT=Other”) :

Generated UDC values as viewed in VW Views

Schedule UDC Import

Creating the schedule is relatively straight-forward: although there is some strong guidance in the VirtualWisdom User Guide, a complete example would like like the following:

  1. Views Application, Setup tab: Views Application, Setup Tab
  2. “Schedules” page, roughly 5th item down: Views Application, Setup Tab, Schedules page
  3. Create a new Schedule, with the action “Import UDC configurations”: Import xyz-UDCs Schedule in Views
  4. …and configure it to use a new UDC Importing Configuration, as follows. NOTE we only use a local filename, all files are in the UDCImport directory of your VirtualWisdomData folder:Vies Application, Setup Tab, UDC Import Configuration (shortened)

The benefit here is that the UDC always replaces existing values without prompting. As well, after import, a UDC re-calculates values for both past summaries and new summaries. This allows you to “fix history” if your UDC is not quite correct the first time.

Use UDCs to Group ISL Trunks or PortChannels

How to, samplecode, UDC No Comments »

I recently worked with a VirtualWisdom Administrator who wanted to group his ISL port utilization to match his ISL trunks, so we worked out a method of doing this, and I wanted to share it.  As a Field Application Engineer at Virtual Instruments, I tend to focus on these lower-level “how to” issues, working with users to achieve the data representation they need to make informed decisions in lieu of guesses and rule-of-thumb.

Initially, this administrator and I spoke of “trunks”, but between Brocade and Cisco terminology, these mean different things. The aggregations of ISLs into single logical units are “Trunking” in Brocade, but “Port Channels” in Cisco. Trunking E ports in Cisco is a different thing. I’ll use “Aggregate” as much as possible to refer to this in terminology as vendor-neutral as VirtualWisdom is.

We discussed why this Admin wanted to see more than just “the top 20″ values on a list of ISLs.  Diving deeper, this was because the top 24 entries are all the same aggregate: essentially, the first entire page is taken up by channels 1 and 2 of a a single aggregate ISL.  He wanted to skip beyond this to see the next 20 or 40 ISLs so he could see which ISLs were getting near 90% utilization.  So… why not combine these into a single filter or expression that matches the aggregate, and make sure that each aggregate uses only one row of this resulting table?

Additionally, one switch vender implements this aggregation of ISLs as balanced utilization across a collection of links between the same endpoints; conversely, another vendor implements this by overflowing one container, then moving to the next. In essence, an abstract aggregation that is has 60% Utilization may look like a collection of ports or links each with utilization 60%, or might look like 6 out of 10 links with Utilization 100%, and 4 of 10 with 0%.  That’s very difficult to separate in the data, and can obfuscate which ISL aggregations are approaching maximum desired load.

VirtualWisdom’s focus is on ports: what type, attached to what, etc.  Using Aliases or Nicknames, we can describe the endpoint, and in VirtualWisdom 3.0.0 and later, those ISL Nicknames are determined for us.  Unfortunately, these all have switch/blade/port, they’re too detailed. We cannot use that combination for a “group-by”expression to separate out the ISL aggregate.

VirtualWisdom is “too detailed” in this case: it wants to show all the ports individually.

A User-Defined Context, or UDC, is a metric with constant values applied using filter expressions. We often use these to automatically apply a logical grouping that better represents the real world implementation. One ISL aggregate between two switches A1 and A2 tends to encompass all E or TE ports on A1 connected to A2, and conversely, all A2 E or TE ports attached to switch A1.  That tends to make this one ISL unique from others.  We create a UDC in the SNMP/Link scope with values based on the “name of the switch” in an ISL: for example, in “SW12A44:3:1 ISL” as a link name, “SW12A44″ is the switch name.  ISLs between two switches share the same switch names, but are distinct by this same manner from ISLs to other switches. All we need is a UDC with values such as “SW12A44″ where “Attached Port Name MATCHES ^SW12A44:*”, and “SW12B44″ where “Attached Port Name MATCHES ^SW12B44:*”.

An example UDC would look like (Using terminology that’s a bit Brocade-leaning for this UDC because the Administrator favoured Brocade terminology) :

UDC to Group ISL Trunks

As you can see, grouping these ISL connects by “Probe Name”, “Channel”, and “TrunkChannel”, and filtering by “Attached ISL” would summarize traffic on all ISLs by the switches each connects, but aggregating bandwidth of all trunk members between each switch. Grouping by Channel continues to help us keep the directions separate so that a trunk loaded with 95% in one direction and 5% in the other shows “95%” and “5%” rather than “50%”.

You’ll notice, too, that we’ve added short-circuit to mark any non-ISL as a “NoTrunk”, the same as the default value. This avoids running the heavier “MATCHES” expression to evaluate ports that aren’t even ISLs. Your Portal server will thank you.

This logic assumes that all ISLs between two switches are in the same Aggregate; if you have any two switches with more than one distinct aggregation of ISLs, our logic no longer applies. One of our Analysts has seen ISLs grouped into multiple distinct aggregates even though they’re between the same switches, but it wasn’t the case in the discussion sponsoring the work I wanted to share.

Some customers have smaller SANs with a few dozen switches; others exceed 280 switches. This number of switches, and the various ISL possibilities between these, makes writing and maintaining a UDC with over 200 values very difficult and labour-intensive.  Because the user is effectively transferring config information from one format to another, accuracy risks can enter where users are transposing digits, or delayed in echoing the updated config information, or (more often) is simply not informed that any change needs to be echoed or copied. These risks are significant detriments to using this method.

To de-risk this implementation and help you try it out, we’ve created a script to convert a basic list of ISLs with nicknames into a UDC: while the blogging engine doesn’t let me upload this file, your VI Support and Services teams can help you get “ISL2TrunkChannelUDC.awk”, and a version of “awk” to run it. If your report with a Data View of “Table” is saved as a CSV with the AttachedPortName in the 4th column, you would run this script as:

awk -v COL=4 -f ISL2TrunkChannelUDC.awk 'Table-(summary).csv' > ISLAggregation.udc

(resulting UDC tested on version 3.0.2 and version 3.1.0 pre-release, incompatible before version 3.0.0)

I hope this helps you keep watch on your ISL utilization, and show the correct justification for adding ISLs to an aggregation, or balancing traffic to another less-used edge switch.

Keep those nicknames updated, and have a great holiday.

WP Theme & Icons by N.Design Studio
Entries RSS Comments RSS Log in