vPivot

Scott Drummonds on Virtualization

vCenter Custom Alarms: Instruction, Tips, Tools

9 Comments »

Half a year ago the Asia Pacific vSpecialist team made a fantastic acquisition in one David “Two Screws” Lloyd. David previously worked for a wonderful VMware and EMC customer in the UK and moved back to Australia. That is when we jumped on the opportunity to snap him up. In working with him in the labs recently, I have come to realize that David has incredible depth and breadth in the space of virtual infrastructure management.

Recently David sat down to share his tips on setting up alarms in vCenter. Most of VMware’s customers understand the power of custom alarming but few harness its value. Using an example of storage path failures, David created a video that walks its audience through the process of configuring a custom alarm using a tailor made executable that generates useful logging message in the vCenter OS’s event log.

Here is David’s demo on YouTube:

(A high resolution version of the video is available here.)

David was kind enough to provide me a write-up of his work with details on the why and how of this process. From here on, this entry will be in his words.

vCenter Based Storage Port Monitoring

The above video demonstrates the creation of custom alarms in vCenter to monitor host level storage port failures.  To add a little bit more functionality to the alarms I wrote a little console application. When executed as a custom action assigned to an alarm, this application writes alarm details to the vCenter OS’s application event log. By adding entries to the local event log, third party monitoring solutions such as Microsoft’s System Center Operations Manager can be utilised to effectively monitor a VMware virtual environment. This is done by monitoring for specific events in the event log.  This is core function of most windows monitoring solutions.

When a command is executed as a custom action, it is run in a Windows shell.  That shell has several environment variables which contain details related to the triggered alarm. These details can then be used to provide a rich level of information to assist in problem identification and resolution. The table below shows the environment variables I have utilised to form the description field of the events raised.

Variable Description
VMWARE_ALARM_NAME The name of the triggered alarm.
VMWARE_ALARM_EVENTDESCRIPTION The textual description of the alarm condition.
VMWARE_ALARM_EVENT_ COMPUTERESOURCE The cluster name of the impacted object.
VMWARE_ALARM_EVENT_HOST The hostname of the impacted object.
VMWARE_ALARM_EVENT_DATASTORE The datastore name of impacted object.
VMWARE_ALARM_EVENT_DATACENTER The datacenter name of the impacted object.
VMWARE_ALARM_EVENT_NETWORK The network port group name of the impacted object.

In addition to the custom description, I have added additional flexibility to further customize the event.  This includes:

  • Event Source – A custom source can be assigned by the executable to events it raises.  This will help easily identifying events. The source name is specified within the applications configuration file ‘vCenter-AlarmLog.exe.config’, which comes with both the binary and source distributions (below).
  • Event ID – Each alarm condition can be assigned a unique ID to help direct of problem tickets resulting from the event. The unique ID is provided via the command line.
  • Event Level – provides the ability to class an event as an Error, Warning or Information. The event level is specified from the command line.

These can assist in the filtering of any events raised for problem ticket routing and classification.

An example of an event created by the tool is shown below. Note the source, ID and level details in addition to the general description. The event entry below was created when a storage port was disabled resulting in a lost storage port.

Example of custom event created by David Lloyd's tool.

Each event description is formatted identically with the field identifier and value (if present) separated by a TAB to help with parsing the details.

Command Line Arguments

Both Event ID and message type can utilise application defaults or can be specified through the command line. By default the application will raise an ‘Error’ entry and event id of ‘9999’, either of which can be changed as shown below:

Command line Description Example
vCenter-AlarmLog.exe This will raise an ‘error’ event with the id of ‘9999’. vCenter-AlarmLog.exe
vCenter-AlarmLog.exe <id> This will raise an ‘error’ event with the numeric id as specified. vCenter-AlarmLog.exe 1234
vCenter-AlarmLog.exe <id> <type> Will raise either an ‘error’, ‘warning’ or ‘information’ entry with the numeric id as specified. vCenter-AlarmLog.exe 1234 warning

Application Requirements

Custom Event Source

The custom event source is defined in the application configuration file ‘vCenter-AlarmLog.exe.config’.  See the ‘appSettings’ entry ‘source’ in this file.  The file is included in both packages but if it does not exist it will be created upon first execution of the application.  For this to work properly, the application needs to be executed once with ‘Administrative’ level access. Failure to do this will prevent the application from writing any events.

Runtime Environment

Requires Microsoft.Net 3.0. This is also a pre-requisite for vCenter, which should already exist on the vCenter server.

Source Code

Not a requirement but available should you wish to tinker with the script. The tool was written with Visual Studio 2010 in C# and its source is available as a download here.

Executable

The executable is available here.

9 Responses

[…] This post was mentioned on Twitter by VMware Planet V12n. VMware Planet V12n said: vCenter Custom Alarms: Instruction, Tips, Tools (Pivot Point) http://bit.ly/e0RHSV […]

  • For USA: RT @drummonds: ANZ #vspecialist @davlloyd finished great demo, docs, and tool for #vmware vCenter alarms: http://vpivot.com/2011/01/11/vcenter-cus

    • @drummonds @davlloyd is gr8! Have U seen his VDI skills yet? Very experienced professional & we presented together at vForum.

  • Very useful. Awesome delivery and lovely voice.

    42 short.

  • […] Drummonds – vCenter Custom Alarms: Instruction, Tips, Tools – Half a year ago the Asia Pacific vSpecialist team made a fantastic acquisition in one David […]

  • […] wpadł mi w ręce artykuł w którym Scott Drummonds umieścił video ze skryptem Davida Lloyda (http://vpivot.com/2011/01/11/vcenter-custom-alarms-instruction-tips-tools/)  po krotce opisujące możliwości uruchamiania komend systemowych po wystąpieniu alertu w […]

  • Hey Scott,

    Great stuff! To be honest I’ve got inspired by this video and created a small tool to forward vcenter alerts to skype (or soon SMS via skype). Thanks!

  • […] last video showed a technique for improving the automated alarming in vCenter.  In these videos David turns to Ionix UIM 2.1, to show Vblock configuration and management. For […]