Half a year ago the Asia Pacific vSpecialist team made a fantastic acquisition in one David “Two Screws” Lloyd. David previously worked for a wonderful VMware and EMC customer in the UK and moved back to Australia. That is when we jumped on the opportunity to snap him up. In working with him in the labs recently, I have come to realize that David has incredible depth and breadth in the space of virtual infrastructure management.
Recently David sat down to share his tips on setting up alarms in vCenter. Most of VMware’s customers understand the power of custom alarming but few harness its value. Using an example of storage path failures, David created a video that walks its audience through the process of configuring a custom alarm using a tailor made executable that generates useful logging message in the vCenter OS’s event log.
Here is David’s demo on YouTube:
(A high resolution version of the video is available here.)
David was kind enough to provide me a write-up of his work with details on the why and how of this process. From here on, this entry will be in his words.
vCenter Based Storage Port Monitoring
The above video demonstrates the creation of custom alarms in vCenter to monitor host level storage port failures. To add a little bit more functionality to the alarms I wrote a little console application. When executed as a custom action assigned to an alarm, this application writes alarm details to the vCenter OS’s application event log. By adding entries to the local event log, third party monitoring solutions such as Microsoft’s System Center Operations Manager can be utilised to effectively monitor a VMware virtual environment. This is done by monitoring for specific events in the event log. This is core function of most windows monitoring solutions.
When a command is executed as a custom action, it is run in a Windows shell. That shell has several environment variables which contain details related to the triggered alarm. These details can then be used to provide a rich level of information to assist in problem identification and resolution. The table below shows the environment variables I have utilised to form the description field of the events raised.
|VMWARE_ALARM_NAME||The name of the triggered alarm.|
|VMWARE_ALARM_EVENTDESCRIPTION||The textual description of the alarm condition.|
|VMWARE_ALARM_EVENT_ COMPUTERESOURCE||The cluster name of the impacted object.|
|VMWARE_ALARM_EVENT_HOST||The hostname of the impacted object.|
|VMWARE_ALARM_EVENT_DATASTORE||The datastore name of impacted object.|
|VMWARE_ALARM_EVENT_DATACENTER||The datacenter name of the impacted object.|
|VMWARE_ALARM_EVENT_NETWORK||The network port group name of the impacted object.|
In addition to the custom description, I have added additional flexibility to further customize the event. This includes:
- Event Source – A custom source can be assigned by the executable to events it raises. This will help easily identifying events. The source name is specified within the applications configuration file ‘vCenter-AlarmLog.exe.config’, which comes with both the binary and source distributions (below).
- Event ID – Each alarm condition can be assigned a unique ID to help direct of problem tickets resulting from the event. The unique ID is provided via the command line.
- Event Level – provides the ability to class an event as an Error, Warning or Information. The event level is specified from the command line.
These can assist in the filtering of any events raised for problem ticket routing and classification.
An example of an event created by the tool is shown below. Note the source, ID and level details in addition to the general description. The event entry below was created when a storage port was disabled resulting in a lost storage port.
Each event description is formatted identically with the field identifier and value (if present) separated by a TAB to help with parsing the details.
Command Line Arguments
Both Event ID and message type can utilise application defaults or can be specified through the command line. By default the application will raise an ‘Error’ entry and event id of ‘9999’, either of which can be changed as shown below:
|vCenter-AlarmLog.exe||This will raise an ‘error’ event with the id of ‘9999’.||vCenter-AlarmLog.exe|
|vCenter-AlarmLog.exe <id>||This will raise an ‘error’ event with the numeric id as specified.||vCenter-AlarmLog.exe 1234|
|vCenter-AlarmLog.exe <id> <type>||Will raise either an ‘error’, ‘warning’ or ‘information’ entry with the numeric id as specified.||vCenter-AlarmLog.exe 1234 warning|
Custom Event Source
The custom event source is defined in the application configuration file ‘vCenter-AlarmLog.exe.config’. See the ‘appSettings’ entry ‘source’ in this file. The file is included in both packages but if it does not exist it will be created upon first execution of the application. For this to work properly, the application needs to be executed once with ‘Administrative’ level access. Failure to do this will prevent the application from writing any events.
Requires Microsoft.Net 3.0. This is also a pre-requisite for vCenter, which should already exist on the vCenter server.
Not a requirement but available should you wish to tinker with the script. The tool was written with Visual Studio 2010 in C# and its source is available as a download here.
The executable is available here.