ServiceNow Event Management

This article covers the benefits of using ServiceNow event management, some example steps to implement, and tips and tricks for usage.

What is ServiceNow Event Management?

From the ServiceNow Wiki on Event Management:

ServiceNow Event Management (SN EM) automatically creates actionable alerts from events captured by third-party monitoring tools. A configurable dashboard provides a consolidated view of all service-impacting events, including a list of current active alerts, impacted services, and associated incidents. In order to generate qualified alerts, events are processed through filters that normalize and de-duplicate the incoming stream. Alerts are then mapped to configuration items (CIs) in the ServiceNow Configuration Management Database (CMDB) and rules may be applied to trigger automated actions.

In short, this allows events from Monitoring tools to go directly into ServiceNow. 

Benefits of using ServiceNow Event Management

1. Event Management Incident Creation. Filter incoming events to reduce noise andautomatically generate only qualified alerts that create incidents. Automatically trigger IT processes for rapid remediation of service-impacting issues.  Prioritize issues appropriately to ensure the availability of critical business services.
2. Costs. The ServiceNow cost per node for SN EM is likely less than using other "middle man" monitoring systems like Tivoli Netcool/Omnibus.  You can save thousands of licensing costs by making the switch.  Companies like FICO and Kimberly-Clark have done this and saved a lot.  I can't speak for FICO due to confidentiality, but I do know about Kimberly-Clark. Kimberly-Clark eliminated duplicate incidents, improved reporting and control, and saved $250,000 in annual maintenance costs.
3. Leverage Existing Infrastructure Monitoring. Connect existing monitoring tools easily to ServiceNow through a simple, high throughput REST API or use SNMP Traps.  Consolidate and eliminate some monitoring tools to all feed into ServiceNow.
4. Central Event/Alert Location.  With SN EM, events and alerts are now stored within ServiceNow instead of just in other monitoring tools.  You can filter and display that information any way you want with highly-customizable ServiceNow.  Show the technician on an incident the exact time an event occurred, the CI, and other critical monitoring information.  Use dashboards to view events, alerts, and associated incidents all within ServiceNow.
5. Market Skillset.  At least in my situation, there are less monitoring developers around than ServiceNow JavaScript developers. You still need monitoring people with SN EM, but ServiceNow developers can assist while a monitoring person is out sick or on vacation.
6. Upgrades.  Upgrades with many monitoring applications can be costly and you will require implementation services.  As you know with ServiceNow, upgrades are pretty easy and amazing.
7. Availability and Scale.  ServiceNow has the proven availability.  Often on-premise solutions can't match 99%+ availability.  Also ServiceNow can easily scale to your enterprise.  
8. Innovation.  ServiceNow is hungry in this space, and is willing to make SN EM a better product and innovative.  Other monitoring tools are not as innovative in comparison.

How it works

Here is the basic process on how ServiceNow Event Management works:

1. Monitoring System: Event Occurs. Event occurs in a monitoring system.  An important event could an unreachable server, free disk space is low, high memory usage, cpu utilization, etc.
2. Monitoring System: Event Trigger. Monitoring system triggers an action to send the event to SN EM.  This can be by REST API, SNMP Traps, Email (not recommended), or any integration method ServiceNow offers.  An integration needs to populate the em_event table in ServiceNow.
3. ServiceNow: Receives Event.  ServiceNow will receive the event, but first it must pass event filters.  ServiceNow has event filters in SN EM that filter out the events you only care about keeping and whether you want events to duplicate.  I encourage keeping the filters within ServiceNow instead of the monitoring system, as you will be able to adjust them easier later in ServiceNow.  
4. ServiceNow: Alert Creation. Using the event details in the "Additional information" field of an event,  Event Transform Rules and Event Transform Maps move event data to a new Alert record.
5. ServiceNow: Incident Creation. Alert Rules determine when incidents or other records should be created from a created alert.  Often it would be a 1:1 ratio of alerts to incidents.  However if you have a maintenance window for certain CIs, an incident could be suppressed at that point.  

Implementation Step 1 - Acquire

You decided you want to use ServiceNow Event Management.  So to do that, you first need to purchase licenses and install the plugin.

To use SN EM you need to follow some steps to have it available in your ServiceNow Instance

1. Get Licenses.  SN EM is licensed per node.  So if you are going to monitor 2000 servers, you need 2000 ServiceNow EM licenses.  Currently it is licensed on a tier setup, the more nodes you license, you get a discount per node.  Contact your ServiceNow account rep to get licenses.
2. Request Plugin.  Once you get the licenses, you login into ServiceNow HI support and request the plugin to be installed.  

Implementation Step 2 - Initial Setup

After the plugin is installed, you can begin configuring SN EM.  Of course, you should do this in a Development instance first!  Without proper testing you can flood your ServiceNow instances with events, alerts, and incidents.  Although we were not able to break ServiceNow with events (and we tried), I think it would be possible if you did this without precaution.

1. Import CIs into the CMDB.  You should have some CIs in your ServiceNow instance so you can automatically bind events to CIs.  I prefer to have discovery import the CIs, but if you have a validated source of CIs you should import them in ServiceNow.
2. Setup Midservers.  Your event monitoring system needs to communicate with ServiceNow.  If it is not exposed to the internet (probably not and not advised), you need a mid server installed to communicate.  You might need more than 1 mid server setup depending on the monitoring systems you use and network setup.  
   1. Extract install files
   2. Install Mid server
   3. Configure config.xml: mid server name, username, password
3. Monitoring User. Create a ServiceNow user to communicate between the monitoring system and ServiceNow.  Roles needed might include: all the soap roles, evt_mgmt_integration, rest_service, itil, mid_server
4. Credentials (Optional).  For some monitoring systems you may need to install a credential.  REST API requires a credential.
   1. Discovery > Credentials > New
   2. Fields to enter: Name, Type, Applies to, Username and Password.
   3. Click Save
5. Configure Connector Instance (Optional).  This part you might need help with a monitoring technician.  You need to add IPs, credentials, ports, and IP ranges for each connector.  You can use Connectors, SNMP Traps, or other methods to create events, so this may be optional depending on the method you want to use.
6. Configure SNMP Trap Collector (Optional).  SNMP Trap Collectors are located in the Left Nav Bar under Mid Server > SNMP Trap Collectors.  You may need help from a monitoring person to get the correct port.  Note the default port to use is usually 162.  Use a different port if that conflicts with your environment setup.
7. Import MIB Files (Optional).  If you are using SNMP traps, you would need to import MIB files and load them in the correct order.
8. Restart Mid Server service.  Sometimes with Credential and mid server changes, you need to restart the mid server service.  That might just be superstition on my part however.

Implementation Step 3 - Configure Filters and Rules

1. Monitoring On. Turn on monitoring to your development ServiceNow instance.  If you have successfully connected, you can watch the events pour in.  If not, go back to Step 2 and reconfigure.
2. Configure Event Filters.  Now that events are coming in, setup some event filters to filter out the events you only care about keeping and whether you want events to duplicate.
3. Configure Event Transform Rules and Event Transform Maps. Using the event details in the "Additional information" field of an event, setup Event Transform Rules and Event Transform Maps move event data to a Alert record.  There should be a ratio of many events to fewer alerts.  
4. Configure Alert Rules.  Alert Rules determine when incidents or other records should be created from a created alert.  Often it would be a 1:1 ratio of alerts to incidents. However if you have a maintenance window for certain CIs or don't want to generate incidents for lower priority alerts, an incident could be suppressed.  That means the alert could be configured to not generate an incident.  
5. Setup Event Properties.  There is a event properties file located at Event Management > Properties.  It contains some important properties you might consider changing such
   1. Resolving an incident closes the associated alerts
   2. Reopening alerts will create new incident
   3. Closing alerts will close the incident

Implementation Step 4 - Add Improvements

By step 4, you should have the the monitoring system creating events, events creating alerts, and alerts creating incidents.  At this point you will notice other improvements you can make. 

• Business Rule to add additional info to Incident.  You can supply a template to map certain data from the alert to the incident.  However there is also other data you might want to move over.  You can create an "after" business rule on the em_alert table to do that.  In that business rule, you use a glide record query on the incident to update it. Hopefully this will be improved in later releases of SN EM so this Business Rules is no longer needed.
• KB Articles to fix alert in incident notes.  Business rule to do that.
• Event or Alert Updates in incident.  Maybe a business rule or related lists
• Can't close incident with open alert.  This could be UI Policy or Business Rule.

There will lots of innovative ideas that can happen here.  This is one of the great benefits of SN EM is that you CAN do this in ServiceNow.

Implementation Step 5 - Testing and Production Cutover

After you get it configured and running.  You need to test it by flooding it with events and see how it is handled.  After testing, you might be ready to cutover to production.  Hopefully you captured all those updates in an update set!  

You may want to consider an "easier" cutover to production and move event management but have the incident creation turned off at first.  That way you can compare alerts and see if it is working properly.  I personally feel that testing should be done in test and not in production, so I don't really like that method.  It does lower the implementation risk however, so maybe it is better though.

Troubleshooting

Here is a list of things to check if things are not going to plan.

• Mid Server Service is running.  You need that for the monitoring system to connect to ServiceNow.
• Mid Server Service is upgraded to latest version.  Version difference might keep event management not working in upgraded versions of ServiceNow.  Mid Servers are supposed to automatically upgrade, but on some networks that may be restricted.
• Monitoring application is running.  Of course that needs to work!
• Events are being processed (Event Processing Statistics).  It will take a lot to break this, 250K event floods just slow it down.  We have seen a node stop and ServiceNow needed a restart to get events processing, but that was due to a mid server version issue.
• Alert Rules, Event Transform Map, and Event Transform Rules are configured correctly. If need these configured correctly to create incidents in the end.  Did you recently make an update?  Check your update sets and versions.
• Errors in Warning or Error logs.  A place to check for errors or warnings.  If you can't figure out what the error means, often ServiceNow HI Support is great at helping with that.

Hope this helps!
Mike