Network Applications Don't Always Play Well With Others: How to Diagnose a Non-Functional LENS Server Software

Recently we've had a couple of customer support issues concerning the LENS Trial Software, where the server was installed and apparently running properly, but not responding to 911 calls from the CS 1000. (For an explanation of LENS, please see my earlier post on the subject.) The messages get to LENS from the CS 1000 by way of SNMP Traps. Typically,
troubleshooting this type of issue requires checking the following things

Are the traps being sent?
The most likely cause that we've seen is that the CS 1000 isn't configured to send its emergency traps to the correct IP address, so it's the first thing we check. The configuration isn't difficult, but it's often overlooked. For specific instructions, see this post. Assuming the traps are being sent to the correct address, we can move on.

Are the traps being received?
This one is more difficult to diagnose, and it involves checking multiple things.

Is the Windows SNMP Trap Service Running?

The first step is to make sure the Windows SNMP Trap Service is running. In both cases, it was. The screen shot on the right shows the Windows Service MMC Snap-in, indicating clearly that the SNMP Trap Service is running. Click the image to enlarge.

The LENS Server also has a handy icon showing the state of the SNMP Trap Service.

Is the Windows Firewall Blocking the Trap?
The trap service may be running, but that doesn't mean the traps are getting through. SNMP traps travel over UDP Port 162. If that port is blocked by whatever firewall rules are active, they'll be blocked by the firewall and never make it to the trap service. Everything will appear to be running fine, but nothing will happen.

Is Something Else Using Port 162?
This is the one that had tripped us up on these two support calls. There was another application handling traps! In and of itself, this isn't out of the realm of possibility. The reason it didn't occur to us is that nothing else complained about the port being tied up by another process. The Windows SNMP Trap Service didn't raise any errors. It started up just fine. And because the Windows Trap Service was happy, LENS was happy.

We found the issue by running this command from the Windows command prompt:

netstat -ano

We then scanned the output, looking for lines containing port 162. On my development machine without any other trap handlers running, I had the following relevant lines in the command's output:

UDP    0.0.0.0:162     *:*       428
UDP    [::]:162        *:*       428

Those lines indicate that port 162 is being used by a process whose Windows Process ID is 428.

To find out what process that is, simply start up the Task Manager and click the Processes tab. Process IDs are not displayed by default, so if you don't see it, click the View Menu and choose Select Columns... In the window that opens, choose PID (Process Identifier) and click OK.

Now sort on the PID column and find the Process ID matching the one from the output from the netstat command (in my case, 428).  You may need to select the Show processes from all users option before it will appear. As you can see from the screen shot here (click the image to enlarge), on my system, Process ID 428 is in fact snmptrap.exe, also known as the SNMP Trap service. Thus, at least on my system, everything is configured properly.

This is the expected scenario, but wasn't the case in our two customer support issues. In both instances, once we reached this step, we found another process holding port 162, even though the Windows SNMP Trap service was also running, apparently without any problems.

The only thing left to do at that point was to stop the conflicting application, or if that wasn't possible, find another Windows Server on which to run the LENS Server.

The Lesson
If the LENS Server software doesn't appear to be responding to emergency calls from the CS 1000, don't rely on the fact that the Windows SNMP Service is running. Take the extra two minutes and make sure nothing else is using that UDP port. Network applications don't always play well together, especially when two of them want to use the same port. Normally, you'd expect one of them would complain. But when they don't, at least there is another way to determine what's going on.

Updated LENS Alert Agent

Back in April I warned of what can happen if you don't pay attention to software requirements, detailing that I got bitten by my own assumptions.

As a follow-up to that post, I'm pleased to announce that as of version 1.4.3, the LENS Alert Agent no longer requires Service Pack 2 of the .NET Framework 2.0. In reviewing the code, we discovered a single line of code that was responsible for the requirement and promptly changed it. We felt that requiring users to upgrade where not absolutely necessary was putting an undue burden on them. The new version loses no functionality from this change, which made the decision even easier for us.

In addition to the .NET change, recent versions of the LENS Alert Agent introduced the following improvements:

• Simplified Acknowledgments of Alerts
  
• Removed redundant audible alarms. Now there is only one audible siren, regardless of how many alerts are present.
• Added an optional timeout feature to audible alerts in the event that a PC is locked and the alarm can't be acknowledged manually.
• Minor bug fixes and display clean up.
• Added support for Nortel CS 2100 (more on this to come).

If you are currently running a version of the LENS Alert Agent prior to version 1.4.2, it probably makes sense to call your support personnel for an upgrade. 

The Dangers of Not Paying Attention to Software Prerequisites

What to do if you get a Method not found error in the LENS Alert Agent...

The other day I was doing a demo of LENS using someone else's laptop. This is a bad idea in most situation, but I was in a hurry. I think that's the first excuse of every software developer whenever something goes wrong. Everything was going well until I tried to switch servers, at which point I received the following error.

Method not found: 'Boolean System.Threading.WaitHandle.WaitOne(Int32)'.

Have you ever been in the middle of a demo and have your own code crash in front of everyone? It's not pleasant. Fortunately, I knew exactly what the problem was. In setting up the demo, I had neglected to check that the .NET Framework 2.0 SP2 was installed (see my April 10 post for more information). It had .NET 2.0, but not SP2. The installer correctly detects .NET 2.0, but fails to complain if SP2 is missing. Upgrading the system to SP2 solved the problem.

The moral of this story is twofold:

1. Always verify the system requirements before installing software. Don't expect the installer to be smart enough to detect subtleties like which service packs are installed.
2. Try not to use other people's PCs when doing demos in front of customers.

I intend to fix this issue in the next Alert Agent release, and hopefully make the installer "service pack aware." In the meantime, should you receive the above exception, you'll know how to fix it: just install .NET 2.0 SP2.

Note: you may also install .NET 3.5, SP1 which contains .NET 2.0 SP2 and provides additional functionality for other (newer) applications you may install.

LENS Server Software Requirements and Configuration

This is a follow up to my post awhile back on Configuring the CS 1000 for LENS and Locate911. I've been receiving a lot of questions on the actual configuration and system requirements for our software version of LENS.

Officially, our system requirements are as follows:

• Windows Server 2003 or later
• Microsoft .NET Framework 2.0 Service Pack 2 or later (Requires users to be current with all .NET updates from Microsoft. Note: These updates are not automatically installed as they are not required security updates. Users must go into Windows Update and select the updates to be installed. A reboot is recommended after such an update.)
• Windows SNMP Service installed and running

So let's go through that list one at a time. As noted above, we officially support Windows Server 2003 or later, though I've been able to get it to run on other Windows versions. Unless you're simply testing the Trial Version of LENS, I highly recommend you stick with a supported server installation for something as critical as 911 notifications. With that in mind, I'm focusing on installing the product on a fresh copy of Windows Server 2003.

On my default installation of Windows Server 2003, the .NET Framework and the Windows SNMP Trap Service were not installed. If that's the case with you, you'll need to install SNMP first. Instructions on doing that can be found here.

The next thing you need is the latest version of the .NET Framework 2.0, which can be found here.

Something worth pointing out is the unintuitive numbering scheme that Microsoft uses on the .NET Framework. Many people think that because they have .NET 3.0 installed, they already have something newer than .NET 2.0 SP2. This is not the case. Most of the .NET Framework installations do include the older versions, but it's not clear by looking at the version number what service packs may also be included. An explanation of what gets installed with which version can be found in this Microsoft Knowledge Base article. According to that article, .NET 2.0 SP2 is not available separately, but that's only the case for Vista and later. If you are running XP or Windows Server 2003, you can install .NET Framework 2.0 SP2 from here. My personal recommendation is that you upgrade your .NET Framework (2.0 or later) to the absolute latest service pack, and keep your framework up to date using Windows Update.

Now that .NET and SNMP are installed, you can get on with installing the LENS Server. The installer is fairly simple and has no user-defined options other than where to install it. On my Windows Server system, the installation requires exactly 1,064,960 bytes (or just under 1 MB).

Once the server is installed, it's still not running. You can use the Windows Services MMC Snap-in if you prefer, but we have provided an application to allow you to configure the server and its dependent services from one place. Look in the Start menu for the "Configure LENS Server" application and run it.

Notice that by default on my system, all three service components are currently stopped. Before you start any of them, make sure the application is configured the way you want it, and click Apply. Then you can start the services, in order, from top to bottom.

The first thing you need to do is configure your server's IP address. This is the address the LENS Alert Agents will connnect to, so make sure you select the right one from the Server IP Address dropdown.

Next, decide whether or not you want to use the map link generator. (This feature is not enabled in the Trial version.) If your Nortel PBX is a CS 1000 Release 5.0 or later, you can use this feature. Essentially, any time it receives an emergency call containing an Emergency Response Location (ERL), it adds a map link to the notification's details. The way it accomplishes this is to append the contents of the two fields, OSN Map Link Label and OSN Map Link to the Alert Details, substituting the string <erl> with the actual ERL number. Thus, if the OSN Map Link is http://www.mycompany.com/maps/map_<erl>.html and the ERL is 105, the link that appears in the Alert Details window will be http://www.mycompany.com/maps/map_105.html. Simple, but effective. Note that these links are not checked for validity, so if you don't actually provide a real target for all possible links, your users may end up with 404: Page Not Found errors when they click on them. If you leave the fields blank, or no ERL exists in the alert, then no link will be generated. ERLs are described in this recent post.

Once you've set these options, feel free to click Apply and start the services. You'll notice that if everything is working, the Messaging Daemon will start automatically with the eTelemetry LENS Server. Should the Messaging Daemon not start, stop the LENS Server, change your IP Address, click Apply, and try again. That usually clears things up. The main cause of the Messaging Daemon not starting is a bad IP address, so make sure you've selected the right one.

I've been asked about memory usage in the LENS Server, so here are some real numbers. According to the Windows Task Manager, the SNMP Trap service is using 2mb and the LENS Server Service is using 16mb. Given that neither application stores information locally, these numbers don't tend to change much during their execution.

If you are running the full LENS Server (as opposed to the trial), it's time to set up the Email functionality. Click the Email tab and fill in all of the available information. The only optional field is the From Name. Make sure you have a working mail server. When you're finished, click Apply, and then restart the LENS Server. The LENS Server only reads its configuration at start up, so you must restart it.

If you want to test your email configuration, I recommend that you use 911 Misdials rather than actual 911 calls. Misdial Prevention is standard on CS 1000 5.0 and later, and is available as a free patch for older systems back to Release 24. See your Nortel support personnel for more information on obtaining and installing this patch.

At this point you should have a working LENS Server up and running. Install the LENS Alert Agent on as many Windows PCs as you'd like (only one on the Trial Version), and use them to know precisely when and where anyone on your network has dialed 911. As always, feel free to post comments and/or questions.

In a future post I will discuss the system requirements and configuration options for the LENS Alert Agent, along with some more creative ways it can be used.

What's an ERL and Why Should I Care?

One of the dangers of immersing yourself in a particular technology is the tendency to assume everyone has the same level of knowledge on the subject that you do. This point hit home recently when I got a support call from an installer about our LENS product. He couldn’t get it to work.

Without going into too many details, the issue was this: LENS was installed in a High School near the school’s CS 1000 (where it belongs, incidentally). One Alert Agent was installed in the office at the high school; another was installed at a nearby middle school. The customer wanted to monitor 911 calls originating in either building, but they wanted the middle school to be notified only on 911 calls from the middle school, and the high school to be notified only on calls from the high school. Given that LENS was designed for precisely that purpose, I was surprised they were having problems.

After a bit of gentle prodding, I determined that the ERL filtering wasn’t enabled on their Alert Agents. It was a simple fix, or so I thought. Just have them turn on ERL filtering in each of their Alert Agents and all will be right with the world. I wasn’t too worried when I was asked, “How do I do that?” No one ever reads manuals anyway, so I talked him through the simple process (click Tools | Options | ERL Filtering and enter the ERLs you want to receive alerts for). I became slightly uneasy when he asked me what values to enter for ERL. I suggested that he enter the middle school‘s ERLs in the middle school agent, and high school’s ERLs in the high school agent. “What’s an ERL?” he asked me. Now I was concerned.

I had fallen into the trap. During the development of LENS and Locate911 over the past year, I had been immersed in the technology, but hadn’t stopped to consider that there are CS 1000 installers in the field who have no idea what ERLs are, how to configure them in the Call Server, or how to map them to physical locations.

This is the definition of ERL, which was conceived by the National Emergency Number Association (NENA):

Emergency Response Location (ERL)
A location to which a 9-1-1 emergency response team may be dispatched. The location should be specific enough to provide a reasonable opportunity for the emergency response team to quickly locate a caller anywhere within it. (See http://www.nena.org/media/files/NENA00-001_V1020070605.pdf, p. 28).

A fantastic video explanation on ERLs and how to configure them can be seen here:

In this customer’s particular case, they had only deployed a single Emergency Location Identification Number (ELIN) covering both buildings. This means that anytime anyone at either location dials 911, operators at the Public Safety Answering Point (PSAP) will only see a single address, and not necessarily the address of the building where the call originated. Imagine expanding this issue to an entire school district spread across many square miles, and you’ll begin to appreciate the scope of the problem. Fortunately, this is precisely the problem that Nortel and eTelemetry address with these products.

The Nortel CS 1000, release 5.0 and later, has native support for the ERL concept. Once a site’s ERLs are mapped to physical locations, each ERL can be assigned an ELIN, and each ELIN can be provisioned in the local PSAP. Now no matter where inside an ERL a 911 call is placed, PSAP operators get much more accurate and valid location information. They know where to send help, even if the individual on the other end of phone can’t speak or the call has been disconnected. Provisioning and changing information in the Public Safety database is both costly and time-consuming. Using ERLs for E911 location provides a huge advantage in this area. Once an ELIN is established, there is hardly any reason ever to change it, saving both time and money.

Once we understood the problem, the fix was simple. We defined two ERLs in the Call Server, 100 and 200. I recommended this approach for future expansion. As the school decides to get more and more granular with their ERLs, they won’t have to start over. ERLs 100-199 were reserved for the high school, and 200-299 for the middle school. These ranges were entered into the respective Alert Agents’ ERL filtering so that each school received the appropriate notifications. Both ERLs use the same ELIN for now, until a new ELIN and address for the other school can be provisioned with Public Safety. Until then, Public Safety still gets a single location, which is far from ideal. Because they are using LENS, however, school officials know not only that a 911 call was placed, but from which school.

In the future, this particular school system can further eliminate confusion by implementing more ERLs locally, such as

• 110: Cafeteria
• 120: Science Labs
• 130: Gymnasium
• and so forth.

They can do this freely, even without updating Public Safety's database. The important thing to understand is that although Public Safety may not know that a 911 call came from the library, local on-site personnel will, can direct local first-responders there quickly, and have people ready to guide public safety responders to the appropriate location when they arrive.

As Voice over IP (VoIP) phones gain more and more traction in the education and business worlds, locating a moving phone is becoming ever more critical. Trying to update Public Safety’s database with every potential phone move is impractical, if not impossible. Not only is it expensive, it’s slow. Implementing ERLs solves this problem by provisioning locations with Public Safety once, and then by relying on a local discovery manager such as Locate911 to keep the Call Server updated with each phone’s location, enabling it to use the appropriate ELIN during a 911 call. Furthmore, by utilizing an appropriate on-site notification solution, like LENS, local first-responders know precisely when and where a 911 call has occurred.

In a future post, I'll describe a strategy to minimize the use of ELINs at the PSAP, while maximizing the number of distinct locations that can be reported internally.

eTelemetry Responds to New NENA Model E911 Legislation

eTelemetry officially responded to the new model legislation proposed by the National Emergency Number Association (NENA) concerning E911 laws. The model legislation can be read here. eTelemetry's response can be read here (PDF here).

Testers Needed for Older Nortel Call Servers

Last year, Nortel certified LENS and Locate911 on their current version of the CS 1000, which at the time was 5.0. They also certified us on 5.5 when it was released. Now we want to expand our support and are looking for people with older Nortel Call Servers, from Release 24 to CS 1000 4.5, who would like to participate in an invitation-only on-site notification test.

In order to participate, you need to have a Nortel Call Server as described above that can be configured to send its open alarms to a remote IP address. I have a publicly-accessible LENS device that you'll be able to monitor along with me as things progress.

If you are interested in participating, you may contact me using Mike AT etelemetry DOT com. Please send me the following information:

• Your name and company
• A list of the Call Servers and versions you're interested in testing
• The number of phones you have (IP and non-IP)
• The number of physical locations at your site.

Once you're accepted, I'll send you the IP address you can use to configure your Call Server, along with some software that will allow you to monitor my LENS device.

As a thank you for your assistance, I am authorized to offer each tester a special deal on a license for the software version of our product. I can provide those details later, since I don’t want this post to sound like an ad. If you have any questions, feel free to comment here or email me at the above address. Thanks!

How to Set Up a CS 1000 for LENS/Locate911

I recently answered a post on tek-tips about how to set up the CS 1000 to relay its E911 emergency call information to our LENS and Locate911 products, and thought I'd replicate the information here also. The process isn't hard, but if you don't spend a lot of time in the CS 1000 overlays, it may seem a little intimidating.

The information in this post applies to the following eTelemetry products:

• Locate911-N
  
• LENS Appliance
• LENS Server Software
• LENS (Trial Version)

I've tested the instructions on the CS 1000 release 4.5, 5.0, and 5.5. If you have a different software release, please post a comment with your experiences. Questions on the process are also welcome. If you have more than one CS 1000 on your ELAN, simply do this once on each.

1. Connect to the CS1000 using the cslogin command from a signaling server (or connect directly if you prefer)
2. Log in as a user with administrative privileges
3. Enter the LD 117 command overlay
4. Use prt open_alarm to see what devices are set to receive alarms
5. Find an empty # and then issue the command: set open_alarm <#> <IP Address of LENS/Locate911>
6. Exit out of the overlay by typing ****
7. Enter the LD 43 command overlay
8. Save and Backup the configuration by issuing the EDD command
9. Exit out of the overlay by typing ****
10. Log out of the CS1000 by typing LOGO
11. Disconnect by typing ~. at the command prompt.

At this point, the eTelemetry server should receive every alarm generated by the CS 1000. Install and run the Alert Agent on any Windows machine, and point it at the IP address of the eTelemetry server.

Now, any time an emergency call or misdial is detected on the CS 1000, the eTelemetry Server will send a message to the Agent,. The Agent will display an Alert Details window like the one on the right and scream like crazy until it is dismissed.

What is Locate911?

Locate911, and its sister product LENS, are network appliances designed to enable companies more easily to comply with local E911 regulations. LENS works by alerting on-site personnel that a 911 call has been made, along with where the call was placed. This enables local first responders to respond to the scene of an emergency, and then to assist public safety officials when they arrive, saving time and perhaps lives in the process.

Locate911 adds to LENS by solving the problem of finding VoIP phones that can potentially move around the enterprise network without administrative involvement. When a VoIP phone makes a 911 call, there is no guarantee that the address information sent to Public Safety bears any resemblences to the phone's actual location. Locate911 tracks these VoIP phones as they move around, updating its internal database with the phone's actual location. This location can then be reported back to the PBX so that it can take appropriate action when routing the 911 call.

LENS is currently available for Nortel's CS 1000 line of PBXes, with other PBX support coming. Locate911 is available in a variety of configurations, including specific releases for Nortel CS 1000 and other PBXes. See www.locate911.com for more information.

From time to time, I will post information about upcoming product releases, technical information, bug fixes, patches, and other product-related issues.