Handling Nagios Client Monitoring System

Procedure to handle Nagios

The procedure that support staff need to follow while handling Nagios is documented under Procedures: Handling Nagios Client Monitoring System

Nagios is accessed via this link: http://monitor.neosys.com/nagios3

Nagios services

Nagios is configured to display information pertaining to all NEOSYS client's server statuses which include multiple services such as:

1. HTTPS: Most of NEOSYS clients are configured to have external web access via secure HTTP protocol (port 4430) from outside office. Nagios is configured to check port 4430 on a regular interval of 10 minutes and display any issues in accessing the same.
2. SSH: As part of the support contract, NEOSYS should have external secure access to the client server usually over port 19580. Nagios is configured to check this port on a regular interval of 10 minutes and display any issues in accessing the same.
3. Ping: Nagios is also configured to ping the client router as a measure to check if router responds incase the NEOSYS server is down.
4. NEOSYS: This service works in a reverse direction, and the NEOSYS installation on the client server sends information such as databases running, current backup status, internal and internet IP addressess etc to Nagios on a regular interval of 10 minutes.

Some key information about Nagios is as follows:

• Nagios is also configured to display information related to internal servers.
• Clients hosted on a NEOSYS cloud server might not have services such as SSH or PING as this is monitored as part of the internal server service.
• Nagios sends out email alerts to support2@neosys.com (which is forwarded to support@neosys.com) from 8 am to 12 midnight on all Dubai working days (Sun-Thu). No alerts are sent out on Fri and Sat, unless they are for NEOSYS internal servers.

Nagios Configuration files:

• Commands to check different services. ~/etc/nagios3/commands.cfg
• Contacts where support2@neosys.com is configured. Emails from nagios will be sent to the email ID configured here. /etc/nagios3/conf.d.backups/contacts.cfg
• For Oman Client. Unclear of purpose. /etc/nagios3/conf.d.backups/dtme.cfg
• Fully commented file of generic hostgroup definitions. /etc/nagios3/conf.d.backups/hostgroups_nagios2.cfg
• Host group definition. /etc/nagios3/conf.d.backups/linux.cfg
• Files where all the neosys hosts and hostgroups are defined. /etc/nagios3/conf.d.backups/neosysclients.cfg
• Config for monitor.neosys.com. /etc/nagios3/conf.d.backups/monitor.cfg

How to handle a service error

1. Nagios Service Info - get there via various routes eg from Service Problems - then click on the service name (not the host name)
2. Service Commands, Acknowledge this service problem (only services with status Warning or Critical have this option)
3. Enter a note - explaining to yourself and your co-workers explaining how the problem is being handled and when to follow up

Notifications will be automatically resumed once the service becomes OK again.

The "Disable notifications" is not quite the same and shows as red on tactical summary screen.

How to stop ALL notifications

Useful to stop a massive number of alerts due to various causes.

1. Nagios Process Info
2. Enable/Disable notifications

Speeding up Nagios web interface

The usual F5 to refresh before the automatic 90 second refresh works but Ctrl+F5 doesnt.

Speeding up NEOSYS process checkins

You can force a neosys service checkin from NEOSYS maintenance mode (any process/database) press F5

MONITOR2

Troubleshooting NAGIOS generally

Fixing “CRITICAL – Socket timeout after 10 secs” error message on NAGIOS

Error Message

Problem

NAGIOS is not updating services like CPU Load, Drive Space C:, Drive Save D:,Explorer, Memory Usage etc.

Solution

Open Windows Task Manager and kill any nscp.exe process. Then, restart NSClient++ from the desktop or by going to Start> Programs> NSClient++ These steps can be carried out even when users are active.

Fixing “NEOSYS has not checked in” error message

Error Message Explained

Nagios reports this error when it is is not able to update the status of NEOSYS for a particular client server.

Possible Causes & Solutions

The maintenance window is left open

Make sure that there is no maintenance window left open in the server.

Hung process on server

Check if there are any hung processes on the server e.g. Fatal Error in Rev Restart. Follow steps in troubleshooting hung process.

The NEOSYS process IS NOT running on the server

If there are no NEOSYS processes running, then start the NEOSYS process and wait for NEOSYS to check into Nagios, or force Nagios to re-check the status of NEOSYS service.

The NEOSYS process IS running on the server but still cannot connect to NAGIOS

NEOSYS connects to Nagios using http and automatically detects and uses any http proxy configuration configured in Internet Explorer.

If Internet Explorer in the server can reach the internet then NEOSYS should be able to update to NAGIOS via the same proxy.

Check if Internet Explorer can reach Nagios by using the below link. The check is successful if you get the Nagios login request shown in the image below.

https://monitor.hosts.neosys.com:4428

If Internet Explorer CANNOT connect to Nagios

The client's IT must enable outbound access preferably on all ports or at least 4428 to our NEOSYS server at monitor.hosts.neosys.com.

If outbound access to monitor.hosts.neosys.com on port 4428 has not been enabled, then Nagios will not be able to update the status of the host and will show the error "NEOSYS not checked in".

If port 4428 cannot be used, then Support can manually configure the network to use standard port 443 in the system configuration file.

This requires that the client's IP number is added to the list of allowed IP numbers on nl10r router since Nagios https server gets multiple probes a day on port 443 if it is left open to everyone.

If Internet Explorer CAN connect to Nagios

View the Internet Explorer proxy configuration as follows:

Check if there is an issue with NEOSYS' http proxy server configuration as follows:

1. Open the UPDATE.$WG or UPDATE.xWG file located in the neosys\neosys or hosts/xxxxxxxx/work folder.
2. You should find a message similar to the following:

Connecting to 192.168.100.145:8080 failed: No such file or directory.

Where the above appears to be some non-functional http proxy server ip/port number and is not the expected nagios server ip number. A windows proxy command shows the same ip and port:

On Windows 2003/XP

proxycfg

On Windows 2008/Win7

NetSH WinHTTP import Proxy ie

Output:

Microsoft (R) WinHTTP Default Proxy Configuration Tool
Copyright (c) Microsoft Corporation. All rights reserved.

Current WinHTTP proxy settings under:
  HKEY_LOCAL_MACHINE\
    SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\Connections\
      WinHttpSettings :

    Proxy Server(s) :  192.168.100.145:8080
    Bypass List     :  192.168.*.*;localhost;<local>

Solution 1 - Remove the above setting to create a direct connection

1. To remove the registry entries that ProxyCfg.exe creates,you must delete the WinHttpSettings value from the following registry key:
   

   HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\Connections\WinHttpSettings

2. After you do the above, confirm that the proxy details are deleted by running the proxycfg command
   
3. Next, restart the NEOSYS processes for the changes to be affected
   

Solution 2 - Configure a functioning proxy ip/port number

1. Use the proxycfg command to enter a working proxy ip/port number/exclusion list: http://msdn.microsoft.com/en-us/library/aa384069%28VS.85%29.aspx
2. After you do the above, confirm that the new proxy details are working by running the proxycfg command again
3. Next, restart the NEOSYS processes for the changes to be affected
   

Note: Please refer to the following link before you restart NEOSYS processes: Closing NEOSYS Services

On the client server, look at the text of UPDATE.$WG or UPDATE.xWG and other UPDATE.* files in the client's NEOSYS installation neosys/neosys or hosts/xxxxxxxx/work folder for clues.

There is a problem with the USB media inserted for backup

Refer here

NEOSYS thinks it sees a new neosys2.exe upgrade file on the location http://www.neosys.com/support/neosys2.exe and attempts to download it

Refer here

Fixing "Cannot make SSL connection" error

Error Message

Users get the message "This page cannot be displayed" when they try to access the HTTPS website. TODO Add screenshot

Cause

When there are multiple HTTPS sites on one server, any subsequent SSL self certifications seems to destroy all other sites with self certification where the export/REMOVE/import step is not done for some reason. See Export, Remove and Import Step

This issue is only evident after the server restart.

Solution

Re-install certificates. See Re-installing Certificates

Fixing "check_ssl_certificates" error

Cause

This error appears when the current ssl certificate to which website binding is done is no longer valid.

Solution

Use the latest valid ssl certificate and redo website binding. Refer to link Setting up HTTPS

Backup -> Impossible alert

Possible Causes and Solutions

If there is an error "Backup->Impossible" on Nagios check if the USB is properly inserted and schedule downtime to Nagios for 2 hours.

Troubleshooting Network Outages reflected in Nagios due to reassigning of router name or IP address

Nagios displays errors if a router name or the ip address it is monitoring have been reassigned.

We can resolve this issue by trying to find the ISP router ip address just before the NEOSYS server.

Steps:-

1. Login to Nagios
2. Click on Tactical Overview -> Network Outages and click on Blocking Outages to view
   
   
   
   
3. You will now see the host/ISP which is down. Click on the status map icon to identify the host associated with the ISP, as shown below:
   
   
   
   
4. From the Network Map displayed, identify the host associated with the ISP.
   
   
   
   
5. If you already know the ip address of the host then skip to next step else, in Nagios, click on Host Detail, then on the hostname identified earlier and From the Host Details shown, save the host url.
   
   
   
   
   
   
6. Log onto www.network-tools.com:
   • Select Trace
   • Enter the host's ip address if already known or host url
   • Click on Go
     
     
     
     
7. The trace route should complete successfully revealing the IP address of the ISP just before the NEOSYS server.
   
   
   
   
8. You can now login to zoneedit and update the ip address of the host.
9. Check Nagios.

Nagios reports a hung process

Possible Causes and Solutions

Refer link here

Explorer.exe not running

Nagios will display this error for only Win3 at the moment.

Possible Causes and Solutions

This error means that the server has (for whatever reasons) rebooted and stuck at the Windows login prompt for someone to enter the username & password. (More info on explorer.exe is available at http://en.wikipedia.org/wiki/Explorer.exe)

Solution to this problem would be to login via Tunnelier and open up Remote Desktop Connection.

Configuring Sonicwall firewall to allow NEOSYS to update Nagios

Configuring Sonicwall firewall to allow NEOSYS to update Nagios

This is documented at Configuring Sonicwall firewall to allow NEOSYS to update Nagios

Counting current active users

NEOSYS gives an estimate of the number of users currently active by counting how many users have been seen to be active, even once, within the last hour.

This can give a sense of the processing requirements for an installation.

The numbers can be seen in nagios installations screen from support.htm

Example:

 Users: 5/4/2 Max: 7/5/2

Users means:

• 5 unique browser session ids seen in the last hour
• 4 unique user codes seen in the last hour
• 2 Unique IP Nos seen in the last hour (60 mins)

Max means the maximums of the above seen in any one of the last 24 hours.

In practice, the middle figure, count of unique user codes, is very close to the real number of people active within an hour. However, since the same user code can be used by different people, even on different computers (although not at the same time) it could technically be an underestimate.

The number of currently active users is usually much lower than the number of registered users. This is because some registered users use the system infrequently and many dont use the system all the time.

The total number of different user codes seen to be active over the whole day is not shown. For that, you can see Usage Statistics.