Handling Nagios Client Monitoring System

Procedure to handle Nagios

The procedure that support staff need to follow while handling Nagios is documented here

Nagios is accessed via this link: http://monitor.neosys.com/nagios3

Nagios services

Nagios is configured to display information pertaining to all NEOSYS client's server statuses which include multiple services such as:

1. HTTPS: Most of NEOSYS clients are configured to have external web access via secure HTTP protocol (port 4430) from outside office. Nagios is configured to check port 4430 on a regular interval of 10 minutes and display any issues in accessing the same.
2. SSH: As part of the support contract, NEOSYS should have external secure access to the client server usually over port 19580. Nagios is configured to check this port on a regular interval of 10 minutes and display any issues in accessing the same.
3. Ping: Nagios is also configured to ping the client router as a measure to check if router responds incase the NEOSYS server is down.
4. NEOSYS: This service works in a reverse direction, and the NEOSYS installation on the client server sends information such as databases running, current backup status, internal and internet IP addressess etc to Nagios on a regular interval of 10 minutes.

Some key information about Nagios is as follows:

• Nagios is also configured to display information related to internal servers.
• Clients hosted on a NEOSYS cloud server might not have services such as SSH or PING as this is monitored as part of the internal server service.
• Nagios sends out email alerts to support2@neosys.com (which is forwarded to support@neosys.com) from 8 am to 12 midnight on all Dubai working days (Sun-Thu). No alerts are sent out on Fri and Sat, unless they are for NEOSYS internal servers.

How to handle a service error

1. Nagios Service Info - get there via various routes eg from Service Problems - then click on the service name (not the host name)
2. Service Commands, Acknowledge this service problem (only services with status Warning or Critical have this option)
3. Enter a note - explaining to yourself and your co-workers explaining how the problem is being handled and when to follow up

Notifications will be automatically resumed once the service becomes OK again.

The "Disable notifications" is not quite the same and shows as red on tactical summary screen.

How to stop ALL notifications

Useful to stop a massive number of alerts due to various causes.

1. Nagios Process Info
2. Enable/Disable notifications

Speeding up Nagios web interface

The usual F5 to refresh before the automatic 90 second refresh works but Ctrl+F5 doesnt.

Speeding up NEOSYS process checkins

You can force a neosys service checkin from NEOSYS maintenance mode (any process/database) press F5

MONITOR2

Adding the client to Nagios

This can be done by IT personnel of NEOSYS and needs to be escalated to them.

Troubleshooting NAGIOS generally

Fixing “CRITICAL – Socket timeout after 10 secs” error message on NAGIOS

Error Message

Problem

NAGIOS is not updating services like CPU Load, Drive Space C:, Drive Save D:,Explorer, Memory Usage etc.

Solution

Open Windows Task Manager and kill any nscp.exe process. Then, restart NSClient++ from the desktop or by going to Start> Programs> NSClient++ These steps can be carried out even when users are active.

Fixing “NEOSYS has not checked in” error message

Error Message Explained

You notice a problem on NAGIOS indicating that NEOSYS not checked in on a particular client server. This happens because NEOSYS is not updating NAGIOS.

Possible Causes & Solutions

1. The maintenance window is left open. Make sure the maintenance window has not been left open in the server.
2. Hung process on server. e.g.:- Fatal Error in Rev Restart. Follow steps in troubleshooting hung process.

The NEOSYS process is NOT running on the server

Start the NEOSYS process and wait for 10 mins for NEOSYS to check into NAGIOS.

The NEOSYS process is running on the server but still cannot connect to NAGIOS

NEOSYS connects to NAGIOS using http. NEOSYS automatically detects and uses any http proxy configuration configured in Internet Explorer. If the Internet Explorer in the server can reach the internet then NEOSYS should be able to update to NAGIOS via the same proxy.

First check if Internet Explorer in the server can reach NAGIOS. Type the following link into the Internet Explorer in the server:

http://monitor.neosys.com

If you are asked to login then the Internet Explorer is working OK. You do not need to login. Just cancel and move on to the next step.

If Internet Explorer CANNOT connect to NAGIOS then resolving that fundamental issue will probably solve the NAGIOS connection issue too.

View the Internet Explorer proxy configuration as follows:

If Internet Explorer CAN connect to NAGIOS then check if there is an issue with NEOSYS's http proxy server configuration as follows:

1. Search for UPDATE.$WG file located in the neosys\neosys. folder and open it using notepad or wordpad.
2. You should find a message similar to the following:

Connecting to 192.168.100.145:8080 failed: No such file or directory.

Where the above appears to be some non-functional http proxy server ip/port number and is not the expected nagios server ip number. A windows proxy command shows the same ip and port:

On Windows 2003/XP

proxycfg

On Windows 2008/Win7

NetSH WinHTTP import Proxy ie

Output:

Microsoft (R) WinHTTP Default Proxy Configuration Tool
Copyright (c) Microsoft Corporation. All rights reserved.

Current WinHTTP proxy settings under:
  HKEY_LOCAL_MACHINE\
    SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\Connections\
      WinHttpSettings :

    Proxy Server(s) :  192.168.100.145:8080
    Bypass List     :  192.168.*.*;localhost;<local>

Solution 1 - Remove the above setting to create a direct connection

1. To remove the registry entries that ProxyCfg.exe creates,you must delete the WinHttpSettings value from the following registry key:
   

   HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\Connections\WinHttpSettings

2. After you do the above, confirm that the proxy details are deleted by running the proxycfg command
   
3. Next, restart the NEOSYS processes for the changes to be affected

Solution 2 - Configure a functioning proxy ip/port number

1. Use the proxycfg command to enter a working proxy ip/port number/exclusion list: http://msdn.microsoft.com/en-us/library/aa384069%28VS.85%29.aspx
2. After you do the above, confirm that the new proxy details are working by running the proxycfg command again
3. Next, restart the NEOSYS processes for the changes to be affected

Note: Please refer to the link before you restart NEOSYS processes Closing NEOSYS Services

On the client server, look at the text of UPDATE.$WG and other UPDATE.* files in the client's NEOSYS installation neosys/neosys folder for clues.

There is a problem with the USB media inserted for backup

Refer here

NEOSYS thinks it sees a new neosys2.exe upgrade file on the location http://www.neosys.com/support/neosys2.exe and attempts to download it

Refer here

Fixing "Cannot make SSL connection"

Error Message

Users get the message "This page cannot be displayed" when they try to access the HTTPS website. TODO Add screenshot

Cause

When there are multiple HTTPS sites on one server, any subsequent SSL self certifications seems to destroy all other sites with self certification where the export/REMOVE/import step is not done for some reason. See Export, Remove and Import Step

This issue is only evident after the server restart.

Solution

Re-install certificates. See Re-installing Certificates

Backup -> Impossible alert

Possible Causes and Solutions

If there is an error "Backup->Impossible" on Nagios check if the USB is properly inserted and schedule downtime to Nagios for 2 hours.

Troubleshooting Network Outages reflected in Nagios due to reassigning of router name or IP address

Nagios displays errors if a router name or the ip address it is monitoring have been reassigned.

We can resolve this issue by trying to find the ISP router ip address just before the NEOSYS server.

Steps:-

1. Login to Nagios
2. Click on Tactical Overview -> Network Outages and click on Blocking Outages to view
   
   
   
   
3. You will now see the host/ISP which is down. Click on the status map icon to identify the host associated with the ISP, as shown below:
   
   
   
   
4. From the Network Map displayed, identify the host associated with the ISP.
   
   
   
   
5. If you already know the ip address of the host then skip to next step else, in Nagios, click on Host Detail, then on the hostname identified earlier and From the Host Details shown, save the host url.
   
   
   
   
   
   
6. Log onto www.network-tools.com:
   • Select Trace
   • Enter the host's ip address if already known or host url
   • Click on Go
     
     
     
     
7. The trace route should complete successfully revealing the IP address of the ISP just before the NEOSYS server.
   
   
   
   
8. You can now login to zoneedit and update the ip address of the host.
9. Check Nagios.

Nagios reports a hung process

Possible Causes and Solutions

Refer link here

Explorer.exe not running

Nagios will display this error for only nl1 at the moment.

Possible Causes and Solutions

This error means that the server has (for whatever reasons) rebooted and stuck at the Windows login prompt for someone to enter the username & password. (More info on explorer.exe is available at http://en.wikipedia.org/wiki/Explorer.exe)

Solution to this problem would be to login via Tunnelier and open up Remote Desktop Connection.

Configuring Sonicwall firewall to allow NEOSYS to update Nagios

Configuring Sonicwall firewall to allow NEOSYS to update Nagios

This is documented at Configuring Sonicwall firewall to allow NEOSYS to update Nagios