Handling Nagios Client Monitoring System
- 1 Procedure to handle Nagios
- 2 Nagios services
- 3 How to handle a service error
- 4 Troubleshooting NAGIOS generally
- 4.1 Fixing “CRITICAL – Socket timeout after 10 secs” error message on NAGIOS
- 4.2 Fixing “NEOSYS has not checked in” error message
- 4.2.1 Error Message Explained
- 4.2.2 Possible Causes & Solutions
- 126.96.36.199 The maintenance window is left open
- 188.8.131.52 Hung process on server
- 184.108.40.206 The NEOSYS process IS NOT running on the server
- 220.127.116.11 The NEOSYS process IS running on the server but still cannot connect to NAGIOS
- 18.104.22.168 There is a problem with the USB media inserted for backup
- 22.214.171.124 NEOSYS thinks it sees a new neosys2.exe upgrade file on the location http://www.neosys.com/support/neosys2.exe and attempts to download it
- 4.3 Fixing "Cannot make SSL connection" error
- 4.4 Troubleshooting Network Outages reflected in Nagios due to reassigning of router name or IP address
- 5 Configuring Sonicwall firewall to allow NEOSYS to update Nagios
- 6 Counting current active users
1 Procedure to handle Nagios
The procedure that support staff need to follow while handling Nagios is documented under Procedures: Handling Nagios Client Monitoring System
Nagios is accessed via this link: http://monitor.neosys.com/nagios3
2 Nagios services
Nagios is configured to display information pertaining to all NEOSYS client's server statuses which include multiple services such as:
- HTTPS: Most of NEOSYS clients are configured to have external web access via secure HTTP protocol (port 4430) from outside office. Nagios is configured to check port 4430 on a regular interval of 10 minutes and display any issues in accessing the same.
- SSH: As part of the support contract, NEOSYS should have external secure access to the client server usually over port 19580. Nagios is configured to check this port on a regular interval of 10 minutes and display any issues in accessing the same.
- Ping: Nagios is also configured to ping the client router as a measure to check if router responds incase the NEOSYS server is down.
- NEOSYS: This service works in a reverse direction, and the NEOSYS installation on the client server sends information such as databases running, current backup status, internal and internet IP addressess etc to Nagios on a regular interval of 10 minutes.
Some key information about Nagios is as follows:
- Nagios is also configured to display information related to internal servers.
- Clients hosted on a NEOSYS cloud server might not have services such as SSH or PING as this is monitored as part of the internal server service.
- Nagios sends out email alerts to firstname.lastname@example.org (which is forwarded to email@example.com) from 8 am to 12 midnight on all Dubai working days (Sun-Thu). No alerts are sent out on Fri and Sat, unless they are for NEOSYS internal servers.
Nagios Configuration files:
- Commands to check different services. ~/etc/nagios3/commands.cfg
- Contacts where firstname.lastname@example.org is configured. Emails from nagios will be sent to the email ID configured here. /etc/nagios3/conf.d.backups/contacts.cfg
- For Oman Client. Unclear of purpose. /etc/nagios3/conf.d.backups/dtme.cfg
- Fully commented file of generic hostgroup definitions. /etc/nagios3/conf.d.backups/hostgroups_nagios2.cfg
- Host group definition. /etc/nagios3/conf.d.backups/linux.cfg
- Files where all the neosys hosts and hostgroups are defined. /etc/nagios3/conf.d.backups/neosysclients.cfg
- Config for monitor.neosys.com. /etc/nagios3/conf.d.backups/monitor.cfg
3 How to handle a service error
- Nagios Service Info - get there via various routes eg from Service Problems - then click on the service name (not the host name)
- Service Commands, Acknowledge this service problem (only services with status Warning or Critical have this option)
- Enter a note - explaining to yourself and your co-workers explaining how the problem is being handled and when to follow up
Notifications will be automatically resumed once the service becomes OK again.
The "Disable notifications" is not quite the same and shows as red on tactical summary screen.
3.2 How to stop ALL notifications
Useful to stop a massive number of alerts due to various causes.
- Nagios Process Info
- Enable/Disable notifications
3.3 Speeding up Nagios web interface
The usual F5 to refresh before the automatic 90 second refresh works but Ctrl+F5 doesnt.
3.4 Speeding up NEOSYS process checkins
You can force a neosys service checkin from NEOSYS maintenance mode (any process/database) press F5
4 Troubleshooting NAGIOS generally
4.1 Fixing “CRITICAL – Socket timeout after 10 secs” error message on NAGIOS
4.1.1 Error Message
NAGIOS is not updating services like CPU Load, Drive Space C:, Drive Save D:,Explorer, Memory Usage etc.
Open Windows Task Manager and kill any nscp.exe process. Then, restart NSClient++ from the desktop or by going to Start> Programs> NSClient++ These steps can be carried out even when users are active.
4.2 Fixing “NEOSYS has not checked in” error message
4.2.1 Error Message Explained
Nagios reports this error when it is is not able to update the status of NEOSYS for a particular client server.
4.2.2 Possible Causes & Solutions
126.96.36.199 The maintenance window is left open
Make sure that there is no maintenance window left open in the server.
188.8.131.52 Hung process on server
Check if there are any hung processes on the server e.g. Fatal Error in Rev Restart. Follow steps in troubleshooting hung process.
184.108.40.206 The NEOSYS process IS NOT running on the server
If there are no NEOSYS processes running, then start the NEOSYS process and wait for NEOSYS to check into Nagios, or force Nagios to re-check the status of NEOSYS service.
220.127.116.11 The NEOSYS process IS running on the server but still cannot connect to NAGIOS
NEOSYS connects to Nagios using http and automatically detects and uses any http proxy configuration configured in Internet Explorer.
If Internet Explorer in the server can reach the internet then NEOSYS should be able to update to NAGIOS via the same proxy.
Check if Internet Explorer can reach Nagios by using the below link. The check is successful if you get the Nagios login request shown in the image below.
18.104.22.168.1 If Internet Explorer CANNOT connect to Nagios
The client's IT must enable outbound access preferably on all ports or at least 4428 to our NEOSYS server at monitor.hosts.neosys.com.
If outbound access to monitor.hosts.neosys.com on port 4428 has not been enabled, then Nagios will not be able to update the status of the host and will show the error "NEOSYS not checked in".
If port 4428 cannot be used, then Support can manually configure the network to use standard port 443 in the system configuration file.
This requires that the client's IP number is added to the list of allowed IP numbers on nl10r router since Nagios https server gets multiple probes a day on port 443 if it is left open to everyone.
22.214.171.124.2 If Internet Explorer CAN connect to Nagios
View the Internet Explorer proxy configuration as follows:
Check if there is an issue with NEOSYS' http proxy server configuration as follows:
- Open the UPDATE.$WG or UPDATE.xWG file located in the neosys\neosys or hosts/xxxxxxxx/work folder.
- You should find a message similar to the following:
Connecting to 192.168.100.145:8080 failed: No such file or directory.
Where the above appears to be some non-functional http proxy server ip/port number and is not the expected nagios server ip number. A windows proxy command shows the same ip and port:
On Windows 2003/XP
On Windows 2008/Win7
NetSH WinHTTP import Proxy ie
Microsoft (R) WinHTTP Default Proxy Configuration Tool Copyright (c) Microsoft Corporation. All rights reserved. Current WinHTTP proxy settings under: HKEY_LOCAL_MACHINE\ SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\Connections\ WinHttpSettings : Proxy Server(s) : 192.168.100.145:8080 Bypass List : 192.168.*.*;localhost;<local>
Solution 1 - Remove the above setting to create a direct connection
- To remove the registry entries that ProxyCfg.exe creates,you must delete the WinHttpSettings value from the following registry key:
- After you do the above, confirm that the proxy details are deleted by running the proxycfg command
- Next, restart the NEOSYS processes for the changes to be affected
Solution 2 - Configure a functioning proxy ip/port number
- Use the proxycfg command to enter a working proxy ip/port number/exclusion list: http://msdn.microsoft.com/en-us/library/aa384069%28VS.85%29.aspx
- After you do the above, confirm that the new proxy details are working by running the proxycfg command again
- Next, restart the NEOSYS processes for the changes to be affected
Note: Please refer to the following link before you restart NEOSYS processes: Closing NEOSYS Services
On the client server, look at the text of UPDATE.$WG or UPDATE.xWG and other UPDATE.* files in the client's NEOSYS installation neosys/neosys or hosts/xxxxxxxx/work folder for clues.
126.96.36.199 There is a problem with the USB media inserted for backup
4.3 Fixing "Cannot make SSL connection" error
4.3.1 Error Message
Users get the message "This page cannot be displayed" when they try to access the HTTPS website. TODO Add screenshot
When there are multiple HTTPS sites on one server, any subsequent SSL self certifications seems to destroy all other sites with self certification where the export/REMOVE/import step is not done for some reason. See Export, Remove and Import Step
This issue is only evident after the server restart.
Re-install certificates. See Re-installing Certificates
4.3.4 Fixing "check_ssl_certificates" error
This error appears when the current ssl certificate to which website binding is done is no longer valid.
Use the latest valid ssl certificate and redo website binding. Refer to link Setting up HTTPS
4.3.5 Backup -> Impossible alert
188.8.131.52 Possible Causes and Solutions
If there is an error "Backup->Impossible" on Nagios check if the USB is properly inserted and schedule downtime to Nagios for 2 hours.
4.3.6 "Change Backup" alert
4.4 Troubleshooting Network Outages reflected in Nagios due to reassigning of router name or IP address
Nagios displays errors if a router name or the ip address it is monitoring have been reassigned.
We can resolve this issue by trying to find the ISP router ip address just before the NEOSYS server.
- Login to Nagios
- Click on Tactical Overview -> Network Outages and click on Blocking Outages to view
- You will now see the host/ISP which is down. Click on the status map icon to identify the host associated with the ISP, as shown below:
- From the Network Map displayed, identify the host associated with the ISP.
- If you already know the ip address of the host then skip to next step else, in Nagios, click on Host Detail, then on the hostname identified earlier and From the Host Details shown, save the host url.
- Log onto www.network-tools.com:
- The trace route should complete successfully revealing the IP address of the ISP just before the NEOSYS server.
- You can now login to zoneedit and update the ip address of the host.
- Check Nagios.
4.4.1 Nagios reports a hung process
184.108.40.206 Possible Causes and Solutions
Refer link here
4.4.2 Explorer.exe not running
Nagios will display this error for only Win3 at the moment.
220.127.116.11 Possible Causes and Solutions
This error means that the server has (for whatever reasons) rebooted and stuck at the Windows login prompt for someone to enter the username & password. (More info on explorer.exe is available at http://en.wikipedia.org/wiki/Explorer.exe)
Solution to this problem would be to login via Tunnelier and open up Remote Desktop Connection.
5 Configuring Sonicwall firewall to allow NEOSYS to update Nagios
5.1 Configuring Sonicwall firewall to allow NEOSYS to update Nagios
This is documented at Configuring Sonicwall firewall to allow NEOSYS to update Nagios
6 Counting current active users
NEOSYS gives an estimate of the number of users currently active by counting how many users have been seen to be active, even once, within the last hour.
This can give a sense of the processing requirements for an installation.
The numbers can be seen in nagios installations screen from support.htm
Users: 5/4/2 Max: 7/5/2
- 5 unique browser session ids seen in the last hour
- 4 unique user codes seen in the last hour
- 2 Unique IP Nos seen in the last hour (60 mins)
Max means the maximums of the above seen in any one of the last 24 hours.
In practice, the middle figure, count of unique user codes, is very close to the real number of people active within an hour. However, since the same user code can be used by different people, even on different computers (although not at the same time) it could technically be an underestimate.
The number of currently active users is usually much lower than the number of registered users. This is because some registered users use the system infrequently and many dont use the system all the time.
The total number of different user codes seen to be active over the whole day is not shown. For that, you can see Usage Statistics.