Troubleshooting NEOSYS Generally
Solving failure to start a NEOSYS server due to disk failure message
Problem
During a reboot process (which maybe due to a Windows update or even done by a support personnel) the NEOSYS server gets hung on the startup and shows a message "Boot Failure - Abort, Retry".
Temporary solution
This typically happens due to the USB being plugged into the server and the boot sequence being wrong - i.e. the server trying to boot from the USB first and fails. The immediate solution would be to unplug the USB and ask the client to reboot the server again and upon successfully rebooting the system, plug the USB back again.
Permanent solution
The above problem will occur every time the computer is rebooted, so you need to immediately talk to the IT Administrator of the client and ask them to rectify the boot sequence to make it boot first from the CD ROM, then the HDD and last the USB.
Allowing "Boot from USB" causes a severe risk of infection by boot sector viruses since the first infected USB device inserted WILL infect the server immediately as anti-virus programs are not active during boot.
Troubleshooting "Document is being updated" message
Troubleshooting the "Database not available" error message
Problem explained
This error appears when you try to login to NEOSYS after you enter your username and password and click the Login button.
Error message :
Cannot login because : Error : The (database code) database is not available right now.
If the error message appears post login i.e. when users are working on the system then check if processes are free to run user request.
Solution explained
- Determine if the processes are running. If they are running and you still get the same message that means that either the processes have hung and need to be killed. Or the processes are busy to run user requests and a new process needs to be started.
- If the processes are not visible on the destop, it is possible that they are running in the background and have hung for some reason. Check the windows task manager to see if any ‘ntvdm’ process is running and fix hung processes.
- If there is no trace of any process running, that means the process probably did not start at the scheduled time. Refer to the 'Hung Process Reports' to verify this claim.
- If the process had not hung, then the server might have restarted due to a power failure or a windows update and the administrator user had not logged in post the scheduled startup time of 6AM. To determine this cause, investigate in the Windows Event Viewer Log file.
- You can now start up the process by clicking on the respective desktop icons.
- Also check if the backup took place successfully or not. If not take a manual backup.
- Look into the logs at the date/time stated for the last transaction processed to investigate why process got hung.See Inspecting logs for more information on logs
Troubleshooting "user not authorised to login from a location" error message
Error message:
xxx is not authorised to login form the location (IP Number. xx.xx.xx.xx)
Solution Explained:
Check the URL used and follow the steps below to check if it is correct and email the user accordingly.
- If the Client installation is hosted on NEOSYS server then users can use only https link to access NEOSYS.
- Check with the client's management if this particular IP is their public IP.
- Add IP on management confirmation (Refer to Handling User Requests to add IP/IPs )
- In case of Client hosted server, users should access NEOSYS via LAN using the http link.
- There can be exceptional cases where user needs to access NEOSYS outside the office Network e.g a client installation with two companies at different locations and NEOSYS installed at one. In this case Support will have to add the IP number of the second company so that users can access NEOSYS.
But before you even suggest to add the IP, get the request from their management saying that the IP number is another office location and needs to be added. (Refer to Handling User Requests to add IP/IPs )
- There can be exceptional cases where user needs to access NEOSYS outside the office Network e.g a client installation with two companies at different locations and NEOSYS installed at one. In this case Support will have to add the IP number of the second company so that users can access NEOSYS.
Handling damaged files
Checking for corrupt database files
Login to NEOSYS Maintenance. This can be done when users are online.
Press F5
CHK.FILES
or
CHK.FILES filename
Sizelock while performing chk.files
Fixing sizelock errors should not be done while other users are online to the same database.
Sizelock errors are not critical and will be fixed automatically during the nightly backup.
Sizelock errors occur if a program or process that is selecting records from a file is aborted in some abnormal way.
Error message:
These Files/Tables have a Sizelock Value of 2 or greater. Tag/Select the Files/Tables to be Fixed. Press F9 to fix selected files
Press F9 to proceed with fixing the selected files or press ESC to continue with chk.files without fixing sizelock as it gets automatically fixed during the nightly backup.
Refer to the Sizelock errors in backup emails for more information.
Determining Database File Name from Operating System File Name
To assess the potential damage and possible remedial measures you need to know the database file name. If the message only refers to the operating system file name you need to follow this procedure to determine the database file name.
Once you have the database file name you can use CHK.FILES XXXXXXX to check if corrupt or not and various other procedures to fix the corruption.
Remember that fixing the corrupt data does not solve the overall problem. The *cause* of the corruption must be identified and eliminated otherwise the problem may reoccur and in a more serious form perhaps with unrecoverable loss of data.
╒═══════════════════════════════════TCL - 2══════════════════════════════════╕ │ │ │ :list FILES WITH ALL CONTAINING 'REV76481' │ │ │ ╘════════════════════════════════════════════════════════════════════════════╛
Finding out when and by whom a record was deleted
In most cases, NEOSYS does not allow users to delete records and instead keeps a record of everything. In some cases however, things are deleted and the only way to get full details about the deletion is to search the logs. This is cumbersome, but there is a quick way to find out when, and by whom, a record was deleted. Prior to NEOSYS software versions dated Mar 2014, and deletions done before the same date, this procedure will only tell you when the record was deleted - but not who deleted it. Knowing exactly when it was deleted will nevertheless help you to search the logs for full details.
In maintenance mode F5
ED SHADOW DELETED*filename*key
For example:
ED SHADOW DELETED*BATCHES*L*JOU*2*U
Journals are stored in the BATCHES file. The key of an unposted batch is x*y*999*U where x is the company code, y is the journal type code, 999 is UNPOSTED batch number and U is just U to indicate unposted batches. Note that unposted batches are normally deleted at the time they are posted - ie converted to posted batches.
╔══════════════════════════┤DELETED*BATCHES*L*JOU*2*U├═════════════════════════╗ ║16831.60706 ║ ║BRUCEL ║ ║╒═══════════════════════════════════TCL - 6══════════════════════════════════╕║ ║│ │║ ║│ :EVAL PRINT 16831.60706 '[DATETIME]' │║ ║│ │║ ║╘════════════════════════════════════════════════════════════════════════════╛║ ║ ║ ║ ║
Once you see the number (in this case 16831.60706) you can convert it to a time and date by typing something like
PRINT 16831.60706 '[DATETIME]'
Using this date and time you can search the logs more effectively to find out who did the deletion and in what circumstances.
Assessing Database Size per File
In NEOSYS maintenance mode
LIST FILES BY BY-DSND SIZE VOLUME.NAME SIZE
This will include all files not just those in the actual dataset in DATA\* folders.
Fixing slow speed
Benchmarking NEOSYS
Investigating slow response in NEOSYS using NEOSYS logs
If users complain about slow speed, but the server CPU performance looks normal, then support MUST investigate NEOSYS logs to find out why NEOSYS was slow at the time when the client complained about slow response.
Search NEOSYS log entries around the time that the user complained about slow response and look for log entries with high response time. Make a note of what requests took long to respond. If multiple users were simultaneously requesting long reports from NEOSYS, then NEOSYS can be expected to respond slowly for other smaller requests that were processed at that time.
Also refer to why is NEOSYS taking a long time to generate a report.
Investigating CPU 100% using Windows Task Manager
Email, to support, a screen-shot of task manager APPLICATIONS, PROCESSES and PERFORMANCE screens MAXIMIZED TO SHOW AS MUCH AS POSSIBLE.
(Sort the processes to show ntvdm, waiting.exe and high cpu% processes clearly)
Steps:
- Right Click on Windows Taskbar and click on Start Task Manager
- Click on Processes and then click on CPU
Note - The HIGH cpu% processes which should usually be the "process" called "System Idle Process"
- Click on the Performance Tab
Note - PF Usage should typically be much less than Physical Memory otherwise there is insufficient real memory in the server to handle the load
- Click on Application Tab then Right Click on a NEOSYS Process and Click on Bring to Front
See what the NEOSYS Process is doing
- Right Click on a NEOSYS Process and Click on Go to Processes
Note the cpu% ntvdm process
- Normally NEOSYS application screens say "LISTENING" in the bottom line and those applications should have very low cpu%
- Look at the difference between the screen of running NEOSYS processes (applications actually) which are idle (listening for requests) and active (processing a request from a user)
- Note the number of cpus or cpu threads in the server from the performance screen graphics
- Take screen-shots of any and ALL hung or long running processes (NEOSYS application screens) and email them to support. Even small details on the screens and user names, the user names may give clues to what problem caused the hanging.A Typical Hung NEOSYS process will look like this:
- Once all hung/long processes are closed then CPU should be low and not near 100%. If it is still 100% then check all high cpu% processes and send a screen-shot of processes sorted to show the high cpu% process names to support.
Solving server CPU% is 100 and all users are extremely slow/stopped
Get the screenshots of Task Manager and ALL processes on the server, the objective is to assess the true issue. No need to get the screens not in use obviously but you can send a parallel screen shot for them if you want to be pedantic or even a comment will do.
Too few CPUs/threads for the number of users
In Windows task manager normally, you should see one ntvdm.exe and one waiting.exe process per NEOSYS process (application). A standard installation has three NEOSYS processes per main database and plus one per test database. This is configured in Support Menu, Configuration File.
If there are MORE ntvdm processes than you expect from the configuration file, then perhaps NEOSYS is auto starting new NEOSYS processes to try and cater for a high number of concurrent users.
If the number of concurrent NEOSYS processes significantly exceeds the number of cpus/hyperthreads available in the server then processing for everybody can become so slow for everybody and almost no work gets done.
Solution
Stop NEOSYS creating new NEOSYS processes automatically. Create a text file with the first and only line as AUTOSTART=NO in the neosys\neosys folder something like this.
notepad d:\neosys\neosys\NET.CFG
AUTOSTART=NO
How do I troubleshoot email not received?
Troubleshooting email not received
Fixing permissions errors while logging in
Problem
While logging in, you get the following error message:
Solution
Add the internet guest account to the security list of the data folder with the default permission of list/read/write
Make sure the read&execute permission is removed
Fixing the 'HTTP Error 500.0 - Internal Server Error' while logging in on IE on a Windows Vista system
Problem
After configuring IIS on Windows Vista you will get this error message while trying to login into NEOSYS from Internet Explorer:
HTTP Error 500.0 - Internal Server Error Description: This application is running in an application pool that uses the Integrated .NET mode. This is the preferred mode for running ASP.NET applications on the current and future version of IIS. In this mode, the application using client impersonation configured with <identity impersonate="true" /> may not behave correctly. Client impersonation is not available in early ASP.NET request processing stages and may lead modules in those stages to execute with process identity instead.
Solution
You can move the application to an application pool that uses the Classic .NET mode by using the following from a command line window (the window must be running as Administrator)
%systemroot%\system32\inetsrv\APPCMD.EXE set app "Default Web Site/neosys" /applicationPool:"Classic .NET AppPool"
Alternatively, you can use any other application pool on your system that is running in the Classic .NET mode. You can also use the IIS Administration tool to move this application to another application pool.
Fixing the 'Class Not Registered' error message while logging in
Problem
While logging into NEOSYS, you will get a popup window giving an error message saying 'Class Not Registered - Server Error'. Typically, you will encounter this error with XP Pro IIS 5.1. As usual, there's way to solve it, however the root cause of this is still unknown.
Anyway, you will get the proper message in the event log:
Event Type: Warning Event Source: W3SVC Event Category: None Event ID: 36 Description: The server failed to load application '/LM/W3SVC/1/ROOT/NEOSYS. The error was 'Class not registered'.
Solution
So, what do you do ? This problem is related to Component Services, and when you open Component Services MMC, you will most probably get Error Code 8004E00F COM + was unable to talk to Microsoft Distributed Transaction Coordinator. So, fix the COM+ services first by using the following KB from Microsoft (PRB: Cannot Expand "My Computer" in Component Services MMC Snap-In http://support.microsoft.com/?id=301919):
To resolve this problem, reinstall Component Services as follows: WARNING:
- Open registry editor, locate HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\COM3, and then delete this key.
- From the Start menu, point to Settings, click Control Panel, and then click Add/Remove Programs.
- Click Add/Remove Windows Components.
- Proceed through the wizard, and accept all of the defaults (including IIS)
- Restart the computer.
If the above didn't solve it, and you still receive the 'Class not registered' error message, then you need to recreate the IIS packages in COM+, try
- Delete IIS related package in Component Services MMC
- IIS In-Process Applications
- IIS Out-of-Process Pooled Applications
- IIS Utilities
Next, if you still get the message, try following before re-install IIS if you can't find Distributed Transaction Coordinator in your Services console.
Launch command prompt and run the following command.
- msdtc -install
- net start msdtc
Then try re-install IIS.
This should solve the problem
Enabling File Security option on Win XP Professional
Problem
In the Properties of any folders, the Security option does not show, hence you cannot modify the Read, Write options.
Solution
The solution would be to untick the 'Simple File Sharing' option from Tools > Folder Options > View:
%00%00%00%00 Errors
Error Message
SYSTEM ERROR in line 162. Amount "-2698.00AED" or base "%00%00%00%00" has been wrongly generated GET NEOSYS SUPPORT. DO NOT ATTEMPT TO CORRECT MANUALLY
Solution
%00%00%00%00 indicates an internal error that NEOSYS programmer has to fix. It is usually random and can be hard to replicate unlike almost all other NEOSYS errors which usually replicate reliably once you find the cause.
B703 Errors
The B703 error is usually always related to something too big for NEOSYS to handle.
These are the only B errors that NEOSYS cant always permanently prevent by fixing the software.
Internet Explorer Menu, View, Text Size doesnt change font size as expected
Cause: This is because the font size is now user definable in NEOSYS and View, Text Size does not override predefined font sizes.
Solution: If you are using Internet Explorer 7 you can scale the screen (including the font size using ctrl + and ctrl - keyboard shortcuts or the font size button on the bottom right hand side of the window.
You can adjust the font size on the User Details form when you login although this permanently applies to all forms not just the one that you are on.
Uploaded jpg files fail to display in internet explorer
Some large jpg files > 2Mb cannot be viewed in internet explorer despite being viewable in image preview, ms paint and other viewers/editors. It is not an issue caused by uploading or downloading the files.
These file appear to have been created on Photoshop CS Macintosh and may be a special type of uncompressed jpg used for production quality files.
Partial solution
Before uploading the files, open them in some editor like MS Paint (right click, edit) and save them. However this results in a loss of quality. Perhaps there is some program that can convert these files to a format understandable by Internet explorer without any loss of quality.
"This document is currently read only"
Cause
The user attempting to modify this document does not have the authorization key to do so.
Solution
Inform the user that he is not authorised to modify the document and give him the list of users within his company who are authorised to do so.
"You have attempted to write to a read-only file"
Message
Error while writing data. You have attempted to write to a read-only file. - or - access to the file has been denied by the operating system. (operating system file name: "..\DATA\ADLINEC\ADAGENCY\REV76467.OV00012618")
Cause
It is almost certainly due to some third party backup or other maintenance software opening the NEOSYS database files when it shouldn't e.g. badly configured third party backup scheduled to backup NEOSYS while NEOSYS is still running. Note that the exact filename varies each time.
Solution
This can be a serious error that causes damaged files in NEOSYS especially if the filename ends in .OV. Use the usual methods of checking for damaged files e.g. do a backup which also looks for damaged files BUT DO NOT OVERWRITE THE LATEST BACKUP SINCE IT MAY BE REQUIRED for restoration. Then fix the damaged files using the usual methods e.g. by rebuilding/using FIXFILE or restoring databases. For more info check Handling damaged files
Prevention
Remove the third party backup or other maintenance software or reschedule it to run at a time that NEOSYS is shutdown. Removal of software may require hunting through the windows process list for unexpected programs running.
Troubleshooting Hung processes
Investigating hung NEOSYS processes
To find out some information about what the hung processes are suffering from, do the following:
When no important clients working (eg first thing in the morning) then request a shutdown of all NEOSYS processes. This should will leave only the hung processes open.
Then find what NEOSYS files are open as follows:
run Sysinternal's "Process Explorer" (from Start, Programs or procexp.exe from desktop)
find, handle
d:\
submit the complete list (maybe more than one page) to support for records
Investigating CPU 100% using Windows Task Manager
Solving server CPU% is 100 and all users are extremely slow/stopped
Error message: "Fatal Error in Rev Restart"
Error message: “Abort, Retry, Fail”
Problem
The following messages may come on older versions of NEOSYS if there is a problem with the USB media inserted for backup.
This results in “NEOSYS has not checked in” message on Nagios since it hangs during the monitoring update and locks all other processes from monitoring too.
General failure writing drive F Abort, Retry, Fail?
Not read reading drive F Abort, Retry, Fail?
Pressing A or F results in the problem happening again in about a minute, perhaps on a different process.
Solution
Temporary solution is to do “Safely remove hardware” from the windows notification area. Using “Eject” from My Computer will not work.
Permanent solution is to replace the defective USB memory stick. Sometime reformatting is sufficient.
Upgrading NEOSYS will probably stop the defective media from causing NEOSYS to hang but the USB will still be useless for backup.
Error message: "Read error in the operating system file"
Problem
Nagios reports a hung process and on the server a process has the following message popup.
Read error in the operating system file "..\DATA\PT0833\ACCOUNTS\REV20049 The file does not exist or the filename is not valid for the operating system.
Cause
Some non-NEOSYS program is directly accessing the NEOSYS database files while NEOSYS processes are running and using the files as well.
- Client IT staff using a backup program to perform backup without ensuring that NEOSYS processes are shutdown.
- A NEOSYS rsync process taking longer than expected due to new USB.
Solution
Kill the NEOSYS process eg with the X button. It is advisable NOT to let it run further while it cannot properly access one of its files.
Prevention
Remove the third party program or arrange for it to operate only while NEOSYS processes are shutdown eg from 3am-6am.
Error message: "Not enough string space - Out of Memory"
Message
‘RTP27’ Line 1. [B29] Not enough string space – Out of Memory.
Not enough memory to execute Debugger; current program aborted. Press any key to continue
Cause
Unknown perhaps related to some large document or report
Solution
Ask users for any hanging transactions. None other than closing and starting another process.
Also refer Enabling EMS memory on Window 2003
Error message: B521
Another user is currently updating indexes. Waiting to make your updates... If you interrupt this process, you may have to rebuild all the indexes
B521 message is usually temporary and goes away by itself. If it does not then, as the text of the B521 message makes clear, some OTHER process is holding up the system, preventing the process with B521 message from continuing.
B521 message is a symptom of other problem so it does not itself have some one simple solution. You must find which other process or issue is causing the hold up and solve whatever problem is causing the hold up, which could be anything.
Error message: " NTVDM encountered a hard error "
The following error message appears:
NTVDM encountered a hard error
Solution
This error is caused by missing or corrupt Windows system files (i.e. command.com, autoexec.nt and config.nt). The solution is to run a program called XP FIX which will reinstall these missing files. You can download it form http://www.visualtour.com/downloads/xp_fix.exe
Sometimes even after running the XP FIX program the error still appears. Then you need to copy all the files under C:/windows/repair to C:/windows/system32 and in the autoexec.nt and config.nt put REM before any lines there which don't have it.
Possible fix for some NTVDM errors
Error message
NTVDM has encountered a System error. The system cannot find the file specified. Choose close to terminate the application.
Possible solution
- Change the TEMP and TMP environment variables to C:\WINDOWS\TEMP. This should be for the user that runs NEOSYS processes - normally administrator.
- Close all NEOSYS processes
- Logout/Login again
- Restart NEOSYS processes
Check solution done properly
You can check that by typing ECHO %TEMP% and ECHO %TMP% at a console prompt after logging out and in again.
Note that windows will actually set the windows environment variables to something like C:\WINDOWS\TEMP\2 for some unknown reason.
d:\hosts>ECHO %temp% C:\WINDOWS\TEMP\2 d:\hosts>ECHO %tmp% C:\WINDOWS\TEMP\2
Error message: "Database not available" post login
Error message: "Cannot backup/restore because PROCESS1 PROCESS2 (etc) is/are online"
How to kill hung NEOSYS processes
NOTE WELL: If you kill actively working NEOSYS processes (those which are "listening" and not hung/crashed) there is a reasonable chance that the database will be damaged and might need a restore losing possibly large amounts of work.
If NEOSYS processes are visible on the server desktop
Look for processes which don't have "Listening ..." on the last but one line. The times on the left hand side are frozen as at the time of the hang.
You can then click the X to kill the process and confirm that this is OK.
Example of a NEOSYS process that has hung due to a software error resulting in a failure to handle a complex query with a lot of brand codes.
If NEOSYS processes are not visible on the server desktop
NEOSYS processes are most of the time visible on the desktop (i.e. the black dos windows) in Windows 2003 OS, except in the case that the process has been scheduled to start on computer restart and no one has logged into the server. In this case it would be running in the background. TODO
You can check if there are any hung processes from the NEOSYS Support Menu, List of Database Processes.
In this case you should follow the below instructions - however all of them need to be done within 30 seconds of starting the first instruction to avoid inconvenience to the users. It is recommend that you keep relevant windows open before proceeding with the same:
- Shutdown NEOSYS by TEMPORARILY putting a file called GLOBAL.END in the parent directory of NEOSYS (if there is already a GLOBAL.END.TEMP file then rename it to GLOBAL.END). Leaving the file there would prevent NEOSYS from starting up again. Shutting down NEOSYS from the Support menu will not work because of the hung processes.
- Use Windows Task Manager to kill all the NTVDM processes - assuming that you have closed all the visible NEOSYS processes, then the NTVDM processes in the task manager would be the hung one.
- Delete GLOBAL.END or rename it to GLOBAL.END.TEMP
- Restart the processes back again. If there are many datasets then you need to restart them all well within the 30 seconds period.
- Restarting a process is not noticed by users.
In case of Patsalides, where we have a thousand datasets which start "on demand" i.e. usually on login; all you need to do is start one dataset which will restart all the other datasets "on demand". If there is no response within 30 seconds then one of the running datasets will start it up so there appears to be a 30 second delay when you login to one of the thousand datasets the first time on any one day.
Temporary workarounds for hung NEOSYS processes
Until the error in the software is fixed users can often get their results by simplifying their requirements. For example select individual clients instead of selecting all the brands for a particular client. If the user has repeated his request (in forlorn hope that it would work finally) then the number of working NEOSYS processes will drop causing severe slowdown for other users and complete stop if all the NEOSYS processes hang.
Fixing " You do not have sufficient privilege to access this file "
This error message may come up while NEOSYS process startup.
Solution: Close the window and look for NEOSYS processes. In case there is no process, start the NEOSYS process.
Fixing 'NEOSYS process which opens up in a notepad instead of Microsoft Windows Based Script Host (i.e. DOS window)'
The NEOSYS process (cmd file) might open up in a notepad, instead of the usual black colour DOS window. This may happen if a JavaScript file is opened using a notepad. Please be very CAREFUL when accessing a .JS and .JSE file and double check that the default program remains wscript.exe. The issue can be fixed by the following:
- Check if Windows Script 5.6/5.7 is installed, IF NOT download and install it from the Microsoft Website.
- Go to Control Panel -> Default Programs -> Associate a file type or protocol with a program and then change the default program for .JS and .JSE to "Microsoft Windows Based Script Host"
Fixing a 'Could not start' error on Scheduled Tasks in Windows Server 2000 SP4
This error occurs because of a change that is made to the data that is stored in the credentials database when you install Windows 2000 SP4. Hence installing SP4 causes the the data that is stored in the credentials database to get converted to an SP4-compatible format. A registry key is configured to indicate that the data has been converted to the SP4 format.
Hence the Scheduled Tasks do not work sometimes. However the Scheduled Tasks works fine sometimes, but when you uninstall SP4, it does not work.
The best solution is to:
- Incase Scheduled Tasks do not work after installing SP4, then uninstall SP4 and it should be fine.
- Incase Scheduled Tasks works after installing SP4, and later after uninstalling SP4, it does not work, then install SP4 and it should be fine.
Checking for server or NEOSYS crashes
- NEOSYS Maintenance Mode
- General Menu, Setup, Processes
- Select the dates and the option Detailed
This report shows a list of dates and times that NEOSYS logged in but did not log out properly.
Ignore the very latest entries since they represent the current NEOSYS processes. For example, if you have four NEOSYS processes running at the time that you get the report (including any in maintenance mode) then you can ignore the last four entries.
The date and time shows for each process that has failed to shutdown correctly when the process logged in. Versions of NEOSYS from January 2008 will also show the date and time that each crashed NEOSYS process was last active (heartbeat) so that the time of failure can be known.
If you see a bunch of NEOSYS processes all started up at around the same time but all failed to shutdown correctly then the cause will be a server failure - usually power failure.
Isolated one-off failures will be related to individual NEOSYS process crashes - most commonly caused by one of the following:
- NEOSYS hanging to due to software failure
- Manually exiting a NEOSYS process on the server either by pressing Ctrl+Alt+Del or clicking the "X" close icon/box and ignoring the warning
- Random server failures eg memory, disk etc
Example:
LOGIN 22/12/2007 06:02 NEOSYS SERVER ADAGENCY Current workstation LOGIN 23/12/2007 06:00 NEOSYS SERVER ADAGENCY Current workstation LOGIN 23/12/2007 06:01 NEOSYS SERVER ADAGENCY Current workstation LOGIN 23/12/2007 06:02 NEOSYS SERVER ADAGENCY Current workstation LOGIN 23/12/2007 08:52 NEOSYS SERVER ADAGENCY Current workstation LOGIN 23/12/2007 08:52 NEOSYS SERVER ADAGENCY Current workstation LOGIN 23/12/2007 08:53 NEOSYS SERVER ADAGENCY Current workstation LOGIN 8/1/2008 06:00 NEOSYS SERVER ADAGENCY Current workstation LOGIN 8/1/2008 06:01 NEOSYS SERVER ADAGENCY Current workstation LOGIN 8/1/2008 06:02 NEOSYS SERVER ADAGENCY Current workstation LOGIN 8/1/2008 13:51 NEOSYS SERVER NEOSYS Current user session
Interpretation:
The first four entries indicate that all four NEOSYS processes started at 06am were suddenly killed probably by power failure
The next four entries indicate that NEOSYS was restarted at around 08:52 and all these processes were AGAIN killed probably by power failure
The last four entries can be ignored because there were four NEOSYS processes running at the time that the report was generated
Searching for word/number in the database files using maintenance mode
You can search for any word/number in the database files of NEOSYS, using the following command line:
F5 FIND FILENAME WORDWITHOUTANYSPACES For eg: FIND CURRENCIES 1.1 (here you are searching for the number 1.1 in the currencies file) You CANNOT search for a phrase ie include spaces like this. Or you can also type: FIND FILENAME <enter> and it will ask you what you want do to search. You can enter an exact phrase with spaces.
Incase you do not know the filenames, you can enter the following command to see all the filenames in the system:
F5 LF
Error Message
The following error message appears in Internet Explorer when you try to open up NEOSYS:
Service Unavailable
Solution
Open the IIS Manager, right click Web Sites and select properties:
Switch to the Service Tab and tick the "Run WWW Service in IIS 5.0 Isolation Mode".
You will be asked for Restart of IIS. Click yes to restart IIS. If you are not asked just restart IIS.
Inspecting IIS log files
At a windows command prompt:
c: cd \Windows\system32\LogFiles\W3SVC1
or
%SystemDrive% cd %SystemRoot% cd system32\LogFiles\W3SVC1
then (substituting the ip number you are interested in)
find "192.168.1.55" *|sort>temp.log
Open temp.log in Excel and use Tools, Data, Text to Columns to split into columns using options "Delimited" and check split on Space.
Autowidth all columns by clicking on the top left box just outside the data to the left of column A and above column 1 then double click the column separator to the right of column "A"
Note that times and dates are in UTC/GMT so you have to add/subtract your timezone offset to get local times.
Inspecting Database LOGS Folder
LOGS folder in NEOSYS installation folder carries records of all changes done in the database which includes adding and removal of data.
Log file is an XML document which should be viewed in Notepad. Each XML document represents commands executed by each NEOSYS process started each day.
The NEOSYS log files e.g. 15123103.xml are created by the database processes. If the database is not available according to the website then no entry will appear in that log. The request will appear in the IIS website log but that log is nothing to do with database processes.
Understanding Log Entries
Inspecting and searching through Logs file allows NEOSYS staff to answer clients queries like "Who deleted schedule XXXX" etc.
While going through the log file you may come across a request "EXECUTE GENERAL GETTASKS NOT", this request is concerned with getting a list of tasks that the user is *not* allowed to do.
To read and understand the log file with more ease, copy the portion of the log file required to be analyzed into another text editor.
The text that appears as %FE, %FE1, %FE2, etc. are basically separators.
- First replace FE0%, FE1%, FE2%, etc. with a separator like "--". There can be occurrences of "%FE" along with some valuable information like "17290", so replacing "%FE1" in "%FE17290" would lead to loss of information.
- Then replace %FE with a bigger separator like "===".
- Replace other entries like %FCB and then %FC and so on with similar separators.
- In the end there may be few "FE" remaining, replace these with a separator like "-".
- The remaining "%" characters MUST be intelligently replaced with a separator like " " (space), because it can either be an actual "%" or it can be a residue from "%FE", "%FE1", etc.
Once replacing all these characters is done, the log file will be more easily readable and vital information will be clearer.
In the log file, you may find numbers like 17290, 17195, etc. These numbers denote dates selected or entered by the NEOSYS user. These are basically the number of days from 31st December 1967 till the date chosen by the user. For example, to convert 17290 to actual date, 31/12/1967 + 17290 = 3/5/2015. So the actual date is 3rd May 2015.
To convert these numbers to dates using maintenance mode, refer to the article Finding out when and by whom a record was deleted
Fixing NEOSYS processes that do not auto start / Recovering from incorrect advanced date or time
This solution is applicable to live database processes only. Test database processes don’t auto-start any other processes.
Problem explained
After starting up the 1st process, the rest of the processes don't start up.
Solution
The possible cause for this could be that the system date/time might have been changed - either manually or by the auto synchronization. Do the following checks in the sequence of order:
- Check for any *.$* files (* after dollar sign should show a number, the highest being the latest one). If it shows OK, then proceed ahead.
- Check for any .end files and delete it to rename to .end.temp
- Check the System Event Viewer log for any 520 or 577 error message (refer http://128.175.24.251/forensics/timechange.htm). Also check for any out of sequence / ahead of today date or time.
- In case of no 520 or 577 error message, go to Administrative Tools > Local Security Policy > Local Policies > Audit Policy > Audit Privilege use - make sure that Success and Failure are selected under this (this will ensure that future changes to the date/time are recorded in the System Log).
- In NEOSYS maintenance mode - F5 ED PROCESSES %UPDATE% - and see what it says, incase of any text (only text, not numbers) there, that means that for sure the system date has been changed. To fix this, exit the editor by pressing the ESCAPE key and then type DELETE PROCESSES "%UPDATE%"
Fixing NEOSYS maintenance mode not starting up
Problem :- While trying to start maintenance mode, the maintenance window closes automatically within a split second.
Solution :- Delete the read-only file REVBOOT file (under d:/neosys/neosys folder). REVBOOT file is recreated when you start maintenance mode.
Recognising and Solving Low Memory Problems
Quick Note: Installing a server class operating system on a workstation class computer with the intention of NEOSYS serving a heavy load is likely to cause problems with low memory.
Quick Fix: Disable *ALL* non-essential features in the power-on setup menu.
Effects
It is speculated but not proven that low memory may cause NEOSYS to fail by hanging, causing damaged files etc.
Checking
F5 MEMORY
On server class machines it should say somewhere around 350Kb to 370Kb Free
Some server class machines have around 330Kb and sometimes even less with no reported problems
The actual effect of low memory is supposed to make NEOSYS slower and perhaps cause hanging and damaged files however this has not been proven in an specific case so far.
On workstation class machines it may often say around 280Kb to 300Kb.
Cause
Although there is plenty of real memory in virtually all computers now, NEOSYS runs in the legacy 16 bit virtual memory space of a windows mode called NTVDM. This is limited to 1Mb plus 4Mb of EMS memory.
The 1Mb memory space is shared with:
- Various non-essential windows drivers which NEOSYS automatically disables them in autoexec.nt
- Various plug and play hardware device drivers for the various adapters in the computer like video, network adapters and various other items that NEOSYS is unable to disable.
In a server class computer the hardware device drivers are usually minimally present in the 1Mb base memory and do not therefore DOESNT a low memory situation for NEOSYS.
In workstation class computers there are often many hardware device drivers present in the 1Mb base memory and this DOES causes a low memory situation for NEOSYS.
When NEOSYS is installed on workstation class computers with XP there is usually not a heavy load expectation and therefore the low memory does not cause a problem.
If Windows Server OS is installed on a workstation class computer NEOSYS may well be expected to serve a heavy load with limited amounts of memory.
Workstation class computers: hardware drivers present and EMS is installed in low memory (0000-9FFF) causing low memory for NEOSYS and possible inability to
Server class computers: Usually few hardware drivers are present in high part (A000-FFFF) of the 1Mb base memory and EMS is able to occupy the high memory leaving the low part (0000-9FFFF) of the 1Mb memory free for NEOSYS. You can find out how much memory is available to NEOSYS and whether EMS is occuping high or low memory using the following sections.
Fixing Low Memory
Start, Run, notepad c:\windows\system32\autoexec.nt
Every time NEOSYS starts it tries to make some changes as follow:
- replaces all lines in C:/WINDOWS/SYSTEM32/AUTOEXEC.NT starting with 'lh ' to start with 'rem NEOSYS LH ' instead.
- changes the line in C:/WINDOWS/SYSTEM32/CONFIG.NT "files=..." to "FILES=200"
The replacement is case sensitive triggered on 'lh' and 'files' so if you manually edit the files and remove the rem or change the number of files and leave the LH and FILES in uppercase then NEOSYS will NOT make further changes. This allows you to do manual amendments to the files without NEOSYS overwriting them.
Check that NEOSYS has successfully disabled all the drivers in the lines starting with LH.
They should be commented out (prefixed) with REM or REM NEOSYS as follows.
After making changes reopen NEOSYS in maintenance mode to use the MEMORY and WHO commands again.
REM Install CD ROM extensions REM NEOSYS LH %SystemRoot%\system32\mscdexnt.exe REM Install network redirector (load before dosx.exe) REM NEOSYS LH %SystemRoot%\system32\redir REM Install DPMI support REM NEOSYS LH %SYSTEMROOT%\SYSTEM32\DOSX
Low Memory Issues in Windows 2003 server can be fixed using instructions mentioned at Fixing Low Memory in Windows 2003 Server
Allowing DOS programs that require DOSX to run on the same computer as NEOSYS
The automatic commenting out DOSX by NEOSYS will prevent some other DOS-like programs from running. If NEOSYS is on dedicated server then there should be no other such programs to fail. However, if you must allow DOS-like programs to work as well as NEOSYS you can do the following configuration:
- leave or restore the original AUTOEXEC.NT and CONFIG.NT files where they are
- copy them to another folder eg neosys folder
- make the necessary REM changes there by hand
- right click the NEOSYS\NEOSYS\AREV.PIF and select properties
- change the location of the AUTOEXEC.NT and CONFIG.NT files in the following location
Checking EMS Memory Configuration
Inspection
F5 WHO
press the up arrow to get to the last part/page
Example of Typical Server EMS Memory
Example of Typical Workstation EMS Memory
Correction
No easy way
Removal of hardware adapters designed for workstations instead of servers eg graphics cards and network cards.
Many of the devices may be located on the motherboard and not relocatable except possibly by BIOS configuration or special manufacturer information.
Use windows device manager, View: Resources by Connection, Open the Memory item and look for items between 000A0000 up to 000FFFFF that might give you a clue as to what hardware could be removed or reconfigured. Actually only 000C0000 to 000FFFFF is candidate for EMS memory since 000A000-000BFFFF is mandatory video memory in all systems.
Fixing issue where NEOSYS processes do not start-up at all or start-up and close immediately
- Find if a file with the name global.end exists in the root directory of the NEOSYS installation. Eg D:\global.end . If you find such a file, rename it to global.end.temp - for more information on global.end and what it does refer to Closing NEOSYS Services
- If the above didn't fix the problem and NEOSYS still does not start, do a windows search for the entire NEOSYS folder for *.end (i.e. any file ending with .end extension). You may find a (databasecode).end file in D:\neosys\neosys folder which is created by the NEOSYS program during backup at 1 am and later on removed automatically. In this case NEOSYS program might have crashed during the backup and left this file behind. (databasecode).end files prevent other other NEOSYS processes starting up on the database while exclusive processes (like backup) are being done.
In case the above didn't fix the problem then escalate to the programmer immediately.
Solving "Control Record" error in maintenance mode
If processes dont start after you log into maintenance mode and you get an error message
╔════[FS152]═══════════════════════════════════════╗ ║ The control record "RECORDS" ║ ║ is too long to be saved. ║ ║ The current record length is 65539 characters. ║ ║ ║ ║ < Press any key > ║ ╚══════════════════════════════════════════════════╝
1. Press space to get rid of the error message and you should then get this menu
2. Press F5 and run this command (case sensitive)
DICT DEFINITIONS
or Press Alt+S and run this command (case sensitive)
EXECUTE DICT DEFINITIONS
3. Press Shift+F3 to get the following message
╔══[B202]══════════════════════════════════╗ ║ "DEFINITIONS" has ║ ║ "QUICKDEX" installed ║ ║ ║ ║ Do you want to remove "QUICKDEX"? [Y/N] ║ ║ ║ ║ <Y >║ ╚══════════════════════════════════════════╝
Do you want to remove "QUICKDEX"? [Y/N]
4. Press Enter to choose Yes.
5.Restart NEOSYS to see if the problem has been solved
Solving “page not found” or "HTTP Error 404.3 - Not Found" when downloading some file types after uploading them successfully
A user gets this error message when trying to download a file that has been uploaded into NEOSYS.
Windows web server will not download file types that it is unaware of. You can enable the download of new file types.
Adding Mime Types in Windows Sever 2003
One by One
Follow below steps to enable the download of new file types one by one.
This process is tedious and error-prone if you have to add many types.
- Computer Management
- Services and Applications
- IIS properties
- Mime Types
- Add
Many
Window Server 2003 is unaware of all the Office 2007+ file types. To add all Office 2007+ file types at once do the following:
Stop IIS
iisreset /stop
Open the list of mime types
- Start, Run, notepad C:\WINDOWS\system32\inetsrv\MetaBase.xml
Search the file for “xlsx” and quit the editor if already inserted.
Otherwise, find the following line,
.xml,text/xml
and insert after that line the following lines. They do not need to be indented.
.docm,application/vnd.ms-word.document.macroEnabled.12 .docx,application/vnd.openxmlformats-officedocument.wordprocessingml.document .dotm,application/vnd.ms-word.template.macroEnabled.12 .dotx,application/vnd.openxmlformats-officedocument.wordprocessingml.template .potm,application/vnd.ms-powerpoint.template.macroEnabled.12 .potx,application/vnd.openxmlformats-officedocument.presentationml.template .ppam,application/vnd.ms-powerpoint.addin.macroEnabled.12 .ppsm,application/vnd.ms-powerpoint.slideshow.macroEnabled.12 .ppsx,application/vnd.openxmlformats-officedocument.presentationml.slideshow .pptm,application/vnd.ms-powerpoint.presentation.macroEnabled.12 .pptx,application/vnd.openxmlformats-officedocument.presentationml.presentation .xlam,application/vnd.ms-excel.addin.macroEnabled.12 .xlsb,application/vnd.ms-excel.sheet.binary.macroEnabled.12 .xlsm,application/vnd.ms-excel.sheet.macroEnabled.12 .xlsx,application/vnd.openxmlformats-officedocument.spreadsheetml.sheet .xltm,application/vnd.ms-excel.template.macroEnabled.12 .xltx,application/vnd.openxmlformats-officedocument.spreadsheetml.template
Save the file with File and Exit
Restart IIS
Adding Mime Types in Windows Server 2008
"The page you are requesting cannot be served because of the extension configuration. If the page is a script, add a handler. If the file should be downloaded, add a MIME map."
This message shows when a user attempts to download a file that has been uploaded into NEOSYS and the file extension is not configured in IIS server Mime Maps
Solution:
Add mime mappings in IIS. Use the GUI or the servers command line. The exact solution depends on the file type. Example for .msg files:
C:\windows\System32\inetsrv\appcmd set config /section:staticContent /+"[fileExtension='.msg',mimeType='application/vnd.ms-outlook']"
The bit that varies in the above example is:
- .msg
- application/vnd.ms-outlook
Other mime types can be found on the web eg https://www.thoughtco.com/file-extensions-and-mime-types-3469109
Getting the 2nd part right enables the user to have the downloaded file open automatically in the right application for the file extension, but only if they have the right application installed.
Handling Neosys Automatic Upgrade Error Messages
Error Message
We will receive an email from the NEOSYS client installation with the following message:
"UPGRADEN.EXE" does not have expected exe MZ signature
Solution
This message is a note that the NEOSYS automatic update procedure failed to download an upgrade file correctly. This happens sometimes due to network issues but can be ignored because the file will be probably be correctly downloaded on the next automatic update check.
Note: This message is only informative to help with any problem with the automatic upgrade procedure and may be removed in a later version of neosys (currently Oct 2012)
Testing https connection
The following procedure tests that the https server is operational and not blocked by firewall etc. It does not detect certificate errors.
From a windows command prompt use the following telnet command:
telnet clientname.hosts.neosys.com 4430
- 4430 is the usual NEOSYS http port but replace it by whatever port is actually used for NEOSYS https on the system being tested. If there are multiple https installations on a particular server then different ports are probably used
- Replace clientname.hosts.neosys.com with the normal https login domain name of the client for whom we want to check the https service
Normal behavior is that it should open a black screen. Pressing Enter or any key returns the _ character. (Close the window using the [X] since there is no keyboard command to do so)
Hanging means that there is some connectivity issue (firewall/ip/server not running/https not installed etc)
Troubleshooting "page not found" error while using https
Problem:
- The https service stops working and gives "Page not found" error.
- The https connection tests fine using telnet (see article above)
Solution:
- Reinstall https using the usual procedure (currently using selfssl)
- Email clients, requesting to reinstall the new security certificate by following the instructions for Internet Explorer or Safari browser.
Installing "QUICKDEX" on some files
"Quickdex" is a type of index that keeps an alphabetical order on small files. In some maintenance procedures you may be asked to "install Quickdex".
For example to add a quickdex to the UNITS file.
Maintenance mode press F5
DICT UNITS
г=================Dictionary=================┐ │ │ │ File name UNITS │ │ Field name ......................... │ │ │ │ Dict type │ │ Single/Multi │ │ Data type │ │ Output format │ │ Validation patterns │ │ │ │ Position Key part │ │ │ │ Column heading │ │ │ │ │ │ Justification Display length │ │ Description │ │ │ │ │ L============================================-
Press Shift+F3.
If it says "Do you want to remove ..." then Quickdex has already been installed and you should not continue. Press Esc to cancel, then Esc to quit.
г=[B202]==================================┐ │ │ │ "UNITS" has │ │ "QUICKDEX" installed. │ │ │ │ Do you want to remove "QUICKDEX"? [Y/N] │ │ │ │<Y >│ L=========================================-
Otherwise it should say "Do you want to install ...". Press Enter to accept.
г=[W963]============================┐ │ │ │ Do you want to install a │ │ Quickdex or Rightdex index? [Q/R] │ │ │ │<Q >│ L===================================-
It should then say "Do you want to update ...". Press Enter to accept.
г=[W901]========================================┐ │ │ │ The "UNITS" file has had │ │ the "QUICKDEX" modifying filing system added. │ │ There are 0 records in "UNITS". │ │ │ │ Do you want to update "QUICKDEX"? [Y/N] │ │ │ │<Y >│ L===============================================-
Afterwards it should return to the initial screen. Press Esc to quit.
Solving NEOSYS smtp server failure
In case the neosys smtp server fails then we can just use the client's smtp server.
The following information is required from the client's smtp server configuration:
- hostname
- username
- password
- port no. (most likely = 25)
These details should be entered in the System Configuration File:
Strange characters in maintenance mode
While in Maintenance mode, pressing keys on keyboard gives strange characters. Even Enter and Esc don’t work.
This problem has been seen using RDP on:
- Window Server 2003 Web Edition
- Windows Server 2003 R2 SP2
Solution:
- Go to Windows -> control panel
- Go to Regional and Language Options
- Click on Languages tab then click on Details
- Click on Settings Tab and Change default input language to English (United States)
(or perhaps something else depending on rdp keyboard)
NEOSYS processes do not start after Windows Update
Problem
Scheduled Task to start the NEOSYS processes fail on Servers with Windows 2008 after Windows update. Support will have to log into the server to start the processes manually Message on the Schedule Task displays " The operation being requested was not performed because the user has not logged on to the network. The specified service does not exit ".
Solution
For clients who cannot tolerate manual intervention after server reboots for any reason (e.g.need to start work before NEOSYS support is available or on NEOSYS support weekends), tick "highest privileges" and "run whether the user is logged in or not". This approach means that NEOSYS processes which are started by the windows scheduled task are not visible on the desktop and run hidden in the background and only listed in task manager, so avoid this approach on small clients (only few users). Avoiding this approach will also serve to act as an indicator of server restarts.
Troubleshooting NEOSYS remote support port forwarding
This assumes that you have already “port forwarded” tcp port 19580 from your public internet router to the NEOSYS server.
Tst 0 - Check if SSH is working on the NEOSYS Server
Type in the following command in command prompt:
telnet 127.0.0.1 19580
IF ALL OK you will see the following:
Test 1 - Check if SSH is basically working on NEOSYS server over the LAN
You must know and use the NEOSYS SERVER LAN IP to do this.
telnet ???.???.???.??? 19580
If you have the WRONG SERVER LAN IP or NEOSYS server SSH is NOT working then it will hang for about 15 seconds and then say “Could not open connection to host …”
OR IF ALL OK you will see the following:
Press Enter to Exit
Test 2 - Check if can connect to the NEOSYS ssh from OUTSIDE the office
From any internet connected computer OUTSIDE the office test if you can connect to NEOSYS ssh service. You cannot do this test from inside the office.
You need to know the public ip or domain name of the router. If the router IP is dynamic then NEOSYS sets up dynamic name server so instead of a static ip number you will have a domain name something like clientxyz.redirectme.net.
Enter the command .. using YOUR public internet IP number (NOT the LAN ip number) or the dynamic domain name.
If everything is working OK you will get a black screen as follows. You will NOT get the “SSH-2.0-OpenSSH-4.7” banner” because NEOSYS ssh remote support is restricted to connect ONLY from LAN ip nos and NEOSYS office ip nos.
If you press Enter a few times then the cursor will just go down. You have click the [X] to close the window.
Test 3 - Check that the ssh connection from step 2 was rejected
On the NEOSYS server, check the Windows Application log to verify that an SSH connection was rejected.
The rejected ip number will be of the system outside the office that you performed the test from.
Sample Email: Solving port 19580 port forwarding issues
Some IT people know how to troubleshoot port forwarding issues but others are mostly just power users who can configure a home router. If the IT person is in the second category then it is quicker for NEOSYS support to offer to configure their router from the NEOSYS server using Teamviewer. Ideally NEOSYS should not be doing client IT work but if client IT allows NEOSYS access to their router then NEOSYS support can make a brief attempt to do the configuration. If the issue is still unresolved then request the client to get a professional IT network expert to do the job and inform them that NEOSYS will not be able to provide them any support till connectivity is fixed. Below is a letter advising a more skilled person to check connections using telnet which is a low level test.
You can usually determine the NEOSYS server LAN IP number from Nagios. If so then adjust the email text appropriately.
Dear XYZ, It is highly critical to fix the connectivity with the NEOSYS server because NEOSYS will not be able to provide any support until connectivity is fixed. User support issues will be delayed and remain unresolved if you do not fix this issue URGENTLY. At the moment there is no connection from the internet via your router to the NEOSYS server and when we do the following, we get no connection. telnet CLIENTNAME.hosts.neosys.com 19580 Normally it should connect and presents a black screen (saying SSH something after pressing Enter) which we close. Please check that you can connect to the NEOSYS server internally by using the NEOSYS server IP address in the following command on any computer in your LAN. telnet 192.168.?.? 19580 If you can connect to the NEOSYS server internally then please check port forwarding. Additionally check the following: 1. Has your router IP changed? The IP we have is x.x.x.x 2. Has your server’s LAN IP changed and are you forwarding to the correct LAN IP? 3. Is the configuration really correct? 4. Check router logs for clues 5. Check NAT settings in the router If you still cannot see the problem, do "telnet CLIENTNAME.hosts.neosys.com 19580" command from OUTSIDE your network to replicate the problem we are facing. For troubleshooting steps refer Troubleshooting NEOSYS remote support http://techwiki.neosys.com/index.php/Troubleshooting_NEOSYS_Generally#Troubleshooting_NEOSYS_remote_support_port_forwarding If the problem still persists, please get a professional IT network expert to fix the issue. Best Regards,
Port mapping restricted by Source IP
On NEOSYS router, port forwarding has been setup only for specific source IP addresses. This means you will not be able to establish a TCP connection to NEOSYS server unless your server's outbound IP is mapped to the NEOSYS router. In other words unless the outbound i.e source IP/port of your server has been granted access on the NEOSYS router for all incoming connections you will not be able to make connections to NEOSYS server.
Outbound IP is used whenever a server tries to make a connection to another server outside its network. On the other hand a server receives all incoming connections using it's Inbound IP.
Therefore to setup ssh connection from a new Client server to NEOSYS server we need to grant access to its outbound IP on the NEOSYS router. See link to find the outbound server IP/ports of the server.
Solving “Cant login … INVALID DATA PATH … permission denied”
Error Message
Error Text:
Cannot login because: ERROR: INVALID DATA PATH “D:\HOSTS\HOSTNAME\DATA\HOSTNAME\~8746345.1$” Permission Denied
Problem Explained
When installing NEOSYS on an existing “non-clean” Windows installation, the standard NEOSYS installation procedure can result in failure to login if the standard windows folder permissions have been modified.
Solution
The solution is to grant IIS permission to write in the \neosys\DATA folder and subfolders as follows:
- First add the IUSR_XXXXXXX user to the list of users. (XXXXXXXX is the server name and therefore varies per server) as follows:
- Right Click on DATA Folder and click on Properties
- Click on Security Tab -> Add -> Advanced
- Click on Find Now, Select the IUSR_XXXXXXX user and Click on OK
- Second, for the newly added IUSR (IIS user) change the permissions as follows:
- REMOVE the read and execute permission (for security, IIS should be unable to execute things that it might have uploaded)
- ADD the write permission
- Login should now be possible.
NEOSYS process window displays message "Upgrade Downloading"
Problem Explained
NEOSYS thinks it sees an new neosys2.exe upgrade file on the location http://www.neosys.com/support/neosys2.exe which is accessed by http so attempts to download it.
Http proxies and various internet issues can cause incorrect info to be sent and there is actually no upgrade available. In this case, eventually it realizes that it cant find an appropriate and it stops.
You don’t have to worry about this case.
Enabling EMS memory on Window 2003
Normally EMS memory is provided by Windows 2003 but this can vary depending on the server hardware/bios configuration
If you get the following messages on Windows 2003
- Backup File Size is 0
- RTP27. [B28] Not enough String Space – Out of Memory
Cause
On servers that had no problem previously, the problem is caused by a windows update in Oct 2012 that disables standard Windows EMS memory.
The patch is issued by Microsoft on 9 Oct 2012 but the installation date in the server depends on when the update was actually installed.
http://support.microsoft.com/kb/2724197
Solution 1 - maximum performance
To re-enable standard windows EMS on older slower servers or servers where NEOSYS performance must be maximized.
The following link contains instructions how to remove the offending windows update
It also shows how to prevent it being reinstalled automatically by Windows.
http://www.columbia.edu/~em36/wpdos/emsxp.html
Don't forget to prevent it being reinstalled again automatically
Solution 2 – ease of installation
This option can also be used if Window 2003 is unable to provide EMS memory for example when the server hardware/bios configuration prevents it.
Install EMSMAGIC in the same way as for Windows server 2008
EMSMAGIC has higher memory consumption and makes NEOSYS processes slower so it is better to use Solution 1 above if NEOSYS performance is an issue.
Fixing no output file in XXX YYY Issue
Error Message
Problem Explained
The message “No output file in XXX YYY” can appear at several instances in NEOSYS, most often when generating reports or documents.
This problem is usually caused by software error and it indicates that the NEOSYS server responded without any output and without any message.
Solution
1) Find proof to check if the data required for the report actually exists. This way we can eliminate lack of data as a cause for this error.
2) Check to see if a similar issue has been fixed in latest version of NEOSYS.
3) Document HOW and WHERE the problem can be duplicated by NEOSYS programmers to identify and correct the software.
An example of this error can be found at No Output File in MEDIAPROXY SCHEDULEPRINT
Troubleshooting Internet Connections
Cannot Connect
While investigating as to why users are not able to access NEOSYS or http://www.neosys.com/ , you can check which ISP the connection issue is on.
whois ipno
Inspect very carefully to get clues as to which ISP and which AREA of the ISP the problematic ip numbers are and which do NOT have problems
Doing tracert on windows command prompt on the user's computer may help locate which point on the route between the user and the server is blocking access
tracert xxxx.hosts.neosys.com
where "xxxx" is the client name. In this example, the output will be something like shown below:
Tracing route to xxxx.hosts.neosys.com [37.48.81.101] over a maximum of 30 hops: 1 2 ms 1 ms 3 ms ukr.sb.com [192.168.2.1] 2 10 ms 10 ms 11 ms losubs.subs.bng2.th-lon.zen.net.uk [62.3.80.21] 3 12 ms 10 ms 67 ms ae1-182.cr1.th-lon.zen.net.uk [62.3.86.80] 4 10 ms 11 ms 11 ms ae0-0.br2.th-lon.zen.net.uk [62.3.80.42] 5 13 ms 14 ms 14 ms peering.thn.lon.leaseweb.net [195.66.225.56] 6 23 ms 23 ms 24 ms te-0-10-0-19.bb01.ams-01.leaseweb.net [31.31.32.71] 7 22 ms 22 ms 23 ms xe-11-2-3.br01.ams-01.nl.leaseweb.net [31.31.38.89] 8 25 ms 28 ms 26 ms be-10.cr02.ams-01.nl.leaseweb.net [81.17.34.21] 9 24 ms 20 ms 25 ms po-1002.ce02.ams-01.nl.leaseweb.net [37.48.95.195] 10 24 ms 22 ms 22 ms nl10r.neosys.com [37.48.81.101]
Troubleshooting TCP/IP Connections
Telnet check
telnet <hostname> 19580
If Telnet does not work
In case telnet does not work, login to the remote host server to investigate the issue. Run the following command simultaneously while doing Telnet from client server to the remote host to check if the TCP packets are reaching the Remote server.
tcpdump -v 'src host client-domain-name/ip' tcpdump -v portno
To check if packets are sent from the client server to the remote host, you can run the following command simultaneously while trying to ssh to the remote server.
netstat -an
You can also check if the outbound ports are open from which you are trying to establish the TCP connection to the remote server.
telnet portquiz.net portno
The outbound IP addresses at times can be different from the public IP of the Client server so be sure that the public IP of the client server is the same as its source IP (which represents an incoming connection from Client to Remote server). One way to find the source IP of the Client server is sending an email from Client server to "support@neosys.com". On receiving the email in Thunderbird, select the email and press Ctrl+u. A new window Opens giving full details of the email received. The third "Received :from" gives the IP of the source.
Troubleshooting DNS failure
NEOSYS clients routers are usually configured to use their ISP DNS service and the ISP DNS service is supposed to contact one of NEOSYS's DNS servers to convert server names like hosts.neosys.com into IP numbers. Misconfiguration of clients routers or problems in the ISP DNS server may cause CANNOT CONNECT problems. Often the connect fails quickly and immediately since if a name cannot be converted to an ip number then the connection cannot even be attempted and therefore there is little or no timeout to wait through.
NEOSYS.COM name servers are listed publically and obtained by whois command.
whois neosys.com
Name Server: DNS1.EASYDNS.COM Name Server: DNS2.EASYDNS.NET Name Server: DNS3.EASYDNS.ORG Name Server: NS12.ZONEEDIT.COM Name Server: NS18.ZONEEDIT.COM
In order to contact NEOSYS DNS servers the ISP's have to use a global DNS to obtain the ip addresses of NEOSYS DNS servers given the host names of NEOSYS DNS servers given in the whois info
Here is an example of DU testing NEOSYS DNS servers. The NEOSYS DNS server ip addresses are listed in the Destination column.
If one DNS server is down or unreachable REGARDLESS OF REASON, the ISP is supposed to use the other DNS servers. It is impossible for all NEOSYS DNS servers to be unreachable except in gross disconnection from the internet of the ISP since it is effectively impossible that all NEOSYS DNS servers which are carefully spread around the internet, to be unreachable.
In the above test one of the NEOSYS DNS servers is unreachable but all the others are reachable therefore DU should have no problem providing DNS service to its clients.
ISP are often worse at providing DNS server than the famous GOOGLE DNS servers, so re configuring client router to use GOOGLE DNS servers is a way to prove that the problem lies with the ISP's DNS service
Additional test for troubleshooting problems with uploading
Verifying that upload.dll can run
This isnt a complete test of everything. It just checks if the upload program can be run by the web server. It doesnt check if uploads work or the image directory is correctly configured with the right permissions and uploads can actually be done.
Error Message
... to be added when discovered ...
Test
Test HTTP if accessible by LAN; Test HTTPS is accessible by Internet; Test both if both are available.
On the server type the following into a browser
LAN/HTTP:
http://localhost/neosys/neosys/dll/upload.dll
WAN/HTTPS:
https://localhost:9999/neosys/dll/upload.dll
Expected Result
Upload Error. ! Please call me from a form !!! The first param must be Filename= name of the uploaded file, TYPE=TEXT The second param must be Filedata= uploaded file, TYPE=FILE The third param is optional PathData= path to uploaded file, default c:\temp\, TYPE=HIDDEN The forth param is optional RedirectPage= name of asp who receive the results, TYPE=HIDDEN Add others params at the end with INPUT tag.
Patching a NEOSYS program
The patch provided will tell you the program name and contain either a whole replacement program text or just some changed lines which you will have to find and edit.
You will need to know the file name which may be provided along with the patch or you can find it using the following code in maintenance mode:
ED VOC programname
ED VOC XYZ tells you what program name is executed and from what file, when you type the command XYZ. Normally the program name is the same as the command.
The file name will normally be BP for agency programs, GBP for general programs or ABP for finance programs.
NEOSYS programs are stored in files just like records of ordinary database files. You can edit either with "ED filename programname" or "TED filename programname". TED is better for editing source code as it opens the the code in a text editor, whereas ED opens the code in the same maintenance window.
Installing patch in live database
In the rare case that the programmer asks Support to install the patch directly on LIVE dataset, start by typing the following command in maintenance mode before commencing:
UTIL
Follow the instructions mentioned in the next section, but skip the instruction to TEST the patched program in test dataset.
The instruction to INSTALL the patched program MUST be followed BEFORE testing the patch in live dataset. Otherwise the changes will not get reflected.
Installing patch in test database
1. EDIT the program source code.
TED programname
If you have to edit or patch a program that starts with the word DICT. and the remainder of the program name is the same as a real file name e.g DICT.INVOICES, then you cannot omit and must type the actual source file name - in this case, "BP".
Otherwise, if you just type "ED DICT.INVOICES" hoping to edit the DICT.INVOICES program in the BP file, then you will end up editing the dictionary of INVOICES, which is not what you are trying to do. Use the command below to edit such type of programs:
ED BP DICT.INVOICES
Next either cut and paste to modify the whole program or edit the program text according to the patch/instructions provided.
Save and close the program source code.
2. COMPILE it. If you get errors then check your edits are correct and recompile otherwise return the patch to programming.
COMPILE filename programname
3. TEST it. Ensure the patched program now works in TEST database.
4. INSTALL the patched program in the LIVE database by typing one of the following command depending on the filename that you patched.
COPYGBP programname COPYABP programname COPYBP programname
Patching NEOSYS dictionaries
A patch to a dictionary applies immediately when you save it and to all datasets regardless of which dataset you work in.
You need:
- the file name eg INVOICES
- the item name (column name) eg DATETIME_AMENDED
- 10 lines of data similar to the example below.
- if the item is an S type dictionary, then either the whole, or only the amended part, of the source code of the dictionary.
ED DICT INVOICES DATETIME_AMENDED
TYPE everything below exactly on the corresponding line numbers except lines 3 and very commonly line 8, as these lines may contain multiple values separated by a superscript 2 (²). Line numbers in the below screen are only for illustrative purpose.
╔═══════════════════════════┤DATETIME_AMENDED├═════════════════════════╗ 1║S ║ 2║ ║ 3║DateTime²Amended ║ 4║S ║ 5║ ║ 6║ ║ 7║[DATETIME] ║ 8║updated=@record<28>²created=@record<31,1>²@ans=''²if created and num(c║..actually this line continues off the screen to the right 9║R ║ 10║10 ║ 11║ ║ : : : : ╚══════════════════════════════════════════════════════════════════════╝
Press Ctrl+E on lines 3 or 8 in order to enter sub-lines.
The multiple values (on line 3 and 8) separated by a superscript 2 (²) automatically appear when sub-lines are entered.
For line 8, you will need to cut and paste the lines of below program to the Ctrl+E screen:
╔═════════════════════════┤Field 8 of DATETIME_AMENDED├════════════════════════╗ ║updated=@record<28> ║ ║created=@record<31,1> ║ ║@ans='' ║ ║if created and num(created) and num(updated) then ║ ║ createdsecs=field(created,'.',1)*86400+field(created,'.',2) ║ ║ updatedsecs=field(updated,'.',1)*86400+field(updated,'.',2) ║ ║ if abs(updatedsecs-createdsecs)>120 then ║ ║ @ans=updated ║ ║ end ║ ║ end ║ ║ ║ ║ ║ ╚══════════════════════════════════════════════════════════════════════════════╝
After you save and exit the Ctrl+E screen, you will see the sublines separated by superscript 2 appear in one line in the ED screen.
Similarly, for line 3, you need to enter its sub-lines in Ctrl+E screen.
To ensure indenting remains nice (although indenting is not important and can be messed up without causing any problem) then copy the double bars at the front of the lines (if available, otherwise insert some char in position 1 of each line perhaps) and then remove them after you paste ... or just edit until the indenting is correct if you really want to.
Press F9 and/or Esc to save and/or exit from Ctrl+E screen
Press F9 and/or Esc to save and/or exit from ED
Searching for strings in files using grep
Use the grep command in Cygwin terminal to search files especially when you are investigating the NEOSYS Logs.
Use the following command to search for a string in any file or directory
grep string path/file
grep -i -a string path/file (use -i to ignore upper/lower case characters in the string and -a to treat the file type=text and display the matching text)
where "string" is the text to be searched, "file" is the type of file you are looking for and "path" is the path of the directory you are looking into. Use "" when having spaces in your string.
e.g when searching in NEOSYS logs the year/month/date is specified in the file name, so if you are looking for a file in year 2016 in the month of Feb, use
grep XXXX path/1602*
In the above command *(asterisk) is a wildcard used to search all files whose file names with "1602".
Sample below of grep command and its output where it is searching for "Dior" in 2016 march logs.
$ grep -i Dior /cygdrive/d/hosts/test/logs/test/2016/1603* Binary file path/16030301.XML matches
Using *(asterisk), a string can also be searched globally across all installations on the server.
Below example will search all files whose file names begin with "NEOS00", in all client installation folders inside the "hosts" folder, for log entries containing text "5th June 2016".
$ grep -a "2016 JUN 05" /cygdrive/d/hosts/*/logs/NEOS00*|less
Use the commands below to display the search string and required number of lines that come either after or before the search string, depending on what you enter in your command. It helps to get more information from files especially when you only know few words and the other information around the searched string also gets displayed.
grep -A NUM string file (NUM lines after the searched text get displayed)
grep -B NUM string file (NUM lines before the searched text get displayed)
See the examples and their respective outputs below, (Note: there is no space between -A and NUM):
$ grep -A2 -i "Dior" 1603* Binary file 16022101.XML matches Binary file 16022901.XML matches
$ grep -A2 -a "Dior" 1603* 16030301.XML:<DataOut>DIOR%FEPOI%FEDior Poison%FE%FE%FE%FE%FEDubai, UAE%FE%FE%FE%FEN%FE%FE%FE%FE%FE%FE%FE%FE%FE%FE%FEN%FE%FE%FE17584.43592%FE%FE%FE%FE%FE%FE%FE%FE%FE%FE%FE%FE%FE%FE%FE1%FE%FE%FE%FE%FE%FE%FE%FE%FENEOSYS%FE17584.43592%FE94_200_49_146%FE1%FE%FE%FE%FE%FE%FE%FE%FE%FE%FE%FE%FEpaulson</DataOut></Message> -- 16030301.XML:<DataOut>DIOR%FEPOI%FEDior Poison%FE%FE%FE%FE%FEDubai, UAE%FE%FE%FE%FEN%FE%FE%FE%FE%FE%FE%FE%FE%FE%FE%FEN%FE%FE%FE17584.43592%FE%FE%FE%FE%FE%FE%FE%FE%FE%FE%FE%FE%FE%FE%FE1%FE%FE%FE%FE%FE%FE%FE%FE%FENEOSYS%FE17584.43592%FE94_200_49_146%FE1%FE%FE%FE%FE%FE%FE%FE%FE%FE%FE%FE%FEpaulson</DataOut></Message>
Some more examples of grep below:
Use the command below when searching for more than one string
grep -a string1 *|grep string2
Use the below command to omit the lines containing specific strings in your search:
grep -B2 -a Processing 160329*|grep -v DataIn|grep -v Message|less
where user wants the output to show lines containing string "Processing" and exclude lines containing string "DataIn" and "Message", |less displays the output in a new screen.
Use "zgrep" command to search in zip files:
zgrep string1 /path |zgrep string2 |less
Managing the queue of reports being delivered by email
In maintenance mode.
Listing
LIST DOCUMENTS WITH SCHEDULED_ONCE
Clearing
SELECT DOCUMENTS WITH SCHEDULED_ONCE DELETE DOCUMENTS
Removing non essential files on win3
Non essential files getting accumulated over time can take up a lot of space on the hard disk resulting in low free space on the server. NEOSYS Client Monitoring system alerts about low free space for win3 server as it monitors the Disk Space of both C & D drive. To fix it Support team will have to create free space on the server.
Making space on the server is time taking and requires patience as you need to go through all the files/folders looking for non-essential files. Do not make any assumptions for not looking into a folder.
Non-essential files are the ones without which we can work properly and will continue to work in future.
Support team should keep in mind the below points while creating space on the server:
- Look into the drive for which nagios is alerting and dig into all the folders.
- Right click Folder > Properties > Size on Disk, to find out the size of a folder.
- On d drive majority of the space is taken by d:\hosts and d:\data.bak folder and on c drive it's taken up by the important windows/cygwin folders.
- The space distribution will give you an idea about which folders to target that can free up a good amount of space on clean up.
- Following are the non essential files/folders which can create a lot of space on removal. Use your intelligence and presence of mind while deleting files/folders because once deleted the information is lost forever.
- Folder: d\data.bak. Check for stopped clients' backups and delete if present.
- Folder: d\hosts\clientname\logs (where clientname stands for all the client folders in hosts) Look into logs prior to the current year and the year before and delete them. In the recent versions of NEOSYS, log folders are compressed and take up a lot less space on disk than presented as their size. You can see the difference in their properties.
- Folder: d\hosts\clientname\downloads. This folder might contain lot of old versions of neosys.exe files. Keep the two latest versions and delete the rest. Do this for all client folders except test installation because Support maintains all the old versions of neosys.exe in it.
- Folder: d\hosts\old. In this folder, delete all client folders that are older than 1 year.
- Apart from above folders look for random backup.zip files present in c/d drive. This happens when Support restores data from a zip file and forgets to delete it after the restore.
Troubleshooting Scripting Disabled error message on browser
Error: NEOSYS requires You have
1. Internet Explorer 6+
or Safari 3.1+
or Firefox 3+
or Chrome 8.0+
2. Scripting enabled Scripting disabled
3. Cookies enabled Unknown
Follow steps in given link to fix Script disable error on browsers : https://wmich.edu/helpdesk/internetenablecookies
