Troubleshooting NEOSYS Generally

Solving failure to start a NEOSYS server due to disk failure message

Problem

During a reboot process (which maybe due to a Windows update or even done by a support personnel) the NEOSYS server gets hung on the startup and shows a message "Boot Failure - Abort, Retry".

Solution

This typically happens due to the USB being plugged into the server and the boot sequence being wrong - i.e. the server trying to boot from the USB first and fails. The immediate solution would be to unplug the USB and ask the client to reboot the server again and upon successfully rebooting the system, plug the USB back again. However this is a temporary solution and the problem would crop up again very soon, hence you need to immediately talk to the IT Administrator of the client and ask them to rectify the boot sequence to make it boot first from the CD ROM, then the HDD and last the USB.

Solving "Read error in the operating system file"

Problem

Nagios reports a hung process and on the server a process has the following message popup.

Read error in the operating system file "..\DATA\PT0833\ACCOUNTS\REV20049
The file does not exist or the filename is
not valid for the operating system.

Cause

Some non-NEOSYS program is directly accessing the NEOSYS database files while NEOSYS processes are running and using the files as well.

• Client IT staff using a backup program to perform backup without ensuring that NEOSYS processes are shutdown.
• A NEOSYS rsync process taking longer than expected due to new USB.

Solution

Kill the NEOSYS process eg with the X button. It is advisable NOT to let it run further while it cannot properly access one of its files.

Prevention

Remove the third party program or arrange for it to operate only while NEOSYS processes are shutdown eg from 3am-6am.

Handling damaged files

Handling damaged files

Checking for corrupt database files

Login to NEOSYS Maintenance

Press F5

CHK.FILES

or

CHK.FILES filename

Assessing Database Size per File

In NEOSYS maintenance mode

LIST FILES BY BY-DSND SIZE VOLUME.NAME SIZE

This will include all files not just those in the actual dataset in DATA\* folders.

Resolving slow speed

Benchmarking NEOSYS

Handling failure and warning on nightly backup alerts

Handling failure and warning messages on nightly backup alerts

Resolving permissions errors while logging in

Problem

While logging in, you get the following error message:

Solution

Add the internet guest account to the security list of the data folder with the default permission of list/read/write

Make sure the read&execute permission is removed

Resolving the 'HTTP Error 500.0 - Internal Server Error' while logging in on IE on a Windows Vista system

Problem

After configuring IIS on Windows Vista you will get this error message while trying to login into NEOSYS from Internet Explorer:

HTTP Error 500.0 - Internal Server Error
Description: This application is running in an application pool that uses the Integrated .NET  
mode. This is the preferred mode for running ASP.NET applications on the current and future 
version of IIS.

In this mode, the application using client impersonation configured with <identity 
impersonate="true" /> may not behave correctly. Client impersonation is not available in early 
ASP.NET request processing stages and may lead modules in those stages to execute with process 
identity instead.

Solution

You can move the application to an application pool that uses the Classic .NET mode by using the following from a command line window (the window must be running as Administrator)

%systemroot%\system32\inetsrv\APPCMD.EXE set app "Default Web Site/neosys" /applicationPool:"Classic .NET AppPool"

Alternatively, you can use any other application pool on your system that is running in the Classic .NET mode. You can also use the IIS Administration tool to move this application to another application pool.

Resolving the 'Class Not Registered' error message while logging in

Problem

While logging into NEOSYS, you will get a popup window giving an error message saying 'Class Not Registered - Server Error'. Typically, you will encounter this error with XP Pro IIS 5.1. As usual, there's way to solve it, however the root cause of this is still unknown.

Anyway, you will get the proper message in the event log:

Event Type: Warning
Event Source: W3SVC
Event Category: None
Event ID: 36
Description: The server failed to load application '/LM/W3SVC/1/ROOT/NEOSYS.
The error was 'Class not registered'. 

Solution

So, what do you do ? This problem is related to Component Services, and when you open Component Services MMC, you will most probably get Error Code 8004E00F COM + was unable to talk to Microsoft Distributed Transaction Coordinator. So, fix the COM+ services first by using the following KB from Microsoft (PRB: Cannot Expand "My Computer" in Component Services MMC Snap-In http://support.microsoft.com/?id=301919):

To resolve this problem, reinstall Component Services as follows: WARNING:

1. Open registry editor, locate HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\COM3, and then delete this key.
2. From the Start menu, point to Settings, click Control Panel, and then click Add/Remove Programs.
3. Click Add/Remove Windows Components.
4. Proceed through the wizard, and accept all of the defaults (including IIS)
5. Restart the computer.

If the above didn't solve it, and you still receive the 'Class not registered' error message, then you need to recreate the IIS packages in COM+, try

1. Delete IIS related package in Component Services MMC
2. IIS In-Process Applications
3. IIS Out-of-Process Pooled Applications
4. IIS Utilities

Next, if you still get the message, try following before re-install IIS if you can't find Distributed Transaction Coordinator in your Services console.

Launch command prompt and run the following command.

1. msdtc -install
2. net start msdtc

Then try re-install IIS.

This should solve the problem

Enabling File Security option on Win XP Professional

Problem

In the Properties of any folders, the Security option does not show, hence you cannot modify the Read, Write options.

Solution

The solution would be to untick the 'Simple File Sharing' option from Tools > Folder Options > View:

B10 & B12 Errors

These errors are in the software and should be immediately escalated to the programmer.

Internet Explorer Menu, View, Text Size doesnt change font size as expected

Cause: This is because the font size is now user definable in NEOSYS and View, Text Size does not override predefined font sizes.

Solution: If you are using Internet Explorer 7 you can scale the screen (including the font size using ctrl + and ctrl - keyboard shortcuts or the font size button on the bottom right hand side of the window.

You can adjust the font size on the User Details form when you login although this permanently applies to all forms not just the one that you are on.

Error on process window "Not enough string space - Out of Memory"

Message

‘RTP27’ Line 1. [B29] Not enough string space – Out of Memory.

Not enough memory to execute Debugger; current program aborted. Press any key to continue

Cause

Unknown perhaps related to some large document or report

Solution

Ask users for any hanging transactions. None other than closing and starting another process

Uploaded jpg files fail to display in internet explorer

Some large jpg files > 2Mb cannot be viewed in internet explorer despite being viewable in image preview, ms paint and other viewers/editors. It is not an issue caused by uploading or downloading the files.

These file appear to have been created on Photoshop CS Macintosh and may be a special type of uncompressed jpg used for production quality files.

Partial solution

Before uploading the files, open them in some editor like MS Paint (right click, edit) and save them. However this results in a loss of quality. Perhaps there is some program that can convert these files to a format understandable by Internet explorer without any loss of quality.

"You have attempted to write to a read-only file"

Message

Error while writing data. You have attempted to write to a read-only file. - or - access to the file has been denied by the operating system. (operating system file name: "..\DATA\ADLINEC\ADAGENCY\REV76467.OV00012618")

Cause

It is almost certainly due to some third party backup or other maintenance software opening the NEOSYS database files when it shouldn't e.g. badly configured third party backup scheduled to backup NEOSYS while NEOSYS is still running. Note that the exact filename varies each time.

Solution

This can be a serious error that causes damaged files in NEOSYS especially if the filename ends in .OV. Use the usual methods of checking for damaged files e.g. do a backup which also looks for damaged files BUT DO NOT OVERWRITE THE LATEST BACKUP SINCE IT MAY BE REQUIRED for restoration. Then fix the damaged files using the usual methods e.g. by rebuilding/using FIXFILE or restoring databases.

Prevention

Remove the third party backup or other maintenance software or reschedule it to run at a time that NEOSYS is shutdown. Removal of software may require hunting through the windows process list for unexpected programs running.

Resolving "Cannot backup/restore because PROCESS1 PROCESS2 (etc) is/are online" message

This can happen for a variety of reasons if all the NEOSYS processes fail to close down at backup time

1. NEOSYS maintenance windows are left open.
2. NEOSYS processes hang due to software error.

How to kill hung NEOSYS processes

NOTE WELL: If you kill actively working NEOSYS processes (those which are "listening" and not hung/crashed) there is a reasonable chance that the database will be damaged and might need a restore losing possibly large amounts of work.

If NEOSYS processes are visible on the server desktop

Look for processes which don't have "Listening ..." on the last but one line. The times on the left hand side are frozen as at the time of the hang.

You can then click the X to kill the process and confirm that this is OK.

Example of a NEOSYS process that has hung due to a software error resulting in a failure to handle a complex query with a lot of brand codes.

If NEOSYS processes are not visible on the server desktop

You can check if there are any hung processes from the NEOSYS Support Menu, List of Database Processes.

1. Shutdown NEOSYS by TEMPORARILY putting a file called GLOBAL.END in the parent directory of neosys. Leaving the file there would prevent NEOSYS from starting up again. Shutting down NEOSYS from the Support menu will not work because of the hung processes.
2. Use Windows Task Manager to kill all the NTVDM tasks. Don't kill active NEOSYS processes.

Temporary workarounds for hung NEOSYS processes

Until the error in the software is fixed users can often get their results by simplifying their requirements. For example select individual clients instead of selecting all the brands for a particular client. If the user has repeated his request (in forlorn hope that it would work finally) then the number of NEOSYS working NEOSYS processes will drop causing severe slowdown for other users and complete stop if all the NEOSYS processes hang.

How to fix a NEOSYS process which opens up in a notepad instead of Microsoft Windows Based Script Host (i.e. DOS window)

The NEOSYS process (cmd file) might open up in a notepad, instead of the usual black colour DOS window. This can be fixed by the following:

1. Download and install the Windows Script 5.6/5.7 from the Microsoft Website
2. Go to any folder, click on Tools > Folder Options > File Types and find the .JS and .JSE and change the default program to wscript.exe (from windows>system32)

Resolving a 'Could not start' error on Scheduled Tasks in Windows Server 2000 SP4

This error occurs because of a change that is made to the data that is stored in the credentials database when you install Windows 2000 SP4. Hence installing SP4 causes the the data that is stored in the credentials database to get converted to an SP4-compatible format. A registry key is configured to indicate that the data has been converted to the SP4 format.

Hence the Scheduled Tasks do not work sometimes. However the Scheduled Tasks works fine sometimes, but when you uninstall SP4, it does not work.

The best solution is to:

1. Incase Scheduled Tasks do not work after installing SP4, then uninstall SP4 and it should be fine.
2. Incase Scheduled Tasks works after installing SP4, and later after uninstalling SP4, it does not work, then install SP4 and it should be fine.

Checking for server or NEOSYS crashes

1. NEOSYS Maintenance Mode
2. General Menu, Setup, Processes
3. Select the dates and the option Detailed

This report shows a list of dates and times that NEOSYS logged in but did not log out properly.

Ignore the very latest entries since they represent the current NEOSYS processes. For example, if you have four NEOSYS processes running at the time that you get the report (including any in maintenance mode) then you can ignore the last four entries.

The date and time shows for each process that has failed to shutdown correctly when the process logged in. Versions of NEOSYS from January 2008 will also show the date and time that each crashed NEOSYS process was last active (heartbeat) so that the time of failure can be known.

If you see a bunch of NEOSYS processes all started up at around the same time but all failed to shutdown correctly then the cause will be a server failure - usually power failure.

Isolated one-off failures will be related to individual NEOSYS process crashes - most commonly caused by one of the following:

1. NEOSYS hanging to due to software failure
2. Manually exiting a NEOSYS process on the server either by pressing Ctrl+Alt+Del or clicking the "X" close icon/box and ignoring the warning
3. Random server failures eg memory, disk etc

Example:

LOGIN 22/12/2007 06:02 NEOSYS SERVER ADAGENCY         Current workstation 
LOGIN 23/12/2007 06:00 NEOSYS SERVER ADAGENCY         Current workstation  
LOGIN 23/12/2007 06:01 NEOSYS SERVER ADAGENCY         Current workstation  
LOGIN 23/12/2007 06:02 NEOSYS SERVER ADAGENCY         Current workstation  
LOGIN 23/12/2007 08:52 NEOSYS SERVER ADAGENCY         Current workstation  
LOGIN 23/12/2007 08:52 NEOSYS SERVER ADAGENCY         Current workstation  
LOGIN 23/12/2007 08:53 NEOSYS SERVER ADAGENCY         Current workstation  
LOGIN 8/1/2008 06:00 NEOSYS SERVER ADAGENCY         Current workstation  
LOGIN 8/1/2008 06:01 NEOSYS SERVER ADAGENCY         Current workstation  
LOGIN 8/1/2008 06:02 NEOSYS SERVER ADAGENCY         Current workstation  
LOGIN 8/1/2008 13:51 NEOSYS SERVER NEOSYS         Current user session  

Interpretation:

The first four entries indicate that all four NEOSYS processes started at 06am were suddenly killed probably by power failure

The next four entries indicate that NEOSYS was restarted at around 08:52 and all these processes were AGAIN killed probably by power failure

The last four entries can be ignored because there were four NEOSYS processes running at the time that the report was generated

Searching for word/number in the database files using maintenance mode

You can search for any word/number in the database files of NEOSYS, using the following command line:

F5
FIND FILENAME WORDWITHOUTANYSPACES

For eg:
FIND CURRENCIES 1.1
(here you are searching for the number 1.1 in the currencies file) You CANNOT search for a phrase ie include spaces like this.

Or you can also type:
FIND FILENAME <enter>
and it will ask you what you want do to search. You can enter an exact phrase with spaces.

Incase you do not know the filenames, you can enter the following command to see all the filenames in the system:

F5
LF

NO LOCK RECORD error when working on NEOSYS

Error Message

The following error message appears:

Cannot save "DOCUMENT NAME" because:
Somebody has updated this record.
Your update cannot be applied.
This session id does not agree "NO LOCK PERIOD"

Possible Causes and Solutions

1. There is some other error in NEOSYS software while saving the record. Probably another error message would come before this one and that message should be sent to NEOSYS in order to solve the actual problem.

2. There has been a temporary loss of connection to the server for some period of time. The NEOSYS web page needs to periodically "refresh" the exclusive lock on the document being edited. If NEOSYS cannot perform the refresh it may lose the rights to save the document.

Loss of connection could be for a huge variety of reasons including server, network, and workstation failures.

3. Bad Internet Explorer Configuration Loss of cookies etc due to third party software (internet toolbars/spyware etc)

Reset Internet Explorer to factory settings.

Troubleshooting the NTVDM Hard error when opening up NEOSYS processes

Error Message

The following error message appears:

NTVDM encountered a hard error

Solution

This error is caused by missing or corrupt Windows system files (i.e. command.com, autoexec.nt and config.nt). The solution is to run a program called XP FIX which will reinstall these missing files. You can download it form http://www.visualtour.com/downloads/xp_fix.exe

Sometimes even after running the XP FIX program the error still appears. Then you need to copy all the files under C:/windows/repair to C:/windows/system32 and in the autoexec.nt and config.nt put REM before any lines there which don't have it.

Troubleshooting a Service Unavailable message on Internet Explorer when opening up NEOSYS

Error Message

The following error message appears in Internet Explorer when you try to open up NEOSYS:

Service Unavailable

Solution

Open the IIS Manager, right click Web Sites and select properties:

Switch to the Service Tab and tick the "Run WWW Service in IIS 5.0 Isolation Mode".

You will be asked for Restart of IIS. Click yes to restart IIS. If you are not asked just restart IIS.

Inspecting IIS log files

At a windows command prompt:

c:
cd \Windows\system32\LogFiles\W3SVC1

or

%SystemDrive%
cd %SystemRoot%
cd system32\LogFiles\W3SVC1

then (substituting the ip number you are interested in)

find "192.168.1.55" *|sort>temp.log

Open temp.log in Excel and use Tools, Data, Text to Columns to split into columns using options "Delimited" and check split on Space.

Autowidth all columns by clicking on the top left box just outside the data to the left of column A and above column 1 then double click the column separator to the right of column "A"

Note that times and dates are in UTC/GMT so you have to add/subtract your timezone offset to get local times.

Fixing NEOSYS processes that do not auto start / Recovering from incorrect advanced date or time

Problem explained

After starting up the 1st process, the rest of the processes don't start up.

Solution

The possible cause for this could be that the system date/time might have been changed - either manually or by the auto synchronization. Do the following checks in the sequence of order:

1. Check for any .$* files (* is a number, the highest being the latest one). If it shows OK, then proceed ahead.
2. Check for any .end files and delete it to rename to .end.temp
3. Check the System Event Viewer log for any 520 or 577 error message (refer http://128.175.24.251/forensics/timechange.htm). Also check for any out of sequence / ahead of today date or time.
4. In case of no 520 or 577 error message, go to Administrative Tools > Local Security Policy > Local Policies > Audit Policy > Audit Privilege use - make sure that Success and Failure are selected under this (this will ensure that future changes to the date/time are recorded in the System Log).
5. In NEOSYS maintenance mode - F5 ED PROCESSES %UPDATE% - and see what it says, incase of any text there, that means that for sure the system date has been changed.
6. Now to fix this, in maintenance mode again enter - F5 DELETE PROCESSES "%UPDATE%"

Recognising and Solving Low Memory Problems

Quick Note: Installing a server class operating system on a workstation class computer with the intention of NEOSYS serving a heavy load is likely to cause problems with low memory.

Effects

It is speculated but not proven that low memory may cause NEOSYS to fail by hanging, causing damaged files etc.

Checking

F5
MEMORY

On server class machines it should say somewhere around 350Kb to 370Kb Free

Some server class machines have around 330Kb and sometimes even less with no reported problems

The actual effect of low memory is supposed to make NEOSYS slower and perhaps cause hanging and damaged files however this has not been proven in an specific case so far.

On workstation class machines it may often say around 280Kb to 300Kb.

Cause

Although there is plenty of real memory in virtually all computers now, NEOSYS runs in the legacy 16 bit virtual memory space of a windows mode called NTVDM. This is limited to 1Mb plus 4Mb of EMS memory.

The 1Mb memory space is shared with:

1. Various non-essential windows drivers which NEOSYS automatically disables them in autoexec.nt
2. Various plug and play hardware device drivers for the various adapters in the computer like video, network adapters and various other items that NEOSYS is unable to disable.

In a server class computer the hardware device drivers are usually minimally present in the 1Mb base memory and do not therefore DOESNT a low memory situation for NEOSYS.

In workstation class computers there are often many hardware device drivers present in the 1Mb base memory and this DOES causes a low memory situation for NEOSYS.

When NEOSYS is installed on workstation class computers with XP there is usually not a heavy load expectation and therefore the low memory does not cause a problem.

If Windows Server OS is installed on a workstation class computer NEOSYS may well be expected to serve a heavy load with limited amounts of memory.

Workstation class computers: hardware drivers present and EMS is installed in low memory (0000-9FFF) causing low memory for NEOSYS and possible inability to

Server class computers: Usually few hardware drivers are present in high part (A000-FFFF) of the 1Mb base memory and EMS is able to occupy the high memory leaving the low part (0000-9FFFF) of the 1Mb memory free for NEOSYS. You can find out how much memory is available to NEOSYS and whether EMS is occuping high or low memory using the following sections.

Fixing Low Memory

Start, Run, notepad c:\windows\system32\autoexec.nt

Every time NEOSYS starts it tries to make some changes as follow:

1. replaces all lines in C:/WINDOWS/SYSTEM32/AUTOEXEC.NT starting with 'lh ' to start with 'rem NEOSYS LH ' instead.
2. changes the line in C:/WINDOWS/SYSTEM32/CONFIG.NT "files=..." to "FILES=200"

The replacement is case sensitive triggered on 'lh' and 'files' so if you manually edit the files and remove the rem or change the number of files and leave the LH and FILES in uppercase then NEOSYS will NOT make further changes. This allows you to do manual amendments to the files without NEOSYS overwriting them.

Check that NEOSYS has successfully disabled all the drivers in the lines starting with LH.

They should be commented out (prefixed) with REM or REM NEOSYS as follows.

After making changes reopen NEOSYS in maintenance mode to use the MEMORY and WHO commands again.

REM Install CD ROM extensions
REM NEOSYS LH %SystemRoot%\system32\mscdexnt.exe

REM Install network redirector (load before dosx.exe)
REM NEOSYS LH %SystemRoot%\system32\redir

REM Install DPMI support
REM NEOSYS LH %SYSTEMROOT%\SYSTEM32\DOSX

Allowing DOS programs that require DOSX to run on the same computer as NEOSYS

The automatic commenting out DOSX by NEOSYS will prevent some other DOS-like programs from running. If NEOSYS is on dedicated server then there should be no other such programs to fail. However, if you must allow DOS-like programs to work as well as NEOSYS you can do the following configuration:

1. leave or restore the original AUTOEXEC.NT and CONFIG.NT files where they are
2. copy them to another folder eg neosys folder
3. make the necessary REM changes there by hand
4. right click the NEOSYS\NEOSYS\AREV.PIF and select properties
5. change the location of the AUTOEXEC.NT and CONFIG.NT files in the following location

Checking EMS Memory Configuration

Inspection

F5
WHO

press the up arrow to get to the last part/page

Example of Typical Server EMS Memory

Example of Typical Workstation EMS Memory

Correction

No easy way

Removal of hardware adapters designed for workstations instead of servers eg graphics cards and network cards.

Many of the devices may be located on the motherboard and not relocatable except possibly by BIOS configuration or special manufacturer information.

Use windows device manager, View: Resources by Connection, Open the Memory item and look for items between 000A0000 up to 000FFFFF that might give you a clue as to what hardware could be removed or reconfigured. Actually only 000C0000 to 000FFFFF is candidate for EMS memory since 000A000-000BFFFF is mandatory video memory in all systems.