Handling damaged files

Symptoms

"A Damaged file has been found..... ((the rest of this message to be completed)) 
Group format Error:
OS File : BOOKING.TEXTS
Group# : 00000932
Record length incorrect or frame linkage error.

Cause

Rare but nasty problem on server 99% caused by power failure, killing NEOSYS or server hardware or operating system failure.

• Did somebody kill NEOSYS processes to reboot the server or any other reason?
• Was there a power cut?
• Any expected or unexpected reboots in the server log? (see Event Viewer > System Log)
• Enough disk free space especially on servers not dedicated to neosys?

Solution

Depends on the cause and usually often requires aggressive interrogation to discover the cause.

Recovery

It is URGENT to stop using NEOSYS and fix the damaged file because it will cause NEOSYS to crash everytime it happens to update that part of the file. NEOSYS crashing while updating often produces unrecoverable inconsistencies in the database that can only be fixed by programmers sitting for hours trying to edit the database .... or by restoring from a backup causing loss of work.

See the following sections how to patch the damaged files.

Files that are automatically fixed

• LISTS
• LOCKS
• !INDEXING

The above files are automatically checked for corruption (and initialised to empty if so) when you startup a NEOSYS process and do not require any manual intervention apart from locating and eliminating the initial cause of the damage.

If the !INDEXING file was damaged then strictly speaking you should also do REINDEXALL to reinstate any possibly missing index entries, but since this will take a long time you should do this when everybody is off the system in the evening probably.

Fixing damaged files that have no process for recreating them

Some damaged files can be rebuilt using NEOSYS utilities especially index files with filenames starting with !. Other damaged files can be “fixed” to prevent NEOSYS crashes as follows but some records will most probably be lost. Depending on the file loss of a few records may not be too bad compared to restoring a database but the client should be informed and make the decision. Depending on the file it is usually best to lose the records. We can make a list of files that one can afford to lose records and those files that can be rebuilt programmatically from other files. Booking order texts they can lose records but inform them and especially the numbers mentioned in FIXFILE

FIXFILE will not run if there are neosys services still running so before doing the FIXFILE command, shut down services from the Support menu or with the Global command:

F5

FIXFILE filename

(Select option 1 to update the file)

In case you get any Overwrite messages – type YES, as these are the actual files being updated.

The only alternative is restore from a previous backup in this case losing a days work. Nasty, and even worse, it will be impossible to reproduce the days work and especially the exact serial numbers of critical documents like orders, invoices, receipts and payments advices.

Using FIXFILE to repair corrupted files

Corrupt files will cause programs to abort every time they try to update that section of the file. Aborting programs causes possibly horrendous inconsistency in the database so it is important to stop all user activity and fix the corrupt file immediately.

You can freely use FIXFILE on any file but it cannot and does not recover any corrupted records therefore, for those files that have specific programs to rebuild them, FIXFILE is not the correct option. Many files do not have programs to rebuild them and FIXFILE is a quick way to get the system back in operation.

FIXFILE does not fix the consistency of information in the database. When the database is is being updated with a transaction, usually a series of files are updated for the one transaction. A damaged file causes the update program to abort in the middle of the series of updates. Therefore some of the updates are done and the remaining updates are not done. This means that the data in the database (even after fixing the corrupt file) is inconsistent.

FIXFILE merely fixes the technical structure of the file so that any further program accesses to that portion of the file do not cause immediate program termination and consequent likely catastrophic database inconsistency.

You must prevent anyone from accessing NEOSYS while FIXFILE is running.

Fixing Damaged DEFINITIONS file

Non-Solution

You will probably not be able to login if the DEFINITIONS FILE is damaged. You can press Alt+S and type EXECUTE to get a TCL command box but this still doesnt help because you cannot use FIXFILE on DEFINITIONS.

CAUTION: DO NOT USE FIXFILE TO TRY AND FIX THE DEFINITIONS FILE.

DOUBLE CAUTION: IN CASE YOU DECIDE TO IGNORE THIS EXPECT ALL KINDS OF HORRIBLE PROBLEMS IN THE DATABASE REQUIRING A COMPLETE RESTORE FROM A GOOD BACKUP AFTER A FEW DAYS OF PAIN BECAUSE RANDOM OLD DATABASE UPGRADE/CONVERSIONS WILL BE PERFORMED *AGAIN* WITHOUT YOU KNOWING IT.

Partial Restore

You can copy recent DEFINITIONS.LK and DEFINITIONS.OV files from the DATA\datasetcode\GENERAL folder.

CAUTION: All changes done in the Configuration File, Authorisation File, Users File and various other changes will be reverted to the date of the DEFINITIONS files being restored. If you are restoring from last nights backup or testdata then this will not be much but could cause serious confusion to people who have changed their passwords etc.

Full Restore

Restore from your latest backup losing all work done since that time and date.

Preventing users from accessing NEOSYS in order to do maintenance or repairs

On most systems you can simply stop the NEOSYS processes. On other systems where NEOSYS processes are automatically started in response to requests from the web, you need to disable the autostart by one of the following methods

1. Stop the autostart program/script
2. Put a file called NET.CFG with a single line saying AUTOSTART=NO in the neosys\neosys folder
3. Stop the IIS server
4. Stop/Pause the particular web in question

Fixing damaged file that have a process to recreate them

ADS file can be recreated with F5 CREATEADS

ANALYSIS file can be recreated with F5 CREATEANALYSIS

Fixing damaged index files (names starting with !)

Whilst backing up or using NEOSYS the following error message might appear:

"A Damaged file has been found.....
Group format Error:
OS File : !ADS
Group# : 00000932
Record length incorrect or frame linkage error.

1. Ensure that no-one is using NEOSYS
2. Log into NEOSYS on the server
3. Press F5 to get a TCL command box
4. Type REINDEXADS and press enter.
5. Wait until it finishes and you get back to the command box
6. Type OFF in the command box and you will quit NEOSYS.

Other reindex commands are:

• REINDEXACCOUNTS recreates !ACCOUNTS, !BATCHES and !VOUCHERS
• REINDEXADDRESSES
• REINDEXADS
• REINDEXBOOKINGS recreates !BOOKING.ORDERS
• REINDEXBRANDS
• REINDEXCLIENTS
• REINDEXINVS recreates !INVOICES
• REINDEXJOBS
• REINDEXMATERIALS
• REINDEXPLANS
• REINDEXPRODINVS recreates !PRODUCTION.INVOICES
• REINDEXPRODORDERS recreates !PRODUCTION.ORDERS
• REINDEXRATECARDS
• REINDEXSCHEDULES
• REINDEXSUPPLIERS
• REINDEXTIMESHEETS
• REINDEXVEHICLES

REINDEXMEDIA does all the following

• REINDEXPLANS
• REINDEXSCHEDULES
• REINDEXADS
• REINDEXMATERIALS
• REINDEXBOOKINGS

REINDEXFILES does all the following

• REINDEXCLIENTS
• REINDEXBRANDS
• REINDEXSUPPLIERS
• REINDEXVEHICLES
• REINDEXRATECARDS

REINDEXPRODUCTION does all the following

• REINDEXJOBS
• REINDEXPRODORDERS
• REINDEXPRODINVS
• REINDEXTIMESHEETS

REINDEXALL does all the following

• REINDEXMEDIA
• REINDEXPRODUCTION
• REINDEXFILES
• REINDEXINVS
• REINDEXACCOUNTS
• REINDEXADDRESSES

Dealing with damaged files that cannot be fixed

If programmers are unable to fix the damage in a reasonably period of time and the damage is caused due to matters beyond NEOSYS control such as non-dedicated servers and power failures then data must be restored and client advised why they are going to suffer loss of data.

THIS DOES NOT RELEASE NEOSYS SUPPORT STAFF FROM THE OBLIGATION TO PROACTIVELY INFORM THE CLIENT OF MEASURES TO PREVENT THE RECURRENCE OF THE EVENT.

The following are the steps to follow:

1. Ascertain the cause of the damage to the database by looking at the internal logs
2. Create a new database from the damaged one as it will contain the up to date but damaged database.
3. Restore the database from the backup media so that the database is in the last good known state.
4. The client can see what work was done over the past couple of days from the damaged database and it will assist them to redo the work in the currently restored out of date main database.
5. Next discuss with the client about the cause of the damage and (a) what to do to prevent the damage occurring again (b) to agree a procedure to perform a manual backup in case the automatic one is not performed for any reason overnight.