Backup and Restore

Description of Backup Procedure for Client Hosted NEOSYS Servers

3 USBs must be maintained and rotated on a weekly basis.

• At 02:21 GMT, live databases are backed up, restored to test and compared with test to verify that the restore was successful.
• Every hour at 40th minute, live databases are backed up without restore and verification.
• Every hour at 45th minute, USB backup is performed, where the client's NEOSYS server is rsynced to USB.
• NEOSYS support staff check the emails every morning Monday through Friday.
• In case of failure, NEOSYS support staff take appropriate action and send an email (to the same people who receive the automated backup alert emails) indicating what action has been taken.
• Additionally offsite backup can be set up on NEOSYS server.

Description of Backup Procedure for NEOSYS Client Hosting Server

NEOSYS backup is a two phase process. It is mandatory that both phases are complete for the process to be considered a backup.

1. NEOSYS server is hosted on Leaseweb in Amsterdam. Following is the backup procedure:

• At 02:21 GMT, live databases are backed up, restored to test and compared with test to verify that the restore was successful.
• Every hour at 40th minute, live databases are backed up without restore and verification.
• Emails for the above backups are sent to backups@neosys.com.
• NEOSYS support staff check the emails every morning Monday through Friday.
• In the case of failure, NEOSYS support staff take appropriate action and send an email (to the same people who receive the automated backup alert emails) indicating what action has been taken.
• The above does not by itself constitute a proper backup because the backup is stored on the same server and physical disk as the actual data.

2. On a separate backup server, also hosted on Leaseweb in Amsterdam:

• Every hour at around 45th minute, NEOSYS server is rsynced to the backup server.
• Every half hour, NEOSYS server container is copied (lxc) to the backup server. Container backups of 7 days are maintained on the backup server.

Backup Procedures

Preparing nightly backup report

1. Note the success, failure and other error of the clients backup mail in an excel sheet and forward the same to your manager.
2. If there is a backup failure or backup is not available, check wiki to take necessary steps.
3. If there is any unknown error, forward the same to your manager.

Send the backup report after every quarter to all the Clients having consolidated backups.

Running a manual backup in case of failures

If a backup has failed, Support should check if any users are active in that particular database. Check Menu -> Support -> List of Documents in Use and also check list of users to see last login time. From backup email inbox, check how much time the backup of that database usually takes.

If no users have logged in yet and if backup of the database usually completes in a few minutes, quickly start a manual backup

This MUST be done first thing in the morning since Dubai clients generally start working around the same time as Support team.

Updating Nagios in case of failures

1. If the backup failure is unsolved, schedule downtime Neosys service in Nagios till 01 am.
2. If the backup did not happen because of server down. Call the IT person; ask him to re-boot the server and check wiki to do necessary step ahead and schedule downtime to Nagios for 2hours.
3. If there is an error "Backup->Impossible" on Nagios, check if the USB is properly inserted. If no USB is found, send a mail to the user asking to ensure that the backup USB is inserted properly. Schedule downtime to Nagios for 2hours.

Interchange backup USB mail reminder

Basically all the clients have different days to change their backup USB. All the notification can be seen on Nagios at 12.00 pm every day.

On the USB change day, at 6:00 am, when the processes start up, the system automatically sends the following email to everyone in the backup email receiver list:

It is time to change the NEOSYS backup media (e.g. USB Flash Drive)   
 
Please change it before 12:00 midday today.

Additional emails reminding IT staff to change backup media are sent out at 11:30am and 5pm if the backup media is not changed according schedule.

Since the system automatically sends a USB change reminder email to the client, support staff do not have to send them any instructions about changing unless they have failed to change the USB on the scheduled day or the scheduled day needs to be moved to another day.

Importance of interchanging backup USBs

If the backup USB is not interchanged on the scheduled day then the NEOSYS automated backup fails. This happens because traditionally, each USB holds backup of 7 days and using 3 different USBs we can store backups for the last 21 days enabling us to restore the system unto a time period beginning 21 days prior. If the USB is not changed then the first backup on the current USB is replaced with the new or latest backup leading to inconsistencies within the backups. Hence we must interchange the USB on schedule to avoid a backup failure the next morning.

The reasons for using multiple USBs for backup are:

• We can keep other USBs out of the office for safety purposes since theft or office fire/water hazards could damage the computer and the USB keys if they are all in

the same place.

• Having multiple USBs provide safety against corrupt USBs which cannot be used to restore any backup data.

Finding out which USB is inserted into the server

As we ask the client to have 3 USB's and interchange them weekly, we also need to sometimes track which one of these 3 USB's are inserted into the server. USB's can be tracked using their volume serial number in most cases. To find this out either go to the command prompt and type VOL <drive letter> where drive letter is the USB drive letter OR in the nightly backup message check for the 2nd line (which looks like this - 14/12/2009 2:45pm Media: 705B-5B5F). However serial numbers can be the same even for different USB's. One of the reasons for this could be that the USBs were imaged from one single USB which caused their volume serial numbers to be the same. However, such a situation is very rare.

Interchanging USB when scheduled USB change day falls on a holiday

When clients ask support which day to interchange USBs when their scheduled USB change day falls on a holidays, send them the below email.

Hi XXX,

Kindly interchange the USB on the last working day before the holiday i.e XXXday DD/MM/YY and then interchange with the next scheduled USB on first working day after the holiday i.e. XXXday DD/MM/YY. Continue to change the USBs as usual on XXXday.

Getting confirmations for Backup email list

The below procedure is done once every year. A task has been set in Thunderbird to remind Support to begin the procedure in November.

The first thing to do is to send the below draft email to Steve and ask if any changes are required.

Subject: YOUR URGENT REPLY IS REQUIRED - NEOSYS Backup Confirmations for <Company>, <City>

Hi <Decision_maker>

We are updating our records of the contact people in your organisation who should receive the various NEOSYS system alert emails and backup confirmations.

You are currently assigned as the decision maker in this matter

Note that it is the responsibility of your company to verify that the backups are successful and to alert us whenever there is an issue with the same.

Please confirm:

1.   The following email addresses should continue to receive these alerts

2.   Any additions/deletions of email addresses from this list

For: <Company>, <City>:

a) Decision Maker:

b) IT Manager (or person responsible to change backup media):

c) Monitors (Managers and/or interested parties):

Best Regards,
NEOSYS Support

1. After you receive confirmation, send emails to all clients.
   • Template email to each client (Thunderbird -> Support -> Templates -> Backup Confirmation Emails)
2. Create a new spreadsheet after copying previous year's confirmations sheet and rename with current year.
   • You can find the previous year's confirmation list in Nextcloud\support\Androulla_Backup Confirmations
   • Clear the contents and fill in as and when you get the confirmations from clients.
   • For hosted clients in the 'IT Manager' field, put 'None'
   • You can keep a list of all pending clients in a personal text file and/or mark as 'Pending' in Spreadsheet.
3. If any changes, update the Spreadsheet and change in NEOSYS.
4. Set reminders to follow-up on these emails to decision makers once a week.
5. In the case where a client asks for more than one decision maker, reply that it will be best to have only one decision maker.
6. If no reply for two weeks, send email to another person other than the decision maker.
   • Order: Another management level user > Main point of contact -> Most active user (check the authorisation file)
7. When all clients confirm, check backup spreadsheet and confirm that it matches the System Configuration file in NEOSYS and any personal records that you have kept.

Manual Backup

See How to do a manual backup in NEOSYS

Manual backups can also be triggered on the server process screen by pressing the lower case letter b key. After the backup is complete the process will resume and restart the child processes. Pressing capital B will result in the process closing on completion of the backup and the child processes will not be restarted. This is useful if you want to trigger a backup and leave without waiting for it to complete.

Configuring NEOSYS automated backup

NEOSYS is designed to do an automated backup of the data (not the program files). The configuration for the automated backup rests in the System Configuration File under Menu > Support.

First step is to configure the automated email settings. After every backup attempt NEOSYS will send out a mail with the status of the backup (either success or failed). You have to configure the SMTP details here along with the list of recipients to the backup email.

Ensure to have atleast 1 person from the client management and 1 person from the client IT, besides the mandatory backups@neosys.com address

Always use the NEOSYS SMTP server details as follows and put in a fictitious client address so that NEOSYS staff can quickly identify which client this mail came from.

Do not configure the backup time unless instructed by your manager. The system will automatically start the backup at 1 am if there is nothing configured in the Backup time.

Type in the backup location (USB/other media) drive letter in the Backup Drive field and the Uploads fields.

Putting a 0 in the Uploads field will disable the backup of the Uploaded files, else by default the uploaded files will be backed up to the drive mentioned in the backup drive field.

The scheduled days of backup can be configured using the checkboxes. If none of the days are checked, backup will be done on all days by default.

Next setup the auto-start of the databases and enter how many processes should be started and which all databases should be backed up.

In the example below DEMO database has been configured to start 3 processes and also do the automated backup, whereas the DEMOTEST database is configured to start 1 process and no automated backup.

Non-Liability for Backup

If a NEOSYS client has not signed the standard NEOSYS contract which excludes liability then NEOSYS needs a specific agreement to the following:

Dear xxxxxxxx, 

NEOSYS has offered to setup a nightly backup of your NEOSYS data. 

We can only do this on condition that you will not hold NEOSYS liable for damages of any kind in the case that our backup procedure fails to 

meet its essential purpose in an emergency. 

You may make your own alternate parallel arrangements to ensure that the whole of the contents of the NEOSYS server are backed up sometime 

between 3am and 6am at night when NEOSYS system is shutdown. 

Please confirm your agreement. 

Best Regards, 
xxxx xxxx 
NEOSYS

Backing up the Images folder

The Images folder under the NEOSYS installation is used to upload images/artworks/files from the Job File section and hence needs to be backed up. NEOSYS will automatically backup this Images folder to the USB drive or other location (specified for the usual data backup) once a week.

To configure this backup, RSYNC needs to have been installed during the initial installation.

In case we need to configure the Images folder backup at another location other than the usual nightly backup location than we need to edit:

line 12 - and specify the Drive of the location to be backed up to. eg. E or F

Backup to other media (i.e. not to USB)

If the backup is going to non-removable media (even if it is a shared folder on another computer) then the NEOSYS user/client/licensee (NOT the NEOSYS support team) can, at their own responsibility, arrange to move (NOT COPY) the NEOSYS backup files from the USB location to a backup location of their choice and avoid the

WARNING message : "Backup media not changed. Overwriting last weeks backup"

Sample alternative response to client requests for additional backups

The existing NEOSYS backup must continue to take place for safety because it is the only well understood standard, controlled and checked procedure in use for all NEOSYS clients.

You are free to setup and operate any additional backup procedure you like but NEOSYS cannot take any responsibility in setting up, monitoring or approving your additional backup procedure because it is beyond our sphere of control, expertise and trust.

What you can backup is the NEOSYS backups on the USB drive. These are readily available online at all times on the NEOSYS server for you to access and copy as you choose.

You can backup the usb anytime using anything you like, but you must not backup anything on any hard disk eg: C or D at any time.

Backup in virtual server

Virtual servers that allow USB passthrough

If NEOSYS is installed on a virtual server that allows USB passthrough (e.g. VMWare vSphere version 4.1 and above), then NEOSYS can be configured to backup onto the inserted external USB drive.

3 USBs should be maintained and rotated on a weekly basis as done by clients installed on physical servers.

VMware website says that when changing the USB, the VMware window must be the active window so that the USB reflects in the virtual machine and not on the host machine. https://www.vmware.com/support/ws45/doc/devices_usb_ws.html

Virtual servers that do not allow USB passthrough

If NEOSYS is installed on a virtual server that DOES NOT allow USB passthrough, then below are the various cases:

Case 1: BACKUP.ZIP file CAN be created on an external removable backup media

The backups can be done onto an USB drive if it is connected to the host machine and configured to appear on the virtual server. The virtual server MUST be restarted for the inserted USB to reflect each time the USB is changed i.e. every week.

Support Staff MUST train the IT person to stop the processes properly i.e. use global.end file and rename it to global.end.temp to allow processes to start after reboot. Support Staff can send a follow up email as mentioned below

Email to IT person after training

Kindly confirm that you will close the NEOSYS processes properly before shutting down the virtual server, i.e. by renaming "global.end" to "global.end.temp" and making sure that no ntvdm.exe processes are running in the task manager. As discussed during the training, this can cause damaged files in NEOSYS if the server is shutdown without closing the NEOSYS processes.

Also please confirm that you will rename "global.end" back to "global.end.temp" before shutting down the virtual server so that the NEOSYS processes start up automatically when the virtual server is restarted.

Case 2: BACKUP.ZIP file CANNOT be created on an external removable backup media

The Client must *at their own responsibility* agree to arrange copy of the backup files to an external backup location every day if the server cannot be restarted every week. In this case, NEOSYS Support staff must send the client below email:

Email for new clients with/old clients moving to virtual servers that do not allow BACKUP.ZIP file to be created on an external removable backup media

If we install NEOSYS on a virtual server, the backups cannot be done onto an external removable backup media as confirmed by your IT. NEOSYS considers backups to be done only if BACKUP.ZIP file is created on an external removable backup media. 

Instead, NEOSYS will be configured to create the BACKUP.ZIP files on the C drive of the virtual server. Your IT team can, at your own responsibility, arrange to move (NOT COPY) these backup files to an external backup location every day. We recommend that the backup location should be removable media (e.g. external hard disk) and not non-removable media (e.g. shared folder on another computer).
 
If any damage happens to the NEOSYS server, the backup files on C drive will not be accessible and will therefore be useless. So, kindly confirm that your IT team will arrange to move the backup files as mentioned above.

In addition to this, offsite backups of your virtual server can be setup by us if required, with an additional cost.

Backup Messages

Interpreting backup alerts messages

Success messages come with a blank body

Failure messages should come with the errors in the body

Both success and failure messages come with an attachment containing a list of files copied (or supposed to be copied but failed) and a summary of the volume and speed of data transferred

Cause and Solution of Backup Warning Messages

NEOSYS gives a backup WARNING message whenever it discovers that it is overwriting backups a week old. It does this because NEOSYS only deems backups completely successful if the backup media is rotated at least on a weekly basis.

If the backup media is removable then it should be changed weekly and the same media not used repeatedly on successive weeks.

If the backup is going to non-removable media (even if it is a shared folder on another computer) then the NEOSYS user/client/licencee (not the NEOSYS support team) can, at their own responsibility, arrange to move (not copy) the NEOSYS backup files from that location to a backup location of their choice and avoid this WARNING message.

Handling failure and warning messages on nightly USB backup alerts

Backup mail not received

Troubleshooting email not received

Handling Change Backup message if the client does not use a USB backup device

IF THE CLIENT IS BACKING UP TO A NON-REMOVABLE DESTINATION THEN EITHER 1. LIVE WITH THE FAILURE MESSAGE OR 2. SUPPRESS THE BACKUP ENTIRELY IN WHICH CASE NO BACKUPS ARE BEING DONE AND PROBABLY THIS WILL SHOW ON NEOSYS PROACTIVE WARNING SYSTEMS LIKE NAGIOS.

Nagios warnings can be acknowledged.

DO *NOT* FOLLOW BELOW PROCEDURE SINCE IS IT IS TOTALLY UNACCEPTABLE FOR NEOSYS TO SEND OUT A "BACKUP SUCCESS" MESSAGE WHEN THE BACKUP IS NOT BEING DONE PROPERLY. IN GENERAL, TAKING SHORT CUTS THAT MAKE THINGS APPEAR SATISFACTORY WHEN THEY ARE NOT IS VERY POOR POLICY.

1. Basically most of the client use USB for Neosys server but there are some clients who do not use USB. They could be on virtual server or just saving their backups in hard drive.
2. As we know that backup take place in Data.bak folder but it does a backup only for a week in a single USB. Over here client do not use USB but save their backup in hard drive/virtual server location in Data.bak folder, once the week is over it will again give Change backup message.
3. In this case rename the Data.bak folder to Data1.bak this is done because system read only Data.bak folder as it is configured in the system. When you rename Data.bak to Data1.bak it automatically create a new folder Data.bak in Backup drive.
4. This process is done so that the backup does not fail next day morning and backup is done in Data.bak folder along with successful backup email.
5. In the third week we will again see the same message Chang backup USB message, so you need to rename Data.bak folder to Data2.bak
6. But in fourth week you rename the Data1.bak to Data.bak and change the Data2.bak to Data1.bak and Data.bak to Data2.bak
7. Keep only three folders and Data.bak, Data1.bak, Data2.bak and keep on renaming these folders every week as shown above.

AS MENTIONED EARLIER *DO NOT FOLLOW* THE ABOVE PROCEDURE ON ANY CLIENT THAT DOES NOT USE A USB BACKUP DEVICE.

Warning Message: Backup media not changed. Overwriting last weeks backup

If the client does not change the USB on scheduled day the next day's backup will fail, with email subject: "NEOSYS: Backup <CLIENT> -> <Drive Letter>: <USB NAME> CHANGED FAILURE"

As per standard "NEOSYS Backup Failure - USB not interchanged" email, the schedule day to interchange backup USB drives should generally remain on the same scheduled day. No cases have yet been documented where we have inconvenienced and forced our clients to change the day.

NEOSYS gives a backup WARNING message whenever it discovers that it is overwriting backups a week old. It does this because NEOSYS only deems backups completely successful if the backup media is rotated at least on a weekly basis.

If the backup media is removable then it should be changed daily or weekly and the same media not used repeatedly on successive weeks.

If the backup is going to non-removable media (even if it is a shared folder on another computer) then the NEOSYS user/client/licensee (not the NEOSYS support team) can, at their own responsibility, arrange to move (not copy) the NEOSYS backup files from that location to a backup location of their choice and avoid this WARNING message. Also refer to Handling Change Backup message if the client does not use a USB backup device

Standard "NEOSYS Backup Failure - USB not interchanged" email:

Dear Team,

 The NEOSYS backup has failed today as the USB was not interchanged on the scheduled day i.e. yesterday. 

 Please interchange the USB immediately today to avoid a backup failure tomorrow morning.

 Also note that your scheduled day to interchange the USB next week remains unchanged to <Schedule day>.

Kind Regards

XXXX XXXX

Error Message: Somebody else was using the dataset

Server=NEOSYS
Client=NEOSYS
User=ADAGENCY
NEOSYS Ver:18:02:13  30 MAY 2006

Backup started at 19/11/2006 2:00am

TEST DATA TESTDATA

To E:\DATA.BAK\TESTDATA\Sunday\BACKUP.ZIP
!!! THE BACKUP HAS FAILED !!!

(because one or more files were not backed up
 - somebody else was using them)

Error Explained

Despite the error message which shows that 'somebody else was using them', it is definitely not correct as NEOSYS shuts down automatically at 2 am (time of the

backup). The actual problem might be that a NEOSYS process must have got stuck and failed to shutdown or a maintenance process was left open by accident.

Action to be taken

Close any maintenance processes that should not have been left open in the first place.

Try to exit normally or kill any other "hung" processes, taking screenshots of any error messages in order to try and prevent the problem happening again.

If not inconvenient to the users consider initiating a manual backup or just check that the next automatic backup works ok.

Error Message: Size Lock

Server=NEOSYS-SERVER
Client=NEOSYS-SERVER
User=NETSERVICE
NEOSYS Ver:19:57:50  08 NOV 2006

Backup started at 19/11/2006 2:00am

ADLINE ADLINE

To F:\DATA.BAK\ADLINE\Sunday\BACKUP.ZIP
The following files had a size lock but have been fixed SHADOW Warning! Process 
U79068                                                      
Other Network Stations are Active!
This utility was designed to be run in a single-user mode!

Potential Errors Could Occur, If You Proceed to fix the sizelock Values with Other Stations   
Active.  These Errors Include;

Invalid SELECT results, Degredation in Network Performance, and other Problems/Concerns which 
may be specific to your Installation/Application.

IT IS STRONGLY RECOMMENDED THAT ALL NETWORK STATIONS BE LOGGED OUT BEFORE PROCEEDING!

Do you wish to Proceed at this time?%B%
!!! THE BACKUP HAS FAILED !!!

(because one or more files were not backed up
- somebody else was using them)

Error Explained

Sizelock errors are not critical and do not stop the backup from being performed. Sizelocks are cleared automatically during backup but the backup alert becomes a

warning that the backup was not quite perfect.

Sizelocks during database checking indicate that some files were not properly closed in normal processing but do not indicate corrupt files. Sizelock means that a file

is prevented from expanding and contracting. Expanding and contracting is necessary in order to speed up access to larger numbers of records. Therefore if sizelocks

were to be left uncleared then the file would become slower and slower to access over time.

In the above example the backup was not completed because of another error "files in use".

Action to be taken

If this is an automatic backup then no action is required since the sizelocks are automatically cleared.

During a manual backup then you should press F9 on the sizelock warning screen to confirm that the sizelocks should be cleared.

Sizelocks can only be cleared if no other process is open.

Failure Message : Backup file size is 0

16/4/2009 9:43am Creating zip

16/4/2009 9:45am !!! THE DATABASE BACKUP HAS FAILED !!!

BECAUSE BACKUP FILE SIZE IS 0

Error Explanation 1

This happens when there is no adequate storage space on the hard-disk which houses the NEOSYS application and data files. When the automated backup is initiated, the

NEOSYS application checks all the data files and creates a zipped file of it which is the databasename.zip (in the above case IMDUB.ZIP). During this zip file

creation, NEOSYS needs additional disk space temporarily and if this is not available then the backup would fail. The IMDUB.ZIP file is created and backed up (as

BACKUP.ZIP under the respective day of the week) to the USB or any other drive specified and is automatically deleted from the drive that it was created in.

Error Explanation 2

This could be due lack of EMSMAGIC.

In case of a Windows Server 2003 installation: Enabling EMS memory on Window 2003

In case of a Windows Server 2008 installation: Installing EMS Magic on Windows 2008

Actions to be taken

1. In this case, you need to clear up the hard-disk and create some space. In the above scenario at Initiative there was a folder of 52 GB in the same drive that houses NEOSYS and the available disk space was just 40 MB. So the un-necessary folder was moved out to make space.
2. If there is sufficient space available on the hard disk which houses NEOSYS, then consider checking the anti-virus settings and escalate the issue to NEOSYS programmer for debugging.

This is another reason why NEOSYS personnel should put their foot down on getting dedicated servers for NEOSYS as it restrains the client IT staff from using the

NEOSYS server to house data or any other applications besides NEOSYS.

Error Message: Inadequate Storage Space on USB

Error 1

 
 Server=SERVER2
 Client=SERVER2
 User=ACCOUNTS
 NEOSYS Ver:11:26:27  26 JUL 2010

 7/2/2012 2:00 Backup started F: BASIC 
 7/2/2012 2:00 Media: FADE-0001

 7/2/2012 2:00 Checking for damaged database files
 7/2/2012 2:01 Checked OK 1.92MB (+1.00KB)

 7/2/2012 2:01 Creating zip
 7/2/2012 2:01 Created OK BASIC.ZIP 251KB

 7/2/2012 2:02 Copying to F:\DATA.BAK\Tuesday\BACKUP.ZIP
 7/2/2012 2:02 Copied OK

 7/2/2012 2:02 Verifying zip OK
 7/2/2012 2:02 Verified OK

 7/2/2012 2:02 Started backup of uploaded files in /cygdrive/E/ to F:
 7/2/2012 2:04 neosys/DATA/ACCOUNTS/REV20050.LK
 neosys/DATA/ACCOUNTS/REV20052.LK
 7/2/2012 2:04 rsync: writefd_unbuffered failed to write 4092 bytes [sender]: Connection reset by peer (104)
 rsync: write failed on "/cygdrive/F/E/neosys/DATA/ACCOUNTS/BATCHES.LK": No space left on device (28)
 rsync: connection unexpectedly closed (9077 bytes received so far) [sender]
 rsync error: error in rsync protocol data stream (code 12) at /home/lapo/packaging/rsync-3.0.4-1/src/rsync-3.0.4/io.c(632) [sender=3.0.4]
 7/2/2012 2:04 Finished backup of uploaded files

 7/2/2012 2:04 !!! THE DATABASE BACKUP HAS FAILED !!!

Error Explained

In the above error log we notice the line:

rsync: write failed on "/cygdrive/F/E/neosys/DATA/PT0011A/ACCOUNTS/BATCHES.LK": No space left on device (28)

This is an indication that the USB being used to store backups does not have adequate storage space to save further backups.

This particular message only comes when using rsync to copy all IMAGES (ie files uploaded to NEOSYS) to the USB.

Sometimes, image backup is used to configured to backup the whole of a disk or the whole of data folder for example if there are hundreds of databases that are not all

used everyday.

Error 2

 Server=SERVER2
 Client=SERVER2
 User=ACCOUNTS
 NEOSYS Ver:11:26:27  26 JUL 2010
 7/2/2012 2:00 Backup started F: PT0621 - GABANG HOLDINGS LIMITED
 7/2/2012 2:00 Media: FADE-0001
 7/2/2012 2:00 Checking for damaged database files
 7/2/2012 2:02 Checked OK 2.25MB (+20.0KB)
 7/2/2012 2:02 Creating zip
 7/2/2012 2:02 Created OK PT0621.ZIP 462KB
 7/2/2012 2:02 !!! THE DATABASE BACKUP HAS FAILED !!!
 BECAUSE NOT ENOUGH FREE SPACE ON BACKUP MEDIA.
 462KB REQUIRED BUT ONLY 452KB AVAILABLE

Error Explained

As the last line of the above error log clearly states, there is not adequate free space available on the USB to perform a backup.

In the above case, 462KB is required space while only 452KB is available.

Action to be taken

The simple solution is to purchase a new USB with greater storage space than the existing USB.

Sample Email to be sent to clients:

The automated NEOSYS backup has failed due to insufficient storage space on the USB.
Currently you are using a 4GB USB, which is insufficient for storing backups.
I recommend that you use a USB with 8 GB or more storage space.

Note - We may also solve this issue by checking the USB for unnecessary files and move/delete them.

Error Message: "Cannot backup/restore because PROCESS1 PROCESS2 (etc) is/are online" message

This can happen for a variety of reasons, e.g if any the NEOSYS processes fail to close down at backup time because the process has hung due to software error.

Error Message: "LISTS file is not available"

Server=NEOSYS	
Client=NEOSYS	
User=ADAGENCY	
NEOSYS Ver:16:07:53  04 JUN 2018	
	
"LISTS" is invalid	
	
The "LISTS" file is not available.	
	
?[FS200]????????????????????????????????????????	
	
The "DATAVOL\NEOS0001\" file is not available.	
	
THERE ARE NO FILES IN THIS DATASET

Solution

Although unlikely, look for third-party software that have been installed on the server without the agreement of NEOSYS and uninstall them (see link below).

Refer: Uninstalling unauthorized third-party software

1. Stop all processes (using global.end method).
2. Navigate to the NEOSYS/DATAVOL folder and delete it.
3. Perform a manual backup. (NEOSYS > Menu > Support > Backup)
4. Backups should be successful. Check by various methods.
5. Start NEOSYS processes.
6. A new email may be sent when the first process is started, (INIT.GENERAL) containing the next section error "volume and cannot be attached".

Error Message: "volume and cannot be attached"

The "DATAVOL	
DATAVOL\NEOS0001\	
08:48:57  27 JUN 2018" volume	
has the same label	
("")	
as the "" volume	
and cannot be attached.	
	

Server:  NEOSYS	
Install: D:\NEOSYS\NEOSYS\	
Version: 16:07:53  04 JUN 2018	
Database:2996420B AMC	
Process: 1	
Client:  NEOSYS	
User:    ADAGENCY	
Stack:   SYSMSG,INIT.GENERAL

Solution

Pending

Fixing Network Connectivity Errors for consolidated Backups

There could be multiple reasons why the backup server might not be able to connect with the backed up server.

Error

[Receiver] io timeout after 600 seconds -- exiting rsync error: timeout in data send/receive (code 30) at /home/lapo/package/rsync-3.0.8-1/src/rsync-3.0.8/io.c(140) [Receiver=3.0.8] [Receiver] io timeout after   600 seconds -- exiting rsync error: timeout in data send/receive (code 30) at /home/lapo/package/rsync-3.0.8-1/src/rsync-3.0.8/io.c(140) [Receiver=3.0.8] tried via hostname.support.neosys.com now trying via hostname2.support.neosys.com
ssh: Could not resolve hostname hostname2.support.neosys.com: hostname nor servname provided, or not known
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] rsync error: error in rsync protocol data stream (code 12) at /home/lapo/package/rsync-3.0.8-1/src/rsync-3.0.8/io.c(601) 

The following tests help in checking why the connectivity was lost between the backup server and the backed up server:

Check whether the server was up

If either of the servers were down, that could be the reason why connectivity was lost between the backup server and the backed up server. If this test fails, proceed to the next step.

Telnet check

 telnet <hostname> 19580  

<hostname> is the name or IP address of the remote server to connect to.

If telnet is not able to connect to the remote host specified in <hostname> or establish a connection on the <port> specified, it will report an error similar to the following:

C:> telnet fred.plus.com <CR>
Connecting To fred.plus.com...Could not open connection to the host, on port 23: Connect failed
C:> telnet mail.plus.net 60 <CR>
Connecting To mail.plus.com...Could not open connection to the host, on port 60: Connect failed.
C:> telnet 60.92.12.56 74 <CR>
Connecting To 60.92.12.56...Could not open connection to the host, on port 74: Connect failed.

Autologin

Error

rsync: failed to connect to 127.0.0.1: Connection refused (111) rsync error: error in socket IO (code 10) at 
/home/lapo/packaging/rsync-3.0.7-1/src/rsync-

3.0.7/clientserver.c(122) [Receiver=3.0.7]

tried via hostname.neosys.com

now trying via hostname2.neosys.com

rsync: failed to connect to 127.0.0.1: Connection refused (111) rsync error: error in socket IO (code 10) at 
/home/lapo/packaging/rsync-3.0.7-1/src/rsync-

3.0.7/clientserver.c(122) [Receiver=3.0.7]

Error Explained

The automated backup cannot use autologin to access the server being backed up and so the backup fails.

Solution

Check autologin and Run autologin

Error Message: "WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!" message

Error

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:9i9JH5xh11rEMIgrsax8A7bEKXfKVRiJr9nZllWXdFg.
Please contact your system administrator.
Add correct host key in /home/administrator/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/administrator/.ssh/known_hosts:208
ED25519 host key for [192.168.10.1]:19580 has changed and you have requested strict checking.
Host key verification failed.
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1]

Solution to the problem is to remove the host key for the host mentioned in the error message in the known_hosts file.

Run the following command in Cygwin terminal.

 ssh-keygen -R [host ip]:port 

e.g in the above message for host ip 192.168.10.1 and port 19580 the command is

 ssh-keygen -R [192.168.10.1]:19580 

Next re run autologin for the host, see Running autologin

Check Scheduled Tasks on the Server

Check if logs of Scheduled Tasks on the server, to see if the remote backup task executed and completed successfully. It is possible that the remote backup scheduled task hung due to some reason and a new instance of the task fails to start up the next day due to the existing instance. To avoid this situation, in Properties of the task, under Settings tab, check "Stop the task if it runs for:" and configure it to less than 24 hours.

Adding/restoring SSH key to backup server

(Will be helpful to understand the concept of asymmetric keys).

This script is usually used to fix the backup procedure. The backup server initiates the backup procedure using the "pull" concept to get data from source servers. The "backed-up server" serves the data (usually using an rsync service) to the backup server on request. They do not use a "push" concept, this is for security reasons. This means that backup servers need to be able to automatically login to the backed-up servers. On most backup servers a NEOSYS Cygwin script called autologin.sh has been installed.

NEOSYS processes on a target server can sometimes reset/ lose autologin (public) keys (particularly when NEOSYS is upgraded) on the target server when it should not actually. This enforces rerunning autologin.sh. You only need to run autologin.sh once per pair of backup and backed-up servers, in the backup server.

This relates to a common procedure used by NEOSYS to provide automatic nightly synchronisation/backups between servers for multi-office configurations. For more information see http://itwiki.neosys.com/index.php/Setting_up_remote_backup

Creating/Upgrading autologin.sh

Follow commands below add autologin.sh to system:

1. Copy script below.
2. run "cd ~/ && nano autologin.sh", paste script into editor and exit.
3. Run "chmod +x autologin.sh"

#!/bin/bash


if [ -z $1 ]; then #syntax
   echo ./autologin.sh targethostname targetusername sshport
elif [ $1 ]; then
   set -e

   REMOTEHOST=$1
   REMOTEUSER=$2
   PORT=$3

   test $REMOTEUSER || REMOTEUSER=administrator
   test $PORT || PORT=19580

   if [ ! $REMOTEUSER ]; then
      echo -n "Remote User (blank=administrator)/root?"
   read REMOTEUSER
   if [ "$REMOTEUSER" == "" ] ; then REMOTEUSER=administrator ; fi
   fi

   if [ ! REMOTEHOST ]; then
   echo -n "Remote Host? "
   read REMOTEHOST
   if [ "$REMOTEHOST" == "" ] ; then exit; fi
   fi

   echo PASSPHRASE MUST BE BLANK IF YOU GET ASKED FOR IT!
   #Generate priv/pub keys in .ssh if not already done
   #Priv key mustnt be accessible except to owner otherwise wont work.
   test -f ~/.ssh/id_rsa || \
   ssh-keygen -t rsa && \
   chmod 600 ~/.ssh/id_rsa

   echo
   echo "Logging in to the remote server (enter the pass again)"
   sed 's/neosys//' ~/.ssh/id_rsa.pub | \
   ssh -p $PORT $REMOTEUSER@$REMOTEHOST \
   "mkdir -p .ssh ; chmod 700 .ssh ;\
   cat >> .ssh/authorized_keys ;\
   chmod 644 .ssh/authorized_keys"

   echo
   echo "Test automatic login to the remote server"
   echo " SHOULD NO LONGER ASK FOR PASSWORD."
   echo " IF SUCCESSFULL, TYPE exit"
   ssh -p $PORT $REMOTEUSER@$REMOTEHOST
fi

Using ./autologin.sh

1. Asks you for the username and hostname and port. (if username is administrator and port is 19580, then you can omit)
2. Asks you to enter the administrator or root password and copies the public key from the source server to the destination server.
   
3. Logs you in to the destination server's Cygwin/bash command line.

Running ./autologin.sh

In Cygwin on the backup server, you can use any of the following syntax depending on your configuration:

Syntax: (for syntax reminder type "./autologin.sh")

./autologin.sh targethostname targetusername sshport

• targethostname - if omitted will be prompted for it
• targetusername - if omitted will be "administrator"
• sshport - if omitted will be "19580"

Example using prompting for parameters (targethostname)

./autologin.sh

Example using default user and port (administrator/19580)

./autologin.sh examplehost.neosys.com

Example with different user (port will be 19580)

./autologin.sh examplehost.neosys.com administrator

Example with different user and port

./autologin.sh examplehost.neosys.com admin 19500

The username is usually administrator for windows server targets.

The hostname can be found in backup email logs or in the CONFIG.CMD file in the backup server.

There are often two alternative hostnames. Try the first one first. If you succeed with the first then there is no need to try the second.

Sometimes one of the host names is based on alternative network access methods like hamachi which uses ip addresses starting with "5." and requires hamachi service to be running in both servers and this sometimes is dependent on being logged in and correct setup of hamachi. You can ping the host names to discover the ip numbers of course.

If Autologin is configured successfully, it logs you in to the backed up server's cygwin/bash command line.
YOU should NOT be required to enter the password each time. You type "exit" to quit the backed up server's command line.

If Autologin is not configured successfully, you might be required to enter a password. In that case, enter the password and follow instructions on screen. The system asks for the password to the target server to transfer an identity file and once again to access the command line of the target server from where you must follow a set of instructions in order to load the identity file properly. Follow the instructions on the screen VERY carefully.

Checking if autologin is configured correctly and working or not

On the source system, in Cygwin/Terminal console, type the following, changing port number, administrator and clientname to suit the case.

ssh –p 19580 administrator@clientname.hosts.neosys.com

After a few seconds, if it is working properly, it should give you a command prompt on the target system. This indicates that autologin was successful. You may exit using the command:

exit

Otherwise, if it gives some error and in particular if it asks you anything at all, for example “confirm fingerprint?” or “password?” then autologin is not working. You can then type in the following command in cygwin to configure or reconfigure autologin:

./autologin.sh

Switching to a backup server

As NEOSYS provides clients with option of backing up their data to a remote NEOSYS server in case of emergencies or server problems, it is crucial you understand the below procedure on how to switch to a backup server in the event of such a situation. Extreme care must be taken when switching over to using a backup server otherwise unnecessary data loss is very likely.

Backup servers are normally switched off and should not be started automatically otherwise there is a serious risk of the client’s staff working on two systems. It is not possible to merge two databases into one database. Before the backup server is enabled the main server must be disabled, and before the main server is re-enabled, the backup server must be disabled again. This can be managed technically without requiring any decision from senior non-technical staff.

However, there are also some potentially hard decisions about unavoidable loss of data versus continued system availability. *** Backup servers should therefore only be started with the written approval of the clients senior staff. A suitable email requesting approval follows.

The following case assumes that the main server has gone down sometime during the working day and that therefore the data on the backup server is out of date. Allowing them to use the backup server therefore implies some loss of data. They may wish to lose the data. They may wish to work on the backup server data and then try to redo the work on the main server once it is restored. There are a variety of options depending on the situation.

If the main server is still functioning AND you are reasonably sure that the database is not damaged (which is perhaps an unlikely situation if you need to use the backup server!), it may be sensible to trigger an additional “backup/sync” process to bring the backup server database up to date with the main server. Before you do this, it is advisable that you take a backup copy of the backup system on the backup server. In this case there would be no data loss in using the backup.

An additional option of providing usage of the backup server in read-only mode so that people can at least access some data is being developed. The backup server could be available continuously at any time in read-only mode. This article would then be related to switching a backup server into main operational mode.

Dear {senior staff} cc {IT staff}

Please note that we can enable the backup server if you wish.

However the data on the backup server is out of date since it is a copy of your main database as at 11/22/33 99:99.

If you wish to allow work to be done on the backup server then any data entered on your main server since the above date will be lost if we 
subsequently copy the data on the backup server to the main server.

If, after using the backup server, we do NOT copy the data on the back server to the main server then any data you have entered on the backup 
server/database will be lost.

Please confirm a) you want to work on HOSTS2 database and that we should therefore enable it and b) you have disconnected your main server 
for the duration.

Best Regards,
xxxxxx xxxxxxx
NEOSYS Support

How to Restore NEOSYS from Backup

For a more detailed step by step go to [Restoring or Installing initial Database]

1. Log into Maintenance Mode
2. Go to General > Backup & Data Management
3. Select (the 4th option) Restore from disk or diskette
4. Select the drive where the latest successful backup.zip file is present and follow the prompts ahead.

Deleting all current and historical backups of a particular directory in NEOSYS

Step 1. Searching for directories to delete without deleting them

There must be no similarly worded directories. Here the example directory is called DDBTEST.

find /backups/current /backups/snapshots -type d -path "*/DDBTEST/*"

Step 2. Deleting the directories using rm

This command uses the VERY DANGEROUS rm command. If the search is done even slightly wrong - which is well known to be very easily done - then rm will happily remove *all* files on the server causing *partial or complete loss of all backups and/or complete loss of the whole server*, requiring installation from scratch.

Blunders with rm are notorious in IT.

The command given below is incomplete and should only be completed by a skilled and responsible IT person who knows what to check and what to add.

In its incomplete form, it is safe to run and will only highlight what directories will be deleted or delete empty directories.

In its completed form, it is DANGEROUS and should not be used except as described above.

find /backups/current /backups/snapshots -type d -path "*/DDBTEST/*" | xargs -n 1 rm

Step 3. Check all gone

Redo step 1

While Moving NEOSYS to a new server, after copying D drive the Maintenance does not contain all data sets

After copying D drive, when you open Maintenance if the data sets are not present as in the old server you MUST restore Backup as restore is a verified/quick method of creating data sets. While restoring last night's Backup for missing data sets, the system gives a warning message "Dataset is already on the computer" as shown below. You MUST select the 2nd option "Overwrite the existing dataset" from the options in the warning message. In this way, all data sets will be created/saved on the new server.

╔═════════════════════════════════════════════════════════╗
║                     *** WARNING ***                     ║
║                       TEST /TEST                        ║
║               IS ALREADY ON THE COMPUTER                ║
║  IF YOU CONTINUE, THE EXISTING FILES WILL BE DELETED.   ║
║                                                         ║
║                What do you want to do?                 ║
║───┬─────────────────────────────────────────────────────║
║  1│Cancel the restore process                           ║
║  2>Overwrite the existing data set                      ║
║  3│Restore and make a new dataset                       ║
╚═════════════════════════════════════════════════════════╝

Accessing historical snapshots of NEOSYS hosted clients data

The main NEOSYS backup server nl19 contains backups of NEOSYS hosted clients along with all NEOSYS internal servers.

It also contains historical snapshots of the backups going back six months.

NEOSYS SUPPORT TEAM *MUST* AVOID COPYING, MOVING OR RENAMING FOLDERS ON NEOSYS WINDOWS SERVERS because this causes severe duplication and consumption of valuable resources for up to six months on nl19. Any exception to this rule must be discussed with NEOSYS IT *before* implementation.

How to restore?

nl19 is only accessible by NEOSYS IT so recovery of any data cannot be done directly by NEOSYS support team and is not a routine procedure.

The secondary NEOSYS backup server hosts2 is accessible to NEOSYS support team but contains only the latest snapshot of the current NEOSYS hosted clients as at last night. No historical versions are available.

What snapshots are available?

As of 2017/05/25 the NEOSYS backup server nl19 takes snapshots (of the backups listed in Nagios) as follows using crontab:

• 120 historical hourly backups done at **:10 every hour (5x24 hours)
• 8 historical daily backups done at 01:11 every night
• 5 historical weekly backups done at 01:12 every Saturday
• 6 historical monthly backups done at 01:13 on the 1st of every month

What is backed up?

Inspecting Nagios shows daily backups of NEOSYS hosted clients to nl19 of win3/d. Complete hourly backup is not possible due to limitations of Windows and/or the backup software used. However, there are hourly backups of win3/d/DATA.BAK so NEOSYS process "backups" to win3/d/DATA.BAK (on the same server and disk and therefore not a proper backup) do get properly backed up to a different server once an hour.