Backup and Restore
Backup Procedures
Preparing daily backup report
- Note the success, failure and other error of the clients backup mail in an excel sheet and forward the same to your manager.
- If there is a backup failure or backup is not available, check wiki to take necessary steps.
- If there is any unknown error, forward the same to your manager.
Updating Nagios incase of failures
- If the backup failure is unsolved, schedule downtime Neosys service in Nagios till 01 am.
- If the backup did not happen because of server down. Call the IT person; ask him to re-boot the server and check wiki to do necessary step ahead and schedule
downtime to Nagios for 2hours.
- If there is an error "Backup->Impossible" on Nagios check wiki and schedule downtime to Nagios for 2hours.
Interchange backup USB mail reminder
- Basically all the clients have different days to change their backup USB. All the notification can be seen on Nagios at 12.00 pm every day.
- When you see the change backup notification on Nagios at 12.00 pm, send a mail to the IT person or the person who changes the USB. As you send the mail schedule
downtime for Neosys services for two hours.
- Check the Nagios at 2.00 pm if the backup USB is changed, don't send another mail and stop there. But if backup is still not yet changed, send a another mail and
this time cc the mail to your manager. After sending the mail, schedule downtime for Neosys service for more two hours.
- Now at 4.00 pm if you still find the backup USB not changed. Call the IT person and ask him to change the USB ASAP and schedule downtime for Neosys services for one
hour.
- Now at 5.00 pm check Nagios and see if the backup USB is changed and if not changed. Send the mail to everyone from the backup mail.
- If the backup is still not changed, escalate the issue to your manager.
Importance of interchanging backup USBs
If the backup USB is not interchanged on the scheduled day then the NEOSYS automated backup fails. This happens because traditionally, each USB holds backup of 7 days
and using 3 different USBs we can store backups for the last 21 days enabling us to restore the system unto a time period beginning 21 days prior. If the USB is not changed then the first backup on the current USB is replaced with the new or latest backup leading to inconsistencies within the backups. Hence we
must interchange the USB on schedule to avoid a backup failure the next morning.
The reasons for using multiple USBs for backup are:
- We can keep other USBs out of the office for safety purposes since theft or office fire/water hazards could damage the computer and the USB keys if they are all in
the same place.
- Having multiple USBs provide safety against corrupt USBs which cannot be used to restore any backup data.
Sample Response: The NEOSYS backup has failed today as the USB was not interchanged on the scheduled day. Please interchange the USB immediately today to avoid a backup failure tomorrow morning. Also note that your scheduled day to interchange the USB next week and hence forth remains unchanged to (Scheduled Day).
Finding out which USB is inserted into the server
As we ask the client to have 3 USB's and interchange them weekly, we also need to sometimes track which one of these 3 USB's are inserted into the server. USB's can be
tracked using their volume serial number in most cases. To find this out either go to the command prompt and type VOL or in the nightly backup message check for the
2nd line (which looks like this - 14/12/2009 2:45pm Media: 705B-5B5F). However serial numbers can be the same even for different USB's.
Description of Backup Procedure for the NEOSYS Client Hosting Server
NEOSYS backup is a two phase process. It is mandatory that both phases are complete for the process to be considered a backup.
- On the main NEOSYS server host(Schuberg Philis in Amsterdam as of Dec 2010)
Every night, the NEOSYS process for selected databases stops serving users at around 1:00 AM to 5:00 AM (local time of the client's main office) and performs the following procedure:
- All the database files are checked for physical corruption.
- All the database files are zipped into a file who's path would be something like C:\DATA.BAK\CLIENTX\MONDAY\BACKUP.ZIP file.
- The file having the same name as that of the corresponding day of the week as last week is overwritten.A "backup alert" email indicating success or failure is sent to senior managers at the client, the client’s IT manager and backups@neosys.com.
- An internal log (accessible on each database's Support Menu) is maintained as a record and in case the emails cannot be sent out for some reason.
- NEOSYS support staff in Dubai check the emails every morning Sunday through Thursday.
- In the case of failure, NEOSYS support staff in Dubai take appropriate action and send an email (to the same people who receive the automated backup alert emails) indicating what action has been taken.
- The above does not by itself constitute a proper backup because the backup is stored on the same server and physical disk as the actual data.
- On the NEOSYS backup server host (in NEOSYS Dubai Office as of Dec 2010 but could be changed without notice to EvoSwitch in Amsterdam).
- At around 02:00-02:30 AM GMT daily, NEOSYS's backup server copies (replicates) all the BACKUP.ZIP files from the main NEOSYS server host to itself. - An email indicating success or failure is sent to NEOSYS support staff (only) who deal with it in a similar way to phase 1, except that only NEOSYS management are updated.
Configuring NEOSYS automated backup
NEOSYS is designed to do an automated backup of the data (not the program files). The configuration for the automated backup rests in the System Configuration File
under Menu > Support.
First step is to configure the automated email settings. After every backup attempt NEOSYS will send out a mail with the status of the backup (either success or
failed). You have to configure the SMTP details here along with the list of recipients to the backup email. Ensure to have atleast 1 person from the client management
and 1 person from the client IT, besides the mandatory backups@neosys.com address
Always use the NEOSYS SMTP server details as follows and put in a fictitious client address so that NEOSYS staff can quickly identify which client this mail came from.
Do not configure the backup time unless instructed by your manager. The system will automatically start the backup at 1 am if there is nothing configured in the Backup
time.
Type in the USB drive letter in the Backup Drive field.
Next setup the auto-start of the databases and enter how many processes should be started and which all databases should be backed up. In the example below DEMO
database has been configured to start 3 processes and also do the automated backup, whereas the DEMOTEST database is configured to start 1 process and no automated
backup.
Non-Liability for Backup
If a NEOSYS client has not signed the standard NEOSYS contract which excludes liability then NEOSYS needs a specific agreement to the following:
Dear xxxxxxxx, NEOSYS has offered to setup a nightly backup of your NEOSYS data. We can only do this on condition that you will not hold NEOSYS liable for damages of any kind in the case that our backup procedure fails to meet its essential purpose in an emergency. You may make your own alternate parallel arrangements to ensure that the whole of the contents of the NEOSYS server are backed up sometime between 3am and 6am at night when NEOSYS system is shutdown. Please confirm your agreement. Best Regards, xxxx xxxx NEOSYS
Backing up the Images folder
The Images folder under the NEOSYS installation is used to upload images/artworks/files from the Job File section and hence needs to be backed up. NEOSYS will
automatically backup this Images folder to the USB drive or other location (specified for the usual data backup) once a week. To configure this backup, RSYNC needs to
have been installed during the initial installation.
In case we need to configure the Images folder backup at another location other than the usual daily backup location than we need to edit:
line 12 - and specify the Drive of the location to be backed up to. eg. E or F
Backup to other media (i.e. not to USB)
If the backup is going to non-removable media (even if it is a shared folder on another computer) then the NEOSYS user/client/licensee (NOT the NEOSYS support
team) can, at their own responsibility, arrange to move (NOT COPY) the NEOSYS backup files from that location to a backup location of their choice and avoid the
WARNING message : "Backup media not changed. Overwriting last weeks backup"
Sample alternative response to client requests for additional backups
The existing NEOSYS backup must continue to take place for safety because it is the only well understood standard, controlled and checked procedure in use for all
NEOSYS clients.
You are free to setup and operate any additional backup procedure you like but NEOSYS cannot take any responsibility in setting up, monitoring or approving your
additional backup procedure because it is beyond our sphere of control, expertise and trust.
What you can backup is the NEOSYS backups on the USB drive. These are readily available online at all times on the NEOSYS server for you to access and copy as you
choose.
You can backup the usb anytime using anything you like, but you must not backup anything on any hard disk eg: C or D at any time.
Backup Messages
Interpreting backup alerts messages
Success messages come with a blank body
Failure messages should come with the errors in the body
Both success and failure messages come with an attachment containing a list of files copied (or supposed to be copied but failed) and a summary of the volume and speed
of data transferred
Cause and Solution of Backup Warning Messages
NEOSYS gives a backup WARNING message whenever it discovers that it is overwriting backups a week old. It does this because NEOSYS only deems backups completely
successful if the backup media is rotated at least on a weekly basis.
If the backup media is removable then it should be changed daily or weekly and the same media not used repeatedly on successive weeks.
If the backup is going to non-removable media (even if it is a shared folder on another computer) then the NEOSYS user/client/licencee (not the NEOSYS support team)
can, at their own responsibility, arrange to move (not copy) the NEOSYS backup files from that location to a backup location of their choice and avoid this WARNING
message.
Handling failure and warning messages on nightly backup alerts
Backup mail not received
Troubleshooting email not received
Warning Message: Backup media not changed. Overwriting last weeks backup
NEOSYS gives a backup WARNING message whenever it discovers that it is overwriting backups a week old. It does this because NEOSYS only deems backups completely
successful if the backup media is rotated at least on a weekly basis.
If the backup media is removable then it should be changed daily or weekly and the same media not used repeatedly on successive weeks.
If the backup is going to non-removable media (even if it is a shared folder on another computer) then the NEOSYS user/client/licensee (not the NEOSYS support team)
can, at their own responsibility, arrange to move (not copy) the NEOSYS backup files from that location to a backup location of their choice and avoid this WARNING
message.
Error Message: Somebody else was using the dataset
Server=NEOSYS Client=NEOSYS User=ADAGENCY NEOSYS Ver:18:02:13 30 MAY 2006 Backup started at 19/11/2006 2:00am TEST DATA TESTDATA To E:\DATA.BAK\TESTDATA\Sunday\BACKUP.ZIP !!! THE BACKUP HAS FAILED !!! (because one or more files were not backed up - somebody else was using them)
Error Explained
Despite the error message which shows that 'somebody else was using them', it is definitely not correct as NEOSYS shuts down automatically at 2 am (time of the
backup). The actual problem might be that a NEOSYS process must have got stuck and failed to shutdown or a maintenance process was left open by accident.
Action to be taken
Close any maintenance processes that should not have been left open in the first place.
Try to exit normally or kill any other "hung" processes, taking screenshots of any error messages in order to try and prevent the problem happening again.
If not inconvenient to the users consider initiating a manual backup or just check that the next automatic backup works ok.
Error Message: Size Lock
Server=NEOSYS-SERVER Client=NEOSYS-SERVER User=NETSERVICE NEOSYS Ver:19:57:50 08 NOV 2006 Backup started at 19/11/2006 2:00am ADLINE ADLINE To F:\DATA.BAK\ADLINE\Sunday\BACKUP.ZIP The following files had a size lock but have been fixed SHADOW Warning! Process U79068 Other Network Stations are Active! This utility was designed to be run in a single-user mode! Potential Errors Could Occur, If You Proceed to fix the sizelock Values with Other Stations Active. These Errors Include; Invalid SELECT results, Degredation in Network Performance, and other Problems/Concerns which may be specific to your Installation/Application. IT IS STRONGLY RECOMMENDED THAT ALL NETWORK STATIONS BE LOGGED OUT BEFORE PROCEEDING! Do you wish to Proceed at this time?%B% !!! THE BACKUP HAS FAILED !!! (because one or more files were not backed up - somebody else was using them)
Error Explained
Sizelock errors are not critical and do not stop the backup from being performed. Sizelocks are cleared automatically during backup but the backup alert becomes a
warning that the backup was not quite perfect.
Sizelocks during database checking indicate that some files were not properly closed in normal processing but do not indicate corrupt files. Sizelock means that a file
is prevented from expanding and contracting. Expanding and contracting is necessary in order to speed up access to larger numbers of records. Therefore if sizelocks
were to be left uncleared then the file would become slower and slower to access over time.
In the above example the backup was not completed because of another error "files in use".
Action to be taken
If this is an automatic backup then no action is required since the sizelocks are automatically cleared.
During a manual backup then you should press F9 on the sizelock warning screen to confirm that the sizelocks should be cleared.
Sizelocks can only be cleared if no other process is open.
Failure Message : Backup file is 0
Server=NEOSYS Client= User=NEOSYS NEOSYS Ver:21:34:54 18 FEB 2009 16/4/2009 9:43am Backup started F: IMDUB - IM DUBAI 16/4/2009 9:43am Media: 5CDB-4C92 16/4/2009 9:43am Checking for damaged database files 16/4/2009 9:43am Checked OK 845MB (+704 bytes) 16/4/2009 9:43am Creating zip 16/4/2009 9:45am !!! THE DATABASE BACKUP HAS FAILED !!! BECAUSE BACKUP FILE SIZE IS 0
Error Explained
This happens when there is no adequate storage space on the hard-disk which houses the NEOSYS application and data files. When the automated backup is initiated, the
NEOSYS application checks all the data files and creates a zipped file of it which is the databasename.zip (in the above case IMDUB.ZIP). During this zip file
creation, NEOSYS needs additional disk space temporarily and if this is not available then the backup would fail. The IMDUB.ZIP file is created and backed up (as
BACKUP.ZIP under the respective day of the week) to the USB or any other drive specified and is automatically deleted from the drive that it was created in.
Actions to be taken
In this case, you need to clear up the hard-disk and create some space. In the above scenario at Initiative there was a folder of 52 GB in the same drive that houses
NEOSYS and the available disk space was just 40 MB. So the un-necessary folder was moved out to make space.
This is another reason why NEOSYS personnel should put their foot down on getting dedicated servers for NEOSYS as it restrains the client IT staff from using the
NEOSYS server to house data or any other applications besides NEOSYS.
Error Message: Inadequate Storage Space on USB
Error 1
Server=SERVER2 Client=SERVER2 User=ACCOUNTS NEOSYS Ver:11:26:27 26 JUL 2010 7/2/2012 2:00 Backup started F: BASIC 7/2/2012 2:00 Media: FADE-0001 7/2/2012 2:00 Checking for damaged database files 7/2/2012 2:01 Checked OK 1.92MB (+1.00KB) 7/2/2012 2:01 Creating zip 7/2/2012 2:01 Created OK BASIC.ZIP 251KB 7/2/2012 2:02 Copying to F:\DATA.BAK\Tuesday\BACKUP.ZIP 7/2/2012 2:02 Copied OK 7/2/2012 2:02 Verifying zip OK 7/2/2012 2:02 Verified OK 7/2/2012 2:02 Started backup of uploaded files in /cygdrive/E/ to F: 7/2/2012 2:04 neosys/DATA/ACCOUNTS/REV20050.LK neosys/DATA/ACCOUNTS/REV20052.LK 7/2/2012 2:04 rsync: writefd_unbuffered failed to write 4092 bytes [sender]: Connection reset by peer (104) rsync: write failed on "/cygdrive/F/E/neosys/DATA/ACCOUNTS/BATCHES.LK": No space left on device (28) rsync: connection unexpectedly closed (9077 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at /home/lapo/packaging/rsync-3.0.4-1/src/rsync-3.0.4/io.c(632) [sender=3.0.4] 7/2/2012 2:04 Finished backup of uploaded files 7/2/2012 2:04 !!! THE DATABASE BACKUP HAS FAILED !!!
Error Explained
In the above error log we notice the line:
rsync: write failed on "/cygdrive/F/E/neosys/DATA/PT0011A/ACCOUNTS/BATCHES.LK": No space left on device (28)
This is an indication that the USB being used to store backups does not have adequate storage space to save further backups.
This particular message only comes when using rsync to copy all IMAGES (ie files uploaded to NEOSYS) to the USB.
Sometimes, image backup is used to configured to backup the whole of a disk or the whole of data folder for example if there are hundreds of databases that are not all
used everyday.
Error 2
Server=SERVER2 Client=SERVER2 User=ACCOUNTS NEOSYS Ver:11:26:27 26 JUL 2010 7/2/2012 2:00 Backup started F: PT0621 - GABANG HOLDINGS LIMITED 7/2/2012 2:00 Media: FADE-0001 7/2/2012 2:00 Checking for damaged database files 7/2/2012 2:02 Checked OK 2.25MB (+20.0KB) 7/2/2012 2:02 Creating zip 7/2/2012 2:02 Created OK PT0621.ZIP 462KB 7/2/2012 2:02 !!! THE DATABASE BACKUP HAS FAILED !!! BECAUSE NOT ENOUGH FREE SPACE ON BACKUP MEDIA. 462KB REQUIRED BUT ONLY 452KB AVAILABLE
Error Explained
As the last line of the above error log clearly states, there is not adequate free space available on the USB to perform a backup.
In the above case, 462KB is required space while only 452KB is available.
Action to be taken
The simple solution is to purchase a new USB with greater storage space than the existing USB.
Sample Email to be sent to clients:
The automated NEOSYS backup has failed due to insufficient storage space on the USB. Currently you are using a 4GB USB, which is insufficient for storing backups. I recommend that you use a USB with 8 GB or more storage space.
Note - We may also solve this issue by checking the USB for unnecessary files and move/delete them.
Checking and Configuring Autologin for Consolidated Backups
Error
rsync: failed to connect to 127.0.0.1: Connection refused (111) rsync error: error in socket IO (code 10) at /home/lapo/packaging/rsync-3.0.7-1/src/rsync- 3.0.7/clientserver.c(122) [Receiver=3.0.7] tried via hostname.neosys.com now trying via hostname2.neosys.com rsync: failed to connect to 127.0.0.1: Connection refused (111) rsync error: error in socket IO (code 10) at /home/lapo/packaging/rsync-3.0.7-1/src/rsync- 3.0.7/clientserver.c(122) [Receiver=3.0.7]
Error Explained
The automated backup cannot use autologin to access the server being backed up and so the backup fails.
Solution
Please ensure that you have the remote server hostname and administrator pass before proceeding.
Enabling backup servers to access backed-up servers
This relates to a common procedure used by NEOSYS to provide automatic nightly synchronisation/backups between servers for multi-office configurations. For more
information see http://itwiki.neosys.com/index.php/Setting_up_remote_backup
Αccording to this procedure, backup processes are actually initiated and run on a "backup server" using a "pull" concept. The "backed-up server" serves the data
(usually using an rsync service) to the backup server on request. They do not use a "push" concept.
This means that backup servers need to be able to automatically login to the backed-up servers. On most backup servers a NEOSYS cygwin script called autologin.sh has
been installed. This helps you configure the backup server and backed-up server.
Normally you only need to run autologin.sh once per pair of backup and backed-up servers but sometimes the configuration is lost (particularly when NEOSYS is upgraded)
and it must be repeated.
Using autologin.sh
- Asks you for the username and hostname
- Copies an identity file from the backup server to the backed up server.
This requires you to enter the administrator or root password. - Logs you in to the backed up server's cygwin/bash command line.
This requires you to enter the password again. - YOU then type a series of commands to install the identity file and finally exit the command console.
The series of commands that you must type are shown on screen
by autologin.sh
- Logs you in to the backed up server's cygwin/bash command line.
This time, you should NOT be required to enter a password. - YOU type "exit" to quit the backed up server's command line.
Running ./autologin.sh
In cygwin on the backup server, you can use any of the following syntax depending on your configuration:
Syntax
./autologin.sh targethostname targetusername sshport
- host and user will be prompted for if omitted
- port will be 19580 if omitted
Example using prompting for parameters
./autologin.sh
Example using default user and port (administrator/19580)
./autologin.sh examplehost.neosys.com
Example with different user (port will be 19580)
./autologin.sh examplehost.neosys.com Administrator
Example with different user and port
./autologin.sh examplehost.neosys.com Administrator 19580
Follow instructions on screen.
The system will ask you once the password to the target server to transfer an identity file and once again to access the command line of the target server from where
you must follow a set of instructions in order to load the identity file properly.
The username is usually administrator for windows server targets.
The hostname can be found in backup email logs or in the CONFIG.CMD file in the backup server.
There are often two alternative hostnames. Try the first one first. If you succeed with the first then there is no need to try the second.
Sometimes one of the host names is based on alternative network access methods like hamachi which uses ip addresses starting with "5." and requires hamachi service to
be running in both servers and this sometimes is dependent on being logged in and correct setup of hamachi. You can ping the host names to discover the ip numbers of
course.
Follow the instructions on the screen VERY carefully.
Checking if autologin is configured correctly and working or not
On the source system, in cygwin console, type something like the following, changing port number, administrator and clientname to suit the case.
ssh –p 19580 administrator@clientname.neosys.com
After a few seconds, if it is working properly, it should give you a command prompt on the target system. This indicates that autologin was successful. You may exit
using the command:
exit
Otherwise, if it gives some error and in particular if it asks you anything at all, for example “confirm fingerprint?” or “password?” then autologin is not working.
You can then type in the following command in cygwin to configure or reconfigure autologin:
./autologin.sh
The edit step
The last step will enter the nano editor where you should do the following:
If present, delete the word neosys near the end of the last line of the file. The last bit of the last line will refer to some user and host that you are enabling
access FROM
BE CAREFUL NOT TO TRIGGER THE AUTOMATIC LINE BREAKING IN NANO because line breaking into two or more lines is automatically triggered when you add characters to long
lines. it is not triggered if you just delete characters. If you do cause a line break then quit the editor WITHOUT saving and try again or then use the editor to
rejoin the two lines. Note that pressing Alt+L will turn long line breaking off and on which avoids the problem in the first place.
Press Ctrl+X to exit the editor
Creating/Upgrading autologin.sh if it doesn’t exist or is out of date
nano autologin.sh chmod +x autologin.sh
#!/bin/bash set -e REMOTEHOST=$1 REMOTEUSER=$2 PORT=$3 test $REMOTEUSER || REMOTEUSER=administrator test $PORT || PORT=19580 if [ ! $REMOTEUSER ]; then echo -n "Remote User (blank=administrator)/root?" read REMOTEUSER if [ "$REMOTEUSER" == "" ] ; then REMOTEUSER=administrator ; fi fi if [ ! REMOTEHOST ]; then echo -n "Remote Host? " read REMOTEHOST if [ "$REMOTEHOST" == "" ] ; then exit; fi fi echo PASSPHRASE MUST BE BLANK IF YOU GET ASKED FOR IT! #Generate priv/pub keys in .ssh if not already done #Priv key mustnt be accessible except to owner otherwise wont work. test -f ~/.ssh/id_dsa || \ ssh-keygen -t dsa -b 1024 -f ~/.ssh/id_dsa -N "" && \ chmod 600 ~/.ssh/id_dsa echo echo "Logging in to the remote server (enter the pass again)" sed 's/neosys//' ~/.ssh/id_dsa.pub | \ ssh -p $PORT $REMOTEUSER@$REMOTEHOST \ "mkdir -p .ssh ; chmod 700 .ssh ;\ cat >> .ssh/authorized_keys ;\ chmod 644 .ssh/authorized_keys" echo echo "Test automatic login to the remote server" echo " SHOULD NO LONGER ASK FOR PASSWORD." echo " IF SUCCESSFULL, TYPE exit" ssh -p $PORT $REMOTEUSER@$REMOTEHOST
