Backup and Restore
Backup Procedures
Preparing nightly backup report
- Note the success, failure and other error of the clients backup mail in an excel sheet and forward the same to your manager.
- If there is a backup failure or backup is not available, check wiki to take necessary steps.
- If there is any unknown error, forward the same to your manager.
Send the backup report after every quarter to all the Clients having consolidated backups.
Updating Nagios incase of failures
- If the backup failure is unsolved, schedule downtime Neosys service in Nagios till 01 am.
- If the backup did not happen because of server down. Call the IT person; ask him to re-boot the server and check wiki to do necessary step ahead and schedule downtime to Nagios for 2hours.
- If there is an error "Backup->Impossible" on Nagios, check if the USB is properly inserted. If no USB is found, send a mail to the user asking to ensure that the backup USB is inserted properly. Schedule downtime to Nagios for 2hours.
Interchange backup USB mail reminder
Basically all the clients have different days to change their backup USB. All the notification can be seen on Nagios at 12.00 pm every day.
On the USB change day, at 6:00 am, when the processes start up, the system automatically sends the following email to everyone in the backup email receiver list:
It is time to change the NEOSYS backup media (e.g. USB Flash Drive) Please change it before 12:00 midday today.
Additional emails reminding IT staff to change backup media are sent out at 11:30am and 5pm if the backup media is not changed according schedule.
Since the system automatically sends a USB change reminder email to the client, support staff do not have to send them any instructions about changing unless they have failed to change the USB on the scheduled day or the scheduled day needs to be moved to another day.
Importance of interchanging backup USBs
If the backup USB is not interchanged on the scheduled day then the NEOSYS automated backup fails. This happens because traditionally, each USB holds backup of 7 days and using 3 different USBs we can store backups for the last 21 days enabling us to restore the system unto a time period beginning 21 days prior. If the USB is not changed then the first backup on the current USB is replaced with the new or latest backup leading to inconsistencies within the backups. Hence we must interchange the USB on schedule to avoid a backup failure the next morning.
The reasons for using multiple USBs for backup are:
- We can keep other USBs out of the office for safety purposes since theft or office fire/water hazards could damage the computer and the USB keys if they are all in
the same place.
- Having multiple USBs provide safety against corrupt USBs which cannot be used to restore any backup data.
Sample Response:
The NEOSYS backup has failed today as the USB was not interchanged on the scheduled day. Please interchange the USB immediately today to avoid a backup failure tomorrow morning. Also note that your scheduled day to interchange the USB next week and hence forth remains unchanged to '''(Scheduled Day)'''.
Finding out which USB is inserted into the server
As we ask the client to have 3 USB's and interchange them weekly, we also need to sometimes track which one of these 3 USB's are inserted into the server. USB's can be tracked using their volume serial number in most cases. To find this out either go to the command prompt and type VOL <drive letter> where drive letter is the USB drive letter OR in the nightly backup message check for the 2nd line (which looks like this - 14/12/2009 2:45pm Media: 705B-5B5F). However serial numbers can be the same even for different USB's. One of the reasons for this could be that the USBs were imaged from one single USB which caused their volume serial numbers to be the same. However, such a situation is very rare.
Interchanging USB when scheduled USB change day falls on a holiday
When clients ask support which day to interchange USBs when their scheduled USB change day falls on a holidays, send them the below email.
Hi XXX, Kindly interchange the USB on the last working day before the holiday i.e XXXday DD/MM/YY and then interchange with the next scheduled USB on first working day after the holiday i.e. XXXday DD/MM/YY. Continue to change the USBs as usual on XXXday.
Description of Backup Procedure for the NEOSYS Client Hosting Server
NEOSYS backup is a two phase process. It is mandatory that both phases are complete for the process to be considered a backup.
- On the main NEOSYS server host(Schuberg Philis in Amsterdam as of Dec 2010)
Every night, the NEOSYS process for selected databases stops serving users at around 1:00 AM to 5:00 AM (local time of the client's main office) and performs the following procedure:
- All the database files are checked for physical corruption.
- All the database files are zipped into a file who's path would be something like C:\DATA.BAK\CLIENTX\MONDAY\BACKUP.ZIP file.
- The file having the same name as that of the corresponding day of the week as last week is overwritten.A "backup alert" email indicating success or failure is sent to senior managers at the client, the client’s IT manager and backups@neosys.com.
- An internal log (accessible on each database's Support Menu) is maintained as a record and in case the emails cannot be sent out for some reason.
- NEOSYS support staff in Dubai check the emails every morning Sunday through Thursday.
- In the case of failure, NEOSYS support staff in Dubai take appropriate action and send an email (to the same people who receive the automated backup alert emails) indicating what action has been taken.
- The above does not by itself constitute a proper backup because the backup is stored on the same server and physical disk as the actual data. - On the NEOSYS backup server host (in NEOSYS Dubai Office as of Dec 2010 but could be changed without notice to EvoSwitch in Amsterdam).
- At around 02:00-02:30 AM GMT daily, NEOSYS's backup server copies (replicates) all the BACKUP.ZIP files from the main NEOSYS server host to itself.
- An email indicating success or failure is sent to NEOSYS support staff (only) who deal with it in a similar way to phase 1, except that only NEOSYS management are updated.
Manual Backup
See How to do a manual backup in NEOSYS
Manual backups can be triggered on the server process screen by pressing the lower case letter b key. After the backup is complete the process will resume and restart the child processes. Pressing capital B will result in the process closing on completion of the backup and the child processes will not be restarted. This is useful if you want to trigger a backup and leave without waiting for it to complete.
Configuring NEOSYS automated backup
NEOSYS is designed to do an automated backup of the data (not the program files). The configuration for the automated backup rests in the System Configuration File
under Menu > Support.
First step is to configure the automated email settings. After every backup attempt NEOSYS will send out a mail with the status of the backup (either success or
failed). You have to configure the SMTP details here along with the list of recipients to the backup email. Ensure to have atleast 1 person from the client management
and 1 person from the client IT, besides the mandatory backups@neosys.com address
Always use the NEOSYS SMTP server details as follows and put in a fictitious client address so that NEOSYS staff can quickly identify which client this mail came from.
Do not configure the backup time unless instructed by your manager. The system will automatically start the backup at 1 am if there is nothing configured in the Backup
time.
Type in the USB drive letter in the Backup Drive field and the Uploads fields. Putting a 0 in the Uploads field will disable the backup of the Uploaded files, else by default the uploaded files will be backed up to the drive mentioned in the backup drive field.
The scheduled days of backup can be configured using the checkboxes. If none of the days are checked, backup will be done on all days by default.
Next setup the auto-start of the databases and enter how many processes should be started and which all databases should be backed up. In the example below DEMO
database has been configured to start 3 processes and also do the automated backup, whereas the DEMOTEST database is configured to start 1 process and no automated
backup.
Non-Liability for Backup
If a NEOSYS client has not signed the standard NEOSYS contract which excludes liability then NEOSYS needs a specific agreement to the following:
Dear xxxxxxxx, NEOSYS has offered to setup a nightly backup of your NEOSYS data. We can only do this on condition that you will not hold NEOSYS liable for damages of any kind in the case that our backup procedure fails to meet its essential purpose in an emergency. You may make your own alternate parallel arrangements to ensure that the whole of the contents of the NEOSYS server are backed up sometime between 3am and 6am at night when NEOSYS system is shutdown. Please confirm your agreement. Best Regards, xxxx xxxx NEOSYS
Backing up the Images folder
The Images folder under the NEOSYS installation is used to upload images/artworks/files from the Job File section and hence needs to be backed up. NEOSYS will
automatically backup this Images folder to the USB drive or other location (specified for the usual data backup) once a week. To configure this backup, RSYNC needs to
have been installed during the initial installation.
In case we need to configure the Images folder backup at another location other than the usual nightly backup location than we need to edit:
line 12 - and specify the Drive of the location to be backed up to. eg. E or F
Backup to other media (i.e. not to USB)
If the backup is going to non-removable media (even if it is a shared folder on another computer) then the NEOSYS user/client/licensee (NOT the NEOSYS support
team) can, at their own responsibility, arrange to move (NOT COPY) the NEOSYS backup files from the USB location to a backup location of their choice and avoid the
WARNING message : "Backup media not changed. Overwriting last weeks backup"
Sample alternative response to client requests for additional backups
The existing NEOSYS backup must continue to take place for safety because it is the only well understood standard, controlled and checked procedure in use for all
NEOSYS clients.
You are free to setup and operate any additional backup procedure you like but NEOSYS cannot take any responsibility in setting up, monitoring or approving your
additional backup procedure because it is beyond our sphere of control, expertise and trust.
What you can backup is the NEOSYS backups on the USB drive. These are readily available online at all times on the NEOSYS server for you to access and copy as you
choose.
You can backup the usb anytime using anything you like, but you must not backup anything on any hard disk eg: C or D at any time.
Backup to virtual server
VMware
If the virtual server is a VMware. Backup can be configured on a USB.
VMware website says that when changing the USB, the VMware window must be the active window, so that the USB reflects in the virtual machine and not on the host machine. https://www.vmware.com/support/ws45/doc/devices_usb_ws.html
Backups can be configured to happen on the inserted USB.
3 USBs can be maintained and rotate on a weekly basis. Refer to 3 USB rotate mail to clients
Email for new clients with virtual server, or old clients moving to virtual server
If we install NEOSYS on a virtual server, backup cannot be done onto removable backup media because the USB drive will have to be mapped to the virtual server, and mapped drives are not reliable. Instead, NEOSYS will be configured to backup onto C drive. Your IT team can, at your own responsibility, arrange to move (NOT COPY) these backup files to an external backup location every day. We recommend that the backup location should be removable media (eg. external harddisk) and not non-removable media (eg. shared folder on another computer). If any damage happens to the NEOSYS server, the backup files on C drive will not be accessible and will therefore be useless. So, kindly confirm that your IT team will arrange to move the backup files as mentioned above. In addition to this, offsite backups of your virtual server can be setup by us if required, with an additional cost.
Backup Messages
Interpreting backup alerts messages
Success messages come with a blank body
Failure messages should come with the errors in the body
Both success and failure messages come with an attachment containing a list of files copied (or supposed to be copied but failed) and a summary of the volume and speed
of data transferred
Cause and Solution of Backup Warning Messages
NEOSYS gives a backup WARNING message whenever it discovers that it is overwriting backups a week old. It does this because NEOSYS only deems backups completely
successful if the backup media is rotated at least on a weekly basis.
If the backup media is removable then it should be changed daily or weekly and the same media not used repeatedly on successive weeks.
If the backup is going to non-removable media (even if it is a shared folder on another computer) then the NEOSYS user/client/licencee (not the NEOSYS support team)
can, at their own responsibility, arrange to move (not copy) the NEOSYS backup files from that location to a backup location of their choice and avoid this WARNING
message.
Handling failure and warning messages on nightly USB backup alerts
Backup mail not received
Troubleshooting email not received
Handling Change Backup message if the client does not use a USB backup device
IF THE CLIENT IS BACKING UP TO A NON-REMOVABLE DESTINATION THEN EITHER 1. LIVE WITH THE FAILURE MESSAGE OR 2. SUPPRESS THE BACKUP ENTIRELY IN WHICH CASE NO BACKUPS ARE BEING DONE AND PROBABLY THIS WILL SHOW ON NEOSYS PROACTIVE WARNING SYSTEMS LIKE NAGIOS.
DO *NOT* FOLLOW BELOW PROCEDURE SINCE IS IT IS TOTALLY UNACCEPTABLE FOR NEOSYS TO SEND OUT A "BACKUP SUCCESS" MESSAGE WHEN THE BACKUP IS NOT BEING DONE PROPERLY. IN GENERAL, TAKING SHORT CUTS THAT MAKE THINGS APPEAR SATISFACTORY WHEN THEY ARE NOT IS VERY POOR POLICY.
- Basically most of the client use USB for Neosys server but there are some clients who do not use USB. They could be on virtual server or just saving their backups in hard drive.
- As we know that backup take place in Data.bak folder but it does a backup only for a week in a single USB. Over here client do not use USB but save their backup in hard drive/virtual server location in Data.bak folder, once the week is over it will again give Change backup message.
- In this case rename the Data.bak folder to Data1.bak this is done because system read only Data.bak folder as it is configured in the system. When you rename Data.bak to Data1.bak it automatically create a new folder Data.bak in Backup drive.
- This process is done so that the backup does not fail next day morning and backup is done in Data.bak folder along with successful backup email.
- In the third week we will again see the same message Chang backup USB message, so you need to rename Data.bak folder to Data2.bak
- But in fourth week you rename the Data1.bak to Data.bak and change the Data2.bak to Data1.bak and Data.bak to Data2.bak
- Keep only three folders and Data.bak, Data1.bak, Data2.bak and keep on renaming these folders every week as shown above.
AS MENTIONED EARLIER *DO NOT FOLLOW* THE ABOVE PROCEDURE ON ANY CLIENT THAT DOES NOT USE A USB BACKUP DEVICE.
Warning Message: Backup media not changed. Overwriting last weeks backup
NEOSYS gives a backup WARNING message whenever it discovers that it is overwriting backups a week old. It does this because NEOSYS only deems backups completely
successful if the backup media is rotated at least on a weekly basis.
If the backup media is removable then it should be changed daily or weekly and the same media not used repeatedly on successive weeks.
If the backup is going to non-removable media (even if it is a shared folder on another computer) then the NEOSYS user/client/licensee (not the NEOSYS support team)
can, at their own responsibility, arrange to move (not copy) the NEOSYS backup files from that location to a backup location of their choice and avoid this WARNING
message.
Error Message: Somebody else was using the dataset
Server=NEOSYS Client=NEOSYS User=ADAGENCY NEOSYS Ver:18:02:13 30 MAY 2006 Backup started at 19/11/2006 2:00am TEST DATA TESTDATA To E:\DATA.BAK\TESTDATA\Sunday\BACKUP.ZIP !!! THE BACKUP HAS FAILED !!! (because one or more files were not backed up - somebody else was using them)
Error Explained
Despite the error message which shows that 'somebody else was using them', it is definitely not correct as NEOSYS shuts down automatically at 2 am (time of the
backup). The actual problem might be that a NEOSYS process must have got stuck and failed to shutdown or a maintenance process was left open by accident.
Action to be taken
Close any maintenance processes that should not have been left open in the first place.
Try to exit normally or kill any other "hung" processes, taking screenshots of any error messages in order to try and prevent the problem happening again.
If not inconvenient to the users consider initiating a manual backup or just check that the next automatic backup works ok.
Error Message: Size Lock
Server=NEOSYS-SERVER Client=NEOSYS-SERVER User=NETSERVICE NEOSYS Ver:19:57:50 08 NOV 2006 Backup started at 19/11/2006 2:00am ADLINE ADLINE To F:\DATA.BAK\ADLINE\Sunday\BACKUP.ZIP The following files had a size lock but have been fixed SHADOW Warning! Process U79068 Other Network Stations are Active! This utility was designed to be run in a single-user mode! Potential Errors Could Occur, If You Proceed to fix the sizelock Values with Other Stations Active. These Errors Include; Invalid SELECT results, Degredation in Network Performance, and other Problems/Concerns which may be specific to your Installation/Application. IT IS STRONGLY RECOMMENDED THAT ALL NETWORK STATIONS BE LOGGED OUT BEFORE PROCEEDING! Do you wish to Proceed at this time?%B% !!! THE BACKUP HAS FAILED !!! (because one or more files were not backed up - somebody else was using them)
Error Explained
Sizelock errors are not critical and do not stop the backup from being performed. Sizelocks are cleared automatically during backup but the backup alert becomes a
warning that the backup was not quite perfect.
Sizelocks during database checking indicate that some files were not properly closed in normal processing but do not indicate corrupt files. Sizelock means that a file
is prevented from expanding and contracting. Expanding and contracting is necessary in order to speed up access to larger numbers of records. Therefore if sizelocks
were to be left uncleared then the file would become slower and slower to access over time.
In the above example the backup was not completed because of another error "files in use".
Action to be taken
If this is an automatic backup then no action is required since the sizelocks are automatically cleared.
During a manual backup then you should press F9 on the sizelock warning screen to confirm that the sizelocks should be cleared.
Sizelocks can only be cleared if no other process is open.
Failure Message : Backup file is 0
16/4/2009 9:43am Creating zip 16/4/2009 9:45am !!! THE DATABASE BACKUP HAS FAILED !!! BECAUSE BACKUP FILE SIZE IS 0
Error Explanation 1
This happens when there is no adequate storage space on the hard-disk which houses the NEOSYS application and data files. When the automated backup is initiated, the
NEOSYS application checks all the data files and creates a zipped file of it which is the databasename.zip (in the above case IMDUB.ZIP). During this zip file
creation, NEOSYS needs additional disk space temporarily and if this is not available then the backup would fail. The IMDUB.ZIP file is created and backed up (as
BACKUP.ZIP under the respective day of the week) to the USB or any other drive specified and is automatically deleted from the drive that it was created in.
Error Explanation 2
This could be due lack of EMSMAGIC.
In case of a Windows Server 2003 installation: Enabling EMS memory on Window 2003
In case of a Windows Server 2008 installation: Installing EMS Magic on Windows 2008
Actions to be taken
- In this case, you need to clear up the hard-disk and create some space. In the above scenario at Initiative there was a folder of 52 GB in the same drive that houses NEOSYS and the available disk space was just 40 MB. So the un-necessary folder was moved out to make space.
- If there is sufficient space available on the hard disk which houses NEOSYS, then consider checking the anti-virus settings and escalate the issue to NEOSYS programmer for debugging.
This is another reason why NEOSYS personnel should put their foot down on getting dedicated servers for NEOSYS as it restrains the client IT staff from using the
NEOSYS server to house data or any other applications besides NEOSYS.
Error Message: Inadequate Storage Space on USB
Error 1
Server=SERVER2 Client=SERVER2 User=ACCOUNTS NEOSYS Ver:11:26:27 26 JUL 2010 7/2/2012 2:00 Backup started F: BASIC 7/2/2012 2:00 Media: FADE-0001 7/2/2012 2:00 Checking for damaged database files 7/2/2012 2:01 Checked OK 1.92MB (+1.00KB) 7/2/2012 2:01 Creating zip 7/2/2012 2:01 Created OK BASIC.ZIP 251KB 7/2/2012 2:02 Copying to F:\DATA.BAK\Tuesday\BACKUP.ZIP 7/2/2012 2:02 Copied OK 7/2/2012 2:02 Verifying zip OK 7/2/2012 2:02 Verified OK 7/2/2012 2:02 Started backup of uploaded files in /cygdrive/E/ to F: 7/2/2012 2:04 neosys/DATA/ACCOUNTS/REV20050.LK neosys/DATA/ACCOUNTS/REV20052.LK 7/2/2012 2:04 rsync: writefd_unbuffered failed to write 4092 bytes [sender]: Connection reset by peer (104) rsync: write failed on "/cygdrive/F/E/neosys/DATA/ACCOUNTS/BATCHES.LK": No space left on device (28) rsync: connection unexpectedly closed (9077 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at /home/lapo/packaging/rsync-3.0.4-1/src/rsync-3.0.4/io.c(632) [sender=3.0.4] 7/2/2012 2:04 Finished backup of uploaded files 7/2/2012 2:04 !!! THE DATABASE BACKUP HAS FAILED !!!
Error Explained
In the above error log we notice the line:
rsync: write failed on "/cygdrive/F/E/neosys/DATA/PT0011A/ACCOUNTS/BATCHES.LK": No space left on device (28)
This is an indication that the USB being used to store backups does not have adequate storage space to save further backups.
This particular message only comes when using rsync to copy all IMAGES (ie files uploaded to NEOSYS) to the USB.
Sometimes, image backup is used to configured to backup the whole of a disk or the whole of data folder for example if there are hundreds of databases that are not all
used everyday.
Error 2
Server=SERVER2 Client=SERVER2 User=ACCOUNTS NEOSYS Ver:11:26:27 26 JUL 2010 7/2/2012 2:00 Backup started F: PT0621 - GABANG HOLDINGS LIMITED 7/2/2012 2:00 Media: FADE-0001 7/2/2012 2:00 Checking for damaged database files 7/2/2012 2:02 Checked OK 2.25MB (+20.0KB) 7/2/2012 2:02 Creating zip 7/2/2012 2:02 Created OK PT0621.ZIP 462KB 7/2/2012 2:02 !!! THE DATABASE BACKUP HAS FAILED !!! BECAUSE NOT ENOUGH FREE SPACE ON BACKUP MEDIA. 462KB REQUIRED BUT ONLY 452KB AVAILABLE
Error Explained
As the last line of the above error log clearly states, there is not adequate free space available on the USB to perform a backup.
In the above case, 462KB is required space while only 452KB is available.
Action to be taken
The simple solution is to purchase a new USB with greater storage space than the existing USB.
Sample Email to be sent to clients:
The automated NEOSYS backup has failed due to insufficient storage space on the USB. Currently you are using a 4GB USB, which is insufficient for storing backups. I recommend that you use a USB with 8 GB or more storage space.
Note - We may also solve this issue by checking the USB for unnecessary files and move/delete them.
Error Message: "Cannot backup/restore because PROCESS1 PROCESS2 (etc) is/are online" message
This can happen for a variety of reasons if any the NEOSYS processes fail to close down at backup time. One could be because NEOSYS processes hang due to software error.
Fixing Network Connectivity Errors for consolidated Backups
There could be multiple reasons why the backup server might not be able to connect with the backed up server.
Error
[Receiver] io timeout after 600 seconds -- exiting rsync error: timeout in data send/receive (code 30) at /home/lapo/package/rsync-3.0.8-1/src/rsync-3.0.8/io.c(140) [Receiver=3.0.8] [Receiver] io timeout after 600 seconds -- exiting rsync error: timeout in data send/receive (code 30) at /home/lapo/package/rsync-3.0.8-1/src/rsync-3.0.8/io.c(140) [Receiver=3.0.8] tried via hostname.support.neosys.com now trying via hostname2.support.neosys.com ssh: Could not resolve hostname hostname2.support.neosys.com: hostname nor servname provided, or not known rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] rsync error: error in rsync protocol data stream (code 12) at /home/lapo/package/rsync-3.0.8-1/src/rsync-3.0.8/io.c(601)
The following tests help in checking why the connectivity was lost between the backup server and the backed up server:
Check whether the server was up
If either of the servers were down, that could be the reason why connectivity was lost between the backup server and the backed up server. If this test fails, proceed to the next step.
Telnet check
telnet <hostname> 19580
<hostname> is the name or IP address of the remote server to connect to.
If telnet is not able to connect to the remote host specified in <hostname> or establish a connection on the <port> specified, it will report an error similar to the following:
C:> telnet fred.plus.com <CR> Connecting To fred.plus.com...Could not open connection to the host, on port 23: Connect failed C:> telnet mail.plus.net 60 <CR> Connecting To mail.plus.com...Could not open connection to the host, on port 60: Connect failed. C:> telnet 60.92.12.56 74 <CR> Connecting To 60.92.12.56...Could not open connection to the host, on port 74: Connect failed.
Autologin
Error
rsync: failed to connect to 127.0.0.1: Connection refused (111) rsync error: error in socket IO (code 10) at /home/lapo/packaging/rsync-3.0.7-1/src/rsync- 3.0.7/clientserver.c(122) [Receiver=3.0.7] tried via hostname.neosys.com now trying via hostname2.neosys.com rsync: failed to connect to 127.0.0.1: Connection refused (111) rsync error: error in socket IO (code 10) at /home/lapo/packaging/rsync-3.0.7-1/src/rsync- 3.0.7/clientserver.c(122) [Receiver=3.0.7]
Error Explained
The automated backup cannot use autologin to access the server being backed up and so the backup fails.
Solution
Check autologin and Run autologin
Error Message: "WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!" message
Error
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the ED25519 key sent by the remote host is SHA256:9i9JH5xh11rEMIgrsax8A7bEKXfKVRiJr9nZllWXdFg. Please contact your system administrator. Add correct host key in /home/administrator/.ssh/known_hosts to get rid of this message. Offending ECDSA key in /home/administrator/.ssh/known_hosts:208 ED25519 host key for [192.168.10.1]:19580 has changed and you have requested strict checking. Host key verification failed. rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1]
Solution to the problem is to remove the host key for the host mentioned in the error message in the known_hosts file.
Run the following command in Cygwin terminal.
ssh-keygen -R [host ip]:port
e.g in the above message for host ip 192.168.10.1 and port 19580 the command is
ssh-keygen -R [192.168.10.1]:19580
Next re run autologin for the host, see Running autologin
Check Scheduled Tasks on the Server
Check if logs of Scheduled Tasks on the server, to see if the remote backup task executed and completed successfully. It is possible that the remote backup scheduled task hung due to some reason and a new instance of the task fails to start up the next day due to the existing instance. To avoid this situation, in Properties of the task, under Settings tab, check "Stop the task if it runs for:" and configure it to less than 24 hours.
Enabling backup servers to access backed-up servers
This relates to a common procedure used by NEOSYS to provide automatic nightly synchronisation/backups between servers for multi-office configurations. For more information see http://itwiki.neosys.com/index.php/Setting_up_remote_backup
Αccording to this procedure, backup processes are actually initiated and run on a "backup server" using a "pull" concept. The "backed-up server" serves the data (usually using an rsync service) to the backup server on request. They do not use a "push" concept. This means that backup servers need to be able to automatically login to the backed-up servers. On most backup servers a NEOSYS cygwin script called autologin.sh has been installed. This helps you configure the backup server and backed-up server. NEOSYS processes on a target server can sometimes reset/ lose autologin codes (particularly when NEOSYS is upgraded) on the target server when it should not actually. This enforces rerunning autologin.sh. You only need to run autologin.sh once per pair of backup and backed-up servers, in the backup server.
Checking if autologin is configured correctly and working or not
On the source system, in cygwin console, type the following, changing port number, administrator and clientname to suit the case.
ssh –p 19580 administrator@clientname.neosys.com
After a few seconds, if it is working properly, it should give you a command prompt on the target system. This indicates that autologin was successful. You may exit using the command:
exit
Otherwise, if it gives some error and in particular if it asks you anything at all, for example “confirm fingerprint?” or “password?” then autologin is not working. You can then type in the following command in cygwin to configure or reconfigure autologin:
./autologin.sh
Running ./autologin.sh
In cygwin on the backup server, you can use any of the following syntax depending on your configuration:
Syntax
./autologin.sh targethostname targetusername sshport
- host and user will be prompted for if omitted
- port will be 19580 if omitted
Example using prompting for parameters
./autologin.sh
Example using default user and port (administrator/19580)
./autologin.sh examplehost.neosys.com
Example with different user (port will be 19580)
./autologin.sh examplehost.neosys.com Administrator
Example with different user and port
./autologin.sh examplehost.neosys.com Administrator 19580
The username is usually administrator for windows server targets.
The hostname can be found in backup email logs or in the CONFIG.CMD file in the backup server.
There are often two alternative hostnames. Try the first one first. If you succeed with the first then there is no need to try the second.
Sometimes one of the host names is based on alternative network access methods like hamachi which uses ip addresses starting with "5." and requires hamachi service to be running in both servers and this sometimes is dependent on being logged in and correct setup of hamachi. You can ping the host names to discover the ip numbers of course.
If Autologin is configured successfully, it logs you in to the backed up server's cygwin/bash command line.
YOU should NOT be required to enter a password. YOU type "exit" to quit the backed up server's command line.
If Autologin is not configured successfully, you might be required to enter a password. In that case, enter the password and follow instructions on screen. The system asks for the password to the target server to transfer an identity file and once again to access the command line of the target server from where you must follow a set of instructions in order to load the identity file properly. Follow the instructions on the screen VERY carefully.
Setting up autologin.sh
- Asks you for the username and hostname
- Copies an identity file from the backup server to the backed up server.
This requires you to enter the administrator or root password. - Logs you in to the backed up server's cygwin/bash command line.
This requires you to enter the password again. - YOU then type a series of commands to install the identity file and finally exit the command console.
The series of commands that you must type are shown on screen
The edit step
The last step will enter the nano editor where you should do the following:
If present, delete the word neosys near the end of the last line of the file. The last bit of the last line will refer to some user and host that you are enabling
access FROM
BE CAREFUL NOT TO TRIGGER THE AUTOMATIC LINE BREAKING IN NANO because line breaking into two or more lines is automatically triggered when you add characters to long
lines. it is not triggered if you just delete characters. If you do cause a line break then quit the editor WITHOUT saving and try again or then use the editor to
rejoin the two lines. Note that pressing Alt+L will turn long line breaking off and on which avoids the problem in the first place.
Press Ctrl+X to exit the editor
Creating/Upgrading autologin.sh if it doesn’t exist or is out of date
nano autologin.sh chmod +x autologin.sh
#!/bin/bash set -e REMOTEHOST=$1 REMOTEUSER=$2 PORT=$3 test $REMOTEUSER || REMOTEUSER=administrator test $PORT || PORT=19580 if [ ! $REMOTEUSER ]; then echo -n "Remote User (blank=administrator)/root?" read REMOTEUSER if [ "$REMOTEUSER" == "" ] ; then REMOTEUSER=administrator ; fi fi if [ ! REMOTEHOST ]; then echo -n "Remote Host? " read REMOTEHOST if [ "$REMOTEHOST" == "" ] ; then exit; fi fi echo PASSPHRASE MUST BE BLANK IF YOU GET ASKED FOR IT! #Generate priv/pub keys in .ssh if not already done #Priv key mustnt be accessible except to owner otherwise wont work. test -f ~/.ssh/id_rsa || \ ssh-keygen -t rsa && \ chmod 600 ~/.ssh/id_rsa echo echo "Logging in to the remote server (enter the pass again)" sed 's/neosys//' ~/.ssh/id_rsa.pub | \ ssh -p $PORT $REMOTEUSER@$REMOTEHOST \ "mkdir -p .ssh ; chmod 700 .ssh ;\ cat >> .ssh/authorized_keys ;\ chmod 644 .ssh/authorized_keys" echo echo "Test automatic login to the remote server" echo " SHOULD NO LONGER ASK FOR PASSWORD." echo " IF SUCCESSFULL, TYPE exit" ssh -p $PORT $REMOTEUSER@$REMOTEHOST
Switching to a backup server
As NEOSYS provides clients with option of backing up their data to a remote NEOSYS server in case of emergencies or server problems, it is crucial you understand the below procedure on how to switch to a backup server in the event of such a situation. Extreme care must be taken when switching over to using a backup server otherwise unnecessary data loss is very likely.
Backup servers are normally switched off and should not be started automatically otherwise there is a serious risk of the client’s staff working on two systems. It is not possible to merge two databases into one database. Before the backup server is enabled the main server must be disabled, and before the main server is re-enabled, the backup server must be disabled again. This can be managed technically without requiring any decision from senior non-technical staff.
However, there are also some potentially hard decisions about unavoidable loss of data versus continued system availability. *** Backup servers should therefore only be started with the written approval of the clients senior staff. A suitable email requesting approval follows.
The following case assumes that the main server has gone down sometime during the working day and that therefore the data on the backup server is out of date. Allowing them to use the backup server therefore implies some loss of data. They may wish to lose the data. They may wish to work on the backup server data and then try to redo the work on the main server once it is restored. There are a variety of options depending on the situation.
If the main server is still functioning AND you are reasonably sure that the database is not damaged (which is perhaps an unlikely situation if you need to use the backup server!), it may be sensible to trigger an additional “backup/sync” process to bring the backup server database up to date with the main server. Before you do this, it is advisable that you take a backup copy of the backup system on the backup server. In this case there would be no data loss in using the backup.
An additional option of providing usage of the backup server in read-only mode so that people can at least access some data is being developed. The backup server could be available continuously at any time in read-only mode. This article would then be related to switching a backup server into main operational mode.
Dear {senior staff} cc {IT staff}
Please note that we can enable the backup server if you wish.
However the data on the backup server is out of date since it is a copy of your main database as at 11/22/33 99:99.
If you wish to allow work to be done on the backup server then any data entered on your main server since the above date will be lost if we
subsequently copy the data on the backup server to the main server.
If, after using the backup server, we do NOT copy the data on the back server to the main server then any data you have entered on the backup
server/database will be lost.
Please confirm a) you want to work on HOSTS2 database and that we should therefore enable it and b) you have disconnected your main server
for the duration.
Best Regards,
xxxxxx xxxxxxx
NEOSYS Support
How to Restore NEOSYS from Backup
- Log into Maintenance Mode
- Go to General > Backup & Data Management
- Select (the 4th option) Restore from disk or diskette
- Select the drive where the latest successful backup.zip file is present and follow the prompts ahead.
While Moving NEOSYS to a new server, after copying D drive the Maintenance does not contain all data sets
After copying D drive, when you open Maintenance if the data sets are not present as in the old server you MUST restore Backup as restore is a verified/quick method of creating data sets. While restoring last nights Backup for missing data sets, the system gives a warning message "Data set is already on the computer" as shown below. You MUST select 2nd option "Overwrite the existing data set" from the options in the warning message. In this way all data sets will be created/saved on the new server.
╔═════════════════════════════════════════════════════════╗ ║ *** WARNING *** ║ ║ TEST /TEST ║ ║ IS ALREADY ON THE COMPUTER ║ ║ IF YOU CONTINUE, THE EXISTING FILES WILL BE DELETED. ║ ║ ║ ║ What do you want to do ? ║ ║───┬─────────────────────────────────────────────────────║ ║ 1│Cancel the restore process ║ ║ 2>Overwrite the existing data set ║ ║ 3│Restore and make a new dataset ║ ╚═════════════════════════════════════════════════════════╝
