Procedures
Here are procedures to be followed by Support Staff in respect to various technical matters in day to day operations of client issues.
NEOSYS Support Policies
In order to maintain good payment speed by clients NEOSYS needs to restrict support to clients that dont pay their bills on time, however the degree of restriction needs to depend on an intimate knowledge of the client which cannot be expected from all NEOSYS support staff. Therefore we will use a simple escalation policy as follows:
NEOSYS SUPPORT MANAGERS WILL maintain an overdue/stop list on a whiteboard visible to all support staff. Generally clients will go on the list when their invoice is seven days overdue and come off only after satisfactory commitment to pay have been obtained.
NEOSYS SUPPORT STAFF WILL discretely refer any calls for support from client on the overdue/stop list to support managers for handling.
This policy excludes:
- Contacting the client to change the USB disk on the scheduled day, in case they haven't done so by 12 pm
- Contacting the client or if the client contacts support staff regarding backup failures and to take measures to fix these failures
Client Password Policy
All client user passwords, including their initial one, are to be obtained via the user's email address using the password reminder/reset button on the login screen.
NEOSYS staff should never know users passwords therefore NEOSYS will not obtain and grant user passwords.
Support requests from ordinary client users
Any support requests concerning inability to obtain passwords will be forwarded to known skilled users on the client staff since this is the most efficient (not fastest) way to handle such issues.
Support requests from senior client management
Any support requests concerning inability to obtain passwords by senior client management users shall be handled directly by NEOSYS support staff in any way convenient to resolve the issue in the quickest possible time rather than the most efficient.
Bearing in mind that NEOSYS staff should never know user's passwords this will probably involve NEOSYS staff using the Password Reminder/Reset button to send a new password to the user.
User Defined Passwords
NEOSYS will provide user defined passwords in very special cases which must be pre-approved case by case by NEOSYS management. For many reasons, NEOSYS will not approve.
Currently this permission has only been granted to one NEOSYS client with several hundred of databases.
Handling client issues
Handling error messages
- The very first step is understanding client problem.
- Ask the client what error does he gets on the screen.
- If error seems to be familiar then resolve it over the phone.
- If error is unknown then ask user to send a screenshot of the error displayed along with the options used (basically you need to know HOW to replicate the error)
- Upon receipt of the error, check in all the wikis for a solution.
- If the issue is unknown or you don’t understand it clearly ask the user then use remote support to gain access to the users desktop to view how to replicate the error.
If it is new issue then report by escalation the same issue to your manager with a brief explanation.
Handling problems with report totals
Questions with regards to report totals should not be escalated to the programmers without following this procedure. If the total can be be broken down into items (as they nearly always can be in NEOSYS) then only a problem where the items dont add up to a total should be escalated, since that would be a system error.
Handling Nagios Client Monitoring system
Nagios is configured to display information pertaining to all NEOSYS client's server statuses which include multiple services such as:
- HTTPS: Most of NEOSYS clients are configured to have external web access via secure HTTP protocol (port 4430) from outside office. Nagios is configured to check port 4430 on a regular interval of 10 minutes and display any issues in accessing the same.
- SSH: As part of the support contract, NEOSYS should have external secure access to the client server usually over port 19580. Nagios is configured to check this port on a regular interval of 10 minutes and display any issues in accessing the same.
- Ping: Nagios is also configured to ping the client router as a measure to check if router responds incase the NEOSYS server is down.
- NEOSYS: This service works in a reverse direction, and the NEOSYS installation on the client server sends information such as databases running, current backup status, internal and internet IP addressess etc to Nagios on a regular interval of 10 minutes.
Some key information about Nagios is as follows:
- Nagios is also configured to display information related to internal servers.
- Clients hosted on a NEOSYS cloud server might not have services such as SSH or PING as this is monitored as part of the internal server service.
- Nagios sends out email alerts to support2@neosys.com (which is forwarded to support@neosys.com) from 8 am to 12 midnight on all Dubai working days (Sun-Thu). No alerts are sent out on Fri and Sat, unless they are for NEOSYS internal servers.
NEOSYS support staff on duty has to follow the below outlined procedures in case of any Nagios items showing a critical or warning message for any service:
- Nagios is required to be checked first thing in the morning and any critical or warning messages need to be dealth with to resolve the same at the earliest.
- Some of the messages could be related to backup failures and the usual procedure as stated in #Backup Procedures needs to be followed. In case the backup issue isn't resolved by 9:30 am, the Nagios service needs to be scheduled with downtime for a minimum of 2 hours and maximum until 1 am next day if the issue cannot be solved.
- In case any HTTPS, SSH or PING service is down, immediate action is required and the relevant IT people at the client side needs to be contacted to get this resolved. A downtime of 2 hours is required to be scheduled with further intervals of 2 hours incase this is not resolved. Support staff shouldn't schedule downtime till 1 am next day, just to get rid of the alerts for the day. Proactive follow up with the client is required to get this resolved before the business day - more so, if there is a weekend ahead.
- In case the HTTPS, SSH or PING service goes down during the day, a grace period of 20 minutes is given before the issue is reported to the client IT. This helps incase there is any temporary internet connection issue at the client or along the internet route.
- In case of "Backup not changed" warning status which occurs if the client has not interchanged the USB before 12 noon on that day, no action is required from the support staff and a downtime until 1 am next day needs to be scheduled.
- PLEASE NOTE: Failure to schedule appropriate downtime will lead to REDUNDANT ALERTS from NAGIOS every hour.
What happens if there is no remote access to NEOSYS server based in client’s premises ?
If access to the NEOSYS server is lost then we must determine the root cause by:
- Checking if the server is UP and running
- If yes, please check internet connectivity on the server
- If there is connectivity, please check the router for connectivity issues
** Do keep NEOSYS support team posted on the server status so we can test connectivity from our side as well and resolve the issue **
What happens if there is no remote access to NEOSYS server based in client’s premises ?
If access to the NEOSYS server is lost then we must determine the root cause by:
- Checking if the server is UP and running
- If yes, please check internet connectivity on the server
- If there is connectivity, please check the router for connectivity issues
** Do keep NEOSYS support team posted on the server status so we can test connectivity from our side as well and resolve the issue **
New Router (Port Forwarding)
If you have changed your router then you may notice that external access to NEOSYS is unavailable.
Solution:
Setup a permanent access for NEOSYS by reconfiguring the Router / Firewall for Port Forwarding from Router to the NEOSYS Server as follows:
- Port 19580 > 19580 for SSH
- Port 4430 > 4430 for HTTPS
** Once this is complete, kindly send an email to the NEOSYS support team for confirmation so that we could test connectivity from our end as well **
Backup Procedures
Preparing daily backup report
- Note the success, failure and other error of the clients backup mail in an excel sheet and forward the same to your manager.
- If there is a backup failure or backup is not available, check wiki to take necessary steps.
- If there is any unknown error, forward the same to your manager.
Updating Nagios incase of failures
- If the backup failure is unsolved, schedule downtime Neosys service in Nagios till 01 am.
- If the backup did not happen because of server down. Call the IT person; ask him to re-boot the server and check wiki to do necessary step ahead and schedule downtime to Nagios for 2hours.
- If there is an error "Backup->Impossible" on Nagios check wiki and schedule downtime to Nagios for 2hours.
Interchange backup USB mail reminder
- Basically all the clients have different days to change their backup USB. All the notification can be seen on Nagios at 12.00 pm every day.
- When you see the change backup notification on Nagios at 12.00 pm, send a mail to the IT person or the person who changes the USB. As you send the mail schedule downtime for Neosys services for two hours.
- Check the Nagios at 2.00 pm if the backup USB is changed, don't send another mail and stop there. But if backup is still not yet changed, send a another mail and this time cc the mail to your manager. After sending the mail, schedule downtime for Neosys service for more two hours.
- Now at 4.00 pm if you still find the backup USB not changed. Call the IT person and ask him to change the USB ASAP and schedule downtime for Neosys services for one hour.
- Now at 5.00 pm check Nagios and see if the backup USB is changed and if not changed. Send the mail to everyone from the backup mail.
- If the backup is still not changed, escalate the issue to your manager.
Note: If the USB is not interchanged on the scheduled day i.e. then The NEOSYS Automated backup will fail. Traditionally, each USB holds backup of 7 days and using 3 different USBs we can store backups for the last 21 days enabling us to restore the system unto a time period beginning 21 days prior. If the USB is not changed then the first backup on the current USB is replaced with the new or latest backup leading to inconsistencies within the backups. Hence we must interchange the USB on schedule to avoid a backup failure the next morning.
Finding out which USB is inserted into the server
As we ask the client to have 3 USB's and interchange them weekly, we also need to sometimes track which one of these 3 USB's are inserted into the server. USB's can be tracked using their volume serial number in most cases. To find this out either go to the command prompt and type VOL or in the nightly backup message check for the 2nd line (which looks like this - 14/12/2009 2:45pm Media: 705B-5B5F). However serial numbers can be the same even for different USB's.
Client relocating operations to a new premises
Ensure that the following procedure is followed by the clients IT staff prior to relocation:
- Login to the NEOSYS server and shut down all the processes by hitting the ESC button on each of the window three times or until they close
- Take a backup of the D drive on another system or network / external drive
- Remove the USB plugged in and secure the same along with the other two USB’s in a safe location
- The NEOSYS server may now be shutdown and moved to the new premises
Post relocation, the clients IT staff must ensure the following immediately:
- Connect the server to the local network and complete all setup required for users to be able to access the system internally
- Configure the firewall and/or port settings to enable NEOSYS support staff to connect to the server externally
- Provide NEOSYS support with the new fixed IP address so that the connection can be tested.
Creating and Handling passwords
Passwords made out of a pass phrase should be at least 10 characters since using initials results in a lot of i's and a's etc which reduces the effectiveness of the password and allows hacking via brute force guessing especially since windows doesnt slow down logins even if it sees thousands of password attempts.
Creating a password
Passwords are generated from a pass phrase and it is important to create a very difficult to guess pass phrase.
For example, a good pass phrase would be: Today is a good day and it is the best time to go for a holiday
The password for this would be Tiagdaiitbt2g4ah
The important instructions for the above are:
- You have to take the first letter of each word and that makes your password (i.e. by using initials)
- Wherever any word starts with a capital, then you have to take first letter as a capital (eg. For Today you will take T)
- Replace and with &
- Replace to with 2
- Replace for with 4
Handling passwords
- Never send the actual password - always send the pass phrase
- Make sure that the password created out of the pass phrase is at least 10 characters long since using initials results in a lot of i's and a's etc which reduces the effectiveness of the password and allows hacking via brute force guessing especially since windows doesnt slow down logins even if it sees thousands of password attempts
- Pass phrases are never to be sent by email, whatever the case maybe.
- Pass phrases can be sent by chat - however they have to be broken down in two parts and sent separately over two different messengers or if you are using Gtalk then use the 'off the record' mode.
- Using SMS to send pass phrases is the best known way as of now.
- If you save the passwords on your system as a file, make sure the file is encrypted with a master password.
NEOSYS Maintenance Window
The NEOSYS server is functional from 6am – 1am. There is a 5hr window gap for the system to perform updates & backups.
The 5hr maintenance window:-
1. At 1am – The server performs a data backup on a USB (for the respective clients) & once the backup has been completed, the system automatically generates an email addressed to the neosys staff & the respective clients.
2. At 2:45am – The main data over writes the test data on the server.
3. At 3:00am – The server by itself performs an update for Windows.
4. At 4:00am – The server performs a backup to the headquarters for clients, and then automatically generates an email addressed to the NEOSYS staff & the respective clients. This process is followed by only two clients i.e. Adline & Farouk.
5. At 6:00am – The server starts up NEOSYS.
Switching to a backup server
As NEOSYS provides clients with option of backing up their data to a remote NEOSYS server in case of emergencies or server problems, it is crucial you understand the below procedure on how to switch to a backup server in the event of such a situation. Extreme care must be taken when switching over to using a backup server otherwise unnecessary data loss is very likely.
Backup servers are normally switched off and should not be started automatically otherwise there is a serious risk of the client’s staff working on two systems. It is not possible to merge two databases into one database. Before the backup server is enabled the main server must be disabled, and before the main server is re-enabled, the backup server must be disabled again. This can be managed technically without requiring any decision from senior non-technical staff.
However, there are also some potentially hard decisions about unavoidable loss of data versus continued system availability. *** Backup servers should therefore only be started with the written approval of the clients senior staff. A suitable email requesting approval follows.
The following case assumes that the main server has gone down sometime during the working day and that therefore the data on the backup server is out of date. Allowing them to use the backup server therefore implies some loss of data. They may wish to lose the data. They may wish to work on the backup server data and then try to redo the work on the main server once it is restored. There are a variety of options depending on the situation.
If the main server is still functioning AND you are reasonably sure that the database is not damaged (which is perhaps an unlikely situation if you need to use the backup server!), it may be sensible to trigger an additional “backup/sync” process to bring the backup server database up to date with the main server. Before you do this, it is advisable that you take a backup copy of the backup system on the backup server. In this case there would be no data loss in using the backup.
An additional option of providing usage of the backup server in read-only mode so that people can at least access some data is being developed. The backup server could be available continuously at any time in read-only mode. This article would then be related to switching a backup server into main operational mode.
Dear {senior staff} cc {IT staff}
Please note that we can enable the backup server if you wish.
However the data on the backup server is out of date since it is a copy of your main database as at 11/22/33 99:99.
If you wish to allow work to be done on the backup server then any data entered on your main server since the above date will be lost if we
subsequently copy the data on the backup server to the main server.
If, after using the backup server, we do NOT copy the data on the back server to the main server then any data you have entered on the backup
server/database will be lost.
Please confirm a) you want to work on HOSTS2 database and that we should therefore enable it and b) you have disconnected your main server
for the duration.
Best Regards,
xxxxxx xxxxxxx
NEOSYS Support
Description of Backup Procedure for the NEOSYS Client Hosting Server
NEOSYS backup is a two phase process. It is mandatory that both phases are complete for the process to be considered a backup.
- On the main NEOSYS server host(Schuberg Philis in Amsterdam as of Dec 2010)
Every night, the NEOSYS process for selected databases stops serving users at around 1:00 AM to 5:00 AM (local time of the client's main office) and performs the following procedure:
- All the database files are checked for physical corruption.
- All the database files are zipped into a file who's path would be something like C:\DATA.BAK\CLIENTX\MONDAY\BACKUP.ZIP file.
- The file having the same name as that of the corresponding day of the week as last week is overwritten.A "backup alert" email indicating success or failure is sent to senior managers at the client, the client’s IT manager and backups@neosys.com.
- An internal log (accessible on each database's Support Menu) is maintained as a record and in case the emails cannot be sent out for some reason.
- NEOSYS support staff in Dubai check the emails every morning Sunday through Thursday.
- In the case of failure, NEOSYS support staff in Dubai take appropriate action and send an email (to the same people who receive the automated backup alert emails) indicating what action has been taken.
- The above does not by itself constitute a proper backup because the backup is stored on the same server and physical disk as the actual data. - On the NEOSYS backup server host (in NEOSYS Dubai Office as of Dec 2010 but could be changed without notice to EvoSwitch in Amsterdam).
- At around 02:00-02:30 AM GMT daily, NEOSYS's backup server copies (replicates) all the BACKUP.ZIP files from the main NEOSYS server host to itself.
- An email indicating success or failure is sent to NEOSYS support staff (only) who deal with it in a similar way to phase 1, except that only NEOSYS management are updated.
Cutting and Pasting NEOSYS Maintenance Mode Messages into Wiki
Error messages expressed as images are not searchable.
Therefore SELECT THE TEXT OF MAINTENANCE MODE WINDOWS using right click on window heading, Edit, Mark, Copy .. not graphical copy.
Then paste the text into wiki and surround with "< pre>" and "< /pre>" tags (without the space) as follows:
the result is searchable text as follows ...
╔════════════════════════════╗ ║ NEOSYS SECURITY ║ ║ What is your name ? ║ ║ ║ ║ [ ] ║ ║ ║ ║ Please enter your name, ║ ║ or press Esc to exit. ║ ╚════════════════════════════╝
