Procedures

From NEOSYS Technical Support Wiki
Jump to navigationJump to search

Here are procedures to be followed by Support Staff in respect to various technical matters in day to day operations of client issues.

NEOSYS Support Policies

In order to maintain good payment speed by clients NEOSYS needs to restrict support to clients that dont pay their bills on time, however the degree of restriction needs to depend on an intimate knowledge of the client which cannot be expected from all NEOSYS support staff. Therefore we will use a simple escalation policy as follows:

Stop Support List

NEOSYS SUPPORT MANAGERS WILL maintain an overdue/stop list on a whiteboard visible to all support staff. Generally clients will go on the list immediately when their invoice is overdue and come off only after satisfactory commitment to pay have been obtained.

NEOSYS SUPPORT STAFF WILL discretely refer any calls for support from client on the overdue/stop list to support managers for handling.

In case clients pressurize neosys staff to provide support then the support staff must inform them that there is an issue with their account and that their request had been forwarded to the manager.

This policy excludes dealing with server failures, backup failures and measures required to be taken to fix these failures.

Client Password Policy

All client user passwords, including their initial one, are to be obtained via the user's email address using the password reminder/reset button on the login screen.

NEOSYS staff should never know users passwords therefore NEOSYS will not obtain and grant user passwords.

Support requests from ordinary client users

Any support requests concerning inability to obtain passwords will be forwarded to known skilled users on the client staff since this is the most efficient (not fastest) way to handle such issues.

Support requests from senior client management

Any support requests concerning inability to obtain passwords by senior client management users shall be handled directly by NEOSYS support staff in any way convenient to resolve the issue in the quickest possible time rather than the most efficient.

Bearing in mind that NEOSYS staff should never know user's passwords this will probably involve NEOSYS staff using the Password Reminder/Reset button to send a new password to the user.

User Defined Passwords

NEOSYS will provide user defined passwords in very special cases which must be pre-approved case by case by NEOSYS management. For many reasons, NEOSYS will not approve.

Currently this permission has only been granted to one NEOSYS client with several hundred of databases.

Handling client issues

Handling error messages

Important: Before Attempting to resolve client issues, please ensure that we have secure access to the NEOSYS server.

  1. The very first step is understanding client problem.
  2. Ask the client what error does he gets on the screen.
  3. If error seems to be familiar then resolve it over the phone.
  4. If error is unknown then ask user to send a screenshot of the error displayed along with the options used (basically you need to know HOW to replicate the error)
  5. Upon receipt of the error, check in all the wikis for a solution.
  6. If the issue is unknown or you don’t understand it clearly ask the user then use remote support to gain access to the users desktop to view how to replicate the error.

If it is new issue then report by escalation the same issue to your manager with a brief explanation.

NEOSYS support for resolving issues with totals on reports

If a client has a problem with any total output by NEOSYS software then NEOSYS support will advise them which other NEOSYS report or reports provide a complete breakdown of the total (if necessary, to individual transactions) and ask the client to locate any offending transactions themselves.

NEOSYS support staff will handle any issues where the total on the breakdown report does not add up to the total on the summary report.

Reconciling totals can be hard if there are many transactions involved. Regardless of how hard it may be, reconciliation is an operational task for users not for support staff since NEOSYS support staff will not get involved in understanding client transactions or data.


Using Support Tools

NEOSYS Software

The NEOSYS Software is currently supported on the following Operating Systems (OS) and browsers:

  • For Media & Finance modules, Internet Explorer 6+ on Windows
  • For Jobs & Timesheets modules, Internet Explorer 6+ on Windows or Safari 5+ on Mac

Users are to ensure that pop-up blockers and any 3rd party toolbars are deactivated/switched off or else certain pages and alert messages while using NEOSYS do not appear as a result of blocking from either the pop-up blocker or toolbars with built-in pop-up blockers.

NEOSYS Support personnel should additionally ensure that under Internet Explorer > Tools > Internet Options > Advanced > Browsing - the items Disable script debugging (Internet Explorer) and Disable script debugging (Other) are UNTICKED. This is because if NEOSYS generates any javascript error message, the same would disappear in the bottom left corner of a window, which in turn helps the programmer fix the error.

Note- The above mentioned process must be carried out after every Factory Reset.

Website Live Support

www.neosys.com is equipped with a Live Support software and clients can visit the website, click on this link and chat with any of our support staff, without the need for any installation. The client has to fill in their name and email address to connect to an available support personnel. During non-working hours, the Live Support icon on the website automatically displays "offline".

NEOSYS Support personnel who are authorised to provide such support, need to download a software called Kayako Live Support from http://hotfix.kayako.com/latest.php?product=lr&platform=win32&buildtype=stable

Once downloaded the account needs to be setup as follows:

  • Account Name: (as provided by NEOSYS IT)
  • SupportSuite URL: http://support.neosys.com/
  • User Name: (as provided by NEOSYS IT and usually same as Account Name)
  • Password: (as provided by NEOSYS IT)

The first time the account is setup, you need to close Kayako Live Support completely and restart for it to login and work properly.

Teamviewer

Since teamviewer allows no restriction on access once a fixed pass is installed, we cannot install fixed pass on teamviewer however convenient it might be.

RULE: NO FIXED PASS TO BE INSTALLED ON TEAMVIEWER IN ANY NEOSYS OR NEOSYS CLIENT COMPUTER

Running teamviewer live from a web link is fine because it does not allow installation of a permanent password

Documenting Processes in Wiki

How to create templates in wiki

Handling Nagios Client Monitoring system

Nagios is configured to display information pertaining to all NEOSYS client's server statuses which include multiple services such as:

  1. HTTPS: Most of NEOSYS clients are configured to have external web access via secure HTTP protocol (port 4430) from outside office. Nagios is configured to check port 4430 on a regular interval of 10 minutes and display any issues in accessing the same.
  2. SSH: As part of the support contract, NEOSYS should have external secure access to the client server usually over port 19580. Nagios is configured to check this port on a regular interval of 10 minutes and display any issues in accessing the same.
  3. Ping: Nagios is also configured to ping the client router as a measure to check if router responds incase the NEOSYS server is down.
  4. NEOSYS: This service works in a reverse direction, and the NEOSYS installation on the client server sends information such as databases running, current backup status, internal and internet IP addressess etc to Nagios on a regular interval of 10 minutes.

Some key information about Nagios is as follows:

  • Nagios is also configured to display information related to internal servers.
  • Clients hosted on a NEOSYS cloud server might not have services such as SSH or PING as this is monitored as part of the internal server service.
  • Nagios sends out email alerts to support2@neosys.com (which is forwarded to support@neosys.com) from 8 am to 12 midnight on all Dubai working days (Sun-Thu). No alerts are sent out on Fri and Sat, unless they are for NEOSYS internal servers.

NEOSYS support staff on duty has to follow the below outlined procedures in case of any Nagios items showing a critical or warning message for any service:

  1. Nagios is required to be checked first thing in the morning and any critical or warning messages need to be dealth with to resolve the same at the earliest.
  2. Some of the messages could be related to backup failures and the usual procedure as stated in #Backup Procedures needs to be followed. In case the backup issue isn't resolved by 9:30 am, the Nagios service needs to be scheduled with downtime for a minimum of 2 hours and maximum until 1 am next day if the issue cannot be solved.
  3. In case any HTTPS, SSH or PING service is down, immediate action is required and the relevant IT people at the client side needs to be contacted to get this resolved. A downtime of 2 hours is required to be scheduled with further intervals of 2 hours incase this is not resolved. Support staff shouldn't schedule downtime till 1 am next day, just to get rid of the alerts for the day. Proactive follow up with the client is required to get this resolved before the business day - more so, if there is a weekend ahead.
  4. In case the HTTPS, SSH or PING service goes down during the day, a grace period of 20 minutes is given before the issue is reported to the client IT. This helps incase there is any temporary internet connection issue at the client or along the internet route.
  5. In case of "Backup not changed" warning status which occurs if the client has not interchanged the USB before 12 noon on that day, no action is required from the support staff and a downtime until 1 am next day needs to be scheduled.
    • PLEASE NOTE: Failure to schedule appropriate downtime will lead to REDUNDANT ALERTS from NAGIOS every hour.

What happens if there is no remote access to NEOSYS server based in client’s premises ?

If access to the NEOSYS server is lost then we must determine the root cause by:

  1. Checking if the server is UP and running
  2. If yes, please check internet connectivity on the server
  3. If there is connectivity, please check the router for connectivity issues


Sample Response: Please note that we have currently lost access to the NEOSYS server. The server seems to be down at the moment and it seems that neosys processes are not running on the server. Kindly check if the server is UP and running. If yes, please check internet connectivity on the server. Do keep us posted on the server status so we can test connectivity from our side as well.0

New Router (Port Forwarding)

If you have changed your router then you may notice that external access to NEOSYS is unavailable.

Solution:

Setup a permanent access for NEOSYS by reconfiguring the Router / Firewall for Port Forwarding from Router to the NEOSYS Server as follows:

  1. Port 19580 > 19580 for SSH
  2. Port 4430 > 4430 for HTTPS

You can click Set Up Port Forwarding to learn how to configure your Router.

Click on the following link to see how to test/ troubleshoot port forwarding settings :
Troubleshooting Port Forwarding

Sample Response: You are requested to kindly setup a permanent access for NEOSYS by reconfiguring the Router / Firewall for Port Forwarding from Router to the NEOSYS Server,i.e. port 19580 for SSH and port 4430 for HTTPS.

Once this is complete, kindly send me an email to confirm the same so that we could test connectivity from our end as well.

Backup Procedures

Preparing daily backup report

  1. Note the success, failure and other error of the clients backup mail in an excel sheet and forward the same to your manager.
  2. If there is a backup failure or backup is not available, check wiki to take necessary steps.
  3. If there is any unknown error, forward the same to your manager.

Updating Nagios incase of failures

  1. If the backup failure is unsolved, schedule downtime Neosys service in Nagios till 01 am.
  2. If the backup did not happen because of server down. Call the IT person; ask him to re-boot the server and check wiki to do necessary step ahead and schedule downtime to Nagios for 2hours.
  3. If there is an error "Backup->Impossible" on Nagios check wiki and schedule downtime to Nagios for 2hours.

Interchange backup USB mail reminder

  1. Basically all the clients have different days to change their backup USB. All the notification can be seen on Nagios at 12.00 pm every day.
  2. When you see the change backup notification on Nagios at 12.00 pm, send a mail to the IT person or the person who changes the USB. As you send the mail schedule downtime for Neosys services for two hours.
  3. Check the Nagios at 2.00 pm if the backup USB is changed, don't send another mail and stop there. But if backup is still not yet changed, send a another mail and this time cc the mail to your manager. After sending the mail, schedule downtime for Neosys service for more two hours.
  4. Now at 4.00 pm if you still find the backup USB not changed. Call the IT person and ask him to change the USB ASAP and schedule downtime for Neosys services for one hour.
  5. Now at 5.00 pm check Nagios and see if the backup USB is changed and if not changed. Send the mail to everyone from the backup mail.
  6. If the backup is still not changed, escalate the issue to your manager.

Note: If the USB is not interchanged on the scheduled day i.e. then The NEOSYS Automated backup will fail. Traditionally, each USB holds backup of 7 days and using 3 different USBs we can store backups for the last 21 days enabling us to restore the system unto a time period beginning 21 days prior. If the USB is not changed then the first backup on the current USB is replaced with the new or latest backup leading to inconsistencies within the backups. Hence we must interchange the USB on schedule to avoid a backup failure the next morning.


Sample Response: The NEOSYS backup has failed today as the USB was not interchanged on the scheduled day. Please interchange the USB immediately today to avoid a backup failure tomorrow morning. Also note that your scheduled day to interchange the USB next week and hence forth remains unchanged to (Scheduled Day).

Finding out which USB is inserted into the server

As we ask the client to have 3 USB's and interchange them weekly, we also need to sometimes track which one of these 3 USB's are inserted into the server. USB's can be tracked using their volume serial number in most cases. To find this out either go to the command prompt and type VOL or in the nightly backup message check for the 2nd line (which looks like this - 14/12/2009 2:45pm Media: 705B-5B5F). However serial numbers can be the same even for different USB's.

Client relocating operations to a new premises

Ensure that the following procedure is followed by the clients IT staff prior to relocation:

  1. Login to the NEOSYS server and shut down all the processes by hitting the ESC button on each of the window three times or until they close
  2. Take a backup of the D drive on another system or network / external drive
  3. Remove the USB plugged in and secure the same along with the other two USB’s in a safe location
  4. The NEOSYS server may now be shutdown and moved to the new premises

Post relocation, the clients IT staff must ensure the following immediately:

  1. Connect the server to the local network and complete all setup required for users to be able to access the system internally
  2. Configure the firewall and/or port settings to enable NEOSYS support staff to connect to the server externally
  3. Provide NEOSYS support with the new fixed IP address so that the connection can be tested.

Creating and Handling passwords

Passwords made out of a pass phrase should be at least 10 characters since using initials results in a lot of i's and a's etc which reduces the effectiveness of the password and allows hacking via brute force guessing especially since windows doesnt slow down logins even if it sees thousands of password attempts.

Creating a password

Passwords are generated from a pass phrase and it is important to create a very difficult to guess pass phrase.

For example, a good pass phrase would be: Today is a good day and it is the best time to go for a holiday

The password for this would be Tiagdaiitbt2g4ah

The important instructions for the above are:

  1. You have to take the first letter of each word and that makes your password (i.e. by using initials)
  2. Wherever any word starts with a capital, then you have to take first letter as a capital (eg. For Today you will take T)
  3. Replace and with &
  4. Replace to with 2
  5. Replace for with 4

Handling passwords

  1. Never send the actual password - always send the pass phrase
  2. Make sure that the password created out of the pass phrase is at least 10 characters long since using initials results in a lot of i's and a's etc which reduces the effectiveness of the password and allows hacking via brute force guessing especially since windows doesnt slow down logins even if it sees thousands of password attempts
  3. Pass phrases are never to be sent by email, whatever the case maybe.
  4. Pass phrases can be sent by chat - however they have to be broken down in two parts and sent separately over two different messengers or if you are using Gtalk then use the 'off the record' mode.
  5. Using SMS to send pass phrases is the best known way as of now.
  6. If you save the passwords on your system as a file, make sure the file is encrypted with a master password.

NEOSYS Maintenance Window

The NEOSYS server is functional from 6am – 1am. There is a 5hr window gap for the system to perform updates & backups.

The 5hr maintenance window:-

1. At 1am – The server performs a data backup on a USB (for the respective clients) & once the backup has been completed, the system automatically generates an email addressed to the neosys staff & the respective clients.

2. At 2:45am – The main data over writes the test data on the server.

3. At 3:00am – The server by itself performs an update for Windows.

4. At 4:00am – The server performs a backup to the headquarters for clients, and then automatically generates an email addressed to the NEOSYS staff & the respective clients. This process is followed by only two clients i.e. Adline & Farouk.

5. At 6:00am – The server starts up NEOSYS.

Switching to a backup server

As NEOSYS provides clients with option of backing up their data to a remote NEOSYS server in case of emergencies or server problems, it is crucial you understand the below procedure on how to switch to a backup server in the event of such a situation. Extreme care must be taken when switching over to using a backup server otherwise unnecessary data loss is very likely.

Backup servers are normally switched off and should not be started automatically otherwise there is a serious risk of the client’s staff working on two systems. It is not possible to merge two databases into one database. Before the backup server is enabled the main server must be disabled, and before the main server is re-enabled, the backup server must be disabled again. This can be managed technically without requiring any decision from senior non-technical staff.

However, there are also some potentially hard decisions about unavoidable loss of data versus continued system availability. *** Backup servers should therefore only be started with the written approval of the clients senior staff. A suitable email requesting approval follows.

The following case assumes that the main server has gone down sometime during the working day and that therefore the data on the backup server is out of date. Allowing them to use the backup server therefore implies some loss of data. They may wish to lose the data. They may wish to work on the backup server data and then try to redo the work on the main server once it is restored. There are a variety of options depending on the situation.

If the main server is still functioning AND you are reasonably sure that the database is not damaged (which is perhaps an unlikely situation if you need to use the backup server!), it may be sensible to trigger an additional “backup/sync” process to bring the backup server database up to date with the main server. Before you do this, it is advisable that you take a backup copy of the backup system on the backup server. In this case there would be no data loss in using the backup.

An additional option of providing usage of the backup server in read-only mode so that people can at least access some data is being developed. The backup server could be available continuously at any time in read-only mode. This article would then be related to switching a backup server into main operational mode. 

Dear {senior staff} cc {IT staff}

Please note that we can enable the backup server if you wish.

However the data on the backup server is out of date since it is a copy of your main database as at 11/22/33 99:99.

If you wish to allow work to be done on the backup server then any data entered on your main server since the above date will be lost if we 
subsequently copy the data on the backup server to the main server.

If, after using the backup server, we do NOT copy the data on the back server to the main server then any data you have entered on the backup 
server/database will be lost.

Please confirm a) you want to work on HOSTS2 database and that we should therefore enable it and b) you have disconnected your main server 
for the duration.

Best Regards,
xxxxxx xxxxxxx
NEOSYS Support


Description of Backup Procedure for the NEOSYS Client Hosting Server

NEOSYS backup is a two phase process. It is mandatory that both phases are complete for the process to be considered a backup.

  1. On the main NEOSYS server host(Schuberg Philis in Amsterdam as of Dec 2010)
    Every night, the NEOSYS process for selected databases stops serving users at around 1:00 AM to 5:00 AM (local time of the client's main office) and performs the following procedure:

    - All the database files are checked for physical corruption.
    - All the database files are zipped into a file who's path would be something like C:\DATA.BAK\CLIENTX\MONDAY\BACKUP.ZIP file.
    - The file having the same name as that of the corresponding day of the week as last week is overwritten.A "backup alert" email indicating success or failure is sent to senior managers at the client, the client’s IT manager and backups@neosys.com.
    - An internal log (accessible on each database's Support Menu) is maintained as a record and in case the emails cannot be sent out for some reason.
    - NEOSYS support staff in Dubai check the emails every morning Sunday through Thursday.
    - In the case of failure, NEOSYS support staff in Dubai take appropriate action and send an email (to the same people who receive the automated backup alert emails) indicating what action has been taken.
    - The above does not by itself constitute a proper backup because the backup is stored on the same server and physical disk as the actual data.

  2. On the NEOSYS backup server host (in NEOSYS Dubai Office as of Dec 2010 but could be changed without notice to EvoSwitch in Amsterdam).
    - At around 02:00-02:30 AM GMT daily, NEOSYS's backup server copies (replicates) all the BACKUP.ZIP files from the main NEOSYS server host to itself.
    - An email indicating success or failure is sent to NEOSYS support staff (only) who deal with it in a similar way to phase 1, except that only NEOSYS management are updated.


Cutting and Pasting NEOSYS Maintenance Mode Messages into Wiki

Error messages expressed as images are not searchable.

Therefore SELECT THE TEXT OF MAINTENANCE MODE WINDOWS using right click on window heading, Edit, Mark, Copy .. not graphical copy.

Then paste the text into wiki and surround with "< pre>" and "< /pre>" tags (without the space) as follows:

Loginmessage.jpg

the result is searchable text as follows ... 

╔════════════════════════════╗
║      NEOSYS SECURITY       ║
║    What is your name ?     ║
║                            ║
║   [                    ]   ║
║                            ║
║  Please enter your name,   ║
║   or press Esc to exit.    ║
╚════════════════════════════╝

Amending/Reposting Journal Entries

NEOSYS does not generally allow reposting or amending of journal entries once posted.

This policy is derived from the manual book keeping practices where once a journal entry is made you cannot remove it as it is written in ink and hence a rectification entry needs to be passed.

However in certain exceptional cases, amending/reposting of journal entries is allowed for a brief period of time to enable clients to present reports in an alternative manner. This would be subject to NEOSYS would requiring a written LETTER OF APPROVAL duly signed and stamped by the highest management of the company.

Before granting such permission NEOSYS will have a conversation with the Management of the client organisation and clearly explain the repercussions of granting permission to edit/report journals, which primarily are:

  1. It is not common practice to do such manipulations
  2. It can lead to possible errors in numbers
  3. Staff can become complacent and careless while posting entries and frequently ask for this permission
  4. Restricting users from editing/reposting journal entries promotes a more healthy and vigilant working environment where users verify each journal entry before posting it which effectively leads to good accounting practices.

Amending/Reposting Procedure

In case the client management still decides to go ahead and allow editing/reposting of journal entries, the following procedure is to be followed:

  1. Client must de-allocate vouchers which need to be amended
  2. NEOSYS support staff must wait for a day so that de-allocated vouchers are copied into Test database
  3. Authorise required users to amend and repost (without record) in Test database only
    ( While reposting, we have 2 options i.e. with record and without record. The 'with record' option causes the system to maintain a history of edits made. Hence, we want to repost without record so that there is no trace of the edit in the system)
  4. Amend a substantial number of vouchers in Test and verify them.
    To verify if the edits made are reflected:
  5. *Print all ledgers for the whole year
  6. *Cross-check all balances
  7. Once you verify the balances are correct in Test database, grant users permission to amend and repost in the Live database.
  8. Ask users to amend and repost vouchers in the Live database.
  9. Cross-check all balances for the current year.
  10. If you successfully verify the balances, revoke permissions immediately. Else, wait for 24 hours and revoke permissions irrespectively.

Removal of unauthorized third-party software on client servers

Rule: Any third party software that is discovered by NEOSYS support staff on client servers that has been installed without the agreement of NEOSYS should be uninstalled immediately on discovery.

However purposeful a software is, NEOSYS is contractually responsible for support and there are too many opportunities for poorly installed software to cause unpredictable damage to the NEOSYS database so NEOSYS has to have a clear and safe and simple policy to ensure the integrity of client data. Installing software without prior discussion with NEOSYS by itself indicates that insufficient care and consideration as been given to possible issues.

Any software required by client IT for some purpose may only be installed after discussion and agreement from NEOSYS support staff concerning the configuration and operation of the software.

The NEOSYS Software Licence and Support agreement requires that where NEOSYS software is installed on client servers that a dedicated server is provided and dedicated implies that no other software may be installed without the agreement of NEOSYS support.

Retrieved from "https://techwiki.neosys.com/index.php?title=Procedures&oldid=970"