Setting up and using remote support: Difference between revisions

From NEOSYS Technical Support Wiki
Jump to navigationJump to search
 
(121 intermediate revisions by 11 users not shown)
Line 1: Line 1:
== Getting agreement of client IT staff to provide remote support ==
==Getting agreement of client IT staff to provide remote access==


[[Letter to obtain agreement of client IT staff to provide remote support]]
[[Letter to obtain agreement of client IT staff to provide remote access]]


== Installing and configuring the server ==
==Initial Connection to the server before setting up permanent remote connection==


=== Initial Connection to the server before setting up permanent remote connection ===
For remote installation you need to get an initial connection to the server before you can setup Cygwin for a permanent remote connection.


In case of a remote installation you need to get an initial connection to the server before you can setup Cygwin for a permanent remote connection. For this purpose you can either use your customised reverse connect UltraVNC SC file or the one-time run Teamviewer utility.
Get the one-time run quick support teamviewer utility from [[https://neosys.com/help|Team Quick Support]]. Otherwise refer to the wiki to install customised reverse connect "UltraVNC" SC file.  


Do not use Microsoft Remote Desktop Client (RDP/RDC) on port 3389 at anytime to access the server from the internet since IT suppliers not aware of the situation often setup the initial administrator password to something obvious like "password" or even blank and in this case there is a good chance internet worms will discover the "open door" and install themselves before you get the chance to put a strong password.
If the client has already gone ahead and provided Microsoft RDP with an obvious/weak system password, then Support MANDATORY MUST get Windows reinstalled from scratch. Antivirus may not be able to tell that the server has been infected and rootkitted and therefore a scan does not prove it has not been infected.


=== Installing and configuring SSH ===
Support MUST not provide NEOSYS support via Microsoft Remote Desktop Client (RDP/RDC) on port 3389 at anytime because it is a BAD idea to simply open port 3389 since an open port 3389 attracts scanners/hackers like flies.


==== Installing Cygwin with OPENSSH ====
Also, IT suppliers not aware of the situation often setup the initial administrator password to something obvious like "password" or the arent-I-clever "P@ssw0rd" or even blank. In this case there is a good chance internet worms will discover the "open door" and install themselves before you get the chance to put a strong password.
 
==Installing and configuring SSH==
===Installing Cygwin with OPENSSH===
 
These instruction are only for installing in a server NOT part of a domain. For installing in a server that is part of a domain, see http://cygwin.com/faq-nochunks.html#faq.using.sshd-in-domain


Watch out for non-intuitive steps like clicking "skip" to install something.
Watch out for non-intuitive steps like clicking "skip" to install something.


# Read [[Avoiding Corrupt Cygwin Installations]]
Read [[Avoid Corrupting Cygwin Installations]]
# ENSURE that you are logged in as the local (NOT DOMAIN) administrator
 
# Run http://www.cygwin.com/setup.exe
#Instruct client to login to server as Administrator.
# Install from Internet
#Connect to client server via Teamviewer or customised reverse connect UltraVNC SC file.
# Root Directory: '''c:\cygwin'''
#ENSURE that you are logged in as the local (NOT DOMAIN) administrator.
# Local Package Directory: '''c:\cygwin.lib'''
#Download/Run/Install http://www.cygwin.com/setup.exe<br>(you might have to go to the home page http://www.cygwin.com and click the link to setup.exe)
# Direct Connection
#Download source: '''Install from Internet'''
# Download Site: '''http://mirror.steadfast.net''' (near the bottom) -- http://mirrors.kernal.org is no longer available
#Root Directory: '''c:\cygwin'''
# You will now receive a Setup Alert. Click '''OK'''
#Local Package Directory: '''c:\cygwin.lib'''
# Select Packages: Maximise window then click '''View''' once to get '''Full'''
#Choose "yes" to "Folder does not exist. Create new?"
# Next to the package '''OPENSSH''', click the word '''Skip''' (once!) to get version 4.4p1-1 or later
#Internet Connection: '''Direct Connection'''
# Next to the package '''NANO''', click the word '''Skip''' (once!) to get the latest version available
#Download Site: '''http://mirrors.kernel.org''' (near the bottom)<br>(If this does not show in the list, key in the URL in the field '''User URL''' and click on Add)
# Next to the package '''WGET''', click the word '''Skip''' (once!) to get the latest version available
#Select Packages: Maximise window then click '''View''' once to get '''Full'''. You can then enter the name of the desired packages in the Search box to speed up location of the desired packages.
# Next to the package '''RSYNC''', click the word '''Skip''' (once!) to get the latest version available
#Next to the package '''OPENSSH''', click the word '''Skip''' (once!) to get version 4.4p1-1 or later
# Next to the package '''EMAIL''', click the word '''Skip''' (once!) to get the latest version available
#Next to the package '''NANO''', click the word '''Skip''' (once!) to get the latest version available
# Next to the package '''WHOIS''', click the word '''Skip''' (once!) to get the latest version available
#Check the NEOSYS INSTALLATION CHECKLIST for any other packages to install like the above.
# Check the NEOSYS INSTALLATION CHECKLIST for any other packages to install like the above.
#Click Next and complete the installation
# Click Next and complete the installation
 
===Win32 Error===
 
The Win32 Error occur when the bad file is cached in internet explorer cache. You can try clearing the internet explorer cache and redownloading or you can try to download from cygwin.com instead of www.cygwin.com so it doesnt look in the cache or www.cygwin.com if your original download was from cygwin.com. All else failing, you can simply upload the setup.exe file from your own pc to the server.
 
All this relates to win32 error when running a downloaded file. Any downloaded file and not just cygwin.com/setup.exe


==== Configuring and starting SSHD ====
===Error during setup===


In case of the following error, check for proxy settings in internet explorer. It is possible that the client uses a proxy setting. In that case, in Step 7 instead of choosing <b>Direct Connection</b>, choose <b>Use Internet Explorer Proxy Setting</b>.
Unable to get setup.ini from <http://mirrors.kernel.org/>
[[File:Cygwin install error.png]]
===Configuring and starting SSHD===
Open the Cygwin icon to get a linux/bash command line and type:
Open the Cygwin icon to get a linux/bash command line and type:


Run the following commands:
Run the following commands: (not needed in recent versions of Cygwin so dont do this)


  chmod +r /etc/passwd
  chmod +r /etc/passwd
  chmod +r /etc/group
  chmod +r /etc/group
  chmod 777 /var
  chmod 777 /var
Refer [[Setting_up_and_using_remote_support#Reinstalling_SSHD_if_service_fails_to_startup| here]] if you get an error while doing the above steps.
Prevent cygwin from using Unix like permissions on files it creates
nano /etc/fstab
add the line or just add ",noacl" to the existing similar line. (What is the effect of omitting this?)
none /cygdrive cygdrive binary,posix=0,user,noacl 0 0


Thereafter start with the ssh configuration:
Thereafter start with the ssh configuration:
Line 51: Line 79:


Then on the following options type:
Then on the following options type:
Only asked if running again:
Overwrite existing /etc/ssh_config file? yes
Overwrite existing /etc/sshd_config file? yes
.
StrictModes                -  no
Privilege                  -  yes
New local sshd account      -  yes
Install SSHD as a service  -  yes
Enter value of daemon      -  Just press Enter
Different name              -  no
Create new privileged user  -  yes
Enter a password now        -  Invent a NEW totally random password with caps and both upper and lower case.
Re-enter the password      -  Enter it again. Dont record it anywhere. Forget it.


Privilege                  –  YES
At the command prompt type
New local sshd account      -  YES
 
Install SSHD as a service  -  YES
  net start cygsshd
Enter value of daemon      -  ntsec
Different name              -  NO
  Create new privileged user  -  YES
Enter a password now        -  Set any random password and should not be the same as the neosys server (8 characters min)


At the command prompt type
For older versions of Cygwin (Before Jan 2019)


  net start sshd
  net start sshd


==== Configuring SSHD to use a non-standard port number ====
===Configuring SSHD to use a non-standard port number===


This is necessary if the router cannot forward port 19580 --> 22 and we don’t want to open port 22 directly.
This is necessary if the router cannot forward port 19580 --> 22 and we don’t want to open port 22 directly.


Capitalization is signification in cygwin/linux commands
Capitalization is SIGNIFICANT AND CANNOT BE IGNORED in cygwin/linux commands


open cygwin command prompt
open cygwin command prompt
  cd /etc
 
  chown administrator sshd_config
  nano /etc/sshd_config
  nano sshd_config (assuming that you have installed the NANO editor)
 
notepad sshd_config (incase you havent installed the NANO editor)
change the Port to look like this:
Move your cursor to '''Port 22''' and change 22 to 19580. ctrl+x to save. On the confirmation type Y and on the next prompt hit enter.
 
  chown system sshd_config
  #Port 22
  Port 19580
 
Also add the last line to the following section. Refer [[Setting_up_and_using_remote_support#Solving_.22Authentication_that_can_continue:_publickey.2Cpassword.22_Error_when_connecting_to_remote_servers_via_remote_access_clients| Error when connecting to remote servers]] to see why this line is added.
 
<pre>
# Change to no to disable s/key passwords
#ChallengeResponseAuthentication yes
ChallengeResponseAuthentication no
</pre>
 
Press Ctrl+x to save. On the confirmation type Y and on the next prompt hit enter.
 
  net stop cygsshd
net start cygsshd
 
For older versions of Cygwin (Before Jan 2019)
 
  net stop sshd
  net stop sshd
  net start sshd
  net start sshd


To check that the connection to port 19580 is successful you can run the following test:
To check that the server is running and listening on port 19580
 
  ssh -p 19580 administrator@localhost
  ssh -p 19580 administrator@localhost


You will be prompted to confirm the connection (say yes)
If you are asked for to confirm the server id is correct or enter password then the check is successful. No need to continue.


Now enter the system password to complete the procedure.
===Changing ssh login from “Administrator” to “administrator”===
Current NEOSYS policy to cater for recent versions of Cygwin is to rename the windows Administrator user to administrator to keep a consistent ssh login across all installations.


==== Error while changing Cygwin port 22 to 19580 ====
If you forget to do this before installing or upgrading Cygwin then you must to the following:
 
#Rename “Administrator” to “administrator” in Windows
#*If you cannot rename Administrator to administrator, follow the procedure mentioned at [[Changing username from Administrator to administrator]]
#In a Cygwin console do:
 
mkpasswd > /etc/passwd
 
It should come back with nothing
 
===Error while changing Cygwin port 22 to 19580===


Error Message:
Error Message:
Could not open file for writring: permission denied


Occurence:
"Could not open file for writing: permission denied"
Sometimes When you edit the sshd_config file through NANO.
 
Occurrence:
Sometimes when you edit the sshd_config file through NANO.


Solution:
Solution:
In SSH shell, follow these commands:
In SSH shell, follow these commands:


  cp sshd_config ashwin_temp (copies sshd_config to a new file ashwin_temp)
  cp sshd_config ashwin_temp #copies sshd_config to a new file ashwin_temp
  rm sshd_config (deletes sshd_config)
  rm sshd_config             #deletes sshd_config
  cp ashwin_temp sshd_config (copies ashwin_temp to sshd_config)
  cp ashwin_temp sshd_config #copies ashwin_temp to sshd_config
 
In case it does not copy sshd_config to ashwin_temp, than check whether an ashwin_temp filename exists and delete it using the rm command.
 
===Enable Remote Connection on Windows to allow connection through Remmina===
 
#Open System Properties from Explorer
#Click on Remote settings
#Under 'Remote' section, check Remote Assistance (disabled on Win 2008 servers) and Remote Desktop to allow remote connections to this computer.


Incase it does not copy sshd_config to ashwin_temp, than check whether an ashwin_temp filename exists and delete it using the rm command.
[[File:Win remote connection.png]]


==== Opening up ssh connections to additional source ip nos ====
===Opening up ssh connections to additional source ip nos===


Starting a NEOSYS process will automatically restrict cygwin ssh to accept connections from known NEOSYS company static ip numbers.
Starting a NEOSYS process will automatically restrict cygwin ssh to accept connections from known NEOSYS company static ip numbers.
Line 114: Line 189:
  nano /etc/hosts.allow
  nano /etc/hosts.allow


sshd: ALL
Enter IP numbers or CIDR format:
 
or a ip numbers or CIDR format


  sshd 12.34.56.78
  sshd 12.34.56.78
  sshd 12.34.0.0/16
  sshd 12.34.0.0/16


==== Setting up email alerts for cygwin ssh logins ====
===Setting up email alerts for cygwin ssh logins===


Use http://www.cygwin.com/setup.exe to install "email" and "whois" packages
1. Use http://www.cygwin.com/setup.exe to install "email" and "whois" packages. MUST READ [[Avoid_Corrupting_Cygwin_Installations#To_see_what_modules_Cygwin_is_going_to_update |See what modules Cygwin is going to update]]


Insert the following script using cygwin command prompt.
2. Run Cygwin and copy & paste script below into new file sshrc. Change it@neosys.com to the email ID to which the alert needs to be sent.


<b>NOTE! it@neosys.com to whatever you want.</b>
  nano /etc/sshrc
 
  cd /etc
nano sshrc


<pre>
<pre>
Line 165: Line 235:
</pre>
</pre>


Make sure that you configure the file permissions
3. Give execute permission to sshrc script file for all groups (owner, group, other):


  chmod a+x sshrc
  chmod a+x sshrc  


Inserted trusted ip nos.
4.Add trusted IPs by copying & pasting text below:


  cd /etc
  cd /etc
Line 175: Line 245:


<pre>
<pre>
#sorry, ip ranges and cidr etc not accepted yet
#IP ranges and CIDR etc not accepted yet


#vm1.neosys.com for remote checking
#vm1.neosys.com for remote checking
Line 193: Line 263:
</pre>
</pre>


Make sure that you configure the file permissions
===Quick way of adding all Support's public keys to allow Remmina support===
 
If a server has lost all authorised keys in .ssh/authorized_keys file, then instead of Support adding their public key individually using "./autologin.sh" use this method:
 
#Connect via SSH to any other client server that has support team's public keys saved.
#Then open Cygwin and type: <pre>cat .ssh/authorized_keys</pre>
#Select and copy all the text in the file. i.e public keys
#Exit and connect to the the new client and open Cygwin and type:<pre>nano .ssh/authorized_keys</pre>
#Right click and paste the copied keys in a new line below any possible existing keys ensuring that each key appears in a separate single line and then save and close the authorized_keys file.
#Check that you can connect to the target server using automatic SSH authentication (SSH Agent or Public key option) in Remmina.


chmod a+x sshrc
===Testing SSH connection to the NEOSYS server over port 19580===


==== Troubleshooting SSH: If SSH connects and then disconnects immediately without exchanging keys ====
If you cannot connect to the server using SSH, see [[Troubleshooting_NEOSYS_Generally#Troubleshooting_NEOSYS_remote_support_port_forwarding|Troubleshooting NEOSYS remote support port forwarding]]
 
===Troubleshooting SSH: If SSH connects and then disconnects immediately without exchanging keys===
   
   
The first time that NEOSYS runs, it automatically adds source ip number restrictions to the sshd remote support configuration in /etc/hosts.allow and /etc/hosts.deny. This is an important security procedure to allow connection to clients systems from NEOSYS ip numbers only. This process allows only local and known NEOSYS ip numbers to connect using SSH. Upgrading NEOSYS will add and/or remove allowable ip numbers as NEOSYS configuration changes.
The first time that NEOSYS runs, it automatically adds source ip number restrictions to the sshd remote support configuration in /etc/hosts.allow and /etc/hosts.deny. This is an important security procedure to allow connection to clients systems from NEOSYS ip numbers only. This process allows only local and known NEOSYS ip numbers to connect using SSH. Upgrading NEOSYS will add and/or remove allowable ip numbers as NEOSYS configuration changes.
Line 211: Line 292:
#Find the ip number of failed connections.
#Find the ip number of failed connections.


===== Possible Problem 1 - Port mapping in router is using NAT =====
====Possible Problem 1 - Port mapping in router is using NAT====


If the ip number of failed connections is some local ip number (of the router for example) then possibly the inbound port forwarding has been done with NAT and the source ip number has been lost. Therefore the NEOSYS ip restrictions are blocking ssh connections because they appear to be coming from an unknown ip number (ie that of the router)
If the ip number of failed connections is some local ip number (of the router for example) then possibly the inbound port forwarding has been done with NAT and the source ip number has been lost. Therefore the NEOSYS ip restrictions are blocking ssh connections because they appear to be coming from an unknown ip number (ie that of the router)


===== Solution 1A =====
====Solution 1A====


Change the router configuration to not use NAT and leave the genuine original source IP number
Change the router configuration to not use NAT and leave the genuine original source IP number


===== Solution 1B =====
====Solution 1B====
The router is sadly using NAT instead of plain old port forwarding.


DO NOT USE THIS PROCEDURE TO BREAK NEOSYS SECURITY
DO NOT USE THIS PROCEDURE TO BREAK NEOSYS SECURITY. DO NOT GRANT ACCESS TO ANY IP OTHER THAN CLIENTS ROUTER IPS
 
The solution is to add NAT router IP to the list of authorised IP numbers on the NEOSYS server. This solution provides access to NEOSYS server from outside office unrestricted by IP number, hence Client Management approval must be obtained before this solution is applied.
 
Sample Email to Management-
<pre>
Dear XXXX,
   
   
DO NOT GRANT ACCESS TO ANY IP OTHER THAN CLIENTS ROUTER IPS
Support must have remote access to the NEOSYS server via SSH but currently we don’t have access.
 
Add the routers IP number to the list of authorised ip numbers in the cygwin hosts.allow file as follows:
This is because your router is using NAT. The NAT router translates the source IP to its own hence the source IP is lost. NEOSYS server
has  a list of allowed source IPs and since the router’s IP is not in the list, connection fails.
The solution to establish successful connectivity is to allow access to NEOSYS server from your NAT router by adding the router’s IP in
list of allowed IPs on the server.
We need your agreement to carry out this solution because authorizing this access means access to NEOSYS from outside office will not be 
restricted by IP any more.
Please confirm that this solution is OK.
Best Regards
</pre>
On receipt of Management approval, add the routers IP number to the list of authorised IP numbers in the cygwin hosts.allow file as follows:


  nano /etc/hosts.allow
  nano /etc/hosts.allow


and add the line as follows ... but put the ip number of your router
and add the line as follows but put the IP number of your router


  sshd: allow 192.168.0.99
  sshd: allow 192.168.0.99


#router - sadly using NAT instead of plain old port forwarding
<b>Warning</b>
#if the router ip changes then neosys remote support will fail until this line is changed
 
#NB do not grant access to 192.168.* etc since this allows local LAN viruses to attack
#If the router IP changes then NEOSYS remote support will fail until this line is changed
#Do not grant access to 192.168.* etc. since this allows local LAN viruses to attack


==== Troubleshooting sshd ====
===Troubleshooting sshd===


You can run the sshd service interactively to see all messages instead of having to search logs/events etc.
You can run the sshd service interactively to see all messages instead of having to search logs/events etc.
Line 246: Line 348:
  /usr/sbin/sshd -D -p 19580
  /usr/sbin/sshd -D -p 19580


==== Reinstalling SSHD if service fails to startup ====
===Reinstalling SSHD if service fails to startup===
 
====Error message====
chmod: cannot access '/etc/passwd': No such file or directory
chmod: cannot access ‘/etc/group’: No such file or directory
 
====Solution====
Sometimes reinstallation isnt necessary and sshd can be made to restart by doing
mkpasswd > /etc/passwd
mkgroup > /etc/group
 
If all else fails:


#Look in '''/var/log/sshd.log''' for errors
#Look in '''/var/log/sshd.log''' for errors
Line 255: Line 369:
Note that you don't have to reinstall cygwin entirely, just sshd with the above steps.
Note that you don't have to reinstall cygwin entirely, just sshd with the above steps.


==== Upgrading SSHD / Cygwin ====
==Upgrading SSHD / Cygwin==
NEOSYS relies on cygwin to provide secure network access and support various linux/unix services under Windows, mainly rsync for interoffice consolidation.
 
Just like MS Windows update, cygwin should be updated at regular intervals to close security holes discovered in the software by its authors. This is particularly important for cygwin's remote access service sshd since it is exposed to the internet although on a non-standard port.
 
Join the cygwin and sshd security news email lists to learn about when cygwin upgrades sshd and/or when there are issues generally with sshd
 
To find out what versions of cygwin/sshd are installed at NEOSYS clients, in Nagios check "Status Information" of the neosys-ssh service
 
SSH OK - OpenSSH_7.7 (protocol 2.0)
 
Before updating Cygwin or its packages you MUST read [[Avoid Corrupting Cygwin Installations]]
 
===Upgrading Cygwin remotely===
 
NEOSYS normal remote server support connection uses cygwin/ssh. Cygwin can be upgraded while in use with a script as explained in the section below.


#Read [[Avoiding Corrupt Cygwin Installations]]
====Upgrading Cygwin with a script====
#Connect normally using ssh/remote desktop etc.
#Establish another non-ssh vnc reverse connection
#Schedule an hourly reverse connect of vnc in case you lose the connection. Add the following to scheduled tasks - "C:\Program Files\UltraVNC\winvnc.exe" -connect address.ofneosys.supportstaff::port
#Make sure you have disconnected the ssh/remote desktop connection at this stage after you have established a reverse connection (i.e. if using Tunnelier, make sure that ssh connection using Tunnelier is not still open and not minimised in the notification area of the taskbar)
#net stop sshd
#Delete any bash/rsync and other cygwin related processes in task manager. You can type ps -ef in a cygwin bash console to find out their names.
#Ensure that you can rename c:\cygwin\bin\cygwin1.dll to xxx and rename back to ensure that no cygwin processes are running. Unfortunately not a perfect guarantee.
#Upgrade cygwin using the same general steps as a normal cygwin install. If any files cannot be upgraded do your best to kill the offending process and retry. Otherwise you may end up having to uninstall/reinstall cygwin from scratch.
#net start sshd


==== How to uninstall/reinstall cygwin ====
The following script can be used to automatically upgrade cygwin to the latest version quite easily even when people are using NEOSYS. However it carries a small risk described below.


With setup.exe (the installer file of cygwin) you can uninstall individual packages but not Cygwin.
WARNING This script temporarily disconnects and disables all ssh remote support connections, including any ssh connection you are using to initiate the process, for the duration of the upgrade.


Before you do this, make sure you have stopped the cygwin service (NET STOP SSHD), removed the sshd server (cygrunsrv -R sshd), deleted the users (sshd & sshd_server).
Since something may go wrong and the script might FAIL to re enable ssh remote connections, you can take one of the precautionary measures listed below.


To uninstall Cygwin you have to run the following in DOS prompt:
*either perform a temporary Teamviewer installation. The quick teamviewer zero installation remote support method will not work under rdp/tunnelier/remmina
*or ensure that client IT support is available ONSITE to provide temporary teamviewer access in the event of any problem
*or be prepared to lose the ability to provide remote support to the installation until the previous item is available


'''TeamViewer 9 issue'''
When attempting to connect to client server via TeamViewer 9 (setup via Tunnelier with unattended access) it shows the error below
[[File:TVerror.jpg]]
SOLUTION: Install TeamViewer 7 which does not give this error. Contact NEOSYS IT for TeamViewer7 commercial license. You must have the client server's administrator password to login using TeamViewer. After the upgrade, REMOVE SETTINGS for unattended access and UNINSTALL Teamviewer. Teamviewer must NOT BE LEFT with permanent login by number and password! Teamviewer options, security, REMOVE "Predefined password (For unattended access)"
=====Running the script=====
[[Setting_up_and_using_remote_support#Finding_the_script|Locate the upgradecygwin.cmd script]] and run it some usual way by clicking and pressing Enter.
You MUST inspect the version of the pre-installed script against the version shown at http://www.neosys.com/support/upgradecygwin.cmd and ensure that the script you are using is the latest one, as the script is updated with fixes for problems faced in the past.
The upgradecygwin.cmd script will try to download the latest version of setup-x86 from cygwin.com. In case it is not possible to download the setup-x86.exe file from cygwin.com due to proxy/firewall or other issues, then follow the below steps before running the cygwin upgrade script.
#Download setup-x86.exe manually from http://www.cygwin.com/setup-x86.exe
#Place it in the same directory as the upgrade script
#Rename it to "setup-x86-manual.exe". The cygwin upgrade script will rename this file to setup-x86.exe
If you initiate the script while connected on ssh using tunnelier/remmina etc. halfway through the script you will be disconnected.
The script will take a few minutes to download and install any cygwin upgrades.
Once the script is finished, it will re-enable creation of new incoming ssh connections and attempt to send an email to support@neosys.com via the standard mailout.neosys.com:2500 email server.
You should then be able to reconnect using ssh and tunnelier/remmina. If you do not get any email then perhaps the script is unable to send the email to the standard mailout.neosys.com:2500 email server due to a firewall. In this case after 10 minutes or so you should be able to reconnect using ssh anyway.
*upgradecygwin.log - contents of the email that would have been sent
*upgradecygwin.err - any errors that prevent sending the email
If you cannot connect on ssh using tunnelier/remmina after say 20 minutes then the script must have failed. To resolve that problem, either use your existing Teamviewer connection or get client IT support to physically access the server to install Teamviewer for you.
Running the script multiple times will not cause any issue. If there is little or nothing to upgrade then the time to complete will be short since there is less to download and install.
=====Verifying successful run=====
#You must carefully inspect the email or log for "error" or "fail" and intelligently and thoughtfully find any other unexpected results and deal with them. It is impossible to give guidelines for everything so this requires brainwork.
#[[Setting_up_and_using_remote_support#How_to_check_Cygwin_version_.3F|You must check the versions of "cygwin" and "openssh"]] at a minimum and ensure they agree with the latest expected version numbers.
#You must check for the word "reboot" especially in the following scenarios:
Installing file cygfile:///usr/bin/cygwin1.dll
io_stream_cygfile: fopen(/usr/bin/cygwin1.dll) failed 13 Permission denied
Failed to open cygfile:///usr/bin/cygwin1.dll for writing.
Scheduled reboot replacement of file C:\cygwin\bin/cygwin1.dll with C:\cygwin\bin/cygwin1.dll.new
mbox note: In-use files have been replaced. You need to reboot as soon as possible to activate the new versions. Cygwin may operate 
incorrectly until you reboot.
note: In-use files have been replaced. You need to reboot as soon as possible to activate the new versions. Cygwin may operate incorrectly
until you reboot.
Ending cygwin install
=====Dealing with reboot required=====
The script attempts to shutdown sshd and some services that may be present in some installations like rsync and exim.
The script attempts to avoid causing "reboot required" by stopping the upgrade if any cygwin processes are found to be running. "Reboot required" indicates that some cygwin program was running while the upgrade process was running and this usually IRRETRIEVABLY BREAKS the cygwin functionality because cygwin's upgrade isnt smart enough to deal with this.
It is quite likely that a reboot will NOT solve various problems.
Rerunning the script will not show the errors again but the problem of bad upgrade.
SOLUTION: You should completely clean out all traces of cygwin in the computer and then reinstall cygwin completely from scratch. How to clean thoroughly is documented in wiki.
=====Finding the script=====
The latest version of the script can be found in the latest version of NEOSYS. The script is installed in the neosys\neosys directory.
For older versions of NEOSYS it can be created or upgraded as follows:
Open http://www.neosys.com/support/upgradecygwin.cmd on your browser to view the script.
Then copy the script onto notepad on the server and save this as a .cmd file in the location mentioned below:
Single installation
x:\neosys\neosys\upgradecygwin.cmd
Multiple installation
x:\hosts\CLIENTCODE\neosys\upgradecygwin.cmd
where x is the drive in which NEOSYS is installed.
====How to check Cygwin version ?====
If you are looking for the version number for the whole Cygwin release, there is none.
Each package in the Cygwin release has its own version. You can find out the Cygwin.dll version by using the following command:
cygcheck -V
To find the version of the Cygwin Package installed, you can use
cygcheck -c PACKAGE_NAME
eg - To check the version of the openssh package you will have to type the following command in cygwin:
cygcheck -c openssh
The output should be as follows:
<pre>
Package          Version          Status
openssh          6.0p1-2            OK
</pre>
==How to uninstall/reinstall cygwin==
Teamviewer required as sshd service will be removed.
1. In Cygwin, list installed service names (repeat step 2 to 3 for each service):
cygrunsrv -L
2. Stop all services in step 1:
cygrunsrv --stop <service_name>
3. Remove service:
cygrunsrv --remove <service_name>
4. Delete sshd and sshd_server users:
net user sshd /DELETE
5. Delete all the files/directories *within* C:\cygwin (in DOS prompt):
  rmdir /s /q C:\cygwin
  rmdir /s /q C:\cygwin
6. Delete desktop shortcut but keep the installation setup-x86.exe in downloads if available to reinstall if required.
The cygwin folder can't be deleted due to a Access Denied error, even when all cygwin services are stopped.
[[#Installing_Cygwin_with_OPENSSH | Install Cygwin]]
Ref: https://cygwin.com/faq/faq.html#faq.setup.uninstall-service
==Adding packages to Cygwin after installation==
Adding packages causes Cygwin to also upgrade but upgrade requires a special process because it cant be upgraded remotely while Cygwin sshd server is working.
#Upgrade Cygwin
#Add the package using Cygwin normal setup program
Step 1 is NOT optional if you want to do step 2.
In the above procedure upgrade Cygwin using the script and follow the precautionary measures listed in [[Setting up and using remote support#Upgrading Cygwin with a script | Upgrade using script]], in case script fails to renable ssh remote connection. Next run setup.exe file present in D:\neosys\neosys to install the required the package.
===Adding individual packages to cygwin without doing a full upgrade===
You can add individual packages to cygwin without doing a full upgrade in many cases. The installed or upgraded version of cygwin should be recent since the current version of the package you want to install might not work with an old version installed cygwin.dll.
To figure out if the cygwin version is recent and will be compatible with the new package, compare the current installed version with the latest version of cygwin.
Cygwin DLL has been named cygwin1.dll and the number 1 is present in the beginning of the release name. Additionally there are DLL major and minor numbers that correspond to the name of the release and a release number respectively. The major version number gets incremented only when a change is made that makes existing software incompatible. The minor version changes every time a new backward compatible Cygwin release is made available. Therefore we need to check the major version of cygwin on the server.
In other words cygwin-1.7.1-2 means cygwin1.dll, major version 7, minor version 1 and release 2.
e.g if the current version of Cygwin DLL is 2.3.0 and latest version is 2.4.1-1 that means there is a change in the major version from 3 to 4 so we cannot go ahead with installing a new package.
Commands below to add or remove packages. Press the View button repeatedly in the installation wizard to get to "Pending" to see what will be installed.
#adding
setup-x86 -P PACKAGE_NAME


You cannot delete the cygwin folder from Windows explorer due to a Access Denied error and this is the best way to uninstall cygwin.
#removing
setup-x86 -x PACKAGE_NAME


==== Getting Ownership and Permissions Correct ====
==Getting Ownership and Permissions Correct==


Installation of cygrin under domain administrator account needs to be fixed as follows:
Installation of cygwin under domain administrator account needs to be fixed as follows:


#c:\cygin Properties, Security, Advanced
#c:\cygwin Properties, Security, Advanced
#Change owner to: Administrators
#Change owner to: Administrators
#Tick: Replace owner on subcontainers
#Tick: Replace owner on subcontainers
Line 296: Line 572:
  chown sshd_server /var/empty
  chown sshd_server /var/empty


==== Configuring Firewall/Router ====
==Configuring Firewall/Router==


You will have to port forward 19580 on the router to port 19580 on the neosys server. Some routers call port forwarding “port mapping” or “virtual servers”
You will have to port forward 19580 on the router to port 19580 on the neosys server. Some routers call port forwarding “port mapping” or “virtual servers”
Line 302: Line 578:
It is BAD idea to simply open port 22 since an open port 22 attracts scanners/hackers like flies.
It is BAD idea to simply open port 22 since an open port 22 attracts scanners/hackers like flies.


Also configure port forwarding of port 4430 to port 4430. This is for remote access via https if desired at a later date.
Configure port forwarding of port 4430 ONLY if access from outside office is required by the client. Support MUST obtain Client management permission before port forwarding 4430.


==== Configuring Specific Client Routers ====
==Configuring Specific Client Routers==


[[Adline Dubai - CISCO PIX Firewall]]
[[Adline Dubai - CISCO PIX Firewall]]
Line 310: Line 586:
[[Sonicwall Firewall Configuration]]
[[Sonicwall Firewall Configuration]]


==== How to install ssh on port 19580 over vnc on port 19580 ====
==How to install ssh on port 19580 over vnc on port 19580==


Install vnc on port 19580
Install vnc on port 19580
Line 332: Line 608:
wait for 10 mins and try to ssh login on port 19580
wait for 10 mins and try to ssh login on port 19580


==== Changing user on Cygwin ====
==Changing user on Cygwin==


On SSH command line:
On SSH command line:
Line 338: Line 614:
  ssh neosys@127.0.0.1 (where 'neosys' is the username)
  ssh neosys@127.0.0.1 (where 'neosys' is the username)


=== Installing and configuring UltraVNC ===
==Installing and configuring UltraVNC==


Download UltraVNC from http://www.ultravnc.com
VNC/Putty is not typically used for NEOSYS remote support anymore and has been replaced by tunnelier/rdp


Run installation – all default options EXCEPT choose INSTALL AS A SERVICE, and START SERVICE.
[[Installing and configuring UltraVNC]]


((screenshot required))
==Remote Desktop Connection==


Run ‘Service Helper’ under UltraVNC Server in case you don’t see vnc tray icon on the notification area.
Servers are normally not exposed to the internet so IT staff and suppliers are often not careful to use strong passwords and use things like "password" or blank.


Admin options should make the following changes:
Given the above, it is NEOSYS policy NOT to use remote desktop via direct access from the internet at all and especially not long term. This is to prevent worms from instantly discovering possible entry points - typically before NEOSYS can even begin to enforce strong administrator password.
[[Image:Vncadmin.png]]
 
# Deselect – Enable Java (to increase security)
If it is otherwise IMPOSSIBLE (difficult or inconvenient does NOT count as impossible!) to avoid using remote desktop protocol to the public internet then a simple and effective way of significantly increasing security is to change the remote desktop port from 3389 to something else e.g. 33890 as per NEOSYS convention.
# Select – Allow loopback connection (essential for connection via ssh)
 
# Select - Loopback only (to increase security)
===Changing RDC port from standard to nonstandard===
 
#Start Registry Editor.
#Locate and then click the following registry subkey:
 
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\TerminalServer\WinStations\RDP-Tcp\PortNumber


[[Image:vncserveraddnewclient55000.png]]
#On the Edit menu, click Modify, and then click Decimal.
#Type the new port number, and then click OK.
#Quit Registry Editor.


==== Installing and configuring the client ====


It is advisabke to use http over ssh rather than remote access IE inside vnc because the user interface is much faster alhough it might not be so bad using vnc to hosts in the same country
==Solving "Authentication that can continue: publickey,password" Error when connecting to remote servers via remote access clients==


==== Installing Putty ====
Some remote access clients cannot connect to ssh servers without special configuration.


http://the.earth.li/~sgtatham/putty/latest/x86/putty.exe
For example remina/ssh cannot connect to windows/cygwin/sshd in their default configuration.


Download putty.exe > run
===Error Message===
[[Image:Sshremmina.jpg]]


===== Configuring Putty =====
SSH password authentication failed: Access denied. Authentication that can continue: publickey,password,keyboard-interactive


ssh tunneling local:127.0.0.1:1235-->remote:127.0.0.1:80 so you can do http://127.0.0.1:1235/neosys on your own computer
===Solution 1===


there are a few computers where it doesnt seem to like the above usually where the target server is on a none-standard port or not listening on 127.0.0.1
If possible configure the client to not perform challenge response during login.


so sometimes you have to map local 127.0.0.1:8080 to remote 10.0.0.255:8080 or whatever the listening ip and port is
There appears to be no way to do this for remina currently


[[Image:puttysession.png|Fill in the host name of the server computer and the port number.]]
===Solution 2===


[[Image:puttyssh.png]]
On the target server:


[[Image:puttytunnel.png]]
Edit the ssh service configuration
Source: 1234 Destination: 127.0.0.1:5900 then click Add


Source: 1235 Destination: 127.0.0.1:80 then click Add
  nano /etc/sshd_config


Once tunnels are setup then return to Session tab in putty and save the options then click open to connect to the server. Login as administrator with the servers password.
Add the last line to the following section


Ultra VNC Viewer:
<pre>
# Change to no to disable s/key passwords
#ChallengeResponseAuthentication yes
ChallengeResponseAuthentication no
</pre>


As the port mapping is complete, enter 127.0.0.1:1234 to login into the computer screen remotely:
Restart the ssh service


==== Configuring Ultra VNC Viewer Listen mode to use a non-standard port ====
net stop sshd
net start sshd


[[Image:vncviewerlisten55000.png]]
Check that you can login using password from one workstation and it will be solved for all workstations for that server


==== Troubleshooting Client ====
===Solution 3===


===== Cygwin login on servers connected to any domain controller =====
On a client workstation:


If NEOSYS is loaded on a server which is connected to any domain controller, it may require the local administrator password reset.
#Use the autologin.sh script to configure automatic login. Refer [[Backup_and_Restore#Creating.2FUpgrading_autologin.sh| Autologin.sh]]
#For "Authentication/Login Method" choose option "Public Key"


==== Resolving VNC Password lockout ====
Check that you can login using password. This will have to be done on every workstation for every server so is rather tedious but it does not require reconfiguration of the server.


If you connect to a client server through CYGWIN and enter the VNC password 3 times wrong in a row, VNC will block you.
===Solution 4===


As you have already entered the server through CYGWIN, you can reactivate the blocked VNC by using the following commands on the prompt:
On the target server, check whether authorized_keys file contains your public key. You can do that by checking the user name displayed at the end of each key.


net stop winvnc
To view the authorized_keys file, open cygwin terminal and type


  net start winvnc
  cat .ssh/authorized_keys


== Remote Desktop Connection ==
If authorized_keys file does not contain your public key, then copy it from authorized_keys.backup file using the below command:


Servers are normally not exposed to the internet so IT staff and suppliers are often not careful to use strong passwords and use things like "password" or blank.
cat .ssh/authorized_keys.backup


Given the above, it is NEOSYS policy NOT to use remote desktop via direct access from the internet at all and especially not long term. This is to prevent worms from instantly discovering possible entry points - typically before NEOSYS can even begin to enforce strong administrator password.
Next edit the authorized_keys file using the below command:


If it is otherwise IMPOSSIBLE (difficult or inconvenient does NOT count as impossible!) to avoid using remote desktop protocol to the public internet then a simple and effective way of significantly increasing security is to change the remote desktop port from 3389 to something else e.g. 33890 as per NEOSYS convention.
nano .ssh/authorized_keys


=== Changing RDC port from standard to nonstandard ===
Then paste the copied key in a new line. Ensure that the key appears in a single line and then close the authorized_keys file.


# Start Registry Editor.
Check that you can connect to the target server using automatic SSH authentication (SSH Agent or Public key)
# Locate and then click the following registry subkey:
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\TerminalServer\WinStations\RDP-Tcp\PortNumber
# On the Edit menu, click Modify, and then click Decimal.
# Type the new port number, and then click OK.
# Quit Registry Editor.

Latest revision as of 11:24, 15 February 2021

Getting agreement of client IT staff to provide remote access

Letter to obtain agreement of client IT staff to provide remote access

Initial Connection to the server before setting up permanent remote connection

For remote installation you need to get an initial connection to the server before you can setup Cygwin for a permanent remote connection.

Get the one-time run quick support teamviewer utility from [Quick Support]. Otherwise refer to the wiki to install customised reverse connect "UltraVNC" SC file.

If the client has already gone ahead and provided Microsoft RDP with an obvious/weak system password, then Support MANDATORY MUST get Windows reinstalled from scratch. Antivirus may not be able to tell that the server has been infected and rootkitted and therefore a scan does not prove it has not been infected.

Support MUST not provide NEOSYS support via Microsoft Remote Desktop Client (RDP/RDC) on port 3389 at anytime because it is a BAD idea to simply open port 3389 since an open port 3389 attracts scanners/hackers like flies.

Also, IT suppliers not aware of the situation often setup the initial administrator password to something obvious like "password" or the arent-I-clever "P@ssw0rd" or even blank. In this case there is a good chance internet worms will discover the "open door" and install themselves before you get the chance to put a strong password.

Installing and configuring SSH

Installing Cygwin with OPENSSH

These instruction are only for installing in a server NOT part of a domain. For installing in a server that is part of a domain, see http://cygwin.com/faq-nochunks.html#faq.using.sshd-in-domain

Watch out for non-intuitive steps like clicking "skip" to install something.

Read Avoid Corrupting Cygwin Installations

  1. Instruct client to login to server as Administrator.
  2. Connect to client server via Teamviewer or customised reverse connect UltraVNC SC file.
  3. ENSURE that you are logged in as the local (NOT DOMAIN) administrator.
  4. Download/Run/Install http://www.cygwin.com/setup.exe
    (you might have to go to the home page http://www.cygwin.com and click the link to setup.exe)
  5. Download source: Install from Internet
  6. Root Directory: c:\cygwin
  7. Local Package Directory: c:\cygwin.lib
  8. Choose "yes" to "Folder does not exist. Create new?"
  9. Internet Connection: Direct Connection
  10. Download Site: http://mirrors.kernel.org (near the bottom)
    (If this does not show in the list, key in the URL in the field User URL and click on Add)
  11. Select Packages: Maximise window then click View once to get Full. You can then enter the name of the desired packages in the Search box to speed up location of the desired packages.
  12. Next to the package OPENSSH, click the word Skip (once!) to get version 4.4p1-1 or later
  13. Next to the package NANO, click the word Skip (once!) to get the latest version available
  14. Check the NEOSYS INSTALLATION CHECKLIST for any other packages to install like the above.
  15. Click Next and complete the installation

Win32 Error

The Win32 Error occur when the bad file is cached in internet explorer cache. You can try clearing the internet explorer cache and redownloading or you can try to download from cygwin.com instead of www.cygwin.com so it doesnt look in the cache or www.cygwin.com if your original download was from cygwin.com. All else failing, you can simply upload the setup.exe file from your own pc to the server.

All this relates to win32 error when running a downloaded file. Any downloaded file and not just cygwin.com/setup.exe

Error during setup

In case of the following error, check for proxy settings in internet explorer. It is possible that the client uses a proxy setting. In that case, in Step 7 instead of choosing Direct Connection, choose Use Internet Explorer Proxy Setting.

Unable to get setup.ini from <http://mirrors.kernel.org/>


Cygwin install error.png

Configuring and starting SSHD

Open the Cygwin icon to get a linux/bash command line and type:

Run the following commands: (not needed in recent versions of Cygwin so dont do this)

chmod +r /etc/passwd
chmod +r /etc/group
chmod 777 /var

Refer here if you get an error while doing the above steps.

Prevent cygwin from using Unix like permissions on files it creates

nano /etc/fstab

add the line or just add ",noacl" to the existing similar line. (What is the effect of omitting this?)

none /cygdrive cygdrive binary,posix=0,user,noacl 0 0

Thereafter start with the ssh configuration:

ssh-host-config

Then on the following options type:

Only asked if running again:
Overwrite existing /etc/ssh_config file? yes
Overwrite existing /etc/sshd_config file? yes
.
StrictModes                 -   no
Privilege                   -   yes
New local sshd account      -   yes
Install SSHD as a service   -   yes
Enter value of daemon       -   Just press Enter
Different name              -   no
Create new privileged user  -   yes
Enter a password now        -   Invent a NEW totally random password with caps and both upper and lower case.
Re-enter the password       -   Enter it again. Dont record it anywhere. Forget it.

At the command prompt type

net start cygsshd

For older versions of Cygwin (Before Jan 2019)

net start sshd

Configuring SSHD to use a non-standard port number

This is necessary if the router cannot forward port 19580 --> 22 and we don’t want to open port 22 directly.

Capitalization is SIGNIFICANT AND CANNOT BE IGNORED in cygwin/linux commands

open cygwin command prompt

nano /etc/sshd_config

change the Port to look like this:

#Port 22
Port 19580

Also add the last line to the following section. Refer Error when connecting to remote servers to see why this line is added.

# Change to no to disable s/key passwords
#ChallengeResponseAuthentication yes
ChallengeResponseAuthentication no

Press Ctrl+x to save. On the confirmation type Y and on the next prompt hit enter.

net stop cygsshd
net start cygsshd

For older versions of Cygwin (Before Jan 2019)

net stop sshd
net start sshd

To check that the server is running and listening on port 19580

ssh -p 19580 administrator@localhost

If you are asked for to confirm the server id is correct or enter password then the check is successful. No need to continue.

Changing ssh login from “Administrator” to “administrator”

Current NEOSYS policy to cater for recent versions of Cygwin is to rename the windows Administrator user to administrator to keep a consistent ssh login across all installations.

If you forget to do this before installing or upgrading Cygwin then you must to the following:

  1. Rename “Administrator” to “administrator” in Windows
  2. In a Cygwin console do:
mkpasswd > /etc/passwd

It should come back with nothing

Error while changing Cygwin port 22 to 19580

Error Message:

"Could not open file for writing: permission denied"

Occurrence: Sometimes when you edit the sshd_config file through NANO.

Solution: In SSH shell, follow these commands:

cp sshd_config ashwin_temp #copies sshd_config to a new file ashwin_temp
rm sshd_config             #deletes sshd_config
cp ashwin_temp sshd_config #copies ashwin_temp to sshd_config

In case it does not copy sshd_config to ashwin_temp, than check whether an ashwin_temp filename exists and delete it using the rm command.

Enable Remote Connection on Windows to allow connection through Remmina

  1. Open System Properties from Explorer
  2. Click on Remote settings
  3. Under 'Remote' section, check Remote Assistance (disabled on Win 2008 servers) and Remote Desktop to allow remote connections to this computer.

Win remote connection.png

Opening up ssh connections to additional source ip nos

Starting a NEOSYS process will automatically restrict cygwin ssh to accept connections from known NEOSYS company static ip numbers.

In the cygwin command line, insert a line in the list of allowable hosts

DO NOT ALLOW ALL OR GENERAL SSH ACCESS TO NEOSYS CLIENTS SERVERS WITHOUT GETTING PERMISSION *AND* INSTALLING EMAIL ALERTS FOR LOGINS AS DESCRIBED BELOW

nano /etc/hosts.allow

Enter IP numbers or CIDR format:

sshd 12.34.56.78
sshd 12.34.0.0/16

Setting up email alerts for cygwin ssh logins

1. Use http://www.cygwin.com/setup.exe to install "email" and "whois" packages. MUST READ See what modules Cygwin is going to update

2. Run Cygwin and copy & paste script below into new file sshrc. Change it@neosys.com to the email ID to which the alert needs to be sent.

nano /etc/sshrc
#!/bin/bash
#
#you configure this

ALERTEMAILADDRESS=it@neosys.com

#
#get the ip number without the ipv6 prefix
FROMIPNO=`echo $SSH_CLIENT|cut -f 1 -d " "|sed 's/::ffff://'`
#
#quit with no message if from a known host

if grep -x $FROMIPNO /etc/trustedipnos
then exit
fi

#
#get the host name by reverse lookup

FROMHOST=`nslookup $FROMIPNO|grep "name ="`

#
#get whois info about the login ip number

#and pipe it into the mail program
#"&" on the end creates a new process in order not to delay login

whois $FROMIPNO|\
email -q -f nl1@neosys.com -s "login $USER $FROMIPNO $FROMHOST" -r \
mailout.neosys.com -p 2500 $ALERTEMAILADDRESS&

3. Give execute permission to sshrc script file for all groups (owner, group, other):

chmod a+x sshrc 

4.Add trusted IPs by copying & pasting text below:

cd /etc
nano trustedipnos
#IP ranges and CIDR etc not accepted yet

#vm1.neosys.com for remote checking
85.17.154.105

#nl1.neosys.com
83.149.104.167

#nl2.neosys.com
85.17.154.66

#uk.neosys.com
78.143.212.191

#nl3.neosys.com
94.75.233.2

Quick way of adding all Support's public keys to allow Remmina support

If a server has lost all authorised keys in .ssh/authorized_keys file, then instead of Support adding their public key individually using "./autologin.sh" use this method:

  1. Connect via SSH to any other client server that has support team's public keys saved.
  2. Then open Cygwin and type:
    cat .ssh/authorized_keys
  3. Select and copy all the text in the file. i.e public keys
  4. Exit and connect to the the new client and open Cygwin and type:
    nano .ssh/authorized_keys
  5. Right click and paste the copied keys in a new line below any possible existing keys ensuring that each key appears in a separate single line and then save and close the authorized_keys file.
  6. Check that you can connect to the target server using automatic SSH authentication (SSH Agent or Public key option) in Remmina.

Testing SSH connection to the NEOSYS server over port 19580

If you cannot connect to the server using SSH, see Troubleshooting NEOSYS remote support port forwarding

Troubleshooting SSH: If SSH connects and then disconnects immediately without exchanging keys

The first time that NEOSYS runs, it automatically adds source ip number restrictions to the sshd remote support configuration in /etc/hosts.allow and /etc/hosts.deny. This is an important security procedure to allow connection to clients systems from NEOSYS ip numbers only. This process allows only local and known NEOSYS ip numbers to connect using SSH. Upgrading NEOSYS will add and/or remove allowable ip numbers as NEOSYS configuration changes.

It is possible that in some client network configurations incoming ssh connections will appear to be from the clients internal routers with an ip unknown to NEOSYS due to NAT configurations. Therefore ssh connections will be blocked unless specifically allow the local ip number or it is added into an upgraded version of NEOSYS.

NOTE: Therefore you must check that remote support via ssh works AFTER you have run NEOSYS once (maintenance mode).

  1. Look in the Windows, Computer Management, System Tools, Event Viewer, Application
  2. Search for entries from source "sshd", double click and look in the Event Properties, Description for ip numbers
  3. Information type sshd entries will give the ip number of successful sshd connections.
  4. Warning type sshd entries will give the ip number of failed sshd connections.
  5. Find the ip number of failed connections.

Possible Problem 1 - Port mapping in router is using NAT

If the ip number of failed connections is some local ip number (of the router for example) then possibly the inbound port forwarding has been done with NAT and the source ip number has been lost. Therefore the NEOSYS ip restrictions are blocking ssh connections because they appear to be coming from an unknown ip number (ie that of the router)

Solution 1A

Change the router configuration to not use NAT and leave the genuine original source IP number

Solution 1B

The router is sadly using NAT instead of plain old port forwarding.

DO NOT USE THIS PROCEDURE TO BREAK NEOSYS SECURITY. DO NOT GRANT ACCESS TO ANY IP OTHER THAN CLIENTS ROUTER IPS

The solution is to add NAT router IP to the list of authorised IP numbers on the NEOSYS server. This solution provides access to NEOSYS server from outside office unrestricted by IP number, hence Client Management approval must be obtained before this solution is applied.

Sample Email to Management-

 Dear XXXX,
 
 Support must have remote access to the NEOSYS server via SSH but currently we don’t have access.
 
 This is because your router is using NAT. The NAT router translates the source IP to its own hence the source IP is lost. NEOSYS server 
 has  a list of allowed source IPs and since the router’s IP is not in the list, connection fails.
 
 The solution to establish successful connectivity is to allow access to NEOSYS server from your NAT router by adding the router’s IP in
 list of allowed IPs on the server.
 
 We need your agreement to carry out this solution because authorizing this access means access to NEOSYS from outside office will not be  
 restricted by IP any more.
 
 Please confirm that this solution is OK.
 
 Best Regards

On receipt of Management approval, add the routers IP number to the list of authorised IP numbers in the cygwin hosts.allow file as follows:

nano /etc/hosts.allow

and add the line as follows but put the IP number of your router

sshd: allow 192.168.0.99

Warning

  1. If the router IP changes then NEOSYS remote support will fail until this line is changed
  2. Do not grant access to 192.168.* etc. since this allows local LAN viruses to attack

Troubleshooting sshd

You can run the sshd service interactively to see all messages instead of having to search logs/events etc.

Unfortunately this will not work the same as the normal windows sshd service unless you assume the identity of the sshd_server user. To assume the identity of the sshd_server user you will have to reset its password to something new (since we dont take a record of it during sshd-host-setup) AND ALSO place the new password in the logon properties of the sshd windows service.

su sshd_server
/usr/sbin/sshd -D -p 19580

Reinstalling SSHD if service fails to startup

Error message

chmod: cannot access '/etc/passwd': No such file or directory
chmod: cannot access ‘/etc/group’: No such file or directory

Solution

Sometimes reinstallation isnt necessary and sshd can be made to restart by doing

mkpasswd > /etc/passwd
mkgroup > /etc/group

If all else fails:

  1. Look in /var/log/sshd.log for errors
  2. Delete the following users: sshd and sshd_server
  3. Remove the sshd service at the cygwin prompt type cygrunsrv –R sshd
  4. Do the above Configuration and starting SSHD step again

Note that you don't have to reinstall cygwin entirely, just sshd with the above steps.

Upgrading SSHD / Cygwin

NEOSYS relies on cygwin to provide secure network access and support various linux/unix services under Windows, mainly rsync for interoffice consolidation.

Just like MS Windows update, cygwin should be updated at regular intervals to close security holes discovered in the software by its authors. This is particularly important for cygwin's remote access service sshd since it is exposed to the internet although on a non-standard port.

Join the cygwin and sshd security news email lists to learn about when cygwin upgrades sshd and/or when there are issues generally with sshd

To find out what versions of cygwin/sshd are installed at NEOSYS clients, in Nagios check "Status Information" of the neosys-ssh service

SSH OK - OpenSSH_7.7 (protocol 2.0) 

Before updating Cygwin or its packages you MUST read Avoid Corrupting Cygwin Installations

Upgrading Cygwin remotely

NEOSYS normal remote server support connection uses cygwin/ssh. Cygwin can be upgraded while in use with a script as explained in the section below.

Upgrading Cygwin with a script

The following script can be used to automatically upgrade cygwin to the latest version quite easily even when people are using NEOSYS. However it carries a small risk described below.

WARNING This script temporarily disconnects and disables all ssh remote support connections, including any ssh connection you are using to initiate the process, for the duration of the upgrade.

Since something may go wrong and the script might FAIL to re enable ssh remote connections, you can take one of the precautionary measures listed below.

  • either perform a temporary Teamviewer installation. The quick teamviewer zero installation remote support method will not work under rdp/tunnelier/remmina
  • or ensure that client IT support is available ONSITE to provide temporary teamviewer access in the event of any problem
  • or be prepared to lose the ability to provide remote support to the installation until the previous item is available

TeamViewer 9 issue

When attempting to connect to client server via TeamViewer 9 (setup via Tunnelier with unattended access) it shows the error below

TVerror.jpg

SOLUTION: Install TeamViewer 7 which does not give this error. Contact NEOSYS IT for TeamViewer7 commercial license. You must have the client server's administrator password to login using TeamViewer. After the upgrade, REMOVE SETTINGS for unattended access and UNINSTALL Teamviewer. Teamviewer must NOT BE LEFT with permanent login by number and password! Teamviewer options, security, REMOVE "Predefined password (For unattended access)"

Running the script

Locate the upgradecygwin.cmd script and run it some usual way by clicking and pressing Enter.

You MUST inspect the version of the pre-installed script against the version shown at http://www.neosys.com/support/upgradecygwin.cmd and ensure that the script you are using is the latest one, as the script is updated with fixes for problems faced in the past.

The upgradecygwin.cmd script will try to download the latest version of setup-x86 from cygwin.com. In case it is not possible to download the setup-x86.exe file from cygwin.com due to proxy/firewall or other issues, then follow the below steps before running the cygwin upgrade script.

  1. Download setup-x86.exe manually from http://www.cygwin.com/setup-x86.exe
  2. Place it in the same directory as the upgrade script
  3. Rename it to "setup-x86-manual.exe". The cygwin upgrade script will rename this file to setup-x86.exe

If you initiate the script while connected on ssh using tunnelier/remmina etc. halfway through the script you will be disconnected.

The script will take a few minutes to download and install any cygwin upgrades.

Once the script is finished, it will re-enable creation of new incoming ssh connections and attempt to send an email to support@neosys.com via the standard mailout.neosys.com:2500 email server.

You should then be able to reconnect using ssh and tunnelier/remmina. If you do not get any email then perhaps the script is unable to send the email to the standard mailout.neosys.com:2500 email server due to a firewall. In this case after 10 minutes or so you should be able to reconnect using ssh anyway.

  • upgradecygwin.log - contents of the email that would have been sent
  • upgradecygwin.err - any errors that prevent sending the email

If you cannot connect on ssh using tunnelier/remmina after say 20 minutes then the script must have failed. To resolve that problem, either use your existing Teamviewer connection or get client IT support to physically access the server to install Teamviewer for you.

Running the script multiple times will not cause any issue. If there is little or nothing to upgrade then the time to complete will be short since there is less to download and install.

Verifying successful run
  1. You must carefully inspect the email or log for "error" or "fail" and intelligently and thoughtfully find any other unexpected results and deal with them. It is impossible to give guidelines for everything so this requires brainwork.
  2. You must check the versions of "cygwin" and "openssh" at a minimum and ensure they agree with the latest expected version numbers.
  3. You must check for the word "reboot" especially in the following scenarios:
Installing file cygfile:///usr/bin/cygwin1.dll
io_stream_cygfile: fopen(/usr/bin/cygwin1.dll) failed 13 Permission denied
Failed to open cygfile:///usr/bin/cygwin1.dll for writing.
Scheduled reboot replacement of file C:\cygwin\bin/cygwin1.dll with C:\cygwin\bin/cygwin1.dll.new
mbox note: In-use files have been replaced. You need to reboot as soon as possible to activate the new versions. Cygwin may operate  
incorrectly until you reboot.
note: In-use files have been replaced. You need to reboot as soon as possible to activate the new versions. Cygwin may operate incorrectly 
until you reboot.
Ending cygwin install
Dealing with reboot required

The script attempts to shutdown sshd and some services that may be present in some installations like rsync and exim.

The script attempts to avoid causing "reboot required" by stopping the upgrade if any cygwin processes are found to be running. "Reboot required" indicates that some cygwin program was running while the upgrade process was running and this usually IRRETRIEVABLY BREAKS the cygwin functionality because cygwin's upgrade isnt smart enough to deal with this.

It is quite likely that a reboot will NOT solve various problems.

Rerunning the script will not show the errors again but the problem of bad upgrade.

SOLUTION: You should completely clean out all traces of cygwin in the computer and then reinstall cygwin completely from scratch. How to clean thoroughly is documented in wiki.

Finding the script

The latest version of the script can be found in the latest version of NEOSYS. The script is installed in the neosys\neosys directory.

For older versions of NEOSYS it can be created or upgraded as follows:

Open http://www.neosys.com/support/upgradecygwin.cmd on your browser to view the script.

Then copy the script onto notepad on the server and save this as a .cmd file in the location mentioned below:

Single installation

x:\neosys\neosys\upgradecygwin.cmd

Multiple installation

x:\hosts\CLIENTCODE\neosys\upgradecygwin.cmd

where x is the drive in which NEOSYS is installed.

How to check Cygwin version ?

If you are looking for the version number for the whole Cygwin release, there is none.

Each package in the Cygwin release has its own version. You can find out the Cygwin.dll version by using the following command:

cygcheck -V

To find the version of the Cygwin Package installed, you can use

cygcheck -c PACKAGE_NAME

eg - To check the version of the openssh package you will have to type the following command in cygwin:

cygcheck -c openssh

The output should be as follows:

 Package           Version           Status
 openssh           6.0p1-2             OK

How to uninstall/reinstall cygwin

Teamviewer required as sshd service will be removed.


1. In Cygwin, list installed service names (repeat step 2 to 3 for each service):

cygrunsrv -L

2. Stop all services in step 1:

cygrunsrv --stop <service_name>

3. Remove service:

cygrunsrv --remove <service_name>

4. Delete sshd and sshd_server users:

net user sshd /DELETE

5. Delete all the files/directories *within* C:\cygwin (in DOS prompt):

rmdir /s /q C:\cygwin

6. Delete desktop shortcut but keep the installation setup-x86.exe in downloads if available to reinstall if required.

The cygwin folder can't be deleted due to a Access Denied error, even when all cygwin services are stopped.

Install Cygwin

Ref: https://cygwin.com/faq/faq.html#faq.setup.uninstall-service

Adding packages to Cygwin after installation

Adding packages causes Cygwin to also upgrade but upgrade requires a special process because it cant be upgraded remotely while Cygwin sshd server is working.

  1. Upgrade Cygwin
  2. Add the package using Cygwin normal setup program

Step 1 is NOT optional if you want to do step 2.

In the above procedure upgrade Cygwin using the script and follow the precautionary measures listed in Upgrade using script, in case script fails to renable ssh remote connection. Next run setup.exe file present in D:\neosys\neosys to install the required the package.

Adding individual packages to cygwin without doing a full upgrade

You can add individual packages to cygwin without doing a full upgrade in many cases. The installed or upgraded version of cygwin should be recent since the current version of the package you want to install might not work with an old version installed cygwin.dll.

To figure out if the cygwin version is recent and will be compatible with the new package, compare the current installed version with the latest version of cygwin.

Cygwin DLL has been named cygwin1.dll and the number 1 is present in the beginning of the release name. Additionally there are DLL major and minor numbers that correspond to the name of the release and a release number respectively. The major version number gets incremented only when a change is made that makes existing software incompatible. The minor version changes every time a new backward compatible Cygwin release is made available. Therefore we need to check the major version of cygwin on the server.

In other words cygwin-1.7.1-2 means cygwin1.dll, major version 7, minor version 1 and release 2.

e.g if the current version of Cygwin DLL is 2.3.0 and latest version is 2.4.1-1 that means there is a change in the major version from 3 to 4 so we cannot go ahead with installing a new package.

Commands below to add or remove packages. Press the View button repeatedly in the installation wizard to get to "Pending" to see what will be installed.

#adding
setup-x86 -P PACKAGE_NAME
#removing
setup-x86 -x PACKAGE_NAME

Getting Ownership and Permissions Correct

Installation of cygwin under domain administrator account needs to be fixed as follows:

  1. c:\cygwin Properties, Security, Advanced
  2. Change owner to: Administrators
  3. Tick: Replace owner on subcontainers

After changing ownership of all cygwin folders to Administrators all ssh login will be blocked and you will get a windows application event log message. "root" actually means sshd's user which is sshd_server by default or can be found in the cygwin ssh windows services properties under log on

fatal: /var/empty must be owned by root and not group or world-writable.

Fix this in cygwin console as follows:

chown sshd_server /var/empty

Configuring Firewall/Router

You will have to port forward 19580 on the router to port 19580 on the neosys server. Some routers call port forwarding “port mapping” or “virtual servers”

It is BAD idea to simply open port 22 since an open port 22 attracts scanners/hackers like flies.

Configure port forwarding of port 4430 ONLY if access from outside office is required by the client. Support MUST obtain Client management permission before port forwarding 4430.

Configuring Specific Client Routers

Adline Dubai - CISCO PIX Firewall

Sonicwall Firewall Configuration

How to install ssh on port 19580 over vnc on port 19580

Install vnc on port 19580

connect on vnc

setup cygwin sshd on port 22

test you can login on port 22

ssh neosys@127.0.0.1

change sshd port to 19580 (but it wont start)

schedule a windows system reboot in 10 mins at windows command prompt

shutdown -t 600

change vnc port to 5900 (if will disconnect you)

wait for 10 mins and try to ssh login on port 19580

Changing user on Cygwin

On SSH command line:

ssh neosys@127.0.0.1 (where 'neosys' is the username)

Installing and configuring UltraVNC

VNC/Putty is not typically used for NEOSYS remote support anymore and has been replaced by tunnelier/rdp

Installing and configuring UltraVNC

Remote Desktop Connection

Servers are normally not exposed to the internet so IT staff and suppliers are often not careful to use strong passwords and use things like "password" or blank.

Given the above, it is NEOSYS policy NOT to use remote desktop via direct access from the internet at all and especially not long term. This is to prevent worms from instantly discovering possible entry points - typically before NEOSYS can even begin to enforce strong administrator password.

If it is otherwise IMPOSSIBLE (difficult or inconvenient does NOT count as impossible!) to avoid using remote desktop protocol to the public internet then a simple and effective way of significantly increasing security is to change the remote desktop port from 3389 to something else e.g. 33890 as per NEOSYS convention.

Changing RDC port from standard to nonstandard

  1. Start Registry Editor.
  2. Locate and then click the following registry subkey:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\TerminalServer\WinStations\RDP-Tcp\PortNumber

  1. On the Edit menu, click Modify, and then click Decimal.
  2. Type the new port number, and then click OK.
  3. Quit Registry Editor.


Solving "Authentication that can continue: publickey,password" Error when connecting to remote servers via remote access clients

Some remote access clients cannot connect to ssh servers without special configuration.

For example remina/ssh cannot connect to windows/cygwin/sshd in their default configuration.

Error Message

Sshremmina.jpg

SSH password authentication failed: Access denied. Authentication that can continue: publickey,password,keyboard-interactive

Solution 1

If possible configure the client to not perform challenge response during login.

There appears to be no way to do this for remina currently

Solution 2

On the target server:

Edit the ssh service configuration

 nano /etc/sshd_config

Add the last line to the following section

# Change to no to disable s/key passwords
#ChallengeResponseAuthentication yes
ChallengeResponseAuthentication no

Restart the ssh service

net stop sshd
net start sshd

Check that you can login using password from one workstation and it will be solved for all workstations for that server

Solution 3

On a client workstation:

  1. Use the autologin.sh script to configure automatic login. Refer Autologin.sh
  2. For "Authentication/Login Method" choose option "Public Key"

Check that you can login using password. This will have to be done on every workstation for every server so is rather tedious but it does not require reconfiguration of the server.

Solution 4

On the target server, check whether authorized_keys file contains your public key. You can do that by checking the user name displayed at the end of each key.

To view the authorized_keys file, open cygwin terminal and type

cat .ssh/authorized_keys

If authorized_keys file does not contain your public key, then copy it from authorized_keys.backup file using the below command:

cat .ssh/authorized_keys.backup

Next edit the authorized_keys file using the below command:

nano .ssh/authorized_keys

Then paste the copied key in a new line. Ensure that the key appears in a single line and then close the authorized_keys file.

Check that you can connect to the target server using automatic SSH authentication (SSH Agent or Public key)