EXODUS Knowledge: Difference between revisions

From NEOSYS Technical Support Wiki
Jump to navigationJump to search
No edit summary
 
(86 intermediate revisions by 4 users not shown)
Line 1: Line 1:
==EXODUS Intro==
==TMUX Screens==


EXODUS is the new version of NEOSYS. The entire NEOSYS environment has been converted to Linux, C++ and postgres.
To create the EXODUS maintenance/programming environment
exodus#: ./tmux.exodus
 
===Screens===
<pre>
0: /
    General work for server services/configs. i.e /etc/nagios, /etc/apache2, /var/log/mail.log or general networking.
 
1: /root/exodus/
    Exodus core is installed. Contains build and make libraries here ./libexodus
 
2: /root/exodus/libexodus/exodus
    Core exodus library source code which emulate the AREV system e.g var.cpp
   
3: /root/exodus/cli/src
    Exodus Command Line Interface E.g edir, edic, list, listfiles, delete, createdb, deletedb.
   
4: root/exodus/service
    EXODUS core default host directory but also contains scripts that create/manage requirements of a service (create_service, create_db, create_site, backup_db, restore_db, start, stop, status)
    The www/3/exodus has generic .htm, php and .js which provide basic website functionality i.e Login page, Support Menu, User details, Authorization.
    NEOSYS www files linked into ./www, from /root/neosys/web, which pulled from NEOSYS www git.
 
5: /root/exodus/service/src
    Exodus core service programs (programs that are not related to NEOSYS agency operations and can be used to support other non-NEOSYS services) eg listen, usersubs, sendmail, openfile, select, sort, list,.
 
    Extra info:   
    The dat/ has generic dictionary files. (See Section Dictionaries & Psql Functions)
    TODO: compall currently throws few warnings. Read them and familiarise. In case new warnings appear, notify Steve.   
       
6: /root/neosys
    NEOSYS specific client installation & management scripts e.g .doall, import_db and import_files (from AREV), ./run ./test (see below for these two commands)
   
7: /root/neosys/src
    NEOSYS Agency programs (see section How AREV programs are distributed in EXODUS src/*.cpp
    /root/neosys has the neosys .git
 
8: /root/hosts
    client specific files  e.g logs, images, (shared ./www -> root/exodus/service/www
 
9: /root/exodus/test/src
    series of programs that test whether subroutines (specifically Exodus core libraries) work as they should eg for the var lib - "assert( var("11111").isnum());"
 
10: bash /bin/top
    Monitor for NEOSYS client processes
    Displays customised format: sorted by total time spend and by processor time.
    Press "=" : to change to standard top format
    Press shift + w : to save format settings, replacing the old customised top format
    To go back to the customised format do:
        Press "o"
        Type "COMMAND=serve_agy"
        shift + t : this sorts by total time spent
        shift + p : this sorts by processor time
        shift + w : to save this setting and enter  'y' to confirm
 
11: /var/log/syslog
    Log of NEOSYS client requests
 
12: /root/exodus/service/quick
    Displays the last time the svr file (in /root/hosts/<client>/data/<dbname>) was updated.
</pre>
 
==Object Code/Libraries==
LIVE and TEST processes use different sets of object code.
TEST processes use libraries in ~/lib/, whereas LIVE processes use object code in ~/neo/lib
 
This means development & testing can be done stress free on TEST database, as opposed to testing on production databases.
 
When compiling using edic, the TEST object code is updated if the compilation is successful. (~/lib)
 
==Dictionaries & Psql Functions==
 
Dictionaries - data used to describe other data; specifically the fields of other files.
 
There are three types of fields: <pre>
(F) Field for raw data E.g "Office 44 Building Rush"
(S) Symbolic or calculated E.g for total ans=field1+field2
(G) General or Group E.g "@CRT" which contains always contains the default columns to show on the output of the list program; if no columns provided in list command.
</pre>
 
Show dictionary using "EXO_DATA=test list dict.clients".
 
===Development and deployment using 'dat' files===
 
====Rationale====
 
Part of system development is the creation of various data that is neither programs nor layout i.e. not cpp, h, html, js, php files etc.


#EXODUS is comprised of two main sections:
For example:
*EXODUS is a complete conversion of the AREV CRUD and web service framework. (https://github.com/exodusdb/exodusdb)
*EXODUS NEOSYS is the complete conversion of the NEOSYS Agency AREV programs. (not public git)


===TMUX Screens===
*Dictionaries are data about data.
*Language files are data about text to use for various languages.
*“Change logs” are data about changes in the system.


To create the EXODUS maintenance/programming environment
Historically, in EXODUS and NEOSYS, the above data has been deployed in exodus database files using SQL text files. However SQL files are not convenient for development.
exodus#: ./tmux.exodus
 
Therefore, 'dat' text files will be used now so that standard development tools including editors and git can be fully exploited.
 
===='dat' files====
 
Each database file is represented by an os directory of the same name.
 
Each record in the database file is represented by an os text file where filename is the primary key.
 
For example a record with key '''DEADLINE''' in a dat file '''dict.materials''' would be represented as an os text file '''dat/dict.materials/DEADLINE'''
 
Each line in the os text file represents one field in the db record. In other words, db record FM characters are represented by new line characters in 'dat' files. Any actual new line characters required in the record, and any backslashes, are escaped and appear as '\n' and '\\' in 'dat' files. Other database field separator characters such as VM, SM, TM and STM are stored without any conversion.
 
====Location of dat files====
 
The development versions are stored in exodus and neosys src/dat dirs. They form part of the standard git repositories in parallel with cpp files.
 
The operational versions are stored in ~/dat and ~/live/dat alongside bin and lib dirs and are automatically installed into databases as database files on service startup. Any database functions embedded in the text files (pgsql) are also automatically installed at the same time.
 
===Delete Database dictionary file===
 
cli syncdat deletes database files if the dat dir is empty or contains a file 'SYNCDAT_DELETE'


<pre>
<pre>
SCREEN NAME      STANDARD PATH                            PURPOSE
touch neosys/src/dat/dict.users/SYNC_DELETE
#ex1_root        - /root                                  - general usage
neosys/src~: syncdat
#exodus          - /root/exodus                          -
#exodus src      - /root/exodus/exodus/libexodus/exodus  -
#exo cli        - /root/exodus/cli/src                  -
#service        - /root/exodus/service                  -
#ser src        - /root/exodus/service/src               -
#neosys          - /root/neosys                          - ./doall
#neo src        - /root/neosys/src                       -
#hosts          - /root/hosts                            - 
#test src        - /root/exodus/test/src                  -
#t10            - ~/                                    -
#t11            - ~/                                    -
#t12            - ~/                                    -
</pre>
</pre>


====Editing and deploying a 'dat' file====
It is currently a three step process to edit and deploy such 'dat' files.
=====Edit the 'dat' file (i.e dict.file record)=====
Note that EXODUS service and NEOSYS service have different src/dat folders.
Editing language items:
edir dat/alanguage/SCHEDULES*ARABIC
Editing a dictionary item:
edir dat/dict.materials/DEADLINE
Editing a pgsql function in a dictionary:


===Source code===
edir dat/dict.materials/DEADLINE 8


===Object code===
=====Copy all 'dat' files to ~/dat =====


===Dictionaries===
This step might be removed at a later date.
The file dictionary records are per database, this means
 
This will cause all test databases to immediately restart and load any 'dat' file changes into dictionaries and data files and also create any new or modified pgsql functions.
 
If any ~/neosys/src/dat files were edited:
 
cd ~/neosys/src
./compall dat
 
and/or, if exodus/service/dat files were edited
 
cd ~/exodus/service/src
./compall dat
 
=====Copy all programs and 'dat' files to ~/live/bin|lib|dat=====
 
This should only be run after testing. It will cause all live databases to automatically restart and do the same as the test databases mentioned above.
 
cd ~/exodus/service
./copyall CONFIRM
 
=====Delete dat record =====
 
Simply open the corresponding dat file, remove all lines and save.
 
===Psql unaccent extension===
 
 
 
The unaccent extension and the immutable_unaccent function are created in the db ‘template1’ during exodus installation while user is postgres which is a superuser.
 
Thereafter all new databases are created as copies of template1 so they contain the extension and function
 
====Error====
<pre>
MVDBException:ERROR: function immutable_unaccent(text) does not exist
ERROR:  permission denied to create extension "unaccent"
HINT:  Must be superuser to create this extension.
</pre>


====Solution====
sudo -u postgres psql -c 'ALTER USER exodus WITH SUPERUSER;'


==Processes==
The TEST process for all database use the same object code stored in /root/lib, whereas all LIVE process use the object code in /root/neo/.
The TEST process for all database use the same object code stored in /root/lib, whereas all LIVE process use the object code in /root/neo/.


==Postgres==
Connect into postgres shell:
alias psql='sudo -u postgres psql'
Delete a database:
sudo -u postgres dropdb <dbcode>
==Debugging Exodus==


===Debugging Live/Test services===


./test <dbname> - execute test programs on _test database.


===./doall===
./run <dbname>  - execute test programs on live database.


===Debugging var values===


===Debugging NEOSYS EXODUS in gdb===
Explanation of var flag value in gdb debug mode.


Screen 6: scripts Two scripts can be used to start a process in debug mode. (WARNING: other running TEST or LIVE processes for client will stop during debugging)
<pre>
*./test will use the TEST program object code.
(gdb) p varx                                                         
*./run will use the LIVE object code.
$2 = {var_str = "", var_int = 0, var_dbl = 0, var_typ = {flags_ = 2}}
</pre>


Each exodus var object contains 4 variables of type std::string (var_str), int (var_int), long double (var_dbl) and another int (var_typ i.e flags).


The first 3 bits of var_typ (flags_) are used to signify which of the other variables are assigned.


First bit = string is un/assigned. Second bit = int is un/assigned. Third bit = double un/assigned (a combination of these bits describes whether var_str/ var_int/ var_dbl are assigned.


Examples:


===Converting AREV to Exodus===
000 (binary) = 0 (decimal) = non are assigned.
====Convert Programs====
In AREV: <br>
#Update program source code and check it can be compiled.
#Ensure the program you want to recompile into exodus has below e.g "*c job" in source code on line 2, col 1
#Command: "attach adecomc"          #attaches exodus recompiler/decompiler
#Command: "adecom <programname>"


In Cygwin: <br>
100 (binary) =  1 (decimal) = var_str is assigned. (var_int & var_dbl are not)
#Run "cygdrive/d/exodus/pickos/syncup.sh"    #syncs all decompiled programs to nl19:/backups/current/hosts/win3/pickos/


In Exodus: <br>
010 (binary) = 2 (decimal) = var_int is assigned. (var_str & var_dbl are not)
Command explanations:
#rsync arev decompiled code from sshfs /backups/current/win3/pickos/
  getpickos
#There are three "getpickos" & "compall".  
TODO down
*cd exodus/service/src getpickos && compall
*cd ~/neosys/src && getpickos && compall
*cd ~/neosys/src && getpickos && compile <progname> or


*WARNING following command is DANGEROUS! all LIVE object code updated and all LIVE/TEXT process restarted! 
001 (binary) =  4 (decimal) = var_dbl is assigned. (var_str var_int are not)
**INCLUDES: "exodus compall" and "neosys compall" and "copyall" and "restart all"!
***Use with caution: "cd ~/neosys && compall"


cd exodus/service copyall
Combinations:
TODO up


====Convert dictionary items====
110 (binary) = 3 (decimal) = var_str & var_int are assigned. (var_dbl is not)
Two methods:
a) Quick exodus patch (next sync-up from overwrite with win10a version)
b) Update exodus from AREV


=====Quick Method=====
#Update dict item in win10a
#In exodus run command "e dicta2e"
# Paste below and edit for your case:


====To inspect a “common” variable====
e,g, a variable called ba.ledgern
p ((subExodusProgram::<b>ba</b>_common)<b>ba</b>).<b>ledgern</b>
====To inspect a common block====
e.g. a common block ba
ptype (subExodusProgram::ba_common)ba
====To dump all variables in a common====
e.g all variables in ba
p (subExodusProgram::ba_common)ba
====What types are available in a program?====
<pre>info types common
info types subExodus
info types libexodus</pre>
====Other commands====
explore subExodusProgram::ba_common
===Process Hung or consuming +100% CPU===
If process is consuming +100% CPU then it is likely stuck in an infinite loop.
./doall <CLIENT> stop
./run <CLIENT>
./test <CLIENT>
Search through recent logs for <CLIENT> looking for requests that didn't return OK at the time nagios alerted of Hung process.
A user may also instigate the error again, so monitor the CLIENT's service CPU usage.
If cause found recreate it in test and once test process hit +100% break in test debugger.
If cause not found, you will have to continue checking logs for the request that caused it or hope a user recreates the issue in live debugged service.
Once you've broken in the debugged with the error, enter "n" and hold the enter key until you just seeing line repeat.
Pay attention to start of loops code. E.g WHILE or FOR. (there is likely an error is there)
==./doall==
====General====
~/neosys/doall [LIVE|TEST] <DBCODE|ALL> < ACTION [OPTIONS,..]  | bash -- COMMAND ]
Example:
./doall TEST ALL create_site t-
Result:
Create site for ALL TEST databases, using domain name prefix option "t-".
Screen 6: ./doall script contains all the necessary information(codes) to setup an installation.
It includes scripts to backup, restore, create an Apache site, create/start/stop/status a service, import an AREV database into postgres and more.
====backup_db====
*Does a backup & restore of a LIVE database into the corresponding TEST database.
*Backup <dbcode>.sql file is written to /root/backups/sql; which is rsynced to nl19:/backups/current/exodus/
*Unlike AREV, postgres can perform a "backup" of a database whilst the system is in use.
==Git==
There are two repositories, one for EXODUS and the other for NEOSYS.
===Using git to make changes===
Before following steps you must have a tested updated to a program/file/script. Do not commit untested changes to avoid a messy git history of reverts.
Update your local repo before committing to local repo using the g alias for "git pull --ff-only ; git push && git status'":
<pre>g</pre>
Check which updates/files have not yet been staged and/or committed:
Add your updates to the staged area:
<pre>git add <filename></pre>
or if all the changes made need to be staged:
<pre>git add -a</pre>
Make a commit with a descriptive message on purpose of updates:
<pre>git commit -m <description n purpose of changes></pre>
Again use g alias:
<pre>g</pre>
===Other useful git cmds===
Do not use this commands unless you know what WILL happen.
*git pull - Instead use the safe "git pull --ff-only"
Stick to the alias "g" which does "git pull --ff-only ; git push && git status'"
*git log - display history of commits to master branch
*git diff - display the differences in files between working files and files in local repo.
*git status - display: which updated files are staged/not staged(tracked)
*git stash -
*git branch - switch to a new branch
*git branch <branchname> - will switch to this branch only if it exists
*git checkout - DESTROYS/Updates the state of local working directory with the state of the files in the local repo. (YOU WILL LOSE non staged updates)
*git restore <filename> - DESTROYS/Updates the file with the version in the local repo. (like with git checkout but for a specific file)
*git checkin -
*git revert <commitHash> - reverses a specific commit (use git log to get the chosen commit hash)
*git ..
==Converting AREV programs to EXODUS==
===Decompile AREV to C++===
Do in Master AREV Installation Maintenance mode: (win7)
#ATTACH ADECOMC
#*ADECOM <programname>  *single program
#*ADECOM <prog1> <prog2>
#*ADECOMALL              *all programs  (CHECK FIRST)
Include the option "(V)" in the command to print the C++ to a notepad, which can easily be copy and pasted.
===Getting C++ program to Exodus Installation===
Two methods: (use the first for one off programs)
====Copy & Paste====
Compile program and open source code in notepad
ADECOM <programname> (V)
Find which directory the program belongs. e.g fin/med/job/sys e.t
find /cygdrive/d/exodus/pickos | grep <program name>
If cpp in SYS, the program belongs in ~/exodus/service/src OR if in MED JOB FIN GEN AGY the program belongs in ~/neosys/src
Copy the source code in the text file from the first command.
In Exodus, create a new .cpp (in correct directory) and paste the source code.
Compile & Test
====Rsync====
/d/exodus/arev/syncup.sh
If cpp in SYS then: ~/exodus/service/src ./getpickos
If cpp in MED JOB FIN GEN AGY then: *~/neosys/src
===Compile C++ files to TEST system===
#*./test <DBNAME>
#*~/neosys ./doall TEST <DBNAME> restart    #to get one service to start start using the new lib files
#*~/neosys ./doall TEST all restart        #to get all the services to start start using the new lib files
===Install C++ files to LIVE System===
#~/exodus/service/ ./copyall            #to copy all the ~/lib and bin files to ~/live/lib and bin ... which is used by all exodus/live services
==Writing Standard Exodus Core Function/Method Testing==
Screen 9: ~/exodus/test/src/
There are a series of test programs that check whether methods/functions behave as intended.
They do this using the function, assert.. a 1 or more argument values produce one and only one output)
e.g test_multilang.cpp or test_sort.cpp
Two methods of running test programs:
*Screen 9: make test
*after compiling using edic/compile/c, enter test_prog_name. (Since compile has moved it to ~/bin)
Difference between the two methods is make calls gdb directly;
whereas ~/bin/test_prog_name uses exodus compile program
#~/neosys ./doall LIVE all restart
==Upgrading Exodus==
#Develop patch in Dev system
===Patching neosys/src or exodus/service/src===
#Manually insert patches and compile into test programs.
#If patch dat files then run neosys/compall to get dat files into ~/dat
*If Patsalides:
*#SITE_DIR=ptcy ./test basic
*#login to BASIC_TEST and test
*Other clients:
*#./test DBCODE
*#login to DBCODE and test patches works
#run exodus/service/copyall and check only the patched programs are listed. (if not, see Troubleshoot below)
#Inform users to log off for 10mins. As a rule of thumb, live services must be stopped before copying test programs to live.
#exodus/service/copyall CONFIRM
====Troubleshoot====
=====./copyall lists programs you have not patched=====
Use exodus/service/copyone to copy only your patched programs one by one. (./copyone will not copy dat files)
===Upgrading to new version===
READ: remember there is no test version of libexodus programs. This means that if libexodus programs are compiled and installed, neosys/src and exodus/services LIVE programs MUST BE UPGRADED.
#Upgrade test system to test
##If a dev system like de2, git stash any uncommitted changes other staff may be working on.
##Run "g" in ~/exodus (to do git pull --ff-only)
##Run "m" in ~/exodus (to compile and install libexodus)
##Run "~/neosys/upgrade.all.sh live" in ~/neosys (to upgrade exodus/service and neosys/src live programs)
##Begin testing
#Upgrade clients
##*Client hosted - Schedule a planned upgrade with client.
##*NEOSYS hosted - Inform all clients/users of planned morning upgrade, 24hours in advanced.      (TODO: 'What's New in NEOSYS') AND (emailallusers.cpp)
##Day of upgrade, check USB backup was successful & make a container snapshot, otherwise do not proceed.
##Stop services
##Run "g" in ~/exodus (to do git pull --ff-only)
##Run "m" in ~/exodus (to compile and install libexodus)
##Run "~/neosys/upgrade.all.sh live".
##Check services started automatically without issues.
#Inform users their system has been upgraded and they can login.
===Roll Back Upgrade===
If for any reason the entire upgrade needs to be reverted (excludes clients database), follow steps below:
#Inform users that there is a problem and services need to be stopped.
#Stop all exodus processes.
#Restore the morning's USB backup by: (Note this backup contains:  source/obj code, /etc, neosys scripts but not client data)
##Login into container's host and rsync the morning's usb backup to the container's /:
#*<pre>rsync --dry-run -avz -e 'ssh -p <C_SSHPORT>' /backups/usb/<C_CODE> <C_CODE>:/      (Example: rsync --dry-run -avz -e 'ssh -p 19582' /backups/usb/ad1 ad1:/</pre>)
#Quick sane check, the latest git commit is not the latest git commit in the remote repo.
#Inform users they can login.
==Configuring services to autostart during login==
For PTCY,
#Create a file 'autostart.cfg' in the data folder
touch /root/hosts/<dbname>/data/autostart.cfg
This will autostart a database service as long as at least one database service in that directory is running.
For example, demo service will autostart during login if,
#/root/hosts/demo/data/autostart.cfg exists and
#demo_test service is running
==Speedup postgresql importing databases==
Can be sped up by a factor of 10 BUT may result in corrupt files in case of hard OS crash where the dirty memory is not flushed to storage.
Therefore it is advisable to use this feature only briefly and REMEMBER TO REMOVE IT!
Email to support@neosys.com whenever using it or use personal notes not memory.
nano /etc/postgresql/12/main/postgresql.conf
Uncomment or add a line
fsync = off
Restart postgres … will break any neosys/exodus running processes.
r postgresql
or reload postgres may also work
==Database Attach==
<pre>
<pre>
#include <exodus/program.h>
root@exodus:~/exodus/cli/src# dbattach
programinit()
dbattach - Attach foreign database files to the current default database
 
Syntax is dbattach TARGETDB [FILENAME,...] {OPTION...}


function main() {
Options:
    var originalvalue;


    //edit the string
  F  Forcibly delete any normal files in the current default database.
    var replacevalue="Line Description";


    //check the value of the field TO replace
  R  Detach foreign files
    //SYNTAX: readv(<record/field> from <filename>, <key> [,field])
  RR  Remove foreign database connection.
    readv(originalvalue from "DICT_JOB_INVOICES", "LINE_TEXT", 3);
  RRR Remove all foreign database connections and attachments.
    outputl("REPLACE:"^quote(originalvalue)^" with "^quote(replacevalue));


    //uncomment below 2 lines AFTER using check above
  L  List attached foreign files in the current default database
    //writev(replacevalue from "DICT_JOB_INVOICES", "LINE_TEXT", 3);
    outputl("REPLACED:"^quote(originalvalue)^" with "^quote(replacevalue));
    return 0;
}


programexit()
If no filenames are provided then a database connection is created.
This is *required* before filenames can be provided and attached.
 
Notes:
 
Attachments are permanent until re-attached or removed.
 
Indexes on attached files behave strangely.
</pre>
</pre>
Issues:


#Save, exit and make sure complied.
TARGETDB must be on the same connection as the current default database.
#./doall TEST all bash dicta2e
#Check changes on a TEST database.
#./doall LIVE all bash dicta2e
 


=====Safe Method=====
Current user must have access to the foreign database.
In AREV: <br>
#Update dict item in win10a
#attach adecomc
#adecom <dictfile>      (e.g  adecom dict.production.invoices)
#base2sql


In Cygwin: <br>
If EXO_USER and EXO_PASS are not set then exodus defaults are used.
#/d/exodus/pickos/syncup.sh


In Exodus: <br>
exodus .config file is not used currently.
#neosys/src/getpickos
#doall all import_db R  (DON'T USE U!) - unable to import if database exists!
#TEST database:
## Renew dict items in a test database:
*doall all/xxx create_testdb
*doall all/xxx backup_db
##Check dictionary item in the <dictfile>.cpp file.

Latest revision as of 13:48, 13 January 2023

TMUX Screens

To create the EXODUS maintenance/programming environment

exodus#: ./tmux.exodus

Screens

0: /
    General work for server services/configs. i.e /etc/nagios, /etc/apache2, /var/log/mail.log or general networking.

1: /root/exodus/
    Exodus core is installed. Contains build and make libraries here ./libexodus

2: /root/exodus/libexodus/exodus
    Core exodus library source code which emulate the AREV system e.g var.cpp
    
3: /root/exodus/cli/src
    Exodus Command Line Interface E.g edir, edic, list, listfiles, delete, createdb, deletedb.
    
4: root/exodus/service
    EXODUS core default host directory but also contains scripts that create/manage requirements of a service (create_service, create_db, create_site, backup_db, restore_db, start, stop, status)
    The www/3/exodus has generic .htm, php and .js which provide basic website functionality i.e Login page, Support Menu, User details, Authorization.
    NEOSYS www files linked into ./www, from /root/neosys/web, which pulled from NEOSYS www git.

5: /root/exodus/service/src
    Exodus core service programs (programs that are not related to NEOSYS agency operations and can be used to support other non-NEOSYS services) eg listen, usersubs, sendmail, openfile, select, sort, list,.

    Extra info:    
    The dat/ has generic dictionary files. (See Section Dictionaries & Psql Functions)
    TODO: compall currently throws few warnings. Read them and familiarise. In case new warnings appear, notify Steve.    
        
6: /root/neosys
    NEOSYS specific client installation & management scripts e.g .doall, import_db and import_files (from AREV), ./run ./test (see below for these two commands)
    
7: /root/neosys/src
    NEOSYS Agency programs (see section How AREV programs are distributed in EXODUS src/*.cpp
    /root/neosys has the neosys .git

8: /root/hosts
    client specific files  e.g logs, images, (shared ./www -> root/exodus/service/www

9: /root/exodus/test/src
    series of programs that test whether subroutines (specifically Exodus core libraries) work as they should eg for the var lib - "assert( var("11111").isnum());"

10: bash /bin/top
    Monitor for NEOSYS client processes
    Displays customised format: sorted by total time spend and by processor time.
     Press "=" : to change to standard top format
     Press shift + w : to save format settings, replacing the old customised top format
     To go back to the customised format do:
        Press "o"
        Type "COMMAND=serve_agy"
        shift + t : this sorts by total time spent
        shift + p : this sorts by processor time
        shift + w : to save this setting and enter  'y' to confirm

11: /var/log/syslog
     Log of NEOSYS client requests

12: /root/exodus/service/quick
     Displays the last time the svr file (in /root/hosts/<client>/data/<dbname>) was updated.

Object Code/Libraries

LIVE and TEST processes use different sets of object code. TEST processes use libraries in ~/lib/, whereas LIVE processes use object code in ~/neo/lib

This means development & testing can be done stress free on TEST database, as opposed to testing on production databases.

When compiling using edic, the TEST object code is updated if the compilation is successful. (~/lib)

Dictionaries & Psql Functions

Dictionaries - data used to describe other data; specifically the fields of other files.

There are three types of fields:

(F) Field for raw data E.g "Office 44 Building Rush"
(S) Symbolic or calculated E.g for total ans=field1+field2
(G) General or Group E.g "@CRT" which contains always contains the default columns to show on the output of the list program; if no columns provided in list command.

Show dictionary using "EXO_DATA=test list dict.clients".

Development and deployment using 'dat' files

Rationale

Part of system development is the creation of various data that is neither programs nor layout i.e. not cpp, h, html, js, php files etc.

For example:

  • Dictionaries are data about data.
  • Language files are data about text to use for various languages.
  • “Change logs” are data about changes in the system.

Historically, in EXODUS and NEOSYS, the above data has been deployed in exodus database files using SQL text files. However SQL files are not convenient for development.

Therefore, 'dat' text files will be used now so that standard development tools including editors and git can be fully exploited.

'dat' files

Each database file is represented by an os directory of the same name.

Each record in the database file is represented by an os text file where filename is the primary key.

For example a record with key DEADLINE in a dat file dict.materials would be represented as an os text file dat/dict.materials/DEADLINE

Each line in the os text file represents one field in the db record. In other words, db record FM characters are represented by new line characters in 'dat' files. Any actual new line characters required in the record, and any backslashes, are escaped and appear as '\n' and '\\' in 'dat' files. Other database field separator characters such as VM, SM, TM and STM are stored without any conversion.

Location of dat files

The development versions are stored in exodus and neosys src/dat dirs. They form part of the standard git repositories in parallel with cpp files.

The operational versions are stored in ~/dat and ~/live/dat alongside bin and lib dirs and are automatically installed into databases as database files on service startup. Any database functions embedded in the text files (pgsql) are also automatically installed at the same time.

Delete Database dictionary file

cli syncdat deletes database files if the dat dir is empty or contains a file 'SYNCDAT_DELETE'

touch neosys/src/dat/dict.users/SYNC_DELETE
neosys/src~: syncdat

Editing and deploying a 'dat' file

It is currently a three step process to edit and deploy such 'dat' files.

Edit the 'dat' file (i.e dict.file record)

Note that EXODUS service and NEOSYS service have different src/dat folders.

Editing language items:

edir dat/alanguage/SCHEDULES*ARABIC

Editing a dictionary item:

edir dat/dict.materials/DEADLINE

Editing a pgsql function in a dictionary:

edir dat/dict.materials/DEADLINE 8
Copy all 'dat' files to ~/dat

This step might be removed at a later date.

This will cause all test databases to immediately restart and load any 'dat' file changes into dictionaries and data files and also create any new or modified pgsql functions.

If any ~/neosys/src/dat files were edited:

cd ~/neosys/src

./compall dat

and/or, if exodus/service/dat files were edited

cd ~/exodus/service/src

./compall dat
Copy all programs and 'dat' files to ~/live/bin|lib|dat

This should only be run after testing. It will cause all live databases to automatically restart and do the same as the test databases mentioned above.

cd ~/exodus/service

./copyall CONFIRM
Delete dat record

Simply open the corresponding dat file, remove all lines and save.

Psql unaccent extension

The unaccent extension and the immutable_unaccent function are created in the db ‘template1’ during exodus installation while user is postgres which is a superuser.

Thereafter all new databases are created as copies of template1 so they contain the extension and function

Error

MVDBException:ERROR: function immutable_unaccent(text) does not exist
ERROR:  permission denied to create extension "unaccent"
HINT:  Must be superuser to create this extension. 

Solution

sudo -u postgres psql -c 'ALTER USER exodus WITH SUPERUSER;'

Processes

The TEST process for all database use the same object code stored in /root/lib, whereas all LIVE process use the object code in /root/neo/.

Postgres

Connect into postgres shell:

alias psql='sudo -u postgres psql'

Delete a database:

sudo -u postgres dropdb <dbcode>

Debugging Exodus

Debugging Live/Test services

./test <dbname> - execute test programs on _test database.

./run <dbname> - execute test programs on live database.

Debugging var values

Explanation of var flag value in gdb debug mode.

(gdb) p varx                                                           
$2 = {var_str = "", var_int = 0, var_dbl = 0, var_typ = {flags_ = 2}}

Each exodus var object contains 4 variables of type std::string (var_str), int (var_int), long double (var_dbl) and another int (var_typ i.e flags).

The first 3 bits of var_typ (flags_) are used to signify which of the other variables are assigned.

First bit = string is un/assigned. Second bit = int is un/assigned. Third bit = double un/assigned (a combination of these bits describes whether var_str/ var_int/ var_dbl are assigned.

Examples:

000 (binary) = 0 (decimal) = non are assigned.

100 (binary) = 1 (decimal) = var_str is assigned. (var_int & var_dbl are not)

010 (binary) = 2 (decimal) = var_int is assigned. (var_str & var_dbl are not)

001 (binary) = 4 (decimal) = var_dbl is assigned. (var_str var_int are not)

Combinations:

110 (binary) = 3 (decimal) = var_str & var_int are assigned. (var_dbl is not)


To inspect a “common” variable

e,g, a variable called ba.ledgern

p ((subExodusProgram::ba_common)ba).ledgern

To inspect a common block

e.g. a common block ba

ptype (subExodusProgram::ba_common)ba

To dump all variables in a common

e.g all variables in ba

p (subExodusProgram::ba_common)ba

What types are available in a program?

info types common
info types subExodus
info types libexodus

Other commands

explore subExodusProgram::ba_common

Process Hung or consuming +100% CPU

If process is consuming +100% CPU then it is likely stuck in an infinite loop.

./doall <CLIENT> stop
./run <CLIENT> 
./test <CLIENT>

Search through recent logs for <CLIENT> looking for requests that didn't return OK at the time nagios alerted of Hung process.

A user may also instigate the error again, so monitor the CLIENT's service CPU usage.


If cause found recreate it in test and once test process hit +100% break in test debugger.

If cause not found, you will have to continue checking logs for the request that caused it or hope a user recreates the issue in live debugged service.


Once you've broken in the debugged with the error, enter "n" and hold the enter key until you just seeing line repeat.

Pay attention to start of loops code. E.g WHILE or FOR. (there is likely an error is there)

./doall

General

~/neosys/doall [LIVE|TEST] <DBCODE|ALL> < ACTION [OPTIONS,..] | bash -- COMMAND ]

Example:

./doall TEST ALL create_site t-

Result: Create site for ALL TEST databases, using domain name prefix option "t-".


Screen 6: ./doall script contains all the necessary information(codes) to setup an installation. It includes scripts to backup, restore, create an Apache site, create/start/stop/status a service, import an AREV database into postgres and more.

backup_db

  • Does a backup & restore of a LIVE database into the corresponding TEST database.
  • Backup <dbcode>.sql file is written to /root/backups/sql; which is rsynced to nl19:/backups/current/exodus/
  • Unlike AREV, postgres can perform a "backup" of a database whilst the system is in use.

Git

There are two repositories, one for EXODUS and the other for NEOSYS.

Using git to make changes

Before following steps you must have a tested updated to a program/file/script. Do not commit untested changes to avoid a messy git history of reverts.

Update your local repo before committing to local repo using the g alias for "git pull --ff-only ; git push && git status'":

g

Check which updates/files have not yet been staged and/or committed: Add your updates to the staged area:

git add <filename>

or if all the changes made need to be staged:

git add -a

Make a commit with a descriptive message on purpose of updates:

git commit -m <description n purpose of changes>

Again use g alias:

g

Other useful git cmds

Do not use this commands unless you know what WILL happen.

  • git pull - Instead use the safe "git pull --ff-only"

Stick to the alias "g" which does "git pull --ff-only ; git push && git status'"

  • git log - display history of commits to master branch
  • git diff - display the differences in files between working files and files in local repo.
  • git status - display: which updated files are staged/not staged(tracked)
  • git stash -
  • git branch - switch to a new branch
  • git branch <branchname> - will switch to this branch only if it exists
  • git checkout - DESTROYS/Updates the state of local working directory with the state of the files in the local repo. (YOU WILL LOSE non staged updates)
  • git restore <filename> - DESTROYS/Updates the file with the version in the local repo. (like with git checkout but for a specific file)
  • git checkin -
  • git revert <commitHash> - reverses a specific commit (use git log to get the chosen commit hash)
  • git ..

Converting AREV programs to EXODUS

Decompile AREV to C++

Do in Master AREV Installation Maintenance mode: (win7)

  1. ATTACH ADECOMC
    • ADECOM <programname> *single program
    • ADECOM <prog1> <prog2>
    • ADECOMALL *all programs (CHECK FIRST)

Include the option "(V)" in the command to print the C++ to a notepad, which can easily be copy and pasted.

Getting C++ program to Exodus Installation

Two methods: (use the first for one off programs)

Copy & Paste

Compile program and open source code in notepad

ADECOM <programname> (V)

Find which directory the program belongs. e.g fin/med/job/sys e.t

find /cygdrive/d/exodus/pickos | grep <program name>

If cpp in SYS, the program belongs in ~/exodus/service/src OR if in MED JOB FIN GEN AGY the program belongs in ~/neosys/src

Copy the source code in the text file from the first command.

In Exodus, create a new .cpp (in correct directory) and paste the source code.

Compile & Test

Rsync

/d/exodus/arev/syncup.sh

If cpp in SYS then: ~/exodus/service/src ./getpickos

If cpp in MED JOB FIN GEN AGY then: *~/neosys/src

Compile C++ files to TEST system

    • ./test <DBNAME>
    • ~/neosys ./doall TEST <DBNAME> restart #to get one service to start start using the new lib files
    • ~/neosys ./doall TEST all restart #to get all the services to start start using the new lib files

Install C++ files to LIVE System

  1. ~/exodus/service/ ./copyall #to copy all the ~/lib and bin files to ~/live/lib and bin ... which is used by all exodus/live services

Writing Standard Exodus Core Function/Method Testing

Screen 9: ~/exodus/test/src/ There are a series of test programs that check whether methods/functions behave as intended. They do this using the function, assert.. a 1 or more argument values produce one and only one output)

e.g test_multilang.cpp or test_sort.cpp

Two methods of running test programs:

  • Screen 9: make test
  • after compiling using edic/compile/c, enter test_prog_name. (Since compile has moved it to ~/bin)

Difference between the two methods is make calls gdb directly; whereas ~/bin/test_prog_name uses exodus compile program

  1. ~/neosys ./doall LIVE all restart



Upgrading Exodus

  1. Develop patch in Dev system

Patching neosys/src or exodus/service/src

  1. Manually insert patches and compile into test programs.
  2. If patch dat files then run neosys/compall to get dat files into ~/dat
  • If Patsalides:
    1. SITE_DIR=ptcy ./test basic
    2. login to BASIC_TEST and test
  • Other clients:
    1. ./test DBCODE
    2. login to DBCODE and test patches works
  1. run exodus/service/copyall and check only the patched programs are listed. (if not, see Troubleshoot below)
  2. Inform users to log off for 10mins. As a rule of thumb, live services must be stopped before copying test programs to live.
  3. exodus/service/copyall CONFIRM

Troubleshoot

./copyall lists programs you have not patched

Use exodus/service/copyone to copy only your patched programs one by one. (./copyone will not copy dat files)

Upgrading to new version

READ: remember there is no test version of libexodus programs. This means that if libexodus programs are compiled and installed, neosys/src and exodus/services LIVE programs MUST BE UPGRADED.

  1. Upgrade test system to test
    1. If a dev system like de2, git stash any uncommitted changes other staff may be working on.
    2. Run "g" in ~/exodus (to do git pull --ff-only)
    3. Run "m" in ~/exodus (to compile and install libexodus)
    4. Run "~/neosys/upgrade.all.sh live" in ~/neosys (to upgrade exodus/service and neosys/src live programs)
    5. Begin testing
  2. Upgrade clients
      • Client hosted - Schedule a planned upgrade with client.
      • NEOSYS hosted - Inform all clients/users of planned morning upgrade, 24hours in advanced. (TODO: 'What's New in NEOSYS') AND (emailallusers.cpp)
    1. Day of upgrade, check USB backup was successful & make a container snapshot, otherwise do not proceed.
    2. Stop services
    3. Run "g" in ~/exodus (to do git pull --ff-only)
    4. Run "m" in ~/exodus (to compile and install libexodus)
    5. Run "~/neosys/upgrade.all.sh live".
    6. Check services started automatically without issues.
  3. Inform users their system has been upgraded and they can login.

Roll Back Upgrade

If for any reason the entire upgrade needs to be reverted (excludes clients database), follow steps below:

  1. Inform users that there is a problem and services need to be stopped.
  2. Stop all exodus processes.
  3. Restore the morning's USB backup by: (Note this backup contains: source/obj code, /etc, neosys scripts but not client data)
    1. Login into container's host and rsync the morning's usb backup to the container's /:
    • rsync --dry-run -avz -e 'ssh -p <C_SSHPORT>' /backups/usb/<C_CODE> <C_CODE>:/      (Example: rsync --dry-run -avz -e 'ssh -p 19582' /backups/usb/ad1 ad1:/
      )
  4. Quick sane check, the latest git commit is not the latest git commit in the remote repo.
  5. Inform users they can login.

Configuring services to autostart during login

For PTCY,

  1. Create a file 'autostart.cfg' in the data folder
touch /root/hosts/<dbname>/data/autostart.cfg

This will autostart a database service as long as at least one database service in that directory is running.

For example, demo service will autostart during login if,

  1. /root/hosts/demo/data/autostart.cfg exists and
  2. demo_test service is running

Speedup postgresql importing databases

Can be sped up by a factor of 10 BUT may result in corrupt files in case of hard OS crash where the dirty memory is not flushed to storage.

Therefore it is advisable to use this feature only briefly and REMEMBER TO REMOVE IT!

Email to support@neosys.com whenever using it or use personal notes not memory.

nano /etc/postgresql/12/main/postgresql.conf

Uncomment or add a line

fsync = off

Restart postgres … will break any neosys/exodus running processes.

r postgresql

or reload postgres may also work

Database Attach

root@exodus:~/exodus/cli/src# dbattach
dbattach - Attach foreign database files to the current default database

Syntax is dbattach TARGETDB [FILENAME,...] {OPTION...}

Options:

  F   Forcibly delete any normal files in the current default database.

  R   Detach foreign files
  RR  Remove foreign database connection.
  RRR Remove all foreign database connections and attachments.

  L   List attached foreign files in the current default database

If no filenames are provided then a database connection is created.
This is *required* before filenames can be provided and attached.

Notes:

Attachments are permanent until re-attached or removed.

Indexes on attached files behave strangely.

Issues:

TARGETDB must be on the same connection as the current default database.

Current user must have access to the foreign database.

If EXO_USER and EXO_PASS are not set then exodus defaults are used.

exodus .config file is not used currently.