EXODUS Knowledge

TMUX Screens

To create the EXODUS maintenance/programming environment

exodus#: ./tmux.exodus

Screens

0: /
    Work regarding exodus server. i.e /etc/nagios or general networking or /var/log/mail.log

1: /root/exodus/
    Exodus core is installed Contains build and make libraries here ./libexodus

2: /root/exodus/libexodus/exodus
    Core exodus (libraries that emulate the AREV system e.g mv.h )
    
3: /root/exodus/cli/src
    Exodus Command Line Interface e.g edir, edic, list, listfiles, delete, createdb
    
4: root/exodus/service
    EXODUS core default host directory but also contains scripts that create/manage requirements of a service (create_service, create_db, create_site, backup_db, restore_db, start, stop, status)
    One exception is NEOSYS www files are in ./www, which pulled from NEOSYS www git.

5: /root/exodus/service/src
    Exodus core service programs (programs that are not related to NEOSYS agency operations and can be used to support other non-NEOSYS services) eg listen, usersubs, sendmail, openfile, select, sort, list,.

    Extra info:    
    sql directory contains basic dictiorary files converted to sql instructions used when creating a EXODUS psql database
    e.g dict_voc.sql is a file that describes what to expect in certain fields of the voc file.
    TODO: compall currently throws few warnings. Read them and familiarise. In case new warnings appear, notify Steve.    
        
6: /root/neosys
    NEOSYS specific client installation management scripts e.g .doall, import_db and import_files (from AREV), ./run ./test (see below for these two commands)
    
7: /root/neosys/src
    NEOSYS Agency programs (see section How AREV programs are distributed in EXODUS src/*.cpp
    ../ has .git

8: /root/hosts
    client specific files  e.g logs, images, (shared ./www -> root/exodus/service/www

9: /root/exodus/test/src
    series of programs that test whether subroutines (specifically Exodus core libraries) work as they should eg for the var lib - "assert( var("11111").isnum());"

10: bash /bin/top
    Monitor for NEOSYS client processes
    Displays customised format: sorted by total time spend and by processor time.
     Press "=" : to change to standard top format
     Press shift + w : to save format settings, replacing the old customised top format
     To go back to the customised format do:
        Press "o"
        Type "COMMAND=serve_agy"
        shift + t : this sorts by total time spent
        shift + p : this sorts by processor time
        shift + w : to save this setting and enter  'y' to confirm

11: /var/log/syslog
     Log of NEOSYS client requests

12: /root/exodus/service/quick
     Displays the last time the svr file (in /root/hosts/<client>/data/<dbname>) was updated.

Object Code/Libraries

LIVE and TEST processes use different sets of object code. TEST processes use libraries in ~/lib/, whereas LIVE processes use object code in ~/neo/lib

This means development & testing can be done stress free on TEST database, as opposed to testing on production databases.

When compiling using edic, the TEST object code is updated if the compilation is successful. (~/lib) In order to apply a tested patch to LIVE see Update LIVE programs.

Dictionaries

Dictionaries, the files used to describe the fields of a file's record. Unlike in AREV, there is a copy of all dictionaries in each pgsql database (In AREV, updating a dictionary would affect all the databases).

Processes

The TEST process for all database use the same object code stored in /root/lib, whereas all LIVE process use the object code in /root/neo/.

Postgres

Connect into postgres shell:

sudo -u postgres psql

List databases once in postgres shell:

\l

Delete a database:

sudo -u postgres dropdb <dbcode>

./doall

General

Screen 6: ./doall script contains all the necessary information(codes) to setup an installation. It includes scripts to backup, restore, create an Apache site, create/start/stop/status a service, import an AREV database into postgres and more.

backup_db

• Does a backup & restore of a LIVE database into the corresponding TEST database.
• Backup <dbcode>.sql file is written to /root/backups/sql; which is rsynced to nl19:/backups/current/exodus/
• Unlike AREV, postgres can perform a "backup" of a database whilst the system is in use.

Git

There are two repositories, one for EXODUS and the other for NEOSYS.

Using git to make changes

Before following steps you must have a tested updated to a program/file/script. Do not commit untested changes to avoid a messy git history of reverts.

Update your local repo before committing to local repo using the g alias for "git pull --ff-only ; git push && git status'":

g

Check which updates/files have not yet been staged and/or committed: Add your updates to the staged area:

git add <filename>

or if all the changes made need to be staged:

git add -a

Make a commit with a descriptive message on purpose of updates:

git commit -m <description n purpose of changes>

Again use g alias:

g

Other useful git cmds

Do not use this commands unless you know what WILL happen.

• git pull - Instead use the safe "git pull --ff-only"

Stick to the alias "g" which does "git pull --ff-only ; git push && git status'"

• git log - display history of commits to master branch
• git diff - display the differences in files between working files and files in local repo.
• git status - display: which updated files are staged/not staged(tracked)
• git stash -
• git branch - switch to a new branch
• git branch <branchname> - will switch to this branch only if it exists
• git checkout - DESTROYS/Updates the state of local working directory with the state of the files in the local repo. (YOU WILL LOSE non staged updates)
• git restore <filename> - DESTROYS/Updates the file with the version in the local repo. (like with git checkout but for a specific file)
• git checkin -
• git revert <commitHash> - reverses a specific commit (use git log to get the chosen commit hash)
• git ..

Converting AREV to EXODUS

Decompile AREV to C++

(Do in win10a Maintenance mode)

1. Apply tested patch to win10a (master AREV Dev system)
2. ATTACH ADECOMC
   • ADECOM <programname> *single program
   • ADECOMALL *all programs (CHECK THIS FIRST)
   • ADECOM <prog1> <prog2> *(CHECK IF THIS WORKS)

Send c++ files from win10a to nl19

(Do in win10a Cygwin)

1. /d/exodus/arev/syncup.sh

Get c++ files from nl19 to exodus

(Do in Exodus system)

1. If cpp in SYS then: ~/exodus/service/src ./getpickos
2. If cpp in MED JOB FIN GEN AGY then: *~/neosys/src ./getpickos
3. Compile single cpp then: c <programname> e.g "c monitor2"
4. Compile all cpp then: ./compall (PENDING WHICH/WHERE? many compall)

Compile C++ files to TEST system

1. • ./test <DBNAME>
   • ~/neosys ./doall TEST <DBNAME> restart #to get one service to start start using the new lib files
   • ~/neosys ./doall TEST all restart #to get all the services to start start using the new lib files

Install C++ files to LIVE System

WARNING

1. ~/exodus/service/ ./copyall #to copy all the ~/lib and bin files to ~/live/lib and bin ... which is used by all exodus/live services

Writing Standard Exodus Core Function/Method Testing

Screen 9: ~/exodus/test/src/ There are a series of test programs that check whether methods/functions behave as intended. They do this using the function, assert.. a 1 or more argument values produce one and only one output)

e.g test_multilang.cpp or test_sort.cpp

Two methods of running test programs:

• Screen 9: make test
• after compiling using edic/compile/c, enter test_prog_name. (Since compile has moved it to ~/bin)

Difference between the two methods is make calls gdb directly; whereas ~/bin/test_prog_name uses exodus compile program

1. ~/neosys ./doall LIVE all restart

Updating a pgsql function in an exodus dictionary

PENDING

Troubleshooting postfix emailing issues

To view current configuration in main.cf

postconf -n

To update the value of a parameter

postconf relayhost=mailout.neosys.com:2500

To restart postfix service

service postfix restart

Send a test email using "mail" and watch the logs to verify

mail <email>
journalctl -f

Development and deployment using 'dat' files

Rationale

Part of system development is the creation of various data that is neither programs nor layout i.e. not cpp, h, html, js, php files etc.

For example:

• Dictionaries are data about data.
• Language files are data about text to use for various languages.
• “Change logs” are data about changes in the system.

Historically, in EXODUS and NEOSYS, the above data has been deployed in exodus database files using SQL text files. However SQL files are not convenient for development.

Therefore, 'dat' text files will be used now so that standard development tools including editors and git can be fully exploited.

'dat' files

Each database file is represented by an os directory of the same name.

Each record in the database file is represented by an os text file where filename is the primary key.

For example a record with key DEADLINE in a dat file dict.materials would be represented as an os text file dat/dict.materials/DEADLINE

Each line in the os text file represents one field in the db record. In other words, db record FM characters are represented by new line characters in 'dat' files. Any actual new line characters required in the record, and any backslashes, are escaped and appear as '\n' and '\\' in 'dat' files. Other database field separator characters such as VM, SM, TM and STM are stored without any conversion.

Location of dat files

The development versions are stored in exodus and neosys src/dat dirs. They form part of the standard git repositories in parallel with cpp files.

The operational versions are stored in ~/dat and ~/live/dat alongside bin and lib dirs and are automatically installed into databases as database files on service startup. Any database functions embedded in the text files (pgsql) are also automatically installed at the same time.

Editing and deploying a 'dat' file

It is currently a three step process to edit and deploy such 'dat' files.

Edit the 'dat' file

Note that EXODUS service and NEOSYS service have different src/dat folders.

Editing language items:

edir dat/alanguage/SCHEDULES*ARABIC

Editing a dictionary item:

edir dat/dict.materials/DEADLINE

Editing a pgsql function in a dictionary:

edir dat/dict.materials/DEADLINE 8

Copy all 'dat' files to ~/dat

This step might be removed at a later date.

This will cause all test databases to immediately restart and load any 'dat' file changes into dictionaries and data files and also create any new or modified pgsql functions.

If any ~/neosys/src/dat files were edited:

cd ~/neosys/src

./compall dat

and/or, if exodus/service/dat files were edited

cd ~/exodus/service/src

./compall dat

Copy all programs and 'dat' files to ~/live/bin|lib|dat

This should only be run after testing. It will cause all live databases to automatically restart and do the same as the test databases mentioned above.

cd ~/exodus/service

./copyall CONFIRM

​

Sharing files between databases

dbattach is an exodus cli tool, used to share records of selected files, in one service's database (primary) with another service (secondary).

E.g clientX can reference clientY's market records. Note that ClientX cannot update clientY's client records.

• These file links are preserved during reboots or service restarts.
• Links to a primary database are also preserved when you copy the secondary database to a new database. (This may be fixed in later versions)

EXO_DATA=<secondaryDBCODE> dbattach <primaryDBCODE> <filename1 [filename2 ..]> <{options}>

Options: (one option must be used)

F   Forcibly delete any normal files in the current default database.    (usual option to use to link secondary to primary)

R   Detach foreign files
RR  Remove foreign database connection.
RRR Remove all foreign database connections and attachments.

L   List attached foreign files in the current default database

Example: EXO_DATA=bateskuw dbattach batesdub markets market_versions clients client_versions brands suppliers supplier_versions vehicles vehicle_versions ratecards ratecard_versions {F}

Possible Errors

INIT.GENERAL will fail to start services if the markets file has been detached. (when attaching files, file records in the secondary database are deleted. ( {F} )