Programming Knowledge

From NEOSYS Technical Support Wiki

1 NEOSYS System

Pick/BASIC Programmers Guide

1.1 Structure Overview

NEOSYS is comprised of a backend, web server and frontend.

  1. The backend is a multi-value database written in the Revelation programming language. Use JBASE programming manuals for standard function syntax.
  2. The webserver role is to forward frontend to backend requests and vice versa.
  3. The frontend is comprised of .htm and javascript.

1.2 Frontend

In NEOSYS frontend there are Files, Screens and Documents.

  • Files allow the user to create, update and delete information about a entity. Such as a Supplier, Client, Ratecard, Schedule or Job.
  • Screens typically allow the user to search, filter and order records.
  • Documents tend to be reports, invoices, orders, cheque printouts.

Written in HTML for bare bone structure of webpages, JavaScript to interactively manipulate HTML and Ajax to send XML requests to server.

1.3 Web Server

TODO

1.4 Backend

The backend database in follows the key value store concept.

  • a row or document is called a "record" or "item". Records are divided by record marker (rm).
  • a column or field is called a field or an "attribute", which is composed of "multi-value attributes" and "sub-value attributes" to store multiple values in the same attribute. Divided by the "fm", "vm" & "svm".
  • a table or collection is called a "file".
  • a database or schema is called an "account".

General code for each module is held in: JOB.SUBS, AGENCY.SUBS, TIMESHEET.SUBS. Subs for subroutines.

The routines that handle production invoices are kept in PRODINVS, however PRODINVS2/3/4 contain more code. Many fields will have a default value, check the DEFAULTVALUE routine.

1.4.1 File & Dictionary

Data is stored using two separate files: a "file" to store raw data and a "dictionary" to store the format for displaying the raw data.

In NEOSYS maintenance mode the command "ed filename [key]" e.g "ed client" will presented one record of the client file and not all client records as you may expect.

Each field of that record is be divided into a line, which line is determined by the file's dictionary file.

Lets look at a simple request example:

09:45:00 192.168.9.30 GREG READ SUPPLIERS "NAME" "MARKETCODE" from "GUNE":

Remember: "ed filename [key]". [] = optional, if key omitted then random record presented. A key is a unique identifier of a record. For a client record it would be the client code.

1. First we need to know which fields/lines of a supplier record, contain the NAME and MARKETCODE. That information is stored in the supplier dictionary file.

2. Command = "ed dict.suppliers NAME" will open a supplier dictionary record with key "NAME" below is what you'd see:

0║ NAME
1║ F 
2║ 1
3║ 20

Line 0 = record key, Line 1 = field, Line 2 = lineNum (in supplier file), Line 3 = maxLength of field. (How this info is obtained is explained later see ***)

OK so now we know that on line 1 of a supplier record we will find the name of a supplier.

3. Open the supplier dictionary file for record with key "MARKETCODE", command = "ed dict.suppliers MARKETCODE":

0║ MARKETCODE
1║ F 
2║ 4
3║ 3

Line 0 = Recordkey, Line 1 = Fieldname, line 2 = lineNum (on supplier file), line 3 = maxlength (character).

OK now we know that on line 4 of a supplier record we will find the market code for a supplier.

4.Finally open supplier record with code "GUNE", command = "ed supplier GUNE" and read lines 1 & 4.

0║ GUNE
1║ GulfNewspaper
2║ GUNE
3║ N
4║ UAE
5║ 045372331
6║ DUBAI

(note that line 0, is not displayed when using "ed")

Lets say that we needed to send this info back to client-side. We know that both these pieces of data are belong to fields. Therefore the raw string we would send back is = "fm GulfNewspaper fm UAE".

      • There's actually master dictionary file "dict.voc" that describes the records of dict.suppliers file, just like the dict.supplier describes how to display supplier data in its raw form.

1.4.2 Processing Client Requests

To determine the nature of a request, you can observe the command on the server process screen.

In order to simplify requests from the client side, part of the command defines which PROXY routine to use, which will assess the remaining arguments (variables) of the request and call the required routines for the task. Each module has it's own PROXY routine, E.g JOBPROXY, GENERALPROXY and MEDIAPROXY.

The general pattern for requests is REQUEST+DATA and RESPONSE+DATA, but DATA is not relevant to all request or responses. DATA usually contains a REV FILE string (always for READ/WRITE). Requests are quite short phrases like "READ CLIENTS XYZ" or "EXECUTE JOBPROXY PRODINVS", although the "PROXY" word is always omitted from the EXECUTE request, so the actual request will be "EXECUTE JOB PRODINVS".

Requests for a document such as an invoice, booking order or report, create a .htm file on the server return the url of the file in RESPONSE+DATA.


1.4.3 Development Notes

Currently an unordered list of facts:

  • Every module has a MODULENAME.js, there is also a virtual module called agency.js which hold a few screens and code files that apply to both media and jobs.
  • if you want to run a script before or after saving you need to know to create a script called form_prewrite() or form_postwrite().
  • "gds" - general data set, is an object that holds a copy of the current document or record. "gds.getx" and "gds.setx" are the main methods. Also "get1" and "getn" are variations of main methods.
  • Arev string from the server is broken up into the gds object.
  • If you want to run a script before or after saving you need to know to create a script called form_prewrite() or form_postwrite().
  • Validation happens at data entry and not at db write.
  • costestimateprint.htm doesnt have a costestimateprint.js like other files. Remember that documents are created at server-side and a url is passed back to the cliets browser. All code that handles a cost estimate print is found in job.subs.
  • "client.js" is the only thing you need to include in the .htm file ... routines in client.htm will automatically load in dbform.js which handles forms in general and dbform.js will include FILEORSCREEN.js
  • PRINTINV rev routine on the server received a record string that contains the parameters that control the formation of the printable htm file.

1.5 Server

  • .asp files runs on the server and responds to requests sending an receiving strings of data from the webpage. Which allow the webpage to update part of the page without reloading the entire document.

ASP merely forward strings to the webpage or the db. (GUESS= its a proxy). ASP asp file receives mainly requests like READ/WRITE/EXECUTE and a few more like LOGON/SELECT/GETVALUES

2 General Programming Concepts

2.1 Understanding Floating Point Number errors in computing

Floating point numbers stored in computers are inherently inaccurate approximations especially after multiplication/division. Multiplication and division are two sides of the same coin and are used all the time in business for finance and operations especially for currency conversion and conversion to base currency.

10.00/3=3.33333333333333...??? ... at what point does the computer stop storing more 3s? and if the computer doesnt store an infinite of 3s, which is an impossibility, then the number stored in the computer is a SLIGHTLY INACCURATE APPROXIMATION.

Even if we store all currency amounts as integer cents, the problem doesn't go away.

10.00 Dollars = 1000 cents and 1000/3=333.333333333333...???

How do computers store floating point numbers?

You can imagine that computers store every single floating point number as TWO numbers.

1. A 15 digit number with a point AFTER the first digit.

2. A separate "scaling" number from +99 through 0 to -99

  • positive means move the decimal point to the right
  • negative means move the decimal point to the left
  • zero means leave the decimal point where it is (after the first digit on the left)

So:

1.23456789012345 and 2 would mean 123.456789012345

1.23456789012345 and -2 would mean 0.0123456789012345

1.23456789012345 and 10 would mean 12345678901.2345

1.23456789012345 and -10 would mean 0.000000000123456789012345

Scientific notation shows the above examples as:

1.23456789012345e+02

1.23456789012345e-02

1.23456789012345e+10

1.23456789012345e-10

So computers always store 15 digits of accuracy and allow scaling by multiples of 10.

Note that by convention the first digit of scientific notation CANNOT be a zero. All the others can be zero, but if the first digit appears to be a zero, then for scientific notation you MUST move the decimal point one to the right and add one to the SCALING number. So 0.123456789012345+e02 becomes 1.23456789012345e01

IMPORTANT: Check carefully for yourself that you agree with the placing of the decimal points in the above examples, and ask/do research if you do not understand it. It is essential. Dont skip.

Consider also that:

10.00/3*3=9.99999999999999..?? in currency if you treat 10.00/3 as 3.33

The end result after adding and subtracting these SLIGHTLY INACCURATE APPROXIMATIONS is that sometimes you can get numbers like the following.

0.00000000000000123456789012345 which in scientific e format is 1.23456789012345e-15 (do you understand why? if not, WORK UNTIL YOU DO!)

Which normally should be rounded off to 0.00 if it is a currency amount.

Therefore when doing calculation we do not round off intermediate results unless it is essential e.g. prices have to be in cents. Whereas for really accurate pricing especially of small items it may be necessary to have unit prices in tiny fractions of cents.

2.1.1 CORRECTLY done without rounding intermediate results

10.00/3=3.33333333333333

3.333333333333333*3=9.999999999999999 (not rounded ... good!)

9.99999999999999 rounded to 2 decimal places = 10.00

2.1.2 WRONGLY done with rounding of intermediate results

10.00/3=3.33333333333333=3.33 (rounded ... wrong!)

3.33*3=9.99

9.99 rounded to 2 decimal places = 9.99 (Wrong result -- we expect 10.00/3*3 to be 10)

2.1.3 Bugs in display or storing of floating point numbers

If fail to ROUND numbers off after calculation to the desired number of decimals, and then convert/display the number to/as a string then you will sometimes see things like 1.23456789012345e-15. This is how many languages, including Javascript work.

3 TODO working notes