doc: Modifying main README.md

README.md

@@ -21,6 +21,7 @@ Features
 - All stored data is indexed and can be quickly searched via a full text phrase search.
 - EliasDB has a GraphQL interface which can be used to store and retrieve data.
 - For more complex queries EliasDB has an own query language called EQL with an sql-like syntax.
+- Includes a scripting interpreter to define alternative actions for database operations or writing backend logic.
 - Written in Go from scratch. Only uses gorilla/websocket to support websockets for GraphQL subscriptions.
 - The database can be embedded or used as a standalone application.
 - When used as a standalone application it comes with an internal HTTPS webserver which provides user management, a REST API and a basic file server.
@@ -68,6 +69,12 @@ There is a separate [tutorial](examples/tutorial/doc/tutorial_graphql.md) on usi
 
 The terminal uses a REST API to communicate with the backend. The REST API can be browsed using a dynamically generated swagger.json definition (https://localhost:9090/db/swagger.json). You can browse the API of EliasDB's latest version [here](http://petstore.swagger.io/?url=https://devt.de/krotik/eliasdb/raw/master/swagger.json).
 
+### Scripting
+
+EliasDB supports a scripting language called [ECAL](ecal.md) to define alternative actions for database operations such as store, update or delete. The actions can be taken before, instead (by calling `db.raiseGraphEventHandled()`) or after the normal database operation. The language is powerful enough to write backend logic for applications.
+
+There is a [VSCode integration](https://devt.de/krotik/ecal/src/master/ecal-support/README.md) available which supports syntax highlighting and debugging via the debug server. More information can be found in the [code repository](https://devt.de/krotik/ecal) of the interpreter.
+
 ### Clustering:
 
 EliasDB supports to be run in a cluster by joining multiple instances of EliasDB together. You can read more about it [here](cluster.md).
@@ -97,6 +104,13 @@ Usage of ./eliasdb server [options]
   -no-serv
     	Do not start the server after initialization
 ```
+If the `EnableECALScripts` configuration option is set the following additional option is available:
+```
+-ecal-console
+    Start an interactive interpreter console for ECAL
+```
+The interactive console can be used to inspect and modify the runtime state of the ECAL interpreter.
+
 Once the server is started the console tool can be used to interact with the server. The options of the console tool are:
 ```
 Usage of ./eliasdb console [options]
@@ -133,9 +147,17 @@ EliasDB uses a single configuration file called eliasdb.config.json. After start
 | ClusterLogHistory | File which is used to store the console history. |
 | ClusterStateInfoFile | File which is used to store the cluster state. |
 | CookieMaxAgeSeconds | Lifetime for cookies used by EliasDB. |
+| ECALDebugServerHost | Hostname the ECAL debug server should listen to. |
+| ECALDebugServerPort | Port on which the debug server should listen on. |
+| ECALEntryScript | Entry script for ECAL interpreter. |
+| ECALLogFile | Logfile for ECAL interpreter. An empty string will cause the logger to write to the console. |
+| ECALLogLevel | Log level for ECAL interpreter. Can be debug, info or error. |
+| ECALScriptFolder | Directory for ECAL scripts. |
 | EnableAccessControl | Flag if access control for EliasDB should be enabled. This provides user authentication and authorization features. |
 | EnableCluster | Flag if EliasDB clustering support should be enabled. EXPERIMENTAL! |
 | EnableClusterTerminal | Flag if the cluster terminal file /web/db/cluster.html should be created. |
+| EnableECALDebugServer | Flag if the ECAL debug server should be started. Note: This will slow ECAL performance significantly. |
+| EnableECALScripts | Flag if ECAL scripts should be executed on startup. |
 | EnableReadOnly | Flag if the datastore should be open read-only. |
 | EnableWebFolder | Flag if the files in the webfolder /web should be served up by the webserver. If false only the REST API is accessible. |
 | EnableWebTerminal | Flag if the web terminal file /web/db/term.html should be created. |
@@ -218,16 +240,16 @@ docker build --tag krotik/eliasdb .
 
 Example Applications
 --------------------
-- [Chat](examples/chat/doc/chat.md) - A simple chat application showing user /management and subscriptions.
+- [Chat](examples/chat/doc/chat.md) - A simple chat application showing user management and subscriptions.
 - [Data-mining](examples/data-mining/doc/data-mining.md) - A more complex application which uses the cluster feature of EliasDB and GraphQL for data queries.
-
+- [Game](examples/game/doc/game.md) - A multiplayer game example using ECAL for simulating the game scene in the backend.
 
 Further Reading
 ---------------
 - A design document which describes the different components of the graph database. [Link](https://devt.de/krotik/eliasdb/src/master/eliasdb_design.md)
-- A reference for the EliasDB query language EQL. [Link](https://devt.de/krotik/eliasdb/src/master/eql.md)
-- A reference for the EliasDB's support for GraphQL. [Link](https://devt.de/krotik/eliasdb/src/master/graphql.md)
-- A quick overview of what you can do when you embed EliasDB in your own Go project. [Link](https://devt.de/krotik/eliasdb/src/master/embedding.md)
+- A reference for EliasDB's custom query language EQL. [Link](eql.md)
+- A reference for EliasDB's support for GraphQL. [Link](graphql.md)
+- A quick overview of what you can do when you embed EliasDB in your own Go project. [Link](embedding.md)
 
 License
 -------

ecal-engine.md

@@ -0,0 +1,150 @@
+ECA Engine
+==========
+The ECA engine is ECAL's low-level event engine which does the actual concurrent event processing. Through ECAL a user can define rules which execute certain actions under certain conditions. The engine is defined in `ecal.engine`.
+
+Priorities
+----------
+The event-based system relies heavily on priorities for control flow. Both events and rules (which are triggered by events) have priorities. By default events and rules have the priority 0 which is the highest priority. Events are processed according to their priority and all triggering rules of a single event are executed according to their priority.
+
+Processor
+---------
+The processor is the central piece of the event engine. It controls the thread pool, contains the rule index and handles the event processing.
+
+The engines behaviour is solely defined by rules. These rules are added before the engine is started. Each added rule has a priority which determines their execution order if multiple rules are triggered by the same event. The main processing cycle, once the engine has been started, can be described as:
+
+Event injection -> Triggering check -> Rule Matching -> Fire Rules
+
+When injecting a new event it is possible to also pass a monitor with a certain scope and a priority. The scope is used by the processor to narrow down the triggering rules. A possible scenario for scopes are different types of analysis (e.g. quick analysis or deep analysis - only a subset of rules is required for the quick analysis). The priority determines when an event is processed - higher priority events are processed first.
+
+After an event is injected the Processor first checks if anything triggers on the event. The result of this is cached. The trigger check is just a first quick check to determine if the event can be discarded right away - even if the event passes the check, it is possible, that no rule will actually fire.
+
+After the first triggering check passed, the event is handed over to a task which runs in the thread pool. The task uses the rule index to determine all triggering rules. After filtering rules which are out of scope or which are suppressed by other rules, the remaining rules are sorted by their priority and then their action is executed.
+
+A rule action can inject new events into the processor which starts the processing cycle again. The processor supports two modes of execution for rule sequences (rules triggered by an event in order of priority):
+
+1. Do not fail on errors: all rules in a trigger sequence for a specific event
+are executed.
+
+2. Fail on first error: the first rule which returns an error will stop
+the trigger sequence. Events which have been added by the failing rule are still processed.
+
+Failing on the first error can be useful in scenarios where authorization is required. High priority rules can block lower priority rules from being executed.
+
+
+Monitor
+-------
+For every event there is a monitor following the event. Monitors form trees as the events cascade. Monitor objects hold additional information such as priority (how quickly should the associated event be processed), processing errors, rule scope, as well as context objects.
+
+
+Rules
+-----
+Rules define the conditions under which a particular action should be executed. Every rule must have the following properties:
+
+- [Name] A name which identifies the rule.
+- [KindMatch] Match on event kinds: A list of strings in dot notation which describes event kinds. May contain '*' characters as wildcards (e.g. core.tests.*).
+- [ScopeMatch] Match on event cascade scope: A list of strings in dot notation which describe the required scopes which are required for this rule to trigger. The included / excluded scopes for an event are stored in its monitor.
+- [StateMatch] Match on event state: A simple list of required key / value states in the event state. Nil values can be used as wildcards (i.e. match is only on key).
+- [Priority] Rules are sorted by their priority before their actions are executed.
+- [SuppressionList] A list of rules (identified by their name) which should be suppressed if this rule fires.
+- [Action] A function which will be executed if this rule fires.
+
+
+Events
+------
+Events are injected into the processor and cause rules to fire. An event is a simple object which contains:
+
+- [Name] A name which identifies the event.
+- [Kind] An event kind - this is checked against the kind match of rules during the triggering check.
+- [State] An event state which contains additional data.
+
+Events are always processed together with a monitor which is either implicitly created or explicitly given together with the event. If the monitor is explicitly given it is possible to specify an event scope which limits the triggering rules and a priority which determines the event processing order. An event with a lower priority is guaranteed to be processed after all events of a higher priority if these have been added before the lower priority event.
+
+Example
+-------
+- A client instantiates a new Processor giving the number of worker threads which should be used to process rules (a good number here are the cores of the physical processor).
+
+```
+proc := NewProcessor(1)
+```
+
+- The client adds rules to the processor.
+
+```
+rule1 := &Rule{
+		"TestRule1",                            // Name
+		"My test rule",                         // Description
+		[]string{"core.main.event1"},           // Kind match
+		[]string{"data"},                       // Match on event cascade scope
+		nil,                                    // No state match
+		2,                                      // Priority of the rule
+		[]string{"TestRule3", "TestRule2"},     // List of suppressed rules by this rule
+		func(p Processor, m Monitor, e *Event) error { // Action of the rule
+      ... code of the rule
+
+			p.AddEvent(&Event{
+							"Next Event",
+							[]string{"core", "main", "event2"},
+							nil,
+						}, m.NewChildMonitor(1))        // New monitor with priority for new event
+		},
+	}
+
+proc.AddRule(rule1)
+...
+```
+
+- The processor is started. At this point the thread pool inside the processor is waiting for tasks with the defined number of worker threads.
+
+```
+proc.SetRootMonitorErrorObserver(func(rm *RootMonitor) { // Called once a root monitor has finished
+	errs := rm.AllErrors()
+	...
+})
+
+proc.Start()
+```
+
+- A root monitor is instantiated and an initial event is added.
+
+```
+e := NewEvent(
+  "InitialEvent",                      // Name
+  []string{"core", "main", "event1"},  // Kind
+  map[interface{}]interface{}{         // State
+    "foo":  "bar",
+  },
+)
+
+rootm := proc.NewRootMonitor(nil, nil)
+
+rootm.SetFinishHandler(func(p Processor) { // Handler for end of event cascade
+  ...
+})
+
+proc.AddEvent(e, rootm)
+```
+
+- The event is processed as follows:
+
+	- The event is injected into the procesor with or without a parent monitor.
+
+		- Quick (not complete!) check if the event triggers any rules. This is to avoid unnecessary computation.
+			- Check that the event kind is not too general (e.g. the rule is for a.b.c event is for a.b)
+			- Check if 	at least one rule matches the kind. At least on rule should either be triggering on all kinds or triggering on the specific kind of the event.
+
+		- Create a new root monitor if no parent monitor has been given.
+
+		- Add a task to the thread pool of the processor (containing the event, parent/root monitor and processor).
+
+	- Thread pool of the processor takes the next task according to the highest priority.
+
+		- Determine the triggering rules (matching via kind, state and scope without suppressed rules).
+
+		- Execute the action of each triggering rule according to their priority.
+
+- The processor can run as long as needed and can be finished when the application should be terminated.
+
+```
+proc.Finish()
+```
+Calling `Finish()` will finish all remaining tasks and then stop the processor.

ecal.md

@@ -1,18 +1,19 @@
 EliasDB Event Condition Action Language
 =======================================
 
-EliasDB supports a scripting language called [Event Condition Action Language (ECAL)](https://devt.de/krotik/ecal/) to enable rule based scripting functionality. ECAL provides [database trigger](https://en.wikipedia.org/wiki/Database_trigger) functionality for EliasDB.
+EliasDB supports a scripting language called [Event Condition Action Language (ECAL)](ecal-lang.md) to enable rule based scripting functionality. ECAL provides [database trigger](https://en.wikipedia.org/wiki/Database_trigger) functionality for EliasDB.
 
 ECAL was added for the following use-cases:
 - Providing a way to manipulate data in response to events
 - Enforce certain aspects of a database schema
 - Providing back-end logic for web applications using EliasDB
 
-The source of EliasDB comes with a game example which demonstrates some aspects of ECAL. See the documentation here: [TODO Link to documentation]
+The source of EliasDB comes with a [game example](examples/game/doc/game.md) which demonstrates a more complex application of ECAL.
 
 ECAL related config values:
 --
 These ECAL related config options are available in `eliasdb.config.json`:
+
 | Configuration Option | Description |
 | --- | --- |
 | EnableECALScripts | Enable ECAL scripting. |
@@ -34,35 +35,34 @@ EliasDB specific events which can be handled:
 --
 The ECAL interpreter in EliasDB receives the following events:
 
-Web Request | ECAL event kind | Event state contents | Description
--|-|-|-
-/db/api/|`db.web.api`| bodyJSON, bodyString, header, method, path, pathList, query | Any web request to /db/api/... These endpoints are public and never require authentication.
-/db/ecal/|`db.web.ecal`| bodyJSON, bodyString, header, method, path, pathList, query | Any web request to /db/ecal/... These endpoints are considered internal and require authentication if access control is enabled.
-/db/sock/|`db.web.sock`| bodyJSON, bodyString, commID, header, method, path, pathList, query | Any web request to /db/sock/... These endpoints are used to initiate websocket connections.
--|`db.web.sock.data`| commID, data, header, method, path, pathList, query | An existing websocket connection received some JSON object data. If the close attribute of the object is set to true then the websocket connection is closed.
+| Web Request | ECAL event kind | Event state contents | Description |
+| --- | ---| --- | --- |
+| /db/api/|`db.web.api`| bodyJSON, bodyString, header, method, path, pathList, query | Any web request to /db/api/... These endpoints are public and never require authentication. |
+| /db/ecal/|`db.web.ecal`| bodyJSON, bodyString, header, method, path, pathList, query | Any web request to /db/ecal/... These endpoints are considered internal and require authentication if access control is enabled. |
+| /db/sock/|`db.web.sock`| bodyJSON, bodyString, commID, header, method, path, pathList, query | Any web request to /db/sock/... These endpoints are used to initiate websocket connections. |
+| - |`db.web.sock.data`| commID, data, header, method, path, pathList, query | An existing websocket connection received some JSON object data. If the close attribute of the object is set to true then the websocket connection is closed. |
+
+| EliasDB Graph Event | ECAL event kind | Event state contents | Description |
+| --- | --- | --- | --- |
+| graph.EventNodeCreated | `db.node.created` | part, trans, node | A node was created. |
+| graph.EventNodeUpdated | `db.node.updated` | part, trans, node, old_node | A node was updated. |
+| graph.EventNodeDeleted | `db.node.deleted` | part, trans, node | A node was deleted. |
+| graph.EventEdgeCreated | `db.edge.created` | part, trans, edge | An edge was created. |
+| graph.EventEdgeUpdated | `db.edge.updated` | part, trans, edge, old_edge | An edge was updated. |
+| graph.EventEdgeDeleted | `db.edge.deleted` | part, trans, edge | An edge was deleted. |
+| graph.EventNodeStore | `db.node.store` | part, trans, node | A node is about to be stored (always overwriting existing values). |
+| graph.EventNodeUpdate | `db.node.update` | part, trans, node | A node is about to be updated. |
+| graph.EventNodeDelete | `db.node.delete` | part, trans, key, kind | A node is about to be deleted. |
+| graph.EventEdgeStore | `db.edge.store` | part, trans, edge | An edge is about to be stored. |
+| graph.EventEdgeDelete | `db.edge.delete` | part, trans, key, kind | An edge is about to be deleted. |
 
-EliasDB can receive the following events from the ECAL interpreter:
+Note: EliasDB will wait for the event cascade to be finished before performing the actual operation (e.g. inserting a node). If the event handling requires a time consuming operation then a new parallel event cascade can be started using `addEvent` with a scope:
 
-ECAL event kind | Event state contents | Description
--|-|-
-db.web.sock.msg | commID, payload, close | The payload is send to a client with an open websocket identified by the commID.
-
-
-EliasDB Graph Event | ECAL event kind | Event state contents | Description
--|-|-|-
-graph.EventNodeCreated | `db.node.created` | part, trans, node | A node was created.
-graph.EventNodeUpdated | `db.node.updated` | part, trans, node, old_node | A node was updated.
-graph.EventNodeDeleted | `db.node.deleted` | part, trans, node | A node was deleted.
-graph.EventEdgeCreated | `db.edge.created` | part, trans, edge | An edge was created.
-graph.EventEdgeUpdated | `db.edge.updated` | part, trans, edge, old_edge | An edge was updated.
-graph.EventEdgeDeleted | `db.edge.deleted` | part, trans, edge | An edge was deleted.
-graph.EventNodeStore | `db.node.store` | part, trans, node | A node is about to be stored (always overwriting existing values).
-graph.EventNodeUpdate | `db.node.update` | part, trans, node | A node is about to be updated.
-graph.EventNodeDelete | `db.node.delete` | part, trans, key, kind | A node is about to be deleted.
-graph.EventEdgeStore | `db.edge.store` | part, trans, edge | An edge is about to be stored.
-graph.EventEdgeDelete | `db.edge.delete` | part, trans, key, kind | An edge is about to be deleted.
+EliasDB can receive the following events from the ECAL interpreter:
 
-Note: EliasDB will wait for the event cascade to be finished before performing the actual operation (e.g. inserting a node). If the event handling requires a time consuming operation then a new parallel event cascade can be started using `addEvent` with a scope:
+| ECAL event kind | Event state contents | Description |
+| --- | --- | --- |
+| db.web.sock.msg | commID, payload, close | The payload is send to a client with an open websocket identified by the commID. |
 ```
 addEvent("request", "foo.bar.xxx", {
    "payload" : 123

examples/game/eliasdb.config.json

@@ -1,34 +0,0 @@
-{
-    "ClusterConfigFile": "cluster.config.json",
-    "ClusterLogHistory": 100,
-    "ClusterStateInfoFile": "cluster.stateinfo",
-    "CookieMaxAgeSeconds": "86400",
-    "ECALDebugServerHost": "127.0.0.1",
-    "ECALDebugServerPort": "33274",
-    "ECALEntryScript": "main.ecal",
-    "ECALLogFile": "",
-    "ECALLogLevel": "info",
-    "ECALScriptFolder": "scripts",
-    "ECALWorkerCount": 10,
-    "EnableAccessControl": false,
-    "EnableCluster": false,
-    "EnableClusterTerminal": false,
-    "EnableECALDebugServer": false,
-    "EnableECALScripts": false,
-    "EnableReadOnly": false,
-    "EnableWebFolder": true,
-    "EnableWebTerminal": true,
-    "HTTPSCertificate": "cert.pem",
-    "HTTPSHost": "127.0.0.1",
-    "HTTPSKey": "key.pem",
-    "HTTPSPort": "9090",
-    "LocationAccessDB": "access.db",
-    "LocationDatastore": "db",
-    "LocationHTTPS": "ssl",
-    "LocationUserDB": "users.db",
-    "LocationWebFolder": "web",
-    "LockFile": "eliasdb.lck",
-    "MemoryOnlyStorage": false,
-    "ResultCacheMaxAgeSeconds": 0,
-    "ResultCacheMaxSize": 0
-}