feat: ECAL support for EliasDB

.gitignore

@@ -16,3 +16,13 @@
 /examples/data-mining/docker-images/eliasdb/eliasdb
 /examples/data-mining/docker-images/frontend/app/node_modules
 /examples/data-mining/docker-images/frontend/app/graphiql
+/examples/game/run/
+/examples/game/res/frontend/*.lock
+/examples/game/res/frontend/*-lock.json
+/examples/game/res/frontend/node_modules
+/examples/tmp
+/ssl
+/web
+/db
+/scripts
+/eliasdb.config.json

README.md

@@ -21,6 +21,7 @@ Features
 - All stored data is indexed and can be quickly searched via a full text phrase search.
 - EliasDB has a GraphQL interface which can be used to store and retrieve data.
 - For more complex queries EliasDB has an own query language called EQL with an sql-like syntax.
+- Includes a scripting interpreter to define alternative actions for database operations or writing backend logic.
 - Written in Go from scratch. Only uses gorilla/websocket to support websockets for GraphQL subscriptions.
 - The database can be embedded or used as a standalone application.
 - When used as a standalone application it comes with an internal HTTPS webserver which provides user management, a REST API and a basic file server.
@@ -68,6 +69,12 @@ There is a separate [tutorial](examples/tutorial/doc/tutorial_graphql.md) on usi
 
 The terminal uses a REST API to communicate with the backend. The REST API can be browsed using a dynamically generated swagger.json definition (https://localhost:9090/db/swagger.json). You can browse the API of EliasDB's latest version [here](http://petstore.swagger.io/?url=https://devt.de/krotik/eliasdb/raw/master/swagger.json).
 
+### Scripting
+
+EliasDB supports a scripting language called [ECAL](ecal.md) to define alternative actions for database operations such as store, update or delete. The actions can be taken before, instead (by calling `db.raiseGraphEventHandled()`) or after the normal database operation. The language is powerful enough to write backend logic for applications.
+
+There is a [VSCode integration](https://devt.de/krotik/ecal/src/master/ecal-support/README.md) available which supports syntax highlighting and debugging via the debug server. More information can be found in the [code repository](https://devt.de/krotik/ecal) of the interpreter.
+
 ### Clustering:
 
 EliasDB supports to be run in a cluster by joining multiple instances of EliasDB together. You can read more about it [here](cluster.md).
@@ -97,6 +104,13 @@ Usage of ./eliasdb server [options]
   -no-serv
     	Do not start the server after initialization
 ```
+If the `EnableECALScripts` configuration option is set the following additional option is available:
+```
+-ecal-console
+    Start an interactive interpreter console for ECAL
+```
+The interactive console can be used to inspect and modify the runtime state of the ECAL interpreter.
+
 Once the server is started the console tool can be used to interact with the server. The options of the console tool are:
 ```
 Usage of ./eliasdb console [options]
@@ -133,9 +147,18 @@ EliasDB uses a single configuration file called eliasdb.config.json. After start
 | ClusterLogHistory | File which is used to store the console history. |
 | ClusterStateInfoFile | File which is used to store the cluster state. |
 | CookieMaxAgeSeconds | Lifetime for cookies used by EliasDB. |
+| ECALDebugServerHost | Hostname the ECAL debug server should listen to. |
+| ECALDebugServerPort | Port on which the debug server should listen on. |
+| ECALEntryScript | Entry script for ECAL interpreter. |
+| ECALLogFile | Logfile for ECAL interpreter. An empty string will cause the logger to write to the console. |
+| ECALLogLevel | Log level for ECAL interpreter. Can be debug, info or error. |
+| ECALScriptFolder | Directory for ECAL scripts. |
+| ECALWorkerCount | Number of worker threads in the ECA engine's thread pool. |
 | EnableAccessControl | Flag if access control for EliasDB should be enabled. This provides user authentication and authorization features. |
 | EnableCluster | Flag if EliasDB clustering support should be enabled. EXPERIMENTAL! |
 | EnableClusterTerminal | Flag if the cluster terminal file /web/db/cluster.html should be created. |
+| EnableECALDebugServer | Flag if the ECAL debug server should be started. Note: This will slow ECAL performance significantly. |
+| EnableECALScripts | Flag if ECAL scripts should be executed on startup. |
 | EnableReadOnly | Flag if the datastore should be open read-only. |
 | EnableWebFolder | Flag if the files in the webfolder /web should be served up by the webserver. If false only the REST API is accessible. |
 | EnableWebTerminal | Flag if the web terminal file /web/db/term.html should be created. |
@@ -218,16 +241,16 @@ docker build --tag krotik/eliasdb .
 
 Example Applications
 --------------------
-- [Chat](examples/chat/doc/chat.md) - A simple chat application showing user /management and subscriptions.
+- [Chat](examples/chat/doc/chat.md) - A simple chat application showing node modification via ECAL script, user management and subscriptions.
 - [Data-mining](examples/data-mining/doc/data-mining.md) - A more complex application which uses the cluster feature of EliasDB and GraphQL for data queries.
-
+- [Game](examples/game/doc/game.md) - A multiplayer game example using ECAL for simulating the game scene in the backend.
 
 Further Reading
 ---------------
-- A design document which describes the different components of the graph database. [Link](https://devt.de/krotik/eliasdb/src/master/eliasdb_design.md)
-- A reference for the EliasDB query language EQL. [Link](https://devt.de/krotik/eliasdb/src/master/eql.md)
-- A reference for the EliasDB's support for GraphQL. [Link](https://devt.de/krotik/eliasdb/src/master/graphql.md)
-- A quick overview of what you can do when you embed EliasDB in your own Go project. [Link](https://devt.de/krotik/eliasdb/src/master/embedding.md)
+- A design document which describes the different components of the graph database. [Link](eliasdb_design.md)
+- A reference for EliasDB's custom query language EQL. [Link](eql.md)
+- A reference for EliasDB's support for GraphQL. [Link](graphql.md)
+- A quick overview of what you can do when you embed EliasDB in your own Go project. [Link](embedding.md)
 
 License
 -------

api/rest.go

@@ -16,6 +16,7 @@ import (
 
 	"devt.de/krotik/common/datautil"
 	"devt.de/krotik/eliasdb/cluster"
+	"devt.de/krotik/eliasdb/ecal"
 	"devt.de/krotik/eliasdb/graph"
 	"devt.de/krotik/eliasdb/graph/graphstorage"
 )
@@ -89,6 +90,11 @@ GM is the GraphManager instance which should be used by the REST API.
 */
 var GM *graph.Manager
 
+/*
+SI is the ScriptingInterpreter instance which is working with the api.GM GraphManager instance.
+*/
+var SI *ecal.ScriptingInterpreter
+
 /*
 GS is the GraphStorage instance which should be used by the REST API.
 */

api/v1/blob.go

@@ -468,7 +468,7 @@ func (be *blobEndpoint) HandleGET(w http.ResponseWriter, r *http.Request, resour
 
 		res, err = sm.FetchCached(loc)
 
-		if err == storage.ErrNotInCache {
+		if sme, ok := err.(*storage.ManagerError); ok && sme.Type == storage.ErrNotInCache {
 			err = sm.Fetch(loc, &ret)
 		} else if err == nil && res != nil {
 			ret = res.([]byte)

api/v1/blob_test.go

@@ -90,7 +90,7 @@ func TestBlob(t *testing.T) {
 
 	st, _, res = sendTestRequest(queryURL, "POST", []byte{0x0b, 0x00, 0x00, 0x0b, 0x01, 0x0e, 0x05})
 
-	if st != "500 Internal Server Error" || res != "Record is already in-use (? - )" {
+	if st != "500 Internal Server Error" || res != "Record is already in-use (<memory> - )" {
 		t.Error("Unexpected response:", st, res)
 		return
 	}

api/v1/cluster_test.go

@@ -30,6 +30,7 @@ import (
 )
 
 func TestClusterStorage(t *testing.T) {
+
 	clusterQueryURL := "http://localhost" + TESTPORT + EndpointClusterQuery
 	graphURL := "http://localhost" + TESTPORT + EndpointGraph
 
@@ -103,6 +104,7 @@ func TestClusterStorage(t *testing.T) {
 }
 
 func TestClusterQuery(t *testing.T) {
+
 	queryURL := "http://localhost" + TESTPORT + EndpointClusterQuery
 
 	st, _, res := sendTestRequest(queryURL, "GET", nil)
@@ -425,6 +427,7 @@ func TestClusterQuery(t *testing.T) {
 }
 
 func TestClusterQueryBigCluster(t *testing.T) {
+
 	queryURL := "http://localhost" + TESTPORT + EndpointClusterQuery
 
 	// Create a big cluster

api/v1/ecal-sock.go

@@ -0,0 +1,185 @@
+/*
+ * EliasDB
+ *
+ * Copyright 2016 Matthias Ladkau. All rights reserved.
+ *
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, You can obtain one at http://mozilla.org/MPL/2.0/.
+ */
+
+package v1
+
+import (
+	"encoding/json"
+	"fmt"
+	"io/ioutil"
+	"net/http"
+	"strings"
+
+	"devt.de/krotik/common/cryptutil"
+	"devt.de/krotik/common/errorutil"
+	"devt.de/krotik/common/stringutil"
+	"devt.de/krotik/ecal/engine"
+	"devt.de/krotik/ecal/scope"
+	"devt.de/krotik/eliasdb/api"
+	"devt.de/krotik/eliasdb/ecal"
+	"github.com/gorilla/websocket"
+)
+
+/*
+EndpointECALSock is the ECAL endpoint URL (rooted) for websocket operations. Handles everything under sock/...
+*/
+const EndpointECALSock = api.APIRoot + "/sock/"
+
+/*
+upgrader can upgrade normal requests to websocket communications
+*/
+var sockUpgrader = websocket.Upgrader{
+	Subprotocols:    []string{"ecal-sock"},
+	ReadBufferSize:  1024,
+	WriteBufferSize: 1024,
+}
+
+var sockCallbackError error
+
+/*
+ECALSockEndpointInst creates a new endpoint handler.
+*/
+func ECALSockEndpointInst() api.RestEndpointHandler {
+	return &ecalSockEndpoint{}
+}
+
+/*
+Handler object for ECAL websocket operations.
+*/
+type ecalSockEndpoint struct {
+	*api.DefaultEndpointHandler
+}
+
+/*
+HandleGET handles ECAL websocket operations.
+*/
+func (e *ecalSockEndpoint) HandleGET(w http.ResponseWriter, r *http.Request, resources []string) {
+
+	if api.SI != nil {
+		var body []byte
+
+		// Update the incomming connection to a websocket
+		// If the upgrade fails then the client gets an HTTP error response.
+
+		conn, err := sockUpgrader.Upgrade(w, r, nil)
+
+		if err != nil {
+
+			// We give details here on what went wrong
+
+			w.Write([]byte(err.Error()))
+			return
+		}
+
+		commID := fmt.Sprintf("%x", cryptutil.GenerateUUID())
+
+		wc := ecal.NewWebsocketConnection(commID, conn)
+
+		wc.Init()
+
+		if body, err = ioutil.ReadAll(r.Body); err == nil {
+
+			var data interface{}
+			json.Unmarshal(body, &data)
+
+			query := map[interface{}]interface{}{}
+			for k, v := range r.URL.Query() {
+				values := make([]interface{}, 0)
+				for _, val := range v {
+					values = append(values, val)
+				}
+				query[k] = values
+			}
+
+			header := map[interface{}]interface{}{}
+			for k, v := range r.Header {
+				header[k] = scope.ConvertJSONToECALObject(v)
+			}
+
+			proc := api.SI.Interpreter.RuntimeProvider.Processor
+			event := engine.NewEvent(fmt.Sprintf("WebSocketRequest"), []string{"db", "web", "sock"},
+				map[interface{}]interface{}{
+					"commID":     commID,
+					"path":       strings.Join(resources, "/"),
+					"pathList":   resources,
+					"bodyString": string(body),
+					"bodyJSON":   scope.ConvertJSONToECALObject(data),
+					"query":      query,
+					"method":     r.Method,
+					"header":     header,
+				})
+
+			// Add event that the websocket has been registered
+
+			if _, err = proc.AddEventAndWait(event, nil); err == nil {
+				api.SI.RegisterECALSock(wc)
+				defer func() {
+					api.SI.DeregisterECALSock(wc)
+				}()
+
+				for {
+					var fatal bool
+					var data map[string]interface{}
+
+					// Read websocket message
+
+					if data, fatal, err = wc.ReadData(); err != nil {
+
+						wc.WriteData(map[string]interface{}{
+							"error": err.Error(),
+						})
+
+						if fatal {
+							break
+						}
+
+						continue
+					}
+
+					if val, ok := data["close"]; ok && stringutil.IsTrueValue(fmt.Sprint(val)) {
+						wc.Close("")
+						break
+					}
+
+					event = engine.NewEvent(fmt.Sprintf("WebSocketRequest"), []string{"db", "web", "sock", "data"},
+						map[interface{}]interface{}{
+							"commID":   commID,
+							"path":     strings.Join(resources, "/"),
+							"pathList": resources,
+							"query":    query,
+							"method":   r.Method,
+							"header":   header,
+							"data":     scope.ConvertJSONToECALObject(data),
+						})
+
+					_, err = proc.AddEvent(event, nil)
+					errorutil.AssertOk(err)
+				}
+			}
+
+		}
+
+		if err != nil {
+			wc.Close(err.Error())
+			api.SI.Interpreter.RuntimeProvider.Logger.LogDebug(err)
+		}
+
+		return
+	}
+
+	http.Error(w, "Resource was not found", http.StatusNotFound)
+}
+
+/*
+SwaggerDefs is used to describe the endpoint in swagger.
+*/
+func (e *ecalSockEndpoint) SwaggerDefs(s map[string]interface{}) {
+	// No swagger definitions for this endpoint as it only handles websocket requests
+}

api/v1/ecal-sock_test.go

@@ -0,0 +1,180 @@
+/*
+ * EliasDB
+ *
+ * Copyright 2016 Matthias Ladkau. All rights reserved.
+ *
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, You can obtain one at http://mozilla.org/MPL/2.0/.
+ */
+
+package v1
+
+import (
+	"fmt"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+
+	"devt.de/krotik/common/errorutil"
+	"devt.de/krotik/ecal/cli/tool"
+	"devt.de/krotik/ecal/engine"
+	"devt.de/krotik/ecal/util"
+	"devt.de/krotik/eliasdb/api"
+	"github.com/gorilla/websocket"
+)
+
+func TestECALSockConnectionErrors(t *testing.T) {
+	queryURL := "http://localhost" + TESTPORT + EndpointECALSock
+
+	_, _, res := sendTestRequest(queryURL+"foo?bar=123", "GET", nil)
+
+	if res != `Bad Request
+websocket: the client is not using the websocket protocol: 'upgrade' token not found in 'Connection' header` {
+		t.Error("Unexpected response:", res)
+		return
+	}
+
+	oldSI := api.SI
+	api.SI = nil
+	defer func() {
+		api.SI = oldSI
+	}()
+
+	_, _, res = sendTestRequest(queryURL+"foo?bar=123", "GET", nil)
+
+	if res != `Resource was not found` {
+		t.Error("Unexpected response:", res)
+		return
+	}
+}
+
+func TestECALSock(t *testing.T) {
+	queryURL := "ws://localhost" + TESTPORT + EndpointECALSock + "foo?bar=123"
+	lastUUID := ""
+	var lastDataEvent *engine.Event
+
+	resetSI()
+	api.SI.Interpreter = tool.NewCLIInterpreter()
+	testScriptDir := "testscripts"
+	api.SI.Interpreter.Dir = &testScriptDir
+	errorutil.AssertOk(api.SI.Interpreter.CreateRuntimeProvider("eliasdb-runtime"))
+	logger := util.NewMemoryLogger(10)
+	api.SI.Interpreter.RuntimeProvider.Logger = logger
+
+	errorutil.AssertOk(api.SI.Interpreter.RuntimeProvider.Processor.AddRule(&engine.Rule{
+		Name:            "WebSocketRegister",                 // Name
+		Desc:            "Handles a websocket communication", // Description
+		KindMatch:       []string{"db.web.sock"},             // Kind match
+		ScopeMatch:      []string{},
+		StateMatch:      nil,
+		Priority:        0,
+		SuppressionList: nil,
+		Action: func(p engine.Processor, m engine.Monitor, e *engine.Event, tid uint64) error {
+			lastUUID = fmt.Sprint(e.State()["commID"])
+			return nil
+		},
+	}))
+
+	wg := &sync.WaitGroup{}
+
+	errorutil.AssertOk(api.SI.Interpreter.RuntimeProvider.Processor.AddRule(&engine.Rule{
+		Name:            "WebSocketHandler",                  // Name
+		Desc:            "Handles a websocket communication", // Description
+		KindMatch:       []string{"db.web.sock.data"},        // Kind match
+		ScopeMatch:      []string{},
+		StateMatch:      nil,
+		Priority:        0,
+		SuppressionList: nil,
+		Action: func(p engine.Processor, m engine.Monitor, e *engine.Event, tid uint64) error {
+			lastDataEvent = e
+			wg.Done()
+			return nil
+		},
+	}))
+
+	api.SI.Interpreter.RuntimeProvider.Processor.Start()
+	defer api.SI.Interpreter.RuntimeProvider.Processor.Finish()
+
+	// Now do the actual testing
+
+	c, _, err := websocket.DefaultDialer.Dial(queryURL, nil)
+	if err != nil {
+		t.Error("Could not open websocket:", err)
+		return
+	}
+
+	_, message, err := c.ReadMessage()
+
+	if msg := formatJSONString(string(message)); err != nil || msg != `{
+  "type": "init_success",
+  "payload": {}
+}` {
+		t.Error("Unexpected response:", msg, err)
+		return
+	}
+
+	err = c.WriteMessage(websocket.TextMessage, []byte("buu"))
+	if err != nil {
+		t.Error("Could not send message:", err)
+		return
+	}
+
+	_, message, err = c.ReadMessage()
+	if msg := formatJSONString(string(message)); err != nil || msg != `{
+  "commID": "`+lastUUID+`",
+  "payload": {
+    "error": "invalid character 'b' looking for beginning of value"
+  },
+  "type": "data"
+}` {
+		t.Error("Unexpected response:", msg, err)
+		return
+	}
+
+	wg.Add(1)
+
+	err = c.WriteMessage(websocket.TextMessage, []byte(`{"foo":"bar"}`))
+	if err != nil {
+		t.Error("Could not send message:", err)
+		return
+	}
+
+	wg.Wait()
+
+	if data := lastDataEvent.State()["data"]; err != nil || fmt.Sprint(data) != `map[foo:bar]` {
+		t.Error("Unexpected response:", data, err)
+		return
+	}
+
+	err = c.WriteMessage(websocket.TextMessage, []byte(`{"close":true}`))
+	if err != nil {
+		t.Error("Could not send message:", err)
+		return
+	}
+
+	// Reset the connection and provoke an error
+
+	c, _, err = websocket.DefaultDialer.Dial(queryURL, nil)
+	if err != nil {
+		t.Error("Could not open websocket:", err)
+		return
+	}
+
+	c.Close()
+
+	for {
+
+		if logger.Size() > 0 {
+			break
+		}
+
+		time.Sleep(10 * time.Millisecond)
+	}
+
+	if !strings.Contains(logger.String(), "unexpected EOF") && !strings.Contains(logger.String(), "connection reset by peer") {
+		t.Error("Unexpected log output:", logger.String())
+		return
+	}
+}

api/v1/ecal.go

@@ -0,0 +1,222 @@
+/*
+ * EliasDB
+ *
+ * Copyright 2016 Matthias Ladkau. All rights reserved.
+ *
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, You can obtain one at http://mozilla.org/MPL/2.0/.
+ */
+
+package v1
+
+import (
+	"encoding/json"
+	"fmt"
+	"io/ioutil"
+	"net/http"
+	"strconv"
+	"strings"
+
+	"devt.de/krotik/ecal/engine"
+	"devt.de/krotik/ecal/scope"
+	"devt.de/krotik/ecal/util"
+	"devt.de/krotik/eliasdb/api"
+	"devt.de/krotik/eliasdb/ecal/dbfunc"
+)
+
+/*
+EndpointECALInternal is the ECAL endpoint URL (rooted) for internal operations. Handles everything under ecal/...
+*/
+const EndpointECALInternal = api.APIRoot + "/ecal/"
+
+/*
+EndpointECALPublic is the ECAL endpoint URL (rooted) for public API operations. Handles everything under api/...
+*/
+const EndpointECALPublic = api.APIRoot + "/api/"
+
+/*
+ECALEndpointInst creates a new endpoint handler.
+*/
+func ECALEndpointInst() api.RestEndpointHandler {
+	return &ecalEndpoint{}
+}
+
+/*
+Handler object for ecal operations.
+*/
+type ecalEndpoint struct {
+	*api.DefaultEndpointHandler
+}
+
+/*
+	HandleGET handles a GET request.
+*/
+func (ee *ecalEndpoint) HandleGET(w http.ResponseWriter, r *http.Request, resources []string) {
+	ee.forwardRequest(w, r, resources)
+}
+
+/*
+	HandlePOST handles a POST request.
+*/
+func (ee *ecalEndpoint) HandlePOST(w http.ResponseWriter, r *http.Request, resources []string) {
+	ee.forwardRequest(w, r, resources)
+}
+
+/*
+	HandlePUT handles a PUT request.
+*/
+func (ee *ecalEndpoint) HandlePUT(w http.ResponseWriter, r *http.Request, resources []string) {
+	ee.forwardRequest(w, r, resources)
+}
+
+/*
+	HandleDELETE handles a DELETE request.
+*/
+func (ee *ecalEndpoint) HandleDELETE(w http.ResponseWriter, r *http.Request, resources []string) {
+	ee.forwardRequest(w, r, resources)
+}
+
+func (ee *ecalEndpoint) forwardRequest(w http.ResponseWriter, r *http.Request, resources []string) {
+
+	if api.SI != nil {
+
+		// Make sure the request we are handling comes from a known path for ECAL
+
+		isPublic := strings.HasPrefix(r.URL.Path, EndpointECALPublic)
+		isInternal := strings.HasPrefix(r.URL.Path, EndpointECALInternal)
+
+		if isPublic || isInternal {
+			var eventKind []string
+
+			body, err := ioutil.ReadAll(r.Body)
+
+			if err == nil {
+				if isPublic {
+					eventKind = []string{"db", "web", "api"}
+				} else {
+					eventKind = []string{"db", "web", "ecal"}
+				}
+
+				var data interface{}
+				json.Unmarshal(body, &data)
+
+				query := map[interface{}]interface{}{}
+				for k, v := range r.URL.Query() {
+					values := make([]interface{}, 0)
+					for _, val := range v {
+						values = append(values, val)
+					}
+					query[k] = values
+				}
+
+				header := map[interface{}]interface{}{}
+				for k, v := range r.Header {
+					header[k] = scope.ConvertJSONToECALObject(v)
+				}
+
+				proc := api.SI.Interpreter.RuntimeProvider.Processor
+				event := engine.NewEvent(fmt.Sprintf("WebRequest"), eventKind,
+					map[interface{}]interface{}{
+						"path":       strings.Join(resources, "/"),
+						"pathList":   resources,
+						"bodyString": string(body),
+						"bodyJSON":   scope.ConvertJSONToECALObject(data),
+						"query":      query,
+						"method":     r.Method,
+						"header":     header,
+					})
+
+				var m engine.Monitor
+
+				if m, err = proc.AddEventAndWait(event, nil); err == nil {
+					if m != nil {
+						var headers map[interface{}]interface{}
+						status := 0
+						var body []byte
+
+						for _, e := range m.(*engine.RootMonitor).AllErrors() {
+							if len(e.ErrorMap) > 0 {
+								for _, e := range e.ErrorMap {
+									if re, ok := e.(*util.RuntimeErrorWithDetail); ok && re.Type == dbfunc.ErrWebEventHandled {
+										res := re.Data.(map[interface{}]interface{})
+
+										if status, err = strconv.Atoi(fmt.Sprint(res["status"])); err == nil {
+											headers, _ = res["header"].(map[interface{}]interface{})
+											body, err = json.Marshal(scope.ConvertECALToJSONObject(res["body"]))
+										}
+									} else {
+										err = e
+									}
+									break
+								}
+								break
+							}
+						}
+
+						if status != 0 {
+							for k, v := range headers {
+								w.Header().Set(fmt.Sprint(k), fmt.Sprint(v))
+							}
+							w.WriteHeader(status)
+							fmt.Fprintln(w, string(body))
+							return
+						}
+					}
+				}
+			}
+
+			if err != nil {
+				api.SI.Interpreter.RuntimeProvider.Logger.LogError(err)
+			}
+		}
+	}
+
+	http.Error(w, "Resource was not found", http.StatusNotFound)
+}
+
+/*
+SwaggerDefs is used to describe the endpoint in swagger.
+*/
+func (ee *ecalEndpoint) SwaggerDefs(s map[string]interface{}) {
+
+	desc := map[string]interface{}{
+		"summary":     "Forward web requests to the ECAL backend.",
+		"description": "The ecal endpoint forwards web requests to the ECAL backend.",
+		"produces": []string{
+			"text/plain",
+			"application/json",
+		},
+		"responses": map[string]interface{}{
+			"200": map[string]interface{}{
+				"description": "A result object generated by ECAL scripts.",
+			},
+			"default": map[string]interface{}{
+				"description": "Error response",
+				"schema": map[string]interface{}{
+					"$ref": "#/definitions/Error",
+				},
+			},
+		},
+	}
+
+	s["paths"].(map[string]interface{})["/ecal"] = map[string]interface{}{
+		"get":    desc,
+		"post":   desc,
+		"put":    desc,
+		"delete": desc,
+	}
+	s["paths"].(map[string]interface{})["/api"] = map[string]interface{}{
+		"get":    desc,
+		"post":   desc,
+		"put":    desc,
+		"delete": desc,
+	}
+
+	// Add generic error object to definition
+
+	s["definitions"].(map[string]interface{})["Error"] = map[string]interface{}{
+		"description": "A human readable error mesage.",
+		"type":        "string",
+	}
+}

api/v1/ecal_test.go

@@ -0,0 +1,186 @@
+/*
+ * EliasDB
+ *
+ * Copyright 2016 Matthias Ladkau. All rights reserved.
+ *
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, You can obtain one at http://mozilla.org/MPL/2.0/.
+ */
+
+package v1
+
+import (
+	"testing"
+
+	"devt.de/krotik/eliasdb/api"
+)
+
+func TestECAL(t *testing.T) {
+	internalURL := "http://localhost" + TESTPORT + EndpointECALInternal
+	publicURL := "http://localhost" + TESTPORT + EndpointECALPublic
+
+	// Test normal log output
+
+	writeScript(`
+log("test insert")
+`)
+
+	if err := api.SI.Run(); err != nil {
+		t.Error("Unexpected result:", err)
+		return
+	}
+
+	if err := checkLog(`test insert
+`); err != nil {
+		t.Error(err)
+	}
+
+	writeScript(`
+log("test sinks")
+sink mysink
+  kindmatch [ "db.web.api" ],
+  statematch { "method" : "POST", "path" : "xx/ss" }
+{
+  del(event.state.header, "Accept-Encoding")
+  del(event.state.header, "User-Agent")
+  log("Got public web request: ", event)
+  log("Body data: ", event.state.bodyJSON.data)
+  db.raiseWebEventHandled({
+	"status" : 201,
+	"body" : {
+		"mydata" : [1,2,3]
+	}
+  })
+}
+sink mysink2
+  kindmatch [ "db.web.ecal" ],
+  statematch { "method" : "GET" }
+{
+  del(event.state.header, "Accept-Encoding")
+  del(event.state.header, "User-Agent")
+  log("Got internal web request: ", event)
+  log("Query data: ", event.state.query.xxx)
+  raise("aaa")
+}
+`)
+
+	if err := api.SI.Run(); err != nil {
+		t.Error("Unexpected result:", err)
+		return
+	}
+
+	st, _, res := sendTestRequest(internalURL+"main/n/Test/bar?xxx=1", "GET", nil)
+
+	if st != "404 Not Found" || res != "Resource was not found" {
+		t.Error("Unexpected result:", st, res)
+		return
+	}
+
+	st, header, res := sendTestRequest(publicURL+"xx/ss/?a=1&b=2", "POST", []byte(`
+{
+  "data": 123
+}
+`[1:]))
+
+	if st != "201 Created" || header["Content-Type"][0] != "application/json; charset=utf-8" || string(res) != `{
+  "mydata": [
+    1,
+    2,
+    3
+  ]
+}` {
+		t.Error("Unexpected result:", st, header, string(res))
+		return
+	}
+
+	if err := checkLog(`test sinks
+Got internal web request: {
+  "kind": "db.web.ecal",
+  "name": "WebRequest",
+  "state": {
+    "bodyJSON": null,
+    "bodyString": "",
+    "header": {
+      "Content-Type": [
+        "application/json"
+      ]
+    },
+    "method": "GET",
+    "path": "main/n/Test/bar",
+    "pathList": [
+      "main",
+      "n",
+      "Test",
+      "bar"
+    ],
+    "query": {
+      "xxx": [
+        "1"
+      ]
+    }
+  }
+}
+Query data: [
+  "1"
+]
+error: ECAL error in eliasdb-runtime (testscripts/main.ecal): aaa () (Line:26 Pos:3)
+Got public web request: {
+  "kind": "db.web.api",
+  "name": "WebRequest",
+  "state": {
+    "bodyJSON": {
+      "data": 123
+    },
+    "bodyString": "{\n  \"data\": 123\n}\n",
+    "header": {
+      "Content-Length": [
+        "18"
+      ],
+      "Content-Type": [
+        "application/json"
+      ]
+    },
+    "method": "POST",
+    "path": "xx/ss",
+    "pathList": [
+      "xx",
+      "ss"
+    ],
+    "query": {
+      "a": [
+        "1"
+      ],
+      "b": [
+        "2"
+      ]
+    }
+  }
+}
+Body data: 123
+`); err != nil {
+		t.Error(err)
+		return
+	}
+
+	oldSI := api.SI
+	defer func() {
+		api.SI = oldSI
+	}()
+
+	api.SI = nil
+
+	st, _, res = sendTestRequest(internalURL, "PUT", nil)
+
+	if st != "404 Not Found" || res != "Resource was not found" {
+		t.Error("Unexpected result:", st, res)
+		return
+	}
+
+	st, _, res = sendTestRequest(internalURL, "DELETE", nil)
+
+	if st != "404 Not Found" || res != "Resource was not found" {
+		t.Error("Unexpected result:", st, res)
+		return
+	}
+}

api/v1/graph_test.go

@@ -315,7 +315,7 @@ func TestGraphQuery(t *testing.T) {
 	st, _, res = sendTestRequest(queryURL+"/main/n/Spam?offset=19&limit=1", "GET", nil)
 
 	if st != "500 Internal Server Error" ||
-		res != "GraphError: Could not read graph information (Record is already in-use (? - ))" {
+		res != "GraphError: Could not read graph information (Record is already in-use (<memory> - ))" {
 		t.Error("Unexpected response:", res)
 		return
 	}
@@ -323,7 +323,7 @@ func TestGraphQuery(t *testing.T) {
 	st, _, res = sendTestRequest(queryURL+"/main/n/Spam", "GET", nil)
 
 	if st != "500 Internal Server Error" ||
-		res != "GraphError: Could not read graph information (Record is already in-use (? - ))" {
+		res != "GraphError: Could not read graph information (Record is already in-use (<memory> - ))" {
 		t.Error("Unexpected response:", res)
 		return
 	}

api/v1/graphql-subscriptions_test.go

@@ -72,8 +72,6 @@ func TestGraphQLSubscriptionMissingPartition(t *testing.T) {
 func TestGraphQLSubscription(t *testing.T) {
 	queryURL := "ws://localhost" + TESTPORT + EndpointGraphQLSubscriptions + "main"
 
-	// Test missing partition
-
 	c, _, err := websocket.DefaultDialer.Dial(queryURL, nil)
 	if err != nil {
 		t.Error("Could not open websocket:", err)
@@ -247,7 +245,7 @@ func TestGraphQLSubscription(t *testing.T) {
             "line": 1
           }
         ],
-        "message": "GraphError: Could not read graph information (Record is already in-use (? - ))",
+        "message": "GraphError: Could not read graph information (Record is already in-use (\u003cmemory\u003e - ))",
         "path": [
           "Author"
         ]

api/v1/rest.go

@@ -49,6 +49,15 @@ var V1EndpointMap = map[string]api.RestEndpointInst{
 	EndpointInfoQuery:            InfoEndpointInst,
 	EndpointQuery:                QueryEndpointInst,
 	EndpointQueryResult:          QueryResultEndpointInst,
+	EndpointECALInternal:         ECALEndpointInst,
+	EndpointECALSock:             ECALSockEndpointInst,
+}
+
+/*
+V1PublicEndpointMap is a map of urls to public endpoints for version 1 of the API
+*/
+var V1PublicEndpointMap = map[string]api.RestEndpointInst{
+	EndpointECALPublic: ECALEndpointInst,
 }
 
 // Helper functions

api/v1/rest_test.go

@@ -14,17 +14,22 @@ import (
 	"bytes"
 	"encoding/json"
 	"flag"
+	"fmt"
 	"io/ioutil"
 	"net/http"
 	"os"
+	"path/filepath"
 	"strconv"
 	"strings"
 	"sync"
 	"testing"
 
 	"devt.de/krotik/common/errorutil"
+	"devt.de/krotik/common/fileutil"
 	"devt.de/krotik/common/httputil"
 	"devt.de/krotik/eliasdb/api"
+	"devt.de/krotik/eliasdb/config"
+	"devt.de/krotik/eliasdb/ecal"
 	"devt.de/krotik/eliasdb/eql"
 	"devt.de/krotik/eliasdb/graph"
 	"devt.de/krotik/eliasdb/graph/data"
@@ -35,16 +40,47 @@ const TESTPORT = ":9090"
 
 var gmMSM *graphstorage.MemoryGraphStorage
 
+const testScriptDir = "testscripts"
+
 // Main function for all tests in this package
 
 func TestMain(m *testing.M) {
 	flag.Parse()
 
+	defer func() {
+		if res, _ := fileutil.PathExists(testScriptDir); res {
+			if err := os.RemoveAll(testScriptDir); err != nil {
+				fmt.Print("Could not remove test directory:", err.Error())
+			}
+		}
+	}()
+
+	if res, _ := fileutil.PathExists(testScriptDir); res {
+		if err := os.RemoveAll(testScriptDir); err != nil {
+			fmt.Print("Could not remove test directory:", err.Error())
+		}
+	}
+
+	ensurePath(testScriptDir)
+
+	data := make(map[string]interface{})
+	for k, v := range config.DefaultConfig {
+		data[k] = v
+	}
+
+	config.Config = data
+
+	config.Config[config.EnableECALScripts] = true
+	config.Config[config.ECALScriptFolder] = testScriptDir
+	config.Config[config.ECALLogFile] = filepath.Join(testScriptDir, "interpreter.log")
+
 	gm, msm := filterGraph()
 	api.GM = gm
 	api.GS = msm
 	gmMSM = msm
 
+	resetSI()
+
 	hs, wg := startServer()
 	if hs == nil {
 		return
@@ -53,16 +89,15 @@ func TestMain(m *testing.M) {
 	// Register endpoints for version 1
 
 	api.RegisterRestEndpoints(V1EndpointMap)
+	api.RegisterRestEndpoints(V1PublicEndpointMap)
 
 	// Run the tests
 
-	res := m.Run()
+	m.Run()
 
 	// Teardown
 
 	stopServer(hs, wg)
-
-	os.Exit(res)
 }
 
 func TestSwaggerDefs(t *testing.T) {
@@ -351,3 +386,40 @@ func filterGraph() (*graph.Manager, *graphstorage.MemoryGraphStorage) {
 
 	return gm, mgs
 }
+
+func ensurePath(path string) {
+	if res, _ := fileutil.PathExists(path); !res {
+		if err := os.Mkdir(path, 0770); err != nil {
+			fmt.Print("Could not create directory:", err.Error())
+			return
+		}
+	}
+}
+
+func resetSI() {
+	api.SI = ecal.NewScriptingInterpreter(testScriptDir, api.GM)
+}
+
+func writeScript(content string) {
+	filename := filepath.Join(testScriptDir, config.Str(config.ECALEntryScript))
+	err := ioutil.WriteFile(
+		filename,
+		[]byte(content), 0600)
+	errorutil.AssertOk(err)
+	os.Remove(config.Str(config.ECALLogFile))
+}
+
+func checkLog(expected string) error {
+	var err error
+
+	content, err := ioutil.ReadFile(config.Str(config.ECALLogFile))
+
+	if err == nil {
+		logtext := string(content)
+
+		if logtext != expected {
+			err = fmt.Errorf("Unexpected log text:\n%v", logtext)
+		}
+	}
+	return err
+}

cli/eliasdb.go

@@ -47,10 +47,12 @@ import (
 	"os"
 	"path/filepath"
 	"strings"
+	"time"
 
 	"devt.de/krotik/common/errorutil"
 	"devt.de/krotik/common/fileutil"
 	"devt.de/krotik/common/termutil"
+	"devt.de/krotik/eliasdb/api"
 	"devt.de/krotik/eliasdb/config"
 	"devt.de/krotik/eliasdb/console"
 	"devt.de/krotik/eliasdb/graph"
@@ -289,10 +291,15 @@ handleServerCommandLine handles all command line options for the server
 */
 func handleServerCommandLine(gm *graph.Manager) bool {
 	var err error
+	var ecalConsole *bool
 
 	importDb := flag.String("import", "", "Import a database from a zip file")
 	exportDb := flag.String("export", "", "Export the current database to a zip file")
 
+	if config.Bool(config.EnableECALScripts) {
+		ecalConsole = flag.Bool("ecal-console", false, "Start an interactive interpreter console for ECAL")
+	}
+
 	noServ := flag.Bool("no-serv", false, "Do not start the server after initialization")
 
 	showHelp := flag.Bool("help", false, "Show this help message")
@@ -368,6 +375,48 @@ func handleServerCommandLine(gm *graph.Manager) bool {
 		}
 	}
 
+	if ecalConsole != nil && *ecalConsole {
+		var term termutil.ConsoleLineTerminal
+
+		isExitLine := func(s string) bool {
+			return s == "exit" || s == "q" || s == "quit" || s == "bye" || s == "\x04"
+		}
+
+		term, err = termutil.NewConsoleLineTerminal(os.Stdout)
+		if err == nil {
+			term, err = termutil.AddHistoryMixin(term, "", isExitLine)
+			if err == nil {
+				tid := api.SI.Interpreter.RuntimeProvider.NewThreadID()
+
+				runECALConsole := func(delay int) {
+					defer term.StopTerm()
+
+					time.Sleep(time.Duration(delay) * time.Millisecond)
+
+					term.WriteString(fmt.Sprintln("Type 'q' or 'quit' to exit the shell and '?' to get help"))
+
+					line, err := term.NextLine()
+					for err == nil && !isExitLine(line) {
+						trimmedLine := strings.TrimSpace(line)
+
+						api.SI.Interpreter.HandleInput(term, trimmedLine, tid)
+
+						line, err = term.NextLine()
+					}
+				}
+
+				if err = term.StartTerm(); err == nil {
+
+					if *noServ {
+						runECALConsole(0)
+					} else {
+						go runECALConsole(3000)
+					}
+				}
+			}
+		}
+	}
+
 	if err != nil {
 		fmt.Println(err.Error())
 		return true

cluster/distributedstorage_test.go

@@ -250,7 +250,7 @@ func TestSimpleDataDistribution(t *testing.T) {
 		return
 	}
 
-	if _, err := sm.FetchCached(5); err != storage.ErrNotInCache {
+	if _, err := sm.FetchCached(5); err.(*storage.ManagerError).Type != storage.ErrNotInCache {
 		t.Error("Unexpected response:", err)
 		return
 	}

cluster/distributedstoragemanager.go

@@ -355,7 +355,7 @@ FetchCached is not implemented for a DistributedStorageManager. Only defined to
 the StorageManager interface.
 */
 func (dsm *DistributedStorageManager) FetchCached(loc uint64) (interface{}, error) {
-	return nil, storage.ErrNotInCache
+	return nil, storage.NewStorageManagerError(storage.ErrNotInCache, "", dsm.Name())
 }
 
 /*

cluster/memberaddresstable_test.go

@@ -127,7 +127,7 @@ func TestAddressTableClusterLoc(t *testing.T) {
 	msm.AccessMap[loc] = storage.AccessCacheAndFetchSeriousError
 
 	loc, err = ms1[0].at.NewClusterLoc("test1")
-	if err.Error() != "Record is already in-use (? - )" {
+	if err.Error() != "Record is already in-use (<memory> - )" {
 		t.Error("Unexpected result:", loc, err)
 		return
 	}

config/config.go

@@ -49,6 +49,8 @@ const (
 	HTTPSPort                = "HTTPSPort"
 	CookieMaxAgeSeconds      = "CookieMaxAgeSeconds"
 	EnableReadOnly           = "EnableReadOnly"
+	EnableECALScripts        = "EnableECALScripts"
+	EnableECALDebugServer    = "EnableECALDebugServer"
 	EnableWebFolder          = "EnableWebFolder"
 	EnableAccessControl      = "EnableAccessControl"
 	EnableWebTerminal        = "EnableWebTerminal"
@@ -59,6 +61,13 @@ const (
 	ClusterStateInfoFile     = "ClusterStateInfoFile"
 	ClusterConfigFile        = "ClusterConfigFile"
 	ClusterLogHistory        = "ClusterLogHistory"
+	ECALScriptFolder         = "ECALScriptFolder"
+	ECALWorkerCount          = "ECALWorkerCount"
+	ECALEntryScript          = "ECALEntryScript"
+	ECALLogLevel             = "ECALLogLevel"
+	ECALLogFile              = "ECALLogFile"
+	ECALDebugServerHost      = "ECALDebugServerHost"
+	ECALDebugServerPort      = "ECALDebugServerPort"
 )
 
 /*
@@ -67,6 +76,8 @@ DefaultConfig is the defaut configuration
 var DefaultConfig = map[string]interface{}{
 	MemoryOnlyStorage:        false,
 	EnableReadOnly:           false,
+	EnableECALScripts:        false,
+	EnableECALDebugServer:    false,
 	EnableWebFolder:          true,
 	EnableAccessControl:      false,
 	EnableWebTerminal:        true,
@@ -88,6 +99,13 @@ var DefaultConfig = map[string]interface{}{
 	ClusterStateInfoFile:     "cluster.stateinfo",
 	ClusterConfigFile:        "cluster.config.json",
 	ClusterLogHistory:        100.0,
+	ECALScriptFolder:         "scripts",
+	ECALWorkerCount:          10,
+	ECALEntryScript:          "main.ecal",
+	ECALLogLevel:             "info",
+	ECALLogFile:              "",
+	ECALDebugServerHost:      "127.0.0.1",
+	ECALDebugServerPort:      "33274",
 }
 
 /*

ecal-engine.md

@@ -0,0 +1,150 @@
+ECA Engine
+==========
+The ECA engine is ECAL's low-level event engine which does the actual concurrent event processing. Through ECAL a user can define rules which execute certain actions under certain conditions. The engine is defined in `ecal.engine`.
+
+Priorities
+----------
+The event-based system relies heavily on priorities for control flow. Both events and rules (which are triggered by events) have priorities. By default events and rules have the priority 0 which is the highest priority. Events are processed according to their priority and all triggering rules of a single event are executed according to their priority.
+
+Processor
+---------
+The processor is the central piece of the event engine. It controls the thread pool, contains the rule index and handles the event processing.
+
+The engines behaviour is solely defined by rules. These rules are added before the engine is started. Each added rule has a priority which determines their execution order if multiple rules are triggered by the same event. The main processing cycle, once the engine has been started, can be described as:
+
+Event injection -> Triggering check -> Rule Matching -> Fire Rules
+
+When injecting a new event it is possible to also pass a monitor with a certain scope and a priority. The scope is used by the processor to narrow down the triggering rules. A possible scenario for scopes are different types of analysis (e.g. quick analysis or deep analysis - only a subset of rules is required for the quick analysis). The priority determines when an event is processed - higher priority events are processed first.
+
+After an event is injected the Processor first checks if anything triggers on the event. The result of this is cached. The trigger check is just a first quick check to determine if the event can be discarded right away - even if the event passes the check, it is possible, that no rule will actually fire.
+
+After the first triggering check passed, the event is handed over to a task which runs in the thread pool. The task uses the rule index to determine all triggering rules. After filtering rules which are out of scope or which are suppressed by other rules, the remaining rules are sorted by their priority and then their action is executed.
+
+A rule action can inject new events into the processor which starts the processing cycle again. The processor supports two modes of execution for rule sequences (rules triggered by an event in order of priority):
+
+1. Do not fail on errors: all rules in a trigger sequence for a specific event
+are executed.
+
+2. Fail on first error: the first rule which returns an error will stop
+the trigger sequence. Events which have been added by the failing rule are still processed.
+
+Failing on the first error can be useful in scenarios where authorization is required. High priority rules can block lower priority rules from being executed.
+
+
+Monitor
+-------
+For every event there is a monitor following the event. Monitors form trees as the events cascade. Monitor objects hold additional information such as priority (how quickly should the associated event be processed), processing errors, rule scope, as well as context objects.
+
+
+Rules
+-----
+Rules define the conditions under which a particular action should be executed. Every rule must have the following properties:
+
+- [Name] A name which identifies the rule.
+- [KindMatch] Match on event kinds: A list of strings in dot notation which describes event kinds. May contain '*' characters as wildcards (e.g. core.tests.*).
+- [ScopeMatch] Match on event cascade scope: A list of strings in dot notation which describe the required scopes which are required for this rule to trigger. The included / excluded scopes for an event are stored in its monitor.
+- [StateMatch] Match on event state: A simple list of required key / value states in the event state. Nil values can be used as wildcards (i.e. match is only on key).
+- [Priority] Rules are sorted by their priority before their actions are executed.
+- [SuppressionList] A list of rules (identified by their name) which should be suppressed if this rule fires.
+- [Action] A function which will be executed if this rule fires.
+
+
+Events
+------
+Events are injected into the processor and cause rules to fire. An event is a simple object which contains:
+
+- [Name] A name which identifies the event.
+- [Kind] An event kind - this is checked against the kind match of rules during the triggering check.
+- [State] An event state which contains additional data.
+
+Events are always processed together with a monitor which is either implicitly created or explicitly given together with the event. If the monitor is explicitly given it is possible to specify an event scope which limits the triggering rules and a priority which determines the event processing order. An event with a lower priority is guaranteed to be processed after all events of a higher priority if these have been added before the lower priority event.
+
+Example
+-------
+- A client instantiates a new Processor giving the number of worker threads which should be used to process rules (a good number here are the cores of the physical processor).
+
+```
+proc := NewProcessor(1)
+```
+
+- The client adds rules to the processor.
+
+```
+rule1 := &Rule{
+		"TestRule1",                            // Name
+		"My test rule",                         // Description
+		[]string{"core.main.event1"},           // Kind match
+		[]string{"data"},                       // Match on event cascade scope
+		nil,                                    // No state match
+		2,                                      // Priority of the rule
+		[]string{"TestRule3", "TestRule2"},     // List of suppressed rules by this rule
+		func(p Processor, m Monitor, e *Event) error { // Action of the rule
+      ... code of the rule
+
+			p.AddEvent(&Event{
+							"Next Event",
+							[]string{"core", "main", "event2"},
+							nil,
+						}, m.NewChildMonitor(1))        // New monitor with priority for new event
+		},
+	}
+
+proc.AddRule(rule1)
+...
+```
+
+- The processor is started. At this point the thread pool inside the processor is waiting for tasks with the defined number of worker threads.
+
+```
+proc.SetRootMonitorErrorObserver(func(rm *RootMonitor) { // Called once a root monitor has finished
+	errs := rm.AllErrors()
+	...
+})
+
+proc.Start()
+```
+
+- A root monitor is instantiated and an initial event is added.
+
+```
+e := NewEvent(
+  "InitialEvent",                      // Name
+  []string{"core", "main", "event1"},  // Kind
+  map[interface{}]interface{}{         // State
+    "foo":  "bar",
+  },
+)
+
+rootm := proc.NewRootMonitor(nil, nil)
+
+rootm.SetFinishHandler(func(p Processor) { // Handler for end of event cascade
+  ...
+})
+
+proc.AddEvent(e, rootm)
+```
+
+- The event is processed as follows:
+
+	- The event is injected into the procesor with or without a parent monitor.
+
+		- Quick (not complete!) check if the event triggers any rules. This is to avoid unnecessary computation.
+			- Check that the event kind is not too general (e.g. the rule is for a.b.c event is for a.b)
+			- Check if 	at least one rule matches the kind. At least on rule should either be triggering on all kinds or triggering on the specific kind of the event.
+
+		- Create a new root monitor if no parent monitor has been given.
+
+		- Add a task to the thread pool of the processor (containing the event, parent/root monitor and processor).
+
+	- Thread pool of the processor takes the next task according to the highest priority.
+
+		- Determine the triggering rules (matching via kind, state and scope without suppressed rules).
+
+		- Execute the action of each triggering rule according to their priority.
+
+- The processor can run as long as needed and can be finished when the application should be terminated.
+
+```
+proc.Finish()
+```
+Calling `Finish()` will finish all remaining tasks and then stop the processor.

ecal.md

@@ -0,0 +1,306 @@
+EliasDB Event Condition Action Language
+=======================================
+
+EliasDB supports a scripting language called [Event Condition Action Language (ECAL)](ecal-lang.md) to enable rule based scripting functionality. ECAL provides [database trigger](https://en.wikipedia.org/wiki/Database_trigger) functionality for EliasDB.
+
+ECAL was added for the following use-cases:
+- Providing a way to manipulate data in response to events
+- Enforce certain aspects of a database schema
+- Providing back-end logic for web applications using EliasDB
+
+The source of EliasDB comes with a [chat example](examples/game/doc/chat.md) containing a simple ECAL script which adds a timestamp to nodes and a [game example](examples/game/doc/game.md) which demonstrates a more complex application of ECAL.
+
+ECAL related config values:
+--
+These ECAL related config options are available in `eliasdb.config.json`:
+
+| Configuration Option | Description |
+| --- | --- |
+| EnableECALScripts | Enable ECAL scripting. |
+| ECALScriptFolder | Scripting folder for ECAL scripts. |
+| ECALEntryScript | Entry script in the script folder. |
+| ECALLogFile | File in which the logs should be written (use an empty string for stdout). |
+| ECALLogLevel | Log level for the printed logs. |
+| EnableECALDebugServer | Enable debugging and start the ECAL debug server. Note: Activating debugging will slow down the interpreter speed significantly! |
+| ECALDebugServerHost | Host for the debug server. |
+| ECALDebugServerPort | Port for the debug server. |
+| ECALWorkerCount | Number of worker threads in the ECA engine's thread pool. |
+
+ECAL Debugging
+--
+If the debug server is enabled in the config file then it is possible to debug ECAL scripts with [VSCode](https://devt.de/krotik/ecal/src/master/ecal-support/README.md). The debugger supports break points and thread state inspection. It is also possible to restart and reload the scripts.
+
+Using the `-ecal-console` parameter it is possible to open an interactive console into the server process. If used together with the debug server additional debug commands are available also there. Enter `?` to see the build-in documentation.
+
+EliasDB specific events which can be handled:
+--
+The ECAL interpreter in EliasDB receives the following events:
+
+| Web Request | ECAL event kind | Event state contents | Description |
+| --- | ---| --- | --- |
+| /db/api/|`db.web.api`| bodyJSON, bodyString, header, method, path, pathList, query | Any web request to /db/api/... These endpoints are public and never require authentication. |
+| /db/ecal/|`db.web.ecal`| bodyJSON, bodyString, header, method, path, pathList, query | Any web request to /db/ecal/... These endpoints are considered internal and require authentication if access control is enabled. |
+| /db/sock/|`db.web.sock`| bodyJSON, bodyString, commID, header, method, path, pathList, query | Any web request to /db/sock/... These endpoints are used to initiate websocket connections. |
+| - |`db.web.sock.data`| commID, data, header, method, path, pathList, query | An existing websocket connection received some JSON object data. If the close attribute of the object is set to true then the websocket connection is closed. |
+
+| EliasDB Graph Event | ECAL event kind | Event state contents | Description |
+| --- | --- | --- | --- |
+| graph.EventNodeCreated | `db.node.created` | part, trans, node | A node was created. |
+| graph.EventNodeUpdated | `db.node.updated` | part, trans, node, old_node | A node was updated. |
+| graph.EventNodeDeleted | `db.node.deleted` | part, trans, node | A node was deleted. |
+| graph.EventEdgeCreated | `db.edge.created` | part, trans, edge | An edge was created. |
+| graph.EventEdgeUpdated | `db.edge.updated` | part, trans, edge, old_edge | An edge was updated. |
+| graph.EventEdgeDeleted | `db.edge.deleted` | part, trans, edge | An edge was deleted. |
+| graph.EventNodeStore | `db.node.store` | part, trans, node | A node is about to be stored (always overwriting existing values). |
+| graph.EventNodeUpdate | `db.node.update` | part, trans, node | A node is about to be updated. |
+| graph.EventNodeDelete | `db.node.delete` | part, trans, key, kind | A node is about to be deleted. |
+| graph.EventEdgeStore | `db.edge.store` | part, trans, edge | An edge is about to be stored. |
+| graph.EventEdgeDelete | `db.edge.delete` | part, trans, key, kind | An edge is about to be deleted. |
+
+Note: EliasDB will wait for the event cascade to be finished before performing the actual operation (e.g. inserting a node). If the event handling requires a time consuming operation then a new parallel event cascade can be started using `addEvent` with a scope:
+
+EliasDB can receive the following events from the ECAL interpreter:
+
+| ECAL event kind | Event state contents | Description |
+| --- | --- | --- |
+| db.web.sock.msg | commID, payload, close | The payload is send to a client with an open websocket identified by the commID. |
+```
+addEvent("request", "foo.bar.xxx", {
+   "payload" : 123
+}, {
+   "": true  # This scope allows all events
+})
+```
+
+EliasDB specific functions:
+--
+The ECAL interpreter in EliasDB supports the following EliasDB specific functions:
+
+#### `db.storeNode(partition, nodeMap, [transaction])`
+Inserts or updates a node in EliasDB.
+
+Parameter | Description
+-|-
+partition | Partition of the node
+nodeMap | Node object as a map with at least a key and a kind attribute
+transaction | Optional a transaction to group a set of changes
+
+Example:
+```
+db.storeNode("main", {
+  "key" : "foo",
+  "kind" : "bar",
+  "data" : 123,
+})
+```
+
+#### `db.updateNode(partition, nodeMap, [transaction])`
+Updates a node in EliasDB (only update the given values of the node).
+
+Parameter | Description
+-|-
+partition | Partition of the node
+nodeMap | Node object as a map with at least a key and a kind attribute
+transaction | Optional a transaction to group a set of changes
+
+Example:
+```
+db.updateNode("main", {
+  "key" : "foo",
+  "kind" : "bar",
+  "data" : 123,
+})
+```
+
+#### `db.removeNode(partition, nodeKey, nodeKind, [transaction])`
+Removes a node in EliasDB.
+
+Parameter | Description
+-|-
+partition | Partition of the node
+nodeKey | Key attribute of the node to remove
+nodeKind | Kind attribute of the node to remove
+transaction | Optional a transaction to group a set of changes
+
+Example:
+```
+db.removeNode("main", "foo", "bar")
+```
+
+#### `db.fetchNode(partition, nodeKey, nodeKind)`
+Fetches a node in EliasDB.
+
+Parameter | Description
+-|-
+partition | Partition of the node
+nodeKey | Key attribute of the node to fetch
+nodeKind | Kind attribute of the node to fetch
+
+Example:
+```
+db.fetchNode("main", "foo", "bar")
+```
+
+#### `db.storeEdge(partition, edgeMap, [transaction])`
+Inserts or updates an edge in EliasDB.
+
+Parameter | Description
+-|-
+partition | Partition of the edge
+edgeMap | Edge object as a map with at least the main attributes: key, kind, end1cascading, end1key, end1kind, end1role, end2cascading, end2key, end2kind, end2role
+transaction | Optional a transaction to group a set of changes
+
+Example:
+```
+db.storeEdge("main", {
+  "key":           "123",
+  "kind":          "myedges",
+  "end1cascading": true,
+  "end1key":       "foo",
+  "end1kind":      "bar",
+  "end1role":      "role1",
+  "end2cascading": false,
+  "end2key":       "key2",
+  "end2kind":      "kind2",
+  "end2role":      "role2",
+})
+```
+
+#### `db.removeEdge(partition, edgeKey, edgeKind, [transaction])`
+Removes an edge in EliasDB.
+
+Parameter | Description
+-|-
+partition | Partition of the edge
+edgeKey | Key attribute of the edge to remove
+edgeKind | Kind attribute of the edge to remove
+transaction | Optional a transaction to group a set of changes
+
+Example:
+```
+db.removeEdge("main", "123", "myedges")
+```
+
+#### `db.fetchEdge(partition, edgeKey, edgeKind)`
+Fetches an edge in EliasDB.
+
+Parameter | Description
+-|-
+partition | Partition of the edge
+edgeKey | Key attribute of the edge to fetch
+edgeKind | Kind attribute of the edge to fetch
+
+Example:
+```
+db.fetchEdge("main", "123", "myedges")
+```
+
+#### `db.traverse(partition, nodeKey, nodeKind, traversalSpec)`
+Traverses an edge in EliasDB from a given node. Returns a list of nodes which were
+reached and a list of edges which were followed.
+
+Parameter | Description
+-|-
+partition | Partition of the node
+nodeKey | Key attribute of the node to traverse from
+nodeKind | Kind attribute of the node to traverse from
+traversalSpec | Traversal spec
+
+Example:
+```
+[nodes, edges] := db.traverse("main", "foo", "bar", "role1:myedges:role2:kind2")
+```
+
+#### `db.newTrans()`
+Creates a new transaction for EliasDB.
+
+Example:
+```
+trans := db.newTrans()
+```
+
+#### `db.newRollingTrans(n)`
+Creates a new rolling transaction for EliasDB. A rolling transaction commits after n entries.
+
+Parameter | Description
+-|-
+n | Rolling threshold (number of operations before rolling)
+
+Example:
+```
+trans := db.newRollingTrans(5)
+```
+
+#### `db.commit(transaction)`
+Commits an existing transaction for EliasDB.
+
+Parameter | Description
+-|-
+transaction | Transaction to execute
+
+Example:
+```
+db.commit(trans)
+```
+
+#### `db.query(partition, query)`
+Run an EQL query.
+
+Parameter | Description
+-|-
+partition | Partition to query
+query | Query to execute
+
+Example:
+```
+db.commit("main", "get bar")
+```
+
+#### `db.graphQL(partition, query, [variables], [operationName])`
+Run a GraphQL query.
+
+Parameter | Description
+-|-
+partition | Partition to query
+query | Query to execute
+variables | Map of variables for the query
+operationName | Operation to execute (useful if the query defines more than a single operation)
+
+Example:
+```
+db.graphQL("main", "query myquery($x: string) { bar(key:$x) { data }}", {
+  "x": "foo",  
+}, "myquery")
+```
+
+#### `db.raiseGraphEventHandled()`
+When handling a graph event, notify the GraphManager of EliasDB that no further action is necessary. This creates a special error object and should not be used inside a `try` block. When using a `try` block this can be used inside an `except` or `otherwise` block.
+
+Example:
+```
+sink mysink
+  kindmatch [ "db.*.*" ],
+{
+  db.raiseGraphEventHandled()
+}
+```
+
+#### `db.raiseWebEventHandled()`
+When handling a web event, notify the web API of EliasDB that the web request was handled. This creates a special error object and should not be used inside a `try` block. When using a `try` block this can be used inside an `except` or `otherwise` block.
+
+Example:
+```
+sink mysink
+  kindmatch [ "web.*.*" ],
+{
+  db.raiseWebEventHandled({
+    "status" : 200,
+    "headers" : {
+      "Date": "today"
+    },
+    "body" : {
+      "mydata" : [1,2,3]
+    }
+  })
+}
+```

ecal/dbfunc/edge.go

@@ -0,0 +1,254 @@
+/*
+ * EliasDB
+ *
+ * Copyright 2016 Matthias Ladkau. All rights reserved.
+ *
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, You can obtain one at http://mozilla.org/MPL/2.0/.
+ */
+
+/*
+Package dbfunc contains EliasDB specific functions for the event condition action language (ECAL).
+*/
+package dbfunc
+
+import (
+	"fmt"
+
+	"devt.de/krotik/ecal/parser"
+	"devt.de/krotik/eliasdb/graph"
+	"devt.de/krotik/eliasdb/graph/data"
+)
+
+/*
+StoreEdgeFunc inserts or updates an edge in EliasDB.
+*/
+type StoreEdgeFunc struct {
+	GM *graph.Manager
+}
+
+/*
+Run executes the ECAL function.
+*/
+func (f *StoreEdgeFunc) Run(instanceID string, vs parser.Scope, is map[string]interface{}, tid uint64, args []interface{}) (interface{}, error) {
+	var err error
+
+	if arglen := len(args); arglen != 2 && arglen != 3 {
+		err = fmt.Errorf("Function requires 2 or 3 parameters: partition, edge" +
+			" map and optionally a transaction")
+	}
+
+	if err == nil {
+		var trans graph.Trans
+
+		part := fmt.Sprint(args[0])
+		nodeMap, ok := args[1].(map[interface{}]interface{})
+
+		// Check parameters
+
+		if !ok {
+			err = fmt.Errorf("Second parameter must be a map")
+		}
+
+		if err == nil && len(args) > 2 {
+			if trans, ok = args[2].(graph.Trans); !ok {
+				err = fmt.Errorf("Third parameter must be a transaction")
+			}
+		}
+
+		// Build up node to store
+
+		edge := data.NewGraphEdgeFromNode(NewGraphNodeFromECALMap(nodeMap))
+
+		// Store the edge
+
+		if err == nil {
+
+			if trans != nil {
+				err = trans.StoreEdge(part, edge)
+			} else {
+				err = f.GM.StoreEdge(part, edge)
+			}
+		}
+	}
+
+	return nil, err
+}
+
+/*
+DocString returns a descriptive string.
+*/
+func (f *StoreEdgeFunc) DocString() (string, error) {
+	return "Inserts or updates an edge in EliasDB.", nil
+}
+
+/*
+RemoveEdgeFunc removes an edge in EliasDB.
+*/
+type RemoveEdgeFunc struct {
+	GM *graph.Manager
+}
+
+/*
+Run executes the ECAL function.
+*/
+func (f *RemoveEdgeFunc) Run(instanceID string, vs parser.Scope, is map[string]interface{}, tid uint64, args []interface{}) (interface{}, error) {
+	var err error
+
+	if arglen := len(args); arglen != 3 && arglen != 4 {
+		err = fmt.Errorf("Function requires 3 or 4 parameters: partition, edge key," +
+			" edge kind and optionally a transaction")
+	}
+
+	if err == nil {
+		var trans graph.Trans
+
+		part := fmt.Sprint(args[0])
+		key := fmt.Sprint(args[1])
+		kind := fmt.Sprint(args[2])
+
+		// Check parameters
+
+		if len(args) > 3 {
+			var ok bool
+
+			if trans, ok = args[3].(graph.Trans); !ok {
+				err = fmt.Errorf("Fourth parameter must be a transaction")
+			}
+		}
+
+		// Remove the edge
+
+		if err == nil {
+
+			if trans != nil {
+				err = trans.RemoveEdge(part, key, kind)
+			} else {
+				_, err = f.GM.RemoveEdge(part, key, kind)
+			}
+		}
+	}
+
+	return nil, err
+}
+
+/*
+DocString returns a descriptive string.
+*/
+func (f *RemoveEdgeFunc) DocString() (string, error) {
+	return "Removes an edge in EliasDB.", nil
+}
+
+/*
+FetchEdgeFunc fetches an edge in EliasDB.
+*/
+type FetchEdgeFunc struct {
+	GM *graph.Manager
+}
+
+/*
+Run executes the ECAL function.
+*/
+func (f *FetchEdgeFunc) Run(instanceID string, vs parser.Scope, is map[string]interface{}, tid uint64, args []interface{}) (interface{}, error) {
+	var res interface{}
+	var err error
+
+	if arglen := len(args); arglen != 3 {
+		err = fmt.Errorf("Function requires 3 parameters: partition, edge key and" +
+			" edge kind")
+	}
+
+	if err == nil {
+		var node data.Node
+
+		part := fmt.Sprint(args[0])
+		key := fmt.Sprint(args[1])
+		kind := fmt.Sprint(args[2])
+
+		conv := func(m map[string]interface{}) map[interface{}]interface{} {
+			c := make(map[interface{}]interface{})
+			for k, v := range m {
+				c[k] = v
+			}
+			return c
+		}
+
+		// Fetch the node
+
+		if node, err = f.GM.FetchEdge(part, key, kind); node != nil {
+			res = conv(node.Data())
+		}
+	}
+
+	return res, err
+}
+
+/*
+DocString returns a descriptive string.
+*/
+func (f *FetchEdgeFunc) DocString() (string, error) {
+	return "Fetches an edge in EliasDB.", nil
+}
+
+/*
+TraverseFunc traverses an edge in EliasDB.
+*/
+type TraverseFunc struct {
+	GM *graph.Manager
+}
+
+/*
+Run executes the ECAL function.
+*/
+func (f *TraverseFunc) Run(instanceID string, vs parser.Scope, is map[string]interface{}, tid uint64, args []interface{}) (interface{}, error) {
+	var res interface{}
+	var err error
+
+	if arglen := len(args); arglen != 4 {
+		err = fmt.Errorf("Function requires 4 parameters: partition, node key," +
+			" node kind and a traversal spec")
+	}
+
+	if err == nil {
+		var nodes []data.Node
+		var edges []data.Edge
+
+		part := fmt.Sprint(args[0])
+		key := fmt.Sprint(args[1])
+		kind := fmt.Sprint(args[2])
+		spec := fmt.Sprint(args[3])
+
+		conv := func(m map[string]interface{}) map[interface{}]interface{} {
+			c := make(map[interface{}]interface{})
+			for k, v := range m {
+				c[k] = v
+			}
+			return c
+		}
+
+		// Do the traversal
+
+		if nodes, edges, err = f.GM.TraverseMulti(part, key, kind, spec, true); err == nil {
+
+			resNodes := make([]interface{}, len(nodes))
+			for i, n := range nodes {
+				resNodes[i] = conv(n.Data())
+			}
+			resEdges := make([]interface{}, len(edges))
+			for i, e := range edges {
+				resEdges[i] = conv(e.Data())
+			}
+			res = []interface{}{resNodes, resEdges}
+		}
+	}
+
+	return res, err
+}
+
+/*
+DocString returns a descriptive string.
+*/
+func (f *TraverseFunc) DocString() (string, error) {
+	return "Traverses an edge in EliasDB from a given node.", nil
+}

ecal/dbfunc/edge_test.go

@@ -0,0 +1,351 @@
+/*
+ * EliasDB
+ *
+ * Copyright 2016 Matthias Ladkau. All rights reserved.
+ *
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, You can obtain one at http://mozilla.org/MPL/2.0/.
+ */
+
+package dbfunc
+
+import (
+	"fmt"
+	"testing"
+
+	"devt.de/krotik/eliasdb/graph"
+	"devt.de/krotik/eliasdb/graph/data"
+	"devt.de/krotik/eliasdb/graph/graphstorage"
+)
+
+func TestStoreAndRemoveEdge(t *testing.T) {
+	mgs := graphstorage.NewMemoryGraphStorage("mystorage")
+	gm := graph.NewGraphManager(mgs)
+
+	se := &StoreEdgeFunc{gm}
+
+	if _, err := se.DocString(); err != nil {
+		t.Error(err)
+		return
+	}
+
+	if _, err := se.Run("", nil, nil, 0, []interface{}{""}); err == nil ||
+		err.Error() != "Function requires 2 or 3 parameters: partition, edge map and optionally a transaction" {
+		t.Error(err)
+		return
+	}
+
+	if _, err := se.Run("", nil, nil, 0, []interface{}{"main", map[interface{}]interface{}{
+		"key": "foo",
+	}}); err == nil ||
+		err.Error() != "GraphError: Invalid data (Edge is missing a kind value)" {
+		t.Error(err)
+		return
+	}
+
+	if _, err := se.Run("", nil, nil, 0, []interface{}{"main", "x"}); err == nil ||
+		err.Error() != "Second parameter must be a map" {
+		t.Error(err)
+		return
+	}
+
+	if _, err := se.Run("", nil, nil, 0, []interface{}{"main", map[interface{}]interface{}{
+		"key": "foo",
+	}, "x"}); err == nil ||
+		err.Error() != "Third parameter must be a transaction" {
+		t.Error(err)
+		return
+	}
+
+	gm.StoreNode("main", data.NewGraphNodeFromMap(map[string]interface{}{
+		"key":  "a",
+		"kind": "b",
+	}))
+	gm.StoreNode("main", data.NewGraphNodeFromMap(map[string]interface{}{
+		"key":  "c",
+		"kind": "d",
+	}))
+
+	if _, err := se.Run("", nil, nil, 0, []interface{}{"main", map[interface{}]interface{}{
+		"key":           "123",
+		"kind":          "e",
+		"end1cascading": true,
+		"end1key":       "a",
+		"end1kind":      "b",
+		"end1role":      "role1",
+		"end2cascading": false,
+		"end2key":       "c",
+		"end2kind":      "d",
+		"end2role":      "role2",
+	}}); err != nil {
+		t.Error(err)
+		return
+	}
+
+	_, err := se.Run("", nil, nil, 0, []interface{}{"main", map[interface{}]interface{}{
+		"key":           "123",
+		"kind":          "e",
+		"end1cascading": true,
+		"end1key":       "a",
+		"end1kind":      "b1",
+		"end1role":      "role1",
+		"end2cascading": false,
+		"end2key":       "c",
+		"end2kind":      "d",
+		"end2role":      "role2",
+	}})
+
+	if err == nil || err.Error() != "GraphError: Invalid data (Can't store edge to non-existing node kind: b1)" {
+		t.Error(err)
+		return
+	}
+
+	fe := &FetchEdgeFunc{gm}
+
+	if _, err := fe.DocString(); err != nil {
+		t.Error(err)
+		return
+	}
+
+	if _, err := fe.Run("", nil, nil, 0, []interface{}{""}); err == nil ||
+		err.Error() != "Function requires 3 parameters: partition, edge key and edge kind" {
+		t.Error(err)
+		return
+	}
+
+	if _, err := fe.Run("", nil, nil, 0, []interface{}{"mai n", "123", "e"}); err == nil ||
+		err.Error() != "GraphError: Invalid data (Partition name mai n is not alphanumeric - can only contain [a-zA-Z0-9_])" {
+		t.Error(err)
+		return
+	}
+
+	res, err := fe.Run("", nil, nil, 0, []interface{}{"main", "123", "e"})
+
+	if fmt.Sprint(data.NewGraphEdgeFromNode(NewGraphNodeFromECALMap(res.(map[interface{}]interface{})))) != `
+GraphEdge:
+              key : 123
+             kind : e
+    end1cascading : true
+          end1key : a
+         end1kind : b
+         end1role : role1
+    end2cascading : false
+          end2key : c
+         end2kind : d
+         end2role : role2
+`[1:] || err != nil {
+		t.Error("Unexpected result:", fmt.Sprint(data.NewGraphEdgeFromNode(NewGraphNodeFromECALMap(res.(map[interface{}]interface{})))), err)
+		return
+	}
+
+	tr := &TraverseFunc{gm}
+
+	if _, err := tr.DocString(); err != nil {
+		t.Error(err)
+		return
+	}
+
+	if _, err := tr.Run("", nil, nil, 0, []interface{}{""}); err == nil ||
+		err.Error() != "Function requires 4 parameters: partition, node key, node kind and a traversal spec" {
+		t.Error(err)
+		return
+	}
+
+	if _, err := tr.Run("", nil, nil, 0, []interface{}{"main", "c", "d", "::"}); err == nil ||
+		err.Error() != "GraphError: Invalid data (Invalid spec: ::)" {
+		t.Error(err)
+		return
+	}
+
+	res, err = tr.Run("", nil, nil, 0, []interface{}{"main", "c", "d", ":::"})
+	if err != nil {
+		t.Error(err)
+		return
+	}
+
+	if fmt.Sprint(data.NewGraphEdgeFromNode(NewGraphNodeFromECALMap(res.([]interface{})[1].([]interface{})[0].(map[interface{}]interface{})))) != `
+GraphEdge:
+              key : 123
+             kind : e
+    end1cascading : false
+          end1key : c
+         end1kind : d
+         end1role : role2
+    end2cascading : true
+          end2key : a
+         end2kind : b
+         end2role : role1
+`[1:] || err != nil {
+		t.Error("Unexpected result:", fmt.Sprint(data.NewGraphEdgeFromNode(NewGraphNodeFromECALMap(res.([]interface{})[1].([]interface{})[0].(map[interface{}]interface{})))), err)
+		return
+	}
+
+	if fmt.Sprint(NewGraphNodeFromECALMap(res.([]interface{})[0].([]interface{})[0].(map[interface{}]interface{}))) != `
+GraphNode:
+     key : a
+    kind : b
+`[1:] || err != nil {
+		t.Error("Unexpected result:", fmt.Sprint(NewGraphNodeFromECALMap(res.([]interface{})[0].([]interface{})[0].(map[interface{}]interface{}))), err)
+		return
+	}
+
+	re := &RemoveEdgeFunc{gm}
+
+	if _, err := re.DocString(); err != nil {
+		t.Error(err)
+		return
+	}
+
+	if _, err := re.Run("", nil, nil, 0, []interface{}{""}); err == nil ||
+		err.Error() != "Function requires 3 or 4 parameters: partition, edge key, edge kind and optionally a transaction" {
+		t.Error(err)
+		return
+	}
+
+	if _, err := re.Run("", nil, nil, 0, []interface{}{"mai n", "123", "e"}); err == nil ||
+		err.Error() != "GraphError: Invalid data (Partition name mai n is not alphanumeric - can only contain [a-zA-Z0-9_])" {
+		t.Error(err)
+		return
+	}
+
+	if _, err := re.Run("", nil, nil, 0, []interface{}{"mai n", "123", "e", "bla"}); err == nil ||
+		err.Error() != "Fourth parameter must be a transaction" {
+		t.Error(err)
+		return
+	}
+
+	if _, err := re.Run("", nil, nil, 0, []interface{}{"main", "123", "e"}); err != nil {
+		t.Error(err)
+		return
+	}
+
+	res, err = fe.Run("", nil, nil, 0, []interface{}{"main", "123", "e"})
+
+	if res != nil || err != nil {
+		t.Error("Unexpected result:", res, err)
+		return
+	}
+}
+
+func TestStoreEdgeTrans(t *testing.T) {
+	mgs := graphstorage.NewMemoryGraphStorage("mystorage")
+	gm := graph.NewGraphManager(mgs)
+
+	sn := &StoreNodeFunc{gm}
+	se := &StoreEdgeFunc{gm}
+	tc := &CommitTransFunc{gm}
+
+	if _, err := tc.DocString(); err != nil {
+		t.Error(err)
+		return
+	}
+
+	trans := graph.NewGraphTrans(gm)
+
+	if _, err := sn.Run("", nil, nil, 0, []interface{}{"main", map[interface{}]interface{}{
+		"key":  "a",
+		"kind": "b",
+	}, trans}); err != nil {
+		t.Error(err)
+		return
+	}
+
+	if _, err := sn.Run("", nil, nil, 0, []interface{}{"main", map[interface{}]interface{}{
+		"key":  "c",
+		"kind": "d",
+	}, trans}); err != nil {
+		t.Error(err)
+		return
+	}
+
+	_, err := se.Run("", nil, nil, 0, []interface{}{"main", map[interface{}]interface{}{
+		"key":           "123",
+		"kind":          "e",
+		"end1cascading": true,
+		"end1key":       "a",
+		"end1kind":      "b",
+		"end1role":      "role1",
+		"end2cascading": false,
+		"end2key":       "c",
+		"end2kind":      "d",
+		"end2role":      "role2",
+	}, trans})
+
+	if err != nil {
+		t.Error(err)
+		return
+	}
+
+	if res := fmt.Sprint(trans.Counts()); res != "2 1 0 0" {
+		t.Error("Unexpected result:", res)
+		return
+	}
+	if _, err := tc.Run("", nil, nil, 0, []interface{}{trans}); err != nil {
+		t.Error(err)
+		return
+	}
+
+	// Check that the nodes have been committed
+
+	if res := fmt.Sprint(trans.Counts()); res != "0 0 0 0" {
+		t.Error("Unexpected result:", res)
+		return
+	}
+
+	if res := gm.EdgeCount("e"); res != 1 {
+		t.Error("Unexpected result:", res)
+		return
+	}
+
+	se.Run("", nil, nil, 0, []interface{}{"main", map[interface{}]interface{}{
+		"key":           "123",
+		"kind":          "e",
+		"end1cascading": true,
+		"end1key":       "a",
+		"end1kind":      "b",
+		"end1role":      "role1",
+		"end2cascading": false,
+		"end2key":       "c1",
+		"end2kind":      "d",
+		"end2role":      "role2",
+	}, trans})
+
+	if _, err := tc.Run("", nil, nil, 0, []interface{}{trans}); err == nil || err.Error() !=
+		"GraphError: Invalid data (Can't find edge endpoint: c1 (d))" {
+		t.Error(err)
+		return
+	}
+
+	re := &RemoveEdgeFunc{}
+
+	if _, err := re.DocString(); err != nil {
+		t.Error(err)
+		return
+	}
+
+	if _, err := re.Run("", nil, nil, 0, []interface{}{"main", "123", "e", trans}); err != nil {
+		t.Error(err)
+		return
+	}
+
+	if res := fmt.Sprint(trans.Counts()); res != "0 0 0 1" {
+		t.Error("Unexpected result:", res)
+		return
+	}
+
+	if _, err := tc.Run("", nil, nil, 0, []interface{}{trans}); err != nil {
+		t.Error(err)
+		return
+	}
+
+	if res := fmt.Sprint(trans.Counts()); res != "0 0 0 0" {
+		t.Error("Unexpected result:", res)
+		return
+	}
+
+	if res := gm.EdgeCount("e"); res != 0 {
+		t.Error("Unexpected result:", res)
+		return
+	}
+}

ecal/dbfunc/eql.go

@@ -0,0 +1,77 @@
+/*
+ * EliasDB
+ *
+ * Copyright 2016 Matthias Ladkau. All rights reserved.
+ *
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, You can obtain one at http://mozilla.org/MPL/2.0/.
+ */
+
+package dbfunc
+
+import (
+	"fmt"
+
+	"devt.de/krotik/ecal/parser"
+	"devt.de/krotik/eliasdb/eql"
+	"devt.de/krotik/eliasdb/graph"
+)
+
+/*
+QueryFunc runs an EQL query.
+*/
+type QueryFunc struct {
+	GM *graph.Manager
+}
+
+/*
+Run executes the ECAL function.
+*/
+func (f *QueryFunc) Run(instanceID string, vs parser.Scope, is map[string]interface{}, tid uint64, args []interface{}) (interface{}, error) {
+	var err error
+	var cols, rows []interface{}
+
+	if arglen := len(args); arglen != 2 {
+		err = fmt.Errorf("Function requires 2 parameters: partition and a query string")
+	}
+
+	if err == nil {
+		var res eql.SearchResult
+
+		part := fmt.Sprint(args[0])
+		query := fmt.Sprint(args[1])
+
+		res, err = eql.RunQuery("db.query", part, query, f.GM)
+
+		if err != nil {
+			return nil, err
+		}
+
+		// Convert result to rumble data structure
+
+		labels := res.Header().Labels()
+		cols = make([]interface{}, len(labels))
+		for i, v := range labels {
+			cols[i] = v
+		}
+
+		rrows := res.Rows()
+		rows = make([]interface{}, len(rrows))
+		for i, v := range rrows {
+			rows[i] = v
+		}
+	}
+
+	return map[interface{}]interface{}{
+		"cols": cols,
+		"rows": rows,
+	}, err
+}
+
+/*
+DocString returns a descriptive string.
+*/
+func (f *QueryFunc) DocString() (string, error) {
+	return "Run an EQL query.", nil
+}

ecal/dbfunc/eql_test.go

@@ -0,0 +1,71 @@
+/*
+ * EliasDB
+ *
+ * Copyright 2016 Matthias Ladkau. All rights reserved.
+ *
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, You can obtain one at http://mozilla.org/MPL/2.0/.
+ */
+
+package dbfunc
+
+import (
+	"fmt"
+	"testing"
+
+	"devt.de/krotik/eliasdb/graph"
+	"devt.de/krotik/eliasdb/graph/data"
+	"devt.de/krotik/eliasdb/graph/graphstorage"
+)
+
+func TestQuery(t *testing.T) {
+	mgs := graphstorage.NewMemoryGraphStorage("mystorage")
+	gm := graph.NewGraphManager(mgs)
+
+	gm.StoreNode("main", data.NewGraphNodeFromMap(map[string]interface{}{
+		"key":  "a",
+		"kind": "b",
+	}))
+	gm.StoreNode("main", data.NewGraphNodeFromMap(map[string]interface{}{
+		"key":  "c",
+		"kind": "d",
+	}))
+
+	q := &QueryFunc{gm}
+
+	if _, err := q.DocString(); err != nil {
+		t.Error(err)
+		return
+	}
+
+	if _, err := q.Run("", nil, nil, 0, []interface{}{""}); err == nil ||
+		err.Error() != "Function requires 2 parameters: partition and a query string" {
+		t.Error(err)
+		return
+	}
+
+	res, err := q.Run("", nil, nil, 0, []interface{}{"main", "get b"})
+
+	if err != nil {
+		t.Error(err)
+		return
+	}
+
+	if res := res.(map[interface{}]interface{})["rows"]; fmt.Sprint(res) != "[[a]]" {
+		t.Error("Unexpected result:", res)
+		return
+	}
+
+	if res := res.(map[interface{}]interface{})["cols"]; fmt.Sprint(res) != "[B Key]" {
+		t.Error("Unexpected result:", res)
+		return
+	}
+
+	_, err = q.Run("", nil, nil, 0, []interface{}{"main", "got b"})
+
+	if err == nil || err.Error() != "EQL error in db.query: Invalid construct (Unknown query type: got) (Line:1 Pos:1)" {
+		t.Error(err)
+		return
+	}
+}

ecal/dbfunc/graphql.go

@@ -0,0 +1,85 @@
+/*
+ * EliasDB
+ *
+ * Copyright 2016 Matthias Ladkau. All rights reserved.
+ *
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, You can obtain one at http://mozilla.org/MPL/2.0/.
+ */
+
+package dbfunc
+
+import (
+	"fmt"
+
+	"devt.de/krotik/ecal/parser"
+	"devt.de/krotik/ecal/scope"
+	"devt.de/krotik/eliasdb/graph"
+	"devt.de/krotik/eliasdb/graphql"
+)
+
+/*
+GraphQLFunc runs a GraphQL query.
+*/
+type GraphQLFunc struct {
+	GM *graph.Manager
+}
+
+/*
+Run executes the ECAL function.
+*/
+func (f *GraphQLFunc) Run(instanceID string, vs parser.Scope, is map[string]interface{}, tid uint64, args []interface{}) (interface{}, error) {
+	var err error
+	var ret interface{}
+
+	if arglen := len(args); arglen < 2 {
+		err = fmt.Errorf("Function requires at least 2 parameters: partition and query with optionally a map of variables and an operation name")
+	}
+
+	if err == nil {
+		var res, varMap map[string]interface{}
+
+		part := fmt.Sprint(args[0])
+		query := fmt.Sprint(args[1])
+		opname := ""
+
+		if err == nil && len(args) > 2 {
+			varECALMap, ok := args[2].(map[interface{}]interface{})
+
+			if !ok {
+				err = fmt.Errorf("Third parameter must be a map")
+			} else {
+				varMap = make(map[string]interface{})
+				for k, v := range varECALMap {
+					varMap[fmt.Sprint(k)] = v
+				}
+			}
+		}
+
+		if err == nil && len(args) > 3 {
+			opname = fmt.Sprint(args[3])
+		}
+
+		if err == nil {
+			res, err = graphql.RunQuery("db.query", part, map[string]interface{}{
+				"operationName": opname,
+				"query":         query,
+				"variables":     varMap,
+			}, f.GM, nil, false)
+
+			if err == nil {
+				ret = scope.ConvertJSONToECALObject(res)
+			}
+		}
+	}
+
+	return ret, err
+}
+
+/*
+DocString returns a descriptive string.
+*/
+func (f *GraphQLFunc) DocString() (string, error) {
+	return "Run a GraphQL query.", nil
+}

ecal/dbfunc/graphql_test.go

@@ -0,0 +1,77 @@
+/*
+ * EliasDB
+ *
+ * Copyright 2016 Matthias Ladkau. All rights reserved.
+ *
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, You can obtain one at http://mozilla.org/MPL/2.0/.
+ */
+
+package dbfunc
+
+import (
+	"fmt"
+	"testing"
+
+	"devt.de/krotik/eliasdb/graph"
+	"devt.de/krotik/eliasdb/graph/data"
+	"devt.de/krotik/eliasdb/graph/graphstorage"
+)
+
+func TestGraphQL(t *testing.T) {
+	mgs := graphstorage.NewMemoryGraphStorage("mystorage")
+	gm := graph.NewGraphManager(mgs)
+
+	gm.StoreNode("main", data.NewGraphNodeFromMap(map[string]interface{}{
+		"key":  "a",
+		"kind": "b",
+		"foo":  "bar1",
+	}))
+	gm.StoreNode("main", data.NewGraphNodeFromMap(map[string]interface{}{
+		"key":  "c",
+		"kind": "b",
+		"foo":  "bar2",
+	}))
+
+	q := &GraphQLFunc{gm}
+
+	if _, err := q.DocString(); err != nil {
+		t.Error(err)
+		return
+	}
+
+	if _, err := q.Run("", nil, nil, 0, []interface{}{""}); err == nil ||
+		err.Error() != "Function requires at least 2 parameters: partition and query with optionally a map of variables and an operation name" {
+		t.Error(err)
+		return
+	}
+
+	if _, err := q.Run("", nil, nil, 0, []interface{}{"", "", ""}); err == nil ||
+		err.Error() != "Third parameter must be a map" {
+		t.Error(err)
+		return
+	}
+
+	res, err := q.Run("", nil, nil, 0, []interface{}{"main",
+		`query foo($x: string) { b(key:$x) { foo }}`, map[interface{}]interface{}{
+			"x": "c",
+		}, "foo"})
+
+	if err != nil {
+		t.Error(err)
+		return
+	}
+
+	if fmt.Sprint(res) != "map[data:map[b:[map[foo:bar2]]]]" {
+		t.Error("Unexpected result:", res)
+		return
+	}
+
+	_, err = q.Run("", nil, nil, 0, []interface{}{"main", "aaaaa"})
+
+	if err == nil || err.Error() != "Fatal GraphQL operation error in db.query: Missing operation (No executable expression found) (Line:1 Pos:1)" {
+		t.Error(err)
+		return
+	}
+}

ecal/dbfunc/node.go

@@ -0,0 +1,261 @@
+/*
+ * EliasDB
+ *
+ * Copyright 2016 Matthias Ladkau. All rights reserved.
+ *
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, You can obtain one at http://mozilla.org/MPL/2.0/.
+ */
+
+package dbfunc
+
+import (
+	"fmt"
+
+	"devt.de/krotik/ecal/parser"
+	"devt.de/krotik/eliasdb/graph"
+	"devt.de/krotik/eliasdb/graph/data"
+)
+
+/*
+StoreNodeFunc inserts a node in EliasDB.
+*/
+type StoreNodeFunc struct {
+	GM *graph.Manager
+}
+
+/*
+Run executes the ECAL function.
+*/
+func (f *StoreNodeFunc) Run(instanceID string, vs parser.Scope, is map[string]interface{}, tid uint64, args []interface{}) (interface{}, error) {
+	var err error
+
+	if arglen := len(args); arglen != 2 && arglen != 3 {
+		err = fmt.Errorf("Function requires 2 or 3 parameters: partition, node" +
+			" map and optionally a transaction")
+	}
+
+	if err == nil {
+		var trans graph.Trans
+
+		part := fmt.Sprint(args[0])
+		nodeMap, ok := args[1].(map[interface{}]interface{})
+
+		// Check parameters
+
+		if !ok {
+			err = fmt.Errorf("Second parameter must be a map")
+		}
+
+		if err == nil && len(args) > 2 {
+			if trans, ok = args[2].(graph.Trans); !ok {
+				err = fmt.Errorf("Third parameter must be a transaction")
+			}
+		}
+
+		// Store the node
+
+		if err == nil {
+			node := NewGraphNodeFromECALMap(nodeMap)
+
+			if trans != nil {
+				err = trans.StoreNode(part, node)
+			} else {
+				err = f.GM.StoreNode(part, node)
+			}
+		}
+	}
+
+	return nil, err
+}
+
+/*
+DocString returns a descriptive string.
+*/
+func (f *StoreNodeFunc) DocString() (string, error) {
+	return "Inserts a node in EliasDB.", nil
+}
+
+/*
+UpdateNodeFunc updates a node in EliasDB (only update the given values of the node).
+*/
+type UpdateNodeFunc struct {
+	GM *graph.Manager
+}
+
+/*
+Run executes the ECAL function.
+*/
+func (f *UpdateNodeFunc) Run(instanceID string, vs parser.Scope, is map[string]interface{}, tid uint64, args []interface{}) (interface{}, error) {
+	var err error
+
+	if arglen := len(args); arglen != 2 && arglen != 3 {
+		err = fmt.Errorf("Function requires 2 or 3 parameters: partition, node" +
+			" map and optionally a transaction")
+	}
+
+	if err == nil {
+		var trans graph.Trans
+
+		part := fmt.Sprint(args[0])
+		nodeMap, ok := args[1].(map[interface{}]interface{})
+
+		// Check parameters
+
+		if !ok {
+			err = fmt.Errorf("Second parameter must be a map")
+		}
+
+		if err == nil && len(args) > 2 {
+			if trans, ok = args[2].(graph.Trans); !ok {
+				err = fmt.Errorf("Third parameter must be a transaction")
+			}
+		}
+
+		// Store the node
+
+		if err == nil {
+			node := NewGraphNodeFromECALMap(nodeMap)
+
+			if trans != nil {
+				err = trans.UpdateNode(part, node)
+			} else {
+				err = f.GM.UpdateNode(part, node)
+			}
+		}
+	}
+
+	return nil, err
+}
+
+/*
+DocString returns a descriptive string.
+*/
+func (f *UpdateNodeFunc) DocString() (string, error) {
+	return "Updates a node in EliasDB (only update the given values of the node).", nil
+}
+
+/*
+RemoveNodeFunc removes a node in EliasDB.
+*/
+type RemoveNodeFunc struct {
+	GM *graph.Manager
+}
+
+/*
+Run executes the ECAL function.
+*/
+func (f *RemoveNodeFunc) Run(instanceID string, vs parser.Scope, is map[string]interface{}, tid uint64, args []interface{}) (interface{}, error) {
+	var err error
+
+	if arglen := len(args); arglen != 3 && arglen != 4 {
+		err = fmt.Errorf("Function requires 3 or 4 parameters: partition, node key" +
+			" node kind and optionally a transaction")
+	}
+
+	if err == nil {
+		var trans graph.Trans
+
+		part := fmt.Sprint(args[0])
+		key := fmt.Sprint(args[1])
+		kind := fmt.Sprint(args[2])
+
+		// Check parameters
+
+		if len(args) > 3 {
+			var ok bool
+
+			if trans, ok = args[3].(graph.Trans); !ok {
+				err = fmt.Errorf("Fourth parameter must be a transaction")
+			}
+		}
+
+		// Remove the node
+
+		if err == nil {
+
+			if trans != nil {
+				err = trans.RemoveNode(part, key, kind)
+			} else {
+				_, err = f.GM.RemoveNode(part, key, kind)
+			}
+		}
+	}
+
+	return nil, err
+}
+
+/*
+DocString returns a descriptive string.
+*/
+func (f *RemoveNodeFunc) DocString() (string, error) {
+	return "Removes a node in EliasDB.", nil
+}
+
+/*
+FetchNodeFunc fetches a node in EliasDB.
+*/
+type FetchNodeFunc struct {
+	GM *graph.Manager
+}
+
+/*
+Run executes the ECAL function.
+*/
+func (f *FetchNodeFunc) Run(instanceID string, vs parser.Scope, is map[string]interface{}, tid uint64, args []interface{}) (interface{}, error) {
+	var res interface{}
+	var err error
+
+	if arglen := len(args); arglen != 3 {
+		err = fmt.Errorf("Function requires 3 parameters: partition, node key" +
+			" node kind")
+	}
+
+	if err == nil {
+		var node data.Node
+
+		part := fmt.Sprint(args[0])
+		key := fmt.Sprint(args[1])
+		kind := fmt.Sprint(args[2])
+
+		conv := func(m map[string]interface{}) map[interface{}]interface{} {
+			c := make(map[interface{}]interface{})
+			for k, v := range m {
+				c[k] = v
+			}
+			return c
+		}
+
+		// Fetch the node
+
+		if node, err = f.GM.FetchNode(part, key, kind); node != nil {
+			res = conv(node.Data())
+		}
+	}
+
+	return res, err
+}
+
+/*
+DocString returns a descriptive string.
+*/
+func (f *FetchNodeFunc) DocString() (string, error) {
+	return "Fetches a node in EliasDB.", nil
+}
+
+// Helper functions
+// ================
+
+/*
+NewGraphNodeFromECALMap creates a new Node instance from a given map.
+*/
+func NewGraphNodeFromECALMap(d map[interface{}]interface{}) data.Node {
+	node := data.NewGraphNode()
+
+	for k, v := range d {
+		node.SetAttr(fmt.Sprint(k), v)
+	}
+
+	return node
+}

ecal/dbfunc/node_test.go

@@ -0,0 +1,375 @@
+/*
+ * EliasDB
+ *
+ * Copyright 2016 Matthias Ladkau. All rights reserved.
+ *
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, You can obtain one at http://mozilla.org/MPL/2.0/.
+ */
+
+package dbfunc
+
+import (
+	"fmt"
+	"testing"
+
+	"devt.de/krotik/eliasdb/graph"
+	"devt.de/krotik/eliasdb/graph/graphstorage"
+)
+
+func TestStoreAndRemoveNode(t *testing.T) {
+
+	mgs := graphstorage.NewMemoryGraphStorage("mystorage")
+	gm := graph.NewGraphManager(mgs)
+
+	sn := &StoreNodeFunc{gm}
+
+	if _, err := sn.DocString(); err != nil {
+		t.Error(err)
+		return
+	}
+
+	if _, err := sn.Run("", nil, nil, 0, []interface{}{""}); err == nil ||
+		err.Error() != "Function requires 2 or 3 parameters: partition, node map and optionally a transaction" {
+		t.Error(err)
+		return
+	}
+
+	if _, err := sn.Run("", nil, nil, 0, []interface{}{"", "bla"}); err == nil ||
+		err.Error() != "Second parameter must be a map" {
+		t.Error(err)
+		return
+	}
+
+	if _, err := sn.Run("", nil, nil, 0, []interface{}{"main", map[interface{}]interface{}{}, "bla"}); err == nil ||
+		err.Error() != "Third parameter must be a transaction" {
+		t.Error(err)
+		return
+	}
+
+	if _, err := sn.Run("", nil, nil, 0, []interface{}{"main", map[interface{}]interface{}{
+		"key": "foo",
+	}}); err == nil ||
+		err.Error() != "GraphError: Invalid data (Node is missing a kind value)" {
+		t.Error(err)
+		return
+	}
+
+	if _, err := sn.Run("", nil, nil, 0, []interface{}{"main", map[interface{}]interface{}{
+		"key":   "foo",
+		"kind":  "bar",
+		"data":  "123",
+		"data2": "1234",
+	}}); err != nil {
+		t.Error(err)
+		return
+	}
+
+	if res := gm.NodeCount("bar"); res != 1 {
+		t.Error("Unexpected result:", res)
+		return
+	}
+
+	un := &UpdateNodeFunc{gm}
+
+	if _, err := un.DocString(); err != nil {
+		t.Error(err)
+		return
+	}
+
+	if _, err := un.Run("", nil, nil, 0, []interface{}{""}); err == nil ||
+		err.Error() != "Function requires 2 or 3 parameters: partition, node map and optionally a transaction" {
+		t.Error(err)
+		return
+	}
+
+	if _, err := un.Run("", nil, nil, 0, []interface{}{"", "bla"}); err == nil ||
+		err.Error() != "Second parameter must be a map" {
+		t.Error(err)
+		return
+	}
+
+	if _, err := un.Run("", nil, nil, 0, []interface{}{"main", map[interface{}]interface{}{}, "bla"}); err == nil ||
+		err.Error() != "Third parameter must be a transaction" {
+		t.Error(err)
+		return
+	}
+
+	if _, err := un.Run("", nil, nil, 0, []interface{}{"main", map[interface{}]interface{}{
+		"key": "foo",
+	}}); err == nil ||
+		err.Error() != "GraphError: Invalid data (Node is missing a kind value)" {
+		t.Error(err)
+		return
+	}
+
+	if _, err := un.Run("", nil, nil, 0, []interface{}{"main", map[interface{}]interface{}{
+		"key":  "foo",
+		"kind": "bar",
+		"data": "1234",
+	}}); err != nil {
+		t.Error(err)
+		return
+	}
+
+	if res := gm.NodeCount("bar"); res != 1 {
+		t.Error("Unexpected result:", res)
+		return
+	}
+
+	fn := &FetchNodeFunc{gm}
+
+	if _, err := fn.DocString(); err != nil {
+		t.Error(err)
+		return
+	}
+
+	if _, err := fn.Run("", nil, nil, 0, []interface{}{""}); err == nil ||
+		err.Error() != "Function requires 3 parameters: partition, node key node kind" {
+		t.Error(err)
+		return
+	}
+
+	if _, err := fn.Run("", nil, nil, 0, []interface{}{"main", "foo", "ba r"}); err == nil ||
+		err.Error() != "GraphError: Invalid data (Node kind ba r is not alphanumeric - can only contain [a-zA-Z0-9_])" {
+		t.Error(err)
+		return
+	}
+
+	res, err := fn.Run("", nil, nil, 0, []interface{}{"main", "foo", "bar"})
+
+	if fmt.Sprint(NewGraphNodeFromECALMap(res.(map[interface{}]interface{}))) != `
+GraphNode:
+      key : foo
+     kind : bar
+     data : 1234
+    data2 : 1234
+`[1:] || err != nil {
+		t.Error("Unexpected result:\n", res, err)
+		return
+	}
+
+	rn := &RemoveNodeFunc{gm}
+
+	if _, err := rn.DocString(); err != nil {
+		t.Error(err)
+		return
+	}
+
+	if _, err := rn.Run("", nil, nil, 0, []interface{}{""}); err == nil ||
+		err.Error() != "Function requires 3 or 4 parameters: partition, node key node kind and optionally a transaction" {
+		t.Error(err)
+		return
+	}
+
+	if _, err := rn.Run("", nil, nil, 0, []interface{}{"mai n", "foo", "bar"}); err == nil ||
+		err.Error() != "GraphError: Invalid data (Partition name mai n is not alphanumeric - can only contain [a-zA-Z0-9_])" {
+		t.Error(err)
+		return
+	}
+
+	_, err = rn.Run("", nil, nil, 0, []interface{}{"main", "foo", "bar"})
+	if err != nil {
+		t.Error(err)
+		return
+	}
+
+	res, err = fn.Run("", nil, nil, 0, []interface{}{"main", "foo", "bar"})
+	if res != nil || err != nil {
+		t.Error("Unexpected result:", res, err)
+		return
+	}
+}
+
+func TestStoreNodeTrans(t *testing.T) {
+	mgs := graphstorage.NewMemoryGraphStorage("mystorage")
+	gm := graph.NewGraphManager(mgs)
+
+	tn := &NewTransFunc{gm}
+
+	if _, err := tn.DocString(); err != nil {
+		t.Error(err)
+		return
+	}
+
+	tn2 := &NewRollingTransFunc{gm}
+
+	if _, err := tn2.DocString(); err != nil {
+		t.Error(err)
+		return
+	}
+
+	tc := &CommitTransFunc{gm}
+
+	if _, err := tc.DocString(); err != nil {
+		t.Error(err)
+		return
+	}
+
+	if _, err := tn.Run("", nil, nil, 0, []interface{}{""}); err == nil ||
+		err.Error() != "Function does not require any parameters" {
+		t.Error(err)
+		return
+	}
+	if _, err := tn2.Run("", nil, nil, 0, []interface{}{"", ""}); err == nil ||
+		err.Error() != "Function requires the rolling threshold (number of operations before rolling)" {
+		t.Error(err)
+		return
+	}
+	if _, err := tc.Run("", nil, nil, 0, []interface{}{"", ""}); err == nil ||
+		err.Error() != "Function requires the transaction to commit as parameter" {
+		t.Error(err)
+		return
+	}
+
+	if _, err := tc.Run("", nil, nil, 0, []interface{}{""}); err == nil ||
+		err.Error() != "Parameter must be a transaction" {
+		t.Error(err)
+		return
+	}
+
+	trans, err := tn.Run("", nil, nil, 0, []interface{}{})
+	if err != nil {
+		t.Error(err)
+		return
+	}
+
+	_, err = tn2.Run("", nil, nil, 0, []interface{}{"foo"})
+	if err == nil || err.Error() != "Rolling threshold must be a number not: foo" {
+		t.Error(err)
+		return
+	}
+
+	_, err = tn2.Run("", nil, nil, 0, []interface{}{1})
+	if err != nil {
+		t.Error(err)
+		return
+	}
+
+	sn := &StoreNodeFunc{gm}
+
+	if _, err := sn.Run("", nil, nil, 0, []interface{}{"main", map[interface{}]interface{}{
+		"key":  "foo1",
+		"kind": "bar",
+	}, trans}); err != nil {
+		t.Error(err)
+		return
+	}
+
+	if _, err := sn.Run("", nil, nil, 0, []interface{}{"main", map[interface{}]interface{}{
+		"key":  "foo2",
+		"kind": "bar",
+	}, trans}); err != nil {
+		t.Error(err)
+		return
+	}
+
+	un := &UpdateNodeFunc{gm}
+
+	if _, err := un.Run("", nil, nil, 0, []interface{}{"main", map[interface{}]interface{}{
+		"key":  "foo3",
+		"kind": "bar",
+	}, trans}); err != nil {
+		t.Error(err)
+		return
+	}
+
+	// Check that the nodes are in the transaction
+
+	if res := fmt.Sprint(trans.(graph.Trans).Counts()); res != "3 0 0 0" {
+		t.Error("Unexpected result:", res)
+		return
+	}
+
+	if res := gm.NodeCount("bar"); res != 0 {
+		t.Error("Unexpected result:", res)
+		return
+	}
+
+	// Commit the nodes
+
+	if _, err := tc.Run("", nil, nil, 0, []interface{}{"main", map[interface{}]interface{}{
+		"key":  "foo3",
+		"kind": "bar",
+	}, trans}); err == nil || err.Error() != "Function requires the transaction to commit as parameter" {
+		t.Error(err)
+		return
+	}
+
+	if _, err := tc.Run("", nil, nil, 0, []interface{}{trans}); err != nil {
+		t.Error(err)
+		return
+	}
+
+	// Check that the nodes have been committed
+
+	if res := fmt.Sprint(trans.(graph.Trans).Counts()); res != "0 0 0 0" {
+		t.Error("Unexpected result:", res)
+		return
+	}
+
+	if res := gm.NodeCount("bar"); res != 3 {
+		t.Error("Unexpected result:", res)
+		return
+	}
+
+	// Remove the nodes
+
+	rn := &RemoveNodeFunc{gm}
+
+	_, err = rn.Run("", nil, nil, 0, []interface{}{"main", "foo1", "bar", nil})
+	if err == nil || err.Error() != "Fourth parameter must be a transaction" {
+		t.Error(err)
+		return
+	}
+
+	_, err = rn.Run("", nil, nil, 0, []interface{}{"main", "foo1", "bar", trans})
+	if err != nil {
+		t.Error(err)
+		return
+	}
+
+	_, err = rn.Run("", nil, nil, 0, []interface{}{"main", "foo2", "bar", trans})
+	if err != nil {
+		t.Error(err)
+		return
+	}
+
+	_, err = rn.Run("", nil, nil, 0, []interface{}{"main", "foo3", "bar", trans})
+	if err != nil {
+		t.Error(err)
+		return
+	}
+
+	// Check that the nodes are in the transaction
+
+	if res := fmt.Sprint(trans.(graph.Trans).Counts()); res != "0 0 3 0" {
+		t.Error("Unexpected result:", res)
+		return
+	}
+
+	if res := gm.NodeCount("bar"); res != 3 {
+		t.Error("Unexpected result:", res)
+		return
+	}
+
+	// Commit the nodes
+
+	if _, err := tc.Run("", nil, nil, 0, []interface{}{trans}); err != nil {
+		t.Error(err)
+		return
+	}
+
+	// Check that the nodes have been committed
+
+	if res := fmt.Sprint(trans.(graph.Trans).Counts()); res != "0 0 0 0" {
+		t.Error("Unexpected result:", res)
+		return
+	}
+
+	if res := gm.NodeCount("bar"); res != 0 {
+		t.Error("Unexpected result:", res)
+		return
+	}
+}

ecal/dbfunc/trans.go

@@ -0,0 +1,127 @@
+/*
+ * EliasDB
+ *
+ * Copyright 2016 Matthias Ladkau. All rights reserved.
+ *
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, You can obtain one at http://mozilla.org/MPL/2.0/.
+ */
+
+package dbfunc
+
+import (
+	"fmt"
+	"strconv"
+
+	"devt.de/krotik/ecal/parser"
+	"devt.de/krotik/eliasdb/graph"
+)
+
+/*
+NewTransFunc creates a new transaction for EliasDB.
+*/
+type NewTransFunc struct {
+	GM *graph.Manager
+}
+
+/*
+Run executes the ECAL function.
+*/
+func (f *NewTransFunc) Run(instanceID string, vs parser.Scope, is map[string]interface{}, tid uint64, args []interface{}) (interface{}, error) {
+	var err error
+
+	if len(args) != 0 {
+		err = fmt.Errorf("Function does not require any parameters")
+	}
+
+	return graph.NewConcurrentGraphTrans(f.GM), err
+}
+
+/*
+DocString returns a descriptive string.
+*/
+func (f *NewTransFunc) DocString() (string, error) {
+	return "Creates a new transaction for EliasDB.", nil
+}
+
+/*
+NewRollingTransFunc creates a new rolling transaction for EliasDB.
+A rolling transaction commits after n entries.
+*/
+type NewRollingTransFunc struct {
+	GM *graph.Manager
+}
+
+/*
+Run executes the ECAL function.
+*/
+func (f *NewRollingTransFunc) Run(instanceID string, vs parser.Scope, is map[string]interface{}, tid uint64, args []interface{}) (interface{}, error) {
+	var err error
+	var trans graph.Trans
+
+	if arglen := len(args); arglen != 1 {
+		err = fmt.Errorf(
+			"Function requires the rolling threshold (number of operations before rolling)")
+	}
+
+	if err == nil {
+		var i int
+
+		if i, err = strconv.Atoi(fmt.Sprint(args[0])); err != nil {
+			err = fmt.Errorf("Rolling threshold must be a number not: %v", args[0])
+		} else {
+			trans = graph.NewRollingTrans(graph.NewConcurrentGraphTrans(f.GM),
+				i, f.GM, graph.NewConcurrentGraphTrans)
+		}
+	}
+
+	return trans, err
+}
+
+/*
+DocString returns a descriptive string.
+*/
+func (f *NewRollingTransFunc) DocString() (string, error) {
+	return "Creates a new rolling transaction for EliasDB. A rolling transaction commits after n entries.", nil
+}
+
+/*
+CommitTransFunc commits an existing transaction for EliasDB.
+*/
+type CommitTransFunc struct {
+	GM *graph.Manager
+}
+
+/*
+Run executes the ECAL function.
+*/
+func (f *CommitTransFunc) Run(instanceID string, vs parser.Scope, is map[string]interface{}, tid uint64, args []interface{}) (interface{}, error) {
+	var err error
+
+	if arglen := len(args); arglen != 1 {
+		err = fmt.Errorf(
+			"Function requires the transaction to commit as parameter")
+	}
+
+	if err == nil {
+		trans, ok := args[0].(graph.Trans)
+
+		// Check parameters
+
+		if !ok {
+			err = fmt.Errorf("Parameter must be a transaction")
+		} else {
+			err = trans.Commit()
+		}
+	}
+
+	return nil, err
+}
+
+/*
+DocString returns a descriptive string.
+*/
+func (f *CommitTransFunc) DocString() (string, error) {
+	return "Commits an existing transaction for EliasDB.", nil
+}

ecal/dbfunc/util.go

@@ -0,0 +1,100 @@
+/*
+ * EliasDB
+ *
+ * Copyright 2016 Matthias Ladkau. All rights reserved.
+ *
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, You can obtain one at http://mozilla.org/MPL/2.0/.
+ */
+
+package dbfunc
+
+import (
+	"fmt"
+
+	"devt.de/krotik/ecal/interpreter"
+	"devt.de/krotik/ecal/parser"
+	"devt.de/krotik/ecal/util"
+	"devt.de/krotik/eliasdb/graph"
+)
+
+/*
+RaiseGraphEventHandledFunc returns the special graph.ErrEventHandled error which a sink,
+handling graph events, can return to notify the GraphManager that no further
+action is necessary.
+*/
+type RaiseGraphEventHandledFunc struct {
+}
+
+/*
+Run executes the ECAL function.
+*/
+func (f *RaiseGraphEventHandledFunc) Run(instanceID string, vs parser.Scope, is map[string]interface{}, tid uint64, args []interface{}) (interface{}, error) {
+	return nil, graph.ErrEventHandled
+}
+
+/*
+DocString returns a descriptive string.
+*/
+func (f *RaiseGraphEventHandledFunc) DocString() (string, error) {
+	return "When handling a graph event, notify the GraphManager of EliasDB that no further action is necessary.", nil
+}
+
+/*
+ErrWebEventHandled is a special error to signal that a web request was handled.
+*/
+var ErrWebEventHandled = fmt.Errorf("Web event handled")
+
+/*
+RaiseWebEventHandledFunc returns a special error which a sink can return to notify
+the web API that a web request was handled.
+*/
+type RaiseWebEventHandledFunc struct {
+}
+
+/*
+Run executes the ECAL function.
+*/
+func (f *RaiseWebEventHandledFunc) Run(instanceID string, vs parser.Scope, is map[string]interface{}, tid uint64, args []interface{}) (interface{}, error) {
+	if arglen := len(args); arglen != 1 {
+		return nil, fmt.Errorf("Function requires 1 parameter: request response object")
+	}
+
+	res := args[0]
+
+	resMap, ok := res.(map[interface{}]interface{})
+
+	if !ok {
+		return nil, fmt.Errorf("Request response object should be a map")
+	}
+
+	if _, ok := resMap["status"]; !ok {
+		resMap["status"] = 200
+	}
+	if _, ok := resMap["headers"]; !ok {
+		resMap["header"] = map[interface{}]interface{}{
+			"Content-Type":           "application/json; charset=utf-8",
+			"X-Content-Type-Options": "nosniff",
+		}
+	}
+	if _, ok := resMap["body"]; !ok {
+		resMap["body"] = map[interface{}]interface{}{}
+	}
+
+	erp := is["erp"].(*interpreter.ECALRuntimeProvider)
+	node := is["astnode"].(*parser.ASTNode)
+
+	return nil, &util.RuntimeErrorWithDetail{
+		RuntimeError: erp.NewRuntimeError(ErrWebEventHandled, "", node).(*util.RuntimeError),
+		Environment:  vs,
+		Data:         res,
+	}
+}
+
+/*
+DocString returns a descriptive string.
+*/
+func (f *RaiseWebEventHandledFunc) DocString() (string, error) {
+	return "When handling a web event, notify the web API of EliasDB that the web request was handled.", nil
+}

ecal/dbfunc/util_test.go

@@ -0,0 +1,71 @@
+/*
+ * EliasDB
+ *
+ * Copyright 2016 Matthias Ladkau. All rights reserved.
+ *
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, You can obtain one at http://mozilla.org/MPL/2.0/.
+ */
+
+package dbfunc
+
+import (
+	"testing"
+
+	"devt.de/krotik/ecal/interpreter"
+	"devt.de/krotik/ecal/parser"
+	"devt.de/krotik/ecal/util"
+	"devt.de/krotik/eliasdb/graph"
+)
+
+func TestRaiseGraphEventHandled(t *testing.T) {
+
+	f := &RaiseGraphEventHandledFunc{}
+
+	if _, err := f.DocString(); err != nil {
+		t.Error(err)
+		return
+	}
+
+	if _, err := f.Run("", nil, nil, 0, []interface{}{}); err != graph.ErrEventHandled {
+		t.Error("Unexpected result:", err)
+		return
+	}
+}
+
+func TestRaiseWebEventHandled(t *testing.T) {
+
+	f := &RaiseWebEventHandledFunc{}
+
+	if _, err := f.DocString(); err != nil {
+		t.Error(err)
+		return
+	}
+
+	if _, err := f.Run("", nil, nil, 0, []interface{}{}); err == nil ||
+		err.Error() != "Function requires 1 parameter: request response object" {
+		t.Error(err)
+		return
+	}
+
+	if _, err := f.Run("", nil, nil, 0, []interface{}{""}); err == nil ||
+		err.Error() != "Request response object should be a map" {
+		t.Error(err)
+		return
+	}
+
+	astnode, _ := parser.ASTFromJSONObject(map[string]interface{}{
+		"name": "foo",
+	})
+
+	_, err := f.Run("", nil, map[string]interface{}{
+		"erp":     interpreter.NewECALRuntimeProvider("", nil, nil),
+		"astnode": astnode,
+	}, 0, []interface{}{map[interface{}]interface{}{}})
+
+	if err.(*util.RuntimeErrorWithDetail).Type != ErrWebEventHandled {
+		t.Error("Unexpected result:", err)
+		return
+	}
+}

ecal/eventbridge.go

@@ -0,0 +1,241 @@
+/*
+ * EliasDB
+ *
+ * Copyright 2016 Matthias Ladkau. All rights reserved.
+ *
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, You can obtain one at http://mozilla.org/MPL/2.0/.
+ */
+
+/*
+Package ecal contains the main API for the event condition action language (ECAL).
+*/
+package ecal
+
+import (
+	"fmt"
+	"strings"
+
+	"devt.de/krotik/common/errorutil"
+	"devt.de/krotik/ecal/engine"
+	"devt.de/krotik/ecal/scope"
+	"devt.de/krotik/ecal/util"
+	"devt.de/krotik/eliasdb/graph"
+	"devt.de/krotik/eliasdb/graph/data"
+)
+
+/*
+EventMapping is a mapping between EliasDB event types to EliasDB specific event kinds in ECAL.
+*/
+var EventMapping = map[int]string{
+
+	/*
+	   EventNodeCreated is thrown when a node was created.
+
+	   Parameters: partition of created node, created node
+	*/
+	graph.EventNodeCreated: "db.node.created",
+
+	/*
+	   EventNodeUpdated is thrown when a node was updated.
+
+	   Parameters: partition of updated node, updated node, old node
+	*/
+	graph.EventNodeUpdated: "db.node.updated",
+
+	/*
+	   EventNodeDeleted is thrown when a node was deleted.
+
+	   Parameters: partition of deleted node, deleted node
+	*/
+	graph.EventNodeDeleted: "db.node.deleted",
+
+	/*
+	   EventEdgeCreated is thrown when an edge was created.
+
+	   Parameters: partition of created edge, created edge
+	*/
+	graph.EventEdgeCreated: "db.edge.created",
+
+	/*
+	   EventEdgeUpdated is thrown when an edge was updated.
+
+	   Parameters: partition of updated edge, updated edge, old edge
+	*/
+	graph.EventEdgeUpdated: "db.edge.updated",
+
+	/*
+	   EventEdgeDeleted is thrown when an edge was deleted.
+
+	   Parameters: partition of deleted edge, deleted edge
+	*/
+	graph.EventEdgeDeleted: "db.edge.deleted",
+
+	/*
+	   EventNodeStore is thrown before a node is stored (always overwriting existing values).
+
+	   Parameters: partition of node to store, node to store
+	*/
+	graph.EventNodeStore: "db.node.store",
+
+	/*
+	   EventNodeUpdate is thrown before a node is updated.
+
+	   Parameters: partition of node to update, node to update
+	*/
+	graph.EventNodeUpdate: "db.node.update",
+
+	/*
+	   EventNodeDelete is thrown before a node is deleted.
+
+	   Parameters: partition of node to delete, key of node to delete, kind of node to delete
+	*/
+	graph.EventNodeDelete: "db.node.delete",
+
+	/*
+	   EventEdgeStore is thrown before an edge is stored (always overwriting existing values).
+
+	   Parameters: partition of stored edge, stored edge
+	*/
+	graph.EventEdgeStore: "db.edge.store",
+
+	/*
+	   EventEdgeDelete is thrown before an edge is deleted.
+
+	   Parameters: partition of deleted edge, deleted edge
+	*/
+	graph.EventEdgeDelete: "db.edge.delete",
+}
+
+/*
+EventBridge is a rule for a graph manager to forward all graph events to ECAL.
+*/
+type EventBridge struct {
+	Processor engine.Processor
+	Logger    util.Logger
+}
+
+/*
+Name returns the name of the rule.
+*/
+func (eb *EventBridge) Name() string {
+	return "ecal.eventbridge"
+}
+
+/*
+Handles returns a list of events which are handled by this rule.
+*/
+func (eb *EventBridge) Handles() []int {
+	return []int{
+		graph.EventNodeCreated,
+		graph.EventNodeUpdated,
+		graph.EventNodeDeleted,
+		graph.EventEdgeCreated,
+		graph.EventEdgeUpdated,
+		graph.EventEdgeDeleted,
+		graph.EventNodeStore,
+		graph.EventNodeUpdate,
+		graph.EventNodeDelete,
+		graph.EventEdgeStore,
+		graph.EventEdgeDelete,
+	}
+}
+
+/*
+Handle handles an event.
+*/
+func (eb *EventBridge) Handle(gm *graph.Manager, trans graph.Trans, event int, ed ...interface{}) error {
+	var err error
+
+	if name, ok := EventMapping[event]; ok {
+		eventName := fmt.Sprintf("EliasDB: %v", name)
+		eventKind := strings.Split(name, ".")
+
+		// Construct an event which can be used to check if any rule will trigger.
+		// This is to avoid the relative costly state construction below for events
+		// which would not trigger any rules.
+
+		triggerCheckEvent := engine.NewEvent(eventName, eventKind, nil)
+
+		if !eb.Processor.IsTriggering(triggerCheckEvent) {
+			return nil
+		}
+
+		// Build up state
+
+		state := map[interface{}]interface{}{
+			"part":  fmt.Sprint(ed[0]),
+			"trans": trans,
+		}
+
+		// Include the right arguments into the state
+
+		switch event {
+		case graph.EventNodeCreated, graph.EventNodeUpdate, graph.EventNodeDeleted, graph.EventNodeStore:
+			state["node"] = scope.ConvertJSONToECALObject(ed[1].(data.Node).Data())
+
+		case graph.EventNodeUpdated:
+			state["node"] = scope.ConvertJSONToECALObject(ed[1].(data.Node).Data())
+			state["old_node"] = scope.ConvertJSONToECALObject(ed[2].(data.Node).Data())
+
+		case graph.EventEdgeCreated, graph.EventEdgeDeleted, graph.EventEdgeStore:
+			state["edge"] = scope.ConvertJSONToECALObject(ed[1].(data.Edge).Data())
+
+		case graph.EventEdgeUpdated:
+			state["edge"] = scope.ConvertJSONToECALObject(ed[1].(data.Edge).Data())
+			state["old_edge"] = scope.ConvertJSONToECALObject(ed[2].(data.Edge).Data())
+
+		case graph.EventNodeDelete, graph.EventEdgeDelete:
+			state["key"] = fmt.Sprint(ed[1])
+			state["kind"] = fmt.Sprint(ed[2])
+		}
+
+		// Try to inject the event
+
+		event := engine.NewEvent(fmt.Sprintf("EliasDB: %v", name), strings.Split(name, "."), state)
+
+		var m engine.Monitor
+		m, err = eb.Processor.AddEventAndWait(event, nil)
+
+		if err == nil {
+
+			// If there was no direct error adding the event then check if an error was
+			// raised in a sink
+
+			if errs := m.(*engine.RootMonitor).AllErrors(); len(errs) > 0 {
+				var errList []error
+
+				for _, e := range errs {
+
+					addError := true
+
+					for _, se := range e.ErrorMap {
+
+						// Check if the sink returned a special graph.ErrEventHandled error
+
+						if re, ok := se.(*util.RuntimeErrorWithDetail); ok && re.Detail == graph.ErrEventHandled.Error() {
+							addError = false
+						}
+					}
+
+					if addError {
+						errList = append(errList, e)
+					}
+				}
+
+				if len(errList) > 0 {
+					err = &errorutil.CompositeError{Errors: errList}
+				} else {
+					err = graph.ErrEventHandled
+				}
+			}
+		}
+
+		if err != nil {
+			eb.Logger.LogDebug(fmt.Sprintf("EliasDB event %v was handled by ECAL and returned: %v", name, err))
+		}
+	}
+
+	return err
+}

ecal/interpreter.go

@@ -0,0 +1,238 @@
+/*
+ * EliasDB
+ *
+ * Copyright 2016 Matthias Ladkau. All rights reserved.
+ *
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, You can obtain one at http://mozilla.org/MPL/2.0/.
+ */
+
+package ecal
+
+import (
+	"fmt"
+	"io/ioutil"
+	"path/filepath"
+	"strings"
+
+	"devt.de/krotik/common/datautil"
+	"devt.de/krotik/common/fileutil"
+	"devt.de/krotik/common/stringutil"
+	"devt.de/krotik/ecal/cli/tool"
+	ecalconfig "devt.de/krotik/ecal/config"
+	"devt.de/krotik/ecal/engine"
+	"devt.de/krotik/ecal/scope"
+	"devt.de/krotik/ecal/stdlib"
+	"devt.de/krotik/ecal/util"
+	"devt.de/krotik/eliasdb/config"
+	"devt.de/krotik/eliasdb/ecal/dbfunc"
+	"devt.de/krotik/eliasdb/graph"
+)
+
+/*
+ScriptingInterpreter models a ECAL script interpreter instance.
+*/
+type ScriptingInterpreter struct {
+	GM          *graph.Manager       // GraphManager for the interpreter
+	Interpreter *tool.CLIInterpreter // ECAL Interpreter object
+
+	Dir       string // Root dir for interpreter
+	EntryFile string // Entry file for the program
+	LogLevel  string // Log level string (Debug, Info, Error)
+	LogFile   string // Logfile (blank for stdout)
+
+	RunDebugServer  bool   // Run a debug server
+	DebugServerHost string // Debug server host
+	DebugServerPort string // Debug server port
+
+	WebsocketConnections *datautil.MapCache
+}
+
+/*
+NewScriptingInterpreter returns a new ECAL scripting interpreter.
+*/
+func NewScriptingInterpreter(scriptFolder string, gm *graph.Manager) *ScriptingInterpreter {
+	return &ScriptingInterpreter{
+		GM:                   gm,
+		Dir:                  scriptFolder,
+		EntryFile:            filepath.Join(scriptFolder, config.Str(config.ECALEntryScript)),
+		LogLevel:             config.Str(config.ECALLogLevel),
+		LogFile:              config.Str(config.ECALLogFile),
+		RunDebugServer:       config.Bool(config.EnableECALDebugServer),
+		DebugServerHost:      config.Str(config.ECALDebugServerHost),
+		DebugServerPort:      config.Str(config.ECALDebugServerPort),
+		WebsocketConnections: datautil.NewMapCache(5000, 0),
+	}
+}
+
+/*
+dummyEntryFile is a small valid ECAL which does not do anything. It is used
+as the default entry file if no entry file exists.
+*/
+const dummyEntryFile = `0 # Write your ECAL code here
+`
+
+/*
+Run runs the ECAL scripting interpreter.
+
+After this function completes:
+- EntryScript in config and all related scripts in the interpreter root dir have been executed
+- ECAL Interpreter object is fully initialized
+- A debug server might be running which can reload the entry script
+- ECAL's event processor has been started
+- GraphManager events are being forwarded to ECAL
+*/
+func (si *ScriptingInterpreter) Run() error {
+	var err error
+
+	// Ensure we have a dummy entry point
+
+	if ok, _ := fileutil.PathExists(si.EntryFile); !ok {
+		err = ioutil.WriteFile(si.EntryFile, []byte(dummyEntryFile), 0600)
+	}
+
+	if err == nil {
+		i := tool.NewCLIInterpreter()
+		si.Interpreter = i
+
+		// Set worker count in ecal config
+
+		ecalconfig.Config[ecalconfig.WorkerCount] = config.Config[config.ECALWorkerCount]
+
+		i.Dir = &si.Dir
+		i.LogFile = &si.LogFile
+		i.LogLevel = &si.LogLevel
+
+		i.EntryFile = si.EntryFile
+		i.LoadPlugins = true
+
+		i.CreateRuntimeProvider("eliasdb-runtime")
+
+		// Adding functions
+
+		AddEliasDBStdlibFunctions(si.GM)
+
+		// Adding rules
+
+		sockRule := &engine.Rule{
+			Name:            "EliasDB-websocket-communication-rule", // Name
+			Desc:            "Handles a websocket communication",    // Description
+			KindMatch:       []string{"db.web.sock.msg"},            // Kind match
+			ScopeMatch:      []string{},
+			StateMatch:      nil,
+			Priority:        0,
+			SuppressionList: nil,
+			Action:          si.HandleECALSockEvent,
+		}
+
+		si.Interpreter.CustomRules = append(si.Interpreter.CustomRules, sockRule)
+
+		if err == nil {
+
+			if si.RunDebugServer {
+				di := tool.NewCLIDebugInterpreter(i)
+
+				addr := fmt.Sprintf("%v:%v", si.DebugServerHost, si.DebugServerPort)
+				di.DebugServerAddr = &addr
+				di.RunDebugServer = &si.RunDebugServer
+				falseFlag := false
+				di.EchoDebugServer = &falseFlag
+				di.Interactive = &falseFlag
+				di.BreakOnStart = &falseFlag
+				di.BreakOnError = &falseFlag
+
+				err = di.Interpret()
+
+			} else {
+
+				err = i.Interpret(false)
+			}
+
+			// EliasDB graph events are now forwarded to ECAL via the eventbridge.
+
+			si.GM.SetGraphRule(&EventBridge{
+				Processor: i.RuntimeProvider.Processor,
+				Logger:    i.RuntimeProvider.Logger,
+			})
+		}
+	}
+
+	// Include a traceback if possible
+
+	if ss, ok := err.(util.TraceableRuntimeError); ok {
+		err = fmt.Errorf("%v\n  %v", err.Error(), strings.Join(ss.GetTraceString(), "\n  "))
+	}
+
+	return err
+}
+
+/*
+RegisterECALSock registers a websocket which should be connected to ECAL events.
+*/
+func (si *ScriptingInterpreter) RegisterECALSock(conn *WebsocketConnection) {
+	si.WebsocketConnections.Put(conn.CommID, conn)
+}
+
+/*
+DeregisterECALSock removes a registered websocket.
+*/
+func (si *ScriptingInterpreter) DeregisterECALSock(conn *WebsocketConnection) {
+	si.WebsocketConnections.Remove(conn.CommID)
+}
+
+/*
+HandleECALSockEvent handles websocket events from the ECAL interpreter (db.web.sock.msg events).
+*/
+func (si *ScriptingInterpreter) HandleECALSockEvent(p engine.Processor, m engine.Monitor, e *engine.Event, tid uint64) error {
+	state := e.State()
+	payload := scope.ConvertECALToJSONObject(state["payload"])
+	shouldClose := stringutil.IsTrueValue(fmt.Sprint(state["close"]))
+
+	id := "null"
+	if commID, ok := state["commID"]; ok {
+		id = fmt.Sprint(commID)
+	}
+
+	err := fmt.Errorf("Could not send data to unknown websocket - commID: %v", id)
+
+	if conn, ok := si.WebsocketConnections.Get(id); ok {
+		err = nil
+		wconn := conn.(*WebsocketConnection)
+		wconn.WriteData(map[string]interface{}{
+			"commID":  id,
+			"payload": payload,
+			"close":   shouldClose,
+		})
+
+		if shouldClose {
+			wconn.Close("")
+		}
+	}
+
+	return err
+}
+
+/*
+AddEliasDBStdlibFunctions adds EliasDB related ECAL stdlib functions.
+*/
+func AddEliasDBStdlibFunctions(gm *graph.Manager) {
+	stdlib.AddStdlibPkg("db", "EliasDB related functions")
+
+	stdlib.AddStdlibFunc("db", "storeNode", &dbfunc.StoreNodeFunc{GM: gm})
+	stdlib.AddStdlibFunc("db", "updateNode", &dbfunc.UpdateNodeFunc{GM: gm})
+	stdlib.AddStdlibFunc("db", "removeNode", &dbfunc.RemoveNodeFunc{GM: gm})
+	stdlib.AddStdlibFunc("db", "fetchNode", &dbfunc.FetchNodeFunc{GM: gm})
+	stdlib.AddStdlibFunc("db", "storeEdge", &dbfunc.StoreEdgeFunc{GM: gm})
+	stdlib.AddStdlibFunc("db", "removeEdge", &dbfunc.RemoveEdgeFunc{GM: gm})
+	stdlib.AddStdlibFunc("db", "fetchEdge", &dbfunc.FetchEdgeFunc{GM: gm})
+	stdlib.AddStdlibFunc("db", "traverse", &dbfunc.TraverseFunc{GM: gm})
+	stdlib.AddStdlibFunc("db", "newTrans", &dbfunc.NewTransFunc{GM: gm})
+	stdlib.AddStdlibFunc("db", "newRollingTrans", &dbfunc.NewRollingTransFunc{GM: gm})
+	stdlib.AddStdlibFunc("db", "commit", &dbfunc.CommitTransFunc{GM: gm})
+	stdlib.AddStdlibFunc("db", "query", &dbfunc.QueryFunc{GM: gm})
+	stdlib.AddStdlibFunc("db", "graphQL", &dbfunc.GraphQLFunc{GM: gm})
+	stdlib.AddStdlibFunc("db", "raiseGraphEventHandled", &dbfunc.RaiseGraphEventHandledFunc{})
+	stdlib.AddStdlibFunc("db", "raiseWebEventHandled", &dbfunc.RaiseWebEventHandledFunc{})
+
+}

ecal/interpreter_test.go

@@ -0,0 +1,426 @@
+/*
+ * EliasDB
+ *
+ * Copyright 2016 Matthias Ladkau. All rights reserved.
+ *
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, You can obtain one at http://mozilla.org/MPL/2.0/.
+ */
+
+package ecal
+
+import (
+	"flag"
+	"fmt"
+	"io/ioutil"
+	"os"
+	"path/filepath"
+	"testing"
+
+	"devt.de/krotik/common/errorutil"
+	"devt.de/krotik/common/fileutil"
+	"devt.de/krotik/eliasdb/config"
+	"devt.de/krotik/eliasdb/graph"
+	"devt.de/krotik/eliasdb/graph/data"
+	"devt.de/krotik/eliasdb/graph/graphstorage"
+)
+
+const testScriptDir = "testscripts"
+
+func TestMain(m *testing.M) {
+	flag.Parse()
+
+	defer func() {
+		if res, _ := fileutil.PathExists(testScriptDir); res {
+			if err := os.RemoveAll(testScriptDir); err != nil {
+				fmt.Print("Could not remove test directory:", err.Error())
+			}
+		}
+	}()
+
+	if res, _ := fileutil.PathExists(testScriptDir); res {
+		if err := os.RemoveAll(testScriptDir); err != nil {
+			fmt.Print("Could not remove test directory:", err.Error())
+		}
+	}
+
+	ensurePath(testScriptDir)
+
+	data := make(map[string]interface{})
+	for k, v := range config.DefaultConfig {
+		data[k] = v
+	}
+
+	config.Config = data
+
+	config.Config[config.EnableECALScripts] = true
+	config.Config[config.ECALScriptFolder] = testScriptDir
+	config.Config[config.ECALLogFile] = filepath.Join(testScriptDir, "interpreter.log")
+
+	// Run the tests
+
+	m.Run()
+}
+
+/*
+ensurePath ensures that a given relative path exists.
+*/
+func ensurePath(path string) {
+	if res, _ := fileutil.PathExists(path); !res {
+		if err := os.Mkdir(path, 0770); err != nil {
+			fmt.Print("Could not create directory:", err.Error())
+			return
+		}
+	}
+}
+
+func writeScript(content string) {
+	filename := filepath.Join(testScriptDir, config.Str(config.ECALEntryScript))
+	err := ioutil.WriteFile(
+		filename,
+		[]byte(content), 0600)
+	errorutil.AssertOk(err)
+	os.Remove(config.Str(config.ECALLogFile))
+}
+
+func checkLog(expected string) error {
+	var err error
+
+	content, err := ioutil.ReadFile(config.Str(config.ECALLogFile))
+	errorutil.AssertOk(err)
+
+	logtext := string(content)
+
+	if logtext != expected {
+		err = fmt.Errorf("Unexpected log text:\n%v", logtext)
+	}
+
+	return err
+}
+
+func TestDebugInterpreter(t *testing.T) {
+
+	config.Config[config.EnableECALDebugServer] = true
+	defer func() {
+		config.Config[config.EnableECALDebugServer] = false
+		errorutil.AssertOk(os.Remove(config.Str(config.ECALLogFile)))
+
+	}()
+
+	mgs := graphstorage.NewMemoryGraphStorage("mystorage")
+	gm := graph.NewGraphManager(mgs)
+
+	ds := NewScriptingInterpreter(testScriptDir, gm)
+
+	filename := filepath.Join(testScriptDir, config.Str(config.ECALEntryScript))
+	os.Remove(filename)
+
+	if err := ds.Run(); err != nil {
+		t.Error("Unexpected result:", err)
+		return
+	}
+}
+
+func TestInterpreter(t *testing.T) {
+
+	mgs := graphstorage.NewMemoryGraphStorage("mystorage")
+	gm := graph.NewGraphManager(mgs)
+
+	ds := NewScriptingInterpreter(testScriptDir, gm)
+
+	// Test normal log output
+
+	writeScript(`
+log("test insert")
+`)
+
+	if err := ds.Run(); err != nil {
+		t.Error("Unexpected result:", err)
+		return
+	}
+
+	if err := checkLog(`test insert
+`); err != nil {
+		t.Error(err)
+	}
+
+	// Test stack trace
+
+	writeScript(`
+raise("some error")
+`)
+
+	if err := ds.Run(); err == nil || err.Error() != `ECAL error in eliasdb-runtime (testscripts/main.ecal): some error () (Line:2 Pos:1)
+  raise("some error") (testscripts/main.ecal:2)` {
+		t.Error("Unexpected result:", err)
+		return
+	}
+
+	// Test db functions
+
+	writeScript(`
+db.storeNode("main", {
+  "key" : "foo",
+  "kind" : "bar",
+  "data" : 123,
+})
+
+db.storeNode("main", {
+  "key" : "key2",
+  "kind" : "kind2",
+  "data" : 456,
+})
+
+db.storeEdge("main", {
+  "key":           "123",
+  "kind":          "myedges",
+  "end1cascading": true,
+  "end1key":       "foo",
+  "end1kind":      "bar",
+  "end1role":      "role1",
+  "end2cascading": false,
+  "end2key":       "key2",
+  "end2kind":      "kind2",
+  "end2role":      "role2",
+})
+
+[n, e] := db.traverse("main", "key2", "kind2", "role2:myedges:role1:bar")
+
+log("nodes: ", n, " edges: ", e)
+`)
+
+	// The store statements should trigger the triggerCheck shortcut in the eventbridge
+	// because no rules are defined to handle the events.
+
+	if err := ds.Run(); err != nil {
+		t.Error("Unexpected result:", err)
+		return
+	}
+
+	if err := checkLog(`nodes: [
+  {
+    "data": 123,
+    "key": "foo",
+    "kind": "bar"
+  }
+] edges: [
+  {
+    "end1cascading": false,
+    "end1key": "key2",
+    "end1kind": "kind2",
+    "end1role": "role2",
+    "end2cascading": true,
+    "end2key": "foo",
+    "end2kind": "bar",
+    "end2role": "role1",
+    "key": "123",
+    "kind": "myedges"
+  }
+]
+`); err != nil {
+		t.Error(err)
+	}
+}
+
+func TestEvents(t *testing.T) {
+	mgs := graphstorage.NewMemoryGraphStorage("mystorage")
+	gm := graph.NewGraphManager(mgs)
+
+	ds := NewScriptingInterpreter(testScriptDir, gm)
+
+	writeScript(`
+sink mysink
+  kindmatch [ "db.*.*" ],
+{
+  log("Got event: ", event)
+  if event.state["node"] != NULL {
+    if event.state.node.key == "foo2" {
+      raise("Oh no")
+    }
+    if event.state.node.key == "foo3" {
+      db.raiseGraphEventHandled()
+    }
+  } elif event.state["edge"] != NULL {
+    if event.state.edge.key == "foo2" {
+      raise("Oh no edge")
+    }
+    if event.state.edge.key == "foo3" and event.kind == "db.edge.created" {
+      raise("Oh no edge2")
+    }
+    if event.state.edge.key == "foo3" and event.kind == "db.edge.updated" {
+      raise("Oh no edge3")
+    }
+  } else {
+    if event.state.key == "foo3" {
+      db.raiseGraphEventHandled()
+    }
+  }
+}
+`)
+
+	if err := ds.Run(); err != nil {
+		t.Error("Unexpected result:", err)
+		return
+	}
+
+	err := gm.StoreNode("main", data.NewGraphNodeFromMap(map[string]interface{}{
+		"key":  "foo",
+		"kind": "bar",
+		"data": 123,
+	}))
+	errorutil.AssertOk(err)
+
+	if err := checkLog(`Got event: {
+  "kind": "db.node.store",
+  "name": "EliasDB: db.node.store",
+  "state": {
+    "node": {
+      "data": 123,
+      "key": "foo",
+      "kind": "bar"
+    },
+    "part": "main",
+    "trans": {}
+  }
+}
+Got event: {
+  "kind": "db.node.created",
+  "name": "EliasDB: db.node.created",
+  "state": {
+    "node": {
+      "data": 123,
+      "key": "foo",
+      "kind": "bar"
+    },
+    "part": "main",
+    "trans": {}
+  }
+}
+`); err != nil {
+		t.Error(err)
+	}
+
+	// Test raising an error before node storage
+
+	err = gm.StoreNode("main", data.NewGraphNodeFromMap(map[string]interface{}{
+		"key":  "foo2",
+		"kind": "bar",
+		"data": 123,
+	}))
+
+	if err == nil || err.Error() != `GraphError: Graph rule error (Taskerror:
+EliasDB: db.node.store -> mysink : ECAL error in eliasdb-runtime (testscripts/main.ecal): Oh no () (Line:8 Pos:7))` {
+		t.Error("Unexpected result:", err)
+		return
+	}
+
+	if res, err := gm.FetchNode("main", "foo2", "bar"); res != nil || err != nil {
+		t.Error("Unexpected result:", res, err)
+		return
+	}
+
+	err = gm.UpdateNode("main", data.NewGraphNodeFromMap(map[string]interface{}{
+		"key":  "foo",
+		"kind": "bar",
+		"data": 1234,
+	}))
+
+	if err != nil {
+		t.Error("Unexpected result:", err)
+		return
+	}
+
+	err = gm.StoreEdge("main", data.NewGraphEdgeFromNode(data.NewGraphNodeFromMap(map[string]interface{}{
+		"key":           "foo2",
+		"kind":          "e",
+		"end1cascading": true,
+		"end1key":       "a",
+		"end1kind":      "b",
+		"end1role":      "role1",
+		"end2cascading": false,
+		"end2key":       "c",
+		"end2kind":      "d",
+		"end2role":      "role2",
+	})))
+
+	if err == nil || err.Error() != `GraphError: Graph rule error (Taskerror:
+EliasDB: db.edge.store -> mysink : ECAL error in eliasdb-runtime (testscripts/main.ecal): Oh no edge () (Line:15 Pos:7))` {
+		t.Error("Unexpected result:", err)
+		return
+	}
+
+	err = gm.StoreEdge("main", data.NewGraphEdgeFromNode(data.NewGraphNodeFromMap(map[string]interface{}{
+		"key":           "foo3",
+		"kind":          "e",
+		"end1cascading": true,
+		"end1key":       "foo",
+		"end1kind":      "bar",
+		"end1role":      "role1",
+		"end2cascading": false,
+		"end2key":       "foo",
+		"end2kind":      "bar",
+		"end2role":      "role2",
+	})))
+
+	if err == nil || err.Error() != `GraphError: Graph rule error (Taskerror:
+EliasDB: db.edge.created -> mysink : ECAL error in eliasdb-runtime (testscripts/main.ecal): Oh no edge2 () (Line:18 Pos:7))` {
+		t.Error("Unexpected result:", err)
+		return
+	}
+
+	err = gm.StoreEdge("main", data.NewGraphEdgeFromNode(data.NewGraphNodeFromMap(map[string]interface{}{
+		"key":           "foo3",
+		"kind":          "e",
+		"end1cascading": true,
+		"end1key":       "foo",
+		"end1kind":      "bar",
+		"end1role":      "role1",
+		"end2cascading": false,
+		"end2key":       "foo",
+		"end2kind":      "bar",
+		"end2role":      "role2",
+	})))
+
+	if err == nil || err.Error() != `GraphError: Graph rule error (Taskerror:
+EliasDB: db.edge.updated -> mysink : ECAL error in eliasdb-runtime (testscripts/main.ecal): Oh no edge3 () (Line:21 Pos:7))` {
+		t.Error("Unexpected result:", err)
+		return
+	}
+
+	// Test preventing node storage without raising an error
+
+	err = gm.StoreNode("main", data.NewGraphNodeFromMap(map[string]interface{}{
+		"key":  "foo3",
+		"kind": "bar",
+		"data": 123,
+	}))
+
+	if err != nil {
+		t.Error("Unexpected result:", err)
+		return
+	}
+
+	if res, err := gm.FetchNode("main", "foo2", "bar"); res != nil || err != nil {
+		t.Error("Unexpected result:", res, err)
+		return
+	}
+
+	err = gm.UpdateNode("main", data.NewGraphNodeFromMap(map[string]interface{}{
+		"key":  "foo3",
+		"kind": "bar",
+		"data": 123,
+	}))
+
+	if err != nil {
+		t.Error("Unexpected result:", err)
+		return
+	}
+
+	_, err = gm.RemoveNode("main", "foo3", "bar")
+
+	if err != nil {
+		t.Error("Unexpected result:", err)
+		return
+	}
+}

ecal/websocket.go

@@ -0,0 +1,98 @@
+/*
+ * EliasDB
+ *
+ * Copyright 2016 Matthias Ladkau. All rights reserved.
+ *
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, You can obtain one at http://mozilla.org/MPL/2.0/.
+ */
+
+package ecal
+
+import (
+	"encoding/json"
+	"sync"
+	"time"
+
+	"github.com/gorilla/websocket"
+)
+
+/*
+WebsocketConnection models a single websocket connection.
+
+Websocket connections support one concurrent reader and one concurrent writer.
+See: https://godoc.org/github.com/gorilla/websocket#hdr-Concurrency
+*/
+type WebsocketConnection struct {
+	CommID string
+	Conn   *websocket.Conn
+	RMutex *sync.Mutex
+	WMutex *sync.Mutex
+}
+
+/*
+NewWebsocketConnection creates a new WebsocketConnection object.
+*/
+func NewWebsocketConnection(commID string, c *websocket.Conn) *WebsocketConnection {
+	return &WebsocketConnection{
+		CommID: commID,
+		Conn:   c,
+		RMutex: &sync.Mutex{},
+		WMutex: &sync.Mutex{}}
+}
+
+/*
+Init initializes the websocket connection.
+*/
+func (wc *WebsocketConnection) Init() {
+	wc.WMutex.Lock()
+	defer wc.WMutex.Unlock()
+	wc.Conn.WriteMessage(websocket.TextMessage, []byte(`{"type":"init_success","payload":{}}`))
+}
+
+/*
+ReadData reads data from the websocket connection.
+*/
+func (wc *WebsocketConnection) ReadData() (map[string]interface{}, bool, error) {
+	var data map[string]interface{}
+	var fatal = true
+
+	wc.RMutex.Lock()
+	_, msg, err := wc.Conn.ReadMessage()
+	wc.RMutex.Unlock()
+
+	if err == nil {
+		fatal = false
+		err = json.Unmarshal(msg, &data)
+	}
+
+	return data, fatal, err
+}
+
+/*
+WriteData writes data to the websocket.
+*/
+func (wc *WebsocketConnection) WriteData(data map[string]interface{}) {
+	wc.WMutex.Lock()
+	defer wc.WMutex.Unlock()
+
+	jsonData, _ := json.Marshal(map[string]interface{}{
+		"commID":  wc.CommID,
+		"type":    "data",
+		"payload": data,
+	})
+
+	wc.Conn.WriteMessage(websocket.TextMessage, jsonData)
+}
+
+/*
+Close closes the websocket connection.
+*/
+func (wc *WebsocketConnection) Close(msg string) {
+	wc.Conn.WriteControl(websocket.CloseMessage,
+		websocket.FormatCloseMessage(
+			websocket.CloseNormalClosure, msg), time.Now().Add(10*time.Second))
+
+	wc.Conn.Close()
+}

ecal/websocket_test.go

@@ -0,0 +1,133 @@
+/*
+ * EliasDB
+ *
+ * Copyright 2016 Matthias Ladkau. All rights reserved.
+ *
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, You can obtain one at http://mozilla.org/MPL/2.0/.
+ */
+
+package ecal
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"sync"
+	"testing"
+
+	"devt.de/krotik/common/errorutil"
+	"devt.de/krotik/common/httputil"
+	"devt.de/krotik/ecal/engine"
+	"github.com/gorilla/websocket"
+)
+
+const TESTPORT = ":9090"
+
+func TestWebsocketHandling(t *testing.T) {
+	sockUpgrader := websocket.Upgrader{
+		Subprotocols:    []string{"ecal-sock"},
+		ReadBufferSize:  1024,
+		WriteBufferSize: 1024,
+	}
+
+	si := NewScriptingInterpreter("", nil)
+
+	http.HandleFunc("/httpserver_test", func(w http.ResponseWriter, r *http.Request) {
+
+		conn, err := sockUpgrader.Upgrade(w, r, nil)
+		errorutil.AssertOk(err)
+
+		wsconn := NewWebsocketConnection("123", conn)
+		si.RegisterECALSock(wsconn)
+		defer func() {
+			si.DeregisterECALSock(wsconn)
+		}()
+
+		wc := NewWebsocketConnection("123", conn)
+
+		wc.Init()
+
+		data, _, err := wc.ReadData()
+		errorutil.AssertOk(err)
+		errorutil.AssertTrue(fmt.Sprint(data) == "map[foo:bar]", fmt.Sprint("data is:", data))
+
+		// Simulate that an event is injectd and writes to the websocket
+
+		event := engine.NewEvent(fmt.Sprintf("WebSocketRequest"), []string{"db", "web", "sock", "msg"},
+			map[interface{}]interface{}{
+				"commID":  "123",
+				"payload": "bla",
+				"close":   true,
+			})
+
+		si.HandleECALSockEvent(nil, nil, event, 0)
+	})
+
+	hs := &httputil.HTTPServer{}
+
+	var wg sync.WaitGroup
+	wg.Add(1)
+
+	go hs.RunHTTPServer(TESTPORT, &wg)
+
+	wg.Wait()
+
+	// Server is started
+
+	if hs.LastError != nil {
+		t.Error(hs.LastError)
+		return
+
+	}
+
+	queryURL := "ws://localhost" + TESTPORT + "/httpserver_test"
+
+	c, _, err := websocket.DefaultDialer.Dial(queryURL, nil)
+	if err != nil {
+		t.Error("Could not open websocket:", err)
+		return
+	}
+
+	_, message, err := c.ReadMessage()
+
+	if msg := formatJSONString(string(message)); err != nil || msg != `{
+  "type": "init_success",
+  "payload": {}
+}` {
+		t.Error("Unexpected response:", msg, err)
+		return
+	}
+
+	err = c.WriteMessage(websocket.TextMessage, []byte(`{"foo":"bar"}`))
+	if err != nil {
+		t.Error("Could not send message:", err)
+		return
+	}
+
+	_, message, err = c.ReadMessage()
+
+	if msg := formatJSONString(string(message)); err != nil || msg != `{
+  "commID": "123",
+  "payload": {
+    "close": true,
+    "commID": "123",
+    "payload": "bla"
+  },
+  "type": "data"
+}` {
+		t.Error("Unexpected response:", msg, err)
+		return
+	}
+}
+
+/*
+formatJSONString formats a given JSON string.
+*/
+func formatJSONString(str string) string {
+	out := bytes.Buffer{}
+	errorutil.AssertOk(json.Indent(&out, []byte(str), "", "  "))
+	return out.String()
+}

eql/interpreter/runtime_test.go

@@ -235,7 +235,7 @@ func TestErrors(t *testing.T) {
 
 	msm.AccessMap[1] = storage.AccessCacheAndFetchSeriousError
 
-	if _, err := rt.nextStartKey(); err.Error() != "GraphError: Could not read graph information (Record is already in-use (? - ))" {
+	if _, err := rt.nextStartKey(); err.Error() != "GraphError: Could not read graph information (Record is already in-use (<memory> - ))" {
 		t.Error(err)
 		return
 	}

examples/chat/doc/chat.md

@@ -1,6 +1,7 @@
 EliasDB Chat Example
 ==
 This example demonstrates a simple application which uses advanced features of EliasDB:
+- Node modification via ECAL script
 - User Management
 - GraphQL subscriptions
 

examples/chat/res/chat/src/component/ChatWindow.vue

@@ -16,7 +16,7 @@ Chat window which displays an ongoing chat channel.
         <div class="chat-msg-window" 
             v-for="msg in messages"
             v-bind:key="msg.key">
-                <div>{{msg.message}}</div>
+                <div>{{msg.message}} at {{new Date(msg.ts / 1000).toLocaleString()}}</div>
         </div>
         <chat-text-area :channel="channel"/>
     </div>
@@ -66,6 +66,7 @@ subscription {
   ${this.channel}(ascending:key, last:11) { # last:11 because channel node will be last
       key,
       message,
+      ts,
   }
 }`,
             data => {

examples/chat/res/eliasdb.config.json

@@ -3,9 +3,18 @@
     "ClusterLogHistory": 100,
     "ClusterStateInfoFile": "cluster.stateinfo",
     "CookieMaxAgeSeconds": "86400",
+    "ECALDebugServerHost": "127.0.0.1",
+    "ECALDebugServerPort": "33274",
+    "ECALEntryScript": "main.ecal",
+    "ECALLogFile": "",
+    "ECALLogLevel": "info",
+    "ECALScriptFolder": "scripts",
+    "ECALWorkerCount": 10,
     "EnableAccessControl": true,
     "EnableCluster": false,
     "EnableClusterTerminal": false,
+    "EnableECALDebugServer": false,
+    "EnableECALScripts": true,
     "EnableReadOnly": false,
     "EnableWebFolder": true,
     "EnableWebTerminal": false,

examples/chat/res/scripts/main.ecal

@@ -0,0 +1,21 @@
+/*
+ Modify all stored nodes which have a message with a timestamp.
+ */
+sink AddNodeTimestamp
+    kindmatch ["db.node.store", "db.node.update"]
+    priority 0
+{
+    log("Node Event: ", event)
+
+    if event.state.node.message != NULL and event.state.node.ts == NULL {
+        try {
+            log("store node")
+            event.state.node.ts := now()
+            db.storeNode(event.state.part, event.state.node)
+        } except e {
+            error(e)
+        } otherwise {
+            db.raiseGraphEventHandled()
+        }
+    }
+}

examples/chat/start.bat

@@ -5,7 +5,9 @@ if NOT EXIST run (
   mkdir run
   cd run
   mkdir web
+  mkdir scripts
   xcopy /e ..\res\chat\* web
+  xcopy /e ..\res\scripts\* scrips
   copy ..\res\eliasdb.config.json .
   copy ..\res\access.db .
   cd ..  

examples/chat/start.sh

@@ -3,7 +3,9 @@ cd "$(dirname "$0")"
 
 if ! [ -d "run" ]; then
   mkdir -p run/web
+  mkdir -p run/scripts
   cp -fR res/chat/* run/web
+  cp -fR res/scripts/* run/scripts
   cp -fR res/eliasdb.config.json run
   cp -fR res/access.db run
 fi

examples/game/doc/display-loop.uxf

@@ -0,0 +1,29 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<diagram program="umlet" version="14.3.0">
+  <help_text/>
+  <zoom_level>9</zoom_level>
+  <element>
+    <id>UMLSequenceAllInOne</id>
+    <coordinates>
+      <x>63</x>
+      <y>243</y>
+      <w>1377</w>
+      <h>450</h>
+    </coordinates>
+    <panel_attributes>title=Display loop in the frontend
+obj=game/game-controller.ts\nupdatePushHandler~a
+obj=game/objects.ts\nPlayer~b
+obj=game/objects.ts\nSprite~c
+obj=display/engine.ts\nMainDisplayController:drawLoop()~d
+obj=display/engine.ts\nMainDisplayController~e
+gate-&gt;a:Websocket messages
+a-&gt;b:setState()
+a-&gt;c:setState()
+d-&gt;e:drawSprites()
+d-&gt;e:drawPlayer()
+d-&gt;e:drawInfoOverlay()
+d-&gt;d+:drawLoop() (using setTimeout)
+</panel_attributes>
+    <additional_attributes/>
+  </element>
+</diagram>

examples/game/doc/fire-shot.uxf

@@ -0,0 +1,30 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<diagram program="umlet" version="14.3.0">
+  <help_text/>
+  <zoom_level>8</zoom_level>
+  <element>
+    <id>UMLSequenceAllInOne</id>
+    <coordinates>
+      <x>0</x>
+      <y>240</y>
+      <w>1432</w>
+      <h>472</h>
+    </coordinates>
+    <panel_attributes>title=Firing a shot
+obj=display/default.ts\nDefaultEngineEventHandler:onkeydown()~a 
+obj=game/objects.ts\nPlayer~b
+obj=main.ecal\nsink WebSocketHandler~c
+obj=engine.ecal\nexecuteAction()~e
+obj=templates.ecal\nPlayerState~d
+gate-&gt;a:The user presses &lt;space&gt;
+a-&gt;b:stateUpdate(action='fire')
+b-&gt;c:send state update to websocket (action='fire')
+c-&gt;d:set property action='fire'
+gate-&gt;e:Game loop
+e-&gt;d:doAction()
+d-&gt;gate:Add shot sprite to model
+d-&gt;gate:Send audio event to all clients
+</panel_attributes>
+    <additional_attributes/>
+  </element>
+</diagram>

examples/game/doc/game-loop.uxf

@@ -0,0 +1,28 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<diagram program="umlet" version="14.3.0">
+  <help_text/>
+  <zoom_level>8</zoom_level>
+  <element>
+    <id>UMLSequenceAllInOne</id>
+    <coordinates>
+      <x>128</x>
+      <y>216</y>
+      <w>1392</w>
+      <h>336</h>
+    </coordinates>
+    <panel_attributes>title=Game loop in the backend
+obj=engine.ecal\nmoveLoop()~a 
+obj=engine.ecal\nmoveObject()~b
+obj=engine.ecal\nexecuteAction()~c
+obj=engine.ecal\ncollisionDetection()~d
+obj=game/game-controller.ts\nupdatePushHandler()~e
+gate-&gt;a:Pulse trigger main.gameloop
+a-&gt;b:loop over game objects
+a-&gt;c:loop over game objects
+b-&gt;e:send state update to all websockets
+a-&gt;d:loop over game objects
+a-&gt;e:send object removals to all websockets
+</panel_attributes>
+    <additional_attributes/>
+  </element>
+</diagram>

examples/game/get_score.sh

@@ -0,0 +1,5 @@
+#!/bin/sh
+# Query the score nodes in the main game world
+../../eliasdb console -exec "get score"
+# Query the conf node in the main game world
+../../eliasdb console -exec "get conf"

examples/game/res/eliasdb.config.json

@@ -0,0 +1,33 @@
+{
+    "ClusterConfigFile": "cluster.config.json",
+    "ClusterLogHistory": 100,
+    "ClusterStateInfoFile": "cluster.stateinfo",
+    "CookieMaxAgeSeconds": "86400",
+    "ECALDebugServerHost": "127.0.0.1",
+    "ECALDebugServerPort": "33274",
+    "ECALEntryScript": "main.ecal",
+    "ECALLogFile": "",
+    "ECALLogLevel": "info",
+    "ECALScriptFolder": "scripts",
+    "EnableAccessControl": false,
+    "EnableCluster": false,
+    "EnableClusterTerminal": false,
+    "EnableECALDebugServer": false,
+    "EnableECALScripts": true,
+    "EnableReadOnly": false,
+    "EnableWebFolder": true,
+    "EnableWebTerminal": true,
+    "HTTPSCertificate": "cert.pem",
+    "HTTPSHost": "127.0.0.1",
+    "HTTPSKey": "key.pem",
+    "HTTPSPort": "9090",
+    "LocationAccessDB": "access.db",
+    "LocationDatastore": "db",
+    "LocationHTTPS": "ssl",
+    "LocationUserDB": "users.db",
+    "LocationWebFolder": "web",
+    "LockFile": "eliasdb.lck",
+    "MemoryOnlyStorage": true,
+    "ResultCacheMaxAgeSeconds": 0,
+    "ResultCacheMaxSize": 0
+}

examples/game/res/frontend/.eslintrc.js

@@ -0,0 +1,20 @@
+module.exports = {
+    env: {
+        browser: true,
+        es2021: true
+    },
+    extends: ['standard'],
+    parser: '@typescript-eslint/parser',
+    parserOptions: {
+        ecmaVersion: 12,
+        sourceType: 'module'
+    },
+    plugins: ['@typescript-eslint'],
+    rules: {
+        eqeqeq: [2, 'allow-null'],
+        indent: [2, 4],
+        quotes: [2, 'single'],
+        semi: [2, 'always'],
+        'no-console': 0
+    }
+};

examples/game/res/frontend/.prettierrc.json

@@ -0,0 +1,5 @@
+{
+    "tabWidth": 4,
+    "singleQuote": true,
+    "trailingComma": "none"
+}

examples/game/res/frontend/assets/asset_license.txt

@@ -0,0 +1,16 @@
+All assets are public domain (CC0)
+
+See: https://opengameart.org/art-search-advanced?keys=space&title=&field_art_tags_tid_op=or&field_art_tags_tid=&name=&field_art_type_tid%5B%5D=9&field_art_type_tid%5B%5D=10&field_art_type_tid%5B%5D=7273&field_art_type_tid%5B%5D=14&field_art_type_tid%5B%5D=12&field_art_type_tid%5B%5D=13&field_art_type_tid%5B%5D=11&field_art_licenses_tid%5B%5D=4&sort_by=score&sort_order=DESC&items_per_page=24&Collection=
+
+Credits:
+
+Asteroid images: BlackMoon Design
+Background image: Cuzco
+Background sound: Aquinn
+Explosion sounds: Kenney (Kenney.nl)
+Ship explosion sound: Bart
+Shot sounds: Kenney (Kenney.nl)
+Shot images: Bonsaiheldin
+Ship images: Kenny (Kenney.nl)
+Vanish sound: Krial
+

examples/game/res/frontend/index.html

@@ -0,0 +1,47 @@
+<html>
+    <head>
+        <meta charset="UTF-8" />
+        <script src="dist/frontend.js"></script>
+        <style>
+            .landing-page-element {
+                height: 30%;
+                max-height: 150px;
+                text-align: center;
+                margin: auto;
+                width: 50%;
+                max-width: 600px;
+                padding: 10px;
+            }
+        </style>
+    </head>
+    <body>
+        <h1 class="landing-page-element">Astro</h1>
+        <form action="world.html" method="GET">
+            <table class="landing-page-element">
+                <tr>
+                    <td>Game:</td>
+                    <td><input type="text" name="game" value="main" /></td>
+                </tr>
+                <tr>
+                    <td>Player:</td>
+                    <td>
+                        <input
+                            id="player-name-input"
+                            type="text"
+                            name="player"
+                            value=""
+                        />
+                    </td>
+                </tr>
+                <tr>
+                    <td colspan="2"><input type="submit" value="Start" /></td>
+                </tr>
+            </table>
+        </form>
+        <script>
+            document.getElementById(
+                'player-name-input'
+            ).value = astro.generatePlayerName();
+        </script>
+    </body>
+</html>