/* * EliasDB * * Copyright 2016 Matthias Ladkau. All rights reserved. * * This Source Code Form is subject to the terms of the Mozilla Public * License, v. 2.0. If a copy of the MPL was not distributed with this * file, You can obtain one at http://mozilla.org/MPL/2.0/. */ /* Package v1 contains EliasDB REST API Version 1. Binary Blob control endpoint /blob The blob endpoint can be used to store and retrieve binary data to/from automatically allocated storage locatons. A new binary blob can be stored by sending a POST request. The body should be the binary data to store. The response should have the following structure: { id : } /blob/ GET requests can be used to retrieve a binary blobs with a specific ID. Binary blobs can be updated by sending a PUT request and removed by sending a DELETE request. Cluster control endpoint /cluster The cluster endpoint returns cluster state specific information. A GET request returns the cluster state info as a key-value map: { : , ... } /cluster/join An instance can join an existing cluster by sending a PUT request to the join endpoint. The body should have the following datastructure: { name : , netaddr : } /cluster/eject A cluster member can eject another cluster member or itself by sending a PUT request to the eject endpoint. The body should have the following datastructure: { name : , } /cluster/ping An instance can ping another instance (provided the secret is correct). Cluster membership is not required for this command. The body should have the following datastructure: { name : , netaddr : } /cluster/memberinfos The memberinfos endpoint returns the static member info of every known cluster member. If a member is not reachable its info contains a single key-value pair with the key error and an error message as value. A GET request returns the member info of every member as a key-value map: { : , ... } /cluster/log Returns the latest cluster related log messages. A DELETE call will clear the current log. EQL parser endpoint /eql The EQL endpoint provides direct access to the EQL parser. It can be used to parse a given EQL query into an Abstract Syntax Tree or pretty print a given Abstract Syntax Tree into an EQL query. A query can be parsed into an Abstract Syntax Tree by sending a POST request. The body should have the following format: { query : } Returns a JSON structure or an error message. { ast : } An Abstract Syntax Tree can be pretty printed into a query by sending a POST request. The body should have the following format: { ast : } Returns a JSON structure or an error message. { query : } Graph request endpoint /graph The graph endpoint is the main entry point to send and request graph data. Data can be send by using POST and PUT requests. POST will store data in the datastore and always overwrite any existing data. PUT requests on nodes will only update the given attributes. PUT requests on edges are handled equally to POST requests. Data can be deleted using DELETE requests. The data structure for DELETE requests requires only the key and kind attributes. A PUT, POST or DELETE request should be send to one of the following endpoints: /graph/ A graph with the following datastructure: { nodes : [ { : }, ... ], edges : [ { : }, ... ] } /graph//n A list of nodes: [ { : }, ... ] /graph//e A list of edges: [ { : }, ... ] GET requests can be used to query single or a series of nodes. The endpoints support the limit and offset parameters for lists: limit - How many list items to return offset - Offset in the dataset (0 to -1) The total number of entries is returned in the X-Total-Count header when a list is returned. /graph//n//[node key]/[traversal spec] /graph//e// The return data is a list of objects unless a specific node / edge or a traversal from a specific node is requested. Each object in the list models a node or edge. [{ key : , ... }] If a specifc object is requested then the return data is a single object. { key : , ... } Traversals return two lists containing traversed nodes and edges. The traversal endpoint does NOT support limit and offset parameters. Also the X-Total-Count header is not set. [ [ ], [ ] ] Index query endpoint /index The index query endpoint should be used to run index search queries against partitions. Index queries look for words or phrases on all nodes of a given node kind. A phrase query finds all nodes/edges where an attribute contains a certain phrase. A request url which runs a new phrase search should be of the following form: /index//n/?phrase=&attr= /index//e/?phrase=&attr= The return data is a list of node keys: [ , , ... ] A word query finds all nodes/edges where an attribute contains a certain word. A request url which runs a new word search should be of the following form: /index//n/?word=&attr= /index//e/?word=&attr= The return data is a map which maps node key to a list of word positions: { key : [ , , ... ], ... } A value search finds all nodes/edges where an attribute has a certain value. A request url which runs a new value search should be of the following form: /index//n/?value=&attr= /index//e/?value=&attr= The return data is a list of node keys: [ , , ... ] Find query endpoint /find The find query endpoint is a simplified index query which looks up nodes in all partitions which do not start with a _ character. It either searches for a word / phrase or an exact value on all available attributes. A phrase query finds all nodes/edges where an attribute contains a certain phrase. A request url should be of the following form: /find?text= /find?value= The return data is a map of partitions to node kinds to a list of nodes: { : { : [ { node1 }, { node2 }, ... ] ... } ... } GraphQL request endpoint /graphql /graphql-query The GraphQL endpoints execute GraphQL queries on EliasDB's datastore. The query endpoint supports only read-queries (i.e. no mutations). EliasDB supports only executable definitions and introspection (i.e. no type system validation). General database information endpoint /info The info endpoint returns general database information such as known node kinds, known attributes, etc .. The return data is a key-value map: { : , ... } /info/kind/ The node kind info endpoint returns general information about a known node or edge kind such as known attributes or known edges. Query endpoint /query The query endpoint should be used to run EQL search queries against partitions. The return value is always a list (even if there is only a single entry). A query result gets an ID and is stored in a cache. The ID is returned in the X-Cache-Id header. Subsequent requests for the same result can use the ID instead of a query. The endpoint supports the optional limit, offset and groups parameter: limit - How many list items to return offset - Offset in the dataset groups - If set then group information are included in the result (depending on the result size this can be an expensive call) The total number of entries in the result is returned in the X-Total-Count header. A request url which runs a new query should be of the following form: /query/?q= /query/?rid= The return data is a result object: { header : { labels : All column labels of the search result. format : All column format definitions of the search result. data : The data which is displayed in each column of the search result. (e.g. 1:n:name - Name of starting nodes, 3:e:key - Key of edge traversed in the second traversal) primary_kind : The primary kind of the search result. }, rows : [ [ , , ... ] ], sources : [ [ , , ... ] ], selections : [ ], total_selections : groups : [ [ ], [ ] ... ] } Query result endpoint /queryresult The query result endpoint is used to run operations on query results. The quickfilter endpoint (GET) is used to determine the most frequent used values in a particular result column. /queryresult//quickfilter/?limit= The optional limit parameter can be used to limit the result items. The return data is a simple object: { values : [ , ... ], frequencies : [ , ... ] } /queryresult//select The select endpoint (GET) returns the (primary) nodes which are currently selected. The primary node of each row is usually the node from which the query started, when constructing the row of the result (unless the primary keyword was used). The return data is a simple object: { keys : [ , ... ], kinds : [ , ... ] } /queryresult//select/ The select endpoint with the row parameter (PUT) is used to select single or multiple rows of a query result. The row parameter can either be a positive number or 'all', 'none' or 'invert'. Returns the new number of total selections. /queryresult//groupselected The groupselected endpoint returns the groups which contain the selected (primary) nodes based on the currently selected rows. The primary node of each row is usually the node from which the query started, when constructing the row of the result (unless the primary keyword was used). The return data is a simple object: { groups : [ , ... ], keys : [ [ ], ... ], kinds : [ [ ], ... ] } The state can be set by sending it to the endpoint via a POST request. /queryresult//groupselected/ The groupselected endpoint with a group name adds (PUT) or removes (DELETE) all selected nodes to/from the given (existing) group. /queryresult//csv The csv endpoint returns the search result as CSV string. */ package v1 import ( "bytes" "encoding/json" "fmt" "net/http" "strconv" "devt.de/krotik/eliasdb/api" "devt.de/krotik/eliasdb/storage" ) /* StorageSuffixBlob is the suffix for binary blob storage */ const StorageSuffixBlob = ".blob" /* EndpointBlob is the blob endpoint URL (rooted). Handles everything under blob/... */ const EndpointBlob = api.APIRoot + APIv1 + "/blob/" /* BlobEndpointInst creates a new endpoint handler. */ func BlobEndpointInst() api.RestEndpointHandler { return &blobEndpoint{} } /* Handler object for blob operations. */ type blobEndpoint struct { *api.DefaultEndpointHandler } /* HandleGET handles REST calls to retrieve binary data. */ func (be *blobEndpoint) HandleGET(w http.ResponseWriter, r *http.Request, resources []string) { var res interface{} var ret []byte // Check parameters if !checkResources(w, resources, 2, 2, "Need a partition and a specific data ID") { return } loc, err := strconv.ParseUint(resources[1], 10, 64) if err != nil { http.Error(w, fmt.Sprint("Could not decode data ID: ", err.Error()), http.StatusBadRequest) return } sm := api.GS.StorageManager(resources[0]+StorageSuffixBlob, false) if sm != nil { res, err = sm.FetchCached(loc) if err == storage.ErrNotInCache { err = sm.Fetch(loc, &ret) } else if err == nil && res != nil { ret = res.([]byte) } } // Write data w.Header().Set("content-type", "application/octet-stream") w.Write(ret) } /* HandlePOST handles a REST call to store new binary data. */ func (be *blobEndpoint) HandlePOST(w http.ResponseWriter, r *http.Request, resources []string) { var buf bytes.Buffer // Check parameters if !checkResources(w, resources, 1, 1, "Need a partition") { return } sm := api.GS.StorageManager(resources[0]+StorageSuffixBlob, true) // Use a memory buffer to read send data buf.ReadFrom(r.Body) loc, err := sm.Insert(buf.Bytes()) if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } sm.Flush() // Write data w.Header().Set("content-type", "application/json; charset=utf-8") ret := json.NewEncoder(w) ret.Encode(map[string]interface{}{ "id": loc, }) } /* HandlePUT handles a REST call to update existing binary data. */ func (be *blobEndpoint) HandlePUT(w http.ResponseWriter, r *http.Request, resources []string) { var buf bytes.Buffer // Check parameters if !checkResources(w, resources, 2, 2, "Need a partition and a specific data ID") { return } loc, err := strconv.ParseUint(resources[1], 10, 64) if err != nil { http.Error(w, fmt.Sprint("Could not decode data ID: ", err.Error()), http.StatusBadRequest) return } sm := api.GS.StorageManager(resources[0]+StorageSuffixBlob, false) if sm != nil { // Use a memory buffer to read send data buf.ReadFrom(r.Body) if err := sm.Update(loc, buf.Bytes()); err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } sm.Flush() } } /* HandleDELETE handles a REST call to remove existing binary data. */ func (be *blobEndpoint) HandleDELETE(w http.ResponseWriter, r *http.Request, resources []string) { // Check parameters if !checkResources(w, resources, 2, 2, "Need a partition and a specific data ID") { return } loc, err := strconv.ParseUint(resources[1], 10, 64) if err != nil { http.Error(w, fmt.Sprint("Could not decode data ID: ", err.Error()), http.StatusBadRequest) return } sm := api.GS.StorageManager(resources[0]+StorageSuffixBlob, false) if sm != nil { if err := sm.Free(loc); err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } sm.Flush() } } /* SwaggerDefs is used to describe the endpoint in swagger. */ func (be *blobEndpoint) SwaggerDefs(s map[string]interface{}) { idParams := []map[string]interface{}{ { "name": "id", "in": "path", "description": "ID of the binary blob.", "required": true, "type": "string", }, } partitionParams := []map[string]interface{}{ { "name": "partition", "in": "path", "description": "Partition to select.", "required": true, "type": "string", }, } binaryData := []map[string]interface{}{ { "name": "data", "in": "body", "description": "The data to store.", "required": true, "schema": map[string]interface{}{ "description": "A blob of binary data.", }, }, } s["paths"].(map[string]interface{})["/v1/blob/{partition}"] = map[string]interface{}{ "post": map[string]interface{}{ "summary": "Create a binary blob of data.", "description": "The blob endpoint can be used to store binary data. Its location will be automatically allocated.", "consumes": []string{ "application/octet-stream", }, "produces": []string{ "text/plain", "application/json", }, "parameters": append(binaryData, partitionParams...), "responses": map[string]interface{}{ "200": map[string]interface{}{ "description": "The operation was successful.", "schema": map[string]interface{}{ "type": "object", "properties": map[string]interface{}{ "id": map[string]interface{}{ "description": "The data ID which can be used to lookup the data.", "type": "number", }, }, }, }, "default": map[string]interface{}{ "description": "Error response", "schema": map[string]interface{}{ "$ref": "#/definitions/Error", }, }, }, }, } s["paths"].(map[string]interface{})["/v1/blob/{partition}/{id}"] = map[string]interface{}{ "get": map[string]interface{}{ "summary": "Retrieve a binary blob of data.", "description": "The blob endpoint can be used to retrieve binary data from a specific location.", "produces": []string{ "text/plain", "application/octet-stream", }, "parameters": append(idParams, partitionParams...), "responses": map[string]interface{}{ "200": map[string]interface{}{ "description": "The requested binary blob.", }, "default": map[string]interface{}{ "description": "Error response", "schema": map[string]interface{}{ "$ref": "#/definitions/Error", }, }, }, }, "put": map[string]interface{}{ "summary": "Update a binary blob of data.", "description": "The blob endpoint can be used to update binary data at a specific location.", "produces": []string{ "text/plain", }, "parameters": append(idParams, partitionParams...), "responses": map[string]interface{}{ "200": map[string]interface{}{ "description": "The operation was successful.", }, "default": map[string]interface{}{ "description": "Error response", "schema": map[string]interface{}{ "$ref": "#/definitions/Error", }, }, }, }, "delete": map[string]interface{}{ "summary": "Remove a binary blob of data.", "description": "The blob endpoint can be used to remove binary data from a specific location.", "produces": []string{ "text/plain", }, "parameters": append(idParams, partitionParams...), "responses": map[string]interface{}{ "200": map[string]interface{}{ "description": "The operation was successful.", }, "default": map[string]interface{}{ "description": "Error response", "schema": map[string]interface{}{ "$ref": "#/definitions/Error", }, }, }, }, } // Add generic error object to definition s["definitions"].(map[string]interface{})["Error"] = map[string]interface{}{ "description": "A human readable error mesage.", "type": "string", } }