A remote union filesystem

Matthias Ladkau ab26b0c56a feat: Initial commit 4 years ago
api ab26b0c56a feat: Initial commit 4 years ago
cli ab26b0c56a feat: Initial commit 4 years ago
config ab26b0c56a feat: Initial commit 4 years ago
examples ab26b0c56a feat: Initial commit 4 years ago
export ab26b0c56a feat: Initial commit 4 years ago
integration ab26b0c56a feat: Initial commit 4 years ago
node ab26b0c56a feat: Initial commit 4 years ago
term ab26b0c56a feat: Initial commit 4 years ago
.gitignore ab26b0c56a feat: Initial commit 4 years ago
LICENSE ab26b0c56a feat: Initial commit 4 years ago
NOTICE ab26b0c56a feat: Initial commit 4 years ago
README.md ab26b0c56a feat: Initial commit 4 years ago
attach_webzip.sh ab26b0c56a feat: Initial commit 4 years ago
branch.go ab26b0c56a feat: Initial commit 4 years ago
branch_test.go ab26b0c56a feat: Initial commit 4 years ago
fileinfo.go ab26b0c56a feat: Initial commit 4 years ago
fileinfo_test.go ab26b0c56a feat: Initial commit 4 years ago
go.mod ab26b0c56a feat: Initial commit 4 years ago
go.sum ab26b0c56a feat: Initial commit 4 years ago
rufs.debug ab26b0c56a feat: Initial commit 4 years ago
swagger.json ab26b0c56a feat: Initial commit 4 years ago
tree.go ab26b0c56a feat: Initial commit 4 years ago
tree_item.go ab26b0c56a feat: Initial commit 4 years ago
tree_test.go ab26b0c56a feat: Initial commit 4 years ago
util.go ab26b0c56a feat: Initial commit 4 years ago
web.zip ab26b0c56a feat: Initial commit 4 years ago

README.md

Rufs

Rufs is a remote union filesystem which aims to provide a lightweight and secure solution for distributed file storage. Rufs uses a client-server system where servers expose branches and clients mount one or several branches into a tree structure. The client can overlay branches providing a union view.

Features

  • Client-Server model using RPC call over ssl.
  • Single executable for client and server.
  • Communication is secured via a secret token which is never transferred over the network and certificate pinning once a client has connected successfully.
  • Clients can provide a unified view with files from different locations.
  • Default client provides CLI, REST API and a web interface.
  • Branches can be read-only.
  • A read-only version of the file system can be exported via FUSE and mounted.

Getting Started

You can download a precompiled package for Windows (win64) or Linux (amd64) here.

The archive contains a single executable which contains the server and client code for Rufs.

Tutorial:

To get an idea of what EliasDB is about have a look at the tutorial.

REST API:

The terminal uses a REST API to communicate with the backend. The REST API can be browsed using a dynamically generated swagger.json definition (https://localhost:9090/fs/swagger.json). You can browse the API of Rufs's latest version here.

Command line options

The main Rufs executable has two main tools:

Rufs 1.0.0

Usage of ./rufs [tool]

The tools are:

    server    Run as a server
    client    Run as a client

Use ./rufs [tool] --help for more information about a tool.

The most important one is server which starts the file server. The server has several options:

Rufs 1.0.0

Usage of ./rufs server [options]

  -config string
    	Server configuration file (default "rufs.server.json")
  -help
    	Show this help message
  -secret string
    	Secret file containing the secret token (default "rufs.secret")
  -ssl-dir string
    	Directory containing the ssl key.pem and cert.pem files (default "ssl")

The server will automatically create a default config file and
default directories if nothing is specified.

Once the server is started the client tool can be used to interact with the server. The options of the client tool are:

Rufs 1.0.0

Usage of ./rufs client [mapping file]

  -fuse-mount string
    	Mount tree as FUSE filesystem at specified path (read-only)
  -help
    	Show this help message
  -secret string
    	Secret file containing the secret token (default "rufs.secret")
  -ssl-dir string
    	Directory containing the ssl key.pem and cert.pem files (default "ssl")
  -web string
    	Export the tree through a https interface on the specified host:port

The mapping file assignes remote branches to the local tree.
The client tries to load rufs.mapping.json if no mapping file is defined.
It starts empty if no mapping file exists. The mapping file
should have the following json format:

{
  "branches" : [
    {
      "branch"      : <branch name>,
      "rpc"         : <rpc interface>,
      "fingerprint" : <fingerprint>
    },
    ...
  ],
  "tree" : [
    {
      "path"      : <path>,
      "branch"    : <branch name>,
      "writeable" : <writable flag>
    },
    ...
  ]
}

On the console type 'q' to exit and 'help' to get an overview of available commands:

Available commands:
----
branch [branch name] [rpc] [fingerprint] : List all known branches or add a new branch to the tree
cat <file>                               : Read and print the contents of a file
cd [path]                                : Show or change the current directory
checksum [path] [glob]                   : Show a directory listing and file checksums
cp <src file/dir> <dst dir>              : Copy a file or directory
dir [path] [glob]                        : Show a directory listing
get <src file> [dst local file]          : Retrieve a file and store it locally (in the current directory)
help [cmd]                               : Show general or command specific help
mkdir <dir>                              : Create a new direectory
mount [path] [branch name] [ro]          : List all mount points or add a new mount point to the tree
ping <branch name> [rpc]                 : Ping a remote branch
put [src local file] [dst file]          : Read a local file and store it
refresh                                  : Refreshes all known branches and reconnect if possible.
ren <file> <newfile>                     : Rename a file or directory
reset [mounts|brances]                   : Remove all mounts or all mounts and all branches
rm <file>                                : Delete a file or directory (* all files; ** all files/recursive)
storeconfig [local file]                 : Store the current tree mapping in a local file
sync <src dir> <dst dir>                 : Make sure dst has the same files and directories as src
tree [path] [glob]                       : Show the listing of a directory and its subdirectories

Configuration

The Rufs client and server use each their own configuration file and require a shared rufs.secret file to be able to talk to each other. The server configuration is called rufs.server.json. After starting the server for the first time it should create a default configuration file. Available configurations are:

Configuration Option Description
BranchName Branch name which the server will export.
EnableReadOnly Export the branch only for read operations.
LocalFolder Local physical folder which is exported.
RPCHost RPC host for communication with clients.
RPCPort RPC port for communication with clients.

Note: It is not (and will never be) possible to access the REST API via HTTP.

Building Rufs

To build Rufs from source you need to have Go installed (go >= 1.12):

Create a directory, change into it and run:

git clone https://devt.de/krotik/rufs/ .

You can build Rufs's executable with:

go build -o rufs ./cli/...

Rufs also has a web interface which should be bundled with the executable. The bundled web interface in web.zip can be attached by running (assuming the rufs executable is in the same folder as the script):

./attach_webzip.sh

License

Rufs source code is available under the MIT License.