CallFS: A High-Performance, Distributed REST API Filesystem
CallFS is an ultra-lightweight, high-performance REST API filesystem that provides precise Linux filesystem semantics over various backends, including local storage, Amazon S3, and a distributed peer-to-peer network. It is designed for speed, reliability, and horizontal scalability.
🚀 Key Features
Multi-Backend Storage : Seamlessly use Local Filesystem, Amazon S3, or other S3-compatible services (like MinIO) as storage backends.
: Seamlessly use Local Filesystem, Amazon S3, or other S3-compatible services (like MinIO) as storage backends. Distributed Architecture : Scale horizontally by adding more CallFS instances. The system automatically routes operations to the correct node.
: Scale horizontally by adding more CallFS instances. The system automatically routes operations to the correct node. Cross-Server Operations : Operations like move, copy, and delete work across the cluster with automatic conflict detection and resolution.
: Operations like move, copy, and delete work across the cluster with automatic conflict detection and resolution. High-Performance API : A clean REST API for all filesystem operations, built for low latency and high throughput.
: A clean REST API for all filesystem operations, built for low latency and high throughput. Secure, Ephemeral Links : Generate secure, time-limited, single-use download links for any file.
: Generate secure, time-limited, single-use download links for any file. Distributed Locking : A Redis-based Distributed Lock Manager ensures data consistency for concurrent operations across the cluster.
: A Redis-based Distributed Lock Manager ensures data consistency for concurrent operations across the cluster. Rich Metadata Store : PostgreSQL backend for robust, queryable, and persistent file metadata.
: PostgreSQL backend for robust, queryable, and persistent file metadata. Comprehensive Security : Enforces TLS, API key authentication, and a full Unix permission model for authorization.
: Enforces TLS, API key authentication, and a full Unix permission model for authorization. First-Class Observability : Structured logging (JSON/console) and extensive Prometheus metrics for deep operational insight.
: Structured logging (JSON/console) and extensive Prometheus metrics for deep operational insight. Zero-Copy I/O: Efficiently streams large files without buffering them in memory, ensuring a low memory footprint.
🏗️ Architecture
CallFS consists of several core components that work together to provide a unified, distributed filesystem API:
API Server : The public-facing HTTP server that handles all incoming REST API requests.
: The public-facing HTTP server that handles all incoming REST API requests. Core Engine : The central orchestrator that processes requests, manages metadata, and selects the appropriate storage backend.
: The central orchestrator that processes requests, manages metadata, and selects the appropriate storage backend. Storage Backends : Pluggable modules for different storage systems: LocalFS : Manages files on the local disk. S3 : Interfaces with Amazon S3 or compatible object stores. Internal Proxy : Routes requests to other CallFS instances in the cluster.
: Pluggable modules for different storage systems: Metadata Store : A PostgreSQL database that stores all filesystem metadata, such as file names, sizes, timestamps, permissions, and backend locations.
: A PostgreSQL database that stores all filesystem metadata, such as file names, sizes, timestamps, permissions, and backend locations. Distributed Lock Manager (DLM) : Uses Redis to provide cluster-wide locks, preventing race conditions during concurrent file modifications.
: Uses Redis to provide cluster-wide locks, preventing race conditions during concurrent file modifications. Link Manager: Generates and validates secure single-use download tokens.
This modular design allows CallFS to be both powerful and flexible, suitable for a wide range of applications from simple file serving to complex, distributed storage architectures.
🔗 API Endpoints
All API endpoints are prefixed with /v1 and require Bearer authentication, except where noted.
File & Directory Operations
GET /files/{path} : Downloads a file or lists a directory's contents.
: Downloads a file or lists a directory's contents. HEAD /files/{path} : Retrieves file metadata. This operation is "enhanced" and can route requests across the cluster to find the correct node.
: Retrieves file metadata. This operation is "enhanced" and can route requests across the cluster to find the correct node. POST /files/{path} : Creates a new file or directory. It features cross-server conflict detection to prevent overwriting.
: Creates a new file or directory. It features cross-server conflict detection to prevent overwriting. PUT /files/{path} : Uploads or updates a file's content. Supports streaming data and cross-server proxying.
: Uploads or updates a file's content. Supports streaming data and cross-server proxying. DELETE /files/{path} : Deletes a file or directory, with cross-server routing capabilities.
Enhanced Directory Listings
GET /directories/{path} : Provides a detailed JSON listing of a directory's contents.
: Provides a detailed JSON listing of a directory's contents. GET /directories/{path}?recursive=true : Performs a recursive listing of all subdirectories.
: Performs a recursive listing of all subdirectories. GET /directories/{path}?recursive=true&max_depth=N : Limits the depth of the recursive listing.
Single-Use Download Links
POST /links/generate : Creates a secure, single-use download link for a file. The link's expiry time is configurable.
: Creates a secure, single-use download link for a file. The link's expiry time is configurable. GET /download/{token} : Downloads a file using a single-use token. (No authentication required).
System Health & Metrics
GET /health : Returns the operational status of the server. (No authentication required) .
: Returns the operational status of the server. . GET /metrics : Exposes detailed performance metrics in Prometheus format. (No authentication required).
🔧 Getting Started
Prerequisites
Go : Version 1.21 or later.
: Version 1.21 or later. PostgreSQL : A running instance for the metadata store.
: A running instance for the metadata store. Redis : A running instance for the distributed lock manager.
: A running instance for the distributed lock manager. Docker & Docker Compose : Recommended for easily running PostgreSQL and Redis.
: Recommended for easily running PostgreSQL and Redis. A valid config.yaml file (see config.yaml.example )
Running the Server
Start dependent services: docker-compose up -d postgres redis Build the binary: go build -o callfs ./cmd/main.go Run the server: ./callfs server --config /path/to/your/config.yaml
Command-Line Interface (CLI)
CallFS includes a simple CLI for managing the server:
callfs server : Starts the main API server. --config, -c : Specifies the path to the configuration file.
: Starts the main API server. callfs config validate : Validates the configuration file and displays the loaded settings.
: Validates the configuration file and displays the loaded settings. callfs --help : Shows all available commands and flags.
📖 Full Documentation
For detailed information on configuration, API usage, security, and more, please refer to the docs_markdown/ directory:
🤝 Contributing
Contributions are welcome! Please see CONTRIBUTING.md for guidelines on how to participate in this project.
📝 License
CallFS is licensed under the MIT License. See the LICENSE file for details.