14 KiB

Raw Blame History

Iron Requirements

1. Product Overview

Iron is a personal cloud drive system built around a self-hosted gateway service.

Users access a unified file space through a Web application in the first phase. The gateway manages metadata, local cache, upload/download pipelines, preview streaming, and data placement across multiple storage backends.

The product goal is not to be a backup tool first. It is a daily-use personal drive focused on:

ease of use
resistance to storage provider failure
low storage cost
optional privacy protections

The system should allow storage across a mix of:

Baidu Netdisk
Aliyun Drive
S3-compatible object storage
local directories

The long-term vision is to expose the same core capabilities to desktop and mobile clients, while keeping storage control and metadata ownership in the user's hands.

2. Product Goals

2.1 Primary Goals

Provide a single virtual drive space across multiple heterogeneous storage backends.
Support daily file management, not only backup and restore.
Allow large media files and documents to coexist in one product.
Support on-demand cloud retrieval so files do not need to be fully cached locally.
Support preview-first workflows, especially for photos, videos, and PDFs.
Keep system-critical metadata under user control instead of delegating it to any one storage vendor.
Allow unstable or free providers to participate as secondary or cold-tier storage.

2.2 Priority Order

Ease of use
Resistance to provider shutdown or lock-in
Low cost
Privacy and encryption

3. Target Users and Usage Context

3.1 Primary User

a single user or household
has one always-on host running the gateway
accesses files from PCs and phones in a home network, with possible future remote access

3.2 Main Data Types

large video files
large photo libraries
documents
PDFs
mixed personal archives

3.3 Usage Style

browse files from a Web UI
upload from browser or client
preview before download when possible
let the system decide actual storage placement
tolerate heavier local infrastructure if it improves experience

4. Product Principles

Users see one logical drive, not multiple backend silos.
Storage placement is system-managed, not user-managed in normal workflows.
Metadata must remain recoverable even if one backend becomes unavailable.
Stable backends and unstable backends must be treated differently by policy.
Media experience should be optimized explicitly, not treated as a side effect of generic file storage.
The MVP should prefer understandable replication over complex erasure coding.

5. Scope Definition

5.1 In Scope for the Product

file browsing and directory navigation
upload and download
delete, rename, move, copy
on-demand retrieval from remote storage
local cache management
image preview
video preview via streaming
PDF preview
multi-backend storage placement
backend health monitoring
metadata backup and recovery tooling
background jobs for sync, repair, and replication

5.2 Out of Scope for MVP

real-time collaborative editing
multi-tenant enterprise permissions
public sharing links with strong abuse controls
fully offline-first desktop synchronization
true distributed peer-to-peer storage nodes
POSIX-complete filesystem semantics
erasure coding as a required storage strategy
full-text OCR and semantic search

6. System Context

6.1 Phase 1 Deployment Model

one self-hosted gateway process
one Web client
one SQLite database
one local cache directory
multiple configured storage backends

6.2 Future Deployment Model

gateway remains the control plane
desktop clients may add richer local sync and mount capabilities
mobile clients may provide browsing, preview, upload, and camera ingestion
metadata database may move from SQLite to a networked database if needed

7. Functional Requirements

The system must expose a single hierarchical directory tree.
The system must allow creating, renaming, moving, and deleting directories.
The system must allow listing files with pagination and sorting.
The system must show key metadata including name, size, type, timestamps, and cache status.
The system must decouple logical path from backend storage location.

7.2 Upload

The system must support browser-based upload.
The system must support resumable upload at least at the gateway boundary.
The system must support large-file upload without requiring full in-memory buffering.
The system must classify uploaded files by size and media type to choose storage strategy.
The system must allow asynchronous replication after initial successful ingestion.

7.3 Download

The system must allow direct file download from the Web UI.
The system must support on-demand fetch from remote backends if data is not in local cache.
The system must expose download progress and failure states.

7.4 Preview

The system must support image preview.
The system must support PDF preview.
The system must support video preview through HTTP range-compatible streaming.
The system should generate thumbnails for common media types.
The system should avoid full-file fetch when previewing large video files if the backend layout permits partial reads.

7.5 Cache Management

The gateway must maintain a local cache for hot files and preview artifacts.
The system must track cache state separately from logical file state.
The system must support configurable cache eviction policies.
The system should allow pinning specific files or directories for local retention in future versions.

7.6 Multi-Backend Storage

The system must support multiple backend types at once.
The system must support backend policies such as stable, opportunistic, and local.
The system must support at least one primary copy and optional secondary copies.
The system must track where each object or file replica is stored.
The system must detect backend unavailability and mark replicas accordingly.

7.7 Metadata and Recovery

The system must maintain its own metadata database independent of backend-native directory layouts.
The system must support export and backup of metadata.
The system must support integrity verification between metadata records and backend objects.
The system must support repair jobs that recreate missing replicas when another valid replica exists.

7.8 Jobs and Operations

The system must support background jobs for uploads, replication, thumbnail generation, and repair.
The system must expose job status in the UI or API.
The system must retain structured logs for user-visible failures.

7.9 Backend Management

The system must allow adding, updating, disabling, and deleting backend configurations.
The system must display backend status, capacity hints if available, and last health-check result.
The system should support backend weighting and placement rules.

8. Non-Functional Requirements

8.1 Usability

A user should not need to understand backend topology during normal file operations.
Common file operations should be understandable from a single Web interface.
Preview should be first-class for photo, video, and PDF content.

8.2 Reliability

Metadata must not depend on a single unstable third-party backend.
The system should tolerate temporary backend outages and recover later.
Background jobs must be restart-safe.

8.3 Performance

Listing a directory from metadata should not require live backend enumeration.
Upload and download pipelines should stream rather than fully buffer large files.
Preview start latency for cached or stable-backend media should be low enough for normal personal use.

8.4 Extensibility

Backend integrations must be pluggable.
Data model must allow migration from SQLite to another relational database.
API design must allow future desktop and mobile clients.

8.5 Security

Authentication is required for the Web UI and APIs.
Secrets for backend credentials must not be stored in plaintext in user-facing logs.
Encryption should be supported in design even if not mandatory for every backend path in MVP.

9. Storage Model Strategy

9.1 MVP Strategy

documents and small files are stored as whole-file objects
photos are stored as whole-file originals plus generated thumbnails
large videos may use chunked storage if needed for upload resilience and streamability
replication is preferred over erasure coding

9.2 Future Strategy

content-addressed chunks for large files
optional object packing for small files
deduplication across identical content
tier-aware migration policies
optional erasure coding for selected backend groups

10. Backend Policy Model

Each backend should have policy metadata, for example:

backend type
stability class
cost class
write priority
read priority
capacity hint
health status
credential state

Suggested policy classes:

stable: trusted for primary storage, such as S3 or a managed local directory on a reliable host
opportunistic: useful for secondary replicas or cold storage, such as free consumer cloud drives
local: fast cache-adjacent storage on the gateway host

11. MVP Definition

11.1 MVP Objective

Deliver a working single-user personal cloud drive that feels usable every day through a Web UI, with support for multi-backend storage, on-demand file retrieval, and media preview.

11.2 MVP Feature Set

Web authentication
file and directory browsing
upload, download, rename, move, delete
backend management for local directory, S3, Aliyun Drive, and Baidu Netdisk
SQLite metadata store
local cache directory
background job queue inside the gateway process
image preview and thumbnail generation
PDF preview
video preview through range-capable gateway streaming
replica tracking
backend health checks
metadata export
repair job for missing replicas

11.3 MVP Constraints

single-user only
one gateway node only
no desktop sync client yet
no mobile-native app yet
no public share links
no full-text search
no cross-file deduplication requirement
no erasure coding requirement

11.4 MVP Success Criteria

A user can upload files and browse them without caring which backend stores them.
A user can preview photos, PDFs, and many common videos from the browser.
A user can survive loss of one configured secondary backend without losing metadata control.
A user can inspect and retry failed background tasks.
A user can back up metadata and recover system state onto a new gateway host.

Current implementation status:

see mvp-status.md for the live gap analysis against this MVP definition
the current repository should be described as a Product MVP Candidate, not yet a polished release
after re-evaluation, the main remaining P0 items are Web UI and browser-facing auth/session UX
the backend now has persisted placement policy controls for file classes, preferred/excluded backends, and size-aware non-local replica caps
most remaining backend work has moved into P1 hardening

12. End-State Vision

12.1 Product Vision

Iron becomes a full personal storage control plane for heterogeneous storage providers, with one logical namespace, rich media workflows, policy-driven placement, and resilient metadata ownership.

12.2 End-State Capabilities

Web, desktop, and mobile clients
remote access support
selective offline sync
folder pinning
advanced media indexing and timeline views
content deduplication
optional end-to-end encryption modes
policy-based lifecycle management
automatic migration between hot, warm, and cold tiers
stronger repair and self-healing workflows
share links and fine-grained permissions
import and migration tools from common cloud drives
optional mount support on desktop systems

13. Version Roadmap

13.1 V0: Foundation

repository setup
gateway skeleton
metadata schema
local backend
basic Web UI shell

Current status:

mostly achieved for backend foundation
a browser-usable Web UI now exists, but it still needs real-user hardening, broader E2E coverage, and polish

13.2 V1: MVP

file operations
local cache
S3 backend
Aliyun backend
Baidu backend
preview pipeline
jobs and repair

Current interpretation:

the first shippable V1 should prioritize auth + Web UI + local + S3 + simple replication + metadata export
the backend side of local + S3 + simple replication + metadata recovery contract is already largely in place
Aliyun and Baidu remain in the roadmap, but are not required to call the first public version a usable MVP

13.3 V2: Usability Expansion

better backend policies
desktop client shell
remote access hardening
richer media views
pinning and offline support

13.4 V3: Advanced Storage Intelligence

deduplication
packed small-object storage
lifecycle movement
stronger repair orchestration
optional erasure-coded backend groups

14. Risks and Product Assumptions

14.1 Assumptions

The user accepts a heavy local gateway service.
Free or consumer cloud backends may be unstable and are therefore not trusted as the only source of critical metadata.
Web-first access is sufficient for MVP.

14.2 Risks

Consumer cloud providers may change APIs, rate limits, or anti-abuse rules.
Preview latency may be inconsistent for files that only exist on slow or unstable backends.
Browser upload behavior for very large files may require resumable protocols early.
Media streaming behavior can vary by codec and container format.

15. Open Questions for Technical Design

What auth model should the first gateway use: simple local account, reverse-proxy auth, or pluggable auth?
Should large video files use chunking from day one, or only after file size thresholds are exceeded?
Should backend adapters be implemented natively or through an abstraction layer such as rclone-compatible gateways?
How should local cache quotas and eviction policies be tuned?
What metadata export format should be treated as the recovery contract?
How much encryption should be mandatory in MVP versus optional by backend policy?

14 KiB Raw Blame History