# Iron Requirements ## 1. Product Overview `Iron` is a personal cloud drive system built around a self-hosted `gateway` service. Users access a unified file space through a Web application in the first phase. The gateway manages metadata, local cache, upload/download pipelines, preview streaming, and data placement across multiple storage backends. The product goal is not to be a backup tool first. It is a daily-use personal drive focused on: - ease of use - resistance to storage provider failure - low storage cost - optional privacy protections The system should allow storage across a mix of: - Baidu Netdisk - Aliyun Drive - S3-compatible object storage - local directories The long-term vision is to expose the same core capabilities to desktop and mobile clients, while keeping storage control and metadata ownership in the user's hands. ## 2. Product Goals ### 2.1 Primary Goals - Provide a single virtual drive space across multiple heterogeneous storage backends. - Support daily file management, not only backup and restore. - Allow large media files and documents to coexist in one product. - Support on-demand cloud retrieval so files do not need to be fully cached locally. - Support preview-first workflows, especially for photos, videos, and PDFs. - Keep system-critical metadata under user control instead of delegating it to any one storage vendor. - Allow unstable or free providers to participate as secondary or cold-tier storage. ### 2.2 Priority Order 1. Ease of use 2. Resistance to provider shutdown or lock-in 3. Low cost 4. Privacy and encryption ## 3. Target Users and Usage Context ### 3.1 Primary User - a single user or household - has one always-on host running the gateway - accesses files from PCs and phones in a home network, with possible future remote access ### 3.2 Main Data Types - large video files - large photo libraries - documents - PDFs - mixed personal archives ### 3.3 Usage Style - browse files from a Web UI - upload from browser or client - preview before download when possible - let the system decide actual storage placement - tolerate heavier local infrastructure if it improves experience ## 4. Product Principles - Users see one logical drive, not multiple backend silos. - Storage placement is system-managed, not user-managed in normal workflows. - Metadata must remain recoverable even if one backend becomes unavailable. - Stable backends and unstable backends must be treated differently by policy. - Media experience should be optimized explicitly, not treated as a side effect of generic file storage. - The MVP should prefer understandable replication over complex erasure coding. ## 5. Scope Definition ### 5.1 In Scope for the Product - file browsing and directory navigation - upload and download - delete, rename, move, copy - on-demand retrieval from remote storage - local cache management - image preview - video preview via streaming - PDF preview - multi-backend storage placement - backend health monitoring - metadata backup and recovery tooling - background jobs for sync, repair, and replication ### 5.2 Out of Scope for MVP - real-time collaborative editing - multi-tenant enterprise permissions - public sharing links with strong abuse controls - fully offline-first desktop synchronization - true distributed peer-to-peer storage nodes - POSIX-complete filesystem semantics - erasure coding as a required storage strategy - full-text OCR and semantic search ## 6. System Context ### 6.1 Phase 1 Deployment Model - one self-hosted gateway process - one Web client - one SQLite database - one local cache directory - multiple configured storage backends ### 6.2 Future Deployment Model - gateway remains the control plane - desktop clients may add richer local sync and mount capabilities - mobile clients may provide browsing, preview, upload, and camera ingestion - metadata database may move from SQLite to a networked database if needed ## 7. Functional Requirements ### 7.1 File Space and Navigation - The system must expose a single hierarchical directory tree. - The system must allow creating, renaming, moving, and deleting directories. - The system must allow listing files with pagination and sorting. - The system must show key metadata including name, size, type, timestamps, and cache status. - The system must decouple logical path from backend storage location. ### 7.2 Upload - The system must support browser-based upload. - The system must support resumable upload at least at the gateway boundary. - The system must support large-file upload without requiring full in-memory buffering. - The system must classify uploaded files by size and media type to choose storage strategy. - The system must allow asynchronous replication after initial successful ingestion. ### 7.3 Download - The system must allow direct file download from the Web UI. - The system must support on-demand fetch from remote backends if data is not in local cache. - The system must expose download progress and failure states. ### 7.4 Preview - The system must support image preview. - The system must support PDF preview. - The system must support video preview through HTTP range-compatible streaming. - The system should generate thumbnails for common media types. - The system should avoid full-file fetch when previewing large video files if the backend layout permits partial reads. ### 7.5 Cache Management - The gateway must maintain a local cache for hot files and preview artifacts. - The system must track cache state separately from logical file state. - The system must support configurable cache eviction policies. - The system should allow pinning specific files or directories for local retention in future versions. ### 7.6 Multi-Backend Storage - The system must support multiple backend types at once. - The system must support backend policies such as `stable`, `opportunistic`, and `local`. - The system must support at least one primary copy and optional secondary copies. - The system must track where each object or file replica is stored. - The system must detect backend unavailability and mark replicas accordingly. ### 7.7 Metadata and Recovery - The system must maintain its own metadata database independent of backend-native directory layouts. - The system must support export and backup of metadata. - The system must support integrity verification between metadata records and backend objects. - The system must support repair jobs that recreate missing replicas when another valid replica exists. ### 7.8 Jobs and Operations - The system must support background jobs for uploads, replication, thumbnail generation, and repair. - The system must expose job status in the UI or API. - The system must retain structured logs for user-visible failures. ### 7.9 Backend Management - The system must allow adding, updating, disabling, and deleting backend configurations. - The system must display backend status, capacity hints if available, and last health-check result. - The system should support backend weighting and placement rules. ## 8. Non-Functional Requirements ### 8.1 Usability - A user should not need to understand backend topology during normal file operations. - Common file operations should be understandable from a single Web interface. - Preview should be first-class for photo, video, and PDF content. ### 8.2 Reliability - Metadata must not depend on a single unstable third-party backend. - The system should tolerate temporary backend outages and recover later. - Background jobs must be restart-safe. ### 8.3 Performance - Listing a directory from metadata should not require live backend enumeration. - Upload and download pipelines should stream rather than fully buffer large files. - Preview start latency for cached or stable-backend media should be low enough for normal personal use. ### 8.4 Extensibility - Backend integrations must be pluggable. - Data model must allow migration from SQLite to another relational database. - API design must allow future desktop and mobile clients. ### 8.5 Security - Authentication is required for the Web UI and APIs. - Secrets for backend credentials must not be stored in plaintext in user-facing logs. - Encryption should be supported in design even if not mandatory for every backend path in MVP. ## 9. Storage Model Strategy ### 9.1 MVP Strategy - documents and small files are stored as whole-file objects - photos are stored as whole-file originals plus generated thumbnails - large videos may use chunked storage if needed for upload resilience and streamability - replication is preferred over erasure coding ### 9.2 Future Strategy - content-addressed chunks for large files - optional object packing for small files - deduplication across identical content - tier-aware migration policies - optional erasure coding for selected backend groups ## 10. Backend Policy Model Each backend should have policy metadata, for example: - backend type - stability class - cost class - write priority - read priority - capacity hint - health status - credential state Suggested policy classes: - `stable`: trusted for primary storage, such as S3 or a managed local directory on a reliable host - `opportunistic`: useful for secondary replicas or cold storage, such as free consumer cloud drives - `local`: fast cache-adjacent storage on the gateway host ## 11. MVP Definition ### 11.1 MVP Objective Deliver a working single-user personal cloud drive that feels usable every day through a Web UI, with support for multi-backend storage, on-demand file retrieval, and media preview. ### 11.2 MVP Feature Set - Web authentication - file and directory browsing - upload, download, rename, move, delete - backend management for local directory, S3, Aliyun Drive, and Baidu Netdisk - SQLite metadata store - local cache directory - background job queue inside the gateway process - image preview and thumbnail generation - PDF preview - video preview through range-capable gateway streaming - replica tracking - backend health checks - metadata export - repair job for missing replicas ### 11.3 MVP Constraints - single-user only - one gateway node only - no desktop sync client yet - no mobile-native app yet - no public share links - no full-text search - no cross-file deduplication requirement - no erasure coding requirement ### 11.4 MVP Success Criteria - A user can upload files and browse them without caring which backend stores them. - A user can preview photos, PDFs, and many common videos from the browser. - A user can survive loss of one configured secondary backend without losing metadata control. - A user can inspect and retry failed background tasks. - A user can back up metadata and recover system state onto a new gateway host. Current implementation status: - see [mvp-status.md](/Users/bytedance/iron/docs/mvp-status.md:1) for the live gap analysis against this MVP definition - the current repository should be described as a `Product MVP Candidate`, not yet a polished release - after re-evaluation, the main remaining `P0` items are Web UI and browser-facing auth/session UX - the backend now has persisted placement policy controls for file classes, preferred/excluded backends, and size-aware non-local replica caps - most remaining backend work has moved into `P1 hardening` ## 12. End-State Vision ### 12.1 Product Vision Iron becomes a full personal storage control plane for heterogeneous storage providers, with one logical namespace, rich media workflows, policy-driven placement, and resilient metadata ownership. ### 12.2 End-State Capabilities - Web, desktop, and mobile clients - remote access support - selective offline sync - folder pinning - advanced media indexing and timeline views - content deduplication - optional end-to-end encryption modes - policy-based lifecycle management - automatic migration between hot, warm, and cold tiers - stronger repair and self-healing workflows - share links and fine-grained permissions - import and migration tools from common cloud drives - optional mount support on desktop systems ## 13. Version Roadmap ### 13.1 V0: Foundation - repository setup - gateway skeleton - metadata schema - local backend - basic Web UI shell Current status: - mostly achieved for backend foundation - a browser-usable Web UI now exists, but it still needs real-user hardening, broader E2E coverage, and polish ### 13.2 V1: MVP - file operations - local cache - S3 backend - Aliyun backend - Baidu backend - preview pipeline - jobs and repair Current interpretation: - the first shippable V1 should prioritize `auth + Web UI + local + S3 + simple replication + metadata export` - the backend side of `local + S3 + simple replication + metadata recovery contract` is already largely in place - Aliyun and Baidu remain in the roadmap, but are not required to call the first public version a usable MVP ### 13.3 V2: Usability Expansion - better backend policies - desktop client shell - remote access hardening - richer media views - pinning and offline support ### 13.4 V3: Advanced Storage Intelligence - deduplication - packed small-object storage - lifecycle movement - stronger repair orchestration - optional erasure-coded backend groups ## 14. Risks and Product Assumptions ### 14.1 Assumptions - The user accepts a heavy local gateway service. - Free or consumer cloud backends may be unstable and are therefore not trusted as the only source of critical metadata. - Web-first access is sufficient for MVP. ### 14.2 Risks - Consumer cloud providers may change APIs, rate limits, or anti-abuse rules. - Preview latency may be inconsistent for files that only exist on slow or unstable backends. - Browser upload behavior for very large files may require resumable protocols early. - Media streaming behavior can vary by codec and container format. ## 15. Open Questions for Technical Design - What auth model should the first gateway use: simple local account, reverse-proxy auth, or pluggable auth? - Should large video files use chunking from day one, or only after file size thresholds are exceeded? - Should backend adapters be implemented natively or through an abstraction layer such as rclone-compatible gateways? - How should local cache quotas and eviction policies be tuned? - What metadata export format should be treated as the recovery contract? - How much encryption should be mandatory in MVP versus optional by backend policy?