diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..548c01d --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,82 @@ +# AGENTS.md + +Guidance for coding agents and automated contributors working in this repository. + +## Project Snapshot + +Iron is a self-hosted personal cloud drive gateway with: + +- FastAPI backend under `app/` +- Vite + React + TypeScript frontend under `frontend/` +- built frontend assets served from `app/web/dist/` +- Alembic migrations under `alembic/` +- tests under `tests/` +- project docs under `docs/` + +The current stage is **Product MVP Candidate**. Preserve the existing +desktop-first drive UX and avoid drifting back toward a generic admin dashboard. + +## Required Setup + +Use the local virtualenv and `uv` cache: + +```bash +UV_CACHE_DIR=.uv-cache uv venv .venv +UV_CACHE_DIR=.uv-cache uv pip install --python .venv/bin/python -e '.[dev]' +npm install +``` + +## Common Commands + +```bash +npm run build +.venv/bin/python -m pytest +.venv/bin/python scripts/ui_playwright_smoke.py +.venv/bin/python -m pytest tests/e2e/test_web_ui_playwright.py +``` + +## Development Rules + +- Keep runtime artifacts out of Git: `.venv/`, `node_modules/`, `output/`, + `.iron-storage/`, `.iron-temp/`, caches, and local databases. +- Add or update tests in the same change as behavior changes. +- For UI work, run the real-service Playwright flow and review screenshots. +- Do not use naked `/api/files/...` media URLs for browser preview/download. + Fetch file content with authorization, create an object URL, and revoke it. +- Keep docs current when changing setup, architecture, test counts, or user flows. +- Prefer small, focused changes over broad rewrites. + +## UI Quality Standard + +For any meaningful Web UI change: + +1. build frontend assets with `npm run build` +2. run `./.venv/bin/python scripts/ui_playwright_smoke.py` +3. inspect screenshots in `output/playwright/` +4. fix layout or interaction issues found in screenshots +5. rerun the Playwright flow + +See [docs/development.md](docs/development.md). + +## Documentation Expectations + +- `README.md` should stay user-facing and concise. +- `CONTRIBUTING.md` should describe contributor workflow. +- `docs/README.md` should index deeper docs. +- `docs/development.md` should describe test and UI validation rules. +- Avoid local absolute paths in committed Markdown. + +## Verification Before Finishing + +At minimum, run: + +```bash +npm run build +.venv/bin/python -m pytest +``` + +If UI behavior changed, also run: + +```bash +.venv/bin/python scripts/ui_playwright_smoke.py +``` diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..d204878 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,62 @@ +# Contributing + +Thanks for helping improve Iron. The project is still early, so the most useful +contributions are focused changes with tests and clear documentation updates. + +## Development Setup + +```bash +UV_CACHE_DIR=.uv-cache uv venv .venv +UV_CACHE_DIR=.uv-cache uv pip install --python .venv/bin/python -e '.[dev]' +npm install +npm run build +``` + +Run the API locally: + +```bash +.venv/bin/python -m uvicorn app.main:app --reload +``` + +Open `http://127.0.0.1:8000/app`. + +## Test Before Submitting + +Run the full suite: + +```bash +.venv/bin/python -m pytest +``` + +For UI changes, also run: + +```bash +npm run build +.venv/bin/python scripts/ui_playwright_smoke.py +``` + +Review screenshots in `output/playwright/` before considering UI work complete. + +## Pull Request Checklist + +- The change is focused and described clearly. +- Tests were added or updated for behavior changes. +- UI changes were validated with Playwright when applicable. +- Documentation was updated if setup, behavior, or architecture changed. +- Generated/runtime files are not included. + +## Code Style + +- Backend code lives in `app/` and uses async FastAPI + SQLAlchemy patterns. +- Frontend code lives in `frontend/` and uses React, React Router, and + TanStack Query. +- Keep browser file preview/download flows authenticated through the API client. +- Prefer clear names and small modules over clever abstractions. + +## Project Status + +Iron is a Product MVP Candidate. See: + +- [docs/product.md](docs/product.md) +- [docs/architecture.md](docs/architecture.md) +- [docs/development.md](docs/development.md) diff --git a/README.md b/README.md index 784b9f6..e5f384e 100644 --- a/README.md +++ b/README.md @@ -1,44 +1,54 @@ # Iron -Personal cloud drive gateway and Web app. +Iron is a self-hosted personal cloud drive gateway. It provides one logical file +namespace over local storage and pluggable backends, with a browser UI for daily +file management and operational visibility. -## Current Status +The project is currently a **Product MVP Candidate**: the core backend and Web +app exist, but the release still needs hardening, broader browser regression +coverage, and more real-world usage. -- product requirements drafted in `docs/` -- technical architecture drafted in `docs/` -- Python backend foundation implemented -- product Web UI implemented with `Vite + React + TypeScript` -- local auth, upload, local persistence, S3 reconciliation, policy-driven replica placement, declarative background jobs, full-system reconcile, preview artifact generation, metadata export/validate/integrity/import, and a product-oriented browser app are available -- the current Web app includes files, uploads, recycle bin, storage, and jobs pages with a desktop-first drive layout -- authenticated download and inline preview flows are covered by Python Playwright E2E tests -- current automated baseline: `66` passing tests -- next primary implementation focus should be hardening the product experience through real-browser regression, not broad backend expansion +## Features -## Backend +- FastAPI backend with SQLite, SQLAlchemy, and Alembic migrations +- Local authentication with persisted bearer-token sessions +- Directory and file operations: browse, create folder, rename, move, delete, + restore, download, and preview +- `tus` upload flow with local object persistence +- Desktop-first React Web app at `/app` +- Storage backend management for local storage and S3 runtime paths +- Placement policies, reconcile jobs, health checks, and retryable background jobs +- Metadata export, validation, integrity checks, restore plans, and guarded import +- Python Playwright end-to-end coverage for core browser flows -Create the local environment with `uv`: +## Quick Start + +Iron uses `uv` for local Python environment setup. ```bash UV_CACHE_DIR=.uv-cache uv venv .venv UV_CACHE_DIR=.uv-cache uv pip install --python .venv/bin/python -e '.[dev]' -``` - -Run the API locally: - -```bash +npm install +npm run build .venv/bin/python -m uvicorn app.main:app --reload ``` -Open the current browser app at: +Open the Web app: -- [http://127.0.0.1:8000/app](http://127.0.0.1:8000/app) +```text +http://127.0.0.1:8000/app +``` Default local bootstrap credentials: -- username: `admin` -- password: `changeme-iron` +```text +username: admin +password: changeme-iron +``` + +## Configuration -Environment overrides: +Common environment variables: - `IRON_DATABASE_URL` - `IRON_BOOTSTRAP_USERNAME` @@ -50,12 +60,18 @@ Environment overrides: - `IRON_JOB_POLL_INTERVAL_SECONDS` - `IRON_JOB_BATCH_SIZE` -Restore safety: +Runtime data is intentionally ignored by Git. By default, local development may +create `.iron-storage/`, `.iron-temp/`, `iron.db`, and `output/`. + +## Development -- metadata import requires `confirm_replace=true` -- metadata import also requires a fresh validation token from `POST /api/exports/metadata/restore-plan` +Build the frontend: + +```bash +npm run build +``` -## Tests +Run the full test suite: ```bash .venv/bin/python -m pytest @@ -63,9 +79,11 @@ Restore safety: Current expected result: -- `66 passed` +```text +66 passed +``` -Run the browser E2E flow directly with: +Run the browser E2E flow directly: ```bash .venv/bin/python scripts/ui_playwright_smoke.py @@ -77,23 +95,45 @@ Or through pytest: .venv/bin/python -m pytest tests/e2e/test_web_ui_playwright.py ``` -## Implemented Foundations +Playwright screenshots and runtime artifacts are written to `output/playwright/`. + +## Repository Layout + +```text +app/ FastAPI application, services, repositories, schemas +alembic/ Database migrations +frontend/ Vite + React + TypeScript source +app/web/dist/ Built frontend assets served by FastAPI +scripts/ Local automation and E2E scripts +tests/ Backend, API, and browser E2E tests +docs/ Product, architecture, API, and maintenance docs +``` + +## Documentation + +Start with: + +- [Documentation index](docs/README.md) +- [Product](docs/product.md) +- [Architecture](docs/architecture.md) +- [API](docs/api.md) +- [Development](docs/development.md) + +For contribution workflow, see [CONTRIBUTING.md](CONTRIBUTING.md). +For AI/coding-agent guidance, see [AGENTS.md](AGENTS.md). + +## Security Notes + +Iron is not yet a hardened public Internet service. Treat the current credentials +and local bearer-token session model as development-oriented defaults. Change the +bootstrap password and review deployment boundaries before exposing the service. -- FastAPI application bootstrap -- async SQLAlchemy engine setup -- initial ORM entity definitions -- Alembic migrations -- local auth bootstrap and bearer-token sessions -- React Web app at `/app` -- authenticated browser download and preview using bearer-token-backed blob fetches -- local + S3 runtime storage foundation -- persisted placement policy controls and placement preview API -- SQLite-backed job records, declarative reconcile flows, and metadata export/validate/integrity/import APIs -- health and readiness endpoints -- baseline automated tests +Metadata import is guarded and requires both: -## Handoff +- `confirm_replace=true` +- a fresh validation token from `POST /api/exports/metadata/restore-plan` -For the most useful handoff view before continuing development, start here: +## License -- [docs/handoff.md](/Users/bytedance/iron/docs/handoff.md:1) +No open source license has been selected yet. Add a `LICENSE` file before +publishing or accepting external contributions. diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..f133d6f --- /dev/null +++ b/docs/README.md @@ -0,0 +1,14 @@ +# Documentation + +This directory contains the stable project documentation for Iron. + +- [Product](product.md): product scope, current stage, MVP cutline, and roadmap. +- [Architecture](architecture.md): backend, frontend, data model, storage, and job design. +- [API](api.md): route map and API groups. +- [Development](development.md): setup, test commands, Playwright workflow, and documentation rules. + +Root-level project docs: + +- [README](../README.md): user-facing overview and quick start. +- [Contributing](../CONTRIBUTING.md): contributor workflow. +- [Agents](../AGENTS.md): coding-agent and automation guidance. diff --git a/docs/api-design.md b/docs/api-design.md deleted file mode 100644 index 9dc2941..0000000 --- a/docs/api-design.md +++ /dev/null @@ -1,726 +0,0 @@ -# Iron API Design - -## 1. Overview - -This document defines the MVP API surface for the Iron gateway. - -Principles: - -- REST-first -- browser-first product usage, with backend auth that can evolve -- `tus` for resumable upload transport -- stable logical file identifiers -- backend topology hidden from normal users - -Base prefix: - -- `/api` - -Non-API upload transport prefix: - -- `/files` - -Notes: - -- `tus` endpoints intentionally live outside the normal JSON API shape because they follow protocol-specific request and response semantics. -- All timestamps should use ISO 8601 UTC strings. -- All entity IDs should use UUIDv7 or another sortable unique ID format. - -## 2. Authentication - -MVP auth model: - -- single local user -- bearer token session for the current backend implementation -- Web UI can later wrap this with cookie-based session handling if desired - -### 2.1 Login - -- `POST /api/auth/login` - -Request: - -```json -{ - "username": "admin", - "password": "secret" -} -``` - -Response: - -```json -{ - "access_token": "iron_xxx", - "token_type": "bearer", - "expires_at": "2026-04-20T00:00:00Z", - "user": { - "id": "usr_01", - "username": "admin" - } -} -``` - -Behavior: - -- implemented in current codebase -- creates a persisted auth session and returns a bearer token -- returns `401` for invalid credentials - -### 2.2 Logout - -- `POST /api/auth/logout` - -Response: - -```json -{ - "success": true -} -``` - -Behavior: - -- implemented in current codebase -- revokes the current bearer-token session - -### 2.3 Current Session - -- `GET /api/auth/me` - -Response: - -```json -{ - "user": { - "id": "usr_01", - "username": "admin" - } -} -``` - -Behavior: - -- implemented in current codebase -- requires `Authorization: Bearer ` - -## 3. Directories - -Route protection: - -- all directory, file, upload, and backend routes require authentication in the current codebase - -### 3.1 List Directory Children - -- `GET /api/directories/{directory_id}/children` - -Query params: - -- `cursor` -- `limit` -- `sort` -- `order` - -Response: - -```json -{ - "directory": { - "id": "dir_root", - "name": "/", - "parent_id": null - }, - "items": [ - { - "kind": "directory", - "id": "dir_01", - "name": "photos", - "created_at": "2026-04-13T00:00:00Z", - "updated_at": "2026-04-13T00:00:00Z" - }, - { - "kind": "file", - "id": "fil_01", - "name": "movie.mp4", - "mime_type": "video/mp4", - "size_bytes": 1048576, - "cache_status": "partial", - "preview_status": "ready", - "created_at": "2026-04-13T00:00:00Z", - "updated_at": "2026-04-13T00:00:00Z" - } - ], - "next_cursor": null -} -``` - -### 3.2 Create Directory - -- `POST /api/directories` - -Request: - -```json -{ - "parent_id": "dir_root", - "name": "photos" -} -``` - -Implemented status: - -- implemented in current codebase -- currently supports create under an existing parent directory -- duplicate sibling names are rejected - -### 3.3 Rename Directory - -- `POST /api/directories/{directory_id}/rename` - -Request: - -```json -{ - "name": "vacation-photos" -} -``` - -Implemented status: - -- implemented in current codebase - -### 3.4 Move Directory - -- `POST /api/directories/{directory_id}/move` - -Request: - -```json -{ - "target_parent_id": "dir_02" -} -``` - -Implemented status: - -- implemented in current codebase -- moving a directory updates descendant logical path keys - -### 3.5 Delete Directory - -- `DELETE /api/directories/{directory_id}` - -Behavior: - -- soft delete in MVP -- reject if non-empty unless `recursive=true` is explicitly supported later - -Current implementation status: - -- `GET /api/directories/{directory_id}/children` is implemented -- current response can contain both directory items and file items -- items are currently returned with directories first, then files - -## 4. Files - -### 4.1 Get File Details - -- `GET /api/files/{file_id}` - -Response: - -```json -{ - "file": { - "id": "fil_01", - "directory_id": "dir_root", - "name": "movie.mp4", - "mime_type": "video/mp4", - "size_bytes": 1048576, - "current_version_id": "ver_01", - "preview_status": "ready", - "cache_status": "partial", - "created_at": "2026-04-13T00:00:00Z", - "updated_at": "2026-04-13T00:00:00Z" - }, - "preview_artifacts": [ - { - "id": "prv_01", - "artifact_type": "thumbnail", - "blob_id": "blb_01", - "status": "ready" - } - ] -} -``` - -Implemented status: - -- implemented in current codebase -- returns logical file metadata from the gateway database -- now includes preview artifact metadata when background generation has run - -### 4.2 Rename File - -- `POST /api/files/{file_id}/rename` - -Request: - -```json -{ - "name": "movie-final.mp4" -} -``` - -Implemented status: - -- implemented in current codebase - -### 4.3 Move File - -- `POST /api/files/{file_id}/move` - -Request: - -```json -{ - "target_parent_id": "dir_02" -} -``` - -Implemented status: - -- implemented in current codebase - -### 4.4 Delete File - -- `DELETE /api/files/{file_id}` - -Behavior: - -- current implementation moves the file to a recycle-bin state -- deleted files disappear from normal directory listings -- deleted files can be listed and restored through recycle-bin APIs - -### 4.5 Download File - -- `GET /api/files/{file_id}/download` - -Behavior: - -- returns attachment response -- gateway resolves best available replica and streams to client - -Implemented status: - -- implemented in current codebase for files persisted to the default local backend -- current implementation resolves the ready local replica and serves it directly - -### 4.6 Stream File - -- `GET /api/files/{file_id}/stream` - -Behavior: - -- supports `Range` requests -- intended for video and browser-native preview flows - -Implemented status: - -- implemented in current codebase for local persisted objects -- supports full reads and single-range byte requests - -### 4.7 File Thumbnail - -- `GET /api/files/{file_id}/thumbnail` - -Behavior: - -- returns preview derivative if present -- returns `404` if not yet available - -### 4.7a File Preview - -- `GET /api/files/{file_id}/preview` - -Behavior: - -- returns an inline response for preview-supported file types - -Implemented status: - -- implemented in current codebase for images and PDFs persisted to the default local backend -- unsupported media types currently return a validation-style error - -### 4.8 Recycle Bin - -- `GET /api/files/recycle-bin` -- `POST /api/files/{file_id}/restore` - -Current implementation status: - -- both endpoints are implemented -- restore returns a conflict if the original directory now contains an active item with the same file name - -## 5. Tus Upload Flow - -Iron uses `tus` as the upload transport and adds one gateway-specific finalize step. - -### 5.1 Create Upload Resource - -- `POST /files` - -Required headers: - -- `Tus-Resumable` -- `Upload-Length` -- `Upload-Metadata` - -Behavior: - -- creates a new upload session -- returns `201` with `Location` - -Implemented status: - -- implemented in current codebase -- currently supports sequential append uploads with declared total length -- returns a gateway-managed upload resource under `/files/{upload_id}` - -### 5.2 Query Upload Offset - -- `HEAD /files/{upload_id}` - -Behavior: - -- returns current `Upload-Offset` - -Implemented status: - -- implemented in current codebase - -### 5.3 Upload Bytes - -- `PATCH /files/{upload_id}` - -Required headers: - -- `Tus-Resumable` -- `Upload-Offset` -- `Content-Type: application/offset+octet-stream` - -Behavior: - -- appends bytes to temporary ingest file -- updates upload progress - -Implemented status: - -- implemented in current codebase -- currently requires exact offset matching and sequential writes - -### 5.4 Finalize Upload - -- `POST /api/uploads/{upload_id}/finalize` - -Request: - -```json -{ - "directory_id": "dir_root", - "filename": "movie.mp4" -} -``` - -Response: - -```json -{ - "file": { - "id": "fil_01", - "name": "movie.mp4" - }, - "upload": { - "id": "upl_01", - "status": "finalized" - } -} -``` - -Behavior: - -- verifies upload completion -- computes content metadata -- creates file and file version records -- performs first durable write -- only exposes the logical file after minimum durability is met - -Implemented status: - -- implemented in current codebase -- currently finalizes into metadata records and persists the blob into the default local backend -- creates a `blob_replicas` record for the local persisted object -- enqueues replication jobs for enabled non-local backends such as `s3` - -## 6. Backends - -### 6.1 List Backends - -- `GET /api/backends` - -### 6.2 Create Backend - -- `POST /api/backends` - -Request example: - -```json -{ - "name": "local-main", - "type": "local", - "stability_class": "local", - "read_priority": 100, - "write_priority": 100, - "config": { - "base_path": "/srv/iron/storage" - } -} -``` - -### 6.3 Update Backend - -- `PATCH /api/backends/{backend_id}` - -### 6.4 Check Backend Health - -- `POST /api/backends/{backend_id}/check` - -Response: - -```json -{ - "backend_id": "bkd_01", - "status": "healthy", - "checked_at": "2026-04-13T00:00:00Z" -} -``` - -### 6.5 Disable Backend - -- `POST /api/backends/{backend_id}/disable` - -## 7. Placement Policies - -### 7.1 List Placement Policies - -- `GET /api/policies/placement` - -Behavior: - -- implemented in current codebase -- returns file-class placement policy rows -- current policy shape includes: - - `require_local` - - `stable_replica_count` - - `opportunistic_replica_count` - - ordered `preferred_backend_ids` - - explicit `excluded_backend_ids` - - optional `max_non_local_size_bytes` - -### 7.2 Upsert Placement Policy - -- `PUT /api/policies/placement/{file_class}` - -Request example: - -```json -{ - "require_local": true, - "stable_replica_count": 1, - "opportunistic_replica_count": 1, - "preferred_backend_ids": ["bkd_s3_primary", "bkd_s3_archive"], - "excluded_backend_ids": ["bkd_s3_legacy"], - "max_non_local_size_bytes": 104857600 -} -``` - -Behavior: - -- implemented in current codebase -- validates that referenced backend ids exist -- rejects overlap between preferred and excluded backend ids -- rejects negative replica counts - -### 7.3 Preview Placement Decision - -- `GET /api/policies/placement/preview` - -Query params: - -- `mime_type` -- `size_bytes` - -Behavior: - -- implemented in current codebase -- returns the matched file class, active policy, and selected backends for the given mime type and optional size -- marks whether non-local replicas are currently allowed under the policy - -## 8. Jobs - -### 8.1 List Jobs - -- `GET /api/jobs` - -Query params: - -- `kind` -- `status` -- `limit` - -### 8.2 Get Job - -- `GET /api/jobs/{job_id}` - -### 8.3 Retry Job - -- `POST /api/jobs/{job_id}/retry` - -### 8.4 Run Pending Jobs - -- `POST /api/jobs/run-pending` - -### 8.5 Enqueue Backend Health Checks - -- `POST /api/jobs/enqueue-health-checks` - -### 8.6 Enqueue Full Reconcile - -- `POST /api/jobs/enqueue-full-reconcile` - -## 9. Exports - -### 9.1 Metadata Export - -- `GET /api/exports/metadata` - -### 9.2 Metadata Import - -- `POST /api/exports/metadata/import` - -Notes: - -- current backend requires `confirm_replace=true` for destructive import -- current backend also requires a fresh `validation_token` from `POST /api/exports/metadata/restore-plan` -- successful import currently schedules follow-up reconcile and backend health-check jobs - -## 9.3 Reconciliation - -- `POST /api/files/{file_id}/reconcile` - -Implemented status: - -- implemented in current codebase -- the backend intention is "make this file converge to policy-defined replica state" - -## 9.4 Metadata Validation and Integrity - -- `POST /api/exports/metadata/validate` -- `GET /api/exports/metadata/integrity` - -## 9.5 Restore Plan - -- `POST /api/exports/metadata/restore-plan` - -## 10. System - -### 10.1 Health Check - -- `GET /api/system/health` - -Response: - -```json -{ - "status": "ok" -} -``` - -### 10.2 Readiness - -- `GET /api/system/ready` - -Response: - -```json -{ - "status": "ready", - "database": "ok" -} -``` - -## 10.3 Backends - -- `GET /api/backends` -- `POST /api/backends` -- `POST /api/backends/{backend_id}/check` -- `POST /api/backends/{backend_id}/disable` - -Implemented status: - -- implemented in current codebase -- supports local and s3 backend configuration metadata -- local and s3 backend health checks are implemented - -Additional implemented status: - -- `POST /api/jobs/run-pending` is implemented in current codebase -- `POST /api/jobs/enqueue-health-checks` is implemented in current codebase -- `GET /api/exports/metadata` is implemented in current codebase -- `POST /api/exports/metadata/validate` is implemented in current codebase -- `GET /api/exports/metadata/integrity` is implemented in current codebase -- `POST /api/exports/metadata/import` is implemented in current codebase -- `POST /api/files/{file_id}/reconcile` is implemented in current codebase - -## 11. Error Shape - -All JSON API errors should use one consistent envelope: - -```json -{ - "error": { - "code": "file_not_found", - "message": "File does not exist.", - "details": null - } -} -``` - -Suggested error codes: - -- `unauthorized` -- `forbidden` -- `validation_error` -- `directory_not_found` -- `file_not_found` -- `upload_not_found` -- `backend_unavailable` -- `backend_config_invalid` -- `conflict` -- `internal_error` - -## 12. Versioning Approach - -MVP can stay unversioned internally if the API is changing quickly. - -Recommendation: - -- keep `/api` in code -- be prepared to introduce `/api/v1` before external client surface becomes stable - -## 13. Implementation Notes - -- `tus` transport handlers should be separated from normal JSON routers. -- Finalization logic must stay in application services, not inside protocol handlers. -- Download and stream endpoints should rely on the same replica resolution service. -- Avoid exposing backend IDs in normal file responses unless operationally useful. diff --git a/docs/api.md b/docs/api.md new file mode 100644 index 0000000..81a6770 --- /dev/null +++ b/docs/api.md @@ -0,0 +1,84 @@ +# API + +Iron exposes JSON APIs under `/api` and `tus` upload transport under `/files`. + +Most `/api` routes require: + +```text +Authorization: Bearer +``` + +## Auth + +- `POST /api/auth/login` +- `POST /api/auth/logout` +- `GET /api/auth/me` + +## Directories + +- `GET /api/directories/{directory_id}/children` +- `POST /api/directories` +- `POST /api/directories/{directory_id}/rename` +- `POST /api/directories/{directory_id}/move` + +## Files + +- `GET /api/files/{file_id}` +- `GET /api/files/{file_id}/download` +- `GET /api/files/{file_id}/preview` +- `GET /api/files/{file_id}/stream` +- `POST /api/files/{file_id}/rename` +- `POST /api/files/{file_id}/move` +- `DELETE /api/files/{file_id}` +- `POST /api/files/{file_id}/restore` +- `GET /api/files/recycle-bin` + +## Uploads + +Protocol endpoints: + +- `POST /files` +- `HEAD /files/{upload_id}` +- `PATCH /files/{upload_id}` + +Finalize endpoint: + +- `POST /api/uploads/{upload_id}/finalize` + +## Backends + +- `GET /api/backends` +- `POST /api/backends` +- `POST /api/backends/{backend_id}/check` +- `POST /api/backends/{backend_id}/disable` + +## Jobs + +- `GET /api/jobs` +- `GET /api/jobs/{job_id}` +- `POST /api/jobs/{job_id}/retry` +- `POST /api/jobs/run-pending` +- `POST /api/jobs/enqueue-health-checks` +- `POST /api/jobs/enqueue-full-reconcile` + +## Placement Policies + +- `GET /api/policies/placement` +- `PUT /api/policies/placement/{file_class}` +- `GET /api/policies/placement/preview` + +## Metadata Export And Recovery + +- `GET /api/exports/metadata` +- `POST /api/exports/metadata/validate` +- `GET /api/exports/metadata/integrity` +- `POST /api/exports/metadata/restore-plan` +- `POST /api/exports/metadata/import` + +Metadata import requires `confirm_replace=true` and a validation token from the +restore-plan endpoint. + +## System + +- `GET /api/system/health` +- `GET /api/system/ready` diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..142660c --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,122 @@ +# Architecture + +Iron is an async Python modular monolith with a React single-page Web app. + +## System Overview + +```text +Browser UI + | +FastAPI app + | +Services + | +Repositories + | +SQLite metadata database + | +Storage adapters: local, S3 +``` + +## Backend + +Core stack: + +- FastAPI +- SQLAlchemy 2.x async ORM +- SQLite via `aiosqlite` +- Alembic migrations +- durable in-process jobs stored in SQLite + +Important packages: + +- `app/api/`: route handlers and dependency wiring +- `app/services/`: business logic +- `app/repositories/`: database access +- `app/models/`: SQLAlchemy entities +- `app/schemas/`: API schemas +- `app/adapters/storage/`: storage backends + +## Frontend + +Core stack: + +- Vite +- React +- TypeScript +- React Router +- TanStack Query + +Source lives in `frontend/`. Built assets are emitted into `app/web/dist/` and +served by FastAPI at `/app`. + +## Data Model + +Main tables: + +- `users` +- `auth_sessions` +- `directories` +- `file_entries` +- `file_versions` +- `blobs` +- `blob_replicas` +- `backends` +- `upload_sessions` +- `upload_session_parts` +- `jobs` +- `placement_policies` +- `preview_artifacts` +- `cache_entries` + +Design principles: + +- logical namespace is separate from physical storage placement +- files have immutable content versions +- physical content is represented as blobs and replicas +- user-facing deletion uses recycle-bin semantics +- jobs are durable and retryable +- backend-specific configuration stays behind adapter boundaries + +## Upload And Read Path + +Uploads use `tus` transport under `/files`. + +Flow: + +1. create upload session +2. append bytes with `PATCH` +3. finalize through the API +4. create file metadata, version, blob, and local replica +5. enqueue replication or preview jobs when needed + +Read path: + +1. resolve file and current version +2. find the best local or cached blob +3. if needed, materialize a ready remote replica into cache +4. serve download, preview, or range stream + +## Browser File Access + +JSON APIs use bearer-token authorization through the shared frontend API client. + +Browser preview and download must not use naked `/api/files/...` media URLs in +`img`, `iframe`, `video`, or `window.open`. Fetch file content with +authorization, create an object URL, use it, then revoke it. + +This is covered by the Playwright E2E flow. + +## Jobs + +Jobs are stored in SQLite and executed by the in-process job loop. + +Current job types include: + +- blob replication +- preview artifact generation +- backend health checks +- full reconcile +- file reconcile + +Every job should be safe to retry. diff --git a/docs/database-schema.md b/docs/database-schema.md deleted file mode 100644 index 6ecc4a6..0000000 --- a/docs/database-schema.md +++ /dev/null @@ -1,507 +0,0 @@ -# Iron Database Schema - -## 1. Overview - -This document defines the initial SQLite schema for Iron MVP. - -Goals: - -- support a single logical file namespace -- separate logical metadata from physical storage placement -- track replicas and background jobs durably -- support `tus` resumable uploads -- keep the schema portable enough for future migration to PostgreSQL - -Conventions: - -- IDs are stored as `TEXT` -- timestamps are stored as UTC datetimes -- soft delete is preferred for logical user-facing entities in MVP -- provider-specific details belong in JSON columns only when no stable relational shape exists - -## 2. Entity Overview - -Main tables: - -- `users` -- `auth_sessions` -- `directories` -- `file_entries` -- `file_versions` -- `blobs` -- `blob_replicas` -- `backends` -- `upload_sessions` -- `upload_session_parts` -- `jobs` -- `placement_policies` -- `preview_artifacts` -- `cache_entries` - -## 3. Table Definitions - -### 3.1 `users` - -Purpose: - -- local gateway login identity - -Columns: - -- `id` TEXT PRIMARY KEY -- `username` TEXT NOT NULL UNIQUE -- `password_hash` TEXT NOT NULL -- `created_at` DATETIME NOT NULL -- `updated_at` DATETIME NOT NULL - -### 3.2 `auth_sessions` - -Purpose: - -- persisted login sessions for bearer-token auth - -Columns: - -- `id` TEXT PRIMARY KEY -- `user_id` TEXT NOT NULL REFERENCES `users`(`id`) -- `token_hash` TEXT NOT NULL UNIQUE -- `expires_at` DATETIME NOT NULL -- `revoked_at` DATETIME NULL -- `created_at` DATETIME NOT NULL -- `updated_at` DATETIME NOT NULL - -Indexes: - -- `idx_auth_sessions_user_status` on (`user_id`, `revoked_at`, `expires_at`) -- unique `idx_auth_sessions_token_hash` on `token_hash` - -Notes: - -- raw session tokens should never be stored directly -- the current implementation stores a SHA-256 token hash and returns the raw bearer token only at login time - -### 3.3 `directories` - -Purpose: - -- logical folder hierarchy - -Columns: - -- `id` TEXT PRIMARY KEY -- `parent_id` TEXT NULL REFERENCES `directories`(`id`) -- `name` TEXT NOT NULL -- `path_key` TEXT NOT NULL UNIQUE -- `created_at` DATETIME NOT NULL -- `updated_at` DATETIME NOT NULL -- `deleted_at` DATETIME NULL - -Constraints: - -- unique sibling name constraint should be enforced in application logic or by an additional composite unique index later - -Indexes: - -- `idx_directories_parent_id` on `parent_id` -- `idx_directories_path_key` on `path_key` - -### 3.4 `file_entries` - -Purpose: - -- logical files visible to users - -Columns: - -- `id` TEXT PRIMARY KEY -- `directory_id` TEXT NOT NULL REFERENCES `directories`(`id`) -- `name` TEXT NOT NULL -- `mime_type` TEXT NULL -- `size_bytes` INTEGER NOT NULL -- `current_version_id` TEXT NULL -- `is_deleted` BOOLEAN NOT NULL DEFAULT 0 -- `deleted_at` DATETIME NULL -- `created_at` DATETIME NOT NULL -- `updated_at` DATETIME NOT NULL - -Constraints: - -- unique active filename per directory - -Indexes: - -- unique `idx_file_entries_directory_name` on (`directory_id`, `name`, `is_deleted`) -- `idx_file_entries_current_version_id` on `current_version_id` -- `idx_file_entries_deleted_at` on `deleted_at` - -Notes: - -- file deletion uses recycle-bin semantics in MVP -- active and deleted files may temporarily share the same logical name in one directory because the uniqueness rule only protects active entries - -### 3.5 `file_versions` - -Purpose: - -- immutable snapshots of file content - -Columns: - -- `id` TEXT PRIMARY KEY -- `file_entry_id` TEXT NOT NULL REFERENCES `file_entries`(`id`) -- `content_hash` TEXT NOT NULL -- `size_bytes` INTEGER NOT NULL -- `storage_layout` TEXT NOT NULL -- `created_at` DATETIME NOT NULL - -Recommended enum values: - -- `single` -- `chunked` - -Indexes: - -- `idx_file_versions_file_entry_id` on `file_entry_id` -- `idx_file_versions_content_hash` on `content_hash` - -### 3.5 `blobs` - -Purpose: - -- physical storage units referenced by file versions - -Columns: - -- `id` TEXT PRIMARY KEY -- `file_version_id` TEXT NOT NULL REFERENCES `file_versions`(`id`) -- `blob_index` INTEGER NOT NULL -- `kind` TEXT NOT NULL -- `content_hash` TEXT NOT NULL -- `size_bytes` INTEGER NOT NULL -- `logical_offset` INTEGER NOT NULL DEFAULT 0 -- `created_at` DATETIME NOT NULL - -Recommended enum values: - -- `file` -- `chunk` -- `thumbnail` -- `poster` - -Indexes: - -- unique `idx_blobs_file_version_index` on (`file_version_id`, `blob_index`) -- `idx_blobs_content_hash` on `content_hash` - -### 3.6 `blob_replicas` - -Purpose: - -- one row per physical copy of a blob on one backend - -Columns: - -- `id` TEXT PRIMARY KEY -- `blob_id` TEXT NOT NULL REFERENCES `blobs`(`id`) -- `backend_id` TEXT NOT NULL REFERENCES `backends`(`id`) -- `storage_key` TEXT NOT NULL -- `status` TEXT NOT NULL -- `etag` TEXT NULL -- `checksum` TEXT NULL -- `size_bytes` INTEGER NOT NULL -- `last_verified_at` DATETIME NULL -- `created_at` DATETIME NOT NULL -- `updated_at` DATETIME NOT NULL - -Recommended enum values: - -- `pending` -- `ready` -- `missing` -- `corrupt` -- `offline` -- `failed` - -Indexes: - -- unique `idx_blob_replicas_blob_backend` on (`blob_id`, `backend_id`) -- `idx_blob_replicas_backend_status` on (`backend_id`, `status`) -- `idx_blob_replicas_storage_key` on `storage_key` - -Current implementation status: - -- local backend writes now create `blob_replicas` rows during upload finalize -- initial persisted replica status is `ready` - -### 3.7 `backends` - -Purpose: - -- configured storage targets and their policy attributes - -Columns: - -- `id` TEXT PRIMARY KEY -- `name` TEXT NOT NULL UNIQUE -- `type` TEXT NOT NULL -- `stability_class` TEXT NOT NULL -- `read_priority` INTEGER NOT NULL -- `write_priority` INTEGER NOT NULL -- `is_enabled` BOOLEAN NOT NULL DEFAULT 1 -- `config_json` TEXT NOT NULL -- `capacity_hint_bytes` INTEGER NULL -- `last_health_status` TEXT NULL -- `last_health_checked_at` DATETIME NULL -- `created_at` DATETIME NOT NULL -- `updated_at` DATETIME NOT NULL - -Recommended `type` values: - -- `local` -- `s3` - -### 3.8 `placement_policies` - -Purpose: - -- persist file-class placement intent separately from backend rows - -Columns: - -- `id` TEXT PRIMARY KEY -- `file_class` TEXT NOT NULL UNIQUE -- `require_local` BOOLEAN NOT NULL DEFAULT 1 -- `stable_replica_count` INTEGER NOT NULL DEFAULT 1 -- `opportunistic_replica_count` INTEGER NOT NULL DEFAULT 0 -- `preferred_backend_ids_json` TEXT NOT NULL DEFAULT `[]` -- `excluded_backend_ids_json` TEXT NOT NULL DEFAULT `[]` -- `max_non_local_size_bytes` INTEGER NULL -- `created_at` DATETIME NOT NULL -- `updated_at` DATETIME NOT NULL - -Notes: - -- the current backend persists preferred and excluded backend ids as JSON lists -- `max_non_local_size_bytes` can force large files to remain local-only even when stable backends exist -- this table is actively used by reconcile jobs and by `/api/policies/placement/preview` -- `aliyun` -- `baidu` -- `bridge` - -Recommended `stability_class` values: - -- `local` -- `stable` -- `opportunistic` - -Indexes: - -- `idx_backends_enabled` on `is_enabled` -- `idx_backends_priority` on (`write_priority`, `read_priority`) - -Current implementation status: - -- the gateway currently bootstraps one default local backend record automatically - -### 3.8 `upload_sessions` - -Purpose: - -- durable record for `tus` upload lifecycle - -Columns: - -- `id` TEXT PRIMARY KEY -- `directory_id` TEXT NULL REFERENCES `directories`(`id`) -- `filename` TEXT NULL -- `total_size_bytes` INTEGER NOT NULL -- `received_size_bytes` INTEGER NOT NULL -- `status` TEXT NOT NULL -- `temp_path` TEXT NOT NULL -- `tus_upload_url` TEXT NULL -- `upload_metadata_json` TEXT NULL -- `created_at` DATETIME NOT NULL -- `updated_at` DATETIME NOT NULL - -Recommended enum values: - -- `created` -- `uploading` -- `uploaded` -- `finalizing` -- `finalized` -- `failed` -- `expired` - -Indexes: - -- `idx_upload_sessions_status` on `status` -- `idx_upload_sessions_updated_at` on `updated_at` - -### 3.9 `upload_session_parts` - -Purpose: - -- optional upload-part detail for resumable recovery and diagnostics - -Columns: - -- `id` TEXT PRIMARY KEY -- `upload_session_id` TEXT NOT NULL REFERENCES `upload_sessions`(`id`) -- `part_number` INTEGER NOT NULL -- `byte_offset` INTEGER NOT NULL -- `size_bytes` INTEGER NOT NULL -- `checksum` TEXT NULL -- `status` TEXT NOT NULL -- `created_at` DATETIME NOT NULL -- `updated_at` DATETIME NOT NULL - -Indexes: - -- unique `idx_upload_session_parts_unique` on (`upload_session_id`, `part_number`) -- `idx_upload_session_parts_upload_id` on `upload_session_id` - -### 3.10 `jobs` - -Purpose: - -- durable background task queue - -Columns: - -- `id` TEXT PRIMARY KEY -- `kind` TEXT NOT NULL -- `status` TEXT NOT NULL -- `payload_json` TEXT NOT NULL -- `attempt_count` INTEGER NOT NULL DEFAULT 0 -- `max_attempts` INTEGER NOT NULL DEFAULT 5 -- `run_after` DATETIME NOT NULL -- `last_error` TEXT NULL -- `created_at` DATETIME NOT NULL -- `updated_at` DATETIME NOT NULL - -Recommended enum values: - -- kinds: `replicate_blob`, `generate_thumbnail`, `verify_replica`, `repair_blob`, `health_check_backend` -- status: `queued`, `running`, `succeeded`, `failed`, `dead` - -Indexes: - -- `idx_jobs_status_run_after` on (`status`, `run_after`) -- `idx_jobs_kind_status` on (`kind`, `status`) - -### 3.11 `preview_artifacts` - -Purpose: - -- preview resources such as thumbnails and poster images - -Columns: - -- `id` TEXT PRIMARY KEY -- `file_version_id` TEXT NOT NULL REFERENCES `file_versions`(`id`) -- `artifact_type` TEXT NOT NULL -- `blob_id` TEXT NOT NULL REFERENCES `blobs`(`id`) -- `status` TEXT NOT NULL -- `created_at` DATETIME NOT NULL -- `updated_at` DATETIME NOT NULL - -Recommended enum values: - -- `thumbnail` -- `poster` -- `pdf_preview` - -Indexes: - -- unique `idx_preview_artifacts_version_type` on (`file_version_id`, `artifact_type`) - -### 3.12 `cache_entries` - -Purpose: - -- local cache state for blobs and derivatives - -Columns: - -- `id` TEXT PRIMARY KEY -- `blob_id` TEXT NOT NULL REFERENCES `blobs`(`id`) -- `local_path` TEXT NOT NULL -- `cache_kind` TEXT NOT NULL -- `size_bytes` INTEGER NOT NULL -- `status` TEXT NOT NULL -- `last_accessed_at` DATETIME NOT NULL -- `expires_at` DATETIME NULL -- `created_at` DATETIME NOT NULL -- `updated_at` DATETIME NOT NULL - -Recommended enum values: - -- cache kinds: `blob`, `preview`, `temp` -- status: `ready`, `partial`, `evicted`, `invalid` - -Indexes: - -- `idx_cache_entries_blob_id` on `blob_id` -- `idx_cache_entries_last_accessed_at` on `last_accessed_at` -- `idx_cache_entries_expires_at` on `expires_at` - -## 4. Relationship Summary - -- one `directory` has many child `directories` -- one `directory` has many `file_entries` -- one `file_entry` has many `file_versions` -- one `file_version` has many `blobs` -- one `blob` has many `blob_replicas` -- one `backend` has many `blob_replicas` -- one `upload_session` may become one `file_entry` -- one `job` targets one logical task payload - -## 5. Migration Guidance - -Recommended migration order: - -1. `users` -2. `directories` -3. `file_entries` -4. `file_versions` -5. `backends` -6. `blobs` -7. `blob_replicas` -8. `upload_sessions` -9. `upload_session_parts` -10. `jobs` -11. `placement_policies` -12. `preview_artifacts` -13. `cache_entries` - -Guidelines: - -- create foreign keys only after referenced tables exist -- add indexes in the same migration as table creation where practical -- keep enum values in application constants for SQLite MVP - -## 6. SQLite Notes - -- enable foreign keys explicitly -- use WAL mode for better concurrent read behavior -- keep transactions short around upload finalization and job claiming -- avoid long-lived write transactions during file streaming - -## 7. Future Schema Evolution - -Likely future additions: - -- `file_tags` -- `share_links` -- `device_clients` -- `sync_rules` -- `search_documents` -- `backend_metrics` -- `replica_verification_events` - -Likely future changes: - -- introduce true version history controls -- move some JSON config into typed backend-specific tables if the surface becomes stable -- expand `placement_policies` if policy shape grows beyond file-class controls, preferred/excluded backend sets, and size-aware replica caps diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 0000000..cdd46a1 --- /dev/null +++ b/docs/development.md @@ -0,0 +1,105 @@ +# Development + +This document captures the development workflow, test expectations, and UI +quality rules for Iron. + +## Setup + +```bash +UV_CACHE_DIR=.uv-cache uv venv .venv +UV_CACHE_DIR=.uv-cache uv pip install --python .venv/bin/python -e '.[dev]' +npm install +``` + +## Run Locally + +```bash +npm run build +.venv/bin/python -m uvicorn app.main:app --reload +``` + +Open: + +```text +http://127.0.0.1:8000/app +``` + +Default development credentials: + +```text +username: admin +password: changeme-iron +``` + +## Tests + +Full suite: + +```bash +.venv/bin/python -m pytest +``` + +Current expected result: + +```text +66 passed +``` + +Browser E2E: + +```bash +.venv/bin/python scripts/ui_playwright_smoke.py +``` + +Pytest E2E entry: + +```bash +.venv/bin/python -m pytest tests/e2e/test_web_ui_playwright.py +``` + +## UI Change Standard + +Any meaningful Web UI change must be validated against a real running service. + +Required workflow: + +1. build frontend assets with `npm run build` +2. run `./.venv/bin/python scripts/ui_playwright_smoke.py` +3. review screenshots in `output/playwright/` +4. fix layout, spacing, overflow, or interaction issues +5. rerun the Playwright flow + +The E2E flow currently covers: + +- login +- files page render +- create folder +- upload text file +- authenticated file download +- upload image file +- authenticated inline image preview +- search +- uploads, recycle bin, storage, and jobs page render +- browser `/api` 4xx/5xx failures during the flow + +## Runtime Artifacts + +Do not commit: + +- `.venv/` +- `.uv-cache/` +- `node_modules/` +- `output/` +- `.iron-storage/` +- `.iron-temp/` +- local databases +- Python caches + +## Documentation Rules + +- Keep `README.md` concise and user-facing. +- Keep contributor instructions in `CONTRIBUTING.md`. +- Keep coding-agent instructions in `AGENTS.md`. +- Keep deeper project docs in `docs/`. +- Use relative links in Markdown. +- Update docs when changing setup, commands, architecture, or user-visible behavior. diff --git a/docs/handoff.md b/docs/handoff.md deleted file mode 100644 index df453a7..0000000 --- a/docs/handoff.md +++ /dev/null @@ -1,233 +0,0 @@ -# Iron Engineering Handoff - -Last updated: 2026-04-15 - -## 1. What This Document Is For - -This is the fastest way for another AI or engineer to continue work on Iron without re-reading the full project history. - -Read this first, then use the other docs as references: - -- product scope: [requirements.md](/Users/bytedance/iron/docs/requirements.md:1) -- gap analysis: [mvp-status.md](/Users/bytedance/iron/docs/mvp-status.md:1) -- implementation architecture: [technical-architecture-draft.md](/Users/bytedance/iron/docs/technical-architecture-draft.md:1) -- API details: [api-design.md](/Users/bytedance/iron/docs/api-design.md:1) -- schema details: [database-schema.md](/Users/bytedance/iron/docs/database-schema.md:1) - -## 2. Current Truth - -The project is no longer architecture-only. - -It already has: - -- async FastAPI backend -- SQLite + SQLAlchemy ORM -- Alembic migrations -- local auth with persisted bearer-token sessions -- directories, files, recycle bin, rename, move -- `tus` uploads -- local backend persistence -- S3 runtime replication and remote fallback reads -- declarative reconcile jobs -- metadata export, validate, integrity, restore-plan, and guarded import -- placement policy persistence and placement preview API -- a frontend app scaffold at `/app` built with `Vite + React + TypeScript` -- product routes for files, uploads, recycle bin, storage, and jobs -- a desktop-first Web UI with file list, contextual inspector, upload queue, recycle bin, storage, and jobs surfaces -- Python Playwright E2E coverage for login, create folder, upload, authenticated download, authenticated image preview, search, and page rendering - -It now has a product-usable Web frontend foundation, but it still needs user-facing hardening before calling it a polished release. - -The most honest stage label is still: - -- `Product MVP Candidate` - -## 3. Current Verified Baseline - -Use `uv` and the local virtualenv: - -```bash -UV_CACHE_DIR=.uv-cache uv venv .venv -UV_CACHE_DIR=.uv-cache uv pip install --python .venv/bin/python -e '.[dev]' -``` - -Run tests with: - -```bash -export UV_CACHE_DIR=.uv-cache && .venv/bin/python -m pytest -``` - -Current expected result: - -- `66 passed` - -If new work changes behavior, update this number in: - -- [README.md](/Users/bytedance/iron/README.md:1) -- [mvp-status.md](/Users/bytedance/iron/docs/mvp-status.md:1) -- this file - -## 4. What Is Implemented - -### 4.1 Backend Capabilities - -- auth: - - bootstrap admin user - - login - - logout - - current user -- namespace: - - create/list directories - - mixed children listing - - file detail - - file and directory rename/move - - recycle-bin delete/restore for files -- upload: - - `tus` create/head/patch - - finalize to file metadata -- storage: - - local backend object persistence - - S3 adapter runtime path - - remote fallback read into local cache -- preview: - - image/PDF inline preview - - generic video stream through HTTP Range - - preview artifact rows and generation jobs -- ops: - - backend list/create/check/disable - - jobs list/detail/retry/run - - enqueue full reconcile - - enqueue backend health checks - - file reconcile -- recovery: - - metadata export - - metadata validate - - metadata integrity - - restore plan with validation token - - guarded metadata import - -### 4.2 Placement Controls - -This area is complete enough for backend MVP work. - -Current placement behavior supports: - -- policies by file class -- local requirement flag -- stable and opportunistic replica counts -- ordered preferred backend ids -- excluded backend ids -- optional `max_non_local_size_bytes` -- placement preview API -- reconcile jobs converging toward policy-selected targets - -Main files: - -- [app/services/policy_service.py](/Users/bytedance/iron/app/services/policy_service.py:1) -- [app/api/routes/policies.py](/Users/bytedance/iron/app/api/routes/policies.py:1) -- [tests/test_policy_routes.py](/Users/bytedance/iron/tests/test_policy_routes.py:1) -- [tests/test_placement_jobs.py](/Users/bytedance/iron/tests/test_placement_jobs.py:1) - -## 5. Main Remaining Work - -### 5.1 P0: Product Frontend - -This is the main next focus. - -Needed next: - -- deepen the new frontend foundation into a more polished daily-usable product -- improve directory navigation and breadcrumbs -- harden upload progress UX and directory targeting -- improve preview and detail-panel behavior -- refine mobile behavior and browser session handling -- extend Playwright E2E coverage as each product flow becomes real - -### 5.2 P1: Backend Hardening - -These are still useful, but they are no longer the main gating path: - -- stronger preview derivative generation -- post-restore operational tooling -- richer replica verification and repair visibility -- more sophisticated cache observability and eviction - -### 5.3 Post-MVP / Roadmap - -- Aliyun adapter -- Baidu adapter -- desktop client -- mobile client -- share links -- search - -## 6. Recommended Next Sequence - -If another AI continues from here, the recommended order is: - -1. preserve the current desktop-first drive UX and avoid reverting to dashboard-style empty panels -2. keep browser file access going through authenticated blob fetches, not naked `/api/files/...` media URLs -3. extend Playwright E2E coverage when adding preview, download, upload, or mutation flows -4. only after product hardening, return to backend polish such as richer preview derivatives - -## 7. Important Repo Realities - -These details are easy to get wrong if someone only reads the older architecture docs: - -- config is currently a lightweight cached dataclass module, not `pydantic-settings` -- the frontend source now lives in `frontend/` and builds into `app/web/dist` -- `tus` endpoints live under `/files`, outside the normal `/api` JSON namespace -- the backend uses persisted bearer tokens today -- metadata import is intentionally guarded by: - - `confirm_replace=true` - - `validation_token` from restore-plan - -## 8. Where To Look In Code - -High-signal entry points: - -- app bootstrap: [app/main.py](/Users/bytedance/iron/app/main.py:1) -- API composition: [app/api/router.py](/Users/bytedance/iron/app/api/router.py:1) -- auth: [app/services/auth_service.py](/Users/bytedance/iron/app/services/auth_service.py:1) -- files: [app/services/file_service.py](/Users/bytedance/iron/app/services/file_service.py:1) -- uploads: [app/services/upload_service.py](/Users/bytedance/iron/app/services/upload_service.py:1) -- jobs: [app/services/job_service.py](/Users/bytedance/iron/app/services/job_service.py:1) -- storage: [app/services/storage_service.py](/Users/bytedance/iron/app/services/storage_service.py:1) -- placement: [app/services/policy_service.py](/Users/bytedance/iron/app/services/policy_service.py:1) -- frontend entry: [frontend/src/app/App.tsx](/Users/bytedance/iron/frontend/src/app/App.tsx:1) -- frontend routes: [frontend/src/app/router.tsx](/Users/bytedance/iron/frontend/src/app/router.tsx:1) -- frontend API client: [frontend/src/lib/api.ts](/Users/bytedance/iron/frontend/src/lib/api.ts:1) -- frontend styles: [frontend/src/styles/app.css](/Users/bytedance/iron/frontend/src/styles/app.css:1) -- Playwright E2E: [scripts/ui_playwright_smoke.py](/Users/bytedance/iron/scripts/ui_playwright_smoke.py:1) -- pytest E2E entry: [tests/e2e/test_web_ui_playwright.py](/Users/bytedance/iron/tests/e2e/test_web_ui_playwright.py:1) -- built app entry: [app/web/dist/index.html](/Users/bytedance/iron/app/web/dist/index.html:1) - -## 9. Testing Guidance - -When continuing development: - -- keep using `uv` + `.venv` -- prefer adding tests in the same slice as each feature -- run Python Playwright for UI-affecting work -- update docs after meaningful milestone changes -- do not let README, MVP status, and actual test count drift apart - -Current test files: - -- [tests/test_auth_routes.py](/Users/bytedance/iron/tests/test_auth_routes.py:1) -- [tests/test_backend_routes.py](/Users/bytedance/iron/tests/test_backend_routes.py:1) -- [tests/test_directory_routes.py](/Users/bytedance/iron/tests/test_directory_routes.py:1) -- [tests/test_export_routes.py](/Users/bytedance/iron/tests/test_export_routes.py:1) -- [tests/test_file_routes.py](/Users/bytedance/iron/tests/test_file_routes.py:1) -- [tests/test_job_routes.py](/Users/bytedance/iron/tests/test_job_routes.py:1) -- [tests/test_placement_jobs.py](/Users/bytedance/iron/tests/test_placement_jobs.py:1) -- [tests/test_policy_routes.py](/Users/bytedance/iron/tests/test_policy_routes.py:1) -- [tests/test_system_routes.py](/Users/bytedance/iron/tests/test_system_routes.py:1) -- [tests/test_upload_routes.py](/Users/bytedance/iron/tests/test_upload_routes.py:1) -- [tests/test_web_routes.py](/Users/bytedance/iron/tests/test_web_routes.py:1) - -## 10. Suggested Handoff Prompt - -If another AI is about to continue, a good short prompt is: - -> Read `/Users/bytedance/iron/docs/handoff.md` first, treat it as the current source of truth, then continue from the frontend-first sequence unless new user instructions override it. diff --git a/docs/mvp-status.md b/docs/mvp-status.md deleted file mode 100644 index 6033e81..0000000 --- a/docs/mvp-status.md +++ /dev/null @@ -1,473 +0,0 @@ -# Iron MVP Status - -Last updated: 2026-04-15 - -## 1. Purpose - -This document fixes one question in a concrete way: - -- what should count as Iron's **usable MVP** -- what is already implemented -- what still blocks that MVP today - -It is the live checkpoint between: - -- [requirements.md](/Users/bytedance/iron/docs/requirements.md:1) -- [technical-architecture-draft.md](/Users/bytedance/iron/docs/technical-architecture-draft.md:1) -- the actual code and tests in `app/` and `tests/` - -## 2. Current Conclusion - -The honest current answer is: - -- Iron already has a **real backend foundation**, not just design documents -- Iron now has a **browser-usable product foundation** -- the current code is best described as a **Product MVP Candidate** - -Why it is not yet a polished release: - -- the Web UI supports the core drive flows, but still needs real-user hardening and polish -- browser auth exists, but session setup, password management, and onboarding are still basic -- `local + S3` both have real runtime behavior, but advanced repair/visibility remains basic -- declarative reconcile, health-check, and preview-generation jobs now exist -- metadata export, validation, integrity checks, restore-plan, and guarded import now exist - -Priority decision after re-evaluation: - -- the remaining **true MVP blockers** are now mostly product-surface blockers -- the remaining backend work is important, but most of it is now `P1 polish / hardening`, not `P0 blocker` - -## 3. What "Usable MVP" Means - -For this project, a usable MVP should mean: - -- one person can deploy a single gateway node -- the person can log in from a browser -- the person can browse, upload, rename, move, delete, restore, download, and preview files -- the person does not need to care which backend stores a file -- at least two backend types work at runtime, not only in configuration -- the system can survive one non-primary backend failure without losing metadata control -- the system exposes enough operational visibility to inspect backend health and failed tasks - -This is still smaller than the long-term vision. It does **not** require yet: - -- desktop sync client -- mobile-native app -- public share links -- full-text search -- cross-file deduplication -- erasure coding -- advanced media indexing - -## 4. Current System Capability - -As of today, Iron already has these working backend slices. - -### 4.1 Core Platform - -- FastAPI service bootstrap -- async SQLAlchemy ORM with SQLite -- Alembic migrations -- `uv`-based local development flow -- automated test suite - -### 4.2 Namespace and Metadata - -- root directory bootstrap -- create directory -- list mixed directory and file children -- file detail lookup -- rename directory -- move directory with descendant `path_key` updates -- rename file -- move file -- recycle-bin file deletion -- restore file from recycle bin - -### 4.3 Upload and Persistence - -- `tus` upload creation -- upload offset inspection -- sequential `PATCH` upload -- finalize upload into file metadata -- persist finalized content into the default `local` backend -- create `blob` and `blob_replica` records - -### 4.4 Read Path - -- file download -- file streaming -- HTTP Range support for local persisted objects -- inline preview for images and PDFs stored in the local backend - -### 4.5 Backend Management - -- list backends -- create backend config entries for `local` and `s3` -- local backend health check -- S3 backend health check -- backend disable flow -- default local backend bootstrap - -### 4.6 Multi-Backend and Operations - -- S3 runtime replication path -- fallback reads from ready remote replicas -- remote blob cache materialization -- policy-based replication target selection -- durable SQLite job records -- in-process job polling loop -- job list/detail/retry/run APIs -- declarative file reconcile API -- backend health-check enqueue API -- full-system reconcile enqueue API -- metadata export, validate, integrity, and import APIs -- staged restore-plan workflow -- preview artifact generation jobs -- preview artifacts are exposed in file detail - -### 4.7 Test Baseline - -Current verified state: - -- `66` automated tests passing -- route coverage exists for the currently implemented backend slices -- Python Playwright E2E covers login, create folder, upload, authenticated download, authenticated inline image preview, search, and major page rendering - -### 4.8 Web UI Shell - -Implemented: - -- browser entry at `/app` -- frontend app scaffold with `Vite + React + TypeScript` -- local login form -- token-backed session bootstrap in browser storage -- files route with listing, selection, preview panel, and basic file actions -- uploads route with browser queue on top of existing `tus` flow -- recycle-bin route -- storage route -- jobs route - -Status: - -- real product foundation and MVP candidate -- still needs user hardening, mobile refinement, richer media behavior, and broader E2E coverage before a release label - -## 5. MVP Gap Matrix - -The table below is the most useful way to read the project right now. - -| Capability | MVP expectation | Current state | Gap level | -| --- | --- | --- | --- | -| Web login and session | browser-usable auth flow | login route and route guard implemented, still local-storage based | medium | -| Web file browser | daily-usable browser entry | desktop-first files workspace with list/grid, pagination, inspector, dialogs, menus | low | -| Namespace operations | browse, rename, move, delete, restore | backend API implemented | low | -| Upload flow | resumable upload from browser | browser upload route and files-page upload implemented, targeting still needs polish | medium | -| Download and preview | browser download and preview | authenticated browser download and image preview covered by E2E; PDF/video paths still need deeper coverage | medium | -| Runtime multi-backend storage | at least `local + S3` really work | `local + S3` both operate at runtime | low | -| Backend abstraction | provider records and management | partially implemented | medium | -| On-demand remote retrieval | read from another backend when local copy is absent | implemented for ready remote replicas | medium | -| Background tasks | replication, repair, health polling | declarative reconcile, health-check, and preview jobs implemented | low | -| Metadata export and restore | backup and recovery path | export, validate, integrity, restore-plan, and import implemented | low | -| Video-first experience | stream-friendly playback, poster/thumbnail | generic stream exists, preview artifact pipeline is still basic | medium | -| Cache model | explicit local cache behavior | remote-fetch cache exists, policy is still basic | medium | -| Repair visibility | inspect and retry failed work | job visibility, retry, and file reconcile enqueue are implemented | low | -| Aliyun/Baidu adapters | roadmap target backends | not implemented | roadmap gap | - -## 6. What Is Already Good Enough - -These areas should no longer be treated as blank or hypothetical: - -- backend service skeleton -- metadata model direction -- upload protocol choice -- recycle-bin semantics -- local object persistence -- local read, stream, and basic preview path -- namespace mutation rules -- baseline test discipline - -This matters because Iron is already past the "architecture-only" phase. The remaining work is now mostly about turning a solid backend core into a user-usable product slice. - -## 7. Biggest MVP Blockers - -If the question is "what prevents someone from really using Iron as their personal cloud drive today?", the answer is below. - -### 7.1 Web UI Exists And Is Now The Main Hardening Surface - -Iron now has a browser entry point with real drive workflows. - -Implemented: - -- login page -- session bootstrap -- root directory listing -- backend listing -- logout -- files workspace with desktop drive layout -- create folder, upload, rename, move, delete, download -- image preview through authenticated blob URLs -- uploads, recycle bin, storage, and jobs pages -- Playwright screenshots and E2E coverage - -Still missing: - -- backend management forms -- better browser-side session handling -- broader PDF/video preview E2E -- mobile/narrow-screen polish -- richer empty-state and operational guidance - -MVP priority: - -- `P0 hardening` - -### 7.2 Authentication Is Browser-Usable But Still Basic - -The backend has a real local auth slice, and the Web app now has a browser-facing login/session flow. - -Implemented: - -- bootstrap local admin user -- login API -- bearer-token session issuance -- current-user API -- logout API -- protected file, directory, backend, upload routes -- browser login page -- authenticated API client -- authenticated blob fetches for download and preview - -Still missing: - -- password rotation or setup UX -- clearer bootstrap credential onboarding - -MVP priority: - -- `P0 blocker`, but much smaller than before because the backend auth layer already exists - -### 7.3 Multi-Backend Runtime Is Real But Still Basic - -This area now has a real first implementation, but it is still early. - -Implemented: - -- real S3 read/write support -- remote fallback reads that can materialize a missing local blob into cache -- S3 health checks -- persisted placement policies by file class -- placement preview API for product-facing inspection -- preferred backend ordering and explicit backend exclusion -- size-aware non-local replica caps -- reconcile jobs now converge toward policy-selected targets instead of a single hardcoded target shape - -Still missing: - -- richer verification and replica selection rules - -MVP priority: - -- `P1 important` -- no longer a first-order blocker for the first usable release - -### 7.4 Background Task Engine Exists But Is Not Complete - -Iron now has a real first task engine, but it is still narrower than the intended MVP. - -Implemented: - -- declarative reconcile jobs -- file reconcile enqueue flow -- backend health-check enqueue flow -- task inspection and retry flow -- in-process polling loop - -Still missing: - -- richer preview derivative generation beyond the current basic artifact flow - -MVP priority: - -- `P1 important` -- the task substrate itself is no longer a blocker - -### 7.5 Metadata Recovery Exists And Has Basic Safety Checks - -This is now substantially addressed. - -Implemented: - -- export command or API -- validation API for metadata snapshots -- integrity check API for live runtime state -- restore-plan dry-run workflow -- import/restore API for metadata snapshots -- restore-triggered follow-up reconcile and health-check jobs - -Still missing: - -- richer post-restore operational tooling - -MVP priority: - -- `P1 important` -- the restore contract already exists; remaining work is hardening - -## 8. Priority Reset - -After re-evaluating the current repository, the most useful classification is: - -### 8.1 P0: Must Finish Before Release Label - -- harden Web UI flows with Playwright-backed regression -- browser-facing session UX beyond local-storage basics -- broader PDF/video/download E2E coverage - -### 8.2 P1: Should Finish Immediately Before or After MVP Cut - -- stronger preview derivative generation -- post-restore operational hardening and checks - -### 8.3 P2: Important But No Longer MVP-Critical - -- Aliyun backend -- Baidu backend -- richer cache policy -- stronger replica verification sophistication -- advanced media workflows beyond current preview artifacts - -## 9. Secondary Gaps - -These do not block the first shippable MVP as hard as the items above, but they still matter. - -### 9.1 Media Experience Is Only Basic - -Current state: - -- image preview works -- PDF preview works -- generic video stream works through HTTP Range -- preview artifacts are tracked and generated through background jobs - -Still missing: - -- richer thumbnails -- richer video posters -- better unsupported-codec fallback behavior - -### 9.2 Cache Behavior Is Not Yet a Product Feature - -The architecture talks about on-demand retrieval and local caching, but current runtime behavior does not yet expose a real cache policy. - -Still missing: - -- explicit cache store -- retention policy -- eviction policy -- cache observability - -### 9.3 Consumer Cloud Backends Are Still Roadmap Work - -Aliyun and Baidu remain important to the long-term goal, but they should be treated as **post-first-MVP** unless they become easier than S3 in practice. - -## 10. Recommended MVP Cutline - -To avoid drifting scope, the recommended first **product-usable MVP** should be defined as: - -### 10.1 Must Have - -- local authentication -- Web UI for file browsing and basic backend management -- `tus` upload from the browser -- upload, rename, move, delete, restore, download -- image and PDF preview -- video playback through range-capable gateway streaming -- `local` backend runtime support -- `S3` backend runtime support -- one policy-driven placement flow with: - - file-class placement rules - - preferred and excluded backend controls - - local plus optional non-local replicas -- backend health check visibility -- metadata export command or API -- one repair or retry path for failed replicas - -### 10.2 Can Wait Until After MVP - -- Aliyun runtime adapter -- Baidu runtime adapter -- thumbnail/poster generation pipeline -- richer cache intelligence -- share links -- search -- desktop client shell -- mobile app shell - -## 11. Stage Labels - -To keep internal communication honest, use these labels: - -### 11.1 Current Stage - -`Product MVP Candidate` - -Meaning: - -- backend APIs exist -- local auth exists -- local persistence works -- core file lifecycle works -- browser product flows exist and are E2E-tested -- still needs polish, hardening, and more coverage before release - -### 11.2 Target Next Stage - -`Product MVP` - -Meaning: - -- browser-usable -- authenticated -- local + S3 both operational -- at least one simple replication flow -- metadata export and basic repair visibility - -### 11.3 Later Stage - -`Multi-Backend Personal Drive` - -Meaning: - -- consumer cloud backends join the runtime pool -- remote fallback retrieval works -- background orchestration is richer -- media workflows are stronger - -## 12. Recommended Gap-Closing Order - -If the goal is to reach the first usable MVP with the shortest path, the order should be: - -1. harden the current browser product flows through Playwright-backed regression -2. add deeper PDF/video preview and download coverage -3. deepen browser session handling on top of the current auth backend -4. improve preview derivative generation and post-restore hardening -5. refine operational pages based on real user tasks -6. add the first consumer-cloud backend - -## 13. Final Assessment - -The most accurate one-line summary today is: - -> Iron has a credible backend and a browser-usable Web product foundation; the next work is hardening and polish. - -The shortest path to changing that is not "more backend breadth everywhere". It is: - -- browser session hardening -- deeper Web UI regression coverage -- stronger media and restore hardening -- repair and restore - -For implementation sequencing and repo-level handoff notes, see [handoff.md](/Users/bytedance/iron/docs/handoff.md:1). - -Once those land, Iron can honestly be called a first usable personal cloud-drive MVP. diff --git a/docs/product.md b/docs/product.md new file mode 100644 index 0000000..41fa1aa --- /dev/null +++ b/docs/product.md @@ -0,0 +1,69 @@ +# Product + +Iron is a self-hosted personal cloud drive gateway. The product goal is to give +one person a familiar browser drive experience while keeping storage placement, +repair, and metadata ownership under their control. + +## Current Stage + +Iron is a **Product MVP Candidate**. + +Implemented: + +- local authentication and browser login +- file browser with folders, search, sort, pagination, and an inspector panel +- create folder, upload, rename, move, delete, restore, download, and image preview +- uploads, recycle bin, storage, and jobs pages +- local backend persistence +- S3 runtime replication and remote fallback reads +- placement policies and reconcile jobs +- metadata export, validate, integrity, restore-plan, and guarded import +- Python Playwright E2E coverage for the core browser flow + +Still needs hardening before a polished release: + +- better browser session and bootstrap credential UX +- deeper PDF/video preview coverage +- richer preview derivative generation +- stronger post-restore and repair visibility +- mobile and narrow-screen refinement + +## MVP Cutline + +A first usable MVP should let one person: + +- deploy a single gateway node +- log in from a browser +- browse, upload, preview, download, rename, move, delete, and restore files +- use local storage and at least one remote backend +- inspect backend health and failed work +- export and validate metadata for recovery + +Not required for the first MVP: + +- desktop sync client +- mobile native app +- public share links +- full-text search +- cross-file deduplication +- erasure coding +- advanced media indexing + +## Roadmap + +Near-term: + +- harden Web UI flows with Playwright-backed regression +- improve session management and onboarding +- expand PDF/video preview tests +- improve upload targeting and progress feedback +- refine storage and jobs pages around real user tasks + +Later: + +- Aliyun and Baidu adapters +- richer cache policy +- share links +- desktop and mobile clients +- stronger media workflows +- advanced repair orchestration diff --git a/docs/requirements.md b/docs/requirements.md deleted file mode 100644 index 3fd022e..0000000 --- a/docs/requirements.md +++ /dev/null @@ -1,397 +0,0 @@ -# Iron Requirements - -## 1. Product Overview - -`Iron` is a personal cloud drive system built around a self-hosted `gateway` service. - -Users access a unified file space through a Web application in the first phase. The gateway manages metadata, local cache, upload/download pipelines, preview streaming, and data placement across multiple storage backends. - -The product goal is not to be a backup tool first. It is a daily-use personal drive focused on: - -- ease of use -- resistance to storage provider failure -- low storage cost -- optional privacy protections - -The system should allow storage across a mix of: - -- Baidu Netdisk -- Aliyun Drive -- S3-compatible object storage -- local directories - -The long-term vision is to expose the same core capabilities to desktop and mobile clients, while keeping storage control and metadata ownership in the user's hands. - -## 2. Product Goals - -### 2.1 Primary Goals - -- Provide a single virtual drive space across multiple heterogeneous storage backends. -- Support daily file management, not only backup and restore. -- Allow large media files and documents to coexist in one product. -- Support on-demand cloud retrieval so files do not need to be fully cached locally. -- Support preview-first workflows, especially for photos, videos, and PDFs. -- Keep system-critical metadata under user control instead of delegating it to any one storage vendor. -- Allow unstable or free providers to participate as secondary or cold-tier storage. - -### 2.2 Priority Order - -1. Ease of use -2. Resistance to provider shutdown or lock-in -3. Low cost -4. Privacy and encryption - -## 3. Target Users and Usage Context - -### 3.1 Primary User - -- a single user or household -- has one always-on host running the gateway -- accesses files from PCs and phones in a home network, with possible future remote access - -### 3.2 Main Data Types - -- large video files -- large photo libraries -- documents -- PDFs -- mixed personal archives - -### 3.3 Usage Style - -- browse files from a Web UI -- upload from browser or client -- preview before download when possible -- let the system decide actual storage placement -- tolerate heavier local infrastructure if it improves experience - -## 4. Product Principles - -- Users see one logical drive, not multiple backend silos. -- Storage placement is system-managed, not user-managed in normal workflows. -- Metadata must remain recoverable even if one backend becomes unavailable. -- Stable backends and unstable backends must be treated differently by policy. -- Media experience should be optimized explicitly, not treated as a side effect of generic file storage. -- The MVP should prefer understandable replication over complex erasure coding. - -## 5. Scope Definition - -### 5.1 In Scope for the Product - -- file browsing and directory navigation -- upload and download -- delete, rename, move, copy -- on-demand retrieval from remote storage -- local cache management -- image preview -- video preview via streaming -- PDF preview -- multi-backend storage placement -- backend health monitoring -- metadata backup and recovery tooling -- background jobs for sync, repair, and replication - -### 5.2 Out of Scope for MVP - -- real-time collaborative editing -- multi-tenant enterprise permissions -- public sharing links with strong abuse controls -- fully offline-first desktop synchronization -- true distributed peer-to-peer storage nodes -- POSIX-complete filesystem semantics -- erasure coding as a required storage strategy -- full-text OCR and semantic search - -## 6. System Context - -### 6.1 Phase 1 Deployment Model - -- one self-hosted gateway process -- one Web client -- one SQLite database -- one local cache directory -- multiple configured storage backends - -### 6.2 Future Deployment Model - -- gateway remains the control plane -- desktop clients may add richer local sync and mount capabilities -- mobile clients may provide browsing, preview, upload, and camera ingestion -- metadata database may move from SQLite to a networked database if needed - -## 7. Functional Requirements - -### 7.1 File Space and Navigation - -- The system must expose a single hierarchical directory tree. -- The system must allow creating, renaming, moving, and deleting directories. -- The system must allow listing files with pagination and sorting. -- The system must show key metadata including name, size, type, timestamps, and cache status. -- The system must decouple logical path from backend storage location. - -### 7.2 Upload - -- The system must support browser-based upload. -- The system must support resumable upload at least at the gateway boundary. -- The system must support large-file upload without requiring full in-memory buffering. -- The system must classify uploaded files by size and media type to choose storage strategy. -- The system must allow asynchronous replication after initial successful ingestion. - -### 7.3 Download - -- The system must allow direct file download from the Web UI. -- The system must support on-demand fetch from remote backends if data is not in local cache. -- The system must expose download progress and failure states. - -### 7.4 Preview - -- The system must support image preview. -- The system must support PDF preview. -- The system must support video preview through HTTP range-compatible streaming. -- The system should generate thumbnails for common media types. -- The system should avoid full-file fetch when previewing large video files if the backend layout permits partial reads. - -### 7.5 Cache Management - -- The gateway must maintain a local cache for hot files and preview artifacts. -- The system must track cache state separately from logical file state. -- The system must support configurable cache eviction policies. -- The system should allow pinning specific files or directories for local retention in future versions. - -### 7.6 Multi-Backend Storage - -- The system must support multiple backend types at once. -- The system must support backend policies such as `stable`, `opportunistic`, and `local`. -- The system must support at least one primary copy and optional secondary copies. -- The system must track where each object or file replica is stored. -- The system must detect backend unavailability and mark replicas accordingly. - -### 7.7 Metadata and Recovery - -- The system must maintain its own metadata database independent of backend-native directory layouts. -- The system must support export and backup of metadata. -- The system must support integrity verification between metadata records and backend objects. -- The system must support repair jobs that recreate missing replicas when another valid replica exists. - -### 7.8 Jobs and Operations - -- The system must support background jobs for uploads, replication, thumbnail generation, and repair. -- The system must expose job status in the UI or API. -- The system must retain structured logs for user-visible failures. - -### 7.9 Backend Management - -- The system must allow adding, updating, disabling, and deleting backend configurations. -- The system must display backend status, capacity hints if available, and last health-check result. -- The system should support backend weighting and placement rules. - -## 8. Non-Functional Requirements - -### 8.1 Usability - -- A user should not need to understand backend topology during normal file operations. -- Common file operations should be understandable from a single Web interface. -- Preview should be first-class for photo, video, and PDF content. - -### 8.2 Reliability - -- Metadata must not depend on a single unstable third-party backend. -- The system should tolerate temporary backend outages and recover later. -- Background jobs must be restart-safe. - -### 8.3 Performance - -- Listing a directory from metadata should not require live backend enumeration. -- Upload and download pipelines should stream rather than fully buffer large files. -- Preview start latency for cached or stable-backend media should be low enough for normal personal use. - -### 8.4 Extensibility - -- Backend integrations must be pluggable. -- Data model must allow migration from SQLite to another relational database. -- API design must allow future desktop and mobile clients. - -### 8.5 Security - -- Authentication is required for the Web UI and APIs. -- Secrets for backend credentials must not be stored in plaintext in user-facing logs. -- Encryption should be supported in design even if not mandatory for every backend path in MVP. - -## 9. Storage Model Strategy - -### 9.1 MVP Strategy - -- documents and small files are stored as whole-file objects -- photos are stored as whole-file originals plus generated thumbnails -- large videos may use chunked storage if needed for upload resilience and streamability -- replication is preferred over erasure coding - -### 9.2 Future Strategy - -- content-addressed chunks for large files -- optional object packing for small files -- deduplication across identical content -- tier-aware migration policies -- optional erasure coding for selected backend groups - -## 10. Backend Policy Model - -Each backend should have policy metadata, for example: - -- backend type -- stability class -- cost class -- write priority -- read priority -- capacity hint -- health status -- credential state - -Suggested policy classes: - -- `stable`: trusted for primary storage, such as S3 or a managed local directory on a reliable host -- `opportunistic`: useful for secondary replicas or cold storage, such as free consumer cloud drives -- `local`: fast cache-adjacent storage on the gateway host - -## 11. MVP Definition - -### 11.1 MVP Objective - -Deliver a working single-user personal cloud drive that feels usable every day through a Web UI, with support for multi-backend storage, on-demand file retrieval, and media preview. - -### 11.2 MVP Feature Set - -- Web authentication -- file and directory browsing -- upload, download, rename, move, delete -- backend management for local directory, S3, Aliyun Drive, and Baidu Netdisk -- SQLite metadata store -- local cache directory -- background job queue inside the gateway process -- image preview and thumbnail generation -- PDF preview -- video preview through range-capable gateway streaming -- replica tracking -- backend health checks -- metadata export -- repair job for missing replicas - -### 11.3 MVP Constraints - -- single-user only -- one gateway node only -- no desktop sync client yet -- no mobile-native app yet -- no public share links -- no full-text search -- no cross-file deduplication requirement -- no erasure coding requirement - -### 11.4 MVP Success Criteria - -- A user can upload files and browse them without caring which backend stores them. -- A user can preview photos, PDFs, and many common videos from the browser. -- A user can survive loss of one configured secondary backend without losing metadata control. -- A user can inspect and retry failed background tasks. -- A user can back up metadata and recover system state onto a new gateway host. - -Current implementation status: - -- see [mvp-status.md](/Users/bytedance/iron/docs/mvp-status.md:1) for the live gap analysis against this MVP definition -- the current repository should be described as a `Product MVP Candidate`, not yet a polished release -- after re-evaluation, the main remaining `P0` items are Web UI and browser-facing auth/session UX -- the backend now has persisted placement policy controls for file classes, preferred/excluded backends, and size-aware non-local replica caps -- most remaining backend work has moved into `P1 hardening` - -## 12. End-State Vision - -### 12.1 Product Vision - -Iron becomes a full personal storage control plane for heterogeneous storage providers, with one logical namespace, rich media workflows, policy-driven placement, and resilient metadata ownership. - -### 12.2 End-State Capabilities - -- Web, desktop, and mobile clients -- remote access support -- selective offline sync -- folder pinning -- advanced media indexing and timeline views -- content deduplication -- optional end-to-end encryption modes -- policy-based lifecycle management -- automatic migration between hot, warm, and cold tiers -- stronger repair and self-healing workflows -- share links and fine-grained permissions -- import and migration tools from common cloud drives -- optional mount support on desktop systems - -## 13. Version Roadmap - -### 13.1 V0: Foundation - -- repository setup -- gateway skeleton -- metadata schema -- local backend -- basic Web UI shell - -Current status: - -- mostly achieved for backend foundation -- a browser-usable Web UI now exists, but it still needs real-user hardening, broader E2E coverage, and polish - -### 13.2 V1: MVP - -- file operations -- local cache -- S3 backend -- Aliyun backend -- Baidu backend -- preview pipeline -- jobs and repair - -Current interpretation: - -- the first shippable V1 should prioritize `auth + Web UI + local + S3 + simple replication + metadata export` -- the backend side of `local + S3 + simple replication + metadata recovery contract` is already largely in place -- Aliyun and Baidu remain in the roadmap, but are not required to call the first public version a usable MVP - -### 13.3 V2: Usability Expansion - -- better backend policies -- desktop client shell -- remote access hardening -- richer media views -- pinning and offline support - -### 13.4 V3: Advanced Storage Intelligence - -- deduplication -- packed small-object storage -- lifecycle movement -- stronger repair orchestration -- optional erasure-coded backend groups - -## 14. Risks and Product Assumptions - -### 14.1 Assumptions - -- The user accepts a heavy local gateway service. -- Free or consumer cloud backends may be unstable and are therefore not trusted as the only source of critical metadata. -- Web-first access is sufficient for MVP. - -### 14.2 Risks - -- Consumer cloud providers may change APIs, rate limits, or anti-abuse rules. -- Preview latency may be inconsistent for files that only exist on slow or unstable backends. -- Browser upload behavior for very large files may require resumable protocols early. -- Media streaming behavior can vary by codec and container format. - -## 15. Open Questions for Technical Design - -- What auth model should the first gateway use: simple local account, reverse-proxy auth, or pluggable auth? -- Should large video files use chunking from day one, or only after file size thresholds are exceeded? -- Should backend adapters be implemented natively or through an abstraction layer such as rclone-compatible gateways? -- How should local cache quotas and eviction policies be tuned? -- What metadata export format should be treated as the recovery contract? -- How much encryption should be mandatory in MVP versus optional by backend policy? diff --git a/docs/technical-architecture-draft.md b/docs/technical-architecture-draft.md deleted file mode 100644 index 1b6c0e4..0000000 --- a/docs/technical-architecture-draft.md +++ /dev/null @@ -1,874 +0,0 @@ -# Iron Technical Architecture Draft - -## 1. Design Goals - -This draft translates the product requirements into an implementation-oriented architecture for the first development phase. - -The design aims to: - -- support a Web-first product with a self-hosted gateway -- keep the codebase simple enough for rapid iteration -- isolate backend-specific complexity behind adapters -- use SQLite first without blocking future database migration -- optimize for media preview and daily file operations - -## 2. Recommended MVP Technical Stack - -## 2.1 Backend - -Recommended: - -- language: `Python` -- HTTP framework: `FastAPI` -- data access: `SQLAlchemy 2.0 ORM` -- migrations: `Alembic` -- schema validation: `Pydantic v2` -- background jobs: in-process worker pool backed by database job records -- object hashing: standard SHA-256 -- config: env-driven settings via a small cached settings module - -Why: - -- Python fits rapid iteration well and has a strong ecosystem for Web APIs, file tooling, storage SDKs, and media processing. -- `FastAPI` provides typed request and response models, automatic OpenAPI generation, and a clean async-friendly API layer. -- MVP is a single-node gateway, so Python is a good fit as long as upload, download, and preview paths are implemented with async streaming semantics instead of full buffering. -- The architecture should keep storage I/O and worker logic explicit so hot paths remain understandable and optimizable. - -## 2.2 Frontend - -Recommended: - -- framework: `React` with `Next.js` App Router or `Vite + React` -- UI library: minimal component primitives such as `shadcn/ui` or `Radix UI` -- data fetching: `TanStack Query` -- video and file preview: native browser capabilities first - -Recommendation between the two: - -- choose `Vite + React` if the Web app is a pure SPA against the gateway API -- choose `Next.js` only if server-rendered routing or later integrated auth flows are clearly desired - -For this project, the current recommendation is: - -- `Vite + React` - -Reason: - -- the gateway already owns the API -- the app is operational rather than content-heavy -- simpler deployment and fewer moving parts for MVP - -## 2.3 Storage Integrations - -Recommended strategy: - -- implement a common backend adapter interface in the gateway -- use native SDKs when stable and practical -- explicitly allow a bridge adapter layer when consumer cloud providers are unstable or poorly documented -- keep bridge usage behind the same storage adapter interface so the rest of the system stays readable - -Proposed initial adapter strategy: - -- local directory: native file operations -- S3: AWS SDK compatible implementation -- Aliyun Drive: adapter with bridge mode support -- Baidu Netdisk: adapter with bridge mode support - -Important note: - -- consumer cloud providers are the highest churn part of the system -- keep their adapters in isolated packages and do not let provider-specific concepts leak into core domain models -- bridge mode is preferred over contaminating core services with provider-specific protocol quirks - -## 3. System Architecture - -```text -Browser - | - v -Web UI - | - v -Gateway API - | - +--> Auth Module - +--> File Service - +--> Preview Service - +--> Backend Service - +--> Job Service - | - +--> Metadata Repository (SQLite) - +--> Cache Manager (local disk) - +--> Storage Engine - | - +--> Placement Planner - +--> Transfer Manager - +--> Replica Manager - +--> Backend Adapters - |- Local - |- S3 - |- Aliyun - |- Baidu -``` - -## 4. Layering Recommendation - -Use a layered async modular monolith for MVP. - -Suggested top-level structure: - -```text -/app - /api - /core - /domain - /services - /repositories - /models - /schemas - /adapters - /db - /storage - /jobs - /preview - /cache - /placement - /workers - /utils -/alembic -/scripts -/tests -/web -/docs -``` - -Alternative package layout if you prefer stricter layering: - -```text -/app - /config - /domain - /service - /repository - /adapter - /http - /db - /storage - /job - /preview - /cache - /placement -``` - -Layer responsibilities: - -- `domain`: core entities, enums, policies, service contracts -- `services`: business orchestration such as upload, move, preview, repair -- `repositories`: persistence operations over SQLite -- `models`: SQLAlchemy ORM models -- `schemas`: API request and response models -- `adapter/db`: SQLite implementations -- `adapter/storage`: backend-specific storage adapters -- `api`: FastAPI routers, dependencies, and HTTP entrypoints -- `jobs`: durable job definitions and scheduling logic -- `cache`: local cache and preview artifact management -- `placement`: backend selection and replication policy logic - -This keeps the codebase simple and still allows later extraction. - -Async design rules: - -- API handlers must be async. -- Repository methods should be async-facing. -- Backend adapters should expose async APIs. -- Blocking SDKs or filesystem-heavy operations must be isolated behind explicit threadpool boundaries. -- No request path should rely on hidden blocking I/O in utility code. - -## 5. Core Domain Model - -The system should not model files as mere backend paths. Use stable IDs. - -Core entities: - -- `User` -- `Directory` -- `FileEntry` -- `FileVersion` -- `Blob` -- `BlobReplica` -- `Backend` -- `UploadSession` -- `Job` -- `PreviewArtifact` -- `CacheEntry` - -### 5.1 Entity Semantics - -- `Directory`: logical folder node in the virtual filesystem -- `FileEntry`: logical file identity tied to a path parent and display name -- `FileVersion`: immutable content snapshot of a file entry -- `Blob`: the stored binary unit, whole-file or chunk object -- `BlobReplica`: one physical copy of a blob on one backend -- `Backend`: configured storage target and its policy metadata -- `PreviewArtifact`: thumbnail, poster frame, or derivative preview asset -- `CacheEntry`: local materialization status for blobs or preview artifacts - -This separation matters because: - -- rename and move should not create a new file identity -- future version history should be possible -- one logical file may map to multiple blobs -- one blob may have multiple replicas - -## 6. SQLite Schema Draft - -Below is the recommended MVP schema direction. Naming can still change, but the concepts should remain stable. - -### 6.1 `users` - -Purpose: - -- local authentication identity for MVP - -Suggested columns: - -- `id` TEXT PRIMARY KEY -- `username` TEXT NOT NULL UNIQUE -- `password_hash` TEXT NOT NULL -- `created_at` DATETIME NOT NULL -- `updated_at` DATETIME NOT NULL - -### 6.2 `directories` - -Purpose: - -- logical folder tree - -Suggested columns: - -- `id` TEXT PRIMARY KEY -- `parent_id` TEXT NULL -- `name` TEXT NOT NULL -- `path_key` TEXT NOT NULL UNIQUE -- `created_at` DATETIME NOT NULL -- `updated_at` DATETIME NOT NULL -- `deleted_at` DATETIME NULL - -Notes: - -- `path_key` is a normalized path index for fast lookups -- root directory can be a special fixed row - -### 6.3 `file_entries` - -Purpose: - -- logical file objects visible in the namespace - -Suggested columns: - -- `id` TEXT PRIMARY KEY -- `directory_id` TEXT NOT NULL -- `name` TEXT NOT NULL -- `mime_type` TEXT NULL -- `size_bytes` INTEGER NOT NULL -- `current_version_id` TEXT NOT NULL -- `is_deleted` BOOLEAN NOT NULL DEFAULT 0 -- `created_at` DATETIME NOT NULL -- `updated_at` DATETIME NOT NULL -- UNIQUE (`directory_id`, `name`) - -### 6.4 `file_versions` - -Purpose: - -- immutable content version records - -Suggested columns: - -- `id` TEXT PRIMARY KEY -- `file_entry_id` TEXT NOT NULL -- `content_hash` TEXT NOT NULL -- `size_bytes` INTEGER NOT NULL -- `storage_layout` TEXT NOT NULL -- `created_at` DATETIME NOT NULL - -Suggested `storage_layout` values: - -- `single` -- `chunked` - -### 6.5 `blobs` - -Purpose: - -- binary storage units - -Suggested columns: - -- `id` TEXT PRIMARY KEY -- `file_version_id` TEXT NOT NULL -- `blob_index` INTEGER NOT NULL -- `kind` TEXT NOT NULL -- `content_hash` TEXT NOT NULL -- `size_bytes` INTEGER NOT NULL -- `logical_offset` INTEGER NOT NULL DEFAULT 0 -- `created_at` DATETIME NOT NULL - -Suggested `kind` values: - -- `file` -- `chunk` -- `thumbnail` -- `poster` - -### 6.6 `blob_replicas` - -Purpose: - -- track every physical copy - -Suggested columns: - -- `id` TEXT PRIMARY KEY -- `blob_id` TEXT NOT NULL -- `backend_id` TEXT NOT NULL -- `storage_key` TEXT NOT NULL -- `status` TEXT NOT NULL -- `etag` TEXT NULL -- `checksum` TEXT NULL -- `size_bytes` INTEGER NOT NULL -- `last_verified_at` DATETIME NULL -- `created_at` DATETIME NOT NULL -- `updated_at` DATETIME NOT NULL - -Suggested `status` values: - -- `pending` -- `ready` -- `missing` -- `corrupt` -- `offline` -- `failed` - -### 6.7 `backends` - -Purpose: - -- backend configuration and policy metadata - -Suggested columns: - -- `id` TEXT PRIMARY KEY -- `name` TEXT NOT NULL UNIQUE -- `type` TEXT NOT NULL -- `stability_class` TEXT NOT NULL -- `read_priority` INTEGER NOT NULL -- `write_priority` INTEGER NOT NULL -- `is_enabled` BOOLEAN NOT NULL DEFAULT 1 -- `config_json` TEXT NOT NULL -- `capacity_hint_bytes` INTEGER NULL -- `last_health_status` TEXT NULL -- `last_health_checked_at` DATETIME NULL -- `created_at` DATETIME NOT NULL -- `updated_at` DATETIME NOT NULL - -Notes: - -- secrets should be encrypted before being stored in `config_json` - -### 6.8 `upload_sessions` - -Purpose: - -- tus-based resumable upload tracking between browser and gateway - -Suggested columns: - -- `id` TEXT PRIMARY KEY -- `directory_id` TEXT NOT NULL -- `filename` TEXT NOT NULL -- `total_size_bytes` INTEGER NOT NULL -- `received_size_bytes` INTEGER NOT NULL -- `status` TEXT NOT NULL -- `temp_path` TEXT NOT NULL -- `tus_upload_url` TEXT NULL -- `upload_metadata_json` TEXT NULL -- `created_at` DATETIME NOT NULL -- `updated_at` DATETIME NOT NULL - -### 6.8a `upload_session_parts` - -Purpose: - -- track uploaded ranges or parts when local resumable state is needed - -Suggested columns: - -- `id` TEXT PRIMARY KEY -- `upload_session_id` TEXT NOT NULL -- `part_number` INTEGER NOT NULL -- `byte_offset` INTEGER NOT NULL -- `size_bytes` INTEGER NOT NULL -- `checksum` TEXT NULL -- `status` TEXT NOT NULL -- `created_at` DATETIME NOT NULL -- `updated_at` DATETIME NOT NULL -- UNIQUE (`upload_session_id`, `part_number`) - -### 6.9 `jobs` - -Purpose: - -- durable background work tracking - -Suggested columns: - -- `id` TEXT PRIMARY KEY -- `kind` TEXT NOT NULL -- `status` TEXT NOT NULL -- `payload_json` TEXT NOT NULL -- `attempt_count` INTEGER NOT NULL DEFAULT 0 -- `max_attempts` INTEGER NOT NULL DEFAULT 5 -- `run_after` DATETIME NOT NULL -- `last_error` TEXT NULL -- `created_at` DATETIME NOT NULL -- `updated_at` DATETIME NOT NULL - -Suggested `kind` values: - -- `replicate_blob` -- `generate_thumbnail` -- `verify_replica` -- `repair_blob` -- `health_check_backend` - -### 6.10 `preview_artifacts` - -Purpose: - -- preview resources mapped back to files or file versions - -Suggested columns: - -- `id` TEXT PRIMARY KEY -- `file_version_id` TEXT NOT NULL -- `artifact_type` TEXT NOT NULL -- `blob_id` TEXT NOT NULL -- `status` TEXT NOT NULL -- `created_at` DATETIME NOT NULL -- `updated_at` DATETIME NOT NULL - -### 6.11 `cache_entries` - -Purpose: - -- local cache tracking - -Suggested columns: - -- `id` TEXT PRIMARY KEY -- `blob_id` TEXT NOT NULL -- `local_path` TEXT NOT NULL -- `cache_kind` TEXT NOT NULL -- `size_bytes` INTEGER NOT NULL -- `status` TEXT NOT NULL -- `last_accessed_at` DATETIME NOT NULL -- `expires_at` DATETIME NULL -- `created_at` DATETIME NOT NULL -- `updated_at` DATETIME NOT NULL - -## 7. API Design Direction - -Use a clean REST API for MVP. gRPC is unnecessary at this stage. - -Suggested route groups: - -- `/api/auth` -- `/api/files` -- `/api/directories` -- `/api/uploads` -- `/api/previews` -- `/api/backends` -- `/api/jobs` -- `/api/system` - -### 7.1 Key Endpoints - -Examples: - -- `POST /api/auth/login` -- `GET /api/directories/:id/children` -- `POST /api/directories` -- `POST /api/uploads` -- `HEAD /api/uploads/:id` -- `PATCH /api/uploads/:id` -- `POST /api/uploads/:id/finalize` -- `GET /api/files/:id` -- `GET /api/files/:id/download` -- `GET /api/files/:id/stream` -- `GET /api/files/:id/thumbnail` -- `POST /api/files/:id/move` -- `POST /api/files/:id/rename` -- `DELETE /api/files/:id` -- `GET /api/backends` -- `POST /api/backends` -- `POST /api/backends/:id/check` -- `GET /api/jobs` -- `POST /api/jobs/:id/retry` - -## 8. Upload Pipeline - -Recommended MVP flow: - -1. Browser creates a `tus` upload resource on the gateway. -2. Browser uploads file content through `tus` patch requests. -3. Gateway writes incoming bytes to a temporary local ingest path. -4. Gateway tracks upload progress in the database. -5. When upload completes, gateway computes metadata such as mime type, hash, and size. -6. Gateway creates logical records in the database. -7. Gateway writes the first required replica synchronously to a chosen primary backend. -8. Gateway marks the file visible when minimum durability criteria are met. -9. Gateway schedules secondary replication and preview generation jobs. - -Important MVP rule: - -- visibility should happen only after at least one durable backend write succeeds - -Recommended upload protocol choice: - -- adopt `tus` for resumable uploads in MVP -- keep upload session state explicit in the database so the gateway can coordinate file creation, placement, and recovery -- treat `tus` as the ingress protocol, while keeping storage placement and replica logic as internal concerns -- add a gateway-side finalize step so a completed `tus` upload becomes a managed logical file only after validation and first durable write - -## 9. Download and Stream Pipeline - -### 9.1 Download - -1. User requests file download. -2. Gateway resolves current file version and blob layout. -3. Gateway checks local cache. -4. If cache misses, gateway chooses the best available replica based on backend health and priority. -5. Gateway streams the response to the client while optionally filling cache. - -### 9.2 Video Streaming - -Recommended strategy: - -- support HTTP Range requests on the gateway -- map byte ranges to full-file or chunked blob reads -- prefer local cache, then stable backends, then opportunistic backends - -Why: - -- this is the most standard Web-compatible approach -- native browser video players work with it -- it avoids inventing a custom preview transport too early - -## 10. Preview Pipeline - -Recommended preview support for MVP: - -- images: generate thumbnails and serve original or scaled versions -- PDFs: browser inline preview using direct file or proxy response -- videos: poster image plus range-based playback - -Suggested preview job flow: - -1. file ingest completes -2. preview worker inspects mime type -3. thumbnail or poster extraction runs if applicable -4. derivative asset is stored as a blob -5. preview artifact record is created - -Tooling options: - -- `ffmpeg` for video poster generation and optional transcoding later -- image library for thumbnails - -Recommendation: - -- do not include full video transcoding in MVP -- stick to poster extraction and native playback for supported source formats - -## 11. Placement and Replica Policy - -Recommended MVP policy engine: - -- every backend has a policy score -- backend selection uses stability class, enabled state, health state, and priority -- every file has a placement rule selected by file class - -Suggested first placement classes: - -- `critical-metadata` -- `document` -- `photo` -- `video` -- `cold-archive` - -Suggested default behavior: - -- metadata exports: local + stable backend -- documents: stable primary, optional opportunistic secondary -- photos: stable primary, opportunistic secondary if available -- videos: stable primary, optional async secondary depending on size and backend capacity - -## 12. Authentication Recommendation - -For MVP: - -- one local user table -- session cookie auth or signed token auth - -Recommendation: - -- use a simple local auth model first -- the current backend implementation uses persisted bearer-token sessions -- the future Web UI may wrap that with cookie handling or translate it into cookie-based sessions -- avoid introducing OAuth or external identity providers in MVP - -Reason: - -- local self-hosted single-user setup does not need auth complexity yet - -## 13. Secrets and Security - -Recommended MVP measures: - -- encrypt backend credentials before storing them in SQLite -- use a gateway master key from environment or local config file -- redact sensitive fields in logs -- require login for all file and backend management routes - -## 14. Caching Strategy - -Suggested MVP cache layers: - -- ingest temp area -- blob cache -- preview cache - -Suggested cache policy: - -- LRU-like eviction based on size budget and recent access -- separate quotas for blob cache and preview cache - -Do not make cache semantics too smart in MVP. Keep them observable and debuggable. - -## 15. Background Jobs - -Recommended architecture: - -- jobs stored in SQLite -- one scheduler loop claims runnable jobs -- small worker pool executes jobs -- failed jobs are retried with backoff -- workers may call synchronous SDKs through thread pools where async-native clients are unavailable - -Why not a separate queue system yet: - -- too much operational complexity for single-node MVP -- SQLite durability is enough for this stage if jobs are idempotent - -Job design rule: - -- every job must be safe to retry - -## 16. Adapter Interface Draft - -Define a storage adapter interface around core object operations. - -Suggested interface shape: - -```python -from typing import Protocol - - -class StorageAdapter(Protocol): - adapter_type: str - - async def check(self) -> None: ... - - async def stat(self, key: str) -> "ObjectInfo": ... - - async def put( - self, - key: str, - stream, - size: int, - options: "PutOptions", - ) -> "ObjectInfo": ... - - async def get( - self, - key: str, - byte_range: "ByteRange | None" = None, - ) -> tuple[object, "ObjectInfo"]: ... - - async def delete(self, key: str) -> None: ... -``` - -Notes: - -- use `key` as the physical storage identifier -- keep adapters object-oriented even for local filesystem storage -- do not let adapters know about directories, files, or logical paths -- if a provider SDK is synchronous, isolate it inside the adapter and offload blocking work explicitly - -Suggested adapter split: - -- `NativeStorageAdapter`: direct provider SDK or protocol implementation -- `BridgeStorageAdapter`: talks to an external bridge service or compatibility layer - -Core services should not need to know which one is in use. - -## 17. Physical Key Strategy - -Do not store files in backends by original user path. - -Recommended key pattern: - -- `blobs/sha256_prefix/full_hash` -- `previews/file_version_id/artifact_type` -- `exports/date/export_id` - -Why: - -- avoids backend path rename costs -- reduces coupling to logical namespace changes -- makes replication and repair much simpler - -## 18. Suggested First Implementation Order - -1. config loading and app bootstrap -2. SQLite migrations -3. domain models and repository interfaces -4. local backend adapter -5. file ingest pipeline -6. file listing and basic Web UI -7. blob cache -8. download pipeline -9. image and PDF preview -10. S3 adapter -11. job system -12. Aliyun adapter -13. Baidu adapter -14. video poster generation and range streaming polish - -## 19. Key Architecture Decisions To Confirm - -These are the most important choices still worth discussing before coding: - -- Backend language: `Python` is confirmed. -- Frontend stack: confirm `Vite + React`. -- Database access: `SQLAlchemy ORM` is confirmed. -- Consumer cloud adapters: bridge-capable adapter strategy is confirmed. -- Upload protocol: `tus` is confirmed. -- Credential encryption: define whether to use a local master key file or environment variable only. -- Async model: async-first architecture is confirmed; blocking operations must be isolated explicitly. - -## 20. Recommendation Summary - -For the first implementation phase, the strongest recommendation is: - -- backend: `Python + FastAPI` -- frontend: `Vite + React` -- database: `SQLite` -- data access: `SQLAlchemy ORM` -- jobs: in-process durable workers -- architecture: layered async modular monolith -- storage strategy: replication first, chunking only where beneficial -- upload strategy: `tus` resumable upload with gateway-managed session records -- preview strategy: native browser preview plus range-based streaming -- backend order: local, S3, Aliyun, Baidu -- adapter policy: native where clean, bridge where provider complexity would otherwise damage maintainability -- Python libraries to prefer: `SQLAlchemy 2.0`, `Alembic`, `Pydantic v2`, `httpx`, `boto3` or compatible S3 SDK, and `ffmpeg` integration for previews - -This combination is the best balance between simplicity, stream handling, self-hosted deployment, and future extensibility. - -## 21. Implemented Foundation Status - -For a product-level view of what is still missing before Iron can be called a usable MVP, see [mvp-status.md](/Users/bytedance/iron/docs/mvp-status.md:1). - -The repository now includes the following implemented baseline pieces: - -- `uv`-managed local virtual environment workflow -- FastAPI application bootstrap and routing shell -- async SQLAlchemy engine setup for SQLite -- initial ORM entity definitions for the core schema -- first Alembic migration for the initial metadata model -- readiness check backed by a real database ping -- baseline automated tests for config, system routes, and migration application -- first auth slice: - - bootstrap local admin user - - login, logout, and current-user APIs - - bearer-token session persistence and route protection -- first Web shell slice: - - `/app` browser entry - - login form and session bootstrap - - root directory listing - - backend listing -- first metadata feature slice for directories: - - root directory bootstrap - - create directory - - list child directories -- first file metadata slice: - - internal file metadata creation service for future upload finalization - - mixed directory and file child listings - - file detail API - - recycle-bin file deletion and restore flow -- first upload slice: - - tus upload session creation - - offset inspection - - sequential patch uploads into temp ingest storage - - finalize flow wired into file metadata creation - - finalize flow persists the primary blob into the default local backend - - local `blob_replicas` records are created for finalized uploads - - finalize enqueues non-local replication jobs -- first download slice: - - file download endpoint resolves local ready replicas - - downloaded content is served from persisted local backend objects - - remote fallback can materialize ready replicas into local cache -- first stream slice: - - file stream endpoint resolves local ready replicas - - HTTP Range support for partial content reads -- first preview slice: - - inline preview endpoint for images and PDFs persisted to local backend -- first backend-visibility slice: - - backend list endpoint for configured backends -- first backend-management slice: - - backend creation for local and s3 config metadata - - local and s3 backend health checks - - backend disable flow -- first jobs-and-ops slice: - - durable SQLite job records - - in-process polling loop - - job list/detail/retry/run APIs - - backend health-check enqueue flow - - declarative file reconcile enqueue flow - - metadata export and import APIs -- first policy slice: - - persisted placement policies by file class - - preferred and excluded backend controls - - size-aware non-local replica caps - - placement preview API for product-facing inspection - - upload finalize uses policy-selected secondary targets - - reconciliation jobs converge actual replicas toward desired state -- first namespace-mutation slice: - - directory rename - - directory move with descendant path updates - - file rename - - file move - -From a product perspective, the next logical step is no longer more basic metadata CRUD. It is: - -- deepen the browser shell into a fuller Web UI -- deepen browser-facing session handling on top of the current auth backend -- add post-restore hardening and stronger preview derivative generation -- add the first consumer-cloud backend - -For the product-level gap assessment, use [mvp-status.md](/Users/bytedance/iron/docs/mvp-status.md:1) as the source of truth. - -For day-to-day continuation guidance and current repo reality, use [handoff.md](/Users/bytedance/iron/docs/handoff.md:1) first. diff --git a/docs/ui-playwright-standard.md b/docs/ui-playwright-standard.md deleted file mode 100644 index 38933db..0000000 --- a/docs/ui-playwright-standard.md +++ /dev/null @@ -1,137 +0,0 @@ -# Iron UI Playwright Standard - -Last updated: 2026-04-14 - -## 1. Purpose - -All future Iron Web UI changes must be validated against a real running service, using Python Playwright flows and screenshot review. - -This is the default UI modification standard for the project. - -## 2. Required Workflow For Any UI Change - -Before considering a UI change complete: - -1. build the frontend assets -2. start the real local service -3. run the Python Playwright regression flow -4. generate screenshots into `output/playwright/` -5. review screenshots for layout, spacing, responsiveness, and interaction states -6. fix issues found in the screenshots or browser flow -7. rerun the Playwright flow after the fixes - -Do not treat UI work as complete based only on code review or unit tests. - -## 3. Required Validation Levels - -Every meaningful UI change should validate: - -- login flow -- files workspace render -- primary file operation flow affected by the change -- upload flow if the change touches file workflows -- screenshots of the affected page states - -## 4. Tooling Standard - -Use: - -- Python Playwright code for repeatable regression checks -- a real locally running Iron service -- screenshot artifacts for visual inspection - -Preferred command: - -```bash -./.venv/bin/python scripts/ui_playwright_smoke.py -``` - -Pytest entrypoint for CI or full local regression: - -```bash -./.venv/bin/python -m pytest tests/e2e/test_web_ui_playwright.py -``` - -Artifacts must be written to: - -```text -output/playwright/ -``` - -## 5. Screenshot Review Checklist - -Every screenshot review should check: - -- overall balance of the layout -- spacing consistency -- text truncation or wrapping problems -- button grouping and hierarchy -- panel heights and overflow behavior -- empty-state clarity -- table/list readability -- dialog layout and action placement -- mobile or narrow-width resilience when the changed surface is responsive - -## 6. Interaction Review Checklist - -Every flow review should check: - -- click targets are obvious -- primary actions are easy to find -- selection state is clear -- navigation state is preserved where expected -- success feedback is visible -- error feedback is actionable -- dialogs do not rely on browser-native prompts -- repetitive tasks do not require unnecessary clicks - -## 7. Regression Scope For Iron - -Current baseline regression should cover: - -- login -- files page -- create folder -- upload from the files page -- authenticated file download -- authenticated inline image preview -- no failing `/api` browser requests during the flow -- files search/filter visibility -- uploads page render - -As new UI capabilities are added, extend the Python Playwright script to cover them. - -## 8. Expected Outputs - -Each run should produce: - -- pass/fail console output -- screenshots -- clear error exit code on failure - -Recommended screenshot set: - -- `login.png` -- `files-empty-or-initial.png` -- `files-after-create-folder.png` -- `files-after-upload.png` -- `uploads-page.png` - -## 9. Standard For Future UI Optimization - -For future layout or interaction polish: - -- first reproduce the current state in Playwright -- capture before screenshots -- make a focused UI change -- capture after screenshots -- compare the changed states visually -- keep the regression script updated if the flow meaningfully changes - -## 10. Non-Negotiable Rule - -No future UI optimization should be merged or considered complete without: - -- a real service run -- Python Playwright flow validation -- screenshot review diff --git a/docs/web-ui-technical-design.md b/docs/web-ui-technical-design.md deleted file mode 100644 index 22d2225..0000000 --- a/docs/web-ui-technical-design.md +++ /dev/null @@ -1,342 +0,0 @@ -# Iron Web UI Technical Design - -Last updated: 2026-04-15 - -## 1. Goal - -This document defines the first real Web UI implementation for Iron as a product, not as a backend demo shell. - -The design target is: - -- a browser-based personal cloud drive that feels familiar to users of mainstream drive products -- a file-first experience, with storage and operational controls available when needed -- a frontend architecture that can grow without forcing a backend rewrite - -## 2. Product Direction - -Iron should feel like: - -- a calm personal drive for everyday browsing, upload, preview, and organization -- a storage-aware control surface when the user wants to inspect health, jobs, or replica state - -Iron should not feel like: - -- a generic admin dashboard -- a developer-only operations console -- a backend demo that exposes storage complexity too early - -The default user mental model should be: - -- there is one drive -- files live in one namespace -- storage placement is handled by the system -- operational details are inspectable but not required for normal use - -## 3. UX Principles - -- file browsing is the center of the product -- preview before download whenever possible -- complex storage details stay secondary -- important system state stays visible but low-noise -- common actions should be reachable in one or two clicks -- the desktop layout should resemble familiar drive products, while mobile should collapse into simple list-detail flows - -## 4. Primary Information Architecture - -Recommended primary navigation: - -1. Files -2. Uploads -3. Recycle Bin -4. Storage -5. Jobs - -Secondary entry points: - -- search -- account/session menu -- future settings/policies entry - -## 5. Layout Model - -The app should use a three-zone desktop layout: - -- left sidebar: primary navigation and directory shortcuts -- main workspace: listing, toolbar, breadcrumbs, upload flows -- right detail panel: preview, metadata, storage status, contextual actions - -Mobile and narrow tablet should collapse to: - -- top bar -- main content -- bottom sheet or dedicated route for file detail - -## 6. Visual Direction - -Reference class: - -- mainstream Web drive layout -- product-first, not enterprise-heavy -- spacious, legible, and calm - -Current visual language: - -- ownCloud-inspired file-product structure, without copying exact visual assets -- desktop-first left navigation, central file workspace, and right contextual panels -- compact first-screen layouts with pagination for long lists instead of page-length scrolling -- restrained neutral surfaces, blue primary actions, high-contrast text, and light borders -- UI changes must be reviewed from real Playwright screenshots before being considered complete - -## 7. Frontend Technical Stack - -Recommended stack: - -- `Vite` -- `React` -- `TypeScript` -- `React Router` -- `TanStack Query` -- `Radix UI` primitives only when needed later - -State strategy: - -- server state: `TanStack Query` -- local UI state: React state and context -- persistent auth/session state: small local storage wrapper - -Rationale: - -- Iron is a long-lived SPA-style product -- the backend already owns API and routing responsibility -- Vite keeps build and iteration simple -- React Router and TanStack Query are enough for current complexity - -## 8. App Structure - -Recommended source layout: - -```text -frontend/ - src/ - app/ - App.tsx - router.tsx - providers.tsx - components/ - layout/ - feedback/ - files/ - storage/ - features/ - auth/ - files/ - uploads/ - recycle-bin/ - storage/ - jobs/ - lib/ - api/ - auth/ - format/ - utils/ - routes/ - styles/ -``` - -Guidance: - -- keep route-level pages under `routes/` -- keep API wrappers under `lib/api/` -- keep reusable visual building blocks under `components/` -- keep feature-specific hooks and rendering helpers in each feature folder - -## 9. Route Plan - -Initial route map: - -- `/app/login` -- `/app/files` -- `/app/files/:directoryId` -- `/app/uploads` -- `/app/recycle-bin` -- `/app/storage` -- `/app/jobs` - -Behavior: - -- `/app` should redirect authenticated users to `/app/files` -- unauthenticated users should be sent to `/app/login` - -## 10. Data and API Integration - -The first Web UI should reuse the current backend API surface. - -Auth: - -- `POST /api/auth/login` -- `GET /api/auth/me` -- `POST /api/auth/logout` - -Files and directories: - -- `GET /api/directories/{directory_id}/children` -- `GET /api/files/{file_id}` -- `GET /api/files/{file_id}/download` -- `GET /api/files/{file_id}/preview` -- `GET /api/files/{file_id}/stream` -- `POST /api/files/{file_id}/rename` -- `POST /api/files/{file_id}/move` -- `DELETE /api/files/{file_id}` -- `POST /api/files/{file_id}/restore` -- `GET /api/files/recycle-bin` -- `POST /api/directories` -- `POST /api/directories/{directory_id}/rename` -- `POST /api/directories/{directory_id}/move` - -Uploads: - -- `POST /files` -- `HEAD /files/{upload_id}` -- `PATCH /files/{upload_id}` -- `POST /api/uploads/{upload_id}/finalize` - -Operations: - -- `GET /api/backends` -- `POST /api/backends/{backend_id}/check` -- `POST /api/backends/{backend_id}/disable` -- `GET /api/jobs` -- `GET /api/jobs/{job_id}` -- `POST /api/jobs/{job_id}/retry` -- `POST /api/jobs/run-pending` -- `POST /api/jobs/enqueue-health-checks` -- `POST /api/jobs/enqueue-full-reconcile` - -Browser content access: - -- JSON APIs use bearer-token `Authorization` headers through the shared API client -- file download, preview, and stream actions must not use naked `/api/files/...` URLs in media tags or `window.open` -- browser preview/download should fetch the file with authorization, create an object URL, and revoke it after use -- Playwright E2E must cover authenticated file access whenever these flows change - -## 11. MVP Page Scope - -### 11.1 Files - -Must include: - -- breadcrumb navigation -- file/folder listing -- list and grid view toggle -- select item -- right-side detail panel -- preview for image, PDF, and video-capable files -- rename, move, delete, download actions -- create folder action - -### 11.2 Uploads - -Must include: - -- file picker -- drag and drop -- upload queue -- progress display -- finalize into current directory - -### 11.3 Recycle Bin - -Must include: - -- deleted file list -- restore action -- metadata visibility - -### 11.4 Storage - -Must include: - -- backend cards/list -- health state -- last checked time -- disable action -- trigger check action - -### 11.5 Jobs - -Must include: - -- recent jobs list -- status emphasis -- retry for retryable/failed jobs -- trigger run pending -- enqueue health checks -- enqueue full reconcile - -## 12. Interaction Model - -Toolbar behavior in Files: - -- primary action: upload -- secondary actions: new folder, refresh, view toggle -- selection actions move into detail panel and contextual menus - -Detail panel behavior: - -- opens on item selection -- shows metadata immediately -- shows preview when supported -- keeps actions grouped by safety - -Feedback behavior: - -- use inline toasts or banners for successful mutations -- keep destructive confirmations explicit -- show loading skeletons instead of blank content - -## 13. Accessibility and Responsiveness - -The first Web UI should include: - -- keyboard reachable navigation -- visible focus states -- semantic buttons, forms, lists, and headings -- color contrast suitable for long sessions -- responsive behavior for widths down to mobile phone layouts - -## 14. Delivery Sequence - -Recommended implementation order: - -1. frontend scaffold and shared layout -2. auth flow and route guards -3. files page and detail panel -4. uploads page and upload client -5. recycle bin -6. storage and jobs pages -7. polish, responsiveness, and interaction refinement - -## 15. Non-Goals For This Slice - -Do not block the first Web UI on: - -- advanced settings UI -- full-text search -- share links -- policy editor UX -- multi-window drag/drop complexity -- perfect media derivative support - -The first goal is a strong file-first personal drive UI built on the current backend. - -## 16. Current Implementation Snapshot - -Implemented as of 2026-04-15: - -- login and authenticated route guard -- files workspace with local folder navigation, breadcrumbs, search, sort, list/grid toggle, pagination, contextual row actions, dialogs, and inspector -- authenticated browser download and inline image preview via blob-backed object URLs -- uploads page with drag/drop, file picker, queue, target folder tree, and pagination -- recycle bin, storage, and jobs pages with compact desktop workspaces -- Python Playwright smoke/E2E flow with screenshots under `output/playwright/` -- pytest E2E entry at `tests/e2e/test_web_ui_playwright.py`