# Iron Technical Architecture Draft

## 1. Design Goals

This draft translates the product requirements into an implementation-oriented architecture for the first development phase.

The design aims to:

- support a Web-first product with a self-hosted gateway
- keep the codebase simple enough for rapid iteration
- isolate backend-specific complexity behind adapters
- use SQLite first without blocking future database migration
- optimize for media preview and daily file operations

## 2. Recommended MVP Technical Stack

## 2.1 Backend

Recommended:

- language: `Python`
- HTTP framework: `FastAPI`
- data access: `SQLAlchemy 2.0 ORM`
- migrations: `Alembic`
- schema validation: `Pydantic v2`
- background jobs: in-process worker pool backed by database job records
- object hashing: standard SHA-256
- config: env-driven settings via a small cached settings module

Why:

- Python fits rapid iteration well and has a strong ecosystem for Web APIs, file tooling, storage SDKs, and media processing.
- `FastAPI` provides typed request and response models, automatic OpenAPI generation, and a clean async-friendly API layer.
- MVP is a single-node gateway, so Python is a good fit as long as upload, download, and preview paths are implemented with async streaming semantics instead of full buffering.
- The architecture should keep storage I/O and worker logic explicit so hot paths remain understandable and optimizable.

## 2.2 Frontend

Recommended:

- framework: `React` with `Next.js` App Router or `Vite + React`
- UI library: minimal component primitives such as `shadcn/ui` or `Radix UI`
- data fetching: `TanStack Query`
- video and file preview: native browser capabilities first

Recommendation between the two:

- choose `Vite + React` if the Web app is a pure SPA against the gateway API
- choose `Next.js` only if server-rendered routing or later integrated auth flows are clearly desired

For this project, the current recommendation is:

- `Vite + React`

Reason:

- the gateway already owns the API
- the app is operational rather than content-heavy
- simpler deployment and fewer moving parts for MVP

## 2.3 Storage Integrations

Recommended strategy:

- implement a common backend adapter interface in the gateway
- use native SDKs when stable and practical
- explicitly allow a bridge adapter layer when consumer cloud providers are unstable or poorly documented
- keep bridge usage behind the same storage adapter interface so the rest of the system stays readable

Proposed initial adapter strategy:

- local directory: native file operations
- S3: AWS SDK compatible implementation
- Aliyun Drive: adapter with bridge mode support
- Baidu Netdisk: adapter with bridge mode support

Important note:

- consumer cloud providers are the highest churn part of the system
- keep their adapters in isolated packages and do not let provider-specific concepts leak into core domain models
- bridge mode is preferred over contaminating core services with provider-specific protocol quirks

## 3. System Architecture

```text
Browser
  |
  v
Web UI
  |
  v
Gateway API
  |
  +--> Auth Module
  +--> File Service
  +--> Preview Service
  +--> Backend Service
  +--> Job Service
  |
  +--> Metadata Repository (SQLite)
  +--> Cache Manager (local disk)
  +--> Storage Engine
          |
          +--> Placement Planner
          +--> Transfer Manager
          +--> Replica Manager
          +--> Backend Adapters
                  |- Local
                  |- S3
                  |- Aliyun
                  |- Baidu
```

## 4. Layering Recommendation

Use a layered async modular monolith for MVP.

Suggested top-level structure:

```text
/app
  /api
  /core
  /domain
  /services
  /repositories
  /models
  /schemas
  /adapters
    /db
    /storage
  /jobs
  /preview
  /cache
  /placement
  /workers
  /utils
/alembic
/scripts
/tests
/web
/docs
```

Alternative package layout if you prefer stricter layering:

```text
/app
  /config
  /domain
  /service
  /repository
  /adapter
    /http
    /db
    /storage
  /job
  /preview
  /cache
  /placement
```

Layer responsibilities:

- `domain`: core entities, enums, policies, service contracts
- `services`: business orchestration such as upload, move, preview, repair
- `repositories`: persistence operations over SQLite
- `models`: SQLAlchemy ORM models
- `schemas`: API request and response models
- `adapter/db`: SQLite implementations
- `adapter/storage`: backend-specific storage adapters
- `api`: FastAPI routers, dependencies, and HTTP entrypoints
- `jobs`: durable job definitions and scheduling logic
- `cache`: local cache and preview artifact management
- `placement`: backend selection and replication policy logic

This keeps the codebase simple and still allows later extraction.

Async design rules:

- API handlers must be async.
- Repository methods should be async-facing.
- Backend adapters should expose async APIs.
- Blocking SDKs or filesystem-heavy operations must be isolated behind explicit threadpool boundaries.
- No request path should rely on hidden blocking I/O in utility code.

## 5. Core Domain Model

The system should not model files as mere backend paths. Use stable IDs.

Core entities:

- `User`
- `Directory`
- `FileEntry`
- `FileVersion`
- `Blob`
- `BlobReplica`
- `Backend`
- `UploadSession`
- `Job`
- `PreviewArtifact`
- `CacheEntry`

### 5.1 Entity Semantics

- `Directory`: logical folder node in the virtual filesystem
- `FileEntry`: logical file identity tied to a path parent and display name
- `FileVersion`: immutable content snapshot of a file entry
- `Blob`: the stored binary unit, whole-file or chunk object
- `BlobReplica`: one physical copy of a blob on one backend
- `Backend`: configured storage target and its policy metadata
- `PreviewArtifact`: thumbnail, poster frame, or derivative preview asset
- `CacheEntry`: local materialization status for blobs or preview artifacts

This separation matters because:

- rename and move should not create a new file identity
- future version history should be possible
- one logical file may map to multiple blobs
- one blob may have multiple replicas

## 6. SQLite Schema Draft

Below is the recommended MVP schema direction. Naming can still change, but the concepts should remain stable.

### 6.1 `users`

Purpose:

- local authentication identity for MVP

Suggested columns:

- `id` TEXT PRIMARY KEY
- `username` TEXT NOT NULL UNIQUE
- `password_hash` TEXT NOT NULL
- `created_at` DATETIME NOT NULL
- `updated_at` DATETIME NOT NULL

### 6.2 `directories`

Purpose:

- logical folder tree

Suggested columns:

- `id` TEXT PRIMARY KEY
- `parent_id` TEXT NULL
- `name` TEXT NOT NULL
- `path_key` TEXT NOT NULL UNIQUE
- `created_at` DATETIME NOT NULL
- `updated_at` DATETIME NOT NULL
- `deleted_at` DATETIME NULL

Notes:

- `path_key` is a normalized path index for fast lookups
- root directory can be a special fixed row

### 6.3 `file_entries`

Purpose:

- logical file objects visible in the namespace

Suggested columns:

- `id` TEXT PRIMARY KEY
- `directory_id` TEXT NOT NULL
- `name` TEXT NOT NULL
- `mime_type` TEXT NULL
- `size_bytes` INTEGER NOT NULL
- `current_version_id` TEXT NOT NULL
- `is_deleted` BOOLEAN NOT NULL DEFAULT 0
- `created_at` DATETIME NOT NULL
- `updated_at` DATETIME NOT NULL
- UNIQUE (`directory_id`, `name`)

### 6.4 `file_versions`

Purpose:

- immutable content version records

Suggested columns:

- `id` TEXT PRIMARY KEY
- `file_entry_id` TEXT NOT NULL
- `content_hash` TEXT NOT NULL
- `size_bytes` INTEGER NOT NULL
- `storage_layout` TEXT NOT NULL
- `created_at` DATETIME NOT NULL

Suggested `storage_layout` values:

- `single`
- `chunked`

### 6.5 `blobs`

Purpose:

- binary storage units

Suggested columns:

- `id` TEXT PRIMARY KEY
- `file_version_id` TEXT NOT NULL
- `blob_index` INTEGER NOT NULL
- `kind` TEXT NOT NULL
- `content_hash` TEXT NOT NULL
- `size_bytes` INTEGER NOT NULL
- `logical_offset` INTEGER NOT NULL DEFAULT 0
- `created_at` DATETIME NOT NULL

Suggested `kind` values:

- `file`
- `chunk`
- `thumbnail`
- `poster`

### 6.6 `blob_replicas`

Purpose:

- track every physical copy

Suggested columns:

- `id` TEXT PRIMARY KEY
- `blob_id` TEXT NOT NULL
- `backend_id` TEXT NOT NULL
- `storage_key` TEXT NOT NULL
- `status` TEXT NOT NULL
- `etag` TEXT NULL
- `checksum` TEXT NULL
- `size_bytes` INTEGER NOT NULL
- `last_verified_at` DATETIME NULL
- `created_at` DATETIME NOT NULL
- `updated_at` DATETIME NOT NULL

Suggested `status` values:

- `pending`
- `ready`
- `missing`
- `corrupt`
- `offline`
- `failed`

### 6.7 `backends`

Purpose:

- backend configuration and policy metadata

Suggested columns:

- `id` TEXT PRIMARY KEY
- `name` TEXT NOT NULL UNIQUE
- `type` TEXT NOT NULL
- `stability_class` TEXT NOT NULL
- `read_priority` INTEGER NOT NULL
- `write_priority` INTEGER NOT NULL
- `is_enabled` BOOLEAN NOT NULL DEFAULT 1
- `config_json` TEXT NOT NULL
- `capacity_hint_bytes` INTEGER NULL
- `last_health_status` TEXT NULL
- `last_health_checked_at` DATETIME NULL
- `created_at` DATETIME NOT NULL
- `updated_at` DATETIME NOT NULL

Notes:

- secrets should be encrypted before being stored in `config_json`

### 6.8 `upload_sessions`

Purpose:

- tus-based resumable upload tracking between browser and gateway

Suggested columns:

- `id` TEXT PRIMARY KEY
- `directory_id` TEXT NOT NULL
- `filename` TEXT NOT NULL
- `total_size_bytes` INTEGER NOT NULL
- `received_size_bytes` INTEGER NOT NULL
- `status` TEXT NOT NULL
- `temp_path` TEXT NOT NULL
- `tus_upload_url` TEXT NULL
- `upload_metadata_json` TEXT NULL
- `created_at` DATETIME NOT NULL
- `updated_at` DATETIME NOT NULL

### 6.8a `upload_session_parts`

Purpose:

- track uploaded ranges or parts when local resumable state is needed

Suggested columns:

- `id` TEXT PRIMARY KEY
- `upload_session_id` TEXT NOT NULL
- `part_number` INTEGER NOT NULL
- `byte_offset` INTEGER NOT NULL
- `size_bytes` INTEGER NOT NULL
- `checksum` TEXT NULL
- `status` TEXT NOT NULL
- `created_at` DATETIME NOT NULL
- `updated_at` DATETIME NOT NULL
- UNIQUE (`upload_session_id`, `part_number`)

### 6.9 `jobs`

Purpose:

- durable background work tracking

Suggested columns:

- `id` TEXT PRIMARY KEY
- `kind` TEXT NOT NULL
- `status` TEXT NOT NULL
- `payload_json` TEXT NOT NULL
- `attempt_count` INTEGER NOT NULL DEFAULT 0
- `max_attempts` INTEGER NOT NULL DEFAULT 5
- `run_after` DATETIME NOT NULL
- `last_error` TEXT NULL
- `created_at` DATETIME NOT NULL
- `updated_at` DATETIME NOT NULL

Suggested `kind` values:

- `replicate_blob`
- `generate_thumbnail`
- `verify_replica`
- `repair_blob`
- `health_check_backend`

### 6.10 `preview_artifacts`

Purpose:

- preview resources mapped back to files or file versions

Suggested columns:

- `id` TEXT PRIMARY KEY
- `file_version_id` TEXT NOT NULL
- `artifact_type` TEXT NOT NULL
- `blob_id` TEXT NOT NULL
- `status` TEXT NOT NULL
- `created_at` DATETIME NOT NULL
- `updated_at` DATETIME NOT NULL

### 6.11 `cache_entries`

Purpose:

- local cache tracking

Suggested columns:

- `id` TEXT PRIMARY KEY
- `blob_id` TEXT NOT NULL
- `local_path` TEXT NOT NULL
- `cache_kind` TEXT NOT NULL
- `size_bytes` INTEGER NOT NULL
- `status` TEXT NOT NULL
- `last_accessed_at` DATETIME NOT NULL
- `expires_at` DATETIME NULL
- `created_at` DATETIME NOT NULL
- `updated_at` DATETIME NOT NULL

## 7. API Design Direction

Use a clean REST API for MVP. gRPC is unnecessary at this stage.

Suggested route groups:

- `/api/auth`
- `/api/files`
- `/api/directories`
- `/api/uploads`
- `/api/previews`
- `/api/backends`
- `/api/jobs`
- `/api/system`

### 7.1 Key Endpoints

Examples:

- `POST /api/auth/login`
- `GET /api/directories/:id/children`
- `POST /api/directories`
- `POST /api/uploads`
- `HEAD /api/uploads/:id`
- `PATCH /api/uploads/:id`
- `POST /api/uploads/:id/finalize`
- `GET /api/files/:id`
- `GET /api/files/:id/download`
- `GET /api/files/:id/stream`
- `GET /api/files/:id/thumbnail`
- `POST /api/files/:id/move`
- `POST /api/files/:id/rename`
- `DELETE /api/files/:id`
- `GET /api/backends`
- `POST /api/backends`
- `POST /api/backends/:id/check`
- `GET /api/jobs`
- `POST /api/jobs/:id/retry`

## 8. Upload Pipeline

Recommended MVP flow:

1. Browser creates a `tus` upload resource on the gateway.
2. Browser uploads file content through `tus` patch requests.
3. Gateway writes incoming bytes to a temporary local ingest path.
4. Gateway tracks upload progress in the database.
5. When upload completes, gateway computes metadata such as mime type, hash, and size.
6. Gateway creates logical records in the database.
7. Gateway writes the first required replica synchronously to a chosen primary backend.
8. Gateway marks the file visible when minimum durability criteria are met.
9. Gateway schedules secondary replication and preview generation jobs.

Important MVP rule:

- visibility should happen only after at least one durable backend write succeeds

Recommended upload protocol choice:

- adopt `tus` for resumable uploads in MVP
- keep upload session state explicit in the database so the gateway can coordinate file creation, placement, and recovery
- treat `tus` as the ingress protocol, while keeping storage placement and replica logic as internal concerns
- add a gateway-side finalize step so a completed `tus` upload becomes a managed logical file only after validation and first durable write

## 9. Download and Stream Pipeline

### 9.1 Download

1. User requests file download.
2. Gateway resolves current file version and blob layout.
3. Gateway checks local cache.
4. If cache misses, gateway chooses the best available replica based on backend health and priority.
5. Gateway streams the response to the client while optionally filling cache.

### 9.2 Video Streaming

Recommended strategy:

- support HTTP Range requests on the gateway
- map byte ranges to full-file or chunked blob reads
- prefer local cache, then stable backends, then opportunistic backends

Why:

- this is the most standard Web-compatible approach
- native browser video players work with it
- it avoids inventing a custom preview transport too early

## 10. Preview Pipeline

Recommended preview support for MVP:

- images: generate thumbnails and serve original or scaled versions
- PDFs: browser inline preview using direct file or proxy response
- videos: poster image plus range-based playback

Suggested preview job flow:

1. file ingest completes
2. preview worker inspects mime type
3. thumbnail or poster extraction runs if applicable
4. derivative asset is stored as a blob
5. preview artifact record is created

Tooling options:

- `ffmpeg` for video poster generation and optional transcoding later
- image library for thumbnails

Recommendation:

- do not include full video transcoding in MVP
- stick to poster extraction and native playback for supported source formats

## 11. Placement and Replica Policy

Recommended MVP policy engine:

- every backend has a policy score
- backend selection uses stability class, enabled state, health state, and priority
- every file has a placement rule selected by file class

Suggested first placement classes:

- `critical-metadata`
- `document`
- `photo`
- `video`
- `cold-archive`

Suggested default behavior:

- metadata exports: local + stable backend
- documents: stable primary, optional opportunistic secondary
- photos: stable primary, opportunistic secondary if available
- videos: stable primary, optional async secondary depending on size and backend capacity

## 12. Authentication Recommendation

For MVP:

- one local user table
- session cookie auth or signed token auth

Recommendation:

- use a simple local auth model first
- the current backend implementation uses persisted bearer-token sessions
- the future Web UI may wrap that with cookie handling or translate it into cookie-based sessions
- avoid introducing OAuth or external identity providers in MVP

Reason:

- local self-hosted single-user setup does not need auth complexity yet

## 13. Secrets and Security

Recommended MVP measures:

- encrypt backend credentials before storing them in SQLite
- use a gateway master key from environment or local config file
- redact sensitive fields in logs
- require login for all file and backend management routes

## 14. Caching Strategy

Suggested MVP cache layers:

- ingest temp area
- blob cache
- preview cache

Suggested cache policy:

- LRU-like eviction based on size budget and recent access
- separate quotas for blob cache and preview cache

Do not make cache semantics too smart in MVP. Keep them observable and debuggable.

## 15. Background Jobs

Recommended architecture:

- jobs stored in SQLite
- one scheduler loop claims runnable jobs
- small worker pool executes jobs
- failed jobs are retried with backoff
- workers may call synchronous SDKs through thread pools where async-native clients are unavailable

Why not a separate queue system yet:

- too much operational complexity for single-node MVP
- SQLite durability is enough for this stage if jobs are idempotent

Job design rule:

- every job must be safe to retry

## 16. Adapter Interface Draft

Define a storage adapter interface around core object operations.

Suggested interface shape:

```python
from typing import Protocol


class StorageAdapter(Protocol):
    adapter_type: str

    async def check(self) -> None: ...

    async def stat(self, key: str) -> "ObjectInfo": ...

    async def put(
        self,
        key: str,
        stream,
        size: int,
        options: "PutOptions",
    ) -> "ObjectInfo": ...

    async def get(
        self,
        key: str,
        byte_range: "ByteRange | None" = None,
    ) -> tuple[object, "ObjectInfo"]: ...

    async def delete(self, key: str) -> None: ...
```

Notes:

- use `key` as the physical storage identifier
- keep adapters object-oriented even for local filesystem storage
- do not let adapters know about directories, files, or logical paths
- if a provider SDK is synchronous, isolate it inside the adapter and offload blocking work explicitly

Suggested adapter split:

- `NativeStorageAdapter`: direct provider SDK or protocol implementation
- `BridgeStorageAdapter`: talks to an external bridge service or compatibility layer

Core services should not need to know which one is in use.

## 17. Physical Key Strategy

Do not store files in backends by original user path.

Recommended key pattern:

- `blobs/sha256_prefix/full_hash`
- `previews/file_version_id/artifact_type`
- `exports/date/export_id`

Why:

- avoids backend path rename costs
- reduces coupling to logical namespace changes
- makes replication and repair much simpler

## 18. Suggested First Implementation Order

1. config loading and app bootstrap
2. SQLite migrations
3. domain models and repository interfaces
4. local backend adapter
5. file ingest pipeline
6. file listing and basic Web UI
7. blob cache
8. download pipeline
9. image and PDF preview
10. S3 adapter
11. job system
12. Aliyun adapter
13. Baidu adapter
14. video poster generation and range streaming polish

## 19. Key Architecture Decisions To Confirm

These are the most important choices still worth discussing before coding:

- Backend language: `Python` is confirmed.
- Frontend stack: confirm `Vite + React`.
- Database access: `SQLAlchemy ORM` is confirmed.
- Consumer cloud adapters: bridge-capable adapter strategy is confirmed.
- Upload protocol: `tus` is confirmed.
- Credential encryption: define whether to use a local master key file or environment variable only.
- Async model: async-first architecture is confirmed; blocking operations must be isolated explicitly.

## 20. Recommendation Summary

For the first implementation phase, the strongest recommendation is:

- backend: `Python + FastAPI`
- frontend: `Vite + React`
- database: `SQLite`
- data access: `SQLAlchemy ORM`
- jobs: in-process durable workers
- architecture: layered async modular monolith
- storage strategy: replication first, chunking only where beneficial
- upload strategy: `tus` resumable upload with gateway-managed session records
- preview strategy: native browser preview plus range-based streaming
- backend order: local, S3, Aliyun, Baidu
- adapter policy: native where clean, bridge where provider complexity would otherwise damage maintainability
- Python libraries to prefer: `SQLAlchemy 2.0`, `Alembic`, `Pydantic v2`, `httpx`, `boto3` or compatible S3 SDK, and `ffmpeg` integration for previews

This combination is the best balance between simplicity, stream handling, self-hosted deployment, and future extensibility.

## 21. Implemented Foundation Status

For a product-level view of what is still missing before Iron can be called a usable MVP, see [mvp-status.md](/Users/bytedance/iron/docs/mvp-status.md:1).

The repository now includes the following implemented baseline pieces:

- `uv`-managed local virtual environment workflow
- FastAPI application bootstrap and routing shell
- async SQLAlchemy engine setup for SQLite
- initial ORM entity definitions for the core schema
- first Alembic migration for the initial metadata model
- readiness check backed by a real database ping
- baseline automated tests for config, system routes, and migration application
- first auth slice:
  - bootstrap local admin user
  - login, logout, and current-user APIs
  - bearer-token session persistence and route protection
- first Web shell slice:
  - `/app` browser entry
  - login form and session bootstrap
  - root directory listing
  - backend listing
- first metadata feature slice for directories:
  - root directory bootstrap
  - create directory
  - list child directories
- first file metadata slice:
  - internal file metadata creation service for future upload finalization
  - mixed directory and file child listings
  - file detail API
  - recycle-bin file deletion and restore flow
- first upload slice:
  - tus upload session creation
  - offset inspection
  - sequential patch uploads into temp ingest storage
  - finalize flow wired into file metadata creation
  - finalize flow persists the primary blob into the default local backend
  - local `blob_replicas` records are created for finalized uploads
  - finalize enqueues non-local replication jobs
- first download slice:
  - file download endpoint resolves local ready replicas
  - downloaded content is served from persisted local backend objects
  - remote fallback can materialize ready replicas into local cache
- first stream slice:
  - file stream endpoint resolves local ready replicas
  - HTTP Range support for partial content reads
- first preview slice:
  - inline preview endpoint for images and PDFs persisted to local backend
- first backend-visibility slice:
  - backend list endpoint for configured backends
- first backend-management slice:
  - backend creation for local and s3 config metadata
  - local and s3 backend health checks
  - backend disable flow
- first jobs-and-ops slice:
  - durable SQLite job records
  - in-process polling loop
  - job list/detail/retry/run APIs
  - backend health-check enqueue flow
  - declarative file reconcile enqueue flow
  - metadata export and import APIs
- first policy slice:
  - persisted placement policies by file class
  - preferred and excluded backend controls
  - size-aware non-local replica caps
  - placement preview API for product-facing inspection
  - upload finalize uses policy-selected secondary targets
  - reconciliation jobs converge actual replicas toward desired state
- first namespace-mutation slice:
  - directory rename
  - directory move with descendant path updates
  - file rename
  - file move

From a product perspective, the next logical step is no longer more basic metadata CRUD. It is:

- deepen the browser shell into a fuller Web UI
- deepen browser-facing session handling on top of the current auth backend
- add post-restore hardening and stronger preview derivative generation
- add the first consumer-cloud backend

For the product-level gap assessment, use [mvp-status.md](/Users/bytedance/iron/docs/mvp-status.md:1) as the source of truth.

For day-to-day continuation guidance and current repo reality, use [handoff.md](/Users/bytedance/iron/docs/handoff.md:1) first.