iron/docs/technical-architecture-draft.md

Iron Technical Architecture Draft

1. Design Goals

This draft translates the product requirements into an implementation-oriented architecture for the first development phase.

The design aims to:

• support a Web-first product with a self-hosted gateway
• keep the codebase simple enough for rapid iteration
• isolate backend-specific complexity behind adapters
• use SQLite first without blocking future database migration
• optimize for media preview and daily file operations

2. Recommended MVP Technical Stack

2.1 Backend

Recommended:

• language: Python
• HTTP framework: FastAPI
• data access: SQLAlchemy 2.0 ORM
• migrations: Alembic
• schema validation: Pydantic v2
• background jobs: in-process worker pool backed by database job records
• object hashing: standard SHA-256
• config: env-driven settings via a small cached settings module

Why:

• Python fits rapid iteration well and has a strong ecosystem for Web APIs, file tooling, storage SDKs, and media processing.
• FastAPI provides typed request and response models, automatic OpenAPI generation, and a clean async-friendly API layer.
• MVP is a single-node gateway, so Python is a good fit as long as upload, download, and preview paths are implemented with async streaming semantics instead of full buffering.
• The architecture should keep storage I/O and worker logic explicit so hot paths remain understandable and optimizable.

2.2 Frontend

Recommended:

• framework: React with Next.js App Router or Vite + React
• UI library: minimal component primitives such as shadcn/ui or Radix UI
• data fetching: TanStack Query
• video and file preview: native browser capabilities first

Recommendation between the two:

• choose Vite + React if the Web app is a pure SPA against the gateway API
• choose Next.js only if server-rendered routing or later integrated auth flows are clearly desired

For this project, the current recommendation is:

• Vite + React

Reason:

• the gateway already owns the API
• the app is operational rather than content-heavy
• simpler deployment and fewer moving parts for MVP

2.3 Storage Integrations

Recommended strategy:

• implement a common backend adapter interface in the gateway
• use native SDKs when stable and practical
• explicitly allow a bridge adapter layer when consumer cloud providers are unstable or poorly documented
• keep bridge usage behind the same storage adapter interface so the rest of the system stays readable

Proposed initial adapter strategy:

• local directory: native file operations
• S3: AWS SDK compatible implementation
• Aliyun Drive: adapter with bridge mode support
• Baidu Netdisk: adapter with bridge mode support

Important note:

• consumer cloud providers are the highest churn part of the system
• keep their adapters in isolated packages and do not let provider-specific concepts leak into core domain models
• bridge mode is preferred over contaminating core services with provider-specific protocol quirks

3. System Architecture

Browser
  |
  v
Web UI
  |
  v
Gateway API
  |
  +--> Auth Module
  +--> File Service
  +--> Preview Service
  +--> Backend Service
  +--> Job Service
  |
  +--> Metadata Repository (SQLite)
  +--> Cache Manager (local disk)
  +--> Storage Engine
          |
          +--> Placement Planner
          +--> Transfer Manager
          +--> Replica Manager
          +--> Backend Adapters
                  |- Local
                  |- S3
                  |- Aliyun
                  |- Baidu

4. Layering Recommendation

Use a layered async modular monolith for MVP.

Suggested top-level structure:

/app
  /api
  /core
  /domain
  /services
  /repositories
  /models
  /schemas
  /adapters
    /db
    /storage
  /jobs
  /preview
  /cache
  /placement
  /workers
  /utils
/alembic
/scripts
/tests
/web
/docs

Alternative package layout if you prefer stricter layering:

/app
  /config
  /domain
  /service
  /repository
  /adapter
    /http
    /db
    /storage
  /job
  /preview
  /cache
  /placement

Layer responsibilities:

• domain: core entities, enums, policies, service contracts
• services: business orchestration such as upload, move, preview, repair
• repositories: persistence operations over SQLite
• models: SQLAlchemy ORM models
• schemas: API request and response models
• adapter/db: SQLite implementations
• adapter/storage: backend-specific storage adapters
• api: FastAPI routers, dependencies, and HTTP entrypoints
• jobs: durable job definitions and scheduling logic
• cache: local cache and preview artifact management
• placement: backend selection and replication policy logic

This keeps the codebase simple and still allows later extraction.

Async design rules:

• API handlers must be async.
• Repository methods should be async-facing.
• Backend adapters should expose async APIs.
• Blocking SDKs or filesystem-heavy operations must be isolated behind explicit threadpool boundaries.
• No request path should rely on hidden blocking I/O in utility code.

5. Core Domain Model

The system should not model files as mere backend paths. Use stable IDs.

Core entities:

• User
• Directory
• FileEntry
• FileVersion
• Blob
• BlobReplica
• Backend
• UploadSession
• Job
• PreviewArtifact
• CacheEntry

5.1 Entity Semantics

• Directory: logical folder node in the virtual filesystem
• FileEntry: logical file identity tied to a path parent and display name
• FileVersion: immutable content snapshot of a file entry
• Blob: the stored binary unit, whole-file or chunk object
• BlobReplica: one physical copy of a blob on one backend
• Backend: configured storage target and its policy metadata
• PreviewArtifact: thumbnail, poster frame, or derivative preview asset
• CacheEntry: local materialization status for blobs or preview artifacts

This separation matters because:

• rename and move should not create a new file identity
• future version history should be possible
• one logical file may map to multiple blobs
• one blob may have multiple replicas

6. SQLite Schema Draft

Below is the recommended MVP schema direction. Naming can still change, but the concepts should remain stable.

6.1 users

Purpose:

• local authentication identity for MVP

Suggested columns:

• id TEXT PRIMARY KEY
• username TEXT NOT NULL UNIQUE
• password_hash TEXT NOT NULL
• created_at DATETIME NOT NULL
• updated_at DATETIME NOT NULL

6.2 directories

Purpose:

• logical folder tree

Suggested columns:

• id TEXT PRIMARY KEY
• parent_id TEXT NULL
• name TEXT NOT NULL
• path_key TEXT NOT NULL UNIQUE
• created_at DATETIME NOT NULL
• updated_at DATETIME NOT NULL
• deleted_at DATETIME NULL

Notes:

• path_key is a normalized path index for fast lookups
• root directory can be a special fixed row

6.3 file_entries

Purpose:

• logical file objects visible in the namespace

Suggested columns:

• id TEXT PRIMARY KEY
• directory_id TEXT NOT NULL
• name TEXT NOT NULL
• mime_type TEXT NULL
• size_bytes INTEGER NOT NULL
• current_version_id TEXT NOT NULL
• is_deleted BOOLEAN NOT NULL DEFAULT 0
• created_at DATETIME NOT NULL
• updated_at DATETIME NOT NULL
• UNIQUE (directory_id, name)

6.4 file_versions

Purpose:

• immutable content version records

Suggested columns:

• id TEXT PRIMARY KEY
• file_entry_id TEXT NOT NULL
• content_hash TEXT NOT NULL
• size_bytes INTEGER NOT NULL
• storage_layout TEXT NOT NULL
• created_at DATETIME NOT NULL

Suggested storage_layout values:

• single
• chunked

6.5 blobs

Purpose:

• binary storage units

Suggested columns:

• id TEXT PRIMARY KEY
• file_version_id TEXT NOT NULL
• blob_index INTEGER NOT NULL
• kind TEXT NOT NULL
• content_hash TEXT NOT NULL
• size_bytes INTEGER NOT NULL
• logical_offset INTEGER NOT NULL DEFAULT 0
• created_at DATETIME NOT NULL

Suggested kind values:

• file
• chunk
• thumbnail
• poster

6.6 blob_replicas

Purpose:

• track every physical copy

Suggested columns:

• id TEXT PRIMARY KEY
• blob_id TEXT NOT NULL
• backend_id TEXT NOT NULL
• storage_key TEXT NOT NULL
• status TEXT NOT NULL
• etag TEXT NULL
• checksum TEXT NULL
• size_bytes INTEGER NOT NULL
• last_verified_at DATETIME NULL
• created_at DATETIME NOT NULL
• updated_at DATETIME NOT NULL

Suggested status values:

• pending
• ready
• missing
• corrupt
• offline
• failed

6.7 backends

Purpose:

• backend configuration and policy metadata

Suggested columns:

• id TEXT PRIMARY KEY
• name TEXT NOT NULL UNIQUE
• type TEXT NOT NULL
• stability_class TEXT NOT NULL
• read_priority INTEGER NOT NULL
• write_priority INTEGER NOT NULL
• is_enabled BOOLEAN NOT NULL DEFAULT 1
• config_json TEXT NOT NULL
• capacity_hint_bytes INTEGER NULL
• last_health_status TEXT NULL
• last_health_checked_at DATETIME NULL
• created_at DATETIME NOT NULL
• updated_at DATETIME NOT NULL

Notes:

• secrets should be encrypted before being stored in config_json

6.8 upload_sessions

Purpose:

• tus-based resumable upload tracking between browser and gateway

Suggested columns:

• id TEXT PRIMARY KEY
• directory_id TEXT NOT NULL
• filename TEXT NOT NULL
• total_size_bytes INTEGER NOT NULL
• received_size_bytes INTEGER NOT NULL
• status TEXT NOT NULL
• temp_path TEXT NOT NULL
• tus_upload_url TEXT NULL
• upload_metadata_json TEXT NULL
• created_at DATETIME NOT NULL
• updated_at DATETIME NOT NULL

6.8a upload_session_parts

Purpose:

• track uploaded ranges or parts when local resumable state is needed

Suggested columns:

• id TEXT PRIMARY KEY
• upload_session_id TEXT NOT NULL
• part_number INTEGER NOT NULL
• byte_offset INTEGER NOT NULL
• size_bytes INTEGER NOT NULL
• checksum TEXT NULL
• status TEXT NOT NULL
• created_at DATETIME NOT NULL
• updated_at DATETIME NOT NULL
• UNIQUE (upload_session_id, part_number)

6.9 jobs

Purpose:

• durable background work tracking

Suggested columns:

• id TEXT PRIMARY KEY
• kind TEXT NOT NULL
• status TEXT NOT NULL
• payload_json TEXT NOT NULL
• attempt_count INTEGER NOT NULL DEFAULT 0
• max_attempts INTEGER NOT NULL DEFAULT 5
• run_after DATETIME NOT NULL
• last_error TEXT NULL
• created_at DATETIME NOT NULL
• updated_at DATETIME NOT NULL

Suggested kind values:

• replicate_blob
• generate_thumbnail
• verify_replica
• repair_blob
• health_check_backend

6.10 preview_artifacts

Purpose:

• preview resources mapped back to files or file versions

Suggested columns:

• id TEXT PRIMARY KEY
• file_version_id TEXT NOT NULL
• artifact_type TEXT NOT NULL
• blob_id TEXT NOT NULL
• status TEXT NOT NULL
• created_at DATETIME NOT NULL
• updated_at DATETIME NOT NULL

6.11 cache_entries

Purpose:

• local cache tracking

Suggested columns:

• id TEXT PRIMARY KEY
• blob_id TEXT NOT NULL
• local_path TEXT NOT NULL
• cache_kind TEXT NOT NULL
• size_bytes INTEGER NOT NULL
• status TEXT NOT NULL
• last_accessed_at DATETIME NOT NULL
• expires_at DATETIME NULL
• created_at DATETIME NOT NULL
• updated_at DATETIME NOT NULL

7. API Design Direction

Use a clean REST API for MVP. gRPC is unnecessary at this stage.

Suggested route groups:

• /api/auth
• /api/files
• /api/directories
• /api/uploads
• /api/previews
• /api/backends
• /api/jobs
• /api/system

7.1 Key Endpoints

Examples:

• POST /api/auth/login
• GET /api/directories/:id/children
• POST /api/directories
• POST /api/uploads
• HEAD /api/uploads/:id
• PATCH /api/uploads/:id
• POST /api/uploads/:id/finalize
• GET /api/files/:id
• GET /api/files/:id/download
• GET /api/files/:id/stream
• GET /api/files/:id/thumbnail
• POST /api/files/:id/move
• POST /api/files/:id/rename
• DELETE /api/files/:id
• GET /api/backends
• POST /api/backends
• POST /api/backends/:id/check
• GET /api/jobs
• POST /api/jobs/:id/retry

8. Upload Pipeline

Recommended MVP flow:

1. Browser creates a tus upload resource on the gateway.
2. Browser uploads file content through tus patch requests.
3. Gateway writes incoming bytes to a temporary local ingest path.
4. Gateway tracks upload progress in the database.
5. When upload completes, gateway computes metadata such as mime type, hash, and size.
6. Gateway creates logical records in the database.
7. Gateway writes the first required replica synchronously to a chosen primary backend.
8. Gateway marks the file visible when minimum durability criteria are met.
9. Gateway schedules secondary replication and preview generation jobs.

Important MVP rule:

• visibility should happen only after at least one durable backend write succeeds

Recommended upload protocol choice:

• adopt tus for resumable uploads in MVP
• keep upload session state explicit in the database so the gateway can coordinate file creation, placement, and recovery
• treat tus as the ingress protocol, while keeping storage placement and replica logic as internal concerns
• add a gateway-side finalize step so a completed tus upload becomes a managed logical file only after validation and first durable write

9. Download and Stream Pipeline

9.1 Download

1. User requests file download.
2. Gateway resolves current file version and blob layout.
3. Gateway checks local cache.
4. If cache misses, gateway chooses the best available replica based on backend health and priority.
5. Gateway streams the response to the client while optionally filling cache.

9.2 Video Streaming

Recommended strategy:

• support HTTP Range requests on the gateway
• map byte ranges to full-file or chunked blob reads
• prefer local cache, then stable backends, then opportunistic backends

Why:

• this is the most standard Web-compatible approach
• native browser video players work with it
• it avoids inventing a custom preview transport too early

10. Preview Pipeline

Recommended preview support for MVP:

• images: generate thumbnails and serve original or scaled versions
• PDFs: browser inline preview using direct file or proxy response
• videos: poster image plus range-based playback

Suggested preview job flow:

1. file ingest completes
2. preview worker inspects mime type
3. thumbnail or poster extraction runs if applicable
4. derivative asset is stored as a blob
5. preview artifact record is created

Tooling options:

• ffmpeg for video poster generation and optional transcoding later
• image library for thumbnails

Recommendation:

• do not include full video transcoding in MVP
• stick to poster extraction and native playback for supported source formats

11. Placement and Replica Policy

Recommended MVP policy engine:

• every backend has a policy score
• backend selection uses stability class, enabled state, health state, and priority
• every file has a placement rule selected by file class

Suggested first placement classes:

• critical-metadata
• document
• photo
• video
• cold-archive

Suggested default behavior:

• metadata exports: local + stable backend
• documents: stable primary, optional opportunistic secondary
• photos: stable primary, opportunistic secondary if available
• videos: stable primary, optional async secondary depending on size and backend capacity

12. Authentication Recommendation

For MVP:

• one local user table
• session cookie auth or signed token auth

Recommendation:

• use a simple local auth model first
• the current backend implementation uses persisted bearer-token sessions
• the future Web UI may wrap that with cookie handling or translate it into cookie-based sessions
• avoid introducing OAuth or external identity providers in MVP

Reason:

• local self-hosted single-user setup does not need auth complexity yet

13. Secrets and Security

Recommended MVP measures:

• encrypt backend credentials before storing them in SQLite
• use a gateway master key from environment or local config file
• redact sensitive fields in logs
• require login for all file and backend management routes

14. Caching Strategy

Suggested MVP cache layers:

• ingest temp area
• blob cache
• preview cache

Suggested cache policy:

• LRU-like eviction based on size budget and recent access
• separate quotas for blob cache and preview cache

Do not make cache semantics too smart in MVP. Keep them observable and debuggable.

15. Background Jobs

Recommended architecture:

• jobs stored in SQLite
• one scheduler loop claims runnable jobs
• small worker pool executes jobs
• failed jobs are retried with backoff
• workers may call synchronous SDKs through thread pools where async-native clients are unavailable

Why not a separate queue system yet:

• too much operational complexity for single-node MVP
• SQLite durability is enough for this stage if jobs are idempotent

Job design rule:

• every job must be safe to retry

16. Adapter Interface Draft

Define a storage adapter interface around core object operations.

Suggested interface shape:

from typing import Protocol


class StorageAdapter(Protocol):
    adapter_type: str

    async def check(self) -> None: ...

    async def stat(self, key: str) -> "ObjectInfo": ...

    async def put(
        self,
        key: str,
        stream,
        size: int,
        options: "PutOptions",
    ) -> "ObjectInfo": ...

    async def get(
        self,
        key: str,
        byte_range: "ByteRange | None" = None,
    ) -> tuple[object, "ObjectInfo"]: ...

    async def delete(self, key: str) -> None: ...

Notes:

• use key as the physical storage identifier
• keep adapters object-oriented even for local filesystem storage
• do not let adapters know about directories, files, or logical paths
• if a provider SDK is synchronous, isolate it inside the adapter and offload blocking work explicitly

Suggested adapter split:

• NativeStorageAdapter: direct provider SDK or protocol implementation
• BridgeStorageAdapter: talks to an external bridge service or compatibility layer

Core services should not need to know which one is in use.

17. Physical Key Strategy

Do not store files in backends by original user path.

Recommended key pattern:

• blobs/sha256_prefix/full_hash
• previews/file_version_id/artifact_type
• exports/date/export_id

Why:

• avoids backend path rename costs
• reduces coupling to logical namespace changes
• makes replication and repair much simpler

18. Suggested First Implementation Order

1. config loading and app bootstrap
2. SQLite migrations
3. domain models and repository interfaces
4. local backend adapter
5. file ingest pipeline
6. file listing and basic Web UI
7. blob cache
8. download pipeline
9. image and PDF preview
10. S3 adapter
11. job system
12. Aliyun adapter
13. Baidu adapter
14. video poster generation and range streaming polish

19. Key Architecture Decisions To Confirm

These are the most important choices still worth discussing before coding:

• Backend language: Python is confirmed.
• Frontend stack: confirm Vite + React.
• Database access: SQLAlchemy ORM is confirmed.
• Consumer cloud adapters: bridge-capable adapter strategy is confirmed.
• Upload protocol: tus is confirmed.
• Credential encryption: define whether to use a local master key file or environment variable only.
• Async model: async-first architecture is confirmed; blocking operations must be isolated explicitly.

20. Recommendation Summary

For the first implementation phase, the strongest recommendation is:

• backend: Python + FastAPI
• frontend: Vite + React
• database: SQLite
• data access: SQLAlchemy ORM
• jobs: in-process durable workers
• architecture: layered async modular monolith
• storage strategy: replication first, chunking only where beneficial
• upload strategy: tus resumable upload with gateway-managed session records
• preview strategy: native browser preview plus range-based streaming
• backend order: local, S3, Aliyun, Baidu
• adapter policy: native where clean, bridge where provider complexity would otherwise damage maintainability
• Python libraries to prefer: SQLAlchemy 2.0, Alembic, Pydantic v2, httpx, boto3 or compatible S3 SDK, and ffmpeg integration for previews

This combination is the best balance between simplicity, stream handling, self-hosted deployment, and future extensibility.

21. Implemented Foundation Status

For a product-level view of what is still missing before Iron can be called a usable MVP, see mvp-status.md.

The repository now includes the following implemented baseline pieces:

• uv-managed local virtual environment workflow
• FastAPI application bootstrap and routing shell
• async SQLAlchemy engine setup for SQLite
• initial ORM entity definitions for the core schema
• first Alembic migration for the initial metadata model
• readiness check backed by a real database ping
• baseline automated tests for config, system routes, and migration application
• first auth slice:
  • bootstrap local admin user
  • login, logout, and current-user APIs
  • bearer-token session persistence and route protection
• first Web shell slice:
  • /app browser entry
  • login form and session bootstrap
  • root directory listing
  • backend listing
• first metadata feature slice for directories:
  • root directory bootstrap
  • create directory
  • list child directories
• first file metadata slice:
  • internal file metadata creation service for future upload finalization
  • mixed directory and file child listings
  • file detail API
  • recycle-bin file deletion and restore flow
• first upload slice:
  • tus upload session creation
  • offset inspection
  • sequential patch uploads into temp ingest storage
  • finalize flow wired into file metadata creation
  • finalize flow persists the primary blob into the default local backend
  • local blob_replicas records are created for finalized uploads
  • finalize enqueues non-local replication jobs
• first download slice:
  • file download endpoint resolves local ready replicas
  • downloaded content is served from persisted local backend objects
  • remote fallback can materialize ready replicas into local cache
• first stream slice:
  • file stream endpoint resolves local ready replicas
  • HTTP Range support for partial content reads
• first preview slice:
  • inline preview endpoint for images and PDFs persisted to local backend
• first backend-visibility slice:
  • backend list endpoint for configured backends
• first backend-management slice:
  • backend creation for local and s3 config metadata
  • local and s3 backend health checks
  • backend disable flow
• first jobs-and-ops slice:
  • durable SQLite job records
  • in-process polling loop
  • job list/detail/retry/run APIs
  • backend health-check enqueue flow
  • declarative file reconcile enqueue flow
  • metadata export and import APIs
• first policy slice:
  • persisted placement policies by file class
  • preferred and excluded backend controls
  • size-aware non-local replica caps
  • placement preview API for product-facing inspection
  • upload finalize uses policy-selected secondary targets
  • reconciliation jobs converge actual replicas toward desired state
• first namespace-mutation slice:
  • directory rename
  • directory move with descendant path updates
  • file rename
  • file move

From a product perspective, the next logical step is no longer more basic metadata CRUD. It is:

• deepen the browser shell into a fuller Web UI
• deepen browser-facing session handling on top of the current auth backend
• add post-restore hardening and stronger preview derivative generation
• add the first consumer-cloud backend

For the product-level gap assessment, use mvp-status.md as the source of truth.

For day-to-day continuation guidance and current repo reality, use handoff.md first.