Compare commits

...

3 Commits

  1. 17
      .gitignore
  2. 82
      AGENTS.md
  3. 62
      CONTRIBUTING.md
  4. 165
      README.md
  5. 37
      alembic.ini
  6. 62
      alembic/env.py
  7. 24
      alembic/script.py.mako
  8. 1
      alembic/versions/.gitkeep
  9. 327
      alembic/versions/20260417_0007_baseline_schema.py
  10. 2
      app/__init__.py
  11. 2
      app/adapters/__init__.py
  12. 2
      app/adapters/storage/__init__.py
  13. 18
      app/adapters/storage/base.py
  14. 50
      app/adapters/storage/local.py
  15. 18
      app/adapters/storage/registry.py
  16. 88
      app/adapters/storage/s3.py
  17. 145
      app/adapters/storage/webdav.py
  18. 2
      app/api/__init__.py
  19. 76
      app/api/dependencies.py
  20. 59
      app/api/router.py
  21. 2
      app/api/routes/__init__.py
  22. 46
      app/api/routes/auth.py
  23. 112
      app/api/routes/backends.py
  24. 72
      app/api/routes/directories.py
  25. 76
      app/api/routes/exports.py
  26. 189
      app/api/routes/files.py
  27. 51
      app/api/routes/jobs.py
  28. 81
      app/api/routes/policies.py
  29. 21
      app/api/routes/system.py
  30. 77
      app/api/routes/tus_uploads.py
  31. 24
      app/api/routes/uploads.py
  32. 2
      app/core/__init__.py
  33. 130
      app/core/backend_config.py
  34. 54
      app/core/config.py
  35. 77
      app/core/exceptions.py
  36. 5
      app/core/ids.py
  37. 50
      app/core/secret_codec.py
  38. 34
      app/core/security.py
  39. 42
      app/core/storage_layout.py
  40. 2
      app/db/__init__.py
  41. 6
      app/db/base.py
  42. 11
      app/db/health.py
  43. 37
      app/db/session.py
  44. 2
      app/domain/__init__.py
  45. 2
      app/jobs/__init__.py
  46. 110
      app/main.py
  47. 33
      app/models/__init__.py
  48. 275
      app/models/entities.py
  49. 2
      app/repositories/__init__.py
  50. 40
      app/repositories/auth_session_repository.py
  51. 64
      app/repositories/backend_repository.py
  52. 75
      app/repositories/blob_replica_repository.py
  53. 31
      app/repositories/cache_entry_repository.py
  54. 58
      app/repositories/directory_repository.py
  55. 103
      app/repositories/file_repository.py
  56. 53
      app/repositories/job_repository.py
  57. 32
      app/repositories/placement_policy_repository.py
  58. 39
      app/repositories/preview_artifact_repository.py
  59. 25
      app/repositories/upload_repository.py
  60. 30
      app/repositories/user_repository.py
  61. 2
      app/schemas/__init__.py
  62. 34
      app/schemas/auth.py
  63. 56
      app/schemas/backend.py
  64. 13
      app/schemas/common.py
  65. 46
      app/schemas/directory.py
  66. 32
      app/schemas/export.py
  67. 84
      app/schemas/file.py
  68. 37
      app/schemas/job.py
  69. 56
      app/schemas/policy.py
  70. 15
      app/schemas/repair.py
  71. 11
      app/schemas/system.py
  72. 20
      app/schemas/upload.py
  73. 2
      app/services/__init__.py
  74. 106
      app/services/auth_service.py
  75. 212
      app/services/backend_service.py
  76. 153
      app/services/directory_service.py
  77. 234
      app/services/export_service.py
  78. 263
      app/services/file_service.py
  79. 254
      app/services/job_service.py
  80. 82
      app/services/local_storage_service.py
  81. 266
      app/services/policy_service.py
  82. 65
      app/services/preview_service.py
  83. 36
      app/services/repair_service.py
  84. 152
      app/services/storage_service.py
  85. 185
      app/services/upload_service.py
  86. 183
      app/web/app.js
  87. 1
      app/web/dist/assets/app.css
  88. 61
      app/web/dist/assets/app.js
  89. 13
      app/web/dist/index.html
  90. 107
      app/web/index.html
  91. 25
      app/web/routes.py
  92. 245
      app/web/styles.css
  93. 10
      deploy/iron/iron.env.example
  94. 29
      deploy/iron/start_iron.sh
  95. 3
      deploy/openlist/.env.example
  96. 9
      deploy/openlist/docker-compose.yml
  97. 17
      deploy/systemd/iron.service.example
  98. 15
      docs/README.md
  99. 108
      docs/api.md
  100. 158
      docs/architecture.md

17
.gitignore

@ -0,0 +1,17 @@
.venv/
.uv-cache/
.pytest_cache/
__pycache__/
*.py[cod]
*.db
*.sqlite
.coverage
htmlcov/
iron_gateway.egg-info/
node_modules/
output/
.iron-storage/
.iron-temp/
.iron-cache/
deploy/openlist/.env
deploy/iron/iron.env

82
AGENTS.md

@ -0,0 +1,82 @@
# AGENTS.md
Guidance for coding agents and automated contributors working in this repository.
## Project Snapshot
Iron is a self-hosted personal cloud drive gateway with:
- FastAPI backend under `app/`
- Vite + React + TypeScript frontend under `frontend/`
- built frontend assets served from `app/web/dist/`
- Alembic migrations under `alembic/`
- tests under `tests/`
- project docs under `docs/`
The current stage is **Product MVP Candidate**. Preserve the existing
desktop-first drive UX and avoid drifting back toward a generic admin dashboard.
## Required Setup
Use the local virtualenv and `uv` cache:
```bash
UV_CACHE_DIR=.uv-cache uv venv .venv
UV_CACHE_DIR=.uv-cache uv pip install --python .venv/bin/python -e '.[dev]'
npm install
```
## Common Commands
```bash
npm run build
.venv/bin/python -m pytest
.venv/bin/python scripts/ui_playwright_smoke.py
.venv/bin/python -m pytest tests/e2e/test_web_ui_playwright.py
```
## Development Rules
- Keep runtime artifacts out of Git: `.venv/`, `node_modules/`, `output/`,
`.iron-storage/`, `.iron-temp/`, caches, and local databases.
- Add or update tests in the same change as behavior changes.
- For UI work, run the real-service Playwright flow and review screenshots.
- Do not use naked `/api/files/...` media URLs for browser preview/download.
Fetch file content with authorization, create an object URL, and revoke it.
- Keep docs current when changing setup, architecture, test counts, or user flows.
- Prefer small, focused changes over broad rewrites.
## UI Quality Standard
For any meaningful Web UI change:
1. build frontend assets with `npm run build`
2. run `./.venv/bin/python scripts/ui_playwright_smoke.py`
3. inspect screenshots in `output/playwright/`
4. fix layout or interaction issues found in screenshots
5. rerun the Playwright flow
See [docs/development.md](docs/development.md).
## Documentation Expectations
- `README.md` should stay user-facing and concise.
- `CONTRIBUTING.md` should describe contributor workflow.
- `docs/README.md` should index deeper docs.
- `docs/development.md` should describe test and UI validation rules.
- Avoid local absolute paths in committed Markdown.
## Verification Before Finishing
At minimum, run:
```bash
npm run build
.venv/bin/python -m pytest
```
If UI behavior changed, also run:
```bash
.venv/bin/python scripts/ui_playwright_smoke.py
```

62
CONTRIBUTING.md

@ -0,0 +1,62 @@
# Contributing
Thanks for helping improve Iron. The project is still early, so the most useful
contributions are focused changes with tests and clear documentation updates.
## Development Setup
```bash
UV_CACHE_DIR=.uv-cache uv venv .venv
UV_CACHE_DIR=.uv-cache uv pip install --python .venv/bin/python -e '.[dev]'
npm install
npm run build
```
Run the API locally:
```bash
.venv/bin/python -m uvicorn app.main:app --reload
```
Open `http://127.0.0.1:8000/app`.
## Test Before Submitting
Run the full suite:
```bash
.venv/bin/python -m pytest
```
For UI changes, also run:
```bash
npm run build
.venv/bin/python scripts/ui_playwright_smoke.py
```
Review screenshots in `output/playwright/` before considering UI work complete.
## Pull Request Checklist
- The change is focused and described clearly.
- Tests were added or updated for behavior changes.
- UI changes were validated with Playwright when applicable.
- Documentation was updated if setup, behavior, or architecture changed.
- Generated/runtime files are not included.
## Code Style
- Backend code lives in `app/` and uses async FastAPI + SQLAlchemy patterns.
- Frontend code lives in `frontend/` and uses React, React Router, and
TanStack Query.
- Keep browser file preview/download flows authenticated through the API client.
- Prefer clear names and small modules over clever abstractions.
## Project Status
Iron is a Product MVP Candidate. See:
- [docs/product.md](docs/product.md)
- [docs/architecture.md](docs/architecture.md)
- [docs/development.md](docs/development.md)

165
README.md

@ -0,0 +1,165 @@
# Iron
Iron is a self-hosted personal cloud drive gateway. It provides one logical file
namespace over local storage and pluggable backends, with a browser UI for daily
file management and operational visibility.
The project is currently a **Product MVP Candidate**: the core backend and Web
app exist, but the release still needs hardening, broader browser regression
coverage, and more real-world usage.
![Iron demo](docs/assets/iron-demo.gif)
## Features
- FastAPI backend with SQLite, SQLAlchemy, and Alembic migrations
- Local authentication with persisted bearer-token sessions
- Directory and file operations: browse, create folder, rename, move, delete,
restore, download, and preview
- `tus` upload flow with local object persistence
- Desktop-first React Web app at `/app`
- Storage backend management for local directories, S3 storage, and
OpenList/WebDAV runtime paths
- Placement policies, reconcile jobs, health checks, and retryable background jobs
- Metadata export, validation, integrity checks, restore plans, and guarded import
- Python Playwright end-to-end coverage for core browser flows
- Real-environment OpenList acceptance scripts for deployment validation
## Quick Start
Iron uses `uv` for local Python environment setup.
```bash
UV_CACHE_DIR=.uv-cache uv venv .venv
UV_CACHE_DIR=.uv-cache uv pip install --python .venv/bin/python -e '.[dev]'
npm install
npm run build
.venv/bin/python -m uvicorn app.main:app --reload
```
Open the Web app:
```text
http://127.0.0.1:8000/app
```
Default local bootstrap credentials:
```text
username: admin
password: changeme-iron
```
For a practical local deployment with OpenList and Aliyun Drive, start with
[docs/openlist-local-deployment-sop.md](docs/openlist-local-deployment-sop.md).
## Configuration
Common environment variables:
- `IRON_DATABASE_URL`
- `IRON_BOOTSTRAP_USERNAME`
- `IRON_BOOTSTRAP_PASSWORD`
- `IRON_SECRET_KEY`
- `IRON_AUTH_SESSION_TTL_HOURS`
- `IRON_UPLOAD_TEMP_DIR`
- `IRON_LOCAL_STORAGE_DIR`
- `IRON_CACHE_DIR`
- `IRON_JOB_POLL_INTERVAL_SECONDS`
- `IRON_JOB_BATCH_SIZE`
Runtime data is intentionally ignored by Git. By default, local development may
create `.iron-storage/`, `.iron-temp/`, `iron.db`, and `output/`.
`IRON_SECRET_KEY` encrypts stored backend credentials such as S3 keys and
WebDAV passwords. Override the development default before using real storage
credentials.
## Development
Build the frontend:
```bash
npm run build
```
Run the full test suite:
```bash
.venv/bin/python -m pytest
```
Current expected result:
```text
78 passed
```
Run the real OpenList acceptance suite against a live local deployment:
```bash
.venv/bin/python scripts/real_openlist_suite.py
```
Run the browser E2E flow directly:
```bash
.venv/bin/python scripts/ui_playwright_smoke.py
```
Regenerate the README demo GIF:
```bash
.venv/bin/python scripts/generate_readme_demo.py
```
Or through pytest:
```bash
.venv/bin/python -m pytest tests/e2e/test_web_ui_playwright.py
```
Playwright screenshots and runtime artifacts are written to `output/playwright/`.
## Repository Layout
```text
app/ FastAPI application, services, repositories, schemas
alembic/ Database migrations
frontend/ Vite + React + TypeScript source
app/web/dist/ Built frontend assets served by FastAPI
scripts/ Local automation and E2E scripts
deploy/ OpenList compose, Iron env templates, and systemd examples
tests/ Backend, API, and browser E2E tests
docs/ Product, architecture, API, and maintenance docs
```
## Documentation
Start with:
- [Documentation index](docs/README.md)
- [Product](docs/product.md)
- [Architecture](docs/architecture.md)
- [API](docs/api.md)
- [Development](docs/development.md)
- [OpenList + Iron Local Deployment SOP](docs/openlist-local-deployment-sop.md)
For contribution workflow, see [CONTRIBUTING.md](CONTRIBUTING.md).
For AI/coding-agent guidance, see [AGENTS.md](AGENTS.md).
## Security Notes
Iron is not yet a hardened public Internet service. Treat the current credentials
and local bearer-token session model as development-oriented defaults. Change the
bootstrap password and review deployment boundaries before exposing the service.
Metadata import is guarded and requires both:
- `confirm_replace=true`
- a fresh validation token from `POST /api/exports/metadata/restore-plan`
## License
No open source license has been selected yet. Add a `LICENSE` file before
publishing or accepting external contributions.

37
alembic.ini

@ -0,0 +1,37 @@
[alembic]
script_location = alembic
prepend_sys_path = .
path_separator = os
sqlalchemy.url = sqlite+aiosqlite:///./iron.db
[loggers]
keys = root,sqlalchemy,alembic
[handlers]
keys = console
[formatters]
keys = generic
[logger_root]
level = WARN
handlers = console
[logger_sqlalchemy]
level = WARN
handlers =
qualname = sqlalchemy.engine
[logger_alembic]
level = INFO
handlers = console
qualname = alembic
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s

62
alembic/env.py

@ -0,0 +1,62 @@
from logging.config import fileConfig
from alembic import context
from sqlalchemy import pool
from sqlalchemy.engine import Connection
from sqlalchemy.ext.asyncio import async_engine_from_config
from app.core.config import get_settings
from app.db.base import Base
import app.models # noqa: F401
config = context.config
if config.config_file_name is not None:
fileConfig(config.config_file_name)
target_metadata = Base.metadata
settings = get_settings()
configured_url = config.get_main_option("sqlalchemy.url")
if not configured_url:
config.set_main_option("sqlalchemy.url", settings.database_url)
def run_migrations_offline() -> None:
url = config.get_main_option("sqlalchemy.url")
context.configure(
url=url,
target_metadata=target_metadata,
literal_binds=True,
dialect_opts={"paramstyle": "named"},
)
with context.begin_transaction():
context.run_migrations()
def do_run_migrations(connection: Connection) -> None:
context.configure(connection=connection, target_metadata=target_metadata)
with context.begin_transaction():
context.run_migrations()
async def run_migrations_online() -> None:
connectable = async_engine_from_config(
config.get_section(config.config_ini_section, {}),
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)
async with connectable.connect() as connection:
await connection.run_sync(do_run_migrations)
await connectable.dispose()
if context.is_offline_mode():
run_migrations_offline()
else:
import asyncio
asyncio.run(run_migrations_online())

24
alembic/script.py.mako

@ -0,0 +1,24 @@
"""${message}
Revision ID: ${up_revision}
Revises: ${down_revision | comma,n}
Create Date: ${create_date}
"""
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = ${repr(up_revision)}
down_revision = ${repr(down_revision)}
branch_labels = ${repr(branch_labels)}
depends_on = ${repr(depends_on)}
def upgrade() -> None:
pass
def downgrade() -> None:
pass

1
alembic/versions/.gitkeep

@ -0,0 +1 @@

327
alembic/versions/20260417_0007_baseline_schema.py

@ -0,0 +1,327 @@
"""baseline schema
Revision ID: 20260417_0007
Revises:
Create Date: 2026-04-17 10:30:00.000000
"""
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = "20260417_0007"
down_revision = None
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"users",
sa.Column("id", sa.String(), nullable=False),
sa.Column("username", sa.String(), nullable=False),
sa.Column("password_hash", sa.String(), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("username"),
)
op.create_table(
"auth_sessions",
sa.Column("id", sa.String(), nullable=False),
sa.Column("user_id", sa.String(), nullable=False),
sa.Column("token_hash", sa.String(), nullable=False),
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("revoked_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.ForeignKeyConstraint(["user_id"], ["users.id"]),
sa.PrimaryKeyConstraint("id"),
)
op.create_index("idx_auth_sessions_user_status", "auth_sessions", ["user_id", "revoked_at", "expires_at"])
op.create_index("idx_auth_sessions_token_hash", "auth_sessions", ["token_hash"], unique=True)
op.create_table(
"directories",
sa.Column("id", sa.String(), nullable=False),
sa.Column("parent_id", sa.String(), nullable=True),
sa.Column("name", sa.String(), nullable=False),
sa.Column("path_key", sa.String(), nullable=False),
sa.Column("deleted_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.ForeignKeyConstraint(["parent_id"], ["directories.id"]),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("path_key"),
)
op.create_index("idx_directories_parent_id", "directories", ["parent_id"])
op.create_index("idx_directories_path_key", "directories", ["path_key"])
op.create_table(
"backends",
sa.Column("id", sa.String(), nullable=False),
sa.Column("name", sa.String(), nullable=False),
sa.Column("type", sa.String(), nullable=False),
sa.Column("stability_class", sa.String(), nullable=False),
sa.Column("read_priority", sa.Integer(), nullable=False),
sa.Column("write_priority", sa.Integer(), nullable=False),
sa.Column("is_enabled", sa.Boolean(), server_default="1", nullable=False),
sa.Column("public_config_json", sa.Text(), nullable=False),
sa.Column("secret_config_json", sa.Text(), nullable=False),
sa.Column("capacity_hint_bytes", sa.Integer(), nullable=True),
sa.Column("last_health_status", sa.String(), nullable=True),
sa.Column("last_health_checked_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("name"),
)
op.create_index("idx_backends_enabled", "backends", ["is_enabled"])
op.create_index("idx_backends_priority", "backends", ["write_priority", "read_priority"])
op.create_table(
"placement_policies",
sa.Column("id", sa.String(), nullable=False),
sa.Column("file_class", sa.String(), nullable=False),
sa.Column("require_local", sa.Boolean(), server_default="1", nullable=False),
sa.Column("stable_replica_count", sa.Integer(), server_default="1", nullable=False),
sa.Column("opportunistic_replica_count", sa.Integer(), server_default="0", nullable=False),
sa.Column("preferred_backend_ids_json", sa.Text(), server_default="[]", nullable=False),
sa.Column("excluded_backend_ids_json", sa.Text(), server_default="[]", nullable=False),
sa.Column("max_non_local_size_bytes", sa.Integer(), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.PrimaryKeyConstraint("id"),
)
op.create_index("idx_placement_policies_file_class", "placement_policies", ["file_class"], unique=True)
op.create_table(
"upload_sessions",
sa.Column("id", sa.String(), nullable=False),
sa.Column("directory_id", sa.String(), nullable=True),
sa.Column("filename", sa.String(), nullable=True),
sa.Column("total_size_bytes", sa.Integer(), nullable=False),
sa.Column("received_size_bytes", sa.Integer(), nullable=False),
sa.Column("status", sa.String(), nullable=False),
sa.Column("temp_path", sa.String(), nullable=False),
sa.Column("tus_upload_url", sa.String(), nullable=True),
sa.Column("upload_metadata_json", sa.Text(), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.ForeignKeyConstraint(["directory_id"], ["directories.id"]),
sa.PrimaryKeyConstraint("id"),
)
op.create_index("idx_upload_sessions_status", "upload_sessions", ["status"])
op.create_index("idx_upload_sessions_updated_at", "upload_sessions", ["updated_at"])
op.create_table(
"file_entries",
sa.Column("id", sa.String(), nullable=False),
sa.Column("directory_id", sa.String(), nullable=False),
sa.Column("name", sa.String(), nullable=False),
sa.Column("mime_type", sa.String(), nullable=True),
sa.Column("size_bytes", sa.Integer(), nullable=False),
sa.Column("current_version_id", sa.String(), nullable=True),
sa.Column("is_deleted", sa.Boolean(), server_default="0", nullable=False),
sa.Column("deleted_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.ForeignKeyConstraint(["directory_id"], ["directories.id"]),
sa.PrimaryKeyConstraint("id"),
)
op.create_index("idx_file_entries_current_version_id", "file_entries", ["current_version_id"])
op.create_index("idx_file_entries_deleted_at", "file_entries", ["deleted_at"])
op.create_index(
"idx_file_entries_directory_name",
"file_entries",
["directory_id", "name", "is_deleted"],
unique=True,
)
op.create_table(
"file_versions",
sa.Column("id", sa.String(), nullable=False),
sa.Column("file_entry_id", sa.String(), nullable=False),
sa.Column("content_hash", sa.String(), nullable=False),
sa.Column("size_bytes", sa.Integer(), nullable=False),
sa.Column("storage_layout", sa.String(), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.ForeignKeyConstraint(["file_entry_id"], ["file_entries.id"]),
sa.PrimaryKeyConstraint("id"),
)
op.create_index("idx_file_versions_content_hash", "file_versions", ["content_hash"])
op.create_index("idx_file_versions_file_entry_id", "file_versions", ["file_entry_id"])
op.create_table(
"blobs",
sa.Column("id", sa.String(), nullable=False),
sa.Column("file_version_id", sa.String(), nullable=False),
sa.Column("blob_index", sa.Integer(), nullable=False),
sa.Column("kind", sa.String(), nullable=False),
sa.Column("content_hash", sa.String(), nullable=False),
sa.Column("size_bytes", sa.Integer(), nullable=False),
sa.Column("logical_offset", sa.Integer(), server_default="0", nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.ForeignKeyConstraint(["file_version_id"], ["file_versions.id"]),
sa.PrimaryKeyConstraint("id"),
)
op.create_index("idx_blobs_content_hash", "blobs", ["content_hash"])
op.create_index("idx_blobs_file_version_index", "blobs", ["file_version_id", "blob_index"], unique=True)
op.create_table(
"blob_replicas",
sa.Column("id", sa.String(), nullable=False),
sa.Column("blob_id", sa.String(), nullable=False),
sa.Column("backend_id", sa.String(), nullable=False),
sa.Column("storage_key", sa.String(), nullable=False),
sa.Column("status", sa.String(), nullable=False),
sa.Column("etag", sa.String(), nullable=True),
sa.Column("checksum", sa.String(), nullable=True),
sa.Column("size_bytes", sa.Integer(), nullable=False),
sa.Column("last_verified_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.ForeignKeyConstraint(["backend_id"], ["backends.id"]),
sa.ForeignKeyConstraint(["blob_id"], ["blobs.id"]),
sa.PrimaryKeyConstraint("id"),
)
op.create_index("idx_blob_replicas_backend_status", "blob_replicas", ["backend_id", "status"])
op.create_index("idx_blob_replicas_blob_backend", "blob_replicas", ["blob_id", "backend_id"], unique=True)
op.create_index("idx_blob_replicas_storage_key", "blob_replicas", ["storage_key"])
op.create_table(
"upload_session_parts",
sa.Column("id", sa.String(), nullable=False),
sa.Column("upload_session_id", sa.String(), nullable=False),
sa.Column("part_number", sa.Integer(), nullable=False),
sa.Column("byte_offset", sa.Integer(), nullable=False),
sa.Column("size_bytes", sa.Integer(), nullable=False),
sa.Column("checksum", sa.String(), nullable=True),
sa.Column("status", sa.String(), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.ForeignKeyConstraint(["upload_session_id"], ["upload_sessions.id"]),
sa.PrimaryKeyConstraint("id"),
)
op.create_index(
"idx_upload_session_parts_unique",
"upload_session_parts",
["upload_session_id", "part_number"],
unique=True,
)
op.create_index("idx_upload_session_parts_upload_id", "upload_session_parts", ["upload_session_id"])
op.create_table(
"jobs",
sa.Column("id", sa.String(), nullable=False),
sa.Column("kind", sa.String(), nullable=False),
sa.Column("status", sa.String(), nullable=False),
sa.Column("payload_json", sa.Text(), nullable=False),
sa.Column("attempt_count", sa.Integer(), server_default="0", nullable=False),
sa.Column("max_attempts", sa.Integer(), server_default="5", nullable=False),
sa.Column("run_after", sa.DateTime(timezone=True), nullable=False),
sa.Column("last_error", sa.Text(), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.PrimaryKeyConstraint("id"),
)
op.create_index("idx_jobs_kind_status", "jobs", ["kind", "status"])
op.create_index("idx_jobs_status_run_after", "jobs", ["status", "run_after"])
op.create_table(
"preview_artifacts",
sa.Column("id", sa.String(), nullable=False),
sa.Column("file_version_id", sa.String(), nullable=False),
sa.Column("artifact_type", sa.String(), nullable=False),
sa.Column("blob_id", sa.String(), nullable=False),
sa.Column("status", sa.String(), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.ForeignKeyConstraint(["blob_id"], ["blobs.id"]),
sa.ForeignKeyConstraint(["file_version_id"], ["file_versions.id"]),
sa.PrimaryKeyConstraint("id"),
)
op.create_index(
"idx_preview_artifacts_version_type",
"preview_artifacts",
["file_version_id", "artifact_type"],
unique=True,
)
op.create_table(
"cache_entries",
sa.Column("id", sa.String(), nullable=False),
sa.Column("blob_id", sa.String(), nullable=False),
sa.Column("local_path", sa.String(), nullable=False),
sa.Column("cache_kind", sa.String(), nullable=False),
sa.Column("size_bytes", sa.Integer(), nullable=False),
sa.Column("status", sa.String(), nullable=False),
sa.Column("last_accessed_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.ForeignKeyConstraint(["blob_id"], ["blobs.id"]),
sa.PrimaryKeyConstraint("id"),
)
op.create_index("idx_cache_entries_blob_id", "cache_entries", ["blob_id"])
op.create_index("idx_cache_entries_expires_at", "cache_entries", ["expires_at"])
op.create_index("idx_cache_entries_last_accessed_at", "cache_entries", ["last_accessed_at"])
def downgrade() -> None:
op.drop_index("idx_cache_entries_last_accessed_at", table_name="cache_entries")
op.drop_index("idx_cache_entries_expires_at", table_name="cache_entries")
op.drop_index("idx_cache_entries_blob_id", table_name="cache_entries")
op.drop_table("cache_entries")
op.drop_index("idx_preview_artifacts_version_type", table_name="preview_artifacts")
op.drop_table("preview_artifacts")
op.drop_index("idx_jobs_status_run_after", table_name="jobs")
op.drop_index("idx_jobs_kind_status", table_name="jobs")
op.drop_table("jobs")
op.drop_index("idx_upload_session_parts_upload_id", table_name="upload_session_parts")
op.drop_index("idx_upload_session_parts_unique", table_name="upload_session_parts")
op.drop_table("upload_session_parts")
op.drop_index("idx_blob_replicas_storage_key", table_name="blob_replicas")
op.drop_index("idx_blob_replicas_blob_backend", table_name="blob_replicas")
op.drop_index("idx_blob_replicas_backend_status", table_name="blob_replicas")
op.drop_table("blob_replicas")
op.drop_index("idx_placement_policies_file_class", table_name="placement_policies")
op.drop_table("placement_policies")
op.drop_index("idx_blobs_file_version_index", table_name="blobs")
op.drop_index("idx_blobs_content_hash", table_name="blobs")
op.drop_table("blobs")
op.drop_index("idx_file_versions_file_entry_id", table_name="file_versions")
op.drop_index("idx_file_versions_content_hash", table_name="file_versions")
op.drop_table("file_versions")
op.drop_index("idx_file_entries_deleted_at", table_name="file_entries")
op.drop_index("idx_file_entries_directory_name", table_name="file_entries")
op.drop_index("idx_file_entries_current_version_id", table_name="file_entries")
op.drop_table("file_entries")
op.drop_index("idx_upload_sessions_updated_at", table_name="upload_sessions")
op.drop_index("idx_upload_sessions_status", table_name="upload_sessions")
op.drop_table("upload_sessions")
op.drop_index("idx_backends_priority", table_name="backends")
op.drop_index("idx_backends_enabled", table_name="backends")
op.drop_table("backends")
op.drop_index("idx_directories_path_key", table_name="directories")
op.drop_index("idx_directories_parent_id", table_name="directories")
op.drop_table("directories")
op.drop_index("idx_auth_sessions_token_hash", table_name="auth_sessions")
op.drop_index("idx_auth_sessions_user_status", table_name="auth_sessions")
op.drop_table("auth_sessions")
op.drop_table("users")

2
app/__init__.py

@ -0,0 +1,2 @@
"""Iron gateway application package."""

2
app/adapters/__init__.py

@ -0,0 +1,2 @@
"""External adapters."""

2
app/adapters/storage/__init__.py

@ -0,0 +1,2 @@
"""Storage adapter implementations."""

18
app/adapters/storage/base.py

@ -0,0 +1,18 @@
from __future__ import annotations
from pathlib import Path
from typing import Protocol
class StorageAdapter(Protocol):
async def check(self) -> dict[str, str]:
...
async def put_file(self, source_path: str, storage_key: str) -> dict[str, str | None]:
...
async def download_file(self, storage_key: str, destination_path: Path) -> Path:
...
async def object_exists(self, storage_key: str) -> bool:
...

50
app/adapters/storage/local.py

@ -0,0 +1,50 @@
from __future__ import annotations
import shutil
from pathlib import Path
import anyio
class LocalStorageAdapter:
def __init__(self, base_path: str) -> None:
self.base_path = Path(base_path)
async def ensure_base_path(self) -> Path:
await anyio.to_thread.run_sync(lambda: self.base_path.mkdir(parents=True, exist_ok=True))
return self.base_path
async def check(self) -> dict[str, str]:
path = await self.ensure_base_path()
is_dir = await anyio.to_thread.run_sync(path.is_dir)
if not is_dir:
raise NotADirectoryError(str(path))
return {"status": "healthy", "detail": str(path)}
async def put_file(self, source_path: str, storage_key: str) -> dict[str, str | None]:
destination = self.base_path / storage_key
await anyio.to_thread.run_sync(lambda: destination.parent.mkdir(parents=True, exist_ok=True))
def _copy_if_needed() -> None:
if not destination.exists():
shutil.copyfile(source_path, destination)
await anyio.to_thread.run_sync(_copy_if_needed)
return {"etag": None, "storage_key": storage_key}
async def download_file(self, storage_key: str, destination_path: Path) -> Path:
source = await self.resolve_path(storage_key)
await anyio.to_thread.run_sync(lambda: destination_path.parent.mkdir(parents=True, exist_ok=True))
await anyio.to_thread.run_sync(lambda: shutil.copyfile(source, destination_path))
return destination_path
async def resolve_path(self, storage_key: str) -> Path:
destination = self.base_path / storage_key
exists = await anyio.to_thread.run_sync(destination.exists)
if not exists:
raise FileNotFoundError(storage_key)
return destination
async def object_exists(self, storage_key: str) -> bool:
destination = self.base_path / storage_key
return await anyio.to_thread.run_sync(destination.exists)

18
app/adapters/storage/registry.py

@ -0,0 +1,18 @@
from __future__ import annotations
from typing import Any
from app.adapters.storage.base import StorageAdapter
from app.adapters.storage.local import LocalStorageAdapter
from app.adapters.storage.s3 import S3StorageAdapter
from app.adapters.storage.webdav import WebDAVStorageAdapter
def build_storage_adapter(backend_type: str, config: dict[str, Any]) -> StorageAdapter:
if backend_type == "local_directory":
return LocalStorageAdapter(config["base_path"])
if backend_type == "s3":
return S3StorageAdapter(config)
if backend_type == "webdav":
return WebDAVStorageAdapter(config)
raise ValueError(f"Unsupported backend type: {backend_type}")

88
app/adapters/storage/s3.py

@ -0,0 +1,88 @@
from __future__ import annotations
from pathlib import Path
from typing import Any
import anyio
import boto3
from botocore.config import Config
from botocore.exceptions import ClientError
class S3StorageAdapter:
def __init__(self, config: dict[str, Any]) -> None:
self.config = config
async def check(self) -> dict[str, str]:
client = self._create_client()
def _check() -> dict[str, str]:
client.head_bucket(Bucket=self.config["bucket"])
return {"status": "healthy", "detail": self.config["bucket"]}
return await anyio.to_thread.run_sync(_check)
async def put_file(self, source_path: str, storage_key: str) -> dict[str, str | None]:
client = self._create_client()
object_key = self._object_key(storage_key)
def _put() -> dict[str, str | None]:
extra_args: dict[str, Any] = {}
if self.config.get("storage_class"):
extra_args["StorageClass"] = self.config["storage_class"]
if self.config.get("server_side_encryption"):
extra_args["ServerSideEncryption"] = self.config["server_side_encryption"]
client.upload_file(source_path, self.config["bucket"], object_key, ExtraArgs=extra_args)
metadata = client.head_object(Bucket=self.config["bucket"], Key=object_key)
return {"etag": metadata.get("ETag"), "storage_key": storage_key}
return await anyio.to_thread.run_sync(_put)
async def download_file(self, storage_key: str, destination_path: Path) -> Path:
client = self._create_client()
object_key = self._object_key(storage_key)
def _download() -> Path:
destination_path.parent.mkdir(parents=True, exist_ok=True)
client.download_file(self.config["bucket"], object_key, str(destination_path))
return destination_path
return await anyio.to_thread.run_sync(_download)
async def object_exists(self, storage_key: str) -> bool:
client = self._create_client()
object_key = self._object_key(storage_key)
def _exists() -> bool:
try:
client.head_object(Bucket=self.config["bucket"], Key=object_key)
return True
except ClientError as exc:
error_code = exc.response.get("Error", {}).get("Code")
if error_code in {"404", "NoSuchKey", "NotFound"}:
return False
raise
return await anyio.to_thread.run_sync(_exists)
def _create_client(self):
client_config = None
if self.config.get("force_path_style"):
client_config = Config(s3={"addressing_style": "path"})
client_kwargs = {
"service_name": "s3",
"region_name": self.config.get("region"),
"endpoint_url": self.config.get("endpoint_url"),
"aws_access_key_id": self.config.get("access_key_id"),
"aws_secret_access_key": self.config.get("secret_access_key"),
"verify": self.config.get("verify_ssl", True),
"config": client_config,
}
cleaned_kwargs = {key: value for key, value in client_kwargs.items() if value is not None}
return boto3.client(**cleaned_kwargs)
def _object_key(self, storage_key: str) -> str:
prefix = (self.config.get("prefix") or "").strip("/")
if not prefix:
return storage_key
return f"{prefix}/{storage_key}"

145
app/adapters/storage/webdav.py

@ -0,0 +1,145 @@
from __future__ import annotations
import base64
import ssl
import urllib.error
import urllib.parse
import urllib.request
import xml.etree.ElementTree as ET
from pathlib import Path
from typing import Any
import anyio
DAV_NAMESPACE = {"d": "DAV:"}
class WebDAVStorageAdapter:
def __init__(self, config: dict[str, Any]) -> None:
self.config = config
self.base_url = config["endpoint_url"].rstrip("/")
self.root_path = self._normalize_relative_path(config.get("root_path") or "iron")
self._ssl_context = None if config.get("verify_ssl", True) else ssl._create_unverified_context()
async def check(self) -> dict[str, str]:
def _check() -> dict[str, str]:
self._ensure_directory(self.root_path)
return {"status": "healthy", "detail": self.root_path}
return await anyio.to_thread.run_sync(_check)
async def put_file(self, source_path: str, storage_key: str) -> dict[str, str | None]:
def _put() -> dict[str, str | None]:
source = Path(source_path)
if not source.exists():
raise FileNotFoundError(source_path)
normalized_key = self._normalize_relative_path(storage_key)
segments = normalized_key.split("/")
if len(segments) > 1:
self._ensure_directory(f"{self.root_path}/{'/'.join(segments[:-1])}")
request = urllib.request.Request(
self._build_url(f"{self.root_path}/{normalized_key}"),
data=source.read_bytes(),
method="PUT",
headers=self._auth_headers(),
)
with urllib.request.urlopen(request, timeout=60, context=self._ssl_context) as response:
etag = response.headers.get("ETag")
return {"etag": etag, "storage_key": normalized_key}
return await anyio.to_thread.run_sync(_put)
async def download_file(self, storage_key: str, destination_path: Path) -> Path:
def _download() -> Path:
normalized_key = self._normalize_relative_path(storage_key)
request = urllib.request.Request(
self._build_url(f"{self.root_path}/{normalized_key}"),
method="GET",
headers=self._auth_headers(),
)
try:
with urllib.request.urlopen(request, timeout=60, context=self._ssl_context) as response:
body = response.read()
except urllib.error.HTTPError as exc:
if exc.code == 404:
raise FileNotFoundError(storage_key) from exc
raise
destination_path.parent.mkdir(parents=True, exist_ok=True)
destination_path.write_bytes(body)
return destination_path
return await anyio.to_thread.run_sync(_download)
async def object_exists(self, storage_key: str) -> bool:
def _exists() -> bool:
normalized_key = self._normalize_relative_path(storage_key)
request = urllib.request.Request(
self._build_url(f"{self.root_path}/{normalized_key}"),
method="HEAD",
headers=self._auth_headers(),
)
try:
with urllib.request.urlopen(request, timeout=20, context=self._ssl_context):
return True
except urllib.error.HTTPError as exc:
if exc.code == 404:
return False
if exc.code in {403, 405}:
return self._path_exists(f"{self.root_path}/{normalized_key}")
raise
return await anyio.to_thread.run_sync(_exists)
def _ensure_directory(self, relative_path: str) -> None:
current = ""
for segment in self._normalize_relative_path(relative_path).split("/"):
current = f"{current}/{segment}".strip("/")
if self._path_exists(current):
continue
request = urllib.request.Request(
self._build_url(current),
method="MKCOL",
headers=self._auth_headers(),
)
try:
with urllib.request.urlopen(request, timeout=20, context=self._ssl_context):
pass
except urllib.error.HTTPError as exc:
if exc.code not in {201, 405}:
raise RuntimeError(f"WebDAV MKCOL failed with {exc.code} for {current}") from exc
def _path_exists(self, relative_path: str) -> bool:
request = urllib.request.Request(
self._build_url(relative_path),
method="PROPFIND",
headers={**self._auth_headers(), "Depth": "0"},
)
try:
with urllib.request.urlopen(request, timeout=20, context=self._ssl_context) as response:
body = response.read()
except urllib.error.HTTPError as exc:
if exc.code == 404:
return False
raise RuntimeError(f"WebDAV PROPFIND failed with {exc.code} for {relative_path}") from exc
if not body:
return True
root = ET.fromstring(body)
return root.find("d:response", DAV_NAMESPACE) is not None
def _build_url(self, relative_path: str) -> str:
safe_path = "/".join(urllib.parse.quote(part, safe="") for part in relative_path.strip("/").split("/") if part)
return f"{self.base_url}/{safe_path}"
def _auth_headers(self) -> dict[str, str]:
username = str(self.config.get("username") or "")
password = str(self.config.get("password") or "")
token = base64.b64encode(f"{username}:{password}".encode("utf-8")).decode("ascii")
return {"Authorization": f"Basic {token}"}
@staticmethod
def _normalize_relative_path(value: str) -> str:
normalized = "/".join(part for part in value.strip().strip("/").split("/") if part)
if not normalized:
raise ValueError("path must not be empty")
return normalized

2
app/api/__init__.py

@ -0,0 +1,2 @@
"""HTTP API layer."""

76
app/api/dependencies.py

@ -0,0 +1,76 @@
from fastapi import Depends
from fastapi import Header
from sqlalchemy.ext.asyncio import AsyncSession
from app.db.session import get_db_session
from app.models.entities import User
from app.services.export_service import ExportService
from app.services.auth_service import AuthService
from app.services.backend_service import BackendService
from app.services.directory_service import DirectoryService
from app.services.file_service import FileService
from app.services.job_service import JobService
from app.services.policy_service import PolicyService
from app.services.repair_service import RepairService
from app.services.upload_service import UploadService
async def get_directory_service(
session: AsyncSession = Depends(get_db_session),
) -> DirectoryService:
return DirectoryService(session)
async def get_file_service(
session: AsyncSession = Depends(get_db_session),
) -> FileService:
return FileService(session)
async def get_upload_service(
session: AsyncSession = Depends(get_db_session),
) -> UploadService:
return UploadService(session)
async def get_backend_service(
session: AsyncSession = Depends(get_db_session),
) -> BackendService:
return BackendService(session)
async def get_job_service(
session: AsyncSession = Depends(get_db_session),
) -> JobService:
return JobService(session)
async def get_export_service(
session: AsyncSession = Depends(get_db_session),
) -> ExportService:
return ExportService(session)
async def get_repair_service(
session: AsyncSession = Depends(get_db_session),
) -> RepairService:
return RepairService(session)
async def get_auth_service(
session: AsyncSession = Depends(get_db_session),
) -> AuthService:
return AuthService(session)
async def get_policy_service(
session: AsyncSession = Depends(get_db_session),
) -> PolicyService:
return PolicyService(session)
async def require_current_user(
authorization: str | None = Header(default=None, alias="Authorization"),
service: AuthService = Depends(get_auth_service),
) -> User:
return await service.get_current_user(authorization)

59
app/api/router.py

@ -0,0 +1,59 @@
from fastapi import APIRouter
from fastapi import Depends
from app.api.dependencies import require_current_user
from app.api.routes.auth import router as auth_router
from app.api.routes.backends import router as backend_router
from app.api.routes.directories import router as directory_router
from app.api.routes.exports import router as export_router
from app.api.routes.files import router as file_router
from app.api.routes.jobs import router as job_router
from app.api.routes.policies import router as policy_router
from app.api.routes.system import router as system_router
from app.api.routes.uploads import router as upload_router
api_router = APIRouter()
api_router.include_router(system_router, prefix="/system", tags=["system"])
api_router.include_router(auth_router, prefix="/auth", tags=["auth"])
api_router.include_router(
export_router,
prefix="/exports",
tags=["exports"],
dependencies=[Depends(require_current_user)],
)
api_router.include_router(
backend_router,
prefix="/backends",
tags=["backends"],
dependencies=[Depends(require_current_user)],
)
api_router.include_router(
job_router,
prefix="/jobs",
tags=["jobs"],
dependencies=[Depends(require_current_user)],
)
api_router.include_router(
policy_router,
prefix="/policies",
tags=["policies"],
dependencies=[Depends(require_current_user)],
)
api_router.include_router(
directory_router,
prefix="/directories",
tags=["directories"],
dependencies=[Depends(require_current_user)],
)
api_router.include_router(
file_router,
prefix="/files",
tags=["files"],
dependencies=[Depends(require_current_user)],
)
api_router.include_router(
upload_router,
prefix="/uploads",
tags=["uploads"],
dependencies=[Depends(require_current_user)],
)

2
app/api/routes/__init__.py

@ -0,0 +1,2 @@
"""API route modules."""

46
app/api/routes/auth.py

@ -0,0 +1,46 @@
from __future__ import annotations
from fastapi import APIRouter
from fastapi import Depends
from fastapi import Header
from app.api.dependencies import get_auth_service
from app.api.dependencies import require_current_user
from app.models.entities import User
from app.schemas.auth import AuthMeResponse
from app.schemas.auth import LoginRequest
from app.schemas.auth import LoginResponse
from app.schemas.auth import LogoutResponse
from app.schemas.auth import UserSummary
from app.services.auth_service import AuthService
router = APIRouter()
@router.post("/login", response_model=LoginResponse)
async def login(
payload: LoginRequest,
service: AuthService = Depends(get_auth_service),
) -> LoginResponse:
result = await service.login(username=payload.username, password=payload.password)
return LoginResponse(
access_token=result.access_token,
expires_at=result.expires_at,
user=UserSummary.model_validate(result.user, from_attributes=True),
)
@router.get("/me", response_model=AuthMeResponse)
async def me(
current_user: User = Depends(require_current_user),
) -> AuthMeResponse:
return AuthMeResponse(user=UserSummary.model_validate(current_user, from_attributes=True))
@router.post("/logout", response_model=LogoutResponse)
async def logout(
authorization: str | None = Header(default=None, alias="Authorization"),
service: AuthService = Depends(get_auth_service),
) -> LogoutResponse:
await service.logout(authorization)
return LogoutResponse()

112
app/api/routes/backends.py

@ -0,0 +1,112 @@
import json
from fastapi import APIRouter, Depends
from fastapi import Response
from app.api.dependencies import get_backend_service
from app.core.secret_codec import list_secret_fields
from app.models.entities import Backend
from app.schemas.backend import (
BackendCheckResponse,
BackendListResponse,
BackendMutationResponse,
BackendSummary,
CreateBackendRequest,
CreateBackendResponse,
UpdateBackendRequest,
)
from app.services.backend_service import BackendService
router = APIRouter()
def summarize_backend(backend: Backend) -> BackendSummary:
summary = BackendSummary.model_validate(backend, from_attributes=True)
public_config = json.loads(backend.public_config_json)
summary.config = public_config
summary.secret_fields = list_secret_fields(backend.secret_config_json)
return summary
@router.get("", response_model=BackendListResponse)
async def list_backends(
service: BackendService = Depends(get_backend_service),
) -> BackendListResponse:
items = await service.list_backends()
return BackendListResponse(
items=[summarize_backend(item) for item in items]
)
@router.post("", response_model=CreateBackendResponse)
async def create_backend(
payload: CreateBackendRequest,
service: BackendService = Depends(get_backend_service),
) -> CreateBackendResponse:
backend = await service.create_backend(
name=payload.name,
backend_type=payload.type,
stability_class=payload.stability_class,
read_priority=payload.read_priority,
write_priority=payload.write_priority,
config=payload.config,
)
return CreateBackendResponse(backend=summarize_backend(backend))
@router.post("/{backend_id}/check", response_model=BackendCheckResponse)
async def check_backend(
backend_id: str,
service: BackendService = Depends(get_backend_service),
) -> BackendCheckResponse:
backend, status, detail = await service.check_backend(backend_id)
return BackendCheckResponse(
backend_id=backend.id,
status=status,
checked_at=backend.last_health_checked_at,
detail=detail,
)
@router.patch("/{backend_id}", response_model=BackendMutationResponse)
async def update_backend(
backend_id: str,
payload: UpdateBackendRequest,
service: BackendService = Depends(get_backend_service),
) -> BackendMutationResponse:
backend = await service.update_backend(
backend_id,
name=payload.name,
stability_class=payload.stability_class,
read_priority=payload.read_priority,
write_priority=payload.write_priority,
config=payload.config,
)
return BackendMutationResponse(backend=summarize_backend(backend))
@router.post("/{backend_id}/disable", response_model=BackendMutationResponse)
async def disable_backend(
backend_id: str,
service: BackendService = Depends(get_backend_service),
) -> BackendMutationResponse:
backend = await service.disable_backend(backend_id)
return BackendMutationResponse(backend=summarize_backend(backend))
@router.post("/{backend_id}/enable", response_model=BackendMutationResponse)
async def enable_backend(
backend_id: str,
service: BackendService = Depends(get_backend_service),
) -> BackendMutationResponse:
backend = await service.enable_backend(backend_id)
return BackendMutationResponse(backend=summarize_backend(backend))
@router.delete("/{backend_id}", status_code=204)
async def delete_backend(
backend_id: str,
service: BackendService = Depends(get_backend_service),
) -> Response:
await service.delete_backend(backend_id)
return Response(status_code=204)

72
app/api/routes/directories.py

@ -0,0 +1,72 @@
from fastapi import APIRouter, Depends, Query, status
from app.api.dependencies import get_directory_service
from app.schemas.directory import (
CreateDirectoryRequest,
CreateDirectoryResponse,
DirectoryChildrenResponse,
DirectoryItem,
DirectorySummary,
MoveDirectoryRequest,
RenameDirectoryRequest,
)
from app.services.directory_service import DirectoryService
router = APIRouter()
@router.get("/{directory_id}/children", response_model=DirectoryChildrenResponse)
async def list_directory_children(
directory_id: str,
_cursor: str | None = Query(default=None, alias="cursor"),
_limit: int = Query(default=100, ge=1, le=500, alias="limit"),
service: DirectoryService = Depends(get_directory_service),
) -> DirectoryChildrenResponse:
result = await service.list_children(directory_id)
return DirectoryChildrenResponse(
directory=DirectorySummary.model_validate(result.directory, from_attributes=True),
items=[
DirectoryItem(
kind=item.kind,
id=item.item.id,
name=item.item.name,
mime_type=getattr(item.item, "mime_type", None),
size_bytes=getattr(item.item, "size_bytes", None),
created_at=item.item.created_at,
updated_at=item.item.updated_at,
)
for item in result.children
],
next_cursor=None,
)
@router.post("", response_model=CreateDirectoryResponse, status_code=status.HTTP_201_CREATED)
async def create_directory(
payload: CreateDirectoryRequest,
service: DirectoryService = Depends(get_directory_service),
) -> CreateDirectoryResponse:
directory = await service.create_directory(parent_id=payload.parent_id, name=payload.name)
return CreateDirectoryResponse(
directory=DirectorySummary.model_validate(directory, from_attributes=True)
)
@router.post("/{directory_id}/rename", response_model=CreateDirectoryResponse)
async def rename_directory(
directory_id: str,
payload: RenameDirectoryRequest,
service: DirectoryService = Depends(get_directory_service),
) -> CreateDirectoryResponse:
directory = await service.rename_directory(directory_id=directory_id, name=payload.name)
return CreateDirectoryResponse(directory=DirectorySummary.model_validate(directory, from_attributes=True))
@router.post("/{directory_id}/move", response_model=CreateDirectoryResponse)
async def move_directory(
directory_id: str,
payload: MoveDirectoryRequest,
service: DirectoryService = Depends(get_directory_service),
) -> CreateDirectoryResponse:
directory = await service.move_directory(directory_id=directory_id, target_parent_id=payload.target_parent_id)
return CreateDirectoryResponse(directory=DirectorySummary.model_validate(directory, from_attributes=True))

76
app/api/routes/exports.py

@ -0,0 +1,76 @@
from __future__ import annotations
from fastapi import APIRouter, Depends
from app.api.dependencies import get_export_service
from app.core.exceptions import ValidationError
from app.schemas.export import (
MetadataExportResponse,
MetadataImportRequest,
MetadataImportResponse,
MetadataRestorePlanResponse,
MetadataValidationResponse,
)
from app.services.export_service import ExportService
from app.services.job_service import JobService
router = APIRouter()
@router.get("/metadata", response_model=MetadataExportResponse)
async def export_metadata(service: ExportService = Depends(get_export_service)) -> MetadataExportResponse:
data = await service.export_metadata()
exported_at = data.pop("exported_at")
return MetadataExportResponse(exported_at=exported_at, data=data)
@router.get("/metadata/integrity", response_model=MetadataValidationResponse)
async def metadata_integrity(service: ExportService = Depends(get_export_service)) -> MetadataValidationResponse:
result = await service.verify_runtime_integrity()
return MetadataValidationResponse(ok=result["ok"], issues=result["issues"])
@router.post("/metadata/validate", response_model=MetadataValidationResponse)
async def validate_metadata(
payload: MetadataImportRequest,
service: ExportService = Depends(get_export_service),
) -> MetadataValidationResponse:
result = await service.validate_metadata_snapshot(payload.data)
return MetadataValidationResponse(ok=result["ok"], issues=result["issues"])
@router.post("/metadata/restore-plan", response_model=MetadataRestorePlanResponse)
async def restore_plan(
payload: MetadataImportRequest,
service: ExportService = Depends(get_export_service),
) -> MetadataRestorePlanResponse:
result = await service.build_restore_plan(payload.data)
return MetadataRestorePlanResponse(
ok=result["ok"],
issues=result["issues"],
validation_token=result["validation_token"],
summary=result["summary"],
)
@router.post("/metadata/import", response_model=MetadataImportResponse)
async def import_metadata(
payload: MetadataImportRequest,
service: ExportService = Depends(get_export_service),
) -> MetadataImportResponse:
if not payload.confirm_replace:
raise ValidationError("Metadata import requires confirm_replace=true.")
restore_plan_result = await service.build_restore_plan(payload.data)
if not restore_plan_result["ok"]:
raise ValidationError("Metadata snapshot is invalid and cannot be imported.")
if payload.validation_token != restore_plan_result["validation_token"]:
raise ValidationError("Metadata import requires a current validation_token from restore-plan.")
imported_tables = await service.import_metadata(payload.data)
job_service = JobService(service.session)
reconcile_job = await job_service.enqueue_full_reconcile()
health_job_count = 0
for backend in await job_service.backend_repository.list_enabled():
await job_service.enqueue_backend_health_check(backend.id)
health_job_count += 1
scheduled_jobs = [reconcile_job.kind] + ["check_backend_health"] * health_job_count
return MetadataImportResponse(imported_tables=imported_tables, scheduled_jobs=scheduled_jobs)

189
app/api/routes/files.py

@ -0,0 +1,189 @@
from typing import AsyncIterator
import anyio
from fastapi import APIRouter, Depends, Header, Response, status
from fastapi.responses import FileResponse, StreamingResponse
from app.api.dependencies import get_file_service, get_repair_service
from app.schemas.file import (
DeletedFileSummary,
FileDetailResponse,
FilePlacementBackendSummary,
FileMutationResponse,
FileReplicaSummary,
FileSummary,
MoveFileRequest,
PreviewArtifactSummary,
RecycleBinResponse,
RenameFileRequest,
)
from app.services.file_service import FileService
from app.schemas.repair import RepairFileResponse
from app.services.repair_service import RepairService
router = APIRouter()
@router.get("/recycle-bin", response_model=RecycleBinResponse)
async def list_recycle_bin(
service: FileService = Depends(get_file_service),
) -> RecycleBinResponse:
items = await service.list_deleted_files()
return RecycleBinResponse(
items=[DeletedFileSummary.model_validate(item, from_attributes=True) for item in items]
)
@router.delete("/{file_id}", response_model=FileMutationResponse)
async def delete_file(
file_id: str,
service: FileService = Depends(get_file_service),
) -> FileMutationResponse:
file_entry = await service.delete_file(file_id)
return FileMutationResponse(file=DeletedFileSummary.model_validate(file_entry, from_attributes=True))
@router.post("/{file_id}/rename", response_model=FileMutationResponse)
async def rename_file(
file_id: str,
payload: RenameFileRequest,
service: FileService = Depends(get_file_service),
) -> FileMutationResponse:
file_entry = await service.rename_file(file_id=file_id, name=payload.name)
return FileMutationResponse(file=FileSummary.model_validate(file_entry, from_attributes=True))
@router.post("/{file_id}/move", response_model=FileMutationResponse)
async def move_file(
file_id: str,
payload: MoveFileRequest,
service: FileService = Depends(get_file_service),
) -> FileMutationResponse:
file_entry = await service.move_file(file_id=file_id, target_parent_id=payload.target_parent_id)
return FileMutationResponse(file=FileSummary.model_validate(file_entry, from_attributes=True))
@router.post("/{file_id}/restore", response_model=FileMutationResponse)
async def restore_file(
file_id: str,
service: FileService = Depends(get_file_service),
) -> FileMutationResponse:
file_entry = await service.restore_file(file_id)
return FileMutationResponse(file=FileSummary.model_validate(file_entry, from_attributes=True))
@router.post("/{file_id}/reconcile", response_model=RepairFileResponse)
async def reconcile_file(
file_id: str,
service: RepairService = Depends(get_repair_service),
) -> RepairFileResponse:
enqueued = await service.enqueue_repairs_for_file(file_id)
return RepairFileResponse(file_id=file_id, enqueued_jobs=enqueued)
@router.get("/{file_id}", response_model=FileDetailResponse)
async def get_file_detail(
file_id: str,
service: FileService = Depends(get_file_service),
) -> FileDetailResponse:
file_entry, artifacts, replicas, decision = await service.get_file_detail(file_id)
replica_items = [
FileReplicaSummary(
id=replica.id,
blob_id=replica.blob_id,
backend_id=backend.id,
backend_name=backend.name,
backend_type=backend.type,
backend_stability_class=backend.stability_class,
backend_enabled=backend.is_enabled,
storage_key=replica.storage_key,
status=replica.status,
size_bytes=replica.size_bytes,
checksum=replica.checksum,
etag=replica.etag,
last_verified_at=replica.last_verified_at,
created_at=replica.created_at,
updated_at=replica.updated_at,
)
for replica, backend in replicas
]
ready_replica_backend_ids = sorted({item.backend_id for item in replica_items if item.status == "ready"})
desired_backend_ids = [backend.id for backend in (decision.backends if decision is not None else [])]
return FileDetailResponse(
file=FileSummary.model_validate(file_entry, from_attributes=True),
preview_artifacts=[
PreviewArtifactSummary.model_validate(artifact, from_attributes=True) for artifact in artifacts
],
replicas=replica_items,
placement_file_class=decision.file_class if decision is not None else None,
desired_backends=[
FilePlacementBackendSummary.model_validate(backend, from_attributes=True)
for backend in (decision.backends if decision is not None else [])
],
ready_replica_backend_ids=ready_replica_backend_ids,
missing_backend_ids=[backend_id for backend_id in desired_backend_ids if backend_id not in ready_replica_backend_ids],
)
@router.get("/{file_id}/download")
async def download_file(
file_id: str,
service: FileService = Depends(get_file_service),
) -> FileResponse:
file_entry, file_path = await service.resolve_download(file_id)
return FileResponse(
path=file_path,
filename=file_entry.name,
media_type=file_entry.mime_type or "application/octet-stream",
)
@router.get("/{file_id}/preview")
async def preview_file(
file_id: str,
service: FileService = Depends(get_file_service),
) -> FileResponse:
file_entry, file_path = await service.resolve_preview(file_id)
return FileResponse(
path=file_path,
filename=file_entry.name,
media_type=file_entry.mime_type or "application/octet-stream",
content_disposition_type="inline",
)
@router.get("/{file_id}/stream")
async def stream_file(
file_id: str,
range_header: str | None = Header(default=None, alias="Range"),
service: FileService = Depends(get_file_service),
) -> Response:
stream_result = await service.resolve_stream(file_id, range_header)
chunk_size = 64 * 1024
async def body_iter() -> AsyncIterator[bytes]:
remaining = stream_result.end - stream_result.start + 1
async with await anyio.open_file(stream_result.file_path, "rb") as file_handle:
await file_handle.seek(stream_result.start)
while remaining > 0:
chunk = await file_handle.read(min(chunk_size, remaining))
if not chunk:
break
remaining -= len(chunk)
yield chunk
headers = {
"Accept-Ranges": "bytes",
"Content-Length": str(stream_result.end - stream_result.start + 1),
}
status_code = status.HTTP_200_OK
if range_header:
headers["Content-Range"] = f"bytes {stream_result.start}-{stream_result.end}/{stream_result.file_size}"
status_code = status.HTTP_206_PARTIAL_CONTENT
return StreamingResponse(
body_iter(),
status_code=status_code,
headers=headers,
media_type=stream_result.file_entry.mime_type or "application/octet-stream",
)

51
app/api/routes/jobs.py

@ -0,0 +1,51 @@
from __future__ import annotations
from fastapi import APIRouter, Depends
from app.api.dependencies import get_job_service, get_repair_service
from app.schemas.job import JobDetailResponse, JobListResponse, JobMutationResponse, JobRunPendingResponse, JobSummary
from app.services.job_service import JobService
from app.services.repair_service import RepairService
from app.schemas.repair import EnqueueHealthChecksResponse
router = APIRouter()
@router.get("", response_model=JobListResponse)
async def list_jobs(service: JobService = Depends(get_job_service)) -> JobListResponse:
items = await service.list_jobs()
return JobListResponse(items=[JobSummary.model_validate(item, from_attributes=True) for item in items])
@router.get("/{job_id}", response_model=JobDetailResponse)
async def get_job(job_id: str, service: JobService = Depends(get_job_service)) -> JobDetailResponse:
job = await service.get_job(job_id)
return JobDetailResponse(job=JobSummary.model_validate(job, from_attributes=True))
@router.post("/{job_id}/retry", response_model=JobMutationResponse)
async def retry_job(job_id: str, service: JobService = Depends(get_job_service)) -> JobMutationResponse:
job = await service.retry_job(job_id)
return JobMutationResponse(job=JobSummary.model_validate(job, from_attributes=True))
@router.post("/run-pending", response_model=JobRunPendingResponse)
async def run_pending_jobs(service: JobService = Depends(get_job_service)) -> JobRunPendingResponse:
result = await service.run_pending_jobs()
return JobRunPendingResponse(processed=result.processed, completed=result.completed, failed=result.failed)
@router.post("/enqueue-health-checks", response_model=EnqueueHealthChecksResponse)
async def enqueue_health_checks(
service: RepairService = Depends(get_repair_service),
) -> EnqueueHealthChecksResponse:
enqueued = await service.enqueue_backend_health_checks()
return EnqueueHealthChecksResponse(enqueued_jobs=enqueued)
@router.post("/enqueue-full-reconcile", response_model=EnqueueHealthChecksResponse)
async def enqueue_full_reconcile(
service: RepairService = Depends(get_repair_service),
) -> EnqueueHealthChecksResponse:
enqueued = await service.enqueue_full_reconcile()
return EnqueueHealthChecksResponse(enqueued_jobs=enqueued)

81
app/api/routes/policies.py

@ -0,0 +1,81 @@
from __future__ import annotations
from fastapi import APIRouter, Depends
from fastapi import Query
from app.api.dependencies import get_policy_service, get_repair_service
from app.schemas.policy import (
PlacementDecisionBackendSummary,
PlacementDecisionResponse,
PlacementPolicyListResponse,
PlacementPolicyMutationResponse,
PlacementPolicySummary,
UpsertPlacementPolicyRequest,
)
from app.services.policy_service import PolicyService
from app.services.repair_service import RepairService
router = APIRouter()
def _policy_to_summary(policy) -> PlacementPolicySummary:
return PlacementPolicySummary(
id=policy.id,
file_class=policy.file_class,
require_local=policy.require_local,
stable_replica_count=policy.stable_replica_count,
opportunistic_replica_count=policy.opportunistic_replica_count,
preferred_backend_ids=PolicyService._decode_backend_ids(policy.preferred_backend_ids_json),
excluded_backend_ids=PolicyService._decode_backend_ids(policy.excluded_backend_ids_json),
max_non_local_size_bytes=policy.max_non_local_size_bytes,
created_at=policy.created_at,
updated_at=policy.updated_at,
)
@router.get("/placement", response_model=PlacementPolicyListResponse)
async def list_placement_policies(
service: PolicyService = Depends(get_policy_service),
) -> PlacementPolicyListResponse:
policies = await service.list_policies()
return PlacementPolicyListResponse(items=[_policy_to_summary(policy) for policy in policies])
@router.get("/placement/preview", response_model=PlacementDecisionResponse)
async def preview_placement_decision(
mime_type: str | None = Query(default=None),
size_bytes: int | None = Query(default=None, ge=0),
service: PolicyService = Depends(get_policy_service),
) -> PlacementDecisionResponse:
decision = await service.get_placement_decision(mime_type, size_bytes=size_bytes)
return PlacementDecisionResponse(
file_class=decision.file_class,
mime_type=mime_type,
size_bytes=size_bytes,
non_local_allowed=decision.non_local_allowed,
policy=_policy_to_summary(decision.policy),
selected_backends=[
PlacementDecisionBackendSummary.model_validate(backend, from_attributes=True)
for backend in decision.backends
],
)
@router.put("/placement/{file_class}", response_model=PlacementPolicyMutationResponse)
async def upsert_placement_policy(
file_class: str,
payload: UpsertPlacementPolicyRequest,
service: PolicyService = Depends(get_policy_service),
repair_service: RepairService = Depends(get_repair_service),
) -> PlacementPolicyMutationResponse:
policy = await service.upsert_policy(
file_class=file_class,
require_local=payload.require_local,
stable_replica_count=payload.stable_replica_count,
opportunistic_replica_count=payload.opportunistic_replica_count,
preferred_backend_ids=payload.preferred_backend_ids,
excluded_backend_ids=payload.excluded_backend_ids,
max_non_local_size_bytes=payload.max_non_local_size_bytes,
)
enqueued_jobs = await repair_service.enqueue_full_reconcile()
return PlacementPolicyMutationResponse(policy=_policy_to_summary(policy), reconcile_enqueued_jobs=enqueued_jobs)

21
app/api/routes/system.py

@ -0,0 +1,21 @@
from fastapi import APIRouter
from app.db.health import ping_database
from app.db.session import create_engine
from app.schemas.system import HealthResponse, ReadyResponse
router = APIRouter()
@router.get("/health", response_model=HealthResponse)
async def health_check() -> HealthResponse:
return HealthResponse(status="ok")
@router.get("/ready", response_model=ReadyResponse)
async def readiness_check() -> ReadyResponse:
database_ok = await ping_database(create_engine())
return ReadyResponse(
status="ready" if database_ok else "degraded",
database="ok" if database_ok else "error",
)

77
app/api/routes/tus_uploads.py

@ -0,0 +1,77 @@
from fastapi import APIRouter, Depends, Header, Request, status
from fastapi.responses import Response
from app.api.dependencies import get_upload_service
from app.schemas.common import ErrorResponse
from app.services.upload_service import CreateUploadInput, UploadService
router = APIRouter()
TUS_VERSION = "1.0.0"
@router.post("", status_code=status.HTTP_201_CREATED, responses={409: {"model": ErrorResponse}})
async def create_tus_upload(
upload_length: int = Header(alias="Upload-Length"),
upload_metadata: str | None = Header(default=None, alias="Upload-Metadata"),
tus_resumable: str = Header(alias="Tus-Resumable"),
service: UploadService = Depends(get_upload_service),
) -> Response:
if tus_resumable != TUS_VERSION:
return Response(status_code=status.HTTP_412_PRECONDITION_FAILED)
upload_session = await service.create_upload(
CreateUploadInput(upload_length=upload_length, upload_metadata=upload_metadata)
)
return Response(
status_code=status.HTTP_201_CREATED,
headers={
"Location": upload_session.tus_upload_url or f"/files/{upload_session.id}",
"Tus-Resumable": TUS_VERSION,
},
)
@router.head("/{upload_id}")
async def head_tus_upload(
upload_id: str,
tus_resumable: str = Header(alias="Tus-Resumable"),
service: UploadService = Depends(get_upload_service),
) -> Response:
if tus_resumable != TUS_VERSION:
return Response(status_code=status.HTTP_412_PRECONDITION_FAILED)
upload_session = await service.get_upload(upload_id)
return Response(
status_code=status.HTTP_200_OK,
headers={
"Tus-Resumable": TUS_VERSION,
"Upload-Offset": str(upload_session.received_size_bytes),
"Upload-Length": str(upload_session.total_size_bytes),
},
)
@router.patch("/{upload_id}", status_code=status.HTTP_204_NO_CONTENT, responses={409: {"model": ErrorResponse}})
async def patch_tus_upload(
upload_id: str,
request: Request,
upload_offset: int = Header(alias="Upload-Offset"),
tus_resumable: str = Header(alias="Tus-Resumable"),
content_type: str = Header(alias="Content-Type"),
service: UploadService = Depends(get_upload_service),
) -> Response:
if tus_resumable != TUS_VERSION:
return Response(status_code=status.HTTP_412_PRECONDITION_FAILED)
if content_type != "application/offset+octet-stream":
return Response(status_code=status.HTTP_415_UNSUPPORTED_MEDIA_TYPE)
chunk = await request.body()
upload_session = await service.append_upload_bytes(upload_id, upload_offset, chunk)
return Response(
status_code=status.HTTP_204_NO_CONTENT,
headers={
"Tus-Resumable": TUS_VERSION,
"Upload-Offset": str(upload_session.received_size_bytes),
},
)

24
app/api/routes/uploads.py

@ -0,0 +1,24 @@
from fastapi import APIRouter, Depends
from app.api.dependencies import get_upload_service
from app.schemas.file import FileSummary
from app.schemas.upload import FinalizeUploadRequest, FinalizeUploadResponse, UploadSummary
from app.services.upload_service import FinalizeUploadInput, UploadService
router = APIRouter()
@router.post("/{upload_id}/finalize", response_model=FinalizeUploadResponse)
async def finalize_upload(
upload_id: str,
payload: FinalizeUploadRequest,
service: UploadService = Depends(get_upload_service),
) -> FinalizeUploadResponse:
upload_session, file_entry = await service.finalize_upload(
upload_id,
FinalizeUploadInput(directory_id=payload.directory_id, filename=payload.filename),
)
return FinalizeUploadResponse(
upload=UploadSummary.model_validate(upload_session, from_attributes=True),
file=FileSummary.model_validate(file_entry, from_attributes=True),
)

2
app/core/__init__.py

@ -0,0 +1,2 @@
"""Core configuration and bootstrap helpers."""

130
app/core/backend_config.py

@ -0,0 +1,130 @@
from __future__ import annotations
from typing import Any, Literal
from pydantic import BaseModel, Field, ValidationError as PydanticValidationError, field_validator
from app.core.exceptions import ValidationError
BackendType = Literal["local_directory", "s3", "webdav"]
class BackendConfigError(ValidationError):
code = "backend_config_invalid"
message = "Backend configuration is invalid."
class LocalDirectoryConfig(BaseModel):
base_path: str = Field(min_length=1)
@field_validator("base_path")
@classmethod
def clean_base_path(cls, value: str) -> str:
cleaned = value.strip()
if not cleaned:
raise ValueError("base_path must not be empty")
return cleaned
class S3CompatibleConfig(BaseModel):
bucket: str = Field(min_length=1)
region: str | None = None
endpoint_url: str | None = None
access_key_id: str | None = None
secret_access_key: str | None = None
prefix: str = ""
force_path_style: bool = False
verify_ssl: bool = True
storage_class: str | None = None
server_side_encryption: str | None = None
@field_validator("bucket")
@classmethod
def clean_bucket(cls, value: str) -> str:
cleaned = value.strip()
if not cleaned:
raise ValueError("bucket must not be empty")
return cleaned
@field_validator("prefix")
@classmethod
def clean_prefix(cls, value: str) -> str:
return value.strip().strip("/")
class WebDAVConfig(BaseModel):
endpoint_url: str = Field(min_length=1)
username: str = Field(min_length=1)
password: str | None = None
root_path: str = Field(default="iron", min_length=1)
verify_ssl: bool = True
@field_validator("endpoint_url", "username")
@classmethod
def clean_required_text(cls, value: str) -> str:
cleaned = value.strip()
if not cleaned:
raise ValueError("value must not be empty")
return cleaned
@field_validator("endpoint_url")
@classmethod
def clean_endpoint_url(cls, value: str) -> str:
return value.strip().rstrip("/")
@field_validator("root_path")
@classmethod
def clean_root_path(cls, value: str) -> str:
cleaned = "/".join(part for part in value.strip().strip("/").split("/") if part)
if not cleaned:
raise ValueError("root_path must not be empty")
return cleaned
CONFIG_MODELS: dict[str, type[BaseModel]] = {
"local_directory": LocalDirectoryConfig,
"s3": S3CompatibleConfig,
"webdav": WebDAVConfig,
}
SECRET_CONFIG_KEYS = {
"access_key_id",
"secret_access_key",
"password",
}
def supported_backend_types() -> set[str]:
return set(CONFIG_MODELS)
def validate_backend_config(backend_type: str, config: dict[str, Any]) -> dict[str, Any]:
model = CONFIG_MODELS.get(backend_type)
if model is None:
raise BackendConfigError(f"Unsupported backend type: {backend_type}.")
try:
return model.model_validate(config).model_dump()
except PydanticValidationError as exc:
first_error = exc.errors()[0] if exc.errors() else {}
location = ".".join(str(part) for part in first_error.get("loc", []))
message = first_error.get("msg", "Invalid backend configuration.")
prefix = f"{location}: " if location else ""
raise BackendConfigError(prefix + message) from exc
def split_backend_config(backend_type: str, config: dict[str, Any]) -> tuple[dict[str, Any], dict[str, Any]]:
normalized = validate_backend_config(backend_type, config)
public_config: dict[str, Any] = {}
secret_config: dict[str, Any] = {}
for key, value in normalized.items():
if key in SECRET_CONFIG_KEYS:
if value:
secret_config[key] = value
else:
public_config[key] = value
return public_config, secret_config
def merge_backend_config(public_config: dict[str, Any], secret_config: dict[str, Any]) -> dict[str, Any]:
return {**public_config, **secret_config}

54
app/core/config.py

@ -0,0 +1,54 @@
import os
from dataclasses import dataclass
from functools import lru_cache
def _get_bool_env(name: str, default: bool) -> bool:
raw = os.getenv(name)
if raw is None:
return default
return raw.strip().lower() in {"1", "true", "yes", "on"}
@dataclass(frozen=True)
class Settings:
app_name: str = "Iron Gateway"
debug: bool = False
api_prefix: str = "/api"
database_url: str = "sqlite+aiosqlite:///./iron.db"
upload_temp_dir: str = "./.iron-temp/uploads"
local_storage_dir: str = "./.iron-storage/local"
cache_dir: str = "./.iron-cache"
bootstrap_username: str = "admin"
bootstrap_password: str = "changeme-iron"
secret_key: str = "iron-local-development-secret-key"
auth_session_ttl_hours: int = 24 * 7
job_poll_interval_seconds: float = 2.0
job_batch_size: int = 10
@classmethod
def from_env(cls) -> "Settings":
return cls(
app_name=os.getenv("IRON_APP_NAME", cls.app_name),
debug=_get_bool_env("IRON_DEBUG", cls.debug),
api_prefix=os.getenv("IRON_API_PREFIX", cls.api_prefix),
database_url=os.getenv("IRON_DATABASE_URL", cls.database_url),
upload_temp_dir=os.getenv("IRON_UPLOAD_TEMP_DIR", cls.upload_temp_dir),
local_storage_dir=os.getenv("IRON_LOCAL_STORAGE_DIR", cls.local_storage_dir),
cache_dir=os.getenv("IRON_CACHE_DIR", cls.cache_dir),
bootstrap_username=os.getenv("IRON_BOOTSTRAP_USERNAME", cls.bootstrap_username),
bootstrap_password=os.getenv("IRON_BOOTSTRAP_PASSWORD", cls.bootstrap_password),
secret_key=os.getenv("IRON_SECRET_KEY", cls.secret_key),
auth_session_ttl_hours=int(
os.getenv("IRON_AUTH_SESSION_TTL_HOURS", str(cls.auth_session_ttl_hours))
),
job_poll_interval_seconds=float(
os.getenv("IRON_JOB_POLL_INTERVAL_SECONDS", str(cls.job_poll_interval_seconds))
),
job_batch_size=int(os.getenv("IRON_JOB_BATCH_SIZE", str(cls.job_batch_size))),
)
@lru_cache(maxsize=1)
def get_settings() -> Settings:
return Settings.from_env()

77
app/core/exceptions.py

@ -0,0 +1,77 @@
class AppError(Exception):
code = "internal_error"
message = "An unexpected error occurred."
def __init__(self, message: str | None = None) -> None:
super().__init__(message or self.message)
self.detail = message or self.message
class NotFoundError(AppError):
code = "not_found"
message = "Resource not found."
class ConflictError(AppError):
code = "conflict"
message = "Resource conflict."
class ValidationError(AppError):
code = "validation_error"
message = "Validation failed."
class UnauthorizedError(AppError):
code = "unauthorized"
message = "Authentication is required."
class AuthenticationFailedError(UnauthorizedError):
code = "authentication_failed"
message = "Invalid username or password."
class AuthSessionExpiredError(UnauthorizedError):
code = "auth_session_expired"
message = "Authentication session is no longer valid."
class DirectoryNotFoundError(NotFoundError):
code = "directory_not_found"
message = "Directory does not exist."
class DirectoryConflictError(ConflictError):
code = "directory_conflict"
message = "Directory already exists."
class FileNotFoundError(NotFoundError):
code = "file_not_found"
message = "File does not exist."
class FileConflictError(ConflictError):
code = "file_conflict"
message = "File already exists."
class UploadNotFoundError(NotFoundError):
code = "upload_not_found"
message = "Upload session does not exist."
class UploadConflictError(ConflictError):
code = "upload_conflict"
message = "Upload state conflict."
class RangeNotSatisfiableError(AppError):
code = "range_not_satisfiable"
message = "Requested byte range is not satisfiable."
class PreviewNotSupportedError(AppError):
code = "preview_not_supported"
message = "Preview is not supported for this file type."

5
app/core/ids.py

@ -0,0 +1,5 @@
from uuid import uuid4
def new_id(prefix: str) -> str:
return f"{prefix}_{uuid4().hex}"

50
app/core/secret_codec.py

@ -0,0 +1,50 @@
from __future__ import annotations
import base64
import hashlib
import json
from typing import Any
from cryptography.fernet import Fernet, InvalidToken
from app.core.config import get_settings
from app.core.exceptions import ValidationError
SECRET_PREFIX = "fernet:v1:"
class SecretConfigError(ValidationError):
code = "secret_config_invalid"
message = "Secret configuration cannot be decrypted."
def encrypt_secret_config(secret_config: dict[str, Any]) -> str:
payload = json.dumps(secret_config, sort_keys=True, separators=(",", ":"), ensure_ascii=True).encode("utf-8")
token = _fernet().encrypt(payload).decode("utf-8")
return SECRET_PREFIX + token
def decrypt_secret_config(raw_value: str) -> dict[str, Any]:
if not raw_value:
return {}
if not raw_value.startswith(SECRET_PREFIX):
value = json.loads(raw_value)
return value if isinstance(value, dict) else {}
token = raw_value.removeprefix(SECRET_PREFIX).encode("utf-8")
try:
payload = _fernet().decrypt(token)
except InvalidToken as exc:
raise SecretConfigError() from exc
value = json.loads(payload.decode("utf-8"))
return value if isinstance(value, dict) else {}
def list_secret_fields(raw_value: str) -> list[str]:
return sorted(decrypt_secret_config(raw_value))
def _fernet() -> Fernet:
digest = hashlib.sha256(get_settings().secret_key.encode("utf-8")).digest()
key = base64.urlsafe_b64encode(digest)
return Fernet(key)

34
app/core/security.py

@ -0,0 +1,34 @@
from __future__ import annotations
import base64
import hashlib
import hmac
import secrets
def hash_password(password: str, salt: str | None = None) -> str:
salt_value = salt or secrets.token_hex(16)
derived = hashlib.pbkdf2_hmac(
"sha256",
password.encode("utf-8"),
salt_value.encode("utf-8"),
200_000,
)
digest = base64.b64encode(derived).decode("utf-8")
return f"pbkdf2_sha256${salt_value}${digest}"
def verify_password(password: str, password_hash: str) -> bool:
algorithm, salt, digest = password_hash.split("$", 2)
if algorithm != "pbkdf2_sha256":
return False
expected = hash_password(password, salt=salt)
return hmac.compare_digest(expected, password_hash)
def generate_session_token() -> str:
return f"iron_{secrets.token_urlsafe(32)}"
def hash_session_token(token: str) -> str:
return hashlib.sha256(token.encode("utf-8")).hexdigest()

42
app/core/storage_layout.py

@ -0,0 +1,42 @@
from __future__ import annotations
from datetime import datetime
from app.models.entities import Blob
def build_blob_storage_key(blob: Blob) -> str:
content_hash = blob.content_hash
prefix_one = content_hash[:2] if len(content_hash) >= 2 else "xx"
prefix_two = content_hash[2:4] if len(content_hash) >= 4 else "xx"
kind = blob.kind or "blob"
return f"objects/{kind}/sha256/{prefix_one}/{prefix_two}/{content_hash}.blob"
def build_preview_artifact_storage_key(*, artifact_type: str, blob: Blob) -> str:
content_hash = blob.content_hash
prefix_one = content_hash[:2] if len(content_hash) >= 2 else "xx"
prefix_two = content_hash[2:4] if len(content_hash) >= 4 else "xx"
artifact_segment = _normalize_segment(artifact_type)
return f"objects/preview/{artifact_segment}/sha256/{prefix_one}/{prefix_two}/{content_hash}.blob"
def build_export_snapshot_storage_key(*, exported_at: datetime, digest: str) -> str:
year = f"{exported_at.year:04d}"
month = f"{exported_at.month:02d}"
day = f"{exported_at.day:02d}"
normalized_digest = _normalize_segment(digest) or "snapshot"
return f"objects/export/metadata/{year}/{month}/{day}/{normalized_digest}.json"
def build_manifest_storage_key(*, scope: str, identifier: str) -> str:
normalized_scope = _normalize_segment(scope) or "general"
normalized_identifier = _normalize_segment(identifier) or "manifest"
return f"objects/manifest/{normalized_scope}/{normalized_identifier}.json"
def _normalize_segment(value: str) -> str:
cleaned = "".join(char if char.isalnum() or char in {"-", "_"} else "-" for char in value.strip().lower())
while "--" in cleaned:
cleaned = cleaned.replace("--", "-")
return cleaned.strip("-_")

2
app/db/__init__.py

@ -0,0 +1,2 @@
"""Database module."""

6
app/db/base.py

@ -0,0 +1,6 @@
from sqlalchemy.orm import DeclarativeBase
class Base(DeclarativeBase):
pass

11
app/db/health.py

@ -0,0 +1,11 @@
from sqlalchemy import text
from sqlalchemy.ext.asyncio import AsyncEngine
async def ping_database(engine: AsyncEngine) -> bool:
try:
async with engine.connect() as connection:
await connection.execute(text("SELECT 1"))
return True
except Exception:
return False

37
app/db/session.py

@ -0,0 +1,37 @@
from collections.abc import AsyncIterator
from functools import lru_cache
from sqlalchemy import event
from sqlalchemy.ext.asyncio import AsyncEngine, AsyncSession, async_sessionmaker, create_async_engine
from app.core.config import get_settings
@lru_cache(maxsize=1)
def create_engine() -> AsyncEngine:
settings = get_settings()
engine = create_async_engine(settings.database_url, future=True)
@event.listens_for(engine.sync_engine, "connect")
def _set_sqlite_pragma(dbapi_connection, _connection_record) -> None:
cursor = dbapi_connection.cursor()
cursor.execute("PRAGMA foreign_keys=ON")
cursor.close()
return engine
def create_session_factory() -> async_sessionmaker[AsyncSession]:
return async_sessionmaker(create_engine(), class_=AsyncSession, expire_on_commit=False)
async def get_db_session() -> AsyncIterator[AsyncSession]:
session_factory = create_session_factory()
async with session_factory() as session:
yield session
async def dispose_engine() -> None:
engine = create_engine()
await engine.dispose()
create_engine.cache_clear()

2
app/domain/__init__.py

@ -0,0 +1,2 @@
"""Domain layer."""

2
app/jobs/__init__.py

@ -0,0 +1,2 @@
"""Background job framework."""

110
app/main.py

@ -0,0 +1,110 @@
from pathlib import Path
from contextlib import asynccontextmanager
import anyio
from fastapi import Depends, FastAPI, Request
from fastapi.responses import JSONResponse
from fastapi.staticfiles import StaticFiles
from app.api.router import api_router
from app.api.dependencies import require_current_user
from app.api.routes.tus_uploads import router as tus_upload_router
from app.web.routes import router as web_router
from app.core.config import get_settings
from app.core.exceptions import AppError
from app.db.session import create_session_factory
from app.services.directory_service import DirectoryService
from app.services.auth_service import AuthService
from app.services.job_service import JobService
from app.services.local_storage_service import LocalStorageService
from app.services.policy_service import PolicyService
from app.services.storage_service import StorageService
from app.services.upload_service import UploadService
WEB_DIST_DIR = Path(__file__).resolve().parent / "web" / "dist"
@asynccontextmanager
async def lifespan(_app: FastAPI):
session_factory = create_session_factory()
async with session_factory() as session:
service = DirectoryService(session)
await service.ensure_root_directory()
auth_service = AuthService(session)
await auth_service.ensure_bootstrap_user()
upload_service = UploadService(session)
await upload_service.ensure_temp_dir()
local_storage_service = LocalStorageService(session)
await local_storage_service.ensure_backend_ready()
policy_service = PolicyService(session)
await policy_service.ensure_default_policies()
storage_service = StorageService(session)
await storage_service.ensure_cache_dir()
settings = get_settings()
async def _job_loop() -> None:
while True:
async with session_factory() as session:
service = JobService(session)
await service.run_pending_jobs(limit=settings.job_batch_size)
await anyio.sleep(settings.job_poll_interval_seconds)
async with anyio.create_task_group() as task_group:
task_group.start_soon(_job_loop)
yield
task_group.cancel_scope.cancel()
def create_app() -> FastAPI:
settings = get_settings()
app = FastAPI(
title=settings.app_name,
debug=settings.debug,
version="0.1.0",
lifespan=lifespan,
)
@app.exception_handler(AppError)
async def app_error_handler(_request: Request, exc: AppError) -> JSONResponse:
status_code = 400
if exc.code.endswith("_not_found") or exc.code == "not_found":
status_code = 404
elif exc.code.endswith("_conflict") or exc.code == "conflict":
status_code = 409
elif exc.code == "unauthorized" or exc.code.startswith("auth_") or exc.code == "authentication_failed":
status_code = 401
elif exc.code == "range_not_satisfiable":
status_code = 416
return JSONResponse(
status_code=status_code,
content={
"error": {
"code": exc.code,
"message": exc.detail,
"details": None,
}
},
)
app.include_router(api_router, prefix=settings.api_prefix)
app.include_router(web_router)
app.include_router(
tus_upload_router,
prefix="/files",
tags=["tus"],
dependencies=[Depends(require_current_user)],
)
if (WEB_DIST_DIR / "assets").exists():
app.mount(
"/assets",
StaticFiles(directory=WEB_DIST_DIR / "assets"),
name="web-assets",
)
return app
app = create_app()

33
app/models/__init__.py

@ -0,0 +1,33 @@
"""SQLAlchemy models package."""
from app.models.entities import AuthSession
from app.models.entities import Backend
from app.models.entities import Blob
from app.models.entities import BlobReplica
from app.models.entities import CacheEntry
from app.models.entities import Directory
from app.models.entities import FileEntry
from app.models.entities import FileVersion
from app.models.entities import Job
from app.models.entities import PlacementPolicy
from app.models.entities import PreviewArtifact
from app.models.entities import UploadSession
from app.models.entities import UploadSessionPart
from app.models.entities import User
__all__ = [
"AuthSession",
"Backend",
"Blob",
"BlobReplica",
"CacheEntry",
"Directory",
"FileEntry",
"FileVersion",
"Job",
"PlacementPolicy",
"PreviewArtifact",
"UploadSession",
"UploadSessionPart",
"User",
]

275
app/models/entities.py

@ -0,0 +1,275 @@
from __future__ import annotations
from datetime import datetime
from typing import Optional
from sqlalchemy import Boolean, DateTime, ForeignKey, Index, Integer, String, Text, func
from sqlalchemy.orm import Mapped, mapped_column
from app.db.base import Base
class TimestampMixin:
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
nullable=False,
server_default=func.now(),
)
updated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
nullable=False,
server_default=func.now(),
onupdate=func.now(),
)
class User(TimestampMixin, Base):
__tablename__ = "users"
id: Mapped[str] = mapped_column(String, primary_key=True)
username: Mapped[str] = mapped_column(String, unique=True, nullable=False)
password_hash: Mapped[str] = mapped_column(String, nullable=False)
class AuthSession(TimestampMixin, Base):
__tablename__ = "auth_sessions"
__table_args__ = (
Index("idx_auth_sessions_user_status", "user_id", "revoked_at", "expires_at"),
Index("idx_auth_sessions_token_hash", "token_hash", unique=True),
)
id: Mapped[str] = mapped_column(String, primary_key=True)
user_id: Mapped[str] = mapped_column(String, ForeignKey("users.id"), nullable=False)
token_hash: Mapped[str] = mapped_column(String, nullable=False)
expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
revoked_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
class Directory(TimestampMixin, Base):
__tablename__ = "directories"
__table_args__ = (
Index("idx_directories_parent_id", "parent_id"),
Index("idx_directories_path_key", "path_key"),
)
id: Mapped[str] = mapped_column(String, primary_key=True)
parent_id: Mapped[Optional[str]] = mapped_column(
String,
ForeignKey("directories.id"),
nullable=True,
)
name: Mapped[str] = mapped_column(String, nullable=False)
path_key: Mapped[str] = mapped_column(String, unique=True, nullable=False)
deleted_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
class FileEntry(TimestampMixin, Base):
__tablename__ = "file_entries"
__table_args__ = (
Index("idx_file_entries_directory_name", "directory_id", "name", "is_deleted", unique=True),
Index("idx_file_entries_current_version_id", "current_version_id"),
Index("idx_file_entries_deleted_at", "deleted_at"),
)
id: Mapped[str] = mapped_column(String, primary_key=True)
directory_id: Mapped[str] = mapped_column(
String,
ForeignKey("directories.id"),
nullable=False,
)
name: Mapped[str] = mapped_column(String, nullable=False)
mime_type: Mapped[Optional[str]] = mapped_column(String, nullable=True)
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False)
current_version_id: Mapped[Optional[str]] = mapped_column(String, nullable=True)
is_deleted: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="0")
deleted_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
class FileVersion(Base):
__tablename__ = "file_versions"
__table_args__ = (
Index("idx_file_versions_file_entry_id", "file_entry_id"),
Index("idx_file_versions_content_hash", "content_hash"),
)
id: Mapped[str] = mapped_column(String, primary_key=True)
file_entry_id: Mapped[str] = mapped_column(
String,
ForeignKey("file_entries.id"),
nullable=False,
)
content_hash: Mapped[str] = mapped_column(String, nullable=False)
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False)
storage_layout: Mapped[str] = mapped_column(String, nullable=False)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
nullable=False,
server_default=func.now(),
)
class Blob(Base):
__tablename__ = "blobs"
__table_args__ = (
Index("idx_blobs_file_version_index", "file_version_id", "blob_index", unique=True),
Index("idx_blobs_content_hash", "content_hash"),
)
id: Mapped[str] = mapped_column(String, primary_key=True)
file_version_id: Mapped[str] = mapped_column(
String,
ForeignKey("file_versions.id"),
nullable=False,
)
blob_index: Mapped[int] = mapped_column(Integer, nullable=False)
kind: Mapped[str] = mapped_column(String, nullable=False)
content_hash: Mapped[str] = mapped_column(String, nullable=False)
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False)
logical_offset: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
nullable=False,
server_default=func.now(),
)
class Backend(TimestampMixin, Base):
__tablename__ = "backends"
__table_args__ = (
Index("idx_backends_enabled", "is_enabled"),
Index("idx_backends_priority", "write_priority", "read_priority"),
)
id: Mapped[str] = mapped_column(String, primary_key=True)
name: Mapped[str] = mapped_column(String, unique=True, nullable=False)
type: Mapped[str] = mapped_column(String, nullable=False)
stability_class: Mapped[str] = mapped_column(String, nullable=False)
read_priority: Mapped[int] = mapped_column(Integer, nullable=False)
write_priority: Mapped[int] = mapped_column(Integer, nullable=False)
is_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="1")
public_config_json: Mapped[str] = mapped_column(Text, nullable=False)
secret_config_json: Mapped[str] = mapped_column(Text, nullable=False)
capacity_hint_bytes: Mapped[Optional[int]] = mapped_column(Integer, nullable=True)
last_health_status: Mapped[Optional[str]] = mapped_column(String, nullable=True)
last_health_checked_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
class PlacementPolicy(TimestampMixin, Base):
__tablename__ = "placement_policies"
__table_args__ = (
Index("idx_placement_policies_file_class", "file_class", unique=True),
)
id: Mapped[str] = mapped_column(String, primary_key=True)
file_class: Mapped[str] = mapped_column(String, nullable=False)
require_local: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="1")
stable_replica_count: Mapped[int] = mapped_column(Integer, nullable=False, default=1, server_default="1")
opportunistic_replica_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
preferred_backend_ids_json: Mapped[str] = mapped_column(Text, nullable=False, default="[]", server_default="[]")
excluded_backend_ids_json: Mapped[str] = mapped_column(Text, nullable=False, default="[]", server_default="[]")
max_non_local_size_bytes: Mapped[Optional[int]] = mapped_column(Integer, nullable=True)
class BlobReplica(TimestampMixin, Base):
__tablename__ = "blob_replicas"
__table_args__ = (
Index("idx_blob_replicas_blob_backend", "blob_id", "backend_id", unique=True),
Index("idx_blob_replicas_backend_status", "backend_id", "status"),
Index("idx_blob_replicas_storage_key", "storage_key"),
)
id: Mapped[str] = mapped_column(String, primary_key=True)
blob_id: Mapped[str] = mapped_column(String, ForeignKey("blobs.id"), nullable=False)
backend_id: Mapped[str] = mapped_column(String, ForeignKey("backends.id"), nullable=False)
storage_key: Mapped[str] = mapped_column(String, nullable=False)
status: Mapped[str] = mapped_column(String, nullable=False)
etag: Mapped[Optional[str]] = mapped_column(String, nullable=True)
checksum: Mapped[Optional[str]] = mapped_column(String, nullable=True)
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False)
last_verified_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
class UploadSession(TimestampMixin, Base):
__tablename__ = "upload_sessions"
__table_args__ = (
Index("idx_upload_sessions_status", "status"),
Index("idx_upload_sessions_updated_at", "updated_at"),
)
id: Mapped[str] = mapped_column(String, primary_key=True)
directory_id: Mapped[Optional[str]] = mapped_column(String, ForeignKey("directories.id"), nullable=True)
filename: Mapped[Optional[str]] = mapped_column(String, nullable=True)
total_size_bytes: Mapped[int] = mapped_column(Integer, nullable=False)
received_size_bytes: Mapped[int] = mapped_column(Integer, nullable=False)
status: Mapped[str] = mapped_column(String, nullable=False)
temp_path: Mapped[str] = mapped_column(String, nullable=False)
tus_upload_url: Mapped[Optional[str]] = mapped_column(String, nullable=True)
upload_metadata_json: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
class UploadSessionPart(TimestampMixin, Base):
__tablename__ = "upload_session_parts"
__table_args__ = (
Index("idx_upload_session_parts_unique", "upload_session_id", "part_number", unique=True),
Index("idx_upload_session_parts_upload_id", "upload_session_id"),
)
id: Mapped[str] = mapped_column(String, primary_key=True)
upload_session_id: Mapped[str] = mapped_column(
String,
ForeignKey("upload_sessions.id"),
nullable=False,
)
part_number: Mapped[int] = mapped_column(Integer, nullable=False)
byte_offset: Mapped[int] = mapped_column(Integer, nullable=False)
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False)
checksum: Mapped[Optional[str]] = mapped_column(String, nullable=True)
status: Mapped[str] = mapped_column(String, nullable=False)
class Job(TimestampMixin, Base):
__tablename__ = "jobs"
__table_args__ = (
Index("idx_jobs_status_run_after", "status", "run_after"),
Index("idx_jobs_kind_status", "kind", "status"),
)
id: Mapped[str] = mapped_column(String, primary_key=True)
kind: Mapped[str] = mapped_column(String, nullable=False)
status: Mapped[str] = mapped_column(String, nullable=False)
payload_json: Mapped[str] = mapped_column(Text, nullable=False)
attempt_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
max_attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=5, server_default="5")
run_after: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
last_error: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
class PreviewArtifact(TimestampMixin, Base):
__tablename__ = "preview_artifacts"
__table_args__ = (
Index("idx_preview_artifacts_version_type", "file_version_id", "artifact_type", unique=True),
)
id: Mapped[str] = mapped_column(String, primary_key=True)
file_version_id: Mapped[str] = mapped_column(String, ForeignKey("file_versions.id"), nullable=False)
artifact_type: Mapped[str] = mapped_column(String, nullable=False)
blob_id: Mapped[str] = mapped_column(String, ForeignKey("blobs.id"), nullable=False)
status: Mapped[str] = mapped_column(String, nullable=False)
class CacheEntry(TimestampMixin, Base):
__tablename__ = "cache_entries"
__table_args__ = (
Index("idx_cache_entries_blob_id", "blob_id"),
Index("idx_cache_entries_last_accessed_at", "last_accessed_at"),
Index("idx_cache_entries_expires_at", "expires_at"),
)
id: Mapped[str] = mapped_column(String, primary_key=True)
blob_id: Mapped[str] = mapped_column(String, ForeignKey("blobs.id"), nullable=False)
local_path: Mapped[str] = mapped_column(String, nullable=False)
cache_kind: Mapped[str] = mapped_column(String, nullable=False)
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False)
status: Mapped[str] = mapped_column(String, nullable=False)
last_accessed_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
expires_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)

2
app/repositories/__init__.py

@ -0,0 +1,2 @@
"""Repository layer."""

40
app/repositories/auth_session_repository.py

@ -0,0 +1,40 @@
from __future__ import annotations
from datetime import datetime
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.models.entities import AuthSession
class AuthSessionRepository:
def __init__(self, session: AsyncSession) -> None:
self.session = session
async def create(self, auth_session: AuthSession) -> AuthSession:
self.session.add(auth_session)
await self.session.commit()
await self.session.refresh(auth_session)
return auth_session
async def get_active_by_token_hash(self, token_hash: str, now: datetime) -> AuthSession | None:
result = await self.session.execute(
select(AuthSession).where(
AuthSession.token_hash == token_hash,
AuthSession.revoked_at.is_(None),
AuthSession.expires_at > now,
)
)
return result.scalar_one_or_none()
async def get_by_token_hash(self, token_hash: str) -> AuthSession | None:
result = await self.session.execute(
select(AuthSession).where(AuthSession.token_hash == token_hash)
)
return result.scalar_one_or_none()
async def save(self, auth_session: AuthSession) -> AuthSession:
await self.session.commit()
await self.session.refresh(auth_session)
return auth_session

64
app/repositories/backend_repository.py

@ -0,0 +1,64 @@
from __future__ import annotations
import json
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.models.entities import Backend, PlacementPolicy
class BackendRepository:
def __init__(self, session: AsyncSession) -> None:
self.session = session
async def get_by_name(self, name: str) -> Backend | None:
result = await self.session.execute(select(Backend).where(Backend.name == name))
return result.scalar_one_or_none()
async def get_by_id(self, backend_id: str) -> Backend | None:
return await self.session.get(Backend, backend_id)
async def create(self, backend: Backend) -> Backend:
self.session.add(backend)
await self.session.flush()
await self.session.refresh(backend)
return backend
async def list_all(self) -> list[Backend]:
result = await self.session.execute(select(Backend).order_by(Backend.write_priority.desc(), Backend.name.asc()))
return list(result.scalars().all())
async def list_enabled(self) -> list[Backend]:
result = await self.session.execute(
select(Backend)
.where(Backend.is_enabled.is_(True))
.order_by(Backend.write_priority.desc(), Backend.read_priority.desc(), Backend.name.asc())
)
return list(result.scalars().all())
async def save(self, backend: Backend) -> Backend:
await self.session.flush()
await self.session.refresh(backend)
return backend
async def delete(self, backend: Backend) -> None:
await self.session.delete(backend)
await self.session.flush()
async def is_referenced_by_policy(self, backend_id: str) -> bool:
result = await self.session.execute(select(PlacementPolicy))
for policy in result.scalars().all():
preferred = json.loads(policy.preferred_backend_ids_json or "[]")
excluded = json.loads(policy.excluded_backend_ids_json or "[]")
if backend_id in preferred or backend_id in excluded:
return True
return False
async def list_enabled_by_type(self, backend_type: str) -> list[Backend]:
result = await self.session.execute(
select(Backend)
.where(Backend.is_enabled.is_(True), Backend.type == backend_type)
.order_by(Backend.write_priority.desc(), Backend.read_priority.desc(), Backend.name.asc())
)
return list(result.scalars().all())

75
app/repositories/blob_replica_repository.py

@ -0,0 +1,75 @@
from __future__ import annotations
from sqlalchemy import func, select
from sqlalchemy.ext.asyncio import AsyncSession
from app.models.entities import Backend
from app.models.entities import BlobReplica
class BlobReplicaRepository:
def __init__(self, session: AsyncSession) -> None:
self.session = session
async def get_by_blob_and_backend(self, blob_id: str, backend_id: str) -> BlobReplica | None:
result = await self.session.execute(
select(BlobReplica).where(
BlobReplica.blob_id == blob_id,
BlobReplica.backend_id == backend_id,
)
)
return result.scalar_one_or_none()
async def create(self, replica: BlobReplica) -> BlobReplica:
self.session.add(replica)
await self.session.flush()
await self.session.refresh(replica)
return replica
async def get_ready_replica_for_blob(self, blob_id: str) -> BlobReplica | None:
result = await self.session.execute(
select(BlobReplica)
.where(
BlobReplica.blob_id == blob_id,
BlobReplica.status == "ready",
)
.order_by(BlobReplica.created_at.asc())
)
return result.scalar_one_or_none()
async def list_ready_replicas_with_backends(self, blob_id: str) -> list[tuple[BlobReplica, Backend]]:
result = await self.session.execute(
select(BlobReplica, Backend)
.join(Backend, BlobReplica.backend_id == Backend.id)
.where(
BlobReplica.blob_id == blob_id,
BlobReplica.status == "ready",
Backend.is_enabled.is_(True),
)
.order_by(Backend.read_priority.desc(), BlobReplica.created_at.asc())
)
return list(result.all())
async def list_replicas_with_backends(self, blob_id: str) -> list[tuple[BlobReplica, Backend]]:
result = await self.session.execute(
select(BlobReplica, Backend)
.join(Backend, BlobReplica.backend_id == Backend.id)
.where(BlobReplica.blob_id == blob_id)
.order_by(Backend.read_priority.desc(), BlobReplica.created_at.asc())
)
return list(result.all())
async def list_by_blob(self, blob_id: str) -> list[BlobReplica]:
result = await self.session.execute(select(BlobReplica).where(BlobReplica.blob_id == blob_id))
return list(result.scalars().all())
async def save(self, replica: BlobReplica) -> BlobReplica:
await self.session.flush()
await self.session.refresh(replica)
return replica
async def count_by_backend(self, backend_id: str) -> int:
result = await self.session.execute(
select(func.count()).select_from(BlobReplica).where(BlobReplica.backend_id == backend_id)
)
return int(result.scalar_one())

31
app/repositories/cache_entry_repository.py

@ -0,0 +1,31 @@
from __future__ import annotations
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.models.entities import CacheEntry
class CacheEntryRepository:
def __init__(self, session: AsyncSession) -> None:
self.session = session
async def get_ready_by_blob(self, blob_id: str) -> CacheEntry | None:
result = await self.session.execute(
select(CacheEntry).where(
CacheEntry.blob_id == blob_id,
CacheEntry.status == "ready",
)
)
return result.scalar_one_or_none()
async def create(self, entry: CacheEntry) -> CacheEntry:
self.session.add(entry)
await self.session.flush()
await self.session.refresh(entry)
return entry
async def save(self, entry: CacheEntry) -> CacheEntry:
await self.session.flush()
await self.session.refresh(entry)
return entry

58
app/repositories/directory_repository.py

@ -0,0 +1,58 @@
from __future__ import annotations
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.models.entities import Directory
class DirectoryRepository:
def __init__(self, session: AsyncSession) -> None:
self.session = session
async def get_by_id(self, directory_id: str) -> Directory | None:
return await self.session.get(Directory, directory_id)
async def get_root(self) -> Directory | None:
result = await self.session.execute(
select(Directory).where(Directory.parent_id.is_(None), Directory.deleted_at.is_(None))
)
return result.scalar_one_or_none()
async def list_children(self, parent_id: str) -> list[Directory]:
result = await self.session.execute(
select(Directory)
.where(Directory.parent_id == parent_id, Directory.deleted_at.is_(None))
.order_by(Directory.name.asc())
)
return list(result.scalars().all())
async def get_by_parent_and_name(self, parent_id: str | None, name: str) -> Directory | None:
result = await self.session.execute(
select(Directory).where(
Directory.parent_id == parent_id,
Directory.name == name,
Directory.deleted_at.is_(None),
)
)
return result.scalar_one_or_none()
async def list_descendants_by_path_prefix(self, path_prefix: str) -> list[Directory]:
like_pattern = f"{path_prefix}/%"
result = await self.session.execute(
select(Directory)
.where(Directory.path_key.like(like_pattern), Directory.deleted_at.is_(None))
.order_by(Directory.path_key.asc())
)
return list(result.scalars().all())
async def create(self, directory: Directory) -> Directory:
self.session.add(directory)
await self.session.flush()
await self.session.refresh(directory)
return directory
async def save(self, directory: Directory) -> Directory:
await self.session.flush()
await self.session.refresh(directory)
return directory

103
app/repositories/file_repository.py

@ -0,0 +1,103 @@
from __future__ import annotations
from datetime import datetime, UTC
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.models.entities import Blob, FileEntry, FileVersion
class FileRepository:
def __init__(self, session: AsyncSession) -> None:
self.session = session
async def get_by_id(self, file_id: str) -> FileEntry | None:
return await self.session.get(FileEntry, file_id)
async def get_by_directory_and_name(self, directory_id: str, name: str) -> FileEntry | None:
result = await self.session.execute(
select(FileEntry).where(
FileEntry.directory_id == directory_id,
FileEntry.name == name,
FileEntry.is_deleted.is_(False),
FileEntry.deleted_at.is_(None),
)
)
return result.scalar_one_or_none()
async def list_by_directory(self, directory_id: str) -> list[FileEntry]:
result = await self.session.execute(
select(FileEntry)
.where(
FileEntry.directory_id == directory_id,
FileEntry.is_deleted.is_(False),
FileEntry.deleted_at.is_(None),
)
.order_by(FileEntry.name.asc())
)
return list(result.scalars().all())
async def list_deleted(self) -> list[FileEntry]:
result = await self.session.execute(
select(FileEntry)
.where(FileEntry.is_deleted.is_(True), FileEntry.deleted_at.is_not(None))
.order_by(FileEntry.deleted_at.desc(), FileEntry.name.asc())
)
return list(result.scalars().all())
async def list_active(self) -> list[FileEntry]:
result = await self.session.execute(
select(FileEntry)
.where(
FileEntry.is_deleted.is_(False),
FileEntry.deleted_at.is_(None),
)
.order_by(FileEntry.created_at.asc())
)
return list(result.scalars().all())
async def create_file_graph(
self,
file_entry: FileEntry,
file_version: FileVersion,
blob: Blob,
) -> FileEntry:
self.session.add(file_entry)
self.session.add(file_version)
await self.session.flush()
self.session.add(blob)
await self.session.flush()
await self.session.refresh(file_entry)
return file_entry
async def get_primary_blob_for_file(self, file_id: str) -> Blob | None:
result = await self.session.execute(
select(Blob)
.join(FileVersion, Blob.file_version_id == FileVersion.id)
.join(FileEntry, FileVersion.id == FileEntry.current_version_id)
.where(FileEntry.id == file_id, Blob.blob_index == 0)
)
return result.scalar_one_or_none()
async def get_blob_by_id(self, blob_id: str) -> Blob | None:
return await self.session.get(Blob, blob_id)
async def soft_delete(self, file_entry: FileEntry) -> FileEntry:
file_entry.is_deleted = True
file_entry.deleted_at = datetime.now(UTC)
await self.session.flush()
await self.session.refresh(file_entry)
return file_entry
async def restore(self, file_entry: FileEntry) -> FileEntry:
file_entry.is_deleted = False
file_entry.deleted_at = None
await self.session.flush()
await self.session.refresh(file_entry)
return file_entry
async def save(self, file_entry: FileEntry) -> FileEntry:
await self.session.flush()
await self.session.refresh(file_entry)
return file_entry

53
app/repositories/job_repository.py

@ -0,0 +1,53 @@
from __future__ import annotations
from datetime import datetime
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.models.entities import Job
class JobRepository:
def __init__(self, session: AsyncSession) -> None:
self.session = session
async def create(self, job: Job) -> Job:
self.session.add(job)
await self.session.flush()
await self.session.refresh(job)
return job
async def get_by_id(self, job_id: str) -> Job | None:
return await self.session.get(Job, job_id)
async def list_all(self) -> list[Job]:
result = await self.session.execute(select(Job).order_by(Job.created_at.desc()))
return list(result.scalars().all())
async def list_runnable(self, now: datetime, limit: int) -> list[Job]:
result = await self.session.execute(
select(Job)
.where(
Job.status.in_(["queued", "retryable"]),
Job.run_after <= now,
)
.order_by(Job.run_after.asc(), Job.created_at.asc())
.limit(limit)
)
return list(result.scalars().all())
async def find_active_by_kind_and_payload(self, kind: str, payload_json: str) -> Job | None:
result = await self.session.execute(
select(Job).where(
Job.kind == kind,
Job.payload_json == payload_json,
Job.status.in_(["queued", "retryable", "running"]),
)
)
return result.scalar_one_or_none()
async def save(self, job: Job) -> Job:
await self.session.flush()
await self.session.refresh(job)
return job

32
app/repositories/placement_policy_repository.py

@ -0,0 +1,32 @@
from __future__ import annotations
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.models.entities import PlacementPolicy
class PlacementPolicyRepository:
def __init__(self, session: AsyncSession) -> None:
self.session = session
async def get_by_file_class(self, file_class: str) -> PlacementPolicy | None:
result = await self.session.execute(
select(PlacementPolicy).where(PlacementPolicy.file_class == file_class)
)
return result.scalar_one_or_none()
async def list_all(self) -> list[PlacementPolicy]:
result = await self.session.execute(select(PlacementPolicy).order_by(PlacementPolicy.file_class.asc()))
return list(result.scalars().all())
async def create(self, policy: PlacementPolicy) -> PlacementPolicy:
self.session.add(policy)
await self.session.flush()
await self.session.refresh(policy)
return policy
async def save(self, policy: PlacementPolicy) -> PlacementPolicy:
await self.session.flush()
await self.session.refresh(policy)
return policy

39
app/repositories/preview_artifact_repository.py

@ -0,0 +1,39 @@
from __future__ import annotations
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.models.entities import PreviewArtifact
class PreviewArtifactRepository:
def __init__(self, session: AsyncSession) -> None:
self.session = session
async def get_by_version_and_type(self, file_version_id: str, artifact_type: str) -> PreviewArtifact | None:
result = await self.session.execute(
select(PreviewArtifact).where(
PreviewArtifact.file_version_id == file_version_id,
PreviewArtifact.artifact_type == artifact_type,
)
)
return result.scalar_one_or_none()
async def list_by_file_version(self, file_version_id: str) -> list[PreviewArtifact]:
result = await self.session.execute(
select(PreviewArtifact)
.where(PreviewArtifact.file_version_id == file_version_id)
.order_by(PreviewArtifact.artifact_type.asc())
)
return list(result.scalars().all())
async def create(self, artifact: PreviewArtifact) -> PreviewArtifact:
self.session.add(artifact)
await self.session.flush()
await self.session.refresh(artifact)
return artifact
async def save(self, artifact: PreviewArtifact) -> PreviewArtifact:
await self.session.flush()
await self.session.refresh(artifact)
return artifact

25
app/repositories/upload_repository.py

@ -0,0 +1,25 @@
from __future__ import annotations
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.models.entities import UploadSession
class UploadRepository:
def __init__(self, session: AsyncSession) -> None:
self.session = session
async def get_by_id(self, upload_id: str) -> UploadSession | None:
return await self.session.get(UploadSession, upload_id)
async def create(self, upload_session: UploadSession) -> UploadSession:
self.session.add(upload_session)
await self.session.flush()
await self.session.refresh(upload_session)
return upload_session
async def save(self, upload_session: UploadSession) -> UploadSession:
await self.session.flush()
await self.session.refresh(upload_session)
return upload_session

30
app/repositories/user_repository.py

@ -0,0 +1,30 @@
from __future__ import annotations
from sqlalchemy import func
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.models.entities import User
class UserRepository:
def __init__(self, session: AsyncSession) -> None:
self.session = session
async def count_users(self) -> int:
result = await self.session.execute(select(func.count()).select_from(User))
return int(result.scalar_one())
async def get_by_username(self, username: str) -> User | None:
result = await self.session.execute(select(User).where(User.username == username))
return result.scalar_one_or_none()
async def get_by_id(self, user_id: str) -> User | None:
result = await self.session.execute(select(User).where(User.id == user_id))
return result.scalar_one_or_none()
async def create(self, user: User) -> User:
self.session.add(user)
await self.session.commit()
await self.session.refresh(user)
return user

2
app/schemas/__init__.py

@ -0,0 +1,2 @@
"""Pydantic schemas."""

34
app/schemas/auth.py

@ -0,0 +1,34 @@
from __future__ import annotations
from datetime import datetime
from pydantic import BaseModel, ConfigDict
class UserSummary(BaseModel):
model_config = ConfigDict(from_attributes=True)
id: str
username: str
created_at: datetime
updated_at: datetime
class LoginRequest(BaseModel):
username: str
password: str
class LoginResponse(BaseModel):
access_token: str
token_type: str = "bearer"
expires_at: datetime
user: UserSummary
class AuthMeResponse(BaseModel):
user: UserSummary
class LogoutResponse(BaseModel):
success: bool = True

56
app/schemas/backend.py

@ -0,0 +1,56 @@
from datetime import datetime
from pydantic import BaseModel, Field
class BackendSummary(BaseModel):
id: str
name: str
type: str
stability_class: str
read_priority: int
write_priority: int
is_enabled: bool
last_health_status: str | None
last_health_checked_at: datetime | None
config: dict | None = None
secret_fields: list[str] = Field(default_factory=list)
created_at: datetime
updated_at: datetime
class BackendListResponse(BaseModel):
items: list[BackendSummary]
class CreateBackendRequest(BaseModel):
name: str
type: str
stability_class: str
read_priority: int
write_priority: int
config: dict
class UpdateBackendRequest(BaseModel):
name: str | None = None
stability_class: str | None = None
read_priority: int | None = None
write_priority: int | None = None
config: dict | None = None
class CreateBackendResponse(BaseModel):
backend: BackendSummary
class BackendCheckResponse(BaseModel):
backend_id: str
status: str
checked_at: datetime | None
detail: str | None = None
class BackendMutationResponse(BaseModel):
ok: bool = True
backend: BackendSummary

13
app/schemas/common.py

@ -0,0 +1,13 @@
from typing import Any
from pydantic import BaseModel
class ErrorDetail(BaseModel):
code: str
message: str
details: Any = None
class ErrorResponse(BaseModel):
error: ErrorDetail

46
app/schemas/directory.py

@ -0,0 +1,46 @@
from __future__ import annotations
from datetime import datetime
from pydantic import BaseModel, Field
class DirectorySummary(BaseModel):
id: str
name: str
parent_id: str | None
created_at: datetime
updated_at: datetime
class DirectoryItem(BaseModel):
kind: str = Field(default="directory")
id: str
name: str
mime_type: str | None = None
size_bytes: int | None = None
created_at: datetime
updated_at: datetime
class DirectoryChildrenResponse(BaseModel):
directory: DirectorySummary
items: list[DirectoryItem]
next_cursor: str | None = None
class CreateDirectoryRequest(BaseModel):
parent_id: str
name: str
class CreateDirectoryResponse(BaseModel):
directory: DirectorySummary
class RenameDirectoryRequest(BaseModel):
name: str
class MoveDirectoryRequest(BaseModel):
target_parent_id: str

32
app/schemas/export.py

@ -0,0 +1,32 @@
from __future__ import annotations
from pydantic import BaseModel
class MetadataExportResponse(BaseModel):
exported_at: str
data: dict
class MetadataImportRequest(BaseModel):
data: dict
confirm_replace: bool = False
validation_token: str | None = None
class MetadataImportResponse(BaseModel):
ok: bool = True
imported_tables: list[str]
scheduled_jobs: list[str] = []
class MetadataValidationResponse(BaseModel):
ok: bool
issues: list[str]
class MetadataRestorePlanResponse(BaseModel):
ok: bool
issues: list[str]
validation_token: str | None
summary: dict

84
app/schemas/file.py

@ -0,0 +1,84 @@
from __future__ import annotations
from datetime import datetime
from pydantic import BaseModel
class FileSummary(BaseModel):
id: str
directory_id: str
name: str
mime_type: str | None
size_bytes: int
current_version_id: str | None
created_at: datetime
updated_at: datetime
class PreviewArtifactSummary(BaseModel):
id: str
artifact_type: str
blob_id: str
status: str
created_at: datetime
updated_at: datetime
class FileReplicaSummary(BaseModel):
id: str
blob_id: str
backend_id: str
backend_name: str
backend_type: str
backend_stability_class: str
backend_enabled: bool
storage_key: str
status: str
size_bytes: int
checksum: str | None
etag: str | None
last_verified_at: datetime | None
created_at: datetime
updated_at: datetime
class FilePlacementBackendSummary(BaseModel):
id: str
name: str
type: str
stability_class: str
read_priority: int
write_priority: int
is_enabled: bool
class FileDetailResponse(BaseModel):
file: FileSummary
preview_artifacts: list[PreviewArtifactSummary] = []
replicas: list[FileReplicaSummary] = []
placement_file_class: str | None = None
desired_backends: list[FilePlacementBackendSummary] = []
ready_replica_backend_ids: list[str] = []
missing_backend_ids: list[str] = []
class DeletedFileSummary(FileSummary):
deleted_at: datetime
class RecycleBinResponse(BaseModel):
items: list[DeletedFileSummary]
class FileMutationResponse(BaseModel):
ok: bool = True
file: FileSummary | DeletedFileSummary
class RenameFileRequest(BaseModel):
name: str
class MoveFileRequest(BaseModel):
target_parent_id: str

37
app/schemas/job.py

@ -0,0 +1,37 @@
from __future__ import annotations
from datetime import datetime
from pydantic import BaseModel
class JobSummary(BaseModel):
id: str
kind: str
status: str
attempt_count: int
max_attempts: int
run_after: datetime
last_error: str | None
created_at: datetime
updated_at: datetime
class JobListResponse(BaseModel):
items: list[JobSummary]
class JobDetailResponse(BaseModel):
job: JobSummary
class JobMutationResponse(BaseModel):
ok: bool = True
job: JobSummary
class JobRunPendingResponse(BaseModel):
ok: bool = True
processed: int
completed: int
failed: int

56
app/schemas/policy.py

@ -0,0 +1,56 @@
from __future__ import annotations
from datetime import datetime
from pydantic import BaseModel
from pydantic import Field
class PlacementPolicySummary(BaseModel):
id: str
file_class: str
require_local: bool
stable_replica_count: int
opportunistic_replica_count: int
preferred_backend_ids: list[str]
excluded_backend_ids: list[str]
max_non_local_size_bytes: int | None
created_at: datetime
updated_at: datetime
class PlacementPolicyListResponse(BaseModel):
items: list[PlacementPolicySummary]
class UpsertPlacementPolicyRequest(BaseModel):
require_local: bool
stable_replica_count: int
opportunistic_replica_count: int
preferred_backend_ids: list[str] = Field(default_factory=list)
excluded_backend_ids: list[str] = Field(default_factory=list)
max_non_local_size_bytes: int | None = None
class PlacementPolicyMutationResponse(BaseModel):
ok: bool = True
policy: PlacementPolicySummary
reconcile_enqueued_jobs: int = 0
class PlacementDecisionBackendSummary(BaseModel):
id: str
name: str
type: str
stability_class: str
read_priority: int
write_priority: int
class PlacementDecisionResponse(BaseModel):
file_class: str
mime_type: str | None
size_bytes: int | None
non_local_allowed: bool
policy: PlacementPolicySummary
selected_backends: list[PlacementDecisionBackendSummary]

15
app/schemas/repair.py

@ -0,0 +1,15 @@
from __future__ import annotations
from pydantic import BaseModel
class RepairFileResponse(BaseModel):
ok: bool = True
file_id: str
enqueued_jobs: int
reconciliation_job_kind: str = "reconcile_file_replicas"
class EnqueueHealthChecksResponse(BaseModel):
ok: bool = True
enqueued_jobs: int

11
app/schemas/system.py

@ -0,0 +1,11 @@
from pydantic import BaseModel
class HealthResponse(BaseModel):
status: str
class ReadyResponse(BaseModel):
status: str
database: str

20
app/schemas/upload.py

@ -0,0 +1,20 @@
from pydantic import BaseModel
from app.schemas.file import FileSummary
class FinalizeUploadRequest(BaseModel):
directory_id: str
filename: str
class UploadSummary(BaseModel):
id: str
status: str
received_size_bytes: int
total_size_bytes: int
class FinalizeUploadResponse(BaseModel):
upload: UploadSummary
file: FileSummary

2
app/services/__init__.py

@ -0,0 +1,2 @@
"""Application services."""

106
app/services/auth_service.py

@ -0,0 +1,106 @@
from __future__ import annotations
from dataclasses import dataclass
from datetime import UTC, datetime, timedelta
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.config import get_settings
from app.core.exceptions import AuthenticationFailedError
from app.core.exceptions import AuthSessionExpiredError
from app.core.exceptions import UnauthorizedError
from app.core.ids import new_id
from app.core.security import generate_session_token
from app.core.security import hash_password
from app.core.security import hash_session_token
from app.core.security import verify_password
from app.models.entities import AuthSession
from app.models.entities import User
from app.repositories.auth_session_repository import AuthSessionRepository
from app.repositories.user_repository import UserRepository
@dataclass
class LoginResult:
user: User
access_token: str
expires_at: datetime
class AuthService:
def __init__(self, session: AsyncSession) -> None:
self.session = session
self.user_repository = UserRepository(session)
self.auth_session_repository = AuthSessionRepository(session)
self.settings = get_settings()
async def ensure_bootstrap_user(self) -> User | None:
if not self.settings.bootstrap_username or not self.settings.bootstrap_password:
return None
if await self.user_repository.count_users() > 0:
return await self.user_repository.get_by_username(self.settings.bootstrap_username)
user = User(
id=new_id("user"),
username=self.settings.bootstrap_username,
password_hash=hash_password(self.settings.bootstrap_password),
)
return await self.user_repository.create(user)
async def login(self, username: str, password: str) -> LoginResult:
user = await self.user_repository.get_by_username(username)
if user is None or not verify_password(password, user.password_hash):
raise AuthenticationFailedError()
expires_at = self._utc_now_naive() + timedelta(hours=self.settings.auth_session_ttl_hours)
access_token = generate_session_token()
auth_session = AuthSession(
id=new_id("sess"),
user_id=user.id,
token_hash=hash_session_token(access_token),
expires_at=expires_at,
revoked_at=None,
)
await self.auth_session_repository.create(auth_session)
return LoginResult(user=user, access_token=access_token, expires_at=expires_at)
async def get_current_user(self, bearer_token: str | None) -> User:
token = self._extract_bearer_token(bearer_token)
now = self._utc_now_naive()
auth_session = await self.auth_session_repository.get_active_by_token_hash(
hash_session_token(token),
now,
)
if auth_session is None:
existing_session = await self.auth_session_repository.get_by_token_hash(hash_session_token(token))
if existing_session is not None and existing_session.expires_at <= now:
raise AuthSessionExpiredError()
raise UnauthorizedError()
user = await self.user_repository.get_by_id(auth_session.user_id)
if user is None:
raise UnauthorizedError()
return user
async def logout(self, bearer_token: str | None) -> None:
token = self._extract_bearer_token(bearer_token)
auth_session = await self.auth_session_repository.get_by_token_hash(hash_session_token(token))
if auth_session is None or auth_session.revoked_at is not None:
raise UnauthorizedError()
auth_session.revoked_at = self._utc_now_naive()
await self.auth_session_repository.save(auth_session)
@staticmethod
def _extract_bearer_token(bearer_token: str | None) -> str:
if bearer_token is None:
raise UnauthorizedError()
scheme, _, token = bearer_token.partition(" ")
if scheme.lower() != "bearer" or not token:
raise UnauthorizedError()
return token
@staticmethod
def _utc_now_naive() -> datetime:
return datetime.now(UTC).replace(tzinfo=None)

212
app/services/backend_service.py

@ -0,0 +1,212 @@
from __future__ import annotations
import json
from datetime import UTC, datetime
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.backend_config import (
SECRET_CONFIG_KEYS,
BackendConfigError,
merge_backend_config,
split_backend_config,
supported_backend_types,
validate_backend_config,
)
from app.core.exceptions import ConflictError, NotFoundError
from app.core.ids import new_id
from app.core.secret_codec import decrypt_secret_config, encrypt_secret_config
from app.models.entities import Backend
from app.repositories.backend_repository import BackendRepository
from app.repositories.blob_replica_repository import BlobReplicaRepository
from app.services.local_storage_service import DEFAULT_LOCAL_BACKEND_ID
from app.services.storage_service import StorageService
class BackendNotFoundError(NotFoundError):
code = "backend_not_found"
message = "Backend does not exist."
class BackendDeleteConflictError(ConflictError):
code = "backend_delete_conflict"
message = "Backend cannot be deleted."
class BackendDisableConflictError(ConflictError):
code = "backend_disable_conflict"
message = "Backend cannot be disabled."
SUPPORTED_BACKEND_TYPES = supported_backend_types()
SUPPORTED_STABILITY_CLASSES = {"local", "stable", "opportunistic"}
class BackendService:
def __init__(self, session: AsyncSession) -> None:
self.session = session
self.repository = BackendRepository(session)
self.blob_replica_repository = BlobReplicaRepository(session)
self.storage_service = StorageService(session)
async def list_backends(self) -> list[Backend]:
return await self.repository.list_all()
async def create_backend(
self,
*,
name: str,
backend_type: str,
stability_class: str,
read_priority: int,
write_priority: int,
config: dict,
) -> Backend:
cleaned_name = name.strip()
if not cleaned_name:
raise BackendConfigError("Backend name must not be empty.")
if backend_type not in SUPPORTED_BACKEND_TYPES:
raise BackendConfigError(f"Unsupported backend type: {backend_type}.")
if stability_class not in SUPPORTED_STABILITY_CLASSES:
raise BackendConfigError(f"Unsupported stability class: {stability_class}.")
if await self.repository.get_by_name(cleaned_name) is not None:
raise ConflictError("A backend with this name already exists.")
public_config, secret_config = self._validate_backend_config(backend_type, config)
backend = Backend(
id=new_id("bkd"),
name=cleaned_name,
type=backend_type,
stability_class=stability_class,
read_priority=read_priority,
write_priority=write_priority,
is_enabled=True,
public_config_json=json.dumps(public_config),
secret_config_json=encrypt_secret_config(secret_config),
)
created = await self.repository.create(backend)
await self.session.commit()
return created
async def check_backend(self, backend_id: str) -> tuple[Backend, str, str | None]:
backend = await self._get_backend(backend_id)
try:
status, detail = await self.storage_service.check_backend(backend)
except Exception as exc:
backend.last_health_status = "unhealthy"
backend.last_health_checked_at = datetime.now(UTC)
await self.repository.save(backend)
await self.session.commit()
return backend, "unhealthy", str(exc)
backend.last_health_status = status
backend.last_health_checked_at = datetime.now(UTC)
await self.repository.save(backend)
await self.session.commit()
return backend, status, detail
async def disable_backend(self, backend_id: str) -> Backend:
backend = await self._get_backend(backend_id)
if backend.type == "local_directory":
enabled_local_backends = await self.repository.list_enabled_by_type("local_directory")
if backend.is_enabled and len(enabled_local_backends) <= 1:
raise BackendDisableConflictError("At least one enabled local_directory backend is required.")
backend.is_enabled = False
saved = await self.repository.save(backend)
await self.session.commit()
return saved
async def enable_backend(self, backend_id: str) -> Backend:
backend = await self._get_backend(backend_id)
backend.is_enabled = True
saved = await self.repository.save(backend)
await self.session.commit()
return saved
async def delete_backend(self, backend_id: str) -> None:
backend = await self._get_backend(backend_id)
if backend.id == DEFAULT_LOCAL_BACKEND_ID:
raise BackendDeleteConflictError("The default local backend cannot be deleted.")
if backend.is_enabled:
raise BackendDeleteConflictError("Disable the backend before deleting it.")
if await self.repository.is_referenced_by_policy(backend.id):
raise BackendDeleteConflictError("Backend is still referenced by a placement policy.")
replica_count = await self.blob_replica_repository.count_by_backend(backend.id)
if replica_count > 0:
raise BackendDeleteConflictError("Backend still has stored blob replicas.")
await self.repository.delete(backend)
await self.session.commit()
async def update_backend(
self,
backend_id: str,
*,
name: str | None = None,
stability_class: str | None = None,
read_priority: int | None = None,
write_priority: int | None = None,
config: dict | None = None,
) -> Backend:
backend = await self._get_backend(backend_id)
if name is not None:
cleaned_name = name.strip()
if not cleaned_name:
raise BackendConfigError("Backend name must not be empty.")
existing = await self.repository.get_by_name(cleaned_name)
if existing is not None and existing.id != backend.id:
raise ConflictError("A backend with this name already exists.")
backend.name = cleaned_name
if stability_class is not None:
if stability_class not in SUPPORTED_STABILITY_CLASSES:
raise BackendConfigError(f"Unsupported stability class: {stability_class}.")
backend.stability_class = stability_class
if read_priority is not None:
backend.read_priority = read_priority
if write_priority is not None:
backend.write_priority = write_priority
if config is not None:
public_config = json.loads(backend.public_config_json)
secret_config = decrypt_secret_config(backend.secret_config_json)
next_config = self._merge_config_update(
backend.type,
public_config=public_config,
secret_config=secret_config,
config_patch=config,
)
next_public_config, next_secret_config = self._validate_backend_config(backend.type, next_config)
backend.public_config_json = json.dumps(next_public_config)
backend.secret_config_json = encrypt_secret_config(next_secret_config)
saved = await self.repository.save(backend)
await self.session.commit()
return saved
async def _get_backend(self, backend_id: str) -> Backend:
backend = await self.repository.get_by_id(backend_id)
if backend is None:
raise BackendNotFoundError()
return backend
@staticmethod
def _validate_backend_config(backend_type: str, config: dict) -> tuple[dict, dict]:
return split_backend_config(backend_type, config)
@staticmethod
def _merge_config_update(
backend_type: str,
*,
public_config: dict,
secret_config: dict,
config_patch: dict,
) -> dict:
merged = merge_backend_config(public_config, secret_config)
normalized_patch = dict(config_patch)
for key in list(normalized_patch):
if key in SECRET_CONFIG_KEYS and not normalized_patch[key]:
normalized_patch.pop(key)
merged.update(normalized_patch)
return validate_backend_config(backend_type, merged)

153
app/services/directory_service.py

@ -0,0 +1,153 @@
from __future__ import annotations
from dataclasses import dataclass
from typing import Literal
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.exceptions import DirectoryConflictError, DirectoryNotFoundError, ValidationError
from app.core.ids import new_id
from app.models.entities import Directory, FileEntry
from app.repositories.directory_repository import DirectoryRepository
from app.repositories.file_repository import FileRepository
ROOT_DIRECTORY_ID = "dir_root"
ROOT_DIRECTORY_NAME = "/"
ROOT_PATH_KEY = "/"
@dataclass
class NamespaceItem:
kind: Literal["directory", "file"]
item: Directory | FileEntry
@dataclass
class DirectoryChildrenResult:
directory: Directory
children: list[NamespaceItem]
class DirectoryService:
def __init__(self, session: AsyncSession) -> None:
self.session = session
self.repository = DirectoryRepository(session)
self.file_repository = FileRepository(session)
async def ensure_root_directory(self) -> Directory:
root = await self.repository.get_root()
if root is not None:
return root
root = Directory(
id=ROOT_DIRECTORY_ID,
parent_id=None,
name=ROOT_DIRECTORY_NAME,
path_key=ROOT_PATH_KEY,
)
await self.repository.create(root)
await self.session.commit()
return root
async def list_children(self, directory_id: str) -> DirectoryChildrenResult:
directory = await self._get_directory(directory_id)
child_directories = await self.repository.list_children(directory.id)
child_files = await self.file_repository.list_by_directory(directory.id)
items = [NamespaceItem(kind="directory", item=child) for child in child_directories]
items.extend(NamespaceItem(kind="file", item=file_entry) for file_entry in child_files)
items.sort(key=lambda item: (item.kind != "directory", item.item.name.lower()))
return DirectoryChildrenResult(directory=directory, children=items)
async def create_directory(self, parent_id: str, name: str) -> Directory:
cleaned_name = self._validate_directory_name(name)
parent = await self._get_directory(parent_id)
await self._ensure_name_available(parent.id, cleaned_name)
directory = Directory(
id=new_id("dir"),
parent_id=parent.id,
name=cleaned_name,
path_key=self._build_path_key(parent.path_key, cleaned_name),
)
await self.repository.create(directory)
await self.session.commit()
return directory
async def rename_directory(self, directory_id: str, name: str) -> Directory:
directory = await self._get_directory(directory_id)
if directory.id == ROOT_DIRECTORY_ID:
raise ValidationError("Root directory cannot be renamed.")
cleaned_name = self._validate_directory_name(name)
parent = await self._get_directory(directory.parent_id or ROOT_DIRECTORY_ID)
if directory.name == cleaned_name:
return directory
await self._ensure_name_available(parent.id, cleaned_name)
old_path_key = directory.path_key
directory.name = cleaned_name
directory.path_key = self._build_path_key(parent.path_key, cleaned_name)
await self._rewrite_descendant_paths(old_path_key, directory.path_key)
saved = await self.repository.save(directory)
await self.session.commit()
return saved
async def move_directory(self, directory_id: str, target_parent_id: str) -> Directory:
directory = await self._get_directory(directory_id)
if directory.id == ROOT_DIRECTORY_ID:
raise ValidationError("Root directory cannot be moved.")
target_parent = await self._get_directory(target_parent_id)
if directory.parent_id == target_parent.id:
return directory
if target_parent.path_key == directory.path_key or target_parent.path_key.startswith(f"{directory.path_key}/"):
raise ValidationError("A directory cannot be moved into itself or its descendants.")
await self._ensure_name_available(target_parent.id, directory.name)
old_path_key = directory.path_key
directory.parent_id = target_parent.id
directory.path_key = self._build_path_key(target_parent.path_key, directory.name)
await self._rewrite_descendant_paths(old_path_key, directory.path_key)
saved = await self.repository.save(directory)
await self.session.commit()
return saved
async def _get_directory(self, directory_id: str) -> Directory:
directory = await self.repository.get_by_id(directory_id)
if directory is None or directory.deleted_at is not None:
raise DirectoryNotFoundError()
return directory
async def _ensure_name_available(self, parent_id: str, name: str) -> None:
conflicting_file = await self.file_repository.get_by_directory_and_name(parent_id, name)
if conflicting_file is not None:
raise DirectoryConflictError("A file with this name already exists in the target location.")
existing = await self.repository.get_by_parent_and_name(parent_id, name)
if existing is not None:
raise DirectoryConflictError("A directory with this name already exists in the target location.")
async def _rewrite_descendant_paths(self, old_prefix: str, new_prefix: str) -> None:
descendants = await self.repository.list_descendants_by_path_prefix(old_prefix)
for descendant in descendants:
suffix = descendant.path_key.removeprefix(old_prefix)
descendant.path_key = f"{new_prefix}{suffix}"
@staticmethod
def _validate_directory_name(name: str) -> str:
cleaned_name = name.strip()
if not cleaned_name:
raise ValidationError("Directory name must not be empty.")
if "/" in cleaned_name:
raise ValidationError("Directory name must not contain '/'.")
return cleaned_name
@staticmethod
def _build_path_key(parent_path_key: str, name: str) -> str:
if parent_path_key == ROOT_PATH_KEY:
return f"/{name}"
return f"{parent_path_key}/{name}"

234
app/services/export_service.py

@ -0,0 +1,234 @@
from __future__ import annotations
from datetime import UTC, datetime
import hashlib
import json
from typing import Any
from sqlalchemy import delete
from sqlalchemy import insert
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.exceptions import ValidationError
from app.models.entities import AuthSession
from app.models.entities import Backend
from app.models.entities import Blob
from app.models.entities import BlobReplica
from app.models.entities import CacheEntry
from app.models.entities import Directory
from app.models.entities import FileEntry
from app.models.entities import FileVersion
from app.models.entities import Job
from app.models.entities import PreviewArtifact
from app.models.entities import UploadSession
from app.models.entities import UploadSessionPart
from app.models.entities import User
from app.services.storage_service import StorageService
class ExportService:
def __init__(self, session: AsyncSession) -> None:
self.session = session
self.storage_service = StorageService(session)
async def export_metadata(self) -> dict:
return {
"users": await self._dump(User),
"auth_sessions": await self._dump(AuthSession),
"directories": await self._dump(Directory),
"file_entries": await self._dump(FileEntry),
"file_versions": await self._dump(FileVersion),
"blobs": await self._dump(Blob),
"blob_replicas": await self._dump(BlobReplica),
"backends": await self._dump(Backend),
"upload_sessions": await self._dump(UploadSession),
"upload_session_parts": await self._dump(UploadSessionPart),
"jobs": await self._dump(Job),
"cache_entries": await self._dump(CacheEntry),
"preview_artifacts": await self._dump(PreviewArtifact),
"exported_at": datetime.now(UTC).isoformat(),
}
async def import_metadata(self, data: dict[str, list[dict[str, Any]]]) -> list[str]:
validation = await self.validate_metadata_snapshot(data)
if not validation["ok"]:
raise ValidationError("Metadata snapshot is invalid and cannot be imported.")
ordered_models = [
User,
AuthSession,
Directory,
Backend,
UploadSession,
UploadSessionPart,
FileEntry,
FileVersion,
Blob,
BlobReplica,
PreviewArtifact,
Job,
CacheEntry,
]
for model in reversed(ordered_models):
await self.session.execute(delete(model))
imported_tables: list[str] = []
for model in ordered_models:
rows = data.get(model.__tablename__, [])
if not rows:
continue
normalized_rows = [self._normalize_row(model, row) for row in rows]
await self.session.execute(insert(model), normalized_rows)
imported_tables.append(model.__tablename__)
await self.session.commit()
return imported_tables
async def build_restore_plan(self, data: dict[str, list[dict[str, Any]]]) -> dict[str, Any]:
validation = await self.validate_metadata_snapshot(data)
summary = {
"users": len(data.get("users", [])),
"directories": len(data.get("directories", [])),
"file_entries": len(data.get("file_entries", [])),
"file_versions": len(data.get("file_versions", [])),
"blobs": len(data.get("blobs", [])),
"blob_replicas": len(data.get("blob_replicas", [])),
"backends": len(data.get("backends", [])),
"jobs": len(data.get("jobs", [])),
"preview_artifacts": len(data.get("preview_artifacts", [])),
}
validation_token = None
if validation["ok"]:
validation_token = self._build_validation_token(data)
return {
"ok": validation["ok"],
"issues": validation["issues"],
"validation_token": validation_token,
"summary": summary,
}
async def validate_metadata_snapshot(self, data: dict[str, list[dict[str, Any]]]) -> dict[str, Any]:
issues: list[str] = []
users = data.get("users", [])
directories = data.get("directories", [])
file_entries = data.get("file_entries", [])
file_versions = data.get("file_versions", [])
blobs = data.get("blobs", [])
blob_replicas = data.get("blob_replicas", [])
backends = data.get("backends", [])
auth_sessions = data.get("auth_sessions", [])
upload_sessions = data.get("upload_sessions", [])
upload_session_parts = data.get("upload_session_parts", [])
jobs = data.get("jobs", [])
cache_entries = data.get("cache_entries", [])
preview_artifacts = data.get("preview_artifacts", [])
if not users:
issues.append("Snapshot must contain at least one user.")
if not any(directory.get("id") == "dir_root" for directory in directories):
issues.append("Snapshot must contain the root directory dir_root.")
user_ids = {row["id"] for row in users if row.get("id")}
directory_ids = {row["id"] for row in directories if row.get("id")}
file_entry_ids = {row["id"] for row in file_entries if row.get("id")}
version_ids = {row["id"] for row in file_versions if row.get("id")}
blob_ids = {row["id"] for row in blobs if row.get("id")}
backend_ids = {row["id"] for row in backends if row.get("id")}
upload_ids = {row["id"] for row in upload_sessions if row.get("id")}
for row in auth_sessions:
if row.get("user_id") not in user_ids:
issues.append(f"Auth session {row.get('id')} references missing user.")
for row in file_entries:
if row.get("directory_id") not in directory_ids:
issues.append(f"File entry {row.get('id')} references missing directory.")
current_version_id = row.get("current_version_id")
if current_version_id is not None and current_version_id not in version_ids:
issues.append(f"File entry {row.get('id')} references missing current version.")
for row in file_versions:
if row.get("file_entry_id") not in file_entry_ids:
issues.append(f"File version {row.get('id')} references missing file entry.")
for row in blobs:
if row.get("file_version_id") not in version_ids:
issues.append(f"Blob {row.get('id')} references missing file version.")
for row in blob_replicas:
if row.get("blob_id") not in blob_ids:
issues.append(f"Blob replica {row.get('id')} references missing blob.")
if row.get("backend_id") not in backend_ids:
issues.append(f"Blob replica {row.get('id')} references missing backend.")
for row in upload_sessions:
directory_id = row.get("directory_id")
if directory_id is not None and directory_id not in directory_ids:
issues.append(f"Upload session {row.get('id')} references missing directory.")
for row in upload_session_parts:
if row.get("upload_session_id") not in upload_ids:
issues.append(f"Upload session part {row.get('id')} references missing upload session.")
for row in preview_artifacts:
if row.get("file_version_id") not in version_ids:
issues.append(f"Preview artifact {row.get('id')} references missing file version.")
if row.get("blob_id") not in blob_ids:
issues.append(f"Preview artifact {row.get('id')} references missing blob.")
for row in cache_entries:
if row.get("blob_id") not in blob_ids:
issues.append(f"Cache entry {row.get('id')} references missing blob.")
return {"ok": not issues, "issues": issues}
async def verify_runtime_integrity(self) -> dict[str, Any]:
snapshot = await self.export_metadata()
validation = await self.validate_metadata_snapshot(snapshot)
issues = list(validation["issues"])
backends = {row["id"]: row for row in snapshot["backends"]}
for replica in snapshot["blob_replicas"]:
if replica.get("status") != "ready":
continue
backend_row = backends.get(replica["backend_id"])
if backend_row is None or not backend_row.get("is_enabled", False):
continue
backend = await self.session.get(Backend, replica["backend_id"])
if backend is None:
continue
try:
adapter = self.storage_service._build_adapter(backend)
exists = await adapter.object_exists(replica["storage_key"])
except Exception as exc:
issues.append(f"Replica {replica['id']} could not be verified: {exc}")
continue
if not exists:
issues.append(f"Replica {replica['id']} is marked ready but object is missing.")
return {"ok": not issues, "issues": issues}
@staticmethod
def _build_validation_token(data: dict[str, list[dict[str, Any]]]) -> str:
canonical = json.dumps(data, sort_keys=True, separators=(",", ":"), ensure_ascii=True)
return hashlib.sha256(canonical.encode("utf-8")).hexdigest()
async def _dump(self, model) -> list[dict]:
result = await self.session.execute(select(model))
items = []
for row in result.scalars().all():
payload = {}
for column in model.__table__.columns:
value = getattr(row, column.name)
if isinstance(value, datetime):
payload[column.name] = value.isoformat()
else:
payload[column.name] = value
items.append(payload)
return items
@staticmethod
def _normalize_row(model, row: dict[str, Any]) -> dict[str, Any]:
normalized = {}
for column in model.__table__.columns:
value = row.get(column.name)
if value is not None and isinstance(value, str):
python_type = getattr(column.type, "python_type", None)
if python_type is datetime:
normalized[column.name] = datetime.fromisoformat(value)
continue
normalized[column.name] = value
return normalized

263
app/services/file_service.py

@ -0,0 +1,263 @@
from __future__ import annotations
from dataclasses import dataclass
from pathlib import Path
from typing import NamedTuple
import anyio
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.exceptions import (
DirectoryConflictError,
DirectoryNotFoundError,
FileConflictError,
FileNotFoundError,
PreviewNotSupportedError,
RangeNotSatisfiableError,
ValidationError,
)
from app.core.ids import new_id
from app.models.entities import Blob, FileEntry, FileVersion
from app.repositories.blob_replica_repository import BlobReplicaRepository
from app.repositories.directory_repository import DirectoryRepository
from app.repositories.file_repository import FileRepository
from app.repositories.preview_artifact_repository import PreviewArtifactRepository
from app.services.local_storage_service import LocalStorageService
from app.services.storage_service import StorageService
@dataclass
class CreateFileMetadataInput:
directory_id: str
name: str
mime_type: str | None
size_bytes: int
content_hash: str
storage_layout: str = "single"
class FileStreamResult(NamedTuple):
file_entry: FileEntry
file_path: Path
file_size: int
start: int
end: int
class FileService:
def __init__(self, session: AsyncSession) -> None:
self.session = session
self.file_repository = FileRepository(session)
self.blob_replica_repository = BlobReplicaRepository(session)
self.directory_repository = DirectoryRepository(session)
self.preview_artifact_repository = PreviewArtifactRepository(session)
self.local_storage_service = LocalStorageService(session)
self.storage_service = StorageService(session)
async def get_file(self, file_id: str) -> FileEntry:
file_entry = await self.file_repository.get_by_id(file_id)
if file_entry is None or file_entry.is_deleted or file_entry.deleted_at is not None:
raise FileNotFoundError()
return file_entry
async def get_file_detail(self, file_id: str) -> tuple[FileEntry, list, list, object | None]:
file_entry = await self.get_file(file_id)
if file_entry.current_version_id is None:
return file_entry, [], [], None
artifacts = await self.preview_artifact_repository.list_by_file_version(file_entry.current_version_id)
blob = await self.file_repository.get_primary_blob_for_file(file_entry.id)
if blob is None:
return file_entry, artifacts, [], None
replicas = await self.blob_replica_repository.list_replicas_with_backends(blob.id)
decision = await self.storage_service.get_replication_targets_for_mime_type(
file_entry.mime_type,
size_bytes=blob.size_bytes,
)
return file_entry, artifacts, replicas, decision
async def list_deleted_files(self) -> list[FileEntry]:
return await self.file_repository.list_deleted()
async def create_file_metadata(self, payload: CreateFileMetadataInput) -> FileEntry:
directory = await self.directory_repository.get_by_id(payload.directory_id)
if directory is None or directory.deleted_at is not None:
raise DirectoryNotFoundError()
cleaned_name = self._validate_file_name(payload.name)
await self._ensure_name_available(directory.id, cleaned_name)
file_entry_id = new_id("fil")
file_version_id = new_id("ver")
blob_id = new_id("blb")
file_entry = FileEntry(
id=file_entry_id,
directory_id=directory.id,
name=cleaned_name,
mime_type=payload.mime_type,
size_bytes=payload.size_bytes,
current_version_id=file_version_id,
is_deleted=False,
)
file_version = FileVersion(
id=file_version_id,
file_entry_id=file_entry_id,
content_hash=payload.content_hash,
size_bytes=payload.size_bytes,
storage_layout=payload.storage_layout,
)
blob = Blob(
id=blob_id,
file_version_id=file_version_id,
blob_index=0,
kind="file",
content_hash=payload.content_hash,
size_bytes=payload.size_bytes,
logical_offset=0,
)
created = await self.file_repository.create_file_graph(file_entry, file_version, blob)
await self.session.commit()
return created
async def rename_file(self, file_id: str, name: str) -> FileEntry:
file_entry = await self.get_file(file_id)
cleaned_name = self._validate_file_name(name)
if file_entry.name == cleaned_name:
return file_entry
await self._ensure_name_available(file_entry.directory_id, cleaned_name)
file_entry.name = cleaned_name
saved = await self.file_repository.save(file_entry)
await self.session.commit()
return saved
async def move_file(self, file_id: str, target_parent_id: str) -> FileEntry:
file_entry = await self.get_file(file_id)
target_parent = await self.directory_repository.get_by_id(target_parent_id)
if target_parent is None or target_parent.deleted_at is not None:
raise DirectoryNotFoundError()
if file_entry.directory_id == target_parent.id:
return file_entry
await self._ensure_name_available(target_parent.id, file_entry.name)
file_entry.directory_id = target_parent.id
saved = await self.file_repository.save(file_entry)
await self.session.commit()
return saved
async def delete_file(self, file_id: str) -> FileEntry:
file_entry = await self.get_file(file_id)
deleted = await self.file_repository.soft_delete(file_entry)
await self.session.commit()
return deleted
async def restore_file(self, file_id: str) -> FileEntry:
file_entry = await self.file_repository.get_by_id(file_id)
if file_entry is None or not file_entry.is_deleted or file_entry.deleted_at is None:
raise FileNotFoundError()
directory = await self.directory_repository.get_by_id(file_entry.directory_id)
if directory is None or directory.deleted_at is not None:
raise DirectoryNotFoundError()
conflicting_directory = await self.directory_repository.get_by_parent_and_name(directory.id, file_entry.name)
if conflicting_directory is not None:
raise DirectoryConflictError("A directory with this name already exists in the target location.")
existing_file = await self.file_repository.get_by_directory_and_name(directory.id, file_entry.name)
if existing_file is not None:
raise FileConflictError("A file with this name already exists in the target location.")
restored = await self.file_repository.restore(file_entry)
await self.session.commit()
return restored
async def resolve_download(self, file_id: str) -> tuple[FileEntry, Path]:
file_entry = await self.get_file(file_id)
blob = await self.file_repository.get_primary_blob_for_file(file_entry.id)
if blob is None:
raise FileNotFoundError()
try:
file_path = await self.storage_service.resolve_best_local_path(blob)
except Exception as exc:
raise FileNotFoundError() from exc
if file_path is None:
raise FileNotFoundError()
return file_entry, file_path
async def resolve_preview(self, file_id: str) -> tuple[FileEntry, Path]:
file_entry, file_path = await self.resolve_download(file_id)
if not self._supports_inline_preview(file_entry.mime_type):
raise PreviewNotSupportedError()
return file_entry, file_path
async def resolve_stream(self, file_id: str, range_header: str | None) -> FileStreamResult:
file_entry, file_path = await self.resolve_download(file_id)
file_size = await anyio.to_thread.run_sync(lambda: file_path.stat().st_size)
start, end = self._parse_range(range_header, file_size)
return FileStreamResult(
file_entry=file_entry,
file_path=file_path,
file_size=file_size,
start=start,
end=end,
)
async def _ensure_name_available(self, directory_id: str, name: str) -> None:
conflicting_directory = await self.directory_repository.get_by_parent_and_name(directory_id, name)
if conflicting_directory is not None:
raise DirectoryConflictError("A directory with this name already exists in the target location.")
existing_file = await self.file_repository.get_by_directory_and_name(directory_id, name)
if existing_file is not None:
raise FileConflictError("A file with this name already exists in the target location.")
@staticmethod
def _validate_file_name(name: str) -> str:
cleaned_name = name.strip()
if not cleaned_name:
raise ValidationError("File name must not be empty.")
if "/" in cleaned_name:
raise ValidationError("File name must not contain '/'.")
return cleaned_name
@staticmethod
def _parse_range(range_header: str | None, file_size: int) -> tuple[int, int]:
if file_size <= 0:
return 0, 0
if not range_header:
return 0, file_size - 1
if not range_header.startswith("bytes="):
raise RangeNotSatisfiableError()
value = range_header.removeprefix("bytes=")
if "," in value:
raise RangeNotSatisfiableError()
start_text, _, end_text = value.partition("-")
try:
if start_text == "":
suffix_length = int(end_text)
if suffix_length <= 0:
raise RangeNotSatisfiableError()
start = max(file_size - suffix_length, 0)
end = file_size - 1
else:
start = int(start_text)
end = file_size - 1 if end_text == "" else int(end_text)
except ValueError as exc:
raise RangeNotSatisfiableError() from exc
if start < 0 or end < start or start >= file_size:
raise RangeNotSatisfiableError()
end = min(end, file_size - 1)
return start, end
@staticmethod
def _supports_inline_preview(mime_type: str | None) -> bool:
if mime_type is None:
return False
return mime_type.startswith("image/") or mime_type == "application/pdf"

254
app/services/job_service.py

@ -0,0 +1,254 @@
from __future__ import annotations
import json
from dataclasses import dataclass
from datetime import UTC, datetime, timedelta
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.config import get_settings
from app.core.exceptions import NotFoundError
from app.core.ids import new_id
from app.models.entities import Job
from app.repositories.backend_repository import BackendRepository
from app.repositories.blob_replica_repository import BlobReplicaRepository
from app.repositories.file_repository import FileRepository
from app.repositories.job_repository import JobRepository
from app.services.policy_service import PolicyService
from app.services.preview_service import PreviewService
from app.services.storage_service import StorageService
class JobNotFoundError(NotFoundError):
code = "job_not_found"
message = "Job does not exist."
@dataclass
class JobRunResult:
processed: int
completed: int
failed: int
class JobService:
def __init__(self, session: AsyncSession) -> None:
self.session = session
self.settings = get_settings()
self.repository = JobRepository(session)
self.file_repository = FileRepository(session)
self.backend_repository = BackendRepository(session)
self.blob_replica_repository = BlobReplicaRepository(session)
self.storage_service = StorageService(session)
self.policy_service = PolicyService(session)
self.preview_service = PreviewService(session)
async def list_jobs(self) -> list[Job]:
return await self.repository.list_all()
async def get_job(self, job_id: str) -> Job:
job = await self.repository.get_by_id(job_id)
if job is None:
raise JobNotFoundError()
return job
async def enqueue_blob_replication(self, blob_id: str, backend_id: str) -> Job:
payload_json = json.dumps({"blob_id": blob_id, "backend_id": backend_id}, sort_keys=True)
existing = await self.repository.find_active_by_kind_and_payload("replicate_blob", payload_json)
if existing is not None:
return existing
job = Job(
id=new_id("job"),
kind="replicate_blob",
status="queued",
payload_json=payload_json,
run_after=datetime.now(UTC),
last_error=None,
)
created = await self.repository.create(job)
await self.session.commit()
return created
async def enqueue_file_reconcile(self, file_id: str) -> Job:
payload_json = json.dumps({"file_id": file_id}, sort_keys=True)
existing = await self.repository.find_active_by_kind_and_payload("reconcile_file_replicas", payload_json)
if existing is not None:
return existing
job = Job(
id=new_id("job"),
kind="reconcile_file_replicas",
status="queued",
payload_json=payload_json,
run_after=datetime.now(UTC),
last_error=None,
)
created = await self.repository.create(job)
await self.session.commit()
return created
async def enqueue_full_reconcile(self) -> Job:
payload_json = json.dumps({}, sort_keys=True)
existing = await self.repository.find_active_by_kind_and_payload("reconcile_all_files", payload_json)
if existing is not None:
return existing
job = Job(
id=new_id("job"),
kind="reconcile_all_files",
status="queued",
payload_json=payload_json,
run_after=datetime.now(UTC),
last_error=None,
)
created = await self.repository.create(job)
await self.session.commit()
return created
async def enqueue_backend_health_check(self, backend_id: str) -> Job:
payload_json = json.dumps({"backend_id": backend_id}, sort_keys=True)
existing = await self.repository.find_active_by_kind_and_payload("check_backend_health", payload_json)
if existing is not None:
return existing
job = Job(
id=new_id("job"),
kind="check_backend_health",
status="queued",
payload_json=payload_json,
run_after=datetime.now(UTC),
last_error=None,
)
created = await self.repository.create(job)
await self.session.commit()
return created
async def enqueue_preview_generation(self, file_id: str) -> Job:
payload_json = json.dumps({"file_id": file_id}, sort_keys=True)
existing = await self.repository.find_active_by_kind_and_payload("generate_preview_artifacts", payload_json)
if existing is not None:
return existing
job = Job(
id=new_id("job"),
kind="generate_preview_artifacts",
status="queued",
payload_json=payload_json,
run_after=datetime.now(UTC),
last_error=None,
)
created = await self.repository.create(job)
await self.session.commit()
return created
async def retry_job(self, job_id: str) -> Job:
job = await self.get_job(job_id)
job.status = "retryable"
job.run_after = datetime.now(UTC)
job.last_error = None
saved = await self.repository.save(job)
await self.session.commit()
return saved
async def run_pending_jobs(self, limit: int | None = None) -> JobRunResult:
now = datetime.now(UTC)
runnable = await self.repository.list_runnable(now, limit or self.settings.job_batch_size)
completed = 0
failed = 0
for job in runnable:
job.status = "running"
await self.repository.save(job)
await self.session.commit()
try:
await self._execute(job)
job.status = "completed"
job.last_error = None
completed += 1
except Exception as exc:
job.attempt_count += 1
job.last_error = str(exc)
if job.attempt_count >= job.max_attempts:
job.status = "failed"
else:
job.status = "retryable"
job.run_after = datetime.now(UTC) + timedelta(seconds=min(60, 2**job.attempt_count))
failed += 1
await self.repository.save(job)
await self.session.commit()
return JobRunResult(processed=len(runnable), completed=completed, failed=failed)
async def _execute(self, job: Job) -> None:
payload = json.loads(job.payload_json)
if job.kind == "reconcile_all_files":
files = await self.file_repository.list_active()
for file_entry in files:
await self.enqueue_file_reconcile(file_entry.id)
return
if job.kind == "reconcile_file_replicas":
file_id = payload["file_id"]
file_entry = await self.file_repository.get_by_id(file_id)
if file_entry is None or file_entry.is_deleted or file_entry.deleted_at is not None:
raise ValueError("File is unavailable.")
blob = await self.file_repository.get_primary_blob_for_file(file_id)
if blob is None:
raise ValueError("Primary blob does not exist.")
decision = await self.policy_service.get_placement_decision(file_entry.mime_type, size_bytes=blob.size_bytes)
desired_backends = decision.backends
existing_replicas = await self.blob_replica_repository.list_by_blob(blob.id)
ready_backend_ids = {replica.backend_id for replica in existing_replicas if replica.status == "ready"}
missing_backends = [backend for backend in desired_backends if backend.id not in ready_backend_ids]
if not missing_backends:
return
source_path = await self.storage_service.resolve_best_local_path(blob)
failed_targets: list[str] = []
for backend in missing_backends:
try:
status, _detail = await self.storage_service.check_backend(backend)
if status != "healthy":
failed_targets.append(backend.name)
continue
await self.storage_service.persist_blob_to_backend(blob, backend, str(source_path))
except Exception:
failed_targets.append(backend.name)
if failed_targets:
raise ValueError(
"Reconciliation is still waiting on backends: " + ", ".join(sorted(set(failed_targets)))
)
return
if job.kind == "replicate_blob":
blob_id = payload["blob_id"]
backend_id = payload["backend_id"]
blob = await self.file_repository.get_blob_by_id(blob_id)
if blob is None:
raise ValueError("Blob does not exist.")
backend = await self.backend_repository.get_by_id(backend_id)
if backend is None or not backend.is_enabled:
raise ValueError("Backend is unavailable.")
source_path = await self.storage_service.resolve_best_local_path(blob)
await self.storage_service.persist_blob_to_backend(blob, backend, str(source_path))
return
if job.kind == "check_backend_health":
backend_id = payload["backend_id"]
backend = await self.backend_repository.get_by_id(backend_id)
if backend is None or not backend.is_enabled:
raise ValueError("Backend is unavailable.")
status, _detail = await self.storage_service.check_backend(backend)
backend.last_health_status = status
backend.last_health_checked_at = datetime.now(UTC)
await self.backend_repository.save(backend)
await self.session.commit()
return
if job.kind == "generate_preview_artifacts":
file_id = payload["file_id"]
await self.preview_service.generate_preview_artifacts_for_file(file_id)
return
raise ValueError(f"Unsupported job kind: {job.kind}")

82
app/services/local_storage_service.py

@ -0,0 +1,82 @@
from __future__ import annotations
import json
from pathlib import Path
import anyio
from sqlalchemy.ext.asyncio import AsyncSession
from app.adapters.storage.local import LocalStorageAdapter
from app.core.config import get_settings
from app.core.ids import new_id
from app.core.secret_codec import encrypt_secret_config
from app.core.storage_layout import build_blob_storage_key
from app.models.entities import Backend, Blob, BlobReplica
from app.repositories.backend_repository import BackendRepository
from app.repositories.blob_replica_repository import BlobReplicaRepository
DEFAULT_LOCAL_BACKEND_ID = "bkd_local_default"
DEFAULT_LOCAL_BACKEND_NAME = "local-default"
class LocalStorageService:
def __init__(self, session: AsyncSession) -> None:
self.session = session
self.settings = get_settings()
self.backend_repository = BackendRepository(session)
self.blob_replica_repository = BlobReplicaRepository(session)
self.adapter = LocalStorageAdapter(self.settings.local_storage_dir)
async def ensure_backend_ready(self) -> Backend:
await self.adapter.ensure_base_path()
backend = await self.backend_repository.get_by_name(DEFAULT_LOCAL_BACKEND_NAME)
if backend is not None:
return backend
backend = Backend(
id=DEFAULT_LOCAL_BACKEND_ID,
name=DEFAULT_LOCAL_BACKEND_NAME,
type="local_directory",
stability_class="local",
read_priority=100,
write_priority=100,
is_enabled=True,
public_config_json=json.dumps({"base_path": self.settings.local_storage_dir}),
secret_config_json=encrypt_secret_config({}),
)
await self.backend_repository.create(backend)
await self.session.commit()
return backend
async def persist_blob_from_temp(self, blob: Blob, temp_path: str) -> BlobReplica:
backend = await self.ensure_backend_ready()
existing = await self.blob_replica_repository.get_by_blob_and_backend(blob.id, backend.id)
if existing is not None:
return existing
storage_key = self._build_storage_key(blob)
await self.adapter.put_file(temp_path, storage_key)
replica = BlobReplica(
id=new_id("rep"),
blob_id=blob.id,
backend_id=backend.id,
storage_key=storage_key,
status="ready",
checksum=blob.content_hash,
size_bytes=blob.size_bytes,
)
created = await self.blob_replica_repository.create(replica)
await self.session.commit()
stored_path = await self.adapter.resolve_path(storage_key)
await anyio.to_thread.run_sync(stored_path.exists)
return created
async def resolve_replica_path(self, replica: BlobReplica) -> Path:
await self.ensure_backend_ready()
return await self.adapter.resolve_path(replica.storage_key)
@staticmethod
def _build_storage_key(blob: Blob) -> str:
return build_blob_storage_key(blob)

266
app/services/policy_service.py

@ -0,0 +1,266 @@
from __future__ import annotations
import json
from dataclasses import dataclass
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.exceptions import ValidationError
from app.core.ids import new_id
from app.models.entities import Backend, PlacementPolicy
from app.repositories.backend_repository import BackendRepository
from app.repositories.placement_policy_repository import PlacementPolicyRepository
DEFAULT_PLACEMENT_POLICIES = {
"document": {
"require_local": True,
"stable_replica_count": 1,
"opportunistic_replica_count": 0,
"preferred_backend_ids": [],
"excluded_backend_ids": [],
"max_non_local_size_bytes": None,
},
"image": {
"require_local": True,
"stable_replica_count": 1,
"opportunistic_replica_count": 1,
"preferred_backend_ids": [],
"excluded_backend_ids": [],
"max_non_local_size_bytes": None,
},
"video": {
"require_local": True,
"stable_replica_count": 1,
"opportunistic_replica_count": 0,
"preferred_backend_ids": [],
"excluded_backend_ids": [],
"max_non_local_size_bytes": None,
},
"other": {
"require_local": True,
"stable_replica_count": 1,
"opportunistic_replica_count": 0,
"preferred_backend_ids": [],
"excluded_backend_ids": [],
"max_non_local_size_bytes": None,
},
}
ALLOWED_FILE_CLASSES = frozenset(DEFAULT_PLACEMENT_POLICIES.keys())
@dataclass
class PlacementDecision:
file_class: str
policy: PlacementPolicy
backends: list[Backend]
non_local_allowed: bool
class PlacementPolicyValidationError(ValidationError):
code = "placement_policy_invalid"
message = "Placement policy is invalid."
class PolicyService:
def __init__(self, session: AsyncSession) -> None:
self.session = session
self.backend_repository = BackendRepository(session)
self.repository = PlacementPolicyRepository(session)
async def ensure_default_policies(self) -> None:
for file_class, values in DEFAULT_PLACEMENT_POLICIES.items():
existing = await self.repository.get_by_file_class(file_class)
if existing is not None:
continue
policy = PlacementPolicy(
id=new_id("plc"),
file_class=file_class,
require_local=values["require_local"],
stable_replica_count=values["stable_replica_count"],
opportunistic_replica_count=values["opportunistic_replica_count"],
preferred_backend_ids_json=json.dumps(values["preferred_backend_ids"]),
excluded_backend_ids_json=json.dumps(values["excluded_backend_ids"]),
max_non_local_size_bytes=values["max_non_local_size_bytes"],
)
await self.repository.create(policy)
await self.session.commit()
async def list_policies(self) -> list[PlacementPolicy]:
await self.ensure_default_policies()
return await self.repository.list_all()
async def upsert_policy(
self,
*,
file_class: str,
require_local: bool,
stable_replica_count: int,
opportunistic_replica_count: int,
preferred_backend_ids: list[str],
excluded_backend_ids: list[str],
max_non_local_size_bytes: int | None,
) -> PlacementPolicy:
await self.ensure_default_policies()
self._validate_file_class(file_class)
await self._validate_policy_inputs(
require_local=require_local,
stable_replica_count=stable_replica_count,
opportunistic_replica_count=opportunistic_replica_count,
preferred_backend_ids=preferred_backend_ids,
excluded_backend_ids=excluded_backend_ids,
max_non_local_size_bytes=max_non_local_size_bytes,
)
existing = await self.repository.get_by_file_class(file_class)
if existing is None:
existing = PlacementPolicy(
id=new_id("plc"),
file_class=file_class,
require_local=require_local,
stable_replica_count=stable_replica_count,
opportunistic_replica_count=opportunistic_replica_count,
preferred_backend_ids_json=json.dumps(preferred_backend_ids),
excluded_backend_ids_json=json.dumps(excluded_backend_ids),
max_non_local_size_bytes=max_non_local_size_bytes,
)
await self.repository.create(existing)
else:
existing.require_local = require_local
existing.stable_replica_count = stable_replica_count
existing.opportunistic_replica_count = opportunistic_replica_count
existing.preferred_backend_ids_json = json.dumps(preferred_backend_ids)
existing.excluded_backend_ids_json = json.dumps(excluded_backend_ids)
existing.max_non_local_size_bytes = max_non_local_size_bytes
await self.repository.save(existing)
await self.session.commit()
return existing
async def get_placement_decision(self, mime_type: str | None, size_bytes: int | None = None) -> PlacementDecision:
await self.ensure_default_policies()
file_class = self.classify_mime_type(mime_type)
policy = await self.repository.get_by_file_class(file_class)
if policy is None:
raise ValueError(f"Placement policy is missing for file class: {file_class}")
preferred_backend_ids = self._decode_backend_ids(policy.preferred_backend_ids_json)
excluded_backend_ids = set(self._decode_backend_ids(policy.excluded_backend_ids_json))
backends = await self.backend_repository.list_enabled()
eligible_backends = [backend for backend in backends if backend.id not in excluded_backend_ids]
local_backends = self._sort_backends(
[backend for backend in eligible_backends if backend.type == "local_directory"],
preferred_backend_ids,
)
stable_backends = [
backend
for backend in eligible_backends
if backend.type != "local_directory" and backend.stability_class == "stable"
]
opportunistic_backends = [
backend
for backend in eligible_backends
if backend.type != "local_directory" and backend.stability_class == "opportunistic"
]
stable_backends = self._sort_backends(stable_backends, preferred_backend_ids)
opportunistic_backends = self._sort_backends(opportunistic_backends, preferred_backend_ids)
selected: list[Backend] = []
if policy.require_local and local_backends:
selected.append(local_backends[0])
non_local_allowed = (
policy.max_non_local_size_bytes is None
or size_bytes is None
or size_bytes <= policy.max_non_local_size_bytes
)
if non_local_allowed:
selected.extend(stable_backends[: policy.stable_replica_count])
selected.extend(opportunistic_backends[: policy.opportunistic_replica_count])
deduped: list[Backend] = []
seen = set()
for backend in selected:
if backend.id in seen:
continue
seen.add(backend.id)
deduped.append(backend)
return PlacementDecision(
file_class=file_class,
policy=policy,
backends=deduped,
non_local_allowed=non_local_allowed,
)
@staticmethod
def classify_mime_type(mime_type: str | None) -> str:
if mime_type is None:
return "other"
if mime_type.startswith("image/"):
return "image"
if mime_type.startswith("video/"):
return "video"
if mime_type == "application/pdf" or mime_type.startswith("text/"):
return "document"
return "other"
@staticmethod
def _decode_backend_ids(raw_value: str | None) -> list[str]:
if not raw_value:
return []
value = json.loads(raw_value)
if not isinstance(value, list):
return []
return [item for item in value if isinstance(item, str) and item]
@staticmethod
def _sort_backends(backends: list[Backend], preferred_backend_ids: list[str]) -> list[Backend]:
preferred_positions = {backend_id: index for index, backend_id in enumerate(preferred_backend_ids)}
return sorted(
backends,
key=lambda backend: (
0 if backend.id in preferred_positions else 1,
preferred_positions.get(backend.id, 0),
-backend.write_priority,
-backend.read_priority,
backend.name,
),
)
@staticmethod
def _validate_file_class(file_class: str) -> None:
if file_class not in ALLOWED_FILE_CLASSES:
raise PlacementPolicyValidationError(
"Unsupported file class. Allowed values: " + ", ".join(sorted(ALLOWED_FILE_CLASSES))
)
async def _validate_policy_inputs(
self,
*,
require_local: bool,
stable_replica_count: int,
opportunistic_replica_count: int,
preferred_backend_ids: list[str],
excluded_backend_ids: list[str],
max_non_local_size_bytes: int | None,
) -> None:
del require_local
if stable_replica_count < 0 or opportunistic_replica_count < 0:
raise PlacementPolicyValidationError("Replica counts must be zero or greater.")
if max_non_local_size_bytes is not None and max_non_local_size_bytes <= 0:
raise PlacementPolicyValidationError("max_non_local_size_bytes must be greater than zero.")
preferred_set = set(preferred_backend_ids)
excluded_set = set(excluded_backend_ids)
if len(preferred_set) != len(preferred_backend_ids):
raise PlacementPolicyValidationError("preferred_backend_ids must not contain duplicates.")
if len(excluded_set) != len(excluded_backend_ids):
raise PlacementPolicyValidationError("excluded_backend_ids must not contain duplicates.")
overlap = preferred_set & excluded_set
if overlap:
raise PlacementPolicyValidationError("A backend cannot be both preferred and excluded.")
known_backend_ids = {backend.id for backend in await self.backend_repository.list_all()}
unknown_backend_ids = sorted((preferred_set | excluded_set) - known_backend_ids)
if unknown_backend_ids:
raise PlacementPolicyValidationError(
"Unknown backend ids referenced by placement policy: " + ", ".join(unknown_backend_ids)
)

65
app/services/preview_service.py

@ -0,0 +1,65 @@
from __future__ import annotations
from app.core.ids import new_id
from app.models.entities import PreviewArtifact
from app.repositories.file_repository import FileRepository
from app.repositories.preview_artifact_repository import PreviewArtifactRepository
class PreviewService:
def __init__(self, session) -> None:
self.session = session
self.file_repository = FileRepository(session)
self.preview_artifact_repository = PreviewArtifactRepository(session)
async def get_preview_artifacts_for_file(self, file_id: str) -> list[PreviewArtifact]:
file_entry = await self.file_repository.get_by_id(file_id)
if file_entry is None or file_entry.current_version_id is None:
return []
return await self.preview_artifact_repository.list_by_file_version(file_entry.current_version_id)
async def generate_preview_artifacts_for_file(self, file_id: str) -> int:
file_entry = await self.file_repository.get_by_id(file_id)
if file_entry is None or file_entry.current_version_id is None:
raise ValueError("File is unavailable.")
blob = await self.file_repository.get_primary_blob_for_file(file_id)
if blob is None:
raise ValueError("Primary blob does not exist.")
artifact_types = self._select_artifact_types(file_entry.mime_type)
created = 0
for artifact_type in artifact_types:
existing = await self.preview_artifact_repository.get_by_version_and_type(
file_entry.current_version_id,
artifact_type,
)
if existing is not None:
if existing.status != "ready":
existing.status = "ready"
existing.blob_id = blob.id
await self.preview_artifact_repository.save(existing)
continue
artifact = PreviewArtifact(
id=new_id("prv"),
file_version_id=file_entry.current_version_id,
artifact_type=artifact_type,
blob_id=blob.id,
status="ready",
)
await self.preview_artifact_repository.create(artifact)
created += 1
await self.session.commit()
return created
@staticmethod
def _select_artifact_types(mime_type: str | None) -> list[str]:
if mime_type is None:
return []
if mime_type.startswith("image/"):
return ["inline_preview", "thumbnail"]
if mime_type == "application/pdf":
return ["inline_preview", "poster"]
if mime_type.startswith("video/"):
return ["poster"]
return []

36
app/services/repair_service.py

@ -0,0 +1,36 @@
from __future__ import annotations
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.exceptions import FileNotFoundError
from app.repositories.backend_repository import BackendRepository
from app.repositories.file_repository import FileRepository
from app.services.job_service import JobService
class RepairService:
def __init__(self, session: AsyncSession) -> None:
self.session = session
self.file_repository = FileRepository(session)
self.backend_repository = BackendRepository(session)
self.job_service = JobService(session)
async def enqueue_repairs_for_file(self, file_id: str) -> int:
file_entry = await self.file_repository.get_by_id(file_id)
if file_entry is None or file_entry.is_deleted or file_entry.deleted_at is not None:
raise FileNotFoundError()
await self.job_service.enqueue_file_reconcile(file_id)
return 1
async def enqueue_backend_health_checks(self) -> int:
backends = await self.backend_repository.list_enabled()
enqueued = 0
for backend in backends:
await self.job_service.enqueue_backend_health_check(backend.id)
enqueued += 1
return enqueued
async def enqueue_full_reconcile(self) -> int:
await self.job_service.enqueue_full_reconcile()
return 1

152
app/services/storage_service.py

@ -0,0 +1,152 @@
from __future__ import annotations
import builtins
import json
from datetime import UTC, datetime
from pathlib import Path
import anyio
from sqlalchemy.ext.asyncio import AsyncSession
from app.adapters.storage.registry import build_storage_adapter
from app.core.backend_config import merge_backend_config
from app.core.config import get_settings
from app.core.ids import new_id
from app.core.secret_codec import decrypt_secret_config
from app.core.storage_layout import build_blob_storage_key
from app.models.entities import Backend, Blob, BlobReplica, CacheEntry
from app.repositories.backend_repository import BackendRepository
from app.repositories.blob_replica_repository import BlobReplicaRepository
from app.repositories.cache_entry_repository import CacheEntryRepository
from app.services.policy_service import PlacementDecision, PolicyService
class StorageService:
def __init__(self, session: AsyncSession) -> None:
self.session = session
self.settings = get_settings()
self.backend_repository = BackendRepository(session)
self.blob_replica_repository = BlobReplicaRepository(session)
self.cache_entry_repository = CacheEntryRepository(session)
self.policy_service = PolicyService(session)
async def ensure_cache_dir(self) -> Path:
cache_dir = Path(self.settings.cache_dir)
await anyio.to_thread.run_sync(lambda: cache_dir.mkdir(parents=True, exist_ok=True))
return cache_dir
async def get_replication_targets_for_mime_type(
self,
mime_type: str | None,
size_bytes: int | None = None,
) -> PlacementDecision:
return await self.policy_service.get_placement_decision(mime_type, size_bytes=size_bytes)
async def persist_blob_to_backend(self, blob: Blob, backend: Backend, source_path: str) -> BlobReplica:
existing = await self.blob_replica_repository.get_by_blob_and_backend(blob.id, backend.id)
storage_key = self._build_storage_key(blob)
adapter = self._build_adapter(backend)
existing_is_usable = False
if existing is not None and existing.status == "ready":
try:
existing_is_usable = await adapter.object_exists(storage_key)
except Exception:
existing_is_usable = False
if existing_is_usable:
return existing
result = await adapter.put_file(source_path, storage_key)
etag = result.get("etag")
replica = existing or BlobReplica(
id=new_id("rep"),
blob_id=blob.id,
backend_id=backend.id,
storage_key=storage_key,
size_bytes=blob.size_bytes,
checksum=blob.content_hash,
status="ready",
etag=etag,
last_verified_at=datetime.now(UTC),
)
replica.storage_key = storage_key
replica.status = "ready"
replica.size_bytes = blob.size_bytes
replica.checksum = blob.content_hash
replica.etag = etag
replica.last_verified_at = datetime.now(UTC)
if existing is None:
created = await self.blob_replica_repository.create(replica)
else:
created = await self.blob_replica_repository.save(replica)
await self.session.commit()
return created
async def materialize_replica_to_local_cache(self, blob: Blob, replica: BlobReplica, backend: Backend) -> Path:
if backend.type == "local_directory":
adapter = self._build_adapter(backend)
return await adapter.resolve_path(replica.storage_key)
cache_dir = await self.ensure_cache_dir()
cache_path = cache_dir / self._build_storage_key(blob)
adapter = self._build_adapter(backend)
await adapter.download_file(replica.storage_key, cache_path)
existing_entry = await self.cache_entry_repository.get_ready_by_blob(blob.id)
now = datetime.now(UTC)
if existing_entry is None:
entry = CacheEntry(
id=new_id("cch"),
blob_id=blob.id,
local_path=str(cache_path),
cache_kind="remote-fetch",
size_bytes=blob.size_bytes,
status="ready",
last_accessed_at=now,
expires_at=None,
)
await self.cache_entry_repository.create(entry)
else:
existing_entry.local_path = str(cache_path)
existing_entry.status = "ready"
existing_entry.size_bytes = blob.size_bytes
existing_entry.last_accessed_at = now
await self.cache_entry_repository.save(existing_entry)
await self.session.commit()
return cache_path
async def resolve_best_local_path(self, blob: Blob) -> Path:
cached_entry = await self.cache_entry_repository.get_ready_by_blob(blob.id)
if cached_entry is not None:
cached_path = Path(cached_entry.local_path)
if await anyio.to_thread.run_sync(cached_path.exists):
cached_entry.last_accessed_at = datetime.now(UTC)
await self.cache_entry_repository.save(cached_entry)
await self.session.commit()
return cached_path
replicas = await self.blob_replica_repository.list_ready_replicas_with_backends(blob.id)
for replica, backend in replicas:
try:
return await self.materialize_replica_to_local_cache(blob, replica, backend)
except builtins.FileNotFoundError:
continue
except Exception:
continue
raise builtins.FileNotFoundError(blob.id)
async def check_backend(self, backend: Backend) -> tuple[str, str | None]:
adapter = self._build_adapter(backend)
result = await adapter.check()
return result["status"], result.get("detail")
def _build_adapter(self, backend: Backend):
public_config = json.loads(backend.public_config_json)
secret_config = decrypt_secret_config(backend.secret_config_json)
config = merge_backend_config(public_config, secret_config)
return build_storage_adapter(backend.type, config)
@staticmethod
def _build_storage_key(blob: Blob) -> str:
return build_blob_storage_key(blob)

185
app/services/upload_service.py

@ -0,0 +1,185 @@
from __future__ import annotations
import base64
import hashlib
from dataclasses import dataclass
from pathlib import Path
import anyio
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.config import get_settings
from app.core.exceptions import UploadConflictError, UploadNotFoundError, ValidationError
from app.core.ids import new_id
from app.models.entities import UploadSession
from app.repositories.file_repository import FileRepository
from app.repositories.upload_repository import UploadRepository
from app.services.file_service import CreateFileMetadataInput, FileService
from app.services.job_service import JobService
from app.services.local_storage_service import LocalStorageService
from app.services.storage_service import StorageService
@dataclass
class CreateUploadInput:
upload_length: int
upload_metadata: str | None = None
@dataclass
class FinalizeUploadInput:
directory_id: str
filename: str
class UploadService:
def __init__(self, session: AsyncSession) -> None:
self.session = session
self.repository = UploadRepository(session)
self.file_service = FileService(session)
self.file_repository = FileRepository(session)
self.local_storage_service = LocalStorageService(session)
self.storage_service = StorageService(session)
self.job_service = JobService(session)
self.settings = get_settings()
async def ensure_temp_dir(self) -> Path:
temp_dir = Path(self.settings.upload_temp_dir)
await anyio.to_thread.run_sync(lambda: temp_dir.mkdir(parents=True, exist_ok=True))
return temp_dir
async def create_upload(self, payload: CreateUploadInput) -> UploadSession:
if payload.upload_length <= 0:
raise ValidationError("Upload-Length must be greater than zero.")
temp_dir = await self.ensure_temp_dir()
upload_id = new_id("upl")
temp_path = temp_dir / f"{upload_id}.bin"
await anyio.Path(temp_path).write_bytes(b"")
upload_session = UploadSession(
id=upload_id,
total_size_bytes=payload.upload_length,
received_size_bytes=0,
status="created",
temp_path=str(temp_path),
tus_upload_url=f"/files/{upload_id}",
upload_metadata_json=payload.upload_metadata,
)
created = await self.repository.create(upload_session)
await self.session.commit()
return created
async def get_upload(self, upload_id: str) -> UploadSession:
upload_session = await self.repository.get_by_id(upload_id)
if upload_session is None:
raise UploadNotFoundError()
return upload_session
async def append_upload_bytes(self, upload_id: str, offset: int, chunk: bytes) -> UploadSession:
upload_session = await self.get_upload(upload_id)
if upload_session.status in {"finalized", "finalizing", "expired"}:
raise UploadConflictError("Upload session is not writable.")
if offset != upload_session.received_size_bytes:
raise UploadConflictError("Upload offset does not match the current stored offset.")
if upload_session.received_size_bytes + len(chunk) > upload_session.total_size_bytes:
raise UploadConflictError("Upload would exceed declared length.")
temp_path = anyio.Path(upload_session.temp_path)
async with await anyio.open_file(temp_path, "ab") as file_handle:
await file_handle.write(chunk)
upload_session.received_size_bytes += len(chunk)
upload_session.status = (
"uploaded"
if upload_session.received_size_bytes == upload_session.total_size_bytes
else "uploading"
)
saved = await self.repository.save(upload_session)
await self.session.commit()
return saved
async def finalize_upload(self, upload_id: str, payload: FinalizeUploadInput):
upload_session = await self.get_upload(upload_id)
if upload_session.received_size_bytes != upload_session.total_size_bytes:
raise UploadConflictError("Upload is not complete.")
if upload_session.status == "finalized":
raise UploadConflictError("Upload has already been finalized.")
upload_session.status = "finalizing"
await self.repository.save(upload_session)
await self.session.flush()
content_hash = await self._compute_sha256(upload_session.temp_path)
mime_type = self._guess_mime_type(payload.filename)
file_entry = await self.file_service.create_file_metadata(
CreateFileMetadataInput(
directory_id=payload.directory_id,
name=payload.filename,
mime_type=mime_type,
size_bytes=upload_session.total_size_bytes,
content_hash=content_hash,
)
)
blob = await self.file_repository.get_primary_blob_for_file(file_entry.id)
if blob is None:
raise UploadConflictError("Primary blob was not created for finalized upload.")
await self.local_storage_service.persist_blob_from_temp(blob, upload_session.temp_path)
await self.job_service.enqueue_file_reconcile(file_entry.id)
if mime_type is not None:
await self.job_service.enqueue_preview_generation(file_entry.id)
upload_session.directory_id = payload.directory_id
upload_session.filename = payload.filename
upload_session.status = "finalized"
await self.repository.save(upload_session)
await self.session.commit()
await anyio.Path(upload_session.temp_path).unlink(missing_ok=True)
return upload_session, file_entry
async def _compute_sha256(self, temp_path: str) -> str:
def _hash_file() -> str:
digest = hashlib.sha256()
with open(temp_path, "rb") as file_handle:
while True:
chunk = file_handle.read(1024 * 1024)
if not chunk:
break
digest.update(chunk)
return digest.hexdigest()
return await anyio.to_thread.run_sync(_hash_file)
@staticmethod
def parse_upload_metadata(raw_metadata: str | None) -> dict[str, str]:
if not raw_metadata:
return {}
parsed: dict[str, str] = {}
for item in raw_metadata.split(","):
item = item.strip()
if not item:
continue
parts = item.split(" ", 1)
key = parts[0]
value = ""
if len(parts) == 2 and parts[1]:
value = base64.b64decode(parts[1]).decode("utf-8")
parsed[key] = value
return parsed
@staticmethod
def _guess_mime_type(filename: str) -> str | None:
lowered = filename.lower()
if lowered.endswith(".mp4"):
return "video/mp4"
if lowered.endswith(".jpg") or lowered.endswith(".jpeg"):
return "image/jpeg"
if lowered.endswith(".png"):
return "image/png"
if lowered.endswith(".pdf"):
return "application/pdf"
return None

183
app/web/app.js

@ -0,0 +1,183 @@
const LOGIN_KEY = "iron_access_token";
const loginCard = document.querySelector("#login-card");
const appCard = document.querySelector("#app-card");
const loginForm = document.querySelector("#login-form");
const loginError = document.querySelector("#login-error");
const appError = document.querySelector("#app-error");
const usernameInput = document.querySelector("#username");
const passwordInput = document.querySelector("#password");
const welcomeTitle = document.querySelector("#welcome-title");
const sessionUser = document.querySelector("#session-user");
const rootName = document.querySelector("#root-name");
const itemsCount = document.querySelector("#items-count");
const rootItems = document.querySelector("#root-items");
const backendCount = document.querySelector("#backend-count");
const backendsList = document.querySelector("#backends-list");
const refreshButton = document.querySelector("#refresh-button");
const logoutButton = document.querySelector("#logout-button");
function getToken() {
return window.localStorage.getItem(LOGIN_KEY);
}
function setToken(token) {
window.localStorage.setItem(LOGIN_KEY, token);
}
function clearToken() {
window.localStorage.removeItem(LOGIN_KEY);
}
async function apiFetch(path, options = {}) {
const headers = new Headers(options.headers || {});
const token = getToken();
if (token) {
headers.set("Authorization", `Bearer ${token}`);
}
const response = await fetch(path, { ...options, headers });
if (!response.ok) {
let message = `Request failed with status ${response.status}`;
try {
const payload = await response.json();
message = payload.error?.message || message;
} catch {
// keep fallback message
}
throw new Error(message);
}
return response;
}
function renderList(container, items, renderItem) {
container.innerHTML = "";
if (items.length === 0) {
const empty = document.createElement("li");
empty.textContent = "Nothing here yet.";
container.append(empty);
return;
}
for (const item of items) {
container.append(renderItem(item));
}
}
function renderRootItem(item) {
const element = document.createElement("li");
const left = document.createElement("div");
const right = document.createElement("div");
right.className = "item-meta";
left.innerHTML = `<strong>${item.name}</strong>`;
if (item.kind === "file" && item.mime_type) {
right.textContent = item.mime_type;
} else {
right.textContent = item.kind;
}
element.append(left, right);
return element;
}
function renderBackendItem(item) {
const element = document.createElement("li");
const left = document.createElement("div");
left.innerHTML = `<strong>${item.name}</strong>`;
const right = document.createElement("div");
right.className = "item-meta";
right.textContent = `${item.type} · ${item.is_enabled ? "enabled" : "disabled"}`;
element.append(left, right);
return element;
}
function showLogin(errorMessage = "") {
loginCard.hidden = false;
appCard.hidden = true;
loginError.hidden = !errorMessage;
loginError.textContent = errorMessage;
}
function showApp() {
loginCard.hidden = true;
appCard.hidden = false;
appError.hidden = true;
}
async function loadWorkspace() {
try {
const [meResponse, directoryResponse, backendResponse] = await Promise.all([
apiFetch("/api/auth/me"),
apiFetch("/api/directories/dir_root/children"),
apiFetch("/api/backends"),
]);
const me = await meResponse.json();
const directory = await directoryResponse.json();
const backends = await backendResponse.json();
welcomeTitle.textContent = `Welcome, ${me.user.username}`;
sessionUser.textContent = me.user.username;
rootName.textContent = directory.directory.name;
itemsCount.textContent = `${directory.items.length} items`;
backendCount.textContent = String(backends.items.length);
renderList(rootItems, directory.items, renderRootItem);
renderList(backendsList, backends.items, renderBackendItem);
showApp();
} catch (error) {
if (String(error.message).includes("Authentication")) {
clearToken();
showLogin("Your session expired. Please sign in again.");
return;
}
appError.hidden = false;
appError.textContent = error.message;
}
}
loginForm.addEventListener("submit", async (event) => {
event.preventDefault();
loginError.hidden = true;
try {
const response = await fetch("/api/auth/login", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
username: usernameInput.value,
password: passwordInput.value,
}),
});
const payload = await response.json();
if (!response.ok) {
throw new Error(payload.error?.message || "Unable to sign in.");
}
setToken(payload.access_token);
passwordInput.value = "";
await loadWorkspace();
} catch (error) {
showLogin(error.message);
}
});
refreshButton.addEventListener("click", async () => {
await loadWorkspace();
});
logoutButton.addEventListener("click", async () => {
try {
await apiFetch("/api/auth/logout", { method: "POST" });
} catch {
// best-effort logout
}
clearToken();
showLogin();
});
if (getToken()) {
void loadWorkspace();
} else {
showLogin();
}

1
app/web/dist/assets/app.css

File diff suppressed because one or more lines are too long

61
app/web/dist/assets/app.js

File diff suppressed because one or more lines are too long

13
app/web/dist/index.html

@ -0,0 +1,13 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Iron</title>
<script type="module" crossorigin src="/assets/app.js"></script>
<link rel="stylesheet" crossorigin href="/assets/app.css">
</head>
<body>
<div id="root"></div>
</body>
</html>

107
app/web/index.html

@ -0,0 +1,107 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Iron</title>
<link rel="stylesheet" href="/assets/styles.css" />
</head>
<body>
<div class="page-shell">
<aside class="hero-panel">
<p class="eyebrow">Iron Gateway</p>
<h1>One personal drive, one calm control plane.</h1>
<p class="lede">
Log in to browse your namespace, inspect storage health, and grow this gateway into
the multi-backend drive defined by the project MVP.
</p>
<div class="hero-card">
<div>
<span class="label">Current shell</span>
<strong>Auth + browser entry</strong>
</div>
<div>
<span class="label">Next up</span>
<strong>Upload UI and S3 runtime path</strong>
</div>
</div>
</aside>
<main class="workspace-panel">
<section id="login-card" class="card">
<div class="card-header">
<p class="eyebrow">Local Login</p>
<h2>Enter the gateway</h2>
</div>
<form id="login-form" class="stack-form">
<label>
<span>Username</span>
<input id="username" name="username" type="text" autocomplete="username" required />
</label>
<label>
<span>Password</span>
<input
id="password"
name="password"
type="password"
autocomplete="current-password"
required
/>
</label>
<button type="submit">Sign in</button>
<p id="login-error" class="error" hidden></p>
</form>
</section>
<section id="app-card" class="card" hidden>
<div class="toolbar">
<div>
<p class="eyebrow">Workspace</p>
<h2 id="welcome-title">Signed in</h2>
</div>
<div class="toolbar-actions">
<button id="refresh-button" type="button" class="ghost">Refresh</button>
<button id="logout-button" type="button" class="ghost">Logout</button>
</div>
</div>
<div class="status-grid">
<article class="status-card">
<span class="label">Session</span>
<strong id="session-user">Unknown</strong>
</article>
<article class="status-card">
<span class="label">Root Directory</span>
<strong id="root-name">/</strong>
</article>
<article class="status-card">
<span class="label">Backends</span>
<strong id="backend-count">0</strong>
</article>
</div>
<section class="panel-grid">
<div class="subpanel">
<div class="subpanel-header">
<h3>Root Items</h3>
<span id="items-count" class="pill">0 items</span>
</div>
<ul id="root-items" class="item-list"></ul>
</div>
<div class="subpanel">
<div class="subpanel-header">
<h3>Configured Backends</h3>
</div>
<ul id="backends-list" class="item-list"></ul>
</div>
</section>
<p id="app-error" class="error" hidden></p>
</section>
</main>
</div>
<script src="/assets/app.js" type="module"></script>
</body>
</html>

25
app/web/routes.py

@ -0,0 +1,25 @@
from __future__ import annotations
from pathlib import Path
from fastapi import APIRouter
from fastapi import HTTPException
from fastapi.responses import FileResponse
router = APIRouter()
WEB_DIR = Path(__file__).resolve().parent
DIST_DIR = WEB_DIR / "dist"
INDEX_FILE = DIST_DIR / "index.html"
def _get_index_file() -> Path:
if not INDEX_FILE.exists():
raise HTTPException(status_code=503, detail="Web UI assets are not built.")
return INDEX_FILE
@router.get("/app", include_in_schema=False)
@router.get("/app/{path:path}", include_in_schema=False)
async def web_app(_path: str = "") -> FileResponse:
return FileResponse(_get_index_file())

245
app/web/styles.css

@ -0,0 +1,245 @@
:root {
--bg: #f3efe6;
--surface: rgba(255, 252, 245, 0.88);
--surface-strong: #fffaf1;
--ink: #1e1b16;
--muted: #6d6457;
--line: rgba(58, 44, 23, 0.12);
--accent: #bb5a2b;
--accent-strong: #8b3d17;
--success: #2d6a4f;
--danger: #a63b2c;
--shadow: 0 18px 50px rgba(72, 44, 20, 0.12);
}
* {
box-sizing: border-box;
}
body {
margin: 0;
min-height: 100vh;
color: var(--ink);
background:
radial-gradient(circle at top left, rgba(219, 164, 103, 0.28), transparent 30%),
radial-gradient(circle at bottom right, rgba(92, 140, 109, 0.24), transparent 28%),
linear-gradient(135deg, #efe7d8 0%, #f8f4ec 45%, #ece3d5 100%);
font-family: Georgia, "Iowan Old Style", "Palatino Linotype", serif;
}
button,
input {
font: inherit;
}
.page-shell {
display: grid;
grid-template-columns: minmax(280px, 0.95fr) minmax(340px, 1.2fr);
min-height: 100vh;
}
.hero-panel,
.workspace-panel {
padding: 48px 40px;
}
.hero-panel {
display: flex;
flex-direction: column;
justify-content: center;
gap: 24px;
}
.hero-panel h1 {
margin: 0;
font-size: clamp(2.3rem, 4vw, 4.4rem);
line-height: 0.95;
max-width: 10ch;
}
.lede {
margin: 0;
max-width: 52ch;
color: var(--muted);
font-size: 1.08rem;
line-height: 1.7;
}
.hero-card,
.card,
.status-card,
.subpanel {
border: 1px solid var(--line);
background: var(--surface);
backdrop-filter: blur(14px);
box-shadow: var(--shadow);
}
.hero-card {
display: grid;
gap: 18px;
padding: 22px 24px;
border-radius: 24px;
max-width: 440px;
}
.workspace-panel {
display: flex;
align-items: center;
justify-content: center;
}
.card {
width: min(860px, 100%);
padding: 28px;
border-radius: 28px;
}
.card-header h2,
.toolbar h2,
.subpanel-header h3 {
margin: 0;
}
.eyebrow,
.label {
display: block;
margin-bottom: 8px;
color: var(--muted);
font-size: 0.8rem;
letter-spacing: 0.12em;
text-transform: uppercase;
}
.stack-form {
display: grid;
gap: 18px;
}
.stack-form label {
display: grid;
gap: 8px;
}
input {
width: 100%;
padding: 14px 16px;
border: 1px solid rgba(61, 45, 24, 0.18);
border-radius: 14px;
background: var(--surface-strong);
}
button {
border: none;
border-radius: 999px;
padding: 13px 18px;
color: #fffaf2;
background: linear-gradient(135deg, var(--accent) 0%, var(--accent-strong) 100%);
cursor: pointer;
transition: transform 140ms ease, opacity 140ms ease;
}
button:hover {
transform: translateY(-1px);
}
.ghost {
color: var(--ink);
background: rgba(255, 250, 241, 0.86);
border: 1px solid var(--line);
}
.toolbar,
.subpanel-header {
display: flex;
align-items: center;
justify-content: space-between;
gap: 16px;
}
.toolbar-actions {
display: flex;
gap: 12px;
}
.status-grid,
.panel-grid {
display: grid;
gap: 16px;
margin-top: 22px;
}
.status-grid {
grid-template-columns: repeat(3, minmax(0, 1fr));
}
.status-card,
.subpanel {
padding: 18px 20px;
border-radius: 22px;
}
.panel-grid {
grid-template-columns: repeat(2, minmax(0, 1fr));
}
.item-list {
list-style: none;
margin: 16px 0 0;
padding: 0;
display: grid;
gap: 10px;
}
.item-list li {
display: flex;
align-items: center;
justify-content: space-between;
gap: 16px;
padding: 12px 14px;
border-radius: 16px;
background: rgba(255, 250, 241, 0.78);
border: 1px solid rgba(61, 45, 24, 0.08);
}
.item-meta {
color: var(--muted);
font-size: 0.92rem;
}
.pill {
padding: 6px 10px;
border-radius: 999px;
background: rgba(187, 90, 43, 0.1);
color: var(--accent-strong);
font-size: 0.86rem;
}
.error {
margin: 14px 0 0;
color: var(--danger);
}
[hidden] {
display: none !important;
}
@media (max-width: 960px) {
.page-shell {
grid-template-columns: 1fr;
}
.hero-panel,
.workspace-panel {
padding: 28px 18px;
}
.hero-panel h1 {
max-width: none;
}
.status-grid,
.panel-grid {
grid-template-columns: 1fr;
}
}

10
deploy/iron/iron.env.example

@ -0,0 +1,10 @@
IRON_DATABASE_URL=sqlite+aiosqlite:////srv/iron/iron.db
IRON_BOOTSTRAP_USERNAME=admin
IRON_BOOTSTRAP_PASSWORD=replace-with-a-strong-password
IRON_SECRET_KEY=replace-with-a-long-random-secret
IRON_UPLOAD_TEMP_DIR=/srv/iron/runtime/temp
IRON_LOCAL_STORAGE_DIR=/srv/iron/runtime/local
IRON_CACHE_DIR=/srv/iron/runtime/cache
IRON_AUTH_SESSION_TTL_HOURS=168
IRON_JOB_POLL_INTERVAL_SECONDS=2
IRON_JOB_BATCH_SIZE=10

29
deploy/iron/start_iron.sh

@ -0,0 +1,29 @@
#!/usr/bin/env bash
set -euo pipefail
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
ENV_FILE="${IRON_ENV_FILE:-$ROOT_DIR/deploy/iron/iron.env}"
HOST="${IRON_HOST:-127.0.0.1}"
PORT="${IRON_PORT:-8000}"
cd "$ROOT_DIR"
if [[ ! -f ".venv/bin/python" ]]; then
echo "missing .venv/bin/python; install dependencies first" >&2
exit 1
fi
if [[ -f "$ENV_FILE" ]]; then
set -a
# shellcheck disable=SC1090
source "$ENV_FILE"
set +a
fi
mkdir -p \
"${IRON_UPLOAD_TEMP_DIR:-./.iron-temp/uploads}" \
"${IRON_LOCAL_STORAGE_DIR:-./.iron-storage/local}" \
"${IRON_CACHE_DIR:-./.iron-cache}"
./.venv/bin/alembic upgrade head
exec ./.venv/bin/python -m uvicorn app.main:app --host "$HOST" --port "$PORT"

3
deploy/openlist/.env.example

@ -0,0 +1,3 @@
OPENLIST_BIND_HOST=127.0.0.1
OPENLIST_PORT=5244
OPENLIST_DATA_DIR=/srv/iron/openlist

9
deploy/openlist/docker-compose.yml

@ -0,0 +1,9 @@
services:
openlist:
image: openlistteam/openlist:latest
container_name: openlist
restart: unless-stopped
ports:
- "${OPENLIST_BIND_HOST:-127.0.0.1}:${OPENLIST_PORT:-5244}:5244"
volumes:
- "${OPENLIST_DATA_DIR:-/srv/iron/openlist}:/opt/openlist/data"

17
deploy/systemd/iron.service.example

@ -0,0 +1,17 @@
[Unit]
Description=Iron personal cloud drive gateway
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=iron
Group=iron
WorkingDirectory=/opt/iron
EnvironmentFile=/opt/iron/deploy/iron/iron.env
ExecStart=/opt/iron/deploy/iron/start_iron.sh
Restart=on-failure
RestartSec=3
[Install]
WantedBy=multi-user.target

15
docs/README.md

@ -0,0 +1,15 @@
# Documentation
This directory contains the stable project documentation for Iron.
- [Product](product.md): product scope, current stage, MVP cutline, and roadmap.
- [Architecture](architecture.md): backend, frontend, data model, storage, and job design.
- [API](api.md): route map and API groups.
- [Development](development.md): setup, test commands, Playwright workflow, and documentation rules.
- [OpenList + Iron Local Deployment SOP](openlist-local-deployment-sop.md): step-by-step local setup for OpenList, Aliyun Drive, WebDAV, and Iron.
Root-level project docs:
- [README](../README.md): user-facing overview and quick start.
- [Contributing](../CONTRIBUTING.md): contributor workflow.
- [Agents](../AGENTS.md): coding-agent and automation guidance.

108
docs/api.md

@ -0,0 +1,108 @@
# API
Iron exposes JSON APIs under `/api` and `tus` upload transport under `/files`.
Most `/api` routes require:
```text
Authorization: Bearer <token>
```
## Auth
- `POST /api/auth/login`
- `POST /api/auth/logout`
- `GET /api/auth/me`
## Directories
- `GET /api/directories/{directory_id}/children`
- `POST /api/directories`
- `POST /api/directories/{directory_id}/rename`
- `POST /api/directories/{directory_id}/move`
## Files
- `GET /api/files/{file_id}`
- `GET /api/files/{file_id}/download`
- `GET /api/files/{file_id}/preview`
- `GET /api/files/{file_id}/stream`
- `POST /api/files/{file_id}/rename`
- `POST /api/files/{file_id}/move`
- `DELETE /api/files/{file_id}`
- `POST /api/files/{file_id}/restore`
- `GET /api/files/recycle-bin`
## Uploads
Protocol endpoints:
- `POST /files`
- `HEAD /files/{upload_id}`
- `PATCH /files/{upload_id}`
Finalize endpoint:
- `POST /api/uploads/{upload_id}/finalize`
## Backends
- `GET /api/backends`
- `POST /api/backends`
- `PATCH /api/backends/{backend_id}`
- `POST /api/backends/{backend_id}/check`
- `POST /api/backends/{backend_id}/disable`
- `POST /api/backends/{backend_id}/enable`
- `DELETE /api/backends/{backend_id}`
Backend `type` values:
- `local_directory`
- `s3`
- `webdav`
Backend responses include public `config` values plus a `secret_fields` list for
configured credentials. Secret fields such as S3 credentials and WebDAV
passwords are never echoed back in clear text.
Backend updates preserve existing secret fields when they are omitted from the
request config, and replace them only when a new value is provided.
WebDAV configs should set `root_path` so Iron uses a dedicated folder tree
inside the remote endpoint, which is a good fit for OpenList-managed cloud
drives.
Backend deletion is guarded: the backend must be disabled first, must not be the
default local backend, must not still store blob replicas, and must not be
referenced by placement policies.
## Jobs
- `GET /api/jobs`
- `GET /api/jobs/{job_id}`
- `POST /api/jobs/{job_id}/retry`
- `POST /api/jobs/run-pending`
- `POST /api/jobs/enqueue-health-checks`
- `POST /api/jobs/enqueue-full-reconcile`
## Placement Policies
- `GET /api/policies/placement`
- `PUT /api/policies/placement/{file_class}`
- `GET /api/policies/placement/preview`
Saving a placement policy also enqueues one full reconcile job so desired
replica placement converges without a separate manual step.
## Metadata Export And Recovery
- `GET /api/exports/metadata`
- `POST /api/exports/metadata/validate`
- `GET /api/exports/metadata/integrity`
- `POST /api/exports/metadata/restore-plan`
- `POST /api/exports/metadata/import`
Metadata import requires `confirm_replace=true` and a validation token from the
restore-plan endpoint.
## System
- `GET /api/system/health`
- `GET /api/system/ready`

158
docs/architecture.md

@ -0,0 +1,158 @@
# Architecture
Iron is an async Python modular monolith with a React single-page Web app.
## System Overview
```text
Browser UI
|
FastAPI app
|
Services
|
Repositories
|
SQLite metadata database
|
Storage adapters: local directory, S3, WebDAV
```
## Backend
Core stack:
- FastAPI
- SQLAlchemy 2.x async ORM
- SQLite via `aiosqlite`
- Alembic migrations
- durable in-process jobs stored in SQLite
Important packages:
- `app/api/`: route handlers and dependency wiring
- `app/services/`: business logic
- `app/repositories/`: database access
- `app/models/`: SQLAlchemy entities
- `app/schemas/`: API schemas
- `app/adapters/storage/`: storage adapter protocol, registry, and backend implementations
## Frontend
Core stack:
- Vite
- React
- TypeScript
- React Router
- TanStack Query
Source lives in `frontend/`. Built assets are emitted into `app/web/dist/` and
served by FastAPI at `/app`.
## Data Model
Main tables:
- `users`
- `auth_sessions`
- `directories`
- `file_entries`
- `file_versions`
- `blobs`
- `blob_replicas`
- `backends`
- `upload_sessions`
- `upload_session_parts`
- `jobs`
- `placement_policies`
- `preview_artifacts`
- `cache_entries`
Design principles:
- logical namespace is separate from physical storage placement
- files have immutable content versions
- physical content is represented as blobs and replicas
- user-facing deletion uses recycle-bin semantics
- jobs are durable and retryable
- backend-specific configuration is validated before persistence and stays
behind adapter boundaries
## Upload And Read Path
Uploads use `tus` transport under `/files`.
Flow:
1. create upload session
2. append bytes with `PATCH`
3. finalize through the API
4. create file metadata, version, blob, and local replica
5. enqueue replication or preview jobs when needed
Read path:
1. resolve file and current version
2. find the best local or cached blob
3. if needed, materialize a ready remote replica into cache
4. serve download, preview, or range stream
## Browser File Access
JSON APIs use bearer-token authorization through the shared frontend API client.
Browser preview and download must not use naked `/api/files/...` media URLs in
`img`, `iframe`, `video`, or `window.open`. Fetch file content with
authorization, create an object URL, use it, then revoke it.
This is covered by the Playwright E2E flow.
## Jobs
Jobs are stored in SQLite and executed by the in-process job loop.
Current job types include:
- blob replication
- preview artifact generation
- backend health checks
- full reconcile
- file reconcile
Every job should be safe to retry.
## Storage Backends
Supported backend types:
- `local_directory`: a filesystem directory used for default local persistence
and low-latency reads
- `s3`: S3 protocol storage, including AWS S3, MinIO, and compatible
object stores
- `webdav`: WebDAV storage, including OpenList-managed cloud drive directories
exposed through a dedicated `root_path`
Replica object keys are written into a structured application-owned layout such
as `objects/file/sha256/73/35/<hash>.blob`. This keeps remote storage organized
under a predictable object tree without trying to disguise replica content as
user-authored media files. During the current development stage, Iron treats
this layout as the only supported remote object format and does not preserve
compatibility with earlier experimental paths.
Reserved object-tree namespaces:
- `objects/file/...`: primary file blobs and replica payloads
- `objects/preview/<artifact_type>/...`: generated previews such as `inline_preview`, `thumbnail`, and `poster`
- `objects/export/metadata/YYYY/MM/DD/...`: metadata snapshot exports
- `objects/manifest/<scope>/...`: future manifest-style control files such as placement or repair manifests
Backend config is normalized through typed schemas before it is stored. Public
config and encrypted secret config are persisted separately. `IRON_SECRET_KEY`
derives the encryption key for stored backend credentials. API responses include
public config plus a list of configured secret fields so operators can identify
a backend without exposing access keys, client secrets, or refresh tokens. Update
requests can change public config and provide secret overrides; omitted secret
fields keep their previous encrypted values. Disabled backends can be deleted
only when they are no longer referenced by policies and no blob replicas remain
stored on them.

Some files were not shown because too many files changed in this diff

Loading…
Cancel
Save