Compare commits
3 Commits
57b9825b2b
...
22fd71cbb1
| Author | SHA1 | Date |
|---|---|---|
|
|
22fd71cbb1 | 3 months ago |
|
|
a32943be95 | 3 months ago |
|
|
17f1530328 | 3 months ago |
156 changed files with 18156 additions and 0 deletions
@ -0,0 +1,17 @@ |
|||
.venv/ |
|||
.uv-cache/ |
|||
.pytest_cache/ |
|||
__pycache__/ |
|||
*.py[cod] |
|||
*.db |
|||
*.sqlite |
|||
.coverage |
|||
htmlcov/ |
|||
iron_gateway.egg-info/ |
|||
node_modules/ |
|||
output/ |
|||
.iron-storage/ |
|||
.iron-temp/ |
|||
.iron-cache/ |
|||
deploy/openlist/.env |
|||
deploy/iron/iron.env |
|||
@ -0,0 +1,82 @@ |
|||
# AGENTS.md |
|||
|
|||
Guidance for coding agents and automated contributors working in this repository. |
|||
|
|||
## Project Snapshot |
|||
|
|||
Iron is a self-hosted personal cloud drive gateway with: |
|||
|
|||
- FastAPI backend under `app/` |
|||
- Vite + React + TypeScript frontend under `frontend/` |
|||
- built frontend assets served from `app/web/dist/` |
|||
- Alembic migrations under `alembic/` |
|||
- tests under `tests/` |
|||
- project docs under `docs/` |
|||
|
|||
The current stage is **Product MVP Candidate**. Preserve the existing |
|||
desktop-first drive UX and avoid drifting back toward a generic admin dashboard. |
|||
|
|||
## Required Setup |
|||
|
|||
Use the local virtualenv and `uv` cache: |
|||
|
|||
```bash |
|||
UV_CACHE_DIR=.uv-cache uv venv .venv |
|||
UV_CACHE_DIR=.uv-cache uv pip install --python .venv/bin/python -e '.[dev]' |
|||
npm install |
|||
``` |
|||
|
|||
## Common Commands |
|||
|
|||
```bash |
|||
npm run build |
|||
.venv/bin/python -m pytest |
|||
.venv/bin/python scripts/ui_playwright_smoke.py |
|||
.venv/bin/python -m pytest tests/e2e/test_web_ui_playwright.py |
|||
``` |
|||
|
|||
## Development Rules |
|||
|
|||
- Keep runtime artifacts out of Git: `.venv/`, `node_modules/`, `output/`, |
|||
`.iron-storage/`, `.iron-temp/`, caches, and local databases. |
|||
- Add or update tests in the same change as behavior changes. |
|||
- For UI work, run the real-service Playwright flow and review screenshots. |
|||
- Do not use naked `/api/files/...` media URLs for browser preview/download. |
|||
Fetch file content with authorization, create an object URL, and revoke it. |
|||
- Keep docs current when changing setup, architecture, test counts, or user flows. |
|||
- Prefer small, focused changes over broad rewrites. |
|||
|
|||
## UI Quality Standard |
|||
|
|||
For any meaningful Web UI change: |
|||
|
|||
1. build frontend assets with `npm run build` |
|||
2. run `./.venv/bin/python scripts/ui_playwright_smoke.py` |
|||
3. inspect screenshots in `output/playwright/` |
|||
4. fix layout or interaction issues found in screenshots |
|||
5. rerun the Playwright flow |
|||
|
|||
See [docs/development.md](docs/development.md). |
|||
|
|||
## Documentation Expectations |
|||
|
|||
- `README.md` should stay user-facing and concise. |
|||
- `CONTRIBUTING.md` should describe contributor workflow. |
|||
- `docs/README.md` should index deeper docs. |
|||
- `docs/development.md` should describe test and UI validation rules. |
|||
- Avoid local absolute paths in committed Markdown. |
|||
|
|||
## Verification Before Finishing |
|||
|
|||
At minimum, run: |
|||
|
|||
```bash |
|||
npm run build |
|||
.venv/bin/python -m pytest |
|||
``` |
|||
|
|||
If UI behavior changed, also run: |
|||
|
|||
```bash |
|||
.venv/bin/python scripts/ui_playwright_smoke.py |
|||
``` |
|||
@ -0,0 +1,62 @@ |
|||
# Contributing |
|||
|
|||
Thanks for helping improve Iron. The project is still early, so the most useful |
|||
contributions are focused changes with tests and clear documentation updates. |
|||
|
|||
## Development Setup |
|||
|
|||
```bash |
|||
UV_CACHE_DIR=.uv-cache uv venv .venv |
|||
UV_CACHE_DIR=.uv-cache uv pip install --python .venv/bin/python -e '.[dev]' |
|||
npm install |
|||
npm run build |
|||
``` |
|||
|
|||
Run the API locally: |
|||
|
|||
```bash |
|||
.venv/bin/python -m uvicorn app.main:app --reload |
|||
``` |
|||
|
|||
Open `http://127.0.0.1:8000/app`. |
|||
|
|||
## Test Before Submitting |
|||
|
|||
Run the full suite: |
|||
|
|||
```bash |
|||
.venv/bin/python -m pytest |
|||
``` |
|||
|
|||
For UI changes, also run: |
|||
|
|||
```bash |
|||
npm run build |
|||
.venv/bin/python scripts/ui_playwright_smoke.py |
|||
``` |
|||
|
|||
Review screenshots in `output/playwright/` before considering UI work complete. |
|||
|
|||
## Pull Request Checklist |
|||
|
|||
- The change is focused and described clearly. |
|||
- Tests were added or updated for behavior changes. |
|||
- UI changes were validated with Playwright when applicable. |
|||
- Documentation was updated if setup, behavior, or architecture changed. |
|||
- Generated/runtime files are not included. |
|||
|
|||
## Code Style |
|||
|
|||
- Backend code lives in `app/` and uses async FastAPI + SQLAlchemy patterns. |
|||
- Frontend code lives in `frontend/` and uses React, React Router, and |
|||
TanStack Query. |
|||
- Keep browser file preview/download flows authenticated through the API client. |
|||
- Prefer clear names and small modules over clever abstractions. |
|||
|
|||
## Project Status |
|||
|
|||
Iron is a Product MVP Candidate. See: |
|||
|
|||
- [docs/product.md](docs/product.md) |
|||
- [docs/architecture.md](docs/architecture.md) |
|||
- [docs/development.md](docs/development.md) |
|||
@ -0,0 +1,165 @@ |
|||
# Iron |
|||
|
|||
Iron is a self-hosted personal cloud drive gateway. It provides one logical file |
|||
namespace over local storage and pluggable backends, with a browser UI for daily |
|||
file management and operational visibility. |
|||
|
|||
The project is currently a **Product MVP Candidate**: the core backend and Web |
|||
app exist, but the release still needs hardening, broader browser regression |
|||
coverage, and more real-world usage. |
|||
|
|||
 |
|||
|
|||
## Features |
|||
|
|||
- FastAPI backend with SQLite, SQLAlchemy, and Alembic migrations |
|||
- Local authentication with persisted bearer-token sessions |
|||
- Directory and file operations: browse, create folder, rename, move, delete, |
|||
restore, download, and preview |
|||
- `tus` upload flow with local object persistence |
|||
- Desktop-first React Web app at `/app` |
|||
- Storage backend management for local directories, S3 storage, and |
|||
OpenList/WebDAV runtime paths |
|||
- Placement policies, reconcile jobs, health checks, and retryable background jobs |
|||
- Metadata export, validation, integrity checks, restore plans, and guarded import |
|||
- Python Playwright end-to-end coverage for core browser flows |
|||
- Real-environment OpenList acceptance scripts for deployment validation |
|||
|
|||
## Quick Start |
|||
|
|||
Iron uses `uv` for local Python environment setup. |
|||
|
|||
```bash |
|||
UV_CACHE_DIR=.uv-cache uv venv .venv |
|||
UV_CACHE_DIR=.uv-cache uv pip install --python .venv/bin/python -e '.[dev]' |
|||
npm install |
|||
npm run build |
|||
.venv/bin/python -m uvicorn app.main:app --reload |
|||
``` |
|||
|
|||
Open the Web app: |
|||
|
|||
```text |
|||
http://127.0.0.1:8000/app |
|||
``` |
|||
|
|||
Default local bootstrap credentials: |
|||
|
|||
```text |
|||
username: admin |
|||
password: changeme-iron |
|||
``` |
|||
|
|||
For a practical local deployment with OpenList and Aliyun Drive, start with |
|||
[docs/openlist-local-deployment-sop.md](docs/openlist-local-deployment-sop.md). |
|||
|
|||
## Configuration |
|||
|
|||
Common environment variables: |
|||
|
|||
- `IRON_DATABASE_URL` |
|||
- `IRON_BOOTSTRAP_USERNAME` |
|||
- `IRON_BOOTSTRAP_PASSWORD` |
|||
- `IRON_SECRET_KEY` |
|||
- `IRON_AUTH_SESSION_TTL_HOURS` |
|||
- `IRON_UPLOAD_TEMP_DIR` |
|||
- `IRON_LOCAL_STORAGE_DIR` |
|||
- `IRON_CACHE_DIR` |
|||
- `IRON_JOB_POLL_INTERVAL_SECONDS` |
|||
- `IRON_JOB_BATCH_SIZE` |
|||
|
|||
Runtime data is intentionally ignored by Git. By default, local development may |
|||
create `.iron-storage/`, `.iron-temp/`, `iron.db`, and `output/`. |
|||
|
|||
`IRON_SECRET_KEY` encrypts stored backend credentials such as S3 keys and |
|||
WebDAV passwords. Override the development default before using real storage |
|||
credentials. |
|||
|
|||
## Development |
|||
|
|||
Build the frontend: |
|||
|
|||
```bash |
|||
npm run build |
|||
``` |
|||
|
|||
Run the full test suite: |
|||
|
|||
```bash |
|||
.venv/bin/python -m pytest |
|||
``` |
|||
|
|||
Current expected result: |
|||
|
|||
```text |
|||
78 passed |
|||
``` |
|||
|
|||
Run the real OpenList acceptance suite against a live local deployment: |
|||
|
|||
```bash |
|||
.venv/bin/python scripts/real_openlist_suite.py |
|||
``` |
|||
|
|||
Run the browser E2E flow directly: |
|||
|
|||
```bash |
|||
.venv/bin/python scripts/ui_playwright_smoke.py |
|||
``` |
|||
|
|||
Regenerate the README demo GIF: |
|||
|
|||
```bash |
|||
.venv/bin/python scripts/generate_readme_demo.py |
|||
``` |
|||
|
|||
Or through pytest: |
|||
|
|||
```bash |
|||
.venv/bin/python -m pytest tests/e2e/test_web_ui_playwright.py |
|||
``` |
|||
|
|||
Playwright screenshots and runtime artifacts are written to `output/playwright/`. |
|||
|
|||
## Repository Layout |
|||
|
|||
```text |
|||
app/ FastAPI application, services, repositories, schemas |
|||
alembic/ Database migrations |
|||
frontend/ Vite + React + TypeScript source |
|||
app/web/dist/ Built frontend assets served by FastAPI |
|||
scripts/ Local automation and E2E scripts |
|||
deploy/ OpenList compose, Iron env templates, and systemd examples |
|||
tests/ Backend, API, and browser E2E tests |
|||
docs/ Product, architecture, API, and maintenance docs |
|||
``` |
|||
|
|||
## Documentation |
|||
|
|||
Start with: |
|||
|
|||
- [Documentation index](docs/README.md) |
|||
- [Product](docs/product.md) |
|||
- [Architecture](docs/architecture.md) |
|||
- [API](docs/api.md) |
|||
- [Development](docs/development.md) |
|||
- [OpenList + Iron Local Deployment SOP](docs/openlist-local-deployment-sop.md) |
|||
|
|||
For contribution workflow, see [CONTRIBUTING.md](CONTRIBUTING.md). |
|||
For AI/coding-agent guidance, see [AGENTS.md](AGENTS.md). |
|||
|
|||
## Security Notes |
|||
|
|||
Iron is not yet a hardened public Internet service. Treat the current credentials |
|||
and local bearer-token session model as development-oriented defaults. Change the |
|||
bootstrap password and review deployment boundaries before exposing the service. |
|||
|
|||
Metadata import is guarded and requires both: |
|||
|
|||
- `confirm_replace=true` |
|||
- a fresh validation token from `POST /api/exports/metadata/restore-plan` |
|||
|
|||
## License |
|||
|
|||
No open source license has been selected yet. Add a `LICENSE` file before |
|||
publishing or accepting external contributions. |
|||
@ -0,0 +1,37 @@ |
|||
[alembic] |
|||
script_location = alembic |
|||
prepend_sys_path = . |
|||
path_separator = os |
|||
sqlalchemy.url = sqlite+aiosqlite:///./iron.db |
|||
|
|||
[loggers] |
|||
keys = root,sqlalchemy,alembic |
|||
|
|||
[handlers] |
|||
keys = console |
|||
|
|||
[formatters] |
|||
keys = generic |
|||
|
|||
[logger_root] |
|||
level = WARN |
|||
handlers = console |
|||
|
|||
[logger_sqlalchemy] |
|||
level = WARN |
|||
handlers = |
|||
qualname = sqlalchemy.engine |
|||
|
|||
[logger_alembic] |
|||
level = INFO |
|||
handlers = console |
|||
qualname = alembic |
|||
|
|||
[handler_console] |
|||
class = StreamHandler |
|||
args = (sys.stderr,) |
|||
level = NOTSET |
|||
formatter = generic |
|||
|
|||
[formatter_generic] |
|||
format = %(levelname)-5.5s [%(name)s] %(message)s |
|||
@ -0,0 +1,62 @@ |
|||
from logging.config import fileConfig |
|||
|
|||
from alembic import context |
|||
from sqlalchemy import pool |
|||
from sqlalchemy.engine import Connection |
|||
from sqlalchemy.ext.asyncio import async_engine_from_config |
|||
|
|||
from app.core.config import get_settings |
|||
from app.db.base import Base |
|||
import app.models # noqa: F401 |
|||
|
|||
config = context.config |
|||
|
|||
if config.config_file_name is not None: |
|||
fileConfig(config.config_file_name) |
|||
|
|||
target_metadata = Base.metadata |
|||
settings = get_settings() |
|||
configured_url = config.get_main_option("sqlalchemy.url") |
|||
if not configured_url: |
|||
config.set_main_option("sqlalchemy.url", settings.database_url) |
|||
|
|||
|
|||
def run_migrations_offline() -> None: |
|||
url = config.get_main_option("sqlalchemy.url") |
|||
context.configure( |
|||
url=url, |
|||
target_metadata=target_metadata, |
|||
literal_binds=True, |
|||
dialect_opts={"paramstyle": "named"}, |
|||
) |
|||
|
|||
with context.begin_transaction(): |
|||
context.run_migrations() |
|||
|
|||
|
|||
def do_run_migrations(connection: Connection) -> None: |
|||
context.configure(connection=connection, target_metadata=target_metadata) |
|||
|
|||
with context.begin_transaction(): |
|||
context.run_migrations() |
|||
|
|||
|
|||
async def run_migrations_online() -> None: |
|||
connectable = async_engine_from_config( |
|||
config.get_section(config.config_ini_section, {}), |
|||
prefix="sqlalchemy.", |
|||
poolclass=pool.NullPool, |
|||
) |
|||
|
|||
async with connectable.connect() as connection: |
|||
await connection.run_sync(do_run_migrations) |
|||
|
|||
await connectable.dispose() |
|||
|
|||
|
|||
if context.is_offline_mode(): |
|||
run_migrations_offline() |
|||
else: |
|||
import asyncio |
|||
|
|||
asyncio.run(run_migrations_online()) |
|||
@ -0,0 +1,24 @@ |
|||
"""${message} |
|||
|
|||
Revision ID: ${up_revision} |
|||
Revises: ${down_revision | comma,n} |
|||
Create Date: ${create_date} |
|||
""" |
|||
from alembic import op |
|||
import sqlalchemy as sa |
|||
|
|||
|
|||
# revision identifiers, used by Alembic. |
|||
revision = ${repr(up_revision)} |
|||
down_revision = ${repr(down_revision)} |
|||
branch_labels = ${repr(branch_labels)} |
|||
depends_on = ${repr(depends_on)} |
|||
|
|||
|
|||
def upgrade() -> None: |
|||
pass |
|||
|
|||
|
|||
def downgrade() -> None: |
|||
pass |
|||
|
|||
@ -0,0 +1 @@ |
|||
|
|||
@ -0,0 +1,327 @@ |
|||
"""baseline schema |
|||
|
|||
Revision ID: 20260417_0007 |
|||
Revises: |
|||
Create Date: 2026-04-17 10:30:00.000000 |
|||
""" |
|||
|
|||
from alembic import op |
|||
import sqlalchemy as sa |
|||
|
|||
|
|||
# revision identifiers, used by Alembic. |
|||
revision = "20260417_0007" |
|||
down_revision = None |
|||
branch_labels = None |
|||
depends_on = None |
|||
|
|||
|
|||
def upgrade() -> None: |
|||
op.create_table( |
|||
"users", |
|||
sa.Column("id", sa.String(), nullable=False), |
|||
sa.Column("username", sa.String(), nullable=False), |
|||
sa.Column("password_hash", sa.String(), nullable=False), |
|||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
|||
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
|||
sa.PrimaryKeyConstraint("id"), |
|||
sa.UniqueConstraint("username"), |
|||
) |
|||
|
|||
op.create_table( |
|||
"auth_sessions", |
|||
sa.Column("id", sa.String(), nullable=False), |
|||
sa.Column("user_id", sa.String(), nullable=False), |
|||
sa.Column("token_hash", sa.String(), nullable=False), |
|||
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=False), |
|||
sa.Column("revoked_at", sa.DateTime(timezone=True), nullable=True), |
|||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
|||
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
|||
sa.ForeignKeyConstraint(["user_id"], ["users.id"]), |
|||
sa.PrimaryKeyConstraint("id"), |
|||
) |
|||
op.create_index("idx_auth_sessions_user_status", "auth_sessions", ["user_id", "revoked_at", "expires_at"]) |
|||
op.create_index("idx_auth_sessions_token_hash", "auth_sessions", ["token_hash"], unique=True) |
|||
|
|||
op.create_table( |
|||
"directories", |
|||
sa.Column("id", sa.String(), nullable=False), |
|||
sa.Column("parent_id", sa.String(), nullable=True), |
|||
sa.Column("name", sa.String(), nullable=False), |
|||
sa.Column("path_key", sa.String(), nullable=False), |
|||
sa.Column("deleted_at", sa.DateTime(timezone=True), nullable=True), |
|||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
|||
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
|||
sa.ForeignKeyConstraint(["parent_id"], ["directories.id"]), |
|||
sa.PrimaryKeyConstraint("id"), |
|||
sa.UniqueConstraint("path_key"), |
|||
) |
|||
op.create_index("idx_directories_parent_id", "directories", ["parent_id"]) |
|||
op.create_index("idx_directories_path_key", "directories", ["path_key"]) |
|||
|
|||
op.create_table( |
|||
"backends", |
|||
sa.Column("id", sa.String(), nullable=False), |
|||
sa.Column("name", sa.String(), nullable=False), |
|||
sa.Column("type", sa.String(), nullable=False), |
|||
sa.Column("stability_class", sa.String(), nullable=False), |
|||
sa.Column("read_priority", sa.Integer(), nullable=False), |
|||
sa.Column("write_priority", sa.Integer(), nullable=False), |
|||
sa.Column("is_enabled", sa.Boolean(), server_default="1", nullable=False), |
|||
sa.Column("public_config_json", sa.Text(), nullable=False), |
|||
sa.Column("secret_config_json", sa.Text(), nullable=False), |
|||
sa.Column("capacity_hint_bytes", sa.Integer(), nullable=True), |
|||
sa.Column("last_health_status", sa.String(), nullable=True), |
|||
sa.Column("last_health_checked_at", sa.DateTime(timezone=True), nullable=True), |
|||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
|||
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
|||
sa.PrimaryKeyConstraint("id"), |
|||
sa.UniqueConstraint("name"), |
|||
) |
|||
op.create_index("idx_backends_enabled", "backends", ["is_enabled"]) |
|||
op.create_index("idx_backends_priority", "backends", ["write_priority", "read_priority"]) |
|||
|
|||
op.create_table( |
|||
"placement_policies", |
|||
sa.Column("id", sa.String(), nullable=False), |
|||
sa.Column("file_class", sa.String(), nullable=False), |
|||
sa.Column("require_local", sa.Boolean(), server_default="1", nullable=False), |
|||
sa.Column("stable_replica_count", sa.Integer(), server_default="1", nullable=False), |
|||
sa.Column("opportunistic_replica_count", sa.Integer(), server_default="0", nullable=False), |
|||
sa.Column("preferred_backend_ids_json", sa.Text(), server_default="[]", nullable=False), |
|||
sa.Column("excluded_backend_ids_json", sa.Text(), server_default="[]", nullable=False), |
|||
sa.Column("max_non_local_size_bytes", sa.Integer(), nullable=True), |
|||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
|||
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
|||
sa.PrimaryKeyConstraint("id"), |
|||
) |
|||
op.create_index("idx_placement_policies_file_class", "placement_policies", ["file_class"], unique=True) |
|||
|
|||
op.create_table( |
|||
"upload_sessions", |
|||
sa.Column("id", sa.String(), nullable=False), |
|||
sa.Column("directory_id", sa.String(), nullable=True), |
|||
sa.Column("filename", sa.String(), nullable=True), |
|||
sa.Column("total_size_bytes", sa.Integer(), nullable=False), |
|||
sa.Column("received_size_bytes", sa.Integer(), nullable=False), |
|||
sa.Column("status", sa.String(), nullable=False), |
|||
sa.Column("temp_path", sa.String(), nullable=False), |
|||
sa.Column("tus_upload_url", sa.String(), nullable=True), |
|||
sa.Column("upload_metadata_json", sa.Text(), nullable=True), |
|||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
|||
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
|||
sa.ForeignKeyConstraint(["directory_id"], ["directories.id"]), |
|||
sa.PrimaryKeyConstraint("id"), |
|||
) |
|||
op.create_index("idx_upload_sessions_status", "upload_sessions", ["status"]) |
|||
op.create_index("idx_upload_sessions_updated_at", "upload_sessions", ["updated_at"]) |
|||
|
|||
op.create_table( |
|||
"file_entries", |
|||
sa.Column("id", sa.String(), nullable=False), |
|||
sa.Column("directory_id", sa.String(), nullable=False), |
|||
sa.Column("name", sa.String(), nullable=False), |
|||
sa.Column("mime_type", sa.String(), nullable=True), |
|||
sa.Column("size_bytes", sa.Integer(), nullable=False), |
|||
sa.Column("current_version_id", sa.String(), nullable=True), |
|||
sa.Column("is_deleted", sa.Boolean(), server_default="0", nullable=False), |
|||
sa.Column("deleted_at", sa.DateTime(timezone=True), nullable=True), |
|||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
|||
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
|||
sa.ForeignKeyConstraint(["directory_id"], ["directories.id"]), |
|||
sa.PrimaryKeyConstraint("id"), |
|||
) |
|||
op.create_index("idx_file_entries_current_version_id", "file_entries", ["current_version_id"]) |
|||
op.create_index("idx_file_entries_deleted_at", "file_entries", ["deleted_at"]) |
|||
op.create_index( |
|||
"idx_file_entries_directory_name", |
|||
"file_entries", |
|||
["directory_id", "name", "is_deleted"], |
|||
unique=True, |
|||
) |
|||
|
|||
op.create_table( |
|||
"file_versions", |
|||
sa.Column("id", sa.String(), nullable=False), |
|||
sa.Column("file_entry_id", sa.String(), nullable=False), |
|||
sa.Column("content_hash", sa.String(), nullable=False), |
|||
sa.Column("size_bytes", sa.Integer(), nullable=False), |
|||
sa.Column("storage_layout", sa.String(), nullable=False), |
|||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
|||
sa.ForeignKeyConstraint(["file_entry_id"], ["file_entries.id"]), |
|||
sa.PrimaryKeyConstraint("id"), |
|||
) |
|||
op.create_index("idx_file_versions_content_hash", "file_versions", ["content_hash"]) |
|||
op.create_index("idx_file_versions_file_entry_id", "file_versions", ["file_entry_id"]) |
|||
|
|||
op.create_table( |
|||
"blobs", |
|||
sa.Column("id", sa.String(), nullable=False), |
|||
sa.Column("file_version_id", sa.String(), nullable=False), |
|||
sa.Column("blob_index", sa.Integer(), nullable=False), |
|||
sa.Column("kind", sa.String(), nullable=False), |
|||
sa.Column("content_hash", sa.String(), nullable=False), |
|||
sa.Column("size_bytes", sa.Integer(), nullable=False), |
|||
sa.Column("logical_offset", sa.Integer(), server_default="0", nullable=False), |
|||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
|||
sa.ForeignKeyConstraint(["file_version_id"], ["file_versions.id"]), |
|||
sa.PrimaryKeyConstraint("id"), |
|||
) |
|||
op.create_index("idx_blobs_content_hash", "blobs", ["content_hash"]) |
|||
op.create_index("idx_blobs_file_version_index", "blobs", ["file_version_id", "blob_index"], unique=True) |
|||
|
|||
op.create_table( |
|||
"blob_replicas", |
|||
sa.Column("id", sa.String(), nullable=False), |
|||
sa.Column("blob_id", sa.String(), nullable=False), |
|||
sa.Column("backend_id", sa.String(), nullable=False), |
|||
sa.Column("storage_key", sa.String(), nullable=False), |
|||
sa.Column("status", sa.String(), nullable=False), |
|||
sa.Column("etag", sa.String(), nullable=True), |
|||
sa.Column("checksum", sa.String(), nullable=True), |
|||
sa.Column("size_bytes", sa.Integer(), nullable=False), |
|||
sa.Column("last_verified_at", sa.DateTime(timezone=True), nullable=True), |
|||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
|||
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
|||
sa.ForeignKeyConstraint(["backend_id"], ["backends.id"]), |
|||
sa.ForeignKeyConstraint(["blob_id"], ["blobs.id"]), |
|||
sa.PrimaryKeyConstraint("id"), |
|||
) |
|||
op.create_index("idx_blob_replicas_backend_status", "blob_replicas", ["backend_id", "status"]) |
|||
op.create_index("idx_blob_replicas_blob_backend", "blob_replicas", ["blob_id", "backend_id"], unique=True) |
|||
op.create_index("idx_blob_replicas_storage_key", "blob_replicas", ["storage_key"]) |
|||
|
|||
op.create_table( |
|||
"upload_session_parts", |
|||
sa.Column("id", sa.String(), nullable=False), |
|||
sa.Column("upload_session_id", sa.String(), nullable=False), |
|||
sa.Column("part_number", sa.Integer(), nullable=False), |
|||
sa.Column("byte_offset", sa.Integer(), nullable=False), |
|||
sa.Column("size_bytes", sa.Integer(), nullable=False), |
|||
sa.Column("checksum", sa.String(), nullable=True), |
|||
sa.Column("status", sa.String(), nullable=False), |
|||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
|||
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
|||
sa.ForeignKeyConstraint(["upload_session_id"], ["upload_sessions.id"]), |
|||
sa.PrimaryKeyConstraint("id"), |
|||
) |
|||
op.create_index( |
|||
"idx_upload_session_parts_unique", |
|||
"upload_session_parts", |
|||
["upload_session_id", "part_number"], |
|||
unique=True, |
|||
) |
|||
op.create_index("idx_upload_session_parts_upload_id", "upload_session_parts", ["upload_session_id"]) |
|||
|
|||
op.create_table( |
|||
"jobs", |
|||
sa.Column("id", sa.String(), nullable=False), |
|||
sa.Column("kind", sa.String(), nullable=False), |
|||
sa.Column("status", sa.String(), nullable=False), |
|||
sa.Column("payload_json", sa.Text(), nullable=False), |
|||
sa.Column("attempt_count", sa.Integer(), server_default="0", nullable=False), |
|||
sa.Column("max_attempts", sa.Integer(), server_default="5", nullable=False), |
|||
sa.Column("run_after", sa.DateTime(timezone=True), nullable=False), |
|||
sa.Column("last_error", sa.Text(), nullable=True), |
|||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
|||
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
|||
sa.PrimaryKeyConstraint("id"), |
|||
) |
|||
op.create_index("idx_jobs_kind_status", "jobs", ["kind", "status"]) |
|||
op.create_index("idx_jobs_status_run_after", "jobs", ["status", "run_after"]) |
|||
|
|||
op.create_table( |
|||
"preview_artifacts", |
|||
sa.Column("id", sa.String(), nullable=False), |
|||
sa.Column("file_version_id", sa.String(), nullable=False), |
|||
sa.Column("artifact_type", sa.String(), nullable=False), |
|||
sa.Column("blob_id", sa.String(), nullable=False), |
|||
sa.Column("status", sa.String(), nullable=False), |
|||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
|||
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
|||
sa.ForeignKeyConstraint(["blob_id"], ["blobs.id"]), |
|||
sa.ForeignKeyConstraint(["file_version_id"], ["file_versions.id"]), |
|||
sa.PrimaryKeyConstraint("id"), |
|||
) |
|||
op.create_index( |
|||
"idx_preview_artifacts_version_type", |
|||
"preview_artifacts", |
|||
["file_version_id", "artifact_type"], |
|||
unique=True, |
|||
) |
|||
|
|||
op.create_table( |
|||
"cache_entries", |
|||
sa.Column("id", sa.String(), nullable=False), |
|||
sa.Column("blob_id", sa.String(), nullable=False), |
|||
sa.Column("local_path", sa.String(), nullable=False), |
|||
sa.Column("cache_kind", sa.String(), nullable=False), |
|||
sa.Column("size_bytes", sa.Integer(), nullable=False), |
|||
sa.Column("status", sa.String(), nullable=False), |
|||
sa.Column("last_accessed_at", sa.DateTime(timezone=True), nullable=False), |
|||
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=True), |
|||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
|||
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
|||
sa.ForeignKeyConstraint(["blob_id"], ["blobs.id"]), |
|||
sa.PrimaryKeyConstraint("id"), |
|||
) |
|||
op.create_index("idx_cache_entries_blob_id", "cache_entries", ["blob_id"]) |
|||
op.create_index("idx_cache_entries_expires_at", "cache_entries", ["expires_at"]) |
|||
op.create_index("idx_cache_entries_last_accessed_at", "cache_entries", ["last_accessed_at"]) |
|||
|
|||
|
|||
def downgrade() -> None: |
|||
op.drop_index("idx_cache_entries_last_accessed_at", table_name="cache_entries") |
|||
op.drop_index("idx_cache_entries_expires_at", table_name="cache_entries") |
|||
op.drop_index("idx_cache_entries_blob_id", table_name="cache_entries") |
|||
op.drop_table("cache_entries") |
|||
|
|||
op.drop_index("idx_preview_artifacts_version_type", table_name="preview_artifacts") |
|||
op.drop_table("preview_artifacts") |
|||
|
|||
op.drop_index("idx_jobs_status_run_after", table_name="jobs") |
|||
op.drop_index("idx_jobs_kind_status", table_name="jobs") |
|||
op.drop_table("jobs") |
|||
|
|||
op.drop_index("idx_upload_session_parts_upload_id", table_name="upload_session_parts") |
|||
op.drop_index("idx_upload_session_parts_unique", table_name="upload_session_parts") |
|||
op.drop_table("upload_session_parts") |
|||
|
|||
op.drop_index("idx_blob_replicas_storage_key", table_name="blob_replicas") |
|||
op.drop_index("idx_blob_replicas_blob_backend", table_name="blob_replicas") |
|||
op.drop_index("idx_blob_replicas_backend_status", table_name="blob_replicas") |
|||
op.drop_table("blob_replicas") |
|||
|
|||
op.drop_index("idx_placement_policies_file_class", table_name="placement_policies") |
|||
op.drop_table("placement_policies") |
|||
|
|||
op.drop_index("idx_blobs_file_version_index", table_name="blobs") |
|||
op.drop_index("idx_blobs_content_hash", table_name="blobs") |
|||
op.drop_table("blobs") |
|||
|
|||
op.drop_index("idx_file_versions_file_entry_id", table_name="file_versions") |
|||
op.drop_index("idx_file_versions_content_hash", table_name="file_versions") |
|||
op.drop_table("file_versions") |
|||
|
|||
op.drop_index("idx_file_entries_deleted_at", table_name="file_entries") |
|||
op.drop_index("idx_file_entries_directory_name", table_name="file_entries") |
|||
op.drop_index("idx_file_entries_current_version_id", table_name="file_entries") |
|||
op.drop_table("file_entries") |
|||
|
|||
op.drop_index("idx_upload_sessions_updated_at", table_name="upload_sessions") |
|||
op.drop_index("idx_upload_sessions_status", table_name="upload_sessions") |
|||
op.drop_table("upload_sessions") |
|||
|
|||
op.drop_index("idx_backends_priority", table_name="backends") |
|||
op.drop_index("idx_backends_enabled", table_name="backends") |
|||
op.drop_table("backends") |
|||
|
|||
op.drop_index("idx_directories_path_key", table_name="directories") |
|||
op.drop_index("idx_directories_parent_id", table_name="directories") |
|||
op.drop_table("directories") |
|||
|
|||
op.drop_index("idx_auth_sessions_token_hash", table_name="auth_sessions") |
|||
op.drop_index("idx_auth_sessions_user_status", table_name="auth_sessions") |
|||
op.drop_table("auth_sessions") |
|||
|
|||
op.drop_table("users") |
|||
@ -0,0 +1,2 @@ |
|||
"""Iron gateway application package.""" |
|||
|
|||
@ -0,0 +1,2 @@ |
|||
"""External adapters.""" |
|||
|
|||
@ -0,0 +1,2 @@ |
|||
"""Storage adapter implementations.""" |
|||
|
|||
@ -0,0 +1,18 @@ |
|||
from __future__ import annotations |
|||
|
|||
from pathlib import Path |
|||
from typing import Protocol |
|||
|
|||
|
|||
class StorageAdapter(Protocol): |
|||
async def check(self) -> dict[str, str]: |
|||
... |
|||
|
|||
async def put_file(self, source_path: str, storage_key: str) -> dict[str, str | None]: |
|||
... |
|||
|
|||
async def download_file(self, storage_key: str, destination_path: Path) -> Path: |
|||
... |
|||
|
|||
async def object_exists(self, storage_key: str) -> bool: |
|||
... |
|||
@ -0,0 +1,50 @@ |
|||
from __future__ import annotations |
|||
|
|||
import shutil |
|||
from pathlib import Path |
|||
|
|||
import anyio |
|||
|
|||
|
|||
class LocalStorageAdapter: |
|||
def __init__(self, base_path: str) -> None: |
|||
self.base_path = Path(base_path) |
|||
|
|||
async def ensure_base_path(self) -> Path: |
|||
await anyio.to_thread.run_sync(lambda: self.base_path.mkdir(parents=True, exist_ok=True)) |
|||
return self.base_path |
|||
|
|||
async def check(self) -> dict[str, str]: |
|||
path = await self.ensure_base_path() |
|||
is_dir = await anyio.to_thread.run_sync(path.is_dir) |
|||
if not is_dir: |
|||
raise NotADirectoryError(str(path)) |
|||
return {"status": "healthy", "detail": str(path)} |
|||
|
|||
async def put_file(self, source_path: str, storage_key: str) -> dict[str, str | None]: |
|||
destination = self.base_path / storage_key |
|||
await anyio.to_thread.run_sync(lambda: destination.parent.mkdir(parents=True, exist_ok=True)) |
|||
|
|||
def _copy_if_needed() -> None: |
|||
if not destination.exists(): |
|||
shutil.copyfile(source_path, destination) |
|||
|
|||
await anyio.to_thread.run_sync(_copy_if_needed) |
|||
return {"etag": None, "storage_key": storage_key} |
|||
|
|||
async def download_file(self, storage_key: str, destination_path: Path) -> Path: |
|||
source = await self.resolve_path(storage_key) |
|||
await anyio.to_thread.run_sync(lambda: destination_path.parent.mkdir(parents=True, exist_ok=True)) |
|||
await anyio.to_thread.run_sync(lambda: shutil.copyfile(source, destination_path)) |
|||
return destination_path |
|||
|
|||
async def resolve_path(self, storage_key: str) -> Path: |
|||
destination = self.base_path / storage_key |
|||
exists = await anyio.to_thread.run_sync(destination.exists) |
|||
if not exists: |
|||
raise FileNotFoundError(storage_key) |
|||
return destination |
|||
|
|||
async def object_exists(self, storage_key: str) -> bool: |
|||
destination = self.base_path / storage_key |
|||
return await anyio.to_thread.run_sync(destination.exists) |
|||
@ -0,0 +1,18 @@ |
|||
from __future__ import annotations |
|||
|
|||
from typing import Any |
|||
|
|||
from app.adapters.storage.base import StorageAdapter |
|||
from app.adapters.storage.local import LocalStorageAdapter |
|||
from app.adapters.storage.s3 import S3StorageAdapter |
|||
from app.adapters.storage.webdav import WebDAVStorageAdapter |
|||
|
|||
|
|||
def build_storage_adapter(backend_type: str, config: dict[str, Any]) -> StorageAdapter: |
|||
if backend_type == "local_directory": |
|||
return LocalStorageAdapter(config["base_path"]) |
|||
if backend_type == "s3": |
|||
return S3StorageAdapter(config) |
|||
if backend_type == "webdav": |
|||
return WebDAVStorageAdapter(config) |
|||
raise ValueError(f"Unsupported backend type: {backend_type}") |
|||
@ -0,0 +1,88 @@ |
|||
from __future__ import annotations |
|||
|
|||
from pathlib import Path |
|||
from typing import Any |
|||
|
|||
import anyio |
|||
import boto3 |
|||
from botocore.config import Config |
|||
from botocore.exceptions import ClientError |
|||
|
|||
|
|||
class S3StorageAdapter: |
|||
def __init__(self, config: dict[str, Any]) -> None: |
|||
self.config = config |
|||
|
|||
async def check(self) -> dict[str, str]: |
|||
client = self._create_client() |
|||
|
|||
def _check() -> dict[str, str]: |
|||
client.head_bucket(Bucket=self.config["bucket"]) |
|||
return {"status": "healthy", "detail": self.config["bucket"]} |
|||
|
|||
return await anyio.to_thread.run_sync(_check) |
|||
|
|||
async def put_file(self, source_path: str, storage_key: str) -> dict[str, str | None]: |
|||
client = self._create_client() |
|||
object_key = self._object_key(storage_key) |
|||
|
|||
def _put() -> dict[str, str | None]: |
|||
extra_args: dict[str, Any] = {} |
|||
if self.config.get("storage_class"): |
|||
extra_args["StorageClass"] = self.config["storage_class"] |
|||
if self.config.get("server_side_encryption"): |
|||
extra_args["ServerSideEncryption"] = self.config["server_side_encryption"] |
|||
client.upload_file(source_path, self.config["bucket"], object_key, ExtraArgs=extra_args) |
|||
metadata = client.head_object(Bucket=self.config["bucket"], Key=object_key) |
|||
return {"etag": metadata.get("ETag"), "storage_key": storage_key} |
|||
|
|||
return await anyio.to_thread.run_sync(_put) |
|||
|
|||
async def download_file(self, storage_key: str, destination_path: Path) -> Path: |
|||
client = self._create_client() |
|||
object_key = self._object_key(storage_key) |
|||
|
|||
def _download() -> Path: |
|||
destination_path.parent.mkdir(parents=True, exist_ok=True) |
|||
client.download_file(self.config["bucket"], object_key, str(destination_path)) |
|||
return destination_path |
|||
|
|||
return await anyio.to_thread.run_sync(_download) |
|||
|
|||
async def object_exists(self, storage_key: str) -> bool: |
|||
client = self._create_client() |
|||
object_key = self._object_key(storage_key) |
|||
|
|||
def _exists() -> bool: |
|||
try: |
|||
client.head_object(Bucket=self.config["bucket"], Key=object_key) |
|||
return True |
|||
except ClientError as exc: |
|||
error_code = exc.response.get("Error", {}).get("Code") |
|||
if error_code in {"404", "NoSuchKey", "NotFound"}: |
|||
return False |
|||
raise |
|||
|
|||
return await anyio.to_thread.run_sync(_exists) |
|||
|
|||
def _create_client(self): |
|||
client_config = None |
|||
if self.config.get("force_path_style"): |
|||
client_config = Config(s3={"addressing_style": "path"}) |
|||
client_kwargs = { |
|||
"service_name": "s3", |
|||
"region_name": self.config.get("region"), |
|||
"endpoint_url": self.config.get("endpoint_url"), |
|||
"aws_access_key_id": self.config.get("access_key_id"), |
|||
"aws_secret_access_key": self.config.get("secret_access_key"), |
|||
"verify": self.config.get("verify_ssl", True), |
|||
"config": client_config, |
|||
} |
|||
cleaned_kwargs = {key: value for key, value in client_kwargs.items() if value is not None} |
|||
return boto3.client(**cleaned_kwargs) |
|||
|
|||
def _object_key(self, storage_key: str) -> str: |
|||
prefix = (self.config.get("prefix") or "").strip("/") |
|||
if not prefix: |
|||
return storage_key |
|||
return f"{prefix}/{storage_key}" |
|||
@ -0,0 +1,145 @@ |
|||
from __future__ import annotations |
|||
|
|||
import base64 |
|||
import ssl |
|||
import urllib.error |
|||
import urllib.parse |
|||
import urllib.request |
|||
import xml.etree.ElementTree as ET |
|||
from pathlib import Path |
|||
from typing import Any |
|||
|
|||
import anyio |
|||
|
|||
|
|||
DAV_NAMESPACE = {"d": "DAV:"} |
|||
|
|||
|
|||
class WebDAVStorageAdapter: |
|||
def __init__(self, config: dict[str, Any]) -> None: |
|||
self.config = config |
|||
self.base_url = config["endpoint_url"].rstrip("/") |
|||
self.root_path = self._normalize_relative_path(config.get("root_path") or "iron") |
|||
self._ssl_context = None if config.get("verify_ssl", True) else ssl._create_unverified_context() |
|||
|
|||
async def check(self) -> dict[str, str]: |
|||
def _check() -> dict[str, str]: |
|||
self._ensure_directory(self.root_path) |
|||
return {"status": "healthy", "detail": self.root_path} |
|||
|
|||
return await anyio.to_thread.run_sync(_check) |
|||
|
|||
async def put_file(self, source_path: str, storage_key: str) -> dict[str, str | None]: |
|||
def _put() -> dict[str, str | None]: |
|||
source = Path(source_path) |
|||
if not source.exists(): |
|||
raise FileNotFoundError(source_path) |
|||
normalized_key = self._normalize_relative_path(storage_key) |
|||
segments = normalized_key.split("/") |
|||
if len(segments) > 1: |
|||
self._ensure_directory(f"{self.root_path}/{'/'.join(segments[:-1])}") |
|||
request = urllib.request.Request( |
|||
self._build_url(f"{self.root_path}/{normalized_key}"), |
|||
data=source.read_bytes(), |
|||
method="PUT", |
|||
headers=self._auth_headers(), |
|||
) |
|||
with urllib.request.urlopen(request, timeout=60, context=self._ssl_context) as response: |
|||
etag = response.headers.get("ETag") |
|||
return {"etag": etag, "storage_key": normalized_key} |
|||
|
|||
return await anyio.to_thread.run_sync(_put) |
|||
|
|||
async def download_file(self, storage_key: str, destination_path: Path) -> Path: |
|||
def _download() -> Path: |
|||
normalized_key = self._normalize_relative_path(storage_key) |
|||
request = urllib.request.Request( |
|||
self._build_url(f"{self.root_path}/{normalized_key}"), |
|||
method="GET", |
|||
headers=self._auth_headers(), |
|||
) |
|||
try: |
|||
with urllib.request.urlopen(request, timeout=60, context=self._ssl_context) as response: |
|||
body = response.read() |
|||
except urllib.error.HTTPError as exc: |
|||
if exc.code == 404: |
|||
raise FileNotFoundError(storage_key) from exc |
|||
raise |
|||
destination_path.parent.mkdir(parents=True, exist_ok=True) |
|||
destination_path.write_bytes(body) |
|||
return destination_path |
|||
|
|||
return await anyio.to_thread.run_sync(_download) |
|||
|
|||
async def object_exists(self, storage_key: str) -> bool: |
|||
def _exists() -> bool: |
|||
normalized_key = self._normalize_relative_path(storage_key) |
|||
request = urllib.request.Request( |
|||
self._build_url(f"{self.root_path}/{normalized_key}"), |
|||
method="HEAD", |
|||
headers=self._auth_headers(), |
|||
) |
|||
try: |
|||
with urllib.request.urlopen(request, timeout=20, context=self._ssl_context): |
|||
return True |
|||
except urllib.error.HTTPError as exc: |
|||
if exc.code == 404: |
|||
return False |
|||
if exc.code in {403, 405}: |
|||
return self._path_exists(f"{self.root_path}/{normalized_key}") |
|||
raise |
|||
|
|||
return await anyio.to_thread.run_sync(_exists) |
|||
|
|||
def _ensure_directory(self, relative_path: str) -> None: |
|||
current = "" |
|||
for segment in self._normalize_relative_path(relative_path).split("/"): |
|||
current = f"{current}/{segment}".strip("/") |
|||
if self._path_exists(current): |
|||
continue |
|||
request = urllib.request.Request( |
|||
self._build_url(current), |
|||
method="MKCOL", |
|||
headers=self._auth_headers(), |
|||
) |
|||
try: |
|||
with urllib.request.urlopen(request, timeout=20, context=self._ssl_context): |
|||
pass |
|||
except urllib.error.HTTPError as exc: |
|||
if exc.code not in {201, 405}: |
|||
raise RuntimeError(f"WebDAV MKCOL failed with {exc.code} for {current}") from exc |
|||
|
|||
def _path_exists(self, relative_path: str) -> bool: |
|||
request = urllib.request.Request( |
|||
self._build_url(relative_path), |
|||
method="PROPFIND", |
|||
headers={**self._auth_headers(), "Depth": "0"}, |
|||
) |
|||
try: |
|||
with urllib.request.urlopen(request, timeout=20, context=self._ssl_context) as response: |
|||
body = response.read() |
|||
except urllib.error.HTTPError as exc: |
|||
if exc.code == 404: |
|||
return False |
|||
raise RuntimeError(f"WebDAV PROPFIND failed with {exc.code} for {relative_path}") from exc |
|||
if not body: |
|||
return True |
|||
root = ET.fromstring(body) |
|||
return root.find("d:response", DAV_NAMESPACE) is not None |
|||
|
|||
def _build_url(self, relative_path: str) -> str: |
|||
safe_path = "/".join(urllib.parse.quote(part, safe="") for part in relative_path.strip("/").split("/") if part) |
|||
return f"{self.base_url}/{safe_path}" |
|||
|
|||
def _auth_headers(self) -> dict[str, str]: |
|||
username = str(self.config.get("username") or "") |
|||
password = str(self.config.get("password") or "") |
|||
token = base64.b64encode(f"{username}:{password}".encode("utf-8")).decode("ascii") |
|||
return {"Authorization": f"Basic {token}"} |
|||
|
|||
@staticmethod |
|||
def _normalize_relative_path(value: str) -> str: |
|||
normalized = "/".join(part for part in value.strip().strip("/").split("/") if part) |
|||
if not normalized: |
|||
raise ValueError("path must not be empty") |
|||
return normalized |
|||
@ -0,0 +1,2 @@ |
|||
"""HTTP API layer.""" |
|||
|
|||
@ -0,0 +1,76 @@ |
|||
from fastapi import Depends |
|||
from fastapi import Header |
|||
from sqlalchemy.ext.asyncio import AsyncSession |
|||
|
|||
from app.db.session import get_db_session |
|||
from app.models.entities import User |
|||
from app.services.export_service import ExportService |
|||
from app.services.auth_service import AuthService |
|||
from app.services.backend_service import BackendService |
|||
from app.services.directory_service import DirectoryService |
|||
from app.services.file_service import FileService |
|||
from app.services.job_service import JobService |
|||
from app.services.policy_service import PolicyService |
|||
from app.services.repair_service import RepairService |
|||
from app.services.upload_service import UploadService |
|||
|
|||
|
|||
async def get_directory_service( |
|||
session: AsyncSession = Depends(get_db_session), |
|||
) -> DirectoryService: |
|||
return DirectoryService(session) |
|||
|
|||
|
|||
async def get_file_service( |
|||
session: AsyncSession = Depends(get_db_session), |
|||
) -> FileService: |
|||
return FileService(session) |
|||
|
|||
|
|||
async def get_upload_service( |
|||
session: AsyncSession = Depends(get_db_session), |
|||
) -> UploadService: |
|||
return UploadService(session) |
|||
|
|||
|
|||
async def get_backend_service( |
|||
session: AsyncSession = Depends(get_db_session), |
|||
) -> BackendService: |
|||
return BackendService(session) |
|||
|
|||
|
|||
async def get_job_service( |
|||
session: AsyncSession = Depends(get_db_session), |
|||
) -> JobService: |
|||
return JobService(session) |
|||
|
|||
|
|||
async def get_export_service( |
|||
session: AsyncSession = Depends(get_db_session), |
|||
) -> ExportService: |
|||
return ExportService(session) |
|||
|
|||
|
|||
async def get_repair_service( |
|||
session: AsyncSession = Depends(get_db_session), |
|||
) -> RepairService: |
|||
return RepairService(session) |
|||
|
|||
|
|||
async def get_auth_service( |
|||
session: AsyncSession = Depends(get_db_session), |
|||
) -> AuthService: |
|||
return AuthService(session) |
|||
|
|||
|
|||
async def get_policy_service( |
|||
session: AsyncSession = Depends(get_db_session), |
|||
) -> PolicyService: |
|||
return PolicyService(session) |
|||
|
|||
|
|||
async def require_current_user( |
|||
authorization: str | None = Header(default=None, alias="Authorization"), |
|||
service: AuthService = Depends(get_auth_service), |
|||
) -> User: |
|||
return await service.get_current_user(authorization) |
|||
@ -0,0 +1,59 @@ |
|||
from fastapi import APIRouter |
|||
from fastapi import Depends |
|||
|
|||
from app.api.dependencies import require_current_user |
|||
from app.api.routes.auth import router as auth_router |
|||
from app.api.routes.backends import router as backend_router |
|||
from app.api.routes.directories import router as directory_router |
|||
from app.api.routes.exports import router as export_router |
|||
from app.api.routes.files import router as file_router |
|||
from app.api.routes.jobs import router as job_router |
|||
from app.api.routes.policies import router as policy_router |
|||
from app.api.routes.system import router as system_router |
|||
from app.api.routes.uploads import router as upload_router |
|||
|
|||
api_router = APIRouter() |
|||
api_router.include_router(system_router, prefix="/system", tags=["system"]) |
|||
api_router.include_router(auth_router, prefix="/auth", tags=["auth"]) |
|||
api_router.include_router( |
|||
export_router, |
|||
prefix="/exports", |
|||
tags=["exports"], |
|||
dependencies=[Depends(require_current_user)], |
|||
) |
|||
api_router.include_router( |
|||
backend_router, |
|||
prefix="/backends", |
|||
tags=["backends"], |
|||
dependencies=[Depends(require_current_user)], |
|||
) |
|||
api_router.include_router( |
|||
job_router, |
|||
prefix="/jobs", |
|||
tags=["jobs"], |
|||
dependencies=[Depends(require_current_user)], |
|||
) |
|||
api_router.include_router( |
|||
policy_router, |
|||
prefix="/policies", |
|||
tags=["policies"], |
|||
dependencies=[Depends(require_current_user)], |
|||
) |
|||
api_router.include_router( |
|||
directory_router, |
|||
prefix="/directories", |
|||
tags=["directories"], |
|||
dependencies=[Depends(require_current_user)], |
|||
) |
|||
api_router.include_router( |
|||
file_router, |
|||
prefix="/files", |
|||
tags=["files"], |
|||
dependencies=[Depends(require_current_user)], |
|||
) |
|||
api_router.include_router( |
|||
upload_router, |
|||
prefix="/uploads", |
|||
tags=["uploads"], |
|||
dependencies=[Depends(require_current_user)], |
|||
) |
|||
@ -0,0 +1,2 @@ |
|||
"""API route modules.""" |
|||
|
|||
@ -0,0 +1,46 @@ |
|||
from __future__ import annotations |
|||
|
|||
from fastapi import APIRouter |
|||
from fastapi import Depends |
|||
from fastapi import Header |
|||
|
|||
from app.api.dependencies import get_auth_service |
|||
from app.api.dependencies import require_current_user |
|||
from app.models.entities import User |
|||
from app.schemas.auth import AuthMeResponse |
|||
from app.schemas.auth import LoginRequest |
|||
from app.schemas.auth import LoginResponse |
|||
from app.schemas.auth import LogoutResponse |
|||
from app.schemas.auth import UserSummary |
|||
from app.services.auth_service import AuthService |
|||
|
|||
router = APIRouter() |
|||
|
|||
|
|||
@router.post("/login", response_model=LoginResponse) |
|||
async def login( |
|||
payload: LoginRequest, |
|||
service: AuthService = Depends(get_auth_service), |
|||
) -> LoginResponse: |
|||
result = await service.login(username=payload.username, password=payload.password) |
|||
return LoginResponse( |
|||
access_token=result.access_token, |
|||
expires_at=result.expires_at, |
|||
user=UserSummary.model_validate(result.user, from_attributes=True), |
|||
) |
|||
|
|||
|
|||
@router.get("/me", response_model=AuthMeResponse) |
|||
async def me( |
|||
current_user: User = Depends(require_current_user), |
|||
) -> AuthMeResponse: |
|||
return AuthMeResponse(user=UserSummary.model_validate(current_user, from_attributes=True)) |
|||
|
|||
|
|||
@router.post("/logout", response_model=LogoutResponse) |
|||
async def logout( |
|||
authorization: str | None = Header(default=None, alias="Authorization"), |
|||
service: AuthService = Depends(get_auth_service), |
|||
) -> LogoutResponse: |
|||
await service.logout(authorization) |
|||
return LogoutResponse() |
|||
@ -0,0 +1,112 @@ |
|||
import json |
|||
|
|||
from fastapi import APIRouter, Depends |
|||
from fastapi import Response |
|||
|
|||
from app.api.dependencies import get_backend_service |
|||
from app.core.secret_codec import list_secret_fields |
|||
from app.models.entities import Backend |
|||
from app.schemas.backend import ( |
|||
BackendCheckResponse, |
|||
BackendListResponse, |
|||
BackendMutationResponse, |
|||
BackendSummary, |
|||
CreateBackendRequest, |
|||
CreateBackendResponse, |
|||
UpdateBackendRequest, |
|||
) |
|||
from app.services.backend_service import BackendService |
|||
|
|||
router = APIRouter() |
|||
|
|||
|
|||
def summarize_backend(backend: Backend) -> BackendSummary: |
|||
summary = BackendSummary.model_validate(backend, from_attributes=True) |
|||
public_config = json.loads(backend.public_config_json) |
|||
summary.config = public_config |
|||
summary.secret_fields = list_secret_fields(backend.secret_config_json) |
|||
return summary |
|||
|
|||
|
|||
@router.get("", response_model=BackendListResponse) |
|||
async def list_backends( |
|||
service: BackendService = Depends(get_backend_service), |
|||
) -> BackendListResponse: |
|||
items = await service.list_backends() |
|||
return BackendListResponse( |
|||
items=[summarize_backend(item) for item in items] |
|||
) |
|||
|
|||
|
|||
@router.post("", response_model=CreateBackendResponse) |
|||
async def create_backend( |
|||
payload: CreateBackendRequest, |
|||
service: BackendService = Depends(get_backend_service), |
|||
) -> CreateBackendResponse: |
|||
backend = await service.create_backend( |
|||
name=payload.name, |
|||
backend_type=payload.type, |
|||
stability_class=payload.stability_class, |
|||
read_priority=payload.read_priority, |
|||
write_priority=payload.write_priority, |
|||
config=payload.config, |
|||
) |
|||
return CreateBackendResponse(backend=summarize_backend(backend)) |
|||
|
|||
|
|||
@router.post("/{backend_id}/check", response_model=BackendCheckResponse) |
|||
async def check_backend( |
|||
backend_id: str, |
|||
service: BackendService = Depends(get_backend_service), |
|||
) -> BackendCheckResponse: |
|||
backend, status, detail = await service.check_backend(backend_id) |
|||
return BackendCheckResponse( |
|||
backend_id=backend.id, |
|||
status=status, |
|||
checked_at=backend.last_health_checked_at, |
|||
detail=detail, |
|||
) |
|||
|
|||
|
|||
@router.patch("/{backend_id}", response_model=BackendMutationResponse) |
|||
async def update_backend( |
|||
backend_id: str, |
|||
payload: UpdateBackendRequest, |
|||
service: BackendService = Depends(get_backend_service), |
|||
) -> BackendMutationResponse: |
|||
backend = await service.update_backend( |
|||
backend_id, |
|||
name=payload.name, |
|||
stability_class=payload.stability_class, |
|||
read_priority=payload.read_priority, |
|||
write_priority=payload.write_priority, |
|||
config=payload.config, |
|||
) |
|||
return BackendMutationResponse(backend=summarize_backend(backend)) |
|||
|
|||
|
|||
@router.post("/{backend_id}/disable", response_model=BackendMutationResponse) |
|||
async def disable_backend( |
|||
backend_id: str, |
|||
service: BackendService = Depends(get_backend_service), |
|||
) -> BackendMutationResponse: |
|||
backend = await service.disable_backend(backend_id) |
|||
return BackendMutationResponse(backend=summarize_backend(backend)) |
|||
|
|||
|
|||
@router.post("/{backend_id}/enable", response_model=BackendMutationResponse) |
|||
async def enable_backend( |
|||
backend_id: str, |
|||
service: BackendService = Depends(get_backend_service), |
|||
) -> BackendMutationResponse: |
|||
backend = await service.enable_backend(backend_id) |
|||
return BackendMutationResponse(backend=summarize_backend(backend)) |
|||
|
|||
|
|||
@router.delete("/{backend_id}", status_code=204) |
|||
async def delete_backend( |
|||
backend_id: str, |
|||
service: BackendService = Depends(get_backend_service), |
|||
) -> Response: |
|||
await service.delete_backend(backend_id) |
|||
return Response(status_code=204) |
|||
@ -0,0 +1,72 @@ |
|||
from fastapi import APIRouter, Depends, Query, status |
|||
|
|||
from app.api.dependencies import get_directory_service |
|||
from app.schemas.directory import ( |
|||
CreateDirectoryRequest, |
|||
CreateDirectoryResponse, |
|||
DirectoryChildrenResponse, |
|||
DirectoryItem, |
|||
DirectorySummary, |
|||
MoveDirectoryRequest, |
|||
RenameDirectoryRequest, |
|||
) |
|||
from app.services.directory_service import DirectoryService |
|||
|
|||
router = APIRouter() |
|||
|
|||
|
|||
@router.get("/{directory_id}/children", response_model=DirectoryChildrenResponse) |
|||
async def list_directory_children( |
|||
directory_id: str, |
|||
_cursor: str | None = Query(default=None, alias="cursor"), |
|||
_limit: int = Query(default=100, ge=1, le=500, alias="limit"), |
|||
service: DirectoryService = Depends(get_directory_service), |
|||
) -> DirectoryChildrenResponse: |
|||
result = await service.list_children(directory_id) |
|||
return DirectoryChildrenResponse( |
|||
directory=DirectorySummary.model_validate(result.directory, from_attributes=True), |
|||
items=[ |
|||
DirectoryItem( |
|||
kind=item.kind, |
|||
id=item.item.id, |
|||
name=item.item.name, |
|||
mime_type=getattr(item.item, "mime_type", None), |
|||
size_bytes=getattr(item.item, "size_bytes", None), |
|||
created_at=item.item.created_at, |
|||
updated_at=item.item.updated_at, |
|||
) |
|||
for item in result.children |
|||
], |
|||
next_cursor=None, |
|||
) |
|||
|
|||
|
|||
@router.post("", response_model=CreateDirectoryResponse, status_code=status.HTTP_201_CREATED) |
|||
async def create_directory( |
|||
payload: CreateDirectoryRequest, |
|||
service: DirectoryService = Depends(get_directory_service), |
|||
) -> CreateDirectoryResponse: |
|||
directory = await service.create_directory(parent_id=payload.parent_id, name=payload.name) |
|||
return CreateDirectoryResponse( |
|||
directory=DirectorySummary.model_validate(directory, from_attributes=True) |
|||
) |
|||
|
|||
|
|||
@router.post("/{directory_id}/rename", response_model=CreateDirectoryResponse) |
|||
async def rename_directory( |
|||
directory_id: str, |
|||
payload: RenameDirectoryRequest, |
|||
service: DirectoryService = Depends(get_directory_service), |
|||
) -> CreateDirectoryResponse: |
|||
directory = await service.rename_directory(directory_id=directory_id, name=payload.name) |
|||
return CreateDirectoryResponse(directory=DirectorySummary.model_validate(directory, from_attributes=True)) |
|||
|
|||
|
|||
@router.post("/{directory_id}/move", response_model=CreateDirectoryResponse) |
|||
async def move_directory( |
|||
directory_id: str, |
|||
payload: MoveDirectoryRequest, |
|||
service: DirectoryService = Depends(get_directory_service), |
|||
) -> CreateDirectoryResponse: |
|||
directory = await service.move_directory(directory_id=directory_id, target_parent_id=payload.target_parent_id) |
|||
return CreateDirectoryResponse(directory=DirectorySummary.model_validate(directory, from_attributes=True)) |
|||
@ -0,0 +1,76 @@ |
|||
from __future__ import annotations |
|||
|
|||
from fastapi import APIRouter, Depends |
|||
|
|||
from app.api.dependencies import get_export_service |
|||
from app.core.exceptions import ValidationError |
|||
from app.schemas.export import ( |
|||
MetadataExportResponse, |
|||
MetadataImportRequest, |
|||
MetadataImportResponse, |
|||
MetadataRestorePlanResponse, |
|||
MetadataValidationResponse, |
|||
) |
|||
from app.services.export_service import ExportService |
|||
from app.services.job_service import JobService |
|||
|
|||
router = APIRouter() |
|||
|
|||
|
|||
@router.get("/metadata", response_model=MetadataExportResponse) |
|||
async def export_metadata(service: ExportService = Depends(get_export_service)) -> MetadataExportResponse: |
|||
data = await service.export_metadata() |
|||
exported_at = data.pop("exported_at") |
|||
return MetadataExportResponse(exported_at=exported_at, data=data) |
|||
|
|||
|
|||
@router.get("/metadata/integrity", response_model=MetadataValidationResponse) |
|||
async def metadata_integrity(service: ExportService = Depends(get_export_service)) -> MetadataValidationResponse: |
|||
result = await service.verify_runtime_integrity() |
|||
return MetadataValidationResponse(ok=result["ok"], issues=result["issues"]) |
|||
|
|||
|
|||
@router.post("/metadata/validate", response_model=MetadataValidationResponse) |
|||
async def validate_metadata( |
|||
payload: MetadataImportRequest, |
|||
service: ExportService = Depends(get_export_service), |
|||
) -> MetadataValidationResponse: |
|||
result = await service.validate_metadata_snapshot(payload.data) |
|||
return MetadataValidationResponse(ok=result["ok"], issues=result["issues"]) |
|||
|
|||
|
|||
@router.post("/metadata/restore-plan", response_model=MetadataRestorePlanResponse) |
|||
async def restore_plan( |
|||
payload: MetadataImportRequest, |
|||
service: ExportService = Depends(get_export_service), |
|||
) -> MetadataRestorePlanResponse: |
|||
result = await service.build_restore_plan(payload.data) |
|||
return MetadataRestorePlanResponse( |
|||
ok=result["ok"], |
|||
issues=result["issues"], |
|||
validation_token=result["validation_token"], |
|||
summary=result["summary"], |
|||
) |
|||
|
|||
|
|||
@router.post("/metadata/import", response_model=MetadataImportResponse) |
|||
async def import_metadata( |
|||
payload: MetadataImportRequest, |
|||
service: ExportService = Depends(get_export_service), |
|||
) -> MetadataImportResponse: |
|||
if not payload.confirm_replace: |
|||
raise ValidationError("Metadata import requires confirm_replace=true.") |
|||
restore_plan_result = await service.build_restore_plan(payload.data) |
|||
if not restore_plan_result["ok"]: |
|||
raise ValidationError("Metadata snapshot is invalid and cannot be imported.") |
|||
if payload.validation_token != restore_plan_result["validation_token"]: |
|||
raise ValidationError("Metadata import requires a current validation_token from restore-plan.") |
|||
imported_tables = await service.import_metadata(payload.data) |
|||
job_service = JobService(service.session) |
|||
reconcile_job = await job_service.enqueue_full_reconcile() |
|||
health_job_count = 0 |
|||
for backend in await job_service.backend_repository.list_enabled(): |
|||
await job_service.enqueue_backend_health_check(backend.id) |
|||
health_job_count += 1 |
|||
scheduled_jobs = [reconcile_job.kind] + ["check_backend_health"] * health_job_count |
|||
return MetadataImportResponse(imported_tables=imported_tables, scheduled_jobs=scheduled_jobs) |
|||
@ -0,0 +1,189 @@ |
|||
from typing import AsyncIterator |
|||
|
|||
import anyio |
|||
from fastapi import APIRouter, Depends, Header, Response, status |
|||
from fastapi.responses import FileResponse, StreamingResponse |
|||
|
|||
from app.api.dependencies import get_file_service, get_repair_service |
|||
from app.schemas.file import ( |
|||
DeletedFileSummary, |
|||
FileDetailResponse, |
|||
FilePlacementBackendSummary, |
|||
FileMutationResponse, |
|||
FileReplicaSummary, |
|||
FileSummary, |
|||
MoveFileRequest, |
|||
PreviewArtifactSummary, |
|||
RecycleBinResponse, |
|||
RenameFileRequest, |
|||
) |
|||
from app.services.file_service import FileService |
|||
from app.schemas.repair import RepairFileResponse |
|||
from app.services.repair_service import RepairService |
|||
|
|||
router = APIRouter() |
|||
|
|||
|
|||
@router.get("/recycle-bin", response_model=RecycleBinResponse) |
|||
async def list_recycle_bin( |
|||
service: FileService = Depends(get_file_service), |
|||
) -> RecycleBinResponse: |
|||
items = await service.list_deleted_files() |
|||
return RecycleBinResponse( |
|||
items=[DeletedFileSummary.model_validate(item, from_attributes=True) for item in items] |
|||
) |
|||
|
|||
|
|||
@router.delete("/{file_id}", response_model=FileMutationResponse) |
|||
async def delete_file( |
|||
file_id: str, |
|||
service: FileService = Depends(get_file_service), |
|||
) -> FileMutationResponse: |
|||
file_entry = await service.delete_file(file_id) |
|||
return FileMutationResponse(file=DeletedFileSummary.model_validate(file_entry, from_attributes=True)) |
|||
|
|||
|
|||
@router.post("/{file_id}/rename", response_model=FileMutationResponse) |
|||
async def rename_file( |
|||
file_id: str, |
|||
payload: RenameFileRequest, |
|||
service: FileService = Depends(get_file_service), |
|||
) -> FileMutationResponse: |
|||
file_entry = await service.rename_file(file_id=file_id, name=payload.name) |
|||
return FileMutationResponse(file=FileSummary.model_validate(file_entry, from_attributes=True)) |
|||
|
|||
|
|||
@router.post("/{file_id}/move", response_model=FileMutationResponse) |
|||
async def move_file( |
|||
file_id: str, |
|||
payload: MoveFileRequest, |
|||
service: FileService = Depends(get_file_service), |
|||
) -> FileMutationResponse: |
|||
file_entry = await service.move_file(file_id=file_id, target_parent_id=payload.target_parent_id) |
|||
return FileMutationResponse(file=FileSummary.model_validate(file_entry, from_attributes=True)) |
|||
|
|||
|
|||
@router.post("/{file_id}/restore", response_model=FileMutationResponse) |
|||
async def restore_file( |
|||
file_id: str, |
|||
service: FileService = Depends(get_file_service), |
|||
) -> FileMutationResponse: |
|||
file_entry = await service.restore_file(file_id) |
|||
return FileMutationResponse(file=FileSummary.model_validate(file_entry, from_attributes=True)) |
|||
|
|||
|
|||
@router.post("/{file_id}/reconcile", response_model=RepairFileResponse) |
|||
async def reconcile_file( |
|||
file_id: str, |
|||
service: RepairService = Depends(get_repair_service), |
|||
) -> RepairFileResponse: |
|||
enqueued = await service.enqueue_repairs_for_file(file_id) |
|||
return RepairFileResponse(file_id=file_id, enqueued_jobs=enqueued) |
|||
|
|||
|
|||
@router.get("/{file_id}", response_model=FileDetailResponse) |
|||
async def get_file_detail( |
|||
file_id: str, |
|||
service: FileService = Depends(get_file_service), |
|||
) -> FileDetailResponse: |
|||
file_entry, artifacts, replicas, decision = await service.get_file_detail(file_id) |
|||
replica_items = [ |
|||
FileReplicaSummary( |
|||
id=replica.id, |
|||
blob_id=replica.blob_id, |
|||
backend_id=backend.id, |
|||
backend_name=backend.name, |
|||
backend_type=backend.type, |
|||
backend_stability_class=backend.stability_class, |
|||
backend_enabled=backend.is_enabled, |
|||
storage_key=replica.storage_key, |
|||
status=replica.status, |
|||
size_bytes=replica.size_bytes, |
|||
checksum=replica.checksum, |
|||
etag=replica.etag, |
|||
last_verified_at=replica.last_verified_at, |
|||
created_at=replica.created_at, |
|||
updated_at=replica.updated_at, |
|||
) |
|||
for replica, backend in replicas |
|||
] |
|||
ready_replica_backend_ids = sorted({item.backend_id for item in replica_items if item.status == "ready"}) |
|||
desired_backend_ids = [backend.id for backend in (decision.backends if decision is not None else [])] |
|||
return FileDetailResponse( |
|||
file=FileSummary.model_validate(file_entry, from_attributes=True), |
|||
preview_artifacts=[ |
|||
PreviewArtifactSummary.model_validate(artifact, from_attributes=True) for artifact in artifacts |
|||
], |
|||
replicas=replica_items, |
|||
placement_file_class=decision.file_class if decision is not None else None, |
|||
desired_backends=[ |
|||
FilePlacementBackendSummary.model_validate(backend, from_attributes=True) |
|||
for backend in (decision.backends if decision is not None else []) |
|||
], |
|||
ready_replica_backend_ids=ready_replica_backend_ids, |
|||
missing_backend_ids=[backend_id for backend_id in desired_backend_ids if backend_id not in ready_replica_backend_ids], |
|||
) |
|||
|
|||
|
|||
@router.get("/{file_id}/download") |
|||
async def download_file( |
|||
file_id: str, |
|||
service: FileService = Depends(get_file_service), |
|||
) -> FileResponse: |
|||
file_entry, file_path = await service.resolve_download(file_id) |
|||
return FileResponse( |
|||
path=file_path, |
|||
filename=file_entry.name, |
|||
media_type=file_entry.mime_type or "application/octet-stream", |
|||
) |
|||
|
|||
|
|||
@router.get("/{file_id}/preview") |
|||
async def preview_file( |
|||
file_id: str, |
|||
service: FileService = Depends(get_file_service), |
|||
) -> FileResponse: |
|||
file_entry, file_path = await service.resolve_preview(file_id) |
|||
return FileResponse( |
|||
path=file_path, |
|||
filename=file_entry.name, |
|||
media_type=file_entry.mime_type or "application/octet-stream", |
|||
content_disposition_type="inline", |
|||
) |
|||
|
|||
|
|||
@router.get("/{file_id}/stream") |
|||
async def stream_file( |
|||
file_id: str, |
|||
range_header: str | None = Header(default=None, alias="Range"), |
|||
service: FileService = Depends(get_file_service), |
|||
) -> Response: |
|||
stream_result = await service.resolve_stream(file_id, range_header) |
|||
chunk_size = 64 * 1024 |
|||
|
|||
async def body_iter() -> AsyncIterator[bytes]: |
|||
remaining = stream_result.end - stream_result.start + 1 |
|||
async with await anyio.open_file(stream_result.file_path, "rb") as file_handle: |
|||
await file_handle.seek(stream_result.start) |
|||
while remaining > 0: |
|||
chunk = await file_handle.read(min(chunk_size, remaining)) |
|||
if not chunk: |
|||
break |
|||
remaining -= len(chunk) |
|||
yield chunk |
|||
|
|||
headers = { |
|||
"Accept-Ranges": "bytes", |
|||
"Content-Length": str(stream_result.end - stream_result.start + 1), |
|||
} |
|||
status_code = status.HTTP_200_OK |
|||
if range_header: |
|||
headers["Content-Range"] = f"bytes {stream_result.start}-{stream_result.end}/{stream_result.file_size}" |
|||
status_code = status.HTTP_206_PARTIAL_CONTENT |
|||
|
|||
return StreamingResponse( |
|||
body_iter(), |
|||
status_code=status_code, |
|||
headers=headers, |
|||
media_type=stream_result.file_entry.mime_type or "application/octet-stream", |
|||
) |
|||
@ -0,0 +1,51 @@ |
|||
from __future__ import annotations |
|||
|
|||
from fastapi import APIRouter, Depends |
|||
|
|||
from app.api.dependencies import get_job_service, get_repair_service |
|||
from app.schemas.job import JobDetailResponse, JobListResponse, JobMutationResponse, JobRunPendingResponse, JobSummary |
|||
from app.services.job_service import JobService |
|||
from app.services.repair_service import RepairService |
|||
from app.schemas.repair import EnqueueHealthChecksResponse |
|||
|
|||
router = APIRouter() |
|||
|
|||
|
|||
@router.get("", response_model=JobListResponse) |
|||
async def list_jobs(service: JobService = Depends(get_job_service)) -> JobListResponse: |
|||
items = await service.list_jobs() |
|||
return JobListResponse(items=[JobSummary.model_validate(item, from_attributes=True) for item in items]) |
|||
|
|||
|
|||
@router.get("/{job_id}", response_model=JobDetailResponse) |
|||
async def get_job(job_id: str, service: JobService = Depends(get_job_service)) -> JobDetailResponse: |
|||
job = await service.get_job(job_id) |
|||
return JobDetailResponse(job=JobSummary.model_validate(job, from_attributes=True)) |
|||
|
|||
|
|||
@router.post("/{job_id}/retry", response_model=JobMutationResponse) |
|||
async def retry_job(job_id: str, service: JobService = Depends(get_job_service)) -> JobMutationResponse: |
|||
job = await service.retry_job(job_id) |
|||
return JobMutationResponse(job=JobSummary.model_validate(job, from_attributes=True)) |
|||
|
|||
|
|||
@router.post("/run-pending", response_model=JobRunPendingResponse) |
|||
async def run_pending_jobs(service: JobService = Depends(get_job_service)) -> JobRunPendingResponse: |
|||
result = await service.run_pending_jobs() |
|||
return JobRunPendingResponse(processed=result.processed, completed=result.completed, failed=result.failed) |
|||
|
|||
|
|||
@router.post("/enqueue-health-checks", response_model=EnqueueHealthChecksResponse) |
|||
async def enqueue_health_checks( |
|||
service: RepairService = Depends(get_repair_service), |
|||
) -> EnqueueHealthChecksResponse: |
|||
enqueued = await service.enqueue_backend_health_checks() |
|||
return EnqueueHealthChecksResponse(enqueued_jobs=enqueued) |
|||
|
|||
|
|||
@router.post("/enqueue-full-reconcile", response_model=EnqueueHealthChecksResponse) |
|||
async def enqueue_full_reconcile( |
|||
service: RepairService = Depends(get_repair_service), |
|||
) -> EnqueueHealthChecksResponse: |
|||
enqueued = await service.enqueue_full_reconcile() |
|||
return EnqueueHealthChecksResponse(enqueued_jobs=enqueued) |
|||
@ -0,0 +1,81 @@ |
|||
from __future__ import annotations |
|||
|
|||
from fastapi import APIRouter, Depends |
|||
from fastapi import Query |
|||
|
|||
from app.api.dependencies import get_policy_service, get_repair_service |
|||
from app.schemas.policy import ( |
|||
PlacementDecisionBackendSummary, |
|||
PlacementDecisionResponse, |
|||
PlacementPolicyListResponse, |
|||
PlacementPolicyMutationResponse, |
|||
PlacementPolicySummary, |
|||
UpsertPlacementPolicyRequest, |
|||
) |
|||
from app.services.policy_service import PolicyService |
|||
from app.services.repair_service import RepairService |
|||
|
|||
router = APIRouter() |
|||
|
|||
|
|||
def _policy_to_summary(policy) -> PlacementPolicySummary: |
|||
return PlacementPolicySummary( |
|||
id=policy.id, |
|||
file_class=policy.file_class, |
|||
require_local=policy.require_local, |
|||
stable_replica_count=policy.stable_replica_count, |
|||
opportunistic_replica_count=policy.opportunistic_replica_count, |
|||
preferred_backend_ids=PolicyService._decode_backend_ids(policy.preferred_backend_ids_json), |
|||
excluded_backend_ids=PolicyService._decode_backend_ids(policy.excluded_backend_ids_json), |
|||
max_non_local_size_bytes=policy.max_non_local_size_bytes, |
|||
created_at=policy.created_at, |
|||
updated_at=policy.updated_at, |
|||
) |
|||
|
|||
|
|||
@router.get("/placement", response_model=PlacementPolicyListResponse) |
|||
async def list_placement_policies( |
|||
service: PolicyService = Depends(get_policy_service), |
|||
) -> PlacementPolicyListResponse: |
|||
policies = await service.list_policies() |
|||
return PlacementPolicyListResponse(items=[_policy_to_summary(policy) for policy in policies]) |
|||
|
|||
|
|||
@router.get("/placement/preview", response_model=PlacementDecisionResponse) |
|||
async def preview_placement_decision( |
|||
mime_type: str | None = Query(default=None), |
|||
size_bytes: int | None = Query(default=None, ge=0), |
|||
service: PolicyService = Depends(get_policy_service), |
|||
) -> PlacementDecisionResponse: |
|||
decision = await service.get_placement_decision(mime_type, size_bytes=size_bytes) |
|||
return PlacementDecisionResponse( |
|||
file_class=decision.file_class, |
|||
mime_type=mime_type, |
|||
size_bytes=size_bytes, |
|||
non_local_allowed=decision.non_local_allowed, |
|||
policy=_policy_to_summary(decision.policy), |
|||
selected_backends=[ |
|||
PlacementDecisionBackendSummary.model_validate(backend, from_attributes=True) |
|||
for backend in decision.backends |
|||
], |
|||
) |
|||
|
|||
|
|||
@router.put("/placement/{file_class}", response_model=PlacementPolicyMutationResponse) |
|||
async def upsert_placement_policy( |
|||
file_class: str, |
|||
payload: UpsertPlacementPolicyRequest, |
|||
service: PolicyService = Depends(get_policy_service), |
|||
repair_service: RepairService = Depends(get_repair_service), |
|||
) -> PlacementPolicyMutationResponse: |
|||
policy = await service.upsert_policy( |
|||
file_class=file_class, |
|||
require_local=payload.require_local, |
|||
stable_replica_count=payload.stable_replica_count, |
|||
opportunistic_replica_count=payload.opportunistic_replica_count, |
|||
preferred_backend_ids=payload.preferred_backend_ids, |
|||
excluded_backend_ids=payload.excluded_backend_ids, |
|||
max_non_local_size_bytes=payload.max_non_local_size_bytes, |
|||
) |
|||
enqueued_jobs = await repair_service.enqueue_full_reconcile() |
|||
return PlacementPolicyMutationResponse(policy=_policy_to_summary(policy), reconcile_enqueued_jobs=enqueued_jobs) |
|||
@ -0,0 +1,21 @@ |
|||
from fastapi import APIRouter |
|||
|
|||
from app.db.health import ping_database |
|||
from app.db.session import create_engine |
|||
from app.schemas.system import HealthResponse, ReadyResponse |
|||
|
|||
router = APIRouter() |
|||
|
|||
|
|||
@router.get("/health", response_model=HealthResponse) |
|||
async def health_check() -> HealthResponse: |
|||
return HealthResponse(status="ok") |
|||
|
|||
|
|||
@router.get("/ready", response_model=ReadyResponse) |
|||
async def readiness_check() -> ReadyResponse: |
|||
database_ok = await ping_database(create_engine()) |
|||
return ReadyResponse( |
|||
status="ready" if database_ok else "degraded", |
|||
database="ok" if database_ok else "error", |
|||
) |
|||
@ -0,0 +1,77 @@ |
|||
from fastapi import APIRouter, Depends, Header, Request, status |
|||
from fastapi.responses import Response |
|||
|
|||
from app.api.dependencies import get_upload_service |
|||
from app.schemas.common import ErrorResponse |
|||
from app.services.upload_service import CreateUploadInput, UploadService |
|||
|
|||
router = APIRouter() |
|||
|
|||
TUS_VERSION = "1.0.0" |
|||
|
|||
|
|||
@router.post("", status_code=status.HTTP_201_CREATED, responses={409: {"model": ErrorResponse}}) |
|||
async def create_tus_upload( |
|||
upload_length: int = Header(alias="Upload-Length"), |
|||
upload_metadata: str | None = Header(default=None, alias="Upload-Metadata"), |
|||
tus_resumable: str = Header(alias="Tus-Resumable"), |
|||
service: UploadService = Depends(get_upload_service), |
|||
) -> Response: |
|||
if tus_resumable != TUS_VERSION: |
|||
return Response(status_code=status.HTTP_412_PRECONDITION_FAILED) |
|||
|
|||
upload_session = await service.create_upload( |
|||
CreateUploadInput(upload_length=upload_length, upload_metadata=upload_metadata) |
|||
) |
|||
return Response( |
|||
status_code=status.HTTP_201_CREATED, |
|||
headers={ |
|||
"Location": upload_session.tus_upload_url or f"/files/{upload_session.id}", |
|||
"Tus-Resumable": TUS_VERSION, |
|||
}, |
|||
) |
|||
|
|||
|
|||
@router.head("/{upload_id}") |
|||
async def head_tus_upload( |
|||
upload_id: str, |
|||
tus_resumable: str = Header(alias="Tus-Resumable"), |
|||
service: UploadService = Depends(get_upload_service), |
|||
) -> Response: |
|||
if tus_resumable != TUS_VERSION: |
|||
return Response(status_code=status.HTTP_412_PRECONDITION_FAILED) |
|||
|
|||
upload_session = await service.get_upload(upload_id) |
|||
return Response( |
|||
status_code=status.HTTP_200_OK, |
|||
headers={ |
|||
"Tus-Resumable": TUS_VERSION, |
|||
"Upload-Offset": str(upload_session.received_size_bytes), |
|||
"Upload-Length": str(upload_session.total_size_bytes), |
|||
}, |
|||
) |
|||
|
|||
|
|||
@router.patch("/{upload_id}", status_code=status.HTTP_204_NO_CONTENT, responses={409: {"model": ErrorResponse}}) |
|||
async def patch_tus_upload( |
|||
upload_id: str, |
|||
request: Request, |
|||
upload_offset: int = Header(alias="Upload-Offset"), |
|||
tus_resumable: str = Header(alias="Tus-Resumable"), |
|||
content_type: str = Header(alias="Content-Type"), |
|||
service: UploadService = Depends(get_upload_service), |
|||
) -> Response: |
|||
if tus_resumable != TUS_VERSION: |
|||
return Response(status_code=status.HTTP_412_PRECONDITION_FAILED) |
|||
if content_type != "application/offset+octet-stream": |
|||
return Response(status_code=status.HTTP_415_UNSUPPORTED_MEDIA_TYPE) |
|||
|
|||
chunk = await request.body() |
|||
upload_session = await service.append_upload_bytes(upload_id, upload_offset, chunk) |
|||
return Response( |
|||
status_code=status.HTTP_204_NO_CONTENT, |
|||
headers={ |
|||
"Tus-Resumable": TUS_VERSION, |
|||
"Upload-Offset": str(upload_session.received_size_bytes), |
|||
}, |
|||
) |
|||
@ -0,0 +1,24 @@ |
|||
from fastapi import APIRouter, Depends |
|||
|
|||
from app.api.dependencies import get_upload_service |
|||
from app.schemas.file import FileSummary |
|||
from app.schemas.upload import FinalizeUploadRequest, FinalizeUploadResponse, UploadSummary |
|||
from app.services.upload_service import FinalizeUploadInput, UploadService |
|||
|
|||
router = APIRouter() |
|||
|
|||
|
|||
@router.post("/{upload_id}/finalize", response_model=FinalizeUploadResponse) |
|||
async def finalize_upload( |
|||
upload_id: str, |
|||
payload: FinalizeUploadRequest, |
|||
service: UploadService = Depends(get_upload_service), |
|||
) -> FinalizeUploadResponse: |
|||
upload_session, file_entry = await service.finalize_upload( |
|||
upload_id, |
|||
FinalizeUploadInput(directory_id=payload.directory_id, filename=payload.filename), |
|||
) |
|||
return FinalizeUploadResponse( |
|||
upload=UploadSummary.model_validate(upload_session, from_attributes=True), |
|||
file=FileSummary.model_validate(file_entry, from_attributes=True), |
|||
) |
|||
@ -0,0 +1,2 @@ |
|||
"""Core configuration and bootstrap helpers.""" |
|||
|
|||
@ -0,0 +1,130 @@ |
|||
from __future__ import annotations |
|||
|
|||
from typing import Any, Literal |
|||
|
|||
from pydantic import BaseModel, Field, ValidationError as PydanticValidationError, field_validator |
|||
|
|||
from app.core.exceptions import ValidationError |
|||
|
|||
BackendType = Literal["local_directory", "s3", "webdav"] |
|||
|
|||
|
|||
class BackendConfigError(ValidationError): |
|||
code = "backend_config_invalid" |
|||
message = "Backend configuration is invalid." |
|||
|
|||
|
|||
class LocalDirectoryConfig(BaseModel): |
|||
base_path: str = Field(min_length=1) |
|||
|
|||
@field_validator("base_path") |
|||
@classmethod |
|||
def clean_base_path(cls, value: str) -> str: |
|||
cleaned = value.strip() |
|||
if not cleaned: |
|||
raise ValueError("base_path must not be empty") |
|||
return cleaned |
|||
|
|||
|
|||
class S3CompatibleConfig(BaseModel): |
|||
bucket: str = Field(min_length=1) |
|||
region: str | None = None |
|||
endpoint_url: str | None = None |
|||
access_key_id: str | None = None |
|||
secret_access_key: str | None = None |
|||
prefix: str = "" |
|||
force_path_style: bool = False |
|||
verify_ssl: bool = True |
|||
storage_class: str | None = None |
|||
server_side_encryption: str | None = None |
|||
|
|||
@field_validator("bucket") |
|||
@classmethod |
|||
def clean_bucket(cls, value: str) -> str: |
|||
cleaned = value.strip() |
|||
if not cleaned: |
|||
raise ValueError("bucket must not be empty") |
|||
return cleaned |
|||
|
|||
@field_validator("prefix") |
|||
@classmethod |
|||
def clean_prefix(cls, value: str) -> str: |
|||
return value.strip().strip("/") |
|||
|
|||
|
|||
class WebDAVConfig(BaseModel): |
|||
endpoint_url: str = Field(min_length=1) |
|||
username: str = Field(min_length=1) |
|||
password: str | None = None |
|||
root_path: str = Field(default="iron", min_length=1) |
|||
verify_ssl: bool = True |
|||
|
|||
@field_validator("endpoint_url", "username") |
|||
@classmethod |
|||
def clean_required_text(cls, value: str) -> str: |
|||
cleaned = value.strip() |
|||
if not cleaned: |
|||
raise ValueError("value must not be empty") |
|||
return cleaned |
|||
|
|||
@field_validator("endpoint_url") |
|||
@classmethod |
|||
def clean_endpoint_url(cls, value: str) -> str: |
|||
return value.strip().rstrip("/") |
|||
|
|||
@field_validator("root_path") |
|||
@classmethod |
|||
def clean_root_path(cls, value: str) -> str: |
|||
cleaned = "/".join(part for part in value.strip().strip("/").split("/") if part) |
|||
if not cleaned: |
|||
raise ValueError("root_path must not be empty") |
|||
return cleaned |
|||
|
|||
|
|||
CONFIG_MODELS: dict[str, type[BaseModel]] = { |
|||
"local_directory": LocalDirectoryConfig, |
|||
"s3": S3CompatibleConfig, |
|||
"webdav": WebDAVConfig, |
|||
} |
|||
|
|||
SECRET_CONFIG_KEYS = { |
|||
"access_key_id", |
|||
"secret_access_key", |
|||
"password", |
|||
} |
|||
|
|||
|
|||
def supported_backend_types() -> set[str]: |
|||
return set(CONFIG_MODELS) |
|||
|
|||
|
|||
def validate_backend_config(backend_type: str, config: dict[str, Any]) -> dict[str, Any]: |
|||
model = CONFIG_MODELS.get(backend_type) |
|||
if model is None: |
|||
raise BackendConfigError(f"Unsupported backend type: {backend_type}.") |
|||
|
|||
try: |
|||
return model.model_validate(config).model_dump() |
|||
except PydanticValidationError as exc: |
|||
first_error = exc.errors()[0] if exc.errors() else {} |
|||
location = ".".join(str(part) for part in first_error.get("loc", [])) |
|||
message = first_error.get("msg", "Invalid backend configuration.") |
|||
prefix = f"{location}: " if location else "" |
|||
raise BackendConfigError(prefix + message) from exc |
|||
|
|||
|
|||
def split_backend_config(backend_type: str, config: dict[str, Any]) -> tuple[dict[str, Any], dict[str, Any]]: |
|||
normalized = validate_backend_config(backend_type, config) |
|||
public_config: dict[str, Any] = {} |
|||
secret_config: dict[str, Any] = {} |
|||
for key, value in normalized.items(): |
|||
if key in SECRET_CONFIG_KEYS: |
|||
if value: |
|||
secret_config[key] = value |
|||
else: |
|||
public_config[key] = value |
|||
return public_config, secret_config |
|||
|
|||
|
|||
def merge_backend_config(public_config: dict[str, Any], secret_config: dict[str, Any]) -> dict[str, Any]: |
|||
return {**public_config, **secret_config} |
|||
@ -0,0 +1,54 @@ |
|||
import os |
|||
from dataclasses import dataclass |
|||
from functools import lru_cache |
|||
|
|||
|
|||
def _get_bool_env(name: str, default: bool) -> bool: |
|||
raw = os.getenv(name) |
|||
if raw is None: |
|||
return default |
|||
return raw.strip().lower() in {"1", "true", "yes", "on"} |
|||
|
|||
|
|||
@dataclass(frozen=True) |
|||
class Settings: |
|||
app_name: str = "Iron Gateway" |
|||
debug: bool = False |
|||
api_prefix: str = "/api" |
|||
database_url: str = "sqlite+aiosqlite:///./iron.db" |
|||
upload_temp_dir: str = "./.iron-temp/uploads" |
|||
local_storage_dir: str = "./.iron-storage/local" |
|||
cache_dir: str = "./.iron-cache" |
|||
bootstrap_username: str = "admin" |
|||
bootstrap_password: str = "changeme-iron" |
|||
secret_key: str = "iron-local-development-secret-key" |
|||
auth_session_ttl_hours: int = 24 * 7 |
|||
job_poll_interval_seconds: float = 2.0 |
|||
job_batch_size: int = 10 |
|||
|
|||
@classmethod |
|||
def from_env(cls) -> "Settings": |
|||
return cls( |
|||
app_name=os.getenv("IRON_APP_NAME", cls.app_name), |
|||
debug=_get_bool_env("IRON_DEBUG", cls.debug), |
|||
api_prefix=os.getenv("IRON_API_PREFIX", cls.api_prefix), |
|||
database_url=os.getenv("IRON_DATABASE_URL", cls.database_url), |
|||
upload_temp_dir=os.getenv("IRON_UPLOAD_TEMP_DIR", cls.upload_temp_dir), |
|||
local_storage_dir=os.getenv("IRON_LOCAL_STORAGE_DIR", cls.local_storage_dir), |
|||
cache_dir=os.getenv("IRON_CACHE_DIR", cls.cache_dir), |
|||
bootstrap_username=os.getenv("IRON_BOOTSTRAP_USERNAME", cls.bootstrap_username), |
|||
bootstrap_password=os.getenv("IRON_BOOTSTRAP_PASSWORD", cls.bootstrap_password), |
|||
secret_key=os.getenv("IRON_SECRET_KEY", cls.secret_key), |
|||
auth_session_ttl_hours=int( |
|||
os.getenv("IRON_AUTH_SESSION_TTL_HOURS", str(cls.auth_session_ttl_hours)) |
|||
), |
|||
job_poll_interval_seconds=float( |
|||
os.getenv("IRON_JOB_POLL_INTERVAL_SECONDS", str(cls.job_poll_interval_seconds)) |
|||
), |
|||
job_batch_size=int(os.getenv("IRON_JOB_BATCH_SIZE", str(cls.job_batch_size))), |
|||
) |
|||
|
|||
|
|||
@lru_cache(maxsize=1) |
|||
def get_settings() -> Settings: |
|||
return Settings.from_env() |
|||
@ -0,0 +1,77 @@ |
|||
class AppError(Exception): |
|||
code = "internal_error" |
|||
message = "An unexpected error occurred." |
|||
|
|||
def __init__(self, message: str | None = None) -> None: |
|||
super().__init__(message or self.message) |
|||
self.detail = message or self.message |
|||
|
|||
|
|||
class NotFoundError(AppError): |
|||
code = "not_found" |
|||
message = "Resource not found." |
|||
|
|||
|
|||
class ConflictError(AppError): |
|||
code = "conflict" |
|||
message = "Resource conflict." |
|||
|
|||
|
|||
class ValidationError(AppError): |
|||
code = "validation_error" |
|||
message = "Validation failed." |
|||
|
|||
|
|||
class UnauthorizedError(AppError): |
|||
code = "unauthorized" |
|||
message = "Authentication is required." |
|||
|
|||
|
|||
class AuthenticationFailedError(UnauthorizedError): |
|||
code = "authentication_failed" |
|||
message = "Invalid username or password." |
|||
|
|||
|
|||
class AuthSessionExpiredError(UnauthorizedError): |
|||
code = "auth_session_expired" |
|||
message = "Authentication session is no longer valid." |
|||
|
|||
|
|||
class DirectoryNotFoundError(NotFoundError): |
|||
code = "directory_not_found" |
|||
message = "Directory does not exist." |
|||
|
|||
|
|||
class DirectoryConflictError(ConflictError): |
|||
code = "directory_conflict" |
|||
message = "Directory already exists." |
|||
|
|||
|
|||
class FileNotFoundError(NotFoundError): |
|||
code = "file_not_found" |
|||
message = "File does not exist." |
|||
|
|||
|
|||
class FileConflictError(ConflictError): |
|||
code = "file_conflict" |
|||
message = "File already exists." |
|||
|
|||
|
|||
class UploadNotFoundError(NotFoundError): |
|||
code = "upload_not_found" |
|||
message = "Upload session does not exist." |
|||
|
|||
|
|||
class UploadConflictError(ConflictError): |
|||
code = "upload_conflict" |
|||
message = "Upload state conflict." |
|||
|
|||
|
|||
class RangeNotSatisfiableError(AppError): |
|||
code = "range_not_satisfiable" |
|||
message = "Requested byte range is not satisfiable." |
|||
|
|||
|
|||
class PreviewNotSupportedError(AppError): |
|||
code = "preview_not_supported" |
|||
message = "Preview is not supported for this file type." |
|||
@ -0,0 +1,5 @@ |
|||
from uuid import uuid4 |
|||
|
|||
|
|||
def new_id(prefix: str) -> str: |
|||
return f"{prefix}_{uuid4().hex}" |
|||
@ -0,0 +1,50 @@ |
|||
from __future__ import annotations |
|||
|
|||
import base64 |
|||
import hashlib |
|||
import json |
|||
from typing import Any |
|||
|
|||
from cryptography.fernet import Fernet, InvalidToken |
|||
|
|||
from app.core.config import get_settings |
|||
from app.core.exceptions import ValidationError |
|||
|
|||
SECRET_PREFIX = "fernet:v1:" |
|||
|
|||
|
|||
class SecretConfigError(ValidationError): |
|||
code = "secret_config_invalid" |
|||
message = "Secret configuration cannot be decrypted." |
|||
|
|||
|
|||
def encrypt_secret_config(secret_config: dict[str, Any]) -> str: |
|||
payload = json.dumps(secret_config, sort_keys=True, separators=(",", ":"), ensure_ascii=True).encode("utf-8") |
|||
token = _fernet().encrypt(payload).decode("utf-8") |
|||
return SECRET_PREFIX + token |
|||
|
|||
|
|||
def decrypt_secret_config(raw_value: str) -> dict[str, Any]: |
|||
if not raw_value: |
|||
return {} |
|||
if not raw_value.startswith(SECRET_PREFIX): |
|||
value = json.loads(raw_value) |
|||
return value if isinstance(value, dict) else {} |
|||
|
|||
token = raw_value.removeprefix(SECRET_PREFIX).encode("utf-8") |
|||
try: |
|||
payload = _fernet().decrypt(token) |
|||
except InvalidToken as exc: |
|||
raise SecretConfigError() from exc |
|||
value = json.loads(payload.decode("utf-8")) |
|||
return value if isinstance(value, dict) else {} |
|||
|
|||
|
|||
def list_secret_fields(raw_value: str) -> list[str]: |
|||
return sorted(decrypt_secret_config(raw_value)) |
|||
|
|||
|
|||
def _fernet() -> Fernet: |
|||
digest = hashlib.sha256(get_settings().secret_key.encode("utf-8")).digest() |
|||
key = base64.urlsafe_b64encode(digest) |
|||
return Fernet(key) |
|||
@ -0,0 +1,34 @@ |
|||
from __future__ import annotations |
|||
|
|||
import base64 |
|||
import hashlib |
|||
import hmac |
|||
import secrets |
|||
|
|||
|
|||
def hash_password(password: str, salt: str | None = None) -> str: |
|||
salt_value = salt or secrets.token_hex(16) |
|||
derived = hashlib.pbkdf2_hmac( |
|||
"sha256", |
|||
password.encode("utf-8"), |
|||
salt_value.encode("utf-8"), |
|||
200_000, |
|||
) |
|||
digest = base64.b64encode(derived).decode("utf-8") |
|||
return f"pbkdf2_sha256${salt_value}${digest}" |
|||
|
|||
|
|||
def verify_password(password: str, password_hash: str) -> bool: |
|||
algorithm, salt, digest = password_hash.split("$", 2) |
|||
if algorithm != "pbkdf2_sha256": |
|||
return False |
|||
expected = hash_password(password, salt=salt) |
|||
return hmac.compare_digest(expected, password_hash) |
|||
|
|||
|
|||
def generate_session_token() -> str: |
|||
return f"iron_{secrets.token_urlsafe(32)}" |
|||
|
|||
|
|||
def hash_session_token(token: str) -> str: |
|||
return hashlib.sha256(token.encode("utf-8")).hexdigest() |
|||
@ -0,0 +1,42 @@ |
|||
from __future__ import annotations |
|||
|
|||
from datetime import datetime |
|||
|
|||
from app.models.entities import Blob |
|||
|
|||
|
|||
def build_blob_storage_key(blob: Blob) -> str: |
|||
content_hash = blob.content_hash |
|||
prefix_one = content_hash[:2] if len(content_hash) >= 2 else "xx" |
|||
prefix_two = content_hash[2:4] if len(content_hash) >= 4 else "xx" |
|||
kind = blob.kind or "blob" |
|||
return f"objects/{kind}/sha256/{prefix_one}/{prefix_two}/{content_hash}.blob" |
|||
|
|||
|
|||
def build_preview_artifact_storage_key(*, artifact_type: str, blob: Blob) -> str: |
|||
content_hash = blob.content_hash |
|||
prefix_one = content_hash[:2] if len(content_hash) >= 2 else "xx" |
|||
prefix_two = content_hash[2:4] if len(content_hash) >= 4 else "xx" |
|||
artifact_segment = _normalize_segment(artifact_type) |
|||
return f"objects/preview/{artifact_segment}/sha256/{prefix_one}/{prefix_two}/{content_hash}.blob" |
|||
|
|||
|
|||
def build_export_snapshot_storage_key(*, exported_at: datetime, digest: str) -> str: |
|||
year = f"{exported_at.year:04d}" |
|||
month = f"{exported_at.month:02d}" |
|||
day = f"{exported_at.day:02d}" |
|||
normalized_digest = _normalize_segment(digest) or "snapshot" |
|||
return f"objects/export/metadata/{year}/{month}/{day}/{normalized_digest}.json" |
|||
|
|||
|
|||
def build_manifest_storage_key(*, scope: str, identifier: str) -> str: |
|||
normalized_scope = _normalize_segment(scope) or "general" |
|||
normalized_identifier = _normalize_segment(identifier) or "manifest" |
|||
return f"objects/manifest/{normalized_scope}/{normalized_identifier}.json" |
|||
|
|||
|
|||
def _normalize_segment(value: str) -> str: |
|||
cleaned = "".join(char if char.isalnum() or char in {"-", "_"} else "-" for char in value.strip().lower()) |
|||
while "--" in cleaned: |
|||
cleaned = cleaned.replace("--", "-") |
|||
return cleaned.strip("-_") |
|||
@ -0,0 +1,2 @@ |
|||
"""Database module.""" |
|||
|
|||
@ -0,0 +1,6 @@ |
|||
from sqlalchemy.orm import DeclarativeBase |
|||
|
|||
|
|||
class Base(DeclarativeBase): |
|||
pass |
|||
|
|||
@ -0,0 +1,11 @@ |
|||
from sqlalchemy import text |
|||
from sqlalchemy.ext.asyncio import AsyncEngine |
|||
|
|||
|
|||
async def ping_database(engine: AsyncEngine) -> bool: |
|||
try: |
|||
async with engine.connect() as connection: |
|||
await connection.execute(text("SELECT 1")) |
|||
return True |
|||
except Exception: |
|||
return False |
|||
@ -0,0 +1,37 @@ |
|||
from collections.abc import AsyncIterator |
|||
from functools import lru_cache |
|||
|
|||
from sqlalchemy import event |
|||
from sqlalchemy.ext.asyncio import AsyncEngine, AsyncSession, async_sessionmaker, create_async_engine |
|||
|
|||
from app.core.config import get_settings |
|||
|
|||
|
|||
@lru_cache(maxsize=1) |
|||
def create_engine() -> AsyncEngine: |
|||
settings = get_settings() |
|||
engine = create_async_engine(settings.database_url, future=True) |
|||
|
|||
@event.listens_for(engine.sync_engine, "connect") |
|||
def _set_sqlite_pragma(dbapi_connection, _connection_record) -> None: |
|||
cursor = dbapi_connection.cursor() |
|||
cursor.execute("PRAGMA foreign_keys=ON") |
|||
cursor.close() |
|||
|
|||
return engine |
|||
|
|||
|
|||
def create_session_factory() -> async_sessionmaker[AsyncSession]: |
|||
return async_sessionmaker(create_engine(), class_=AsyncSession, expire_on_commit=False) |
|||
|
|||
|
|||
async def get_db_session() -> AsyncIterator[AsyncSession]: |
|||
session_factory = create_session_factory() |
|||
async with session_factory() as session: |
|||
yield session |
|||
|
|||
|
|||
async def dispose_engine() -> None: |
|||
engine = create_engine() |
|||
await engine.dispose() |
|||
create_engine.cache_clear() |
|||
@ -0,0 +1,2 @@ |
|||
"""Domain layer.""" |
|||
|
|||
@ -0,0 +1,2 @@ |
|||
"""Background job framework.""" |
|||
|
|||
@ -0,0 +1,110 @@ |
|||
from pathlib import Path |
|||
|
|||
from contextlib import asynccontextmanager |
|||
|
|||
import anyio |
|||
from fastapi import Depends, FastAPI, Request |
|||
from fastapi.responses import JSONResponse |
|||
from fastapi.staticfiles import StaticFiles |
|||
|
|||
from app.api.router import api_router |
|||
from app.api.dependencies import require_current_user |
|||
from app.api.routes.tus_uploads import router as tus_upload_router |
|||
from app.web.routes import router as web_router |
|||
from app.core.config import get_settings |
|||
from app.core.exceptions import AppError |
|||
from app.db.session import create_session_factory |
|||
from app.services.directory_service import DirectoryService |
|||
from app.services.auth_service import AuthService |
|||
from app.services.job_service import JobService |
|||
from app.services.local_storage_service import LocalStorageService |
|||
from app.services.policy_service import PolicyService |
|||
from app.services.storage_service import StorageService |
|||
from app.services.upload_service import UploadService |
|||
|
|||
WEB_DIST_DIR = Path(__file__).resolve().parent / "web" / "dist" |
|||
|
|||
|
|||
@asynccontextmanager |
|||
async def lifespan(_app: FastAPI): |
|||
session_factory = create_session_factory() |
|||
|
|||
async with session_factory() as session: |
|||
service = DirectoryService(session) |
|||
await service.ensure_root_directory() |
|||
auth_service = AuthService(session) |
|||
await auth_service.ensure_bootstrap_user() |
|||
upload_service = UploadService(session) |
|||
await upload_service.ensure_temp_dir() |
|||
local_storage_service = LocalStorageService(session) |
|||
await local_storage_service.ensure_backend_ready() |
|||
policy_service = PolicyService(session) |
|||
await policy_service.ensure_default_policies() |
|||
storage_service = StorageService(session) |
|||
await storage_service.ensure_cache_dir() |
|||
|
|||
settings = get_settings() |
|||
|
|||
async def _job_loop() -> None: |
|||
while True: |
|||
async with session_factory() as session: |
|||
service = JobService(session) |
|||
await service.run_pending_jobs(limit=settings.job_batch_size) |
|||
await anyio.sleep(settings.job_poll_interval_seconds) |
|||
|
|||
async with anyio.create_task_group() as task_group: |
|||
task_group.start_soon(_job_loop) |
|||
yield |
|||
task_group.cancel_scope.cancel() |
|||
|
|||
|
|||
def create_app() -> FastAPI: |
|||
settings = get_settings() |
|||
app = FastAPI( |
|||
title=settings.app_name, |
|||
debug=settings.debug, |
|||
version="0.1.0", |
|||
lifespan=lifespan, |
|||
) |
|||
|
|||
@app.exception_handler(AppError) |
|||
async def app_error_handler(_request: Request, exc: AppError) -> JSONResponse: |
|||
status_code = 400 |
|||
if exc.code.endswith("_not_found") or exc.code == "not_found": |
|||
status_code = 404 |
|||
elif exc.code.endswith("_conflict") or exc.code == "conflict": |
|||
status_code = 409 |
|||
elif exc.code == "unauthorized" or exc.code.startswith("auth_") or exc.code == "authentication_failed": |
|||
status_code = 401 |
|||
elif exc.code == "range_not_satisfiable": |
|||
status_code = 416 |
|||
|
|||
return JSONResponse( |
|||
status_code=status_code, |
|||
content={ |
|||
"error": { |
|||
"code": exc.code, |
|||
"message": exc.detail, |
|||
"details": None, |
|||
} |
|||
}, |
|||
) |
|||
|
|||
app.include_router(api_router, prefix=settings.api_prefix) |
|||
app.include_router(web_router) |
|||
app.include_router( |
|||
tus_upload_router, |
|||
prefix="/files", |
|||
tags=["tus"], |
|||
dependencies=[Depends(require_current_user)], |
|||
) |
|||
if (WEB_DIST_DIR / "assets").exists(): |
|||
app.mount( |
|||
"/assets", |
|||
StaticFiles(directory=WEB_DIST_DIR / "assets"), |
|||
name="web-assets", |
|||
) |
|||
return app |
|||
|
|||
|
|||
app = create_app() |
|||
@ -0,0 +1,33 @@ |
|||
"""SQLAlchemy models package.""" |
|||
|
|||
from app.models.entities import AuthSession |
|||
from app.models.entities import Backend |
|||
from app.models.entities import Blob |
|||
from app.models.entities import BlobReplica |
|||
from app.models.entities import CacheEntry |
|||
from app.models.entities import Directory |
|||
from app.models.entities import FileEntry |
|||
from app.models.entities import FileVersion |
|||
from app.models.entities import Job |
|||
from app.models.entities import PlacementPolicy |
|||
from app.models.entities import PreviewArtifact |
|||
from app.models.entities import UploadSession |
|||
from app.models.entities import UploadSessionPart |
|||
from app.models.entities import User |
|||
|
|||
__all__ = [ |
|||
"AuthSession", |
|||
"Backend", |
|||
"Blob", |
|||
"BlobReplica", |
|||
"CacheEntry", |
|||
"Directory", |
|||
"FileEntry", |
|||
"FileVersion", |
|||
"Job", |
|||
"PlacementPolicy", |
|||
"PreviewArtifact", |
|||
"UploadSession", |
|||
"UploadSessionPart", |
|||
"User", |
|||
] |
|||
@ -0,0 +1,275 @@ |
|||
from __future__ import annotations |
|||
|
|||
from datetime import datetime |
|||
from typing import Optional |
|||
|
|||
from sqlalchemy import Boolean, DateTime, ForeignKey, Index, Integer, String, Text, func |
|||
from sqlalchemy.orm import Mapped, mapped_column |
|||
|
|||
from app.db.base import Base |
|||
|
|||
|
|||
class TimestampMixin: |
|||
created_at: Mapped[datetime] = mapped_column( |
|||
DateTime(timezone=True), |
|||
nullable=False, |
|||
server_default=func.now(), |
|||
) |
|||
updated_at: Mapped[datetime] = mapped_column( |
|||
DateTime(timezone=True), |
|||
nullable=False, |
|||
server_default=func.now(), |
|||
onupdate=func.now(), |
|||
) |
|||
|
|||
|
|||
class User(TimestampMixin, Base): |
|||
__tablename__ = "users" |
|||
|
|||
id: Mapped[str] = mapped_column(String, primary_key=True) |
|||
username: Mapped[str] = mapped_column(String, unique=True, nullable=False) |
|||
password_hash: Mapped[str] = mapped_column(String, nullable=False) |
|||
|
|||
|
|||
class AuthSession(TimestampMixin, Base): |
|||
__tablename__ = "auth_sessions" |
|||
__table_args__ = ( |
|||
Index("idx_auth_sessions_user_status", "user_id", "revoked_at", "expires_at"), |
|||
Index("idx_auth_sessions_token_hash", "token_hash", unique=True), |
|||
) |
|||
|
|||
id: Mapped[str] = mapped_column(String, primary_key=True) |
|||
user_id: Mapped[str] = mapped_column(String, ForeignKey("users.id"), nullable=False) |
|||
token_hash: Mapped[str] = mapped_column(String, nullable=False) |
|||
expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False) |
|||
revoked_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True) |
|||
|
|||
|
|||
class Directory(TimestampMixin, Base): |
|||
__tablename__ = "directories" |
|||
__table_args__ = ( |
|||
Index("idx_directories_parent_id", "parent_id"), |
|||
Index("idx_directories_path_key", "path_key"), |
|||
) |
|||
|
|||
id: Mapped[str] = mapped_column(String, primary_key=True) |
|||
parent_id: Mapped[Optional[str]] = mapped_column( |
|||
String, |
|||
ForeignKey("directories.id"), |
|||
nullable=True, |
|||
) |
|||
name: Mapped[str] = mapped_column(String, nullable=False) |
|||
path_key: Mapped[str] = mapped_column(String, unique=True, nullable=False) |
|||
deleted_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True) |
|||
|
|||
|
|||
class FileEntry(TimestampMixin, Base): |
|||
__tablename__ = "file_entries" |
|||
__table_args__ = ( |
|||
Index("idx_file_entries_directory_name", "directory_id", "name", "is_deleted", unique=True), |
|||
Index("idx_file_entries_current_version_id", "current_version_id"), |
|||
Index("idx_file_entries_deleted_at", "deleted_at"), |
|||
) |
|||
|
|||
id: Mapped[str] = mapped_column(String, primary_key=True) |
|||
directory_id: Mapped[str] = mapped_column( |
|||
String, |
|||
ForeignKey("directories.id"), |
|||
nullable=False, |
|||
) |
|||
name: Mapped[str] = mapped_column(String, nullable=False) |
|||
mime_type: Mapped[Optional[str]] = mapped_column(String, nullable=True) |
|||
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False) |
|||
current_version_id: Mapped[Optional[str]] = mapped_column(String, nullable=True) |
|||
is_deleted: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="0") |
|||
deleted_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True) |
|||
|
|||
|
|||
class FileVersion(Base): |
|||
__tablename__ = "file_versions" |
|||
__table_args__ = ( |
|||
Index("idx_file_versions_file_entry_id", "file_entry_id"), |
|||
Index("idx_file_versions_content_hash", "content_hash"), |
|||
) |
|||
|
|||
id: Mapped[str] = mapped_column(String, primary_key=True) |
|||
file_entry_id: Mapped[str] = mapped_column( |
|||
String, |
|||
ForeignKey("file_entries.id"), |
|||
nullable=False, |
|||
) |
|||
content_hash: Mapped[str] = mapped_column(String, nullable=False) |
|||
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False) |
|||
storage_layout: Mapped[str] = mapped_column(String, nullable=False) |
|||
created_at: Mapped[datetime] = mapped_column( |
|||
DateTime(timezone=True), |
|||
nullable=False, |
|||
server_default=func.now(), |
|||
) |
|||
|
|||
|
|||
class Blob(Base): |
|||
__tablename__ = "blobs" |
|||
__table_args__ = ( |
|||
Index("idx_blobs_file_version_index", "file_version_id", "blob_index", unique=True), |
|||
Index("idx_blobs_content_hash", "content_hash"), |
|||
) |
|||
|
|||
id: Mapped[str] = mapped_column(String, primary_key=True) |
|||
file_version_id: Mapped[str] = mapped_column( |
|||
String, |
|||
ForeignKey("file_versions.id"), |
|||
nullable=False, |
|||
) |
|||
blob_index: Mapped[int] = mapped_column(Integer, nullable=False) |
|||
kind: Mapped[str] = mapped_column(String, nullable=False) |
|||
content_hash: Mapped[str] = mapped_column(String, nullable=False) |
|||
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False) |
|||
logical_offset: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0") |
|||
created_at: Mapped[datetime] = mapped_column( |
|||
DateTime(timezone=True), |
|||
nullable=False, |
|||
server_default=func.now(), |
|||
) |
|||
|
|||
|
|||
class Backend(TimestampMixin, Base): |
|||
__tablename__ = "backends" |
|||
__table_args__ = ( |
|||
Index("idx_backends_enabled", "is_enabled"), |
|||
Index("idx_backends_priority", "write_priority", "read_priority"), |
|||
) |
|||
|
|||
id: Mapped[str] = mapped_column(String, primary_key=True) |
|||
name: Mapped[str] = mapped_column(String, unique=True, nullable=False) |
|||
type: Mapped[str] = mapped_column(String, nullable=False) |
|||
stability_class: Mapped[str] = mapped_column(String, nullable=False) |
|||
read_priority: Mapped[int] = mapped_column(Integer, nullable=False) |
|||
write_priority: Mapped[int] = mapped_column(Integer, nullable=False) |
|||
is_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="1") |
|||
public_config_json: Mapped[str] = mapped_column(Text, nullable=False) |
|||
secret_config_json: Mapped[str] = mapped_column(Text, nullable=False) |
|||
capacity_hint_bytes: Mapped[Optional[int]] = mapped_column(Integer, nullable=True) |
|||
last_health_status: Mapped[Optional[str]] = mapped_column(String, nullable=True) |
|||
last_health_checked_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True) |
|||
|
|||
|
|||
class PlacementPolicy(TimestampMixin, Base): |
|||
__tablename__ = "placement_policies" |
|||
__table_args__ = ( |
|||
Index("idx_placement_policies_file_class", "file_class", unique=True), |
|||
) |
|||
|
|||
id: Mapped[str] = mapped_column(String, primary_key=True) |
|||
file_class: Mapped[str] = mapped_column(String, nullable=False) |
|||
require_local: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="1") |
|||
stable_replica_count: Mapped[int] = mapped_column(Integer, nullable=False, default=1, server_default="1") |
|||
opportunistic_replica_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0") |
|||
preferred_backend_ids_json: Mapped[str] = mapped_column(Text, nullable=False, default="[]", server_default="[]") |
|||
excluded_backend_ids_json: Mapped[str] = mapped_column(Text, nullable=False, default="[]", server_default="[]") |
|||
max_non_local_size_bytes: Mapped[Optional[int]] = mapped_column(Integer, nullable=True) |
|||
|
|||
|
|||
class BlobReplica(TimestampMixin, Base): |
|||
__tablename__ = "blob_replicas" |
|||
__table_args__ = ( |
|||
Index("idx_blob_replicas_blob_backend", "blob_id", "backend_id", unique=True), |
|||
Index("idx_blob_replicas_backend_status", "backend_id", "status"), |
|||
Index("idx_blob_replicas_storage_key", "storage_key"), |
|||
) |
|||
|
|||
id: Mapped[str] = mapped_column(String, primary_key=True) |
|||
blob_id: Mapped[str] = mapped_column(String, ForeignKey("blobs.id"), nullable=False) |
|||
backend_id: Mapped[str] = mapped_column(String, ForeignKey("backends.id"), nullable=False) |
|||
storage_key: Mapped[str] = mapped_column(String, nullable=False) |
|||
status: Mapped[str] = mapped_column(String, nullable=False) |
|||
etag: Mapped[Optional[str]] = mapped_column(String, nullable=True) |
|||
checksum: Mapped[Optional[str]] = mapped_column(String, nullable=True) |
|||
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False) |
|||
last_verified_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True) |
|||
|
|||
|
|||
class UploadSession(TimestampMixin, Base): |
|||
__tablename__ = "upload_sessions" |
|||
__table_args__ = ( |
|||
Index("idx_upload_sessions_status", "status"), |
|||
Index("idx_upload_sessions_updated_at", "updated_at"), |
|||
) |
|||
|
|||
id: Mapped[str] = mapped_column(String, primary_key=True) |
|||
directory_id: Mapped[Optional[str]] = mapped_column(String, ForeignKey("directories.id"), nullable=True) |
|||
filename: Mapped[Optional[str]] = mapped_column(String, nullable=True) |
|||
total_size_bytes: Mapped[int] = mapped_column(Integer, nullable=False) |
|||
received_size_bytes: Mapped[int] = mapped_column(Integer, nullable=False) |
|||
status: Mapped[str] = mapped_column(String, nullable=False) |
|||
temp_path: Mapped[str] = mapped_column(String, nullable=False) |
|||
tus_upload_url: Mapped[Optional[str]] = mapped_column(String, nullable=True) |
|||
upload_metadata_json: Mapped[Optional[str]] = mapped_column(Text, nullable=True) |
|||
|
|||
|
|||
class UploadSessionPart(TimestampMixin, Base): |
|||
__tablename__ = "upload_session_parts" |
|||
__table_args__ = ( |
|||
Index("idx_upload_session_parts_unique", "upload_session_id", "part_number", unique=True), |
|||
Index("idx_upload_session_parts_upload_id", "upload_session_id"), |
|||
) |
|||
|
|||
id: Mapped[str] = mapped_column(String, primary_key=True) |
|||
upload_session_id: Mapped[str] = mapped_column( |
|||
String, |
|||
ForeignKey("upload_sessions.id"), |
|||
nullable=False, |
|||
) |
|||
part_number: Mapped[int] = mapped_column(Integer, nullable=False) |
|||
byte_offset: Mapped[int] = mapped_column(Integer, nullable=False) |
|||
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False) |
|||
checksum: Mapped[Optional[str]] = mapped_column(String, nullable=True) |
|||
status: Mapped[str] = mapped_column(String, nullable=False) |
|||
|
|||
|
|||
class Job(TimestampMixin, Base): |
|||
__tablename__ = "jobs" |
|||
__table_args__ = ( |
|||
Index("idx_jobs_status_run_after", "status", "run_after"), |
|||
Index("idx_jobs_kind_status", "kind", "status"), |
|||
) |
|||
|
|||
id: Mapped[str] = mapped_column(String, primary_key=True) |
|||
kind: Mapped[str] = mapped_column(String, nullable=False) |
|||
status: Mapped[str] = mapped_column(String, nullable=False) |
|||
payload_json: Mapped[str] = mapped_column(Text, nullable=False) |
|||
attempt_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0") |
|||
max_attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=5, server_default="5") |
|||
run_after: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False) |
|||
last_error: Mapped[Optional[str]] = mapped_column(Text, nullable=True) |
|||
|
|||
|
|||
class PreviewArtifact(TimestampMixin, Base): |
|||
__tablename__ = "preview_artifacts" |
|||
__table_args__ = ( |
|||
Index("idx_preview_artifacts_version_type", "file_version_id", "artifact_type", unique=True), |
|||
) |
|||
|
|||
id: Mapped[str] = mapped_column(String, primary_key=True) |
|||
file_version_id: Mapped[str] = mapped_column(String, ForeignKey("file_versions.id"), nullable=False) |
|||
artifact_type: Mapped[str] = mapped_column(String, nullable=False) |
|||
blob_id: Mapped[str] = mapped_column(String, ForeignKey("blobs.id"), nullable=False) |
|||
status: Mapped[str] = mapped_column(String, nullable=False) |
|||
|
|||
|
|||
class CacheEntry(TimestampMixin, Base): |
|||
__tablename__ = "cache_entries" |
|||
__table_args__ = ( |
|||
Index("idx_cache_entries_blob_id", "blob_id"), |
|||
Index("idx_cache_entries_last_accessed_at", "last_accessed_at"), |
|||
Index("idx_cache_entries_expires_at", "expires_at"), |
|||
) |
|||
|
|||
id: Mapped[str] = mapped_column(String, primary_key=True) |
|||
blob_id: Mapped[str] = mapped_column(String, ForeignKey("blobs.id"), nullable=False) |
|||
local_path: Mapped[str] = mapped_column(String, nullable=False) |
|||
cache_kind: Mapped[str] = mapped_column(String, nullable=False) |
|||
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False) |
|||
status: Mapped[str] = mapped_column(String, nullable=False) |
|||
last_accessed_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False) |
|||
expires_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True) |
|||
@ -0,0 +1,2 @@ |
|||
"""Repository layer.""" |
|||
|
|||
@ -0,0 +1,40 @@ |
|||
from __future__ import annotations |
|||
|
|||
from datetime import datetime |
|||
|
|||
from sqlalchemy import select |
|||
from sqlalchemy.ext.asyncio import AsyncSession |
|||
|
|||
from app.models.entities import AuthSession |
|||
|
|||
|
|||
class AuthSessionRepository: |
|||
def __init__(self, session: AsyncSession) -> None: |
|||
self.session = session |
|||
|
|||
async def create(self, auth_session: AuthSession) -> AuthSession: |
|||
self.session.add(auth_session) |
|||
await self.session.commit() |
|||
await self.session.refresh(auth_session) |
|||
return auth_session |
|||
|
|||
async def get_active_by_token_hash(self, token_hash: str, now: datetime) -> AuthSession | None: |
|||
result = await self.session.execute( |
|||
select(AuthSession).where( |
|||
AuthSession.token_hash == token_hash, |
|||
AuthSession.revoked_at.is_(None), |
|||
AuthSession.expires_at > now, |
|||
) |
|||
) |
|||
return result.scalar_one_or_none() |
|||
|
|||
async def get_by_token_hash(self, token_hash: str) -> AuthSession | None: |
|||
result = await self.session.execute( |
|||
select(AuthSession).where(AuthSession.token_hash == token_hash) |
|||
) |
|||
return result.scalar_one_or_none() |
|||
|
|||
async def save(self, auth_session: AuthSession) -> AuthSession: |
|||
await self.session.commit() |
|||
await self.session.refresh(auth_session) |
|||
return auth_session |
|||
@ -0,0 +1,64 @@ |
|||
from __future__ import annotations |
|||
|
|||
import json |
|||
|
|||
from sqlalchemy import select |
|||
from sqlalchemy.ext.asyncio import AsyncSession |
|||
|
|||
from app.models.entities import Backend, PlacementPolicy |
|||
|
|||
|
|||
class BackendRepository: |
|||
def __init__(self, session: AsyncSession) -> None: |
|||
self.session = session |
|||
|
|||
async def get_by_name(self, name: str) -> Backend | None: |
|||
result = await self.session.execute(select(Backend).where(Backend.name == name)) |
|||
return result.scalar_one_or_none() |
|||
|
|||
async def get_by_id(self, backend_id: str) -> Backend | None: |
|||
return await self.session.get(Backend, backend_id) |
|||
|
|||
async def create(self, backend: Backend) -> Backend: |
|||
self.session.add(backend) |
|||
await self.session.flush() |
|||
await self.session.refresh(backend) |
|||
return backend |
|||
|
|||
async def list_all(self) -> list[Backend]: |
|||
result = await self.session.execute(select(Backend).order_by(Backend.write_priority.desc(), Backend.name.asc())) |
|||
return list(result.scalars().all()) |
|||
|
|||
async def list_enabled(self) -> list[Backend]: |
|||
result = await self.session.execute( |
|||
select(Backend) |
|||
.where(Backend.is_enabled.is_(True)) |
|||
.order_by(Backend.write_priority.desc(), Backend.read_priority.desc(), Backend.name.asc()) |
|||
) |
|||
return list(result.scalars().all()) |
|||
|
|||
async def save(self, backend: Backend) -> Backend: |
|||
await self.session.flush() |
|||
await self.session.refresh(backend) |
|||
return backend |
|||
|
|||
async def delete(self, backend: Backend) -> None: |
|||
await self.session.delete(backend) |
|||
await self.session.flush() |
|||
|
|||
async def is_referenced_by_policy(self, backend_id: str) -> bool: |
|||
result = await self.session.execute(select(PlacementPolicy)) |
|||
for policy in result.scalars().all(): |
|||
preferred = json.loads(policy.preferred_backend_ids_json or "[]") |
|||
excluded = json.loads(policy.excluded_backend_ids_json or "[]") |
|||
if backend_id in preferred or backend_id in excluded: |
|||
return True |
|||
return False |
|||
|
|||
async def list_enabled_by_type(self, backend_type: str) -> list[Backend]: |
|||
result = await self.session.execute( |
|||
select(Backend) |
|||
.where(Backend.is_enabled.is_(True), Backend.type == backend_type) |
|||
.order_by(Backend.write_priority.desc(), Backend.read_priority.desc(), Backend.name.asc()) |
|||
) |
|||
return list(result.scalars().all()) |
|||
@ -0,0 +1,75 @@ |
|||
from __future__ import annotations |
|||
|
|||
from sqlalchemy import func, select |
|||
from sqlalchemy.ext.asyncio import AsyncSession |
|||
|
|||
from app.models.entities import Backend |
|||
from app.models.entities import BlobReplica |
|||
|
|||
|
|||
class BlobReplicaRepository: |
|||
def __init__(self, session: AsyncSession) -> None: |
|||
self.session = session |
|||
|
|||
async def get_by_blob_and_backend(self, blob_id: str, backend_id: str) -> BlobReplica | None: |
|||
result = await self.session.execute( |
|||
select(BlobReplica).where( |
|||
BlobReplica.blob_id == blob_id, |
|||
BlobReplica.backend_id == backend_id, |
|||
) |
|||
) |
|||
return result.scalar_one_or_none() |
|||
|
|||
async def create(self, replica: BlobReplica) -> BlobReplica: |
|||
self.session.add(replica) |
|||
await self.session.flush() |
|||
await self.session.refresh(replica) |
|||
return replica |
|||
|
|||
async def get_ready_replica_for_blob(self, blob_id: str) -> BlobReplica | None: |
|||
result = await self.session.execute( |
|||
select(BlobReplica) |
|||
.where( |
|||
BlobReplica.blob_id == blob_id, |
|||
BlobReplica.status == "ready", |
|||
) |
|||
.order_by(BlobReplica.created_at.asc()) |
|||
) |
|||
return result.scalar_one_or_none() |
|||
|
|||
async def list_ready_replicas_with_backends(self, blob_id: str) -> list[tuple[BlobReplica, Backend]]: |
|||
result = await self.session.execute( |
|||
select(BlobReplica, Backend) |
|||
.join(Backend, BlobReplica.backend_id == Backend.id) |
|||
.where( |
|||
BlobReplica.blob_id == blob_id, |
|||
BlobReplica.status == "ready", |
|||
Backend.is_enabled.is_(True), |
|||
) |
|||
.order_by(Backend.read_priority.desc(), BlobReplica.created_at.asc()) |
|||
) |
|||
return list(result.all()) |
|||
|
|||
async def list_replicas_with_backends(self, blob_id: str) -> list[tuple[BlobReplica, Backend]]: |
|||
result = await self.session.execute( |
|||
select(BlobReplica, Backend) |
|||
.join(Backend, BlobReplica.backend_id == Backend.id) |
|||
.where(BlobReplica.blob_id == blob_id) |
|||
.order_by(Backend.read_priority.desc(), BlobReplica.created_at.asc()) |
|||
) |
|||
return list(result.all()) |
|||
|
|||
async def list_by_blob(self, blob_id: str) -> list[BlobReplica]: |
|||
result = await self.session.execute(select(BlobReplica).where(BlobReplica.blob_id == blob_id)) |
|||
return list(result.scalars().all()) |
|||
|
|||
async def save(self, replica: BlobReplica) -> BlobReplica: |
|||
await self.session.flush() |
|||
await self.session.refresh(replica) |
|||
return replica |
|||
|
|||
async def count_by_backend(self, backend_id: str) -> int: |
|||
result = await self.session.execute( |
|||
select(func.count()).select_from(BlobReplica).where(BlobReplica.backend_id == backend_id) |
|||
) |
|||
return int(result.scalar_one()) |
|||
@ -0,0 +1,31 @@ |
|||
from __future__ import annotations |
|||
|
|||
from sqlalchemy import select |
|||
from sqlalchemy.ext.asyncio import AsyncSession |
|||
|
|||
from app.models.entities import CacheEntry |
|||
|
|||
|
|||
class CacheEntryRepository: |
|||
def __init__(self, session: AsyncSession) -> None: |
|||
self.session = session |
|||
|
|||
async def get_ready_by_blob(self, blob_id: str) -> CacheEntry | None: |
|||
result = await self.session.execute( |
|||
select(CacheEntry).where( |
|||
CacheEntry.blob_id == blob_id, |
|||
CacheEntry.status == "ready", |
|||
) |
|||
) |
|||
return result.scalar_one_or_none() |
|||
|
|||
async def create(self, entry: CacheEntry) -> CacheEntry: |
|||
self.session.add(entry) |
|||
await self.session.flush() |
|||
await self.session.refresh(entry) |
|||
return entry |
|||
|
|||
async def save(self, entry: CacheEntry) -> CacheEntry: |
|||
await self.session.flush() |
|||
await self.session.refresh(entry) |
|||
return entry |
|||
@ -0,0 +1,58 @@ |
|||
from __future__ import annotations |
|||
|
|||
from sqlalchemy import select |
|||
from sqlalchemy.ext.asyncio import AsyncSession |
|||
|
|||
from app.models.entities import Directory |
|||
|
|||
|
|||
class DirectoryRepository: |
|||
def __init__(self, session: AsyncSession) -> None: |
|||
self.session = session |
|||
|
|||
async def get_by_id(self, directory_id: str) -> Directory | None: |
|||
return await self.session.get(Directory, directory_id) |
|||
|
|||
async def get_root(self) -> Directory | None: |
|||
result = await self.session.execute( |
|||
select(Directory).where(Directory.parent_id.is_(None), Directory.deleted_at.is_(None)) |
|||
) |
|||
return result.scalar_one_or_none() |
|||
|
|||
async def list_children(self, parent_id: str) -> list[Directory]: |
|||
result = await self.session.execute( |
|||
select(Directory) |
|||
.where(Directory.parent_id == parent_id, Directory.deleted_at.is_(None)) |
|||
.order_by(Directory.name.asc()) |
|||
) |
|||
return list(result.scalars().all()) |
|||
|
|||
async def get_by_parent_and_name(self, parent_id: str | None, name: str) -> Directory | None: |
|||
result = await self.session.execute( |
|||
select(Directory).where( |
|||
Directory.parent_id == parent_id, |
|||
Directory.name == name, |
|||
Directory.deleted_at.is_(None), |
|||
) |
|||
) |
|||
return result.scalar_one_or_none() |
|||
|
|||
async def list_descendants_by_path_prefix(self, path_prefix: str) -> list[Directory]: |
|||
like_pattern = f"{path_prefix}/%" |
|||
result = await self.session.execute( |
|||
select(Directory) |
|||
.where(Directory.path_key.like(like_pattern), Directory.deleted_at.is_(None)) |
|||
.order_by(Directory.path_key.asc()) |
|||
) |
|||
return list(result.scalars().all()) |
|||
|
|||
async def create(self, directory: Directory) -> Directory: |
|||
self.session.add(directory) |
|||
await self.session.flush() |
|||
await self.session.refresh(directory) |
|||
return directory |
|||
|
|||
async def save(self, directory: Directory) -> Directory: |
|||
await self.session.flush() |
|||
await self.session.refresh(directory) |
|||
return directory |
|||
@ -0,0 +1,103 @@ |
|||
from __future__ import annotations |
|||
|
|||
from datetime import datetime, UTC |
|||
|
|||
from sqlalchemy import select |
|||
from sqlalchemy.ext.asyncio import AsyncSession |
|||
|
|||
from app.models.entities import Blob, FileEntry, FileVersion |
|||
|
|||
|
|||
class FileRepository: |
|||
def __init__(self, session: AsyncSession) -> None: |
|||
self.session = session |
|||
|
|||
async def get_by_id(self, file_id: str) -> FileEntry | None: |
|||
return await self.session.get(FileEntry, file_id) |
|||
|
|||
async def get_by_directory_and_name(self, directory_id: str, name: str) -> FileEntry | None: |
|||
result = await self.session.execute( |
|||
select(FileEntry).where( |
|||
FileEntry.directory_id == directory_id, |
|||
FileEntry.name == name, |
|||
FileEntry.is_deleted.is_(False), |
|||
FileEntry.deleted_at.is_(None), |
|||
) |
|||
) |
|||
return result.scalar_one_or_none() |
|||
|
|||
async def list_by_directory(self, directory_id: str) -> list[FileEntry]: |
|||
result = await self.session.execute( |
|||
select(FileEntry) |
|||
.where( |
|||
FileEntry.directory_id == directory_id, |
|||
FileEntry.is_deleted.is_(False), |
|||
FileEntry.deleted_at.is_(None), |
|||
) |
|||
.order_by(FileEntry.name.asc()) |
|||
) |
|||
return list(result.scalars().all()) |
|||
|
|||
async def list_deleted(self) -> list[FileEntry]: |
|||
result = await self.session.execute( |
|||
select(FileEntry) |
|||
.where(FileEntry.is_deleted.is_(True), FileEntry.deleted_at.is_not(None)) |
|||
.order_by(FileEntry.deleted_at.desc(), FileEntry.name.asc()) |
|||
) |
|||
return list(result.scalars().all()) |
|||
|
|||
async def list_active(self) -> list[FileEntry]: |
|||
result = await self.session.execute( |
|||
select(FileEntry) |
|||
.where( |
|||
FileEntry.is_deleted.is_(False), |
|||
FileEntry.deleted_at.is_(None), |
|||
) |
|||
.order_by(FileEntry.created_at.asc()) |
|||
) |
|||
return list(result.scalars().all()) |
|||
|
|||
async def create_file_graph( |
|||
self, |
|||
file_entry: FileEntry, |
|||
file_version: FileVersion, |
|||
blob: Blob, |
|||
) -> FileEntry: |
|||
self.session.add(file_entry) |
|||
self.session.add(file_version) |
|||
await self.session.flush() |
|||
self.session.add(blob) |
|||
await self.session.flush() |
|||
await self.session.refresh(file_entry) |
|||
return file_entry |
|||
|
|||
async def get_primary_blob_for_file(self, file_id: str) -> Blob | None: |
|||
result = await self.session.execute( |
|||
select(Blob) |
|||
.join(FileVersion, Blob.file_version_id == FileVersion.id) |
|||
.join(FileEntry, FileVersion.id == FileEntry.current_version_id) |
|||
.where(FileEntry.id == file_id, Blob.blob_index == 0) |
|||
) |
|||
return result.scalar_one_or_none() |
|||
|
|||
async def get_blob_by_id(self, blob_id: str) -> Blob | None: |
|||
return await self.session.get(Blob, blob_id) |
|||
|
|||
async def soft_delete(self, file_entry: FileEntry) -> FileEntry: |
|||
file_entry.is_deleted = True |
|||
file_entry.deleted_at = datetime.now(UTC) |
|||
await self.session.flush() |
|||
await self.session.refresh(file_entry) |
|||
return file_entry |
|||
|
|||
async def restore(self, file_entry: FileEntry) -> FileEntry: |
|||
file_entry.is_deleted = False |
|||
file_entry.deleted_at = None |
|||
await self.session.flush() |
|||
await self.session.refresh(file_entry) |
|||
return file_entry |
|||
|
|||
async def save(self, file_entry: FileEntry) -> FileEntry: |
|||
await self.session.flush() |
|||
await self.session.refresh(file_entry) |
|||
return file_entry |
|||
@ -0,0 +1,53 @@ |
|||
from __future__ import annotations |
|||
|
|||
from datetime import datetime |
|||
|
|||
from sqlalchemy import select |
|||
from sqlalchemy.ext.asyncio import AsyncSession |
|||
|
|||
from app.models.entities import Job |
|||
|
|||
|
|||
class JobRepository: |
|||
def __init__(self, session: AsyncSession) -> None: |
|||
self.session = session |
|||
|
|||
async def create(self, job: Job) -> Job: |
|||
self.session.add(job) |
|||
await self.session.flush() |
|||
await self.session.refresh(job) |
|||
return job |
|||
|
|||
async def get_by_id(self, job_id: str) -> Job | None: |
|||
return await self.session.get(Job, job_id) |
|||
|
|||
async def list_all(self) -> list[Job]: |
|||
result = await self.session.execute(select(Job).order_by(Job.created_at.desc())) |
|||
return list(result.scalars().all()) |
|||
|
|||
async def list_runnable(self, now: datetime, limit: int) -> list[Job]: |
|||
result = await self.session.execute( |
|||
select(Job) |
|||
.where( |
|||
Job.status.in_(["queued", "retryable"]), |
|||
Job.run_after <= now, |
|||
) |
|||
.order_by(Job.run_after.asc(), Job.created_at.asc()) |
|||
.limit(limit) |
|||
) |
|||
return list(result.scalars().all()) |
|||
|
|||
async def find_active_by_kind_and_payload(self, kind: str, payload_json: str) -> Job | None: |
|||
result = await self.session.execute( |
|||
select(Job).where( |
|||
Job.kind == kind, |
|||
Job.payload_json == payload_json, |
|||
Job.status.in_(["queued", "retryable", "running"]), |
|||
) |
|||
) |
|||
return result.scalar_one_or_none() |
|||
|
|||
async def save(self, job: Job) -> Job: |
|||
await self.session.flush() |
|||
await self.session.refresh(job) |
|||
return job |
|||
@ -0,0 +1,32 @@ |
|||
from __future__ import annotations |
|||
|
|||
from sqlalchemy import select |
|||
from sqlalchemy.ext.asyncio import AsyncSession |
|||
|
|||
from app.models.entities import PlacementPolicy |
|||
|
|||
|
|||
class PlacementPolicyRepository: |
|||
def __init__(self, session: AsyncSession) -> None: |
|||
self.session = session |
|||
|
|||
async def get_by_file_class(self, file_class: str) -> PlacementPolicy | None: |
|||
result = await self.session.execute( |
|||
select(PlacementPolicy).where(PlacementPolicy.file_class == file_class) |
|||
) |
|||
return result.scalar_one_or_none() |
|||
|
|||
async def list_all(self) -> list[PlacementPolicy]: |
|||
result = await self.session.execute(select(PlacementPolicy).order_by(PlacementPolicy.file_class.asc())) |
|||
return list(result.scalars().all()) |
|||
|
|||
async def create(self, policy: PlacementPolicy) -> PlacementPolicy: |
|||
self.session.add(policy) |
|||
await self.session.flush() |
|||
await self.session.refresh(policy) |
|||
return policy |
|||
|
|||
async def save(self, policy: PlacementPolicy) -> PlacementPolicy: |
|||
await self.session.flush() |
|||
await self.session.refresh(policy) |
|||
return policy |
|||
@ -0,0 +1,39 @@ |
|||
from __future__ import annotations |
|||
|
|||
from sqlalchemy import select |
|||
from sqlalchemy.ext.asyncio import AsyncSession |
|||
|
|||
from app.models.entities import PreviewArtifact |
|||
|
|||
|
|||
class PreviewArtifactRepository: |
|||
def __init__(self, session: AsyncSession) -> None: |
|||
self.session = session |
|||
|
|||
async def get_by_version_and_type(self, file_version_id: str, artifact_type: str) -> PreviewArtifact | None: |
|||
result = await self.session.execute( |
|||
select(PreviewArtifact).where( |
|||
PreviewArtifact.file_version_id == file_version_id, |
|||
PreviewArtifact.artifact_type == artifact_type, |
|||
) |
|||
) |
|||
return result.scalar_one_or_none() |
|||
|
|||
async def list_by_file_version(self, file_version_id: str) -> list[PreviewArtifact]: |
|||
result = await self.session.execute( |
|||
select(PreviewArtifact) |
|||
.where(PreviewArtifact.file_version_id == file_version_id) |
|||
.order_by(PreviewArtifact.artifact_type.asc()) |
|||
) |
|||
return list(result.scalars().all()) |
|||
|
|||
async def create(self, artifact: PreviewArtifact) -> PreviewArtifact: |
|||
self.session.add(artifact) |
|||
await self.session.flush() |
|||
await self.session.refresh(artifact) |
|||
return artifact |
|||
|
|||
async def save(self, artifact: PreviewArtifact) -> PreviewArtifact: |
|||
await self.session.flush() |
|||
await self.session.refresh(artifact) |
|||
return artifact |
|||
@ -0,0 +1,25 @@ |
|||
from __future__ import annotations |
|||
|
|||
from sqlalchemy import select |
|||
from sqlalchemy.ext.asyncio import AsyncSession |
|||
|
|||
from app.models.entities import UploadSession |
|||
|
|||
|
|||
class UploadRepository: |
|||
def __init__(self, session: AsyncSession) -> None: |
|||
self.session = session |
|||
|
|||
async def get_by_id(self, upload_id: str) -> UploadSession | None: |
|||
return await self.session.get(UploadSession, upload_id) |
|||
|
|||
async def create(self, upload_session: UploadSession) -> UploadSession: |
|||
self.session.add(upload_session) |
|||
await self.session.flush() |
|||
await self.session.refresh(upload_session) |
|||
return upload_session |
|||
|
|||
async def save(self, upload_session: UploadSession) -> UploadSession: |
|||
await self.session.flush() |
|||
await self.session.refresh(upload_session) |
|||
return upload_session |
|||
@ -0,0 +1,30 @@ |
|||
from __future__ import annotations |
|||
|
|||
from sqlalchemy import func |
|||
from sqlalchemy import select |
|||
from sqlalchemy.ext.asyncio import AsyncSession |
|||
|
|||
from app.models.entities import User |
|||
|
|||
|
|||
class UserRepository: |
|||
def __init__(self, session: AsyncSession) -> None: |
|||
self.session = session |
|||
|
|||
async def count_users(self) -> int: |
|||
result = await self.session.execute(select(func.count()).select_from(User)) |
|||
return int(result.scalar_one()) |
|||
|
|||
async def get_by_username(self, username: str) -> User | None: |
|||
result = await self.session.execute(select(User).where(User.username == username)) |
|||
return result.scalar_one_or_none() |
|||
|
|||
async def get_by_id(self, user_id: str) -> User | None: |
|||
result = await self.session.execute(select(User).where(User.id == user_id)) |
|||
return result.scalar_one_or_none() |
|||
|
|||
async def create(self, user: User) -> User: |
|||
self.session.add(user) |
|||
await self.session.commit() |
|||
await self.session.refresh(user) |
|||
return user |
|||
@ -0,0 +1,2 @@ |
|||
"""Pydantic schemas.""" |
|||
|
|||
@ -0,0 +1,34 @@ |
|||
from __future__ import annotations |
|||
|
|||
from datetime import datetime |
|||
|
|||
from pydantic import BaseModel, ConfigDict |
|||
|
|||
|
|||
class UserSummary(BaseModel): |
|||
model_config = ConfigDict(from_attributes=True) |
|||
|
|||
id: str |
|||
username: str |
|||
created_at: datetime |
|||
updated_at: datetime |
|||
|
|||
|
|||
class LoginRequest(BaseModel): |
|||
username: str |
|||
password: str |
|||
|
|||
|
|||
class LoginResponse(BaseModel): |
|||
access_token: str |
|||
token_type: str = "bearer" |
|||
expires_at: datetime |
|||
user: UserSummary |
|||
|
|||
|
|||
class AuthMeResponse(BaseModel): |
|||
user: UserSummary |
|||
|
|||
|
|||
class LogoutResponse(BaseModel): |
|||
success: bool = True |
|||
@ -0,0 +1,56 @@ |
|||
from datetime import datetime |
|||
|
|||
from pydantic import BaseModel, Field |
|||
|
|||
|
|||
class BackendSummary(BaseModel): |
|||
id: str |
|||
name: str |
|||
type: str |
|||
stability_class: str |
|||
read_priority: int |
|||
write_priority: int |
|||
is_enabled: bool |
|||
last_health_status: str | None |
|||
last_health_checked_at: datetime | None |
|||
config: dict | None = None |
|||
secret_fields: list[str] = Field(default_factory=list) |
|||
created_at: datetime |
|||
updated_at: datetime |
|||
|
|||
|
|||
class BackendListResponse(BaseModel): |
|||
items: list[BackendSummary] |
|||
|
|||
|
|||
class CreateBackendRequest(BaseModel): |
|||
name: str |
|||
type: str |
|||
stability_class: str |
|||
read_priority: int |
|||
write_priority: int |
|||
config: dict |
|||
|
|||
|
|||
class UpdateBackendRequest(BaseModel): |
|||
name: str | None = None |
|||
stability_class: str | None = None |
|||
read_priority: int | None = None |
|||
write_priority: int | None = None |
|||
config: dict | None = None |
|||
|
|||
|
|||
class CreateBackendResponse(BaseModel): |
|||
backend: BackendSummary |
|||
|
|||
|
|||
class BackendCheckResponse(BaseModel): |
|||
backend_id: str |
|||
status: str |
|||
checked_at: datetime | None |
|||
detail: str | None = None |
|||
|
|||
|
|||
class BackendMutationResponse(BaseModel): |
|||
ok: bool = True |
|||
backend: BackendSummary |
|||
@ -0,0 +1,13 @@ |
|||
from typing import Any |
|||
|
|||
from pydantic import BaseModel |
|||
|
|||
|
|||
class ErrorDetail(BaseModel): |
|||
code: str |
|||
message: str |
|||
details: Any = None |
|||
|
|||
|
|||
class ErrorResponse(BaseModel): |
|||
error: ErrorDetail |
|||
@ -0,0 +1,46 @@ |
|||
from __future__ import annotations |
|||
|
|||
from datetime import datetime |
|||
|
|||
from pydantic import BaseModel, Field |
|||
|
|||
|
|||
class DirectorySummary(BaseModel): |
|||
id: str |
|||
name: str |
|||
parent_id: str | None |
|||
created_at: datetime |
|||
updated_at: datetime |
|||
|
|||
|
|||
class DirectoryItem(BaseModel): |
|||
kind: str = Field(default="directory") |
|||
id: str |
|||
name: str |
|||
mime_type: str | None = None |
|||
size_bytes: int | None = None |
|||
created_at: datetime |
|||
updated_at: datetime |
|||
|
|||
|
|||
class DirectoryChildrenResponse(BaseModel): |
|||
directory: DirectorySummary |
|||
items: list[DirectoryItem] |
|||
next_cursor: str | None = None |
|||
|
|||
|
|||
class CreateDirectoryRequest(BaseModel): |
|||
parent_id: str |
|||
name: str |
|||
|
|||
|
|||
class CreateDirectoryResponse(BaseModel): |
|||
directory: DirectorySummary |
|||
|
|||
|
|||
class RenameDirectoryRequest(BaseModel): |
|||
name: str |
|||
|
|||
|
|||
class MoveDirectoryRequest(BaseModel): |
|||
target_parent_id: str |
|||
@ -0,0 +1,32 @@ |
|||
from __future__ import annotations |
|||
|
|||
from pydantic import BaseModel |
|||
|
|||
|
|||
class MetadataExportResponse(BaseModel): |
|||
exported_at: str |
|||
data: dict |
|||
|
|||
|
|||
class MetadataImportRequest(BaseModel): |
|||
data: dict |
|||
confirm_replace: bool = False |
|||
validation_token: str | None = None |
|||
|
|||
|
|||
class MetadataImportResponse(BaseModel): |
|||
ok: bool = True |
|||
imported_tables: list[str] |
|||
scheduled_jobs: list[str] = [] |
|||
|
|||
|
|||
class MetadataValidationResponse(BaseModel): |
|||
ok: bool |
|||
issues: list[str] |
|||
|
|||
|
|||
class MetadataRestorePlanResponse(BaseModel): |
|||
ok: bool |
|||
issues: list[str] |
|||
validation_token: str | None |
|||
summary: dict |
|||
@ -0,0 +1,84 @@ |
|||
from __future__ import annotations |
|||
|
|||
from datetime import datetime |
|||
|
|||
from pydantic import BaseModel |
|||
|
|||
|
|||
class FileSummary(BaseModel): |
|||
id: str |
|||
directory_id: str |
|||
name: str |
|||
mime_type: str | None |
|||
size_bytes: int |
|||
current_version_id: str | None |
|||
created_at: datetime |
|||
updated_at: datetime |
|||
|
|||
|
|||
class PreviewArtifactSummary(BaseModel): |
|||
id: str |
|||
artifact_type: str |
|||
blob_id: str |
|||
status: str |
|||
created_at: datetime |
|||
updated_at: datetime |
|||
|
|||
|
|||
class FileReplicaSummary(BaseModel): |
|||
id: str |
|||
blob_id: str |
|||
backend_id: str |
|||
backend_name: str |
|||
backend_type: str |
|||
backend_stability_class: str |
|||
backend_enabled: bool |
|||
storage_key: str |
|||
status: str |
|||
size_bytes: int |
|||
checksum: str | None |
|||
etag: str | None |
|||
last_verified_at: datetime | None |
|||
created_at: datetime |
|||
updated_at: datetime |
|||
|
|||
|
|||
class FilePlacementBackendSummary(BaseModel): |
|||
id: str |
|||
name: str |
|||
type: str |
|||
stability_class: str |
|||
read_priority: int |
|||
write_priority: int |
|||
is_enabled: bool |
|||
|
|||
|
|||
class FileDetailResponse(BaseModel): |
|||
file: FileSummary |
|||
preview_artifacts: list[PreviewArtifactSummary] = [] |
|||
replicas: list[FileReplicaSummary] = [] |
|||
placement_file_class: str | None = None |
|||
desired_backends: list[FilePlacementBackendSummary] = [] |
|||
ready_replica_backend_ids: list[str] = [] |
|||
missing_backend_ids: list[str] = [] |
|||
|
|||
|
|||
class DeletedFileSummary(FileSummary): |
|||
deleted_at: datetime |
|||
|
|||
|
|||
class RecycleBinResponse(BaseModel): |
|||
items: list[DeletedFileSummary] |
|||
|
|||
|
|||
class FileMutationResponse(BaseModel): |
|||
ok: bool = True |
|||
file: FileSummary | DeletedFileSummary |
|||
|
|||
|
|||
class RenameFileRequest(BaseModel): |
|||
name: str |
|||
|
|||
|
|||
class MoveFileRequest(BaseModel): |
|||
target_parent_id: str |
|||
@ -0,0 +1,37 @@ |
|||
from __future__ import annotations |
|||
|
|||
from datetime import datetime |
|||
|
|||
from pydantic import BaseModel |
|||
|
|||
|
|||
class JobSummary(BaseModel): |
|||
id: str |
|||
kind: str |
|||
status: str |
|||
attempt_count: int |
|||
max_attempts: int |
|||
run_after: datetime |
|||
last_error: str | None |
|||
created_at: datetime |
|||
updated_at: datetime |
|||
|
|||
|
|||
class JobListResponse(BaseModel): |
|||
items: list[JobSummary] |
|||
|
|||
|
|||
class JobDetailResponse(BaseModel): |
|||
job: JobSummary |
|||
|
|||
|
|||
class JobMutationResponse(BaseModel): |
|||
ok: bool = True |
|||
job: JobSummary |
|||
|
|||
|
|||
class JobRunPendingResponse(BaseModel): |
|||
ok: bool = True |
|||
processed: int |
|||
completed: int |
|||
failed: int |
|||
@ -0,0 +1,56 @@ |
|||
from __future__ import annotations |
|||
|
|||
from datetime import datetime |
|||
|
|||
from pydantic import BaseModel |
|||
from pydantic import Field |
|||
|
|||
|
|||
class PlacementPolicySummary(BaseModel): |
|||
id: str |
|||
file_class: str |
|||
require_local: bool |
|||
stable_replica_count: int |
|||
opportunistic_replica_count: int |
|||
preferred_backend_ids: list[str] |
|||
excluded_backend_ids: list[str] |
|||
max_non_local_size_bytes: int | None |
|||
created_at: datetime |
|||
updated_at: datetime |
|||
|
|||
|
|||
class PlacementPolicyListResponse(BaseModel): |
|||
items: list[PlacementPolicySummary] |
|||
|
|||
|
|||
class UpsertPlacementPolicyRequest(BaseModel): |
|||
require_local: bool |
|||
stable_replica_count: int |
|||
opportunistic_replica_count: int |
|||
preferred_backend_ids: list[str] = Field(default_factory=list) |
|||
excluded_backend_ids: list[str] = Field(default_factory=list) |
|||
max_non_local_size_bytes: int | None = None |
|||
|
|||
|
|||
class PlacementPolicyMutationResponse(BaseModel): |
|||
ok: bool = True |
|||
policy: PlacementPolicySummary |
|||
reconcile_enqueued_jobs: int = 0 |
|||
|
|||
|
|||
class PlacementDecisionBackendSummary(BaseModel): |
|||
id: str |
|||
name: str |
|||
type: str |
|||
stability_class: str |
|||
read_priority: int |
|||
write_priority: int |
|||
|
|||
|
|||
class PlacementDecisionResponse(BaseModel): |
|||
file_class: str |
|||
mime_type: str | None |
|||
size_bytes: int | None |
|||
non_local_allowed: bool |
|||
policy: PlacementPolicySummary |
|||
selected_backends: list[PlacementDecisionBackendSummary] |
|||
@ -0,0 +1,15 @@ |
|||
from __future__ import annotations |
|||
|
|||
from pydantic import BaseModel |
|||
|
|||
|
|||
class RepairFileResponse(BaseModel): |
|||
ok: bool = True |
|||
file_id: str |
|||
enqueued_jobs: int |
|||
reconciliation_job_kind: str = "reconcile_file_replicas" |
|||
|
|||
|
|||
class EnqueueHealthChecksResponse(BaseModel): |
|||
ok: bool = True |
|||
enqueued_jobs: int |
|||
@ -0,0 +1,11 @@ |
|||
from pydantic import BaseModel |
|||
|
|||
|
|||
class HealthResponse(BaseModel): |
|||
status: str |
|||
|
|||
|
|||
class ReadyResponse(BaseModel): |
|||
status: str |
|||
database: str |
|||
|
|||
@ -0,0 +1,20 @@ |
|||
from pydantic import BaseModel |
|||
|
|||
from app.schemas.file import FileSummary |
|||
|
|||
|
|||
class FinalizeUploadRequest(BaseModel): |
|||
directory_id: str |
|||
filename: str |
|||
|
|||
|
|||
class UploadSummary(BaseModel): |
|||
id: str |
|||
status: str |
|||
received_size_bytes: int |
|||
total_size_bytes: int |
|||
|
|||
|
|||
class FinalizeUploadResponse(BaseModel): |
|||
upload: UploadSummary |
|||
file: FileSummary |
|||
@ -0,0 +1,2 @@ |
|||
"""Application services.""" |
|||
|
|||
@ -0,0 +1,106 @@ |
|||
from __future__ import annotations |
|||
|
|||
from dataclasses import dataclass |
|||
from datetime import UTC, datetime, timedelta |
|||
|
|||
from sqlalchemy.ext.asyncio import AsyncSession |
|||
|
|||
from app.core.config import get_settings |
|||
from app.core.exceptions import AuthenticationFailedError |
|||
from app.core.exceptions import AuthSessionExpiredError |
|||
from app.core.exceptions import UnauthorizedError |
|||
from app.core.ids import new_id |
|||
from app.core.security import generate_session_token |
|||
from app.core.security import hash_password |
|||
from app.core.security import hash_session_token |
|||
from app.core.security import verify_password |
|||
from app.models.entities import AuthSession |
|||
from app.models.entities import User |
|||
from app.repositories.auth_session_repository import AuthSessionRepository |
|||
from app.repositories.user_repository import UserRepository |
|||
|
|||
|
|||
@dataclass |
|||
class LoginResult: |
|||
user: User |
|||
access_token: str |
|||
expires_at: datetime |
|||
|
|||
|
|||
class AuthService: |
|||
def __init__(self, session: AsyncSession) -> None: |
|||
self.session = session |
|||
self.user_repository = UserRepository(session) |
|||
self.auth_session_repository = AuthSessionRepository(session) |
|||
self.settings = get_settings() |
|||
|
|||
async def ensure_bootstrap_user(self) -> User | None: |
|||
if not self.settings.bootstrap_username or not self.settings.bootstrap_password: |
|||
return None |
|||
|
|||
if await self.user_repository.count_users() > 0: |
|||
return await self.user_repository.get_by_username(self.settings.bootstrap_username) |
|||
|
|||
user = User( |
|||
id=new_id("user"), |
|||
username=self.settings.bootstrap_username, |
|||
password_hash=hash_password(self.settings.bootstrap_password), |
|||
) |
|||
return await self.user_repository.create(user) |
|||
|
|||
async def login(self, username: str, password: str) -> LoginResult: |
|||
user = await self.user_repository.get_by_username(username) |
|||
if user is None or not verify_password(password, user.password_hash): |
|||
raise AuthenticationFailedError() |
|||
|
|||
expires_at = self._utc_now_naive() + timedelta(hours=self.settings.auth_session_ttl_hours) |
|||
access_token = generate_session_token() |
|||
auth_session = AuthSession( |
|||
id=new_id("sess"), |
|||
user_id=user.id, |
|||
token_hash=hash_session_token(access_token), |
|||
expires_at=expires_at, |
|||
revoked_at=None, |
|||
) |
|||
await self.auth_session_repository.create(auth_session) |
|||
return LoginResult(user=user, access_token=access_token, expires_at=expires_at) |
|||
|
|||
async def get_current_user(self, bearer_token: str | None) -> User: |
|||
token = self._extract_bearer_token(bearer_token) |
|||
now = self._utc_now_naive() |
|||
auth_session = await self.auth_session_repository.get_active_by_token_hash( |
|||
hash_session_token(token), |
|||
now, |
|||
) |
|||
if auth_session is None: |
|||
existing_session = await self.auth_session_repository.get_by_token_hash(hash_session_token(token)) |
|||
if existing_session is not None and existing_session.expires_at <= now: |
|||
raise AuthSessionExpiredError() |
|||
raise UnauthorizedError() |
|||
|
|||
user = await self.user_repository.get_by_id(auth_session.user_id) |
|||
if user is None: |
|||
raise UnauthorizedError() |
|||
return user |
|||
|
|||
async def logout(self, bearer_token: str | None) -> None: |
|||
token = self._extract_bearer_token(bearer_token) |
|||
auth_session = await self.auth_session_repository.get_by_token_hash(hash_session_token(token)) |
|||
if auth_session is None or auth_session.revoked_at is not None: |
|||
raise UnauthorizedError() |
|||
|
|||
auth_session.revoked_at = self._utc_now_naive() |
|||
await self.auth_session_repository.save(auth_session) |
|||
|
|||
@staticmethod |
|||
def _extract_bearer_token(bearer_token: str | None) -> str: |
|||
if bearer_token is None: |
|||
raise UnauthorizedError() |
|||
scheme, _, token = bearer_token.partition(" ") |
|||
if scheme.lower() != "bearer" or not token: |
|||
raise UnauthorizedError() |
|||
return token |
|||
|
|||
@staticmethod |
|||
def _utc_now_naive() -> datetime: |
|||
return datetime.now(UTC).replace(tzinfo=None) |
|||
@ -0,0 +1,212 @@ |
|||
from __future__ import annotations |
|||
|
|||
import json |
|||
from datetime import UTC, datetime |
|||
|
|||
from sqlalchemy.ext.asyncio import AsyncSession |
|||
|
|||
from app.core.backend_config import ( |
|||
SECRET_CONFIG_KEYS, |
|||
BackendConfigError, |
|||
merge_backend_config, |
|||
split_backend_config, |
|||
supported_backend_types, |
|||
validate_backend_config, |
|||
) |
|||
from app.core.exceptions import ConflictError, NotFoundError |
|||
from app.core.ids import new_id |
|||
from app.core.secret_codec import decrypt_secret_config, encrypt_secret_config |
|||
from app.models.entities import Backend |
|||
from app.repositories.backend_repository import BackendRepository |
|||
from app.repositories.blob_replica_repository import BlobReplicaRepository |
|||
from app.services.local_storage_service import DEFAULT_LOCAL_BACKEND_ID |
|||
from app.services.storage_service import StorageService |
|||
|
|||
|
|||
class BackendNotFoundError(NotFoundError): |
|||
code = "backend_not_found" |
|||
message = "Backend does not exist." |
|||
|
|||
|
|||
class BackendDeleteConflictError(ConflictError): |
|||
code = "backend_delete_conflict" |
|||
message = "Backend cannot be deleted." |
|||
|
|||
|
|||
class BackendDisableConflictError(ConflictError): |
|||
code = "backend_disable_conflict" |
|||
message = "Backend cannot be disabled." |
|||
|
|||
|
|||
SUPPORTED_BACKEND_TYPES = supported_backend_types() |
|||
SUPPORTED_STABILITY_CLASSES = {"local", "stable", "opportunistic"} |
|||
|
|||
|
|||
class BackendService: |
|||
def __init__(self, session: AsyncSession) -> None: |
|||
self.session = session |
|||
self.repository = BackendRepository(session) |
|||
self.blob_replica_repository = BlobReplicaRepository(session) |
|||
self.storage_service = StorageService(session) |
|||
|
|||
async def list_backends(self) -> list[Backend]: |
|||
return await self.repository.list_all() |
|||
|
|||
async def create_backend( |
|||
self, |
|||
*, |
|||
name: str, |
|||
backend_type: str, |
|||
stability_class: str, |
|||
read_priority: int, |
|||
write_priority: int, |
|||
config: dict, |
|||
) -> Backend: |
|||
cleaned_name = name.strip() |
|||
if not cleaned_name: |
|||
raise BackendConfigError("Backend name must not be empty.") |
|||
if backend_type not in SUPPORTED_BACKEND_TYPES: |
|||
raise BackendConfigError(f"Unsupported backend type: {backend_type}.") |
|||
if stability_class not in SUPPORTED_STABILITY_CLASSES: |
|||
raise BackendConfigError(f"Unsupported stability class: {stability_class}.") |
|||
if await self.repository.get_by_name(cleaned_name) is not None: |
|||
raise ConflictError("A backend with this name already exists.") |
|||
|
|||
public_config, secret_config = self._validate_backend_config(backend_type, config) |
|||
backend = Backend( |
|||
id=new_id("bkd"), |
|||
name=cleaned_name, |
|||
type=backend_type, |
|||
stability_class=stability_class, |
|||
read_priority=read_priority, |
|||
write_priority=write_priority, |
|||
is_enabled=True, |
|||
public_config_json=json.dumps(public_config), |
|||
secret_config_json=encrypt_secret_config(secret_config), |
|||
) |
|||
created = await self.repository.create(backend) |
|||
await self.session.commit() |
|||
return created |
|||
|
|||
async def check_backend(self, backend_id: str) -> tuple[Backend, str, str | None]: |
|||
backend = await self._get_backend(backend_id) |
|||
try: |
|||
status, detail = await self.storage_service.check_backend(backend) |
|||
except Exception as exc: |
|||
backend.last_health_status = "unhealthy" |
|||
backend.last_health_checked_at = datetime.now(UTC) |
|||
await self.repository.save(backend) |
|||
await self.session.commit() |
|||
return backend, "unhealthy", str(exc) |
|||
|
|||
backend.last_health_status = status |
|||
backend.last_health_checked_at = datetime.now(UTC) |
|||
await self.repository.save(backend) |
|||
await self.session.commit() |
|||
return backend, status, detail |
|||
|
|||
async def disable_backend(self, backend_id: str) -> Backend: |
|||
backend = await self._get_backend(backend_id) |
|||
if backend.type == "local_directory": |
|||
enabled_local_backends = await self.repository.list_enabled_by_type("local_directory") |
|||
if backend.is_enabled and len(enabled_local_backends) <= 1: |
|||
raise BackendDisableConflictError("At least one enabled local_directory backend is required.") |
|||
backend.is_enabled = False |
|||
saved = await self.repository.save(backend) |
|||
await self.session.commit() |
|||
return saved |
|||
|
|||
async def enable_backend(self, backend_id: str) -> Backend: |
|||
backend = await self._get_backend(backend_id) |
|||
backend.is_enabled = True |
|||
saved = await self.repository.save(backend) |
|||
await self.session.commit() |
|||
return saved |
|||
|
|||
async def delete_backend(self, backend_id: str) -> None: |
|||
backend = await self._get_backend(backend_id) |
|||
if backend.id == DEFAULT_LOCAL_BACKEND_ID: |
|||
raise BackendDeleteConflictError("The default local backend cannot be deleted.") |
|||
if backend.is_enabled: |
|||
raise BackendDeleteConflictError("Disable the backend before deleting it.") |
|||
if await self.repository.is_referenced_by_policy(backend.id): |
|||
raise BackendDeleteConflictError("Backend is still referenced by a placement policy.") |
|||
replica_count = await self.blob_replica_repository.count_by_backend(backend.id) |
|||
if replica_count > 0: |
|||
raise BackendDeleteConflictError("Backend still has stored blob replicas.") |
|||
await self.repository.delete(backend) |
|||
await self.session.commit() |
|||
|
|||
async def update_backend( |
|||
self, |
|||
backend_id: str, |
|||
*, |
|||
name: str | None = None, |
|||
stability_class: str | None = None, |
|||
read_priority: int | None = None, |
|||
write_priority: int | None = None, |
|||
config: dict | None = None, |
|||
) -> Backend: |
|||
backend = await self._get_backend(backend_id) |
|||
|
|||
if name is not None: |
|||
cleaned_name = name.strip() |
|||
if not cleaned_name: |
|||
raise BackendConfigError("Backend name must not be empty.") |
|||
existing = await self.repository.get_by_name(cleaned_name) |
|||
if existing is not None and existing.id != backend.id: |
|||
raise ConflictError("A backend with this name already exists.") |
|||
backend.name = cleaned_name |
|||
|
|||
if stability_class is not None: |
|||
if stability_class not in SUPPORTED_STABILITY_CLASSES: |
|||
raise BackendConfigError(f"Unsupported stability class: {stability_class}.") |
|||
backend.stability_class = stability_class |
|||
|
|||
if read_priority is not None: |
|||
backend.read_priority = read_priority |
|||
if write_priority is not None: |
|||
backend.write_priority = write_priority |
|||
|
|||
if config is not None: |
|||
public_config = json.loads(backend.public_config_json) |
|||
secret_config = decrypt_secret_config(backend.secret_config_json) |
|||
next_config = self._merge_config_update( |
|||
backend.type, |
|||
public_config=public_config, |
|||
secret_config=secret_config, |
|||
config_patch=config, |
|||
) |
|||
next_public_config, next_secret_config = self._validate_backend_config(backend.type, next_config) |
|||
backend.public_config_json = json.dumps(next_public_config) |
|||
backend.secret_config_json = encrypt_secret_config(next_secret_config) |
|||
|
|||
saved = await self.repository.save(backend) |
|||
await self.session.commit() |
|||
return saved |
|||
|
|||
async def _get_backend(self, backend_id: str) -> Backend: |
|||
backend = await self.repository.get_by_id(backend_id) |
|||
if backend is None: |
|||
raise BackendNotFoundError() |
|||
return backend |
|||
|
|||
@staticmethod |
|||
def _validate_backend_config(backend_type: str, config: dict) -> tuple[dict, dict]: |
|||
return split_backend_config(backend_type, config) |
|||
|
|||
@staticmethod |
|||
def _merge_config_update( |
|||
backend_type: str, |
|||
*, |
|||
public_config: dict, |
|||
secret_config: dict, |
|||
config_patch: dict, |
|||
) -> dict: |
|||
merged = merge_backend_config(public_config, secret_config) |
|||
normalized_patch = dict(config_patch) |
|||
for key in list(normalized_patch): |
|||
if key in SECRET_CONFIG_KEYS and not normalized_patch[key]: |
|||
normalized_patch.pop(key) |
|||
merged.update(normalized_patch) |
|||
return validate_backend_config(backend_type, merged) |
|||
@ -0,0 +1,153 @@ |
|||
from __future__ import annotations |
|||
|
|||
from dataclasses import dataclass |
|||
from typing import Literal |
|||
|
|||
from sqlalchemy.ext.asyncio import AsyncSession |
|||
|
|||
from app.core.exceptions import DirectoryConflictError, DirectoryNotFoundError, ValidationError |
|||
from app.core.ids import new_id |
|||
from app.models.entities import Directory, FileEntry |
|||
from app.repositories.directory_repository import DirectoryRepository |
|||
from app.repositories.file_repository import FileRepository |
|||
|
|||
ROOT_DIRECTORY_ID = "dir_root" |
|||
ROOT_DIRECTORY_NAME = "/" |
|||
ROOT_PATH_KEY = "/" |
|||
|
|||
|
|||
@dataclass |
|||
class NamespaceItem: |
|||
kind: Literal["directory", "file"] |
|||
item: Directory | FileEntry |
|||
|
|||
|
|||
@dataclass |
|||
class DirectoryChildrenResult: |
|||
directory: Directory |
|||
children: list[NamespaceItem] |
|||
|
|||
|
|||
class DirectoryService: |
|||
def __init__(self, session: AsyncSession) -> None: |
|||
self.session = session |
|||
self.repository = DirectoryRepository(session) |
|||
self.file_repository = FileRepository(session) |
|||
|
|||
async def ensure_root_directory(self) -> Directory: |
|||
root = await self.repository.get_root() |
|||
if root is not None: |
|||
return root |
|||
|
|||
root = Directory( |
|||
id=ROOT_DIRECTORY_ID, |
|||
parent_id=None, |
|||
name=ROOT_DIRECTORY_NAME, |
|||
path_key=ROOT_PATH_KEY, |
|||
) |
|||
await self.repository.create(root) |
|||
await self.session.commit() |
|||
return root |
|||
|
|||
async def list_children(self, directory_id: str) -> DirectoryChildrenResult: |
|||
directory = await self._get_directory(directory_id) |
|||
child_directories = await self.repository.list_children(directory.id) |
|||
child_files = await self.file_repository.list_by_directory(directory.id) |
|||
|
|||
items = [NamespaceItem(kind="directory", item=child) for child in child_directories] |
|||
items.extend(NamespaceItem(kind="file", item=file_entry) for file_entry in child_files) |
|||
items.sort(key=lambda item: (item.kind != "directory", item.item.name.lower())) |
|||
return DirectoryChildrenResult(directory=directory, children=items) |
|||
|
|||
async def create_directory(self, parent_id: str, name: str) -> Directory: |
|||
cleaned_name = self._validate_directory_name(name) |
|||
|
|||
parent = await self._get_directory(parent_id) |
|||
await self._ensure_name_available(parent.id, cleaned_name) |
|||
|
|||
directory = Directory( |
|||
id=new_id("dir"), |
|||
parent_id=parent.id, |
|||
name=cleaned_name, |
|||
path_key=self._build_path_key(parent.path_key, cleaned_name), |
|||
) |
|||
await self.repository.create(directory) |
|||
await self.session.commit() |
|||
return directory |
|||
|
|||
async def rename_directory(self, directory_id: str, name: str) -> Directory: |
|||
directory = await self._get_directory(directory_id) |
|||
if directory.id == ROOT_DIRECTORY_ID: |
|||
raise ValidationError("Root directory cannot be renamed.") |
|||
|
|||
cleaned_name = self._validate_directory_name(name) |
|||
parent = await self._get_directory(directory.parent_id or ROOT_DIRECTORY_ID) |
|||
if directory.name == cleaned_name: |
|||
return directory |
|||
|
|||
await self._ensure_name_available(parent.id, cleaned_name) |
|||
|
|||
old_path_key = directory.path_key |
|||
directory.name = cleaned_name |
|||
directory.path_key = self._build_path_key(parent.path_key, cleaned_name) |
|||
await self._rewrite_descendant_paths(old_path_key, directory.path_key) |
|||
saved = await self.repository.save(directory) |
|||
await self.session.commit() |
|||
return saved |
|||
|
|||
async def move_directory(self, directory_id: str, target_parent_id: str) -> Directory: |
|||
directory = await self._get_directory(directory_id) |
|||
if directory.id == ROOT_DIRECTORY_ID: |
|||
raise ValidationError("Root directory cannot be moved.") |
|||
|
|||
target_parent = await self._get_directory(target_parent_id) |
|||
if directory.parent_id == target_parent.id: |
|||
return directory |
|||
if target_parent.path_key == directory.path_key or target_parent.path_key.startswith(f"{directory.path_key}/"): |
|||
raise ValidationError("A directory cannot be moved into itself or its descendants.") |
|||
|
|||
await self._ensure_name_available(target_parent.id, directory.name) |
|||
|
|||
old_path_key = directory.path_key |
|||
directory.parent_id = target_parent.id |
|||
directory.path_key = self._build_path_key(target_parent.path_key, directory.name) |
|||
await self._rewrite_descendant_paths(old_path_key, directory.path_key) |
|||
saved = await self.repository.save(directory) |
|||
await self.session.commit() |
|||
return saved |
|||
|
|||
async def _get_directory(self, directory_id: str) -> Directory: |
|||
directory = await self.repository.get_by_id(directory_id) |
|||
if directory is None or directory.deleted_at is not None: |
|||
raise DirectoryNotFoundError() |
|||
return directory |
|||
|
|||
async def _ensure_name_available(self, parent_id: str, name: str) -> None: |
|||
conflicting_file = await self.file_repository.get_by_directory_and_name(parent_id, name) |
|||
if conflicting_file is not None: |
|||
raise DirectoryConflictError("A file with this name already exists in the target location.") |
|||
|
|||
existing = await self.repository.get_by_parent_and_name(parent_id, name) |
|||
if existing is not None: |
|||
raise DirectoryConflictError("A directory with this name already exists in the target location.") |
|||
|
|||
async def _rewrite_descendant_paths(self, old_prefix: str, new_prefix: str) -> None: |
|||
descendants = await self.repository.list_descendants_by_path_prefix(old_prefix) |
|||
for descendant in descendants: |
|||
suffix = descendant.path_key.removeprefix(old_prefix) |
|||
descendant.path_key = f"{new_prefix}{suffix}" |
|||
|
|||
@staticmethod |
|||
def _validate_directory_name(name: str) -> str: |
|||
cleaned_name = name.strip() |
|||
if not cleaned_name: |
|||
raise ValidationError("Directory name must not be empty.") |
|||
if "/" in cleaned_name: |
|||
raise ValidationError("Directory name must not contain '/'.") |
|||
return cleaned_name |
|||
|
|||
@staticmethod |
|||
def _build_path_key(parent_path_key: str, name: str) -> str: |
|||
if parent_path_key == ROOT_PATH_KEY: |
|||
return f"/{name}" |
|||
return f"{parent_path_key}/{name}" |
|||
@ -0,0 +1,234 @@ |
|||
from __future__ import annotations |
|||
|
|||
from datetime import UTC, datetime |
|||
import hashlib |
|||
import json |
|||
from typing import Any |
|||
|
|||
from sqlalchemy import delete |
|||
from sqlalchemy import insert |
|||
from sqlalchemy import select |
|||
from sqlalchemy.ext.asyncio import AsyncSession |
|||
|
|||
from app.core.exceptions import ValidationError |
|||
from app.models.entities import AuthSession |
|||
from app.models.entities import Backend |
|||
from app.models.entities import Blob |
|||
from app.models.entities import BlobReplica |
|||
from app.models.entities import CacheEntry |
|||
from app.models.entities import Directory |
|||
from app.models.entities import FileEntry |
|||
from app.models.entities import FileVersion |
|||
from app.models.entities import Job |
|||
from app.models.entities import PreviewArtifact |
|||
from app.models.entities import UploadSession |
|||
from app.models.entities import UploadSessionPart |
|||
from app.models.entities import User |
|||
from app.services.storage_service import StorageService |
|||
|
|||
|
|||
class ExportService: |
|||
def __init__(self, session: AsyncSession) -> None: |
|||
self.session = session |
|||
self.storage_service = StorageService(session) |
|||
|
|||
async def export_metadata(self) -> dict: |
|||
return { |
|||
"users": await self._dump(User), |
|||
"auth_sessions": await self._dump(AuthSession), |
|||
"directories": await self._dump(Directory), |
|||
"file_entries": await self._dump(FileEntry), |
|||
"file_versions": await self._dump(FileVersion), |
|||
"blobs": await self._dump(Blob), |
|||
"blob_replicas": await self._dump(BlobReplica), |
|||
"backends": await self._dump(Backend), |
|||
"upload_sessions": await self._dump(UploadSession), |
|||
"upload_session_parts": await self._dump(UploadSessionPart), |
|||
"jobs": await self._dump(Job), |
|||
"cache_entries": await self._dump(CacheEntry), |
|||
"preview_artifacts": await self._dump(PreviewArtifact), |
|||
"exported_at": datetime.now(UTC).isoformat(), |
|||
} |
|||
|
|||
async def import_metadata(self, data: dict[str, list[dict[str, Any]]]) -> list[str]: |
|||
validation = await self.validate_metadata_snapshot(data) |
|||
if not validation["ok"]: |
|||
raise ValidationError("Metadata snapshot is invalid and cannot be imported.") |
|||
|
|||
ordered_models = [ |
|||
User, |
|||
AuthSession, |
|||
Directory, |
|||
Backend, |
|||
UploadSession, |
|||
UploadSessionPart, |
|||
FileEntry, |
|||
FileVersion, |
|||
Blob, |
|||
BlobReplica, |
|||
PreviewArtifact, |
|||
Job, |
|||
CacheEntry, |
|||
] |
|||
for model in reversed(ordered_models): |
|||
await self.session.execute(delete(model)) |
|||
|
|||
imported_tables: list[str] = [] |
|||
for model in ordered_models: |
|||
rows = data.get(model.__tablename__, []) |
|||
if not rows: |
|||
continue |
|||
normalized_rows = [self._normalize_row(model, row) for row in rows] |
|||
await self.session.execute(insert(model), normalized_rows) |
|||
imported_tables.append(model.__tablename__) |
|||
|
|||
await self.session.commit() |
|||
return imported_tables |
|||
|
|||
async def build_restore_plan(self, data: dict[str, list[dict[str, Any]]]) -> dict[str, Any]: |
|||
validation = await self.validate_metadata_snapshot(data) |
|||
summary = { |
|||
"users": len(data.get("users", [])), |
|||
"directories": len(data.get("directories", [])), |
|||
"file_entries": len(data.get("file_entries", [])), |
|||
"file_versions": len(data.get("file_versions", [])), |
|||
"blobs": len(data.get("blobs", [])), |
|||
"blob_replicas": len(data.get("blob_replicas", [])), |
|||
"backends": len(data.get("backends", [])), |
|||
"jobs": len(data.get("jobs", [])), |
|||
"preview_artifacts": len(data.get("preview_artifacts", [])), |
|||
} |
|||
validation_token = None |
|||
if validation["ok"]: |
|||
validation_token = self._build_validation_token(data) |
|||
return { |
|||
"ok": validation["ok"], |
|||
"issues": validation["issues"], |
|||
"validation_token": validation_token, |
|||
"summary": summary, |
|||
} |
|||
|
|||
async def validate_metadata_snapshot(self, data: dict[str, list[dict[str, Any]]]) -> dict[str, Any]: |
|||
issues: list[str] = [] |
|||
users = data.get("users", []) |
|||
directories = data.get("directories", []) |
|||
file_entries = data.get("file_entries", []) |
|||
file_versions = data.get("file_versions", []) |
|||
blobs = data.get("blobs", []) |
|||
blob_replicas = data.get("blob_replicas", []) |
|||
backends = data.get("backends", []) |
|||
auth_sessions = data.get("auth_sessions", []) |
|||
upload_sessions = data.get("upload_sessions", []) |
|||
upload_session_parts = data.get("upload_session_parts", []) |
|||
jobs = data.get("jobs", []) |
|||
cache_entries = data.get("cache_entries", []) |
|||
preview_artifacts = data.get("preview_artifacts", []) |
|||
|
|||
if not users: |
|||
issues.append("Snapshot must contain at least one user.") |
|||
if not any(directory.get("id") == "dir_root" for directory in directories): |
|||
issues.append("Snapshot must contain the root directory dir_root.") |
|||
|
|||
user_ids = {row["id"] for row in users if row.get("id")} |
|||
directory_ids = {row["id"] for row in directories if row.get("id")} |
|||
file_entry_ids = {row["id"] for row in file_entries if row.get("id")} |
|||
version_ids = {row["id"] for row in file_versions if row.get("id")} |
|||
blob_ids = {row["id"] for row in blobs if row.get("id")} |
|||
backend_ids = {row["id"] for row in backends if row.get("id")} |
|||
upload_ids = {row["id"] for row in upload_sessions if row.get("id")} |
|||
|
|||
for row in auth_sessions: |
|||
if row.get("user_id") not in user_ids: |
|||
issues.append(f"Auth session {row.get('id')} references missing user.") |
|||
for row in file_entries: |
|||
if row.get("directory_id") not in directory_ids: |
|||
issues.append(f"File entry {row.get('id')} references missing directory.") |
|||
current_version_id = row.get("current_version_id") |
|||
if current_version_id is not None and current_version_id not in version_ids: |
|||
issues.append(f"File entry {row.get('id')} references missing current version.") |
|||
for row in file_versions: |
|||
if row.get("file_entry_id") not in file_entry_ids: |
|||
issues.append(f"File version {row.get('id')} references missing file entry.") |
|||
for row in blobs: |
|||
if row.get("file_version_id") not in version_ids: |
|||
issues.append(f"Blob {row.get('id')} references missing file version.") |
|||
for row in blob_replicas: |
|||
if row.get("blob_id") not in blob_ids: |
|||
issues.append(f"Blob replica {row.get('id')} references missing blob.") |
|||
if row.get("backend_id") not in backend_ids: |
|||
issues.append(f"Blob replica {row.get('id')} references missing backend.") |
|||
for row in upload_sessions: |
|||
directory_id = row.get("directory_id") |
|||
if directory_id is not None and directory_id not in directory_ids: |
|||
issues.append(f"Upload session {row.get('id')} references missing directory.") |
|||
for row in upload_session_parts: |
|||
if row.get("upload_session_id") not in upload_ids: |
|||
issues.append(f"Upload session part {row.get('id')} references missing upload session.") |
|||
for row in preview_artifacts: |
|||
if row.get("file_version_id") not in version_ids: |
|||
issues.append(f"Preview artifact {row.get('id')} references missing file version.") |
|||
if row.get("blob_id") not in blob_ids: |
|||
issues.append(f"Preview artifact {row.get('id')} references missing blob.") |
|||
for row in cache_entries: |
|||
if row.get("blob_id") not in blob_ids: |
|||
issues.append(f"Cache entry {row.get('id')} references missing blob.") |
|||
|
|||
return {"ok": not issues, "issues": issues} |
|||
|
|||
async def verify_runtime_integrity(self) -> dict[str, Any]: |
|||
snapshot = await self.export_metadata() |
|||
validation = await self.validate_metadata_snapshot(snapshot) |
|||
issues = list(validation["issues"]) |
|||
|
|||
backends = {row["id"]: row for row in snapshot["backends"]} |
|||
for replica in snapshot["blob_replicas"]: |
|||
if replica.get("status") != "ready": |
|||
continue |
|||
backend_row = backends.get(replica["backend_id"]) |
|||
if backend_row is None or not backend_row.get("is_enabled", False): |
|||
continue |
|||
backend = await self.session.get(Backend, replica["backend_id"]) |
|||
if backend is None: |
|||
continue |
|||
try: |
|||
adapter = self.storage_service._build_adapter(backend) |
|||
exists = await adapter.object_exists(replica["storage_key"]) |
|||
except Exception as exc: |
|||
issues.append(f"Replica {replica['id']} could not be verified: {exc}") |
|||
continue |
|||
if not exists: |
|||
issues.append(f"Replica {replica['id']} is marked ready but object is missing.") |
|||
|
|||
return {"ok": not issues, "issues": issues} |
|||
|
|||
@staticmethod |
|||
def _build_validation_token(data: dict[str, list[dict[str, Any]]]) -> str: |
|||
canonical = json.dumps(data, sort_keys=True, separators=(",", ":"), ensure_ascii=True) |
|||
return hashlib.sha256(canonical.encode("utf-8")).hexdigest() |
|||
|
|||
async def _dump(self, model) -> list[dict]: |
|||
result = await self.session.execute(select(model)) |
|||
items = [] |
|||
for row in result.scalars().all(): |
|||
payload = {} |
|||
for column in model.__table__.columns: |
|||
value = getattr(row, column.name) |
|||
if isinstance(value, datetime): |
|||
payload[column.name] = value.isoformat() |
|||
else: |
|||
payload[column.name] = value |
|||
items.append(payload) |
|||
return items |
|||
|
|||
@staticmethod |
|||
def _normalize_row(model, row: dict[str, Any]) -> dict[str, Any]: |
|||
normalized = {} |
|||
for column in model.__table__.columns: |
|||
value = row.get(column.name) |
|||
if value is not None and isinstance(value, str): |
|||
python_type = getattr(column.type, "python_type", None) |
|||
if python_type is datetime: |
|||
normalized[column.name] = datetime.fromisoformat(value) |
|||
continue |
|||
normalized[column.name] = value |
|||
return normalized |
|||
@ -0,0 +1,263 @@ |
|||
from __future__ import annotations |
|||
|
|||
from dataclasses import dataclass |
|||
from pathlib import Path |
|||
from typing import NamedTuple |
|||
|
|||
import anyio |
|||
from sqlalchemy.ext.asyncio import AsyncSession |
|||
|
|||
from app.core.exceptions import ( |
|||
DirectoryConflictError, |
|||
DirectoryNotFoundError, |
|||
FileConflictError, |
|||
FileNotFoundError, |
|||
PreviewNotSupportedError, |
|||
RangeNotSatisfiableError, |
|||
ValidationError, |
|||
) |
|||
from app.core.ids import new_id |
|||
from app.models.entities import Blob, FileEntry, FileVersion |
|||
from app.repositories.blob_replica_repository import BlobReplicaRepository |
|||
from app.repositories.directory_repository import DirectoryRepository |
|||
from app.repositories.file_repository import FileRepository |
|||
from app.repositories.preview_artifact_repository import PreviewArtifactRepository |
|||
from app.services.local_storage_service import LocalStorageService |
|||
from app.services.storage_service import StorageService |
|||
|
|||
|
|||
@dataclass |
|||
class CreateFileMetadataInput: |
|||
directory_id: str |
|||
name: str |
|||
mime_type: str | None |
|||
size_bytes: int |
|||
content_hash: str |
|||
storage_layout: str = "single" |
|||
|
|||
|
|||
class FileStreamResult(NamedTuple): |
|||
file_entry: FileEntry |
|||
file_path: Path |
|||
file_size: int |
|||
start: int |
|||
end: int |
|||
|
|||
|
|||
class FileService: |
|||
def __init__(self, session: AsyncSession) -> None: |
|||
self.session = session |
|||
self.file_repository = FileRepository(session) |
|||
self.blob_replica_repository = BlobReplicaRepository(session) |
|||
self.directory_repository = DirectoryRepository(session) |
|||
self.preview_artifact_repository = PreviewArtifactRepository(session) |
|||
self.local_storage_service = LocalStorageService(session) |
|||
self.storage_service = StorageService(session) |
|||
|
|||
async def get_file(self, file_id: str) -> FileEntry: |
|||
file_entry = await self.file_repository.get_by_id(file_id) |
|||
if file_entry is None or file_entry.is_deleted or file_entry.deleted_at is not None: |
|||
raise FileNotFoundError() |
|||
return file_entry |
|||
|
|||
async def get_file_detail(self, file_id: str) -> tuple[FileEntry, list, list, object | None]: |
|||
file_entry = await self.get_file(file_id) |
|||
if file_entry.current_version_id is None: |
|||
return file_entry, [], [], None |
|||
artifacts = await self.preview_artifact_repository.list_by_file_version(file_entry.current_version_id) |
|||
blob = await self.file_repository.get_primary_blob_for_file(file_entry.id) |
|||
if blob is None: |
|||
return file_entry, artifacts, [], None |
|||
replicas = await self.blob_replica_repository.list_replicas_with_backends(blob.id) |
|||
decision = await self.storage_service.get_replication_targets_for_mime_type( |
|||
file_entry.mime_type, |
|||
size_bytes=blob.size_bytes, |
|||
) |
|||
return file_entry, artifacts, replicas, decision |
|||
|
|||
async def list_deleted_files(self) -> list[FileEntry]: |
|||
return await self.file_repository.list_deleted() |
|||
|
|||
async def create_file_metadata(self, payload: CreateFileMetadataInput) -> FileEntry: |
|||
directory = await self.directory_repository.get_by_id(payload.directory_id) |
|||
if directory is None or directory.deleted_at is not None: |
|||
raise DirectoryNotFoundError() |
|||
|
|||
cleaned_name = self._validate_file_name(payload.name) |
|||
await self._ensure_name_available(directory.id, cleaned_name) |
|||
|
|||
file_entry_id = new_id("fil") |
|||
file_version_id = new_id("ver") |
|||
blob_id = new_id("blb") |
|||
|
|||
file_entry = FileEntry( |
|||
id=file_entry_id, |
|||
directory_id=directory.id, |
|||
name=cleaned_name, |
|||
mime_type=payload.mime_type, |
|||
size_bytes=payload.size_bytes, |
|||
current_version_id=file_version_id, |
|||
is_deleted=False, |
|||
) |
|||
file_version = FileVersion( |
|||
id=file_version_id, |
|||
file_entry_id=file_entry_id, |
|||
content_hash=payload.content_hash, |
|||
size_bytes=payload.size_bytes, |
|||
storage_layout=payload.storage_layout, |
|||
) |
|||
blob = Blob( |
|||
id=blob_id, |
|||
file_version_id=file_version_id, |
|||
blob_index=0, |
|||
kind="file", |
|||
content_hash=payload.content_hash, |
|||
size_bytes=payload.size_bytes, |
|||
logical_offset=0, |
|||
) |
|||
|
|||
created = await self.file_repository.create_file_graph(file_entry, file_version, blob) |
|||
await self.session.commit() |
|||
return created |
|||
|
|||
async def rename_file(self, file_id: str, name: str) -> FileEntry: |
|||
file_entry = await self.get_file(file_id) |
|||
cleaned_name = self._validate_file_name(name) |
|||
if file_entry.name == cleaned_name: |
|||
return file_entry |
|||
|
|||
await self._ensure_name_available(file_entry.directory_id, cleaned_name) |
|||
file_entry.name = cleaned_name |
|||
saved = await self.file_repository.save(file_entry) |
|||
await self.session.commit() |
|||
return saved |
|||
|
|||
async def move_file(self, file_id: str, target_parent_id: str) -> FileEntry: |
|||
file_entry = await self.get_file(file_id) |
|||
target_parent = await self.directory_repository.get_by_id(target_parent_id) |
|||
if target_parent is None or target_parent.deleted_at is not None: |
|||
raise DirectoryNotFoundError() |
|||
if file_entry.directory_id == target_parent.id: |
|||
return file_entry |
|||
|
|||
await self._ensure_name_available(target_parent.id, file_entry.name) |
|||
file_entry.directory_id = target_parent.id |
|||
saved = await self.file_repository.save(file_entry) |
|||
await self.session.commit() |
|||
return saved |
|||
|
|||
async def delete_file(self, file_id: str) -> FileEntry: |
|||
file_entry = await self.get_file(file_id) |
|||
deleted = await self.file_repository.soft_delete(file_entry) |
|||
await self.session.commit() |
|||
return deleted |
|||
|
|||
async def restore_file(self, file_id: str) -> FileEntry: |
|||
file_entry = await self.file_repository.get_by_id(file_id) |
|||
if file_entry is None or not file_entry.is_deleted or file_entry.deleted_at is None: |
|||
raise FileNotFoundError() |
|||
|
|||
directory = await self.directory_repository.get_by_id(file_entry.directory_id) |
|||
if directory is None or directory.deleted_at is not None: |
|||
raise DirectoryNotFoundError() |
|||
|
|||
conflicting_directory = await self.directory_repository.get_by_parent_and_name(directory.id, file_entry.name) |
|||
if conflicting_directory is not None: |
|||
raise DirectoryConflictError("A directory with this name already exists in the target location.") |
|||
|
|||
existing_file = await self.file_repository.get_by_directory_and_name(directory.id, file_entry.name) |
|||
if existing_file is not None: |
|||
raise FileConflictError("A file with this name already exists in the target location.") |
|||
|
|||
restored = await self.file_repository.restore(file_entry) |
|||
await self.session.commit() |
|||
return restored |
|||
|
|||
async def resolve_download(self, file_id: str) -> tuple[FileEntry, Path]: |
|||
file_entry = await self.get_file(file_id) |
|||
blob = await self.file_repository.get_primary_blob_for_file(file_entry.id) |
|||
if blob is None: |
|||
raise FileNotFoundError() |
|||
try: |
|||
file_path = await self.storage_service.resolve_best_local_path(blob) |
|||
except Exception as exc: |
|||
raise FileNotFoundError() from exc |
|||
|
|||
if file_path is None: |
|||
raise FileNotFoundError() |
|||
return file_entry, file_path |
|||
|
|||
async def resolve_preview(self, file_id: str) -> tuple[FileEntry, Path]: |
|||
file_entry, file_path = await self.resolve_download(file_id) |
|||
if not self._supports_inline_preview(file_entry.mime_type): |
|||
raise PreviewNotSupportedError() |
|||
return file_entry, file_path |
|||
|
|||
async def resolve_stream(self, file_id: str, range_header: str | None) -> FileStreamResult: |
|||
file_entry, file_path = await self.resolve_download(file_id) |
|||
file_size = await anyio.to_thread.run_sync(lambda: file_path.stat().st_size) |
|||
start, end = self._parse_range(range_header, file_size) |
|||
return FileStreamResult( |
|||
file_entry=file_entry, |
|||
file_path=file_path, |
|||
file_size=file_size, |
|||
start=start, |
|||
end=end, |
|||
) |
|||
|
|||
async def _ensure_name_available(self, directory_id: str, name: str) -> None: |
|||
conflicting_directory = await self.directory_repository.get_by_parent_and_name(directory_id, name) |
|||
if conflicting_directory is not None: |
|||
raise DirectoryConflictError("A directory with this name already exists in the target location.") |
|||
|
|||
existing_file = await self.file_repository.get_by_directory_and_name(directory_id, name) |
|||
if existing_file is not None: |
|||
raise FileConflictError("A file with this name already exists in the target location.") |
|||
|
|||
@staticmethod |
|||
def _validate_file_name(name: str) -> str: |
|||
cleaned_name = name.strip() |
|||
if not cleaned_name: |
|||
raise ValidationError("File name must not be empty.") |
|||
if "/" in cleaned_name: |
|||
raise ValidationError("File name must not contain '/'.") |
|||
return cleaned_name |
|||
|
|||
@staticmethod |
|||
def _parse_range(range_header: str | None, file_size: int) -> tuple[int, int]: |
|||
if file_size <= 0: |
|||
return 0, 0 |
|||
if not range_header: |
|||
return 0, file_size - 1 |
|||
if not range_header.startswith("bytes="): |
|||
raise RangeNotSatisfiableError() |
|||
|
|||
value = range_header.removeprefix("bytes=") |
|||
if "," in value: |
|||
raise RangeNotSatisfiableError() |
|||
|
|||
start_text, _, end_text = value.partition("-") |
|||
try: |
|||
if start_text == "": |
|||
suffix_length = int(end_text) |
|||
if suffix_length <= 0: |
|||
raise RangeNotSatisfiableError() |
|||
start = max(file_size - suffix_length, 0) |
|||
end = file_size - 1 |
|||
else: |
|||
start = int(start_text) |
|||
end = file_size - 1 if end_text == "" else int(end_text) |
|||
except ValueError as exc: |
|||
raise RangeNotSatisfiableError() from exc |
|||
|
|||
if start < 0 or end < start or start >= file_size: |
|||
raise RangeNotSatisfiableError() |
|||
|
|||
end = min(end, file_size - 1) |
|||
return start, end |
|||
|
|||
@staticmethod |
|||
def _supports_inline_preview(mime_type: str | None) -> bool: |
|||
if mime_type is None: |
|||
return False |
|||
return mime_type.startswith("image/") or mime_type == "application/pdf" |
|||
@ -0,0 +1,254 @@ |
|||
from __future__ import annotations |
|||
|
|||
import json |
|||
from dataclasses import dataclass |
|||
from datetime import UTC, datetime, timedelta |
|||
|
|||
from sqlalchemy.ext.asyncio import AsyncSession |
|||
|
|||
from app.core.config import get_settings |
|||
from app.core.exceptions import NotFoundError |
|||
from app.core.ids import new_id |
|||
from app.models.entities import Job |
|||
from app.repositories.backend_repository import BackendRepository |
|||
from app.repositories.blob_replica_repository import BlobReplicaRepository |
|||
from app.repositories.file_repository import FileRepository |
|||
from app.repositories.job_repository import JobRepository |
|||
from app.services.policy_service import PolicyService |
|||
from app.services.preview_service import PreviewService |
|||
from app.services.storage_service import StorageService |
|||
|
|||
|
|||
class JobNotFoundError(NotFoundError): |
|||
code = "job_not_found" |
|||
message = "Job does not exist." |
|||
|
|||
|
|||
@dataclass |
|||
class JobRunResult: |
|||
processed: int |
|||
completed: int |
|||
failed: int |
|||
|
|||
|
|||
class JobService: |
|||
def __init__(self, session: AsyncSession) -> None: |
|||
self.session = session |
|||
self.settings = get_settings() |
|||
self.repository = JobRepository(session) |
|||
self.file_repository = FileRepository(session) |
|||
self.backend_repository = BackendRepository(session) |
|||
self.blob_replica_repository = BlobReplicaRepository(session) |
|||
self.storage_service = StorageService(session) |
|||
self.policy_service = PolicyService(session) |
|||
self.preview_service = PreviewService(session) |
|||
|
|||
async def list_jobs(self) -> list[Job]: |
|||
return await self.repository.list_all() |
|||
|
|||
async def get_job(self, job_id: str) -> Job: |
|||
job = await self.repository.get_by_id(job_id) |
|||
if job is None: |
|||
raise JobNotFoundError() |
|||
return job |
|||
|
|||
async def enqueue_blob_replication(self, blob_id: str, backend_id: str) -> Job: |
|||
payload_json = json.dumps({"blob_id": blob_id, "backend_id": backend_id}, sort_keys=True) |
|||
existing = await self.repository.find_active_by_kind_and_payload("replicate_blob", payload_json) |
|||
if existing is not None: |
|||
return existing |
|||
job = Job( |
|||
id=new_id("job"), |
|||
kind="replicate_blob", |
|||
status="queued", |
|||
payload_json=payload_json, |
|||
run_after=datetime.now(UTC), |
|||
last_error=None, |
|||
) |
|||
created = await self.repository.create(job) |
|||
await self.session.commit() |
|||
return created |
|||
|
|||
async def enqueue_file_reconcile(self, file_id: str) -> Job: |
|||
payload_json = json.dumps({"file_id": file_id}, sort_keys=True) |
|||
existing = await self.repository.find_active_by_kind_and_payload("reconcile_file_replicas", payload_json) |
|||
if existing is not None: |
|||
return existing |
|||
job = Job( |
|||
id=new_id("job"), |
|||
kind="reconcile_file_replicas", |
|||
status="queued", |
|||
payload_json=payload_json, |
|||
run_after=datetime.now(UTC), |
|||
last_error=None, |
|||
) |
|||
created = await self.repository.create(job) |
|||
await self.session.commit() |
|||
return created |
|||
|
|||
async def enqueue_full_reconcile(self) -> Job: |
|||
payload_json = json.dumps({}, sort_keys=True) |
|||
existing = await self.repository.find_active_by_kind_and_payload("reconcile_all_files", payload_json) |
|||
if existing is not None: |
|||
return existing |
|||
job = Job( |
|||
id=new_id("job"), |
|||
kind="reconcile_all_files", |
|||
status="queued", |
|||
payload_json=payload_json, |
|||
run_after=datetime.now(UTC), |
|||
last_error=None, |
|||
) |
|||
created = await self.repository.create(job) |
|||
await self.session.commit() |
|||
return created |
|||
|
|||
async def enqueue_backend_health_check(self, backend_id: str) -> Job: |
|||
payload_json = json.dumps({"backend_id": backend_id}, sort_keys=True) |
|||
existing = await self.repository.find_active_by_kind_and_payload("check_backend_health", payload_json) |
|||
if existing is not None: |
|||
return existing |
|||
job = Job( |
|||
id=new_id("job"), |
|||
kind="check_backend_health", |
|||
status="queued", |
|||
payload_json=payload_json, |
|||
run_after=datetime.now(UTC), |
|||
last_error=None, |
|||
) |
|||
created = await self.repository.create(job) |
|||
await self.session.commit() |
|||
return created |
|||
|
|||
async def enqueue_preview_generation(self, file_id: str) -> Job: |
|||
payload_json = json.dumps({"file_id": file_id}, sort_keys=True) |
|||
existing = await self.repository.find_active_by_kind_and_payload("generate_preview_artifacts", payload_json) |
|||
if existing is not None: |
|||
return existing |
|||
job = Job( |
|||
id=new_id("job"), |
|||
kind="generate_preview_artifacts", |
|||
status="queued", |
|||
payload_json=payload_json, |
|||
run_after=datetime.now(UTC), |
|||
last_error=None, |
|||
) |
|||
created = await self.repository.create(job) |
|||
await self.session.commit() |
|||
return created |
|||
|
|||
async def retry_job(self, job_id: str) -> Job: |
|||
job = await self.get_job(job_id) |
|||
job.status = "retryable" |
|||
job.run_after = datetime.now(UTC) |
|||
job.last_error = None |
|||
saved = await self.repository.save(job) |
|||
await self.session.commit() |
|||
return saved |
|||
|
|||
async def run_pending_jobs(self, limit: int | None = None) -> JobRunResult: |
|||
now = datetime.now(UTC) |
|||
runnable = await self.repository.list_runnable(now, limit or self.settings.job_batch_size) |
|||
completed = 0 |
|||
failed = 0 |
|||
for job in runnable: |
|||
job.status = "running" |
|||
await self.repository.save(job) |
|||
await self.session.commit() |
|||
try: |
|||
await self._execute(job) |
|||
job.status = "completed" |
|||
job.last_error = None |
|||
completed += 1 |
|||
except Exception as exc: |
|||
job.attempt_count += 1 |
|||
job.last_error = str(exc) |
|||
if job.attempt_count >= job.max_attempts: |
|||
job.status = "failed" |
|||
else: |
|||
job.status = "retryable" |
|||
job.run_after = datetime.now(UTC) + timedelta(seconds=min(60, 2**job.attempt_count)) |
|||
failed += 1 |
|||
await self.repository.save(job) |
|||
await self.session.commit() |
|||
return JobRunResult(processed=len(runnable), completed=completed, failed=failed) |
|||
|
|||
async def _execute(self, job: Job) -> None: |
|||
payload = json.loads(job.payload_json) |
|||
if job.kind == "reconcile_all_files": |
|||
files = await self.file_repository.list_active() |
|||
for file_entry in files: |
|||
await self.enqueue_file_reconcile(file_entry.id) |
|||
return |
|||
|
|||
if job.kind == "reconcile_file_replicas": |
|||
file_id = payload["file_id"] |
|||
file_entry = await self.file_repository.get_by_id(file_id) |
|||
if file_entry is None or file_entry.is_deleted or file_entry.deleted_at is not None: |
|||
raise ValueError("File is unavailable.") |
|||
|
|||
blob = await self.file_repository.get_primary_blob_for_file(file_id) |
|||
if blob is None: |
|||
raise ValueError("Primary blob does not exist.") |
|||
|
|||
decision = await self.policy_service.get_placement_decision(file_entry.mime_type, size_bytes=blob.size_bytes) |
|||
desired_backends = decision.backends |
|||
existing_replicas = await self.blob_replica_repository.list_by_blob(blob.id) |
|||
ready_backend_ids = {replica.backend_id for replica in existing_replicas if replica.status == "ready"} |
|||
|
|||
missing_backends = [backend for backend in desired_backends if backend.id not in ready_backend_ids] |
|||
if not missing_backends: |
|||
return |
|||
|
|||
source_path = await self.storage_service.resolve_best_local_path(blob) |
|||
failed_targets: list[str] = [] |
|||
for backend in missing_backends: |
|||
try: |
|||
status, _detail = await self.storage_service.check_backend(backend) |
|||
if status != "healthy": |
|||
failed_targets.append(backend.name) |
|||
continue |
|||
await self.storage_service.persist_blob_to_backend(blob, backend, str(source_path)) |
|||
except Exception: |
|||
failed_targets.append(backend.name) |
|||
|
|||
if failed_targets: |
|||
raise ValueError( |
|||
"Reconciliation is still waiting on backends: " + ", ".join(sorted(set(failed_targets))) |
|||
) |
|||
return |
|||
|
|||
if job.kind == "replicate_blob": |
|||
blob_id = payload["blob_id"] |
|||
backend_id = payload["backend_id"] |
|||
|
|||
blob = await self.file_repository.get_blob_by_id(blob_id) |
|||
if blob is None: |
|||
raise ValueError("Blob does not exist.") |
|||
|
|||
backend = await self.backend_repository.get_by_id(backend_id) |
|||
if backend is None or not backend.is_enabled: |
|||
raise ValueError("Backend is unavailable.") |
|||
|
|||
source_path = await self.storage_service.resolve_best_local_path(blob) |
|||
await self.storage_service.persist_blob_to_backend(blob, backend, str(source_path)) |
|||
return |
|||
|
|||
if job.kind == "check_backend_health": |
|||
backend_id = payload["backend_id"] |
|||
backend = await self.backend_repository.get_by_id(backend_id) |
|||
if backend is None or not backend.is_enabled: |
|||
raise ValueError("Backend is unavailable.") |
|||
status, _detail = await self.storage_service.check_backend(backend) |
|||
backend.last_health_status = status |
|||
backend.last_health_checked_at = datetime.now(UTC) |
|||
await self.backend_repository.save(backend) |
|||
await self.session.commit() |
|||
return |
|||
|
|||
if job.kind == "generate_preview_artifacts": |
|||
file_id = payload["file_id"] |
|||
await self.preview_service.generate_preview_artifacts_for_file(file_id) |
|||
return |
|||
|
|||
raise ValueError(f"Unsupported job kind: {job.kind}") |
|||
@ -0,0 +1,82 @@ |
|||
from __future__ import annotations |
|||
|
|||
import json |
|||
from pathlib import Path |
|||
|
|||
import anyio |
|||
from sqlalchemy.ext.asyncio import AsyncSession |
|||
|
|||
from app.adapters.storage.local import LocalStorageAdapter |
|||
from app.core.config import get_settings |
|||
from app.core.ids import new_id |
|||
from app.core.secret_codec import encrypt_secret_config |
|||
from app.core.storage_layout import build_blob_storage_key |
|||
from app.models.entities import Backend, Blob, BlobReplica |
|||
from app.repositories.backend_repository import BackendRepository |
|||
from app.repositories.blob_replica_repository import BlobReplicaRepository |
|||
|
|||
DEFAULT_LOCAL_BACKEND_ID = "bkd_local_default" |
|||
DEFAULT_LOCAL_BACKEND_NAME = "local-default" |
|||
|
|||
|
|||
class LocalStorageService: |
|||
def __init__(self, session: AsyncSession) -> None: |
|||
self.session = session |
|||
self.settings = get_settings() |
|||
self.backend_repository = BackendRepository(session) |
|||
self.blob_replica_repository = BlobReplicaRepository(session) |
|||
self.adapter = LocalStorageAdapter(self.settings.local_storage_dir) |
|||
|
|||
async def ensure_backend_ready(self) -> Backend: |
|||
await self.adapter.ensure_base_path() |
|||
|
|||
backend = await self.backend_repository.get_by_name(DEFAULT_LOCAL_BACKEND_NAME) |
|||
if backend is not None: |
|||
return backend |
|||
|
|||
backend = Backend( |
|||
id=DEFAULT_LOCAL_BACKEND_ID, |
|||
name=DEFAULT_LOCAL_BACKEND_NAME, |
|||
type="local_directory", |
|||
stability_class="local", |
|||
read_priority=100, |
|||
write_priority=100, |
|||
is_enabled=True, |
|||
public_config_json=json.dumps({"base_path": self.settings.local_storage_dir}), |
|||
secret_config_json=encrypt_secret_config({}), |
|||
) |
|||
await self.backend_repository.create(backend) |
|||
await self.session.commit() |
|||
return backend |
|||
|
|||
async def persist_blob_from_temp(self, blob: Blob, temp_path: str) -> BlobReplica: |
|||
backend = await self.ensure_backend_ready() |
|||
existing = await self.blob_replica_repository.get_by_blob_and_backend(blob.id, backend.id) |
|||
if existing is not None: |
|||
return existing |
|||
|
|||
storage_key = self._build_storage_key(blob) |
|||
await self.adapter.put_file(temp_path, storage_key) |
|||
replica = BlobReplica( |
|||
id=new_id("rep"), |
|||
blob_id=blob.id, |
|||
backend_id=backend.id, |
|||
storage_key=storage_key, |
|||
status="ready", |
|||
checksum=blob.content_hash, |
|||
size_bytes=blob.size_bytes, |
|||
) |
|||
created = await self.blob_replica_repository.create(replica) |
|||
await self.session.commit() |
|||
|
|||
stored_path = await self.adapter.resolve_path(storage_key) |
|||
await anyio.to_thread.run_sync(stored_path.exists) |
|||
return created |
|||
|
|||
async def resolve_replica_path(self, replica: BlobReplica) -> Path: |
|||
await self.ensure_backend_ready() |
|||
return await self.adapter.resolve_path(replica.storage_key) |
|||
|
|||
@staticmethod |
|||
def _build_storage_key(blob: Blob) -> str: |
|||
return build_blob_storage_key(blob) |
|||
@ -0,0 +1,266 @@ |
|||
from __future__ import annotations |
|||
|
|||
import json |
|||
from dataclasses import dataclass |
|||
|
|||
from sqlalchemy.ext.asyncio import AsyncSession |
|||
|
|||
from app.core.exceptions import ValidationError |
|||
from app.core.ids import new_id |
|||
from app.models.entities import Backend, PlacementPolicy |
|||
from app.repositories.backend_repository import BackendRepository |
|||
from app.repositories.placement_policy_repository import PlacementPolicyRepository |
|||
|
|||
DEFAULT_PLACEMENT_POLICIES = { |
|||
"document": { |
|||
"require_local": True, |
|||
"stable_replica_count": 1, |
|||
"opportunistic_replica_count": 0, |
|||
"preferred_backend_ids": [], |
|||
"excluded_backend_ids": [], |
|||
"max_non_local_size_bytes": None, |
|||
}, |
|||
"image": { |
|||
"require_local": True, |
|||
"stable_replica_count": 1, |
|||
"opportunistic_replica_count": 1, |
|||
"preferred_backend_ids": [], |
|||
"excluded_backend_ids": [], |
|||
"max_non_local_size_bytes": None, |
|||
}, |
|||
"video": { |
|||
"require_local": True, |
|||
"stable_replica_count": 1, |
|||
"opportunistic_replica_count": 0, |
|||
"preferred_backend_ids": [], |
|||
"excluded_backend_ids": [], |
|||
"max_non_local_size_bytes": None, |
|||
}, |
|||
"other": { |
|||
"require_local": True, |
|||
"stable_replica_count": 1, |
|||
"opportunistic_replica_count": 0, |
|||
"preferred_backend_ids": [], |
|||
"excluded_backend_ids": [], |
|||
"max_non_local_size_bytes": None, |
|||
}, |
|||
} |
|||
ALLOWED_FILE_CLASSES = frozenset(DEFAULT_PLACEMENT_POLICIES.keys()) |
|||
|
|||
|
|||
@dataclass |
|||
class PlacementDecision: |
|||
file_class: str |
|||
policy: PlacementPolicy |
|||
backends: list[Backend] |
|||
non_local_allowed: bool |
|||
|
|||
|
|||
class PlacementPolicyValidationError(ValidationError): |
|||
code = "placement_policy_invalid" |
|||
message = "Placement policy is invalid." |
|||
|
|||
|
|||
class PolicyService: |
|||
def __init__(self, session: AsyncSession) -> None: |
|||
self.session = session |
|||
self.backend_repository = BackendRepository(session) |
|||
self.repository = PlacementPolicyRepository(session) |
|||
|
|||
async def ensure_default_policies(self) -> None: |
|||
for file_class, values in DEFAULT_PLACEMENT_POLICIES.items(): |
|||
existing = await self.repository.get_by_file_class(file_class) |
|||
if existing is not None: |
|||
continue |
|||
policy = PlacementPolicy( |
|||
id=new_id("plc"), |
|||
file_class=file_class, |
|||
require_local=values["require_local"], |
|||
stable_replica_count=values["stable_replica_count"], |
|||
opportunistic_replica_count=values["opportunistic_replica_count"], |
|||
preferred_backend_ids_json=json.dumps(values["preferred_backend_ids"]), |
|||
excluded_backend_ids_json=json.dumps(values["excluded_backend_ids"]), |
|||
max_non_local_size_bytes=values["max_non_local_size_bytes"], |
|||
) |
|||
await self.repository.create(policy) |
|||
await self.session.commit() |
|||
|
|||
async def list_policies(self) -> list[PlacementPolicy]: |
|||
await self.ensure_default_policies() |
|||
return await self.repository.list_all() |
|||
|
|||
async def upsert_policy( |
|||
self, |
|||
*, |
|||
file_class: str, |
|||
require_local: bool, |
|||
stable_replica_count: int, |
|||
opportunistic_replica_count: int, |
|||
preferred_backend_ids: list[str], |
|||
excluded_backend_ids: list[str], |
|||
max_non_local_size_bytes: int | None, |
|||
) -> PlacementPolicy: |
|||
await self.ensure_default_policies() |
|||
self._validate_file_class(file_class) |
|||
await self._validate_policy_inputs( |
|||
require_local=require_local, |
|||
stable_replica_count=stable_replica_count, |
|||
opportunistic_replica_count=opportunistic_replica_count, |
|||
preferred_backend_ids=preferred_backend_ids, |
|||
excluded_backend_ids=excluded_backend_ids, |
|||
max_non_local_size_bytes=max_non_local_size_bytes, |
|||
) |
|||
existing = await self.repository.get_by_file_class(file_class) |
|||
if existing is None: |
|||
existing = PlacementPolicy( |
|||
id=new_id("plc"), |
|||
file_class=file_class, |
|||
require_local=require_local, |
|||
stable_replica_count=stable_replica_count, |
|||
opportunistic_replica_count=opportunistic_replica_count, |
|||
preferred_backend_ids_json=json.dumps(preferred_backend_ids), |
|||
excluded_backend_ids_json=json.dumps(excluded_backend_ids), |
|||
max_non_local_size_bytes=max_non_local_size_bytes, |
|||
) |
|||
await self.repository.create(existing) |
|||
else: |
|||
existing.require_local = require_local |
|||
existing.stable_replica_count = stable_replica_count |
|||
existing.opportunistic_replica_count = opportunistic_replica_count |
|||
existing.preferred_backend_ids_json = json.dumps(preferred_backend_ids) |
|||
existing.excluded_backend_ids_json = json.dumps(excluded_backend_ids) |
|||
existing.max_non_local_size_bytes = max_non_local_size_bytes |
|||
await self.repository.save(existing) |
|||
await self.session.commit() |
|||
return existing |
|||
|
|||
async def get_placement_decision(self, mime_type: str | None, size_bytes: int | None = None) -> PlacementDecision: |
|||
await self.ensure_default_policies() |
|||
file_class = self.classify_mime_type(mime_type) |
|||
policy = await self.repository.get_by_file_class(file_class) |
|||
if policy is None: |
|||
raise ValueError(f"Placement policy is missing for file class: {file_class}") |
|||
|
|||
preferred_backend_ids = self._decode_backend_ids(policy.preferred_backend_ids_json) |
|||
excluded_backend_ids = set(self._decode_backend_ids(policy.excluded_backend_ids_json)) |
|||
backends = await self.backend_repository.list_enabled() |
|||
eligible_backends = [backend for backend in backends if backend.id not in excluded_backend_ids] |
|||
local_backends = self._sort_backends( |
|||
[backend for backend in eligible_backends if backend.type == "local_directory"], |
|||
preferred_backend_ids, |
|||
) |
|||
stable_backends = [ |
|||
backend |
|||
for backend in eligible_backends |
|||
if backend.type != "local_directory" and backend.stability_class == "stable" |
|||
] |
|||
opportunistic_backends = [ |
|||
backend |
|||
for backend in eligible_backends |
|||
if backend.type != "local_directory" and backend.stability_class == "opportunistic" |
|||
] |
|||
stable_backends = self._sort_backends(stable_backends, preferred_backend_ids) |
|||
opportunistic_backends = self._sort_backends(opportunistic_backends, preferred_backend_ids) |
|||
|
|||
selected: list[Backend] = [] |
|||
if policy.require_local and local_backends: |
|||
selected.append(local_backends[0]) |
|||
|
|||
non_local_allowed = ( |
|||
policy.max_non_local_size_bytes is None |
|||
or size_bytes is None |
|||
or size_bytes <= policy.max_non_local_size_bytes |
|||
) |
|||
if non_local_allowed: |
|||
selected.extend(stable_backends[: policy.stable_replica_count]) |
|||
selected.extend(opportunistic_backends[: policy.opportunistic_replica_count]) |
|||
|
|||
deduped: list[Backend] = [] |
|||
seen = set() |
|||
for backend in selected: |
|||
if backend.id in seen: |
|||
continue |
|||
seen.add(backend.id) |
|||
deduped.append(backend) |
|||
|
|||
return PlacementDecision( |
|||
file_class=file_class, |
|||
policy=policy, |
|||
backends=deduped, |
|||
non_local_allowed=non_local_allowed, |
|||
) |
|||
|
|||
@staticmethod |
|||
def classify_mime_type(mime_type: str | None) -> str: |
|||
if mime_type is None: |
|||
return "other" |
|||
if mime_type.startswith("image/"): |
|||
return "image" |
|||
if mime_type.startswith("video/"): |
|||
return "video" |
|||
if mime_type == "application/pdf" or mime_type.startswith("text/"): |
|||
return "document" |
|||
return "other" |
|||
|
|||
@staticmethod |
|||
def _decode_backend_ids(raw_value: str | None) -> list[str]: |
|||
if not raw_value: |
|||
return [] |
|||
value = json.loads(raw_value) |
|||
if not isinstance(value, list): |
|||
return [] |
|||
return [item for item in value if isinstance(item, str) and item] |
|||
|
|||
@staticmethod |
|||
def _sort_backends(backends: list[Backend], preferred_backend_ids: list[str]) -> list[Backend]: |
|||
preferred_positions = {backend_id: index for index, backend_id in enumerate(preferred_backend_ids)} |
|||
return sorted( |
|||
backends, |
|||
key=lambda backend: ( |
|||
0 if backend.id in preferred_positions else 1, |
|||
preferred_positions.get(backend.id, 0), |
|||
-backend.write_priority, |
|||
-backend.read_priority, |
|||
backend.name, |
|||
), |
|||
) |
|||
|
|||
@staticmethod |
|||
def _validate_file_class(file_class: str) -> None: |
|||
if file_class not in ALLOWED_FILE_CLASSES: |
|||
raise PlacementPolicyValidationError( |
|||
"Unsupported file class. Allowed values: " + ", ".join(sorted(ALLOWED_FILE_CLASSES)) |
|||
) |
|||
|
|||
async def _validate_policy_inputs( |
|||
self, |
|||
*, |
|||
require_local: bool, |
|||
stable_replica_count: int, |
|||
opportunistic_replica_count: int, |
|||
preferred_backend_ids: list[str], |
|||
excluded_backend_ids: list[str], |
|||
max_non_local_size_bytes: int | None, |
|||
) -> None: |
|||
del require_local |
|||
if stable_replica_count < 0 or opportunistic_replica_count < 0: |
|||
raise PlacementPolicyValidationError("Replica counts must be zero or greater.") |
|||
if max_non_local_size_bytes is not None and max_non_local_size_bytes <= 0: |
|||
raise PlacementPolicyValidationError("max_non_local_size_bytes must be greater than zero.") |
|||
|
|||
preferred_set = set(preferred_backend_ids) |
|||
excluded_set = set(excluded_backend_ids) |
|||
if len(preferred_set) != len(preferred_backend_ids): |
|||
raise PlacementPolicyValidationError("preferred_backend_ids must not contain duplicates.") |
|||
if len(excluded_set) != len(excluded_backend_ids): |
|||
raise PlacementPolicyValidationError("excluded_backend_ids must not contain duplicates.") |
|||
overlap = preferred_set & excluded_set |
|||
if overlap: |
|||
raise PlacementPolicyValidationError("A backend cannot be both preferred and excluded.") |
|||
|
|||
known_backend_ids = {backend.id for backend in await self.backend_repository.list_all()} |
|||
unknown_backend_ids = sorted((preferred_set | excluded_set) - known_backend_ids) |
|||
if unknown_backend_ids: |
|||
raise PlacementPolicyValidationError( |
|||
"Unknown backend ids referenced by placement policy: " + ", ".join(unknown_backend_ids) |
|||
) |
|||
@ -0,0 +1,65 @@ |
|||
from __future__ import annotations |
|||
|
|||
from app.core.ids import new_id |
|||
from app.models.entities import PreviewArtifact |
|||
from app.repositories.file_repository import FileRepository |
|||
from app.repositories.preview_artifact_repository import PreviewArtifactRepository |
|||
|
|||
|
|||
class PreviewService: |
|||
def __init__(self, session) -> None: |
|||
self.session = session |
|||
self.file_repository = FileRepository(session) |
|||
self.preview_artifact_repository = PreviewArtifactRepository(session) |
|||
|
|||
async def get_preview_artifacts_for_file(self, file_id: str) -> list[PreviewArtifact]: |
|||
file_entry = await self.file_repository.get_by_id(file_id) |
|||
if file_entry is None or file_entry.current_version_id is None: |
|||
return [] |
|||
return await self.preview_artifact_repository.list_by_file_version(file_entry.current_version_id) |
|||
|
|||
async def generate_preview_artifacts_for_file(self, file_id: str) -> int: |
|||
file_entry = await self.file_repository.get_by_id(file_id) |
|||
if file_entry is None or file_entry.current_version_id is None: |
|||
raise ValueError("File is unavailable.") |
|||
blob = await self.file_repository.get_primary_blob_for_file(file_id) |
|||
if blob is None: |
|||
raise ValueError("Primary blob does not exist.") |
|||
|
|||
artifact_types = self._select_artifact_types(file_entry.mime_type) |
|||
created = 0 |
|||
for artifact_type in artifact_types: |
|||
existing = await self.preview_artifact_repository.get_by_version_and_type( |
|||
file_entry.current_version_id, |
|||
artifact_type, |
|||
) |
|||
if existing is not None: |
|||
if existing.status != "ready": |
|||
existing.status = "ready" |
|||
existing.blob_id = blob.id |
|||
await self.preview_artifact_repository.save(existing) |
|||
continue |
|||
|
|||
artifact = PreviewArtifact( |
|||
id=new_id("prv"), |
|||
file_version_id=file_entry.current_version_id, |
|||
artifact_type=artifact_type, |
|||
blob_id=blob.id, |
|||
status="ready", |
|||
) |
|||
await self.preview_artifact_repository.create(artifact) |
|||
created += 1 |
|||
await self.session.commit() |
|||
return created |
|||
|
|||
@staticmethod |
|||
def _select_artifact_types(mime_type: str | None) -> list[str]: |
|||
if mime_type is None: |
|||
return [] |
|||
if mime_type.startswith("image/"): |
|||
return ["inline_preview", "thumbnail"] |
|||
if mime_type == "application/pdf": |
|||
return ["inline_preview", "poster"] |
|||
if mime_type.startswith("video/"): |
|||
return ["poster"] |
|||
return [] |
|||
@ -0,0 +1,36 @@ |
|||
from __future__ import annotations |
|||
|
|||
from sqlalchemy.ext.asyncio import AsyncSession |
|||
|
|||
from app.core.exceptions import FileNotFoundError |
|||
from app.repositories.backend_repository import BackendRepository |
|||
from app.repositories.file_repository import FileRepository |
|||
from app.services.job_service import JobService |
|||
|
|||
|
|||
class RepairService: |
|||
def __init__(self, session: AsyncSession) -> None: |
|||
self.session = session |
|||
self.file_repository = FileRepository(session) |
|||
self.backend_repository = BackendRepository(session) |
|||
self.job_service = JobService(session) |
|||
|
|||
async def enqueue_repairs_for_file(self, file_id: str) -> int: |
|||
file_entry = await self.file_repository.get_by_id(file_id) |
|||
if file_entry is None or file_entry.is_deleted or file_entry.deleted_at is not None: |
|||
raise FileNotFoundError() |
|||
|
|||
await self.job_service.enqueue_file_reconcile(file_id) |
|||
return 1 |
|||
|
|||
async def enqueue_backend_health_checks(self) -> int: |
|||
backends = await self.backend_repository.list_enabled() |
|||
enqueued = 0 |
|||
for backend in backends: |
|||
await self.job_service.enqueue_backend_health_check(backend.id) |
|||
enqueued += 1 |
|||
return enqueued |
|||
|
|||
async def enqueue_full_reconcile(self) -> int: |
|||
await self.job_service.enqueue_full_reconcile() |
|||
return 1 |
|||
@ -0,0 +1,152 @@ |
|||
from __future__ import annotations |
|||
|
|||
import builtins |
|||
import json |
|||
from datetime import UTC, datetime |
|||
from pathlib import Path |
|||
|
|||
import anyio |
|||
from sqlalchemy.ext.asyncio import AsyncSession |
|||
|
|||
from app.adapters.storage.registry import build_storage_adapter |
|||
from app.core.backend_config import merge_backend_config |
|||
from app.core.config import get_settings |
|||
from app.core.ids import new_id |
|||
from app.core.secret_codec import decrypt_secret_config |
|||
from app.core.storage_layout import build_blob_storage_key |
|||
from app.models.entities import Backend, Blob, BlobReplica, CacheEntry |
|||
from app.repositories.backend_repository import BackendRepository |
|||
from app.repositories.blob_replica_repository import BlobReplicaRepository |
|||
from app.repositories.cache_entry_repository import CacheEntryRepository |
|||
from app.services.policy_service import PlacementDecision, PolicyService |
|||
|
|||
|
|||
class StorageService: |
|||
def __init__(self, session: AsyncSession) -> None: |
|||
self.session = session |
|||
self.settings = get_settings() |
|||
self.backend_repository = BackendRepository(session) |
|||
self.blob_replica_repository = BlobReplicaRepository(session) |
|||
self.cache_entry_repository = CacheEntryRepository(session) |
|||
self.policy_service = PolicyService(session) |
|||
|
|||
async def ensure_cache_dir(self) -> Path: |
|||
cache_dir = Path(self.settings.cache_dir) |
|||
await anyio.to_thread.run_sync(lambda: cache_dir.mkdir(parents=True, exist_ok=True)) |
|||
return cache_dir |
|||
|
|||
async def get_replication_targets_for_mime_type( |
|||
self, |
|||
mime_type: str | None, |
|||
size_bytes: int | None = None, |
|||
) -> PlacementDecision: |
|||
return await self.policy_service.get_placement_decision(mime_type, size_bytes=size_bytes) |
|||
|
|||
async def persist_blob_to_backend(self, blob: Blob, backend: Backend, source_path: str) -> BlobReplica: |
|||
existing = await self.blob_replica_repository.get_by_blob_and_backend(blob.id, backend.id) |
|||
storage_key = self._build_storage_key(blob) |
|||
adapter = self._build_adapter(backend) |
|||
existing_is_usable = False |
|||
if existing is not None and existing.status == "ready": |
|||
try: |
|||
existing_is_usable = await adapter.object_exists(storage_key) |
|||
except Exception: |
|||
existing_is_usable = False |
|||
if existing_is_usable: |
|||
return existing |
|||
|
|||
result = await adapter.put_file(source_path, storage_key) |
|||
etag = result.get("etag") |
|||
|
|||
replica = existing or BlobReplica( |
|||
id=new_id("rep"), |
|||
blob_id=blob.id, |
|||
backend_id=backend.id, |
|||
storage_key=storage_key, |
|||
size_bytes=blob.size_bytes, |
|||
checksum=blob.content_hash, |
|||
status="ready", |
|||
etag=etag, |
|||
last_verified_at=datetime.now(UTC), |
|||
) |
|||
replica.storage_key = storage_key |
|||
replica.status = "ready" |
|||
replica.size_bytes = blob.size_bytes |
|||
replica.checksum = blob.content_hash |
|||
replica.etag = etag |
|||
replica.last_verified_at = datetime.now(UTC) |
|||
|
|||
if existing is None: |
|||
created = await self.blob_replica_repository.create(replica) |
|||
else: |
|||
created = await self.blob_replica_repository.save(replica) |
|||
await self.session.commit() |
|||
return created |
|||
|
|||
async def materialize_replica_to_local_cache(self, blob: Blob, replica: BlobReplica, backend: Backend) -> Path: |
|||
if backend.type == "local_directory": |
|||
adapter = self._build_adapter(backend) |
|||
return await adapter.resolve_path(replica.storage_key) |
|||
|
|||
cache_dir = await self.ensure_cache_dir() |
|||
cache_path = cache_dir / self._build_storage_key(blob) |
|||
adapter = self._build_adapter(backend) |
|||
await adapter.download_file(replica.storage_key, cache_path) |
|||
|
|||
existing_entry = await self.cache_entry_repository.get_ready_by_blob(blob.id) |
|||
now = datetime.now(UTC) |
|||
if existing_entry is None: |
|||
entry = CacheEntry( |
|||
id=new_id("cch"), |
|||
blob_id=blob.id, |
|||
local_path=str(cache_path), |
|||
cache_kind="remote-fetch", |
|||
size_bytes=blob.size_bytes, |
|||
status="ready", |
|||
last_accessed_at=now, |
|||
expires_at=None, |
|||
) |
|||
await self.cache_entry_repository.create(entry) |
|||
else: |
|||
existing_entry.local_path = str(cache_path) |
|||
existing_entry.status = "ready" |
|||
existing_entry.size_bytes = blob.size_bytes |
|||
existing_entry.last_accessed_at = now |
|||
await self.cache_entry_repository.save(existing_entry) |
|||
await self.session.commit() |
|||
return cache_path |
|||
|
|||
async def resolve_best_local_path(self, blob: Blob) -> Path: |
|||
cached_entry = await self.cache_entry_repository.get_ready_by_blob(blob.id) |
|||
if cached_entry is not None: |
|||
cached_path = Path(cached_entry.local_path) |
|||
if await anyio.to_thread.run_sync(cached_path.exists): |
|||
cached_entry.last_accessed_at = datetime.now(UTC) |
|||
await self.cache_entry_repository.save(cached_entry) |
|||
await self.session.commit() |
|||
return cached_path |
|||
|
|||
replicas = await self.blob_replica_repository.list_ready_replicas_with_backends(blob.id) |
|||
for replica, backend in replicas: |
|||
try: |
|||
return await self.materialize_replica_to_local_cache(blob, replica, backend) |
|||
except builtins.FileNotFoundError: |
|||
continue |
|||
except Exception: |
|||
continue |
|||
raise builtins.FileNotFoundError(blob.id) |
|||
|
|||
async def check_backend(self, backend: Backend) -> tuple[str, str | None]: |
|||
adapter = self._build_adapter(backend) |
|||
result = await adapter.check() |
|||
return result["status"], result.get("detail") |
|||
|
|||
def _build_adapter(self, backend: Backend): |
|||
public_config = json.loads(backend.public_config_json) |
|||
secret_config = decrypt_secret_config(backend.secret_config_json) |
|||
config = merge_backend_config(public_config, secret_config) |
|||
return build_storage_adapter(backend.type, config) |
|||
|
|||
@staticmethod |
|||
def _build_storage_key(blob: Blob) -> str: |
|||
return build_blob_storage_key(blob) |
|||
@ -0,0 +1,185 @@ |
|||
from __future__ import annotations |
|||
|
|||
import base64 |
|||
import hashlib |
|||
from dataclasses import dataclass |
|||
from pathlib import Path |
|||
|
|||
import anyio |
|||
from sqlalchemy.ext.asyncio import AsyncSession |
|||
|
|||
from app.core.config import get_settings |
|||
from app.core.exceptions import UploadConflictError, UploadNotFoundError, ValidationError |
|||
from app.core.ids import new_id |
|||
from app.models.entities import UploadSession |
|||
from app.repositories.file_repository import FileRepository |
|||
from app.repositories.upload_repository import UploadRepository |
|||
from app.services.file_service import CreateFileMetadataInput, FileService |
|||
from app.services.job_service import JobService |
|||
from app.services.local_storage_service import LocalStorageService |
|||
from app.services.storage_service import StorageService |
|||
|
|||
|
|||
@dataclass |
|||
class CreateUploadInput: |
|||
upload_length: int |
|||
upload_metadata: str | None = None |
|||
|
|||
|
|||
@dataclass |
|||
class FinalizeUploadInput: |
|||
directory_id: str |
|||
filename: str |
|||
|
|||
|
|||
class UploadService: |
|||
def __init__(self, session: AsyncSession) -> None: |
|||
self.session = session |
|||
self.repository = UploadRepository(session) |
|||
self.file_service = FileService(session) |
|||
self.file_repository = FileRepository(session) |
|||
self.local_storage_service = LocalStorageService(session) |
|||
self.storage_service = StorageService(session) |
|||
self.job_service = JobService(session) |
|||
self.settings = get_settings() |
|||
|
|||
async def ensure_temp_dir(self) -> Path: |
|||
temp_dir = Path(self.settings.upload_temp_dir) |
|||
await anyio.to_thread.run_sync(lambda: temp_dir.mkdir(parents=True, exist_ok=True)) |
|||
return temp_dir |
|||
|
|||
async def create_upload(self, payload: CreateUploadInput) -> UploadSession: |
|||
if payload.upload_length <= 0: |
|||
raise ValidationError("Upload-Length must be greater than zero.") |
|||
|
|||
temp_dir = await self.ensure_temp_dir() |
|||
upload_id = new_id("upl") |
|||
temp_path = temp_dir / f"{upload_id}.bin" |
|||
await anyio.Path(temp_path).write_bytes(b"") |
|||
|
|||
upload_session = UploadSession( |
|||
id=upload_id, |
|||
total_size_bytes=payload.upload_length, |
|||
received_size_bytes=0, |
|||
status="created", |
|||
temp_path=str(temp_path), |
|||
tus_upload_url=f"/files/{upload_id}", |
|||
upload_metadata_json=payload.upload_metadata, |
|||
) |
|||
created = await self.repository.create(upload_session) |
|||
await self.session.commit() |
|||
return created |
|||
|
|||
async def get_upload(self, upload_id: str) -> UploadSession: |
|||
upload_session = await self.repository.get_by_id(upload_id) |
|||
if upload_session is None: |
|||
raise UploadNotFoundError() |
|||
return upload_session |
|||
|
|||
async def append_upload_bytes(self, upload_id: str, offset: int, chunk: bytes) -> UploadSession: |
|||
upload_session = await self.get_upload(upload_id) |
|||
|
|||
if upload_session.status in {"finalized", "finalizing", "expired"}: |
|||
raise UploadConflictError("Upload session is not writable.") |
|||
if offset != upload_session.received_size_bytes: |
|||
raise UploadConflictError("Upload offset does not match the current stored offset.") |
|||
if upload_session.received_size_bytes + len(chunk) > upload_session.total_size_bytes: |
|||
raise UploadConflictError("Upload would exceed declared length.") |
|||
|
|||
temp_path = anyio.Path(upload_session.temp_path) |
|||
async with await anyio.open_file(temp_path, "ab") as file_handle: |
|||
await file_handle.write(chunk) |
|||
|
|||
upload_session.received_size_bytes += len(chunk) |
|||
upload_session.status = ( |
|||
"uploaded" |
|||
if upload_session.received_size_bytes == upload_session.total_size_bytes |
|||
else "uploading" |
|||
) |
|||
saved = await self.repository.save(upload_session) |
|||
await self.session.commit() |
|||
return saved |
|||
|
|||
async def finalize_upload(self, upload_id: str, payload: FinalizeUploadInput): |
|||
upload_session = await self.get_upload(upload_id) |
|||
if upload_session.received_size_bytes != upload_session.total_size_bytes: |
|||
raise UploadConflictError("Upload is not complete.") |
|||
if upload_session.status == "finalized": |
|||
raise UploadConflictError("Upload has already been finalized.") |
|||
|
|||
upload_session.status = "finalizing" |
|||
await self.repository.save(upload_session) |
|||
await self.session.flush() |
|||
|
|||
content_hash = await self._compute_sha256(upload_session.temp_path) |
|||
mime_type = self._guess_mime_type(payload.filename) |
|||
file_entry = await self.file_service.create_file_metadata( |
|||
CreateFileMetadataInput( |
|||
directory_id=payload.directory_id, |
|||
name=payload.filename, |
|||
mime_type=mime_type, |
|||
size_bytes=upload_session.total_size_bytes, |
|||
content_hash=content_hash, |
|||
) |
|||
) |
|||
blob = await self.file_repository.get_primary_blob_for_file(file_entry.id) |
|||
if blob is None: |
|||
raise UploadConflictError("Primary blob was not created for finalized upload.") |
|||
|
|||
await self.local_storage_service.persist_blob_from_temp(blob, upload_session.temp_path) |
|||
await self.job_service.enqueue_file_reconcile(file_entry.id) |
|||
if mime_type is not None: |
|||
await self.job_service.enqueue_preview_generation(file_entry.id) |
|||
|
|||
upload_session.directory_id = payload.directory_id |
|||
upload_session.filename = payload.filename |
|||
upload_session.status = "finalized" |
|||
await self.repository.save(upload_session) |
|||
await self.session.commit() |
|||
await anyio.Path(upload_session.temp_path).unlink(missing_ok=True) |
|||
|
|||
return upload_session, file_entry |
|||
|
|||
async def _compute_sha256(self, temp_path: str) -> str: |
|||
def _hash_file() -> str: |
|||
digest = hashlib.sha256() |
|||
with open(temp_path, "rb") as file_handle: |
|||
while True: |
|||
chunk = file_handle.read(1024 * 1024) |
|||
if not chunk: |
|||
break |
|||
digest.update(chunk) |
|||
return digest.hexdigest() |
|||
|
|||
return await anyio.to_thread.run_sync(_hash_file) |
|||
|
|||
@staticmethod |
|||
def parse_upload_metadata(raw_metadata: str | None) -> dict[str, str]: |
|||
if not raw_metadata: |
|||
return {} |
|||
|
|||
parsed: dict[str, str] = {} |
|||
for item in raw_metadata.split(","): |
|||
item = item.strip() |
|||
if not item: |
|||
continue |
|||
parts = item.split(" ", 1) |
|||
key = parts[0] |
|||
value = "" |
|||
if len(parts) == 2 and parts[1]: |
|||
value = base64.b64decode(parts[1]).decode("utf-8") |
|||
parsed[key] = value |
|||
return parsed |
|||
|
|||
@staticmethod |
|||
def _guess_mime_type(filename: str) -> str | None: |
|||
lowered = filename.lower() |
|||
if lowered.endswith(".mp4"): |
|||
return "video/mp4" |
|||
if lowered.endswith(".jpg") or lowered.endswith(".jpeg"): |
|||
return "image/jpeg" |
|||
if lowered.endswith(".png"): |
|||
return "image/png" |
|||
if lowered.endswith(".pdf"): |
|||
return "application/pdf" |
|||
return None |
|||
@ -0,0 +1,183 @@ |
|||
const LOGIN_KEY = "iron_access_token"; |
|||
|
|||
const loginCard = document.querySelector("#login-card"); |
|||
const appCard = document.querySelector("#app-card"); |
|||
const loginForm = document.querySelector("#login-form"); |
|||
const loginError = document.querySelector("#login-error"); |
|||
const appError = document.querySelector("#app-error"); |
|||
const usernameInput = document.querySelector("#username"); |
|||
const passwordInput = document.querySelector("#password"); |
|||
const welcomeTitle = document.querySelector("#welcome-title"); |
|||
const sessionUser = document.querySelector("#session-user"); |
|||
const rootName = document.querySelector("#root-name"); |
|||
const itemsCount = document.querySelector("#items-count"); |
|||
const rootItems = document.querySelector("#root-items"); |
|||
const backendCount = document.querySelector("#backend-count"); |
|||
const backendsList = document.querySelector("#backends-list"); |
|||
const refreshButton = document.querySelector("#refresh-button"); |
|||
const logoutButton = document.querySelector("#logout-button"); |
|||
|
|||
function getToken() { |
|||
return window.localStorage.getItem(LOGIN_KEY); |
|||
} |
|||
|
|||
function setToken(token) { |
|||
window.localStorage.setItem(LOGIN_KEY, token); |
|||
} |
|||
|
|||
function clearToken() { |
|||
window.localStorage.removeItem(LOGIN_KEY); |
|||
} |
|||
|
|||
async function apiFetch(path, options = {}) { |
|||
const headers = new Headers(options.headers || {}); |
|||
const token = getToken(); |
|||
if (token) { |
|||
headers.set("Authorization", `Bearer ${token}`); |
|||
} |
|||
|
|||
const response = await fetch(path, { ...options, headers }); |
|||
if (!response.ok) { |
|||
let message = `Request failed with status ${response.status}`; |
|||
try { |
|||
const payload = await response.json(); |
|||
message = payload.error?.message || message; |
|||
} catch { |
|||
// keep fallback message
|
|||
} |
|||
throw new Error(message); |
|||
} |
|||
return response; |
|||
} |
|||
|
|||
function renderList(container, items, renderItem) { |
|||
container.innerHTML = ""; |
|||
if (items.length === 0) { |
|||
const empty = document.createElement("li"); |
|||
empty.textContent = "Nothing here yet."; |
|||
container.append(empty); |
|||
return; |
|||
} |
|||
for (const item of items) { |
|||
container.append(renderItem(item)); |
|||
} |
|||
} |
|||
|
|||
function renderRootItem(item) { |
|||
const element = document.createElement("li"); |
|||
const left = document.createElement("div"); |
|||
const right = document.createElement("div"); |
|||
right.className = "item-meta"; |
|||
|
|||
left.innerHTML = `<strong>${item.name}</strong>`; |
|||
if (item.kind === "file" && item.mime_type) { |
|||
right.textContent = item.mime_type; |
|||
} else { |
|||
right.textContent = item.kind; |
|||
} |
|||
|
|||
element.append(left, right); |
|||
return element; |
|||
} |
|||
|
|||
function renderBackendItem(item) { |
|||
const element = document.createElement("li"); |
|||
const left = document.createElement("div"); |
|||
left.innerHTML = `<strong>${item.name}</strong>`; |
|||
const right = document.createElement("div"); |
|||
right.className = "item-meta"; |
|||
right.textContent = `${item.type} · ${item.is_enabled ? "enabled" : "disabled"}`; |
|||
element.append(left, right); |
|||
return element; |
|||
} |
|||
|
|||
function showLogin(errorMessage = "") { |
|||
loginCard.hidden = false; |
|||
appCard.hidden = true; |
|||
loginError.hidden = !errorMessage; |
|||
loginError.textContent = errorMessage; |
|||
} |
|||
|
|||
function showApp() { |
|||
loginCard.hidden = true; |
|||
appCard.hidden = false; |
|||
appError.hidden = true; |
|||
} |
|||
|
|||
async function loadWorkspace() { |
|||
try { |
|||
const [meResponse, directoryResponse, backendResponse] = await Promise.all([ |
|||
apiFetch("/api/auth/me"), |
|||
apiFetch("/api/directories/dir_root/children"), |
|||
apiFetch("/api/backends"), |
|||
]); |
|||
|
|||
const me = await meResponse.json(); |
|||
const directory = await directoryResponse.json(); |
|||
const backends = await backendResponse.json(); |
|||
|
|||
welcomeTitle.textContent = `Welcome, ${me.user.username}`; |
|||
sessionUser.textContent = me.user.username; |
|||
rootName.textContent = directory.directory.name; |
|||
itemsCount.textContent = `${directory.items.length} items`; |
|||
backendCount.textContent = String(backends.items.length); |
|||
|
|||
renderList(rootItems, directory.items, renderRootItem); |
|||
renderList(backendsList, backends.items, renderBackendItem); |
|||
showApp(); |
|||
} catch (error) { |
|||
if (String(error.message).includes("Authentication")) { |
|||
clearToken(); |
|||
showLogin("Your session expired. Please sign in again."); |
|||
return; |
|||
} |
|||
appError.hidden = false; |
|||
appError.textContent = error.message; |
|||
} |
|||
} |
|||
|
|||
loginForm.addEventListener("submit", async (event) => { |
|||
event.preventDefault(); |
|||
loginError.hidden = true; |
|||
|
|||
try { |
|||
const response = await fetch("/api/auth/login", { |
|||
method: "POST", |
|||
headers: { "Content-Type": "application/json" }, |
|||
body: JSON.stringify({ |
|||
username: usernameInput.value, |
|||
password: passwordInput.value, |
|||
}), |
|||
}); |
|||
const payload = await response.json(); |
|||
if (!response.ok) { |
|||
throw new Error(payload.error?.message || "Unable to sign in."); |
|||
} |
|||
|
|||
setToken(payload.access_token); |
|||
passwordInput.value = ""; |
|||
await loadWorkspace(); |
|||
} catch (error) { |
|||
showLogin(error.message); |
|||
} |
|||
}); |
|||
|
|||
refreshButton.addEventListener("click", async () => { |
|||
await loadWorkspace(); |
|||
}); |
|||
|
|||
logoutButton.addEventListener("click", async () => { |
|||
try { |
|||
await apiFetch("/api/auth/logout", { method: "POST" }); |
|||
} catch { |
|||
// best-effort logout
|
|||
} |
|||
clearToken(); |
|||
showLogin(); |
|||
}); |
|||
|
|||
if (getToken()) { |
|||
void loadWorkspace(); |
|||
} else { |
|||
showLogin(); |
|||
} |
|||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@ -0,0 +1,13 @@ |
|||
<!doctype html> |
|||
<html lang="en"> |
|||
<head> |
|||
<meta charset="UTF-8" /> |
|||
<meta name="viewport" content="width=device-width, initial-scale=1.0" /> |
|||
<title>Iron</title> |
|||
<script type="module" crossorigin src="/assets/app.js"></script> |
|||
<link rel="stylesheet" crossorigin href="/assets/app.css"> |
|||
</head> |
|||
<body> |
|||
<div id="root"></div> |
|||
</body> |
|||
</html> |
|||
@ -0,0 +1,107 @@ |
|||
<!doctype html> |
|||
<html lang="en"> |
|||
<head> |
|||
<meta charset="utf-8" /> |
|||
<meta name="viewport" content="width=device-width, initial-scale=1" /> |
|||
<title>Iron</title> |
|||
<link rel="stylesheet" href="/assets/styles.css" /> |
|||
</head> |
|||
<body> |
|||
<div class="page-shell"> |
|||
<aside class="hero-panel"> |
|||
<p class="eyebrow">Iron Gateway</p> |
|||
<h1>One personal drive, one calm control plane.</h1> |
|||
<p class="lede"> |
|||
Log in to browse your namespace, inspect storage health, and grow this gateway into |
|||
the multi-backend drive defined by the project MVP. |
|||
</p> |
|||
<div class="hero-card"> |
|||
<div> |
|||
<span class="label">Current shell</span> |
|||
<strong>Auth + browser entry</strong> |
|||
</div> |
|||
<div> |
|||
<span class="label">Next up</span> |
|||
<strong>Upload UI and S3 runtime path</strong> |
|||
</div> |
|||
</div> |
|||
</aside> |
|||
|
|||
<main class="workspace-panel"> |
|||
<section id="login-card" class="card"> |
|||
<div class="card-header"> |
|||
<p class="eyebrow">Local Login</p> |
|||
<h2>Enter the gateway</h2> |
|||
</div> |
|||
<form id="login-form" class="stack-form"> |
|||
<label> |
|||
<span>Username</span> |
|||
<input id="username" name="username" type="text" autocomplete="username" required /> |
|||
</label> |
|||
<label> |
|||
<span>Password</span> |
|||
<input |
|||
id="password" |
|||
name="password" |
|||
type="password" |
|||
autocomplete="current-password" |
|||
required |
|||
/> |
|||
</label> |
|||
<button type="submit">Sign in</button> |
|||
<p id="login-error" class="error" hidden></p> |
|||
</form> |
|||
</section> |
|||
|
|||
<section id="app-card" class="card" hidden> |
|||
<div class="toolbar"> |
|||
<div> |
|||
<p class="eyebrow">Workspace</p> |
|||
<h2 id="welcome-title">Signed in</h2> |
|||
</div> |
|||
<div class="toolbar-actions"> |
|||
<button id="refresh-button" type="button" class="ghost">Refresh</button> |
|||
<button id="logout-button" type="button" class="ghost">Logout</button> |
|||
</div> |
|||
</div> |
|||
|
|||
<div class="status-grid"> |
|||
<article class="status-card"> |
|||
<span class="label">Session</span> |
|||
<strong id="session-user">Unknown</strong> |
|||
</article> |
|||
<article class="status-card"> |
|||
<span class="label">Root Directory</span> |
|||
<strong id="root-name">/</strong> |
|||
</article> |
|||
<article class="status-card"> |
|||
<span class="label">Backends</span> |
|||
<strong id="backend-count">0</strong> |
|||
</article> |
|||
</div> |
|||
|
|||
<section class="panel-grid"> |
|||
<div class="subpanel"> |
|||
<div class="subpanel-header"> |
|||
<h3>Root Items</h3> |
|||
<span id="items-count" class="pill">0 items</span> |
|||
</div> |
|||
<ul id="root-items" class="item-list"></ul> |
|||
</div> |
|||
|
|||
<div class="subpanel"> |
|||
<div class="subpanel-header"> |
|||
<h3>Configured Backends</h3> |
|||
</div> |
|||
<ul id="backends-list" class="item-list"></ul> |
|||
</div> |
|||
</section> |
|||
|
|||
<p id="app-error" class="error" hidden></p> |
|||
</section> |
|||
</main> |
|||
</div> |
|||
|
|||
<script src="/assets/app.js" type="module"></script> |
|||
</body> |
|||
</html> |
|||
@ -0,0 +1,25 @@ |
|||
from __future__ import annotations |
|||
|
|||
from pathlib import Path |
|||
|
|||
from fastapi import APIRouter |
|||
from fastapi import HTTPException |
|||
from fastapi.responses import FileResponse |
|||
|
|||
|
|||
router = APIRouter() |
|||
WEB_DIR = Path(__file__).resolve().parent |
|||
DIST_DIR = WEB_DIR / "dist" |
|||
INDEX_FILE = DIST_DIR / "index.html" |
|||
|
|||
|
|||
def _get_index_file() -> Path: |
|||
if not INDEX_FILE.exists(): |
|||
raise HTTPException(status_code=503, detail="Web UI assets are not built.") |
|||
return INDEX_FILE |
|||
|
|||
|
|||
@router.get("/app", include_in_schema=False) |
|||
@router.get("/app/{path:path}", include_in_schema=False) |
|||
async def web_app(_path: str = "") -> FileResponse: |
|||
return FileResponse(_get_index_file()) |
|||
@ -0,0 +1,245 @@ |
|||
:root { |
|||
--bg: #f3efe6; |
|||
--surface: rgba(255, 252, 245, 0.88); |
|||
--surface-strong: #fffaf1; |
|||
--ink: #1e1b16; |
|||
--muted: #6d6457; |
|||
--line: rgba(58, 44, 23, 0.12); |
|||
--accent: #bb5a2b; |
|||
--accent-strong: #8b3d17; |
|||
--success: #2d6a4f; |
|||
--danger: #a63b2c; |
|||
--shadow: 0 18px 50px rgba(72, 44, 20, 0.12); |
|||
} |
|||
|
|||
* { |
|||
box-sizing: border-box; |
|||
} |
|||
|
|||
body { |
|||
margin: 0; |
|||
min-height: 100vh; |
|||
color: var(--ink); |
|||
background: |
|||
radial-gradient(circle at top left, rgba(219, 164, 103, 0.28), transparent 30%), |
|||
radial-gradient(circle at bottom right, rgba(92, 140, 109, 0.24), transparent 28%), |
|||
linear-gradient(135deg, #efe7d8 0%, #f8f4ec 45%, #ece3d5 100%); |
|||
font-family: Georgia, "Iowan Old Style", "Palatino Linotype", serif; |
|||
} |
|||
|
|||
button, |
|||
input { |
|||
font: inherit; |
|||
} |
|||
|
|||
.page-shell { |
|||
display: grid; |
|||
grid-template-columns: minmax(280px, 0.95fr) minmax(340px, 1.2fr); |
|||
min-height: 100vh; |
|||
} |
|||
|
|||
.hero-panel, |
|||
.workspace-panel { |
|||
padding: 48px 40px; |
|||
} |
|||
|
|||
.hero-panel { |
|||
display: flex; |
|||
flex-direction: column; |
|||
justify-content: center; |
|||
gap: 24px; |
|||
} |
|||
|
|||
.hero-panel h1 { |
|||
margin: 0; |
|||
font-size: clamp(2.3rem, 4vw, 4.4rem); |
|||
line-height: 0.95; |
|||
max-width: 10ch; |
|||
} |
|||
|
|||
.lede { |
|||
margin: 0; |
|||
max-width: 52ch; |
|||
color: var(--muted); |
|||
font-size: 1.08rem; |
|||
line-height: 1.7; |
|||
} |
|||
|
|||
.hero-card, |
|||
.card, |
|||
.status-card, |
|||
.subpanel { |
|||
border: 1px solid var(--line); |
|||
background: var(--surface); |
|||
backdrop-filter: blur(14px); |
|||
box-shadow: var(--shadow); |
|||
} |
|||
|
|||
.hero-card { |
|||
display: grid; |
|||
gap: 18px; |
|||
padding: 22px 24px; |
|||
border-radius: 24px; |
|||
max-width: 440px; |
|||
} |
|||
|
|||
.workspace-panel { |
|||
display: flex; |
|||
align-items: center; |
|||
justify-content: center; |
|||
} |
|||
|
|||
.card { |
|||
width: min(860px, 100%); |
|||
padding: 28px; |
|||
border-radius: 28px; |
|||
} |
|||
|
|||
.card-header h2, |
|||
.toolbar h2, |
|||
.subpanel-header h3 { |
|||
margin: 0; |
|||
} |
|||
|
|||
.eyebrow, |
|||
.label { |
|||
display: block; |
|||
margin-bottom: 8px; |
|||
color: var(--muted); |
|||
font-size: 0.8rem; |
|||
letter-spacing: 0.12em; |
|||
text-transform: uppercase; |
|||
} |
|||
|
|||
.stack-form { |
|||
display: grid; |
|||
gap: 18px; |
|||
} |
|||
|
|||
.stack-form label { |
|||
display: grid; |
|||
gap: 8px; |
|||
} |
|||
|
|||
input { |
|||
width: 100%; |
|||
padding: 14px 16px; |
|||
border: 1px solid rgba(61, 45, 24, 0.18); |
|||
border-radius: 14px; |
|||
background: var(--surface-strong); |
|||
} |
|||
|
|||
button { |
|||
border: none; |
|||
border-radius: 999px; |
|||
padding: 13px 18px; |
|||
color: #fffaf2; |
|||
background: linear-gradient(135deg, var(--accent) 0%, var(--accent-strong) 100%); |
|||
cursor: pointer; |
|||
transition: transform 140ms ease, opacity 140ms ease; |
|||
} |
|||
|
|||
button:hover { |
|||
transform: translateY(-1px); |
|||
} |
|||
|
|||
.ghost { |
|||
color: var(--ink); |
|||
background: rgba(255, 250, 241, 0.86); |
|||
border: 1px solid var(--line); |
|||
} |
|||
|
|||
.toolbar, |
|||
.subpanel-header { |
|||
display: flex; |
|||
align-items: center; |
|||
justify-content: space-between; |
|||
gap: 16px; |
|||
} |
|||
|
|||
.toolbar-actions { |
|||
display: flex; |
|||
gap: 12px; |
|||
} |
|||
|
|||
.status-grid, |
|||
.panel-grid { |
|||
display: grid; |
|||
gap: 16px; |
|||
margin-top: 22px; |
|||
} |
|||
|
|||
.status-grid { |
|||
grid-template-columns: repeat(3, minmax(0, 1fr)); |
|||
} |
|||
|
|||
.status-card, |
|||
.subpanel { |
|||
padding: 18px 20px; |
|||
border-radius: 22px; |
|||
} |
|||
|
|||
.panel-grid { |
|||
grid-template-columns: repeat(2, minmax(0, 1fr)); |
|||
} |
|||
|
|||
.item-list { |
|||
list-style: none; |
|||
margin: 16px 0 0; |
|||
padding: 0; |
|||
display: grid; |
|||
gap: 10px; |
|||
} |
|||
|
|||
.item-list li { |
|||
display: flex; |
|||
align-items: center; |
|||
justify-content: space-between; |
|||
gap: 16px; |
|||
padding: 12px 14px; |
|||
border-radius: 16px; |
|||
background: rgba(255, 250, 241, 0.78); |
|||
border: 1px solid rgba(61, 45, 24, 0.08); |
|||
} |
|||
|
|||
.item-meta { |
|||
color: var(--muted); |
|||
font-size: 0.92rem; |
|||
} |
|||
|
|||
.pill { |
|||
padding: 6px 10px; |
|||
border-radius: 999px; |
|||
background: rgba(187, 90, 43, 0.1); |
|||
color: var(--accent-strong); |
|||
font-size: 0.86rem; |
|||
} |
|||
|
|||
.error { |
|||
margin: 14px 0 0; |
|||
color: var(--danger); |
|||
} |
|||
|
|||
[hidden] { |
|||
display: none !important; |
|||
} |
|||
|
|||
@media (max-width: 960px) { |
|||
.page-shell { |
|||
grid-template-columns: 1fr; |
|||
} |
|||
|
|||
.hero-panel, |
|||
.workspace-panel { |
|||
padding: 28px 18px; |
|||
} |
|||
|
|||
.hero-panel h1 { |
|||
max-width: none; |
|||
} |
|||
|
|||
.status-grid, |
|||
.panel-grid { |
|||
grid-template-columns: 1fr; |
|||
} |
|||
} |
|||
@ -0,0 +1,10 @@ |
|||
IRON_DATABASE_URL=sqlite+aiosqlite:////srv/iron/iron.db |
|||
IRON_BOOTSTRAP_USERNAME=admin |
|||
IRON_BOOTSTRAP_PASSWORD=replace-with-a-strong-password |
|||
IRON_SECRET_KEY=replace-with-a-long-random-secret |
|||
IRON_UPLOAD_TEMP_DIR=/srv/iron/runtime/temp |
|||
IRON_LOCAL_STORAGE_DIR=/srv/iron/runtime/local |
|||
IRON_CACHE_DIR=/srv/iron/runtime/cache |
|||
IRON_AUTH_SESSION_TTL_HOURS=168 |
|||
IRON_JOB_POLL_INTERVAL_SECONDS=2 |
|||
IRON_JOB_BATCH_SIZE=10 |
|||
@ -0,0 +1,29 @@ |
|||
#!/usr/bin/env bash |
|||
set -euo pipefail |
|||
|
|||
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" |
|||
ENV_FILE="${IRON_ENV_FILE:-$ROOT_DIR/deploy/iron/iron.env}" |
|||
HOST="${IRON_HOST:-127.0.0.1}" |
|||
PORT="${IRON_PORT:-8000}" |
|||
|
|||
cd "$ROOT_DIR" |
|||
|
|||
if [[ ! -f ".venv/bin/python" ]]; then |
|||
echo "missing .venv/bin/python; install dependencies first" >&2 |
|||
exit 1 |
|||
fi |
|||
|
|||
if [[ -f "$ENV_FILE" ]]; then |
|||
set -a |
|||
# shellcheck disable=SC1090 |
|||
source "$ENV_FILE" |
|||
set +a |
|||
fi |
|||
|
|||
mkdir -p \ |
|||
"${IRON_UPLOAD_TEMP_DIR:-./.iron-temp/uploads}" \ |
|||
"${IRON_LOCAL_STORAGE_DIR:-./.iron-storage/local}" \ |
|||
"${IRON_CACHE_DIR:-./.iron-cache}" |
|||
|
|||
./.venv/bin/alembic upgrade head |
|||
exec ./.venv/bin/python -m uvicorn app.main:app --host "$HOST" --port "$PORT" |
|||
@ -0,0 +1,3 @@ |
|||
OPENLIST_BIND_HOST=127.0.0.1 |
|||
OPENLIST_PORT=5244 |
|||
OPENLIST_DATA_DIR=/srv/iron/openlist |
|||
@ -0,0 +1,9 @@ |
|||
services: |
|||
openlist: |
|||
image: openlistteam/openlist:latest |
|||
container_name: openlist |
|||
restart: unless-stopped |
|||
ports: |
|||
- "${OPENLIST_BIND_HOST:-127.0.0.1}:${OPENLIST_PORT:-5244}:5244" |
|||
volumes: |
|||
- "${OPENLIST_DATA_DIR:-/srv/iron/openlist}:/opt/openlist/data" |
|||
@ -0,0 +1,17 @@ |
|||
[Unit] |
|||
Description=Iron personal cloud drive gateway |
|||
After=network-online.target |
|||
Wants=network-online.target |
|||
|
|||
[Service] |
|||
Type=simple |
|||
User=iron |
|||
Group=iron |
|||
WorkingDirectory=/opt/iron |
|||
EnvironmentFile=/opt/iron/deploy/iron/iron.env |
|||
ExecStart=/opt/iron/deploy/iron/start_iron.sh |
|||
Restart=on-failure |
|||
RestartSec=3 |
|||
|
|||
[Install] |
|||
WantedBy=multi-user.target |
|||
@ -0,0 +1,15 @@ |
|||
# Documentation |
|||
|
|||
This directory contains the stable project documentation for Iron. |
|||
|
|||
- [Product](product.md): product scope, current stage, MVP cutline, and roadmap. |
|||
- [Architecture](architecture.md): backend, frontend, data model, storage, and job design. |
|||
- [API](api.md): route map and API groups. |
|||
- [Development](development.md): setup, test commands, Playwright workflow, and documentation rules. |
|||
- [OpenList + Iron Local Deployment SOP](openlist-local-deployment-sop.md): step-by-step local setup for OpenList, Aliyun Drive, WebDAV, and Iron. |
|||
|
|||
Root-level project docs: |
|||
|
|||
- [README](../README.md): user-facing overview and quick start. |
|||
- [Contributing](../CONTRIBUTING.md): contributor workflow. |
|||
- [Agents](../AGENTS.md): coding-agent and automation guidance. |
|||
@ -0,0 +1,108 @@ |
|||
# API |
|||
|
|||
Iron exposes JSON APIs under `/api` and `tus` upload transport under `/files`. |
|||
|
|||
Most `/api` routes require: |
|||
|
|||
```text |
|||
Authorization: Bearer <token> |
|||
``` |
|||
|
|||
## Auth |
|||
|
|||
- `POST /api/auth/login` |
|||
- `POST /api/auth/logout` |
|||
- `GET /api/auth/me` |
|||
|
|||
## Directories |
|||
|
|||
- `GET /api/directories/{directory_id}/children` |
|||
- `POST /api/directories` |
|||
- `POST /api/directories/{directory_id}/rename` |
|||
- `POST /api/directories/{directory_id}/move` |
|||
|
|||
## Files |
|||
|
|||
- `GET /api/files/{file_id}` |
|||
- `GET /api/files/{file_id}/download` |
|||
- `GET /api/files/{file_id}/preview` |
|||
- `GET /api/files/{file_id}/stream` |
|||
- `POST /api/files/{file_id}/rename` |
|||
- `POST /api/files/{file_id}/move` |
|||
- `DELETE /api/files/{file_id}` |
|||
- `POST /api/files/{file_id}/restore` |
|||
- `GET /api/files/recycle-bin` |
|||
|
|||
## Uploads |
|||
|
|||
Protocol endpoints: |
|||
|
|||
- `POST /files` |
|||
- `HEAD /files/{upload_id}` |
|||
- `PATCH /files/{upload_id}` |
|||
|
|||
Finalize endpoint: |
|||
|
|||
- `POST /api/uploads/{upload_id}/finalize` |
|||
|
|||
## Backends |
|||
|
|||
- `GET /api/backends` |
|||
- `POST /api/backends` |
|||
- `PATCH /api/backends/{backend_id}` |
|||
- `POST /api/backends/{backend_id}/check` |
|||
- `POST /api/backends/{backend_id}/disable` |
|||
- `POST /api/backends/{backend_id}/enable` |
|||
- `DELETE /api/backends/{backend_id}` |
|||
|
|||
Backend `type` values: |
|||
|
|||
- `local_directory` |
|||
- `s3` |
|||
- `webdav` |
|||
|
|||
Backend responses include public `config` values plus a `secret_fields` list for |
|||
configured credentials. Secret fields such as S3 credentials and WebDAV |
|||
passwords are never echoed back in clear text. |
|||
Backend updates preserve existing secret fields when they are omitted from the |
|||
request config, and replace them only when a new value is provided. |
|||
WebDAV configs should set `root_path` so Iron uses a dedicated folder tree |
|||
inside the remote endpoint, which is a good fit for OpenList-managed cloud |
|||
drives. |
|||
Backend deletion is guarded: the backend must be disabled first, must not be the |
|||
default local backend, must not still store blob replicas, and must not be |
|||
referenced by placement policies. |
|||
|
|||
## Jobs |
|||
|
|||
- `GET /api/jobs` |
|||
- `GET /api/jobs/{job_id}` |
|||
- `POST /api/jobs/{job_id}/retry` |
|||
- `POST /api/jobs/run-pending` |
|||
- `POST /api/jobs/enqueue-health-checks` |
|||
- `POST /api/jobs/enqueue-full-reconcile` |
|||
|
|||
## Placement Policies |
|||
|
|||
- `GET /api/policies/placement` |
|||
- `PUT /api/policies/placement/{file_class}` |
|||
- `GET /api/policies/placement/preview` |
|||
|
|||
Saving a placement policy also enqueues one full reconcile job so desired |
|||
replica placement converges without a separate manual step. |
|||
|
|||
## Metadata Export And Recovery |
|||
|
|||
- `GET /api/exports/metadata` |
|||
- `POST /api/exports/metadata/validate` |
|||
- `GET /api/exports/metadata/integrity` |
|||
- `POST /api/exports/metadata/restore-plan` |
|||
- `POST /api/exports/metadata/import` |
|||
|
|||
Metadata import requires `confirm_replace=true` and a validation token from the |
|||
restore-plan endpoint. |
|||
|
|||
## System |
|||
|
|||
- `GET /api/system/health` |
|||
- `GET /api/system/ready` |
|||
@ -0,0 +1,158 @@ |
|||
# Architecture |
|||
|
|||
Iron is an async Python modular monolith with a React single-page Web app. |
|||
|
|||
## System Overview |
|||
|
|||
```text |
|||
Browser UI |
|||
| |
|||
FastAPI app |
|||
| |
|||
Services |
|||
| |
|||
Repositories |
|||
| |
|||
SQLite metadata database |
|||
| |
|||
Storage adapters: local directory, S3, WebDAV |
|||
``` |
|||
|
|||
## Backend |
|||
|
|||
Core stack: |
|||
|
|||
- FastAPI |
|||
- SQLAlchemy 2.x async ORM |
|||
- SQLite via `aiosqlite` |
|||
- Alembic migrations |
|||
- durable in-process jobs stored in SQLite |
|||
|
|||
Important packages: |
|||
|
|||
- `app/api/`: route handlers and dependency wiring |
|||
- `app/services/`: business logic |
|||
- `app/repositories/`: database access |
|||
- `app/models/`: SQLAlchemy entities |
|||
- `app/schemas/`: API schemas |
|||
- `app/adapters/storage/`: storage adapter protocol, registry, and backend implementations |
|||
|
|||
## Frontend |
|||
|
|||
Core stack: |
|||
|
|||
- Vite |
|||
- React |
|||
- TypeScript |
|||
- React Router |
|||
- TanStack Query |
|||
|
|||
Source lives in `frontend/`. Built assets are emitted into `app/web/dist/` and |
|||
served by FastAPI at `/app`. |
|||
|
|||
## Data Model |
|||
|
|||
Main tables: |
|||
|
|||
- `users` |
|||
- `auth_sessions` |
|||
- `directories` |
|||
- `file_entries` |
|||
- `file_versions` |
|||
- `blobs` |
|||
- `blob_replicas` |
|||
- `backends` |
|||
- `upload_sessions` |
|||
- `upload_session_parts` |
|||
- `jobs` |
|||
- `placement_policies` |
|||
- `preview_artifacts` |
|||
- `cache_entries` |
|||
|
|||
Design principles: |
|||
|
|||
- logical namespace is separate from physical storage placement |
|||
- files have immutable content versions |
|||
- physical content is represented as blobs and replicas |
|||
- user-facing deletion uses recycle-bin semantics |
|||
- jobs are durable and retryable |
|||
- backend-specific configuration is validated before persistence and stays |
|||
behind adapter boundaries |
|||
|
|||
## Upload And Read Path |
|||
|
|||
Uploads use `tus` transport under `/files`. |
|||
|
|||
Flow: |
|||
|
|||
1. create upload session |
|||
2. append bytes with `PATCH` |
|||
3. finalize through the API |
|||
4. create file metadata, version, blob, and local replica |
|||
5. enqueue replication or preview jobs when needed |
|||
|
|||
Read path: |
|||
|
|||
1. resolve file and current version |
|||
2. find the best local or cached blob |
|||
3. if needed, materialize a ready remote replica into cache |
|||
4. serve download, preview, or range stream |
|||
|
|||
## Browser File Access |
|||
|
|||
JSON APIs use bearer-token authorization through the shared frontend API client. |
|||
|
|||
Browser preview and download must not use naked `/api/files/...` media URLs in |
|||
`img`, `iframe`, `video`, or `window.open`. Fetch file content with |
|||
authorization, create an object URL, use it, then revoke it. |
|||
|
|||
This is covered by the Playwright E2E flow. |
|||
|
|||
## Jobs |
|||
|
|||
Jobs are stored in SQLite and executed by the in-process job loop. |
|||
|
|||
Current job types include: |
|||
|
|||
- blob replication |
|||
- preview artifact generation |
|||
- backend health checks |
|||
- full reconcile |
|||
- file reconcile |
|||
|
|||
Every job should be safe to retry. |
|||
|
|||
## Storage Backends |
|||
|
|||
Supported backend types: |
|||
|
|||
- `local_directory`: a filesystem directory used for default local persistence |
|||
and low-latency reads |
|||
- `s3`: S3 protocol storage, including AWS S3, MinIO, and compatible |
|||
object stores |
|||
- `webdav`: WebDAV storage, including OpenList-managed cloud drive directories |
|||
exposed through a dedicated `root_path` |
|||
|
|||
Replica object keys are written into a structured application-owned layout such |
|||
as `objects/file/sha256/73/35/<hash>.blob`. This keeps remote storage organized |
|||
under a predictable object tree without trying to disguise replica content as |
|||
user-authored media files. During the current development stage, Iron treats |
|||
this layout as the only supported remote object format and does not preserve |
|||
compatibility with earlier experimental paths. |
|||
|
|||
Reserved object-tree namespaces: |
|||
|
|||
- `objects/file/...`: primary file blobs and replica payloads |
|||
- `objects/preview/<artifact_type>/...`: generated previews such as `inline_preview`, `thumbnail`, and `poster` |
|||
- `objects/export/metadata/YYYY/MM/DD/...`: metadata snapshot exports |
|||
- `objects/manifest/<scope>/...`: future manifest-style control files such as placement or repair manifests |
|||
|
|||
Backend config is normalized through typed schemas before it is stored. Public |
|||
config and encrypted secret config are persisted separately. `IRON_SECRET_KEY` |
|||
derives the encryption key for stored backend credentials. API responses include |
|||
public config plus a list of configured secret fields so operators can identify |
|||
a backend without exposing access keys, client secrets, or refresh tokens. Update |
|||
requests can change public config and provide secret overrides; omitted secret |
|||
fields keep their previous encrypted values. Disabled backends can be deleted |
|||
only when they are no longer referenced by policies and no blob replicas remain |
|||
stored on them. |
|||
Some files were not shown because too many files changed in this diff
Loading…
Reference in new issue