Compare commits
3 Commits
57b9825b2b
...
22fd71cbb1
| Author | SHA1 | Date |
|---|---|---|
|
|
22fd71cbb1 | 3 months ago |
|
|
a32943be95 | 3 months ago |
|
|
17f1530328 | 3 months ago |
156 changed files with 18156 additions and 0 deletions
@ -0,0 +1,17 @@ |
|||||
|
.venv/ |
||||
|
.uv-cache/ |
||||
|
.pytest_cache/ |
||||
|
__pycache__/ |
||||
|
*.py[cod] |
||||
|
*.db |
||||
|
*.sqlite |
||||
|
.coverage |
||||
|
htmlcov/ |
||||
|
iron_gateway.egg-info/ |
||||
|
node_modules/ |
||||
|
output/ |
||||
|
.iron-storage/ |
||||
|
.iron-temp/ |
||||
|
.iron-cache/ |
||||
|
deploy/openlist/.env |
||||
|
deploy/iron/iron.env |
||||
@ -0,0 +1,82 @@ |
|||||
|
# AGENTS.md |
||||
|
|
||||
|
Guidance for coding agents and automated contributors working in this repository. |
||||
|
|
||||
|
## Project Snapshot |
||||
|
|
||||
|
Iron is a self-hosted personal cloud drive gateway with: |
||||
|
|
||||
|
- FastAPI backend under `app/` |
||||
|
- Vite + React + TypeScript frontend under `frontend/` |
||||
|
- built frontend assets served from `app/web/dist/` |
||||
|
- Alembic migrations under `alembic/` |
||||
|
- tests under `tests/` |
||||
|
- project docs under `docs/` |
||||
|
|
||||
|
The current stage is **Product MVP Candidate**. Preserve the existing |
||||
|
desktop-first drive UX and avoid drifting back toward a generic admin dashboard. |
||||
|
|
||||
|
## Required Setup |
||||
|
|
||||
|
Use the local virtualenv and `uv` cache: |
||||
|
|
||||
|
```bash |
||||
|
UV_CACHE_DIR=.uv-cache uv venv .venv |
||||
|
UV_CACHE_DIR=.uv-cache uv pip install --python .venv/bin/python -e '.[dev]' |
||||
|
npm install |
||||
|
``` |
||||
|
|
||||
|
## Common Commands |
||||
|
|
||||
|
```bash |
||||
|
npm run build |
||||
|
.venv/bin/python -m pytest |
||||
|
.venv/bin/python scripts/ui_playwright_smoke.py |
||||
|
.venv/bin/python -m pytest tests/e2e/test_web_ui_playwright.py |
||||
|
``` |
||||
|
|
||||
|
## Development Rules |
||||
|
|
||||
|
- Keep runtime artifacts out of Git: `.venv/`, `node_modules/`, `output/`, |
||||
|
`.iron-storage/`, `.iron-temp/`, caches, and local databases. |
||||
|
- Add or update tests in the same change as behavior changes. |
||||
|
- For UI work, run the real-service Playwright flow and review screenshots. |
||||
|
- Do not use naked `/api/files/...` media URLs for browser preview/download. |
||||
|
Fetch file content with authorization, create an object URL, and revoke it. |
||||
|
- Keep docs current when changing setup, architecture, test counts, or user flows. |
||||
|
- Prefer small, focused changes over broad rewrites. |
||||
|
|
||||
|
## UI Quality Standard |
||||
|
|
||||
|
For any meaningful Web UI change: |
||||
|
|
||||
|
1. build frontend assets with `npm run build` |
||||
|
2. run `./.venv/bin/python scripts/ui_playwright_smoke.py` |
||||
|
3. inspect screenshots in `output/playwright/` |
||||
|
4. fix layout or interaction issues found in screenshots |
||||
|
5. rerun the Playwright flow |
||||
|
|
||||
|
See [docs/development.md](docs/development.md). |
||||
|
|
||||
|
## Documentation Expectations |
||||
|
|
||||
|
- `README.md` should stay user-facing and concise. |
||||
|
- `CONTRIBUTING.md` should describe contributor workflow. |
||||
|
- `docs/README.md` should index deeper docs. |
||||
|
- `docs/development.md` should describe test and UI validation rules. |
||||
|
- Avoid local absolute paths in committed Markdown. |
||||
|
|
||||
|
## Verification Before Finishing |
||||
|
|
||||
|
At minimum, run: |
||||
|
|
||||
|
```bash |
||||
|
npm run build |
||||
|
.venv/bin/python -m pytest |
||||
|
``` |
||||
|
|
||||
|
If UI behavior changed, also run: |
||||
|
|
||||
|
```bash |
||||
|
.venv/bin/python scripts/ui_playwright_smoke.py |
||||
|
``` |
||||
@ -0,0 +1,62 @@ |
|||||
|
# Contributing |
||||
|
|
||||
|
Thanks for helping improve Iron. The project is still early, so the most useful |
||||
|
contributions are focused changes with tests and clear documentation updates. |
||||
|
|
||||
|
## Development Setup |
||||
|
|
||||
|
```bash |
||||
|
UV_CACHE_DIR=.uv-cache uv venv .venv |
||||
|
UV_CACHE_DIR=.uv-cache uv pip install --python .venv/bin/python -e '.[dev]' |
||||
|
npm install |
||||
|
npm run build |
||||
|
``` |
||||
|
|
||||
|
Run the API locally: |
||||
|
|
||||
|
```bash |
||||
|
.venv/bin/python -m uvicorn app.main:app --reload |
||||
|
``` |
||||
|
|
||||
|
Open `http://127.0.0.1:8000/app`. |
||||
|
|
||||
|
## Test Before Submitting |
||||
|
|
||||
|
Run the full suite: |
||||
|
|
||||
|
```bash |
||||
|
.venv/bin/python -m pytest |
||||
|
``` |
||||
|
|
||||
|
For UI changes, also run: |
||||
|
|
||||
|
```bash |
||||
|
npm run build |
||||
|
.venv/bin/python scripts/ui_playwright_smoke.py |
||||
|
``` |
||||
|
|
||||
|
Review screenshots in `output/playwright/` before considering UI work complete. |
||||
|
|
||||
|
## Pull Request Checklist |
||||
|
|
||||
|
- The change is focused and described clearly. |
||||
|
- Tests were added or updated for behavior changes. |
||||
|
- UI changes were validated with Playwright when applicable. |
||||
|
- Documentation was updated if setup, behavior, or architecture changed. |
||||
|
- Generated/runtime files are not included. |
||||
|
|
||||
|
## Code Style |
||||
|
|
||||
|
- Backend code lives in `app/` and uses async FastAPI + SQLAlchemy patterns. |
||||
|
- Frontend code lives in `frontend/` and uses React, React Router, and |
||||
|
TanStack Query. |
||||
|
- Keep browser file preview/download flows authenticated through the API client. |
||||
|
- Prefer clear names and small modules over clever abstractions. |
||||
|
|
||||
|
## Project Status |
||||
|
|
||||
|
Iron is a Product MVP Candidate. See: |
||||
|
|
||||
|
- [docs/product.md](docs/product.md) |
||||
|
- [docs/architecture.md](docs/architecture.md) |
||||
|
- [docs/development.md](docs/development.md) |
||||
@ -0,0 +1,165 @@ |
|||||
|
# Iron |
||||
|
|
||||
|
Iron is a self-hosted personal cloud drive gateway. It provides one logical file |
||||
|
namespace over local storage and pluggable backends, with a browser UI for daily |
||||
|
file management and operational visibility. |
||||
|
|
||||
|
The project is currently a **Product MVP Candidate**: the core backend and Web |
||||
|
app exist, but the release still needs hardening, broader browser regression |
||||
|
coverage, and more real-world usage. |
||||
|
|
||||
|
 |
||||
|
|
||||
|
## Features |
||||
|
|
||||
|
- FastAPI backend with SQLite, SQLAlchemy, and Alembic migrations |
||||
|
- Local authentication with persisted bearer-token sessions |
||||
|
- Directory and file operations: browse, create folder, rename, move, delete, |
||||
|
restore, download, and preview |
||||
|
- `tus` upload flow with local object persistence |
||||
|
- Desktop-first React Web app at `/app` |
||||
|
- Storage backend management for local directories, S3 storage, and |
||||
|
OpenList/WebDAV runtime paths |
||||
|
- Placement policies, reconcile jobs, health checks, and retryable background jobs |
||||
|
- Metadata export, validation, integrity checks, restore plans, and guarded import |
||||
|
- Python Playwright end-to-end coverage for core browser flows |
||||
|
- Real-environment OpenList acceptance scripts for deployment validation |
||||
|
|
||||
|
## Quick Start |
||||
|
|
||||
|
Iron uses `uv` for local Python environment setup. |
||||
|
|
||||
|
```bash |
||||
|
UV_CACHE_DIR=.uv-cache uv venv .venv |
||||
|
UV_CACHE_DIR=.uv-cache uv pip install --python .venv/bin/python -e '.[dev]' |
||||
|
npm install |
||||
|
npm run build |
||||
|
.venv/bin/python -m uvicorn app.main:app --reload |
||||
|
``` |
||||
|
|
||||
|
Open the Web app: |
||||
|
|
||||
|
```text |
||||
|
http://127.0.0.1:8000/app |
||||
|
``` |
||||
|
|
||||
|
Default local bootstrap credentials: |
||||
|
|
||||
|
```text |
||||
|
username: admin |
||||
|
password: changeme-iron |
||||
|
``` |
||||
|
|
||||
|
For a practical local deployment with OpenList and Aliyun Drive, start with |
||||
|
[docs/openlist-local-deployment-sop.md](docs/openlist-local-deployment-sop.md). |
||||
|
|
||||
|
## Configuration |
||||
|
|
||||
|
Common environment variables: |
||||
|
|
||||
|
- `IRON_DATABASE_URL` |
||||
|
- `IRON_BOOTSTRAP_USERNAME` |
||||
|
- `IRON_BOOTSTRAP_PASSWORD` |
||||
|
- `IRON_SECRET_KEY` |
||||
|
- `IRON_AUTH_SESSION_TTL_HOURS` |
||||
|
- `IRON_UPLOAD_TEMP_DIR` |
||||
|
- `IRON_LOCAL_STORAGE_DIR` |
||||
|
- `IRON_CACHE_DIR` |
||||
|
- `IRON_JOB_POLL_INTERVAL_SECONDS` |
||||
|
- `IRON_JOB_BATCH_SIZE` |
||||
|
|
||||
|
Runtime data is intentionally ignored by Git. By default, local development may |
||||
|
create `.iron-storage/`, `.iron-temp/`, `iron.db`, and `output/`. |
||||
|
|
||||
|
`IRON_SECRET_KEY` encrypts stored backend credentials such as S3 keys and |
||||
|
WebDAV passwords. Override the development default before using real storage |
||||
|
credentials. |
||||
|
|
||||
|
## Development |
||||
|
|
||||
|
Build the frontend: |
||||
|
|
||||
|
```bash |
||||
|
npm run build |
||||
|
``` |
||||
|
|
||||
|
Run the full test suite: |
||||
|
|
||||
|
```bash |
||||
|
.venv/bin/python -m pytest |
||||
|
``` |
||||
|
|
||||
|
Current expected result: |
||||
|
|
||||
|
```text |
||||
|
78 passed |
||||
|
``` |
||||
|
|
||||
|
Run the real OpenList acceptance suite against a live local deployment: |
||||
|
|
||||
|
```bash |
||||
|
.venv/bin/python scripts/real_openlist_suite.py |
||||
|
``` |
||||
|
|
||||
|
Run the browser E2E flow directly: |
||||
|
|
||||
|
```bash |
||||
|
.venv/bin/python scripts/ui_playwright_smoke.py |
||||
|
``` |
||||
|
|
||||
|
Regenerate the README demo GIF: |
||||
|
|
||||
|
```bash |
||||
|
.venv/bin/python scripts/generate_readme_demo.py |
||||
|
``` |
||||
|
|
||||
|
Or through pytest: |
||||
|
|
||||
|
```bash |
||||
|
.venv/bin/python -m pytest tests/e2e/test_web_ui_playwright.py |
||||
|
``` |
||||
|
|
||||
|
Playwright screenshots and runtime artifacts are written to `output/playwright/`. |
||||
|
|
||||
|
## Repository Layout |
||||
|
|
||||
|
```text |
||||
|
app/ FastAPI application, services, repositories, schemas |
||||
|
alembic/ Database migrations |
||||
|
frontend/ Vite + React + TypeScript source |
||||
|
app/web/dist/ Built frontend assets served by FastAPI |
||||
|
scripts/ Local automation and E2E scripts |
||||
|
deploy/ OpenList compose, Iron env templates, and systemd examples |
||||
|
tests/ Backend, API, and browser E2E tests |
||||
|
docs/ Product, architecture, API, and maintenance docs |
||||
|
``` |
||||
|
|
||||
|
## Documentation |
||||
|
|
||||
|
Start with: |
||||
|
|
||||
|
- [Documentation index](docs/README.md) |
||||
|
- [Product](docs/product.md) |
||||
|
- [Architecture](docs/architecture.md) |
||||
|
- [API](docs/api.md) |
||||
|
- [Development](docs/development.md) |
||||
|
- [OpenList + Iron Local Deployment SOP](docs/openlist-local-deployment-sop.md) |
||||
|
|
||||
|
For contribution workflow, see [CONTRIBUTING.md](CONTRIBUTING.md). |
||||
|
For AI/coding-agent guidance, see [AGENTS.md](AGENTS.md). |
||||
|
|
||||
|
## Security Notes |
||||
|
|
||||
|
Iron is not yet a hardened public Internet service. Treat the current credentials |
||||
|
and local bearer-token session model as development-oriented defaults. Change the |
||||
|
bootstrap password and review deployment boundaries before exposing the service. |
||||
|
|
||||
|
Metadata import is guarded and requires both: |
||||
|
|
||||
|
- `confirm_replace=true` |
||||
|
- a fresh validation token from `POST /api/exports/metadata/restore-plan` |
||||
|
|
||||
|
## License |
||||
|
|
||||
|
No open source license has been selected yet. Add a `LICENSE` file before |
||||
|
publishing or accepting external contributions. |
||||
@ -0,0 +1,37 @@ |
|||||
|
[alembic] |
||||
|
script_location = alembic |
||||
|
prepend_sys_path = . |
||||
|
path_separator = os |
||||
|
sqlalchemy.url = sqlite+aiosqlite:///./iron.db |
||||
|
|
||||
|
[loggers] |
||||
|
keys = root,sqlalchemy,alembic |
||||
|
|
||||
|
[handlers] |
||||
|
keys = console |
||||
|
|
||||
|
[formatters] |
||||
|
keys = generic |
||||
|
|
||||
|
[logger_root] |
||||
|
level = WARN |
||||
|
handlers = console |
||||
|
|
||||
|
[logger_sqlalchemy] |
||||
|
level = WARN |
||||
|
handlers = |
||||
|
qualname = sqlalchemy.engine |
||||
|
|
||||
|
[logger_alembic] |
||||
|
level = INFO |
||||
|
handlers = console |
||||
|
qualname = alembic |
||||
|
|
||||
|
[handler_console] |
||||
|
class = StreamHandler |
||||
|
args = (sys.stderr,) |
||||
|
level = NOTSET |
||||
|
formatter = generic |
||||
|
|
||||
|
[formatter_generic] |
||||
|
format = %(levelname)-5.5s [%(name)s] %(message)s |
||||
@ -0,0 +1,62 @@ |
|||||
|
from logging.config import fileConfig |
||||
|
|
||||
|
from alembic import context |
||||
|
from sqlalchemy import pool |
||||
|
from sqlalchemy.engine import Connection |
||||
|
from sqlalchemy.ext.asyncio import async_engine_from_config |
||||
|
|
||||
|
from app.core.config import get_settings |
||||
|
from app.db.base import Base |
||||
|
import app.models # noqa: F401 |
||||
|
|
||||
|
config = context.config |
||||
|
|
||||
|
if config.config_file_name is not None: |
||||
|
fileConfig(config.config_file_name) |
||||
|
|
||||
|
target_metadata = Base.metadata |
||||
|
settings = get_settings() |
||||
|
configured_url = config.get_main_option("sqlalchemy.url") |
||||
|
if not configured_url: |
||||
|
config.set_main_option("sqlalchemy.url", settings.database_url) |
||||
|
|
||||
|
|
||||
|
def run_migrations_offline() -> None: |
||||
|
url = config.get_main_option("sqlalchemy.url") |
||||
|
context.configure( |
||||
|
url=url, |
||||
|
target_metadata=target_metadata, |
||||
|
literal_binds=True, |
||||
|
dialect_opts={"paramstyle": "named"}, |
||||
|
) |
||||
|
|
||||
|
with context.begin_transaction(): |
||||
|
context.run_migrations() |
||||
|
|
||||
|
|
||||
|
def do_run_migrations(connection: Connection) -> None: |
||||
|
context.configure(connection=connection, target_metadata=target_metadata) |
||||
|
|
||||
|
with context.begin_transaction(): |
||||
|
context.run_migrations() |
||||
|
|
||||
|
|
||||
|
async def run_migrations_online() -> None: |
||||
|
connectable = async_engine_from_config( |
||||
|
config.get_section(config.config_ini_section, {}), |
||||
|
prefix="sqlalchemy.", |
||||
|
poolclass=pool.NullPool, |
||||
|
) |
||||
|
|
||||
|
async with connectable.connect() as connection: |
||||
|
await connection.run_sync(do_run_migrations) |
||||
|
|
||||
|
await connectable.dispose() |
||||
|
|
||||
|
|
||||
|
if context.is_offline_mode(): |
||||
|
run_migrations_offline() |
||||
|
else: |
||||
|
import asyncio |
||||
|
|
||||
|
asyncio.run(run_migrations_online()) |
||||
@ -0,0 +1,24 @@ |
|||||
|
"""${message} |
||||
|
|
||||
|
Revision ID: ${up_revision} |
||||
|
Revises: ${down_revision | comma,n} |
||||
|
Create Date: ${create_date} |
||||
|
""" |
||||
|
from alembic import op |
||||
|
import sqlalchemy as sa |
||||
|
|
||||
|
|
||||
|
# revision identifiers, used by Alembic. |
||||
|
revision = ${repr(up_revision)} |
||||
|
down_revision = ${repr(down_revision)} |
||||
|
branch_labels = ${repr(branch_labels)} |
||||
|
depends_on = ${repr(depends_on)} |
||||
|
|
||||
|
|
||||
|
def upgrade() -> None: |
||||
|
pass |
||||
|
|
||||
|
|
||||
|
def downgrade() -> None: |
||||
|
pass |
||||
|
|
||||
@ -0,0 +1 @@ |
|||||
|
|
||||
@ -0,0 +1,327 @@ |
|||||
|
"""baseline schema |
||||
|
|
||||
|
Revision ID: 20260417_0007 |
||||
|
Revises: |
||||
|
Create Date: 2026-04-17 10:30:00.000000 |
||||
|
""" |
||||
|
|
||||
|
from alembic import op |
||||
|
import sqlalchemy as sa |
||||
|
|
||||
|
|
||||
|
# revision identifiers, used by Alembic. |
||||
|
revision = "20260417_0007" |
||||
|
down_revision = None |
||||
|
branch_labels = None |
||||
|
depends_on = None |
||||
|
|
||||
|
|
||||
|
def upgrade() -> None: |
||||
|
op.create_table( |
||||
|
"users", |
||||
|
sa.Column("id", sa.String(), nullable=False), |
||||
|
sa.Column("username", sa.String(), nullable=False), |
||||
|
sa.Column("password_hash", sa.String(), nullable=False), |
||||
|
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
||||
|
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
||||
|
sa.PrimaryKeyConstraint("id"), |
||||
|
sa.UniqueConstraint("username"), |
||||
|
) |
||||
|
|
||||
|
op.create_table( |
||||
|
"auth_sessions", |
||||
|
sa.Column("id", sa.String(), nullable=False), |
||||
|
sa.Column("user_id", sa.String(), nullable=False), |
||||
|
sa.Column("token_hash", sa.String(), nullable=False), |
||||
|
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=False), |
||||
|
sa.Column("revoked_at", sa.DateTime(timezone=True), nullable=True), |
||||
|
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
||||
|
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
||||
|
sa.ForeignKeyConstraint(["user_id"], ["users.id"]), |
||||
|
sa.PrimaryKeyConstraint("id"), |
||||
|
) |
||||
|
op.create_index("idx_auth_sessions_user_status", "auth_sessions", ["user_id", "revoked_at", "expires_at"]) |
||||
|
op.create_index("idx_auth_sessions_token_hash", "auth_sessions", ["token_hash"], unique=True) |
||||
|
|
||||
|
op.create_table( |
||||
|
"directories", |
||||
|
sa.Column("id", sa.String(), nullable=False), |
||||
|
sa.Column("parent_id", sa.String(), nullable=True), |
||||
|
sa.Column("name", sa.String(), nullable=False), |
||||
|
sa.Column("path_key", sa.String(), nullable=False), |
||||
|
sa.Column("deleted_at", sa.DateTime(timezone=True), nullable=True), |
||||
|
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
||||
|
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
||||
|
sa.ForeignKeyConstraint(["parent_id"], ["directories.id"]), |
||||
|
sa.PrimaryKeyConstraint("id"), |
||||
|
sa.UniqueConstraint("path_key"), |
||||
|
) |
||||
|
op.create_index("idx_directories_parent_id", "directories", ["parent_id"]) |
||||
|
op.create_index("idx_directories_path_key", "directories", ["path_key"]) |
||||
|
|
||||
|
op.create_table( |
||||
|
"backends", |
||||
|
sa.Column("id", sa.String(), nullable=False), |
||||
|
sa.Column("name", sa.String(), nullable=False), |
||||
|
sa.Column("type", sa.String(), nullable=False), |
||||
|
sa.Column("stability_class", sa.String(), nullable=False), |
||||
|
sa.Column("read_priority", sa.Integer(), nullable=False), |
||||
|
sa.Column("write_priority", sa.Integer(), nullable=False), |
||||
|
sa.Column("is_enabled", sa.Boolean(), server_default="1", nullable=False), |
||||
|
sa.Column("public_config_json", sa.Text(), nullable=False), |
||||
|
sa.Column("secret_config_json", sa.Text(), nullable=False), |
||||
|
sa.Column("capacity_hint_bytes", sa.Integer(), nullable=True), |
||||
|
sa.Column("last_health_status", sa.String(), nullable=True), |
||||
|
sa.Column("last_health_checked_at", sa.DateTime(timezone=True), nullable=True), |
||||
|
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
||||
|
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
||||
|
sa.PrimaryKeyConstraint("id"), |
||||
|
sa.UniqueConstraint("name"), |
||||
|
) |
||||
|
op.create_index("idx_backends_enabled", "backends", ["is_enabled"]) |
||||
|
op.create_index("idx_backends_priority", "backends", ["write_priority", "read_priority"]) |
||||
|
|
||||
|
op.create_table( |
||||
|
"placement_policies", |
||||
|
sa.Column("id", sa.String(), nullable=False), |
||||
|
sa.Column("file_class", sa.String(), nullable=False), |
||||
|
sa.Column("require_local", sa.Boolean(), server_default="1", nullable=False), |
||||
|
sa.Column("stable_replica_count", sa.Integer(), server_default="1", nullable=False), |
||||
|
sa.Column("opportunistic_replica_count", sa.Integer(), server_default="0", nullable=False), |
||||
|
sa.Column("preferred_backend_ids_json", sa.Text(), server_default="[]", nullable=False), |
||||
|
sa.Column("excluded_backend_ids_json", sa.Text(), server_default="[]", nullable=False), |
||||
|
sa.Column("max_non_local_size_bytes", sa.Integer(), nullable=True), |
||||
|
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
||||
|
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
||||
|
sa.PrimaryKeyConstraint("id"), |
||||
|
) |
||||
|
op.create_index("idx_placement_policies_file_class", "placement_policies", ["file_class"], unique=True) |
||||
|
|
||||
|
op.create_table( |
||||
|
"upload_sessions", |
||||
|
sa.Column("id", sa.String(), nullable=False), |
||||
|
sa.Column("directory_id", sa.String(), nullable=True), |
||||
|
sa.Column("filename", sa.String(), nullable=True), |
||||
|
sa.Column("total_size_bytes", sa.Integer(), nullable=False), |
||||
|
sa.Column("received_size_bytes", sa.Integer(), nullable=False), |
||||
|
sa.Column("status", sa.String(), nullable=False), |
||||
|
sa.Column("temp_path", sa.String(), nullable=False), |
||||
|
sa.Column("tus_upload_url", sa.String(), nullable=True), |
||||
|
sa.Column("upload_metadata_json", sa.Text(), nullable=True), |
||||
|
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
||||
|
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
||||
|
sa.ForeignKeyConstraint(["directory_id"], ["directories.id"]), |
||||
|
sa.PrimaryKeyConstraint("id"), |
||||
|
) |
||||
|
op.create_index("idx_upload_sessions_status", "upload_sessions", ["status"]) |
||||
|
op.create_index("idx_upload_sessions_updated_at", "upload_sessions", ["updated_at"]) |
||||
|
|
||||
|
op.create_table( |
||||
|
"file_entries", |
||||
|
sa.Column("id", sa.String(), nullable=False), |
||||
|
sa.Column("directory_id", sa.String(), nullable=False), |
||||
|
sa.Column("name", sa.String(), nullable=False), |
||||
|
sa.Column("mime_type", sa.String(), nullable=True), |
||||
|
sa.Column("size_bytes", sa.Integer(), nullable=False), |
||||
|
sa.Column("current_version_id", sa.String(), nullable=True), |
||||
|
sa.Column("is_deleted", sa.Boolean(), server_default="0", nullable=False), |
||||
|
sa.Column("deleted_at", sa.DateTime(timezone=True), nullable=True), |
||||
|
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
||||
|
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
||||
|
sa.ForeignKeyConstraint(["directory_id"], ["directories.id"]), |
||||
|
sa.PrimaryKeyConstraint("id"), |
||||
|
) |
||||
|
op.create_index("idx_file_entries_current_version_id", "file_entries", ["current_version_id"]) |
||||
|
op.create_index("idx_file_entries_deleted_at", "file_entries", ["deleted_at"]) |
||||
|
op.create_index( |
||||
|
"idx_file_entries_directory_name", |
||||
|
"file_entries", |
||||
|
["directory_id", "name", "is_deleted"], |
||||
|
unique=True, |
||||
|
) |
||||
|
|
||||
|
op.create_table( |
||||
|
"file_versions", |
||||
|
sa.Column("id", sa.String(), nullable=False), |
||||
|
sa.Column("file_entry_id", sa.String(), nullable=False), |
||||
|
sa.Column("content_hash", sa.String(), nullable=False), |
||||
|
sa.Column("size_bytes", sa.Integer(), nullable=False), |
||||
|
sa.Column("storage_layout", sa.String(), nullable=False), |
||||
|
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
||||
|
sa.ForeignKeyConstraint(["file_entry_id"], ["file_entries.id"]), |
||||
|
sa.PrimaryKeyConstraint("id"), |
||||
|
) |
||||
|
op.create_index("idx_file_versions_content_hash", "file_versions", ["content_hash"]) |
||||
|
op.create_index("idx_file_versions_file_entry_id", "file_versions", ["file_entry_id"]) |
||||
|
|
||||
|
op.create_table( |
||||
|
"blobs", |
||||
|
sa.Column("id", sa.String(), nullable=False), |
||||
|
sa.Column("file_version_id", sa.String(), nullable=False), |
||||
|
sa.Column("blob_index", sa.Integer(), nullable=False), |
||||
|
sa.Column("kind", sa.String(), nullable=False), |
||||
|
sa.Column("content_hash", sa.String(), nullable=False), |
||||
|
sa.Column("size_bytes", sa.Integer(), nullable=False), |
||||
|
sa.Column("logical_offset", sa.Integer(), server_default="0", nullable=False), |
||||
|
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
||||
|
sa.ForeignKeyConstraint(["file_version_id"], ["file_versions.id"]), |
||||
|
sa.PrimaryKeyConstraint("id"), |
||||
|
) |
||||
|
op.create_index("idx_blobs_content_hash", "blobs", ["content_hash"]) |
||||
|
op.create_index("idx_blobs_file_version_index", "blobs", ["file_version_id", "blob_index"], unique=True) |
||||
|
|
||||
|
op.create_table( |
||||
|
"blob_replicas", |
||||
|
sa.Column("id", sa.String(), nullable=False), |
||||
|
sa.Column("blob_id", sa.String(), nullable=False), |
||||
|
sa.Column("backend_id", sa.String(), nullable=False), |
||||
|
sa.Column("storage_key", sa.String(), nullable=False), |
||||
|
sa.Column("status", sa.String(), nullable=False), |
||||
|
sa.Column("etag", sa.String(), nullable=True), |
||||
|
sa.Column("checksum", sa.String(), nullable=True), |
||||
|
sa.Column("size_bytes", sa.Integer(), nullable=False), |
||||
|
sa.Column("last_verified_at", sa.DateTime(timezone=True), nullable=True), |
||||
|
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
||||
|
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
||||
|
sa.ForeignKeyConstraint(["backend_id"], ["backends.id"]), |
||||
|
sa.ForeignKeyConstraint(["blob_id"], ["blobs.id"]), |
||||
|
sa.PrimaryKeyConstraint("id"), |
||||
|
) |
||||
|
op.create_index("idx_blob_replicas_backend_status", "blob_replicas", ["backend_id", "status"]) |
||||
|
op.create_index("idx_blob_replicas_blob_backend", "blob_replicas", ["blob_id", "backend_id"], unique=True) |
||||
|
op.create_index("idx_blob_replicas_storage_key", "blob_replicas", ["storage_key"]) |
||||
|
|
||||
|
op.create_table( |
||||
|
"upload_session_parts", |
||||
|
sa.Column("id", sa.String(), nullable=False), |
||||
|
sa.Column("upload_session_id", sa.String(), nullable=False), |
||||
|
sa.Column("part_number", sa.Integer(), nullable=False), |
||||
|
sa.Column("byte_offset", sa.Integer(), nullable=False), |
||||
|
sa.Column("size_bytes", sa.Integer(), nullable=False), |
||||
|
sa.Column("checksum", sa.String(), nullable=True), |
||||
|
sa.Column("status", sa.String(), nullable=False), |
||||
|
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
||||
|
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
||||
|
sa.ForeignKeyConstraint(["upload_session_id"], ["upload_sessions.id"]), |
||||
|
sa.PrimaryKeyConstraint("id"), |
||||
|
) |
||||
|
op.create_index( |
||||
|
"idx_upload_session_parts_unique", |
||||
|
"upload_session_parts", |
||||
|
["upload_session_id", "part_number"], |
||||
|
unique=True, |
||||
|
) |
||||
|
op.create_index("idx_upload_session_parts_upload_id", "upload_session_parts", ["upload_session_id"]) |
||||
|
|
||||
|
op.create_table( |
||||
|
"jobs", |
||||
|
sa.Column("id", sa.String(), nullable=False), |
||||
|
sa.Column("kind", sa.String(), nullable=False), |
||||
|
sa.Column("status", sa.String(), nullable=False), |
||||
|
sa.Column("payload_json", sa.Text(), nullable=False), |
||||
|
sa.Column("attempt_count", sa.Integer(), server_default="0", nullable=False), |
||||
|
sa.Column("max_attempts", sa.Integer(), server_default="5", nullable=False), |
||||
|
sa.Column("run_after", sa.DateTime(timezone=True), nullable=False), |
||||
|
sa.Column("last_error", sa.Text(), nullable=True), |
||||
|
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
||||
|
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
||||
|
sa.PrimaryKeyConstraint("id"), |
||||
|
) |
||||
|
op.create_index("idx_jobs_kind_status", "jobs", ["kind", "status"]) |
||||
|
op.create_index("idx_jobs_status_run_after", "jobs", ["status", "run_after"]) |
||||
|
|
||||
|
op.create_table( |
||||
|
"preview_artifacts", |
||||
|
sa.Column("id", sa.String(), nullable=False), |
||||
|
sa.Column("file_version_id", sa.String(), nullable=False), |
||||
|
sa.Column("artifact_type", sa.String(), nullable=False), |
||||
|
sa.Column("blob_id", sa.String(), nullable=False), |
||||
|
sa.Column("status", sa.String(), nullable=False), |
||||
|
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
||||
|
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
||||
|
sa.ForeignKeyConstraint(["blob_id"], ["blobs.id"]), |
||||
|
sa.ForeignKeyConstraint(["file_version_id"], ["file_versions.id"]), |
||||
|
sa.PrimaryKeyConstraint("id"), |
||||
|
) |
||||
|
op.create_index( |
||||
|
"idx_preview_artifacts_version_type", |
||||
|
"preview_artifacts", |
||||
|
["file_version_id", "artifact_type"], |
||||
|
unique=True, |
||||
|
) |
||||
|
|
||||
|
op.create_table( |
||||
|
"cache_entries", |
||||
|
sa.Column("id", sa.String(), nullable=False), |
||||
|
sa.Column("blob_id", sa.String(), nullable=False), |
||||
|
sa.Column("local_path", sa.String(), nullable=False), |
||||
|
sa.Column("cache_kind", sa.String(), nullable=False), |
||||
|
sa.Column("size_bytes", sa.Integer(), nullable=False), |
||||
|
sa.Column("status", sa.String(), nullable=False), |
||||
|
sa.Column("last_accessed_at", sa.DateTime(timezone=True), nullable=False), |
||||
|
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=True), |
||||
|
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
||||
|
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False), |
||||
|
sa.ForeignKeyConstraint(["blob_id"], ["blobs.id"]), |
||||
|
sa.PrimaryKeyConstraint("id"), |
||||
|
) |
||||
|
op.create_index("idx_cache_entries_blob_id", "cache_entries", ["blob_id"]) |
||||
|
op.create_index("idx_cache_entries_expires_at", "cache_entries", ["expires_at"]) |
||||
|
op.create_index("idx_cache_entries_last_accessed_at", "cache_entries", ["last_accessed_at"]) |
||||
|
|
||||
|
|
||||
|
def downgrade() -> None: |
||||
|
op.drop_index("idx_cache_entries_last_accessed_at", table_name="cache_entries") |
||||
|
op.drop_index("idx_cache_entries_expires_at", table_name="cache_entries") |
||||
|
op.drop_index("idx_cache_entries_blob_id", table_name="cache_entries") |
||||
|
op.drop_table("cache_entries") |
||||
|
|
||||
|
op.drop_index("idx_preview_artifacts_version_type", table_name="preview_artifacts") |
||||
|
op.drop_table("preview_artifacts") |
||||
|
|
||||
|
op.drop_index("idx_jobs_status_run_after", table_name="jobs") |
||||
|
op.drop_index("idx_jobs_kind_status", table_name="jobs") |
||||
|
op.drop_table("jobs") |
||||
|
|
||||
|
op.drop_index("idx_upload_session_parts_upload_id", table_name="upload_session_parts") |
||||
|
op.drop_index("idx_upload_session_parts_unique", table_name="upload_session_parts") |
||||
|
op.drop_table("upload_session_parts") |
||||
|
|
||||
|
op.drop_index("idx_blob_replicas_storage_key", table_name="blob_replicas") |
||||
|
op.drop_index("idx_blob_replicas_blob_backend", table_name="blob_replicas") |
||||
|
op.drop_index("idx_blob_replicas_backend_status", table_name="blob_replicas") |
||||
|
op.drop_table("blob_replicas") |
||||
|
|
||||
|
op.drop_index("idx_placement_policies_file_class", table_name="placement_policies") |
||||
|
op.drop_table("placement_policies") |
||||
|
|
||||
|
op.drop_index("idx_blobs_file_version_index", table_name="blobs") |
||||
|
op.drop_index("idx_blobs_content_hash", table_name="blobs") |
||||
|
op.drop_table("blobs") |
||||
|
|
||||
|
op.drop_index("idx_file_versions_file_entry_id", table_name="file_versions") |
||||
|
op.drop_index("idx_file_versions_content_hash", table_name="file_versions") |
||||
|
op.drop_table("file_versions") |
||||
|
|
||||
|
op.drop_index("idx_file_entries_deleted_at", table_name="file_entries") |
||||
|
op.drop_index("idx_file_entries_directory_name", table_name="file_entries") |
||||
|
op.drop_index("idx_file_entries_current_version_id", table_name="file_entries") |
||||
|
op.drop_table("file_entries") |
||||
|
|
||||
|
op.drop_index("idx_upload_sessions_updated_at", table_name="upload_sessions") |
||||
|
op.drop_index("idx_upload_sessions_status", table_name="upload_sessions") |
||||
|
op.drop_table("upload_sessions") |
||||
|
|
||||
|
op.drop_index("idx_backends_priority", table_name="backends") |
||||
|
op.drop_index("idx_backends_enabled", table_name="backends") |
||||
|
op.drop_table("backends") |
||||
|
|
||||
|
op.drop_index("idx_directories_path_key", table_name="directories") |
||||
|
op.drop_index("idx_directories_parent_id", table_name="directories") |
||||
|
op.drop_table("directories") |
||||
|
|
||||
|
op.drop_index("idx_auth_sessions_token_hash", table_name="auth_sessions") |
||||
|
op.drop_index("idx_auth_sessions_user_status", table_name="auth_sessions") |
||||
|
op.drop_table("auth_sessions") |
||||
|
|
||||
|
op.drop_table("users") |
||||
@ -0,0 +1,2 @@ |
|||||
|
"""Iron gateway application package.""" |
||||
|
|
||||
@ -0,0 +1,2 @@ |
|||||
|
"""External adapters.""" |
||||
|
|
||||
@ -0,0 +1,2 @@ |
|||||
|
"""Storage adapter implementations.""" |
||||
|
|
||||
@ -0,0 +1,18 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from pathlib import Path |
||||
|
from typing import Protocol |
||||
|
|
||||
|
|
||||
|
class StorageAdapter(Protocol): |
||||
|
async def check(self) -> dict[str, str]: |
||||
|
... |
||||
|
|
||||
|
async def put_file(self, source_path: str, storage_key: str) -> dict[str, str | None]: |
||||
|
... |
||||
|
|
||||
|
async def download_file(self, storage_key: str, destination_path: Path) -> Path: |
||||
|
... |
||||
|
|
||||
|
async def object_exists(self, storage_key: str) -> bool: |
||||
|
... |
||||
@ -0,0 +1,50 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
import shutil |
||||
|
from pathlib import Path |
||||
|
|
||||
|
import anyio |
||||
|
|
||||
|
|
||||
|
class LocalStorageAdapter: |
||||
|
def __init__(self, base_path: str) -> None: |
||||
|
self.base_path = Path(base_path) |
||||
|
|
||||
|
async def ensure_base_path(self) -> Path: |
||||
|
await anyio.to_thread.run_sync(lambda: self.base_path.mkdir(parents=True, exist_ok=True)) |
||||
|
return self.base_path |
||||
|
|
||||
|
async def check(self) -> dict[str, str]: |
||||
|
path = await self.ensure_base_path() |
||||
|
is_dir = await anyio.to_thread.run_sync(path.is_dir) |
||||
|
if not is_dir: |
||||
|
raise NotADirectoryError(str(path)) |
||||
|
return {"status": "healthy", "detail": str(path)} |
||||
|
|
||||
|
async def put_file(self, source_path: str, storage_key: str) -> dict[str, str | None]: |
||||
|
destination = self.base_path / storage_key |
||||
|
await anyio.to_thread.run_sync(lambda: destination.parent.mkdir(parents=True, exist_ok=True)) |
||||
|
|
||||
|
def _copy_if_needed() -> None: |
||||
|
if not destination.exists(): |
||||
|
shutil.copyfile(source_path, destination) |
||||
|
|
||||
|
await anyio.to_thread.run_sync(_copy_if_needed) |
||||
|
return {"etag": None, "storage_key": storage_key} |
||||
|
|
||||
|
async def download_file(self, storage_key: str, destination_path: Path) -> Path: |
||||
|
source = await self.resolve_path(storage_key) |
||||
|
await anyio.to_thread.run_sync(lambda: destination_path.parent.mkdir(parents=True, exist_ok=True)) |
||||
|
await anyio.to_thread.run_sync(lambda: shutil.copyfile(source, destination_path)) |
||||
|
return destination_path |
||||
|
|
||||
|
async def resolve_path(self, storage_key: str) -> Path: |
||||
|
destination = self.base_path / storage_key |
||||
|
exists = await anyio.to_thread.run_sync(destination.exists) |
||||
|
if not exists: |
||||
|
raise FileNotFoundError(storage_key) |
||||
|
return destination |
||||
|
|
||||
|
async def object_exists(self, storage_key: str) -> bool: |
||||
|
destination = self.base_path / storage_key |
||||
|
return await anyio.to_thread.run_sync(destination.exists) |
||||
@ -0,0 +1,18 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from typing import Any |
||||
|
|
||||
|
from app.adapters.storage.base import StorageAdapter |
||||
|
from app.adapters.storage.local import LocalStorageAdapter |
||||
|
from app.adapters.storage.s3 import S3StorageAdapter |
||||
|
from app.adapters.storage.webdav import WebDAVStorageAdapter |
||||
|
|
||||
|
|
||||
|
def build_storage_adapter(backend_type: str, config: dict[str, Any]) -> StorageAdapter: |
||||
|
if backend_type == "local_directory": |
||||
|
return LocalStorageAdapter(config["base_path"]) |
||||
|
if backend_type == "s3": |
||||
|
return S3StorageAdapter(config) |
||||
|
if backend_type == "webdav": |
||||
|
return WebDAVStorageAdapter(config) |
||||
|
raise ValueError(f"Unsupported backend type: {backend_type}") |
||||
@ -0,0 +1,88 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from pathlib import Path |
||||
|
from typing import Any |
||||
|
|
||||
|
import anyio |
||||
|
import boto3 |
||||
|
from botocore.config import Config |
||||
|
from botocore.exceptions import ClientError |
||||
|
|
||||
|
|
||||
|
class S3StorageAdapter: |
||||
|
def __init__(self, config: dict[str, Any]) -> None: |
||||
|
self.config = config |
||||
|
|
||||
|
async def check(self) -> dict[str, str]: |
||||
|
client = self._create_client() |
||||
|
|
||||
|
def _check() -> dict[str, str]: |
||||
|
client.head_bucket(Bucket=self.config["bucket"]) |
||||
|
return {"status": "healthy", "detail": self.config["bucket"]} |
||||
|
|
||||
|
return await anyio.to_thread.run_sync(_check) |
||||
|
|
||||
|
async def put_file(self, source_path: str, storage_key: str) -> dict[str, str | None]: |
||||
|
client = self._create_client() |
||||
|
object_key = self._object_key(storage_key) |
||||
|
|
||||
|
def _put() -> dict[str, str | None]: |
||||
|
extra_args: dict[str, Any] = {} |
||||
|
if self.config.get("storage_class"): |
||||
|
extra_args["StorageClass"] = self.config["storage_class"] |
||||
|
if self.config.get("server_side_encryption"): |
||||
|
extra_args["ServerSideEncryption"] = self.config["server_side_encryption"] |
||||
|
client.upload_file(source_path, self.config["bucket"], object_key, ExtraArgs=extra_args) |
||||
|
metadata = client.head_object(Bucket=self.config["bucket"], Key=object_key) |
||||
|
return {"etag": metadata.get("ETag"), "storage_key": storage_key} |
||||
|
|
||||
|
return await anyio.to_thread.run_sync(_put) |
||||
|
|
||||
|
async def download_file(self, storage_key: str, destination_path: Path) -> Path: |
||||
|
client = self._create_client() |
||||
|
object_key = self._object_key(storage_key) |
||||
|
|
||||
|
def _download() -> Path: |
||||
|
destination_path.parent.mkdir(parents=True, exist_ok=True) |
||||
|
client.download_file(self.config["bucket"], object_key, str(destination_path)) |
||||
|
return destination_path |
||||
|
|
||||
|
return await anyio.to_thread.run_sync(_download) |
||||
|
|
||||
|
async def object_exists(self, storage_key: str) -> bool: |
||||
|
client = self._create_client() |
||||
|
object_key = self._object_key(storage_key) |
||||
|
|
||||
|
def _exists() -> bool: |
||||
|
try: |
||||
|
client.head_object(Bucket=self.config["bucket"], Key=object_key) |
||||
|
return True |
||||
|
except ClientError as exc: |
||||
|
error_code = exc.response.get("Error", {}).get("Code") |
||||
|
if error_code in {"404", "NoSuchKey", "NotFound"}: |
||||
|
return False |
||||
|
raise |
||||
|
|
||||
|
return await anyio.to_thread.run_sync(_exists) |
||||
|
|
||||
|
def _create_client(self): |
||||
|
client_config = None |
||||
|
if self.config.get("force_path_style"): |
||||
|
client_config = Config(s3={"addressing_style": "path"}) |
||||
|
client_kwargs = { |
||||
|
"service_name": "s3", |
||||
|
"region_name": self.config.get("region"), |
||||
|
"endpoint_url": self.config.get("endpoint_url"), |
||||
|
"aws_access_key_id": self.config.get("access_key_id"), |
||||
|
"aws_secret_access_key": self.config.get("secret_access_key"), |
||||
|
"verify": self.config.get("verify_ssl", True), |
||||
|
"config": client_config, |
||||
|
} |
||||
|
cleaned_kwargs = {key: value for key, value in client_kwargs.items() if value is not None} |
||||
|
return boto3.client(**cleaned_kwargs) |
||||
|
|
||||
|
def _object_key(self, storage_key: str) -> str: |
||||
|
prefix = (self.config.get("prefix") or "").strip("/") |
||||
|
if not prefix: |
||||
|
return storage_key |
||||
|
return f"{prefix}/{storage_key}" |
||||
@ -0,0 +1,145 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
import base64 |
||||
|
import ssl |
||||
|
import urllib.error |
||||
|
import urllib.parse |
||||
|
import urllib.request |
||||
|
import xml.etree.ElementTree as ET |
||||
|
from pathlib import Path |
||||
|
from typing import Any |
||||
|
|
||||
|
import anyio |
||||
|
|
||||
|
|
||||
|
DAV_NAMESPACE = {"d": "DAV:"} |
||||
|
|
||||
|
|
||||
|
class WebDAVStorageAdapter: |
||||
|
def __init__(self, config: dict[str, Any]) -> None: |
||||
|
self.config = config |
||||
|
self.base_url = config["endpoint_url"].rstrip("/") |
||||
|
self.root_path = self._normalize_relative_path(config.get("root_path") or "iron") |
||||
|
self._ssl_context = None if config.get("verify_ssl", True) else ssl._create_unverified_context() |
||||
|
|
||||
|
async def check(self) -> dict[str, str]: |
||||
|
def _check() -> dict[str, str]: |
||||
|
self._ensure_directory(self.root_path) |
||||
|
return {"status": "healthy", "detail": self.root_path} |
||||
|
|
||||
|
return await anyio.to_thread.run_sync(_check) |
||||
|
|
||||
|
async def put_file(self, source_path: str, storage_key: str) -> dict[str, str | None]: |
||||
|
def _put() -> dict[str, str | None]: |
||||
|
source = Path(source_path) |
||||
|
if not source.exists(): |
||||
|
raise FileNotFoundError(source_path) |
||||
|
normalized_key = self._normalize_relative_path(storage_key) |
||||
|
segments = normalized_key.split("/") |
||||
|
if len(segments) > 1: |
||||
|
self._ensure_directory(f"{self.root_path}/{'/'.join(segments[:-1])}") |
||||
|
request = urllib.request.Request( |
||||
|
self._build_url(f"{self.root_path}/{normalized_key}"), |
||||
|
data=source.read_bytes(), |
||||
|
method="PUT", |
||||
|
headers=self._auth_headers(), |
||||
|
) |
||||
|
with urllib.request.urlopen(request, timeout=60, context=self._ssl_context) as response: |
||||
|
etag = response.headers.get("ETag") |
||||
|
return {"etag": etag, "storage_key": normalized_key} |
||||
|
|
||||
|
return await anyio.to_thread.run_sync(_put) |
||||
|
|
||||
|
async def download_file(self, storage_key: str, destination_path: Path) -> Path: |
||||
|
def _download() -> Path: |
||||
|
normalized_key = self._normalize_relative_path(storage_key) |
||||
|
request = urllib.request.Request( |
||||
|
self._build_url(f"{self.root_path}/{normalized_key}"), |
||||
|
method="GET", |
||||
|
headers=self._auth_headers(), |
||||
|
) |
||||
|
try: |
||||
|
with urllib.request.urlopen(request, timeout=60, context=self._ssl_context) as response: |
||||
|
body = response.read() |
||||
|
except urllib.error.HTTPError as exc: |
||||
|
if exc.code == 404: |
||||
|
raise FileNotFoundError(storage_key) from exc |
||||
|
raise |
||||
|
destination_path.parent.mkdir(parents=True, exist_ok=True) |
||||
|
destination_path.write_bytes(body) |
||||
|
return destination_path |
||||
|
|
||||
|
return await anyio.to_thread.run_sync(_download) |
||||
|
|
||||
|
async def object_exists(self, storage_key: str) -> bool: |
||||
|
def _exists() -> bool: |
||||
|
normalized_key = self._normalize_relative_path(storage_key) |
||||
|
request = urllib.request.Request( |
||||
|
self._build_url(f"{self.root_path}/{normalized_key}"), |
||||
|
method="HEAD", |
||||
|
headers=self._auth_headers(), |
||||
|
) |
||||
|
try: |
||||
|
with urllib.request.urlopen(request, timeout=20, context=self._ssl_context): |
||||
|
return True |
||||
|
except urllib.error.HTTPError as exc: |
||||
|
if exc.code == 404: |
||||
|
return False |
||||
|
if exc.code in {403, 405}: |
||||
|
return self._path_exists(f"{self.root_path}/{normalized_key}") |
||||
|
raise |
||||
|
|
||||
|
return await anyio.to_thread.run_sync(_exists) |
||||
|
|
||||
|
def _ensure_directory(self, relative_path: str) -> None: |
||||
|
current = "" |
||||
|
for segment in self._normalize_relative_path(relative_path).split("/"): |
||||
|
current = f"{current}/{segment}".strip("/") |
||||
|
if self._path_exists(current): |
||||
|
continue |
||||
|
request = urllib.request.Request( |
||||
|
self._build_url(current), |
||||
|
method="MKCOL", |
||||
|
headers=self._auth_headers(), |
||||
|
) |
||||
|
try: |
||||
|
with urllib.request.urlopen(request, timeout=20, context=self._ssl_context): |
||||
|
pass |
||||
|
except urllib.error.HTTPError as exc: |
||||
|
if exc.code not in {201, 405}: |
||||
|
raise RuntimeError(f"WebDAV MKCOL failed with {exc.code} for {current}") from exc |
||||
|
|
||||
|
def _path_exists(self, relative_path: str) -> bool: |
||||
|
request = urllib.request.Request( |
||||
|
self._build_url(relative_path), |
||||
|
method="PROPFIND", |
||||
|
headers={**self._auth_headers(), "Depth": "0"}, |
||||
|
) |
||||
|
try: |
||||
|
with urllib.request.urlopen(request, timeout=20, context=self._ssl_context) as response: |
||||
|
body = response.read() |
||||
|
except urllib.error.HTTPError as exc: |
||||
|
if exc.code == 404: |
||||
|
return False |
||||
|
raise RuntimeError(f"WebDAV PROPFIND failed with {exc.code} for {relative_path}") from exc |
||||
|
if not body: |
||||
|
return True |
||||
|
root = ET.fromstring(body) |
||||
|
return root.find("d:response", DAV_NAMESPACE) is not None |
||||
|
|
||||
|
def _build_url(self, relative_path: str) -> str: |
||||
|
safe_path = "/".join(urllib.parse.quote(part, safe="") for part in relative_path.strip("/").split("/") if part) |
||||
|
return f"{self.base_url}/{safe_path}" |
||||
|
|
||||
|
def _auth_headers(self) -> dict[str, str]: |
||||
|
username = str(self.config.get("username") or "") |
||||
|
password = str(self.config.get("password") or "") |
||||
|
token = base64.b64encode(f"{username}:{password}".encode("utf-8")).decode("ascii") |
||||
|
return {"Authorization": f"Basic {token}"} |
||||
|
|
||||
|
@staticmethod |
||||
|
def _normalize_relative_path(value: str) -> str: |
||||
|
normalized = "/".join(part for part in value.strip().strip("/").split("/") if part) |
||||
|
if not normalized: |
||||
|
raise ValueError("path must not be empty") |
||||
|
return normalized |
||||
@ -0,0 +1,2 @@ |
|||||
|
"""HTTP API layer.""" |
||||
|
|
||||
@ -0,0 +1,76 @@ |
|||||
|
from fastapi import Depends |
||||
|
from fastapi import Header |
||||
|
from sqlalchemy.ext.asyncio import AsyncSession |
||||
|
|
||||
|
from app.db.session import get_db_session |
||||
|
from app.models.entities import User |
||||
|
from app.services.export_service import ExportService |
||||
|
from app.services.auth_service import AuthService |
||||
|
from app.services.backend_service import BackendService |
||||
|
from app.services.directory_service import DirectoryService |
||||
|
from app.services.file_service import FileService |
||||
|
from app.services.job_service import JobService |
||||
|
from app.services.policy_service import PolicyService |
||||
|
from app.services.repair_service import RepairService |
||||
|
from app.services.upload_service import UploadService |
||||
|
|
||||
|
|
||||
|
async def get_directory_service( |
||||
|
session: AsyncSession = Depends(get_db_session), |
||||
|
) -> DirectoryService: |
||||
|
return DirectoryService(session) |
||||
|
|
||||
|
|
||||
|
async def get_file_service( |
||||
|
session: AsyncSession = Depends(get_db_session), |
||||
|
) -> FileService: |
||||
|
return FileService(session) |
||||
|
|
||||
|
|
||||
|
async def get_upload_service( |
||||
|
session: AsyncSession = Depends(get_db_session), |
||||
|
) -> UploadService: |
||||
|
return UploadService(session) |
||||
|
|
||||
|
|
||||
|
async def get_backend_service( |
||||
|
session: AsyncSession = Depends(get_db_session), |
||||
|
) -> BackendService: |
||||
|
return BackendService(session) |
||||
|
|
||||
|
|
||||
|
async def get_job_service( |
||||
|
session: AsyncSession = Depends(get_db_session), |
||||
|
) -> JobService: |
||||
|
return JobService(session) |
||||
|
|
||||
|
|
||||
|
async def get_export_service( |
||||
|
session: AsyncSession = Depends(get_db_session), |
||||
|
) -> ExportService: |
||||
|
return ExportService(session) |
||||
|
|
||||
|
|
||||
|
async def get_repair_service( |
||||
|
session: AsyncSession = Depends(get_db_session), |
||||
|
) -> RepairService: |
||||
|
return RepairService(session) |
||||
|
|
||||
|
|
||||
|
async def get_auth_service( |
||||
|
session: AsyncSession = Depends(get_db_session), |
||||
|
) -> AuthService: |
||||
|
return AuthService(session) |
||||
|
|
||||
|
|
||||
|
async def get_policy_service( |
||||
|
session: AsyncSession = Depends(get_db_session), |
||||
|
) -> PolicyService: |
||||
|
return PolicyService(session) |
||||
|
|
||||
|
|
||||
|
async def require_current_user( |
||||
|
authorization: str | None = Header(default=None, alias="Authorization"), |
||||
|
service: AuthService = Depends(get_auth_service), |
||||
|
) -> User: |
||||
|
return await service.get_current_user(authorization) |
||||
@ -0,0 +1,59 @@ |
|||||
|
from fastapi import APIRouter |
||||
|
from fastapi import Depends |
||||
|
|
||||
|
from app.api.dependencies import require_current_user |
||||
|
from app.api.routes.auth import router as auth_router |
||||
|
from app.api.routes.backends import router as backend_router |
||||
|
from app.api.routes.directories import router as directory_router |
||||
|
from app.api.routes.exports import router as export_router |
||||
|
from app.api.routes.files import router as file_router |
||||
|
from app.api.routes.jobs import router as job_router |
||||
|
from app.api.routes.policies import router as policy_router |
||||
|
from app.api.routes.system import router as system_router |
||||
|
from app.api.routes.uploads import router as upload_router |
||||
|
|
||||
|
api_router = APIRouter() |
||||
|
api_router.include_router(system_router, prefix="/system", tags=["system"]) |
||||
|
api_router.include_router(auth_router, prefix="/auth", tags=["auth"]) |
||||
|
api_router.include_router( |
||||
|
export_router, |
||||
|
prefix="/exports", |
||||
|
tags=["exports"], |
||||
|
dependencies=[Depends(require_current_user)], |
||||
|
) |
||||
|
api_router.include_router( |
||||
|
backend_router, |
||||
|
prefix="/backends", |
||||
|
tags=["backends"], |
||||
|
dependencies=[Depends(require_current_user)], |
||||
|
) |
||||
|
api_router.include_router( |
||||
|
job_router, |
||||
|
prefix="/jobs", |
||||
|
tags=["jobs"], |
||||
|
dependencies=[Depends(require_current_user)], |
||||
|
) |
||||
|
api_router.include_router( |
||||
|
policy_router, |
||||
|
prefix="/policies", |
||||
|
tags=["policies"], |
||||
|
dependencies=[Depends(require_current_user)], |
||||
|
) |
||||
|
api_router.include_router( |
||||
|
directory_router, |
||||
|
prefix="/directories", |
||||
|
tags=["directories"], |
||||
|
dependencies=[Depends(require_current_user)], |
||||
|
) |
||||
|
api_router.include_router( |
||||
|
file_router, |
||||
|
prefix="/files", |
||||
|
tags=["files"], |
||||
|
dependencies=[Depends(require_current_user)], |
||||
|
) |
||||
|
api_router.include_router( |
||||
|
upload_router, |
||||
|
prefix="/uploads", |
||||
|
tags=["uploads"], |
||||
|
dependencies=[Depends(require_current_user)], |
||||
|
) |
||||
@ -0,0 +1,2 @@ |
|||||
|
"""API route modules.""" |
||||
|
|
||||
@ -0,0 +1,46 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from fastapi import APIRouter |
||||
|
from fastapi import Depends |
||||
|
from fastapi import Header |
||||
|
|
||||
|
from app.api.dependencies import get_auth_service |
||||
|
from app.api.dependencies import require_current_user |
||||
|
from app.models.entities import User |
||||
|
from app.schemas.auth import AuthMeResponse |
||||
|
from app.schemas.auth import LoginRequest |
||||
|
from app.schemas.auth import LoginResponse |
||||
|
from app.schemas.auth import LogoutResponse |
||||
|
from app.schemas.auth import UserSummary |
||||
|
from app.services.auth_service import AuthService |
||||
|
|
||||
|
router = APIRouter() |
||||
|
|
||||
|
|
||||
|
@router.post("/login", response_model=LoginResponse) |
||||
|
async def login( |
||||
|
payload: LoginRequest, |
||||
|
service: AuthService = Depends(get_auth_service), |
||||
|
) -> LoginResponse: |
||||
|
result = await service.login(username=payload.username, password=payload.password) |
||||
|
return LoginResponse( |
||||
|
access_token=result.access_token, |
||||
|
expires_at=result.expires_at, |
||||
|
user=UserSummary.model_validate(result.user, from_attributes=True), |
||||
|
) |
||||
|
|
||||
|
|
||||
|
@router.get("/me", response_model=AuthMeResponse) |
||||
|
async def me( |
||||
|
current_user: User = Depends(require_current_user), |
||||
|
) -> AuthMeResponse: |
||||
|
return AuthMeResponse(user=UserSummary.model_validate(current_user, from_attributes=True)) |
||||
|
|
||||
|
|
||||
|
@router.post("/logout", response_model=LogoutResponse) |
||||
|
async def logout( |
||||
|
authorization: str | None = Header(default=None, alias="Authorization"), |
||||
|
service: AuthService = Depends(get_auth_service), |
||||
|
) -> LogoutResponse: |
||||
|
await service.logout(authorization) |
||||
|
return LogoutResponse() |
||||
@ -0,0 +1,112 @@ |
|||||
|
import json |
||||
|
|
||||
|
from fastapi import APIRouter, Depends |
||||
|
from fastapi import Response |
||||
|
|
||||
|
from app.api.dependencies import get_backend_service |
||||
|
from app.core.secret_codec import list_secret_fields |
||||
|
from app.models.entities import Backend |
||||
|
from app.schemas.backend import ( |
||||
|
BackendCheckResponse, |
||||
|
BackendListResponse, |
||||
|
BackendMutationResponse, |
||||
|
BackendSummary, |
||||
|
CreateBackendRequest, |
||||
|
CreateBackendResponse, |
||||
|
UpdateBackendRequest, |
||||
|
) |
||||
|
from app.services.backend_service import BackendService |
||||
|
|
||||
|
router = APIRouter() |
||||
|
|
||||
|
|
||||
|
def summarize_backend(backend: Backend) -> BackendSummary: |
||||
|
summary = BackendSummary.model_validate(backend, from_attributes=True) |
||||
|
public_config = json.loads(backend.public_config_json) |
||||
|
summary.config = public_config |
||||
|
summary.secret_fields = list_secret_fields(backend.secret_config_json) |
||||
|
return summary |
||||
|
|
||||
|
|
||||
|
@router.get("", response_model=BackendListResponse) |
||||
|
async def list_backends( |
||||
|
service: BackendService = Depends(get_backend_service), |
||||
|
) -> BackendListResponse: |
||||
|
items = await service.list_backends() |
||||
|
return BackendListResponse( |
||||
|
items=[summarize_backend(item) for item in items] |
||||
|
) |
||||
|
|
||||
|
|
||||
|
@router.post("", response_model=CreateBackendResponse) |
||||
|
async def create_backend( |
||||
|
payload: CreateBackendRequest, |
||||
|
service: BackendService = Depends(get_backend_service), |
||||
|
) -> CreateBackendResponse: |
||||
|
backend = await service.create_backend( |
||||
|
name=payload.name, |
||||
|
backend_type=payload.type, |
||||
|
stability_class=payload.stability_class, |
||||
|
read_priority=payload.read_priority, |
||||
|
write_priority=payload.write_priority, |
||||
|
config=payload.config, |
||||
|
) |
||||
|
return CreateBackendResponse(backend=summarize_backend(backend)) |
||||
|
|
||||
|
|
||||
|
@router.post("/{backend_id}/check", response_model=BackendCheckResponse) |
||||
|
async def check_backend( |
||||
|
backend_id: str, |
||||
|
service: BackendService = Depends(get_backend_service), |
||||
|
) -> BackendCheckResponse: |
||||
|
backend, status, detail = await service.check_backend(backend_id) |
||||
|
return BackendCheckResponse( |
||||
|
backend_id=backend.id, |
||||
|
status=status, |
||||
|
checked_at=backend.last_health_checked_at, |
||||
|
detail=detail, |
||||
|
) |
||||
|
|
||||
|
|
||||
|
@router.patch("/{backend_id}", response_model=BackendMutationResponse) |
||||
|
async def update_backend( |
||||
|
backend_id: str, |
||||
|
payload: UpdateBackendRequest, |
||||
|
service: BackendService = Depends(get_backend_service), |
||||
|
) -> BackendMutationResponse: |
||||
|
backend = await service.update_backend( |
||||
|
backend_id, |
||||
|
name=payload.name, |
||||
|
stability_class=payload.stability_class, |
||||
|
read_priority=payload.read_priority, |
||||
|
write_priority=payload.write_priority, |
||||
|
config=payload.config, |
||||
|
) |
||||
|
return BackendMutationResponse(backend=summarize_backend(backend)) |
||||
|
|
||||
|
|
||||
|
@router.post("/{backend_id}/disable", response_model=BackendMutationResponse) |
||||
|
async def disable_backend( |
||||
|
backend_id: str, |
||||
|
service: BackendService = Depends(get_backend_service), |
||||
|
) -> BackendMutationResponse: |
||||
|
backend = await service.disable_backend(backend_id) |
||||
|
return BackendMutationResponse(backend=summarize_backend(backend)) |
||||
|
|
||||
|
|
||||
|
@router.post("/{backend_id}/enable", response_model=BackendMutationResponse) |
||||
|
async def enable_backend( |
||||
|
backend_id: str, |
||||
|
service: BackendService = Depends(get_backend_service), |
||||
|
) -> BackendMutationResponse: |
||||
|
backend = await service.enable_backend(backend_id) |
||||
|
return BackendMutationResponse(backend=summarize_backend(backend)) |
||||
|
|
||||
|
|
||||
|
@router.delete("/{backend_id}", status_code=204) |
||||
|
async def delete_backend( |
||||
|
backend_id: str, |
||||
|
service: BackendService = Depends(get_backend_service), |
||||
|
) -> Response: |
||||
|
await service.delete_backend(backend_id) |
||||
|
return Response(status_code=204) |
||||
@ -0,0 +1,72 @@ |
|||||
|
from fastapi import APIRouter, Depends, Query, status |
||||
|
|
||||
|
from app.api.dependencies import get_directory_service |
||||
|
from app.schemas.directory import ( |
||||
|
CreateDirectoryRequest, |
||||
|
CreateDirectoryResponse, |
||||
|
DirectoryChildrenResponse, |
||||
|
DirectoryItem, |
||||
|
DirectorySummary, |
||||
|
MoveDirectoryRequest, |
||||
|
RenameDirectoryRequest, |
||||
|
) |
||||
|
from app.services.directory_service import DirectoryService |
||||
|
|
||||
|
router = APIRouter() |
||||
|
|
||||
|
|
||||
|
@router.get("/{directory_id}/children", response_model=DirectoryChildrenResponse) |
||||
|
async def list_directory_children( |
||||
|
directory_id: str, |
||||
|
_cursor: str | None = Query(default=None, alias="cursor"), |
||||
|
_limit: int = Query(default=100, ge=1, le=500, alias="limit"), |
||||
|
service: DirectoryService = Depends(get_directory_service), |
||||
|
) -> DirectoryChildrenResponse: |
||||
|
result = await service.list_children(directory_id) |
||||
|
return DirectoryChildrenResponse( |
||||
|
directory=DirectorySummary.model_validate(result.directory, from_attributes=True), |
||||
|
items=[ |
||||
|
DirectoryItem( |
||||
|
kind=item.kind, |
||||
|
id=item.item.id, |
||||
|
name=item.item.name, |
||||
|
mime_type=getattr(item.item, "mime_type", None), |
||||
|
size_bytes=getattr(item.item, "size_bytes", None), |
||||
|
created_at=item.item.created_at, |
||||
|
updated_at=item.item.updated_at, |
||||
|
) |
||||
|
for item in result.children |
||||
|
], |
||||
|
next_cursor=None, |
||||
|
) |
||||
|
|
||||
|
|
||||
|
@router.post("", response_model=CreateDirectoryResponse, status_code=status.HTTP_201_CREATED) |
||||
|
async def create_directory( |
||||
|
payload: CreateDirectoryRequest, |
||||
|
service: DirectoryService = Depends(get_directory_service), |
||||
|
) -> CreateDirectoryResponse: |
||||
|
directory = await service.create_directory(parent_id=payload.parent_id, name=payload.name) |
||||
|
return CreateDirectoryResponse( |
||||
|
directory=DirectorySummary.model_validate(directory, from_attributes=True) |
||||
|
) |
||||
|
|
||||
|
|
||||
|
@router.post("/{directory_id}/rename", response_model=CreateDirectoryResponse) |
||||
|
async def rename_directory( |
||||
|
directory_id: str, |
||||
|
payload: RenameDirectoryRequest, |
||||
|
service: DirectoryService = Depends(get_directory_service), |
||||
|
) -> CreateDirectoryResponse: |
||||
|
directory = await service.rename_directory(directory_id=directory_id, name=payload.name) |
||||
|
return CreateDirectoryResponse(directory=DirectorySummary.model_validate(directory, from_attributes=True)) |
||||
|
|
||||
|
|
||||
|
@router.post("/{directory_id}/move", response_model=CreateDirectoryResponse) |
||||
|
async def move_directory( |
||||
|
directory_id: str, |
||||
|
payload: MoveDirectoryRequest, |
||||
|
service: DirectoryService = Depends(get_directory_service), |
||||
|
) -> CreateDirectoryResponse: |
||||
|
directory = await service.move_directory(directory_id=directory_id, target_parent_id=payload.target_parent_id) |
||||
|
return CreateDirectoryResponse(directory=DirectorySummary.model_validate(directory, from_attributes=True)) |
||||
@ -0,0 +1,76 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from fastapi import APIRouter, Depends |
||||
|
|
||||
|
from app.api.dependencies import get_export_service |
||||
|
from app.core.exceptions import ValidationError |
||||
|
from app.schemas.export import ( |
||||
|
MetadataExportResponse, |
||||
|
MetadataImportRequest, |
||||
|
MetadataImportResponse, |
||||
|
MetadataRestorePlanResponse, |
||||
|
MetadataValidationResponse, |
||||
|
) |
||||
|
from app.services.export_service import ExportService |
||||
|
from app.services.job_service import JobService |
||||
|
|
||||
|
router = APIRouter() |
||||
|
|
||||
|
|
||||
|
@router.get("/metadata", response_model=MetadataExportResponse) |
||||
|
async def export_metadata(service: ExportService = Depends(get_export_service)) -> MetadataExportResponse: |
||||
|
data = await service.export_metadata() |
||||
|
exported_at = data.pop("exported_at") |
||||
|
return MetadataExportResponse(exported_at=exported_at, data=data) |
||||
|
|
||||
|
|
||||
|
@router.get("/metadata/integrity", response_model=MetadataValidationResponse) |
||||
|
async def metadata_integrity(service: ExportService = Depends(get_export_service)) -> MetadataValidationResponse: |
||||
|
result = await service.verify_runtime_integrity() |
||||
|
return MetadataValidationResponse(ok=result["ok"], issues=result["issues"]) |
||||
|
|
||||
|
|
||||
|
@router.post("/metadata/validate", response_model=MetadataValidationResponse) |
||||
|
async def validate_metadata( |
||||
|
payload: MetadataImportRequest, |
||||
|
service: ExportService = Depends(get_export_service), |
||||
|
) -> MetadataValidationResponse: |
||||
|
result = await service.validate_metadata_snapshot(payload.data) |
||||
|
return MetadataValidationResponse(ok=result["ok"], issues=result["issues"]) |
||||
|
|
||||
|
|
||||
|
@router.post("/metadata/restore-plan", response_model=MetadataRestorePlanResponse) |
||||
|
async def restore_plan( |
||||
|
payload: MetadataImportRequest, |
||||
|
service: ExportService = Depends(get_export_service), |
||||
|
) -> MetadataRestorePlanResponse: |
||||
|
result = await service.build_restore_plan(payload.data) |
||||
|
return MetadataRestorePlanResponse( |
||||
|
ok=result["ok"], |
||||
|
issues=result["issues"], |
||||
|
validation_token=result["validation_token"], |
||||
|
summary=result["summary"], |
||||
|
) |
||||
|
|
||||
|
|
||||
|
@router.post("/metadata/import", response_model=MetadataImportResponse) |
||||
|
async def import_metadata( |
||||
|
payload: MetadataImportRequest, |
||||
|
service: ExportService = Depends(get_export_service), |
||||
|
) -> MetadataImportResponse: |
||||
|
if not payload.confirm_replace: |
||||
|
raise ValidationError("Metadata import requires confirm_replace=true.") |
||||
|
restore_plan_result = await service.build_restore_plan(payload.data) |
||||
|
if not restore_plan_result["ok"]: |
||||
|
raise ValidationError("Metadata snapshot is invalid and cannot be imported.") |
||||
|
if payload.validation_token != restore_plan_result["validation_token"]: |
||||
|
raise ValidationError("Metadata import requires a current validation_token from restore-plan.") |
||||
|
imported_tables = await service.import_metadata(payload.data) |
||||
|
job_service = JobService(service.session) |
||||
|
reconcile_job = await job_service.enqueue_full_reconcile() |
||||
|
health_job_count = 0 |
||||
|
for backend in await job_service.backend_repository.list_enabled(): |
||||
|
await job_service.enqueue_backend_health_check(backend.id) |
||||
|
health_job_count += 1 |
||||
|
scheduled_jobs = [reconcile_job.kind] + ["check_backend_health"] * health_job_count |
||||
|
return MetadataImportResponse(imported_tables=imported_tables, scheduled_jobs=scheduled_jobs) |
||||
@ -0,0 +1,189 @@ |
|||||
|
from typing import AsyncIterator |
||||
|
|
||||
|
import anyio |
||||
|
from fastapi import APIRouter, Depends, Header, Response, status |
||||
|
from fastapi.responses import FileResponse, StreamingResponse |
||||
|
|
||||
|
from app.api.dependencies import get_file_service, get_repair_service |
||||
|
from app.schemas.file import ( |
||||
|
DeletedFileSummary, |
||||
|
FileDetailResponse, |
||||
|
FilePlacementBackendSummary, |
||||
|
FileMutationResponse, |
||||
|
FileReplicaSummary, |
||||
|
FileSummary, |
||||
|
MoveFileRequest, |
||||
|
PreviewArtifactSummary, |
||||
|
RecycleBinResponse, |
||||
|
RenameFileRequest, |
||||
|
) |
||||
|
from app.services.file_service import FileService |
||||
|
from app.schemas.repair import RepairFileResponse |
||||
|
from app.services.repair_service import RepairService |
||||
|
|
||||
|
router = APIRouter() |
||||
|
|
||||
|
|
||||
|
@router.get("/recycle-bin", response_model=RecycleBinResponse) |
||||
|
async def list_recycle_bin( |
||||
|
service: FileService = Depends(get_file_service), |
||||
|
) -> RecycleBinResponse: |
||||
|
items = await service.list_deleted_files() |
||||
|
return RecycleBinResponse( |
||||
|
items=[DeletedFileSummary.model_validate(item, from_attributes=True) for item in items] |
||||
|
) |
||||
|
|
||||
|
|
||||
|
@router.delete("/{file_id}", response_model=FileMutationResponse) |
||||
|
async def delete_file( |
||||
|
file_id: str, |
||||
|
service: FileService = Depends(get_file_service), |
||||
|
) -> FileMutationResponse: |
||||
|
file_entry = await service.delete_file(file_id) |
||||
|
return FileMutationResponse(file=DeletedFileSummary.model_validate(file_entry, from_attributes=True)) |
||||
|
|
||||
|
|
||||
|
@router.post("/{file_id}/rename", response_model=FileMutationResponse) |
||||
|
async def rename_file( |
||||
|
file_id: str, |
||||
|
payload: RenameFileRequest, |
||||
|
service: FileService = Depends(get_file_service), |
||||
|
) -> FileMutationResponse: |
||||
|
file_entry = await service.rename_file(file_id=file_id, name=payload.name) |
||||
|
return FileMutationResponse(file=FileSummary.model_validate(file_entry, from_attributes=True)) |
||||
|
|
||||
|
|
||||
|
@router.post("/{file_id}/move", response_model=FileMutationResponse) |
||||
|
async def move_file( |
||||
|
file_id: str, |
||||
|
payload: MoveFileRequest, |
||||
|
service: FileService = Depends(get_file_service), |
||||
|
) -> FileMutationResponse: |
||||
|
file_entry = await service.move_file(file_id=file_id, target_parent_id=payload.target_parent_id) |
||||
|
return FileMutationResponse(file=FileSummary.model_validate(file_entry, from_attributes=True)) |
||||
|
|
||||
|
|
||||
|
@router.post("/{file_id}/restore", response_model=FileMutationResponse) |
||||
|
async def restore_file( |
||||
|
file_id: str, |
||||
|
service: FileService = Depends(get_file_service), |
||||
|
) -> FileMutationResponse: |
||||
|
file_entry = await service.restore_file(file_id) |
||||
|
return FileMutationResponse(file=FileSummary.model_validate(file_entry, from_attributes=True)) |
||||
|
|
||||
|
|
||||
|
@router.post("/{file_id}/reconcile", response_model=RepairFileResponse) |
||||
|
async def reconcile_file( |
||||
|
file_id: str, |
||||
|
service: RepairService = Depends(get_repair_service), |
||||
|
) -> RepairFileResponse: |
||||
|
enqueued = await service.enqueue_repairs_for_file(file_id) |
||||
|
return RepairFileResponse(file_id=file_id, enqueued_jobs=enqueued) |
||||
|
|
||||
|
|
||||
|
@router.get("/{file_id}", response_model=FileDetailResponse) |
||||
|
async def get_file_detail( |
||||
|
file_id: str, |
||||
|
service: FileService = Depends(get_file_service), |
||||
|
) -> FileDetailResponse: |
||||
|
file_entry, artifacts, replicas, decision = await service.get_file_detail(file_id) |
||||
|
replica_items = [ |
||||
|
FileReplicaSummary( |
||||
|
id=replica.id, |
||||
|
blob_id=replica.blob_id, |
||||
|
backend_id=backend.id, |
||||
|
backend_name=backend.name, |
||||
|
backend_type=backend.type, |
||||
|
backend_stability_class=backend.stability_class, |
||||
|
backend_enabled=backend.is_enabled, |
||||
|
storage_key=replica.storage_key, |
||||
|
status=replica.status, |
||||
|
size_bytes=replica.size_bytes, |
||||
|
checksum=replica.checksum, |
||||
|
etag=replica.etag, |
||||
|
last_verified_at=replica.last_verified_at, |
||||
|
created_at=replica.created_at, |
||||
|
updated_at=replica.updated_at, |
||||
|
) |
||||
|
for replica, backend in replicas |
||||
|
] |
||||
|
ready_replica_backend_ids = sorted({item.backend_id for item in replica_items if item.status == "ready"}) |
||||
|
desired_backend_ids = [backend.id for backend in (decision.backends if decision is not None else [])] |
||||
|
return FileDetailResponse( |
||||
|
file=FileSummary.model_validate(file_entry, from_attributes=True), |
||||
|
preview_artifacts=[ |
||||
|
PreviewArtifactSummary.model_validate(artifact, from_attributes=True) for artifact in artifacts |
||||
|
], |
||||
|
replicas=replica_items, |
||||
|
placement_file_class=decision.file_class if decision is not None else None, |
||||
|
desired_backends=[ |
||||
|
FilePlacementBackendSummary.model_validate(backend, from_attributes=True) |
||||
|
for backend in (decision.backends if decision is not None else []) |
||||
|
], |
||||
|
ready_replica_backend_ids=ready_replica_backend_ids, |
||||
|
missing_backend_ids=[backend_id for backend_id in desired_backend_ids if backend_id not in ready_replica_backend_ids], |
||||
|
) |
||||
|
|
||||
|
|
||||
|
@router.get("/{file_id}/download") |
||||
|
async def download_file( |
||||
|
file_id: str, |
||||
|
service: FileService = Depends(get_file_service), |
||||
|
) -> FileResponse: |
||||
|
file_entry, file_path = await service.resolve_download(file_id) |
||||
|
return FileResponse( |
||||
|
path=file_path, |
||||
|
filename=file_entry.name, |
||||
|
media_type=file_entry.mime_type or "application/octet-stream", |
||||
|
) |
||||
|
|
||||
|
|
||||
|
@router.get("/{file_id}/preview") |
||||
|
async def preview_file( |
||||
|
file_id: str, |
||||
|
service: FileService = Depends(get_file_service), |
||||
|
) -> FileResponse: |
||||
|
file_entry, file_path = await service.resolve_preview(file_id) |
||||
|
return FileResponse( |
||||
|
path=file_path, |
||||
|
filename=file_entry.name, |
||||
|
media_type=file_entry.mime_type or "application/octet-stream", |
||||
|
content_disposition_type="inline", |
||||
|
) |
||||
|
|
||||
|
|
||||
|
@router.get("/{file_id}/stream") |
||||
|
async def stream_file( |
||||
|
file_id: str, |
||||
|
range_header: str | None = Header(default=None, alias="Range"), |
||||
|
service: FileService = Depends(get_file_service), |
||||
|
) -> Response: |
||||
|
stream_result = await service.resolve_stream(file_id, range_header) |
||||
|
chunk_size = 64 * 1024 |
||||
|
|
||||
|
async def body_iter() -> AsyncIterator[bytes]: |
||||
|
remaining = stream_result.end - stream_result.start + 1 |
||||
|
async with await anyio.open_file(stream_result.file_path, "rb") as file_handle: |
||||
|
await file_handle.seek(stream_result.start) |
||||
|
while remaining > 0: |
||||
|
chunk = await file_handle.read(min(chunk_size, remaining)) |
||||
|
if not chunk: |
||||
|
break |
||||
|
remaining -= len(chunk) |
||||
|
yield chunk |
||||
|
|
||||
|
headers = { |
||||
|
"Accept-Ranges": "bytes", |
||||
|
"Content-Length": str(stream_result.end - stream_result.start + 1), |
||||
|
} |
||||
|
status_code = status.HTTP_200_OK |
||||
|
if range_header: |
||||
|
headers["Content-Range"] = f"bytes {stream_result.start}-{stream_result.end}/{stream_result.file_size}" |
||||
|
status_code = status.HTTP_206_PARTIAL_CONTENT |
||||
|
|
||||
|
return StreamingResponse( |
||||
|
body_iter(), |
||||
|
status_code=status_code, |
||||
|
headers=headers, |
||||
|
media_type=stream_result.file_entry.mime_type or "application/octet-stream", |
||||
|
) |
||||
@ -0,0 +1,51 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from fastapi import APIRouter, Depends |
||||
|
|
||||
|
from app.api.dependencies import get_job_service, get_repair_service |
||||
|
from app.schemas.job import JobDetailResponse, JobListResponse, JobMutationResponse, JobRunPendingResponse, JobSummary |
||||
|
from app.services.job_service import JobService |
||||
|
from app.services.repair_service import RepairService |
||||
|
from app.schemas.repair import EnqueueHealthChecksResponse |
||||
|
|
||||
|
router = APIRouter() |
||||
|
|
||||
|
|
||||
|
@router.get("", response_model=JobListResponse) |
||||
|
async def list_jobs(service: JobService = Depends(get_job_service)) -> JobListResponse: |
||||
|
items = await service.list_jobs() |
||||
|
return JobListResponse(items=[JobSummary.model_validate(item, from_attributes=True) for item in items]) |
||||
|
|
||||
|
|
||||
|
@router.get("/{job_id}", response_model=JobDetailResponse) |
||||
|
async def get_job(job_id: str, service: JobService = Depends(get_job_service)) -> JobDetailResponse: |
||||
|
job = await service.get_job(job_id) |
||||
|
return JobDetailResponse(job=JobSummary.model_validate(job, from_attributes=True)) |
||||
|
|
||||
|
|
||||
|
@router.post("/{job_id}/retry", response_model=JobMutationResponse) |
||||
|
async def retry_job(job_id: str, service: JobService = Depends(get_job_service)) -> JobMutationResponse: |
||||
|
job = await service.retry_job(job_id) |
||||
|
return JobMutationResponse(job=JobSummary.model_validate(job, from_attributes=True)) |
||||
|
|
||||
|
|
||||
|
@router.post("/run-pending", response_model=JobRunPendingResponse) |
||||
|
async def run_pending_jobs(service: JobService = Depends(get_job_service)) -> JobRunPendingResponse: |
||||
|
result = await service.run_pending_jobs() |
||||
|
return JobRunPendingResponse(processed=result.processed, completed=result.completed, failed=result.failed) |
||||
|
|
||||
|
|
||||
|
@router.post("/enqueue-health-checks", response_model=EnqueueHealthChecksResponse) |
||||
|
async def enqueue_health_checks( |
||||
|
service: RepairService = Depends(get_repair_service), |
||||
|
) -> EnqueueHealthChecksResponse: |
||||
|
enqueued = await service.enqueue_backend_health_checks() |
||||
|
return EnqueueHealthChecksResponse(enqueued_jobs=enqueued) |
||||
|
|
||||
|
|
||||
|
@router.post("/enqueue-full-reconcile", response_model=EnqueueHealthChecksResponse) |
||||
|
async def enqueue_full_reconcile( |
||||
|
service: RepairService = Depends(get_repair_service), |
||||
|
) -> EnqueueHealthChecksResponse: |
||||
|
enqueued = await service.enqueue_full_reconcile() |
||||
|
return EnqueueHealthChecksResponse(enqueued_jobs=enqueued) |
||||
@ -0,0 +1,81 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from fastapi import APIRouter, Depends |
||||
|
from fastapi import Query |
||||
|
|
||||
|
from app.api.dependencies import get_policy_service, get_repair_service |
||||
|
from app.schemas.policy import ( |
||||
|
PlacementDecisionBackendSummary, |
||||
|
PlacementDecisionResponse, |
||||
|
PlacementPolicyListResponse, |
||||
|
PlacementPolicyMutationResponse, |
||||
|
PlacementPolicySummary, |
||||
|
UpsertPlacementPolicyRequest, |
||||
|
) |
||||
|
from app.services.policy_service import PolicyService |
||||
|
from app.services.repair_service import RepairService |
||||
|
|
||||
|
router = APIRouter() |
||||
|
|
||||
|
|
||||
|
def _policy_to_summary(policy) -> PlacementPolicySummary: |
||||
|
return PlacementPolicySummary( |
||||
|
id=policy.id, |
||||
|
file_class=policy.file_class, |
||||
|
require_local=policy.require_local, |
||||
|
stable_replica_count=policy.stable_replica_count, |
||||
|
opportunistic_replica_count=policy.opportunistic_replica_count, |
||||
|
preferred_backend_ids=PolicyService._decode_backend_ids(policy.preferred_backend_ids_json), |
||||
|
excluded_backend_ids=PolicyService._decode_backend_ids(policy.excluded_backend_ids_json), |
||||
|
max_non_local_size_bytes=policy.max_non_local_size_bytes, |
||||
|
created_at=policy.created_at, |
||||
|
updated_at=policy.updated_at, |
||||
|
) |
||||
|
|
||||
|
|
||||
|
@router.get("/placement", response_model=PlacementPolicyListResponse) |
||||
|
async def list_placement_policies( |
||||
|
service: PolicyService = Depends(get_policy_service), |
||||
|
) -> PlacementPolicyListResponse: |
||||
|
policies = await service.list_policies() |
||||
|
return PlacementPolicyListResponse(items=[_policy_to_summary(policy) for policy in policies]) |
||||
|
|
||||
|
|
||||
|
@router.get("/placement/preview", response_model=PlacementDecisionResponse) |
||||
|
async def preview_placement_decision( |
||||
|
mime_type: str | None = Query(default=None), |
||||
|
size_bytes: int | None = Query(default=None, ge=0), |
||||
|
service: PolicyService = Depends(get_policy_service), |
||||
|
) -> PlacementDecisionResponse: |
||||
|
decision = await service.get_placement_decision(mime_type, size_bytes=size_bytes) |
||||
|
return PlacementDecisionResponse( |
||||
|
file_class=decision.file_class, |
||||
|
mime_type=mime_type, |
||||
|
size_bytes=size_bytes, |
||||
|
non_local_allowed=decision.non_local_allowed, |
||||
|
policy=_policy_to_summary(decision.policy), |
||||
|
selected_backends=[ |
||||
|
PlacementDecisionBackendSummary.model_validate(backend, from_attributes=True) |
||||
|
for backend in decision.backends |
||||
|
], |
||||
|
) |
||||
|
|
||||
|
|
||||
|
@router.put("/placement/{file_class}", response_model=PlacementPolicyMutationResponse) |
||||
|
async def upsert_placement_policy( |
||||
|
file_class: str, |
||||
|
payload: UpsertPlacementPolicyRequest, |
||||
|
service: PolicyService = Depends(get_policy_service), |
||||
|
repair_service: RepairService = Depends(get_repair_service), |
||||
|
) -> PlacementPolicyMutationResponse: |
||||
|
policy = await service.upsert_policy( |
||||
|
file_class=file_class, |
||||
|
require_local=payload.require_local, |
||||
|
stable_replica_count=payload.stable_replica_count, |
||||
|
opportunistic_replica_count=payload.opportunistic_replica_count, |
||||
|
preferred_backend_ids=payload.preferred_backend_ids, |
||||
|
excluded_backend_ids=payload.excluded_backend_ids, |
||||
|
max_non_local_size_bytes=payload.max_non_local_size_bytes, |
||||
|
) |
||||
|
enqueued_jobs = await repair_service.enqueue_full_reconcile() |
||||
|
return PlacementPolicyMutationResponse(policy=_policy_to_summary(policy), reconcile_enqueued_jobs=enqueued_jobs) |
||||
@ -0,0 +1,21 @@ |
|||||
|
from fastapi import APIRouter |
||||
|
|
||||
|
from app.db.health import ping_database |
||||
|
from app.db.session import create_engine |
||||
|
from app.schemas.system import HealthResponse, ReadyResponse |
||||
|
|
||||
|
router = APIRouter() |
||||
|
|
||||
|
|
||||
|
@router.get("/health", response_model=HealthResponse) |
||||
|
async def health_check() -> HealthResponse: |
||||
|
return HealthResponse(status="ok") |
||||
|
|
||||
|
|
||||
|
@router.get("/ready", response_model=ReadyResponse) |
||||
|
async def readiness_check() -> ReadyResponse: |
||||
|
database_ok = await ping_database(create_engine()) |
||||
|
return ReadyResponse( |
||||
|
status="ready" if database_ok else "degraded", |
||||
|
database="ok" if database_ok else "error", |
||||
|
) |
||||
@ -0,0 +1,77 @@ |
|||||
|
from fastapi import APIRouter, Depends, Header, Request, status |
||||
|
from fastapi.responses import Response |
||||
|
|
||||
|
from app.api.dependencies import get_upload_service |
||||
|
from app.schemas.common import ErrorResponse |
||||
|
from app.services.upload_service import CreateUploadInput, UploadService |
||||
|
|
||||
|
router = APIRouter() |
||||
|
|
||||
|
TUS_VERSION = "1.0.0" |
||||
|
|
||||
|
|
||||
|
@router.post("", status_code=status.HTTP_201_CREATED, responses={409: {"model": ErrorResponse}}) |
||||
|
async def create_tus_upload( |
||||
|
upload_length: int = Header(alias="Upload-Length"), |
||||
|
upload_metadata: str | None = Header(default=None, alias="Upload-Metadata"), |
||||
|
tus_resumable: str = Header(alias="Tus-Resumable"), |
||||
|
service: UploadService = Depends(get_upload_service), |
||||
|
) -> Response: |
||||
|
if tus_resumable != TUS_VERSION: |
||||
|
return Response(status_code=status.HTTP_412_PRECONDITION_FAILED) |
||||
|
|
||||
|
upload_session = await service.create_upload( |
||||
|
CreateUploadInput(upload_length=upload_length, upload_metadata=upload_metadata) |
||||
|
) |
||||
|
return Response( |
||||
|
status_code=status.HTTP_201_CREATED, |
||||
|
headers={ |
||||
|
"Location": upload_session.tus_upload_url or f"/files/{upload_session.id}", |
||||
|
"Tus-Resumable": TUS_VERSION, |
||||
|
}, |
||||
|
) |
||||
|
|
||||
|
|
||||
|
@router.head("/{upload_id}") |
||||
|
async def head_tus_upload( |
||||
|
upload_id: str, |
||||
|
tus_resumable: str = Header(alias="Tus-Resumable"), |
||||
|
service: UploadService = Depends(get_upload_service), |
||||
|
) -> Response: |
||||
|
if tus_resumable != TUS_VERSION: |
||||
|
return Response(status_code=status.HTTP_412_PRECONDITION_FAILED) |
||||
|
|
||||
|
upload_session = await service.get_upload(upload_id) |
||||
|
return Response( |
||||
|
status_code=status.HTTP_200_OK, |
||||
|
headers={ |
||||
|
"Tus-Resumable": TUS_VERSION, |
||||
|
"Upload-Offset": str(upload_session.received_size_bytes), |
||||
|
"Upload-Length": str(upload_session.total_size_bytes), |
||||
|
}, |
||||
|
) |
||||
|
|
||||
|
|
||||
|
@router.patch("/{upload_id}", status_code=status.HTTP_204_NO_CONTENT, responses={409: {"model": ErrorResponse}}) |
||||
|
async def patch_tus_upload( |
||||
|
upload_id: str, |
||||
|
request: Request, |
||||
|
upload_offset: int = Header(alias="Upload-Offset"), |
||||
|
tus_resumable: str = Header(alias="Tus-Resumable"), |
||||
|
content_type: str = Header(alias="Content-Type"), |
||||
|
service: UploadService = Depends(get_upload_service), |
||||
|
) -> Response: |
||||
|
if tus_resumable != TUS_VERSION: |
||||
|
return Response(status_code=status.HTTP_412_PRECONDITION_FAILED) |
||||
|
if content_type != "application/offset+octet-stream": |
||||
|
return Response(status_code=status.HTTP_415_UNSUPPORTED_MEDIA_TYPE) |
||||
|
|
||||
|
chunk = await request.body() |
||||
|
upload_session = await service.append_upload_bytes(upload_id, upload_offset, chunk) |
||||
|
return Response( |
||||
|
status_code=status.HTTP_204_NO_CONTENT, |
||||
|
headers={ |
||||
|
"Tus-Resumable": TUS_VERSION, |
||||
|
"Upload-Offset": str(upload_session.received_size_bytes), |
||||
|
}, |
||||
|
) |
||||
@ -0,0 +1,24 @@ |
|||||
|
from fastapi import APIRouter, Depends |
||||
|
|
||||
|
from app.api.dependencies import get_upload_service |
||||
|
from app.schemas.file import FileSummary |
||||
|
from app.schemas.upload import FinalizeUploadRequest, FinalizeUploadResponse, UploadSummary |
||||
|
from app.services.upload_service import FinalizeUploadInput, UploadService |
||||
|
|
||||
|
router = APIRouter() |
||||
|
|
||||
|
|
||||
|
@router.post("/{upload_id}/finalize", response_model=FinalizeUploadResponse) |
||||
|
async def finalize_upload( |
||||
|
upload_id: str, |
||||
|
payload: FinalizeUploadRequest, |
||||
|
service: UploadService = Depends(get_upload_service), |
||||
|
) -> FinalizeUploadResponse: |
||||
|
upload_session, file_entry = await service.finalize_upload( |
||||
|
upload_id, |
||||
|
FinalizeUploadInput(directory_id=payload.directory_id, filename=payload.filename), |
||||
|
) |
||||
|
return FinalizeUploadResponse( |
||||
|
upload=UploadSummary.model_validate(upload_session, from_attributes=True), |
||||
|
file=FileSummary.model_validate(file_entry, from_attributes=True), |
||||
|
) |
||||
@ -0,0 +1,2 @@ |
|||||
|
"""Core configuration and bootstrap helpers.""" |
||||
|
|
||||
@ -0,0 +1,130 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from typing import Any, Literal |
||||
|
|
||||
|
from pydantic import BaseModel, Field, ValidationError as PydanticValidationError, field_validator |
||||
|
|
||||
|
from app.core.exceptions import ValidationError |
||||
|
|
||||
|
BackendType = Literal["local_directory", "s3", "webdav"] |
||||
|
|
||||
|
|
||||
|
class BackendConfigError(ValidationError): |
||||
|
code = "backend_config_invalid" |
||||
|
message = "Backend configuration is invalid." |
||||
|
|
||||
|
|
||||
|
class LocalDirectoryConfig(BaseModel): |
||||
|
base_path: str = Field(min_length=1) |
||||
|
|
||||
|
@field_validator("base_path") |
||||
|
@classmethod |
||||
|
def clean_base_path(cls, value: str) -> str: |
||||
|
cleaned = value.strip() |
||||
|
if not cleaned: |
||||
|
raise ValueError("base_path must not be empty") |
||||
|
return cleaned |
||||
|
|
||||
|
|
||||
|
class S3CompatibleConfig(BaseModel): |
||||
|
bucket: str = Field(min_length=1) |
||||
|
region: str | None = None |
||||
|
endpoint_url: str | None = None |
||||
|
access_key_id: str | None = None |
||||
|
secret_access_key: str | None = None |
||||
|
prefix: str = "" |
||||
|
force_path_style: bool = False |
||||
|
verify_ssl: bool = True |
||||
|
storage_class: str | None = None |
||||
|
server_side_encryption: str | None = None |
||||
|
|
||||
|
@field_validator("bucket") |
||||
|
@classmethod |
||||
|
def clean_bucket(cls, value: str) -> str: |
||||
|
cleaned = value.strip() |
||||
|
if not cleaned: |
||||
|
raise ValueError("bucket must not be empty") |
||||
|
return cleaned |
||||
|
|
||||
|
@field_validator("prefix") |
||||
|
@classmethod |
||||
|
def clean_prefix(cls, value: str) -> str: |
||||
|
return value.strip().strip("/") |
||||
|
|
||||
|
|
||||
|
class WebDAVConfig(BaseModel): |
||||
|
endpoint_url: str = Field(min_length=1) |
||||
|
username: str = Field(min_length=1) |
||||
|
password: str | None = None |
||||
|
root_path: str = Field(default="iron", min_length=1) |
||||
|
verify_ssl: bool = True |
||||
|
|
||||
|
@field_validator("endpoint_url", "username") |
||||
|
@classmethod |
||||
|
def clean_required_text(cls, value: str) -> str: |
||||
|
cleaned = value.strip() |
||||
|
if not cleaned: |
||||
|
raise ValueError("value must not be empty") |
||||
|
return cleaned |
||||
|
|
||||
|
@field_validator("endpoint_url") |
||||
|
@classmethod |
||||
|
def clean_endpoint_url(cls, value: str) -> str: |
||||
|
return value.strip().rstrip("/") |
||||
|
|
||||
|
@field_validator("root_path") |
||||
|
@classmethod |
||||
|
def clean_root_path(cls, value: str) -> str: |
||||
|
cleaned = "/".join(part for part in value.strip().strip("/").split("/") if part) |
||||
|
if not cleaned: |
||||
|
raise ValueError("root_path must not be empty") |
||||
|
return cleaned |
||||
|
|
||||
|
|
||||
|
CONFIG_MODELS: dict[str, type[BaseModel]] = { |
||||
|
"local_directory": LocalDirectoryConfig, |
||||
|
"s3": S3CompatibleConfig, |
||||
|
"webdav": WebDAVConfig, |
||||
|
} |
||||
|
|
||||
|
SECRET_CONFIG_KEYS = { |
||||
|
"access_key_id", |
||||
|
"secret_access_key", |
||||
|
"password", |
||||
|
} |
||||
|
|
||||
|
|
||||
|
def supported_backend_types() -> set[str]: |
||||
|
return set(CONFIG_MODELS) |
||||
|
|
||||
|
|
||||
|
def validate_backend_config(backend_type: str, config: dict[str, Any]) -> dict[str, Any]: |
||||
|
model = CONFIG_MODELS.get(backend_type) |
||||
|
if model is None: |
||||
|
raise BackendConfigError(f"Unsupported backend type: {backend_type}.") |
||||
|
|
||||
|
try: |
||||
|
return model.model_validate(config).model_dump() |
||||
|
except PydanticValidationError as exc: |
||||
|
first_error = exc.errors()[0] if exc.errors() else {} |
||||
|
location = ".".join(str(part) for part in first_error.get("loc", [])) |
||||
|
message = first_error.get("msg", "Invalid backend configuration.") |
||||
|
prefix = f"{location}: " if location else "" |
||||
|
raise BackendConfigError(prefix + message) from exc |
||||
|
|
||||
|
|
||||
|
def split_backend_config(backend_type: str, config: dict[str, Any]) -> tuple[dict[str, Any], dict[str, Any]]: |
||||
|
normalized = validate_backend_config(backend_type, config) |
||||
|
public_config: dict[str, Any] = {} |
||||
|
secret_config: dict[str, Any] = {} |
||||
|
for key, value in normalized.items(): |
||||
|
if key in SECRET_CONFIG_KEYS: |
||||
|
if value: |
||||
|
secret_config[key] = value |
||||
|
else: |
||||
|
public_config[key] = value |
||||
|
return public_config, secret_config |
||||
|
|
||||
|
|
||||
|
def merge_backend_config(public_config: dict[str, Any], secret_config: dict[str, Any]) -> dict[str, Any]: |
||||
|
return {**public_config, **secret_config} |
||||
@ -0,0 +1,54 @@ |
|||||
|
import os |
||||
|
from dataclasses import dataclass |
||||
|
from functools import lru_cache |
||||
|
|
||||
|
|
||||
|
def _get_bool_env(name: str, default: bool) -> bool: |
||||
|
raw = os.getenv(name) |
||||
|
if raw is None: |
||||
|
return default |
||||
|
return raw.strip().lower() in {"1", "true", "yes", "on"} |
||||
|
|
||||
|
|
||||
|
@dataclass(frozen=True) |
||||
|
class Settings: |
||||
|
app_name: str = "Iron Gateway" |
||||
|
debug: bool = False |
||||
|
api_prefix: str = "/api" |
||||
|
database_url: str = "sqlite+aiosqlite:///./iron.db" |
||||
|
upload_temp_dir: str = "./.iron-temp/uploads" |
||||
|
local_storage_dir: str = "./.iron-storage/local" |
||||
|
cache_dir: str = "./.iron-cache" |
||||
|
bootstrap_username: str = "admin" |
||||
|
bootstrap_password: str = "changeme-iron" |
||||
|
secret_key: str = "iron-local-development-secret-key" |
||||
|
auth_session_ttl_hours: int = 24 * 7 |
||||
|
job_poll_interval_seconds: float = 2.0 |
||||
|
job_batch_size: int = 10 |
||||
|
|
||||
|
@classmethod |
||||
|
def from_env(cls) -> "Settings": |
||||
|
return cls( |
||||
|
app_name=os.getenv("IRON_APP_NAME", cls.app_name), |
||||
|
debug=_get_bool_env("IRON_DEBUG", cls.debug), |
||||
|
api_prefix=os.getenv("IRON_API_PREFIX", cls.api_prefix), |
||||
|
database_url=os.getenv("IRON_DATABASE_URL", cls.database_url), |
||||
|
upload_temp_dir=os.getenv("IRON_UPLOAD_TEMP_DIR", cls.upload_temp_dir), |
||||
|
local_storage_dir=os.getenv("IRON_LOCAL_STORAGE_DIR", cls.local_storage_dir), |
||||
|
cache_dir=os.getenv("IRON_CACHE_DIR", cls.cache_dir), |
||||
|
bootstrap_username=os.getenv("IRON_BOOTSTRAP_USERNAME", cls.bootstrap_username), |
||||
|
bootstrap_password=os.getenv("IRON_BOOTSTRAP_PASSWORD", cls.bootstrap_password), |
||||
|
secret_key=os.getenv("IRON_SECRET_KEY", cls.secret_key), |
||||
|
auth_session_ttl_hours=int( |
||||
|
os.getenv("IRON_AUTH_SESSION_TTL_HOURS", str(cls.auth_session_ttl_hours)) |
||||
|
), |
||||
|
job_poll_interval_seconds=float( |
||||
|
os.getenv("IRON_JOB_POLL_INTERVAL_SECONDS", str(cls.job_poll_interval_seconds)) |
||||
|
), |
||||
|
job_batch_size=int(os.getenv("IRON_JOB_BATCH_SIZE", str(cls.job_batch_size))), |
||||
|
) |
||||
|
|
||||
|
|
||||
|
@lru_cache(maxsize=1) |
||||
|
def get_settings() -> Settings: |
||||
|
return Settings.from_env() |
||||
@ -0,0 +1,77 @@ |
|||||
|
class AppError(Exception): |
||||
|
code = "internal_error" |
||||
|
message = "An unexpected error occurred." |
||||
|
|
||||
|
def __init__(self, message: str | None = None) -> None: |
||||
|
super().__init__(message or self.message) |
||||
|
self.detail = message or self.message |
||||
|
|
||||
|
|
||||
|
class NotFoundError(AppError): |
||||
|
code = "not_found" |
||||
|
message = "Resource not found." |
||||
|
|
||||
|
|
||||
|
class ConflictError(AppError): |
||||
|
code = "conflict" |
||||
|
message = "Resource conflict." |
||||
|
|
||||
|
|
||||
|
class ValidationError(AppError): |
||||
|
code = "validation_error" |
||||
|
message = "Validation failed." |
||||
|
|
||||
|
|
||||
|
class UnauthorizedError(AppError): |
||||
|
code = "unauthorized" |
||||
|
message = "Authentication is required." |
||||
|
|
||||
|
|
||||
|
class AuthenticationFailedError(UnauthorizedError): |
||||
|
code = "authentication_failed" |
||||
|
message = "Invalid username or password." |
||||
|
|
||||
|
|
||||
|
class AuthSessionExpiredError(UnauthorizedError): |
||||
|
code = "auth_session_expired" |
||||
|
message = "Authentication session is no longer valid." |
||||
|
|
||||
|
|
||||
|
class DirectoryNotFoundError(NotFoundError): |
||||
|
code = "directory_not_found" |
||||
|
message = "Directory does not exist." |
||||
|
|
||||
|
|
||||
|
class DirectoryConflictError(ConflictError): |
||||
|
code = "directory_conflict" |
||||
|
message = "Directory already exists." |
||||
|
|
||||
|
|
||||
|
class FileNotFoundError(NotFoundError): |
||||
|
code = "file_not_found" |
||||
|
message = "File does not exist." |
||||
|
|
||||
|
|
||||
|
class FileConflictError(ConflictError): |
||||
|
code = "file_conflict" |
||||
|
message = "File already exists." |
||||
|
|
||||
|
|
||||
|
class UploadNotFoundError(NotFoundError): |
||||
|
code = "upload_not_found" |
||||
|
message = "Upload session does not exist." |
||||
|
|
||||
|
|
||||
|
class UploadConflictError(ConflictError): |
||||
|
code = "upload_conflict" |
||||
|
message = "Upload state conflict." |
||||
|
|
||||
|
|
||||
|
class RangeNotSatisfiableError(AppError): |
||||
|
code = "range_not_satisfiable" |
||||
|
message = "Requested byte range is not satisfiable." |
||||
|
|
||||
|
|
||||
|
class PreviewNotSupportedError(AppError): |
||||
|
code = "preview_not_supported" |
||||
|
message = "Preview is not supported for this file type." |
||||
@ -0,0 +1,5 @@ |
|||||
|
from uuid import uuid4 |
||||
|
|
||||
|
|
||||
|
def new_id(prefix: str) -> str: |
||||
|
return f"{prefix}_{uuid4().hex}" |
||||
@ -0,0 +1,50 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
import base64 |
||||
|
import hashlib |
||||
|
import json |
||||
|
from typing import Any |
||||
|
|
||||
|
from cryptography.fernet import Fernet, InvalidToken |
||||
|
|
||||
|
from app.core.config import get_settings |
||||
|
from app.core.exceptions import ValidationError |
||||
|
|
||||
|
SECRET_PREFIX = "fernet:v1:" |
||||
|
|
||||
|
|
||||
|
class SecretConfigError(ValidationError): |
||||
|
code = "secret_config_invalid" |
||||
|
message = "Secret configuration cannot be decrypted." |
||||
|
|
||||
|
|
||||
|
def encrypt_secret_config(secret_config: dict[str, Any]) -> str: |
||||
|
payload = json.dumps(secret_config, sort_keys=True, separators=(",", ":"), ensure_ascii=True).encode("utf-8") |
||||
|
token = _fernet().encrypt(payload).decode("utf-8") |
||||
|
return SECRET_PREFIX + token |
||||
|
|
||||
|
|
||||
|
def decrypt_secret_config(raw_value: str) -> dict[str, Any]: |
||||
|
if not raw_value: |
||||
|
return {} |
||||
|
if not raw_value.startswith(SECRET_PREFIX): |
||||
|
value = json.loads(raw_value) |
||||
|
return value if isinstance(value, dict) else {} |
||||
|
|
||||
|
token = raw_value.removeprefix(SECRET_PREFIX).encode("utf-8") |
||||
|
try: |
||||
|
payload = _fernet().decrypt(token) |
||||
|
except InvalidToken as exc: |
||||
|
raise SecretConfigError() from exc |
||||
|
value = json.loads(payload.decode("utf-8")) |
||||
|
return value if isinstance(value, dict) else {} |
||||
|
|
||||
|
|
||||
|
def list_secret_fields(raw_value: str) -> list[str]: |
||||
|
return sorted(decrypt_secret_config(raw_value)) |
||||
|
|
||||
|
|
||||
|
def _fernet() -> Fernet: |
||||
|
digest = hashlib.sha256(get_settings().secret_key.encode("utf-8")).digest() |
||||
|
key = base64.urlsafe_b64encode(digest) |
||||
|
return Fernet(key) |
||||
@ -0,0 +1,34 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
import base64 |
||||
|
import hashlib |
||||
|
import hmac |
||||
|
import secrets |
||||
|
|
||||
|
|
||||
|
def hash_password(password: str, salt: str | None = None) -> str: |
||||
|
salt_value = salt or secrets.token_hex(16) |
||||
|
derived = hashlib.pbkdf2_hmac( |
||||
|
"sha256", |
||||
|
password.encode("utf-8"), |
||||
|
salt_value.encode("utf-8"), |
||||
|
200_000, |
||||
|
) |
||||
|
digest = base64.b64encode(derived).decode("utf-8") |
||||
|
return f"pbkdf2_sha256${salt_value}${digest}" |
||||
|
|
||||
|
|
||||
|
def verify_password(password: str, password_hash: str) -> bool: |
||||
|
algorithm, salt, digest = password_hash.split("$", 2) |
||||
|
if algorithm != "pbkdf2_sha256": |
||||
|
return False |
||||
|
expected = hash_password(password, salt=salt) |
||||
|
return hmac.compare_digest(expected, password_hash) |
||||
|
|
||||
|
|
||||
|
def generate_session_token() -> str: |
||||
|
return f"iron_{secrets.token_urlsafe(32)}" |
||||
|
|
||||
|
|
||||
|
def hash_session_token(token: str) -> str: |
||||
|
return hashlib.sha256(token.encode("utf-8")).hexdigest() |
||||
@ -0,0 +1,42 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from datetime import datetime |
||||
|
|
||||
|
from app.models.entities import Blob |
||||
|
|
||||
|
|
||||
|
def build_blob_storage_key(blob: Blob) -> str: |
||||
|
content_hash = blob.content_hash |
||||
|
prefix_one = content_hash[:2] if len(content_hash) >= 2 else "xx" |
||||
|
prefix_two = content_hash[2:4] if len(content_hash) >= 4 else "xx" |
||||
|
kind = blob.kind or "blob" |
||||
|
return f"objects/{kind}/sha256/{prefix_one}/{prefix_two}/{content_hash}.blob" |
||||
|
|
||||
|
|
||||
|
def build_preview_artifact_storage_key(*, artifact_type: str, blob: Blob) -> str: |
||||
|
content_hash = blob.content_hash |
||||
|
prefix_one = content_hash[:2] if len(content_hash) >= 2 else "xx" |
||||
|
prefix_two = content_hash[2:4] if len(content_hash) >= 4 else "xx" |
||||
|
artifact_segment = _normalize_segment(artifact_type) |
||||
|
return f"objects/preview/{artifact_segment}/sha256/{prefix_one}/{prefix_two}/{content_hash}.blob" |
||||
|
|
||||
|
|
||||
|
def build_export_snapshot_storage_key(*, exported_at: datetime, digest: str) -> str: |
||||
|
year = f"{exported_at.year:04d}" |
||||
|
month = f"{exported_at.month:02d}" |
||||
|
day = f"{exported_at.day:02d}" |
||||
|
normalized_digest = _normalize_segment(digest) or "snapshot" |
||||
|
return f"objects/export/metadata/{year}/{month}/{day}/{normalized_digest}.json" |
||||
|
|
||||
|
|
||||
|
def build_manifest_storage_key(*, scope: str, identifier: str) -> str: |
||||
|
normalized_scope = _normalize_segment(scope) or "general" |
||||
|
normalized_identifier = _normalize_segment(identifier) or "manifest" |
||||
|
return f"objects/manifest/{normalized_scope}/{normalized_identifier}.json" |
||||
|
|
||||
|
|
||||
|
def _normalize_segment(value: str) -> str: |
||||
|
cleaned = "".join(char if char.isalnum() or char in {"-", "_"} else "-" for char in value.strip().lower()) |
||||
|
while "--" in cleaned: |
||||
|
cleaned = cleaned.replace("--", "-") |
||||
|
return cleaned.strip("-_") |
||||
@ -0,0 +1,2 @@ |
|||||
|
"""Database module.""" |
||||
|
|
||||
@ -0,0 +1,6 @@ |
|||||
|
from sqlalchemy.orm import DeclarativeBase |
||||
|
|
||||
|
|
||||
|
class Base(DeclarativeBase): |
||||
|
pass |
||||
|
|
||||
@ -0,0 +1,11 @@ |
|||||
|
from sqlalchemy import text |
||||
|
from sqlalchemy.ext.asyncio import AsyncEngine |
||||
|
|
||||
|
|
||||
|
async def ping_database(engine: AsyncEngine) -> bool: |
||||
|
try: |
||||
|
async with engine.connect() as connection: |
||||
|
await connection.execute(text("SELECT 1")) |
||||
|
return True |
||||
|
except Exception: |
||||
|
return False |
||||
@ -0,0 +1,37 @@ |
|||||
|
from collections.abc import AsyncIterator |
||||
|
from functools import lru_cache |
||||
|
|
||||
|
from sqlalchemy import event |
||||
|
from sqlalchemy.ext.asyncio import AsyncEngine, AsyncSession, async_sessionmaker, create_async_engine |
||||
|
|
||||
|
from app.core.config import get_settings |
||||
|
|
||||
|
|
||||
|
@lru_cache(maxsize=1) |
||||
|
def create_engine() -> AsyncEngine: |
||||
|
settings = get_settings() |
||||
|
engine = create_async_engine(settings.database_url, future=True) |
||||
|
|
||||
|
@event.listens_for(engine.sync_engine, "connect") |
||||
|
def _set_sqlite_pragma(dbapi_connection, _connection_record) -> None: |
||||
|
cursor = dbapi_connection.cursor() |
||||
|
cursor.execute("PRAGMA foreign_keys=ON") |
||||
|
cursor.close() |
||||
|
|
||||
|
return engine |
||||
|
|
||||
|
|
||||
|
def create_session_factory() -> async_sessionmaker[AsyncSession]: |
||||
|
return async_sessionmaker(create_engine(), class_=AsyncSession, expire_on_commit=False) |
||||
|
|
||||
|
|
||||
|
async def get_db_session() -> AsyncIterator[AsyncSession]: |
||||
|
session_factory = create_session_factory() |
||||
|
async with session_factory() as session: |
||||
|
yield session |
||||
|
|
||||
|
|
||||
|
async def dispose_engine() -> None: |
||||
|
engine = create_engine() |
||||
|
await engine.dispose() |
||||
|
create_engine.cache_clear() |
||||
@ -0,0 +1,2 @@ |
|||||
|
"""Domain layer.""" |
||||
|
|
||||
@ -0,0 +1,2 @@ |
|||||
|
"""Background job framework.""" |
||||
|
|
||||
@ -0,0 +1,110 @@ |
|||||
|
from pathlib import Path |
||||
|
|
||||
|
from contextlib import asynccontextmanager |
||||
|
|
||||
|
import anyio |
||||
|
from fastapi import Depends, FastAPI, Request |
||||
|
from fastapi.responses import JSONResponse |
||||
|
from fastapi.staticfiles import StaticFiles |
||||
|
|
||||
|
from app.api.router import api_router |
||||
|
from app.api.dependencies import require_current_user |
||||
|
from app.api.routes.tus_uploads import router as tus_upload_router |
||||
|
from app.web.routes import router as web_router |
||||
|
from app.core.config import get_settings |
||||
|
from app.core.exceptions import AppError |
||||
|
from app.db.session import create_session_factory |
||||
|
from app.services.directory_service import DirectoryService |
||||
|
from app.services.auth_service import AuthService |
||||
|
from app.services.job_service import JobService |
||||
|
from app.services.local_storage_service import LocalStorageService |
||||
|
from app.services.policy_service import PolicyService |
||||
|
from app.services.storage_service import StorageService |
||||
|
from app.services.upload_service import UploadService |
||||
|
|
||||
|
WEB_DIST_DIR = Path(__file__).resolve().parent / "web" / "dist" |
||||
|
|
||||
|
|
||||
|
@asynccontextmanager |
||||
|
async def lifespan(_app: FastAPI): |
||||
|
session_factory = create_session_factory() |
||||
|
|
||||
|
async with session_factory() as session: |
||||
|
service = DirectoryService(session) |
||||
|
await service.ensure_root_directory() |
||||
|
auth_service = AuthService(session) |
||||
|
await auth_service.ensure_bootstrap_user() |
||||
|
upload_service = UploadService(session) |
||||
|
await upload_service.ensure_temp_dir() |
||||
|
local_storage_service = LocalStorageService(session) |
||||
|
await local_storage_service.ensure_backend_ready() |
||||
|
policy_service = PolicyService(session) |
||||
|
await policy_service.ensure_default_policies() |
||||
|
storage_service = StorageService(session) |
||||
|
await storage_service.ensure_cache_dir() |
||||
|
|
||||
|
settings = get_settings() |
||||
|
|
||||
|
async def _job_loop() -> None: |
||||
|
while True: |
||||
|
async with session_factory() as session: |
||||
|
service = JobService(session) |
||||
|
await service.run_pending_jobs(limit=settings.job_batch_size) |
||||
|
await anyio.sleep(settings.job_poll_interval_seconds) |
||||
|
|
||||
|
async with anyio.create_task_group() as task_group: |
||||
|
task_group.start_soon(_job_loop) |
||||
|
yield |
||||
|
task_group.cancel_scope.cancel() |
||||
|
|
||||
|
|
||||
|
def create_app() -> FastAPI: |
||||
|
settings = get_settings() |
||||
|
app = FastAPI( |
||||
|
title=settings.app_name, |
||||
|
debug=settings.debug, |
||||
|
version="0.1.0", |
||||
|
lifespan=lifespan, |
||||
|
) |
||||
|
|
||||
|
@app.exception_handler(AppError) |
||||
|
async def app_error_handler(_request: Request, exc: AppError) -> JSONResponse: |
||||
|
status_code = 400 |
||||
|
if exc.code.endswith("_not_found") or exc.code == "not_found": |
||||
|
status_code = 404 |
||||
|
elif exc.code.endswith("_conflict") or exc.code == "conflict": |
||||
|
status_code = 409 |
||||
|
elif exc.code == "unauthorized" or exc.code.startswith("auth_") or exc.code == "authentication_failed": |
||||
|
status_code = 401 |
||||
|
elif exc.code == "range_not_satisfiable": |
||||
|
status_code = 416 |
||||
|
|
||||
|
return JSONResponse( |
||||
|
status_code=status_code, |
||||
|
content={ |
||||
|
"error": { |
||||
|
"code": exc.code, |
||||
|
"message": exc.detail, |
||||
|
"details": None, |
||||
|
} |
||||
|
}, |
||||
|
) |
||||
|
|
||||
|
app.include_router(api_router, prefix=settings.api_prefix) |
||||
|
app.include_router(web_router) |
||||
|
app.include_router( |
||||
|
tus_upload_router, |
||||
|
prefix="/files", |
||||
|
tags=["tus"], |
||||
|
dependencies=[Depends(require_current_user)], |
||||
|
) |
||||
|
if (WEB_DIST_DIR / "assets").exists(): |
||||
|
app.mount( |
||||
|
"/assets", |
||||
|
StaticFiles(directory=WEB_DIST_DIR / "assets"), |
||||
|
name="web-assets", |
||||
|
) |
||||
|
return app |
||||
|
|
||||
|
|
||||
|
app = create_app() |
||||
@ -0,0 +1,33 @@ |
|||||
|
"""SQLAlchemy models package.""" |
||||
|
|
||||
|
from app.models.entities import AuthSession |
||||
|
from app.models.entities import Backend |
||||
|
from app.models.entities import Blob |
||||
|
from app.models.entities import BlobReplica |
||||
|
from app.models.entities import CacheEntry |
||||
|
from app.models.entities import Directory |
||||
|
from app.models.entities import FileEntry |
||||
|
from app.models.entities import FileVersion |
||||
|
from app.models.entities import Job |
||||
|
from app.models.entities import PlacementPolicy |
||||
|
from app.models.entities import PreviewArtifact |
||||
|
from app.models.entities import UploadSession |
||||
|
from app.models.entities import UploadSessionPart |
||||
|
from app.models.entities import User |
||||
|
|
||||
|
__all__ = [ |
||||
|
"AuthSession", |
||||
|
"Backend", |
||||
|
"Blob", |
||||
|
"BlobReplica", |
||||
|
"CacheEntry", |
||||
|
"Directory", |
||||
|
"FileEntry", |
||||
|
"FileVersion", |
||||
|
"Job", |
||||
|
"PlacementPolicy", |
||||
|
"PreviewArtifact", |
||||
|
"UploadSession", |
||||
|
"UploadSessionPart", |
||||
|
"User", |
||||
|
] |
||||
@ -0,0 +1,275 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from datetime import datetime |
||||
|
from typing import Optional |
||||
|
|
||||
|
from sqlalchemy import Boolean, DateTime, ForeignKey, Index, Integer, String, Text, func |
||||
|
from sqlalchemy.orm import Mapped, mapped_column |
||||
|
|
||||
|
from app.db.base import Base |
||||
|
|
||||
|
|
||||
|
class TimestampMixin: |
||||
|
created_at: Mapped[datetime] = mapped_column( |
||||
|
DateTime(timezone=True), |
||||
|
nullable=False, |
||||
|
server_default=func.now(), |
||||
|
) |
||||
|
updated_at: Mapped[datetime] = mapped_column( |
||||
|
DateTime(timezone=True), |
||||
|
nullable=False, |
||||
|
server_default=func.now(), |
||||
|
onupdate=func.now(), |
||||
|
) |
||||
|
|
||||
|
|
||||
|
class User(TimestampMixin, Base): |
||||
|
__tablename__ = "users" |
||||
|
|
||||
|
id: Mapped[str] = mapped_column(String, primary_key=True) |
||||
|
username: Mapped[str] = mapped_column(String, unique=True, nullable=False) |
||||
|
password_hash: Mapped[str] = mapped_column(String, nullable=False) |
||||
|
|
||||
|
|
||||
|
class AuthSession(TimestampMixin, Base): |
||||
|
__tablename__ = "auth_sessions" |
||||
|
__table_args__ = ( |
||||
|
Index("idx_auth_sessions_user_status", "user_id", "revoked_at", "expires_at"), |
||||
|
Index("idx_auth_sessions_token_hash", "token_hash", unique=True), |
||||
|
) |
||||
|
|
||||
|
id: Mapped[str] = mapped_column(String, primary_key=True) |
||||
|
user_id: Mapped[str] = mapped_column(String, ForeignKey("users.id"), nullable=False) |
||||
|
token_hash: Mapped[str] = mapped_column(String, nullable=False) |
||||
|
expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False) |
||||
|
revoked_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True) |
||||
|
|
||||
|
|
||||
|
class Directory(TimestampMixin, Base): |
||||
|
__tablename__ = "directories" |
||||
|
__table_args__ = ( |
||||
|
Index("idx_directories_parent_id", "parent_id"), |
||||
|
Index("idx_directories_path_key", "path_key"), |
||||
|
) |
||||
|
|
||||
|
id: Mapped[str] = mapped_column(String, primary_key=True) |
||||
|
parent_id: Mapped[Optional[str]] = mapped_column( |
||||
|
String, |
||||
|
ForeignKey("directories.id"), |
||||
|
nullable=True, |
||||
|
) |
||||
|
name: Mapped[str] = mapped_column(String, nullable=False) |
||||
|
path_key: Mapped[str] = mapped_column(String, unique=True, nullable=False) |
||||
|
deleted_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True) |
||||
|
|
||||
|
|
||||
|
class FileEntry(TimestampMixin, Base): |
||||
|
__tablename__ = "file_entries" |
||||
|
__table_args__ = ( |
||||
|
Index("idx_file_entries_directory_name", "directory_id", "name", "is_deleted", unique=True), |
||||
|
Index("idx_file_entries_current_version_id", "current_version_id"), |
||||
|
Index("idx_file_entries_deleted_at", "deleted_at"), |
||||
|
) |
||||
|
|
||||
|
id: Mapped[str] = mapped_column(String, primary_key=True) |
||||
|
directory_id: Mapped[str] = mapped_column( |
||||
|
String, |
||||
|
ForeignKey("directories.id"), |
||||
|
nullable=False, |
||||
|
) |
||||
|
name: Mapped[str] = mapped_column(String, nullable=False) |
||||
|
mime_type: Mapped[Optional[str]] = mapped_column(String, nullable=True) |
||||
|
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False) |
||||
|
current_version_id: Mapped[Optional[str]] = mapped_column(String, nullable=True) |
||||
|
is_deleted: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="0") |
||||
|
deleted_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True) |
||||
|
|
||||
|
|
||||
|
class FileVersion(Base): |
||||
|
__tablename__ = "file_versions" |
||||
|
__table_args__ = ( |
||||
|
Index("idx_file_versions_file_entry_id", "file_entry_id"), |
||||
|
Index("idx_file_versions_content_hash", "content_hash"), |
||||
|
) |
||||
|
|
||||
|
id: Mapped[str] = mapped_column(String, primary_key=True) |
||||
|
file_entry_id: Mapped[str] = mapped_column( |
||||
|
String, |
||||
|
ForeignKey("file_entries.id"), |
||||
|
nullable=False, |
||||
|
) |
||||
|
content_hash: Mapped[str] = mapped_column(String, nullable=False) |
||||
|
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False) |
||||
|
storage_layout: Mapped[str] = mapped_column(String, nullable=False) |
||||
|
created_at: Mapped[datetime] = mapped_column( |
||||
|
DateTime(timezone=True), |
||||
|
nullable=False, |
||||
|
server_default=func.now(), |
||||
|
) |
||||
|
|
||||
|
|
||||
|
class Blob(Base): |
||||
|
__tablename__ = "blobs" |
||||
|
__table_args__ = ( |
||||
|
Index("idx_blobs_file_version_index", "file_version_id", "blob_index", unique=True), |
||||
|
Index("idx_blobs_content_hash", "content_hash"), |
||||
|
) |
||||
|
|
||||
|
id: Mapped[str] = mapped_column(String, primary_key=True) |
||||
|
file_version_id: Mapped[str] = mapped_column( |
||||
|
String, |
||||
|
ForeignKey("file_versions.id"), |
||||
|
nullable=False, |
||||
|
) |
||||
|
blob_index: Mapped[int] = mapped_column(Integer, nullable=False) |
||||
|
kind: Mapped[str] = mapped_column(String, nullable=False) |
||||
|
content_hash: Mapped[str] = mapped_column(String, nullable=False) |
||||
|
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False) |
||||
|
logical_offset: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0") |
||||
|
created_at: Mapped[datetime] = mapped_column( |
||||
|
DateTime(timezone=True), |
||||
|
nullable=False, |
||||
|
server_default=func.now(), |
||||
|
) |
||||
|
|
||||
|
|
||||
|
class Backend(TimestampMixin, Base): |
||||
|
__tablename__ = "backends" |
||||
|
__table_args__ = ( |
||||
|
Index("idx_backends_enabled", "is_enabled"), |
||||
|
Index("idx_backends_priority", "write_priority", "read_priority"), |
||||
|
) |
||||
|
|
||||
|
id: Mapped[str] = mapped_column(String, primary_key=True) |
||||
|
name: Mapped[str] = mapped_column(String, unique=True, nullable=False) |
||||
|
type: Mapped[str] = mapped_column(String, nullable=False) |
||||
|
stability_class: Mapped[str] = mapped_column(String, nullable=False) |
||||
|
read_priority: Mapped[int] = mapped_column(Integer, nullable=False) |
||||
|
write_priority: Mapped[int] = mapped_column(Integer, nullable=False) |
||||
|
is_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="1") |
||||
|
public_config_json: Mapped[str] = mapped_column(Text, nullable=False) |
||||
|
secret_config_json: Mapped[str] = mapped_column(Text, nullable=False) |
||||
|
capacity_hint_bytes: Mapped[Optional[int]] = mapped_column(Integer, nullable=True) |
||||
|
last_health_status: Mapped[Optional[str]] = mapped_column(String, nullable=True) |
||||
|
last_health_checked_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True) |
||||
|
|
||||
|
|
||||
|
class PlacementPolicy(TimestampMixin, Base): |
||||
|
__tablename__ = "placement_policies" |
||||
|
__table_args__ = ( |
||||
|
Index("idx_placement_policies_file_class", "file_class", unique=True), |
||||
|
) |
||||
|
|
||||
|
id: Mapped[str] = mapped_column(String, primary_key=True) |
||||
|
file_class: Mapped[str] = mapped_column(String, nullable=False) |
||||
|
require_local: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, server_default="1") |
||||
|
stable_replica_count: Mapped[int] = mapped_column(Integer, nullable=False, default=1, server_default="1") |
||||
|
opportunistic_replica_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0") |
||||
|
preferred_backend_ids_json: Mapped[str] = mapped_column(Text, nullable=False, default="[]", server_default="[]") |
||||
|
excluded_backend_ids_json: Mapped[str] = mapped_column(Text, nullable=False, default="[]", server_default="[]") |
||||
|
max_non_local_size_bytes: Mapped[Optional[int]] = mapped_column(Integer, nullable=True) |
||||
|
|
||||
|
|
||||
|
class BlobReplica(TimestampMixin, Base): |
||||
|
__tablename__ = "blob_replicas" |
||||
|
__table_args__ = ( |
||||
|
Index("idx_blob_replicas_blob_backend", "blob_id", "backend_id", unique=True), |
||||
|
Index("idx_blob_replicas_backend_status", "backend_id", "status"), |
||||
|
Index("idx_blob_replicas_storage_key", "storage_key"), |
||||
|
) |
||||
|
|
||||
|
id: Mapped[str] = mapped_column(String, primary_key=True) |
||||
|
blob_id: Mapped[str] = mapped_column(String, ForeignKey("blobs.id"), nullable=False) |
||||
|
backend_id: Mapped[str] = mapped_column(String, ForeignKey("backends.id"), nullable=False) |
||||
|
storage_key: Mapped[str] = mapped_column(String, nullable=False) |
||||
|
status: Mapped[str] = mapped_column(String, nullable=False) |
||||
|
etag: Mapped[Optional[str]] = mapped_column(String, nullable=True) |
||||
|
checksum: Mapped[Optional[str]] = mapped_column(String, nullable=True) |
||||
|
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False) |
||||
|
last_verified_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True) |
||||
|
|
||||
|
|
||||
|
class UploadSession(TimestampMixin, Base): |
||||
|
__tablename__ = "upload_sessions" |
||||
|
__table_args__ = ( |
||||
|
Index("idx_upload_sessions_status", "status"), |
||||
|
Index("idx_upload_sessions_updated_at", "updated_at"), |
||||
|
) |
||||
|
|
||||
|
id: Mapped[str] = mapped_column(String, primary_key=True) |
||||
|
directory_id: Mapped[Optional[str]] = mapped_column(String, ForeignKey("directories.id"), nullable=True) |
||||
|
filename: Mapped[Optional[str]] = mapped_column(String, nullable=True) |
||||
|
total_size_bytes: Mapped[int] = mapped_column(Integer, nullable=False) |
||||
|
received_size_bytes: Mapped[int] = mapped_column(Integer, nullable=False) |
||||
|
status: Mapped[str] = mapped_column(String, nullable=False) |
||||
|
temp_path: Mapped[str] = mapped_column(String, nullable=False) |
||||
|
tus_upload_url: Mapped[Optional[str]] = mapped_column(String, nullable=True) |
||||
|
upload_metadata_json: Mapped[Optional[str]] = mapped_column(Text, nullable=True) |
||||
|
|
||||
|
|
||||
|
class UploadSessionPart(TimestampMixin, Base): |
||||
|
__tablename__ = "upload_session_parts" |
||||
|
__table_args__ = ( |
||||
|
Index("idx_upload_session_parts_unique", "upload_session_id", "part_number", unique=True), |
||||
|
Index("idx_upload_session_parts_upload_id", "upload_session_id"), |
||||
|
) |
||||
|
|
||||
|
id: Mapped[str] = mapped_column(String, primary_key=True) |
||||
|
upload_session_id: Mapped[str] = mapped_column( |
||||
|
String, |
||||
|
ForeignKey("upload_sessions.id"), |
||||
|
nullable=False, |
||||
|
) |
||||
|
part_number: Mapped[int] = mapped_column(Integer, nullable=False) |
||||
|
byte_offset: Mapped[int] = mapped_column(Integer, nullable=False) |
||||
|
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False) |
||||
|
checksum: Mapped[Optional[str]] = mapped_column(String, nullable=True) |
||||
|
status: Mapped[str] = mapped_column(String, nullable=False) |
||||
|
|
||||
|
|
||||
|
class Job(TimestampMixin, Base): |
||||
|
__tablename__ = "jobs" |
||||
|
__table_args__ = ( |
||||
|
Index("idx_jobs_status_run_after", "status", "run_after"), |
||||
|
Index("idx_jobs_kind_status", "kind", "status"), |
||||
|
) |
||||
|
|
||||
|
id: Mapped[str] = mapped_column(String, primary_key=True) |
||||
|
kind: Mapped[str] = mapped_column(String, nullable=False) |
||||
|
status: Mapped[str] = mapped_column(String, nullable=False) |
||||
|
payload_json: Mapped[str] = mapped_column(Text, nullable=False) |
||||
|
attempt_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0") |
||||
|
max_attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=5, server_default="5") |
||||
|
run_after: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False) |
||||
|
last_error: Mapped[Optional[str]] = mapped_column(Text, nullable=True) |
||||
|
|
||||
|
|
||||
|
class PreviewArtifact(TimestampMixin, Base): |
||||
|
__tablename__ = "preview_artifacts" |
||||
|
__table_args__ = ( |
||||
|
Index("idx_preview_artifacts_version_type", "file_version_id", "artifact_type", unique=True), |
||||
|
) |
||||
|
|
||||
|
id: Mapped[str] = mapped_column(String, primary_key=True) |
||||
|
file_version_id: Mapped[str] = mapped_column(String, ForeignKey("file_versions.id"), nullable=False) |
||||
|
artifact_type: Mapped[str] = mapped_column(String, nullable=False) |
||||
|
blob_id: Mapped[str] = mapped_column(String, ForeignKey("blobs.id"), nullable=False) |
||||
|
status: Mapped[str] = mapped_column(String, nullable=False) |
||||
|
|
||||
|
|
||||
|
class CacheEntry(TimestampMixin, Base): |
||||
|
__tablename__ = "cache_entries" |
||||
|
__table_args__ = ( |
||||
|
Index("idx_cache_entries_blob_id", "blob_id"), |
||||
|
Index("idx_cache_entries_last_accessed_at", "last_accessed_at"), |
||||
|
Index("idx_cache_entries_expires_at", "expires_at"), |
||||
|
) |
||||
|
|
||||
|
id: Mapped[str] = mapped_column(String, primary_key=True) |
||||
|
blob_id: Mapped[str] = mapped_column(String, ForeignKey("blobs.id"), nullable=False) |
||||
|
local_path: Mapped[str] = mapped_column(String, nullable=False) |
||||
|
cache_kind: Mapped[str] = mapped_column(String, nullable=False) |
||||
|
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False) |
||||
|
status: Mapped[str] = mapped_column(String, nullable=False) |
||||
|
last_accessed_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False) |
||||
|
expires_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True) |
||||
@ -0,0 +1,2 @@ |
|||||
|
"""Repository layer.""" |
||||
|
|
||||
@ -0,0 +1,40 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from datetime import datetime |
||||
|
|
||||
|
from sqlalchemy import select |
||||
|
from sqlalchemy.ext.asyncio import AsyncSession |
||||
|
|
||||
|
from app.models.entities import AuthSession |
||||
|
|
||||
|
|
||||
|
class AuthSessionRepository: |
||||
|
def __init__(self, session: AsyncSession) -> None: |
||||
|
self.session = session |
||||
|
|
||||
|
async def create(self, auth_session: AuthSession) -> AuthSession: |
||||
|
self.session.add(auth_session) |
||||
|
await self.session.commit() |
||||
|
await self.session.refresh(auth_session) |
||||
|
return auth_session |
||||
|
|
||||
|
async def get_active_by_token_hash(self, token_hash: str, now: datetime) -> AuthSession | None: |
||||
|
result = await self.session.execute( |
||||
|
select(AuthSession).where( |
||||
|
AuthSession.token_hash == token_hash, |
||||
|
AuthSession.revoked_at.is_(None), |
||||
|
AuthSession.expires_at > now, |
||||
|
) |
||||
|
) |
||||
|
return result.scalar_one_or_none() |
||||
|
|
||||
|
async def get_by_token_hash(self, token_hash: str) -> AuthSession | None: |
||||
|
result = await self.session.execute( |
||||
|
select(AuthSession).where(AuthSession.token_hash == token_hash) |
||||
|
) |
||||
|
return result.scalar_one_or_none() |
||||
|
|
||||
|
async def save(self, auth_session: AuthSession) -> AuthSession: |
||||
|
await self.session.commit() |
||||
|
await self.session.refresh(auth_session) |
||||
|
return auth_session |
||||
@ -0,0 +1,64 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
import json |
||||
|
|
||||
|
from sqlalchemy import select |
||||
|
from sqlalchemy.ext.asyncio import AsyncSession |
||||
|
|
||||
|
from app.models.entities import Backend, PlacementPolicy |
||||
|
|
||||
|
|
||||
|
class BackendRepository: |
||||
|
def __init__(self, session: AsyncSession) -> None: |
||||
|
self.session = session |
||||
|
|
||||
|
async def get_by_name(self, name: str) -> Backend | None: |
||||
|
result = await self.session.execute(select(Backend).where(Backend.name == name)) |
||||
|
return result.scalar_one_or_none() |
||||
|
|
||||
|
async def get_by_id(self, backend_id: str) -> Backend | None: |
||||
|
return await self.session.get(Backend, backend_id) |
||||
|
|
||||
|
async def create(self, backend: Backend) -> Backend: |
||||
|
self.session.add(backend) |
||||
|
await self.session.flush() |
||||
|
await self.session.refresh(backend) |
||||
|
return backend |
||||
|
|
||||
|
async def list_all(self) -> list[Backend]: |
||||
|
result = await self.session.execute(select(Backend).order_by(Backend.write_priority.desc(), Backend.name.asc())) |
||||
|
return list(result.scalars().all()) |
||||
|
|
||||
|
async def list_enabled(self) -> list[Backend]: |
||||
|
result = await self.session.execute( |
||||
|
select(Backend) |
||||
|
.where(Backend.is_enabled.is_(True)) |
||||
|
.order_by(Backend.write_priority.desc(), Backend.read_priority.desc(), Backend.name.asc()) |
||||
|
) |
||||
|
return list(result.scalars().all()) |
||||
|
|
||||
|
async def save(self, backend: Backend) -> Backend: |
||||
|
await self.session.flush() |
||||
|
await self.session.refresh(backend) |
||||
|
return backend |
||||
|
|
||||
|
async def delete(self, backend: Backend) -> None: |
||||
|
await self.session.delete(backend) |
||||
|
await self.session.flush() |
||||
|
|
||||
|
async def is_referenced_by_policy(self, backend_id: str) -> bool: |
||||
|
result = await self.session.execute(select(PlacementPolicy)) |
||||
|
for policy in result.scalars().all(): |
||||
|
preferred = json.loads(policy.preferred_backend_ids_json or "[]") |
||||
|
excluded = json.loads(policy.excluded_backend_ids_json or "[]") |
||||
|
if backend_id in preferred or backend_id in excluded: |
||||
|
return True |
||||
|
return False |
||||
|
|
||||
|
async def list_enabled_by_type(self, backend_type: str) -> list[Backend]: |
||||
|
result = await self.session.execute( |
||||
|
select(Backend) |
||||
|
.where(Backend.is_enabled.is_(True), Backend.type == backend_type) |
||||
|
.order_by(Backend.write_priority.desc(), Backend.read_priority.desc(), Backend.name.asc()) |
||||
|
) |
||||
|
return list(result.scalars().all()) |
||||
@ -0,0 +1,75 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from sqlalchemy import func, select |
||||
|
from sqlalchemy.ext.asyncio import AsyncSession |
||||
|
|
||||
|
from app.models.entities import Backend |
||||
|
from app.models.entities import BlobReplica |
||||
|
|
||||
|
|
||||
|
class BlobReplicaRepository: |
||||
|
def __init__(self, session: AsyncSession) -> None: |
||||
|
self.session = session |
||||
|
|
||||
|
async def get_by_blob_and_backend(self, blob_id: str, backend_id: str) -> BlobReplica | None: |
||||
|
result = await self.session.execute( |
||||
|
select(BlobReplica).where( |
||||
|
BlobReplica.blob_id == blob_id, |
||||
|
BlobReplica.backend_id == backend_id, |
||||
|
) |
||||
|
) |
||||
|
return result.scalar_one_or_none() |
||||
|
|
||||
|
async def create(self, replica: BlobReplica) -> BlobReplica: |
||||
|
self.session.add(replica) |
||||
|
await self.session.flush() |
||||
|
await self.session.refresh(replica) |
||||
|
return replica |
||||
|
|
||||
|
async def get_ready_replica_for_blob(self, blob_id: str) -> BlobReplica | None: |
||||
|
result = await self.session.execute( |
||||
|
select(BlobReplica) |
||||
|
.where( |
||||
|
BlobReplica.blob_id == blob_id, |
||||
|
BlobReplica.status == "ready", |
||||
|
) |
||||
|
.order_by(BlobReplica.created_at.asc()) |
||||
|
) |
||||
|
return result.scalar_one_or_none() |
||||
|
|
||||
|
async def list_ready_replicas_with_backends(self, blob_id: str) -> list[tuple[BlobReplica, Backend]]: |
||||
|
result = await self.session.execute( |
||||
|
select(BlobReplica, Backend) |
||||
|
.join(Backend, BlobReplica.backend_id == Backend.id) |
||||
|
.where( |
||||
|
BlobReplica.blob_id == blob_id, |
||||
|
BlobReplica.status == "ready", |
||||
|
Backend.is_enabled.is_(True), |
||||
|
) |
||||
|
.order_by(Backend.read_priority.desc(), BlobReplica.created_at.asc()) |
||||
|
) |
||||
|
return list(result.all()) |
||||
|
|
||||
|
async def list_replicas_with_backends(self, blob_id: str) -> list[tuple[BlobReplica, Backend]]: |
||||
|
result = await self.session.execute( |
||||
|
select(BlobReplica, Backend) |
||||
|
.join(Backend, BlobReplica.backend_id == Backend.id) |
||||
|
.where(BlobReplica.blob_id == blob_id) |
||||
|
.order_by(Backend.read_priority.desc(), BlobReplica.created_at.asc()) |
||||
|
) |
||||
|
return list(result.all()) |
||||
|
|
||||
|
async def list_by_blob(self, blob_id: str) -> list[BlobReplica]: |
||||
|
result = await self.session.execute(select(BlobReplica).where(BlobReplica.blob_id == blob_id)) |
||||
|
return list(result.scalars().all()) |
||||
|
|
||||
|
async def save(self, replica: BlobReplica) -> BlobReplica: |
||||
|
await self.session.flush() |
||||
|
await self.session.refresh(replica) |
||||
|
return replica |
||||
|
|
||||
|
async def count_by_backend(self, backend_id: str) -> int: |
||||
|
result = await self.session.execute( |
||||
|
select(func.count()).select_from(BlobReplica).where(BlobReplica.backend_id == backend_id) |
||||
|
) |
||||
|
return int(result.scalar_one()) |
||||
@ -0,0 +1,31 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from sqlalchemy import select |
||||
|
from sqlalchemy.ext.asyncio import AsyncSession |
||||
|
|
||||
|
from app.models.entities import CacheEntry |
||||
|
|
||||
|
|
||||
|
class CacheEntryRepository: |
||||
|
def __init__(self, session: AsyncSession) -> None: |
||||
|
self.session = session |
||||
|
|
||||
|
async def get_ready_by_blob(self, blob_id: str) -> CacheEntry | None: |
||||
|
result = await self.session.execute( |
||||
|
select(CacheEntry).where( |
||||
|
CacheEntry.blob_id == blob_id, |
||||
|
CacheEntry.status == "ready", |
||||
|
) |
||||
|
) |
||||
|
return result.scalar_one_or_none() |
||||
|
|
||||
|
async def create(self, entry: CacheEntry) -> CacheEntry: |
||||
|
self.session.add(entry) |
||||
|
await self.session.flush() |
||||
|
await self.session.refresh(entry) |
||||
|
return entry |
||||
|
|
||||
|
async def save(self, entry: CacheEntry) -> CacheEntry: |
||||
|
await self.session.flush() |
||||
|
await self.session.refresh(entry) |
||||
|
return entry |
||||
@ -0,0 +1,58 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from sqlalchemy import select |
||||
|
from sqlalchemy.ext.asyncio import AsyncSession |
||||
|
|
||||
|
from app.models.entities import Directory |
||||
|
|
||||
|
|
||||
|
class DirectoryRepository: |
||||
|
def __init__(self, session: AsyncSession) -> None: |
||||
|
self.session = session |
||||
|
|
||||
|
async def get_by_id(self, directory_id: str) -> Directory | None: |
||||
|
return await self.session.get(Directory, directory_id) |
||||
|
|
||||
|
async def get_root(self) -> Directory | None: |
||||
|
result = await self.session.execute( |
||||
|
select(Directory).where(Directory.parent_id.is_(None), Directory.deleted_at.is_(None)) |
||||
|
) |
||||
|
return result.scalar_one_or_none() |
||||
|
|
||||
|
async def list_children(self, parent_id: str) -> list[Directory]: |
||||
|
result = await self.session.execute( |
||||
|
select(Directory) |
||||
|
.where(Directory.parent_id == parent_id, Directory.deleted_at.is_(None)) |
||||
|
.order_by(Directory.name.asc()) |
||||
|
) |
||||
|
return list(result.scalars().all()) |
||||
|
|
||||
|
async def get_by_parent_and_name(self, parent_id: str | None, name: str) -> Directory | None: |
||||
|
result = await self.session.execute( |
||||
|
select(Directory).where( |
||||
|
Directory.parent_id == parent_id, |
||||
|
Directory.name == name, |
||||
|
Directory.deleted_at.is_(None), |
||||
|
) |
||||
|
) |
||||
|
return result.scalar_one_or_none() |
||||
|
|
||||
|
async def list_descendants_by_path_prefix(self, path_prefix: str) -> list[Directory]: |
||||
|
like_pattern = f"{path_prefix}/%" |
||||
|
result = await self.session.execute( |
||||
|
select(Directory) |
||||
|
.where(Directory.path_key.like(like_pattern), Directory.deleted_at.is_(None)) |
||||
|
.order_by(Directory.path_key.asc()) |
||||
|
) |
||||
|
return list(result.scalars().all()) |
||||
|
|
||||
|
async def create(self, directory: Directory) -> Directory: |
||||
|
self.session.add(directory) |
||||
|
await self.session.flush() |
||||
|
await self.session.refresh(directory) |
||||
|
return directory |
||||
|
|
||||
|
async def save(self, directory: Directory) -> Directory: |
||||
|
await self.session.flush() |
||||
|
await self.session.refresh(directory) |
||||
|
return directory |
||||
@ -0,0 +1,103 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from datetime import datetime, UTC |
||||
|
|
||||
|
from sqlalchemy import select |
||||
|
from sqlalchemy.ext.asyncio import AsyncSession |
||||
|
|
||||
|
from app.models.entities import Blob, FileEntry, FileVersion |
||||
|
|
||||
|
|
||||
|
class FileRepository: |
||||
|
def __init__(self, session: AsyncSession) -> None: |
||||
|
self.session = session |
||||
|
|
||||
|
async def get_by_id(self, file_id: str) -> FileEntry | None: |
||||
|
return await self.session.get(FileEntry, file_id) |
||||
|
|
||||
|
async def get_by_directory_and_name(self, directory_id: str, name: str) -> FileEntry | None: |
||||
|
result = await self.session.execute( |
||||
|
select(FileEntry).where( |
||||
|
FileEntry.directory_id == directory_id, |
||||
|
FileEntry.name == name, |
||||
|
FileEntry.is_deleted.is_(False), |
||||
|
FileEntry.deleted_at.is_(None), |
||||
|
) |
||||
|
) |
||||
|
return result.scalar_one_or_none() |
||||
|
|
||||
|
async def list_by_directory(self, directory_id: str) -> list[FileEntry]: |
||||
|
result = await self.session.execute( |
||||
|
select(FileEntry) |
||||
|
.where( |
||||
|
FileEntry.directory_id == directory_id, |
||||
|
FileEntry.is_deleted.is_(False), |
||||
|
FileEntry.deleted_at.is_(None), |
||||
|
) |
||||
|
.order_by(FileEntry.name.asc()) |
||||
|
) |
||||
|
return list(result.scalars().all()) |
||||
|
|
||||
|
async def list_deleted(self) -> list[FileEntry]: |
||||
|
result = await self.session.execute( |
||||
|
select(FileEntry) |
||||
|
.where(FileEntry.is_deleted.is_(True), FileEntry.deleted_at.is_not(None)) |
||||
|
.order_by(FileEntry.deleted_at.desc(), FileEntry.name.asc()) |
||||
|
) |
||||
|
return list(result.scalars().all()) |
||||
|
|
||||
|
async def list_active(self) -> list[FileEntry]: |
||||
|
result = await self.session.execute( |
||||
|
select(FileEntry) |
||||
|
.where( |
||||
|
FileEntry.is_deleted.is_(False), |
||||
|
FileEntry.deleted_at.is_(None), |
||||
|
) |
||||
|
.order_by(FileEntry.created_at.asc()) |
||||
|
) |
||||
|
return list(result.scalars().all()) |
||||
|
|
||||
|
async def create_file_graph( |
||||
|
self, |
||||
|
file_entry: FileEntry, |
||||
|
file_version: FileVersion, |
||||
|
blob: Blob, |
||||
|
) -> FileEntry: |
||||
|
self.session.add(file_entry) |
||||
|
self.session.add(file_version) |
||||
|
await self.session.flush() |
||||
|
self.session.add(blob) |
||||
|
await self.session.flush() |
||||
|
await self.session.refresh(file_entry) |
||||
|
return file_entry |
||||
|
|
||||
|
async def get_primary_blob_for_file(self, file_id: str) -> Blob | None: |
||||
|
result = await self.session.execute( |
||||
|
select(Blob) |
||||
|
.join(FileVersion, Blob.file_version_id == FileVersion.id) |
||||
|
.join(FileEntry, FileVersion.id == FileEntry.current_version_id) |
||||
|
.where(FileEntry.id == file_id, Blob.blob_index == 0) |
||||
|
) |
||||
|
return result.scalar_one_or_none() |
||||
|
|
||||
|
async def get_blob_by_id(self, blob_id: str) -> Blob | None: |
||||
|
return await self.session.get(Blob, blob_id) |
||||
|
|
||||
|
async def soft_delete(self, file_entry: FileEntry) -> FileEntry: |
||||
|
file_entry.is_deleted = True |
||||
|
file_entry.deleted_at = datetime.now(UTC) |
||||
|
await self.session.flush() |
||||
|
await self.session.refresh(file_entry) |
||||
|
return file_entry |
||||
|
|
||||
|
async def restore(self, file_entry: FileEntry) -> FileEntry: |
||||
|
file_entry.is_deleted = False |
||||
|
file_entry.deleted_at = None |
||||
|
await self.session.flush() |
||||
|
await self.session.refresh(file_entry) |
||||
|
return file_entry |
||||
|
|
||||
|
async def save(self, file_entry: FileEntry) -> FileEntry: |
||||
|
await self.session.flush() |
||||
|
await self.session.refresh(file_entry) |
||||
|
return file_entry |
||||
@ -0,0 +1,53 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from datetime import datetime |
||||
|
|
||||
|
from sqlalchemy import select |
||||
|
from sqlalchemy.ext.asyncio import AsyncSession |
||||
|
|
||||
|
from app.models.entities import Job |
||||
|
|
||||
|
|
||||
|
class JobRepository: |
||||
|
def __init__(self, session: AsyncSession) -> None: |
||||
|
self.session = session |
||||
|
|
||||
|
async def create(self, job: Job) -> Job: |
||||
|
self.session.add(job) |
||||
|
await self.session.flush() |
||||
|
await self.session.refresh(job) |
||||
|
return job |
||||
|
|
||||
|
async def get_by_id(self, job_id: str) -> Job | None: |
||||
|
return await self.session.get(Job, job_id) |
||||
|
|
||||
|
async def list_all(self) -> list[Job]: |
||||
|
result = await self.session.execute(select(Job).order_by(Job.created_at.desc())) |
||||
|
return list(result.scalars().all()) |
||||
|
|
||||
|
async def list_runnable(self, now: datetime, limit: int) -> list[Job]: |
||||
|
result = await self.session.execute( |
||||
|
select(Job) |
||||
|
.where( |
||||
|
Job.status.in_(["queued", "retryable"]), |
||||
|
Job.run_after <= now, |
||||
|
) |
||||
|
.order_by(Job.run_after.asc(), Job.created_at.asc()) |
||||
|
.limit(limit) |
||||
|
) |
||||
|
return list(result.scalars().all()) |
||||
|
|
||||
|
async def find_active_by_kind_and_payload(self, kind: str, payload_json: str) -> Job | None: |
||||
|
result = await self.session.execute( |
||||
|
select(Job).where( |
||||
|
Job.kind == kind, |
||||
|
Job.payload_json == payload_json, |
||||
|
Job.status.in_(["queued", "retryable", "running"]), |
||||
|
) |
||||
|
) |
||||
|
return result.scalar_one_or_none() |
||||
|
|
||||
|
async def save(self, job: Job) -> Job: |
||||
|
await self.session.flush() |
||||
|
await self.session.refresh(job) |
||||
|
return job |
||||
@ -0,0 +1,32 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from sqlalchemy import select |
||||
|
from sqlalchemy.ext.asyncio import AsyncSession |
||||
|
|
||||
|
from app.models.entities import PlacementPolicy |
||||
|
|
||||
|
|
||||
|
class PlacementPolicyRepository: |
||||
|
def __init__(self, session: AsyncSession) -> None: |
||||
|
self.session = session |
||||
|
|
||||
|
async def get_by_file_class(self, file_class: str) -> PlacementPolicy | None: |
||||
|
result = await self.session.execute( |
||||
|
select(PlacementPolicy).where(PlacementPolicy.file_class == file_class) |
||||
|
) |
||||
|
return result.scalar_one_or_none() |
||||
|
|
||||
|
async def list_all(self) -> list[PlacementPolicy]: |
||||
|
result = await self.session.execute(select(PlacementPolicy).order_by(PlacementPolicy.file_class.asc())) |
||||
|
return list(result.scalars().all()) |
||||
|
|
||||
|
async def create(self, policy: PlacementPolicy) -> PlacementPolicy: |
||||
|
self.session.add(policy) |
||||
|
await self.session.flush() |
||||
|
await self.session.refresh(policy) |
||||
|
return policy |
||||
|
|
||||
|
async def save(self, policy: PlacementPolicy) -> PlacementPolicy: |
||||
|
await self.session.flush() |
||||
|
await self.session.refresh(policy) |
||||
|
return policy |
||||
@ -0,0 +1,39 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from sqlalchemy import select |
||||
|
from sqlalchemy.ext.asyncio import AsyncSession |
||||
|
|
||||
|
from app.models.entities import PreviewArtifact |
||||
|
|
||||
|
|
||||
|
class PreviewArtifactRepository: |
||||
|
def __init__(self, session: AsyncSession) -> None: |
||||
|
self.session = session |
||||
|
|
||||
|
async def get_by_version_and_type(self, file_version_id: str, artifact_type: str) -> PreviewArtifact | None: |
||||
|
result = await self.session.execute( |
||||
|
select(PreviewArtifact).where( |
||||
|
PreviewArtifact.file_version_id == file_version_id, |
||||
|
PreviewArtifact.artifact_type == artifact_type, |
||||
|
) |
||||
|
) |
||||
|
return result.scalar_one_or_none() |
||||
|
|
||||
|
async def list_by_file_version(self, file_version_id: str) -> list[PreviewArtifact]: |
||||
|
result = await self.session.execute( |
||||
|
select(PreviewArtifact) |
||||
|
.where(PreviewArtifact.file_version_id == file_version_id) |
||||
|
.order_by(PreviewArtifact.artifact_type.asc()) |
||||
|
) |
||||
|
return list(result.scalars().all()) |
||||
|
|
||||
|
async def create(self, artifact: PreviewArtifact) -> PreviewArtifact: |
||||
|
self.session.add(artifact) |
||||
|
await self.session.flush() |
||||
|
await self.session.refresh(artifact) |
||||
|
return artifact |
||||
|
|
||||
|
async def save(self, artifact: PreviewArtifact) -> PreviewArtifact: |
||||
|
await self.session.flush() |
||||
|
await self.session.refresh(artifact) |
||||
|
return artifact |
||||
@ -0,0 +1,25 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from sqlalchemy import select |
||||
|
from sqlalchemy.ext.asyncio import AsyncSession |
||||
|
|
||||
|
from app.models.entities import UploadSession |
||||
|
|
||||
|
|
||||
|
class UploadRepository: |
||||
|
def __init__(self, session: AsyncSession) -> None: |
||||
|
self.session = session |
||||
|
|
||||
|
async def get_by_id(self, upload_id: str) -> UploadSession | None: |
||||
|
return await self.session.get(UploadSession, upload_id) |
||||
|
|
||||
|
async def create(self, upload_session: UploadSession) -> UploadSession: |
||||
|
self.session.add(upload_session) |
||||
|
await self.session.flush() |
||||
|
await self.session.refresh(upload_session) |
||||
|
return upload_session |
||||
|
|
||||
|
async def save(self, upload_session: UploadSession) -> UploadSession: |
||||
|
await self.session.flush() |
||||
|
await self.session.refresh(upload_session) |
||||
|
return upload_session |
||||
@ -0,0 +1,30 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from sqlalchemy import func |
||||
|
from sqlalchemy import select |
||||
|
from sqlalchemy.ext.asyncio import AsyncSession |
||||
|
|
||||
|
from app.models.entities import User |
||||
|
|
||||
|
|
||||
|
class UserRepository: |
||||
|
def __init__(self, session: AsyncSession) -> None: |
||||
|
self.session = session |
||||
|
|
||||
|
async def count_users(self) -> int: |
||||
|
result = await self.session.execute(select(func.count()).select_from(User)) |
||||
|
return int(result.scalar_one()) |
||||
|
|
||||
|
async def get_by_username(self, username: str) -> User | None: |
||||
|
result = await self.session.execute(select(User).where(User.username == username)) |
||||
|
return result.scalar_one_or_none() |
||||
|
|
||||
|
async def get_by_id(self, user_id: str) -> User | None: |
||||
|
result = await self.session.execute(select(User).where(User.id == user_id)) |
||||
|
return result.scalar_one_or_none() |
||||
|
|
||||
|
async def create(self, user: User) -> User: |
||||
|
self.session.add(user) |
||||
|
await self.session.commit() |
||||
|
await self.session.refresh(user) |
||||
|
return user |
||||
@ -0,0 +1,2 @@ |
|||||
|
"""Pydantic schemas.""" |
||||
|
|
||||
@ -0,0 +1,34 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from datetime import datetime |
||||
|
|
||||
|
from pydantic import BaseModel, ConfigDict |
||||
|
|
||||
|
|
||||
|
class UserSummary(BaseModel): |
||||
|
model_config = ConfigDict(from_attributes=True) |
||||
|
|
||||
|
id: str |
||||
|
username: str |
||||
|
created_at: datetime |
||||
|
updated_at: datetime |
||||
|
|
||||
|
|
||||
|
class LoginRequest(BaseModel): |
||||
|
username: str |
||||
|
password: str |
||||
|
|
||||
|
|
||||
|
class LoginResponse(BaseModel): |
||||
|
access_token: str |
||||
|
token_type: str = "bearer" |
||||
|
expires_at: datetime |
||||
|
user: UserSummary |
||||
|
|
||||
|
|
||||
|
class AuthMeResponse(BaseModel): |
||||
|
user: UserSummary |
||||
|
|
||||
|
|
||||
|
class LogoutResponse(BaseModel): |
||||
|
success: bool = True |
||||
@ -0,0 +1,56 @@ |
|||||
|
from datetime import datetime |
||||
|
|
||||
|
from pydantic import BaseModel, Field |
||||
|
|
||||
|
|
||||
|
class BackendSummary(BaseModel): |
||||
|
id: str |
||||
|
name: str |
||||
|
type: str |
||||
|
stability_class: str |
||||
|
read_priority: int |
||||
|
write_priority: int |
||||
|
is_enabled: bool |
||||
|
last_health_status: str | None |
||||
|
last_health_checked_at: datetime | None |
||||
|
config: dict | None = None |
||||
|
secret_fields: list[str] = Field(default_factory=list) |
||||
|
created_at: datetime |
||||
|
updated_at: datetime |
||||
|
|
||||
|
|
||||
|
class BackendListResponse(BaseModel): |
||||
|
items: list[BackendSummary] |
||||
|
|
||||
|
|
||||
|
class CreateBackendRequest(BaseModel): |
||||
|
name: str |
||||
|
type: str |
||||
|
stability_class: str |
||||
|
read_priority: int |
||||
|
write_priority: int |
||||
|
config: dict |
||||
|
|
||||
|
|
||||
|
class UpdateBackendRequest(BaseModel): |
||||
|
name: str | None = None |
||||
|
stability_class: str | None = None |
||||
|
read_priority: int | None = None |
||||
|
write_priority: int | None = None |
||||
|
config: dict | None = None |
||||
|
|
||||
|
|
||||
|
class CreateBackendResponse(BaseModel): |
||||
|
backend: BackendSummary |
||||
|
|
||||
|
|
||||
|
class BackendCheckResponse(BaseModel): |
||||
|
backend_id: str |
||||
|
status: str |
||||
|
checked_at: datetime | None |
||||
|
detail: str | None = None |
||||
|
|
||||
|
|
||||
|
class BackendMutationResponse(BaseModel): |
||||
|
ok: bool = True |
||||
|
backend: BackendSummary |
||||
@ -0,0 +1,13 @@ |
|||||
|
from typing import Any |
||||
|
|
||||
|
from pydantic import BaseModel |
||||
|
|
||||
|
|
||||
|
class ErrorDetail(BaseModel): |
||||
|
code: str |
||||
|
message: str |
||||
|
details: Any = None |
||||
|
|
||||
|
|
||||
|
class ErrorResponse(BaseModel): |
||||
|
error: ErrorDetail |
||||
@ -0,0 +1,46 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from datetime import datetime |
||||
|
|
||||
|
from pydantic import BaseModel, Field |
||||
|
|
||||
|
|
||||
|
class DirectorySummary(BaseModel): |
||||
|
id: str |
||||
|
name: str |
||||
|
parent_id: str | None |
||||
|
created_at: datetime |
||||
|
updated_at: datetime |
||||
|
|
||||
|
|
||||
|
class DirectoryItem(BaseModel): |
||||
|
kind: str = Field(default="directory") |
||||
|
id: str |
||||
|
name: str |
||||
|
mime_type: str | None = None |
||||
|
size_bytes: int | None = None |
||||
|
created_at: datetime |
||||
|
updated_at: datetime |
||||
|
|
||||
|
|
||||
|
class DirectoryChildrenResponse(BaseModel): |
||||
|
directory: DirectorySummary |
||||
|
items: list[DirectoryItem] |
||||
|
next_cursor: str | None = None |
||||
|
|
||||
|
|
||||
|
class CreateDirectoryRequest(BaseModel): |
||||
|
parent_id: str |
||||
|
name: str |
||||
|
|
||||
|
|
||||
|
class CreateDirectoryResponse(BaseModel): |
||||
|
directory: DirectorySummary |
||||
|
|
||||
|
|
||||
|
class RenameDirectoryRequest(BaseModel): |
||||
|
name: str |
||||
|
|
||||
|
|
||||
|
class MoveDirectoryRequest(BaseModel): |
||||
|
target_parent_id: str |
||||
@ -0,0 +1,32 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from pydantic import BaseModel |
||||
|
|
||||
|
|
||||
|
class MetadataExportResponse(BaseModel): |
||||
|
exported_at: str |
||||
|
data: dict |
||||
|
|
||||
|
|
||||
|
class MetadataImportRequest(BaseModel): |
||||
|
data: dict |
||||
|
confirm_replace: bool = False |
||||
|
validation_token: str | None = None |
||||
|
|
||||
|
|
||||
|
class MetadataImportResponse(BaseModel): |
||||
|
ok: bool = True |
||||
|
imported_tables: list[str] |
||||
|
scheduled_jobs: list[str] = [] |
||||
|
|
||||
|
|
||||
|
class MetadataValidationResponse(BaseModel): |
||||
|
ok: bool |
||||
|
issues: list[str] |
||||
|
|
||||
|
|
||||
|
class MetadataRestorePlanResponse(BaseModel): |
||||
|
ok: bool |
||||
|
issues: list[str] |
||||
|
validation_token: str | None |
||||
|
summary: dict |
||||
@ -0,0 +1,84 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from datetime import datetime |
||||
|
|
||||
|
from pydantic import BaseModel |
||||
|
|
||||
|
|
||||
|
class FileSummary(BaseModel): |
||||
|
id: str |
||||
|
directory_id: str |
||||
|
name: str |
||||
|
mime_type: str | None |
||||
|
size_bytes: int |
||||
|
current_version_id: str | None |
||||
|
created_at: datetime |
||||
|
updated_at: datetime |
||||
|
|
||||
|
|
||||
|
class PreviewArtifactSummary(BaseModel): |
||||
|
id: str |
||||
|
artifact_type: str |
||||
|
blob_id: str |
||||
|
status: str |
||||
|
created_at: datetime |
||||
|
updated_at: datetime |
||||
|
|
||||
|
|
||||
|
class FileReplicaSummary(BaseModel): |
||||
|
id: str |
||||
|
blob_id: str |
||||
|
backend_id: str |
||||
|
backend_name: str |
||||
|
backend_type: str |
||||
|
backend_stability_class: str |
||||
|
backend_enabled: bool |
||||
|
storage_key: str |
||||
|
status: str |
||||
|
size_bytes: int |
||||
|
checksum: str | None |
||||
|
etag: str | None |
||||
|
last_verified_at: datetime | None |
||||
|
created_at: datetime |
||||
|
updated_at: datetime |
||||
|
|
||||
|
|
||||
|
class FilePlacementBackendSummary(BaseModel): |
||||
|
id: str |
||||
|
name: str |
||||
|
type: str |
||||
|
stability_class: str |
||||
|
read_priority: int |
||||
|
write_priority: int |
||||
|
is_enabled: bool |
||||
|
|
||||
|
|
||||
|
class FileDetailResponse(BaseModel): |
||||
|
file: FileSummary |
||||
|
preview_artifacts: list[PreviewArtifactSummary] = [] |
||||
|
replicas: list[FileReplicaSummary] = [] |
||||
|
placement_file_class: str | None = None |
||||
|
desired_backends: list[FilePlacementBackendSummary] = [] |
||||
|
ready_replica_backend_ids: list[str] = [] |
||||
|
missing_backend_ids: list[str] = [] |
||||
|
|
||||
|
|
||||
|
class DeletedFileSummary(FileSummary): |
||||
|
deleted_at: datetime |
||||
|
|
||||
|
|
||||
|
class RecycleBinResponse(BaseModel): |
||||
|
items: list[DeletedFileSummary] |
||||
|
|
||||
|
|
||||
|
class FileMutationResponse(BaseModel): |
||||
|
ok: bool = True |
||||
|
file: FileSummary | DeletedFileSummary |
||||
|
|
||||
|
|
||||
|
class RenameFileRequest(BaseModel): |
||||
|
name: str |
||||
|
|
||||
|
|
||||
|
class MoveFileRequest(BaseModel): |
||||
|
target_parent_id: str |
||||
@ -0,0 +1,37 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from datetime import datetime |
||||
|
|
||||
|
from pydantic import BaseModel |
||||
|
|
||||
|
|
||||
|
class JobSummary(BaseModel): |
||||
|
id: str |
||||
|
kind: str |
||||
|
status: str |
||||
|
attempt_count: int |
||||
|
max_attempts: int |
||||
|
run_after: datetime |
||||
|
last_error: str | None |
||||
|
created_at: datetime |
||||
|
updated_at: datetime |
||||
|
|
||||
|
|
||||
|
class JobListResponse(BaseModel): |
||||
|
items: list[JobSummary] |
||||
|
|
||||
|
|
||||
|
class JobDetailResponse(BaseModel): |
||||
|
job: JobSummary |
||||
|
|
||||
|
|
||||
|
class JobMutationResponse(BaseModel): |
||||
|
ok: bool = True |
||||
|
job: JobSummary |
||||
|
|
||||
|
|
||||
|
class JobRunPendingResponse(BaseModel): |
||||
|
ok: bool = True |
||||
|
processed: int |
||||
|
completed: int |
||||
|
failed: int |
||||
@ -0,0 +1,56 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from datetime import datetime |
||||
|
|
||||
|
from pydantic import BaseModel |
||||
|
from pydantic import Field |
||||
|
|
||||
|
|
||||
|
class PlacementPolicySummary(BaseModel): |
||||
|
id: str |
||||
|
file_class: str |
||||
|
require_local: bool |
||||
|
stable_replica_count: int |
||||
|
opportunistic_replica_count: int |
||||
|
preferred_backend_ids: list[str] |
||||
|
excluded_backend_ids: list[str] |
||||
|
max_non_local_size_bytes: int | None |
||||
|
created_at: datetime |
||||
|
updated_at: datetime |
||||
|
|
||||
|
|
||||
|
class PlacementPolicyListResponse(BaseModel): |
||||
|
items: list[PlacementPolicySummary] |
||||
|
|
||||
|
|
||||
|
class UpsertPlacementPolicyRequest(BaseModel): |
||||
|
require_local: bool |
||||
|
stable_replica_count: int |
||||
|
opportunistic_replica_count: int |
||||
|
preferred_backend_ids: list[str] = Field(default_factory=list) |
||||
|
excluded_backend_ids: list[str] = Field(default_factory=list) |
||||
|
max_non_local_size_bytes: int | None = None |
||||
|
|
||||
|
|
||||
|
class PlacementPolicyMutationResponse(BaseModel): |
||||
|
ok: bool = True |
||||
|
policy: PlacementPolicySummary |
||||
|
reconcile_enqueued_jobs: int = 0 |
||||
|
|
||||
|
|
||||
|
class PlacementDecisionBackendSummary(BaseModel): |
||||
|
id: str |
||||
|
name: str |
||||
|
type: str |
||||
|
stability_class: str |
||||
|
read_priority: int |
||||
|
write_priority: int |
||||
|
|
||||
|
|
||||
|
class PlacementDecisionResponse(BaseModel): |
||||
|
file_class: str |
||||
|
mime_type: str | None |
||||
|
size_bytes: int | None |
||||
|
non_local_allowed: bool |
||||
|
policy: PlacementPolicySummary |
||||
|
selected_backends: list[PlacementDecisionBackendSummary] |
||||
@ -0,0 +1,15 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from pydantic import BaseModel |
||||
|
|
||||
|
|
||||
|
class RepairFileResponse(BaseModel): |
||||
|
ok: bool = True |
||||
|
file_id: str |
||||
|
enqueued_jobs: int |
||||
|
reconciliation_job_kind: str = "reconcile_file_replicas" |
||||
|
|
||||
|
|
||||
|
class EnqueueHealthChecksResponse(BaseModel): |
||||
|
ok: bool = True |
||||
|
enqueued_jobs: int |
||||
@ -0,0 +1,11 @@ |
|||||
|
from pydantic import BaseModel |
||||
|
|
||||
|
|
||||
|
class HealthResponse(BaseModel): |
||||
|
status: str |
||||
|
|
||||
|
|
||||
|
class ReadyResponse(BaseModel): |
||||
|
status: str |
||||
|
database: str |
||||
|
|
||||
@ -0,0 +1,20 @@ |
|||||
|
from pydantic import BaseModel |
||||
|
|
||||
|
from app.schemas.file import FileSummary |
||||
|
|
||||
|
|
||||
|
class FinalizeUploadRequest(BaseModel): |
||||
|
directory_id: str |
||||
|
filename: str |
||||
|
|
||||
|
|
||||
|
class UploadSummary(BaseModel): |
||||
|
id: str |
||||
|
status: str |
||||
|
received_size_bytes: int |
||||
|
total_size_bytes: int |
||||
|
|
||||
|
|
||||
|
class FinalizeUploadResponse(BaseModel): |
||||
|
upload: UploadSummary |
||||
|
file: FileSummary |
||||
@ -0,0 +1,2 @@ |
|||||
|
"""Application services.""" |
||||
|
|
||||
@ -0,0 +1,106 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from dataclasses import dataclass |
||||
|
from datetime import UTC, datetime, timedelta |
||||
|
|
||||
|
from sqlalchemy.ext.asyncio import AsyncSession |
||||
|
|
||||
|
from app.core.config import get_settings |
||||
|
from app.core.exceptions import AuthenticationFailedError |
||||
|
from app.core.exceptions import AuthSessionExpiredError |
||||
|
from app.core.exceptions import UnauthorizedError |
||||
|
from app.core.ids import new_id |
||||
|
from app.core.security import generate_session_token |
||||
|
from app.core.security import hash_password |
||||
|
from app.core.security import hash_session_token |
||||
|
from app.core.security import verify_password |
||||
|
from app.models.entities import AuthSession |
||||
|
from app.models.entities import User |
||||
|
from app.repositories.auth_session_repository import AuthSessionRepository |
||||
|
from app.repositories.user_repository import UserRepository |
||||
|
|
||||
|
|
||||
|
@dataclass |
||||
|
class LoginResult: |
||||
|
user: User |
||||
|
access_token: str |
||||
|
expires_at: datetime |
||||
|
|
||||
|
|
||||
|
class AuthService: |
||||
|
def __init__(self, session: AsyncSession) -> None: |
||||
|
self.session = session |
||||
|
self.user_repository = UserRepository(session) |
||||
|
self.auth_session_repository = AuthSessionRepository(session) |
||||
|
self.settings = get_settings() |
||||
|
|
||||
|
async def ensure_bootstrap_user(self) -> User | None: |
||||
|
if not self.settings.bootstrap_username or not self.settings.bootstrap_password: |
||||
|
return None |
||||
|
|
||||
|
if await self.user_repository.count_users() > 0: |
||||
|
return await self.user_repository.get_by_username(self.settings.bootstrap_username) |
||||
|
|
||||
|
user = User( |
||||
|
id=new_id("user"), |
||||
|
username=self.settings.bootstrap_username, |
||||
|
password_hash=hash_password(self.settings.bootstrap_password), |
||||
|
) |
||||
|
return await self.user_repository.create(user) |
||||
|
|
||||
|
async def login(self, username: str, password: str) -> LoginResult: |
||||
|
user = await self.user_repository.get_by_username(username) |
||||
|
if user is None or not verify_password(password, user.password_hash): |
||||
|
raise AuthenticationFailedError() |
||||
|
|
||||
|
expires_at = self._utc_now_naive() + timedelta(hours=self.settings.auth_session_ttl_hours) |
||||
|
access_token = generate_session_token() |
||||
|
auth_session = AuthSession( |
||||
|
id=new_id("sess"), |
||||
|
user_id=user.id, |
||||
|
token_hash=hash_session_token(access_token), |
||||
|
expires_at=expires_at, |
||||
|
revoked_at=None, |
||||
|
) |
||||
|
await self.auth_session_repository.create(auth_session) |
||||
|
return LoginResult(user=user, access_token=access_token, expires_at=expires_at) |
||||
|
|
||||
|
async def get_current_user(self, bearer_token: str | None) -> User: |
||||
|
token = self._extract_bearer_token(bearer_token) |
||||
|
now = self._utc_now_naive() |
||||
|
auth_session = await self.auth_session_repository.get_active_by_token_hash( |
||||
|
hash_session_token(token), |
||||
|
now, |
||||
|
) |
||||
|
if auth_session is None: |
||||
|
existing_session = await self.auth_session_repository.get_by_token_hash(hash_session_token(token)) |
||||
|
if existing_session is not None and existing_session.expires_at <= now: |
||||
|
raise AuthSessionExpiredError() |
||||
|
raise UnauthorizedError() |
||||
|
|
||||
|
user = await self.user_repository.get_by_id(auth_session.user_id) |
||||
|
if user is None: |
||||
|
raise UnauthorizedError() |
||||
|
return user |
||||
|
|
||||
|
async def logout(self, bearer_token: str | None) -> None: |
||||
|
token = self._extract_bearer_token(bearer_token) |
||||
|
auth_session = await self.auth_session_repository.get_by_token_hash(hash_session_token(token)) |
||||
|
if auth_session is None or auth_session.revoked_at is not None: |
||||
|
raise UnauthorizedError() |
||||
|
|
||||
|
auth_session.revoked_at = self._utc_now_naive() |
||||
|
await self.auth_session_repository.save(auth_session) |
||||
|
|
||||
|
@staticmethod |
||||
|
def _extract_bearer_token(bearer_token: str | None) -> str: |
||||
|
if bearer_token is None: |
||||
|
raise UnauthorizedError() |
||||
|
scheme, _, token = bearer_token.partition(" ") |
||||
|
if scheme.lower() != "bearer" or not token: |
||||
|
raise UnauthorizedError() |
||||
|
return token |
||||
|
|
||||
|
@staticmethod |
||||
|
def _utc_now_naive() -> datetime: |
||||
|
return datetime.now(UTC).replace(tzinfo=None) |
||||
@ -0,0 +1,212 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
import json |
||||
|
from datetime import UTC, datetime |
||||
|
|
||||
|
from sqlalchemy.ext.asyncio import AsyncSession |
||||
|
|
||||
|
from app.core.backend_config import ( |
||||
|
SECRET_CONFIG_KEYS, |
||||
|
BackendConfigError, |
||||
|
merge_backend_config, |
||||
|
split_backend_config, |
||||
|
supported_backend_types, |
||||
|
validate_backend_config, |
||||
|
) |
||||
|
from app.core.exceptions import ConflictError, NotFoundError |
||||
|
from app.core.ids import new_id |
||||
|
from app.core.secret_codec import decrypt_secret_config, encrypt_secret_config |
||||
|
from app.models.entities import Backend |
||||
|
from app.repositories.backend_repository import BackendRepository |
||||
|
from app.repositories.blob_replica_repository import BlobReplicaRepository |
||||
|
from app.services.local_storage_service import DEFAULT_LOCAL_BACKEND_ID |
||||
|
from app.services.storage_service import StorageService |
||||
|
|
||||
|
|
||||
|
class BackendNotFoundError(NotFoundError): |
||||
|
code = "backend_not_found" |
||||
|
message = "Backend does not exist." |
||||
|
|
||||
|
|
||||
|
class BackendDeleteConflictError(ConflictError): |
||||
|
code = "backend_delete_conflict" |
||||
|
message = "Backend cannot be deleted." |
||||
|
|
||||
|
|
||||
|
class BackendDisableConflictError(ConflictError): |
||||
|
code = "backend_disable_conflict" |
||||
|
message = "Backend cannot be disabled." |
||||
|
|
||||
|
|
||||
|
SUPPORTED_BACKEND_TYPES = supported_backend_types() |
||||
|
SUPPORTED_STABILITY_CLASSES = {"local", "stable", "opportunistic"} |
||||
|
|
||||
|
|
||||
|
class BackendService: |
||||
|
def __init__(self, session: AsyncSession) -> None: |
||||
|
self.session = session |
||||
|
self.repository = BackendRepository(session) |
||||
|
self.blob_replica_repository = BlobReplicaRepository(session) |
||||
|
self.storage_service = StorageService(session) |
||||
|
|
||||
|
async def list_backends(self) -> list[Backend]: |
||||
|
return await self.repository.list_all() |
||||
|
|
||||
|
async def create_backend( |
||||
|
self, |
||||
|
*, |
||||
|
name: str, |
||||
|
backend_type: str, |
||||
|
stability_class: str, |
||||
|
read_priority: int, |
||||
|
write_priority: int, |
||||
|
config: dict, |
||||
|
) -> Backend: |
||||
|
cleaned_name = name.strip() |
||||
|
if not cleaned_name: |
||||
|
raise BackendConfigError("Backend name must not be empty.") |
||||
|
if backend_type not in SUPPORTED_BACKEND_TYPES: |
||||
|
raise BackendConfigError(f"Unsupported backend type: {backend_type}.") |
||||
|
if stability_class not in SUPPORTED_STABILITY_CLASSES: |
||||
|
raise BackendConfigError(f"Unsupported stability class: {stability_class}.") |
||||
|
if await self.repository.get_by_name(cleaned_name) is not None: |
||||
|
raise ConflictError("A backend with this name already exists.") |
||||
|
|
||||
|
public_config, secret_config = self._validate_backend_config(backend_type, config) |
||||
|
backend = Backend( |
||||
|
id=new_id("bkd"), |
||||
|
name=cleaned_name, |
||||
|
type=backend_type, |
||||
|
stability_class=stability_class, |
||||
|
read_priority=read_priority, |
||||
|
write_priority=write_priority, |
||||
|
is_enabled=True, |
||||
|
public_config_json=json.dumps(public_config), |
||||
|
secret_config_json=encrypt_secret_config(secret_config), |
||||
|
) |
||||
|
created = await self.repository.create(backend) |
||||
|
await self.session.commit() |
||||
|
return created |
||||
|
|
||||
|
async def check_backend(self, backend_id: str) -> tuple[Backend, str, str | None]: |
||||
|
backend = await self._get_backend(backend_id) |
||||
|
try: |
||||
|
status, detail = await self.storage_service.check_backend(backend) |
||||
|
except Exception as exc: |
||||
|
backend.last_health_status = "unhealthy" |
||||
|
backend.last_health_checked_at = datetime.now(UTC) |
||||
|
await self.repository.save(backend) |
||||
|
await self.session.commit() |
||||
|
return backend, "unhealthy", str(exc) |
||||
|
|
||||
|
backend.last_health_status = status |
||||
|
backend.last_health_checked_at = datetime.now(UTC) |
||||
|
await self.repository.save(backend) |
||||
|
await self.session.commit() |
||||
|
return backend, status, detail |
||||
|
|
||||
|
async def disable_backend(self, backend_id: str) -> Backend: |
||||
|
backend = await self._get_backend(backend_id) |
||||
|
if backend.type == "local_directory": |
||||
|
enabled_local_backends = await self.repository.list_enabled_by_type("local_directory") |
||||
|
if backend.is_enabled and len(enabled_local_backends) <= 1: |
||||
|
raise BackendDisableConflictError("At least one enabled local_directory backend is required.") |
||||
|
backend.is_enabled = False |
||||
|
saved = await self.repository.save(backend) |
||||
|
await self.session.commit() |
||||
|
return saved |
||||
|
|
||||
|
async def enable_backend(self, backend_id: str) -> Backend: |
||||
|
backend = await self._get_backend(backend_id) |
||||
|
backend.is_enabled = True |
||||
|
saved = await self.repository.save(backend) |
||||
|
await self.session.commit() |
||||
|
return saved |
||||
|
|
||||
|
async def delete_backend(self, backend_id: str) -> None: |
||||
|
backend = await self._get_backend(backend_id) |
||||
|
if backend.id == DEFAULT_LOCAL_BACKEND_ID: |
||||
|
raise BackendDeleteConflictError("The default local backend cannot be deleted.") |
||||
|
if backend.is_enabled: |
||||
|
raise BackendDeleteConflictError("Disable the backend before deleting it.") |
||||
|
if await self.repository.is_referenced_by_policy(backend.id): |
||||
|
raise BackendDeleteConflictError("Backend is still referenced by a placement policy.") |
||||
|
replica_count = await self.blob_replica_repository.count_by_backend(backend.id) |
||||
|
if replica_count > 0: |
||||
|
raise BackendDeleteConflictError("Backend still has stored blob replicas.") |
||||
|
await self.repository.delete(backend) |
||||
|
await self.session.commit() |
||||
|
|
||||
|
async def update_backend( |
||||
|
self, |
||||
|
backend_id: str, |
||||
|
*, |
||||
|
name: str | None = None, |
||||
|
stability_class: str | None = None, |
||||
|
read_priority: int | None = None, |
||||
|
write_priority: int | None = None, |
||||
|
config: dict | None = None, |
||||
|
) -> Backend: |
||||
|
backend = await self._get_backend(backend_id) |
||||
|
|
||||
|
if name is not None: |
||||
|
cleaned_name = name.strip() |
||||
|
if not cleaned_name: |
||||
|
raise BackendConfigError("Backend name must not be empty.") |
||||
|
existing = await self.repository.get_by_name(cleaned_name) |
||||
|
if existing is not None and existing.id != backend.id: |
||||
|
raise ConflictError("A backend with this name already exists.") |
||||
|
backend.name = cleaned_name |
||||
|
|
||||
|
if stability_class is not None: |
||||
|
if stability_class not in SUPPORTED_STABILITY_CLASSES: |
||||
|
raise BackendConfigError(f"Unsupported stability class: {stability_class}.") |
||||
|
backend.stability_class = stability_class |
||||
|
|
||||
|
if read_priority is not None: |
||||
|
backend.read_priority = read_priority |
||||
|
if write_priority is not None: |
||||
|
backend.write_priority = write_priority |
||||
|
|
||||
|
if config is not None: |
||||
|
public_config = json.loads(backend.public_config_json) |
||||
|
secret_config = decrypt_secret_config(backend.secret_config_json) |
||||
|
next_config = self._merge_config_update( |
||||
|
backend.type, |
||||
|
public_config=public_config, |
||||
|
secret_config=secret_config, |
||||
|
config_patch=config, |
||||
|
) |
||||
|
next_public_config, next_secret_config = self._validate_backend_config(backend.type, next_config) |
||||
|
backend.public_config_json = json.dumps(next_public_config) |
||||
|
backend.secret_config_json = encrypt_secret_config(next_secret_config) |
||||
|
|
||||
|
saved = await self.repository.save(backend) |
||||
|
await self.session.commit() |
||||
|
return saved |
||||
|
|
||||
|
async def _get_backend(self, backend_id: str) -> Backend: |
||||
|
backend = await self.repository.get_by_id(backend_id) |
||||
|
if backend is None: |
||||
|
raise BackendNotFoundError() |
||||
|
return backend |
||||
|
|
||||
|
@staticmethod |
||||
|
def _validate_backend_config(backend_type: str, config: dict) -> tuple[dict, dict]: |
||||
|
return split_backend_config(backend_type, config) |
||||
|
|
||||
|
@staticmethod |
||||
|
def _merge_config_update( |
||||
|
backend_type: str, |
||||
|
*, |
||||
|
public_config: dict, |
||||
|
secret_config: dict, |
||||
|
config_patch: dict, |
||||
|
) -> dict: |
||||
|
merged = merge_backend_config(public_config, secret_config) |
||||
|
normalized_patch = dict(config_patch) |
||||
|
for key in list(normalized_patch): |
||||
|
if key in SECRET_CONFIG_KEYS and not normalized_patch[key]: |
||||
|
normalized_patch.pop(key) |
||||
|
merged.update(normalized_patch) |
||||
|
return validate_backend_config(backend_type, merged) |
||||
@ -0,0 +1,153 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from dataclasses import dataclass |
||||
|
from typing import Literal |
||||
|
|
||||
|
from sqlalchemy.ext.asyncio import AsyncSession |
||||
|
|
||||
|
from app.core.exceptions import DirectoryConflictError, DirectoryNotFoundError, ValidationError |
||||
|
from app.core.ids import new_id |
||||
|
from app.models.entities import Directory, FileEntry |
||||
|
from app.repositories.directory_repository import DirectoryRepository |
||||
|
from app.repositories.file_repository import FileRepository |
||||
|
|
||||
|
ROOT_DIRECTORY_ID = "dir_root" |
||||
|
ROOT_DIRECTORY_NAME = "/" |
||||
|
ROOT_PATH_KEY = "/" |
||||
|
|
||||
|
|
||||
|
@dataclass |
||||
|
class NamespaceItem: |
||||
|
kind: Literal["directory", "file"] |
||||
|
item: Directory | FileEntry |
||||
|
|
||||
|
|
||||
|
@dataclass |
||||
|
class DirectoryChildrenResult: |
||||
|
directory: Directory |
||||
|
children: list[NamespaceItem] |
||||
|
|
||||
|
|
||||
|
class DirectoryService: |
||||
|
def __init__(self, session: AsyncSession) -> None: |
||||
|
self.session = session |
||||
|
self.repository = DirectoryRepository(session) |
||||
|
self.file_repository = FileRepository(session) |
||||
|
|
||||
|
async def ensure_root_directory(self) -> Directory: |
||||
|
root = await self.repository.get_root() |
||||
|
if root is not None: |
||||
|
return root |
||||
|
|
||||
|
root = Directory( |
||||
|
id=ROOT_DIRECTORY_ID, |
||||
|
parent_id=None, |
||||
|
name=ROOT_DIRECTORY_NAME, |
||||
|
path_key=ROOT_PATH_KEY, |
||||
|
) |
||||
|
await self.repository.create(root) |
||||
|
await self.session.commit() |
||||
|
return root |
||||
|
|
||||
|
async def list_children(self, directory_id: str) -> DirectoryChildrenResult: |
||||
|
directory = await self._get_directory(directory_id) |
||||
|
child_directories = await self.repository.list_children(directory.id) |
||||
|
child_files = await self.file_repository.list_by_directory(directory.id) |
||||
|
|
||||
|
items = [NamespaceItem(kind="directory", item=child) for child in child_directories] |
||||
|
items.extend(NamespaceItem(kind="file", item=file_entry) for file_entry in child_files) |
||||
|
items.sort(key=lambda item: (item.kind != "directory", item.item.name.lower())) |
||||
|
return DirectoryChildrenResult(directory=directory, children=items) |
||||
|
|
||||
|
async def create_directory(self, parent_id: str, name: str) -> Directory: |
||||
|
cleaned_name = self._validate_directory_name(name) |
||||
|
|
||||
|
parent = await self._get_directory(parent_id) |
||||
|
await self._ensure_name_available(parent.id, cleaned_name) |
||||
|
|
||||
|
directory = Directory( |
||||
|
id=new_id("dir"), |
||||
|
parent_id=parent.id, |
||||
|
name=cleaned_name, |
||||
|
path_key=self._build_path_key(parent.path_key, cleaned_name), |
||||
|
) |
||||
|
await self.repository.create(directory) |
||||
|
await self.session.commit() |
||||
|
return directory |
||||
|
|
||||
|
async def rename_directory(self, directory_id: str, name: str) -> Directory: |
||||
|
directory = await self._get_directory(directory_id) |
||||
|
if directory.id == ROOT_DIRECTORY_ID: |
||||
|
raise ValidationError("Root directory cannot be renamed.") |
||||
|
|
||||
|
cleaned_name = self._validate_directory_name(name) |
||||
|
parent = await self._get_directory(directory.parent_id or ROOT_DIRECTORY_ID) |
||||
|
if directory.name == cleaned_name: |
||||
|
return directory |
||||
|
|
||||
|
await self._ensure_name_available(parent.id, cleaned_name) |
||||
|
|
||||
|
old_path_key = directory.path_key |
||||
|
directory.name = cleaned_name |
||||
|
directory.path_key = self._build_path_key(parent.path_key, cleaned_name) |
||||
|
await self._rewrite_descendant_paths(old_path_key, directory.path_key) |
||||
|
saved = await self.repository.save(directory) |
||||
|
await self.session.commit() |
||||
|
return saved |
||||
|
|
||||
|
async def move_directory(self, directory_id: str, target_parent_id: str) -> Directory: |
||||
|
directory = await self._get_directory(directory_id) |
||||
|
if directory.id == ROOT_DIRECTORY_ID: |
||||
|
raise ValidationError("Root directory cannot be moved.") |
||||
|
|
||||
|
target_parent = await self._get_directory(target_parent_id) |
||||
|
if directory.parent_id == target_parent.id: |
||||
|
return directory |
||||
|
if target_parent.path_key == directory.path_key or target_parent.path_key.startswith(f"{directory.path_key}/"): |
||||
|
raise ValidationError("A directory cannot be moved into itself or its descendants.") |
||||
|
|
||||
|
await self._ensure_name_available(target_parent.id, directory.name) |
||||
|
|
||||
|
old_path_key = directory.path_key |
||||
|
directory.parent_id = target_parent.id |
||||
|
directory.path_key = self._build_path_key(target_parent.path_key, directory.name) |
||||
|
await self._rewrite_descendant_paths(old_path_key, directory.path_key) |
||||
|
saved = await self.repository.save(directory) |
||||
|
await self.session.commit() |
||||
|
return saved |
||||
|
|
||||
|
async def _get_directory(self, directory_id: str) -> Directory: |
||||
|
directory = await self.repository.get_by_id(directory_id) |
||||
|
if directory is None or directory.deleted_at is not None: |
||||
|
raise DirectoryNotFoundError() |
||||
|
return directory |
||||
|
|
||||
|
async def _ensure_name_available(self, parent_id: str, name: str) -> None: |
||||
|
conflicting_file = await self.file_repository.get_by_directory_and_name(parent_id, name) |
||||
|
if conflicting_file is not None: |
||||
|
raise DirectoryConflictError("A file with this name already exists in the target location.") |
||||
|
|
||||
|
existing = await self.repository.get_by_parent_and_name(parent_id, name) |
||||
|
if existing is not None: |
||||
|
raise DirectoryConflictError("A directory with this name already exists in the target location.") |
||||
|
|
||||
|
async def _rewrite_descendant_paths(self, old_prefix: str, new_prefix: str) -> None: |
||||
|
descendants = await self.repository.list_descendants_by_path_prefix(old_prefix) |
||||
|
for descendant in descendants: |
||||
|
suffix = descendant.path_key.removeprefix(old_prefix) |
||||
|
descendant.path_key = f"{new_prefix}{suffix}" |
||||
|
|
||||
|
@staticmethod |
||||
|
def _validate_directory_name(name: str) -> str: |
||||
|
cleaned_name = name.strip() |
||||
|
if not cleaned_name: |
||||
|
raise ValidationError("Directory name must not be empty.") |
||||
|
if "/" in cleaned_name: |
||||
|
raise ValidationError("Directory name must not contain '/'.") |
||||
|
return cleaned_name |
||||
|
|
||||
|
@staticmethod |
||||
|
def _build_path_key(parent_path_key: str, name: str) -> str: |
||||
|
if parent_path_key == ROOT_PATH_KEY: |
||||
|
return f"/{name}" |
||||
|
return f"{parent_path_key}/{name}" |
||||
@ -0,0 +1,234 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from datetime import UTC, datetime |
||||
|
import hashlib |
||||
|
import json |
||||
|
from typing import Any |
||||
|
|
||||
|
from sqlalchemy import delete |
||||
|
from sqlalchemy import insert |
||||
|
from sqlalchemy import select |
||||
|
from sqlalchemy.ext.asyncio import AsyncSession |
||||
|
|
||||
|
from app.core.exceptions import ValidationError |
||||
|
from app.models.entities import AuthSession |
||||
|
from app.models.entities import Backend |
||||
|
from app.models.entities import Blob |
||||
|
from app.models.entities import BlobReplica |
||||
|
from app.models.entities import CacheEntry |
||||
|
from app.models.entities import Directory |
||||
|
from app.models.entities import FileEntry |
||||
|
from app.models.entities import FileVersion |
||||
|
from app.models.entities import Job |
||||
|
from app.models.entities import PreviewArtifact |
||||
|
from app.models.entities import UploadSession |
||||
|
from app.models.entities import UploadSessionPart |
||||
|
from app.models.entities import User |
||||
|
from app.services.storage_service import StorageService |
||||
|
|
||||
|
|
||||
|
class ExportService: |
||||
|
def __init__(self, session: AsyncSession) -> None: |
||||
|
self.session = session |
||||
|
self.storage_service = StorageService(session) |
||||
|
|
||||
|
async def export_metadata(self) -> dict: |
||||
|
return { |
||||
|
"users": await self._dump(User), |
||||
|
"auth_sessions": await self._dump(AuthSession), |
||||
|
"directories": await self._dump(Directory), |
||||
|
"file_entries": await self._dump(FileEntry), |
||||
|
"file_versions": await self._dump(FileVersion), |
||||
|
"blobs": await self._dump(Blob), |
||||
|
"blob_replicas": await self._dump(BlobReplica), |
||||
|
"backends": await self._dump(Backend), |
||||
|
"upload_sessions": await self._dump(UploadSession), |
||||
|
"upload_session_parts": await self._dump(UploadSessionPart), |
||||
|
"jobs": await self._dump(Job), |
||||
|
"cache_entries": await self._dump(CacheEntry), |
||||
|
"preview_artifacts": await self._dump(PreviewArtifact), |
||||
|
"exported_at": datetime.now(UTC).isoformat(), |
||||
|
} |
||||
|
|
||||
|
async def import_metadata(self, data: dict[str, list[dict[str, Any]]]) -> list[str]: |
||||
|
validation = await self.validate_metadata_snapshot(data) |
||||
|
if not validation["ok"]: |
||||
|
raise ValidationError("Metadata snapshot is invalid and cannot be imported.") |
||||
|
|
||||
|
ordered_models = [ |
||||
|
User, |
||||
|
AuthSession, |
||||
|
Directory, |
||||
|
Backend, |
||||
|
UploadSession, |
||||
|
UploadSessionPart, |
||||
|
FileEntry, |
||||
|
FileVersion, |
||||
|
Blob, |
||||
|
BlobReplica, |
||||
|
PreviewArtifact, |
||||
|
Job, |
||||
|
CacheEntry, |
||||
|
] |
||||
|
for model in reversed(ordered_models): |
||||
|
await self.session.execute(delete(model)) |
||||
|
|
||||
|
imported_tables: list[str] = [] |
||||
|
for model in ordered_models: |
||||
|
rows = data.get(model.__tablename__, []) |
||||
|
if not rows: |
||||
|
continue |
||||
|
normalized_rows = [self._normalize_row(model, row) for row in rows] |
||||
|
await self.session.execute(insert(model), normalized_rows) |
||||
|
imported_tables.append(model.__tablename__) |
||||
|
|
||||
|
await self.session.commit() |
||||
|
return imported_tables |
||||
|
|
||||
|
async def build_restore_plan(self, data: dict[str, list[dict[str, Any]]]) -> dict[str, Any]: |
||||
|
validation = await self.validate_metadata_snapshot(data) |
||||
|
summary = { |
||||
|
"users": len(data.get("users", [])), |
||||
|
"directories": len(data.get("directories", [])), |
||||
|
"file_entries": len(data.get("file_entries", [])), |
||||
|
"file_versions": len(data.get("file_versions", [])), |
||||
|
"blobs": len(data.get("blobs", [])), |
||||
|
"blob_replicas": len(data.get("blob_replicas", [])), |
||||
|
"backends": len(data.get("backends", [])), |
||||
|
"jobs": len(data.get("jobs", [])), |
||||
|
"preview_artifacts": len(data.get("preview_artifacts", [])), |
||||
|
} |
||||
|
validation_token = None |
||||
|
if validation["ok"]: |
||||
|
validation_token = self._build_validation_token(data) |
||||
|
return { |
||||
|
"ok": validation["ok"], |
||||
|
"issues": validation["issues"], |
||||
|
"validation_token": validation_token, |
||||
|
"summary": summary, |
||||
|
} |
||||
|
|
||||
|
async def validate_metadata_snapshot(self, data: dict[str, list[dict[str, Any]]]) -> dict[str, Any]: |
||||
|
issues: list[str] = [] |
||||
|
users = data.get("users", []) |
||||
|
directories = data.get("directories", []) |
||||
|
file_entries = data.get("file_entries", []) |
||||
|
file_versions = data.get("file_versions", []) |
||||
|
blobs = data.get("blobs", []) |
||||
|
blob_replicas = data.get("blob_replicas", []) |
||||
|
backends = data.get("backends", []) |
||||
|
auth_sessions = data.get("auth_sessions", []) |
||||
|
upload_sessions = data.get("upload_sessions", []) |
||||
|
upload_session_parts = data.get("upload_session_parts", []) |
||||
|
jobs = data.get("jobs", []) |
||||
|
cache_entries = data.get("cache_entries", []) |
||||
|
preview_artifacts = data.get("preview_artifacts", []) |
||||
|
|
||||
|
if not users: |
||||
|
issues.append("Snapshot must contain at least one user.") |
||||
|
if not any(directory.get("id") == "dir_root" for directory in directories): |
||||
|
issues.append("Snapshot must contain the root directory dir_root.") |
||||
|
|
||||
|
user_ids = {row["id"] for row in users if row.get("id")} |
||||
|
directory_ids = {row["id"] for row in directories if row.get("id")} |
||||
|
file_entry_ids = {row["id"] for row in file_entries if row.get("id")} |
||||
|
version_ids = {row["id"] for row in file_versions if row.get("id")} |
||||
|
blob_ids = {row["id"] for row in blobs if row.get("id")} |
||||
|
backend_ids = {row["id"] for row in backends if row.get("id")} |
||||
|
upload_ids = {row["id"] for row in upload_sessions if row.get("id")} |
||||
|
|
||||
|
for row in auth_sessions: |
||||
|
if row.get("user_id") not in user_ids: |
||||
|
issues.append(f"Auth session {row.get('id')} references missing user.") |
||||
|
for row in file_entries: |
||||
|
if row.get("directory_id") not in directory_ids: |
||||
|
issues.append(f"File entry {row.get('id')} references missing directory.") |
||||
|
current_version_id = row.get("current_version_id") |
||||
|
if current_version_id is not None and current_version_id not in version_ids: |
||||
|
issues.append(f"File entry {row.get('id')} references missing current version.") |
||||
|
for row in file_versions: |
||||
|
if row.get("file_entry_id") not in file_entry_ids: |
||||
|
issues.append(f"File version {row.get('id')} references missing file entry.") |
||||
|
for row in blobs: |
||||
|
if row.get("file_version_id") not in version_ids: |
||||
|
issues.append(f"Blob {row.get('id')} references missing file version.") |
||||
|
for row in blob_replicas: |
||||
|
if row.get("blob_id") not in blob_ids: |
||||
|
issues.append(f"Blob replica {row.get('id')} references missing blob.") |
||||
|
if row.get("backend_id") not in backend_ids: |
||||
|
issues.append(f"Blob replica {row.get('id')} references missing backend.") |
||||
|
for row in upload_sessions: |
||||
|
directory_id = row.get("directory_id") |
||||
|
if directory_id is not None and directory_id not in directory_ids: |
||||
|
issues.append(f"Upload session {row.get('id')} references missing directory.") |
||||
|
for row in upload_session_parts: |
||||
|
if row.get("upload_session_id") not in upload_ids: |
||||
|
issues.append(f"Upload session part {row.get('id')} references missing upload session.") |
||||
|
for row in preview_artifacts: |
||||
|
if row.get("file_version_id") not in version_ids: |
||||
|
issues.append(f"Preview artifact {row.get('id')} references missing file version.") |
||||
|
if row.get("blob_id") not in blob_ids: |
||||
|
issues.append(f"Preview artifact {row.get('id')} references missing blob.") |
||||
|
for row in cache_entries: |
||||
|
if row.get("blob_id") not in blob_ids: |
||||
|
issues.append(f"Cache entry {row.get('id')} references missing blob.") |
||||
|
|
||||
|
return {"ok": not issues, "issues": issues} |
||||
|
|
||||
|
async def verify_runtime_integrity(self) -> dict[str, Any]: |
||||
|
snapshot = await self.export_metadata() |
||||
|
validation = await self.validate_metadata_snapshot(snapshot) |
||||
|
issues = list(validation["issues"]) |
||||
|
|
||||
|
backends = {row["id"]: row for row in snapshot["backends"]} |
||||
|
for replica in snapshot["blob_replicas"]: |
||||
|
if replica.get("status") != "ready": |
||||
|
continue |
||||
|
backend_row = backends.get(replica["backend_id"]) |
||||
|
if backend_row is None or not backend_row.get("is_enabled", False): |
||||
|
continue |
||||
|
backend = await self.session.get(Backend, replica["backend_id"]) |
||||
|
if backend is None: |
||||
|
continue |
||||
|
try: |
||||
|
adapter = self.storage_service._build_adapter(backend) |
||||
|
exists = await adapter.object_exists(replica["storage_key"]) |
||||
|
except Exception as exc: |
||||
|
issues.append(f"Replica {replica['id']} could not be verified: {exc}") |
||||
|
continue |
||||
|
if not exists: |
||||
|
issues.append(f"Replica {replica['id']} is marked ready but object is missing.") |
||||
|
|
||||
|
return {"ok": not issues, "issues": issues} |
||||
|
|
||||
|
@staticmethod |
||||
|
def _build_validation_token(data: dict[str, list[dict[str, Any]]]) -> str: |
||||
|
canonical = json.dumps(data, sort_keys=True, separators=(",", ":"), ensure_ascii=True) |
||||
|
return hashlib.sha256(canonical.encode("utf-8")).hexdigest() |
||||
|
|
||||
|
async def _dump(self, model) -> list[dict]: |
||||
|
result = await self.session.execute(select(model)) |
||||
|
items = [] |
||||
|
for row in result.scalars().all(): |
||||
|
payload = {} |
||||
|
for column in model.__table__.columns: |
||||
|
value = getattr(row, column.name) |
||||
|
if isinstance(value, datetime): |
||||
|
payload[column.name] = value.isoformat() |
||||
|
else: |
||||
|
payload[column.name] = value |
||||
|
items.append(payload) |
||||
|
return items |
||||
|
|
||||
|
@staticmethod |
||||
|
def _normalize_row(model, row: dict[str, Any]) -> dict[str, Any]: |
||||
|
normalized = {} |
||||
|
for column in model.__table__.columns: |
||||
|
value = row.get(column.name) |
||||
|
if value is not None and isinstance(value, str): |
||||
|
python_type = getattr(column.type, "python_type", None) |
||||
|
if python_type is datetime: |
||||
|
normalized[column.name] = datetime.fromisoformat(value) |
||||
|
continue |
||||
|
normalized[column.name] = value |
||||
|
return normalized |
||||
@ -0,0 +1,263 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from dataclasses import dataclass |
||||
|
from pathlib import Path |
||||
|
from typing import NamedTuple |
||||
|
|
||||
|
import anyio |
||||
|
from sqlalchemy.ext.asyncio import AsyncSession |
||||
|
|
||||
|
from app.core.exceptions import ( |
||||
|
DirectoryConflictError, |
||||
|
DirectoryNotFoundError, |
||||
|
FileConflictError, |
||||
|
FileNotFoundError, |
||||
|
PreviewNotSupportedError, |
||||
|
RangeNotSatisfiableError, |
||||
|
ValidationError, |
||||
|
) |
||||
|
from app.core.ids import new_id |
||||
|
from app.models.entities import Blob, FileEntry, FileVersion |
||||
|
from app.repositories.blob_replica_repository import BlobReplicaRepository |
||||
|
from app.repositories.directory_repository import DirectoryRepository |
||||
|
from app.repositories.file_repository import FileRepository |
||||
|
from app.repositories.preview_artifact_repository import PreviewArtifactRepository |
||||
|
from app.services.local_storage_service import LocalStorageService |
||||
|
from app.services.storage_service import StorageService |
||||
|
|
||||
|
|
||||
|
@dataclass |
||||
|
class CreateFileMetadataInput: |
||||
|
directory_id: str |
||||
|
name: str |
||||
|
mime_type: str | None |
||||
|
size_bytes: int |
||||
|
content_hash: str |
||||
|
storage_layout: str = "single" |
||||
|
|
||||
|
|
||||
|
class FileStreamResult(NamedTuple): |
||||
|
file_entry: FileEntry |
||||
|
file_path: Path |
||||
|
file_size: int |
||||
|
start: int |
||||
|
end: int |
||||
|
|
||||
|
|
||||
|
class FileService: |
||||
|
def __init__(self, session: AsyncSession) -> None: |
||||
|
self.session = session |
||||
|
self.file_repository = FileRepository(session) |
||||
|
self.blob_replica_repository = BlobReplicaRepository(session) |
||||
|
self.directory_repository = DirectoryRepository(session) |
||||
|
self.preview_artifact_repository = PreviewArtifactRepository(session) |
||||
|
self.local_storage_service = LocalStorageService(session) |
||||
|
self.storage_service = StorageService(session) |
||||
|
|
||||
|
async def get_file(self, file_id: str) -> FileEntry: |
||||
|
file_entry = await self.file_repository.get_by_id(file_id) |
||||
|
if file_entry is None or file_entry.is_deleted or file_entry.deleted_at is not None: |
||||
|
raise FileNotFoundError() |
||||
|
return file_entry |
||||
|
|
||||
|
async def get_file_detail(self, file_id: str) -> tuple[FileEntry, list, list, object | None]: |
||||
|
file_entry = await self.get_file(file_id) |
||||
|
if file_entry.current_version_id is None: |
||||
|
return file_entry, [], [], None |
||||
|
artifacts = await self.preview_artifact_repository.list_by_file_version(file_entry.current_version_id) |
||||
|
blob = await self.file_repository.get_primary_blob_for_file(file_entry.id) |
||||
|
if blob is None: |
||||
|
return file_entry, artifacts, [], None |
||||
|
replicas = await self.blob_replica_repository.list_replicas_with_backends(blob.id) |
||||
|
decision = await self.storage_service.get_replication_targets_for_mime_type( |
||||
|
file_entry.mime_type, |
||||
|
size_bytes=blob.size_bytes, |
||||
|
) |
||||
|
return file_entry, artifacts, replicas, decision |
||||
|
|
||||
|
async def list_deleted_files(self) -> list[FileEntry]: |
||||
|
return await self.file_repository.list_deleted() |
||||
|
|
||||
|
async def create_file_metadata(self, payload: CreateFileMetadataInput) -> FileEntry: |
||||
|
directory = await self.directory_repository.get_by_id(payload.directory_id) |
||||
|
if directory is None or directory.deleted_at is not None: |
||||
|
raise DirectoryNotFoundError() |
||||
|
|
||||
|
cleaned_name = self._validate_file_name(payload.name) |
||||
|
await self._ensure_name_available(directory.id, cleaned_name) |
||||
|
|
||||
|
file_entry_id = new_id("fil") |
||||
|
file_version_id = new_id("ver") |
||||
|
blob_id = new_id("blb") |
||||
|
|
||||
|
file_entry = FileEntry( |
||||
|
id=file_entry_id, |
||||
|
directory_id=directory.id, |
||||
|
name=cleaned_name, |
||||
|
mime_type=payload.mime_type, |
||||
|
size_bytes=payload.size_bytes, |
||||
|
current_version_id=file_version_id, |
||||
|
is_deleted=False, |
||||
|
) |
||||
|
file_version = FileVersion( |
||||
|
id=file_version_id, |
||||
|
file_entry_id=file_entry_id, |
||||
|
content_hash=payload.content_hash, |
||||
|
size_bytes=payload.size_bytes, |
||||
|
storage_layout=payload.storage_layout, |
||||
|
) |
||||
|
blob = Blob( |
||||
|
id=blob_id, |
||||
|
file_version_id=file_version_id, |
||||
|
blob_index=0, |
||||
|
kind="file", |
||||
|
content_hash=payload.content_hash, |
||||
|
size_bytes=payload.size_bytes, |
||||
|
logical_offset=0, |
||||
|
) |
||||
|
|
||||
|
created = await self.file_repository.create_file_graph(file_entry, file_version, blob) |
||||
|
await self.session.commit() |
||||
|
return created |
||||
|
|
||||
|
async def rename_file(self, file_id: str, name: str) -> FileEntry: |
||||
|
file_entry = await self.get_file(file_id) |
||||
|
cleaned_name = self._validate_file_name(name) |
||||
|
if file_entry.name == cleaned_name: |
||||
|
return file_entry |
||||
|
|
||||
|
await self._ensure_name_available(file_entry.directory_id, cleaned_name) |
||||
|
file_entry.name = cleaned_name |
||||
|
saved = await self.file_repository.save(file_entry) |
||||
|
await self.session.commit() |
||||
|
return saved |
||||
|
|
||||
|
async def move_file(self, file_id: str, target_parent_id: str) -> FileEntry: |
||||
|
file_entry = await self.get_file(file_id) |
||||
|
target_parent = await self.directory_repository.get_by_id(target_parent_id) |
||||
|
if target_parent is None or target_parent.deleted_at is not None: |
||||
|
raise DirectoryNotFoundError() |
||||
|
if file_entry.directory_id == target_parent.id: |
||||
|
return file_entry |
||||
|
|
||||
|
await self._ensure_name_available(target_parent.id, file_entry.name) |
||||
|
file_entry.directory_id = target_parent.id |
||||
|
saved = await self.file_repository.save(file_entry) |
||||
|
await self.session.commit() |
||||
|
return saved |
||||
|
|
||||
|
async def delete_file(self, file_id: str) -> FileEntry: |
||||
|
file_entry = await self.get_file(file_id) |
||||
|
deleted = await self.file_repository.soft_delete(file_entry) |
||||
|
await self.session.commit() |
||||
|
return deleted |
||||
|
|
||||
|
async def restore_file(self, file_id: str) -> FileEntry: |
||||
|
file_entry = await self.file_repository.get_by_id(file_id) |
||||
|
if file_entry is None or not file_entry.is_deleted or file_entry.deleted_at is None: |
||||
|
raise FileNotFoundError() |
||||
|
|
||||
|
directory = await self.directory_repository.get_by_id(file_entry.directory_id) |
||||
|
if directory is None or directory.deleted_at is not None: |
||||
|
raise DirectoryNotFoundError() |
||||
|
|
||||
|
conflicting_directory = await self.directory_repository.get_by_parent_and_name(directory.id, file_entry.name) |
||||
|
if conflicting_directory is not None: |
||||
|
raise DirectoryConflictError("A directory with this name already exists in the target location.") |
||||
|
|
||||
|
existing_file = await self.file_repository.get_by_directory_and_name(directory.id, file_entry.name) |
||||
|
if existing_file is not None: |
||||
|
raise FileConflictError("A file with this name already exists in the target location.") |
||||
|
|
||||
|
restored = await self.file_repository.restore(file_entry) |
||||
|
await self.session.commit() |
||||
|
return restored |
||||
|
|
||||
|
async def resolve_download(self, file_id: str) -> tuple[FileEntry, Path]: |
||||
|
file_entry = await self.get_file(file_id) |
||||
|
blob = await self.file_repository.get_primary_blob_for_file(file_entry.id) |
||||
|
if blob is None: |
||||
|
raise FileNotFoundError() |
||||
|
try: |
||||
|
file_path = await self.storage_service.resolve_best_local_path(blob) |
||||
|
except Exception as exc: |
||||
|
raise FileNotFoundError() from exc |
||||
|
|
||||
|
if file_path is None: |
||||
|
raise FileNotFoundError() |
||||
|
return file_entry, file_path |
||||
|
|
||||
|
async def resolve_preview(self, file_id: str) -> tuple[FileEntry, Path]: |
||||
|
file_entry, file_path = await self.resolve_download(file_id) |
||||
|
if not self._supports_inline_preview(file_entry.mime_type): |
||||
|
raise PreviewNotSupportedError() |
||||
|
return file_entry, file_path |
||||
|
|
||||
|
async def resolve_stream(self, file_id: str, range_header: str | None) -> FileStreamResult: |
||||
|
file_entry, file_path = await self.resolve_download(file_id) |
||||
|
file_size = await anyio.to_thread.run_sync(lambda: file_path.stat().st_size) |
||||
|
start, end = self._parse_range(range_header, file_size) |
||||
|
return FileStreamResult( |
||||
|
file_entry=file_entry, |
||||
|
file_path=file_path, |
||||
|
file_size=file_size, |
||||
|
start=start, |
||||
|
end=end, |
||||
|
) |
||||
|
|
||||
|
async def _ensure_name_available(self, directory_id: str, name: str) -> None: |
||||
|
conflicting_directory = await self.directory_repository.get_by_parent_and_name(directory_id, name) |
||||
|
if conflicting_directory is not None: |
||||
|
raise DirectoryConflictError("A directory with this name already exists in the target location.") |
||||
|
|
||||
|
existing_file = await self.file_repository.get_by_directory_and_name(directory_id, name) |
||||
|
if existing_file is not None: |
||||
|
raise FileConflictError("A file with this name already exists in the target location.") |
||||
|
|
||||
|
@staticmethod |
||||
|
def _validate_file_name(name: str) -> str: |
||||
|
cleaned_name = name.strip() |
||||
|
if not cleaned_name: |
||||
|
raise ValidationError("File name must not be empty.") |
||||
|
if "/" in cleaned_name: |
||||
|
raise ValidationError("File name must not contain '/'.") |
||||
|
return cleaned_name |
||||
|
|
||||
|
@staticmethod |
||||
|
def _parse_range(range_header: str | None, file_size: int) -> tuple[int, int]: |
||||
|
if file_size <= 0: |
||||
|
return 0, 0 |
||||
|
if not range_header: |
||||
|
return 0, file_size - 1 |
||||
|
if not range_header.startswith("bytes="): |
||||
|
raise RangeNotSatisfiableError() |
||||
|
|
||||
|
value = range_header.removeprefix("bytes=") |
||||
|
if "," in value: |
||||
|
raise RangeNotSatisfiableError() |
||||
|
|
||||
|
start_text, _, end_text = value.partition("-") |
||||
|
try: |
||||
|
if start_text == "": |
||||
|
suffix_length = int(end_text) |
||||
|
if suffix_length <= 0: |
||||
|
raise RangeNotSatisfiableError() |
||||
|
start = max(file_size - suffix_length, 0) |
||||
|
end = file_size - 1 |
||||
|
else: |
||||
|
start = int(start_text) |
||||
|
end = file_size - 1 if end_text == "" else int(end_text) |
||||
|
except ValueError as exc: |
||||
|
raise RangeNotSatisfiableError() from exc |
||||
|
|
||||
|
if start < 0 or end < start or start >= file_size: |
||||
|
raise RangeNotSatisfiableError() |
||||
|
|
||||
|
end = min(end, file_size - 1) |
||||
|
return start, end |
||||
|
|
||||
|
@staticmethod |
||||
|
def _supports_inline_preview(mime_type: str | None) -> bool: |
||||
|
if mime_type is None: |
||||
|
return False |
||||
|
return mime_type.startswith("image/") or mime_type == "application/pdf" |
||||
@ -0,0 +1,254 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
import json |
||||
|
from dataclasses import dataclass |
||||
|
from datetime import UTC, datetime, timedelta |
||||
|
|
||||
|
from sqlalchemy.ext.asyncio import AsyncSession |
||||
|
|
||||
|
from app.core.config import get_settings |
||||
|
from app.core.exceptions import NotFoundError |
||||
|
from app.core.ids import new_id |
||||
|
from app.models.entities import Job |
||||
|
from app.repositories.backend_repository import BackendRepository |
||||
|
from app.repositories.blob_replica_repository import BlobReplicaRepository |
||||
|
from app.repositories.file_repository import FileRepository |
||||
|
from app.repositories.job_repository import JobRepository |
||||
|
from app.services.policy_service import PolicyService |
||||
|
from app.services.preview_service import PreviewService |
||||
|
from app.services.storage_service import StorageService |
||||
|
|
||||
|
|
||||
|
class JobNotFoundError(NotFoundError): |
||||
|
code = "job_not_found" |
||||
|
message = "Job does not exist." |
||||
|
|
||||
|
|
||||
|
@dataclass |
||||
|
class JobRunResult: |
||||
|
processed: int |
||||
|
completed: int |
||||
|
failed: int |
||||
|
|
||||
|
|
||||
|
class JobService: |
||||
|
def __init__(self, session: AsyncSession) -> None: |
||||
|
self.session = session |
||||
|
self.settings = get_settings() |
||||
|
self.repository = JobRepository(session) |
||||
|
self.file_repository = FileRepository(session) |
||||
|
self.backend_repository = BackendRepository(session) |
||||
|
self.blob_replica_repository = BlobReplicaRepository(session) |
||||
|
self.storage_service = StorageService(session) |
||||
|
self.policy_service = PolicyService(session) |
||||
|
self.preview_service = PreviewService(session) |
||||
|
|
||||
|
async def list_jobs(self) -> list[Job]: |
||||
|
return await self.repository.list_all() |
||||
|
|
||||
|
async def get_job(self, job_id: str) -> Job: |
||||
|
job = await self.repository.get_by_id(job_id) |
||||
|
if job is None: |
||||
|
raise JobNotFoundError() |
||||
|
return job |
||||
|
|
||||
|
async def enqueue_blob_replication(self, blob_id: str, backend_id: str) -> Job: |
||||
|
payload_json = json.dumps({"blob_id": blob_id, "backend_id": backend_id}, sort_keys=True) |
||||
|
existing = await self.repository.find_active_by_kind_and_payload("replicate_blob", payload_json) |
||||
|
if existing is not None: |
||||
|
return existing |
||||
|
job = Job( |
||||
|
id=new_id("job"), |
||||
|
kind="replicate_blob", |
||||
|
status="queued", |
||||
|
payload_json=payload_json, |
||||
|
run_after=datetime.now(UTC), |
||||
|
last_error=None, |
||||
|
) |
||||
|
created = await self.repository.create(job) |
||||
|
await self.session.commit() |
||||
|
return created |
||||
|
|
||||
|
async def enqueue_file_reconcile(self, file_id: str) -> Job: |
||||
|
payload_json = json.dumps({"file_id": file_id}, sort_keys=True) |
||||
|
existing = await self.repository.find_active_by_kind_and_payload("reconcile_file_replicas", payload_json) |
||||
|
if existing is not None: |
||||
|
return existing |
||||
|
job = Job( |
||||
|
id=new_id("job"), |
||||
|
kind="reconcile_file_replicas", |
||||
|
status="queued", |
||||
|
payload_json=payload_json, |
||||
|
run_after=datetime.now(UTC), |
||||
|
last_error=None, |
||||
|
) |
||||
|
created = await self.repository.create(job) |
||||
|
await self.session.commit() |
||||
|
return created |
||||
|
|
||||
|
async def enqueue_full_reconcile(self) -> Job: |
||||
|
payload_json = json.dumps({}, sort_keys=True) |
||||
|
existing = await self.repository.find_active_by_kind_and_payload("reconcile_all_files", payload_json) |
||||
|
if existing is not None: |
||||
|
return existing |
||||
|
job = Job( |
||||
|
id=new_id("job"), |
||||
|
kind="reconcile_all_files", |
||||
|
status="queued", |
||||
|
payload_json=payload_json, |
||||
|
run_after=datetime.now(UTC), |
||||
|
last_error=None, |
||||
|
) |
||||
|
created = await self.repository.create(job) |
||||
|
await self.session.commit() |
||||
|
return created |
||||
|
|
||||
|
async def enqueue_backend_health_check(self, backend_id: str) -> Job: |
||||
|
payload_json = json.dumps({"backend_id": backend_id}, sort_keys=True) |
||||
|
existing = await self.repository.find_active_by_kind_and_payload("check_backend_health", payload_json) |
||||
|
if existing is not None: |
||||
|
return existing |
||||
|
job = Job( |
||||
|
id=new_id("job"), |
||||
|
kind="check_backend_health", |
||||
|
status="queued", |
||||
|
payload_json=payload_json, |
||||
|
run_after=datetime.now(UTC), |
||||
|
last_error=None, |
||||
|
) |
||||
|
created = await self.repository.create(job) |
||||
|
await self.session.commit() |
||||
|
return created |
||||
|
|
||||
|
async def enqueue_preview_generation(self, file_id: str) -> Job: |
||||
|
payload_json = json.dumps({"file_id": file_id}, sort_keys=True) |
||||
|
existing = await self.repository.find_active_by_kind_and_payload("generate_preview_artifacts", payload_json) |
||||
|
if existing is not None: |
||||
|
return existing |
||||
|
job = Job( |
||||
|
id=new_id("job"), |
||||
|
kind="generate_preview_artifacts", |
||||
|
status="queued", |
||||
|
payload_json=payload_json, |
||||
|
run_after=datetime.now(UTC), |
||||
|
last_error=None, |
||||
|
) |
||||
|
created = await self.repository.create(job) |
||||
|
await self.session.commit() |
||||
|
return created |
||||
|
|
||||
|
async def retry_job(self, job_id: str) -> Job: |
||||
|
job = await self.get_job(job_id) |
||||
|
job.status = "retryable" |
||||
|
job.run_after = datetime.now(UTC) |
||||
|
job.last_error = None |
||||
|
saved = await self.repository.save(job) |
||||
|
await self.session.commit() |
||||
|
return saved |
||||
|
|
||||
|
async def run_pending_jobs(self, limit: int | None = None) -> JobRunResult: |
||||
|
now = datetime.now(UTC) |
||||
|
runnable = await self.repository.list_runnable(now, limit or self.settings.job_batch_size) |
||||
|
completed = 0 |
||||
|
failed = 0 |
||||
|
for job in runnable: |
||||
|
job.status = "running" |
||||
|
await self.repository.save(job) |
||||
|
await self.session.commit() |
||||
|
try: |
||||
|
await self._execute(job) |
||||
|
job.status = "completed" |
||||
|
job.last_error = None |
||||
|
completed += 1 |
||||
|
except Exception as exc: |
||||
|
job.attempt_count += 1 |
||||
|
job.last_error = str(exc) |
||||
|
if job.attempt_count >= job.max_attempts: |
||||
|
job.status = "failed" |
||||
|
else: |
||||
|
job.status = "retryable" |
||||
|
job.run_after = datetime.now(UTC) + timedelta(seconds=min(60, 2**job.attempt_count)) |
||||
|
failed += 1 |
||||
|
await self.repository.save(job) |
||||
|
await self.session.commit() |
||||
|
return JobRunResult(processed=len(runnable), completed=completed, failed=failed) |
||||
|
|
||||
|
async def _execute(self, job: Job) -> None: |
||||
|
payload = json.loads(job.payload_json) |
||||
|
if job.kind == "reconcile_all_files": |
||||
|
files = await self.file_repository.list_active() |
||||
|
for file_entry in files: |
||||
|
await self.enqueue_file_reconcile(file_entry.id) |
||||
|
return |
||||
|
|
||||
|
if job.kind == "reconcile_file_replicas": |
||||
|
file_id = payload["file_id"] |
||||
|
file_entry = await self.file_repository.get_by_id(file_id) |
||||
|
if file_entry is None or file_entry.is_deleted or file_entry.deleted_at is not None: |
||||
|
raise ValueError("File is unavailable.") |
||||
|
|
||||
|
blob = await self.file_repository.get_primary_blob_for_file(file_id) |
||||
|
if blob is None: |
||||
|
raise ValueError("Primary blob does not exist.") |
||||
|
|
||||
|
decision = await self.policy_service.get_placement_decision(file_entry.mime_type, size_bytes=blob.size_bytes) |
||||
|
desired_backends = decision.backends |
||||
|
existing_replicas = await self.blob_replica_repository.list_by_blob(blob.id) |
||||
|
ready_backend_ids = {replica.backend_id for replica in existing_replicas if replica.status == "ready"} |
||||
|
|
||||
|
missing_backends = [backend for backend in desired_backends if backend.id not in ready_backend_ids] |
||||
|
if not missing_backends: |
||||
|
return |
||||
|
|
||||
|
source_path = await self.storage_service.resolve_best_local_path(blob) |
||||
|
failed_targets: list[str] = [] |
||||
|
for backend in missing_backends: |
||||
|
try: |
||||
|
status, _detail = await self.storage_service.check_backend(backend) |
||||
|
if status != "healthy": |
||||
|
failed_targets.append(backend.name) |
||||
|
continue |
||||
|
await self.storage_service.persist_blob_to_backend(blob, backend, str(source_path)) |
||||
|
except Exception: |
||||
|
failed_targets.append(backend.name) |
||||
|
|
||||
|
if failed_targets: |
||||
|
raise ValueError( |
||||
|
"Reconciliation is still waiting on backends: " + ", ".join(sorted(set(failed_targets))) |
||||
|
) |
||||
|
return |
||||
|
|
||||
|
if job.kind == "replicate_blob": |
||||
|
blob_id = payload["blob_id"] |
||||
|
backend_id = payload["backend_id"] |
||||
|
|
||||
|
blob = await self.file_repository.get_blob_by_id(blob_id) |
||||
|
if blob is None: |
||||
|
raise ValueError("Blob does not exist.") |
||||
|
|
||||
|
backend = await self.backend_repository.get_by_id(backend_id) |
||||
|
if backend is None or not backend.is_enabled: |
||||
|
raise ValueError("Backend is unavailable.") |
||||
|
|
||||
|
source_path = await self.storage_service.resolve_best_local_path(blob) |
||||
|
await self.storage_service.persist_blob_to_backend(blob, backend, str(source_path)) |
||||
|
return |
||||
|
|
||||
|
if job.kind == "check_backend_health": |
||||
|
backend_id = payload["backend_id"] |
||||
|
backend = await self.backend_repository.get_by_id(backend_id) |
||||
|
if backend is None or not backend.is_enabled: |
||||
|
raise ValueError("Backend is unavailable.") |
||||
|
status, _detail = await self.storage_service.check_backend(backend) |
||||
|
backend.last_health_status = status |
||||
|
backend.last_health_checked_at = datetime.now(UTC) |
||||
|
await self.backend_repository.save(backend) |
||||
|
await self.session.commit() |
||||
|
return |
||||
|
|
||||
|
if job.kind == "generate_preview_artifacts": |
||||
|
file_id = payload["file_id"] |
||||
|
await self.preview_service.generate_preview_artifacts_for_file(file_id) |
||||
|
return |
||||
|
|
||||
|
raise ValueError(f"Unsupported job kind: {job.kind}") |
||||
@ -0,0 +1,82 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
import json |
||||
|
from pathlib import Path |
||||
|
|
||||
|
import anyio |
||||
|
from sqlalchemy.ext.asyncio import AsyncSession |
||||
|
|
||||
|
from app.adapters.storage.local import LocalStorageAdapter |
||||
|
from app.core.config import get_settings |
||||
|
from app.core.ids import new_id |
||||
|
from app.core.secret_codec import encrypt_secret_config |
||||
|
from app.core.storage_layout import build_blob_storage_key |
||||
|
from app.models.entities import Backend, Blob, BlobReplica |
||||
|
from app.repositories.backend_repository import BackendRepository |
||||
|
from app.repositories.blob_replica_repository import BlobReplicaRepository |
||||
|
|
||||
|
DEFAULT_LOCAL_BACKEND_ID = "bkd_local_default" |
||||
|
DEFAULT_LOCAL_BACKEND_NAME = "local-default" |
||||
|
|
||||
|
|
||||
|
class LocalStorageService: |
||||
|
def __init__(self, session: AsyncSession) -> None: |
||||
|
self.session = session |
||||
|
self.settings = get_settings() |
||||
|
self.backend_repository = BackendRepository(session) |
||||
|
self.blob_replica_repository = BlobReplicaRepository(session) |
||||
|
self.adapter = LocalStorageAdapter(self.settings.local_storage_dir) |
||||
|
|
||||
|
async def ensure_backend_ready(self) -> Backend: |
||||
|
await self.adapter.ensure_base_path() |
||||
|
|
||||
|
backend = await self.backend_repository.get_by_name(DEFAULT_LOCAL_BACKEND_NAME) |
||||
|
if backend is not None: |
||||
|
return backend |
||||
|
|
||||
|
backend = Backend( |
||||
|
id=DEFAULT_LOCAL_BACKEND_ID, |
||||
|
name=DEFAULT_LOCAL_BACKEND_NAME, |
||||
|
type="local_directory", |
||||
|
stability_class="local", |
||||
|
read_priority=100, |
||||
|
write_priority=100, |
||||
|
is_enabled=True, |
||||
|
public_config_json=json.dumps({"base_path": self.settings.local_storage_dir}), |
||||
|
secret_config_json=encrypt_secret_config({}), |
||||
|
) |
||||
|
await self.backend_repository.create(backend) |
||||
|
await self.session.commit() |
||||
|
return backend |
||||
|
|
||||
|
async def persist_blob_from_temp(self, blob: Blob, temp_path: str) -> BlobReplica: |
||||
|
backend = await self.ensure_backend_ready() |
||||
|
existing = await self.blob_replica_repository.get_by_blob_and_backend(blob.id, backend.id) |
||||
|
if existing is not None: |
||||
|
return existing |
||||
|
|
||||
|
storage_key = self._build_storage_key(blob) |
||||
|
await self.adapter.put_file(temp_path, storage_key) |
||||
|
replica = BlobReplica( |
||||
|
id=new_id("rep"), |
||||
|
blob_id=blob.id, |
||||
|
backend_id=backend.id, |
||||
|
storage_key=storage_key, |
||||
|
status="ready", |
||||
|
checksum=blob.content_hash, |
||||
|
size_bytes=blob.size_bytes, |
||||
|
) |
||||
|
created = await self.blob_replica_repository.create(replica) |
||||
|
await self.session.commit() |
||||
|
|
||||
|
stored_path = await self.adapter.resolve_path(storage_key) |
||||
|
await anyio.to_thread.run_sync(stored_path.exists) |
||||
|
return created |
||||
|
|
||||
|
async def resolve_replica_path(self, replica: BlobReplica) -> Path: |
||||
|
await self.ensure_backend_ready() |
||||
|
return await self.adapter.resolve_path(replica.storage_key) |
||||
|
|
||||
|
@staticmethod |
||||
|
def _build_storage_key(blob: Blob) -> str: |
||||
|
return build_blob_storage_key(blob) |
||||
@ -0,0 +1,266 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
import json |
||||
|
from dataclasses import dataclass |
||||
|
|
||||
|
from sqlalchemy.ext.asyncio import AsyncSession |
||||
|
|
||||
|
from app.core.exceptions import ValidationError |
||||
|
from app.core.ids import new_id |
||||
|
from app.models.entities import Backend, PlacementPolicy |
||||
|
from app.repositories.backend_repository import BackendRepository |
||||
|
from app.repositories.placement_policy_repository import PlacementPolicyRepository |
||||
|
|
||||
|
DEFAULT_PLACEMENT_POLICIES = { |
||||
|
"document": { |
||||
|
"require_local": True, |
||||
|
"stable_replica_count": 1, |
||||
|
"opportunistic_replica_count": 0, |
||||
|
"preferred_backend_ids": [], |
||||
|
"excluded_backend_ids": [], |
||||
|
"max_non_local_size_bytes": None, |
||||
|
}, |
||||
|
"image": { |
||||
|
"require_local": True, |
||||
|
"stable_replica_count": 1, |
||||
|
"opportunistic_replica_count": 1, |
||||
|
"preferred_backend_ids": [], |
||||
|
"excluded_backend_ids": [], |
||||
|
"max_non_local_size_bytes": None, |
||||
|
}, |
||||
|
"video": { |
||||
|
"require_local": True, |
||||
|
"stable_replica_count": 1, |
||||
|
"opportunistic_replica_count": 0, |
||||
|
"preferred_backend_ids": [], |
||||
|
"excluded_backend_ids": [], |
||||
|
"max_non_local_size_bytes": None, |
||||
|
}, |
||||
|
"other": { |
||||
|
"require_local": True, |
||||
|
"stable_replica_count": 1, |
||||
|
"opportunistic_replica_count": 0, |
||||
|
"preferred_backend_ids": [], |
||||
|
"excluded_backend_ids": [], |
||||
|
"max_non_local_size_bytes": None, |
||||
|
}, |
||||
|
} |
||||
|
ALLOWED_FILE_CLASSES = frozenset(DEFAULT_PLACEMENT_POLICIES.keys()) |
||||
|
|
||||
|
|
||||
|
@dataclass |
||||
|
class PlacementDecision: |
||||
|
file_class: str |
||||
|
policy: PlacementPolicy |
||||
|
backends: list[Backend] |
||||
|
non_local_allowed: bool |
||||
|
|
||||
|
|
||||
|
class PlacementPolicyValidationError(ValidationError): |
||||
|
code = "placement_policy_invalid" |
||||
|
message = "Placement policy is invalid." |
||||
|
|
||||
|
|
||||
|
class PolicyService: |
||||
|
def __init__(self, session: AsyncSession) -> None: |
||||
|
self.session = session |
||||
|
self.backend_repository = BackendRepository(session) |
||||
|
self.repository = PlacementPolicyRepository(session) |
||||
|
|
||||
|
async def ensure_default_policies(self) -> None: |
||||
|
for file_class, values in DEFAULT_PLACEMENT_POLICIES.items(): |
||||
|
existing = await self.repository.get_by_file_class(file_class) |
||||
|
if existing is not None: |
||||
|
continue |
||||
|
policy = PlacementPolicy( |
||||
|
id=new_id("plc"), |
||||
|
file_class=file_class, |
||||
|
require_local=values["require_local"], |
||||
|
stable_replica_count=values["stable_replica_count"], |
||||
|
opportunistic_replica_count=values["opportunistic_replica_count"], |
||||
|
preferred_backend_ids_json=json.dumps(values["preferred_backend_ids"]), |
||||
|
excluded_backend_ids_json=json.dumps(values["excluded_backend_ids"]), |
||||
|
max_non_local_size_bytes=values["max_non_local_size_bytes"], |
||||
|
) |
||||
|
await self.repository.create(policy) |
||||
|
await self.session.commit() |
||||
|
|
||||
|
async def list_policies(self) -> list[PlacementPolicy]: |
||||
|
await self.ensure_default_policies() |
||||
|
return await self.repository.list_all() |
||||
|
|
||||
|
async def upsert_policy( |
||||
|
self, |
||||
|
*, |
||||
|
file_class: str, |
||||
|
require_local: bool, |
||||
|
stable_replica_count: int, |
||||
|
opportunistic_replica_count: int, |
||||
|
preferred_backend_ids: list[str], |
||||
|
excluded_backend_ids: list[str], |
||||
|
max_non_local_size_bytes: int | None, |
||||
|
) -> PlacementPolicy: |
||||
|
await self.ensure_default_policies() |
||||
|
self._validate_file_class(file_class) |
||||
|
await self._validate_policy_inputs( |
||||
|
require_local=require_local, |
||||
|
stable_replica_count=stable_replica_count, |
||||
|
opportunistic_replica_count=opportunistic_replica_count, |
||||
|
preferred_backend_ids=preferred_backend_ids, |
||||
|
excluded_backend_ids=excluded_backend_ids, |
||||
|
max_non_local_size_bytes=max_non_local_size_bytes, |
||||
|
) |
||||
|
existing = await self.repository.get_by_file_class(file_class) |
||||
|
if existing is None: |
||||
|
existing = PlacementPolicy( |
||||
|
id=new_id("plc"), |
||||
|
file_class=file_class, |
||||
|
require_local=require_local, |
||||
|
stable_replica_count=stable_replica_count, |
||||
|
opportunistic_replica_count=opportunistic_replica_count, |
||||
|
preferred_backend_ids_json=json.dumps(preferred_backend_ids), |
||||
|
excluded_backend_ids_json=json.dumps(excluded_backend_ids), |
||||
|
max_non_local_size_bytes=max_non_local_size_bytes, |
||||
|
) |
||||
|
await self.repository.create(existing) |
||||
|
else: |
||||
|
existing.require_local = require_local |
||||
|
existing.stable_replica_count = stable_replica_count |
||||
|
existing.opportunistic_replica_count = opportunistic_replica_count |
||||
|
existing.preferred_backend_ids_json = json.dumps(preferred_backend_ids) |
||||
|
existing.excluded_backend_ids_json = json.dumps(excluded_backend_ids) |
||||
|
existing.max_non_local_size_bytes = max_non_local_size_bytes |
||||
|
await self.repository.save(existing) |
||||
|
await self.session.commit() |
||||
|
return existing |
||||
|
|
||||
|
async def get_placement_decision(self, mime_type: str | None, size_bytes: int | None = None) -> PlacementDecision: |
||||
|
await self.ensure_default_policies() |
||||
|
file_class = self.classify_mime_type(mime_type) |
||||
|
policy = await self.repository.get_by_file_class(file_class) |
||||
|
if policy is None: |
||||
|
raise ValueError(f"Placement policy is missing for file class: {file_class}") |
||||
|
|
||||
|
preferred_backend_ids = self._decode_backend_ids(policy.preferred_backend_ids_json) |
||||
|
excluded_backend_ids = set(self._decode_backend_ids(policy.excluded_backend_ids_json)) |
||||
|
backends = await self.backend_repository.list_enabled() |
||||
|
eligible_backends = [backend for backend in backends if backend.id not in excluded_backend_ids] |
||||
|
local_backends = self._sort_backends( |
||||
|
[backend for backend in eligible_backends if backend.type == "local_directory"], |
||||
|
preferred_backend_ids, |
||||
|
) |
||||
|
stable_backends = [ |
||||
|
backend |
||||
|
for backend in eligible_backends |
||||
|
if backend.type != "local_directory" and backend.stability_class == "stable" |
||||
|
] |
||||
|
opportunistic_backends = [ |
||||
|
backend |
||||
|
for backend in eligible_backends |
||||
|
if backend.type != "local_directory" and backend.stability_class == "opportunistic" |
||||
|
] |
||||
|
stable_backends = self._sort_backends(stable_backends, preferred_backend_ids) |
||||
|
opportunistic_backends = self._sort_backends(opportunistic_backends, preferred_backend_ids) |
||||
|
|
||||
|
selected: list[Backend] = [] |
||||
|
if policy.require_local and local_backends: |
||||
|
selected.append(local_backends[0]) |
||||
|
|
||||
|
non_local_allowed = ( |
||||
|
policy.max_non_local_size_bytes is None |
||||
|
or size_bytes is None |
||||
|
or size_bytes <= policy.max_non_local_size_bytes |
||||
|
) |
||||
|
if non_local_allowed: |
||||
|
selected.extend(stable_backends[: policy.stable_replica_count]) |
||||
|
selected.extend(opportunistic_backends[: policy.opportunistic_replica_count]) |
||||
|
|
||||
|
deduped: list[Backend] = [] |
||||
|
seen = set() |
||||
|
for backend in selected: |
||||
|
if backend.id in seen: |
||||
|
continue |
||||
|
seen.add(backend.id) |
||||
|
deduped.append(backend) |
||||
|
|
||||
|
return PlacementDecision( |
||||
|
file_class=file_class, |
||||
|
policy=policy, |
||||
|
backends=deduped, |
||||
|
non_local_allowed=non_local_allowed, |
||||
|
) |
||||
|
|
||||
|
@staticmethod |
||||
|
def classify_mime_type(mime_type: str | None) -> str: |
||||
|
if mime_type is None: |
||||
|
return "other" |
||||
|
if mime_type.startswith("image/"): |
||||
|
return "image" |
||||
|
if mime_type.startswith("video/"): |
||||
|
return "video" |
||||
|
if mime_type == "application/pdf" or mime_type.startswith("text/"): |
||||
|
return "document" |
||||
|
return "other" |
||||
|
|
||||
|
@staticmethod |
||||
|
def _decode_backend_ids(raw_value: str | None) -> list[str]: |
||||
|
if not raw_value: |
||||
|
return [] |
||||
|
value = json.loads(raw_value) |
||||
|
if not isinstance(value, list): |
||||
|
return [] |
||||
|
return [item for item in value if isinstance(item, str) and item] |
||||
|
|
||||
|
@staticmethod |
||||
|
def _sort_backends(backends: list[Backend], preferred_backend_ids: list[str]) -> list[Backend]: |
||||
|
preferred_positions = {backend_id: index for index, backend_id in enumerate(preferred_backend_ids)} |
||||
|
return sorted( |
||||
|
backends, |
||||
|
key=lambda backend: ( |
||||
|
0 if backend.id in preferred_positions else 1, |
||||
|
preferred_positions.get(backend.id, 0), |
||||
|
-backend.write_priority, |
||||
|
-backend.read_priority, |
||||
|
backend.name, |
||||
|
), |
||||
|
) |
||||
|
|
||||
|
@staticmethod |
||||
|
def _validate_file_class(file_class: str) -> None: |
||||
|
if file_class not in ALLOWED_FILE_CLASSES: |
||||
|
raise PlacementPolicyValidationError( |
||||
|
"Unsupported file class. Allowed values: " + ", ".join(sorted(ALLOWED_FILE_CLASSES)) |
||||
|
) |
||||
|
|
||||
|
async def _validate_policy_inputs( |
||||
|
self, |
||||
|
*, |
||||
|
require_local: bool, |
||||
|
stable_replica_count: int, |
||||
|
opportunistic_replica_count: int, |
||||
|
preferred_backend_ids: list[str], |
||||
|
excluded_backend_ids: list[str], |
||||
|
max_non_local_size_bytes: int | None, |
||||
|
) -> None: |
||||
|
del require_local |
||||
|
if stable_replica_count < 0 or opportunistic_replica_count < 0: |
||||
|
raise PlacementPolicyValidationError("Replica counts must be zero or greater.") |
||||
|
if max_non_local_size_bytes is not None and max_non_local_size_bytes <= 0: |
||||
|
raise PlacementPolicyValidationError("max_non_local_size_bytes must be greater than zero.") |
||||
|
|
||||
|
preferred_set = set(preferred_backend_ids) |
||||
|
excluded_set = set(excluded_backend_ids) |
||||
|
if len(preferred_set) != len(preferred_backend_ids): |
||||
|
raise PlacementPolicyValidationError("preferred_backend_ids must not contain duplicates.") |
||||
|
if len(excluded_set) != len(excluded_backend_ids): |
||||
|
raise PlacementPolicyValidationError("excluded_backend_ids must not contain duplicates.") |
||||
|
overlap = preferred_set & excluded_set |
||||
|
if overlap: |
||||
|
raise PlacementPolicyValidationError("A backend cannot be both preferred and excluded.") |
||||
|
|
||||
|
known_backend_ids = {backend.id for backend in await self.backend_repository.list_all()} |
||||
|
unknown_backend_ids = sorted((preferred_set | excluded_set) - known_backend_ids) |
||||
|
if unknown_backend_ids: |
||||
|
raise PlacementPolicyValidationError( |
||||
|
"Unknown backend ids referenced by placement policy: " + ", ".join(unknown_backend_ids) |
||||
|
) |
||||
@ -0,0 +1,65 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from app.core.ids import new_id |
||||
|
from app.models.entities import PreviewArtifact |
||||
|
from app.repositories.file_repository import FileRepository |
||||
|
from app.repositories.preview_artifact_repository import PreviewArtifactRepository |
||||
|
|
||||
|
|
||||
|
class PreviewService: |
||||
|
def __init__(self, session) -> None: |
||||
|
self.session = session |
||||
|
self.file_repository = FileRepository(session) |
||||
|
self.preview_artifact_repository = PreviewArtifactRepository(session) |
||||
|
|
||||
|
async def get_preview_artifacts_for_file(self, file_id: str) -> list[PreviewArtifact]: |
||||
|
file_entry = await self.file_repository.get_by_id(file_id) |
||||
|
if file_entry is None or file_entry.current_version_id is None: |
||||
|
return [] |
||||
|
return await self.preview_artifact_repository.list_by_file_version(file_entry.current_version_id) |
||||
|
|
||||
|
async def generate_preview_artifacts_for_file(self, file_id: str) -> int: |
||||
|
file_entry = await self.file_repository.get_by_id(file_id) |
||||
|
if file_entry is None or file_entry.current_version_id is None: |
||||
|
raise ValueError("File is unavailable.") |
||||
|
blob = await self.file_repository.get_primary_blob_for_file(file_id) |
||||
|
if blob is None: |
||||
|
raise ValueError("Primary blob does not exist.") |
||||
|
|
||||
|
artifact_types = self._select_artifact_types(file_entry.mime_type) |
||||
|
created = 0 |
||||
|
for artifact_type in artifact_types: |
||||
|
existing = await self.preview_artifact_repository.get_by_version_and_type( |
||||
|
file_entry.current_version_id, |
||||
|
artifact_type, |
||||
|
) |
||||
|
if existing is not None: |
||||
|
if existing.status != "ready": |
||||
|
existing.status = "ready" |
||||
|
existing.blob_id = blob.id |
||||
|
await self.preview_artifact_repository.save(existing) |
||||
|
continue |
||||
|
|
||||
|
artifact = PreviewArtifact( |
||||
|
id=new_id("prv"), |
||||
|
file_version_id=file_entry.current_version_id, |
||||
|
artifact_type=artifact_type, |
||||
|
blob_id=blob.id, |
||||
|
status="ready", |
||||
|
) |
||||
|
await self.preview_artifact_repository.create(artifact) |
||||
|
created += 1 |
||||
|
await self.session.commit() |
||||
|
return created |
||||
|
|
||||
|
@staticmethod |
||||
|
def _select_artifact_types(mime_type: str | None) -> list[str]: |
||||
|
if mime_type is None: |
||||
|
return [] |
||||
|
if mime_type.startswith("image/"): |
||||
|
return ["inline_preview", "thumbnail"] |
||||
|
if mime_type == "application/pdf": |
||||
|
return ["inline_preview", "poster"] |
||||
|
if mime_type.startswith("video/"): |
||||
|
return ["poster"] |
||||
|
return [] |
||||
@ -0,0 +1,36 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from sqlalchemy.ext.asyncio import AsyncSession |
||||
|
|
||||
|
from app.core.exceptions import FileNotFoundError |
||||
|
from app.repositories.backend_repository import BackendRepository |
||||
|
from app.repositories.file_repository import FileRepository |
||||
|
from app.services.job_service import JobService |
||||
|
|
||||
|
|
||||
|
class RepairService: |
||||
|
def __init__(self, session: AsyncSession) -> None: |
||||
|
self.session = session |
||||
|
self.file_repository = FileRepository(session) |
||||
|
self.backend_repository = BackendRepository(session) |
||||
|
self.job_service = JobService(session) |
||||
|
|
||||
|
async def enqueue_repairs_for_file(self, file_id: str) -> int: |
||||
|
file_entry = await self.file_repository.get_by_id(file_id) |
||||
|
if file_entry is None or file_entry.is_deleted or file_entry.deleted_at is not None: |
||||
|
raise FileNotFoundError() |
||||
|
|
||||
|
await self.job_service.enqueue_file_reconcile(file_id) |
||||
|
return 1 |
||||
|
|
||||
|
async def enqueue_backend_health_checks(self) -> int: |
||||
|
backends = await self.backend_repository.list_enabled() |
||||
|
enqueued = 0 |
||||
|
for backend in backends: |
||||
|
await self.job_service.enqueue_backend_health_check(backend.id) |
||||
|
enqueued += 1 |
||||
|
return enqueued |
||||
|
|
||||
|
async def enqueue_full_reconcile(self) -> int: |
||||
|
await self.job_service.enqueue_full_reconcile() |
||||
|
return 1 |
||||
@ -0,0 +1,152 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
import builtins |
||||
|
import json |
||||
|
from datetime import UTC, datetime |
||||
|
from pathlib import Path |
||||
|
|
||||
|
import anyio |
||||
|
from sqlalchemy.ext.asyncio import AsyncSession |
||||
|
|
||||
|
from app.adapters.storage.registry import build_storage_adapter |
||||
|
from app.core.backend_config import merge_backend_config |
||||
|
from app.core.config import get_settings |
||||
|
from app.core.ids import new_id |
||||
|
from app.core.secret_codec import decrypt_secret_config |
||||
|
from app.core.storage_layout import build_blob_storage_key |
||||
|
from app.models.entities import Backend, Blob, BlobReplica, CacheEntry |
||||
|
from app.repositories.backend_repository import BackendRepository |
||||
|
from app.repositories.blob_replica_repository import BlobReplicaRepository |
||||
|
from app.repositories.cache_entry_repository import CacheEntryRepository |
||||
|
from app.services.policy_service import PlacementDecision, PolicyService |
||||
|
|
||||
|
|
||||
|
class StorageService: |
||||
|
def __init__(self, session: AsyncSession) -> None: |
||||
|
self.session = session |
||||
|
self.settings = get_settings() |
||||
|
self.backend_repository = BackendRepository(session) |
||||
|
self.blob_replica_repository = BlobReplicaRepository(session) |
||||
|
self.cache_entry_repository = CacheEntryRepository(session) |
||||
|
self.policy_service = PolicyService(session) |
||||
|
|
||||
|
async def ensure_cache_dir(self) -> Path: |
||||
|
cache_dir = Path(self.settings.cache_dir) |
||||
|
await anyio.to_thread.run_sync(lambda: cache_dir.mkdir(parents=True, exist_ok=True)) |
||||
|
return cache_dir |
||||
|
|
||||
|
async def get_replication_targets_for_mime_type( |
||||
|
self, |
||||
|
mime_type: str | None, |
||||
|
size_bytes: int | None = None, |
||||
|
) -> PlacementDecision: |
||||
|
return await self.policy_service.get_placement_decision(mime_type, size_bytes=size_bytes) |
||||
|
|
||||
|
async def persist_blob_to_backend(self, blob: Blob, backend: Backend, source_path: str) -> BlobReplica: |
||||
|
existing = await self.blob_replica_repository.get_by_blob_and_backend(blob.id, backend.id) |
||||
|
storage_key = self._build_storage_key(blob) |
||||
|
adapter = self._build_adapter(backend) |
||||
|
existing_is_usable = False |
||||
|
if existing is not None and existing.status == "ready": |
||||
|
try: |
||||
|
existing_is_usable = await adapter.object_exists(storage_key) |
||||
|
except Exception: |
||||
|
existing_is_usable = False |
||||
|
if existing_is_usable: |
||||
|
return existing |
||||
|
|
||||
|
result = await adapter.put_file(source_path, storage_key) |
||||
|
etag = result.get("etag") |
||||
|
|
||||
|
replica = existing or BlobReplica( |
||||
|
id=new_id("rep"), |
||||
|
blob_id=blob.id, |
||||
|
backend_id=backend.id, |
||||
|
storage_key=storage_key, |
||||
|
size_bytes=blob.size_bytes, |
||||
|
checksum=blob.content_hash, |
||||
|
status="ready", |
||||
|
etag=etag, |
||||
|
last_verified_at=datetime.now(UTC), |
||||
|
) |
||||
|
replica.storage_key = storage_key |
||||
|
replica.status = "ready" |
||||
|
replica.size_bytes = blob.size_bytes |
||||
|
replica.checksum = blob.content_hash |
||||
|
replica.etag = etag |
||||
|
replica.last_verified_at = datetime.now(UTC) |
||||
|
|
||||
|
if existing is None: |
||||
|
created = await self.blob_replica_repository.create(replica) |
||||
|
else: |
||||
|
created = await self.blob_replica_repository.save(replica) |
||||
|
await self.session.commit() |
||||
|
return created |
||||
|
|
||||
|
async def materialize_replica_to_local_cache(self, blob: Blob, replica: BlobReplica, backend: Backend) -> Path: |
||||
|
if backend.type == "local_directory": |
||||
|
adapter = self._build_adapter(backend) |
||||
|
return await adapter.resolve_path(replica.storage_key) |
||||
|
|
||||
|
cache_dir = await self.ensure_cache_dir() |
||||
|
cache_path = cache_dir / self._build_storage_key(blob) |
||||
|
adapter = self._build_adapter(backend) |
||||
|
await adapter.download_file(replica.storage_key, cache_path) |
||||
|
|
||||
|
existing_entry = await self.cache_entry_repository.get_ready_by_blob(blob.id) |
||||
|
now = datetime.now(UTC) |
||||
|
if existing_entry is None: |
||||
|
entry = CacheEntry( |
||||
|
id=new_id("cch"), |
||||
|
blob_id=blob.id, |
||||
|
local_path=str(cache_path), |
||||
|
cache_kind="remote-fetch", |
||||
|
size_bytes=blob.size_bytes, |
||||
|
status="ready", |
||||
|
last_accessed_at=now, |
||||
|
expires_at=None, |
||||
|
) |
||||
|
await self.cache_entry_repository.create(entry) |
||||
|
else: |
||||
|
existing_entry.local_path = str(cache_path) |
||||
|
existing_entry.status = "ready" |
||||
|
existing_entry.size_bytes = blob.size_bytes |
||||
|
existing_entry.last_accessed_at = now |
||||
|
await self.cache_entry_repository.save(existing_entry) |
||||
|
await self.session.commit() |
||||
|
return cache_path |
||||
|
|
||||
|
async def resolve_best_local_path(self, blob: Blob) -> Path: |
||||
|
cached_entry = await self.cache_entry_repository.get_ready_by_blob(blob.id) |
||||
|
if cached_entry is not None: |
||||
|
cached_path = Path(cached_entry.local_path) |
||||
|
if await anyio.to_thread.run_sync(cached_path.exists): |
||||
|
cached_entry.last_accessed_at = datetime.now(UTC) |
||||
|
await self.cache_entry_repository.save(cached_entry) |
||||
|
await self.session.commit() |
||||
|
return cached_path |
||||
|
|
||||
|
replicas = await self.blob_replica_repository.list_ready_replicas_with_backends(blob.id) |
||||
|
for replica, backend in replicas: |
||||
|
try: |
||||
|
return await self.materialize_replica_to_local_cache(blob, replica, backend) |
||||
|
except builtins.FileNotFoundError: |
||||
|
continue |
||||
|
except Exception: |
||||
|
continue |
||||
|
raise builtins.FileNotFoundError(blob.id) |
||||
|
|
||||
|
async def check_backend(self, backend: Backend) -> tuple[str, str | None]: |
||||
|
adapter = self._build_adapter(backend) |
||||
|
result = await adapter.check() |
||||
|
return result["status"], result.get("detail") |
||||
|
|
||||
|
def _build_adapter(self, backend: Backend): |
||||
|
public_config = json.loads(backend.public_config_json) |
||||
|
secret_config = decrypt_secret_config(backend.secret_config_json) |
||||
|
config = merge_backend_config(public_config, secret_config) |
||||
|
return build_storage_adapter(backend.type, config) |
||||
|
|
||||
|
@staticmethod |
||||
|
def _build_storage_key(blob: Blob) -> str: |
||||
|
return build_blob_storage_key(blob) |
||||
@ -0,0 +1,185 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
import base64 |
||||
|
import hashlib |
||||
|
from dataclasses import dataclass |
||||
|
from pathlib import Path |
||||
|
|
||||
|
import anyio |
||||
|
from sqlalchemy.ext.asyncio import AsyncSession |
||||
|
|
||||
|
from app.core.config import get_settings |
||||
|
from app.core.exceptions import UploadConflictError, UploadNotFoundError, ValidationError |
||||
|
from app.core.ids import new_id |
||||
|
from app.models.entities import UploadSession |
||||
|
from app.repositories.file_repository import FileRepository |
||||
|
from app.repositories.upload_repository import UploadRepository |
||||
|
from app.services.file_service import CreateFileMetadataInput, FileService |
||||
|
from app.services.job_service import JobService |
||||
|
from app.services.local_storage_service import LocalStorageService |
||||
|
from app.services.storage_service import StorageService |
||||
|
|
||||
|
|
||||
|
@dataclass |
||||
|
class CreateUploadInput: |
||||
|
upload_length: int |
||||
|
upload_metadata: str | None = None |
||||
|
|
||||
|
|
||||
|
@dataclass |
||||
|
class FinalizeUploadInput: |
||||
|
directory_id: str |
||||
|
filename: str |
||||
|
|
||||
|
|
||||
|
class UploadService: |
||||
|
def __init__(self, session: AsyncSession) -> None: |
||||
|
self.session = session |
||||
|
self.repository = UploadRepository(session) |
||||
|
self.file_service = FileService(session) |
||||
|
self.file_repository = FileRepository(session) |
||||
|
self.local_storage_service = LocalStorageService(session) |
||||
|
self.storage_service = StorageService(session) |
||||
|
self.job_service = JobService(session) |
||||
|
self.settings = get_settings() |
||||
|
|
||||
|
async def ensure_temp_dir(self) -> Path: |
||||
|
temp_dir = Path(self.settings.upload_temp_dir) |
||||
|
await anyio.to_thread.run_sync(lambda: temp_dir.mkdir(parents=True, exist_ok=True)) |
||||
|
return temp_dir |
||||
|
|
||||
|
async def create_upload(self, payload: CreateUploadInput) -> UploadSession: |
||||
|
if payload.upload_length <= 0: |
||||
|
raise ValidationError("Upload-Length must be greater than zero.") |
||||
|
|
||||
|
temp_dir = await self.ensure_temp_dir() |
||||
|
upload_id = new_id("upl") |
||||
|
temp_path = temp_dir / f"{upload_id}.bin" |
||||
|
await anyio.Path(temp_path).write_bytes(b"") |
||||
|
|
||||
|
upload_session = UploadSession( |
||||
|
id=upload_id, |
||||
|
total_size_bytes=payload.upload_length, |
||||
|
received_size_bytes=0, |
||||
|
status="created", |
||||
|
temp_path=str(temp_path), |
||||
|
tus_upload_url=f"/files/{upload_id}", |
||||
|
upload_metadata_json=payload.upload_metadata, |
||||
|
) |
||||
|
created = await self.repository.create(upload_session) |
||||
|
await self.session.commit() |
||||
|
return created |
||||
|
|
||||
|
async def get_upload(self, upload_id: str) -> UploadSession: |
||||
|
upload_session = await self.repository.get_by_id(upload_id) |
||||
|
if upload_session is None: |
||||
|
raise UploadNotFoundError() |
||||
|
return upload_session |
||||
|
|
||||
|
async def append_upload_bytes(self, upload_id: str, offset: int, chunk: bytes) -> UploadSession: |
||||
|
upload_session = await self.get_upload(upload_id) |
||||
|
|
||||
|
if upload_session.status in {"finalized", "finalizing", "expired"}: |
||||
|
raise UploadConflictError("Upload session is not writable.") |
||||
|
if offset != upload_session.received_size_bytes: |
||||
|
raise UploadConflictError("Upload offset does not match the current stored offset.") |
||||
|
if upload_session.received_size_bytes + len(chunk) > upload_session.total_size_bytes: |
||||
|
raise UploadConflictError("Upload would exceed declared length.") |
||||
|
|
||||
|
temp_path = anyio.Path(upload_session.temp_path) |
||||
|
async with await anyio.open_file(temp_path, "ab") as file_handle: |
||||
|
await file_handle.write(chunk) |
||||
|
|
||||
|
upload_session.received_size_bytes += len(chunk) |
||||
|
upload_session.status = ( |
||||
|
"uploaded" |
||||
|
if upload_session.received_size_bytes == upload_session.total_size_bytes |
||||
|
else "uploading" |
||||
|
) |
||||
|
saved = await self.repository.save(upload_session) |
||||
|
await self.session.commit() |
||||
|
return saved |
||||
|
|
||||
|
async def finalize_upload(self, upload_id: str, payload: FinalizeUploadInput): |
||||
|
upload_session = await self.get_upload(upload_id) |
||||
|
if upload_session.received_size_bytes != upload_session.total_size_bytes: |
||||
|
raise UploadConflictError("Upload is not complete.") |
||||
|
if upload_session.status == "finalized": |
||||
|
raise UploadConflictError("Upload has already been finalized.") |
||||
|
|
||||
|
upload_session.status = "finalizing" |
||||
|
await self.repository.save(upload_session) |
||||
|
await self.session.flush() |
||||
|
|
||||
|
content_hash = await self._compute_sha256(upload_session.temp_path) |
||||
|
mime_type = self._guess_mime_type(payload.filename) |
||||
|
file_entry = await self.file_service.create_file_metadata( |
||||
|
CreateFileMetadataInput( |
||||
|
directory_id=payload.directory_id, |
||||
|
name=payload.filename, |
||||
|
mime_type=mime_type, |
||||
|
size_bytes=upload_session.total_size_bytes, |
||||
|
content_hash=content_hash, |
||||
|
) |
||||
|
) |
||||
|
blob = await self.file_repository.get_primary_blob_for_file(file_entry.id) |
||||
|
if blob is None: |
||||
|
raise UploadConflictError("Primary blob was not created for finalized upload.") |
||||
|
|
||||
|
await self.local_storage_service.persist_blob_from_temp(blob, upload_session.temp_path) |
||||
|
await self.job_service.enqueue_file_reconcile(file_entry.id) |
||||
|
if mime_type is not None: |
||||
|
await self.job_service.enqueue_preview_generation(file_entry.id) |
||||
|
|
||||
|
upload_session.directory_id = payload.directory_id |
||||
|
upload_session.filename = payload.filename |
||||
|
upload_session.status = "finalized" |
||||
|
await self.repository.save(upload_session) |
||||
|
await self.session.commit() |
||||
|
await anyio.Path(upload_session.temp_path).unlink(missing_ok=True) |
||||
|
|
||||
|
return upload_session, file_entry |
||||
|
|
||||
|
async def _compute_sha256(self, temp_path: str) -> str: |
||||
|
def _hash_file() -> str: |
||||
|
digest = hashlib.sha256() |
||||
|
with open(temp_path, "rb") as file_handle: |
||||
|
while True: |
||||
|
chunk = file_handle.read(1024 * 1024) |
||||
|
if not chunk: |
||||
|
break |
||||
|
digest.update(chunk) |
||||
|
return digest.hexdigest() |
||||
|
|
||||
|
return await anyio.to_thread.run_sync(_hash_file) |
||||
|
|
||||
|
@staticmethod |
||||
|
def parse_upload_metadata(raw_metadata: str | None) -> dict[str, str]: |
||||
|
if not raw_metadata: |
||||
|
return {} |
||||
|
|
||||
|
parsed: dict[str, str] = {} |
||||
|
for item in raw_metadata.split(","): |
||||
|
item = item.strip() |
||||
|
if not item: |
||||
|
continue |
||||
|
parts = item.split(" ", 1) |
||||
|
key = parts[0] |
||||
|
value = "" |
||||
|
if len(parts) == 2 and parts[1]: |
||||
|
value = base64.b64decode(parts[1]).decode("utf-8") |
||||
|
parsed[key] = value |
||||
|
return parsed |
||||
|
|
||||
|
@staticmethod |
||||
|
def _guess_mime_type(filename: str) -> str | None: |
||||
|
lowered = filename.lower() |
||||
|
if lowered.endswith(".mp4"): |
||||
|
return "video/mp4" |
||||
|
if lowered.endswith(".jpg") or lowered.endswith(".jpeg"): |
||||
|
return "image/jpeg" |
||||
|
if lowered.endswith(".png"): |
||||
|
return "image/png" |
||||
|
if lowered.endswith(".pdf"): |
||||
|
return "application/pdf" |
||||
|
return None |
||||
@ -0,0 +1,183 @@ |
|||||
|
const LOGIN_KEY = "iron_access_token"; |
||||
|
|
||||
|
const loginCard = document.querySelector("#login-card"); |
||||
|
const appCard = document.querySelector("#app-card"); |
||||
|
const loginForm = document.querySelector("#login-form"); |
||||
|
const loginError = document.querySelector("#login-error"); |
||||
|
const appError = document.querySelector("#app-error"); |
||||
|
const usernameInput = document.querySelector("#username"); |
||||
|
const passwordInput = document.querySelector("#password"); |
||||
|
const welcomeTitle = document.querySelector("#welcome-title"); |
||||
|
const sessionUser = document.querySelector("#session-user"); |
||||
|
const rootName = document.querySelector("#root-name"); |
||||
|
const itemsCount = document.querySelector("#items-count"); |
||||
|
const rootItems = document.querySelector("#root-items"); |
||||
|
const backendCount = document.querySelector("#backend-count"); |
||||
|
const backendsList = document.querySelector("#backends-list"); |
||||
|
const refreshButton = document.querySelector("#refresh-button"); |
||||
|
const logoutButton = document.querySelector("#logout-button"); |
||||
|
|
||||
|
function getToken() { |
||||
|
return window.localStorage.getItem(LOGIN_KEY); |
||||
|
} |
||||
|
|
||||
|
function setToken(token) { |
||||
|
window.localStorage.setItem(LOGIN_KEY, token); |
||||
|
} |
||||
|
|
||||
|
function clearToken() { |
||||
|
window.localStorage.removeItem(LOGIN_KEY); |
||||
|
} |
||||
|
|
||||
|
async function apiFetch(path, options = {}) { |
||||
|
const headers = new Headers(options.headers || {}); |
||||
|
const token = getToken(); |
||||
|
if (token) { |
||||
|
headers.set("Authorization", `Bearer ${token}`); |
||||
|
} |
||||
|
|
||||
|
const response = await fetch(path, { ...options, headers }); |
||||
|
if (!response.ok) { |
||||
|
let message = `Request failed with status ${response.status}`; |
||||
|
try { |
||||
|
const payload = await response.json(); |
||||
|
message = payload.error?.message || message; |
||||
|
} catch { |
||||
|
// keep fallback message
|
||||
|
} |
||||
|
throw new Error(message); |
||||
|
} |
||||
|
return response; |
||||
|
} |
||||
|
|
||||
|
function renderList(container, items, renderItem) { |
||||
|
container.innerHTML = ""; |
||||
|
if (items.length === 0) { |
||||
|
const empty = document.createElement("li"); |
||||
|
empty.textContent = "Nothing here yet."; |
||||
|
container.append(empty); |
||||
|
return; |
||||
|
} |
||||
|
for (const item of items) { |
||||
|
container.append(renderItem(item)); |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
function renderRootItem(item) { |
||||
|
const element = document.createElement("li"); |
||||
|
const left = document.createElement("div"); |
||||
|
const right = document.createElement("div"); |
||||
|
right.className = "item-meta"; |
||||
|
|
||||
|
left.innerHTML = `<strong>${item.name}</strong>`; |
||||
|
if (item.kind === "file" && item.mime_type) { |
||||
|
right.textContent = item.mime_type; |
||||
|
} else { |
||||
|
right.textContent = item.kind; |
||||
|
} |
||||
|
|
||||
|
element.append(left, right); |
||||
|
return element; |
||||
|
} |
||||
|
|
||||
|
function renderBackendItem(item) { |
||||
|
const element = document.createElement("li"); |
||||
|
const left = document.createElement("div"); |
||||
|
left.innerHTML = `<strong>${item.name}</strong>`; |
||||
|
const right = document.createElement("div"); |
||||
|
right.className = "item-meta"; |
||||
|
right.textContent = `${item.type} · ${item.is_enabled ? "enabled" : "disabled"}`; |
||||
|
element.append(left, right); |
||||
|
return element; |
||||
|
} |
||||
|
|
||||
|
function showLogin(errorMessage = "") { |
||||
|
loginCard.hidden = false; |
||||
|
appCard.hidden = true; |
||||
|
loginError.hidden = !errorMessage; |
||||
|
loginError.textContent = errorMessage; |
||||
|
} |
||||
|
|
||||
|
function showApp() { |
||||
|
loginCard.hidden = true; |
||||
|
appCard.hidden = false; |
||||
|
appError.hidden = true; |
||||
|
} |
||||
|
|
||||
|
async function loadWorkspace() { |
||||
|
try { |
||||
|
const [meResponse, directoryResponse, backendResponse] = await Promise.all([ |
||||
|
apiFetch("/api/auth/me"), |
||||
|
apiFetch("/api/directories/dir_root/children"), |
||||
|
apiFetch("/api/backends"), |
||||
|
]); |
||||
|
|
||||
|
const me = await meResponse.json(); |
||||
|
const directory = await directoryResponse.json(); |
||||
|
const backends = await backendResponse.json(); |
||||
|
|
||||
|
welcomeTitle.textContent = `Welcome, ${me.user.username}`; |
||||
|
sessionUser.textContent = me.user.username; |
||||
|
rootName.textContent = directory.directory.name; |
||||
|
itemsCount.textContent = `${directory.items.length} items`; |
||||
|
backendCount.textContent = String(backends.items.length); |
||||
|
|
||||
|
renderList(rootItems, directory.items, renderRootItem); |
||||
|
renderList(backendsList, backends.items, renderBackendItem); |
||||
|
showApp(); |
||||
|
} catch (error) { |
||||
|
if (String(error.message).includes("Authentication")) { |
||||
|
clearToken(); |
||||
|
showLogin("Your session expired. Please sign in again."); |
||||
|
return; |
||||
|
} |
||||
|
appError.hidden = false; |
||||
|
appError.textContent = error.message; |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
loginForm.addEventListener("submit", async (event) => { |
||||
|
event.preventDefault(); |
||||
|
loginError.hidden = true; |
||||
|
|
||||
|
try { |
||||
|
const response = await fetch("/api/auth/login", { |
||||
|
method: "POST", |
||||
|
headers: { "Content-Type": "application/json" }, |
||||
|
body: JSON.stringify({ |
||||
|
username: usernameInput.value, |
||||
|
password: passwordInput.value, |
||||
|
}), |
||||
|
}); |
||||
|
const payload = await response.json(); |
||||
|
if (!response.ok) { |
||||
|
throw new Error(payload.error?.message || "Unable to sign in."); |
||||
|
} |
||||
|
|
||||
|
setToken(payload.access_token); |
||||
|
passwordInput.value = ""; |
||||
|
await loadWorkspace(); |
||||
|
} catch (error) { |
||||
|
showLogin(error.message); |
||||
|
} |
||||
|
}); |
||||
|
|
||||
|
refreshButton.addEventListener("click", async () => { |
||||
|
await loadWorkspace(); |
||||
|
}); |
||||
|
|
||||
|
logoutButton.addEventListener("click", async () => { |
||||
|
try { |
||||
|
await apiFetch("/api/auth/logout", { method: "POST" }); |
||||
|
} catch { |
||||
|
// best-effort logout
|
||||
|
} |
||||
|
clearToken(); |
||||
|
showLogin(); |
||||
|
}); |
||||
|
|
||||
|
if (getToken()) { |
||||
|
void loadWorkspace(); |
||||
|
} else { |
||||
|
showLogin(); |
||||
|
} |
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@ -0,0 +1,13 @@ |
|||||
|
<!doctype html> |
||||
|
<html lang="en"> |
||||
|
<head> |
||||
|
<meta charset="UTF-8" /> |
||||
|
<meta name="viewport" content="width=device-width, initial-scale=1.0" /> |
||||
|
<title>Iron</title> |
||||
|
<script type="module" crossorigin src="/assets/app.js"></script> |
||||
|
<link rel="stylesheet" crossorigin href="/assets/app.css"> |
||||
|
</head> |
||||
|
<body> |
||||
|
<div id="root"></div> |
||||
|
</body> |
||||
|
</html> |
||||
@ -0,0 +1,107 @@ |
|||||
|
<!doctype html> |
||||
|
<html lang="en"> |
||||
|
<head> |
||||
|
<meta charset="utf-8" /> |
||||
|
<meta name="viewport" content="width=device-width, initial-scale=1" /> |
||||
|
<title>Iron</title> |
||||
|
<link rel="stylesheet" href="/assets/styles.css" /> |
||||
|
</head> |
||||
|
<body> |
||||
|
<div class="page-shell"> |
||||
|
<aside class="hero-panel"> |
||||
|
<p class="eyebrow">Iron Gateway</p> |
||||
|
<h1>One personal drive, one calm control plane.</h1> |
||||
|
<p class="lede"> |
||||
|
Log in to browse your namespace, inspect storage health, and grow this gateway into |
||||
|
the multi-backend drive defined by the project MVP. |
||||
|
</p> |
||||
|
<div class="hero-card"> |
||||
|
<div> |
||||
|
<span class="label">Current shell</span> |
||||
|
<strong>Auth + browser entry</strong> |
||||
|
</div> |
||||
|
<div> |
||||
|
<span class="label">Next up</span> |
||||
|
<strong>Upload UI and S3 runtime path</strong> |
||||
|
</div> |
||||
|
</div> |
||||
|
</aside> |
||||
|
|
||||
|
<main class="workspace-panel"> |
||||
|
<section id="login-card" class="card"> |
||||
|
<div class="card-header"> |
||||
|
<p class="eyebrow">Local Login</p> |
||||
|
<h2>Enter the gateway</h2> |
||||
|
</div> |
||||
|
<form id="login-form" class="stack-form"> |
||||
|
<label> |
||||
|
<span>Username</span> |
||||
|
<input id="username" name="username" type="text" autocomplete="username" required /> |
||||
|
</label> |
||||
|
<label> |
||||
|
<span>Password</span> |
||||
|
<input |
||||
|
id="password" |
||||
|
name="password" |
||||
|
type="password" |
||||
|
autocomplete="current-password" |
||||
|
required |
||||
|
/> |
||||
|
</label> |
||||
|
<button type="submit">Sign in</button> |
||||
|
<p id="login-error" class="error" hidden></p> |
||||
|
</form> |
||||
|
</section> |
||||
|
|
||||
|
<section id="app-card" class="card" hidden> |
||||
|
<div class="toolbar"> |
||||
|
<div> |
||||
|
<p class="eyebrow">Workspace</p> |
||||
|
<h2 id="welcome-title">Signed in</h2> |
||||
|
</div> |
||||
|
<div class="toolbar-actions"> |
||||
|
<button id="refresh-button" type="button" class="ghost">Refresh</button> |
||||
|
<button id="logout-button" type="button" class="ghost">Logout</button> |
||||
|
</div> |
||||
|
</div> |
||||
|
|
||||
|
<div class="status-grid"> |
||||
|
<article class="status-card"> |
||||
|
<span class="label">Session</span> |
||||
|
<strong id="session-user">Unknown</strong> |
||||
|
</article> |
||||
|
<article class="status-card"> |
||||
|
<span class="label">Root Directory</span> |
||||
|
<strong id="root-name">/</strong> |
||||
|
</article> |
||||
|
<article class="status-card"> |
||||
|
<span class="label">Backends</span> |
||||
|
<strong id="backend-count">0</strong> |
||||
|
</article> |
||||
|
</div> |
||||
|
|
||||
|
<section class="panel-grid"> |
||||
|
<div class="subpanel"> |
||||
|
<div class="subpanel-header"> |
||||
|
<h3>Root Items</h3> |
||||
|
<span id="items-count" class="pill">0 items</span> |
||||
|
</div> |
||||
|
<ul id="root-items" class="item-list"></ul> |
||||
|
</div> |
||||
|
|
||||
|
<div class="subpanel"> |
||||
|
<div class="subpanel-header"> |
||||
|
<h3>Configured Backends</h3> |
||||
|
</div> |
||||
|
<ul id="backends-list" class="item-list"></ul> |
||||
|
</div> |
||||
|
</section> |
||||
|
|
||||
|
<p id="app-error" class="error" hidden></p> |
||||
|
</section> |
||||
|
</main> |
||||
|
</div> |
||||
|
|
||||
|
<script src="/assets/app.js" type="module"></script> |
||||
|
</body> |
||||
|
</html> |
||||
@ -0,0 +1,25 @@ |
|||||
|
from __future__ import annotations |
||||
|
|
||||
|
from pathlib import Path |
||||
|
|
||||
|
from fastapi import APIRouter |
||||
|
from fastapi import HTTPException |
||||
|
from fastapi.responses import FileResponse |
||||
|
|
||||
|
|
||||
|
router = APIRouter() |
||||
|
WEB_DIR = Path(__file__).resolve().parent |
||||
|
DIST_DIR = WEB_DIR / "dist" |
||||
|
INDEX_FILE = DIST_DIR / "index.html" |
||||
|
|
||||
|
|
||||
|
def _get_index_file() -> Path: |
||||
|
if not INDEX_FILE.exists(): |
||||
|
raise HTTPException(status_code=503, detail="Web UI assets are not built.") |
||||
|
return INDEX_FILE |
||||
|
|
||||
|
|
||||
|
@router.get("/app", include_in_schema=False) |
||||
|
@router.get("/app/{path:path}", include_in_schema=False) |
||||
|
async def web_app(_path: str = "") -> FileResponse: |
||||
|
return FileResponse(_get_index_file()) |
||||
@ -0,0 +1,245 @@ |
|||||
|
:root { |
||||
|
--bg: #f3efe6; |
||||
|
--surface: rgba(255, 252, 245, 0.88); |
||||
|
--surface-strong: #fffaf1; |
||||
|
--ink: #1e1b16; |
||||
|
--muted: #6d6457; |
||||
|
--line: rgba(58, 44, 23, 0.12); |
||||
|
--accent: #bb5a2b; |
||||
|
--accent-strong: #8b3d17; |
||||
|
--success: #2d6a4f; |
||||
|
--danger: #a63b2c; |
||||
|
--shadow: 0 18px 50px rgba(72, 44, 20, 0.12); |
||||
|
} |
||||
|
|
||||
|
* { |
||||
|
box-sizing: border-box; |
||||
|
} |
||||
|
|
||||
|
body { |
||||
|
margin: 0; |
||||
|
min-height: 100vh; |
||||
|
color: var(--ink); |
||||
|
background: |
||||
|
radial-gradient(circle at top left, rgba(219, 164, 103, 0.28), transparent 30%), |
||||
|
radial-gradient(circle at bottom right, rgba(92, 140, 109, 0.24), transparent 28%), |
||||
|
linear-gradient(135deg, #efe7d8 0%, #f8f4ec 45%, #ece3d5 100%); |
||||
|
font-family: Georgia, "Iowan Old Style", "Palatino Linotype", serif; |
||||
|
} |
||||
|
|
||||
|
button, |
||||
|
input { |
||||
|
font: inherit; |
||||
|
} |
||||
|
|
||||
|
.page-shell { |
||||
|
display: grid; |
||||
|
grid-template-columns: minmax(280px, 0.95fr) minmax(340px, 1.2fr); |
||||
|
min-height: 100vh; |
||||
|
} |
||||
|
|
||||
|
.hero-panel, |
||||
|
.workspace-panel { |
||||
|
padding: 48px 40px; |
||||
|
} |
||||
|
|
||||
|
.hero-panel { |
||||
|
display: flex; |
||||
|
flex-direction: column; |
||||
|
justify-content: center; |
||||
|
gap: 24px; |
||||
|
} |
||||
|
|
||||
|
.hero-panel h1 { |
||||
|
margin: 0; |
||||
|
font-size: clamp(2.3rem, 4vw, 4.4rem); |
||||
|
line-height: 0.95; |
||||
|
max-width: 10ch; |
||||
|
} |
||||
|
|
||||
|
.lede { |
||||
|
margin: 0; |
||||
|
max-width: 52ch; |
||||
|
color: var(--muted); |
||||
|
font-size: 1.08rem; |
||||
|
line-height: 1.7; |
||||
|
} |
||||
|
|
||||
|
.hero-card, |
||||
|
.card, |
||||
|
.status-card, |
||||
|
.subpanel { |
||||
|
border: 1px solid var(--line); |
||||
|
background: var(--surface); |
||||
|
backdrop-filter: blur(14px); |
||||
|
box-shadow: var(--shadow); |
||||
|
} |
||||
|
|
||||
|
.hero-card { |
||||
|
display: grid; |
||||
|
gap: 18px; |
||||
|
padding: 22px 24px; |
||||
|
border-radius: 24px; |
||||
|
max-width: 440px; |
||||
|
} |
||||
|
|
||||
|
.workspace-panel { |
||||
|
display: flex; |
||||
|
align-items: center; |
||||
|
justify-content: center; |
||||
|
} |
||||
|
|
||||
|
.card { |
||||
|
width: min(860px, 100%); |
||||
|
padding: 28px; |
||||
|
border-radius: 28px; |
||||
|
} |
||||
|
|
||||
|
.card-header h2, |
||||
|
.toolbar h2, |
||||
|
.subpanel-header h3 { |
||||
|
margin: 0; |
||||
|
} |
||||
|
|
||||
|
.eyebrow, |
||||
|
.label { |
||||
|
display: block; |
||||
|
margin-bottom: 8px; |
||||
|
color: var(--muted); |
||||
|
font-size: 0.8rem; |
||||
|
letter-spacing: 0.12em; |
||||
|
text-transform: uppercase; |
||||
|
} |
||||
|
|
||||
|
.stack-form { |
||||
|
display: grid; |
||||
|
gap: 18px; |
||||
|
} |
||||
|
|
||||
|
.stack-form label { |
||||
|
display: grid; |
||||
|
gap: 8px; |
||||
|
} |
||||
|
|
||||
|
input { |
||||
|
width: 100%; |
||||
|
padding: 14px 16px; |
||||
|
border: 1px solid rgba(61, 45, 24, 0.18); |
||||
|
border-radius: 14px; |
||||
|
background: var(--surface-strong); |
||||
|
} |
||||
|
|
||||
|
button { |
||||
|
border: none; |
||||
|
border-radius: 999px; |
||||
|
padding: 13px 18px; |
||||
|
color: #fffaf2; |
||||
|
background: linear-gradient(135deg, var(--accent) 0%, var(--accent-strong) 100%); |
||||
|
cursor: pointer; |
||||
|
transition: transform 140ms ease, opacity 140ms ease; |
||||
|
} |
||||
|
|
||||
|
button:hover { |
||||
|
transform: translateY(-1px); |
||||
|
} |
||||
|
|
||||
|
.ghost { |
||||
|
color: var(--ink); |
||||
|
background: rgba(255, 250, 241, 0.86); |
||||
|
border: 1px solid var(--line); |
||||
|
} |
||||
|
|
||||
|
.toolbar, |
||||
|
.subpanel-header { |
||||
|
display: flex; |
||||
|
align-items: center; |
||||
|
justify-content: space-between; |
||||
|
gap: 16px; |
||||
|
} |
||||
|
|
||||
|
.toolbar-actions { |
||||
|
display: flex; |
||||
|
gap: 12px; |
||||
|
} |
||||
|
|
||||
|
.status-grid, |
||||
|
.panel-grid { |
||||
|
display: grid; |
||||
|
gap: 16px; |
||||
|
margin-top: 22px; |
||||
|
} |
||||
|
|
||||
|
.status-grid { |
||||
|
grid-template-columns: repeat(3, minmax(0, 1fr)); |
||||
|
} |
||||
|
|
||||
|
.status-card, |
||||
|
.subpanel { |
||||
|
padding: 18px 20px; |
||||
|
border-radius: 22px; |
||||
|
} |
||||
|
|
||||
|
.panel-grid { |
||||
|
grid-template-columns: repeat(2, minmax(0, 1fr)); |
||||
|
} |
||||
|
|
||||
|
.item-list { |
||||
|
list-style: none; |
||||
|
margin: 16px 0 0; |
||||
|
padding: 0; |
||||
|
display: grid; |
||||
|
gap: 10px; |
||||
|
} |
||||
|
|
||||
|
.item-list li { |
||||
|
display: flex; |
||||
|
align-items: center; |
||||
|
justify-content: space-between; |
||||
|
gap: 16px; |
||||
|
padding: 12px 14px; |
||||
|
border-radius: 16px; |
||||
|
background: rgba(255, 250, 241, 0.78); |
||||
|
border: 1px solid rgba(61, 45, 24, 0.08); |
||||
|
} |
||||
|
|
||||
|
.item-meta { |
||||
|
color: var(--muted); |
||||
|
font-size: 0.92rem; |
||||
|
} |
||||
|
|
||||
|
.pill { |
||||
|
padding: 6px 10px; |
||||
|
border-radius: 999px; |
||||
|
background: rgba(187, 90, 43, 0.1); |
||||
|
color: var(--accent-strong); |
||||
|
font-size: 0.86rem; |
||||
|
} |
||||
|
|
||||
|
.error { |
||||
|
margin: 14px 0 0; |
||||
|
color: var(--danger); |
||||
|
} |
||||
|
|
||||
|
[hidden] { |
||||
|
display: none !important; |
||||
|
} |
||||
|
|
||||
|
@media (max-width: 960px) { |
||||
|
.page-shell { |
||||
|
grid-template-columns: 1fr; |
||||
|
} |
||||
|
|
||||
|
.hero-panel, |
||||
|
.workspace-panel { |
||||
|
padding: 28px 18px; |
||||
|
} |
||||
|
|
||||
|
.hero-panel h1 { |
||||
|
max-width: none; |
||||
|
} |
||||
|
|
||||
|
.status-grid, |
||||
|
.panel-grid { |
||||
|
grid-template-columns: 1fr; |
||||
|
} |
||||
|
} |
||||
@ -0,0 +1,10 @@ |
|||||
|
IRON_DATABASE_URL=sqlite+aiosqlite:////srv/iron/iron.db |
||||
|
IRON_BOOTSTRAP_USERNAME=admin |
||||
|
IRON_BOOTSTRAP_PASSWORD=replace-with-a-strong-password |
||||
|
IRON_SECRET_KEY=replace-with-a-long-random-secret |
||||
|
IRON_UPLOAD_TEMP_DIR=/srv/iron/runtime/temp |
||||
|
IRON_LOCAL_STORAGE_DIR=/srv/iron/runtime/local |
||||
|
IRON_CACHE_DIR=/srv/iron/runtime/cache |
||||
|
IRON_AUTH_SESSION_TTL_HOURS=168 |
||||
|
IRON_JOB_POLL_INTERVAL_SECONDS=2 |
||||
|
IRON_JOB_BATCH_SIZE=10 |
||||
@ -0,0 +1,29 @@ |
|||||
|
#!/usr/bin/env bash |
||||
|
set -euo pipefail |
||||
|
|
||||
|
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" |
||||
|
ENV_FILE="${IRON_ENV_FILE:-$ROOT_DIR/deploy/iron/iron.env}" |
||||
|
HOST="${IRON_HOST:-127.0.0.1}" |
||||
|
PORT="${IRON_PORT:-8000}" |
||||
|
|
||||
|
cd "$ROOT_DIR" |
||||
|
|
||||
|
if [[ ! -f ".venv/bin/python" ]]; then |
||||
|
echo "missing .venv/bin/python; install dependencies first" >&2 |
||||
|
exit 1 |
||||
|
fi |
||||
|
|
||||
|
if [[ -f "$ENV_FILE" ]]; then |
||||
|
set -a |
||||
|
# shellcheck disable=SC1090 |
||||
|
source "$ENV_FILE" |
||||
|
set +a |
||||
|
fi |
||||
|
|
||||
|
mkdir -p \ |
||||
|
"${IRON_UPLOAD_TEMP_DIR:-./.iron-temp/uploads}" \ |
||||
|
"${IRON_LOCAL_STORAGE_DIR:-./.iron-storage/local}" \ |
||||
|
"${IRON_CACHE_DIR:-./.iron-cache}" |
||||
|
|
||||
|
./.venv/bin/alembic upgrade head |
||||
|
exec ./.venv/bin/python -m uvicorn app.main:app --host "$HOST" --port "$PORT" |
||||
@ -0,0 +1,3 @@ |
|||||
|
OPENLIST_BIND_HOST=127.0.0.1 |
||||
|
OPENLIST_PORT=5244 |
||||
|
OPENLIST_DATA_DIR=/srv/iron/openlist |
||||
@ -0,0 +1,9 @@ |
|||||
|
services: |
||||
|
openlist: |
||||
|
image: openlistteam/openlist:latest |
||||
|
container_name: openlist |
||||
|
restart: unless-stopped |
||||
|
ports: |
||||
|
- "${OPENLIST_BIND_HOST:-127.0.0.1}:${OPENLIST_PORT:-5244}:5244" |
||||
|
volumes: |
||||
|
- "${OPENLIST_DATA_DIR:-/srv/iron/openlist}:/opt/openlist/data" |
||||
@ -0,0 +1,17 @@ |
|||||
|
[Unit] |
||||
|
Description=Iron personal cloud drive gateway |
||||
|
After=network-online.target |
||||
|
Wants=network-online.target |
||||
|
|
||||
|
[Service] |
||||
|
Type=simple |
||||
|
User=iron |
||||
|
Group=iron |
||||
|
WorkingDirectory=/opt/iron |
||||
|
EnvironmentFile=/opt/iron/deploy/iron/iron.env |
||||
|
ExecStart=/opt/iron/deploy/iron/start_iron.sh |
||||
|
Restart=on-failure |
||||
|
RestartSec=3 |
||||
|
|
||||
|
[Install] |
||||
|
WantedBy=multi-user.target |
||||
@ -0,0 +1,15 @@ |
|||||
|
# Documentation |
||||
|
|
||||
|
This directory contains the stable project documentation for Iron. |
||||
|
|
||||
|
- [Product](product.md): product scope, current stage, MVP cutline, and roadmap. |
||||
|
- [Architecture](architecture.md): backend, frontend, data model, storage, and job design. |
||||
|
- [API](api.md): route map and API groups. |
||||
|
- [Development](development.md): setup, test commands, Playwright workflow, and documentation rules. |
||||
|
- [OpenList + Iron Local Deployment SOP](openlist-local-deployment-sop.md): step-by-step local setup for OpenList, Aliyun Drive, WebDAV, and Iron. |
||||
|
|
||||
|
Root-level project docs: |
||||
|
|
||||
|
- [README](../README.md): user-facing overview and quick start. |
||||
|
- [Contributing](../CONTRIBUTING.md): contributor workflow. |
||||
|
- [Agents](../AGENTS.md): coding-agent and automation guidance. |
||||
@ -0,0 +1,108 @@ |
|||||
|
# API |
||||
|
|
||||
|
Iron exposes JSON APIs under `/api` and `tus` upload transport under `/files`. |
||||
|
|
||||
|
Most `/api` routes require: |
||||
|
|
||||
|
```text |
||||
|
Authorization: Bearer <token> |
||||
|
``` |
||||
|
|
||||
|
## Auth |
||||
|
|
||||
|
- `POST /api/auth/login` |
||||
|
- `POST /api/auth/logout` |
||||
|
- `GET /api/auth/me` |
||||
|
|
||||
|
## Directories |
||||
|
|
||||
|
- `GET /api/directories/{directory_id}/children` |
||||
|
- `POST /api/directories` |
||||
|
- `POST /api/directories/{directory_id}/rename` |
||||
|
- `POST /api/directories/{directory_id}/move` |
||||
|
|
||||
|
## Files |
||||
|
|
||||
|
- `GET /api/files/{file_id}` |
||||
|
- `GET /api/files/{file_id}/download` |
||||
|
- `GET /api/files/{file_id}/preview` |
||||
|
- `GET /api/files/{file_id}/stream` |
||||
|
- `POST /api/files/{file_id}/rename` |
||||
|
- `POST /api/files/{file_id}/move` |
||||
|
- `DELETE /api/files/{file_id}` |
||||
|
- `POST /api/files/{file_id}/restore` |
||||
|
- `GET /api/files/recycle-bin` |
||||
|
|
||||
|
## Uploads |
||||
|
|
||||
|
Protocol endpoints: |
||||
|
|
||||
|
- `POST /files` |
||||
|
- `HEAD /files/{upload_id}` |
||||
|
- `PATCH /files/{upload_id}` |
||||
|
|
||||
|
Finalize endpoint: |
||||
|
|
||||
|
- `POST /api/uploads/{upload_id}/finalize` |
||||
|
|
||||
|
## Backends |
||||
|
|
||||
|
- `GET /api/backends` |
||||
|
- `POST /api/backends` |
||||
|
- `PATCH /api/backends/{backend_id}` |
||||
|
- `POST /api/backends/{backend_id}/check` |
||||
|
- `POST /api/backends/{backend_id}/disable` |
||||
|
- `POST /api/backends/{backend_id}/enable` |
||||
|
- `DELETE /api/backends/{backend_id}` |
||||
|
|
||||
|
Backend `type` values: |
||||
|
|
||||
|
- `local_directory` |
||||
|
- `s3` |
||||
|
- `webdav` |
||||
|
|
||||
|
Backend responses include public `config` values plus a `secret_fields` list for |
||||
|
configured credentials. Secret fields such as S3 credentials and WebDAV |
||||
|
passwords are never echoed back in clear text. |
||||
|
Backend updates preserve existing secret fields when they are omitted from the |
||||
|
request config, and replace them only when a new value is provided. |
||||
|
WebDAV configs should set `root_path` so Iron uses a dedicated folder tree |
||||
|
inside the remote endpoint, which is a good fit for OpenList-managed cloud |
||||
|
drives. |
||||
|
Backend deletion is guarded: the backend must be disabled first, must not be the |
||||
|
default local backend, must not still store blob replicas, and must not be |
||||
|
referenced by placement policies. |
||||
|
|
||||
|
## Jobs |
||||
|
|
||||
|
- `GET /api/jobs` |
||||
|
- `GET /api/jobs/{job_id}` |
||||
|
- `POST /api/jobs/{job_id}/retry` |
||||
|
- `POST /api/jobs/run-pending` |
||||
|
- `POST /api/jobs/enqueue-health-checks` |
||||
|
- `POST /api/jobs/enqueue-full-reconcile` |
||||
|
|
||||
|
## Placement Policies |
||||
|
|
||||
|
- `GET /api/policies/placement` |
||||
|
- `PUT /api/policies/placement/{file_class}` |
||||
|
- `GET /api/policies/placement/preview` |
||||
|
|
||||
|
Saving a placement policy also enqueues one full reconcile job so desired |
||||
|
replica placement converges without a separate manual step. |
||||
|
|
||||
|
## Metadata Export And Recovery |
||||
|
|
||||
|
- `GET /api/exports/metadata` |
||||
|
- `POST /api/exports/metadata/validate` |
||||
|
- `GET /api/exports/metadata/integrity` |
||||
|
- `POST /api/exports/metadata/restore-plan` |
||||
|
- `POST /api/exports/metadata/import` |
||||
|
|
||||
|
Metadata import requires `confirm_replace=true` and a validation token from the |
||||
|
restore-plan endpoint. |
||||
|
|
||||
|
## System |
||||
|
|
||||
|
- `GET /api/system/health` |
||||
|
- `GET /api/system/ready` |
||||
@ -0,0 +1,158 @@ |
|||||
|
# Architecture |
||||
|
|
||||
|
Iron is an async Python modular monolith with a React single-page Web app. |
||||
|
|
||||
|
## System Overview |
||||
|
|
||||
|
```text |
||||
|
Browser UI |
||||
|
| |
||||
|
FastAPI app |
||||
|
| |
||||
|
Services |
||||
|
| |
||||
|
Repositories |
||||
|
| |
||||
|
SQLite metadata database |
||||
|
| |
||||
|
Storage adapters: local directory, S3, WebDAV |
||||
|
``` |
||||
|
|
||||
|
## Backend |
||||
|
|
||||
|
Core stack: |
||||
|
|
||||
|
- FastAPI |
||||
|
- SQLAlchemy 2.x async ORM |
||||
|
- SQLite via `aiosqlite` |
||||
|
- Alembic migrations |
||||
|
- durable in-process jobs stored in SQLite |
||||
|
|
||||
|
Important packages: |
||||
|
|
||||
|
- `app/api/`: route handlers and dependency wiring |
||||
|
- `app/services/`: business logic |
||||
|
- `app/repositories/`: database access |
||||
|
- `app/models/`: SQLAlchemy entities |
||||
|
- `app/schemas/`: API schemas |
||||
|
- `app/adapters/storage/`: storage adapter protocol, registry, and backend implementations |
||||
|
|
||||
|
## Frontend |
||||
|
|
||||
|
Core stack: |
||||
|
|
||||
|
- Vite |
||||
|
- React |
||||
|
- TypeScript |
||||
|
- React Router |
||||
|
- TanStack Query |
||||
|
|
||||
|
Source lives in `frontend/`. Built assets are emitted into `app/web/dist/` and |
||||
|
served by FastAPI at `/app`. |
||||
|
|
||||
|
## Data Model |
||||
|
|
||||
|
Main tables: |
||||
|
|
||||
|
- `users` |
||||
|
- `auth_sessions` |
||||
|
- `directories` |
||||
|
- `file_entries` |
||||
|
- `file_versions` |
||||
|
- `blobs` |
||||
|
- `blob_replicas` |
||||
|
- `backends` |
||||
|
- `upload_sessions` |
||||
|
- `upload_session_parts` |
||||
|
- `jobs` |
||||
|
- `placement_policies` |
||||
|
- `preview_artifacts` |
||||
|
- `cache_entries` |
||||
|
|
||||
|
Design principles: |
||||
|
|
||||
|
- logical namespace is separate from physical storage placement |
||||
|
- files have immutable content versions |
||||
|
- physical content is represented as blobs and replicas |
||||
|
- user-facing deletion uses recycle-bin semantics |
||||
|
- jobs are durable and retryable |
||||
|
- backend-specific configuration is validated before persistence and stays |
||||
|
behind adapter boundaries |
||||
|
|
||||
|
## Upload And Read Path |
||||
|
|
||||
|
Uploads use `tus` transport under `/files`. |
||||
|
|
||||
|
Flow: |
||||
|
|
||||
|
1. create upload session |
||||
|
2. append bytes with `PATCH` |
||||
|
3. finalize through the API |
||||
|
4. create file metadata, version, blob, and local replica |
||||
|
5. enqueue replication or preview jobs when needed |
||||
|
|
||||
|
Read path: |
||||
|
|
||||
|
1. resolve file and current version |
||||
|
2. find the best local or cached blob |
||||
|
3. if needed, materialize a ready remote replica into cache |
||||
|
4. serve download, preview, or range stream |
||||
|
|
||||
|
## Browser File Access |
||||
|
|
||||
|
JSON APIs use bearer-token authorization through the shared frontend API client. |
||||
|
|
||||
|
Browser preview and download must not use naked `/api/files/...` media URLs in |
||||
|
`img`, `iframe`, `video`, or `window.open`. Fetch file content with |
||||
|
authorization, create an object URL, use it, then revoke it. |
||||
|
|
||||
|
This is covered by the Playwright E2E flow. |
||||
|
|
||||
|
## Jobs |
||||
|
|
||||
|
Jobs are stored in SQLite and executed by the in-process job loop. |
||||
|
|
||||
|
Current job types include: |
||||
|
|
||||
|
- blob replication |
||||
|
- preview artifact generation |
||||
|
- backend health checks |
||||
|
- full reconcile |
||||
|
- file reconcile |
||||
|
|
||||
|
Every job should be safe to retry. |
||||
|
|
||||
|
## Storage Backends |
||||
|
|
||||
|
Supported backend types: |
||||
|
|
||||
|
- `local_directory`: a filesystem directory used for default local persistence |
||||
|
and low-latency reads |
||||
|
- `s3`: S3 protocol storage, including AWS S3, MinIO, and compatible |
||||
|
object stores |
||||
|
- `webdav`: WebDAV storage, including OpenList-managed cloud drive directories |
||||
|
exposed through a dedicated `root_path` |
||||
|
|
||||
|
Replica object keys are written into a structured application-owned layout such |
||||
|
as `objects/file/sha256/73/35/<hash>.blob`. This keeps remote storage organized |
||||
|
under a predictable object tree without trying to disguise replica content as |
||||
|
user-authored media files. During the current development stage, Iron treats |
||||
|
this layout as the only supported remote object format and does not preserve |
||||
|
compatibility with earlier experimental paths. |
||||
|
|
||||
|
Reserved object-tree namespaces: |
||||
|
|
||||
|
- `objects/file/...`: primary file blobs and replica payloads |
||||
|
- `objects/preview/<artifact_type>/...`: generated previews such as `inline_preview`, `thumbnail`, and `poster` |
||||
|
- `objects/export/metadata/YYYY/MM/DD/...`: metadata snapshot exports |
||||
|
- `objects/manifest/<scope>/...`: future manifest-style control files such as placement or repair manifests |
||||
|
|
||||
|
Backend config is normalized through typed schemas before it is stored. Public |
||||
|
config and encrypted secret config are persisted separately. `IRON_SECRET_KEY` |
||||
|
derives the encryption key for stored backend credentials. API responses include |
||||
|
public config plus a list of configured secret fields so operators can identify |
||||
|
a backend without exposing access keys, client secrets, or refresh tokens. Update |
||||
|
requests can change public config and provide secret overrides; omitted secret |
||||
|
fields keep their previous encrypted values. Disabled backends can be deleted |
||||
|
only when they are no longer referenced by policies and no blob replicas remain |
||||
|
stored on them. |
||||
Some files were not shown because too many files changed in this diff
Loading…
Reference in new issue