Rawls Operations Handbook

Warning

This page is a read-only mirror of /Users/srvo/dev/workbench/rawls.qmd. Update the canonical file and rerun python scripts/sync_mirrors.py.

1 Summary

rawls is the primary analysis and automation server. It hosts the backtest runtime, exclusions ingestion cron jobs, and associated tooling. The system doubles as the staging ground for environment changes before they propagate to object storage, MotherDuck, and other downstream services.

2 Hardware & Capacity

Provider: Hetzner (ARM-based Ubuntu 24.04.3 LTS)
CPU: multi-core ARM64 (sufficient for VectorBT workloads)
Memory: sized for DuckDB operations and UV-managed environments
Storage:
- /srv/data/exclusions (primary exclusions DuckDB, logs, sync artifacts)
- /home/srvo/dev/ (clones of git repos: exclusions-manager, backtest, workbench-api, hooks, etc.)
- /home/srvo/data/exclusions_legacy_YYYYMMDD/ retains historical snapshots for reference
Object Storage: https://fsn1.your-objectstorage.com/rawls/data/ receives zipped DuckDB and parquet snapshots via AWS CLI + Doppler credentials

3 Software Stack

OS: Ubuntu 24.04.3 LTS
Package managers: apt, uv (Python), npm (UI), uv-managed virtual environments (~/.local/bin/uv)
Python versions: 3.12.x (system), uv-managed per-repo envs (~/.venv, vectorbtpro env)
CLI utilities: Doppler CLI, AWS CLI v2, Git, jq, etc.

4 Repositories & Paths

Path	Purpose
`/home/srvo/dev/exclusions-manager`	Latest exclusions toolkit (uv sync). Cron jobs call scripts from here.
`/home/srvo/dev/backtest`	Latest backtest repo (uv sync, Typer CLI).
`/home/srvo/dev/workbench-api`	Flask API.
`/home/srvo/dev/workbench-ui`	React UI sources.
`/srv/data/exclusions`	Canonical exclusions data (DuckDB, review queue, sync reports, logs).
Symlinks under `/home/srvo/data/`	Compatibility links to `/srv/data/exclusions`.
`/home/srvo/data/exclusions_legacy_*`	Archived pre-centralization tree.

5 Automation & Cron

Current crontab (crontab -l):

0 2 1 * * – uv run python -m backtest strategies batch -- ... (monthly AFSC sync)
0 8 * * * – uv run python -m backtest universe refresh-monthly (daily review workflow)
30 2 1 * * – uv run python -m backtest universe refresh-monthly (monthly all-sources sync)
0 2 22 * * – uv run python -m backtest universe refresh-monthly (lunar-based sync)
Sharadar update job remains in /home/srvo/backtest/data (to be consolidated)

All jobs run under ~/dev/exclusions-manager with EXCLUSIONS_DB_PATH=/srv/data/exclusions/exclusions_unified.duckdb etc., relying on python -m backtest or direct script calls.

6 Security & Access

Authentication handled via SSH keys; root privileges available through sudo as needed.
Secrets managed via Doppler (project rawls, config dev), e.g., MOTHERDUCK_TOKEN, AWS_ACCESS_KEY_ID, etc.

Object storage commands use Doppler-wrapped AWS CLI:

doppler run --project rawls --config dev -- aws --endpoint-url https://fsn1.your-objectstorage.com s3 cp ...

Cloudflare Access protects external services (n8n, APIs). Scripts in dev/labs/hooks/ use service tokens from the macOS keychain when run locally.

7 Backups & Snapshots

DuckDB snapshot uploaded manually/cron: aws s3 cp /srv/data/exclusions/exclusions_unified.duckdb s3://rawls/data/exclusions_unified.duckdb
MotherDuck tables updated via CLI (python -m backtest sync ...).
Legacy archives preserved under /home/srvo/data/exclusions_legacy_YYYYMMDD/ until decommissioned.

8 Future Enhancements

Automate S3 sync via cron/CLI wrapper to ensure nightly backups without manual intervention.
Centralize remaining scripts in ~/dev/exclusions-manager or Typer CLI to reduce duplication in /home/srvo/data/exclusions_legacy_*.
Add monitoring/log rotation for /srv/data/exclusions/logs/ (rsyslog or similar).
Document new secrets or service tokens when onboarding additional integrations (MotherDuck staging, Appwrite, etc.).
Align Sharadar/Backtest data setup with the Typer CLI to eliminate legacy shell scripts.

Keep this doc updated as infrastructure evolves.