Workbench Overview

Warning

This page is a read-only mirror of /Users/srvo/dev/workbench/workbench.qmd. Update the canonical file and rerun python scripts/sync_mirrors.py.

1 Components

flowchart LR
    Backtest[backtest] -->|depends on| Exclusions[exclusions-manager]
    Backtest --> API[workbench-api]
    API --> UI[workbench-ui]
    Hooks --> API
    Hooks --> Backtest
    Backtest --> S3[(S3: rawls/data)]
    Exclusions --> S3
    API --> MotherDuck[(MotherDuck)]

backtest/
- Python research runtime (VectorBT, DuckDB catalog, factorial runs)
- Typer CLI (python -m backtest), Makefile wrappers now thin shims
- Consumes exclusions-manager (editable uv source) for config and DB APIs
- Pushes results to MotherDuck (sync_* scripts) and object storage snapshots
exclusions-manager/
- Canonical exclusions toolkit (ingestion, CLI, config)
- Python package + CLI entry points (exclusions-sync, exclusions-monthly-sync, etc.)
- Shared config resolves DuckDB path (/srv/data/exclusions/... on rawls)
- Supplies config helpers to backtest/API and exports docs/centralization plan
workbench-api/
- Flask service powering the 3-panel workbench
- Makes direct DuckDB connections (currently evolving toward the shared config)
- Depends on catalog outputs (Sharadar, exclusions, results) and reuses backtest utilities for certain tasks
workbench-ui/
- Vite/React front-end; npm scripts orchestrate build/lint/test
- Communicates with workbench-api for data
dev/labs/hooks/
- Operational scripts (n8n, LACRM, Cal.com) plus automation docs
- Several scripts sync data back into the API/backtest systems (e.g., schema exports)

2 Data & Storage

flowchart TD
    subgraph Rawls
        A[/srv/data/exclusions/]
        B[/srv/data/exclusions/logs/]
        C[/srv/data/exclusions/sync_reports/]
        D[/srv/data/exclusions/review_queue/]
        BacktestCatalog[/home/srvo/dev/exclusions-manager]
    end
    A --> Backtest
    A --> API
    Backtest -->|aws s3 cp| S3[(s3://rawls/data/)]
    Exclusions -> S3
    Backtest -->|MotherDuck| MD[(MotherDuck workbench.main)]
    API --> MD

Local authoritative data (rawls) now under /srv/data/exclusions (symlinked from /home/srvo/data/...).
S3 bucket rawls.fsn1.your-objectstorage.com/rawls/data/ stores DuckDB snapshots; manual/cron sync uses Doppler + AWS CLI.
MotherDuck holds aggregations (results, exclusions_pit, catalog tables) accessed by both backtest jobs and the API.

3 Shared Tooling & Entry Points

Repo	Primary Commands	Notes
`backtest`	`python -m backtest ...` (Typer)	Groups: `setup`, `data`, `strategies`, `universe`, `tests`, `sync`; Makefile now delegates here.
`exclusions-manager`	`python -m exclusions_manager ...` or provided CLI entry points	Cron jobs call `uv run python monthly_afsc_sync.py` etc.; config module consumed by others.
`workbench-api`	`make run` / `python -m workbench_api.app`	Flask app; integration with new exclusions-manager config is WIP.
`workbench-ui`	`npm run dev`, `npm run build`	Vite/React front-end.
`hooks`	`./scripts/export-workflows.sh`, `uv run scripts/export_lacrm_schema.py`	Focused on ops integrations (Cal.com, LACRM, DocuSeal).

Backtest commands default to running on rawls via SSH. Pass --local (or set BACKTEST_REMOTE_HOST=) when operating directly on the server or targeting another host.

4 Git & Deployment Notes

backtest: pushes to github.com/srvo/backtest. README/CLI usage kept up to date. CI pending.
exclusions-manager: pushes to github.com/ethicalcapital/exclusions-manager. Rawls runs from ~/dev/exclusions-manager via uv sync.
workbench-api & workbench-ui: currently manual deployment; consider adding Typer/uv packaging for parity.
hooks: scripts expect environment secrets via keychain/Doppler; not packaged as CLI yet.

5 Future Work

Backtest CLI polish: richer subcommands (strategy run summaries, warehouse queries) and vectorbt.pro setup integration.
API/Backtest alignment: ensure workbench-api imports the shared exclusions config and uses updated paths (/srv/data/exclusions).
Unified top-level CLI: once each repo exposes reliable entry points, add a workbench command (Typer or just) to invoke sub-commands (workbench backtest ..., workbench exclusions ...).
Documentation sync: migrate any relevant material from archive/ directories into the active docs and prune stale guides.
Automation: schedule S3 snapshots and MotherDuck sync via cron/CLI wrappers to prevent drift.

Refer back to docs/ in each repo for deeper dives.