Developer Quickstart
Prerequisites
Section titled “Prerequisites”Clone and install
Section titled “Clone and install”git clone https://github.com/refgenie/refgenie.gitcd refgenieuv sync # coreuv sync --extra dash # plus the local web UIuv sync --extra server # plus the server (includes dash and MCP)Running from source
Section titled “Running from source”Run any refgenie command through uv. This sets up a database, syncs the public data channel, and builds the fasta asset for the small test genome:
uv run refgenie inituv run refgenie data-channel add refgenie https https://refgenie.github.io/refgenie-registry/index.yamluv run refgenie data-channel sync refgenie --exists-okuv run refgenie genome init --fasta tests/data/rCRSd.fa --name rCRSd -d "Human mitochondrial reference"uv run refgenie listrefgenie list then shows:
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━┓┃ Aliases ┃ Genome digest ┃ Asset group ┃ Asset ┃┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━┩│ rCRSd │ jthDpfNIgzM5AGJlOkRtfnky4rXMBIUP │ fasta │ default │└─────────┴──────────────────────────────────┴─────────────┴─────────┘To start over, uv run refgenie purge -f deletes the local database and
assets.
To register a genome from a remote RefgetStore instead, browse the store and init by seqcol digest:
uv run refgenie genome browse --server-url https://refgenie.s3.us-east-1.amazonaws.com/refget-store/jungle/uv run refgenie genome init \ --store https://refgenie.s3.us-east-1.amazonaws.com/refget-store/jungle/ \ --digest="2Ls1P5eUdKbvtOhjJx3s2R5r0_I-IB5Z" \ --name my_genomeuv run refgenie build my_genome/fastaIf a digest starts with -, write --digest="<DIGEST>" (with =) so it is not
read as a flag.
Running the web UI in dev mode
Section titled “Running the web UI in dev mode”The web UI is a React + Vite + TypeScript app in frontend/. Vite builds it
into refgenie/server/webui/, and the refgenie app serves it from there. A
source checkout has to build it once:
npm --prefix frontend installnpm --prefix frontend run buildThen start the local app:
uv run refgenie dash # or: task dashThis binds 127.0.0.1:8080 and opens your browser. Without a built bundle the
app still starts and the API still answers; / returns a 503 page telling you
to run the build.
For UI work, run the two halves separately so both hot-reload:
# terminal 1 -- the API, with auto-reloaduv run uvicorn "refgenie.server.main:create_local_app" --factory --reload --port 8080
# terminal 2 -- vite, proxying /v4, /v1 and /service-info to port 8080npm --prefix frontend run dev # or: task dash-devVite serves the UI on port 5173 and proxies API calls to the backend, so you edit TypeScript and Python at the same time without rebuilding either.
To catch base-path and hashed-asset problems the way CI would, do a
production-style preview instead: task web-build && uv run refgenie dash,
then open http://localhost:8080/.
To see which UI build a running server is serving, fetch GET /build-info.json
or read the web_ui block under refgenie in GET /service-info. Both report
the commit the bundle was built from, whether that checkout was clean, and the
build timestamp.
Running MCP server
Section titled “Running MCP server”uv sync --extra serveruv run refgenie-mcpRunning tests
Section titled “Running tests”task test runs the unit tier, and task ktest TARGET=alias runs tests whose
names match alias. The suite has four tiers (unit, component, e2e,
integration); see the tier table in CONTRIBUTING.md
for how to run each one. Integration tests run only through
./tests/scripts/test-integration.sh (or task integration), which starts and
stops the Docker services they need.
Filesystem side effects of the ORM
Section titled “Filesystem side effects of the ORM”Refgenie keeps state in three places that must agree: the SQLite catalog, the RefgetStore, and plain files on disk. Only the catalog has real transactions, so it is the participant that decides.
The rule: commit the catalog first, then clean up the filesystem, and make the cleanup safe to re-run. No code path may destroy managed data before the catalog agrees it is gone. This applies to the event listeners below and equally to any manager doing filesystem work of its own.
refgenie/db/events.py registers the mapper-level listeners (via
register_events()). The delete listeners run on flush, but they do not
touch the disk. They resolve the paths the delete makes obsolete (possible only
while the row is still present) and queue them through
refgenie/db/cleanup.py. Session-level listeners registered alongside them drain
that queue on after_commit and discard it on after_rollback /
after_soft_rollback.
So an ordinary
session.delete(asset)session.commit()still deletes files and directories on disk, but only once the COMMIT
succeeds. Read this table before writing code that inserts or deletes these rows.
| Model | Event | What it does on disk |
|---|---|---|
Asset | before_delete | Queues genome_folder / asset.path — the asset's content directory — for removal after commit. Logs a warning and proceeds if the directory is already gone. |
Alias | before_delete | Queues the trees alias_owned_paths names — alias/<name>/ and builds/<name>/ — for removal after commit. Only fires for SQL-backed aliases; the local store's aliases (_StoreAliases in managers/alias.py) remove the same paths themselves, after the store has dropped the alias. |
StagedAsset | before_delete | Queues, for after commit: mode="archive" the staged tarball; mode="file" the staging directory of per-file symlinks. Then prunes the parent group directory if it is empty. Originals in genome_folder are untouched. |
SeekKey | before_insert | Read-only on disk. Stats the target file / directory / prefix glob to populate seek_key.size. Raises ValueError for an unsupported SeekKeyType. |
| (session) | after_commit | Runs the queued removals. Failures are logged, never raised: the catalog is already correct, so a file left behind is drift to sweep, not a reason to fail a command that succeeded. Must not emit SQL. |
| (session) | after_rollback, after_soft_rollback | Discards the queue. |
No filesystem walks in listeners. A listener runs inside the flush, so a walk there holds the database write lock for as long as the walk takes, for work that has nothing to do with the transaction. Values that need a walk are computed before the insert, by the code that wrote the files:
- The staged tarball's
tarball_digest/tarball_sizeare computed inStageManager.create, right after the tarball is written. - An asset's
sizeis computed inAssetContentManager.add(utils.digest.directory_size), before the insert.
Consequences:
- A rollback never destroys data. Rows come back and the files were never
removed. A committed delete whose file removal failed leaves content with no
catalog row, which is inert: no read path reaches it, and the
write paths reclaim it (see
AssetManager._place_content,AssetPuller.pull). - The path listeners resolve
genome_folderfrom the newestconfigurationrow, not from the liveRefgenieobject. A test or script that pointsgenome_foldersomewhere unexpected will have these listeners act there. - Registration is global and idempotent.
register_events()guards on a module-level_events_registeredflag, so listeners attach once per process and stay attached for every engine and session in it.
This covers interruption of a single process. It does not coordinate concurrent writers in separate processes.
Code formatting
Section titled “Code formatting”task reformatThis runs ruff to format the codebase.
Building a wheel
Section titled “Building a wheel”uv buildThe wheel is created in the dist/ directory.
To validate the built package includes all required files:
./tests/scripts/test-package.shThis builds the wheel, installs it in an isolated environment, and runs smoke tests.
Task runner commands
Section titled “Task runner commands”All available task runner targets:
| Command | Description |
|---|---|
task test | Run unit tests |
task ktest TARGET=<pattern> | Run tests matching a pattern |
task integration | Run integration tests |
task lint | Lint with ruff |
task lint-fix | Lint with ruff and apply fixes |
task reformat | Format code with ruff |
task init | Initialize refgenie backend |
task purge | Purge backend and remove all assets |
task reinit | Purge then re-initialize |
task dash | Run the local web UI (serves the built bundle) |
task dash-dev | Run the vite dev server for the web UI |
task web-install | Install frontend dependencies (npm ci) |
task web-build | Build the web UI bundle into refgenie/server/webui/ |
task web-dev | Run the vite dev server for the web UI (alias of dash-dev) |
task web-check | Typecheck and lint the frontend |
task generate-snakefile | Generate snakemake pipeline |
task archive JOBS_COUNT=<n> | Run snakemake archive pipeline |
task alembic-revision MESSAGE=<msg> DB_CONN_STR=<url> | Create new DB migration |
task alembic-upgrade DB_CONN_STR=<url> | Upgrade DB schema to latest |
task build | Build package distribution |
task package | Build the wheel/sdist and validate them in an isolated venv |