Building and serving tutorial
This tutorial follows an asset from start to finish: set up refgenie, register a genome, build assets, stage them, serve them, and push them to cloud storage. It uses a tiny genome (about 16 KB), so each step runs in seconds.
Refgenie handles an asset in three steps:
build → stage → push| Step | What happens | Result |
|---|---|---|
| build | Run a recipe | Asset files in the genome folder, and a database record |
| stage | Make a built asset ready to serve | A symlink (file mode) or .tgz (archive mode) in the stage folder |
| push | Upload staged assets to cloud storage | Files in the cloud, and a record of what was pushed |
You choose what to build, what to stage, and what to push.
Prerequisites
Section titled “Prerequisites”- Python 3.11 or newer
- refgenie (
pip install "refgenie>=1.0.0a3"; see Install) - Docker or Apptainer, to run each recipe's tools in its container
- For section 6:
pip install "refgenie[server]>=1.0.0a3" - For section 7:
pip install "refgenie[snakemake]>=1.0.0a3" - For section 8: a cloud command-line tool, such as
aws
1. Set up refgenie
Section titled “1. Set up refgenie”Create a configuration with its own genome folder and stage folder:
mkdir -p /tmp/rg_tutorial/{genomes,staged}
refgenie init \ --genome-folder /tmp/rg_tutorial/genomes \ --genome-stage-folder /tmp/rg_tutorial/stagedA new install has no recipes. Add the refgenie data channel to get them:
refgenie data-channel add refgenie https https://refgenie.github.io/refgenie-registry/index.yamlrefgenie data-channel sync refgenie --exists-okCheck the result with refgenie config get.
2. Register a genome
Section titled “2. Register a genome”Download rCRSd, a doubled copy of the human mitochondrial reference sequence:
curl -L https://raw.githubusercontent.com/refgenie/refgenie/master/tests/data/rCRSd.fa \ -o /tmp/rCRSd.faRegister it:
refgenie genome init \ --fasta /tmp/rCRSd.fa \ --name rCRSd \ --description "Revised Cambridge Reference Sequence (mitochondrial)" \ --species "Homo sapiens"Refgenie computes a digest for each sequence and stores the sequences in its RefgetStore, computes a digest for the whole genome, and sets the alias rCRSd to point to it. Because a fasta recipe is registered, it also builds the genome's fasta asset (FASTA file, index, and chromosome sizes). Add --no-build to skip that.
refgenie genome listrefgenie seek rCRSd/fasta # FASTA filerefgenie seek rCRSd/fasta.fai # FASTA indexrefgenie seek rCRSd/fasta.chrom_sizes # chromosome sizesSee the Genome initialization tutorial for other ways to register a genome.
3. Asset classes and recipes
Section titled “3. Asset classes and recipes”An asset class says what an asset contains (its seek keys and serving modes). A recipe says how to build one. See Asset classes and recipes.
refgenie asset-class listrefgenie recipe listrefgenie recipe requirements bowtie2_indexThe last command shows that bowtie2_index needs one input asset, a fasta, and an optional threads parameter. Recipes take three kinds of input:
| Input | Flag | Example |
|---|---|---|
| asset | --assets | --assets fasta=rCRSd/fasta |
| file | --files | --files gtf=/path/to/genes.gtf |
| parameter | --params | --params threads=4 |
4. Build an asset
Section titled “4. Build an asset”Build a bowtie2 index. Refgenie uses the genome's fasta asset as input, and --container runs bowtie2-build in the recipe's container image:
refgenie build rCRSd/bowtie2_index --containerTo name the input asset yourself:
refgenie build rCRSd/bowtie2_index --container --assets fasta=rCRSd/fasta:defaultAdd --stage to build and stage in one step:
refgenie build rCRSd/bwa_index --container --stagerefgenie list shows every asset you have.
5. Stage assets
Section titled “5. Stage assets”Staging prepares a built asset in the stage folder, in the form its asset class's serving modes ask for:
| Mode | What goes in the stage folder | Use |
|---|---|---|
file | A folder symlink to the built asset | Each file has its own download address |
archive | A .tgz archive | One download, for aligner indexes and other bundles |
none | Nothing | Metadata only; users build it themselves |
The fasta class uses both file and archive; bowtie2_index uses archive.
refgenie stage add rCRSd/fastarefgenie stage add rCRSd/bowtie2_indexls -la /tmp/rg_tutorial/staged/refgenie stage listTo unstage, run refgenie stage remove rCRSd/fasta. The built asset stays.
6. Serve assets
Section titled “6. Serve assets”refgenie serve # http://localhost:8000refgenie serve --port 9000Some of the endpoints:
| Endpoint | What it returns |
|---|---|
GET /v4/assets | All assets |
GET /v4/assets/{digest}/files | The files of a file-mode asset |
GET /v4/assets/{digest}/files/{path} | One file |
GET /v4/archives/{digest}/download | The .tgz of an archive-mode asset |
GET /ga4gh/drs/v1/objects/{id} | A GA4GH DRS object |
Get the address of a file without downloading it:
refgenie seekr rCRSd/fasta -s http://localhost:8000On another computer, subscribe to your server and pull as usual:
refgenie subscribe -s http://yourserver.example.comrefgenie pull rCRSd/fastaA pull downloads and unpacks the .tgz for archive-mode assets, downloads files one by one for file-mode assets, and fails with a message to build locally for none-mode assets.
7. Build many assets with PEP and Snakemake
Section titled “7. Build many assets with PEP and Snakemake”To build many assets for many genomes, list them in a PEP and let refgenie write a Snakemake workflow that builds them in parallel, parents first.
Make a project folder:
/tmp/rg_pep/ config.yaml # Snakemake config; may be empty pep/ config.yaml samples.csvpep/config.yaml:
pep_version: 2.1.0sample_table: samples.csv
sample_modifiers: derive: attributes: [fasta_file_path] sources: local: /tmp/{genome_name}.fapep/samples.csv has one row per asset to build. sample_name is the genome name, and asset_group_name must match a recipe name:
sample_name,genome_name,asset_group_name,fasta_file_pathrCRSd,rCRSd,fasta,localrCRSd,rCRSd,bowtie2_index,localrCRSd,rCRSd,bwa_index,localGenerate the Snakefile and run it from the project folder. The workflow requires REFGENIE_INPUTS to be set:
cd /tmp/rg_pepecho "{}" > config.yamlrefgenie generate snakefile --output-path Snakefileexport REFGENIE_INPUTS=/tmpsnakemake --jobs 4The Snakefile has a rule that registers each genome from its FASTA file, and one build rule per recipe that builds and stages the asset. It is written from the recipes registered when you generate it, so generate it again after you add recipes.
On a SLURM cluster, use a Snakemake profile with executor: slurm:
snakemake --jobs unlimited --workflow-profile /path/to/snakemake_profile_dirBy default the generated build rules run refgenie build without --container, so each recipe's tools must be on your PATH, for example through bulker. To run each recipe in its container image instead, add --container to refgenie generate snakefile.
8. Push to cloud storage
Section titled “8. Push to cloud storage”Pushing uploads staged assets to cloud storage (S3 and others) so the public can download them. refgenie push uploads the assets you staged with --push-to, so add the remote first:
refgenie remote add \ --type s3 \ --prefix my-bucket/refgenie \ --name production \ --push-command "aws s3 cp {local_path} s3://{prefix}/{relative_path}"Then build and stage with --push-to, and push:
refgenie build rCRSd/bowtie2_index --container --stage --push-to productionrefgenie push --dry-run # show what would runrefgenie pushrefgenie remote status # pushed and unpushed countsSee Configure remote storage for the push command placeholders and the folder_sync strategy, which uploads the whole stage folder in one command.
Summary
Section titled “Summary”# 1. Set uprefgenie init --genome-folder /path/to/genomes --genome-stage-folder /path/to/stagedrefgenie data-channel add refgenie https https://refgenie.github.io/refgenie-registry/index.yamlrefgenie data-channel sync refgenie --exists-ok
# 2. Register a genome (also builds its fasta asset)refgenie genome init --fasta rCRSd.fa --name rCRSd
# 3. Add a remoterefgenie remote add --type s3 --prefix bucket/refgenie --name production \ --push-command "aws s3 cp {local_path} s3://{prefix}/{relative_path}"
# 4. Build, stage, and mark for pushrefgenie build rCRSd/bowtie2_index --container --stage --push-to production
# 5. Serve locally, or push to the cloudrefgenie serverefgenie push| Command | What it does |
|---|---|
refgenie init | Create the configuration and database |
refgenie data-channel add / sync | Get asset classes and recipes |
refgenie genome init --fasta FILE --name NAME | Register a genome from a FASTA file |
refgenie recipe requirements NAME | Show a recipe's inputs |
refgenie build GENOME/ASSET | Build an asset |
refgenie build GENOME/ASSET --stage | Build and stage |
refgenie list | List local assets |
refgenie seek GENOME/ASSET | Get a local file path |
refgenie stage add / remove / list | Manage staged assets |
refgenie serve | Start a server |
refgenie seekr GENOME/ASSET -s URL | Get a file's address on a server |
refgenie generate snakefile -o FILE | Write a Snakemake build workflow |
refgenie remote add / list / status | Manage remotes |
refgenie push | Upload assets marked for push |