Upgrade from legacy refgenie
This guide moves a refgenie 0.x setup (config versions 0.3 and 0.4) to refgenie 1.0. The refgenie-upgrade tool reads your old config, registers your genomes and assets in a new 1.0 database, and links your existing asset files, so you do not have to rebuild anything. For old commands and their 1.0 forms, and changes that affect code and pipelines, see the migration reference.
Before you start
Section titled “Before you start”You need:
- Your 0.x config file (the YAML file that
$REFGENIEpoints to) - Python 3.11 or higher
Back up the config:
cp $REFGENIE genome_config_backup.yamlInstall the upgrade tool from GitHub. It also installs refgenie 1.0 if you do not have it.
pip install git+https://github.com/refgenie/refgenie-upgrade.gitWhich path is yours?
Section titled “Which path is yours?”- You built assets locally (you ran
refgenie build, have FASTA files on disk, or added custom assets): follow the steps below. - You only pulled assets from a server: skip to Start fresh.
Step 1: Preview the upgrade
Section titled “Step 1: Preview the upgrade”A dry run shows what the tool will migrate, without changing anything:
refgenie-upgrade /path/to/genome_config.yaml --dry-runIt lists the genomes, assets, and tags it will migrate, warns about missing files, and ends with a summary such as:
Summary: 2 genome(s) (5 asset(s)) would be migratedIf a genome is skipped because its FASTA file is missing, see Troubleshooting.
Step 2: Run the upgrade
Section titled “Step 2: Run the upgrade”refgenie-upgrade /path/to/genome_config.yamlIf $REFGENIE is set, you can leave out the path. The tool:
- Creates a new refgenie 1.0 instance (database and genome folder).
- Computes GA4GH digests for your genomes.
- Registers your genome aliases, like
hg38. - Links your existing asset files into the new folder layout.
By default the new instance goes in a refgenie1_data folder next to your old genome folder. Use --target-dir to put it somewhere else:
refgenie-upgrade genome_config.yaml --target-dir /data/refgenieBy default the tool symlinks to your original files and leaves them in place. You can copy or move them instead:
refgenie-upgrade genome_config.yaml --symlink # default: symlink to originalsrefgenie-upgrade genome_config.yaml --copy # copy files (doubles disk use)refgenie-upgrade genome_config.yaml --move # move files (frees the old space)Step 3: Point refgenie at the new instance
Section titled “Step 3: Point refgenie at the new instance”When it finishes, the tool prints the line to add to your shell profile (.bashrc, .zshrc, etc.), for example:
export REFGENIE_HOME_PATH=/data/refgenie1_dataAlso remove $REFGENIE from your shell profile. Refgenie 1.0 does not read it and prints a warning while it is set.
| 0.x setting | 1.0 setting | Purpose |
|---|---|---|
$REFGENIE | $REFGENIE_HOME_PATH (or $REFGENIE_DB_CONFIG_PATH) | Where refgenie finds its configuration |
genome_folder (in YAML) | $REFGENIE_GENOME_FOLDER | Where assets are stored |
genome_archive_folder (in YAML) | $REFGENIE_GENOME_STAGE_FOLDER | Where assets are staged for serving |
See Configure for all settings.
Step 4: Check the result
Section titled “Step 4: Check the result”refgenie listrefgenie seek hg38/fastarefgenie seek hg38/bowtie2_indexseek returns paths in the human-readable alias tree. Add --abs for the digest-addressed path. See Retrieve paths to assets.
Step 5: Update your pipelines
Section titled “Step 5: Update your pipelines”Seek keys ending in .dir no longer exist. In most cases, drop the suffix; aligner indexes now return the index prefix directly:
--genome-index { refgenie[sample.genome].bowtie2_index.dir }--genome-index { refgenie[sample.genome].bowtie2_index }The migration reference lists this and the other changes to seek keys, the Python API, looper integration, and plugins.
Start fresh if you only pulled assets
Section titled “Start fresh if you only pulled assets”If you only pulled pre-built assets, set up a new instance and pull again from the public server:
pip install "refgenie>=1.0.0a3"refgenie initrefgenie pull hg38/fastarefgenie init subscribes to https://api.refgenie.org, which serves assets in the 1.0 format.
Troubleshooting
Section titled “Troubleshooting”A genome was skipped because its FASTA is missing. The tool knows the digests of common genomes. For other genomes it needs the FASTA file on disk to compute the digest. Put the FASTA back and run the upgrade again. If your files have moved, point the tool at them with --genome-folder.
The upgrade stopped partway, for example because the disk filled up. Run it again. It skips assets that are already migrated.
Pulling a new asset onto a migrated genome fails. Your migrated FASTA keeps its original bytes. The FASTA on api.refgenie.org is rebuilt from the genome's sequences: all uppercase and wrapped at 80 bases per line. The sequences and genome digest are the same, but the files differ, so the two FASTA assets have different digests. This affects most legacy genomes, including hg38, hg19 and mm10. Server assets such as bowtie2_index list the server's FASTA as their parent, so pulling one stops with an error that names both FASTAs. Either replace your FASTA with the server's (refgenie pull hg38/fasta --force, then pull the asset you wanted), or keep your FASTA, for example for its soft-masking, and build the asset locally with refgenie build.
Some things are not migrated. The tool skips assets stored at absolute paths (symlinked child assets). It recomputes asset digests with SHA-256 instead of keeping the old MD5 digests. Build logs, build recipes, archive and server settings, and download statistics stay in your old config file and genome folder.
Clean up
Section titled “Clean up”After everything works:
- Delete the old
genome_config.yaml(you have a backup). - Uninstall the old packages:
pip uninstall refgenconf refgenieserver. - Make sure
$REFGENIEis gone from your shell profile.
If you used the default --symlink mode, keep your old genome folder: the new instance links to the files in it.