Skip to content

Upgrade from legacy refgenie

This guide moves a refgenie 0.x setup (config versions 0.3 and 0.4) to refgenie 1.0. The refgenie-upgrade tool reads your old config, registers your genomes and assets in a new 1.0 database, and links your existing asset files, so you do not have to rebuild anything. For old commands and their 1.0 forms, and changes that affect code and pipelines, see the migration reference.

You need:

  • Your 0.x config file (the YAML file that $REFGENIE points to)
  • Python 3.11 or higher

Back up the config:

Terminal window
cp $REFGENIE genome_config_backup.yaml

Install the upgrade tool from GitHub. It also installs refgenie 1.0 if you do not have it.

Terminal window
pip install git+https://github.com/refgenie/refgenie-upgrade.git
  • You built assets locally (you ran refgenie build, have FASTA files on disk, or added custom assets): follow the steps below.
  • You only pulled assets from a server: skip to Start fresh.

A dry run shows what the tool will migrate, without changing anything:

Terminal window
refgenie-upgrade /path/to/genome_config.yaml --dry-run

It lists the genomes, assets, and tags it will migrate, warns about missing files, and ends with a summary such as:

Summary: 2 genome(s) (5 asset(s)) would be migrated

If a genome is skipped because its FASTA file is missing, see Troubleshooting.

Terminal window
refgenie-upgrade /path/to/genome_config.yaml

If $REFGENIE is set, you can leave out the path. The tool:

  1. Creates a new refgenie 1.0 instance (database and genome folder).
  2. Computes GA4GH digests for your genomes.
  3. Registers your genome aliases, like hg38.
  4. Links your existing asset files into the new folder layout.

By default the new instance goes in a refgenie1_data folder next to your old genome folder. Use --target-dir to put it somewhere else:

Terminal window
refgenie-upgrade genome_config.yaml --target-dir /data/refgenie

By default the tool symlinks to your original files and leaves them in place. You can copy or move them instead:

Terminal window
refgenie-upgrade genome_config.yaml --symlink # default: symlink to originals
refgenie-upgrade genome_config.yaml --copy # copy files (doubles disk use)
refgenie-upgrade genome_config.yaml --move # move files (frees the old space)

Step 3: Point refgenie at the new instance

Section titled “Step 3: Point refgenie at the new instance”

When it finishes, the tool prints the line to add to your shell profile (.bashrc, .zshrc, etc.), for example:

Terminal window
export REFGENIE_HOME_PATH=/data/refgenie1_data

Also remove $REFGENIE from your shell profile. Refgenie 1.0 does not read it and prints a warning while it is set.

0.x setting1.0 settingPurpose
$REFGENIE$REFGENIE_HOME_PATH (or $REFGENIE_DB_CONFIG_PATH)Where refgenie finds its configuration
genome_folder (in YAML)$REFGENIE_GENOME_FOLDERWhere assets are stored
genome_archive_folder (in YAML)$REFGENIE_GENOME_STAGE_FOLDERWhere assets are staged for serving

See Configure for all settings.

Terminal window
refgenie list
refgenie seek hg38/fasta
refgenie seek hg38/bowtie2_index

seek returns paths in the human-readable alias tree. Add --abs for the digest-addressed path. See Retrieve paths to assets.

Seek keys ending in .dir no longer exist. In most cases, drop the suffix; aligner indexes now return the index prefix directly:

--genome-index { refgenie[sample.genome].bowtie2_index.dir }
--genome-index { refgenie[sample.genome].bowtie2_index }

The migration reference lists this and the other changes to seek keys, the Python API, looper integration, and plugins.

If you only pulled pre-built assets, set up a new instance and pull again from the public server:

Terminal window
pip install "refgenie>=1.0.0a3"
refgenie init
refgenie pull hg38/fasta

refgenie init subscribes to https://api.refgenie.org, which serves assets in the 1.0 format.

A genome was skipped because its FASTA is missing. The tool knows the digests of common genomes. For other genomes it needs the FASTA file on disk to compute the digest. Put the FASTA back and run the upgrade again. If your files have moved, point the tool at them with --genome-folder.

The upgrade stopped partway, for example because the disk filled up. Run it again. It skips assets that are already migrated.

Pulling a new asset onto a migrated genome fails. Your migrated FASTA keeps its original bytes. The FASTA on api.refgenie.org is rebuilt from the genome's sequences: all uppercase and wrapped at 80 bases per line. The sequences and genome digest are the same, but the files differ, so the two FASTA assets have different digests. This affects most legacy genomes, including hg38, hg19 and mm10. Server assets such as bowtie2_index list the server's FASTA as their parent, so pulling one stops with an error that names both FASTAs. Either replace your FASTA with the server's (refgenie pull hg38/fasta --force, then pull the asset you wanted), or keep your FASTA, for example for its soft-masking, and build the asset locally with refgenie build.

Some things are not migrated. The tool skips assets stored at absolute paths (symlinked child assets). It recomputes asset digests with SHA-256 instead of keeping the old MD5 digests. Build logs, build recipes, archive and server settings, and download statistics stay in your old config file and genome folder.

After everything works:

  1. Delete the old genome_config.yaml (you have a backup).
  2. Uninstall the old packages: pip uninstall refgenconf refgenieserver.
  3. Make sure $REFGENIE is gone from your shell profile.

If you used the default --symlink mode, keep your old genome folder: the new instance links to the files in it.