Configure remote storage
A remote is cloud storage or a web host (such as an S3 bucket) where you upload staged assets so others can download them. You add a remote, mark staged assets for it, and run refgenie push.
Add a remote
Section titled “Add a remote”refgenie remote add \ --type s3 \ --prefix my-bucket/refgenie \ --name my-s3-remote \ --push-command "aws s3 cp {local_path} s3://{prefix}/{relative_path}"--typeishttp,https, ors3.--prefixis the base address where the files will be found after upload.--nameis what other commands use to refer to the remote (build --push-to,stage add --push-to,push --remote,remote status,remote remove). It must be unique and not all digits.--push-commandis the shell command refgenie runs to upload. A remote without one is skipped byrefgenie push.
The push command can use these placeholders:
| Placeholder | Value |
|---|---|
{local_path} | Full path to the staged file or folder |
{relative_path} | Its path inside the stage folder |
{prefix} | The remote's prefix |
{genome_stage_folder} | The stage folder (for the folder_sync strategy) |
refgenie push uploads assets that were staged with --push-to:
refgenie build hg38/fasta --stage --push-to my-s3-remoterefgenie stage add mm10/fasta --push-to my-s3-remote # an asset already builtrefgenie pushNarrow it down, or preview it:
refgenie push --genome hg38 # one genomerefgenie push --remote my-s3-remote # one remoterefgenie push --dry-run # show what would runRefgenie runs the remote's push command for each asset and records it as pushed. For s3 remotes it checks that the upload landed, and skips archives that are already in the bucket.
Sync the whole stage folder
Section titled “Sync the whole stage folder”The default strategy, per_asset, runs the push command once per asset. The folder_sync strategy runs it once per remote, so the command should copy the whole stage folder:
refgenie remote add \ --type s3 \ --prefix my-bucket/refgenie \ --name my-s3-sync \ --push-command "aws s3 sync {genome_stage_folder} s3://{prefix}/ --follow-symlinks"refgenie push --strategy folder_syncRefgenie then marks each waiting asset as pushed if its staged file exists.
List, check, and remove remotes
Section titled “List, check, and remove remotes”refgenie remote list # each remote and its pushed/unpushed countsrefgenie remote status # pushed/unpushed counts and unpushed digestsrefgenie remote status --remote my-s3-remoterefgenie remote remove my-s3-remote # by name or idRemotes and the server
Section titled “Remotes and the server”When a refgenie server knows an asset was pushed to a remote, it sends download requests to the remote's address instead of serving the file itself. The server then acts as a catalog while the cloud storage moves the data.