RefgetStore File Format

The RefgetStore is a directory-based file format for storing reference genome sequences with content-addressable access via GA4GH refget digests. It provides efficient storage, deduplication, and retrieval of sequences across multiple genome assemblies.

Overview

A RefgetStore is a self-contained directory that stores sequence data in individual files (one per sequence), sequence metadata (names, lengths, digests, alphabets), collection metadata (grouping sequences by genome assembly), and index files for efficient lookup.

Directory Structure

refget-store/
├── rgstore.json                  # Store metadata and configuration
├── sequences.rgsi                # refget sequence index file, for all sequences
├── collections.rgci              # refget collection index file
├── sequences/                    # Sequence data files
│   ├── Ab/                       # Subdirectories by digest prefix
│   │   ├── AbCdEf123....seq     # Individual sequence file
│   │   └── AbXyZ789....seq
│   └── Xy/
│       └── XyZabc456....seq
├── collections/                  # Collection metadata
│   ├── collection1.rgsi          # Each collection is represented as a refget sequence index file
│   ├── collection1.fhr.json      # FHR sidecar metadata file (optional)
│   └── collection2.rgsi
└── aliases/                      # Human-readable name mappings
    ├── sequences/                # Sequence alias namespaces
    │   ├── insdc.tsv
    │   └── genbank.tsv
    └── collections/              # Collection alias namespaces
        ├── ncbi.tsv
        └── ucsc.tsv

File Specifications

rgstore.json

The root metadata file containing store configuration.

Location: <store-root>/rgstore.json

Format: JSON

Schema:

{
  "version": 1,
  "seqdata_path_template": "sequences/%s2/%s.seq",
  "collections_path_template": "collections/%s.rgsi",
  "sequence_index": "sequences.rgsi",
  "collection_index": "collections.rgci",
  "mode": "Encoded",
  "created_at": "2025-01-15T10:30:00Z",
  "modified": "2025-03-02T18:04:11Z",
  "ancillary_digests": true,
  "attribute_index": false,
  "sequence_alias_namespaces": ["insdc", "genbank"],
  "collection_alias_namespaces": ["ncbi", "ucsc"],
  "sequences_digest": "9f2c...",
  "collections_digest": "1a7e...",
  "aliases_digest": "c30b...",
  "fhr_digest": "48d1...",
  "logical_sequence_bytes": 782345216
}

Fields:

• version (integer): Format version number (currently 1)
• seqdata_path_template (string): Template for sequence file paths
  • %s = full digest string
  • %s2 = first 2 characters of digest
  • %s4 = first 4 characters of digest
  • Example: "sequences/%s2/%s.seq" → "sequences/Ab/AbCdEf123....seq"
• collections_path_template (string): Template for collection file paths
  • Example: "collections/%s.rgsi"
• sequence_index (string): Path to the sequence metadata index file
  • Default: "sequences.rgsi"
• collection_index (string, optional): Path to the collection metadata index file
  • Default: "collections.rgci"
• mode (string): Storage mode for sequence data
  • "Raw": Uncompressed sequence data
  • "Encoded": Bit-packed encoded sequences (space efficient)
• created_at (string): ISO 8601 timestamp of store creation
• modified (string, optional): RFC 3339 timestamp, updated on every index write
• ancillary_digests (boolean): Whether the ancillary collection digests (name_length_pairs, sorted_name_length_pairs, sorted_sequences) are computed and stored
  • Default: true
• attribute_index (boolean): Whether an on-disk attribute reverse index is maintained
  • Default: false
• sequence_alias_namespaces (array of strings, optional): List of available sequence alias namespace names (e.g., ["insdc", "genbank"])
• collection_alias_namespaces (array of strings, optional): List of available collection alias namespace names (e.g., ["ncbi", "ucsc"])
• sequences_digest (string, optional): SHA-256 digest of sequences.rgsi
• collections_digest (string, optional): SHA-256 digest of collections.rgci
• aliases_digest (string, optional): SHA-256 digest of the combined alias data
• fhr_digest (string, optional): SHA-256 digest of the combined FHR sidecar data
• logical_sequence_bytes (integer, optional): Total size in bytes of all sequence payloads, computed at index-write time as each sequence’s length converted through the storage mode’s bits-per-symbol
  • Lets a client report the store’s data size without downloading sequences.rgsi
  • Excludes index files, alias files, FHR sidecars, and this manifest
  • Absent in manifests written before the field existed

The alias namespace lists are authoritative: a store loads exactly the alias namespaces the manifest advertises rather than scanning the aliases/ directory. This makes local and remote behavior identical, since remote access cannot list a directory.

sequences.rgsi

Master index of all sequences in the store.

Location: <store-root>/sequences.rgsi

Format: Tab-separated values (TSV)

Schema:

#name    length    alphabet    sha512t24u                md5                               description
chr1     248956422  dna2bit    AbCdEf123GhIjK...         a1b2c3d4e5f6...                   Homo sapiens chromosome 1
chr2     242193529  dna2bit    XyZabc456DefGh...         f7e8d9c0b1a2...
chrM     16569      dna2bit    MnOpQr789StUv...          1a2b3c4d5e6f...                   Mitochondrial DNA

The header line starts with # and defines column names.

Data Columns:

1. name (string): Sequence name (the first word from the FASTA header line)
2. length (integer): Sequence length in base pairs
3. alphabet (string): Alphabet type
   • dna2bit: 2-bit DNA encoding (ACGT only)
   • dna3bit: 3-bit DNA encoding (includes N)
   • dnaio: Full IUPAC DNA alphabet
   • protein: Protein sequences
   • ASCII: Generic ASCII sequences
4. sha512t24u (string): GA4GH SHA-512/24u digest (base64url, 32 chars)
   • Content-addressable identifier
   • Used as primary key for sequence lookup
5. md5 (string): MD5 digest (hex, 32 chars)
   • Enables cross-referencing with MD5-based systems
6. description (string, optional): Text from the FASTA header after the first word
   • May be empty; trailing whitespace is stripped

Each sequence occupies one line, with lines starting with # serving as comments or headers. Fields are tab-separated and no quoting is required since sequence names cannot contain tabs.

Sequence Files (.seq)

Individual sequence data files, one per sequence.

Location: Determined by seqdata_path_template in rgstore.json

Naming: Based on SHA-512/24u digest

Format: Binary

Content depends on storage mode: Raw mode stores plain sequence data as bytes (DNA as ASCII characters like A, C, G, T, N; protein as A, R, N, D, C, etc.) that is directly readable as text, while encoded mode uses bit-packed sequence data (DNA 2-bit packs 4 nucleotides per byte for ACGT; DNA 3-bit stores ~2.67 nucleotides per byte including N) that is more space-efficient but requires decoding to read. For example, human chr1 (248 Mbp) takes ~248 MB in raw mode but only ~62 MB in encoded 2-bit mode (4× compression).

Collection Files (.rgsi)

Metadata files grouping sequences into collections (e.g., genome assemblies).

Location: <store-root>/collections/<collection-digest>.rgsi

Format: Tab-separated values (TSV) with header sections

Structure:

##seqcol_digest=uC_UorBNf3YUu1YIDainBhI94CedlNeH
##names_digest=zxcvbnmasdfghjkl
##sequences_digest=qwertyuiopasdfgh
##lengths_digest=poiuytrewqlkjhgf
##name_length_pairs_digest=abcdefghijklmnop
##sorted_name_length_pairs_digest=mnopqrstuvwxyzab
##sorted_sequences_digest=stuvwxyzabcdefgh
#name  length  alphabet  sha512t24u  md5  description
chr1  248956422  dna2bit  AbCdEf123GhIjK...  a1b2c3d4e5f6...  Homo sapiens chromosome 1
chr2  242193529  dna2bit  XyZabc456DefGh...  f7e8d9c0b1a2...

The header section uses ## (double hash) for collection-level metadata headers. These include:

• ##seqcol_digest: The GA4GH sequence collection digest (level 1 digest)
• ##names_digest: Digest of the names array
• ##sequences_digest: Digest of the sequences array
• ##lengths_digest: Digest of the lengths array
• ##name_length_pairs_digest: Digest of the name_length_pairs array
• ##sorted_name_length_pairs_digest: Digest of the sorted_name_length_pairs array (coordinate system identifier)
• ##sorted_sequences_digest: Digest of the sorted_sequences array

The data section header uses # (single hash) and is tab-separated.

Data Section: Same format as sequences.rgsi (6 columns including description), but only the sequences in this collection.

Alias Files (.tsv)

Alias files map human-readable names to sequence or collection digests, organized by namespace.

Location: <store-root>/aliases/sequences/<namespace>.tsv or <store-root>/aliases/collections/<namespace>.tsv

Format: Tab-separated values (TSV), two columns

Schema:

alias  digest
chr1  AbCdEf123GhIjKlMnOpQrStUvWxYzAb
NC_000001.11  AbCdEf123GhIjKlMnOpQrStUvWxYzAb
hg38  uC_UorBNf3YUu1YIDainBhI94CedlNeH

Each row maps one alias string (column 1) to one digest (column 2), tab-separated. No header line is used. Multiple aliases from different namespaces can point to the same digest.

Namespace directories:

• aliases/sequences/ — contains one file per namespace for sequence aliases (e.g., insdc.tsv, genbank.tsv)
• aliases/collections/ — contains one file per namespace for collection aliases (e.g., ucsc.tsv, ncbi.tsv)

FHR Sidecar Files (.fhr.json)

FHR (FAIR Headers Reference genome) sidecar files attach structured metadata to a collection, following the FHR specification.

Location: <store-root>/collections/<collection-digest>.fhr.json

Format: JSON

Example:

{
  "schema_version": 1,
  "accessionID": {"id": "GCA_000001405.15", "source": "NCBI"},
  "taxon": [{"name": "Homo sapiens", "taxid": "9606"}],
  "description": "GRCh38 human reference genome",
  "scholarly_article": "https://doi.org/10.1038/nature15393",
  "funding": "National Human Genome Research Institute"
}

Each collection may have at most one .fhr.json sidecar. The sidecar is loaded automatically when the collection is opened, and an empty metadata object is stored if no sidecar file exists (zero-cost). See the FHR specification for the full field list.

Storage Modes

RefgetStore supports two storage modes: Raw mode stores sequences as plain text (for DNA/protein), making them simple to debug and inspect with no decoding overhead, but results in larger file sizes without compression—use this when storage space is not a concern, you need human-readable sequences, or during debugging and development. Encoded mode provides 2-4× smaller file sizes through efficient bit-packing and faster I/O, though it requires decoding and is slightly more complex—use this for production deployments and storing large genomes where storage space matters.

For details on how RefgetStore’s bit-packed encoding works and how it compares to the UCSC 2bit format, see RefgetStore encoding.

Path Templates

Templates use placeholders to organize files hierarchically:

Sequence Path Templates

Pattern: sequences/%s2/%s.seq

Placeholders include %s (full 32-character digest), %s2 (first 2 characters), and %s4 (first 4 characters).

Example:

Digest: AbCdEf123GhIjKlMnOpQrStUvWxYzAb
Template: sequences/%s2/%s.seq
Result: sequences/Ab/AbCdEf123GhIjKlMnOpQrStUvWxYzAb.seq

Using digest prefixes prevents directories with millions of files, provides better filesystem performance, and cleaner organization. Common patterns include sequences/%s2/%s.seq (2-char prefix, 256 subdirectories), sequences/%s4/%s.seq (4-char prefix, 65,536 subdirectories), and sequences/%s.seq (flat structure, not recommended for large stores).

Content-Addressable Storage

RefgetStore uses content-addressable storage: sequences are identified by their digest (hash of content), not by name.

Benefits

Content-addressable storage enables deduplication by storing identical sequences only once, even when they appear in different assemblies (like chrM shared between GRCh38 and GRCh37). The digest-based approach ensures integrity by verifying that content hasn’t been corrupted, providing tamper-evident storage. Finally, it creates universal identifiers where the same sequence has the same digest everywhere, enabling distributed, federated stores that are portable across systems.

Example

GRCh38 chr1: sha512t24u = AbCdEf123...
GRCh37 chr1: sha512t24u = XyZabc456...  (different sequence)
GRCh38 chrM: sha512t24u = MnOpQr789...
GRCh37 chrM: sha512t24u = MnOpQr789...  (same sequence as GRCh38!)

Only 3 sequence files needed, even though we have 4 sequence references.

GA4GH Compliance

RefgetStore implements the GA4GH refget specification, using SHA-512/24u digests (truncated SHA-512, base64url encoded) and supporting both Level 1 and Level 2 sequence collection digests.

Using RefgetStore

For Python API usage (creating, loading, querying, and exporting), see the RefgetStore tutorial. For API reference documentation, see the Python API reference.

Distribution

Local Distribution

Package the entire directory and distribute:

tar -czf refget-store.tar.gz /path/to/refget-store/

Remote Distribution

Host on any static file server or object storage:

# S3
aws s3 sync /path/to/refget-store/ s3://bucket/refget-store/

# HTTP server
python -m http.server -d /path/to/refget-store/

Remote access provides lazy loading (only downloading sequences when requested), user-controlled caching (you specify where cached data is stored), bandwidth efficiency (only transferring needed data), and selective downloads (skipping sequences you don’t need). See the RefgetStore tutorial for Python API examples of connecting to remote stores.

Cache Directory

When a remote store is accessed, a local cache directory mirrors the remote store structure. The cache location is user-specified (not automatic), giving explicit control over disk usage and placement.

The cache directory has the same on-disk layout as any RefgetStore:

• rgstore.json and collections.rgci are downloaded on the initial connection
• sequences.rgsi is deferred: it is downloaded the first time a sequence is requested, not at connection time. For a large store this file reaches tens of megabytes, and browsing or listing collections does not need it. Use the manifest’s logical_sequence_bytes to report the store’s data size before the sequence index has been fetched.
• sequences/ and collections/ files are downloaded on-demand only when a specific sequence or collection is first accessed
• The cache can be shared between processes and is cleaned up by simply deleting the directory

See How RefgetStore defers loading for the record-level and index-level deferral rules this layout supports.

Design Rationale

Why separate files per sequence?

Separate files per sequence enable selective memory mapping (mmap only the sequences you need, not an entire archive), automatic deduplication (identical sequences naturally share the same digest-named file), and simplified remote access (download only the specific sequence files you need with standard HTTP range requests). The key advantage over indexed single files is granular resource management—you can load, cache, and mmap individual sequences independently, which is particularly beneficial for distributed storage systems, content delivery networks, and partial synchronization where you don’t want to handle a monolithic file.

Why use digest prefixes in paths?

Digest prefixes avoid filesystem limits (directories with millions of files), improve directory lookup performance, and make it easier to shard across multiple servers or buckets.

Why support both Raw and Encoded?

Supporting both modes provides flexibility to trade space for simplicity, with raw mode easier to debug during development and encoded mode providing efficiency for production.

Why include MD5?

MD5 support provides compatibility with legacy systems, easier migration from MD5-based systems, and cross-referencing between old and new identifiers.

See Also

• GA4GH refget specification
• RefgetStore encoding — bit-packing details and comparison with UCSC 2bit
• RefgetStore Python API
• RefgetStore tutorial