Refget Python API Documentation

FASTA Processing

fasta_to_seqcol_dict(fasta_file_path: Union[str, Path])

Convert a FASTA file into a Sequence Collection dict.

Args: fasta_file_path: Path to the FASTA file

Returns: dict: A canonical sequence collection dictionary

Raises: ImportError: If gtars is not installed (required for FASTA processing)

compare_seqcols(A: SeqColDict, B: SeqColDict)

Workhorse comparison function

Args: A: Sequence collection A B: Sequence collection B

Returns: dict: Following formal seqcol specification comparison function return value

calc_jaccard_similarities(A: SeqColDict, B: SeqColDict)

Takes two sequence collections and calculates jaccard similarties for all attributes

Args: A: Sequence collection A B: Sequence collection B

Returns: dict: Jaccard similarities for all attributes

validate_seqcol(seqcol_obj: SeqColDict, schema = None)

Validate a seqcol object against the seqcol schema. Returns True if valid, raises InvalidSeqColError if not, which enumerates the errors. Retrieve individual errors with exception.errors

validate_seqcol_bool(seqcol_obj: SeqColDict, schema = None)

Validate a seqcol object against the seqcol schema. Returns True if valid, False if not.

To enumerate the errors, use validate_seqcol instead.

FastAPI Integration

create_refget_router(sequences: bool = False, collections: bool = True, pangenomes: bool = False, fasta_drs: bool = False, compliance: bool = True, refget_store_url: str = None, mount_prefix: str = '')

Create a FastAPI router for the sequence collection API. This router provides endpoints for retrieving and comparing sequence collections. You can choose which endpoints to include by setting the sequences, collections, pangenomes, or fasta_drs flags.

Args: sequences (bool): Include sequence endpoints collections (bool): Include sequence collection endpoints pangenomes (bool): Include pangenome endpoints fasta_drs (bool): Include FASTA DRS endpoints refget_store_url (str): URL of backing RefgetStore (e.g., s3://bucket/store/) mount_prefix (str): The path prefix this router will be included under, when it is included with include_router(..., prefix=...) rather than mounted as a sub-application. Only used so the compliance endpoints self-target the seqcol service instead of the server root. Leave empty when mounting an app (scope["root_path"] covers it).

Returns: (APIRouter): A FastAPI router with the specified endpoints

Examples: app.include_router(create_refget_router(fasta_drs=True), prefix="/seqcol")

Client Classes

The client module provides interfaces for interacting with refget-compliant servers.

class SequenceClient(urls: list[str] = ['https://www.ebi.ac.uk/ena/cram'], raise_errors: Optional[bool] = None)

A client for interacting with a refget sequences API.

Properties

raise_errors

urls

Methods

get_metadata(digest: str)

Retrieves metadata for a given sequence digest.

Args: digest (str): The digest of the sequence.

Returns: (dict): The metadata.

get_sequence(digest: str, start: Optional[int] = None, end: Optional[int] = None)

Retrieves a sequence for a given digest.

Args: digest (str): The digest of the sequence.

Returns: (str): The sequence.

class SequenceCollectionClient(urls: list[str] = ['https://seqcolapi.databio.org'], raise_errors: Optional[bool] = None)

A client for interacting with a refget sequence collections API.

Properties

raise_errors

urls

Methods

aliases_for(digest: str, kind: str = 'collection')

Reverse lookup: list all (namespace, alias) pairs for a digest.

Args: digest (str): The digest to look up. kind (str): “collection” (default) or “sequence”.

Returns: (dict): {“digest”: …, “aliases”: [[namespace, alias], …]}.

build_chrom_sizes(digest: str)

Build a chrom.sizes file content for a sequence collection.

Format per line: NAME\tLENGTH

Args: digest (str): The sequence collection digest

Returns: (str): String content of the chrom.sizes file

build_fai(digest: str)

Build a complete .fai index file content for a FASTA.

FAI format per line: NAME\tLENGTH\tOFFSET\tLINEBASES\tLINEWIDTH

Args: digest (str): The sequence collection digest

Returns: (str): String content of the .fai file

compare(digest1: str, digest2: str)

Compares two sequence collections hosted on the server.

Args: digest1 (str): The digest of the first sequence collection. digest2 (str): The digest of the second sequence collection.

Returns: (dict): The JSON response containing the comparison of the two sequence collections.

compare_local(digest: str, local_collection: dict)

Compares a server-hosted sequence collection with a local collection.

Args: digest (str): The digest of the server-hosted sequence collection. local_collection (dict): A level 2 sequence collection representation.

Returns: (dict): The JSON response containing the comparison.

download_fasta(digest: str, dest_path: str = None, access_id: str = None)

Download the FASTA file to a local path.

Args: digest (str): The sequence collection digest dest_path (str, optional): Destination file path. If None, uses object name. access_id (str, optional): Specific access method to use. If None, tries all.

Returns: (str): Path to downloaded file

Raises: ValueError: If no access methods available or specified access_id not found

download_fasta_to_store(digest: str, store: 'RefgetStore', access_id: str = None, temp_dir: str = None, namespaces: Optional[list[str]] = None)

Download the FASTA file and import it into a RefgetStore.

This method downloads the FASTA file from the DRS endpoint and immediately imports it into the provided RefgetStore, enabling local sequence retrieval by digest without re-downloading.

Args: digest (str): The sequence collection digest store (RefgetStore): The RefgetStore instance to import into access_id (str, optional): Specific access method to use. If None, tries all. temp_dir (str, optional): Directory for temporary download. If None, uses system temp.

Returns: (str): The collection digest of the imported sequences

Raises: ValueError: If no access methods available or specified access_id not found ImportError: If gtars/RefgetStore is not available

Example: >>> from refget.store import RefgetStore >>> from refget.clients import SequenceCollectionClient >>> store = RefgetStore.in_memory() >>> client = SequenceCollectionClient() >>> collection_digest = client.download_fasta_to_store(“abc123”, store) >>> # Now you can retrieve sequences by digest from the local store >>> seq = store.get_substring(sequence_digest, 0, 100)

get_attribute(attribute: str, digest: str)

Retrieves a specific attribute value by its digest.

Args: attribute (str): The attribute name (e.g., “names”, “lengths”, “sequences”). digest (str): The level 1 digest of the attribute.

Returns: (dict): The JSON response containing the attribute value.

get_collection(digest: str, level: int = 2)

Retrieves a sequence collection for a given digest and detail level.

Args: digest (str): The digest of the sequence collection. level (int, optional): The level of detail for the sequence collection. Defaults to 2.

Returns: (dict): The JSON response containing the sequence collection.

get_fasta(digest: str)

Get DRS object metadata for a FASTA file.

Args: digest (str): The sequence collection digest (which is also the DRS object ID)

Returns: (dict): DRS object with id, self_uri, size, checksums, access_methods, etc.

get_fasta_index(digest: str)

Get FAI index data for a FASTA file.

Args: digest (str): The sequence collection digest

Returns: (dict): Dict with line_bases, extra_line_bytes, offsets

get_fhr(digest: str)

Get FHR metadata for a collection.

Args: digest (str): The collection digest.

Returns: (dict): FHR metadata, or None if not found.

get_refget_store(cache_dir: str)

Get a RefgetStore instance connected to the server’s backing store.

Args: cache_dir (str): Local directory for caching store data

Returns: (RefgetStore): RefgetStore instance loaded from remote

Raises: ValueError: If server doesn’t have a RefgetStore configured ImportError: If gtars is not installed

get_refget_store_url()

Discover RefgetStore URL from service-info if available.

Returns: (str): The RefgetStore URL if configured, None otherwise.

get_regions(digest: str, regions: list)

Extract region substrings from a server-hosted sequence collection.

Posts a list of regions to the server’s region-extraction endpoint and returns the structured results. Requires a store-backed server (the database backend responds with HTTP 501).

Args: digest (str): The collection digest to extract regions from. regions (list): A list of {“chrom”, “start”, “end”} dicts.

Returns: (list): A list of {“chrom_name”, “start”, “end”, “sequence”} dicts.

is_aliases_enabled()

Check if alias endpoints are advertised in service-info.

Returns: (bool): True if aliases are enabled, False otherwise.

is_fasta_drs_enabled()

Check if FastaDRS endpoints are available.

Returns: (bool): True if FastaDRS is enabled, False otherwise.

is_fhr_enabled()

Check if FHR metadata is advertised in service-info.

Returns: (bool): True if FHR metadata is enabled, False otherwise.

list_alias_namespaces(kind: str = 'collection')

List alias namespaces for the given kind.

Args: kind (str): “collection” (default) or “sequence”.

Returns: (dict): {“namespaces”: […]}.

list_aliases(namespace: str, kind: str = 'collection')

List aliases within a namespace.

Args: namespace (str): The alias namespace. kind (str): “collection” (default) or “sequence”.

Returns: (dict): {“namespace”: …, “aliases”: […]}.

list_attributes(attribute: str, page: Optional[int] = None, page_size: Optional[int] = None)

Lists all available values for a given attribute with optional paging support.

Args: attribute (str): The attribute to list values for. page (int, optional): The page number to retrieve. Defaults to None. page_size (int, optional): The number of items per page. Defaults to None.

Returns: (dict): The JSON response containing the list of available values for the attribute.

list_collections(page: Optional[int] = None, page_size: Optional[int] = None, filters = {})

Lists all available sequence collections with optional paging and attribute filtering support.

Args: page (int, optional): The page number to retrieve. Defaults to None. page_size (int, optional): The number of items per page. Defaults to None. **filters (Any): Optional attribute filters (e.g., names=“abc123”, lengths=“def456”). Values should be level 1 digests of the attributes.

Returns: (dict): The JSON response containing the list of available sequence collections.

list_fhr()

List collections that have FHR metadata.

Returns: (dict): {“collections”: […]}.

resolve_alias(namespace: str, alias: str, kind: str = 'collection')

Resolve a namespace:alias to a digest.

Args: namespace (str): The alias namespace. alias (str): The alias name. kind (str): “collection” (default) or “sequence”.

Returns: (dict): {“namespace”: …, “alias”: …, “digest”: …} or None.

service_info()

Retrieves information about the service.

Returns: (dict): The service information.

write_chrom_sizes(digest: str, dest_path: str)

Write a chrom.sizes file for a sequence collection.

Args: digest (str): The sequence collection digest dest_path (str): Path to write the chrom.sizes file

Returns: (str): Path to the written file

write_fai(digest: str, dest_path: str)

Write a .fai index file for a FASTA.

Args: digest (str): The sequence collection digest dest_path (str): Path to write the .fai file

Returns: (str): Path to the written file

class FastaDrsClient(urls: list[str] = ['https://seqcolapi.databio.org/fasta'], raise_errors: Optional[bool] = None)

A client for interacting with FASTA files via GA4GH DRS endpoints.

Properties

raise_errors

urls

Methods

build_fai(digest: str, seqcol_client: 'SequenceCollectionClient' = None)

Build a complete .fai index file content for a FASTA.

FAI format per line: NAME LENGTH OFFSET LINEBASES LINEWIDTH

Args: digest (str): The sequence collection digest seqcol_client (SequenceCollectionClient, optional): SequenceCollectionClient to use. If None, uses parent client or creates one.

Returns: (str): String content of the .fai file

download(digest: str, dest_path: str = None, access_id: str = None)

Download the FASTA file to a local path.

Args: digest (str): The sequence collection digest dest_path (str, optional): Destination file path. If None, uses object name. access_id (str, optional): Specific access method to use. If None, tries all.

Returns: (str): Path to downloaded file

Raises: ValueError: If no access methods available or specified access_id not found

download_to_store(digest: str, store: 'RefgetStore', access_id: str = None, temp_dir: str = None, namespaces: Optional[list[str]] = None)

Download the FASTA file and import it into a RefgetStore.

This method downloads the FASTA file from the DRS endpoint and immediately imports it into the provided RefgetStore, enabling local sequence retrieval by digest without re-downloading.

Args: digest (str): The sequence collection digest store (RefgetStore): The RefgetStore instance to import into access_id (str, optional): Specific access method to use. If None, tries all. temp_dir (str, optional): Directory for temporary download. If None, uses system temp. namespaces (list[str], optional): Namespace prefixes to extract aliases from FASTA headers when importing into the store.

Returns: (str): The collection digest of the imported sequences

Raises: ValueError: If no access methods available or specified access_id not found ImportError: If gtars/RefgetStore is not available

Example: >>> from refget.store import RefgetStore >>> store = RefgetStore.in_memory() >>> client = FastaDrsClient() >>> collection_digest = client.download_to_store(“abc123”, store)

get_access_url(digest: str, access_id: str)

Get access URL for a specific access method.

Args: digest (str): The sequence collection digest access_id (str): The access ID from the access method

Returns: (dict): Access URL object

get_index(digest: str)

Get FAI index data for a FASTA file.

Args: digest (str): The sequence collection digest

Returns: (dict): Dict with line_bases, extra_line_bytes, offsets

get_object(digest: str)

Get DRS object metadata for a FASTA file.

Args: digest (str): The sequence collection digest (which is also the DRS object ID)

Returns: (dict): DRS object with id, self_uri, size, checksums, access_methods, etc.

service_info()

Get DRS service info.

Returns: (dict): The service information.

write_fai(digest: str, dest_path: str, seqcol_client: 'SequenceCollectionClient' = None)

Write a .fai index file for a FASTA.

Args: digest (str): The sequence collection digest dest_path (str): Path to write the .fai file seqcol_client (SequenceCollectionClient, optional): SequenceCollectionClient to use

Returns: (str): Path to the written file

class PangenomeClient

Agent Classes

Agents provide higher-level abstractions for working with refget data in a PostgreSQL database.

class RefgetDBAgent(engine: Optional[SqlalchemyDatabaseEngine] = None, postgres_str: Optional[str] = None, schema = SEQCOL_SCHEMA_PATH, inherent_attrs: List[str] = DEFAULT_INHERENT_ATTRS, fasta_drs_url_prefix: Optional[str] = None)

Primary aggregator agent, interface to all other agents

Parameterized it via these environment variables:

• POSTGRES_HOST
• POSTGRES_DB
• POSTGRES_USER
• POSTGRES_PASSWORD

Properties

attribute: AttributeAgent

engine

fasta_drs: FastaDrsAgent

inherent_attrs

pangenome: PangenomeAgent

schema_dict

seq: SequenceAgent

seqcol: SequenceCollectionAgent

Methods

aliases_for(kind: str, digest: str)

calc_similarities(digestA: str, digestB: str)

Calculates the Jaccard similarity between two sequence collections.

This method retrieves two sequence collections using their digests and then computes jaccard similarities for all attributes.

Args: digestA (str): The digest (identifier) for the first sequence collection. digestB (str): The digest (identifier) for the second sequence collection.

Returns: dict: The Jaccard similarity score between the two sequence collections for all present and shared attributes.

calc_similarities_seqcol_dicts(seqcolA: dict, seqcolB: dict)

Calculates the Jaccard similarity between two sequence collections.

This method retrieves one sequence collections using a digests and then computes jaccard similarities versus another input sequence collection dictionary.

Args: seqcolA (dict): the first sequence collection in dict format. seqcolB (dict): the second sequence collection in dict format.

Returns: dict: The Jaccard similarity score between the two sequence collections for all present and shared attributes.

capabilities()

collection_count()

compare_1_digest(digestA: str, seqcolB: dict)

compare_digest_with_level2(digest: str, level2_b: dict)

compare_digests(digestA: str, digestB: str)

get_attribute(attribute_name: str, attribute_digest: str)

get_collection(digest: str, level: int = 2)

get_collection_attribute(digest: str, attribute: str)

get_collection_itemwise(digest: str, limit: int | None = None)

get_fhr(digest: str)

list_alias_namespaces(kind: str)

list_aliases(kind: str, namespace: str)

list_attributes(attribute: str, page: int = 0, page_size: int = 100)

list_collections(page: int = 0, page_size: int = 100, filters: dict | None = None)

list_fhr()

resolve_alias(kind: str, namespace: str, alias: str)

retrieve_level2_digest(seqcoldigest: str)

truncate()

Delete all records from the database

class SequenceCollectionAgent(engine: SqlalchemyDatabaseEngine, inherent_attrs: Optional[List[str]] = None, parent: Optional['RefgetDBAgent'] = None)

Agent for interacting with database of sequence collection

Properties

engine

inherent_attrs

parent

Methods

add(seqcol: SequenceCollection, update: bool = False)

Add a sequence collection to the database or update it if it exists

Args: seqcol: The sequence collection to add update: If True, update an existing collection if it exists

Returns: The added or updated sequence collection

add_from_dict(seqcol_dict: dict, update: bool = False)

Add a sequence collection from a seqcol dictionary

Args: seqcol_dict (dict): The sequence collection in dictionary form update (bool): If True, update an existing collection if it exists

Returns: (SequenceCollection): The added or updated sequence collection

add_from_fasta_file(fasta_file_path: str, update: bool = False, create_fasta_drs: bool = True, human_readable_name: str = None)

Given a path to a fasta file, load the sequences into the refget database.

Args: fasta_file_path (str): Path to the fasta file update (bool): If True, update an existing collection if it exists create_fasta_drs (bool): If True, create a FastaDrsObject for the FASTA file human_readable_name (str): Optional human-readable name for the collection

Returns: (SequenceCollection): The added or updated sequence collection

add_from_fasta_file_with_name(fasta_file_path: str, human_readable_name: str, update: bool = False, create_fasta_drs: bool = True)

Given a path to a fasta file, and a human-readable name, load the sequences into the refget database.

Deprecated: Use add_from_fasta_file(fasta_file_path, human_readable_name=name) instead.

add_from_fasta_pep(pep: 'peppy.Project', fa_root: str, update: bool = False, create_fasta_drs: bool = True)

Given a PEP project and a root directory containing the fasta files, load the fasta files into the refget database.

Args: pep (peppy.Project): PEP project object containing sample metadata fa_root (str): Root directory containing the fasta files update (bool): If True, update existing sequence collections create_fasta_drs (bool): If True, create FastaDrsObjects for the FASTA files

Returns: (dict): A dictionary of the digests of the added sequence collections

get(digest: str, return_format: str = 'level2', attribute: Optional[str] = None, itemwise_limit: Optional[int] = None)

Get a sequence collection by digest

Args: digest (str): The digest of the sequence collection return_format (str): The format in which to return the sequence collection attribute (str): Name of an attribute to return, if you just want an attribute itemwise_limit (int): Limit the number of items returned in itemwise format

Returns: (SequenceCollection): The sequence collection (in requested format)

get_many_level2_offset(limit: int = 50, offset: int = 0, target_digests: Optional[List[str]] = None)

list(page_size: int = 100, cursor: Optional[str] = None)

list_by_offset(limit: int = 50, offset: int = 0)

search_by_attributes(filters: dict, offset: int = 0, limit: int = 50)

Search sequence collections by multiple attribute filters (AND logic).

Args: filters: Dict of {attribute_name: digest} pairs offset: Pagination offset limit: Max results to return

Returns: Dict with pagination info and results

class SequenceAgent(engine: SqlalchemyDatabaseEngine)

Agent for interacting with database of sequences

Properties

engine

Methods

add(sequence: Sequence)

get(digest: str, start: int | None = None, end: int | None = None)

list(offset: int = 0, limit: int = 50)

class PangenomeAgent(parent: 'RefgetDBAgent')

Agent for interacting with database of pangenomes

Properties

engine

parent

Methods

add(pangenome: Pangenome)

add_from_fasta_pep(pep: 'peppy.Project', fa_root: str, update: bool = False)

get(digest: str, return_format: str = 'level2')

list_by_offset(limit: int = 50, offset: int = 0)

class AttributeAgent(engine: SqlalchemyDatabaseEngine)

Properties

engine

Methods

get(attribute_type: str, digest: str)

list(attribute_type: str, offset: int = 0, limit: int = 50)

search(attribute_type: str, digest: str, offset: int = 0, limit: int = 50)

class FastaDrsAgent(engine: SqlalchemyDatabaseEngine, url_prefix: Optional[str] = None)

Agent for interacting with database of FASTA DRS objects

Properties

engine

url_prefix

Methods

add(fasta_drs: FastaDrsObject)

Add a FastaDrsObject to the database

add_access_method(digest: str, access_method: AccessMethod)

Add an access method to an existing FastaDrsObject.

Args: digest: The digest (object_id) of the DRS object access_method: The AccessMethod to add

Returns: The updated FastaDrsObject

get(digest: str)

Get a FastaDrsObject by its digest (object_id)

list_by_offset(limit: int = 50, offset: int = 0)

List FastaDrsObjects with pagination

RefgetStore (gtars)

RefgetStore provides high-performance local sequence storage implemented in Rust. It supports:

• In-memory and on-disk storage with optional compression
• Remote store access with local caching
• Sequence retrieval by digest or by collection + name
• BED file region extraction for batch operations
• FASTA export for individual sequences or regions

See the RefgetStore tutorial for usage examples.

RefgetStore

class RefgetStore

A global store for GA4GH refget sequences with lazy-loading support.

RefgetStore provides content-addressable storage for reference genome sequences following the GA4GH refget specification. Supports both local and remote stores with on-demand sequence loading.

Attributes: cache_path: Local directory path where the store is located or cached. None for in-memory stores. remote_url: Remote URL of the store if loaded remotely, None otherwise. quiet: Whether the store suppresses progress output. storage_mode: Current storage mode (Raw or Encoded).

Note: Boolean evaluation: RefgetStore follows Python container semantics, meaning bool(store) is False for empty stores (like list, dict, etc.). To check if a store variable is initialized (not None), use if store is not None: rather than if store:.

Example::

    store = RefgetStore.in_memory()  # Empty store
    bool(store)  # False (empty container)
    len(store)   # 0

    # Wrong: checks emptiness, not initialization
    if store:
        process(store)

    # Right: checks if variable is set
    if store is not None:
        process(store)

Examples: Create a new store and import sequences::

    from gtars.refget import RefgetStore
    store = RefgetStore.in_memory()
    store.add_sequence_collection_from_fasta("genome.fa")

Open an existing local store::

    store = RefgetStore.open_local("/data/hg38")
    seq = store.get_substring("chr1_digest", 0, 1000)

Open a remote store with caching::

    store = RefgetStore.open_remote(
        "/local/cache",
        "https://example.com/hg38"
    )

Properties

cache_path: Optional[str]

is_persisting: bool : Whether the store is currently persisting to disk.

quiet: bool : Whether the store is in quiet mode.

remote_url: Optional[str]

storage_mode: StorageMode : Current storage mode (Raw or Encoded).

Class Methods

in_memory()

Create a new in-memory RefgetStore.

Creates a store that keeps all sequences in memory. Use this for temporary processing or when you don’t need disk persistence.

Returns: New empty RefgetStore with Encoded storage mode.

Example::

store = RefgetStore.in_memory()
store.add_sequence_collection_from_fasta("genome.fa")

on_disk(cache_path: Union[str, PathLike])

Create or load a disk-backed RefgetStore.

If the directory contains an existing store (rgstore.json), loads it. Otherwise creates a new store with Encoded mode.

Args: cache_path: Directory path for the store. Created if it doesn’t exist.

Returns: RefgetStore (new or loaded from disk).

Example::

store = RefgetStore.on_disk("/data/my_store")
store.add_sequence_collection_from_fasta("genome.fa")
# Store is automatically persisted to disk

open_local(path: Union[str, PathLike])

Open a local RefgetStore from a directory.

Loads only lightweight metadata and stubs. Collections and sequences remain as stubs until explicitly accessed with get_collection()/get_sequence().

Expects: rgstore.json, sequences.rgsi, collections.rgci, collections/*.rgsi

Args: path: Local directory containing the refget store.

Returns: RefgetStore with metadata loaded, sequences lazy-loaded.

Raises: IOError: If the store directory or index files cannot be read.

Example::

store = RefgetStore.open_local("/data/hg38_store")
seq = store.get_substring("chr1_digest", 0, 1000)

open_remote(cache_path: Union[str, PathLike], remote_url: str)

Open a remote RefgetStore with local caching.

Loads only lightweight metadata and stubs from the remote URL. Data is fetched on-demand when get_collection()/get_sequence() is called.

By default, persistence is enabled (sequences are cached to disk). Call disable_persistence() after loading to keep only in memory.

Args: cache_path: Local directory to cache downloaded metadata and sequences. Created if it doesn’t exist. remote_url: Base URL of the remote refget store (e.g., “https://example.com/hg38” or “s3://bucket/hg38”).

Returns: RefgetStore with metadata loaded, sequences fetched on-demand.

Raises: IOError: If remote metadata cannot be fetched or cache cannot be written.

Example::

store = RefgetStore.open_remote(
    "/data/cache/hg38",
    "https://refget-server.com/hg38"
)
# First access fetches from remote and caches
seq = store.get_substring("chr1_digest", 0, 1000)
# Second access uses cache
seq2 = store.get_substring("chr1_digest", 1000, 2000)

store_exists(path: Union[str, PathLike])

Check whether a valid RefgetStore exists at the given path.

Returns True if the path contains a store manifest file, indicating the store has been initialized. Returns False if the path does not exist or does not contain a store.

This avoids hardcoding knowledge of the store’s internal file format in calling code.

Args: path: Path to the store directory.

Returns: True if a store exists at the path, False otherwise.

Example::

from gtars.refget import RefgetStore
RefgetStore.store_exists("/data/hg38_store")  # True
RefgetStore.store_exists("/tmp/empty")  # False

Methods

add_collection_alias(namespace: str, alias: str, digest: str)

Add a collection alias: namespace/alias maps to collection digest.

add_sequence(sequence: SequenceRecord, force: bool = False)

Add a sequence to the store without collection association.

The sequence can be created using digest_sequence() and later retrieved by its digest via get_sequence().

Args: sequence: A SequenceRecord created by digest_sequence(). force: If True, overwrite existing. If False (default), skip duplicates.

Raises: IOError: If the sequence cannot be stored.

Example::

from gtars.refget import RefgetStore, digest_sequence
store = RefgetStore.in_memory()
seq = digest_sequence(b"ACGTACGT")
store.add_sequence(seq)
retrieved = store.get_sequence(seq.metadata.sha512t24u)

add_sequence_alias(namespace: str, alias: str, digest: str)

Add a sequence alias: namespace/alias maps to sequence digest.

add_sequence_collection(collection: SequenceCollection, force: bool = False)

Add a pre-built SequenceCollection to the store.

Adds a SequenceCollection (created via digest_fasta() or programmatically) directly to the store without reading from a FASTA file.

Args: collection: A SequenceCollection to add. force: If True, overwrite existing collections/sequences. If False (default), skip duplicates.

Raises: IOError: If the collection cannot be stored.

Example::

from gtars.refget import RefgetStore, digest_fasta
store = RefgetStore.in_memory()
collection = digest_fasta("genome.fa")
store.add_sequence_collection(collection)

add_sequence_collection_from_fasta(file_path: Union[str, PathLike], force: bool = False, namespaces: Optional[List[str]] = None)

Add a sequence collection from a FASTA file.

Reads a FASTA file, digests the sequences, creates a SequenceCollection, and adds it to the store along with all its sequences.

Args: file_path: Path to the FASTA file to import. force: If True, overwrite existing collections/sequences. If False (default), skip duplicates. namespaces: Optional list of namespace prefixes to extract aliases from FASTA headers. For example, [“ncbi”, “refseq”] will scan headers for tokens like ncbi:NC_000001.11 and register them as aliases.

Returns: A tuple containing: - SequenceCollectionMetadata: Metadata for the collection. - bool: True if the collection was newly added, False if it already existed.

Raises: IOError: If the file cannot be read or processed.

Example::

store = RefgetStore.in_memory()
metadata, was_new = store.add_sequence_collection_from_fasta("genome.fa")
print(f"{'Added' if was_new else 'Skipped'}: {metadata.digest}")

# Extract aliases from FASTA headers
metadata, was_new = store.add_sequence_collection_from_fasta(
    "genome.fa", namespaces=["ncbi", "refseq"]
)

add_sequence_collections_from_fastas(fastas: Union[str, PathLike, List[Union[str, PathLike]]], file_list: Optional[Union[str, PathLike]] = None, jobs: int = 0, force: bool = False, namespaces: Optional[List[str]] = None)

Import multiple FASTA files into the store.

fastas accepts a single path, a glob pattern (e.g. “fasta/*.fa.gz”), a directory of FASTAs, or a list mixing any of these. file_list points at a file-of-filenames (one path per line; blank lines and #-comments ignored; may itself contain globs/directories). Inputs are expanded by gtars into a de-duplicated, deterministic order (explicit paths first, then file_list lines; directory/glob matches sorted lexicographically). jobs is the number of files imported concurrently (0 = auto, 1 = serial); it does not affect ordering or the resulting store.

Args: fastas: A path, glob, directory, or list of any of these. file_list: Optional path to a file-of-filenames. jobs: Number of files imported concurrently (0 = auto, 1 = serial). force: If True, overwrite existing collections/sequences. namespaces: Optional namespace prefixes to extract aliases from headers.

Returns: A list of (SequenceCollectionMetadata, bool) tuples in expanded-input order; the bool is True if newly added.

Raises: ValueError: If the inputs cannot be expanded (e.g. glob matches nothing). IOError: If a file cannot be read or processed.

Example::

store = RefgetStore.in_memory()
results = store.add_sequence_collections_from_fastas("data/*.fa.gz", jobs=4)

compare(digest_a: str, digest_b: str)

Compare two collections by digest.

Args: digest_a: First collection digest. digest_b: Second collection digest.

Returns: dict with keys: digests, attributes, array_elements.

disable_ancillary_digests()

Disable computation of ancillary digests.

disable_attribute_index()

Disable indexed attribute lookup, using brute-force scan instead.

disable_encoding()

Disable encoding, use raw byte storage.

Decodes any existing Encoded sequences in memory.

Example::

store = RefgetStore.in_memory()
store.disable_encoding()  # Switch to Raw mode

disable_persistence()

Disable disk persistence for this store.

New sequences will be kept in memory only. Existing Stub sequences can still be loaded from disk if local_path is set.

Example::

store = RefgetStore.open_remote("/cache", "https://example.com")
store.disable_persistence()  # Stop caching new sequences

enable_ancillary_digests()

Enable computation of ancillary digests.

enable_attribute_index()

Enable indexed attribute lookup (not yet implemented).

enable_encoding()

Enable 2-bit encoding for space efficiency.

Re-encodes any existing Raw sequences in memory.

Example::

store = RefgetStore.in_memory()
store.disable_encoding()  # Switch to Raw
store.enable_encoding()   # Back to Encoded

enable_persistence(path: Union[str, PathLike])

Enable disk persistence for this store.

Sets up the store to write sequences to disk. Any in-memory Full sequences are flushed to disk and converted to Stubs.

Args: path: Directory for storing sequences and metadata.

Raises: IOError: If the directory cannot be created or written to.

Example::

store = RefgetStore.in_memory()
store.add_sequence_collection_from_fasta("genome.fa")
store.enable_persistence("/data/store")  # Flush to disk

export_fasta(collection_digest: str, output_path: Union[str, PathLike], sequence_names: Optional[List[str]] = None, line_width: Optional[int] = None)

Export sequences from a collection to a FASTA file.

Args: collection_digest: Collection to export from. output_path: Path to write FASTA file. sequence_names: Optional list of sequence names to export. If None, exports all sequences in the collection. line_width: Optional line width for wrapping sequences. If None, uses default of 80.

export_fasta_by_digests(seq_digests: List[str], output_path: Union[str, PathLike], line_width: Optional[int] = None)

Export sequences by their digests to a FASTA file.

Args: seq_digests: List of sequence digests to export. output_path: Path to write FASTA file. line_width: Optional line width for wrapping sequences. If None, uses default of 80.

export_fasta_from_regions(collection_digest: str, bed_file_path: Union[str, PathLike], output_file_path: Union[str, PathLike])

Export sequences from BED file regions to a FASTA file.

Reads a BED file defining genomic regions and exports the sequences for those regions to a FASTA file.

Args: collection_digest: The collection’s SHA-512/24u digest. bed_file_path: Path to BED file defining regions. output_file_path: Path to write the output FASTA file.

Raises: IOError: If files cannot be read/written or sequences not found.

Example::

store.export_fasta_from_regions(
    "uC_UorBNf3YUu1YIDainBhI94CedlNeH",
    "regions.bed",
    "output.fa"
)

find_collections_by_attribute(attr_name: str, attr_digest: str)

Find collections by attribute digest.

Args: attr_name: Attribute name (names, lengths, sequences, name_length_pairs, sorted_name_length_pairs, sorted_sequences). attr_digest: The digest to search for.

Returns: List of collection digests that have the matching attribute.

get_aliases_for_collection(digest: str)

Reverse lookup: find all (namespace, alias) pairs pointing to this collection digest.

get_aliases_for_sequence(digest: str)

Reverse lookup: find all (namespace, alias) pairs pointing to this sequence digest.

get_attribute(attr_name: str, attr_digest: str)

Get attribute array by digest.

Args: attr_name: Attribute name (names, lengths, or sequences). attr_digest: The digest to search for.

Returns: The attribute array, or None if not found.

get_collection(collection_digest: str)

Get a collection by digest with all sequences loaded.

Loads the collection and all its sequence data into memory. Use this when you need full access to sequence content.

Args: collection_digest: The collection’s SHA-512/24u digest.

Returns: The collection with all sequence data loaded.

Raises: IOError: If the collection cannot be loaded.

Example::

collection = store.get_collection("uC_UorBNf3YUu1YIDainBhI94CedlNeH")
for seq in collection.sequences:
    print(f"{seq.metadata.name}: {seq.decode()[:20]}...")

get_collection_by_alias(namespace: str, alias: str)

Resolve a collection alias and return the loaded collection.

Returns None if the alias is not found.

get_collection_level1(digest: str)

Get level 1 representation (attribute digests) for a collection.

Args: digest: Collection digest.

Returns: dict with spec-compliant field names (names, lengths, sequences, plus optional name_length_pairs, sorted_name_length_pairs, sorted_sequences).

get_collection_level2(digest: str)

Get level 2 representation (full arrays, spec format) for a collection.

Args: digest: Collection digest.

Returns: dict with names (list[str]), lengths (list[int]), sequences (list[str]).

get_collection_metadata(collection_digest: str)

Get metadata for a collection by digest.

Returns lightweight metadata without loading the full collection. Use this for quick lookups of collection information.

Args: collection_digest: The collection’s SHA-512/24u digest.

Returns: Collection metadata if found, None otherwise.

Example::

meta = store.get_collection_metadata("uC_UorBNf3YUu1YIDainBhI94CedlNeH")
if meta:
    print(f"Collection has {meta.n_sequences} sequences")

get_collection_metadata_by_alias(namespace: str, alias: str)

Resolve a collection alias to collection metadata (no data loading).

get_fhr_metadata(collection_digest: str)

Get FHR metadata for a collection. Returns None if missing.

get_sequence(digest: str)

Retrieve a sequence record by its digest (SHA-512/24u or MD5).

Loads the sequence data if not already in memory. Supports lookup by either SHA-512/24u (preferred) or MD5 digest. Automatically strips “SQ.” prefix if present (case-insensitive).

Args: digest: Sequence digest (SHA-512/24u base64url or MD5 hex string), optionally with “SQ.” prefix.

Returns: The sequence record with data.

Raises: KeyError: If the sequence is not found.

Example::

record = store.get_sequence("aKF498dAxcJAqme6QYQ7EZ07-fiw8Kw2")
print(f"Found: {record.metadata.name}")
# Also works with SQ. prefix
record = store.get_sequence("SQ.aKF498dAxcJAqme6QYQ7EZ07-fiw8Kw2")

get_sequence_by_alias(namespace: str, alias: str)

Resolve a sequence alias and return the loaded sequence record.

Returns None if the alias is not found.

get_sequence_by_name(collection_digest: str, sequence_name: str)

Retrieve a sequence by collection digest and sequence name.

Looks up a sequence within a specific collection using its name (e.g., “chr1”, “chrM”). Loads the sequence data if needed. Automatically strips “SQ.” prefix from collection digest if present.

Args: collection_digest: The collection’s SHA-512/24u digest, optionally with “SQ.” prefix. sequence_name: Name of the sequence within that collection.

Returns: The sequence record with data.

Raises: KeyError: If the sequence is not found.

Example::

record = store.get_sequence_by_name(
    "uC_UorBNf3YUu1YIDainBhI94CedlNeH",
    "chr1"
)
print(f"Sequence: {record.decode()[:50]}...")

get_sequence_metadata(seq_digest: str)

Get metadata for a sequence by digest (no data loaded).

Use this for lightweight lookups when you don’t need the actual sequence. Automatically strips “SQ.” prefix from digest if present.

Args: seq_digest: The sequence’s SHA-512/24u digest, optionally with “SQ.” prefix.

Returns: Sequence metadata if found, None otherwise.

get_sequence_metadata_by_alias(namespace: str, alias: str)

Resolve a sequence alias to sequence metadata (no data loading).

get_substring(seq_digest: str, start: int, end: int)

Extract a substring from a sequence (flow 1: lean partial read).

Retrieves a specific region using 0-based, half-open coordinates [start, end), reading only the bytes that cover the region. Source resolution is resident -> local .seq -> remote HTTP byte-range, so a remote-backed store is served directly with no manual preload and without downloading or caching the whole sequence. For repeat-heavy access call load_sequence() (flow 3) first. Automatically strips “SQ.” prefix from digest if present.

Args: seq_digest: Sequence digest (SHA-512/24u), optionally with “SQ.” prefix. start: Start position (0-based, inclusive). end: End position (0-based, exclusive).

Returns: The substring sequence.

Raises: KeyError: If the sequence is not found.

Example::

# Get first 1000 bases of chr1
seq = store.get_substring("chr1_digest", 0, 1000)
print(f"First 50bp: {seq[:50]}")

has_ancillary_digests()

Returns whether ancillary digests are enabled.

has_attribute_index()

Returns whether the on-disk attribute index is enabled.

into_readonly()

Convert to a ReadonlyRefgetStore for concurrent read access.

Consumes this store (replacing it with an empty in-memory store) and returns a ReadonlyRefgetStore whose read methods all use &self (no mutable borrow), making it suitable for Arc<ReadonlyRefgetStore> in servers.

Call load_all_collections() or load_collection() before converting, since ReadonlyRefgetStore cannot lazy-load.

Returns: ReadonlyRefgetStore: An immutable store suitable for concurrent access.

Example::

store = RefgetStore.open_remote("/cache", "https://example.com")
store.load_all_collections()
readonly = store.into_readonly()
coll = readonly.get_collection("abc123")

is_collection_loaded(collection_digest: str)

Check if a collection is fully loaded.

Returns True if the collection’s sequence list is loaded in memory, False if it’s only metadata (stub).

Args: collection_digest: The collection’s SHA-512/24u digest.

Returns: True if loaded, False otherwise.

iter_collections()

Iterate over all collections with their sequences loaded.

This loads all collection data upfront and returns a list of SequenceCollection objects with full sequence data.

For browsing without loading data, use list_collections() instead.

Returns: List of all collections with loaded sequences.

Example::

for coll in store.iter_collections():
    print(f"{coll.digest}: {len(coll.sequences)} sequences")

iter_sequences()

Iterate over all sequences with their data loaded.

This ensures all sequence data is loaded and returns a list of SequenceRecord objects with full sequence data.

For browsing without loading data, use list_sequences() instead.

Returns: List of all sequences with loaded data.

Example::

for seq in store.iter_sequences():
    print(f"{seq.metadata.name}: {seq.decode()[:20]}...")

list_collection_alias_namespaces()

List all collection alias namespaces.

list_collection_aliases(namespace: str)

List all aliases in a collection alias namespace.

list_collections(page: int = 0, page_size: int = 100, filters: Optional[Dict[str, str]] = None)

List collections with pagination and optional attribute filtering.

Args: page: 0-indexed page number. page_size: Number of results per page. filters: Optional attribute filters (AND logic). Keys are attribute names (names, lengths, sequences, name_length_pairs, sorted_name_length_pairs, sorted_sequences), values are digests.

Returns: Dict with “results” (list of SequenceCollectionMetadata) and “pagination” (dict with page, page_size, total).

Example::

# Get first page of all collections
result = store.list_collections()
for meta in result["results"]:
    print(f"{meta.digest}: {meta.n_sequences} sequences")
print(f"Total: {result['pagination']['total']}")

# Filter by names digest
result = store.list_collections(filters={"names": "abc123"})

list_fhr_metadata()

List all collection digests that have FHR metadata.

list_sequence_alias_namespaces()

List all sequence alias namespaces.

list_sequence_aliases(namespace: str)

List all aliases in a sequence alias namespace.

list_sequences()

List all sequence metadata in the store.

Returns metadata for all sequences without loading sequence data. Use this for browsing/inventory operations.

Returns: List of metadata for all sequences in the store.

Example::

for meta in store.list_sequences():
    print(f"{meta.name}: {meta.length} bp")

load_collection_aliases(namespace: str, path: str)

Load collection aliases from a TSV file (alias\tdigest per line).

load_fhr_metadata(collection_digest: str, path: str)

Load FHR metadata from a JSON file and attach it to a collection.

load_sequence_aliases(namespace: str, path: str)

Load sequence aliases from a TSV file (alias\tdigest per line).

remove_collection(digest: str, remove_orphan_sequences: bool = False)

Remove a collection from the store.

Args: digest: The collection’s SHA-512/24u digest string. remove_orphan_sequences: If True, also remove sequences no longer referenced by any remaining collection. Default: False.

Returns: True if the collection was found and removed, False if not found.

remove_collection_alias(namespace: str, alias: str)

Remove a single collection alias. Returns True if it existed.

remove_fhr_metadata(collection_digest: str)

Remove FHR metadata for a collection.

remove_sequence_alias(namespace: str, alias: str)

Remove a single sequence alias. Returns True if it existed.

set_encoding_mode(mode: StorageMode)

Change the storage mode, re-encoding/decoding existing sequences as needed.

When switching from Raw to Encoded, all Full sequences in memory are encoded (2-bit packed). When switching from Encoded to Raw, all Full sequences in memory are decoded back to raw bytes.

Args: mode: The storage mode to switch to (StorageMode.Raw or StorageMode.Encoded).

Example::

store = RefgetStore.in_memory()
store.set_encoding_mode(StorageMode.Raw)

set_fhr_metadata(collection_digest: str, metadata: FhrMetadata)

Set FHR metadata for a collection.

set_quiet(quiet: bool)

Set whether to suppress progress output.

When quiet is True, operations like add_sequence_collection_from_fasta will not print progress messages.

Args: quiet: Whether to suppress progress output.

Example::

store = RefgetStore.in_memory()
store.set_quiet(True)
store.add_sequence_collection_from_fasta("genome.fa")  # No output

stats()

Returns statistics about the store.

Returns: dict with keys: - ‘n_sequences’: Total number of sequences (Stub + Full) - ‘n_sequences_loaded’: Number of sequences with data loaded (Full) - ‘n_collections’: Total number of collections (Stub + Full) - ‘n_collections_loaded’: Number of collections with sequences loaded (Full) - ‘storage_mode’: Storage mode (‘Raw’ or ‘Encoded’)

Note: n_collections_loaded only reflects collections fully loaded in memory. For remote stores, collections are loaded on-demand when accessed.

Example::

stats = store.stats()
print(f"Store has {stats['n_sequences']} sequences")
print(f"Collections: {stats['n_collections']} total, {stats['n_collections_loaded']} loaded")

stream_sequence(seq_digest: str, start: Optional[int] = None, end: Optional[int] = None, chunk_size: int = 65536)

Stream a (sub)sequence as decoded bases without loading it (flow 2).

Returns a :class:SequenceStream that yields str chunks lazily, reading from resident bytes, a local .seq, or a remote HTTP byte-range as needed. Peak memory is O(1) in the region length. Omit start/end to stream the whole sequence.

Args: seq_digest: Sequence digest (SHA-512/24u), optionally with “SQ.” prefix. start: Start position (0-based, inclusive). Defaults to 0. end: End position (0-based, exclusive). Defaults to the sequence length. chunk_size: Bytes read per chunk (default 64 KiB).

Returns: A SequenceStream iterator of decoded base chunks.

substrings_from_regions(collection_digest: str, bed_file_path: Union[str, PathLike])

Get substrings for BED file regions as a list.

Reads a BED file and returns a list of sequences for each region.

Args: collection_digest: The collection’s SHA-512/24u digest. bed_file_path: Path to BED file defining regions.

Returns: List of retrieved sequence segments.

Raises: IOError: If files cannot be read or sequences not found.

Example::

sequences = store.substrings_from_regions(
    "uC_UorBNf3YUu1YIDainBhI94CedlNeH",
    "regions.bed"
)
for seq in sequences:
    print(f"{seq.chrom_name}:{seq.start}-{seq.end}")

write()

Write the store using its configured paths.

Convenience method for disk-backed stores. Uses the store’s own local_path and seqdata_path_template.

Raises: IOError: If the store cannot be written.

write_store_to_dir(root_path: Union[str, PathLike], seqdata_path_template: Optional[str] = None)

Write the store to a directory on disk.

Persists the store with all sequences and metadata to disk using the RefgetStore directory format.

Args: root_path: Directory path to write the store to. seqdata_path_template: Optional path template for sequence files (e.g., “sequences/%s2/%s.seq” where %s2 = first 2 chars of digest, %s = full digest). Uses default if not specified.

Example::

store.write_store_to_dir("/data/my_store")
store.write_store_to_dir("/data/my_store", "sequences/%s2/%s.seq")

Functions

digest_fasta(fasta: Union[str, PathLike])

Digest all sequences in a FASTA file and compute collection-level digests.

This function reads a FASTA file and computes GA4GH-compliant digests for each sequence, as well as collection-level digests (Level 1 and Level 2) following the GA4GH refget specification.

Args: fasta: Path to FASTA file (str or PathLike).

Returns: Collection containing all sequences with their metadata and computed digests.

Raises: IOError: If the FASTA file cannot be read or parsed.

Example:: from gtars.refget import digest_fasta collection = digest_fasta(“genome.fa”) print(f”Collection digest: {collection.digest}”) print(f”Number of sequences: {len(collection)}“)

compute_fai(fasta: Union[str, PathLike])

Compute FASTA index (FAI) metadata for all sequences in a FASTA file.

This function computes the FAI index metadata (offset, line_bases, line_bytes) for each sequence in a FASTA file, compatible with samtools faidx format. Only works with uncompressed FASTA files.

Args: fasta: Path to FASTA file (str or PathLike). Must be uncompressed.

Returns: List of FAI records, one per sequence, containing name, length, and FAI metadata (offset, line_bases, line_bytes).

Raises: IOError: If the FASTA file cannot be read or is compressed.

Example:: from gtars.refget import compute_fai fai_records = compute_fai(“genome.fa”) for record in fai_records: print(f”{record.name}: {record.length} bp”)

digest_sequence(data: bytes, name: Optional[str] = None, description: Optional[str] = None)

Create a SequenceRecord from raw data, computing all metadata.

This is the sequence-level parallel to digest_fasta() for collections. It computes the GA4GH sha512t24u digest, MD5 digest, detects the alphabet, and returns a SequenceRecord with computed metadata and the original data.

The input data is automatically uppercased to ensure consistent digest computation (matching FASTA processing behavior).

Args: data: The raw sequence bytes (e.g., b”ACGTACGT”). name: Optional sequence name (e.g., “chr1”). Defaults to "" if not provided. description: Optional description text for the sequence.

Returns: A SequenceRecord with computed metadata and the original data (uppercased).

Example:: from gtars.refget import digest_sequence seq = digest_sequence(b”ACGTACGT”) print(seq.metadata.length) # Output: 8

seq = digest_sequence(b"ACGT", name="chr1")
print(seq.metadata.name, seq.metadata.length)
# Output: chr1 4

# With description
seq2 = digest_sequence(b"ACGT", name="chr1", description="Chromosome 1")
print(seq2.metadata.description)
# Output: Chromosome 1

Data Classes

class SequenceCollection

A collection of biological sequences (e.g., a genome assembly).

SequenceCollection represents a set of sequences with collection-level digests following the GA4GH seqcol specification. Supports iteration, indexing, and len().

Attributes: sequences: List of sequence records. digest: Collection-level SHA-512/24u digest (Level 2). lvl1: Level 1 digests for names, lengths, sequences. file_path: Source file path if loaded from FASTA.

Examples: Iterate over sequences::

    for seq in collection:
        print(f"{seq.metadata.name}: {seq.metadata.length} bp")

Access by index::

    first_seq = collection[0]
    last_seq = collection[-1]

Get length::

    n = len(collection)

Properties

digest: str

file_path: Optional[str]

lvl1: SeqColDigestLvl1

sequences: List[SequenceRecord]

Methods

write_fasta(file_path: str, line_width: Optional[int] = None)

Write the collection to a FASTA file.

Args: file_path: Path to the output FASTA file. line_width: Number of bases per line (default: 70).

Raises: IOError: If any sequence doesn’t have data loaded.

Example::

collection = load_fasta("genome.fa")
collection.write_fasta("output.fa")
collection.write_fasta("output.fa", line_width=60)

class StorageMode

Defines how sequence data is stored in the Refget store.

Variants: Raw: Store sequences as raw bytes (1 byte per base). Encoded: Store sequences with 2-bit encoding (4 bases per byte).

Properties

Encoded: int

Raw: int

Digest Functions

Low-level functions for computing GA4GH digests:

canonical_str(item: dict)

Convert a dict into a canonical string representation