Finding Duplicate Genomes with Sorted Sequences
The sorted_sequences ancillary digest identifies the set of sequences in a
collection regardless of chromosome ordering or naming. Two FASTA files that
contain identical sequences but use different chromosome names or a different
order will share the same sorted_sequences digest.
This guide shows how to use sorted_sequences to find genome collections
that contain the same sequences.
Prerequisites: You should know how to install the refget package, create a RefgetStore, and load FASTA files (see the RefgetStore tutorial). For what ancillary digests are, see Seqcol Operations.
Create three FASTA files with identical sequences but different chromosome names and ordering, then load them into an in-memory store.
import osimport tempfile
from refget.store import RefgetStore, digest_fasta
temp_dir = tempfile.mkdtemp(prefix="sorted_seq_howto_")
# Genome A: original names and orderfasta_a = os.path.join(temp_dir, "genome_a.fa")with open(fasta_a, "w") as f: f.write(">chr1\nACGTACGTACGTACGT\n>chr2\nGGGGAAAATTTTCCCC\n")
# Genome B: same sequences, different chromosome namesfasta_b = os.path.join(temp_dir, "genome_b.fa")with open(fasta_b, "w") as f: f.write(">seq_alpha\nACGTACGTACGTACGT\n>seq_beta\nGGGGAAAATTTTCCCC\n")
# Genome C: same sequences, reversed orderfasta_c = os.path.join(temp_dir, "genome_c.fa")with open(fasta_c, "w") as f: f.write(">chr2\nGGGGAAAATTTTCCCC\n>chr1\nACGTACGTACGTACGT\n")
store = RefgetStore.in_memory()store.set_quiet(True)
store.add_sequence_collection_from_fasta(fasta_a)store.add_sequence_collection_from_fasta(fasta_b)store.add_sequence_collection_from_fasta(fasta_c)
digest_a = digest_fasta(fasta_a).digestdigest_b = digest_fasta(fasta_b).digestdigest_c = digest_fasta(fasta_c).digest
print(f"Genome A: {digest_a}")print(f"Genome B: {digest_b}")print(f"Genome C: {digest_c}")Genome A: g-n-_M2U_wMQJAH0LqKM0LxopwLHFB5xGenome B: t4O7L7jEV67yz6UlqrPYRwLHLhqjL6DfGenome C: _GLBHLerQ-8yDg8P9nihPs6ZX5MvBHjwAll three collections have different top-level digests because they differ in names or ordering.
Retrieve the sorted_sequences digest
Section titled “Retrieve the sorted_sequences digest”The level 1 representation includes a sorted_sequences key. This digest
is computed from the sorted list of individual sequence digests, so it is
invariant to both naming and ordering.
lvl1_a = store.get_collection_level1(digest_a)lvl1_b = store.get_collection_level1(digest_b)lvl1_c = store.get_collection_level1(digest_c)
print(f"Genome A sorted_sequences: {lvl1_a['sorted_sequences']}")print(f"Genome B sorted_sequences: {lvl1_b['sorted_sequences']}")print(f"Genome C sorted_sequences: {lvl1_c['sorted_sequences']}")print(f"\nAll match: {lvl1_a['sorted_sequences'] == lvl1_b['sorted_sequences'] == lvl1_c['sorted_sequences']}")Genome A sorted_sequences: EjQi8LUJO9s914OI7uxOdfo_FLKgwufVGenome B sorted_sequences: EjQi8LUJO9s914OI7uxOdfo_FLKgwufVGenome C sorted_sequences: EjQi8LUJO9s914OI7uxOdfo_FLKgwufV
All match: TrueFind all collections sharing the same sequences
Section titled “Find all collections sharing the same sequences”Use find_collections_by_attribute with the sorted_sequences digest to
retrieve every collection in the store that contains the same set of sequences.
This works the same way as the attribute lookups in
Seqcol Operations.
sorted_seq_digest = lvl1_a["sorted_sequences"]# The store returns matches in no fixed order; sort them for a stable listingmatches = sorted(store.find_collections_by_attribute("sorted_sequences", sorted_seq_digest))
print(f"Collections with sorted_sequences = {sorted_seq_digest}:")for m in matches: print(f" {m}")print(f"\nTotal: {len(matches)}")Collections with sorted_sequences = EjQi8LUJO9s914OI7uxOdfo_FLKgwufV: _GLBHLerQ-8yDg8P9nihPs6ZX5MvBHjw g-n-_M2U_wMQJAH0LqKM0LxopwLHFB5x t4O7L7jEV67yz6UlqrPYRwLHLhqjL6Df
Total: 3Retrieve sequences by sorted_sequences digest
Section titled “Retrieve sequences by sorted_sequences digest”The level 1 sorted_sequences value is a digest of the entire sorted array.
Use get_attribute to expand it into the individual sequence digests, then
retrieve each sequence.
# Level 1 digest -> array of individual sequence digestssorted_seq_digest = lvl1_a["sorted_sequences"]individual_seq_digests = store.get_attribute("sorted_sequences", sorted_seq_digest)print(f"Individual sequence digests: {individual_seq_digests}")
# Each of those is a refget sequence digest -- use it to retrieve the sequencefor seq_digest in individual_seq_digests: record = store.get_sequence(seq_digest) print(f" {seq_digest} -> {record.metadata.name} ({record.metadata.length} bp)")Individual sequence digests: ['SQ.8zS0M3VBpV7-TNdB7RjfpMbC8hrz6SbH', 'SQ.U1x5Qz66jmXmv_oJtfYmtUNkwq1wZ75u'] SQ.8zS0M3VBpV7-TNdB7RjfpMbC8hrz6SbH -> chr2 (16 bp) SQ.U1x5Qz66jmXmv_oJtfYmtUNkwq1wZ75u -> chr1 (16 bp)A sequence's name comes from the first collection that added it, so the names above are the ones from Genome A. To go further and extract subsequences from these digests, see Retrieving subsequences by digest.
Check an incoming FASTA against existing collections
Section titled “Check an incoming FASTA against existing collections”To test whether a new FASTA file duplicates sequences already in the store,
compute its digest, load it, and query by sorted_sequences.
# A new FASTA arrives -- same sequences, yet another naming schemefasta_new = os.path.join(temp_dir, "genome_new.fa")with open(fasta_new, "w") as f: f.write(">scaffold_1\nACGTACGTACGTACGT\n>scaffold_2\nGGGGAAAATTTTCCCC\n")
# Load it into the storestore.add_sequence_collection_from_fasta(fasta_new)new_digest = digest_fasta(fasta_new).digest
# Get its sorted_sequences digest and search for duplicatesnew_lvl1 = store.get_collection_level1(new_digest)new_sorted = new_lvl1["sorted_sequences"]duplicates = store.find_collections_by_attribute("sorted_sequences", new_sorted)
# Exclude the new collection itself to see pre-existing matchesexisting = sorted(d for d in duplicates if d != new_digest)
print(f"New collection: {new_digest}")print(f"Existing collections with identical sequences: {len(existing)}")for d in existing: print(f" {d}")New collection: GEeVdhZgRle9w-p82SuPOvPhiU4AYJh5Existing collections with identical sequences: 3 _GLBHLerQ-8yDg8P9nihPs6ZX5MvBHjw g-n-_M2U_wMQJAH0LqKM0LxopwLHFB5x t4O7L7jEV67yz6UlqrPYRwLHLhqjL6DfRelated guides
Section titled “Related guides”- Coordinate system matching -- Use
sorted_name_length_pairsto find collections sharing the same coordinate system - Retrieving subsequences by digest -- Expand a
sorted_sequencesdigest and extract subsequences - Seqcol Operations -- Compare collections and retrieve attribute arrays
Cleanup
Section titled “Cleanup”import shutilshutil.rmtree(temp_dir)