Skip to content

Using HyenaDNA Models in DNALLM

HyenaDNA is a class of genomic foundation models designed for long-range sequence modeling at single-nucleotide resolution. It is notable for its attention-free architecture, which replaces the quadratic-cost attention mechanism of Transformers with long convolutions. This allows it to scale to extremely long sequences (up to 1 million tokens).

DNALLM Examples: HyenaDNA

1. Architecture Overview

HyenaDNA is based on the Hyena operator, a sub-quadratic alternative to attention.

  • Attention-Free: Instead of using self-attention, HyenaDNA relies on implicit long convolutions parameterized by a small neural network. This design choice significantly reduces computational complexity from O(N²) to O(N log N), where N is the sequence length.
  • Causal Language Modeling: It is pre-trained as a causal (autoregressive) language model, meaning it predicts the next nucleotide in a sequence given the preceding ones.
  • Single-Nucleotide Resolution: The model operates directly on single characters (A, C, G, T, N), avoiding k-mer tokenization and preserving the full resolution of the genomic sequence.

This architecture makes HyenaDNA exceptionally well-suited for tasks involving very long-range dependencies, such as modeling entire genes or regulatory regions.

2. Environment and Installation

HyenaDNA models load through the standard DNALLM installation; no additional dependencies are required.

Installation

pip install dnallm

3. Model Loading and Configuration

HyenaDNA models are registered in the DNALLM model registry (dnallm/models/modeling_auto.py) and load through the generic AutoModelForCausalLM route in load_model_and_tokenizer.

Loading a Model

Here’s how to load a HyenaDNA model for a causal language modeling task. 

from dnallm.configuration.configs import TaskConfig
from dnallm.models import load_model_and_tokenizer

# Use a specific HyenaDNA model
model_name = "LongSafari/hyenadna-small-32k-seqlen-hf"

# HyenaDNA is pre-trained as a causal (autoregressive) language model
task_config = TaskConfig(task_type="generation")

# Load model and tokenizer (source="huggingface" resolves the bare repo id)
model, tokenizer = load_model_and_tokenizer(model_name, task_config, source="huggingface")

print("Model:", type(model))
print("Tokenizer:", type(tokenizer))

4. Inference Example

Let's use a HyenaDNA model to get embeddings for a DNA sequence. 

import torch
from dnallm.configuration.configs import TaskConfig
from dnallm.models import load_model_and_tokenizer

# 1. Load the pre-trained model and tokenizer
model_name = "LongSafari/hyenadna-tiny-1k-seqlen-hf"
task_config = TaskConfig(task_type="generation")
model, tokenizer = load_model_and_tokenizer(model_name, task_config, source="huggingface")
model.eval()

# 2. Prepare and tokenize the DNA sequence
dna_sequence = "GATTACAGATTACAGATTACAGATTACAGATTACAGATTACA"
inputs = tokenizer(dna_sequence, return_tensors="pt")

# 3. Perform inference
with torch.no_grad():
    outputs = model(**inputs)

embeddings = outputs.hidden_states[-1]  # Hyena outputs hidden states differently
print("Shape of embeddings:", embeddings.shape)