Carbon: Decoding the Language of Life
Loubna Ben Allal*,1, Qiuyi Li*,2, Maurizio Fiusco*,3,4, Lewis Tunstall*,1, Kashif Rasul*,1, Ed Beeching1,
Dana Aubakirova1, Carlos Pati˜no1, Thibaud Frere1, Anton Lozhkov1, Georgia Channing1,
Thomas Wolf1, Diego di Bernardo†,3,4, Leandro von Werra†,1
1Hugging Face 2Zhongguancun Academy 3TIGEM 4University of Naples Federico II
*Equal core contribution †Senior author
huggingface.co/collections/HuggingFaceBio/carbon
ABSTRACT
Genomic foundation models have emerged alongside the rapid progress of large language models,
offering a promising framework for learning general-purpose sequence priors for DNA understand-
ing, generation, and design. This connection to LLMs creates a major opportunity: modern archi-
tectures, scaling infrastructure, autoregressive training, and token-based modeling provide powerful
tools for genomic sequence modeling. At the same time, DNA differs fundamentally from natu-
ral language. Genomic sequences are noisy, redundant, sparsely constrained, unevenly annotated,
and shaped by evolutionary rather than communicative pressures. As a result, key components of
the standard LLM recipe, including data construction, tokenization, and training objectives, must
be reconsidered in the biological sequence setting. A central challenge in DNA modeling is rec-
onciling single-nucleotide resolution with long-context reasoning. Single-nucleotide resolution is
essential for variant effect prediction, splice-site analysis, and codon-level reasoning. Long-context
modeling is equally important, as many genomic mechanisms depend on distal regulatory elements,
gene neighborhoods, and long-range evolutionary constraints. However, the most direct path to
nucleotide-level reasoning, single-nucleotide tokenization, makes genomic sequences extremely
long and imposes substantial computational cost on Transformer models. We present CARBON,
a family of efficient generative DNA language models designed as a practical reference point for
this setting. CARBON includes 3B- and 8B-parameter decoder-only autoregressive models using
non-overlapping 6-mer tokenization. CARBON-3B supports a maximum context length of 65,536
tokens, corresponding to approximately 393 kbp of DNA; CARBON-8B supports up to 131,072 to-
kens, roughly 786 kbp. This simple and controlled setup helps isolate a central question for DNA
language modeling: whether current progress is limited primarily by model architecture and nom-
inal context length, or by more basic alignment between data, tokenization, objectives, evaluation,
and the biological structure of genomic sequence. In our training-free evaluation suite, CARBON-3B
is competitive with Evo2-7B despite having less than half the parameters. CARBON-8B improves
on CARBON-3B on every training-free task, with the largest gain on long-context retrieval. Both
models deliver tens-fold faster inference under comparable settings. The CARBON recipe combines
annotation-aware data curation, deterministic 6-mer tokenization, and a staged CE-to-FNS objective
schedule, adapting the LLM recipe to the statistical and biological properties of DNA rather than
directly transplanting it. We release the models, data, training code, and evaluation suite, including
new training-free probes for sequence-level perturbation and DNA long-context retrieval. CARBON
is intended as an open recipe for efficient generative DNA modeling rather than an argument for any
specific architecture, tokenization strategy, or objective design as the optimal solution. Its strong
performance provides grounded evidence that substantial room remains for domain-aware model
design carefully aligned with the genomic sequence itself.
Correspondence to: loubna@hf.co, lewis@hf.co, leandro@hf.co, qiuyi.li1993@gmail.com, dibernardo@tigem.it
Loubna Ben Allal*,1, Qiuyi Li*,2, Maurizio Fiusco*,3,4, Lewis Tunstall*,1, Kashif Rasul*,1, Ed Beeching1,
Dana Aubakirova1, Carlos Pati˜no1, Thibaud Frere1, Anton Lozhkov1, Georgia Channing1,
Thomas Wolf1, Diego di Bernardo†,3,4, Leandro von Werra†,1
1Hugging Face 2Zhongguancun Academy 3TIGEM 4University of Naples Federico II
*Equal core contribution †Senior author
huggingface.co/collections/HuggingFaceBio/carbon
ABSTRACT
Genomic foundation models have emerged alongside the rapid progress of large language models,
offering a promising framework for learning general-purpose sequence priors for DNA understand-
ing, generation, and design. This connection to LLMs creates a major opportunity: modern archi-
tectures, scaling infrastructure, autoregressive training, and token-based modeling provide powerful
tools for genomic sequence modeling. At the same time, DNA differs fundamentally from natu-
ral language. Genomic sequences are noisy, redundant, sparsely constrained, unevenly annotated,
and shaped by evolutionary rather than communicative pressures. As a result, key components of
the standard LLM recipe, including data construction, tokenization, and training objectives, must
be reconsidered in the biological sequence setting. A central challenge in DNA modeling is rec-
onciling single-nucleotide resolution with long-context reasoning. Single-nucleotide resolution is
essential for variant effect prediction, splice-site analysis, and codon-level reasoning. Long-context
modeling is equally important, as many genomic mechanisms depend on distal regulatory elements,
gene neighborhoods, and long-range evolutionary constraints. However, the most direct path to
nucleotide-level reasoning, single-nucleotide tokenization, makes genomic sequences extremely
long and imposes substantial computational cost on Transformer models. We present CARBON,
a family of efficient generative DNA language models designed as a practical reference point for
this setting. CARBON includes 3B- and 8B-parameter decoder-only autoregressive models using
non-overlapping 6-mer tokenization. CARBON-3B supports a maximum context length of 65,536
tokens, corresponding to approximately 393 kbp of DNA; CARBON-8B supports up to 131,072 to-
kens, roughly 786 kbp. This simple and controlled setup helps isolate a central question for DNA
language modeling: whether current progress is limited primarily by model architecture and nom-
inal context length, or by more basic alignment between data, tokenization, objectives, evaluation,
and the biological structure of genomic sequence. In our training-free evaluation suite, CARBON-3B
is competitive with Evo2-7B despite having less than half the parameters. CARBON-8B improves
on CARBON-3B on every training-free task, with the largest gain on long-context retrieval. Both
models deliver tens-fold faster inference under comparable settings. The CARBON recipe combines
annotation-aware data curation, deterministic 6-mer tokenization, and a staged CE-to-FNS objective
schedule, adapting the LLM recipe to the statistical and biological properties of DNA rather than
directly transplanting it. We release the models, data, training code, and evaluation suite, including
new training-free probes for sequence-level perturbation and DNA long-context retrieval. CARBON
is intended as an open recipe for efficient generative DNA modeling rather than an argument for any
specific architecture, tokenization strategy, or objective design as the optimal solution. Its strong
performance provides grounded evidence that substantial room remains for domain-aware model
design carefully aligned with the genomic sequence itself.
Correspondence to: loubna@hf.co, lewis@hf.co, leandro@hf.co, qiuyi.li1993@gmail.com, dibernardo@tigem.it
Carbon Technical Report
1 Introduction200 500 1k 2k 5k 10k 20k 50k 100k 200k
Throughput
0
20
40
60
80
100
Win rate (%)
Base pairs per second
better
faster
Evo2 20B
Evo2 7B
Evo2 1B
Carbon 8B
GENERator-v2 3B
GENERator-v2 1.2B
Carbon 3B
Figure 1: CARBON models obtain comparable or better performance than Evo2 7B, while being over 150 times faster.
Win rates computed across seven zero-shot DNA benchmarks. Throughput computed over 256 DNA sequences, with
1080 base pairs for prefill and decode.
Genomic DNA encodes protein-coding genes, regulatory elements, non-coding RNAs, repetitive structures, and evo-
lutionary constraints accumulated over long evolutionary timescales [50]. A foundation model that can learn the dis-
tribution of genomic sequence has the potential to support a broad range of biological applications, including sequence
completion, variant interpretation, functional annotation, comparative genomics, and programmable DNA design. In-
spired by the rapid progress of large language models [67, 16, 1] and protein language models [37, 48], genomic
foundation models have become an increasingly active direction for learning general-purpose sequence priors directly
from DNA [24, 12, 11, 68, 9].
In this technical report, we present CARBON, a family of efficient generative DNA language models. CARBON in-
cludes 3B- and 8B-parameter decoder-only autoregressive models with non-overlapping 6-mer tokenization. The
architecture follows a standard open-weight Transformer design (RMSNorm, SwiGLU, RoPE, GQA), as described
in Section 5.1. CARBON-3B supports a maximum context length of 65,536 tokens (approximately 393 kbp of DNA),
and CARBON-8B extends to 131,072 tokens (approximately 786 kbp). While recent genomic models often use long-
convolution, state-space, or hybrid sequence architectures [46, 61, 6, 11], CARBON shows that a standard Transformer
backbone combined with compact 6-mer tokenization can achieve strong performance while supporting long-context
inference with substantially higher throughput. This choice also makes CARBON easier to use with familiar open-
source LLM tools, and keeps the empirical focus on the components we ablate directly: data curation, tokenization,
and the training objective. Across our training-free benchmarks, CARBON-3B is competitive with Evo2-7B [11] at
less than half the parameter count. CARBON-8B beats Evo2-7B on the majority of variant-effect prediction, sequence
generation, and perturbation tasks, and leads on long-context retrieval at 786 kbp. Both models run tens-fold faster at
inference.
To build CARBON, we studied how the standard LLM recipe should be adapted for DNA. The most important dif-
ferences we found arise at three levels: data, tokenization, and the training objective. The first difference is the
signal-to-noise structure of genomic sequences. Unlike natural language text, which is produced for communication
and is therefore relatively compact and information-dense, genomes are the product of evolutionary processes shaped
by mutation, genetic drift, selection, duplication, transposition, and evolutionary contingency. Large portions of raw
genomic sequence can be weakly constrained, repetitive, or only sparsely informative for the biological capabilities we
2
1 Introduction200 500 1k 2k 5k 10k 20k 50k 100k 200k
Throughput
0
20
40
60
80
100
Win rate (%)
Base pairs per second
better
faster
Evo2 20B
Evo2 7B
Evo2 1B
Carbon 8B
GENERator-v2 3B
GENERator-v2 1.2B
Carbon 3B
Figure 1: CARBON models obtain comparable or better performance than Evo2 7B, while being over 150 times faster.
Win rates computed across seven zero-shot DNA benchmarks. Throughput computed over 256 DNA sequences, with
1080 base pairs for prefill and decode.
Genomic DNA encodes protein-coding genes, regulatory elements, non-coding RNAs, repetitive structures, and evo-
lutionary constraints accumulated over long evolutionary timescales [50]. A foundation model that can learn the dis-
tribution of genomic sequence has the potential to support a broad range of biological applications, including sequence
completion, variant interpretation, functional annotation, comparative genomics, and programmable DNA design. In-
spired by the rapid progress of large language models [67, 16, 1] and protein language models [37, 48], genomic
foundation models have become an increasingly active direction for learning general-purpose sequence priors directly
from DNA [24, 12, 11, 68, 9].
In this technical report, we present CARBON, a family of efficient generative DNA language models. CARBON in-
cludes 3B- and 8B-parameter decoder-only autoregressive models with non-overlapping 6-mer tokenization. The
architecture follows a standard open-weight Transformer design (RMSNorm, SwiGLU, RoPE, GQA), as described
in Section 5.1. CARBON-3B supports a maximum context length of 65,536 tokens (approximately 393 kbp of DNA),
and CARBON-8B extends to 131,072 tokens (approximately 786 kbp). While recent genomic models often use long-
convolution, state-space, or hybrid sequence architectures [46, 61, 6, 11], CARBON shows that a standard Transformer
backbone combined with compact 6-mer tokenization can achieve strong performance while supporting long-context
inference with substantially higher throughput. This choice also makes CARBON easier to use with familiar open-
source LLM tools, and keeps the empirical focus on the components we ablate directly: data curation, tokenization,
and the training objective. Across our training-free benchmarks, CARBON-3B is competitive with Evo2-7B [11] at
less than half the parameter count. CARBON-8B beats Evo2-7B on the majority of variant-effect prediction, sequence
generation, and perturbation tasks, and leads on long-context retrieval at 786 kbp. Both models run tens-fold faster at
inference.
To build CARBON, we studied how the standard LLM recipe should be adapted for DNA. The most important dif-
ferences we found arise at three levels: data, tokenization, and the training objective. The first difference is the
signal-to-noise structure of genomic sequences. Unlike natural language text, which is produced for communication
and is therefore relatively compact and information-dense, genomes are the product of evolutionary processes shaped
by mutation, genetic drift, selection, duplication, transposition, and evolutionary contingency. Large portions of raw
genomic sequence can be weakly constrained, repetitive, or only sparsely informative for the biological capabilities we
2
Carbon Technical Report
want the model to acquire. This motivates annotation-aware data curation [68, 11] rather than treating raw nucleotide
count as the only measure of data quality.
The second difference is the absence of stable word boundaries. BPE and related subword tokenizers [26] are pow-
erful in natural language because repeated character patterns often correspond to reusable lexical or sublexical units.
DNA contains motifs, but these motifs are often degenerate, context-dependent, strand-sensitive, length-variable, and
tolerant to substitutions or insertions [5]. In our experiments, this made learned BPE tokenization poorly matched to
autoregressive DNA generation. Non-overlapping 6-mer tokenization [68, 33] provided a more stable alternative: it
fixes the prediction step length, avoids segmentation ambiguity, reduces token length by a factor of six, and substan-
tially improves inference efficiency with little performance cost.
The third difference is the structure of the training objective. Standard next-token cross-entropy works remarkably
well in natural language modeling, where tokenizer units often behave as meaningful prediction targets. In a 6-mer
DNA model, however, cross-entropy treats each 6-mer as an atomic class among 4096 possibilities. This creates an
all-or-nothing supervision signal: a prediction that matches five out of six nucleotides is penalized in the same way
as a completely unrelated 6-mer. During CARBON training, we found that this exact-token objective can lead to a
persistent “loss staircase” phenomenon and increased sensitivity to numerical precision in late-stage training. Fac-
torized Nucleotide Supervision (FNS, [33]) addresses this problem by exposing nucleotide-level supervision through
probability marginalization, providing structured partial credit while preserving the efficiency of 6-mer tokenization.
In practice, switching from cross-entropy to FNS at the appropriate stage stabilized training and allowed the model to
continue improving.
Together, these observations suggest that DNA language modeling is not simply natural language modeling with a four-
letter alphabet. The standard LLM recipe remains extremely valuable as a starting point, including the use of large-
scale data, token-based sequence representation, and next-token cross-entropy. However, these components need to be
reconsidered in the biological sequence setting. For CARBON, this led to an annotation-aware mixed corpus, focused
primarily on eukaryotic genes, non-overlapping 6-mer tokenization, and a staged CE-to-FNS objective schedule.
Developing this recipe required ablations guided by evaluations that are stable, inexpensive, and directly comparable
across pre-training runs. DNA foundation models are commonly evaluated through a mixture of zero-shot likelihood-
based benchmarks, supervised finetuning tasks, and embedding-based analyses [11, 68, 9]. Each evaluation mode
emphasizes a different aspect of model behavior: finetuning benchmarks measure task-specific transfer, embedding
analyses probe learned representations, and zero-shot likelihood-based tasks test the base model directly without ad-
ditional training. For pre-training ablations and base-model comparison, we focus on zero-shot, training-free evalu-
ations, since they avoid confounding factors introduced by finetuning protocols or embedding classifiers. To select
reliable DNA probes, we follow criteria used in recent LLM evaluation work [53]: a useful task should score above
random, improve with training, exhibit low seed-to-seed noise, and produce broadly consistent model rankings across
checkpoints. The resulting suite spans generation (Sequence Recovery [68]) and variant-effect prediction (ClinVar
coding and non-coding [29, 7], BRCA2 [23], TraitGym [8]). To cover capabilities not addressed by existing public
benchmarks, we additionally introduce two sequence-level perturbation probes (nucleotide triplet-expansion and syn-
onymous codon replacement) and Genomic-NIAH, a DNA long-context retrieval benchmark built from real genomic
haystacks.
The performance of CARBON suggests that the field is still far from a settled optimum: a relatively standard Trans-
former can match much larger and longer-context baselines. This means that there is substantial room for progress
through careful algorithmic and data-centric work, not only by increasing the parameter count or the nominal context
length. Several questions remain open, including how to construct genomic pre-training data; how to represent DNA
sequences; how to recover nucleotide-level reasoning efficiently; how to measure effective long-context utilization;
and how to connect sequence-level models to practical biological applications.
More broadly, CARBON reinforces the view that DNA language modeling should not be treated as a direct transplant
of natural language modeling. LLM technology provides powerful tools, but DNA sequence is governed by different
statistical and biological principles. The purpose of this report is therefore not only to introduce a strong model
family, but also to share the implementation path, empirical observations, and design considerations behind it. We
hope CARBON can serve as a grounded baseline for efficient generative DNA modeling and help clarify where the
field currently stands, what a strong practical baseline can look like, and which questions may matter most for future
progress.
Main contributions. The main contribution of this report is an open, evaluation-driven recipe for efficient generative
DNA modeling. We release the resulting models, training data, training code, and evaluation suite, together with
ablations showing how data, tokenization, and the training objective shape performance.
3
want the model to acquire. This motivates annotation-aware data curation [68, 11] rather than treating raw nucleotide
count as the only measure of data quality.
The second difference is the absence of stable word boundaries. BPE and related subword tokenizers [26] are pow-
erful in natural language because repeated character patterns often correspond to reusable lexical or sublexical units.
DNA contains motifs, but these motifs are often degenerate, context-dependent, strand-sensitive, length-variable, and
tolerant to substitutions or insertions [5]. In our experiments, this made learned BPE tokenization poorly matched to
autoregressive DNA generation. Non-overlapping 6-mer tokenization [68, 33] provided a more stable alternative: it
fixes the prediction step length, avoids segmentation ambiguity, reduces token length by a factor of six, and substan-
tially improves inference efficiency with little performance cost.
The third difference is the structure of the training objective. Standard next-token cross-entropy works remarkably
well in natural language modeling, where tokenizer units often behave as meaningful prediction targets. In a 6-mer
DNA model, however, cross-entropy treats each 6-mer as an atomic class among 4096 possibilities. This creates an
all-or-nothing supervision signal: a prediction that matches five out of six nucleotides is penalized in the same way
as a completely unrelated 6-mer. During CARBON training, we found that this exact-token objective can lead to a
persistent “loss staircase” phenomenon and increased sensitivity to numerical precision in late-stage training. Fac-
torized Nucleotide Supervision (FNS, [33]) addresses this problem by exposing nucleotide-level supervision through
probability marginalization, providing structured partial credit while preserving the efficiency of 6-mer tokenization.
In practice, switching from cross-entropy to FNS at the appropriate stage stabilized training and allowed the model to
continue improving.
Together, these observations suggest that DNA language modeling is not simply natural language modeling with a four-
letter alphabet. The standard LLM recipe remains extremely valuable as a starting point, including the use of large-
scale data, token-based sequence representation, and next-token cross-entropy. However, these components need to be
reconsidered in the biological sequence setting. For CARBON, this led to an annotation-aware mixed corpus, focused
primarily on eukaryotic genes, non-overlapping 6-mer tokenization, and a staged CE-to-FNS objective schedule.
Developing this recipe required ablations guided by evaluations that are stable, inexpensive, and directly comparable
across pre-training runs. DNA foundation models are commonly evaluated through a mixture of zero-shot likelihood-
based benchmarks, supervised finetuning tasks, and embedding-based analyses [11, 68, 9]. Each evaluation mode
emphasizes a different aspect of model behavior: finetuning benchmarks measure task-specific transfer, embedding
analyses probe learned representations, and zero-shot likelihood-based tasks test the base model directly without ad-
ditional training. For pre-training ablations and base-model comparison, we focus on zero-shot, training-free evalu-
ations, since they avoid confounding factors introduced by finetuning protocols or embedding classifiers. To select
reliable DNA probes, we follow criteria used in recent LLM evaluation work [53]: a useful task should score above
random, improve with training, exhibit low seed-to-seed noise, and produce broadly consistent model rankings across
checkpoints. The resulting suite spans generation (Sequence Recovery [68]) and variant-effect prediction (ClinVar
coding and non-coding [29, 7], BRCA2 [23], TraitGym [8]). To cover capabilities not addressed by existing public
benchmarks, we additionally introduce two sequence-level perturbation probes (nucleotide triplet-expansion and syn-
onymous codon replacement) and Genomic-NIAH, a DNA long-context retrieval benchmark built from real genomic
haystacks.
The performance of CARBON suggests that the field is still far from a settled optimum: a relatively standard Trans-
former can match much larger and longer-context baselines. This means that there is substantial room for progress
through careful algorithmic and data-centric work, not only by increasing the parameter count or the nominal context
length. Several questions remain open, including how to construct genomic pre-training data; how to represent DNA
sequences; how to recover nucleotide-level reasoning efficiently; how to measure effective long-context utilization;
and how to connect sequence-level models to practical biological applications.
More broadly, CARBON reinforces the view that DNA language modeling should not be treated as a direct transplant
of natural language modeling. LLM technology provides powerful tools, but DNA sequence is governed by different
statistical and biological principles. The purpose of this report is therefore not only to introduce a strong model
family, but also to share the implementation path, empirical observations, and design considerations behind it. We
hope CARBON can serve as a grounded baseline for efficient generative DNA modeling and help clarify where the
field currently stands, what a strong practical baseline can look like, and which questions may matter most for future
progress.
Main contributions. The main contribution of this report is an open, evaluation-driven recipe for efficient generative
DNA modeling. We release the resulting models, training data, training code, and evaluation suite, together with
ablations showing how data, tokenization, and the training objective shape performance.
3
- Xet hash:
- 9758fd5690f9a56d24deee7f8a754e3eccc0091eeb3c9cbf5e3272979fa9b8aa
- Size of remote file:
- 4.07 MB
- SHA256:
- 19f6584884aeb1690838b38a7cbcad694100565813e43fc422c82ddf6b418d95
·
Xet efficiently stores Large Files inside Git, intelligently splitting files into unique chunks and accelerating uploads and downloads. More info.
