Emergence of Biological Structural Discovery in General-Purpose Language Models
Abstract
Large language models (LLMs) are evolving into engines for scientific discovery, yet the assumption that biological understanding requires domain-specific pre-training remains largely unchallenged. Here we report that general-purpose LLMs possess an emergent capability for biological structural discovery. Under strict, shortcut-controlled evaluation, a small-scale GPT-2 (124M) fine-tuned solely on English paraphrase discrimination detects protein homology zero-shot at ROC-AUC 0.79 on a shortcut-controlled benchmark. Controls establish that the ability is conferred by pre-training, not architecture: a randomly initialized GPT-2 is at chance (0.52). To exclude the possibility that public checkpoints were contaminated with biological data, we train our own GPT-2 from scratch on an English-only web corpus; it reproduces the transfer (0.76), proving the effect arises from linguistic pre-training alone. Network-based interpretability reveals a deep structural isomorphism: the discriminative signal localizes to deep layers (0.97 at layer 9), and attention analysis surfaces modality-agnostic "difference" operators. Scaling to massive instruction-tuned models further improves performance, including in the remote-homology "twilight zone", which we report as an exploratory upper bound because those models' training corpora are undisclosed. We formalize these tasks through the BioPAWS benchmark. Our controlled results, obtained entirely on models with known training data, establish that abstract logical structures distilled from human language constitute a genuine, if bounded, cognitive prior for decoding the syntax of biology.
Lifecycle
- biorxiv v2 2026-07-09 source ↗
Discussion
Moderated digest of third-party discussion on Bluesky — substantive endorsement and critique. 2 post(s) filtered (self-promotion, off-topic, or low-substance).
-
AI x Bio Discovery @aixbiobot.bsky.social · 787 followers neutral
Emergence of Biological Structural Discovery in General-Purpose Language Models [new] LMs: Discover bio structures & protein homology via structural understanding sans domain-specific training.