We set out to audit the reinforcement learning results for IDiom, a recent protein language model for designing intrinsically disordered regions from the Rotskoff and Dunn labs at Stanford. Originally, we wanted to explore whether there was any reward hacking, owing to the susceptibility of RL-optimized generative models to exploit shallow features in their reward signals. Instead, we found something more interesting: the biological distinctions that emerge after RL are neither reproduced by simple base-model filtering nor fully resolved by the reward model itself.

The Setup

IDiom is a 122M parameter autoregressive model trained on 37 million intrinsically disordered region sequences from the AlphaFold Database. After pretraining, Liu et al. post-train the model with GRPO on ProtGPS — a localization predictor built on ESM2 embeddings with the goal of steering IDiom to generate disordered sequences localizing to specific subcellular compartments. The paper presents results for four compartments: nucleolus, chromosomes, P-bodies, and stress granules.

We run four additional computational controls.

Test 1: Scrambling

For each RL-optimized set, we shuffled amino acids while preserving exact composition and scored with ProtGPS. We also tested block shuffles (chunks of 10-20 residues rearranged) to preserve local motifs while disrupting long-range order. This was done to determine precisely how much ProtGPS cares about sequence order. Results were mostly unsurprising— ProtGPS is not purely compositional, and the block-shuffle results suggest that much of the signal lives in local motif structure rather than residue-level global order.

ProtGPS cares a lot about sequence order for the nucleolus and stress granule. For the P-body and chromosome, composition retains most of the score. ProtGPS is not a pure composition detector, but composition does most of the work for some compartments.

Test 2: Shallow Feature Probe

We then train a random forest regressor to predict ProtGPS scores from 35 simple features like amino acid frequencies, motif counts, charge, and entropy. This was done to understand whether ProtGPS can be gamed by relatively simple features.

When we hold out an entire RL cellular subcompartment and test on it, R² collapses to deeply negative values. Clearly, the shallow model completely fails to generalize to RL distributions it hasn’t seen.

Test 3: Specificity

The paper states on page 12, “…these results demonstrate that RL post-training using the ProtGPS predictor as a reward model is able to teach IDiom the global amino acid compositions, sequence patterning, and specific motifs which are necessary for compartment-specific localization.” Because ProtGPS returns twelve compartment scores, the natural specificity check is not only whether the optimized target score increases, but whether the same sequences also score highly for other ProtGPS compartments. The paper reports the target optimization and downstream sequence features, but not the full twelve-compartment off-target vectors. We score every RL sequence across all 12 ProtGPS compartments. Here’s where it gets interesting, specifically for Stress Granules.

Stress granule RL sequences score 84% on stress granule, and 77% on P-body. This is surprising: unlike the others, the median specificity is near zero! According to ProtGPS, these sequences are roughly equally likely to localize to two different compartments.

At this point, we thought we had our finding: stress granule RL targeting doesn’t work across these two components. The RL learned a generic RNA-granule-adjacent profile that ProtGPS scores high across both compartments. At least according to the reward function in the paper! One may argue that RL optimized each set to maximize its target compartment score on ProtGPS. So ProtGPS should be maximally favorable to showing specificity — each set was literally trained to score high on its target! If ProtGPS can't show specificity even under these ideal conditions, what does this tell us about the reward function?

But then we landed on something more surprising.

Before explaining what it is, a caveat. P-bodies and stress granules are related but distinct cytoplasmic mRNP granules. Prior work shows that they can physically associate, share some components, harbor overlapping mRNA species, and participate in a dynamic mRNA cycle in which mRNPs move between polysomes, stress granules, and P-bodies (Kedersha et al., 2005; Decker & Parker, 2012). At the same time, they retain distinct molecular compositions: P-bodies are enriched for mRNA decay and repression machinery, whereas stress granules are enriched for stalled translation-initiation complexes and factors such as TIA-1 and G3BP (Kedersha et al., 2005). Some ProtGPS cross-reactivity between P-body and stress-granule scores is biologically plausible — but it should be interpreted as overlap between related RNA-granule programs, not evidence that the compartments are interchangeable.

Test 4: The unexpected finding

We then evaluated the RL sequences with DeepLoc 2.0, another localization predictor built on ESM1b and trained with a separate supervised objective over broad subcellular localization classes. Unlike ProtGPS, which predicts twelve condensate-associated compartments, DeepLoc predicts 10 broad subcellular classes. It doesn’t have P-body or stress granule as separate classes — both fall under “Cytoplasm” — so it can’t directly validate sub-compartment targeting. But the full ten-class probability vectors can be used to ask a simpler question: do P-body and stress granule sequences look the same or different?

We trained a logistic regression model on DeepLoc’s ten-class probability vectors to classify sequences by their RL origin.

83% accuracy. DeepLoc clearly distinguishes the two sets.

Clearly, the RL-generated stress granule and P-body sequences are not the same. They differ in ways that an independent predictor can detect: P-body sequences show higher Cytoplasm probability (0.55 vs 0.47), stress granule sequences show higher Nucleus probability (0.63 vs 0.57), consistent with the known nuclear-cytoplasmic shuttling behavior of stress granule proteins. But IDiom’s reward function cannot distinguish between which compartment the RL model trained on stress granules is supposed to go to.

Where does the distinction come from?

Our first assumption was prior preservation: the pretrained model already encodes P-body vs stress-granule differences, and the regularizers mentioned, such as the KL penalty, kept them intact through RL. We then tested this directly.

Using 10K base-IDiom sequences already scored by ProtGPS, we selected the top-scoring sequences for each compartment via rejection sampling and ran the same DeepLoc classifier.

Clearly, that isn’t the case, as the prior alone doesn’t produce the separation. High-ProtGPS-scoring base sequences are not DeepLoc-distinguishable between P-bodies and stress granules. The 83% separation emerges only after RL optimization. Something about the overall training creates distinctions that neither the base prior nor the reward model individually contains.

Disentangling this further — whether interaction effects between the KL penalty & other regularizers and the reward landscape are responsible — remains an open question.

Also, this is not an indictment of ProtGPS as a localization predictor. Across most provided target-output pairs, the off-target scores are low, and the cross-reactivity we observe is concentrated in biologically related compartments. The problem is the way ProtGPS is used as a reward. Optimizing a single compartment score does not penalize sequences that also score highly for a related compartment. A specificity-aware reward may resolve this.

Implications

For IDiom specifically, the paper’s direct post-training readout is the optimized ProtGPS target score, supplemented by composition and motif analyses of the generated sequences. That does not make the result invalid, but it leaves an important identification gap: target-score improvement alone cannot tell us whether the generated sequences are selectively compartment-specific, broadly high-scoring across related compartments, or exploiting shortcuts in the reward model. Adding off-target ProtGPS scores changes the interpretation. Future evaluations should report the full twelve-compartment ProtGPS score vector to evaluate the extent of off-target effects.

For RL on protein language models, the relationship between pretraining, reward modeling, and post-training deserves more attention. In IDiom, the P-body/stress-granule separation we observed is not well explained by the ProtGPS reward alone, but it is also not reproduced by simply filtering base-model samples for high ProtGPS scores. In our rejection-sampling test, high-ProtGPS base sequences remained only weakly separable by DeepLoc, while RL-generated P-body and stress-granule sequences were strongly separable. This suggests that RL changed the generator beyond what the scalar ProtGPS reward could evaluate.

For RL alignment more broadly, this illustrates the limits of combining reward improvement with hand-selected feature analyses: those checks can support a biological interpretation, but they may still miss distributional structure that appears under independent evaluation. At the same time, base-model rejection sampling did not reproduce that separation, so the distinction cannot be attributed to the pretrained prior alone.

Methods

All sequences were obtained from the IDiom HuggingFace repository (datasets: jxliu2/idiom-datasets; models: jxliu2/idiom). ProtGPS scoring used the checkpoint bundled with the IDiom model repository. DeepLoc 2.0 predictions used the Fast model (ESM1b) from the public GitHub repository. Scrambled sequences were generated by random permutation of amino acids preserving exact composition and length, and block shuffles used chunks of 10-20 residues. (3 full scrambles + 3 block shuffles per sequence). The random forest probe used scikit-learn’s RandomForestRegressor (500 trees, max depth 15) with an 80/20 train/test split. The P-body vs stress granule classifier was a logistic regression on DeepLoc’s ten-class probability vectors with 5-fold cross-validation. Specificity was defined as the target compartment score minus the maximum off-target compartment score. Cross-compartment correlations are Pearson correlations computed across all RL and base model sequences. Code is available at: Github-IdIom reviewer

TL;DR: IDiom is still pretty cool. Our audit does not show that RL failed, or that the generated IDRs are meaningless. It shows that for some targets, their reward gains and hand-picked motif checks are not enough to establish fine-grained compartment specificity. The reward model blurs some related compartments, while still inducing separable biological structure.

Thanks to Richard Dzeng for providing feedback on an early draft.

Note: I shared this analysis with the IDiom authors before public promotion and received a gracious response. I’ll update the post if further context or corrections come up.

The models have been getting better at generating first-pass binders. Brian Naughton’s guide walks you through designing a VHH from scratch: pick a target, run your generative model of choice, filter by your in-silico structure metric of choice, and send it away to the lab for in vitro testing.

As Nick Boyd(founder of Escalante) mentions, there are currently two main approaches to computational protein binder design: optimization (exemplified by BindCraft) and generative models (like BoltzGen). At the risk of repeating something already mentioned, BoltzGen is faster, but the per-binder design quality is much lower than that of something like BindCraft, which has the opposite properties. So the net computational cost is roughly the same. In his essay, Boyd shows how you can improve upon BoltzGen by borrowing the LLM posttraining playbook — finetune on high-quality hallucinated binders from optimization-based methods, apply GRPO, and watch your binder quality improve.

It would seem as if the binder generation problem has been solved, but that’s far from the case.

Leash Bio showed that when we analyze what ChEMBL-trained models actually learn, it’s less about molecular binding and more about which medicinal chemists tend to make which kinds of molecules. Their model Hermes — a lightweight 50M parameter sequence-only transformer trained entirely on combinatorially synthesized molecules with no human design intent, outperforms Boltz2 on out-of-distribution chemistry despite being several orders of magnitude smaller.

There are uncomfortable parallels to be observed. ChEMBL is biased by what medicinal chemists chose to synthesize: their preferences, their intuitions, their career-long optimization toward molecules that look drug-like to a human expert. PDB has the same problem. It’s biased by what structural biologists chose to crystallize: proteins that are stable enough to survive purification, well-behaved enough to form crystals, interesting enough to justify the grant money. Both are human-curated snapshots of a tiny corner of an astronomically vast space.

As Nick Boyd emphasizes in his essay, BoltzGen, Protenix, and ProteinMPNN are all downstream of PDB— structures of what structural biologists chose to crystallize. When you apply GRPO with a reward signal that's just PDB all the way down, can you confidently say you’re truly learning what makes a good binder in a general sense? Or are we optimizing for something a structural biologist would have crystallized?

In this article, we explore whether applying reinforcement learning causes a collapse in generated binders. We generated 200 structures per condition across three model checkpoints — base BoltzGen, an SFT checkpoint finetuned on high-quality Mosaic hallucinated binders, and an RL checkpoint trained on top of that — and evaluated structural diversity using Foldseek across five targets spanning in-distribution (ACE2, CCL2, PDL1), near-OOD (EGFR), and far-OOD (KRAS) regimes.

Methods

Trimming the targets

For PDL1, CCL2, and KRAS, full protein sequences were used. For EGFR and ACE2, truncations were necessary to fit within GPU memory constraints on an RTX 5090. For EGFR, residues 190–505 were retained, preserving domains II and III — the core dimerization arm and primary ligand-binding domain. For ACE2, the full M2 peptidase domain was kept, covering the entire SARS-CoV-2 binding interface. Both truncated structures were refolded with AlphaFold3 to confirm iPTM did not drop meaningfully.

Evaluating Structural Diversity amongst the Binders

Once we had binders for all targets, we used Foldseek to perform three steps: database construction, intra-set structural clustering at TM-score threshold 0.5, and easy-search against PDB100. We report Shannon entropy of the cluster distribution as our primary diversity metric, cluster diversity (unique clusters / N) as a secondary metric, and mean TM-score to nearest PDB hit as our PDB sociology signal.

Measuring Collapse: Two Lenses on the Same Problem

To measure structural diversity within each generated set, we compute the Shannon entropy of the Foldseek cluster distribution:

where p_i is the fraction of structures belonging to cluster i. Higher entropy means the generated set spans many distinct structural solutions. Lower entropy means the set has collapsed toward a small number of dominant folds --- in the extreme case of PDL1 baseline, a single cluster, giving H = 0.

To measure how PDB-like the generated structures are, we run each binder against PDB100 using Foldseek’s structural search and record the TM-score of the nearest hit. TM-score ranges from 0 to 1, where scores above 0.5 indicate the same overall fold and scores above 0.8 indicate near-identical structures.

Results

PDL1 - Clones all the way down

On PDL1, base BoltzGen produced 200 structurally distinct binders that Foldseek collapsed into a single cluster with a Shannon entropy of 0.0. We also see the Mean TM-score to nearest PDB hit: 0.946. The model had already memorized the answer before any RL ever happened.

This could be due to its possible overrepresentation: it’s one of the most studied drug targets in structural biology, with drugs like atezolizumab and durvalumab, all having crystal structures in PDB. This reinforces our hypothesis that the training data is so saturated with examples that the model can’t generate anything else.

For less PDB-saturated targets, RL induces measurable collapse.

On KRAS, an oncology target with far fewer known binders in PDB, we see base BoltzGen generates a wide range of binders: 29 clusters from 200 structures. After RL finetuning, this collapses to 10 clusters and a 68% reduction in structural entropy. We see the pattern in CCL2 as well, where the model generates fewer diverse structures.

On EGFR, finetuning increases structural diversity relative to base BoltzGen (entropy 1.06 → 2.14), before RL collapses it back down. The RL PDB TM-score actually decreases on EGFR, same as PDL1. Finetuning breaks that specific memorized mode and recovers diversity, before RL imposes a new, different form of collapse. Whether this recovered diversity is meaningful or simply a different memorized mode is an open question we can't resolve without experimental validation.

Conclusion

For targets heavily represented in PDB, the base model is already collapsed, and RL can’t push further, and may slightly, if at all, break the memorized mode. It is possible that RL’ing for longer horizons may change this. For targets less represented (KRAS, CCL2), the base model does seem to have diversity that RL destroys. This is counterintuitive to what RL should be doing. The reward signal, Protenix iPTM, itself PDB-trained, reinforces structures that resemble PDB proteins, and the severity of collapse tracks how well the target is represented in that training data.

This may have a grim therapeutic implication for hard-to-drug targets. KRAS is one of the most clinically important undruggable oncology targets because its shallow binding pocket requires non-helical contacts — beta-strand mimetics, cyclic peptides, and designed loops that can engage the flat RAS surface.¹ An RL-finetuned model that has collapsed toward helical bundle solutions is less likely to design binders specific to KRAS and rather generate its favorite motif, hoping it sticks.

As for whether to use RL, it might be worth doing depending on how similar your target is to the data your model has been trained on.

What comes next

It seems evident, then, that the next task to focus on is harnessing signals that are qualitatively different from adding another PDB-derived term to your loss function, or looking at entropy regularizers. As Nick Boyd discusses, one alternative is to explore physics-based reward signals (e.g., Rosetta energy). Whether integrating that distinction into the RL loop resolves the mode collapse is what I want to explore next.

Thanks to Brian Naughton and Dr Aaron Ring for feedback on an early version of this post.

1. Does Effective Altruism’s emphasis on the future generations, belittle the needs of the current, and if so, is this morally appropriate?

2. Why are calls to people's emotions more ‘effective’ than rationality?

3. How does effective forecasting like Fermicasting provide any plausible benefit? Are people reasonably confident to make meaningful decisions based on the heuristics of other people?

4. Does crypto really have any inherent value? And what is something we can do with crypto that we’d be unable to do with the money?

5. Artificial intelligence is trained on human data. Why then are we outraged when a word-predicting model outputs something outrageous?

6. It is not entirely impossible that artificial intelligence might be better at decision-making than humans, if so, would it be better to align AI to human values, or leave it in some area for independent decision-making?

7. There are several opinions that AI would help create not reduce net jobs. For unskilled blue-collar workers. What are some of these jobs?

8. Is it possible to satisfy the need for human craving for loneliness solely through artificial intelligence?

9. What's the best way to make a positive impact on the world?

10. What's the end goal of humans, is it to optimize individual happiness?

11. What's the best way to form meaningful relationships with people?

12. What are some of the main issues plaguing AI Alignment?

13. Some of the current language models are owned by companies as these are expensive to run. Will Open Source models ever come to be as competitive as them?

14. How should developing countries optimize for development and progress while ensuring that they’re not accelerating climate change?

15. How does one manage to build depth in a specific niche of ai, while managing to stay ‘dangerous’ in other fields

16. What are the best ways to solve problems of distribution like hunger and poverty?

17. What's the best solution to the current problem plaguing chatgpt making up nonexistent information and sources?

18. To what extent of freedom of speech be allowed?

19. With students using chatgpt for almost all essay prompts, what are examples of areas where only humans will be able to make intelligible responses if any?

20. What are some of the best ways to learn about opinions in philosophy, and try to answer questions?

21. With OpenAI constantly patching the different jailbreaks being used to bypass its content policies, 

22. What’s more likely to make people mad: something that’s false or true?

23. Is there a reproducible process for making pop songs that AI can replicate?

24. Will AI ever be able to better understand us than we do ourselves?

25. Is dissociation from emotions better or worse for making decisions? Particularly in a field like friendship or family

26. Should powerful AI systems should behave in the way users want or their creators intend?

27. What's the best way to depolarise society from its current state?

28. The current language models are being trained on data that don't accurately reflect all strata of society, what are the best ways to overcome this?

29. What are some of the best ways to form contrarian ideas that are right?

30. What measures does crypto have once, hypothetically prop-shops start trading crypto and intentionally boost or deflate their value?

31. What are some of the best ways to get better at forecasting?

32. Would Universal basic income promote innovation or deaccelerate it?

33. How do we get better at noticing things overlooked by others

34. Would it be possible to build a programming language that generates additional syntax to solve different needs?

35. What are the best ways to help solve the rising issues of loneliness

36. How does one get to regularly interact and learn from successful people?

37. Is it ever truly possible to overcome our biases, and how can we do this?

38. What are some of the positive outcomes we can get through gene editing, and how can we make an impact in this field?

39. Is there a thing like free thought?

40. Do Animals Dream?

41.  How do various religions differ in the nature and magnitude of their effects?

42. What influences when people to act in accordance with their self-interest and when they don't?

43. How does mental imagery work? How do we improve its function?

44. Do people have different levels of self-control or do they just experience temptation differently?

45. What makes a good life? How do we study this?

46. We remember dreams almost perfectly right after waking up and then the memory rapidly recedes and disappears completely, unless we write them down. This isn’t how normal memories function. So, why the difference?

47. What is “personal productivity” and why does it vary from day to day so much (eg. Weinberger et al 2018)? And why does it not seem to correlate with environmental variables like weather or sleep quality?

48. Does listening to music improve or worsen memory?

49. What is consciousness?

50. What would happen if we could travel faster than the speed of light?

51. How much of our behavior is determined by nature versus nurture?

52. How does language shape the way we think?

53. What makes some memories more vivid than others?

54. What does it really mean to be ‘self-aware’?

55. What laws should be imposed by governments on generative AI, if any?

56. Is rationality a universal trait, or are there cultural differences in what is considered rational behavior?

57. What determines how we perceive time? Is it the same for everyone?

58. How do people make major decisions in their lives? When and why does it come up, and how do they go about making those decisions?

59. Are we all fundamentally selfish, even when we do things for others' benefit? Or are there truly settings (intrinsic and/or exogenous) where we do things that are good for others but bad for both our short-term future and long-term future selves?

60. Do people have different levels of self-control or do they just experience temptation differently?

61. Is there a way to conduct research without bias in funding? How?

62. Would it be feasible for prop trading shops to be owned by the government to ensure market liquidity?

63. What would be the best way to go about building a large language model to rival that of GPT-3?

64. Are we all fundamentally selfish, even when we do things for others' benefit? Or are there truly settings (intrinsic and/or exogenous) where we do things that are good for others but bad for both our short-term future and long-term future selves?

65. What basically goes on in the brain, when we design or think of something ‘new’ or never seen before?

66. Are people able to concentrate more effectively under total silence?

67. what are the factors that influence the speed and accuracy of learning a new language?

68. What is the best way to factor in risk while making uncertain decisions?

69. How does trusting the ‘gut’ work?

70. If we ever find a way to significantly extend the human health span or reverse ageing, what could that post-death society look like? 

71. What are the neural mechanisms underlying consciousness, and how can we study and manipulate these mechanisms?

72. Would it be possible to prevent shrinkage of the brain?

73. How does neuroplasticity differ between the developing brain and the adult brain,

74. What is happening in the brain when a human questions?

75. What is the probability there is microbial-like life (other than from earth) in our solar system?"

76. Is string theory more closely correct than any other current theory of physics?

77. What's the best way to determine if someone would be a good friend for you?

78. How do you ask the right questions?

79. How do I get people to like me? 

80. How do you tell the difference between a preference and a bias 

81. What is the probability that I might be sleep deprived if I wake up before my alarm goes off more than 95% of the time?

82. What do other people subjectively experience when they are thinking? To me it’s like talking to myself (in verbal English sentences) but I'm told that isn't universal.

83. When is self-denial useful in altering your desires, vs satisfying them so you can devote time to other things?

84. How does one define wisdom?

85. What happens to consciousness once you fall asleep?

86. Can charisma be taught?

87. Why is it so hard to predict success?

88. Why are we so fascinated by coincidences?

89. Is It Wrong to Enjoy Yourself While the World Is Burning?

90. Is it more important to help society or to help yourself?

91. how can we stop confusing correlation with causation?

92. Why Do We Want What We Can’t Have?

93. Which Matters More, a First or Last Impression?

94. How do I improve my ability to simulate/guess other people's internal states and future behaviours?

95. How do I work out what I want and what I should do?

96. 2) Would the human race be eradicated if there is a worst-possible-scenario nuclear incident? Or merely a lot of people?

97. What could be the potential downsides of building a universal sign language?

98. How do people ascertain emotions in certain songs?

99. Do animals ever 'ask questions'?

100. When you forget a thought, where does this thought go?