Publications
November 06, 2025
Crystal: Illuminating LLM Abilities on Language and Code
Tianhua Tao, Junbo Li, Bowen Tan, Hongyi Wang, William Marshall, Bhargav M Kanakiya, Joel Hestness, Natalia Vassilieva, Zhiqiang Shen, Eric P. Xing, Zhengzhong Liu
[arXiv]
October 25, 2025
REAP the Experts: Why Pruning Prevails for One-Shot MoE Compression
ICLR, 2026
Mike Lasby, Ivan Lazarevich, Nish Sinnadurai, Sean Lie, Yani Ioannou, Vithursan Thangarasa
[OpenReview] [arXiv]
September 29, 2025
Scaling with Collapse: Efficient and Predictable Training of LLM Families
ICLR, 2026
Shane Bergsma, Bin Claire Zhang, Nolan Dey, Shaheer Muhammad, Gurpreet Gosal, Joel Hestness
[OpenReview] [arXiv]
September 29, 2025
Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMs
July 18, 2025
Wafer-Scale Integration for AI – The Holy Grail?
May 25, 2025
DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding
NeurIPS, 2025
DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding
[OpenReview] [arXiv]
May 19, 2025
Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training
NeurIPS, 2025
Shane Bergsma, Nolan Dey, Gurpreet Gosal, Gavia Gray, Daria Soboleva, Joel Hestness
[OpenReview] [arXiv]
May 02, 2025
Don't be lazy: CompleteP enables compute-efficient deep transformers
NeurIPS, 2025
Nolan Dey*, Bin Claire Zhang*, Lorenzo Noci, Mufan Li, Blake Bordelon, Shane Bergsma, Cengiz Pehlevan, Boris Hanin, Joel Hestness
[OpenReview] [arXiv]
March 25, 2025
Sherkala-Chat: Building a State-of-the-Art LLM for Kazakh in a Moderately Resourced Setting
Fajri Koto; Rituraj Joshi; Nurdaulet Mukhituly; Yuxia Wang; Zhuohan Xie; Rahul Pal; Daniil Orel; Parvez Mullah; Diana Turmakhan; Maiya Goloburda; Mohammed Kamran; Samujjwal Ghosh; Bokang Jia; Jonibek Mansurov; Mukhammed Togmanov; Debopriyo Banerjee; Nurkhan Laiyk; Akhmed Sakip; Xudong Han; Ekaterina Kochmar; Alham Fikri Aji; Aaryamonvikram Singh; Alok Anil Jadhav; Satheesh Katipomu; Samta Kamboj; Monojit Choudhury; Gurpreet Gosal; Gokul Ramakrishnan; Biswajit Mishra; Sarath Chandran; Avraham Sheinin; Natalia Vassilieva; Neha Sengupta; Larry Murray; Preslav Nakov
[arXiv]
February 21, 2025
Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs
ICLR, 2025
Shane Bergsma, Nolan Dey, Gurpreet Gosal, Gavia Gray, Daria Soboleva, Joel Hestness
November 18, 2024
Empirical Upper Bounds for Unstructured Sparsity in Compute-Efficient Language Modeling
Machine Learning and Compression NeurIPS Workshop, 2024
Esha Singh, Shane Bergsma, Nolan Dey, Joel Hestness, Gavia Gray
November 01, 2024
Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers
October 31, 2024
Sparse maximal update parameterization: A holistic approach to sparse training dynamics
October 13, 2024
Self-Data Distillation for Recovering Quality in Pruned Large Language Models
September 04, 2024
Bilingual Adaptation of Monolingual Foundation Models
FM-Wild ICML Workshop, 2024
Gurpreet Gosal, Yishi Xu, Gokul Ramakrishnan, Rituraj Joshi, Avraham Sheinin, Zhiming (Charles)Chen, Biswajit Mishra, Natalia Vassilieva, Joel Hestness, Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Onkar Pandit, Satheesh Katipomu, Samta Kamboj, Samujjwal Ghosh, Rahul Pal, Parvez Mullah, Soundar Doraiswamy, Mohamed El Karim Chami, Preslav Nakov
[arXiv] [OpenReview]
June 01, 2024
The practitioner's guide to the maximal update parameterization
Blog & open-source code, 2024
Nolan Dey, Quentin Anthony, Joel Hestness
May 20, 2024
MediSwift: Efficient Sparse Pre-trained Biomedical Language Models
Vithursan Thangarasa, Mahmoud Salem, Shreyas Saxena, Kevin Leong, Joel Hestness, Sean Lie
[arXiv]
May 15, 2024
Breaking the Molecular Dynamics Timescale Barrier Using a Wafer-Scale System
Kylee Santos, Stan Moore, Tomas Oppelstrup, Amirali Sharifian, Ilya Sharapov, Aidan Thompson, Delyan Z Kalchev, Danny Perez, Robert Schreiber, Scott Pakin, Edgar A Leon, James H Laros III, Michael James, Sivasankaran Rajamanickam
[arXiv]
May 15, 2024
Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deployment
Abhinav Agarwalla, Abhay Gupta, Alexandre Marques, Shubhra Pandit, Michael Goin, Eldar Kurtic, Kevin Leong, Tuan Nguyen, Mahmoud Salem, Dan Alistarh, Sean Lie, Mark Kurtz
[arXiv]
November 30, 2023
Efficient and Approximate Per-Example Gradient Norms for Gradient Noise Scale
Gavia Gray, Anshul Samar, Joel Hestness
November 13, 2023
Efficient Algorithms for Monte Carlo Particle Transport on AI Accelerator Hardware
John Tramm, Bryce Allen, Kazutomo Yoshii, Andrew Siegel, Leighton Wilson
[arXiv]
November 08, 2023
Position Interpolation Improves ALiBi Extrapolation
Faisal Al-Khateeb, Nolan Dey, Daria Soboleva, Joel Hestness
[arXiv]
September 26, 2023
Scaling the “Memory Wall” for Multi-Dimensional Seismic Processing with Algebraic Compression on Cerebras CS-2 Systems
Hatem Ltaief, Yuxi Hong, Leighton Wilson, Mathias Jacquelin, Matteo Ravasi, David Keyes
September 22, 2023
BTLM-3B-8K: 7B Parameter Performance in a 3B Parameter Model
Efficient Natural Language and Speech Processing NeurIPS Workshop, 2023
Nolan Dey*, Daria Soboleva*, Faisal Al-Khateeb, Ribhu Pathria, Hemant Khachane, Shaheer Muhammad, Zhiming (Charles) Chen, Bowen Yang, Siyun Li, Abhay Gupta, Shreyas Saxena, Robert Myers, Jacob Robert Steeves, Marvin Tom, Joel Hestness
[arXiv] [Workshop Paper] [Blog] [Hugging Face] [1.08M downloads and 10th most popular text generation model in first month]