Publications

publications by categories in reversed chronological order.

2026

  1. Diffusion language models know the answer before decoding
    Li, Pengxiang and Zhou, Yefan and Muhtar, Dilxat and Yin, Lu and Yan, Shilin and Shen, Li and Vosoughi, Soroush and Liu, Shiwei
    ICLR 2026 Oral Paper Code Models
  2. Neural Sum-of-Squares: Certifying the Nonnegativity of Polynomials with Transformers
    Pelleriti, Nico and Spiegel, Christoph and Liu, Shiwei and Martı́nez-Rubio, David and Zimmer, Max and Pokutta, Sebastian
    ICLR 2026 Paper Code Models
  3. GPTailor: Large Language Model Pruning Through Layer Cutting and Stitching
    Su, Guinan and Shen, Li and Yin, Lu and Liu, Shiwei and Yang, Yanwu and Geiping, Jonas
    ICLR 2026 Paper Code Models

2025

  1. The Curse of Depth in Large Language Models
    Sun, Wenfang and Song, Xinyuan and Li, Pengxiang and Yin, Lu and Zheng, Yefeng and Liu, Shiwei
    NeurIPS 2025 Paper Code Models
  2. AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMs
    He, Di and Jaiswal, Ajay and Tu, Songjun and Shen, Li and Yuan, Ganzhao and Liu, Shiwei and Yin, Lu
    NeurIPS 2025 Paper Code Models
  3. GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling
    Chen, Tianhao and Xu, Xin and Liu, Zijing and Li, Pengxiang and Song, Xinyuan and Jaiswal, Ajay Kumar and Zhang, Fan and Hu, Jishan and Wang, Yang and Chen, Hao and others
    NeurIPS 2025 Paper Code Models
  4. LIFT the Veil for the Truth: Principal Weights Emerge after Rank Reduction for Reasoning-Focused Supervised Fine-Tuning
    Liu, Zihang and Pang, Tianyu and Balabanov, Oleg and Yang, Chaoqun and Huang, Tianjin and Yin, Lu and Yang, Yaoqing and Liu, Shiwei
    ICML 2025 Paper Code Models
  5. Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn More
    Zhuang, Xialie and Jia, Zhikai and Li, Jianjin and Zhang, Zhenyu and Shen, Li and Cao, Zheng and Liu, Shiwei
    ICML 2025 Paper Code Models
  6. SoS1: O1 and R1-Like Reasoning LLMs are Sum-of-Square Solvers
    Li, Kechen and Zhu, Wenqi and Cartis, Coralia and Ji, Tianbo and Liu, Shiwei
    Preprint Paper
  7. Stable-SPAM: How to Train in 4-Bit More Stably than 16-Bit Adam
    Huang, Tianjin and Hu, Haotian and Zhang, Zhenyu and Jin, Gaojie and Li, Xiang and Shen, Li and Chen, Tianlong and Liu, Lu and Wen, Qingsong and Wang, Zhangyang and Liu, Shiwei
    Preprint Paper
  8. Mix-ln: Unleashing the power of deeper layers by combining pre-ln and post-ln
    Li, Pengxiang and Yin, Lu and Liu, Shiwei
    ICLR 2025 Paper Code Models
  9. SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training
    Huang, Tianjin and Zhu, Ziquan and Jin, Gaojie and Liu, Lu and Wang, Zhangyang and Liu, Shiwei
    ICLR 2025 Paper Code Models
  10. Outlier-weighed layerwise sampling for LLM fine-tuning
    Li, Pengxiang and Yin, Lu and Gao, Xiaowei and Liu, Shiwei
    Findings of the Association for Computational Linguistics: ACL 2025 Paper Code Models

2024

  1. OwLore: Outlier-weighed Layerwise Sampled Low-Rank Projection for Memory-Efficient LLM Fine-tuning
    Li, Pengxiang and Yin, Lu and Gao, Xiaowei and Liu, Shiwei
    ICLR 2025 Paper Code Models
  2. Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity
    Yin, Lu and Wu, You and Zhang, Zhenyu and Hsieh, Cheng-Yu and Wang, Yaqing and Jia, Yiling and Pechenizkiy, Mykola and Liang, Yi and Wang, Zhangyang and Liu, Shiwei
    ICML 2024 Paper Code Models
  3. CaM: Cache Merging for Memory-efficient LLMs Inference
    Zhang, Yuxin and Du, Yuxuan and Luo, Gen and Zhong, Yunshan and Zhang, Zhenyu and Liu, Shiwei and Ji, Rongrong
    ICML 2024
  4. Junk DNA Hypothesis: Pruning Small Pre-Trained Weights $\backslashtextit {Irreversibly} and \backslashtextit {Monotonically} $ Impairs“Difficult" Downstream Tasks in LLMs
    Yin, Lu and JAISWAL, AJAY KUMAR and Liu, Shiwei and Kundu, Souvik and Wang, Zhangyang
    ICML 2024
  5. Dynamic Data Pruning for Automatic Speech Recognition
    Xiao, Qiao and Ma, Pingchuan and Fernandez-Lopez, Adriana and Wu, Boqian and Yin, Lu and Petridis, Stavros and Pechenizkiy, Mykola and Pantic, Maja and Mocanu, Decebal Constantin and Liu, Shiwei
    Interspeech 2024 Paper
  6. MSRS: Training Multimodal Speech Recognition Models from Scratch with Sparse Mask Optimization
    Fernandez-Lopez, Adriana and Chen, Honglie and Ma, Pingchuan and Yin, Lu and Xiao, Qiao and Petridis, Stavros and Liu, Shiwei and Pantic, Maja
    Interspeech 2024 Paper
  7. Dynamic sparse no training: Training-free fine-tuning for sparse llms
    Zhang, Yuxin and Zhao, Lirui and Lin, Mingbao and Sun, Yunyun and Yao, Yiwu and Han, Xingjia and Tanner, Jared and Liu, Shiwei and Ji, Rongrong
    ICLR 2024 Paper Code
  8. Adamerging: Adaptive model merging for multi-task learning
    Yang, Enneng and Wang, Zhenyi and Shen, Li and Liu, Shiwei and Guo, Guibing and Wang, Xingwei and Tao, Dacheng
    ICLR 2024 Paper Code

2023

  1. Don’t be so dense: Sparse-to-sparse gan training without sacrificing performance
    Liu, Shiwei and Tian, Yuesong and Chen, Tianlong and Shen, Li
    IJCV
  2. More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsity
    Liu, Shiwei and Chen, Tianlong and Chen, Xiaohan and Chen, Xuxi and Xiao, Qiao and Wu, Boqian and Kärkkäinen, Tommi and Pechenizkiy, Mykola and Mocanu, Decebal and Wang, Zhangyang
    ICLR 2023 Paper Code
  3. Revisiting pruning at initialization through the lens of ramanujan graph
    Hoang, Duc NM and Liu, Shiwei and Marculescu, Radu and Wang, Zhangyang
    ICLR 2023 Paper Code
  4. Sparse moe as the new dropout: Scaling dense and self-slimmable transformers
    Chen, Tianlong and Zhang, Zhenyu and Jaiswal, Ajay and Liu, Shiwei and Wang, Zhangyang
    ICLR 2023 Paper Code
  5. Sparsity may cry: Let us fail (current) sparse neural networks together!
    Liu, Shiwei and Chen, Tianlong and Zhang, Zhenyu and Chen, Xuxi and Huang, Tianjin and Jaiswal, Ajay and Wang, Zhangyang
    ICLR 2023 Paper Code Models

2022

  1. You can have better graph neural networks by not training weights at all: Finding untrained gnns tickets
    Huang, Tianjin and Chen, Tianlong and Fang, Meng and Menkovski, Vlado and Zhao, Jiaxu and Yin, Lu and Pei, Yulong and Mocanu, Decebal Constantin and Wang, Zhangyang and Pechenizkiy, Mykola and others
    LoG 2022 Code
  2. The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training
    Liu, Shiwei and Chen, Tianlong and Chen, Xiaohan and Shen, Li and Mocanu, Decebal Constantin and Wang, Zhangyang and Pechenizkiy, Mykola
    ICLR 2022 Paper Code Models
  3. Deep ensembling with no overhead for either training or testing: The all-round blessings of dynamic sparsity
    Liu, Shiwei and Chen, Tianlong and Atashgahi, Zahra and Chen, Xiaohan and Sokar, Ghada and Mocanu, Elena and Pechenizkiy, Mykola and Wang, Zhangyang and Mocanu, Decebal Constantin
    ICLR 2022 Paper

2021

  1. Sparse training via boosting pruning plasticity with neuroregeneration
    Liu, Shiwei and Chen, Tianlong and Chen, Xiaohan and Atashgahi, Zahra and Yin, Lu and Kou, Huanyu and Shen, Li and Pechenizkiy, Mykola and Wang, Zhangyang and Mocanu, Decebal Constantin
    NeurIPS 2021 Paper Code Models
  2. Do we actually need dense over-parameterization? in-time over-parameterization in sparse training
    Liu, Shiwei and Yin, Lu and Mocanu, Decebal Constantin and Pechenizkiy, Mykola
    ICML 2021 Paper Code Models
  3. Selfish sparse rnn training
    Liu, Shiwei and Mocanu, Decebal Constantin and Pei, Yulong and Pechenizkiy, Mykola
    ICML 2021 Paper Code Models