Shiwei Liu
Office S2.23
Mathematics Institute
University of Oxford
Hi, I am a Royal Society Newton International Fellow at University of Oxford, and a Junior Research Fellow (JRF) at Somerville College. Previously, I was a postdoctoral fellow in the VITA group funded by IFML at UT Austin, working with Atlas Wang. I obtained my Ph.D. at the Eindhoven University of Technology (TU/e), the Netherlands, under the supervision of Mykola Pechenizkiy and Decebal Constantin Mocanu.
Research Interests
My general research interests are to understand and leverage the role of low dimensionality in machine learning, whose impacts span many important topics, including but not limited to efficient training/inference of large foundation models, algorithm and system co-design, and ML interpretability.
Potential Collaborations: If you are interested in these topics, feel free to reach out for more information!
news
Sep 30, 2024 | I serve as an Area Chair for ICASSP 2025. |
---|---|
Sep 21, 2024 | 📝 3 papers are accepted by NeurIPS 2024: Found in the Middle, Sparse 3D Medical, Alpha Pruning. |
Sep 21, 2024 | 📝 2 papers are accepted by EMNLP 2024: Layer-skip LLM and Is c4 enough for LLM Pruning? |
Jul 22, 2024 | I am honored to join in the organization committee of the Conference on Parsimony and Learning (CPAL). See you in Stanford. |
Jun 25, 2024 | I am very happy to be offered as a Junior Research Fellow (JRF) at Somerville College, one of the first two women’s colleges at Oxford. |
Jun 21, 2024 | I will have a talk tour around Europe at the NLP group at University of Sheffield, LTL group at University of Cambridge, and BlueNN group at University of Luxembourg. |
Jun 15, 2024 | 📝 2 papers got accepted by Interspeech 2024: Sparse Multimodal from Scratch, Dynamic Data Pruning for Speech. |
May 21, 2024 | Our “Edge LLMs: Edge-Device Large Language Model Competition” competition has been accepted by NeurIPS 2024. Submission opens Link. |
May 15, 2024 | 📝 1 paper Q-Hitter: Quantized-Sparse KV Cache got accepted by MLSys 2024. |
May 02, 2024 | 📝 5 papers got accepted by ICML 2024: Layerwise Importance for LLMs, LLM Junk DNA, Bi-Level DST, KV Cache Merging, Saprse Cocktail. |
Jan 16, 2024 | 📝 3 papers got accepted by ICLR 2024: Training-Free Sparse LLM Fine-tuning, Multi-Task Vector Merging, Sparse Training with Neuron Revitalization. |
Jan 16, 2024 | 📝 4 papers got accepted by NeurIPS 2023: Channel-Level DST, Essential Sparsity, Pruning Topology, Note-Path Balance. |
Jan 05, 2024 | 🏆 I am highly honored to receive the Rising Star in AI from KAUST and will give a talk at Rising Stars in AI Symposium. |
Nov 01, 2023 | 📝 Block Sparse Training accepted by CPAL. |
Oct 16, 2023 | 🏆 I am highly grateful to receive the Best PhD Dissertation Runner-up Award from the Informatics Europe. |
Oct 10, 2023 | 🏆 I am highly honered to receive the Rising Star Award from CPAL and will give a presentation at HKU in Jan 2024. |
Sep 20, 2023 | 🚀 I am grateful to receive the prestigious Newton International Fellowship from the British Academy and the Royal Society. |
May 20, 2023 | The work I conducted during my internship at JD Academy has been accepted by International Journal of Computer Vision (IJCV) - STU-GAN. |
Apr 20, 2023 | 📝 3 papers got accepted in ICML 2023, Instant Soup (Oral), Large Kernel Distillation, and Graph Ladling. |
Mar 20, 2023 | 📝 2 papers SNN Ten Lessons and Channel-Level DST paper has been accepted as spotlight presentations at the SNN workshop. |
Jan 20, 2023 | 📝 4 papers got accepted in ICLR 2023, Ramanujan Graph Pruning (oral), Sparsity May Cry Benchmark (spotlight%), MoE as Dropout (spotlight), SLaK:51x51 Large Conv. |
Dec 01, 2022 | 📝 Our Untrained GNNs paper received the Best Paper Award from LoG 2022.11/2022*, |
Nov 01, 2022 | 📝 One paper Lottery-Pools got accepted in AAAI 2023.9/2022*, |
Apr 06, 2022 | I got my PhD thesis abstract accepted by IDA 2022, which was also the first conference (symposium) that I attended in the first year of my PhD. PhD life is a cycle :). |
Apr 01, 2022 | Our tutorial Sparse Neural Networks Training has been accepted at ECMLPKDD 2022. |
Jan 06, 2022 | 📝 Two of my first-author papers are accepted by ICLR 2022: Random pruning and FreeTickets. |
Sep 06, 2021 | 🏆 I receive the “Outstanding Intern” honor in JD Academy Explore |
Sep 05, 2021 | 📝 One paper got accepted by NeurIPs 2021: GraNet. |
May 05, 2021 | 📝 2 papers are accepted by ICML 2021: In-Time Over-Parameterization and Selfish RNN. |
selected publications
- PreprintOwLore: Outlier-weighed Layerwise Sampled Low-Rank Projection for Memory-Efficient LLM Fine-tuningarXiv preprint arXiv:2405.18380, 2024
- ICML2024Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity2024
- ICLR2023More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsityarXiv preprint arXiv:2207.03620, 2023
- LoG2022You can have better graph neural networks by not training weights at all: Finding untrained gnns tickets2022
- ICLR2022The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse trainingarXiv preprint arXiv:2202.02643, 2022
- NeurIPS2021Sparse training via boosting pruning plasticity with neuroregenerationAdvances in Neural Information Processing Systems, 2021
- ICML2021Do we actually need dense over-parameterization? in-time over-parameterization in sparse trainingIn International Conference on Machine Learning, 2021