MoE-RBench: Towards Building Reliable Language Models with Sparse Mixture-of-Experts
Published in International Conference on Machine Learning (ICML), 2024
We study the reliability of sparse mixture-of-experts language models and provide a comprehensive benchmark for comparing MoE and dense architectures.
Recommended citation: Guanjie Chen, Xinyu Zhao, Tianlong Chen, and Yu Cheng. MoE-RBench: Towards Building Reliable Language Models with Sparse Mixture-of-Experts. ICML, 2024.
Download Paper