Zhaoye Fei

Fudan University · FudanNLP

Hey! I'm Zhaoye.

Foundation models. Embodied intelligence.
Human–AI interaction.

Zhaoye Fei's Google Scholar avatar

About me

I am a PhD student at Fudan University, advised by Prof. Xipeng Qiu, and a member of FudanNLP. I received my bachelor’s degree from Hangzhou Dianzi University.

My research centers on foundation models and intelligent agents, with three main interests:

  • LLM Pre-training & Multimodal Integration: Learning general-purpose representations and integrating knowledge across language, vision, and audio.
  • Embodied Intelligence: World modeling, grounded reasoning, and learning to plan and act in physical environments.
  • Human–AI Interaction: Speech and audio understanding and generation for natural, real-time interaction.

My work has appeared at NeurIPS, ICML, ICLR, ACL, and CVPR. I also contribute to the open-source InternLM and MOSS model families.

I am a member of vidar-team and the dedicated photographer of @blueblueO908.

Google Scholar · GitHub

News

  1. I will curate the Real-Time AI Infra session at iRTE 2026 in Beijing on October 24, 14:00–17:00! We will explore low-latency, high-concurrency and streaming inference for multimodal AI.
  2. We announce the release of EasyWAM, a unified and efficient framework for training, fine-tuning and evaluating World Action Models!
  3. The team I mentored won 1st place in Track 1 and 2nd place in Track 2 of the NVVSpeech Challenge at ISCSLP 2026, on understanding and generating speech with non-verbal vocalizations!
  4. The team I mentored won 1st place in both tracks of the SmartGlasses Challenge 2026, covering dyadic dialogue and multi-party meeting understanding!
  5. Teams I mentored won 1st place in both tasks of the 2nd MLC-SLM Challenge: multilingual speech diarization and recognition, and conversational speech understanding!
  6. We released MOSS-Transcribe-Diarize 0.9B, a lightweight model for transcription, speaker diarization and timestamps, with 691k total downloads on Hugging Face as of October 2026!
  7. We introduce HiMe, a hierarchical memory framework for long-horizon robot control!
  8. We introduce OmniAct, an omnimodal embodied agent that coordinates robot skills and digital tools for sustained physical autonomy!
  9. We released MOSS-TTS-Local-Transformer v1.5, with 48 kHz stereo output, multilingual synthesis and more stable voice cloning! It has reached 393k total downloads on Hugging Face as of October 2026.
  10. One paper accepted to ICML 2026: Scaling Transformers for End-to-End Discrete Audio Tokenization!
  11. We released MOSS-Audio, our open-source model family for speech, sound and music understanding, with time-aware captioning and audio-grounded reasoning! Its four 4B/8B Instruct and Thinking checkpoints have reached approximately 576k total downloads on Hugging Face as of October 2026. Technical report (June 2026).
  12. Two papers accepted to ACL 2026: XY-Tokenizer (Main) and WESR (Findings)!
  13. We introduce our MOSS-TTS family: MOSS-TTS for multilingual, controllable speech synthesis and voice cloning; MOSS-TTSD for expressive, long-form multilingual dialogue with consistent speaker voices; and MOSS-VoiceGenerator for voice design from text descriptions! Total Hugging Face downloads as of October 2026: MOSS-TTS (base) 1.56M, TTSD v0.5/v1.0 combined approximately 177k, and VoiceGenerator 154k.
  14. One paper accepted to CVPR 2026: LIBERO-Plus!
  15. We introduce MOSS-Audio-Tokenizer, achieving state-of-the-art audio reconstruction across speech, sound and music with a fully end-to-end causal Transformer architecture! The base checkpoint has reached 1.1M total downloads on Hugging Face as of October 2026.
  16. We introduce MOVA, an open-source model for jointly generating synchronized video and audio from image and text inputs! The MOVA-360p checkpoint has reached 734k total downloads on Hugging Face as of October 2026.
  17. Two papers accepted to ICLR 2026: MOSS-Speech and RoboOmni!
  18. We introduce MOSS Transcribe Diarize, a unified end-to-end model for transcription, speaker diarization and timestamps!
  19. Our paper on World-aware Planning was accepted to NeurIPS 2025! It strengthens vision-language planning through environment-grounded instruction enhancement and curriculum learning.
  20. We released MOSS-TTSD v0.5, an open-source model for expressive two-speaker dialogue in Chinese and English, with voice cloning and long-form generation!
  21. VLABench was accepted to ICCV 2025! The benchmark evaluates language-conditioned robot manipulation through implicit instructions, world knowledge and long-horizon reasoning.
  22. Three papers were accepted to ACL 2025: D²PO for joint world modeling and embodied planning; weak-to-strong generalization through better supervision and question selection; and VisuoThink for visual-textual reasoning with tree search!
  23. We launched MOSI Intelligence, a startup focused on contextual intelligence and AI systems that understand and interact with their surroundings.
  24. We released SpeechGPT 2.0-preview, an end-to-end Chinese spoken dialogue model for low-latency, real-time interaction, with natural interruptions and controllable voice styles!
  25. Turn Waste into Worth was accepted to EMNLP 2024! Our Rectify-Router recovers dropped tokens and reduces padding waste in mixture-of-experts models.
  26. Balanced Data Sampling was accepted to Findings of ACL 2024! ClusterClip balances common and rare training examples while limiting repetition to reduce overfitting.
  27. We released WanJuan-CC, an English web corpus curated through deduplication, safety filtering and quality screening, with 100B tokens available for language-model pre-training!
  28. We shared the InternLM-Math technical report, unifying mathematical problem solving, solution verification, code-assisted reasoning and Lean theorem proving in one model family!
  29. We released InternLM2, with improved reasoning, coding and long-context capabilities! I was responsible for exploring its post-pretraining (mid-training) stage.
  30. We updated InternLM-7B-Chat and InternLM-20B-Chat, refining their instruction tuning for higher-quality responses and greater stylistic diversity!
  31. We released InternLM-20B, expanding the model family with a larger multilingual model trained on English, Chinese and code, alongside a chat version!
  32. We released InternLM-7B, an open 7B-parameter language model trained on trillions of tokens for general-purpose language understanding and generation!
  33. We released the first InternLM technical report, presenting a 104B-parameter multilingual model built through progressive pre-training and human-preference alignment!
  34. I joined Shanghai AI Laboratory to work on large language model pre-training and contribute to the InternLM project.
  35. Coarse-to-Fine was accepted to COLING 2022! We develop hierarchical multi-task learning that combines shared representations with task-group and task-specific layers for natural language understanding.
  36. I joined Huawei’s 2012 Laboratories as a research intern, working on sparse mixture-of-experts models and efficient expert routing.
  37. I joined FudanNLP under the supervision of Prof. Xipeng Qiu, beginning my research in natural language processing.

Selected publications

Selected peer-reviewed papers. * denotes equal contribution where indicated.

Full list of papers can be found at Google Scholar / Semantic Scholar / DBLP.

Technical reports & preprints

Selected technical reports and preprints.

Method overview for MOSS-TTS Technical Report

MOSS-TTS Technical Report

Technical Report · 2026
Yitian Gong, Botian Jiang, Yiwei Zhao, Yucheng Yuan, Kuangwei Chen, Yaozhou Jiang, Cheng Chang, Dong Hong, Mingshu Chen, Ruixiao Li, Yiyang Zhang, Yang Gao, Hanfu Chen, Ke Chen, Songlin Wang, Xiaogui Yang, Yuqian Zhang, Kexin Huang, ZhengYuan Lin, Kang Yu, Ziqi Chen, Jin Wang, Zhaoye Fei, Qinyuan Cheng, Shimin Li, Xipeng Qiu
Paper Code
Method overview for MOVA: Towards Scalable and Synchronized Video-Audio Generation

MOVA: Towards Scalable and Synchronized Video-Audio Generation

Technical Report · 2026
OpenMOSS Team, Donghua Yu, Mingshu Chen, Qi Chen, Qi Luo, Qianyi Wu, Qinyuan Cheng, Ruixiao Li, Tianyi Liang, Wenbo Zhang, Wenming Tu, Xiangyu Peng, Yang Gao, Yanru Huo, Ying Zhu, Yinze Luo, Yiyang Zhang, Yuerong Song, Zhe Xu, Zhiyu Zhang, Chenchen Yang, Cheng Chang, Chushu Zhou, Hanfu Chen, Hongnan Ma, Jiaxi Li, Jingqi Tong, Junxi Liu, Ke Chen, Shimin Li, Shiqi Jiang, Songlin Wang, Wei Jiang, Zhaoye Fei, Zhiyuan Ning, Chunguo Li, Chenhui Li, Ziwei He, Zengfeng Huang, Xie Chen, Xipeng Qiu
Paper Project Page Code
Method overview for MOSS Transcribe Diarize Technical Report

MOSS Transcribe Diarize Technical Report

Technical Report · 2026
MOSI.AI, Donghua Yu, Zhengyuan Lin, Hanfu Chen, Chen Yang, Yiyang Zhang, Jingqi Chen, Ke Chen, Liwei Fan, Yi Jiang, Jie Zhu, Muchen Li, Wenxuan Wang, Yang Wang, Zhe Xu, Botian Jiang, Yitian Gong, Yuqian Zhang, Wenbo Zhang, Songlin Wang, Zhiyu Wu, Zhaoye Fei, Qinyuan Cheng, Shimin Li, Xipeng Qiu
Paper Code

Honors & awards

Academic service

Program Committee: AAAI (2027)

Reviewer:

  • ARR (2024, 2025, 2026)
  • NeurIPS (2024, 2025, 2026)
  • ICLR (2024, 2025, 2026, 2027)
  • ICML (2025, 2026)
  • AISTATS (2025, 2026, 2027)
  • ECCV (2026)
  • TMLR

Find me online

LLM pre-training & multimodal integration
Embodied intelligence · Human–AI interaction

MOSI AI
HuaFa Road St.699 No.3
shanghai

Paper figure

Open original image