Invited Speakers

Abstract

Recent advances in speech and audio-language models have enabled increasingly natural voice interactions and sophisticated auditory reasoning. However, extending language models from text to speech introduces new safety and trustworthiness challenges that cannot be fully captured by text-based evaluations. Drawing on recent research, this talk highlights several overlooked vulnerabilities of audio-language models.

The research shows that different emotional deliveries of the same harmful instruction can lead to substantially different safety responses, while speaker characteristics such as gender and accent can introduce systematic biases. The talk also examines whether audio-language models truly ground their responses in acoustic evidence, revealing vulnerabilities such as sensitivity to irrelevant sounds and failures to recognize unanswerable questions.

The talk concludes with approaches to improving audio grounding and mitigating hallucinations. Together, these findings highlight the need for speech-aware approaches to evaluating and improving the safety and trustworthiness of audio-language models.

Bio

Hung-yi Lee is a professor in the Department of Electrical Engineering at National Taiwan University (NTU), with a joint appointment in the Department of Computer Science and Information Engineering. His recent research focuses on self-supervised learning and foundation models for speech. He received the Salesforce Research Deep Learning Grant in 2019, the AWS ML Research Award in 2020, the Outstanding Young Engineer Award from the Chinese Institute of Electrical Engineering in 2018, the Young Scholar Innovation Award from the Foundation for the Advancement of Outstanding Scholarship in 2019, the Ta-You Wu Memorial Award from Taiwan’s Ministry of Science and Technology in 2019, and the 59th Ten Outstanding Young Persons Award of Taiwan in the category of Science and Technology Research and Development. He is a Fellow of the International Speech Communication Association (ISCA). He also runs a YouTube channel that teaches deep learning technologies in Mandarin and has more than 400,000 subscribers.

Abstract

Building speech foundation models is no longer the exclusive domain of large companies, but reproducing one still takes more than a published architecture. The hard part is the groundwork: collecting, cleaning, and automatically annotating data before training begins. Large-scale audio collections are abundant, yet raw audio and transcripts are often misaligned, mislabeled, or otherwise unusable without careful processing.

This talk uses fully transparent open Whisper-style speech models (OWSM) as a case study and focuses on the groundwork behind them. It shows how OWSM v4 narrowed a web-scale pool of multilingual speech, around 500K hours from YODAS v2, to a usable training set through alignment, language identification, and CTC-based scoring. It also examines how the same kind of pipeline carries over to YODAS v3, a larger, more multilingual release recorded at higher fidelity.

The talk argues that task-specific models are not a stepping stone to be replaced by foundation models; they are the tools that make the data usable. It presents the Phonetic Open Whisper-Style Speech Foundation Model (POWSM), which applies the same recipe to phone recognition, showing that task-specific linguistic and phonetic tools are key to this groundwork.

The talk concludes with a brief discussion of unified speech models spanning generation and understanding as a direction that follows the same open-science principles.

Bio

Shinji Watanabe is an Associate Professor at Carnegie Mellon University, Pittsburgh, PA. He received his B.S., M.S., and Ph.D. (Dr. Eng.) degrees from Waseda University, Japan. He was a research scientist at NTT Communication Science Laboratories, Kyoto, Japan, from 2001 to 2011, a visiting scholar at Georgia Institute of Technology, Atlanta, GA, in 2009, and a senior principal research scientist at Mitsubishi Electric Research Laboratories (MERL), Cambridge, MA, USA, from 2012 to 2017. Before Carnegie Mellon University, he was an associate research professor at Johns Hopkins University, Baltimore, MD, USA, from 2017 to 2020. His research interests include automatic speech recognition, speech enhancement, spoken language understanding, and machine learning for speech and language processing. He has published over 600 papers in peer-reviewed journals and conferences and received several awards, including the best paper award from Interspeech 2024 and EMNLP 2024. He has been a member of several technical committees, including the IEEE Signal Processing Society Speech and Language Technical Committee (SLTC, Chair). He is an IEEE and ISCA Fellow.

Abstract

Speech generative AI can now produce voices that are difficult for listeners to distinguish from real ones. These advances enable valuable applications, but also lower the barrier to impersonation fraud and disinformation. This talk examines how passive detection and proactive protection can work together to address these risks.

Passive defense uses deepfake detectors to identify traces of synthesis in speech signals. It requires no cooperation from the generator, but must continually adapt to new and unseen generation methods. The talk reviews recent progress in detection and its remaining limitations.

Proactive defense intervenes when speech is generated or published. Alongside C2PA provenance metadata, the talk introduces self-embedding watermarking, which conceals a compact representation of the original speech within the signal so that manipulated segments can be localized and partially restored. Such marks can provide strong evidence, but are available only for content marked in advance and may be stripped, forged, or degraded.

The talk concludes by exploring how the two approaches can cover each other’s blind spots: detection can serve as a fallback when marks are removed or absent, while watermarks can ease the burden on detectors. It also considers open questions in designing and evaluating multi-layer defenses for real-world use.

Bio

Junichi Yamagishi received a Ph.D. from the Tokyo Institute of Technology in 2006 for a thesis that pioneered speaker-adaptive speech synthesis. He is a professor at the National Institute of Informatics, Tokyo, Japan. Previously, he held an EPSRC Career Acceleration Fellowship in the Centre for Speech Technology Research (CSTR) at the University of Edinburgh, U.K., from 2011 to 2016. He has authored and co-authored more than 400 refereed papers in international journals and conferences.

He served as an elected member of the IEEE Speech and Language Technical Committee from 2013 to 2019, as an Associate Editor for the IEEE/ACM Transactions on Audio, Speech, and Language Processing from 2014 to 2017, as a chairperson of ISCA SynSIG from 2017 to 2021, as a Senior Associate Editor for the IEEE/ACM Transaction on Audio, Speech and Language Processing from 2019 to 2023, and as a technical committee of the Asia-Pacific Signal and Information Processing Association (APSIPA) Multimedia Security and Forensics from 2018 to 2023. He has been a member of the IEEE Signal Processing Society Education Board since 2021.

He received the Itakura Prize from the Acoustic Society of Japan, the Kiyasu Special Industrial Achievement Award from the Information Processing Society of Japan, the Young Scientists’ Prize from the Minister of Education, Science and Technology, the JSPS prize from the Japan Society for the Promotion of Science, the best paper award at the IEEE International Workshop on Information Forensics and Security (WIFS), the Docmo mobile science award, and IEEE Biometrics Council’s BTAS/IJCB 5-Year Highest Impact Award in 2010, 2013, 2014, 2016, 2017, 2018, and 2023, respectively.