西村 良太 Ryota NISHIMURA

研究Research

人と機械が，人同士のようにリズムよく会話できるようにする研究です．核となるのは対話の時間構造（間・相槌・話者交替・応答タイミング）です．

My research aims to let humans and machines converse with the rhythm of human-to-human conversation. Its core is the temporal structure of dialogue: pauses, backchannels, turn-taking, and response timing.

学部4年（2004年）の最初の研究テーマが「自然な発話タイミングを生成する決定木」で，以来20年以上，同じ問いを追い続けています．「あらゆる機械にAIが載る時代，音声対話が最も自然な情報入出力手段になる」と考え，実際に役立つものを作り，公開しながら研究を進めています．

My first research topic as an undergraduate in 2004 was a decision tree that generates natural utterance timing, and I have pursued the same question for more than 20 years. I believe that as AI is embedded in every machine, spoken dialogue will become the most natural way to exchange information, and I work by building useful systems and releasing them publicly.

下の動画は2009年に公開したデモです．タイミング，リズム，韻律などを制御し，自然な対話ができる音声対話システムでした．当時は1台のコンピュータで動かすため，音声認識は語彙と文法を制限し，出力音声は合成ではなく録音を使うなどの工夫が必要でした．現在は大語彙連続音声認識や高品質な音声合成がリアルタイムで動き，大規模言語モデルも登場し，より自然な音声対話への期待が高まっています．

The video below is a demo released in 2009. The system controlled timing, rhythm, and prosody to hold natural conversations. To run on a single computer at the time, speech recognition was restricted in vocabulary and grammar, and recorded rather than synthesized speech was used. Today, large-vocabulary recognition and high-quality synthesis run in real time, and large language models have appeared, raising expectations for more natural spoken dialogue.

研究テーマResearch themes

1. 応答タイミング制御（間・相槌・話者交替）Response timing control (pauses, backchannels, turn-taking)

相手の声の高さや強さの変化，ポーズの長さなどから，100ミリ秒ごとに「今あいづちを打つ」「今話し始める」「まだ待つ」を判断する技術です．音声認識が終わってから応答するのでは遅いため，話している最中に判断します．決定木を用いた応答判定から始め，2026年には対話コーパス上で決定木・MLP・LSTM・HuBERT+LSTMの4モデルを再構築して比較しました．相槌の予測（HuBERT中間層特徴，Interspeech 2025）や，発話終端の事前推定にも取り組んでいます．

These methods decide every 100 ms whether to give a backchannel, start speaking, or keep waiting, using changes in pitch and power and the length of pauses. Waiting for speech recognition to finish is too late, so the decision is made while the other person is still speaking. Starting from decision-tree response judgment, in 2026 I rebuilt and compared four models (decision tree, MLP, LSTM, and HuBERT+LSTM) on a dialogue corpus. I also work on backchannel prediction (HuBERT intermediate-layer features, Interspeech 2025) and on predicting the end of an utterance in advance.

2. マルチモーダル対話・CGエージェントMultimodal dialogue and CG agents

音声に加えて表情・うなずき・視線などを扱う対話システムの研究です．音声対話ツールキットMMDAgentの開発に携わり，フォトリアルなCGエージェント「Saya」との音声対話では2022年にACC TOKYO CREATIVITY AWARDSゴールドを受賞しました．相手の声に合わせて「聞いている顔」がうなずく頭部動作の生成にも取り組んでいます（APSIPA ASC 2025）．

This line of work handles facial expressions, nodding, gaze, and more in addition to speech. I was involved in developing the MMDAgent spoken-dialogue toolkit, and spoken dialogue with the photorealistic CG agent "Saya" received the ACC TOKYO CREATIVITY AWARDS Gold in 2022. I also generate head movements in which a listening face nods in response to the other person's voice (APSIPA ASC 2025).

3. リアルタイム対話基盤 DiaROS・MMSDSReal-time dialogue platform DiaROS and MMSDS

音声認識・大規模言語モデル・音声合成を直列ではなく並列に動かし，応答の遅延を削るROS上の対話基盤DiaROSを開発しています（科研費基盤研究(C) 19K04311の成果）．ブラウザから話しかけると3D顔が聞いている間はうなずき，話す間は口・表情・首が動くマルチモーダル音声対話システム（MMSDS）として発展させています．遅延を重視した設計で，全メッセージにタイムスタンプを付け，処理段階ごとの遅延を可視化できます．

DiaROS is a ROS-based dialogue platform that runs speech recognition, a large language model, and speech synthesis in parallel rather than in series to cut response latency (a result of KAKENHI 19K04311). It has been extended into a multimodal spoken-dialogue system (MMSDS) in which a 3D face nods while listening and moves its mouth, expression, and neck while speaking when addressed from a browser. The design prioritizes latency: every message is time-stamped so that delay at each processing stage can be visualized.

4. 高齢者・構音障害者の音声認識Speech recognition for elderly speakers and speakers with dysarthria

平均83.4歳の超高齢者音声コーパスEARSを構築し，高齢者の声に対する音声認識の精度向上に取り組んでいます（Computer Speech & Language 2022，Oriental-COCOSDA 2019 Best Paper Award）．少量の高齢者音声を成人データに繰り返し混ぜて学習する方法などを検討しました．あわせて，構音障害のある方の音声認識にも取り組んでいます．

I built EARS, a corpus of speech from very elderly speakers (mean age 83.4), and work on improving speech recognition for older voices (Computer Speech & Language 2022; Oriental-COCOSDA 2019 Best Paper Award). Methods such as repeatedly mixing a small amount of elderly speech into adult training data were examined. I also work on recognition of speech from people with dysarthria.

5. 医療向け音声認識Speech recognition for healthcare

看護師が患者との会話をスマートフォンで録音すると，院内の音声認識とローカルの大規模言語モデルによりSOAP形式の看護記録を自動生成するシステムを，患者情報を院外に出さない完全院内完結の形で構築しています．豊橋ハートセンターで実証運用を行っており，JST A-STEP（電子カルテ音声入力，研究代表者：北岡教英）として事業化に向けた取り組みも進めています．

I am building a system in which a nurse records a conversation with a patient on a smartphone, and in-hospital speech recognition and a local large language model automatically generate nursing records in SOAP format, entirely inside the hospital so that patient information never leaves it. It is in trial operation at Toyohashi Heart Center, and commercialization is being pursued under JST A-STEP (electronic medical record voice input; PI: Norihide Kitaoka).

6. 認知症検出・医工連携Dementia detection and medical-engineering collaboration

会話音声（および方言）の音響特徴から認知症の傾向を検出する研究に取り組んでいます（LREC 2022，NLP2026）．泌尿器科CT画像に基づく深達度予測AIや歯科画像診断AIなど，医療画像のAIでも共同研究を行い，日本泌尿器科学会総会で2年連続の受賞をしました．

I study detecting signs of dementia from acoustic features of conversational speech, including dialects (LREC 2022, NLP2026). I also collaborate on medical-image AI, such as depth prediction from urological CT images and dental image diagnosis, and received awards at the Japanese Urological Association annual meeting two years in a row.

7. VADレス音声認識・低資源言語・その他VAD-less recognition, low-resource languages, and others

音声区間検出（VAD）を用いずに，雑音や無音を棄却しながらリアルタイムで認識する技術（APSIPA ASC 2025）や，言いよどみをラベル化する自発音声認識（IEICE Trans. 2026）に取り組んでいます．モンゴル語など低資源言語の音声合成・転写正規化，Web会議音声の劣化が音声認識に及ぼす影響の分析，カウンセリング対話コーパスの公開なども行っています．

I work on real-time recognition that rejects noise and silence without voice activity detection (APSIPA ASC 2025) and on spontaneous speech recognition that labels disfluencies (IEICE Trans. 2026). Other work includes speech synthesis and transcription normalization for low-resource languages such as Mongolian, analysis of how degraded web-conference audio affects recognition, and release of a counseling dialogue corpus.

経歴の流れCareer at a glance

研究室メンバーと共同で進めている研究も含みます．詳細は業績ページを参照してください．

This includes work carried out jointly with laboratory members. See the publications page for details.

業績一覧を見るSee publications