Speech Breathing in Virtual Humans: An Interactive Model and Empirical Study

Ulysses Bernardet, Sin-hwa Kanq, Andrew Feng, Steve Dipaola, Ari Shapiro

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Human speech production requires the dynamic regulation of air through the vocal system. While virtual character systems commonly are capable of speech output, they rarely take breathing during speaking - speech breathing - into account. We believe that integrating dynamic speech breathing systems in virtual characters can significantly contribute to augmenting their realism. Here, we present a novel control architecture aimed at generating speech breathing in virtual characters. This architecture is informed by behavioral, linguistic and anatomical knowledge of human speech breathing. Based on textual input and controlled by a set of low-and high-level parameters, the system produces dynamic signals in real-time that control the virtual character's anatomy (thorax, abdomen, head, nostrils, and mouth) and sound production (speech and breathing). In addition, we perform a study to determine the effects of including breathing-motivated speech movements, such as head tilts and chest expansions during dialogue on a virtual character, as well as breathing sounds. This study includes speech that is generated both from a text-to-speech engine as well as from recorded voice.
Original languageEnglish
Title of host publication2019 IEEE Virtual Humans and Crowds for Immersive Environments, VHCIE 2019
PublisherIEEE
ISBN (Electronic)9781728132198
DOIs
Publication statusPublished - 14 May 2019
Event2019 IEEE Virtual Humans and Crowds for Immersive Environments (VHCIE) - Osaka, Japan
Duration: 24 Mar 201924 Mar 2019

Conference

Conference2019 IEEE Virtual Humans and Crowds for Immersive Environments (VHCIE)
Period24/03/1924/03/19

Fingerprint

Acoustic waves
Real time control
Linguistics
Dynamical systems
Engines
Air

Bibliographical note

© 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

Keywords

  • 500 [Human-centered computing]
  • Computing methodologies - Virtual reality
  • Human-centered computing - Empirical studies in HCI 500 [Computing methodologies]
  • Procedural animation - [500]
  • Virtual reality - [500]

Cite this

Bernardet, U., Kanq, S., Feng, A., Dipaola, S., & Shapiro, A. (2019). Speech Breathing in Virtual Humans: An Interactive Model and Empirical Study. In 2019 IEEE Virtual Humans and Crowds for Immersive Environments, VHCIE 2019 [8714737] IEEE. https://doi.org/10.1109/VHCIE.2019.8714737
Bernardet, Ulysses ; Kanq, Sin-hwa ; Feng, Andrew ; Dipaola, Steve ; Shapiro, Ari. / Speech Breathing in Virtual Humans: An Interactive Model and Empirical Study. 2019 IEEE Virtual Humans and Crowds for Immersive Environments, VHCIE 2019. IEEE, 2019.
@inproceedings{fe870317c2b440a8967e1de76a9d143c,
title = "Speech Breathing in Virtual Humans: An Interactive Model and Empirical Study",
abstract = "Human speech production requires the dynamic regulation of air through the vocal system. While virtual character systems commonly are capable of speech output, they rarely take breathing during speaking - speech breathing - into account. We believe that integrating dynamic speech breathing systems in virtual characters can significantly contribute to augmenting their realism. Here, we present a novel control architecture aimed at generating speech breathing in virtual characters. This architecture is informed by behavioral, linguistic and anatomical knowledge of human speech breathing. Based on textual input and controlled by a set of low-and high-level parameters, the system produces dynamic signals in real-time that control the virtual character's anatomy (thorax, abdomen, head, nostrils, and mouth) and sound production (speech and breathing). In addition, we perform a study to determine the effects of including breathing-motivated speech movements, such as head tilts and chest expansions during dialogue on a virtual character, as well as breathing sounds. This study includes speech that is generated both from a text-to-speech engine as well as from recorded voice.",
keywords = "500 [Human-centered computing], Computing methodologies - Virtual reality, Human-centered computing - Empirical studies in HCI 500 [Computing methodologies], Procedural animation - [500], Virtual reality - [500]",
author = "Ulysses Bernardet and Sin-hwa Kanq and Andrew Feng and Steve Dipaola and Ari Shapiro",
note = "{\circledC} 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.",
year = "2019",
month = "5",
day = "14",
doi = "10.1109/VHCIE.2019.8714737",
language = "English",
booktitle = "2019 IEEE Virtual Humans and Crowds for Immersive Environments, VHCIE 2019",
publisher = "IEEE",
address = "United States",

}

Bernardet, U, Kanq, S, Feng, A, Dipaola, S & Shapiro, A 2019, Speech Breathing in Virtual Humans: An Interactive Model and Empirical Study. in 2019 IEEE Virtual Humans and Crowds for Immersive Environments, VHCIE 2019., 8714737, IEEE, 2019 IEEE Virtual Humans and Crowds for Immersive Environments (VHCIE), 24/03/19. https://doi.org/10.1109/VHCIE.2019.8714737

Speech Breathing in Virtual Humans: An Interactive Model and Empirical Study. / Bernardet, Ulysses; Kanq, Sin-hwa; Feng, Andrew; Dipaola, Steve; Shapiro, Ari.

2019 IEEE Virtual Humans and Crowds for Immersive Environments, VHCIE 2019. IEEE, 2019. 8714737.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Speech Breathing in Virtual Humans: An Interactive Model and Empirical Study

AU - Bernardet, Ulysses

AU - Kanq, Sin-hwa

AU - Feng, Andrew

AU - Dipaola, Steve

AU - Shapiro, Ari

N1 - © 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

PY - 2019/5/14

Y1 - 2019/5/14

N2 - Human speech production requires the dynamic regulation of air through the vocal system. While virtual character systems commonly are capable of speech output, they rarely take breathing during speaking - speech breathing - into account. We believe that integrating dynamic speech breathing systems in virtual characters can significantly contribute to augmenting their realism. Here, we present a novel control architecture aimed at generating speech breathing in virtual characters. This architecture is informed by behavioral, linguistic and anatomical knowledge of human speech breathing. Based on textual input and controlled by a set of low-and high-level parameters, the system produces dynamic signals in real-time that control the virtual character's anatomy (thorax, abdomen, head, nostrils, and mouth) and sound production (speech and breathing). In addition, we perform a study to determine the effects of including breathing-motivated speech movements, such as head tilts and chest expansions during dialogue on a virtual character, as well as breathing sounds. This study includes speech that is generated both from a text-to-speech engine as well as from recorded voice.

AB - Human speech production requires the dynamic regulation of air through the vocal system. While virtual character systems commonly are capable of speech output, they rarely take breathing during speaking - speech breathing - into account. We believe that integrating dynamic speech breathing systems in virtual characters can significantly contribute to augmenting their realism. Here, we present a novel control architecture aimed at generating speech breathing in virtual characters. This architecture is informed by behavioral, linguistic and anatomical knowledge of human speech breathing. Based on textual input and controlled by a set of low-and high-level parameters, the system produces dynamic signals in real-time that control the virtual character's anatomy (thorax, abdomen, head, nostrils, and mouth) and sound production (speech and breathing). In addition, we perform a study to determine the effects of including breathing-motivated speech movements, such as head tilts and chest expansions during dialogue on a virtual character, as well as breathing sounds. This study includes speech that is generated both from a text-to-speech engine as well as from recorded voice.

KW - 500 [Human-centered computing]

KW - Computing methodologies - Virtual reality

KW - Human-centered computing - Empirical studies in HCI 500 [Computing methodologies]

KW - Procedural animation - [500]

KW - Virtual reality - [500]

UR - https://ieeexplore.ieee.org/document/8714737/

UR - http://www.scopus.com/inward/record.url?scp=85066763102&partnerID=8YFLogxK

U2 - 10.1109/VHCIE.2019.8714737

DO - 10.1109/VHCIE.2019.8714737

M3 - Conference contribution

BT - 2019 IEEE Virtual Humans and Crowds for Immersive Environments, VHCIE 2019

PB - IEEE

ER -

Bernardet U, Kanq S, Feng A, Dipaola S, Shapiro A. Speech Breathing in Virtual Humans: An Interactive Model and Empirical Study. In 2019 IEEE Virtual Humans and Crowds for Immersive Environments, VHCIE 2019. IEEE. 2019. 8714737 https://doi.org/10.1109/VHCIE.2019.8714737