Synthesizing Expressive Facial and Speech Animation by Text-to-IPA Translation with Emotion Control

Andreea Stef, Kaveen Perera, Hubert P. H. Shum and Edmond S. L. Ho
Proceedings of the 2018 International Conference on Software, Knowledge, Information Management and Applications (SKIMA), 2018

Synthesizing Expressive Facial and Speech Animation by Text-to-IPA Translation with Emotion Control

Abstract

Given the complexity of the human facial anatomy, animating facial expressions and lip movements for speech is a very time-consuming and tedious task. In this paper, a new text-to-animation framework for facial animation synthesis is proposed. The core idea is to improve the expressiveness of lip-sync animation by incorporating facial expressions in 3D animated characters. This idea is realized as a plug-in in Autodesk Maya, one of the most popular animation platforms in the industry, such that professional animators can effectively apply the method in their existing work. We evaluate the proposed system by conducting two sets of surveys, in which both novice and experienced users participate in the user study to provide feedback and evaluations from different perspectives. The results of the survey highlights the effectiveness of creating realistic facial animations with the use of emotion expressions. Video demos of the synthesized animations are available online at https://git.io/fx5U3

Downloads

YouTube

Citations

BibTeX

@inproceedings{stef18synthesizing,
 author={Stef, Andreea and Perera, Kaveen and Shum, Hubert P. H. and Ho, Edmond S. L.},
 booktitle={Proceedings of the 2018 International Conference on Software, Knowledge, Information Management and Applications},
 series={SKIMA '18},
 title={Synthesizing Expressive Facial and Speech Animation by Text-to-IPA Translation with Emotion Control},
 year={2018},
 month={12},
 pages={1--8},
 numpages={8},
 doi={10.1109/SKIMA.2018.8631536},
 issn={2573-3214},
 publisher={IEEE},
 location={Phnom Penh, Cambodia},
}

RIS

TY  - CONF
AU  - Stef, Andreea
AU  - Perera, Kaveen
AU  - Shum, Hubert P. H.
AU  - Ho, Edmond S. L.
T2  - Proceedings of the 2018 International Conference on Software, Knowledge, Information Management and Applications
TI  - Synthesizing Expressive Facial and Speech Animation by Text-to-IPA Translation with Emotion Control
PY  - 2018
Y1  - 12 2018
SP  - 1
EP  - 8
DO  - 10.1109/SKIMA.2018.8631536
SN  - 2573-3214
PB  - IEEE
ER  - 

Plain Text

Andreea Stef, Kaveen Perera, Hubert P. H. Shum and Edmond S. L. Ho, "Synthesizing Expressive Facial and Speech Animation by Text-to-IPA Translation with Emotion Control," in SKIMA '18: Proceedings of the 2018 International Conference on Software, Knowledge, Information Management and Applications, pp. 1-8, Phnom Penh, Cambodia, IEEE, Dec 2018.

Supporting Grants

Similar Research

Jacky C. P. Chan, Hubert P. H. Shum, He Wang, Li Yi, Wei Wei and Edmond S. L. Ho, "A Generic Framework for Editing and Synthesizing Multimodal Data with Relative Emotion Strength", Computer Animation and Virtual Worlds (CAVW), 2019
Shanfeng Hu, Hubert P. H. Shum, Xiaohui Liang, Frederick W. B. Li and Nauman Aslam, "Facial Reshaping Operator for Controllable Face Beautification", Expert Systems with Applications (ESWA), 2021
Daniel Organisciak, Edmond S. L. Ho and Hubert P. H. Shum, "Makeup Style Transfer on Low-Quality Images with Weighted Multi-Scale Attention", Proceedings of the 2020 International Conference on Pattern Recognition (ICPR), 2020

 

 

Last updated on 4 June 2024
RSS Feed