Facial cues, bilingual input and AI assistance, with the operator controlling the final reply.
HRFEST 2026 · Paper 230 IHCI 2026 · Paper 99
Two distinct conference papers on one shared platform: perception and bilingual speech (second author of nine), and human-in-the-loop edge/cloud decision support (fourth author of eight).
MediaPipe facial landmarks with learned CNN/ONNX and interpretable action-unit routes.
My focusFacial perception
Vision-pipeline implementation and Raspberry Pi integration within the wider team system.
Decision authorityHuman operator
Visible facial movement and proposed replies support the operator’s judgement.
Project overview
Bangbang is a collaborative tele-operated avatar-robot platform. Facial-expression and bilingual speech cues support an operator who can inspect the interaction and choose how to respond.
The project produced two distinct conference papers: a perception study (HRFEST Paper 230) and an operator-assistance study (IHCI Paper 99). The submitted FURP 2026 poster brings these strands together.
Bangbang 2.0 · vision workflow graphic prepared for the project. The vision node used 16 GB RAM; the 8 GB label in the original graphic is incorrect.Full image ↗
Prototype and scope
The public release contains facial-landmark and action-unit processing, emotion-model definitions, training/export scripts and Raspberry Pi vision examples. Datasets and trained model files are not bundled.
An earlier stationary speech-and-vision branch uses Vosk with an English example vocabulary. It is retained separately from the later vision work and does not reproduce the complete bilingual speech or teleoperation system described in the team research.
The Paper 99 presentation shows local and cloud AI assistance, bilingual input and a browser interface for reviewing suggested replies. Chinese speech recognition uses a cloud service. Its timing figures are development observations with varying prompts and settings; they do not form a controlled model ranking.
Functional checks and demonstrations show particular prototype functions. They do not establish end-to-end reliability, accurate interpretation of emotion, reduced operator workload or user benefit. Those questions need further evaluation.
15-slide team presentation credited to Yanjie Zhou, saved as the October 2026 research revision. It accompanies the operator-assistance paper on the shared Bangbang platform, distinct from the HRFEST Paper 230 perception study.
Bangbang 2.0: Multimodal Perception and AI Assistance for Inclusive Telepresence. Poster team: Changshuai Liu, Yanjie Zhou, Siheng Ye and Yuechen Cao. Supervisor: David Chieng. Co-supervisor: Younghoon Chang.
The HRFEST links open the organisers’ copies on Google Drive. The full Paper 230 PDF is not hosted on this site. The local Paper 99 slides cover the operator-assistance study; embedded video appears as a still image in that PDF.
The repository releases the perception components, with setup notes for the required models and runtime. It does not reproduce the full team speech and teleoperation system.
Bangbang V2: On-Device Facial Emotion and Bilingual Speech Recognition for a Raspberry Pi-Based Tele-Operated Avatar Robot
Younghoon Chang · Siheng Ye · Yuechen Cao · Yanjie Zhou · Changshuai Liu · Hoe Yin Khong · Yuhong Dai · David Chieng · Christian Fernando Libaque-Saenz
Siheng Ye · second author of nine.
Accepted for presentation. The official programme schedules the paper for 5 November 2026, 17:18–17:23 · Auditorio Biblioteca · T3 · Peru time (UTC−5). Proceedings publication is not yet confirmed.
Paper metadata updated on 7 November 2026 using the paper files, acceptance evidence and official programme records. Proceedings publication, DOI and indexing are not yet confirmed.
