Above: Emergency telemedicine system. Left: Expert annotates patient remotely using Z-Space system. Right top: Annotations are projected on patient using Kinect and projector. Right bottom: Detail of Z-Space patient visualization.
The telehealth project uses principles of 3D teleimmersion — a two-way remote communication system in which each party maintains an audiovisual representation of each other — to remotely deliver medical care in various scenarios. The telehealth framework uses Microsoft Kinect cameras to stream real-time images of each party (e.g., a doctor and a patient) into a 3D virtual environment, allowing them to interact in the same virtual space. In addition, the Kinect-acquired human kinematics can be used (in combination with other sensors) to obtain quantitative measurements on the performance of the exercises or to assess the patient’s abilities.
This process can be used to greatly streamline and improve the rehabilitation process when regular in-person meetings cannot be scheduled between a patient and therapist (or doctor). The telehealth project is carried out in collaboration with UC Irvine and UC Davis Medical Center. (Image: Two parties interacting in the same virtual space.)
In many emergency scenarios, medical care is first provided by first responders in the field and later by physicians at designated trauma centers. In the so-called “golden hour,” the efficiency of patient triage and medical transport greatly affects the outcomes of emergency treatment.
In current practice, the initial communication and interaction between physician and first responders is limited to voice or video conferencing, and the attending physicians cannot receive more comprehensive, critical patient information.
The emergency telemedicine project leverages 3D teleimmersion and augmented reality (AR) technologies to address this communication gap. First, a real-time patient data acquisition and visualization module captures patients’ external trauma and vital signals in the field and precisely renders the signals for remote physicians using 3D teleimmersion. Second, an AR module allows physicians and first responders to examine patient data and collaborate on simple medical procedures. Finally, a telemedicine communication module enables secure and robust deployment of the above two modules over long distance through the internet.
We envision that extension of our technology can bring about disruptive new applications in telemedicine in developing countries, such as connecting patients and clinics in poor rural areas with better-equipped hospitals in urban areas. (Image: Emergency telemedicine system, expert side.)
Research has shown the positive effect of physical activity and exercise on both physical and cognitive abilities. In collaboration with Northeastern University, Oregon Health and Science University, and the University of Washington, our team has developed a semi-autonomous system for exercise for the elderly.
The system includes interactive video feedback based on real-time data collected by a Microsoft Kinect camera. During exercise, the system collects kinematic data and provides feedback on the performance to encourage the user. The software was designed in collaboration with an exercise coach to accommodate elderly users with respect to exercise options and user interface.
This project is carried out in collaboration with Oregon Health & Science University. (Image: Kinect exercise system user interface.)
David Anton, Gregorij Kurillo, Alfredo Goñi, Arantza Illarramendi, and Ruzena Bajcsy. Real-time communication for Kinect-based telerehabilitation. In Future Generation Computer Systems, 75, pages 72-81, October 2017.
David Anton, Gregorij Kurillo, Allen Y Yang, and Ruzena Bajcsy. Augmented telemedicine platform for real-time remote medical consultation. In International Conference on Multimedia Modeling, pages 77–89. Springer, 2017.
Gregorij Kurillo, Allen Y Yang, Victor Shia, Aaron Bair, and Ruzena Bajcsy. New emergency medicine paradigm via augmented telemedicine. In International Conference on Virtual, Augmented and Mixed Reality, pages 502–511. Springer, 2016.
Ferda Ofli, Gregorij Kurillo, Štěpán Obdržálek, Ruzena Bajcsy, Holly Brugge Jimison, and Misha Pavel. Design and evaluation of an interactive exercise coaching system for older adults: Lessons learned. IEEE journal of biomedical and health informatics, 20(1):201–212, 2016.
Qifei Wang, Gregorij Kurillo, Ferda Ofli, and Ruzena Bajcsy. Evaluation of pose tracking accuracy in the first and second generations of Microsoft Kinect. In IEEE International Conference on Healthcare Informatics (ICHI), pages 380–389. IEEE, 2015.
Qifei Wang, Gregorij Kurillo, Ferda Ofli, and Ruzena Bajcsy. Unsupervised temporal segmentation of repetitive human actions based on kinematic modeling and frequency analysis. In IEEE International Conference on 3D Vision (3DV), pages 562–570. IEEE, 2015.
Gregorij Kurillo, Ferda Ofli, Jennifer Marcoe, Paul Gorman, Holly Jimison, Misha Pavel, and Ruzena Bajcsy. Multi-disciplinary design and in-home evaluation of Kinect-based exercise coaching system for elderly. In International Conference on Human Aspects of IT for the Aged Population, pages 101–113. Springer, 2015.
Gregorij Kurillo, Jay J Han, Alina Nicorici, and Ruzena Bajcsy. Tele-MFAsT: Kinect-based tele-medicine tool for remote motion and function assessment. Stud Health Technol Inform, 196:215–221, 2014.
Štěpán Obdržálek, Gregorij Kurillo, Edmund Seto, and Ruzena Bajcsy. Architecture of an automated coaching system for elderly population. Studies in health technology and informatics, 184:309, 2013.
