Towards a More Interactive Virtual Coach for Rehabilitation Exercises
Class Project: 15-821 Mobile and Pervasive Computing(IoT)
Team: Catherine Yu, Andong Jing
Work shared Equally
Mentors: Asim Smailagic, Dan Siewiorek, and Min Lee.
Date: December, 2021
Length: 2.5 months
Team: Catherine Yu, Andong Jing
Work shared Equally
Mentors: Asim Smailagic, Dan Siewiorek, and Min Lee.
Date: December, 2021
Length: 2.5 months
Problem:
For rehabilitation, clinical tests require direct, visual observation of patient's exercises, but:
The adoption of rehabilitation monitoring system remains a challenge due to:
Solution:
Virtual Coach for patients with musculoskeletal and neurological diseases, that:
![]()
Patients are instructed to raise their wrists to their mouths as if drinking water as an exercise for elbow flexion[1]. The hybrid model then identifies and corrects the following 3 compensation:
![]()
User Intefaces:
![]()
There are two UIs: one for the patients and one for the therapists. In the patients view, they can see the detected compensations and suggested corrections. In the therpists view, they can review patients exercises, label compensations, and adjust thresholds.
For rehabilitation, clinical tests require direct, visual observation of patient's exercises, but:
•tests
are time-consuming
•therapists
have limited
availabilities
The adoption of rehabilitation monitoring system remains a challenge due to:
•the
lack of user-centered
designs
•opaqueness of the machine learning algorithms[1]
Solution:
Virtual Coach for patients with musculoskeletal and neurological diseases, that:
•identifies
the compensations
•provide
feedback
& corrections
Final System Overview:
Final System Overview:
User Intefaces:
Model Comparisons:
Performance: For each patient(15 total), we applied leave-one-trial-out cross validation(10 trials/patient). For patients that do not have labeled compensation, KNN can not be applied.
Explainability: Predictions can be used to generate corrections(e.g. identified head tilted to the right can be corrected with keep head to the left)
New-patient friendly: The patient-specific KNN requires training. The patient specific RB has default values when training is absent. DNN is a universal model for all patients.
Performance: For each patient(15 total), we applied leave-one-trial-out cross validation(10 trials/patient). For patients that do not have labeled compensation, KNN can not be applied.
Explainability: Predictions can be used to generate corrections(e.g. identified head tilted to the right can be corrected with keep head to the left)
New-patient friendly: The patient-specific KNN requires training. The patient specific RB has default values when training is absent. DNN is a universal model for all patients.
Our Hybrid Model has:
•Comparative performance as NN when NN has good performance
•Comparative performance as NN when NN has good performance
•Better performance than NN when NN has low performance(patients 11&12)
Future Works:
•Include more rehabilitation exercises
•Get therapists insights on generated corrections
•Conduct user studies with patients
•Explore corrections beyond textual feedbacks
•Use computer vision to eliminate the need of Kinect sensor and thus make the system more accessible
Takeaways:
To facilitate Human-AI interactions for AI systems that supplement expert’s decision making and correct non-expert’s behaviors, it’s essential and valuable to analyze the task itself to construct a hybrid model that has both good performance and clear explainability.
Future Works:
•Include more rehabilitation exercises
•Get therapists insights on generated corrections
•Conduct user studies with patients
•Explore corrections beyond textual feedbacks
•Use computer vision to eliminate the need of Kinect sensor and thus make the system more accessible
Takeaways:
To facilitate Human-AI interactions for AI systems that supplement expert’s decision making and correct non-expert’s behaviors, it’s essential and valuable to analyze the task itself to construct a hybrid model that has both good performance and clear explainability.
[1] Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A., & Bermúdez i Badia, S. B. (2021, May). A Human-AI Collaborative Approach for Clinical Decision Making on Rehabilitation Assessment. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-14).
[2] Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A., & Bermúdez i Badia, S. (2020). Co-Design and Evaluation of an Intelligent Decision Support System for Stroke Rehabilitation Assessment. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW2), 1-27.