Towards a More Interactive Virtual Coach for Rehabilitation Exercises

Class Project:  15-821 Mobile and Pervasive Computing(IoT)

Catherine Yu, Andong Jing
Work shared Equally

Mentors: Asim Smailagic, Dan Siewiorek, and Min Lee.

Date: December, 2021

Length: 2.5 months

For rehabilitation, clinical tests require direct, visual observation of patient's exercises, but:
•tests are time-consuming
•therapists have limited availabilities

The adoption of rehabilitation monitoring system remains a challenge due to:
•the lack of user-centered designs
•opaqueness of the machine learning algorithms[1]

Virtual Coach for patients with musculoskeletal and neurological diseases, that:
•identifies the compensations
•provide feedback & corrections

Final System Overview:

Patients are instructed to raise their wrists to their mouths as if drinking water as an exercise for elbow flexion[1]. The hybrid model then identifies and corrects the following 3 compensation:

User Intefaces:

There are two UIs: one for the patients and one for the therapists. In the patients view, they can see the detected compensations and suggested corrections. In the therpists view, they can review patients exercises, label compensations, and adjust thresholds.
Model Comparisons:
Performance: For each patient(15 total), we applied leave-one-trial-out cross validation(10 trials/patient). For patients that do not have labeled compensation, KNN can not be applied.

Explainability:  Predictions can be used to generate corrections(e.g. identified head tilted to the right can be corrected with keep head to the left)

New-patient friendly: The patient-specific KNN requires training. The patient specific RB has default values when training is absent. DNN is a universal model for all patients.

Our Hybrid Model has:
•Comparative performance as NN when NN has good performance
•Better performance than NN when NN has low performance(patients 11&12)

Future Works:
•Include more rehabilitation exercises
•Get therapists insights on generated corrections
•Conduct user studies with patients
•Explore corrections beyond textual feedbacks
•Use computer vision to eliminate the need of Kinect sensor and thus make the system more accessible

To facilitate Human-AI interactions for AI systems that supplement expert’s decision making and correct non-expert’s behaviors, it’s essential and valuable to analyze the task itself to construct a hybrid model that has both good performance and clear explainability.

[1] Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A., & Bermúdez i Badia, S. B. (2021, May). A Human-AI Collaborative Approach for Clinical Decision Making on Rehabilitation Assessment. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-14).
[2] Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A., & Bermúdez i Badia, S. (2020). Co-Design and Evaluation of an Intelligent Decision Support System for Stroke Rehabilitation Assessment. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW2), 1-27.