RESEARCH & DEVELOPMENT * AR & WEB

RehabAR

: Providing Real-Time Feedback to Patients During Stroke Rehabilitation
in Mixed Reality

OVERVIEW

ROLE

AR/VR Developer

AR/VR Designer

ADVISORS

Dr. Edward Kim

SKILLS

Unity

Google Mediapipes

XR Interaction toolkit

Figma

TOOLS

Meta Quest3

TIMELINE

Sep 2024 - Dec 2024

TEAM

Ryugwang(Josh) Jang

James Hu

Mia Wu

Tian(Iris) Qin

May Liu

ABOUT MY EXPERIENCE

During my first semester at UC Berkeley, I had the opportunity to work with Dr. Edward Kim as a research assistant on an AR/VR rehabilitation project.


Stroke patients often face significant challenges in rehabilitation due to limited mobility, and those in non-urban areas experience disparities in access to care. The project aimed to develop a solution leveraging AR/VR technologies to provide remote rehabilitation support and enhance patient outcomes.


Throughout my involvement, I contributed to the development and refinement of this innovative system. The project was structured into three phases:


  • Problem identification and scope definition


    I collaborated with the team to identify key challenges faced by stroke patients in rehabilitation. This involved analyzing user needs, reviewing rehabilitation protocols, and exploring the application of AR/VR technologies to address these challenges.


  • AR-based feedback system design

    My primary contribution was designing a real-time feedback system to assist patients during rehabilitation exercises. The system used AR to overlay corrective guidance, helping patients perform movements more accurately. This process included prototyping, testing the AR interface, and refining the interaction flow to ensure accessibility and usability.


  • Evaluation and iteration

    In this phase, we conducted user testing sessions to evaluate the effectiveness of the AR feedback system. Feedback from stroke patients and healthcare professionals guided iterative improvements, ensuring the solution was both intuitive and effective in promoting recovery.


This experience not only deepened my understanding of AR/VR applications in healthcare but also strengthened my skills in user-centered design, prototyping, and interdisciplinary collaboration. It reaffirmed my passion for leveraging technology to address real-world problems and improve lives.

Challenges Identified

  • Feedback Accuracy: The system often provided inconsistent scoring compared to in-person evaluations.

  • Engagement: Generic videos for errors frustrated patients who believed they performed movements correctly.

  • Accessibility: Stroke patients’ limited motor abilities and unfamiliarity with MR interfaces made usability challenging.

Problem

How might we make stroke rehabilitation more affordable, accessible, and scalable while overcoming patients’ limited mobility and unfamiliarity with VR/AR technologies?

In-person rehabilitation is costly, inaccessible to many, and lacks scalability. Stroke patients often struggle with limited mobility and are unfamiliar with VR/AR technologies.

Context

Each year, 795,000 people in the US experience a stroke, with over $56 billion spent annually on stroke-related care. Patients in non-urban areas face disparities in rehabilitation access. Stroke rehabilitation often requires consistent and timely access to specialized care, which is not always available in rural or underserved areas. This gap leads to delayed recovery and reduced quality of life for many patients.


Our goal is to bridge this gap by leveraging technology, such as AR/VR, to deliver accessible and effective rehabilitation solutions. By creating immersive and remote rehabilitation environments, we aim to provide equitable care to all patients, regardless of their geographic location.

Goals

Enhance Accessibility & Scalability:


Develop an AR/VR-based rehabilitation system that provides equitable and scalable therapy solutions, ensuring stroke patients in underserved areas can access care regardless of their geographic location.

Improve Affordability & Usability:


Create a cost-effective and user-friendly platform that reduces the financial burden while accommodating patients with limited mobility and unfamiliarity with AR/VR technologies through intuitive design and support.

Promote Effective Recovery:


Deliver immersive and engaging rehabilitation experiences that encourage consistent participation, enabling better adherence to therapy routines and improved recovery outcomes.

Key Features

  • Body Tracking: Used Meta Movement SDK and Google MediaPipe to track joint angles for precise movement evaluation.

  • Real-Time Feedback: Integrated mirror feedback and color-coded visual cues to help patients correct their movements.

  • Kiosk Mode: Simplified interaction by launching the rehabilitation app automatically, bypassing complex menus.

  • Remote Customization: Therapists uploaded 3D scans of patients’ home environments and prescribed personalized rehab tasks through a web portal.

  • Standardized Motion Terminology: Adopted the Fugl-Meyer Assessment and standardized motion terms (e.g., Flexion, Extension, Radial Deviation) with accompanying images, ensuring clear and consistent communication within the team.

Implementation

Technologies Used:

  • Meta Movement SDK for body tracking.

  • Google MediaPipe for motion analysis.

  • Unity for AR/VR integration.

Visual Feedback:

  • Colored spheres indicating correct/incorrect movement alignment.

  • Mirror functionality for self-assessment in real-time.

Web Integration:

  • Therapists annotated and prescribed exercises remotely via a user-friendly web portal.

Results & Evaluation

  • Tracking Accuracy Comparison: Conducted a detailed comparison of Meta Movement SDK and Google MediaPipe to evaluate their accuracy and reliability in tracking joint angles for rehabilitation exercises.

  • Accessibility Enhancement Plan: Proposed a kiosk mode for Meta Quest to improve accessibility for stroke patients, enabling direct access to the rehabilitation app upon wearing the headset. Submitted a support request due to the feature being part of a paid service.

  • Future Evaluation Needs: Identified the need for further evaluation through user testing and therapist feedback, focusing on usability, accessibility, and patient outcomes.

LEARNINGS & NEXT STEPS

Learnings

1. Regulatory Challenges: Regulatory hurdles posed significant delays, highlighting the need to account for legal and compliance factors early in the development process.

2. Visualization Limitations: The mesh-only visualization in the web portal, while functional, lacked realism, which could impact patient engagement and comprehension.

3. Technical Integration: Comparing Meta Movement SDK and Google MediaPipe provided valuable insights into tracking accuracy, but also revealed challenges in achieving seamless integration.

4. Accessibility Focus: Developing the kiosk mode concept reinforced the importance of accessibility for stroke patients who may face difficulties navigating complex interfaces.

Next Steps

1. Patient Trials: Secure necessary regulatory approvals to conduct trials with stroke patients, focusing on usability, accessibility, and therapeutic outcomes.

2. Enhanced Visualization: Improve the visual realism of the web portal by integrating more detailed 3D models and environmental textures.

3. Expanded Task Library: Add diverse rehabilitation tasks tailored to different stages of stroke recovery, increasing system flexibility and effectiveness.

4. Kiosk Mode Implementation: Collaborate with Meta Quest to implement kiosk mode functionality, enabling seamless app access for patients.

5. Feedback Integration: Gather detailed feedback from therapists and caregivers to iteratively refine the system and address real-world needs.

Next

Curious? Feedback? Collaboration?

I'd love to meet you :)

Feel free to grab a virtual coffee with me via email!

2025 Josh Jang