What if museum visitors have AR glasses? I spent my nights and weekends to explore opportunities and problem spaces:
• Bridge information gap between art and art label
• Break down linear art learning
• Restore damaged art
• Research general spatial UI interaction patterns
• Quest 2, XR interaction toolkit, Figma, Unity 3D
• Solo project
• User tested with ex-NetEase QA, UX designers, PMs
Bridge art and art labels
A larger and more contextual placement of art label, rather than the small printout next to the bottom of the painting.
Break down linear art learning
Visual highlights couple with traditional yet chunked audio guide
enable an easier and more engaging learning experience.
Research interaction patterns
Insights discovered after experimenting with direct hand interactions on 3D UIs:
• hands/fingers don't move in straight line, but an arc/squiggle line
• 15-45 degree tilt directly facing users is easier for tapping this kind of 3D buttons
• a curved interface layout is needed for readability and reach
Other findings after 50 hours in VR:
• virtual grab still feels odd and unnatural because there is no physical resistance from the virtual objects
• the lower the visual fidelity of the virtual environment, the harder to perceive depth
• dry eye after 20 mins of use
• forehead and cheekbore pressure after 20 mins of use
Next steps/working on now:
• Use Meta interaction SDK
• Experiment with phone input
• Experiment with eye + hand input
• Gnerative AI x Art
Known issues (to think about):
• How are these experience UIs activiated?
• How are the UI placed in space and in relation to the user?
• How would this experience work with multiple paintings?
Past Explorations
Art Stories, Like button, and Similar Arts
❌ unclear reference and hard to interact in story mode
2-state buttons with large volumetric trigger zones
✅ easy interaction
❌ unclear which tile corresponds to which story
3-state buttons with height-based volumetric trigger zones
Tilting head/body to see explore stories
✅ reminiscent of tilted head thinking and moving physically to see art at different angles
✅ minimal readability impact despite reading in an angle
❌ unclear how many stories to explore
Volumetric zones as trigger target
✅ low effort
❌ failed to communicate "zone" with these shapes
Controller point and hover with Bezel/Bezi and Quest 2