PointAR

Your personal museum tour guide

An Augmented Reality museum guide tailor-made to each user. It can be experienced throughout the museum or remotely at any other location.

Time: FALL 2022

Role: AR UX / UI Design

Tools: Unity, Blender, NReal Glasses



TEAM

This project is a joint effort between Benjamin Glasser and Mario Santanilla. While we both were R&D XR Design interns at BMW in Munich, Germany, we spent much of our time in museums of all sorts.

APPROACH

After experiencing the literal and metaphorical red velvet rope that distances the viewer from their favorite works, we were inspired by the affordances of Unity, Blender, Unreal Engine and LiDAR sensors of which we were exploring extensively to augment the traditional museum experience and allow users to get closer to their favorite works of art inside and outside of the museum.


Cute cat Serious cat Serious cat

CONTEXT

Museums with huge collections are bound to their geographical location, as well as their building size. Their curatorial layouts are far from customized to the interests of the visitors by attempting to encompass massive timespans, materials and dimensional scopes that overwhelm the visitors and leave them feeling as though their visit was incomplete. This is heightened by experiencing a sense of distance from the intimacy that is desired with particular pieces.


AIM

How might we leverage the power of Augmented Reality to enhance the way visitors experience a museum visually and tactilely?




Serious cat

Feature #1

Survey

In our research, one of the key factors that led to museum disappointment was a sense of confusion and overwhelm felt when entering vast collections. Even when one goes on a guided tour, they are far from tailored to the individual's interests. Therefore, it was imperative that PointAR start with a survey to understand the user’s specific desires and interests in order to be able to structure the most precisely tailored tour.

For this, it was necessary to define a language that could easily transition to the AR experience. Through field research, we broke down the museum into three categories: time period, geography, and material form. We focused on these attributes to guide the user through the survey. This took us on a rich UI visual exploration, guided by rigorous UX research and testing. Serious cat


Feature #2

Tour

Serious cat
Serious cat
Serious cat
Serious cat
When brainstorming how to improve the experience of touring through a museum, immersion was an aspect that we wanted to improve through helpful guidance. We knew we would want to focus the user's attention by removing unnecessary stimuli visitors mentioned. We would do this by removing excess noise and highlighting targeted points of interest.
An aspect of AR that excites us is the opportunity to get rid of the frame and restrictions of traditional screen borders. This is why, through various explorations on innovative guidance systems that AR affords, we landed on the inspiration of Particle Systems: a fluid tool that is malleable and adapts to each unique environment. This system helps focus the visual attention of the user while following it, aided by a voice over tour guide that immerses the spectator on the journey from piece to piece.



Serious cat
Feature #3

Haptic Experience

While discussing ways to improve museum immersion, we realized that we did not need to be limited to augmenting museum experience through visual cues alone. We thought specifically of old carved sculptures and that a common aspect of the modern museum experience is that users cannot interact directly with the pieces in fear of damaging the expensive, fragile works. However, this adds another layer of distancing in a museum. Our hope is to provide the user with the experience of touch to get them closer to the intimacy of what the sculptor was feeling through their creative process.

The haptic experience is where PointAR ties all of these various elements together. Inspired by experiments in particle systems, we explored Point Cloud LiDAR scanners that have become more accessible in recent years. We decided to push the limits of this tool, to record not only visual 3D assets, but to integrate them in a system of haptic rings that allows the user to feel and touch the sculptures. We prototyped a haptic set of rings that could communicate with our application to get the individual that much closer to their favorite pieces.

This also allows for PointAR to reach users that are unable to visit particular museums and still have the opportunity to experience these pieces in the comfort of their own environment to explore, feel, and study.
Serious cat
Serious cat

PROCESS

The goal of this project was to explore how we could use the tools we know with AR prototyping and gain a deeper understanding of them by focusing on a possible use case in one of our areas of interests, art.


Our initial approach to the project was based on long hours of conversation we had on the current affordances of using individual vertices to structure computer generated imagery. Advancements in the usability of LiDAR Point Cloud recordings of 3D spaces and Particle Systems to simulate real life physical behaviors rid the need for the physical restrictions of the traditional frame imposed by standard screen hardware with fixed resolutions.




Serious cat


Our three part system consists of the survey, the tour and the haptic experience. The survey is the initial encounter the visitor has, when a series of multiple choice questions asked on their phone application determines their specific areas of interest. The walk through uses this information to create a tailored AR tour that uses a malleable Particle System to guide the user from the different locations. The haptics are activated when the user reaches a specific art piece that they want to explore further so that they can feel and interact with the piece.Using particle systems, PointAR can successfully and dynamically transition between realities and technologies. For instance, as many AR applications are currently hosted on phone apps, we felt it would be intuitive to host the survey on the phone screen which is where users currently feel most comfortable interacting with virtual buttons, but then transition seamlessly into AR space via a particle system that begins on the phone screen and transitions into the augmented space. This continues with the consistency of our visual system. Through this strategic prototyping, we created a system that could adapt in real time to have a dynamic and useful prototype to showcase and express our idea.



Serious cat


Using particle systems, PointAR can successfully and dynamically transition between realities and technologies. For instance, as many AR applications are currently hosted on phone apps, we felt it would be intuitive to host the survey on the phone screen which is where users currently feel most comfortable interacting with virtual buttons, but then transition seamlessly into AR space via a particle system that begins on the phone screen and transitions into the augmented space. This continues with the consistency of our visual system. Through this strategic prototyping, we created a system that could adapt in real time to have a dynamic and useful prototype to showcase and express our idea.


Serious cat
Serious cat







To prototype this guidance experience, we used Unity to create the Particle System that guides the user around the museum. This system is based on a Bezier curve that dynamically adapts to a series of objects that function as “magnets” that guide the particle system to distinct points in real time.



For the prototype we created a UI system that would allow us to manually move, rotate and activate the start, middle and end points of each Particle System section. This was necessary so that we could go into a museum environment to test this system in various spaces. By testing this, we found that these systems are successfully free of their borders and can grow, shrink, and move in space outside of screen restrictions which allows them to more properly delineate directional information within tight enclosed spaces. Furthermore, these systems that are made up of individual dots allow for the creation of digital twins of untouchable objects. This also keeps a consistent visual language throughout the museum tour.
Serious cat











NEXT STEPS

  • One of the most important next steps that we encountered was the possibility of turning the haptic system into a much more high definition prototype to test this design concept.
  • Iterate the project using other AR glasses to have a much more broader understanding of the state of the art of the AR tech.
  • We are always hungry for user feedback and as the project keeps on evolving, a never ending step is to go out and user test.