Periodically I share my views with others. Here’s something I feel is still highly relevant and worth sharing with all.
AR/VR/MR - On Multi-Modal Input
If the goal is to create a human centric natural experience, I would go as far as possible in this area. You’ve already started by integrating a mouse - I would go full monty on this, consider the following:
You should be able to quickly do this in your system. Identify a trigger in order to enter a mode (one that is forgiving, and easy to escape) to use gestures. If you can set a trigger (clapping hands) you could then enter a gesture mode, where both left and right hand are independently tracked, and gestures are allowed to do whatever your heart fancies. Eventually complicated gestures can be built, and muscle memory will help in “remembering”. I have no idea what the neuroscience is behind this, but know this is something you should accomplish.
Head Gaze (The third eye)
You should be able to easily support head gaze. Considering a “dot” in field of view as a reference point for additional action. Normal Mode - you might place informational affordances in view that when the head gaze is co-located on the affordance (with a time delay, similar to your grabbing delay), an action is automatically triggered. It might pop up a bubble, or something else. In a move mode, when your hands are already occupied, you may be able to use this head gaze to perform additional functions.
Your system is already tracking hands. Track more then that - convert any object into a dial, button, or even an 2D touch application or touch surface. Think about a series of platonic solids sitting in your workspace, either as tools or physical representations of virtual objects.
Absolutely required for text input, but you could go beyond far beyond that. Wont spend too much time chatting about this, as it’s not hard to imagine, but might be lower priority, and more useful for applications.
If you could track what your eyes are looking, you might be able to do a lot more. I’m pretty sure your current product wouldn’t support eye gaze. It may or may not be worth it - however it would be a wonderful experiment to try.
Think about clicking. Done. I know nothing about this. But I’m sure you could go to town with it. Your product sits around the head, it also has a lot of sensors already incorporated into it (meaning increasing the BOM cost might not be a problem).
On Cross Computing Devices
You’ve already started to think about this area. How do you create a cohesive unified & collaborative experience across all surfaces & spaces. I absolutely love the pushing and pulling content from your phone. I also know its very difficult to get that level of integration. Its very hard from a PHONE to share content to other applications, let alone devices. And their are so many layers of control (OS, then the application). You may be able to try different approaches (ie Object ID’s, visual markers, or SDK integration). Ideally creating a suite of your own Desktop / Mobile applications that are super friendly with this space.
The Reciprocal must be true. It should be effortless and painless to peer into this space from a Desktop or mobile device.
If you are looking to develop a fluid system for humans, then you must not ignore that they will live on different devices.
On Design Principals
Your principles are a solid way to move forward. I am a firm believer and advocate of developing design systems that reinforce a point of view. This is really no different then Jakob Neilson & Alan cooper producing their seminal works on usability and persona’s. (BTW, I routinely broke both Neilson’s rules :).
In order to evolve your design system, I suggest you move to patterns, and larger compositions before you solidify/modify your next set of principles. Diverge first and then Converge. What does this mean?
It means identifying and/or creating archetypes of the different applications that might be used in your space. From workspaces for viewing and making 3D models, to medical applications, information spaces, and other productivity applications that might be transformed by re-interpreteing them in this space.
After identifying these, then one would start to go through an exercise of ideation to build out the epicentres for those applications and then build patterns, for direct manipulation, content navigation, data entry, dialoging systems, notification systems, etc in-context to each of these applications. Before any normalizing exercise happens, make sure the purpose built applications directly solve those problems.
After variants of each application type is created, then one would evaluate based on quantitive, qualitative and intuition.
Then you normalize and create pattern libraries, and SDK components.
And finally you can codify foundational design principals. (you would always have a running list)
And let people break those rules.
Sanjiv Sirpal, October 2017