Forums | NoesisGUI

Hi

Will Noesis work (with suitable modifications) in an AR or VR environment?

If not, is it possible to include in a project but selectively exclude it from VR/AR builds?

== John ==

The latest biggest VR game I heard about, Sansar, is made by Second Life creator Linden Lab and it uses NoesisGUI, according to this tweet by NoesisGUI https://twitter.com/noesisengine/status ... 2292831232

For VR -- and AR -- you just float a primitive in space, anchor it somewhere convenient, and then drop Noesis as a texture on it. There's some event forwarding for interaction. It's not super complicated. Once I get my main computer up and running again, I can share a Vive and leap motion example on Unity doing the menu system from Sword Art Online, a popular Anime, complete with a disabled logout button to freak out your users. Maniacal laughter you must obtain elsewhere, though, I'm fresh out.

I also have a project where I was testing Tilt Brush's menu system using Noesis. Cube centered on hand, each face is a Noesis Panel, and then you interact with it using your other hand.

Noesis only renders in 2D, and you likely won't like what happens if you assign it to the overlay plane in VR. However, if you have the Curved UI for VR asset in Unity, for example, it's pretty trivial to adapt that to run with Noesis driving the UI. At the end of the day, Noesis is just writing pixels to a texture anyway -- it's just a matter of how you're going to map that to the screen.

For AR on iOS or Android, where you're not using a head mounted display, you can also just use Noesis as is on the overlay buffer. Works fine, in that case, since it's just drawing pixels to overlay either way, use Opaque or not as desired based on where you want the UI to cover or be clear.

Thanks Dave (I assume)

I liked Noesis because of the XAML / Vector based approach. In VR, I don't want to float textures over objects - I was looking for something more 'line' based.

Thanks, I appreciate the input. I'll try it.

Will look forward to the example too!

Right now, I seem to have broken my Blend (really VS 2015) integration Will go deal with that ... sigh.

== John ==

If I attach the Noesis GUI to an object, it seems to render but it's "inside" the object.

I get the 'attach as a texture' but I'd prefer to have it drawn in screen space like the regular UI on flat devices. What I like about Noesis is that lines are lines, not anti-aliased to a bitmap and would like to keep that. With VR you can zoom into things and it seems to me Noesis would have the unique ability that this wouldn't then pixelate.

Can you get it to render without an object rendering? I could then create an empty GameObject and just position it somewhere.

I tried this with a cube, turning off the mesh renderer, and an empty game object and the UI doesn't show up - which may mean rendering to a texture (which is in the docs) is the only way to go.

== John ==

If I attach the Noesis GUI to an object, it seems to render but it's "inside" the object.

I get the 'attach as a texture' but I'd prefer to have it drawn in screen space like the regular UI on flat devices. What I like about Noesis is that lines are lines, not anti-aliased to a bitmap and would like to keep that. With VR you can zoom into things and it seems to me Noesis would have the unique ability that this wouldn't then pixelate.

Can you get it to render without an object rendering? I could then create an empty GameObject and just position it somewhere.

I tried this with a cube, turning off the mesh renderer, and an empty game object and the UI doesn't show up - which may mean rendering to a texture (which is in the docs) is the only way to go.

== John ==

Start with this:
https://unity3d.com/learn/tutorials/top ... erfaces-vr

In Unity, adding HUD style non-diegetic UI is usually accomplished by using Screen Space - Overlay or Screen Space - Camera render modes on a UI Canvas.
... image removed...
This approach usually doesn’t work in VR - our eyes are unable to focus on something so close, and Screen Space-Overlay is not supported in Unity VR.
...
Instead, we generally need to position our UI within the environment itself using World Space Canvas render mode. This will allow our eyes to focus on the UI. This is known as Spatial UI.

So here's the thing, right -- screen space is a plane across the display. It works because the distance between the screen and whatever you're drawing is at 0 distance. At the end of the day, we're using Pythagorean Theorem to figure out which pixels to light, and this is a special case where the math works out and no scaling is needed.

When you move into VR, the lens on the HMD isn't a triangle to the LCD -- and so if you draw a straight line on the LCD, it will be curved. The pixels also map to different sizes on the lens, there's a higher density near the middle. And so the math there becomes super hard and has to be done for each brand of HMD. So in general, we'd like to leverage what the various SDKs know about the headset.

And so you make a primitive using the engine, and texture it with the UI.

Noesis wouldn't have to do the math, that's all built in. It would just have to draw a line; the VR rendering software that's built into the game engine will do the warping for you (and for that matter, also the latency reduction, etc).

If someone 'renders' a cube, they don't have to warp it. They just draw a cube.

Noesis wouldn't have to do much more than that.

I imagine that it might have to have a scaling factor, so that it knows where to draw, but that could be based on the objects in-game dimensions. If I put a XAML on top of a cube that's going from 0 to 4, then it knows 0 to 300 becomes 0 to 4 when it renders the XAML. From what I can gather (still learning the UI), it obviously already does that mapping, as if my screen changes dimensions the XAML still stretches the whole way. So, Noesis is getting the screen dimensions and scaling the XAML.

BTW, Whoops, typo, I said 'screen space' but meant World space. Touches / mouse / controller clicks are a little more interesting but there's helper functions that do the mapping.

== John ==

P.S. Yes, I know that section ... there's other reasons to avoid near to the POV objects, called the Vergence-accommodation conflict. In fact, that article says "our eyes are unable to focus on something so close," which is inaccurate. In an HMD, you CAN focus on everything; it's the way the optics are built. The HMD lenses focus the display at a simulated infinite distance, so every pixel that is rendered is in focus. The problem isn't that you can't focus. The problem is that your brain 'knows' something up close requires different focusing than something far away, in addition to the stereoscopic effect. So, you have a higher chance of getting queasy because your brain says "wait, something that close would require focusing differently, and I don't have to do that, it's right in focus ..." See this article for example: https://medium.com/vrinflux-dot-com/ver ... ab1a7d9ba

Noesis only renders in 2D, and you likely won't like what happens if you assign it to the overlay plane in VR.

Regarding this, Noesis can render in real 3D, without rendering to texture, just direct to the camera space. Each node of the visual tree (a button for example) can be fully rotated using a 4x4 matrix. We have a few C++ clients already using it for VR. Although we didn't have time yet to properly integrate this into Unity VR. Will do it soon or later.

Thanks, that's good to hear!

== John ==

Regarding this, Noesis can render in real 3D, without rendering to texture, just direct to the camera space. Each node of the visual tree (a button for example) can be fully rotated using a 4x4 matrix. We have a few C++ clients already using it for VR. Although we didn't have time yet to properly integrate this into Unity VR. Will do it soon or later.

Did later ever happen with integrating this properly?