Forums | NoesisGUI

I was wondering if you guys working on getting NoesisGUI working on the Apple Vision Pro using Apple's Swift-based RealityKit 3D Engine?
https://developer.apple.com/documentation/realitykit

There are basically 5 types of applications on the Apple Vision Pro.
iPad/iPhone Windowed apps
Vision Pro Windowed apps
VR Immersive apps
AR Shared apps
AR Immersive apps

iPad/iPhone apps with NoesisGUI should already work. Vision Pro Windowed apps and VR Immersive apps both allow access to the Metal API so it should be straight forward to get NoesisGUI working there. That just leaves AR apps. The only way to create AR apps is using RealityKit, which does not give you access to the Metal API. Is there any work being done to get NoesisGUI working using RealityKit? Is that even possible?

Unity are building a Unity to RealityKit translator called PolySpatial to allow Unity users to create Vision Pro AR apps. Did you guys sign up for their beta program?
https://create.unity.com/spatial
Maybe it is possible to get NoesisGUI working via Unity.

Hello! We haven't had enough time to explore this. We have a few important VR clients and as soon as they demand more things I assume we will move forward. For now, I think the most viable path is going through Unity but definitely we are going to explore other alternatives.

Thanks for the detailed post!

Last week, Unity opened up the PolySpatial beta to all Unity Pro, Unity Enterprise and Unity Industry subscribers.
https://discussions.unity.com/t/welcome ... ram/270282

A lot of rendering system components are not supported compared to regular Unity.
https://docs.unity3d.com/Packages/com.u ... tures.html
Render Textures do seem to be supported though so than might be another path.

It seems native plugins are already supported. Wondering if we will be able to render directly in 3D with our Metal renderer. Definitely interesting. Thanks for sharing!

You'll be able to use your metal renderer to render directly in 3D for VR apps, since these apps have access to metal. This should work in Unity without making any new changes as it doesn't use the PolySpatial system.
You should also be able to make VR apps with the Native SDK version using this as a guide: https://github.com/metal-by-example/met ... -rendering

I don't think you'll be able to render directly in 3D for AR apps though. To make AR apps you have to use Apple's RealityKit high-level rendering engine, which does not give you a way to render your own 3D metal code. Unity's PolySpatial system simply translates Unity projects into RealityKit, so if it it is not possible in RealityKit, it won't be possible in Unity PolySpatial either.

While I am not sure how to do this using Unity PolySpatial, I think it might be possible to use your metal renderer in the Native SDK to render a UI and display it in 2D using RealityKit. The metal renderer in the Native SDK seems to use CAMetalLayer API which is also available in visionOS. There are a couple posts in the apple forums that seems to indicate you can render using metal into a CAMetalLayer and then display that in a flat 2D window in RealityKit:
https://developer.apple.com/forums/thread/732355
https://developer.apple.com/forums/thread/733398

The latest version of Xcode (15.2) now has official visionOS support. So you no longer need the beta version, if you happen to want to build the core library for visionOS. ;)
The Apple Vision Pro will be released in 3 weeks (Feb 2, 2024).

If you want to build Vision Pro Windowed support into the application framework, I think these two examples will help you get started:

CAMetalLayer

https://github.com/satoshi0212/visionOS ... main/Day35

Metal Shader and Transparency

https://github.com/satoshi0212/visionOS ... main/Day36

Any confirmation of NeosisGUI used on Vision Pro yet? Any special steps involved? I am interested.

You can't use NoesisGUI natively on the Vision Pro yet. They have not released NoesisGUI library compiled for visionOS.

You can run iPad apps on the Vision Pro, so any iPad apps using NoesisGUI, including their own sample app, should work in this compatibility mode.

I hope they release a compiled library for visionOS soon.

In the short term, such a library would allow support for:
An eventual NoesisApp for visionOS allowing for NoesisGUI UI in a 2D windows in an AR/MR app (see linked tweets above).
Unity Windowed Support - https://docs.unity3d.com/Packages/com.u ... dApps.html
Unity VR Support - https://docs.unity3d.com/Packages/com.u ... RApps.html
Unreal 5.4 VR Support - https://forums.unrealengine.com/t/suppo ... 1190042/40

(Side note: yes it was confirmed that Epic is added VR Vision Pro support to Unreal 5.4 whose first public preview will drop in late March at GDC24)

Over the long term, it would be great if there was some sort of RealityKit renderer in addition to the existing Metal renderer.

This would enable support for:
NoesisGUI UI in a 3D volume or immersive space aka AR/MR
Eye Tracking Hover for any UI element
3D World Space UI
RealityKit Physics

You can't use NoesisGUI natively on the Vision Pro yet. They have not released NoesisGUI library compiled for visionOS.

Could you create a ticket similar to #2988 but for visionOS? I will try to provide an experimenal core library.

Created: https://www.noesisengine.com/bugs/view.php?id=3124