### Placing a UI element in 3D with matrices

Hello, I've been trying to setup 3d transform behaviour for our UI and have run into some issues. The intention is for an icon to be placed in world and have it anchored to a specific position. I've ended up running into a bit of trouble getting some of the matrix math right so I just wanted to talk through my setup and ask if there is something I may be missing or need to account for?

In my xaml I'm doing this:
And for testing purposes I have a view model that has a transform that is set to an identity matrix for it to be at 0,0,0 in world with no rotation.

And then the view projection matrix which I am currently passing to the render call like so (I realise you can also call SetProjectionMatrix on the view but for now I'm using Render):
For the view projection matrix, I used the default one (from SetSize in View.cpp) as a starting point to jump off from where there is an awayFromCamera offset, our games own view matrix and then the projection matrix (but for now I have used infinite near/far) and then multiply by a 'viewport' matrix. So it looks something like this:
The problem is the camera rotation works perfectly it seems but the position of the icon is still following the camera but you can see it moving ever so slightly almost is if it is really far away, so the scale for the position must be wrong. My only guess is that the position needs to be converted to screen space e.g. 1.0 on the x axis becomes 1920 * 0.5 or something like that? So to try and solve that I decide to multiply the position of the view matrix by the awayFromCamera's translation component, but I ended up having to add some arbitrary offsets that break when the fov changes.

Here's the full code for the view projection matrix, it's gotten a bit unwieldy and I definitely feel I've over complicated it and made a bit of a mess so please have a look and let me know what I should potentially be doing. One thing to note, our game has the y and z axis flipped hence the weird order in the view matrix and projection matrix.
The main two things that seem wrong, 1. the arbitrary offset values, and 2. scaling the position to get it into screen space:
I feel like there must be some matrix I can factor in to solve this problem right? Or is there even maybe a way I can make noesis use the same coordinate scale as our game? I thought maybe that viewport matrix should be responsible for that but it doesn't seem to impact the positioning anyway. I have a feeling I would need to do this same scaling on the transform of each icon in their view model too currently, so is there a way to avoid that as well?

In my xaml I'm doing this:

Code: Select all

```
<noesis:Element.Transform3D>
<noesis:MatrixTransform3D Matrix="{Binding Transform}"/>
</noesis:Element.Transform3D>
```

And then the view projection matrix which I am currently passing to the render call like so (I realise you can also call SetProjectionMatrix on the view but for now I'm using Render):

Code: Select all

```
m_View->GetRenderer()->Render(Get3DViewProjectionViewportMatrix());
```

Code: Select all

```
return awayFromCamera * (viewMatrix * projectionMatrix) * viewport;
```

Here's the full code for the view projection matrix, it's gotten a bit unwieldy and I definitely feel I've over complicated it and made a bit of a mess so please have a look and let me know what I should potentially be doing. One thing to note, our game has the y and z axis flipped hence the weird order in the view matrix and projection matrix.

Code: Select all

```
Noesis::Matrix4 C_NoesisGUIRenderExt::C_RenderCallback::Get3DViewProjectionViewportMatrix()
{
#pragma UE_TODO("2021-10-18: Robert.Glen: This math will need to be revisted. Currently there are a lot of magic numbers and this solution doesn't work as soon as the fov changes.")
//get the game camera and retrieve view and projection matrices, we need these independently so we can manipulate the view before multiplying them together.
camera::I_Camera const* const pCamera = camera::I_GameCamera::GetInstance()->GetPlayerCamera(0);
sys::math::C_Matrix view = pCamera->GetViewMatrix();
sys::math::C_Matrix4 projection = pCamera->GetProjectionMatrix();
// view matrix, we need to flip y and z axies.
Noesis::Matrix4 viewMatrix = Noesis::Matrix4(
view.m[0][0], view.m[2][0], view.m[1][0], 0.0f,
view.m[0][2], view.m[2][2], view.m[1][2], 0.0f,
view.m[0][1], view.m[2][1], view.m[1][1], 0.0f,
view.m[0][3], view.m[2][3], view.m[1][3], 1.0f);
// projection y and z axies flipped also but not y and z component of each axis for some reason.
// for now we are using an infinite near and far - but these should be replaced with their correct values eventually. Current problem is with our position scaling on the view matrix - it may mean we need to scale near and far as well.
#pragma UE_TODO("2021-10-18: Robert.Glen: Reintegrate true projection near/far (left commented for now).")
Noesis::Matrix4 projectionMatrix(
projection.m[0][0], projection.m[1][0], projection.m[2][0], projection.m[3][0],
projection.m[0][2], projection.m[1][2], projection.m[2][2], projection.m[3][2],
projection.m[0][1], projection.m[1][1], 1.0f/*projection.m[2][1]*/, 1.0f/*projection.m[3][1]*/,
projection.m[0][3], projection.m[1][3], -1.0f/*projection.m[2][3]*/, projection.m[3][3]);
float const width = static_cast<float>(m_BackBufferWidth);
float const height = static_cast<float>(m_BackBufferHeight);
// Adjust field of view depending on resolution (similar to UWP)
float const hfov = pCamera->GetFOV() * Noesis::DegToRad;
// awayFromCamera was included in default noesis projection matrix, it seems to be some offset that moves the matrix by half the width/height/depth to be in the right place. We need to determine offsets and 'away' or depth.
// currently arbitrary, we will need something more concrete in future.
float const offsetX = 0.5f;
float const offsetY = 0.6f;
float const offsetZ = 2.0f;
float const cot = cosf(hfov) / sinf(hfov);
float const away = (height * offsetZ) * cot;
// these offsets currently seem to be arbitrary. It's likely we are missing something in the math below but for now this works as a temporary solution.
float const extraPositionalOffsetX = 0.6f;
float const extraPositionalOffsetY = 0.9f;
Noesis::Matrix4 awayFromCamera = Noesis::Transform3::Trans(-width * offsetX, -height * offsetY, away).ToMatrix4();
Noesis::Matrix4 viewport = Noesis::Matrix4::Viewport(width, height);
// scale the position by viewport size, due to Noesis coords being screenspace rather than normal.
viewMatrix[3].x *= -awayFromCamera[3].x * extraPositionalOffsetX;
viewMatrix[3].y *= -awayFromCamera[3].y * extraPositionalOffsetY;
viewMatrix[3].z *= awayFromCamera[3].z;
return awayFromCamera * (viewMatrix * projectionMatrix) * viewport;
}
```

Code: Select all

```
viewMatrix[3].x *= -awayFromCamera[3].x * extraPositionalOffsetX;
viewMatrix[3].y *= -awayFromCamera[3].y * extraPositionalOffsetY;
viewMatrix[3].z *= awayFromCamera[3].z;
```

### Re: Placing a UI element in 3D with matrices

Hi Rob,

Before we analyze your post in more detail, I wonder if you read our OculusVR example.

In that example, we are calculating the projection matrix from the headset per frame. Note the magic numbers scaling Noesis View dimensions to real world units

Before we analyze your post in more detail, I wonder if you read our OculusVR example.

In that example, we are calculating the projection matrix from the headset per frame. Note the magic numbers scaling Noesis View dimensions to real world units

Code: Select all

```
// UI location attached to the camera
XMMATRIX prod_ = XMMatrixMultiply(Camera(eyePos, eyeQuat).GetViewMatrix(), proj);
Noesis::Matrix4 viewport = Noesis::Matrix4::Viewport(eyeWidth, eyeHeight);
Noesis::Matrix4 offset = Noesis::Transform3::Trans(-1.4f, -0.2f, -1.25f).ToMatrix4();
Noesis::Matrix4 scale = Noesis::Transform3::Scale(0.002f, 0.002f, -0.002f).ToMatrix4();
Noesis::Matrix4 eyeMtx = scale * offset * (*(Noesis::Matrix4*)&prod_) * viewport;
```

### Re: Placing a UI element in 3D with matrices

I'm actually going to be handing this problem off to our rendering team (I'm a UI programmer so this isn't something I usually work on) so it's not in my hands anymore, but I can pass any info on.

I did see that oculus VR code, as well as another forum post that covers a similar issue: viewtopic.php?f=3&t=2378.

That scalar matrix could be what I was missing though, I think my awayFromCamera matrix already fulfills the same purpose as that offset matrix maybe? I'm assuming those numbers would need to be different for a standard setup e.g. a single 1080p or 4k monitor, or do you think they should work across the board? And are those numbers trial and error or is there some more context to them?

I did see that oculus VR code, as well as another forum post that covers a similar issue: viewtopic.php?f=3&t=2378.

That scalar matrix could be what I was missing though, I think my awayFromCamera matrix already fulfills the same purpose as that offset matrix maybe? I'm assuming those numbers would need to be different for a standard setup e.g. a single 1080p or 4k monitor, or do you think they should work across the board? And are those numbers trial and error or is there some more context to them?

### Re: Placing a UI element in 3D with matrices

That scalar matrix could be what I was missing though, I think my awayFromCamera matrix already fulfills the same purpose as that offset matrix maybe? I'm assuming those numbers would need to be different for a standard setup e.g. a single 1080p or 4k monitor, or do you think they should work across the board? And are those numbers trial and error or is there some more context to them?

- viewport depends on the resolution, so it is different for 1080p and 4K
- offset just moves the origin of the UI in the world. This probably can be omitted and directly applied to the matrix in the xaml
- scale adjusts the view dimensions of the UI to real world dimensions. So if you have a button 200x100 in your view, scales adjust the coordinates to the same space rest of 3D objects are using.

### Re: Placing a UI element in 3D with matrices

Thank you for your replies. The issues with the matrices has been resolved. :-)

### Re: Placing a UI element in 3D with matrices

Happy to hear!

### Who is online

Users browsing this forum: tongkuisu and 1 guest