User avatar
krupitskas
Topic Author
Posts: 23
Joined: 16 Apr 2021, 17:42

3D text with perspective projection

04 Aug 2021, 11:34

Hello!
I'm working on 3D text labels, which should be affected by camera view and projection matrices. I'm creating text dynamically in C++ backend and add as a child to a grid (it's kinda a viewport).
The problem is that this text batches with other SDF texts, which are drawn also in UI, and it's only attached ortho matrix.
So it's multiple problems here
1) While experimenting with XAML (but I should use C++, because of dynamic text creation), I've added TextBlock to a Grid (Viewport), which is part of bigger Grid (Whole screen)
        <TextBlock Text="This should be 3d transformed text">
            <noesis:Element.Transform3D>
                <noesis:CompositeTransform3D TranslateX = "50" TranslateY = "25" RotationY="-40"/>
            </noesis:Element.Transform3D>
        </TextBlock>
 
But instead it translated whole UI and rotation doesn't work
2) How to provide model and view matrix, and even custom projection matrix? While model and view can be combined into one, should I use ModelTransform3D for this? And for custom projection?
3) While setting CompositeTransform3D from C++ like

    _temp_text_block = Noesis::MakePtr<Noesis::TextBlock>();
    grid_text_tool->GetChildren()->Add(_temp_text_block);

    auto transform3d = Noesis::MakePtr<Noesis::CompositeTransform3D>();

    transform3d->SetRotationX(45);
    transform3d->SetCenterX(40);
    transform3d->SetTranslateX(50);

    _temp_text_block->SetText("Dynamic text inserted");
    _temp_text_block->SetTransform3D(transform3d);
Nothing happens, text beign rendered in the top left.
 
User avatar
sfernandez
Site Admin
Posts: 2984
Joined: 22 Dec 2011, 19:20

Re: 3D text with perspective projection

06 Aug 2021, 20:29

Maybe I'm not understanding something, but just applying a CompositeTransform3D to a TextBlock element should be enough, and should only affect that text, not the whole UI. See the following xamltoy:



If I load that xaml in our XamlPlayer I can see that each text is being drawn in a separate batch (Ctrl+B).

The code you posted for dynamically creating texts is correct.
 
User avatar
krupitskas
Topic Author
Posts: 23
Joined: 16 Apr 2021, 17:42

Re: 3D text with perspective projection

08 Aug 2021, 23:58

Thanks for reply and for example! It didn't worked either, so I figured out that this was a bug in our noesis integration (bgfx batched "view concept" and matrices was overrited). However, I have more questions.
I was able to provide custom perspective projection and view matrix, but seems like "Noesis DrawBatch(batch)" has baked model matrix into batch.mtxProj?
Why I think so:
I'm providing my view*proj matrices right in the rendering code and I have now "world" space positioned UI, which is also moving with the camera, etc, which is great,
but 3D text, which I'ce created like in the code below, Is being attached to 0,0 origin and renders without transformations.
    auto grid_text_tool = content->FindName<Noesis::Grid>("TextToolGrid");

    auto mtx_transform_3d = Noesis::MakePtr<Noesis::MatrixTransform3D>();

    _text_block_transform = Noesis::Transform3::Identity();
    _text_block_transform *= Noesis::Transform3::Scale(6, 3, 0);
    _text_block_transform *= Noesis::Transform3::RotX(1);
    _text_block_transform *= Noesis::Transform3::Trans(20, -20, 0);
    mtx_transform_3d->SetMatrix(_text_block_transform);

    _temp_text_block = Noesis::MakePtr<Noesis::TextBlock>();
    grid_text_tool->GetChildren()->Add(_temp_text_block);

    _temp_text_block->SetText("Dynamic text inserted");
    _temp_text_block->SetTransform3D(mtx_transform_3d);
I've also tried to do a view->SetProjectionMatrix(view * proj) but UI dissapears (for sure just broken matrix)

I'm using glm as a math library, it's column-major. Is Noesis row-major, right?

So questions are:
1) How to get just the model matrix from the batch?
2) Is it baked into batch.mtxProj?
3) Is it normal to create 2 views, where one is 2D simple UI and second is view for 3D stuff?
4) Noesis row major matrices?
 
User avatar
jsantos
Site Admin
Posts: 3906
Joined: 20 Jan 2012, 17:18
Contact:

Re: 3D text with perspective projection

09 Aug 2021, 16:17

The matrix in the batch is sent to the shader in column_major order. The first thing I would do is making sure your integration is correct, just take for example, the XAML that sfernandez gave you and open it with XamlPlayer to capture a frame with RenderDoc. Just do the same with your own integration and compare steps in RenderDoc, your shaders should be the same we are using in our reference implementations.

Once you have that working, you can continue making modifications to the projection matrix of the view using view->SetProjectionMatrix. Please, have a look at the following tutorial.

So, answering your questions:

1. It is not possible without dirty tricks we don't recommend as they may break in the future.
2. Yes, but not always.
3. We recommend using the same view, but yes, you could use two view for this.
4. column_major
 
User avatar
krupitskas
Topic Author
Posts: 23
Joined: 16 Apr 2021, 17:42

Re: 3D text with perspective projection

09 Aug 2021, 18:06

Thanks for the tutorial link. Honestly, it's a bit sad that this requires Oculus SDK && Oculus Device to run it, so no way for me to run/inspect/debug.
I've expected something simple like a cube moving with a text label on top in a world (or kinda a health bar)
Still more couple of questions:
In the documentation: "same projection matrix must be used for the UI and for the 3-D world." It's not working out of the box for me.

What I mean is:
If I replace view_proj variable directly in RenderDevice code, when drawing a batch, everything works fine (however Ioose all model transformations).
But If I call
```
_view_3d_text->GetRenderer()->RenderOffscreen(view_proj);
_view_3d_text->GetRenderer()->Render(view_proj);
```
nothing even draws in the frame capture (I feel like it's frustrum culling doing it's job).
Also from sample, it has viewport matrix multiplication, some magic scale and translate. Im not sure why this 3 matrices are there, when I need to use "same projection matrix".
I've did the same and sometimes, broken version of 3d text randomly appears.
    Noesis::Matrix4 viewport = Noesis::Matrix4::Viewport(window_w, window_h);
    Noesis::Matrix4 offset = Noesis::Transform3::Trans(-1.4f, -0.2f, -1.25f).ToMatrix4();
    Noesis::Matrix4 scale = Noesis::Transform3::Scale(0.002f, 0.002f, -0.002f).ToMatrix4();
    Noesis::Matrix4 view = Noesis::Matrix4(glm::value_ptr(camera.get_view()));
    Noesis::Matrix4 proj = Noesis::Matrix4(glm::value_ptr(camera.get_proj()));

    Noesis::Matrix4 prod = scale * offset * view * proj * viewport;
    
     _view_3d_text->GetRenderer()->RenderOffscreen(prod);
    _view_3d_text->GetRenderer()->Render(prod);
Any thoughts?
 
User avatar
krupitskas
Topic Author
Posts: 23
Joined: 16 Apr 2021, 17:42

Re: 3D text with perspective projection

09 Aug 2021, 19:27

Just for a simplicity, I'm trying to put a text on a plane in 3d space, where plane is 6 vertices glm::vec3[...] = {..., {0.5, 0.5, 0.0} ... } and 4 indices and camera with a simple view and projection like glm::look_at and glm::perspective.
I can see that positions or Noesis UI input vertices already pre transformed, so I feel that it's not just a perspective projection should be provide? Viewport matrix too?
 
User avatar
jsantos
Site Admin
Posts: 3906
Joined: 20 Jan 2012, 17:18
Contact:

Re: 3D text with perspective projection

10 Aug 2021, 12:58

Thanks for the tutorial link. Honestly, it's a bit sad that this requires Oculus SDK && Oculus Device to run it, so no way for me to run/inspect/debug.
Yes, sorry about this, as we don't have yet a 3D engine in the application framework we can't provide a better solution but we are working on improving this. Thanks for the feedback.

The wording of the documentation is not very accurate when it says "the same projection matrix must be used for the UI and for the 3-D world", it depends on how you read it, because you must use indeed the same projection but the matrix being sent to SetProjectionMatrix transforms to a space -width,+witdh -height,+height while a "normal" projection matrix transforms to normalized 0...1 space, so you basically needs to scale by the viewport matrix.

So you need to send:
custom_scale * view_camera * projection_camera * viewport
custom_scale gives a custom size to the UI in the world.
 
User avatar
krupitskas
Topic Author
Posts: 23
Joined: 16 Apr 2021, 17:42

Re: 3D text with perspective projection

12 Aug 2021, 16:45

Thanks for the help, finally I have everything working.
Two problems I've faced
1) My camera projection was configured incorrectly for UI, zNear was 10 and zFar 15000. Set zNear to 0.1 showed UI instantly.
2) SetProjection(matrix) actually cause slight delay in the update, and when camera moves, feels like UI have kinda inertion. When set matrix directly for render, there no such problem.

It would be really cool if you can note in the docs that for simple perspective matrix you also need viewport matrix.

Thanks!
 
User avatar
jsantos
Site Admin
Posts: 3906
Joined: 20 Jan 2012, 17:18
Contact:

Re: 3D text with perspective projection

12 Aug 2021, 17:50

Glad to know your found a solution!
2) SetProjection(matrix) actually cause slight delay in the update, and when camera moves, feels like UI have kinda inertion. When set matrix directly for render, there no such problem.
That's unexpected, were you calling SetProjection before updating the view? Are you rendering in a different thread?
It would be really cool if you can note in the docs that for simple perspective matrix you also need viewport matrix.
The documentation right now is not very clear about this:
/// Sets the projection matrix that transforms the coordinates of each primitive to a
/// non-normalized homogeneous space where (0, 0) is at the lower-left corner and
/// (width, height) is at the upper-right corner after projection
virtual void SetProjectionMatrix(const Matrix4& projection) = 0;
We will improve it. Thanks for the feedback.

Who is online

Users browsing this forum: Google [Bot] and 12 guests