View Issue Details

IDProjectCategoryView StatusLast Update
0002202NoesisGUIC++ SDKpublic2022-02-03 10:43
Reporterkrupitskas Assigned Tohcpizzi  
PrioritynormalSeverityblockReproducibilityalways
Status resolvedResolutionfixed 
Product Version3.1 
Target Version3.1.3Fixed in Version3.1.3 
Summary0002202: 3D UI interaction broken
DescriptionHello!
I'm trying to implement 3D ui elements, which I want to click and drag around. Imagine "image placer" in 2D world.
What I've did is created a canvas with custom name, obtained it through:

[code]
application_text_tool = Noesis::Ptr<AppTextTool>{Noesis::DynamicCast<AppTextTool*>(noesis_3d_view->GetContent())};
auto content_text_tool = Noesis::DynamicCast<Noesis::FrameworkElement*>(application_text_tool->GetContent());
world_space_canvas = Noesis::Ptr(content_text_tool->FindName<Noesis::Canvas>("TextToolCanvas"));
noesis_3d_view->SetSize(static_cast<uint16_t>(p.win_width), static_cast<uint16_t>(p.win_height));
[/code]

Later I render it:
[code]
    const auto viewport = Noesis::Matrix4::Viewport(window_w, window_h);
    const auto view = Noesis::Matrix4(glm::value_ptr(camera.get_view()));
    const auto proj = Noesis::Matrix4(glm::value_ptr(camera.get_proj()));
    Noesis::Matrix4 offset = Noesis::Transform3::Trans(0.0, static_cast<float>(-window_h), 0.0).ToMatrix4();

    const auto prod = offset * view * proj * viewport;

    noesis_3d_view->SetProjectionMatrix(prod);
    noesis_3d_view->GetRenderer()->RenderOffscreen(prod);
    noesis_3d_view->GetRenderer()->Render(prod);
[/code]

Element itself:

[code]
    <Canvas x:Name="TextToolCanvas">
       <Viewbox Stretch="Uniform" StretchDirection="Both">
            <Rectangle Opacity="0.8" Width="736" Height="951" Fill="Red">
                    <i:Interaction.Behaviors>
                        <ei:MouseDragElementBehavior ConstrainToParentBounds="False" />
                    </i:Interaction.Behaviors>
            </Rectangle>
        </Viewbox>

    </Canvas>
[/code]

So result looks like this: [url]http://www.giphy.com/gifs/HyE5flQi9qF40jmQ3t[/url]

But if I insert rectangle into 2D ui, everything works perfect. [url]http://www.giphy.com/gifs/JMkTe0Rms1CqDyoJi5[/url]\

Forum post:
https://www.noesisengine.com/forums/viewtopic.php?f=3&t=2482
Steps To ReproduceCreate two views, provide world space matrix to one of them, add rectangle with mouseinteraction behaviour, try to click and move.
TagsC++
PlatformAny

Relationships

has duplicate 0002257 resolvedsfernandez VIsual::PointFromScreen called from a button click - zero determinent, invalid matrix inverse 

Activities

sfernandez

sfernandez

2021-12-20 11:20

manager   ~0007687

Just to confirm, are you able to reproduce these issues with NoesisGUI 3.1.1 version?
jsantos

jsantos

2021-12-20 12:24

manager   ~0007688

The following XAML is working fine in XamlToy. Also we tried this same XAML in a local example changing the perspective projection of the camera (this cannot be done in xamltoy)

<Grid
  xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
  xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
  xmlns:ei="http://schemas.microsoft.com/expression/2010/interactions"
  xmlns:i="http://schemas.microsoft.com/expression/2010/interactivity"
  xmlns:noesis="clr-namespace:NoesisGUIExtensions;assembly=Noesis.GUI.Extensions">
  
  <noesis:Element.Transform3D>
    <noesis:CompositeTransform3D RotationY="50" />
  </noesis:Element.Transform3D>
  
    <Canvas x:Name="TextToolCanvas">
       <Viewbox Stretch="Uniform" StretchDirection="Both" Canvas.Left="200" Canvas.Top="200">
            <Rectangle Opacity="0.8" Width="100" Height="200" Fill="Red">
                    <i:Interaction.Behaviors>
                        <ei:MouseDragElementBehavior ConstrainToParentBounds="False" />
                    </i:Interaction.Behaviors>
            </Rectangle>
        </Viewbox>

    </Canvas>
  
</Grid>
krupitskas

krupitskas

2021-12-27 19:26

reporter   ~0007692

Hello. I've reproduced this issue with glut example. Attaching file Main.cpp
+ Also try to increase or decrease size of the window in `glutInitWindowSize(1000, 600);`
For some reason clicking become broken.

P.S. const uint8_t AppGlut_xaml[] is

```
<Grid
  xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
  xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
  xmlns:ei="http://schemas.microsoft.com/expression/2010/interactions"
  xmlns:i="http://schemas.microsoft.com/expression/2010/interactivity"
  xmlns:noesis="clr-namespace:NoesisGUIExtensions;assembly=Noesis.GUI.Extensions">
  
    <Canvas x:Name="TextToolCanvas">
       <Viewbox Stretch="Uniform" StretchDirection="Both" Canvas.Left="200" Canvas.Top="200">
            <Rectangle Opacity="0.8" Width="1000" Height="2000" Fill="Red">
                    <i:Interaction.Behaviors>
                        <ei:MouseDragElementBehavior ConstrainToParentBounds="False" />
                    </i:Interaction.Behaviors>
            </Rectangle>
        </Viewbox>

    </Canvas>
  
</Grid>
```
Main.cpp (11,012 bytes)   
////////////////////////////////////////////////////////////////////////////////////////////////////
// NoesisGUI - http://www.noesisengine.com
// Copyright (c) 2013 Noesis Technologies S.L. All Rights Reserved.
////////////////////////////////////////////////////////////////////////////////////////////////////


// This is a minimal integration example. For simplification purposes only basic input events are
// handled, no resource providers are used and the shutdown procedure is omitted. A more complete
// multiplatform integration sample with step by step comments can be found at 'Samples/Integration'


#ifdef _WIN32
#include "glut.h"
#pragma comment(linker,"/SUBSYSTEM:CONSOLE")
#endif

#ifdef __APPLE__
#include <GLUT/glut.h>
#endif

#ifdef __EMSCRIPTEN__
#include <GL/glut.h>
#include <GLES3/gl32.h>
#include <emscripten/html5.h>
#endif

#ifdef __linux__
#define GL_GLEXT_PROTOTYPES
#include <GL/gl.h>
#include <GL/glut.h>
#endif

#ifdef _MSC_VER
#define UNUSED_ARGS(...) (void)(true ? (void)0 : ((void)(__VA_ARGS__)))
#else
#define UNUSED_ARGS(...)
#endif


#include <NsApp/ThemeProviders.h>
#include <NsRender/GLFactory.h>
#include <NsGui/FontProperties.h>
#include <NsGui/IntegrationAPI.h>
#include <NsGui/IRenderer.h>
#include <NsGui/IView.h>
#include <NsGui/Grid.h>
#include <NsApp/EmbeddedXamlProvider.h>

const uint8_t AppGlut_xaml[] =
{
    60,71,114,105,100,13,10,32,32,120,109,108,110,115,61,34,104,116,116,112,58,47,47,115,99,104,101,
    109,97,115,46,109,105,99,114,111,115,111,102,116,46,99,111,109,47,119,105,110,102,120,47,50,48,48,
    54,47,120,97,109,108,47,112,114,101,115,101,110,116,97,116,105,111,110,34,13,10,32,32,120,109,108,
    110,115,58,120,61,34,104,116,116,112,58,47,47,115,99,104,101,109,97,115,46,109,105,99,114,111,115,
    111,102,116,46,99,111,109,47,119,105,110,102,120,47,50,48,48,54,47,120,97,109,108,34,13,10,32,32,
    120,109,108,110,115,58,101,105,61,34,104,116,116,112,58,47,47,115,99,104,101,109,97,115,46,109,105,
    99,114,111,115,111,102,116,46,99,111,109,47,101,120,112,114,101,115,115,105,111,110,47,50,48,49,48,
    47,105,110,116,101,114,97,99,116,105,111,110,115,34,13,10,32,32,120,109,108,110,115,58,105,61,34,
    104,116,116,112,58,47,47,115,99,104,101,109,97,115,46,109,105,99,114,111,115,111,102,116,46,99,111,
    109,47,101,120,112,114,101,115,115,105,111,110,47,50,48,49,48,47,105,110,116,101,114,97,99,116,105,
    118,105,116,121,34,13,10,32,32,120,109,108,110,115,58,110,111,101,115,105,115,61,34,99,108,114,45,
    110,97,109,101,115,112,97,99,101,58,78,111,101,115,105,115,71,85,73,69,120,116,101,110,115,105,111,
    110,115,59,97,115,115,101,109,98,108,121,61,78,111,101,115,105,115,46,71,85,73,46,69,120,116,101,
    110,115,105,111,110,115,34,62,13,10,32,32,13,10,32,32,32,32,60,67,97,110,118,97,115,32,120,58,78,
    97,109,101,61,34,84,101,120,116,84,111,111,108,67,97,110,118,97,115,34,62,13,10,32,32,32,32,32,32,
    32,60,86,105,101,119,98,111,120,32,83,116,114,101,116,99,104,61,34,85,110,105,102,111,114,109,34,
    32,83,116,114,101,116,99,104,68,105,114,101,99,116,105,111,110,61,34,66,111,116,104,34,32,67,97,
    110,118,97,115,46,76,101,102,116,61,34,50,48,48,34,32,67,97,110,118,97,115,46,84,111,112,61,34,50,
    48,48,34,62,13,10,32,32,32,32,32,32,32,32,32,32,32,32,60,82,101,99,116,97,110,103,108,101,32,79,
    112,97,99,105,116,121,61,34,48,46,56,34,32,87,105,100,116,104,61,34,49,48,48,48,34,32,72,101,105,
    103,104,116,61,34,50,48,48,48,34,32,70,105,108,108,61,34,82,101,100,34,62,13,10,32,32,32,32,32,32,
    32,32,32,32,32,32,32,32,32,32,32,32,32,32,60,105,58,73,110,116,101,114,97,99,116,105,111,110,46,66,
    101,104,97,118,105,111,114,115,62,13,10,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,
    32,32,32,32,32,60,101,105,58,77,111,117,115,101,68,114,97,103,69,108,101,109,101,110,116,66,101,
    104,97,118,105,111,114,32,67,111,110,115,116,114,97,105,110,84,111,80,97,114,101,110,116,66,111,
    117,110,100,115,61,34,70,97,108,115,101,34,32,47,62,13,10,32,32,32,32,32,32,32,32,32,32,32,32,32,
    32,32,32,32,32,32,32,60,47,105,58,73,110,116,101,114,97,99,116,105,111,110,46,66,101,104,97,118,
    105,111,114,115,62,13,10,32,32,32,32,32,32,32,32,32,32,32,32,60,47,82,101,99,116,97,110,103,108,
    101,62,13,10,32,32,32,32,32,32,32,32,60,47,86,105,101,119,98,111,120,62,13,10,13,10,32,32,32,32,60,
    47,67,97,110,118,97,115,62,13,10,32,32,13,10,60,47,71,114,105,100,62
};




#include <NsApp/Launcher.h>

static Noesis::IView* _view;

///////////////////////////////////////////////////////////////////////////////////////////////////
static void NoesisInit()
{
    // A logging handler is installed here. You can also install a custom error handler and memory
    // allocator. By default errors are redirected to the logging handler
    Noesis::GUI::SetLogHandler([](const char*, uint32_t, uint32_t level, const char*, const char* msg)
    {
        // [TRACE] [DEBUG] [INFO] [WARNING] [ERROR]
        const char* prefixes[] = { "T", "D", "I", "W", "E" };
        printf("[NOESIS/%s] %s\n", prefixes[level], msg);
    });

    NoesisApp::Launcher::RegisterAppComponents();

    // https://www.noesisengine.com/docs/Gui.Core.Licensing.html
    Noesis::GUI::SetLicense(NS_LICENSE_NAME, NS_LICENSE_KEY);

    // Noesis initialization. This must be the first step before using any NoesisGUI functionality
    Noesis::GUI::Init();

    NoesisApp::EmbeddedXaml xamls[] = { { "AppGlut.xaml", AppGlut_xaml } };
    Noesis::Ptr<Noesis::XamlProvider> xamlProvider = *new NoesisApp::EmbeddedXamlProvider(xamls);

    // Setup theme
    NoesisApp::SetThemeProviders(xamlProvider);
    Noesis::GUI::LoadApplicationResources("Theme/NoesisTheme.DarkBlue.xaml");

                    
    // <Button Content="Hello World!" Margin="0,30,0,0"/>

    // For simplicity purposes we are not using resource providers in this sample. ParseXaml() is
    // enough if there is no extra XAML dependencies
    Noesis::Ptr<Noesis::Grid> xaml = Noesis::GUI::LoadXaml<Noesis::Grid>("AppGlut.xaml");
    
    /*(Noesis::GUI::ParseXaml<Noesis::Grid>(R"(
            <Grid
              xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
              xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
              xmlns:ei="http://schemas.microsoft.com/expression/2010/interactions"
              xmlns:i="http://schemas.microsoft.com/expression/2010/interactivity"
              xmlns:noesis="clr-namespace:NoesisGUIExtensions;assembly=Noesis.GUI.Extensions">
  
              <noesis:Element.Transform3D>
                <noesis:CompositeTransform3D RotationY="50" />
              </noesis:Element.Transform3D>
  
                <Canvas x:Name="TextToolCanvas">
                   <Viewbox Stretch="Uniform" StretchDirection="Both" Canvas.Left="200" Canvas.Top="200">
                        <Rectangle Opacity="0.8" Width="100" Height="200" Fill="Red">
                                <i:Interaction.Behaviors>
                                    <ei:MouseDragElementBehavior ConstrainToParentBounds="False" />
                                </i:Interaction.Behaviors>
                        </Rectangle>
                    </Viewbox>

                </Canvas>
  
            </Grid>
    )"));*/

    // View creation to render and interact with the user interface
    // We transfer the ownership to a global pointer instead of a Ptr<> because there is no way
    // in GLUT to do shutdown and we don't want the Ptr<> to be released at global time
    _view = Noesis::GUI::CreateView(xaml).GiveOwnership();
    _view->SetFlags(Noesis::RenderFlags_PPAA | Noesis::RenderFlags_LCD);

    // Renderer initialization with an OpenGL device
    _view->GetRenderer()->Init(NoesisApp::GLFactory::CreateDevice(false));
}

///////////////////////////////////////////////////////////////////////////////////////////////////
static void DisplayFunc(void)
{
    const auto perspective = Noesis::Matrix4::PerspectiveFov(90.0f, float(glutGet(GLUT_WINDOW_WIDTH)) / float(glutGet(GLUT_WINDOW_HEIGHT)), 0.01f);
    const auto viewport = Noesis::Matrix4::Viewport(float(glutGet(GLUT_WINDOW_WIDTH)), float(glutGet(GLUT_WINDOW_HEIGHT)));
    const auto offset = Noesis::Transform3::Trans(0.0f, 0.0f, 10000.0f).ToMatrix4();

    const auto prod = offset * perspective * viewport;

    _view->SetProjectionMatrix(prod);

    // Update view (layout, animations, ...)
    _view->Update(glutGet(GLUT_ELAPSED_TIME) / 1000.0);

    // Offscreen rendering phase populates textures needed by the on-screen rendering
    _view->GetRenderer()->UpdateRenderTree();
    _view->GetRenderer()->RenderOffscreen();

    // If you are going to render here with your own engine you need to restore the GPU state
    // because noesis changes it. In this case only framebuffer and viewport need to be restored
    glBindFramebuffer(GL_FRAMEBUFFER, 0);
    glViewport(0, 0, glutGet(GLUT_WINDOW_WIDTH), glutGet(GLUT_WINDOW_HEIGHT));

    glClearColor(0.0f, 0.0f, 0.25f, 0.0f);
    glClearStencil(0);
    glClear(GL_COLOR_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);

    // Rendering is done in the active framebuffer
    _view->GetRenderer()->Render();

    glutSwapBuffers();
    glutPostRedisplay();
}

////////////////////////////////////////////////////////////////////////////////////////////////////
static void ReshapeFunc(int width, int height)
{
    _view->SetSize(width, height);
}

////////////////////////////////////////////////////////////////////////////////////////////////////
static void MouseMoveFunc(int x, int y)
{
    _view->MouseMove(x, y);
}

////////////////////////////////////////////////////////////////////////////////////////////////////
static void MouseFunc(int button, int state, int x, int y)
{
    if (button == GLUT_LEFT_BUTTON)
    {
        if (state == GLUT_DOWN)
        {
            _view->MouseButtonDown(x, y, Noesis::MouseButton_Left);
        }
        else
        {
            _view->MouseButtonUp(x, y, Noesis::MouseButton_Left);
        }
    }
}

////////////////////////////////////////////////////////////////////////////////////////////////////
int main(int argc, char **argv)
{
    glutInit(&argc, argv);
    glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE | GLUT_STENCIL);
    glutInitWindowSize(1000, 600);

#ifdef __EMSCRIPTEN__
    double width, height;
    emscripten_get_element_css_size("#canvas", &width, &height);
    emscripten_set_canvas_element_size("#canvas", (uint32_t)width, (uint32_t)height);
    glutInitWindowSize((uint32_t)width, (uint32_t)height);
#endif

    glutCreateWindow("NoesisGUI - GLUT integration");
    NoesisInit();

    glutDisplayFunc(&DisplayFunc);
    glutReshapeFunc(&ReshapeFunc);
    glutPassiveMotionFunc(&MouseMoveFunc);
    glutMouseFunc(&MouseFunc);

    glutMainLoop();
    return 0;
}
Main.cpp (11,012 bytes)   
krupitskas

krupitskas

2022-01-11 17:22

reporter   ~0007726

Hey guys, any update? It's a blocker for us for the next release unfortunately :(
sfernandez

sfernandez

2022-01-12 10:43

manager   ~0007727

Hello, sorry for the delay, we've been on holidays the past weeks.
We are now working on the issue, related to float precision on the world-screen operations when the z range is big.

I guess the projection matrix you are setting is the same you have in your 3D scene, right?
hcpizzi

hcpizzi

2022-01-12 17:43

developer   ~0007730

Could you paste the values from view matrix taken from your application? I want to make sure the fix doesn't just work with the kind of matrices that we use when none is set.

Thanks.
krupitskas

krupitskas

2022-01-14 10:54

reporter   ~0007737

GLM projection matrix: mat4x4((1.014229, 0.000000, 0.000000, 0.000000), (0.000000, 1.732051, 0.000000, 0.000000), (-0.000000, -0.000000, 1.000000, 1.000000), (0.000000, 0.000000, -0.010000, 0.000000))

GLM view matrix: mat4x4((1.000000, 0.000000, 0.000000, 0.000000), (0.000000, 1.000000, 0.000000, 0.000000), (0.000000, 0.000000, 1.000000, 0.000000), (-0.000000, -0.000000, 16525.679688, 1.000000))

Yes, I use multiplication of these two matrices for my props and for a Noesis UI 3d view.
hcpizzi

hcpizzi

2022-01-28 16:05

developer   ~0007757

Hi,

Sorry this is taking so long. We think we have a robust solution now, and would like to build a library with the fix for you to test.

Which version of the Noesis SDK are you using, and what platform would you like the library for?

Thank you for your patience.

Issue History

Date Modified Username Field Change
2021-11-30 08:02 krupitskas New Issue
2021-11-30 08:02 krupitskas Tag Attached: C++
2021-11-30 11:38 jsantos Target Version => 3.1.2
2021-11-30 11:38 jsantos Assigned To => sfernandez
2021-11-30 11:38 jsantos Status new => assigned
2021-12-20 11:20 sfernandez Status assigned => feedback
2021-12-20 11:20 sfernandez Note Added: 0007687
2021-12-20 12:24 jsantos Note Added: 0007688
2021-12-21 21:34 jsantos Target Version 3.1.2 => 3.1.3
2021-12-27 19:26 krupitskas File Added: Main.cpp
2021-12-27 19:26 krupitskas Note Added: 0007692
2021-12-27 19:26 krupitskas Status feedback => assigned
2022-01-11 17:22 krupitskas Note Added: 0007726
2022-01-12 10:43 sfernandez Status assigned => feedback
2022-01-12 10:43 sfernandez Note Added: 0007727
2022-01-12 10:43 sfernandez Assigned To sfernandez => hcpizzi
2022-01-12 17:43 hcpizzi Note Added: 0007730
2022-01-14 10:54 krupitskas Note Added: 0007737
2022-01-14 10:54 krupitskas Status feedback => assigned
2022-01-28 16:05 hcpizzi Status assigned => feedback
2022-01-28 16:05 hcpizzi Note Added: 0007757
2022-01-31 16:32 hcpizzi Status feedback => resolved
2022-01-31 16:32 hcpizzi Resolution open => fixed
2022-01-31 16:32 hcpizzi Fixed in Version => 3.1.3
2022-02-03 10:43 sfernandez Relationship added has duplicate 0002257