Unity3d and gaze based input support

aroszykS · 10 Nov 2025, 21:38

Hello,
I want to create a custom interaction with noesis GUI in Unity- so that the ui will react not to the pointer/cursor events, but rather to a reticle(centred on the screen) + timed gaze system(Look at element to hover + wait 3 seconds to click on it).
I cannot lock the cursor in the screen centre and instead I'd like to use Unity ray casting for implementing this feature.

Such system worked well for me with with UGUI:

ExecuteEvents.ExecuteHierarchy(gameObject, eventData, ExecuteEvents.pointerUpHandler/pointerClickHandler);

Is something like that feasible with Noesis UGUI?
I haven't seen anything like this in your samples, do you have any tips on how this could be achieved?
Thanks!

11 Nov 2025, 19:55

I recommend reviewing NoesisView::UpdateActions. In that function, we manually send rays using VisualTreeHelper.HitTest3D and then emulate mouse events using the returned hit information.

private void UpdateActions(float t)
{
    int x = int.MaxValue;
    int y = int.MaxValue;
    bool trackedPos = false;

    if (_trackedPositionAction?.activeControl != null && _trackedRotationAction?.activeControl != null)
    {
        Vector3 pos_ = _trackedPositionAction.ReadValue<Vector3>();
        Vector3 dir_ = _trackedRotationAction.ReadValue<Quaternion>() * Vector3.forward;

        if (_xrTrackingOrigin != null)
        {
            pos_ = _xrTrackingOrigin.TransformPoint(pos_);
            dir_ = _xrTrackingOrigin.TransformVector(dir_);
        }

        Point3D pos = new Point3D(pos_.x, pos_.y, pos_.z);
        Vector3D dir = new Vector3D(dir_.x, dir_.y, dir_.z);

        var captured = Content.Mouse.Captured;
        if (captured != null)
        {
            // Visuals capturing the mouse must always utilize its 3D plane for hit testing
            // https://www.noesisengine.com/bugs/view.php?id=2837
            if (VisualTreeHelper.IntersectPlane(captured, pos, dir, out Point3D worldPos))
            {
                Noesis.Matrix4 mtx = CameraMatrix(_camera);
                Noesis.Vector4 cameraPos = new Noesis.Vector4(worldPos.X, worldPos.Y, worldPos.Z, 1.0f) * mtx;

                x = (int)(cameraPos.X / cameraPos.W);
                y = (int)(cameraPos.Y / cameraPos.W);

                trackedPos = true;
            }
        }
        else
        {
            Visual root = (Visual)VisualTreeHelper.GetRoot(_uiView.Content);
            var hit = VisualTreeHelper.HitTest3D(root, pos, dir);

            if (hit.VisualHit != null)
            {
                Noesis.Matrix4 mtx = CameraMatrix(_camera);
                Noesis.Vector4 cameraPos = new Noesis.Vector4(hit.WorldPos.X, hit.WorldPos.Y, hit.WorldPos.Z, 1.0f) * mtx;

                x = (int)(cameraPos.X / cameraPos.W);
                y = (int)(cameraPos.Y / cameraPos.W);

                trackedPos = true;
            }
        }

        MouseMove(x, y);
    }

    if (_trackedTriggerAction != null)
    {
        if (_trackedTriggerAction.WasPressedThisFrame())
        {
            MouseButtonDown(x, y, Noesis.MouseButton.Left);
        }

        if (_trackedTriggerAction.WasReleasedThisFrame())
        {
            MouseButtonUp(x, y, Noesis.MouseButton.Left);
        }
    }

Unity3d and gaze based input support

Unity3d and gaze based input support

Re: Unity3d and gaze based input support

Who is online