View Issue Details

IDProjectCategoryView StatusLast Update
0002472NoesisGUIUnrealpublic2023-04-11 18:13
Reportertsilmer Assigned Tohcpizzi  
PriorityhighSeveritymajor 
Status resolvedResolutionfixed 
Product Version3.1.6 
Target Version3.1.7Fixed in Version3.2.0 
Summary0002472: Incompatibility with Unreal Engine CommonUI plugin
Description

Hi,

Epic has been rolling out a new UI system called CommonUI (https://docs.unrealengine.com/5.0/en-US/common-ui-plugin-for-advanced-user-interfaces-in-unreal-engine/) which is build on top of UMG. The plugin comes standard with UE5, but also works in 4.27. Epic currently uses it for Fortnite and it is currently being used by most of our games too.

There seems to be an incompatibility between the Noesis Plugin and CommonUI in input handling.
Currently after we install, enable and setup CommonUI in a project the "Accept" button in gamepads no longer works, although all the other buttons work just fine.

Steps To Reproduce

Reproduction steps for UE5 or UE4.27 (same thing happen on both), using the Noesis UE sample app (downloaded from https://www.noesisengine.com/developers/downloads.php) and the Buttons level:

  • Edit noesisgui-ue4.27-3.1.6-indie\Assets\Buttons\Buttons\MainWindow.xaml to add GamepadTriggers such as:
          <Button x:Name="StartButton" Grid.Row="1" Content="START" Margin="60,0,0,0" Command="{Binding StartCommand}" RenderTransformOrigin="0.5,0.5">
            <b:Interaction.Triggers>
              <b:EventTrigger EventName="MouseEnter">
                <noesis:SetFocusAction/>
              </b:EventTrigger>
              <b:EventTrigger EventName="GotFocus">
                <b:PlaySoundAction Source="AudioSlide.wav" Volume="0.2"/>
              </b:EventTrigger>

              <noesis:GamepadTrigger FiredOn="ButtonDown" Button="Accept">
                <b:PlaySoundAction Source="AudioClick.wav" Volume="0.3"/>
              </noesis:GamepadTrigger>

            <noesis:GamepadTrigger FiredOn="ButtonDown" Button="Cancel">
                <b:PlaySoundAction Source="AudioClick.wav" Volume="0.3"/>
              </noesis:GamepadTrigger>
            </b:Interaction.Triggers>

            <Button.RenderTransform>
              <TransformGroup>
                <ScaleTransform/>
                <SkewTransform/>
                <RotateTransform/>
                <TranslateTransform/>
              </TransformGroup>
            </Button.RenderTransform>
          </Button>
  • Install the Unreal Engine CommonUI plugin
  • Change the Viewport class to CommonGameViewportClient as instructed in https://docs.unrealengine.com/5.0/en-US/common-ui-quickstart-guide-for-unreal-engine/
  • Notice that the the Accept button never triggers, while Cancel triggers as expected (you can try other buttons and they all work).
  • Add a breakpoint inside the UNoesisInstance::NativeOnKeyDown method in NoesisGUI-UE4.27-3.1.6-Indie\Plugins\NoesisGUI\Source\NoesisRuntime\Private\NoesisInstance.cpp and also note that the breakpoint is never hit when pressing the Accept button.
PlatformWindows

Activities

rg_moliver

rg_moliver

2022-12-08 02:46

reporter   ~0008183

Wanted to check in and see if there was any update or ETA on this? thanks so much!

hcpizzi

hcpizzi

2022-12-12 12:46

developer   ~0008184

Hi,

I've been looking into this. Turns out CommonUI turns the accept gamepad button press into a MouseButtonDown event, in FAnalogCursor::HandleKeyDownEvent (via FCommonAnalogCursor::HandleKeyDownEvent).

It looks like the way to avoid this would be to have the ActionRouter handle the event, but I haven't figured out how to do it. Here's the relevant line in FCommonAnalogCursor::HandleKeyDownEvent:

if (bIsVirtualAccept && ActionRouter.ProcessInput(InKeyEvent.GetKey(), InputEventType) == ERouteUIInputResult::Handled)

sgonchar

sgonchar

2023-01-09 18:27

reporter   ~0008195

Hello, I'm wondering when can there be an ETA on this? Thank you.

hcpizzi

hcpizzi

2023-01-10 17:32

developer   ~0008198

Hi,

I'm currently looking into having the ActionRouter handle the input event so that it doesn't decay into a left mouse button down event, but I still haven't figured out how to do it.

sgonchar

sgonchar

2023-01-10 19:38

reporter   ~0008200

Thank you for looking into it! It's becoming higher and higher pri for us. Eager to see your solution for it.

hcpizzi

hcpizzi

2023-01-11 14:20

developer   ~0008202

OK, I've got a workaround for this.

The code in FCommonAnalogCursor::HandleKeyDownEvent that handles the Virtual_Accept key does this:

        //@todo DanH: This is a major bummer to have to flip this flag on the input subsystem here, but there is no awareness on a mouse event of whether it's real or not
        //      Though tbh, any place that cares should be able to just check the live input mode and infer from that whether this is a mouse click or virtual gamepad click
        UCommonInputSubsystem& InputSubsytem = ActionRouter.GetInputSubsystem();
        InputSubsytem.SetIsGamepadSimulatedClick(bIsVirtualAccept);
        bool bReturnValue = FAnalogCursor::HandleKeyDownEvent(SlateApp, InKeyEvent);
        InputSubsytem.SetIsGamepadSimulatedClick(false);

I can check that IsGamepadSimulatedClick flag in our MouseButtonDown event handler, and convert it back to a Key_GamepadAccept KeyDown/Up event pair.

It's not pretty, but it gets the job done. I can send you a patch if you want to try it first.

What do you think?

sgonchar

sgonchar

2023-01-11 20:27

reporter   ~0008205

Thank you! If it works in your samples it would work for us as well, I think.
Please send us a patch / change PR or etc we can try on our end when it's ready. I'll see what it takes to go through a release cycle to get this to the game team that is having the issue.
Cheers.

hcpizzi

hcpizzi

2023-01-13 13:22

developer   ~0008207

Here's the patch for you to try. Please, let us know if it resolves your issue. Thanks.

CommonUIWorkaround.patch (6,287 bytes)   
Index: NoesisGUI.uplugin
===================================================================
--- NoesisGUI.uplugin	(revision 11845)
+++ NoesisGUI.uplugin	(working copy)
@@ -28,5 +28,11 @@
 			"Type": "Editor",
 			"LoadingPhase": "Default"
 		}
+	],
+	"Plugins": [
+		{
+		  "Name": "CommonUI",
+		  "Enabled": false
+		}
 	]
 }
\ No newline at end of file
Index: Source/NoesisRuntime/Classes/NoesisInstance.h
===================================================================
--- Source/NoesisRuntime/Classes/NoesisInstance.h	(revision 11846)
+++ Source/NoesisRuntime/Classes/NoesisInstance.h	(working copy)
@@ -129,6 +129,7 @@
 	float CurrentTime;
 	FSceneInterface* Scene;
 	FGameTime WorldTime;
+	bool IsGamepadSimulatedClick = false;
 
 	typedef TSharedPtr<class FNoesisSlateElement, ESPMode::ThreadSafe> FNoesisSlateElementPtr;
 	FNoesisSlateElementPtr NoesisSlateElement;
Index: Source/NoesisRuntime/NoesisRuntime.Build.cs
===================================================================
--- Source/NoesisRuntime/NoesisRuntime.Build.cs	(revision 11846)
+++ Source/NoesisRuntime/NoesisRuntime.Build.cs	(working copy)
@@ -98,5 +98,32 @@
 		{
 			PrivateDefinitions.Add("WITH_ENHANCED_INPUT=0");
 		}
+		
+		bool WithCommonUI = false;
+		if (Target.ProjectFile != null)
+		{
+			var Project = ProjectDescriptor.FromFile(Target.ProjectFile);
+
+			if (Project.Plugins != null)
+			{
+				WithCommonUI = (Array.Find(Project.Plugins, p => { return p.Name == "CommonUI"; }) != null);
+			}
+		}
+
+		if (WithCommonUI)
+		{
+			System.Console.WriteLine("NoesisGUI: It looks like your project is using the CommonUI plugin. The warning below is harmless but, if you want to get rid of it, you can edit NoesisGUI.uplugin and change the CommonUI plugin dependency to be enabled.");
+			PublicDependencyModuleNames.AddRange(
+				new string[]
+				{
+					"CommonInput"
+				}
+			);
+			PublicDefinitions.Add("WITH_COMMON_UI=1");
+		}
+		else
+		{
+			PublicDefinitions.Add("WITH_COMMON_UI=0");
+		}
 	}
 }
Index: Source/NoesisRuntime/Private/NoesisInstance.cpp
===================================================================
--- Source/NoesisRuntime/Private/NoesisInstance.cpp	(revision 11846)
+++ Source/NoesisRuntime/Private/NoesisInstance.cpp	(working copy)
@@ -53,6 +53,11 @@
 #include "NoesisSupport.h"
 #include "Extensions/BackgroundImage.h"
 
+#if WITH_COMMON_UI
+// CommonInput includes
+#include "CommonInputSubsystem.h"
+#endif
+
 DECLARE_CYCLE_STAT(TEXT("Update"), STAT_NoesisInstance_Update, STATGROUP_Noesis);
 DECLARE_CYCLE_STAT(TEXT("UpdateRenderTree"), STAT_NoesisInstance_UpdateRenderTree, STATGROUP_Noesis);
 DECLARE_CYCLE_STAT(TEXT("RenderOffscreen"), STAT_NoesisInstance_DrawOffscreen, STATGROUP_Noesis);
@@ -1233,8 +1238,8 @@
 	KeyMap.Add(EKeys::Gamepad_LeftStick_Left, Noesis::Key_GamepadLeft);
 	KeyMap.Add(EKeys::Gamepad_DPad_Right, Noesis::Key_GamepadRight);
 	KeyMap.Add(EKeys::Gamepad_LeftStick_Right, Noesis::Key_GamepadRight);
-	KeyMap.Add(EKeys::Gamepad_FaceButton_Bottom, Noesis::Key_GamepadAccept);
-	KeyMap.Add(EKeys::Gamepad_FaceButton_Right, Noesis::Key_GamepadCancel);
+	KeyMap.Add(EKeys::Virtual_Accept, Noesis::Key_GamepadAccept);
+	KeyMap.Add(EKeys::Virtual_Back, Noesis::Key_GamepadCancel);
 	KeyMap.Add(EKeys::Gamepad_LeftShoulder, Noesis::Key_GamepadPageLeft);
 	KeyMap.Add(EKeys::Gamepad_RightShoulder, Noesis::Key_GamepadPageRight);
 	KeyMap.Add(EKeys::Gamepad_LeftTrigger, Noesis::Key_GamepadPageUp);
@@ -1319,16 +1324,35 @@
 	SCOPE_CYCLE_COUNTER(STAT_NoesisInstance_OnMouseButtonDown);
 	if (XamlView)
 	{
-		FVector2D Position = MyGeometry.AbsoluteToLocal(MouseEvent.GetScreenSpacePosition()) * MyGeometry.Scale;
-		bool hit = HitTest(Position);
+#if WITH_COMMON_UI
+		ULocalPlayer* LocalPlayer = GetOwningLocalPlayer();
+		if (LocalPlayer != nullptr)
+		{
+			if (UCommonInputSubsystem* CommonInputSubsystem = UCommonInputSubsystem::Get(GetOwningLocalPlayer()))
+			{
+				IsGamepadSimulatedClick = CommonInputSubsystem->GetIsGamepadSimulatedClick();
+			}
+		}
+#endif
 
-		Noesis::MouseButton MouseButton = GetNoesisMouseButton(MouseEvent.GetEffectingButton());
-		XamlView->MouseButtonDown(FPlatformMath::RoundToInt(Position.X), FPlatformMath::RoundToInt(Position.Y), MouseButton);
-
-		if (hit)
+		if (IsGamepadSimulatedClick)
 		{
-			return FReply::Handled().PreventThrottling();
+			XamlView->KeyDown(Noesis::Key_GamepadAccept);
+			return FReply::Handled().PreventThrottling().CaptureMouse(MyWidget.Pin().ToSharedRef());
 		}
+		else
+		{
+			FVector2D Position = MyGeometry.AbsoluteToLocal(MouseEvent.GetScreenSpacePosition()) * MyGeometry.Scale;
+			bool hit = HitTest(Position);
+
+			Noesis::MouseButton MouseButton = GetNoesisMouseButton(MouseEvent.GetEffectingButton());
+			XamlView->MouseButtonDown(FPlatformMath::RoundToInt(Position.X), FPlatformMath::RoundToInt(Position.Y), MouseButton);
+
+			if (hit)
+			{
+				return FReply::Handled().PreventThrottling();
+			}
+		}
 	}
 
 	return FReply::Unhandled();
@@ -1339,16 +1363,25 @@
 	SCOPE_CYCLE_COUNTER(STAT_NoesisInstance_OnMouseButtonUp);
 	if (XamlView)
 	{
-		FVector2D Position = MyGeometry.AbsoluteToLocal(MouseEvent.GetScreenSpacePosition()) * MyGeometry.Scale;
-		bool hit = HitTest(Position);
-
-		Noesis::MouseButton MouseButton = GetNoesisMouseButton(MouseEvent.GetEffectingButton());
-		XamlView->MouseButtonUp(FPlatformMath::RoundToInt(Position.X), FPlatformMath::RoundToInt(Position.Y), MouseButton);
-
-		if (hit)
+		if (IsGamepadSimulatedClick)
 		{
+			IsGamepadSimulatedClick = false;
+			XamlView->KeyUp(Noesis::Key_GamepadAccept);
 			return FReply::Handled().PreventThrottling();
 		}
+		else
+		{
+			FVector2D Position = MyGeometry.AbsoluteToLocal(MouseEvent.GetScreenSpacePosition()) * MyGeometry.Scale;
+			bool hit = HitTest(Position);
+
+			Noesis::MouseButton MouseButton = GetNoesisMouseButton(MouseEvent.GetEffectingButton());
+			XamlView->MouseButtonUp(FPlatformMath::RoundToInt(Position.X), FPlatformMath::RoundToInt(Position.Y), MouseButton);
+
+			if (hit)
+			{
+				return FReply::Handled().PreventThrottling();
+			}
+		}
 	}
 
 	return FReply::Unhandled();
CommonUIWorkaround.patch (6,287 bytes)   
sgonchar

sgonchar

2023-01-13 18:28

reporter   ~0008208

Thank you very much! We'll patch and try to push through release process shortly!

sgonchar

sgonchar

2023-04-06 03:06

reporter   ~0008408

Hello!
Update, please excuse late reply, release / integration process was much longer then we expected.
Fix confirmed, it fixed the issue in live / in game.

  • Input works when using common game view port
  • input still works when switching back to game view port (I didn't have to use the .cs portion)
    Cheers.

Issue History

Date Modified Username Field Change
2022-11-28 17:49 tsilmer New Issue
2022-11-28 18:03 jsantos Assigned To => hcpizzi
2022-11-28 18:03 jsantos Status new => assigned
2022-11-29 11:49 jsantos View Status private => public
2022-12-02 00:31 tsilmer Priority normal => high
2022-12-08 02:46 rg_moliver Note Added: 0008183
2022-12-12 12:46 hcpizzi Note Added: 0008184
2023-01-09 18:27 sgonchar Note Added: 0008195
2023-01-10 16:22 jsantos Target Version => 3.1.7
2023-01-10 17:32 hcpizzi Note Added: 0008198
2023-01-10 19:38 sgonchar Note Added: 0008200
2023-01-11 14:20 hcpizzi Note Added: 0008202
2023-01-11 14:30 jsantos Status assigned => feedback
2023-01-11 20:27 sgonchar Note Added: 0008205
2023-01-13 13:22 hcpizzi Note Added: 0008207
2023-01-13 13:22 hcpizzi File Added: CommonUIWorkaround.patch
2023-01-13 18:28 sgonchar Note Added: 0008208
2023-04-06 03:06 sgonchar Note Added: 0008408
2023-04-11 18:13 jsantos Status feedback => resolved
2023-04-11 18:13 jsantos Resolution open => fixed
2023-04-11 18:13 jsantos Fixed in Version => 3.2.0