Handling Input in NoesisGUI for Unreal Engine (Potential bug with NoesisInstance key mappings?)

KCoppinsIventis · 22 Mar 2024, 12:28

Hi There,

I am currently working on supporting input on my in game menus in Unreal Engine. I have a main menu that can be opened pressing ESC or Start button on a controller and that is handled by Noesis using the xaml binding below:

<noesis:InputActionTrigger Action="Pause" Type="Pressed" Consume="False">
    <b:InvokeCommandAction Command="{Binding ToggleLocationSelector}"/>
</noesis:InputActionTrigger>

When the menu is opened the following code is ran to switch the input mode to UI only, this is so movement input isn't propagated to the controller causing the possessed pawn to move whilst a menu is open:

// Where location selector is my ViewModel object - I set it to null if I am closing the menu
if (LocationSelector)
{
    LocationSelector->Initialise(UserController);

    if (NoesisInstance)
    {
        auto InputMode = FInputModeUIOnly();
        InputMode.SetWidgetToFocus(NoesisInstance->GetCachedWidget());
        UserController->SetInputMode(InputMode);
    }
}

else 
{
    if (NoesisInstance)
    {
        auto InputMode = FInputModeGameAndUI();
        InputMode.SetWidgetToFocus(NoesisInstance->GetCachedWidget());
        UserController->SetInputMode(InputMode);
    }
}

The issue I am having is that it appears that the widget is not responsible for its user inputs. When I switch to UI only and set the instance to be the focused widget, all input is suppressed, meaning I cannot close the menu once opened and I cannot navigate the menu using directional inputs using a controller. This gives me the impression that Noesis gets its UI input from the player controller? Is there a way to change this so that it handles its own input? I could in theory have a state bool that stores if the user is currently on the menu, but I'd rather use the input mode feature built into unreal

KCoppinsIventis · 23 Mar 2024, 12:23

I have done some more investigating into this issue and it appears to be part of the Noesis design. When a NoesisInstance initialises it uses the UserWidgets input component like so:

// NoesisInstance.cpp:Line562
auto ThisInputComponent = GetInputComponent();

The implementation of the GetInputComponent comes from the UserWidget's InitialiseInputComponent:

void UUserWidget::InitializeInputComponent()
{
	if ( APlayerController* Controller = GetOwningPlayer() )
	{
		// Use the existing PC's input class, or fallback to the project default. We should use the existing class
		// instead of just the default one because if you have a plugin that has a PC with a different default input
		// class then this would fail
		UClass* InputClass = Controller->InputComponent ? Controller->InputComponent->GetClass() : UInputSettings::GetDefaultInputComponentClass();
		InputComponent = NewObject< UInputComponent >( this, InputClass, NAME_None, RF_Transient );
PRAGMA_DISABLE_DEPRECATION_WARNINGS
		InputComponent->bBlockInput = bStopAction;
		InputComponent->Priority = Priority;
PRAGMA_ENABLE_DEPRECATION_WARNINGS
		Controller->PushInputComponent( InputComponent );
	}
	else
	{
		FMessageLog("PIE").Info(FText::Format(LOCTEXT("NoInputListeningWithoutPlayerController", "Unable to listen to input actions without a player controller in {0}."), FText::FromName(GetClass()->GetFName())));
	}
}

This uses the player controller's input component. When setting Input Mode to UI Only, all input is blocked to the player controller. However, if you have the widget focused, its OnKeyDown event is still called, which in turn calls Noesis's NativeOnKeyDown function:

FReply UNoesisInstance::NativeOnKeyDown(const FGeometry& MyGeometry, const FKeyEvent& KeyEvent)
{
	SupportsKeyboardFocus = !EnableActions;

	SCOPE_CYCLE_COUNTER(STAT_NoesisInstance_OnKeyDown);
	if ((EnableKeyboard || GNoesisIsEnteringText) && XamlView)
	{
		FKey Key = KeyEvent.GetKey();
		Noesis::Key* NoesisKey = KeyToNoesisKey(Key);
		if (NoesisKey != nullptr)
		{
			XamlView->KeyDown(*NoesisKey);
		}
	}

	return Super::NativeOnKeyDown(MyGeometry, KeyEvent);
}

This sends the raw key input to the xaml file so you handle the key like so, it will be handled:

<b:KeyTrigger Key="Esc">
    <b:InvokeCommandAction Command="{Binding SomeCommand}"/>
</b:KeyTrigger>

However, the issue is when we want to use actions to handle user input:

<noesis:InputActionTrigger Action="Pause" Type="Pressed" Consume="False">
    <b:InvokeCommandAction Command="{Binding SomeCommand}"/>
</noesis:InputActionTrigger>

Actions are handled by using the player controllers input component:

void UNoesisInstance::RegisterInputAction(FInputActionBinding Binding)
{
	if (!InputComponent)
	{
		InitializeInputComponent();
	}

	if (InputComponent)
	{
		InputComponent->AddActionBinding(Binding);
	}
}

Or if you are using Unreal Engines enhanced input it will bind these events, again using the input controller:

TPair<const TCHAR*, Noesis::Key> InputEvents[] = {
	{ TEXT("/NoesisGUI/EnhancedInput/Left"), Noesis::Key_GamepadLeft },
	{ TEXT("/NoesisGUI/EnhancedInput/Up"), Noesis::Key_GamepadUp },
	{ TEXT("/NoesisGUI/EnhancedInput/Right"), Noesis::Key_GamepadRight },
	{ TEXT("/NoesisGUI/EnhancedInput/Down"), Noesis::Key_GamepadDown },
	{ TEXT("/NoesisGUI/EnhancedInput/Accept"), Noesis::Key_GamepadAccept },
	{ TEXT("/NoesisGUI/EnhancedInput/Cancel"), Noesis::Key_GamepadCancel },
	{ TEXT("/NoesisGUI/EnhancedInput/Menu"), Noesis::Key_GamepadMenu },
	{ TEXT("/NoesisGUI/EnhancedInput/View"), Noesis::Key_GamepadView },
	{ TEXT("/NoesisGUI/EnhancedInput/PageUp"), Noesis::Key_GamepadPageUp },
	{ TEXT("/NoesisGUI/EnhancedInput/PageDown"), Noesis::Key_GamepadPageDown },
	{ TEXT("/NoesisGUI/EnhancedInput/PageLeft"), Noesis::Key_GamepadPageLeft },
	{ TEXT("/NoesisGUI/EnhancedInput/PageRight"), Noesis::Key_GamepadPageRight },
	{ TEXT("/NoesisGUI/EnhancedInput/Context1"), Noesis::Key_GamepadContext1 },
	{ TEXT("/NoesisGUI/EnhancedInput/Context2"), Noesis::Key_GamepadContext2 },
	{ TEXT("/NoesisGUI/EnhancedInput/Context3"), Noesis::Key_GamepadContext3 },
	{ TEXT("/NoesisGUI/EnhancedInput/Context4"), Noesis::Key_GamepadContext4 }
};

for (const auto& Event : InputEvents)
{
	UInputAction* Action = LoadObject<UInputAction>(nullptr, Event.Key, nullptr, LOAD_NoWarn);

	if (Action != nullptr)
	{
		EnhancedInputComponent->BindAction(Action, ETriggerEvent::Triggered, this, &UNoesisInstance::OnEnhancedInputActionTriggered, Event.Value);
		EnhancedInputComponent->BindAction(Action, ETriggerEvent::Completed, this, &UNoesisInstance::OnEnhancedInputActionCompleted, Event.Value);
		EnhancedInputComponent->BindAction(Action, ETriggerEvent::Canceled, this, &UNoesisInstance::OnEnhancedInputActionCompleted, Event.Value);
	}
}

Since in Input Mode UI Only, the controller doesn't receive the inputs and therefore these events are not called. Is there anyway we can map our raw key inputs to Unreal Engine's actions without the use of the player controller? I have also noted that OnKeyDown is registered for gamepad inputs as well, however noesis can't map the key in the NativeOnKeyDown function and therefore is not sent to the xaml view. I haven't investigated the use of GamepadTrigger in Xaml yet but if the key down is never sent to the view then I assume it wouldnt work anyway?

<noesis:GamepadTrigger Button="Menu">
    <b:InvokeCommandAction Command="{Binding SomeCommand}"/>
</noesis:GamepadTrigger>

KCoppinsIventis · 23 Mar 2024, 13:10

To expand on my findings, I decide to modify the NoesisInstance.cpp file and change the InitKeyMap function to include mapping for gamepad, for testing I decided to add a mapping for the menu button:

KeyMap.Add(EKeys::Gamepad_Special_Right, Noesis::Key_GamepadMenu);

Then, within my xaml I added:

<noesis:GamepadTrigger Button="Menu" FiredOn="ButtonDown">
    <b:InvokeCommandAction Command="{Binding SomeCommand}"/>
</noesis:GamepadTrigger>

This produced the correct results and sent the correct key to the xaml file. Could someone on the Noesis team let me know if there's a particular reason why the mapping for gamepad isn't included here? Should a bug be raised for this or am I using the system incorrectly? This still doesn't totally solve my issue as I wanted to use actions but I think that maybe an unreal engine problem as it appears to require the player controller's input component

Edit:
Doing this does appear to block some inputs to my player controller whilst in Input Mode Game and UI - specifically dpad entries?

27 Mar 2024, 10:01

Hi,

I've been looking at the way the Common UI plugin works in conjunction with Enhanced Input. It basically bypasses it, by looking at the keys that are mapped to the Input Actions and reacting to those key events. That means the Input Mapping Contentxt's Triggers and Modifiers are ignored. Since Epic took this approach, I think there's not much more that can be done differently. If you create a ticket I can send you a patch for you to try.

Of course, InputActionTriggers won't work, but it does work for the key mappings we have related to the Input Actions in /NoesisGUI/EnhancedInput (Gamepad_Accept and so on).

Common UI seems to have another way of specifying inputs, using UI Tags. I still have to look into that. Seems like it may be a more natural fit than Input Actions for this.

However, may I ask: Why not, instead of setting the input mode to UI only, you don't instead remove the Input Mappings that are relevant to gameplay and add the ones relevant to UI interaction to the Enhanced Input Subsystem?

Handling Input in NoesisGUI for Unreal Engine (Potential bug with NoesisInstance key mappings?)

Handling Input in NoesisGUI for Unreal Engine (Potential bug with NoesisInstance key mappings?)

Re: Handling Input in NoesisGUI for Unreal Engine

Re: Handling Input in NoesisGUI for Unreal Engine

Re: Handling Input in NoesisGUI for Unreal Engine (Potential bug with NoesisInstance key mappings?)

Who is online