View Issue Details

IDProjectCategoryView StatusLast Update
0003356NoesisGUIUnity3Dpublic2024-06-10 21:56
Reportersaboehnke Assigned Tojsantos  
PrioritynormalSeverityminorReproducibilitysometimes
Status resolvedResolutionwon't fix 
Product Version3.2 
Summary0003356: TextureSource is not updating at runtime
DescriptionI am attempting to play a video stream on a worldspace UI component but the TextureSource is not being updated at runtime while the texture that is being used to update it is being updated. This has been confirmed by using a RawImage and setting its source to the same texture. The RawImage updates as it should while the worldspace UI will only show the first frame of the video.
TagsC#, texturesource, Unity, worldspaceUI
PlatformWindows

Activities

jsantos

jsantos

2024-05-31 11:48

manager   ~0009604

Last edited: 2024-05-31 11:49

Have you tried using the MediaElement (that in Unity is implemented using the VideoPlayer) instead of raw Textures?

It seems your approach is creating a new texture per frame and this seems to be quite inefficient.
saboehnke

saboehnke

2024-05-31 14:17

reporter   ~0009606

So I'm receiving a continuous real time stream of video. The streaming third party I am using lets me specify a Texture2D (which I then use in a worldspace UI by creating a TextureSource with it). This texture is continuously updated (the texture is being altered instead of creating a new one over and over).
jsantos

jsantos

2024-05-31 19:07

manager   ~0009608

It seems that Unity is internally creating a new texture (or changing the texture handle to a new one that is free) and we need to notify Noesis about this.

I will work on this ASAP.
saboehnke

saboehnke

2024-05-31 19:59

reporter   ~0009609

I'm not sure I understand. It is just one texture and it is set only once. If I set the texture for the RawImage in unity I'm only setting it one time. It stays up to date because what is considered the data of that texture is changing. You're saying that worldspace UI can't handle this? Because it works sometimes. Not often, but sometimes.
jsantos

jsantos

2024-05-31 20:05

manager   ~0009610

Last edited: 2024-05-31 20:05

You are doing Texture2D.SetPixels32 or something similar right?

If I remember this correctly, invoking that function is changing the internal low-level handle of the texture to a new one while Noesis is still using the old one.
saboehnke

saboehnke

2024-05-31 21:31

reporter   ~0009611

I believe that is how the third party we are using most likely handles it. I just specify a texture for their Unity SDK and they do the writing to it. But it is always the same texture.

I appreciate you looking into the fix. Do you have any idea how long the change may take? Or maybe how long it'd take to get a patch?
saboehnke

saboehnke

2024-06-04 15:12

reporter   ~0009635

Hi there, I'm just following up on this to see what you think about my last response. I have to determine if I will be moving forward with worldspace UI or if I need to swap to raw images in unity instead.
jsantos

jsantos

2024-06-04 18:04

manager   ~0009636

Even though it is the same texture, every time it is updated, Unity is assigning a new handle. And we need that new handle to be sent to our render thread, but the only way to get that handle is using GetNativeTexturePtr. I have been profiling this function, and it is extremely slow, it is taking more than 10ms.

Unity offers a better alternative for updating Textures in native code, by using CommandBuffer.IssuePluginCustomTextureUpdateV2.

This would require changes in your Third Party. Could you please indicate which Third Party is this one?

I will try to find more time this week to analyze this with more detail.
saboehnke

saboehnke

2024-06-04 19:01

reporter   ~0009638

Sure thing. We use Agora's Unity SDK for video streaming (sharing screens, etc.). There is no way they are going to change their implementation. They essentially have an event that I listen for "ApplyTexture". It hands me a Texture2D. I then setup the worldspace ui for the video with a user control with this inside:
<Image
    x:Name="StreamImage"
    Stretch="Uniform"
    Source="{Binding state.VideoSource}"
    MaxWidth="{Binding state.VideoWidth}"
    MaxHeight="{Binding state.VideoHeight}"
    HorizontalAlignment="Center"
    VerticalAlignment="Center">
    <Image.RenderTransform>
        <TransformGroup>
            <ScaleTransform ScaleY="-1" />
            <TranslateTransform x:Name="TranslateTransform" Y="{Binding ActualHeight, ElementName=StreamImage}" />
        </TransformGroup>
    </Image.RenderTransform>
</Image>

Regardless, if this requires the third party to make a change rather than Noesis being able to update the ui accordingly based on the texture being changed I will need to move to another solution. Please let me know.
jsantos

jsantos

2024-06-04 19:22

manager   ~0009639

I will work on this task this week. Could you give me more information about this "ApplyTexture" event? In general, the process used by Agora (filling the texture with CPU and in the main thread) doesn't seem ideal to me.

We also sent an email to "Ryan Hesse" in charge of this license. Could you please confirm if the email was received?
saboehnke

saboehnke

2024-06-04 19:46

reporter   ~0009640

It's just an event that hands me the texture to apply to my worldspace UI. I misspoke before when I said I handed them one. Ryan is my boss. He says he hasn't received the email yet.
jsantos

jsantos

2024-06-04 20:06

manager   ~0009642

Last edited: 2024-06-04 20:06

We sent the email yesterday. Please, check the spam folder. Thank you!

Is there any web documentation about this ApplyTexture event offered by Agora?
Is this event raised per frame?
saboehnke

saboehnke

2024-06-04 20:17

reporter   ~0009643

It is called once to specify the texture. So I get the texture from that callback, I use it in my worldspace ui, and the updates to the texture are on their side.

I let him know to check his spam.
jsantos

jsantos

2024-06-05 11:37

manager   ~0009649

Please, have a look at this post

https://www.noesisengine.com/forums/viewtopic.php?t=2130

The proposal workaround should work for you too. You need to Blit the texture that Agora is filling to a new texture that is wrapped by Noesis. This is a GPU-GPU process, so should be quite fast, happens in the render thread and won't change internal handles.

What is your platform by the way? And what renderer are you using (OpenGL?)
saboehnke

saboehnke

2024-06-05 15:18

reporter   ~0009651

Last edited: 2024-06-05 15:38

The platform is windows.

So currently I take the Texture2D and set the VideoSource (used as the source in the image in my xml from a previous response of mine) like so:
VideoSource = new TextureSource(texture);

You're saying that if I do something like this, that it'll fix the issue?

RenderTexture blitTexture = new RenderTexture(256, 256, 16, RenderTextureFormat.ARGB32);
Graphics.Blit(texture, blitTexture);
VideoSource = new TextureSource(blitTexture);

Because if I go this route I get a still image with no updates as before.
jsantos

jsantos

2024-06-05 16:04

manager   ~0009652

Yes, that should work.

What renderer is this by the way? (GL, D3D11, D3D12, Vulkan)
saboehnke

saboehnke

2024-06-05 16:10

reporter   ~0009653

In Unity we have "Auto Graphics API for Windows" turned on so Unity will determine the renderer based on the platform and ability of the pc.
jsantos

jsantos

2024-06-05 16:20

manager   ~0009654

So, in your machine where you are reproducing the issue, this is D3D12 renderer right?
saboehnke

saboehnke

2024-06-05 16:37

reporter   ~0009656

I'm not sure how to check that to be honest.
jsantos

jsantos

2024-06-05 16:47

manager   ~0009657

Last edited: 2024-06-05 16:47

Unity dumps that information to the log when launching.

This can be useful for a few things I want to check,
saboehnke

saboehnke

2024-06-05 17:13

reporter   ~0009659

Initialize engine version: 2022.3.0f1 (fb119bb0b476)
GfxDevice: creating device client; threaded=1; jobified=1
Direct3D:
    Version: Direct3D 11.0 [level 11.1]
    Renderer: NVIDIA GeForce RTX 4080 Laptop GPU (ID=0x27e0)
    Vendor: NVIDIA
    VRAM: 11997 MB
    Driver: 31.0.15.5212

Is any of that useful? I don't see anything in the player log about a renderer other than this.
jsantos

jsantos

2024-06-05 17:15

manager   ~0009660

Yes, Direct3D 11, thanks.
jsantos

jsantos

2024-06-07 19:25

manager   ~0009672

"Because if I go this route I get a still image with no updates as before."

You need to Blit each time the texture changes. If Agora doesn't give you this information you can use Texture.updateCount.

I think this ticket can be closed.

Please, let me know.
saboehnke

saboehnke

2024-06-10 18:14

reporter   ~0009682

So you're saying I should check the textures updateCount at every frame and blit to a new RenderTexture and reassign the video source to a new TextureSource created from the new RenderTexture? Hopefully I won't have to go with the updateCount route...because that is not a great solution.
jsantos

jsantos

2024-06-10 18:17

manager   ~0009683

Last edited: 2024-06-10 19:09

The updateCount is a workaround in case Agora is not giving you the information about when the texture is updated. Checking that per frame should be trivial.

The RenderTexture and the associated TextureSource must be created only once. You only need to perform the Blit per frame (from the texture given to Agora to the RenderTarget you created).
saboehnke

saboehnke

2024-06-10 19:19

reporter   ~0009685

Sure...things like this are trivial in the simplified case I have provided. But when it gets more involved like in the case where we can have multiple video streams active at once owned by different users, things become more difficult to manage. We were really hoping that this was something that could be handled on the noesis side so there wouldn't have to be a manual update process on our end. We will probably just move to using raw images instead of using noesis worldspace ui.
jsantos

jsantos

2024-06-10 21:56

manager   ~0009687

There are different things involved here, first, Agora current API is not very friendly with native plugins and I am pretty sure they have high cpu usage in the main thread. Unity provide solutions for that and, as mentioned before, I would be more than happy to have a conversation with Agora about this.

Just having a manager of all your texture sources that check per frame if the texture count was updated seems a reasonable workaround to me in comparison with the alternative of mixing Raw Images with other Noesis in 3D.

But that's up to you of course. I am going to close this ticket.

Issue History

Date Modified Username Field Change
2024-05-30 17:04 saboehnke New Issue
2024-05-30 17:04 saboehnke Tag Attached: C#
2024-05-30 17:04 saboehnke Tag Attached: texturesource
2024-05-30 17:04 saboehnke Tag Attached: Unity
2024-05-30 17:04 saboehnke Tag Attached: worldspaceUI
2024-05-31 11:46 jsantos Assigned To => jsantos
2024-05-31 11:46 jsantos Status new => assigned
2024-05-31 11:48 jsantos Note Added: 0009604
2024-05-31 11:48 jsantos Status assigned => feedback
2024-05-31 11:48 jsantos Note Edited: 0009604
2024-05-31 11:49 jsantos Note Edited: 0009604
2024-05-31 14:17 saboehnke Note Added: 0009606
2024-05-31 14:17 saboehnke Status feedback => assigned
2024-05-31 19:07 jsantos Note Added: 0009608
2024-05-31 19:07 jsantos Target Version => 3.2.4
2024-05-31 19:59 saboehnke Note Added: 0009609
2024-05-31 20:05 jsantos Note Added: 0009610
2024-05-31 20:05 jsantos Note Edited: 0009610
2024-05-31 21:31 saboehnke Note Added: 0009611
2024-06-04 15:12 saboehnke Note Added: 0009635
2024-06-04 18:04 jsantos Note Added: 0009636
2024-06-04 18:04 jsantos Status assigned => feedback
2024-06-04 19:01 saboehnke Note Added: 0009638
2024-06-04 19:01 saboehnke Status feedback => assigned
2024-06-04 19:22 jsantos Note Added: 0009639
2024-06-04 19:22 jsantos Status assigned => feedback
2024-06-04 19:46 saboehnke Note Added: 0009640
2024-06-04 19:46 saboehnke Status feedback => assigned
2024-06-04 20:06 jsantos Note Added: 0009642
2024-06-04 20:06 jsantos Status assigned => feedback
2024-06-04 20:06 jsantos Note Edited: 0009642
2024-06-04 20:17 saboehnke Note Added: 0009643
2024-06-04 20:17 saboehnke Status feedback => assigned
2024-06-05 11:37 jsantos Note Added: 0009649
2024-06-05 11:37 jsantos Status assigned => feedback
2024-06-05 15:18 saboehnke Note Added: 0009651
2024-06-05 15:18 saboehnke Status feedback => assigned
2024-06-05 15:36 saboehnke Note Edited: 0009651
2024-06-05 15:38 saboehnke Note Edited: 0009651
2024-06-05 16:04 jsantos Note Added: 0009652
2024-06-05 16:04 jsantos Status assigned => feedback
2024-06-05 16:10 saboehnke Note Added: 0009653
2024-06-05 16:10 saboehnke Status feedback => assigned
2024-06-05 16:20 jsantos Note Added: 0009654
2024-06-05 16:20 jsantos Status assigned => feedback
2024-06-05 16:37 saboehnke Note Added: 0009656
2024-06-05 16:37 saboehnke Status feedback => assigned
2024-06-05 16:47 jsantos Note Added: 0009657
2024-06-05 16:47 jsantos Note Edited: 0009657
2024-06-05 17:13 saboehnke Note Added: 0009659
2024-06-05 17:15 jsantos Note Added: 0009660
2024-06-07 19:25 jsantos Note Added: 0009672
2024-06-07 19:25 jsantos Status assigned => feedback
2024-06-07 19:25 jsantos Target Version 3.2.4 =>
2024-06-10 18:14 saboehnke Note Added: 0009682
2024-06-10 18:14 saboehnke Status feedback => assigned
2024-06-10 18:17 jsantos Note Added: 0009683
2024-06-10 18:18 jsantos Status assigned => feedback
2024-06-10 19:09 jsantos Note Edited: 0009683
2024-06-10 19:19 saboehnke Note Added: 0009685
2024-06-10 19:19 saboehnke Status feedback => assigned
2024-06-10 21:56 jsantos Note Added: 0009687
2024-06-10 21:56 jsantos Status assigned => resolved
2024-06-10 21:56 jsantos Resolution open => won't fix