nokola
Topic Author
Posts: 188
Joined: 10 Mar 2015, 05:29

How to avoid unnecessary full screen redraws on Unity?

15 Jan 2016, 01:01

I currently have this setup for a mobile painting app in Unity:
1. Screen texture (e.g. 1280x720), Image texture (e.g. 2000x2000), noesis renderer to screen
2. A camera that never deletes the screen, I only redraw changes

This is what happens when the user paints: OnPaint:
1. Calculate dirty rects (those are the regions that have changes painted by the user)
2. Update ImageTexture with the new strokes
3. DrawOnScreen(ImageTexture, dirty rects) -> redraws only the changes
4. DrawNoesisXamlThatIsCurrentlyVisible()

The issue is that on Step 4, noesis ends up drawing overlaying text, especially when I have animations. That's because my camera never deletes the scene, and only draws the changes. I'd have to add a step 3.5 and change step 4:
3.5. noesisDirtyRects = GetNoesisDirtyRectsThisFrame()
4. If (noesisDirtyRects is not empty) { DrawOnScreen(ImageTexture, noesisDirtyRects); DrawNoesisXamlThatIsCurrentlyVisible(noesisDirtyRects); }

However, I don't know how to get dirty rects from Noesis (opened a bug.)
Without dirty rects, I'd have to redraw my texture on every frame and the XAML on every frame, which will impact battery life significantly.

Another possible simplification (not optimal but much better) could be:
4. If Noesis_HasChangedSinceLastFrame() { RedrawImageTexture(); DrawNoesisXamlThatIsCurrentlyVisible() }

Is there a better solution that I can currently use without having to redraw the whole screen twice on every frame? Or a way to know if there are visual changes since last XAML frame from Noesis?
 
User avatar
jsantos
Site Admin
Posts: 2904
Joined: 20 Jan 2012, 17:18
Contact:

Re: How to avoid unnecessary full screen redraws on Unity?

15 Jan 2016, 13:35

I currently have this setup for a mobile painting app in Unity:
1. Screen texture (e.g. 1280x720), Image texture (e.g. 2000x2000), noesis renderer to screen
2. A camera that never deletes the screen, I only redraw changes
Before going on with the rest of the message, are you sure this is a supported scenario by Unity? I mean, you are supposing that the framebuffer content is preserved from frame to frame. What happens if the mobile is using double or triple buffer for example?

We are working in supporting dirty rectangles for v1.3, by activating a flag in the component saying that the backbuffer is preserved. We are not exposing the dirty rectangles because they are calculated internally in the render thread. Anyway, with this flag available you could do the following:

1. Render NoesisGUI in a texture enabling the Dirty Rectangles flag. With this flag, the texture will be only updated whenever it changes.
2. Render your application to the main buffer as you are doing now, using your own dirty rectangles and copying the parts you need from the NoesisGUI texture.

What do you think about this approach?
 
nokola
Topic Author
Posts: 188
Joined: 10 Mar 2015, 05:29

Re: How to avoid unnecessary full screen redraws on Unity?

15 Jan 2016, 17:43

Yes, no-redraw camera is supported scenario. It's in the docs and I've been working with it on Android for the last few months.

The solution you mention will not work without an extra step that renders the Noesis dirty rectangles on screen too.

Here is an example screen that won't work.
This is the phone screen with changes for the next frame:
***********
*         *
*  PPP    *
*         *
*     NN  *
***********
The "PPP" is a dirty rectangle for my app (where the user just painted) and "NN" is a dirty rectangle from Noesis (e.g. XAML animation in progress)

If I follow this algorithm:
1. Render NoesisGUI in a texture enabling the Dirty Rectangles flag. With this flag, the texture will be only updated whenever it changes.
2. Render your application to the main buffer as you are doing now, using your own dirty rectangles and copying the parts you need from the NoesisGUI texture.
I won't know what the Noesis "NN" dirty rects are to only copy those from the Noesis texture in step 2.
Plus, I'll need an additional texture for Noesis render-to-texture with this approach (not bad solution even though not optimal.)

In case of noesis render-to-texture, would it be possible to get the dirty rects after the render completes? For a solution like this:
1. PPP = DrawUsersPaintingThisFrameToImageTexture() // PPP are the dirty rects
2. NN = DrawNoesisToTexture(NoesisTexture) // NN are the dirty rects
3. AllDirtyRects = Union(PPP, NN);
4. DrawImageToScreen(AllDirtyRects) // refresh the background on screen, this includes user's painting in step 1
5. DrawNoesisTextureToScreen(NN) // draw XAML changes to screen

How does this look?
 
User avatar
jsantos
Site Admin
Posts: 2904
Joined: 20 Jan 2012, 17:18
Contact:

Re: How to avoid unnecessary full screen redraws on Unity?

18 Jan 2016, 17:07

I won't know what the Noesis "NN" dirty rects are to only copy those from the Noesis texture in step 2.
Could you evaluate the impact in the battery between copying all the texture and copying only a small sub-rectangle?
Plus, I'll need an additional texture for Noesis render-to-texture with this approach (not bad solution even though not optimal.)
Another question, could you use NoesisGUI for painting everything or do you have 3D elements that must be painted using Unity?
 
nokola
Topic Author
Posts: 188
Joined: 10 Mar 2015, 05:29

Re: How to avoid unnecessary full screen redraws on Unity?

19 Jan 2016, 01:17

Could you evaluate the impact in the battery between copying all the texture and copying only a small sub-rectangle?
Short answer: No, however I will make those tests and update you. Once I have my first alpha released in a month or so, I'll have a better view of performance in the real-world.

More detailed answer: Not for Android (the platform I focus on currently), however Yes for Windows Phone
I don't have data for "small rectangle" vs "full screen" however I have some data for "no painting if no changes" vs "full screen repaint" scenario from previous tests:
With one-redraw per frame, the phone gets hotter and runs out of battery as if playing a game (few hours)
With no-redraw if changes, the battery load is reduced almost by 50%, because a significant part of the time the screen doesn't redraw at all (e.g. when you're just looking at it.) The moment a person lifts their finger and stops painting, the screen stops refreshing since there are no changes.

Note that if I do a full screen redraw of Noesis I'd have to do a full screen redraw of the underlying texture as well. Those are bigger image textures (several megapixels) that are costly to draw on every frame. My point is: even if the noesis full screen redraw is not very expensive, the need to redraw the huge image texture below the noesis gui is expensive.
Another question, could you use NoesisGUI for painting everything or do you have 3D elements that must be painted using Unity?
Yes, for the moment. I plan to add some interactive elements later (half a year+ from now) however if a noesis-only solution works better I'll look into it as well.
Do you have something in mind?

Additional info:
I also want to start with the right foundation, because the GPU is otherwise heavily used. For example, I have brushes that paint up to 200,000 lines on some frames (usually few thousand; when painting fast about 10-20k, and when painting like crazy across the screen they can reach into the 200k in very rare cases.) To make the line drawing efficient, I spent a couple months writing my own shaders that produce both high-quality antialiased lines at a reasonable speed.

I also have some post-processing effects that run in realtime, and can be painted on and off, those eat a lot of GPU power.

My point is there are already a bunch of things taxing the GPU and everything that can be squeezed in terms of performance leads to better app for the customer :)
 
User avatar
jsantos
Site Admin
Posts: 2904
Joined: 20 Jan 2012, 17:18
Contact:

Re: How to avoid unnecessary full screen redraws on Unity?

19 Jan 2016, 15:21

Do you have something in mind?
Yes, I was thinking about reversing the process and render the Unity elements you need to a texture and use that texture as part of the tree of NoesisGUI elements. That way:

1. You know when and where to render your texture.
2. NoesisGUI only updates the parts of the screen that really changes. No updates if no changes. We will need a mechanism to inform about the dirty rectangles in the textures that you are using...

I don't know, this approach is easier for us because, as said, the dirty rectangles are calculated in the render thread, and having access to that information in the main thread will imply a lock or having obsolete information (a few frames).
 
nokola
Topic Author
Posts: 188
Joined: 10 Mar 2015, 05:29

Re: How to avoid unnecessary full screen redraws on Unity?

19 Jan 2016, 19:02

1. You know when a where to render your texture.
2. NoesisGUI only updates the parts of the screen that really changes. No updates if no changes. We will need a mechanism to inform about the dirty rectangles in the textures that you are using...
I think this is a great idea! It could be done with something like SwapChainBackgroundPanel: http://blogs.msdn.com/b/windowsappdev/a ... rectx.aspx

Ideally it should have a function that can say only part of the background texture is changed, e.g. Invalidate(RECT rect); // if rect == null invalidate all the texture

I don't think we need a list of dirty rects to pass in btw, 1 rect should be enough.

If that is done, then I can setup a camera that never clears and we'll truly have "redraw only on changes."

Is this similar to what you have in mind?
 
User avatar
jsantos
Site Admin
Posts: 2904
Joined: 20 Jan 2012, 17:18
Contact:

Re: How to avoid unnecessary full screen redraws on Unity?

21 Jan 2016, 14:11

Is this similar to what you have in mind?
Yes, something like that should work. Thanks for creating the ticket in the tracker.
 
nokola
Topic Author
Posts: 188
Joined: 10 Mar 2015, 05:29

Re: How to avoid unnecessary full screen redraws on Unity?

22 Jan 2016, 05:57

Awesome, thanks!

Who is online

Users browsing this forum: No registered users and 1 guest