cpasjuste
Topic Author
Posts: 11
Joined: 15 Nov 2022, 17:23

Linux (x64/arm64) MediaPlayer

21 Nov 2022, 13:39

Hi,

I need to be able to play some video files in my current project. I saw here that it's not supported on linux, so i wanted to implement a "libmpv" media player (with a "DynamicTextureSource").

After a few research i discovered the "GEMediaPlayer" module which seems to target linux arm64 only. I then compiled the "libMediaPlayer.so" and "mp" binary targeting linux x64, but i do get a lot of "threading" errors until it crash:
GEMediaPlayer: Loading "Particles.mp4"
[NOESIS/I] '/HelloWorld;component/Particles.mp4' loaded
GEMediaPlayer: CreateState
GEMediaPlayer: Alloc
GEMediaPlayer: OpenMedia
GEMediaPlayer: OpenMedia OK
GEMediaPlayer: stream OK
[NOESIS/E] The calling thread (46905) cannot access this HelloWorld.MainWindow because a different thread (46886) owns it
[NOESIS/E] The calling thread (46905) cannot access this AdornerDecorator because a different thread (46886) owns it
[NOESIS/E] The calling thread (46905) cannot access this RootVisual because a different thread (46886) owns it
[NOESIS/E] The calling thread (46905) cannot access this View because a different thread (46886) owns it
GEMediaPlayer: stream OK
GEMediaPlayer: Play
[NOESIS/E] The calling thread (46905) cannot access this View because a different thread (46886) owns it
[NOESIS/E] The calling thread (46905) cannot access this VirtualizingStackPanel because a different thread (46886) owns it
[NOESIS/E] The calling thread (46905) cannot access this ListBox because a different thread (46886) owns it
.....
I'm a little new to C# and such... I wonder if the problem comes from the "GEMediaPlayer" which may not be supported anymore (even on linux arm64 which is the original target) or if i'm doing something wrong. I'd like to move to the mpv implementation, but i'm a little worried about loosing my time on this if i'm unable to fix this threading problem first...

By the way, is there a discord server (or IRC or something) to talk to you guys ?

Many thanks in advance for your help!
 
cpasjuste
Topic Author
Posts: 11
Joined: 15 Nov 2022, 17:23

Re: Linux (x64/arm64) MediaPlayer

22 Nov 2022, 19:08

Hi,

Since i was not able to solve the threading problem, i went ahead and wrote a "libmpv" UserControl today, and it seems to works very fine on linux ! :)
Here is a preview of the work in progress (the recording is very laggy but you get the point): http://files.mydedibox.fr/files/temp/noesis_mpv.webm

I still need to cleanup stuff and add a few "events/triggers", then i'll share the code. It should work on windows too (will check that later, someday). Also note that it use software rendering but that's enough for my usage.

That said, i guess it would be better to rewrite it as a "GEMediaPlayer" alternative (to use the mediaelement tag), but i would probably need some help to understand the "threading problem".
 
User avatar
jsantos
Site Admin
Posts: 3906
Joined: 20 Jan 2012, 17:18
Contact:

Re: Linux (x64/arm64) MediaPlayer

23 Nov 2022, 10:57

Here is a preview of the work in progress (the recording is very laggy but you get the point): http://files.mydedibox.fr/files/temp/noesis_mpv.webm
Thanks for sharing your progress. This looks definitely good! We don't recommend using "GEMediaPlayer", it is an internal implementation that's using a separate process for rendering the video (to avoid licensing problems with the video library). Your approach is much better, even if for now it is only using software rendering.

Regarding the threading problem. Accessing Noesis objects from different threads is not allowed. You must serialize the access to any property or any kind of interaction with objects in the Noesis namespace.
 
cpasjuste
Topic Author
Posts: 11
Joined: 15 Nov 2022, 17:23

Re: Linux (x64/arm64) MediaPlayer

23 Nov 2022, 12:20

Hi jsantos,

First thanks for your reply.

This morning i was successful in moving my work in progress mpv player to a "MediaPlayer" derived class, and so with "MediaElement" compatibility!

All seems to works fine, but i'm struggling with the three events "RaiseMediaEnded", "RaiseMediaFailed" and "RaiseMediaOpened", i don't know how to implement them and so i have manually set the image source in the constructor (since "RaiseMediaOpened" is responsible of setting the source it seems):
((Noesis.Image)owner.Child).Source = pixels.TextureSource;
I tried calling "RaiseMediaOpened" manually but this does seems to do nothing... a little help would probably be cool here :)

You'll find below my current implementation (i need to handle a few more function override but will see that later...)
#if NOESIS
using Noesis;
#else
using System.Runtime.InteropServices;
#endif
using System;

namespace NoesisApp
{
    public class MPVMediaPlayer : MediaPlayer
    {
        private Texture texture;
        Pixels pixels;
        IntPtr mpvHandle;
        IntPtr mpvRenderContext;

        MPVMediaPlayer(MediaElement owner, Uri uri)
        {
            Console.WriteLine("MPVMediaPlayer: owner: " + owner.ToString() + ", uri: " + uri.ToString());

            // init mpv player
            mpvHandle = libmpv.mpv_create();
            if (mpvHandle == IntPtr.Zero)
            {
                throw new Exception("mpv_create");
            }

            // mpv properties/parameters
            libmpv.SetPropertyString(mpvHandle, "terminal", "yes");
            libmpv.SetPropertyString(mpvHandle, "msg-level", "all=v");
            libmpv.SetPropertyString(mpvHandle, "profile", "sw-fast");

            // mpv init
            libmpv.mpv_error err = libmpv.mpv_initialize(mpvHandle);
            if (err < 0)
            {
                throw new Exception("mpv_initialize error: " + libmpv.GetError(err));
            }

            IntPtr api = Marshal.StringToHGlobalAnsi("sw");
            libmpv.mpv_render_param[] renderParams = {
                new libmpv.mpv_render_param(libmpv.mpv_render_param_type.MPV_RENDER_PARAM_API_TYPE, api),
                new libmpv.mpv_render_param(libmpv.mpv_render_param_type.MPV_RENDER_PARAM_INVALID, IntPtr.Zero)
            };
            err = libmpv.mpv_render_context_create(out mpvRenderContext, mpvHandle, renderParams);
            if (err < 0)
            {
                throw new Exception("mpv_render_context_create error: " + libmpv.GetError(err));
            }
            Marshal.FreeHGlobal(api);

            // pixels data
            pixels = new Pixels((uint)owner.ActualWidth, (uint)owner.ActualHeight,
                new DynamicTextureSource((uint)owner.ActualWidth, (uint)owner.ActualHeight, TextureRender, this));

            // load media
            libmpv.CommandV(mpvHandle, "loadfile", "/" + uri.GetPath(), "replace");

            // TODO: handle RaiseMediaOpened
            ((Noesis.Image)owner.Child).Source = pixels.TextureSource;
            //RaiseMediaOpened();
        }

        ~MPVMediaPlayer()
        {
            Close();
        }

        public override void Close()
        {
            if (mpvRenderContext != IntPtr.Zero)
            {
                libmpv.mpv_render_context_free(mpvRenderContext);
                mpvRenderContext = IntPtr.Zero;
            }

            if (mpvHandle != IntPtr.Zero)
            {
                libmpv.mpv_destroy(mpvHandle);
                mpvHandle = IntPtr.Zero;
            }
        }

        public override uint Width
        {
            get { return !Stopped() ? (uint)libmpv.GetPropertyInt(mpvHandle, "width") : 0; }
        }

        public override uint Height
        {
            get { return !Stopped() ? (uint)libmpv.GetPropertyInt(mpvHandle, "height") : 0; }
        }

        public override bool CanPause
        {
            get { return true; }
        }

        public override bool HasAudio
        {
            get { return true; }
        }

        public override bool HasVideo
        {
            get { return true; }
        }

        public override float BufferingProgress
        {
            get { return 1; }
        }

        public override float DownloadProgress
        {
            get { return 1; }
        }

        public override double Duration
        {
            get { return !Stopped() ? libmpv.GetPropertyDouble(mpvHandle, "duration") : 0; }
        }

        public override double Position
        {
            get { return !Stopped() ? libmpv.GetPropertyDouble(mpvHandle, "playback-time") : 0; }
            set { if (!Stopped()) libmpv.Command(mpvHandle, "no-osd seek " + value.ToString()); }
        }

        public override float SpeedRatio
        {
            get { return !Stopped() ? (float)libmpv.GetPropertyDouble(mpvHandle, "speed") : 0; }
            set { if (!Stopped()) libmpv.Command(mpvHandle, "no-osd set speed " + value.ToString()); }
        }

        public override float Volume
        {
            get { return 0.5f; }
        }

        public override float Balance
        {
            get { return 0.5f; }
        }

        public override bool IsMuted
        {
            get { return false; }
        }

        public override bool ScrubbingEnabled
        {
            get { return false; }
        }

        public override void Play()
        {
            Console.WriteLine("MPVMediaPlayer: Play");
            if (!Stopped()) libmpv.Command(mpvHandle, "set pause no");
        }

        public override void Pause()
        {
            Console.WriteLine("MPVMediaPlayer: Pause");
            if (!Stopped()) libmpv.Command(mpvHandle, "set pause yes");
        }

        public override void Stop()
        {
            Console.WriteLine("MPVMediaPlayer: Stop");
            if (!Stopped()) libmpv.Command(mpvHandle, "stop");
        }

        public bool Stopped()
        {
            return mpvRenderContext == IntPtr.Zero || libmpv.GetPropertyBool(mpvHandle, "playback-abort");
        }

        public override ImageSource TextureSource
        {
            get
            {
                Console.WriteLine("MPVMediaPlayer: TextureSource");
                return pixels.TextureSource;
            }
        }

        public static MediaPlayer Create(MediaElement owner, Uri uri, object user)
        {
            return new MPVMediaPlayer(owner, uri);
        }

        private static Texture TextureRender(RenderDevice device, object user)
        {
            MPVMediaPlayer mediaPlayer = user as MPVMediaPlayer;
            return mediaPlayer.GetTexture(device);
        }

        unsafe private Texture GetTexture(RenderDevice device)
        {
            if (texture == null || pixels.Width != (uint)Width || pixels.Height != (uint)Height)
            {
                Console.WriteLine("MpvMediaPlayer: resizing pixels (" + Width + " x " + Height + ")");
                pixels.Resize((uint)Width, (uint)Height);
                texture = device.CreateTexture("mpv", pixels.Width, pixels.Height, 1, TextureFormat.RGBA8, IntPtr.Zero);
            }

            libmpv.mpv_render_context_flag flags = libmpv.mpv_render_context_update(mpvRenderContext);
            if (flags.HasFlag(libmpv.mpv_render_context_flag.MPV_RENDER_UPDATE_FRAME))
            {
                lock (pixels)
                {
                    fixed (byte* data = pixels.Data)
                    {
                        long p = pixels.Pitch;
                        IntPtr pitch = (IntPtr)(long*)&p;
                        int[] s = { (int)pixels.Width, (int)pixels.Height };
                        fixed (int* size = s)
                        {
                            libmpv.mpv_render_param[] renderParams = {
                                new libmpv.mpv_render_param(libmpv.mpv_render_param_type.MPV_RENDER_PARAM_SW_SIZE, new IntPtr(size)),
                                new libmpv.mpv_render_param(libmpv.mpv_render_param_type.MPV_RENDER_PARAM_SW_FORMAT, pixels.FormatPtr),
                                new libmpv.mpv_render_param(libmpv.mpv_render_param_type.MPV_RENDER_PARAM_SW_STRIDE, pitch),
                                new libmpv.mpv_render_param(libmpv.mpv_render_param_type.MPV_RENDER_PARAM_SW_POINTER, (IntPtr)data),
                                new libmpv.mpv_render_param(libmpv.mpv_render_param_type.MPV_RENDER_PARAM_INVALID, IntPtr.Zero)
                            };

                            libmpv.mpv_error err = libmpv.mpv_render_context_render(mpvRenderContext, renderParams);
                            if (err < 0)
                            {
                                throw new Exception("mpv_render_context_render error: " + libmpv.GetError(err));
                            }
                        }

                        device.UpdateTexture(texture, 0, 0, 0, pixels.Width, pixels.Height, (IntPtr)data);
                    }
                }
            }

            return texture;
        }

        private class Pixels
        {
            public Pixels(uint w, uint h, DynamicTextureSource tex)
            {
                FormatPtr = Marshal.StringToHGlobalAnsi("rgba");
                TextureSource = tex;
                Resize(w, h);
            }

            ~Pixels()
            {
                Marshal.FreeHGlobal(FormatPtr);
            }

            public void Resize(uint w, uint h)
            {
                Width = (uint)w;
                Height = (uint)h;
                Pitch = w * Bpp;
                Data = new byte[w * h * Bpp];
                TextureSource.Resize(w, h);
            }

            public uint Width;
            public uint Height;
            public uint Bpp { get => 4; }
            public long Pitch;
            public byte[] Data;
            public int Length { get => Data.Length; }
            public IntPtr FormatPtr;
            public DynamicTextureSource TextureSource;
        }
    }
}
Note that OpenGL rendering should not be hard to add, but i'd like to avoid the need to handle another native library/wrapper. For this, we would need to be able to create an "fbo" framebuffer/texture from managed (as i did here: https://github.com/Cpasjuste/libcross2d ... er.cpp#L20) and be able to get the fbo id (as i did here: https://github.com/Cpasjuste/pemu/blob/ ... re.cpp#L22).
This is something we could look at later...

Finally, the player can only play files from filesystem (or http/ftp...) for now. It would probably require some work to handle embedded resources with mpv...

Edit: fixed the "RaiseMediaOpened, ..." things, didn't get it right at first :) Will polish all that then share.
 
cpasjuste
Topic Author
Posts: 11
Joined: 15 Nov 2022, 17:23

Re: Linux (x64/arm64) MediaPlayer

23 Nov 2022, 16:41

Hi, I'm making a new post to be more "clear"...

So the implementation is going very well, i fixed a crash when changing the "Source" property, and did some cleanup. I have a few questions though:
  • When changing the MediaElement "Source" property, a new player is created, Close() is called but the (MediaPlayer) objects "destructors" are ony called when the application close/exit. Is this a normal behavior (i'm not used to C#) ?
  • How to handle the "Play", "Pause" and "Stop" functions? Should the "Play" function restart the media if not paused? Because there is no argument to the "Pause" function, i guess it should rewind/restart on "Play" if it's not paused, else resume ? Edit: definitive answer found here
Thanks,
cpasjuste
 
User avatar
sfernandez
Site Admin
Posts: 2984
Joined: 22 Dec 2011, 19:20

Re: Linux (x64/arm64) MediaPlayer

25 Nov 2022, 19:26

Hi,

Glad to hear you solved the problem with MediaOpened event. I guess you saw that MediaElement is hooking to that event after creating the MediaPlayer, so you can't raise it from the media player constructor, it must be done later.

Regarding your questions, MediaPlayer object should be released whenever the GC decides, if no other object keeps a strong reference to it. Perhaps we should implement IDisposable on the MediaPlayer and call Dispose from the MediaElement before destroying it, to allow inheritors release any unmanaged resources in the correct thread.

The Play method should start or resume a paused player. And Stop should stop playing and rewind to the start of the media.

Who is online

Users browsing this forum: Google [Bot] and 57 guests