HLS video stream fullscreen and seek time

Feb 21, 2014 at 4:16 PM
Hello,
I was happy to find this framework as I need to play HLS streams and the m3u8 files. Playback works fine, but unfortunately the long seek time required for the video stream and issues with the video size(resolution) is holding me back from going on.

I'm using VS2012Prof and a build of SM from the source in "phonesm-20130921" zip file, testing on "Emulator WVGA 512MB." I need to work with WP8 and not WP8.1.

1) I have red the Random To do notes and it mentions the issues with the seek time. I'm having some difficulties building the latest source code, so while I get it work I'm wondering if this seeking would still be an issue in the latest source, and also if it the latest code will be still usable on WP8.
I'm also using the Player framework; does it matter or I should look for a solution in the streaming media framework?

2a) As for the full screen, I can't get the video to stretch correctly. When using fill, It works unless the video is "widescreen?" (or resolutions of 720x648 and 720x576 or something). Then the player has those black margins below and at the top of the display. Using UniformToFill end even worse, there is usually a black margin on the top and the picture is also too big to fit the display or is "pushed out" at the bottom of the display. On Android we did not have these kind of issues and everything gets played correctly on full screen, no black bars.
Does anybody have any ideas ??
I don't know how much work is done with this within your code aside reading the video size and so on, I just wanted to know if this is possibly an issue on your side or with the System.Windows.Controls.MediaElement.

2b) Setting the Height and Width directly also gives the same results, depending on if I set the Width or the Height values. And According to Microsofts' documentation there is no way to set both the Height and Width at once (changing the aspect ratio) ?

Thanks for any answers and sorry if some of the questions are totally wrong, but right now I'm just really confused and could not find relevant info on the web ...
Feb 21, 2014 at 5:34 PM
You may change the AR by using Stretch == Fill and changing size of the MediaElement, or by using a RenderTransform. Fortunately, WP/WPF/SL media model is very flexible...

P.S. AFAIK, this project has nothing to do with aspect ratio; it's your video source problem.
Feb 21, 2014 at 5:51 PM
Thanks,
good to know that there are no factors influencing the video size in the project. One place less to look at. As i mentioned fill and setting the size does not work as expected, but i will take a look on RenderTransform.
Any ideas for the 1st part, the seek time. May this be an issue with PlayerFramework or with this project? And would it get better by getting and using the latest source code ? THanks again.
Feb 21, 2014 at 8:35 PM
I can recommend you to use "plain" MediaElement for your tests instead of using PlayerFramework. At least you can eliminate possible framework's bugs. And I don't understand #1 question: what is the hardship?
Coordinator
Feb 21, 2014 at 11:23 PM
AlexBq wrote:
Hello,
I was happy to find this framework as I need to play HLS streams and the m3u8 files. Playback works fine, but unfortunately the long seek time required for the video stream and issues with the video size(resolution) is holding me back from going on.
The current source code should significantly improve seek times for you.
I'm using VS2012Prof and a build of SM from the source in "phonesm-20130921" zip file, testing on "Emulator WVGA 512MB." I need to work with WP8 and not WP8.1.

1) I have red the Random To do notes and it mentions the issues with the seek time. I'm having some difficulties building the latest source code, so while I get it work
I'm wondering if this seeking would still be an issue in the latest source, and also if it the latest code will be still usable on WP8.
What problems are you having with the build?
I'm also using the Player framework; does it matter or I should look for a solution in the streaming media framework?
The Player Framework plugin from phonesm-20130921 has some issues if you wanted to change streams. I would second SeNS' comment about giving your stream a try with HlsView.WP8 ("plain" MediaElement) if you suspect anything odd with the Player Framework plugin.
Thanks for any answers and sorry if some of the questions are totally wrong, but right now I'm just really confused and could not find relevant info on the web ...
Nothing to be sorry about. I'm trying to get a new build together, and updating the docs is definitely on the TODO list.
Coordinator
Feb 22, 2014 at 2:30 AM
As for the build, I broke SM.Media.Legacy this morning. I'll push a fix shortly (after I go dig out my dunce cap).

If that isn't the problem you are having, you might want to...
  • Make sure you have populated the SMF directory (see the readme.txt in that folder).
  • Check which version of Player Framework you have installed (it should show up as 1.8.2.2).
  • Make sure you have the current version of NuGet installed (check for updates under Tools -> Extensions and Updates...).
Feb 24, 2014 at 9:47 AM
Edited Feb 24, 2014 at 10:00 AM
Thanks for the feedback, I will continue to experiment with only the MediaElement and HLS.View.WP8.

As for the build error, it may be an error only on my side. Did not investigate further, so consider the below ex details only as informative, just for the case if it would be relevant. :

I was working with the source as of 21.feb.2014 using VS2012 update 4, have latest NuGet installed and HLS.View.WP8 is functional.
Player Framwork version is 1.8.2.2
Trying SamplePlayer.WP8 (building as debug and target OS version WP OS 8.0) ends with an error.
This error also shows up in the designer view, so i might have still missing something.
(If I understood well, SMF is only required for WP7 projects, not WP8 right ? ) :

System.ArgumentNullException was unhandled by user code
HResult=-2147467261
Message=Value cannot be null.
Parameter name: instance
Source=Autofac
ParamName=instance
StackTrace:
   at Autofac.RegistrationExtensions.RegisterInstance[T](ContainerBuilder builder, T instance)
   at SM.Media.Builder.BuilderBase.RegisterSingleton[TService](TService instance)
   at SM.Media.TsMediaManagerBuilder..ctor(IHttpClients httpClients)
   at SM.Media.MediaStreamFascade.CreateBuilder(IHttpClients httpClients)
   at SM.Media.MediaStreamFascade..ctor(IHttpClients httpClients, Func`2 setSourceAsync)
   at SM.Media.DefaultMediaStreamFascadeParameters.<.cctor>b__0(IHttpClients httpClients, Func`2 mediaStreamSourceSetter)
   at SM.Media.DefaultMediaStreamFascadeParameters.Create(MediaStreamFascadeParameters parameters, IHttpClients httpClients, Func`2 mediaStreamSourceSetter)
   at SM.Media.Me0diaPlayer.StreamingMediaPlugin.<get_MediaElement>b__0(Func`2 setSource)
   at SM.Media.MediaPlayer.MediaElementWrapper..ctor(Func`2 mediaStreamFascadeFactory)
   at SM.Media.MediaPlayer.StreamingMediaPlugin.get_MediaElement()
   at Microsoft.PlayerFramework.MediaPlayer.GetTemplateChildren()
   at Microsoft.PlayerFramework.MediaPlayer.<OnApplyTemplate>d__1.MoveNext()
InnerException:
Coordinator
Feb 24, 2014 at 11:05 AM
I think you ran into the problem fixed in ce9995d0a868. Could you try updating your code and see if it works for you now?

Thanks.

Apart from WP7's background audio client not working, the only weirdness I am aware of is a problem I ran into on a box with an old video card (HD 4870 X2). With the AMD Legacy 13.1 or 13.9 beta drivers installed, the Win8.1 client would not run properly. Neither the MediaStreamSource.Started nor the MediaStreamSource.SampleRequested events ever fired for phonesm or for Microsoft's MediaStreamSource streaming sample (well, the JavaScript sample worked, but the C# and C++ samples would fail). The most meaningful hint I got was that the C++ sample threw some kind of COM exception deep down in what appeared to be DirectX code when the MediaStreamSource was assigned to the MediaElement. Removing the AMD drivers and reverting to Microsoft's default driver for the card made the problem go away. I never tried a clean install of the AMD drivers, so I don't know if it is a problem with those driver or with something that was just wonky on that particular computer. (And, yes, I did try another thing or two before mucking with the video driver...)
Marked as answer by AlexBq on 2/24/2014 at 5:30 AM
Feb 24, 2014 at 12:29 PM
Yes, ce9995d0a868 helps. Thanks.
Coordinator
Feb 24, 2014 at 1:18 PM
Ok, good.

Sorry for the problem and thanks for reporting it.
Feb 24, 2014 at 3:56 PM
Edited Feb 24, 2014 at 4:09 PM
Hi, I need to add another question. Looks like I can't play the playlists I receive using the latest code. Though they work using version phonesm-20130921.

Looks like it comes from TsMediaManager:

System.NotSupportedException occurred
HResult=-2146233067
Message=Unknown content type
Source=SM.Media
StackTrace:
   at SM.Media.TsMediaManager.<CreateReaderPipeline>d__24.MoveNext()
InnerException:

m3u8 like :
#EXTM3U
#EXT-X-VERSION:1
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-ALLOW-CACHE:YES
#EXT-X-TARGETDURATION:13
#EXTINF:13,
#EXT-X-PROGRAM-DATE-TIME:2014-02-24T14:50:00Z
/x/xxxxxxxxxxx/aaaaaaaaa?p=hls&r=c&s=
#EXTINF:13,
#EXT-X-PROGRAM-DATE-TIME:2014-02-24T14:50:12Z
/x/xxxxxxxxxxx/aaaaaaaaa?p=hls&r=c&s=AQ==
#EXTINF:12,
#EXT-X-PROGRAM-DATE-TIME:2014-02-24T14:50:24Z
/x/xxxxxxxxxxx/aaaaaaaaa?p=hls&r=c&s=Ag==
#EXTINF:12,
#EXT-X-PROGRAM-DATE-TIME:2014-02-24T14:50:35Z
/x/xxxxxxxxxxx/aaaaaaaaa?p=hls&r=c&s=Aw==
...
/*removed to save space*/
...
#EXTINF:13,
#EXT-X-PROGRAM-DATE-TIME:2014-02-24T16:59:32Z
/x/xxxxxxxxxxx/bbbbbbbbb?p=hls&r=c&s=Ao0=
#EXTINF:12,
#EXT-X-PROGRAM-DATE-TIME:2014-02-24T16:59:45Z
/x/xxxxxxxxxxx/bbbbbbbbb?p=hls&r=c&s=Ao4=
#EXTINF:7,
#EXT-X-PROGRAM-DATE-TIME:2014-02-24T16:59:57Z
/x/xxxxxxxxxxx/bbbbbbbbb?p=hls&r=c&s=Ao8=
#EXT-X-ENDLIST
Coordinator
Feb 24, 2014 at 4:46 PM
Hmmmm... I went back and forth on how to deal with this. Those segments don't give much of a hint as to what they are. Do you happen to know the MIME type (Content-Type header) returned for those segments?

The code used to assume that everything was .ts unless it was obviously something else (e.g., there was an .aac or .mp3 extension).

If you want to change this behavior, then go to TsMediaManager.CreateReaderPipeline() and default contentType to ContentTypes.TransportStream:
    await InitializeMediaParser(reader, contentType ?? ContentTypes.TransportStream).ConfigureAwait(false);
And, of course, drop the preceding throw new NotSupportedException().

I think I'll change it back to defaulting to .TS, but with a Debug.WriteLine() noting that it couldn't determine the type. (At some point, I'll do a global search/replace to change the Debug.WriteLine()s call to some proper logging...)

Much of that code probably belongs in ReaderPipeline instead of TsMediaManager, and that is one of the few remaining things I've been meaning to look at before doing another build. If you do make local changes and are tracking phonesm source changes, you may get some conflicts later this week.
Feb 25, 2014 at 9:39 AM
Y, thanks a lot again, defaulting to transport stream helped.
Coordinator
Feb 25, 2014 at 10:29 AM
Good to hear.

I'll be pushing some changes in a day or so that, among other things, sets the default to Transport Stream.

The original reason for not having a default was to get quick failures when the server was returning something other than expected (typically, a status code 200 response along with a 404ish HTML body instead of an MPEG-2 TS body). The other change needed to accomplish this is a check on the content type header when the callback reader actually starts shoveling data. This second part was never implemented. Intelligent retries, particularly when alternate servers are available, need to know as quickly as possible that something is wrong. Anyhow, some of the infrastructure for doing this is in place; much is not. I don't expect to be looking at that again for a while, and when I do, I'll try to make sure the policies used--be that for what timeouts to use or what, if any, default content type to use--is easily configurable without recompiling SM.Media.

BTW, are you seeing better seek times with the new code compared to phonesm-20130921? (You may need to test that in release mode, all those Debug.WriteLine() calls...)
Feb 25, 2014 at 2:45 PM
Edited Feb 25, 2014 at 2:46 PM
Seek times using the player framework are now somewhere between 15-30 seconds (release build), which is most of the times more then the time it takes to start the playback. But I still need to do some test using the Hls.View setup ...
Coordinator
Feb 25, 2014 at 4:31 PM
That sounds pretty slow for seeking. You may want to look at the buffering properties in DefaultBufferingPolicy. Both BytesMinimum and DurationBufferingDone might be too high for your application. They should probably not be constants, but that is a can-o-worms I have not yet opened.

The only code that looks at those parameters is in DefaultBufferingPolicy members, so they are safe to muck around with. You probably want to keep the minimums below the maximums and such.

The best way to get a sense of what is going on with the buffering is to take a look at the debug output after a seek. You should see stuff along the lines of:
BufferingManager.UpdateBuffering: 88.14%, 00:00:07.9330000 duration, 1018193 size, 1.91 MiB memory
BufferingManager.UpdateBuffering: 89.63%, 00:00:08.0670000 duration, 1083249 size, 2.25 MiB memory
BufferingManager.UpdateBuffering: 93.70%, 00:00:08.4330000 duration, 1142904 size, 2.29 MiB memory
BufferingManager.UpdateBuffering: 99.85%, 00:00:08.9861222 duration, 1212383 size, 2.33 MiB memory
BufferingManager.UpdateBuffering done buffering: 00:00:09 duration, 1216133 size, 2.35 MiB memory
  Video count 275 size 1106584 newest 16:13:05.2106000 oldest 16:12:56.2106000
  Audio count 131 size 109549 newest 16:13:05.2065000 oldest 16:12:56.1507111
The first line says that it has 7.93 seconds and about 1MB buffered. The two lines after the "done buffering" show 275 video packets totaling 1106584 bytes of data and 131 audio packets totaling 109549 of data.

You can "new" a DefaultBufferingPolicy, set the properties you want to change, and pass it to IMediaStreamFascade.SetParameters() before you start playing.
Coordinator
Feb 28, 2014 at 10:35 AM
There have been some changes in the buffering code to use different limits when buffering before starting playback and when buffering after running out of packets. You may find this helpful for reducing your seek times.

I had thought of add such code for a while, but it seemed like it would need a bit of messy surgery to let BufferingManager know what caused things to start buffering. It didn't occur to me until recently that since flush is only called when starting playback, BufferingManager already has all the information it needs.

Another thing you may want to do is to run with SimulatedPlayer and Fiddler (the SimulatedPlayers app.config is configured to always use Fiddler--or, at least, whatever proxy is on http://localhost:8888. This usually makes it much easier to see what is taking so much time (e.g., playlists that take a long time to download, segments that keep needing retries, or--sometimes--daft behavior on the part of the phonesm code).
Mar 4, 2014 at 1:17 PM
Edited Mar 4, 2014 at 3:55 PM
Hi, thanks for the feedback again. I'm afraid the buffering will not be the key in my case, but I will try to use Fiddler ass you suggested and try to figure out what's going on. I got some information that the streams might also lack some information ?? Don't know yet.

Edit: I deleted a part of this comment. I had some problem with full screen of resolution of 720x576 which did get rendered letterboxed in the mediaelement, no matter what i tried, but i just noticed that this seems to be a problem only in the emulator not on the real device. So just in case somebody would made the same mistake, always check the real device before starting to dig deeper ...
Coordinator
Mar 4, 2014 at 3:59 PM
My familiarity with the MediaElement tends to be focused around what not to do to make it angry...

Apart from not being able to get the .Processed event working, the Windows 8.1 version of MediaElement (well, MediaStreamSource) has been a joy to work with. No funny races, no blowing up at the slightest provocation, and no more CodecPrivateData. If we are lucky, a Win8.1ish MediaElement will have made it's way to Windows Phone 8.1 (no, I don't know and have not seen the WP 8.1 SDK). If not, perhaps it will at least have some bugs fixed (e.g., work item 15).

I wasn't clear from you said: Is the video displayed in the wrong aspect ratio or is it letterboxed (with the correct aspect ratio)?

Have you tried editing the values in question to see how the MediaElement responds? I mean, encode a bunch of streams with different settings and see what happens when you play them. If you can figure out something that helps, the CodecPrivateData code could be hacked to change the SPS to match what you need--assuming (as you asked) that the stuff in CodecPrivateData takes precedence over what is in the stream. Even if you don't care to do it the hard way (parse it, change the items in question, then regenerate it), one can override the code that returns the CodecPrivateData so that it uses a fixed string (the AAC code has settings for this).

And yes, the SPS is parsed just enough so that the code can figure out the height and width. There are four things passed to the OS, the VideoFourCC (fixed for H.264), the height, the width, and the CodecPrivateData (see TsMediaStreamSource.ConfigureVideoStream()). The same blob that is passed into the ParseSps method is passed to the code that build the CodedPrivateData string. The only mucking phonesm does with the bits is the RBSP decoding that happens when reading and the RBSP encoding used when generating CodecPrivateData (I don't have confirmation that is what it should be doing--the documentation for that sort of thing apparently consists of a vague background noise from crickets chirping--but it is that way now since it seems to work). Come to think of it, have you verified that the phonesm code isn't getting the height or width wrong and causing your problem?

In your first post, you said you had played with the height and width properties. Did you mean the ones on MediaElement or the ones sent to the decoder (MediaStreamAttributeKeys.Height and MediaStreamAttributeKeys.Width)?

Let me know what you find out.
Coordinator
Mar 4, 2014 at 4:05 PM
This may also be of interest, perhaps more so when it gets resolved:

"Multiple simultaneous seeks take longer to complete than they should" (work Item 37)
Mar 4, 2014 at 5:02 PM
Sorry for the fuss with my latest comment, i see i did not made the edit in time to save your time. As I wrote in my edit, the aspect ratio looks ok on the real device.
I just did not have the mobile device at hand that time and in the emulator the video was not displayed in the right aspect ratio. It was displayed letterboxed. I have checked later, the source did not contain the black matter (letterbox) in it.
Checked with http://www.codecian.com/ (it's a trial soft). And http://www.w6rz.net/h264_parse.zip is not bad for a quick h264 header parse check.

But anyway at least to answer the questions:
Come to think of it, have you verified that the phonesm code isn't getting the height or width wrong and causing your problem?
Yes I did check those parts of the code and it's correct.
Mar 5, 2014 at 9:14 AM
The last version of the source I have tried yesterday looks much better, the seek times are now much lower. Thanks a lot.
Coordinator
Mar 5, 2014 at 6:23 PM
Ok, good to hear that seek, aspect rations, and such are working for you.