This project has moved and is read-only. For the latest updates, please go here.

Switching Streams based on Network Bandwidth

Feb 13, 2014 at 6:35 AM
Hi henric,
          This code works wonderfully, but i need to switch between different bandwidth streams depending on current network bitrate,, how can i do this with this lib.
would appreciate your help.

menues
Coordinator
Feb 13, 2014 at 9:07 PM
The short answer is, "not easily."

The PlaylistSegmentManager, implementing ISegmentManager, is responsible for maintaining the list of URLs to read (read once for static playlists and re-read every once in a while for dynamic playlists). The CallbackReader iterates through that list using ISegmentManager's "IAsyncEnumerable<ISegment> Playlist" (where IAsyncEnumerable<T> is like IEnumerable<T> but with GetEnumerator() returning an IAsyncEnumerator<T> with "Task<bool> MoveNextAsync()" instead of an IEnumerator<T> with a "bool MoveNext()").

Modifying PlaylistSegmentManager or creating a new ISegmentManager, that keeps a list of alternate playlists should not be terribly difficult. Some mechanism of estimating the current bandwidth needs to be implemented (perhaps involving BufferingManager--and yes, I still dislike the name of that class; perhaps not). However, there are some complications.

Consider what needs to happen if the Media Playlists have more than one type of media segment. For example, if the lowest bandwidth playlist contains audio only .aac segments instead of audio/video .ts segments, then both the MediaParser handling the stream would need to change from a TsMediaParser to an AacMediaParser and the MediaElement would need to be reconfigured. Alternately, one could ignore the oddball Media Playlists or, I suppose, one might be able to feed the MediaElement a loop of fake video data when playing that Media Playlist.

The application may have more than one reader running if the program has an alternate audio track. This case is not currently supported by the code, but I did get it running once by hacking TsMediaManager to work with just one particular playlist. Having independent bandwidth estimates for the readers may be misleading since the code tries to encourage reading segments in bursts in order to let the radio idle and thereby reduce battery usage (IIRC, the default is to have about 12 seconds of hysteresis by enabling the reader if there is less than 12 seconds buffered and disabling the reader once 25 seconds are buffered; see IBufferingPolicy for details). As such, multiple readers may sometimes read simultaneously and sometimes not. (Perhaps this is something to be managed explicitly...?) When someone (i.e., me...) gets around to implementing captioning support, then there could be up to three readers, but the captioning data should be small enough to be irrelevant.

There may be alternate servers for the same stream. Some pieces of support for failover is already in the code, but not to a point where it is really useful. I'm not sure exactly how such failover logic should interact with any bandwidth management logic (i.e., if the connection is bad, should it try the other server or try a lower-bitrate stream from the same server). The most important place for this to happen is in the bowels of CallbackReader, but the current code does a couple of retries to the same URL, then gives up. For failover to work well, it would need to interact with some yet to be determined service to both let that service know about network problems and to get that service's guess at the best URL for retrieving the current segment for every retry.

The underlying operating system's web cache may also cause bandwidth estimate complications. One might want to customize HttpClients to have IHttpClients.CreateSegmentClient() return an HttpClient that uses a custom, sockets-based HttpMessageHandler instead of HttpClientHandler.

Note that a solution to the general case may allow neat features like being able to switch between an audio/video player and audio-only background audio mode.

If your application does not have to deal with alternate audio tracks, varying media segment types, and such, then I suspect that a modified PlaylistSegmentManager that estimates bandwidth by looking at how quickly stream segments are iterated and selects the Media Playlist to use for the next segment based on that estimate may be sufficient. This might take a while to recover if, say, just after 10 seconds segment starts reading and then the device switches from a good WiFi signal to a poor mobile connection. The CallbackReader only grabs a new ISegment when it has finished the previous one. You would also need a way to distinguish a slow network from the BufferingManager throttling the reader due to the user hitting "Pause".

This is all complicated by a design decision I made when first putting this thing together to try to minimize coupling by preferring "Func<X>" over "ISomeInterface" where possible. This has made it awkward to pass meta-information along with the flow of the media data (e.g., there is a "discontinuity here"). It has also led to some ugly code to keep a copy of an object reference in one place and pass a delegate to the code that really needs the object reference (e.g., see how TsDecoder deals with PES stream handlers). I've been spending the past week or so refactoring things to try to untangle some of this. I'll probably push something to CodePlex this week or early next week that rearranges some of the internals. If you need implement this but can wait a week or so, I would encourage you to wait until after I do that push.
Coordinator
Feb 13, 2014 at 9:35 PM
This is much simpler, if you don't mind the playback glitching and letting the user pick the bandwidth:
  1. Read the current position
  2. Stop playback
  3. Set the PlaylistSegmentManagerFactory.SelectSubProgram delegate to pick the right sub program ("Media Playlist", in HLS RFC terms)
  4. Start playback again at the old position (or old position minus a few seconds).
Feb 14, 2014 at 4:26 AM
Hi Henric really appreciate your reply ,

i will wait for your updates and check how can i do the switching and one another thing can we put in a logic to estimate bandwidth based on the time it takes to download the segment and scale stream (up or down) downloading next segment based on how the prev segment does buffering ( i don't know if this sounds ideal), would appreciate your thoughts on this.

again thanks a lot.

menues
Feb 14, 2014 at 4:31 AM
and adding to the prev post , assuming we are not changing the encoder/codec and transport container, which we have fixed it to certain type.
Coordinator
Feb 18, 2014 at 9:41 AM
I've pushed the set of changes I mentioned earlier. There may be some more changes in the next week or so, but not at the scale of this last bunch. (What I currently have in mind: TsMediaManager.ReaderPipeline needs refactoring, TsTimestamp should be integrated a bit more elegantly, and BufferingManager should probably create the StreamBuffers rather than going through the current convolutions) After that, the idea would be to limit things to bug fixes until I can get a new build together.

My guess is that if you measure the time CallbackReader.ReadSegment() takes, minus the time spent in _bufferPool.AllocateAsync(), and divide by the total number of bytes read, you will get a reasonable estimate of the bandwidth. You may want to look at few samples with an FIR, IIR, or what-not type estimator. The TCP socket will keep reading until its buffers are full and the platform's underlying web caching infrastructure may keep reading from the network while CallbackReader is blocked, so subtracting all the _bufferPool.AllocateAsync() time may cause the bandwidth to be overestimated since the network read will continue in parallel with the blocked call to allocate (and the next few reads after _bufferPool.AllocateAsync() would get satisfied as fast as memory can be copied from data already on the device).

Another way to handle this would be to measure the total time taken by CallbackReader.ReadSegment() and discard any samples where _bufferPool.AllocateAsync() blocked. I mean, something like this:
    var allocateAsync = _bufferPool.AllocateAsync(cancellationToken);

    if (!allocateAsync.IsCompleted)
        blocked = true;

    buffer = await allocateAsync.ConfigureAwait(false);
Then have ReadSegment() return Task<TimeSpan?> instead of Task, where the TimeSpan? would be null if "blocked" is true (and, yeah, that method should be called "ReadSegmentAsync()" not "ReadSegment()").
Feb 24, 2014 at 1:32 PM
Hi Henric,
          appreciate your help, i was going through your new code update, now the question is how do i associate new playlist for every url present for different bandwidth, tricky part is associate playlist with SegmentReaderManager, i have thought long and hard , this does not seem easier that i thought, perhaps you could help me with this, creating alternate playlist  and associating it with SegmentReaderManager.
Feb 24, 2014 at 1:39 PM
Edited Feb 24, 2014 at 2:28 PM
or do you think it makes sense to modify PlaylistSegmentManagerFactory to return Playlistsegmentmanager for every url, and pass the array to SegmentReaderManager, but i guess there would be separate CallbackReader for each of them that seems little inappropriate.
Coordinator
Feb 25, 2014 at 5:10 PM
I've moved some things around to deal with some problems handling Media Playlists when there is no Master Playlist (dynamic playlists should work better and it should stop the double-read on startup). The responsibility for reading the playlist has moved from PlaylistSegmentManager to IProgramStream, which is created by ProgramManager (in ProgramManagerBase). It also cleans things up a bit. I hope it has not made too much of a mess for you.

Since IProgramStream has responsibility for both reading the playlist and creating the ISegment collection, and is created by the very thing responsible for reading the Master Playlist, you should be able to crate a fancier implementation that switches requests from the client (PlaylistSegmentManager) to the most appropriate real IProgramStream (the one matching the bandwidth that you want). As for getting information about bandwidth to the new IProgramStream implementation, perhaps something like (completely off the top of my head):
   public interface INetworkStatus
   {
      void ReportBandwidth(Uri url, float bytesPerSecond);
      void ReportError(Uri url, ... status code or something...);

      float EstimateBandwidth(Uri url);
   }
Configure a singleton for INetworkStatus in the builder and have ProgramStream and PlaylistSegmentManager both get INetworkStatus instances. (Obviously some kind of "class NetworkStatus : INetworkStatus" could prove useful....)

Anyhow, the idea is to make it easier to move the reporting somewhere closer to where the network is actually being read, not just where you can count segments, without having to change the code that selects which Media Playlist to use.

Perhaps that interface should be split into two parts, one for reporting status and one for getting estimates...?