This project has moved. For the latest updates, please go here.

id3 Tag Support

May 2, 2014 at 8:25 AM
Edited May 2, 2014 at 8:47 AM
Hi,

I need id3 support in the HLS stream for my project and I was wondering if this is supported?

The id3 information is provided in the video file as XML.

Greetings,
Mark Hijdra
Coordinator
May 2, 2014 at 10:04 AM
The MP3 parser identifies and skips ID3v2 tags; it does not actually do anything with them. If you need to do something with the data, then take a look at SM.Media/MP3/Mp3Parser.cs. The code that skips "_skip" bytes is where the data will be available:
if (_skip > 0)
{
    if (_skip >= length)
    {
        _skip -= length;
        return;
    }

    offset += _skip;
    length -= _skip;
    _skip = 0;
}
If you save the data to, say, a MemoryStream inside the "if (_skip > 0) { ... }" and then feed it to the ID3 tag library of your choice when _skip hits zero, then you should have the full ID3v2 tag.

If you just want to pass the ID3v2 tag on to the rest of the system, then comment out the bit that calls GetId3Length().

If you have MP3 data inside an MPEG-2 transport stream, you will need to set "UseParser" to true in SM.Media/MP3/Mp3StreamHandler.cs.

Something like this should work (some idiot checking on the tag's length before calling the MemoryStream constructor would be advisable):
    public sealed class Mp3Parser : AudioParserBase
    {
        bool _hasSeenValidFrames;
        int _skip;
        MemoryStream _id3Stream;

        public Mp3Parser(ITsPesPacketPool pesPacketPool, Action<IAudioFrameHeader> configurationHandler, Action<TsPesPacket> submitPacket)
            : base(new Mp3FrameHeader(), pesPacketPool, configurationHandler, submitPacket)
        { }

        public override void ProcessData(byte[] buffer, int offset, int length)
        {
            Debug.Assert(length > 0);
            Debug.Assert(offset + length <= buffer.Length);

            if (0 == _skip && 0 == _index && 0 == _startIndex)
            {
                // This will not find all ID3 headers if someone is playing
                // games with the read buffer size.  However, for any reasonable
                // buffer size, the first 10 bytes of the file should wind up here
                // in one block.

                var id3Length = GetId3Length(buffer, offset, length);

                if (id3Length.HasValue)
                {
                    _skip = id3Length.Value + 10;

                    _id3Stream = new MemoryStream(_skip);

                    Debug.WriteLine("Mp3Parser.ProcessData() ID3 detected, length {0}", _skip);
                }
            }

            if (_skip > 0)
            {
                if (null != _id3Stream)
                    _id3Stream.Write(buffer, offset, Math.Min(_skip, length));

                if (_skip >= length)
                {
                    _skip -= length;
                    return;
                }

                if (null != _id3Stream)
                {
                    // Do something with the binary ID3v2 data...
                    DoSomethingWithTheTag(_id3Stream.ToArray());
                }

                offset += _skip;
                length -= _skip;
                _skip = 0;
            }

            var endOffset = offset + length;
    ...
May 2, 2014 at 11:24 AM
Thank you for the fast reply!

Unfortunately this solution does not work for me because I work with Live Streams (HLS).
The metadata I want to access is added to the .ts files.

For more information about id3 in HLS: https://developer.apple.com/library/ios/documentation/AudioVideo/Conceptual/HTTP_Live_Streaming_Metadata_Spec/HTTP_Live_Streaming_Metadata_Spec.pdf

Is this supported or can you provide me the location where I can build in support for this?
The project solution is massive, so I would be glad to know in what files and methods I need to look.

Greetings,
Mark Hijdra
Coordinator
May 2, 2014 at 12:40 PM
Take a look at SM.Media/Pes/PesHandlerFactory. The TsStreamTypeContentTypes Dictionary maps from the TS' "stream_type" to an instance of the "ContentType" class. You will need to add a stream_type 0x15 that maps to a "ContentType" instance representing the metadata.

An instance of a PesStreamHandler is used for each stream, so you will need to add one for your metadata. I would suggest copying H262StreamHandler.

You will also need to add a stream handler factory for that content type and register it something like so (again, just copy H262StreamHandlerFactory and change the KnownContentTypes to return the ContentType for the metadata matching the one from above):
_mediaStreamFacade.Builder.RegisterType<YourMetadataStreamHandlerFactory>().As<IPesStreamFactoryInstance>().InstancePerLifetimeScope().PreserveExistingDefaults();
You may also need to muck with the ProgramStreamsHandler, or your stream may be filtered out. Take a look at SM.Media.BackgroundAudioStreamingAgent.AudioTrackStreamer for an example of a custom stream filter.

I would suggest your start with feeding the TsDump console app an example .TS segment make sure that phonesm parses things the way you expect. You should wind up with a file for each demuxed stream.

I can take a look if you PM me a short sample .TS segment. I don't have a lot of time to spend on this, but I think I can add the required stuff to SM.Media so that you don't need to carry local changes in your tree; that way, you would hopefully only need to do the above .RegisterType<>.
May 2, 2014 at 2:18 PM
Thanks!

Had to do some other stuff but Monday I'll take more time to look into this.
Coordinator
May 2, 2014 at 4:06 PM
I may make some changes before then. Looking at that code, it seems like the extra stream_type -> ContentType indirection serves no useful purpose other than to fit the PES stream factory stuff into the content service factory framework. Perhaps that is cute and artificially consistent with other parts of the system, but it adds complexity without contributing much else.
Coordinator
May 3, 2014 at 6:06 PM
I've checked in some changes to avoid the extra conversion to ContentType and to fix some other problems with streams that are neither audio nor video.

You should be able to do something like this:
    class TimedMetadataStreamHandler : PesStreamHandler
    {
        readonly ITsPesPacketPool _pesPacketPool;

        public TimedMetadataStreamHandler(ITsPesPacketPool pesPacketPool, uint pid, TsStreamType streamType)
            : base(pid, streamType)
        {
            if (null == pesPacketPool)
                throw new ArgumentNullException("pesPacketPool");

            _pesPacketPool = pesPacketPool;
        }

        public override IConfigurationSource Configurator
        {
            get { return null; }
        }

        public override void PacketHandler(TsPesPacket packet)
        {
            base.PacketHandler(packet);

            if (null != packet)
               _pesPacketPool.FreePesPacket(packet);
        }
    }

    class TimedMetadataStreamHandlerFactory : IPesStreamFactoryInstance
    {
        static readonly byte[] Types = { 0x15 };

        public ICollection<byte> SupportedStreamTypes { get { return Types; } }

        public PesStreamHandler Create(PesStreamParameters parameter)
        {
            return new TimedMetadataStreamHandler(parameter.PesPacketPool, parameter.Pid, parameter.StreamType);
        }
    }
and then feed it to AutoFac like this:
    var builder = (BuilderBase) _mediaStreamFacade.Builder;

    builder.ContainerBuilder.RegisterType<TimedMetadataStreamHandlerFactory>()
        .As<IPesStreamFactoryInstance>()
        .InstancePerLifetimeScope()
        .PreserveExistingDefaults();
or by customizing the DI container initialization code.

Be aware that end-of-stream is indicated by a null packet, so keep that in mind when mucking about in PacketHandler().

If you want to forward the packet to the buffer the way the audio and video packets are handled, then grab the parameter.NextHandler in the factory's "Create()" and pass it to the instance constructor (and then either call "_nextHandler(packet)" or "_pesPacketPool.FreePesPacket(packet)"; do not call both for the same packet).
May 5, 2014 at 11:49 AM
Edited May 5, 2014 at 11:57 AM
I don't know why but it this does not work for me.

In the TSDump Console application I managed to do something but I don't know how the same is implemented in the player project.
It's pretty quick and dirty but shows what I want.

PS: I am trying to get a sample .ts file from a collegae (with id3 tag) without any copyrights or something.

Look for this code in the Method below.
                var streamBuffer = stream.StreamSource as StreamBuffer;
                if (streamBuffer != null)
                {
                    if (streamBuffer.StreamType.StreamType == 0x15)
                    {
                        var value = Encoding.Default.GetString(packet.Buffer);
                        Console.WriteLine(value);
                    }
                }
        static bool DumpStream(IMediaParserMediaStream stream)
        {
            var streamSource = stream.StreamSource;

            if (null == streamSource)
                return false;

            Console.WriteLine("Stream " + stream.ConfigurationSource.Name);

            var sawData = false;

            var sb = new StringBuilder();

            for (; ; )
            {
                var packet = streamSource.GetNextSample();

                if (null == packet)
                {
                    if (streamSource.IsEof)
                        Console.WriteLine("EOF");

                    return sawData;
                }

                sawData = true;

                sb.AppendFormat("{0}/{1} {2} {3}", packet.PresentationTimestamp, packet.DecodeTimestamp, packet.Duration, packet.Length);
                sb.AppendLine();

                for (var i = 0; i < packet.Length; ++i)
                {
                    if (i > 0 && 0 == (i & 0x03))
                    {
                        if (0 == (i & 0x1f))
                            sb.AppendLine();
                        else
                            sb.Append(' ');
                    }

                    sb.Append(packet.Buffer[packet.Index + i].ToString("x2"));
                }

                var streamBuffer = stream.StreamSource as StreamBuffer;
                if (streamBuffer != null)
                {
                    if (streamBuffer.StreamType.StreamType == 0x15)
                    {
                        var value = Encoding.Default.GetString(packet.Buffer);
                        Console.WriteLine(value);
                    }
                }

                Console.WriteLine(sb);
                sb.Clear();

                streamSource.FreeSample(packet);
            }
        }
Coordinator
May 5, 2014 at 2:03 PM
I'm not clear on where you are putting that code.

For the player (assuming WP8), TsMediaStreamSource.Configure() would probably be the best place to pick up a the stream (assuming you have something like TimedMetadataStreamHandler calling "next handler" instead of freeing the packages. Win81 (and, when the NuGet vs Autofac stuff is sorted, WP8.1) would do pretty much the same in WinRtMediaStreamSource.

BTW, if you want all the streams written to disk by TsDump, then change PesStreamCopyHandler's ctor to look something like this:
    public PesStreamCopyHandler(uint pid, TsStreamType streamType, Action<TsPesPacket> nextHandler)
        : base(pid, streamType)
    {
        _nextHandler = nextHandler;

        var ext = streamType.FileExtension;
        if (string.IsNullOrWhiteSpace(ext))
            ext = "_" + streamType.StreamType.ToString("x2") + ".bin";

        _stream = File.Create(string.Format("TS_PID{0}{1}", pid, ext));
    }
The demuxed files should be the same as from demuxing with ffmpeg.
May 5, 2014 at 2:32 PM
Excuse me for being not clear, the code I added to the TsDump project was inside the 'MediaDumpBase.cs' file > DumpStream (method).
This was only to test the file quicker.

I used your TimedMetadataStreamHandler and TimedMetadataStreamHandlerFactory code and put the
builder.RegisterType<TimedMetadataStreamHandlerFactory>().As<IPesStreamFactoryInstance>().InstancePerLifetimeScope().PreserveExistingDefaults();
inside SM.Media.Builder (project) > SmMediaModule.cs > Load

after the other HandlerFactories
            builder.RegisterType<AacStreamHandlerFactory>().As<IPesStreamFactoryInstance>().InstancePerLifetimeScope().PreserveExistingDefaults();
            builder.RegisterType<Ac3StreamHandlerFactory>().As<IPesStreamFactoryInstance>().InstancePerLifetimeScope().PreserveExistingDefaults();
            builder.RegisterType<H262StreamHandlerFactory>().As<IPesStreamFactoryInstance>().InstancePerLifetimeScope().PreserveExistingDefaults();
            builder.RegisterType<H264StreamHandlerFactory>().As<IPesStreamFactoryInstance>().InstancePerLifetimeScope().PreserveExistingDefaults();
            builder.RegisterType<Mp3StreamHandlerFactory>().As<IPesStreamFactoryInstance>().InstancePerLifetimeScope().PreserveExistingDefaults();
            builder.RegisterType<TimedMetadataStreamHandlerFactory>().As<IPesStreamFactoryInstance>().InstancePerLifetimeScope().PreserveExistingDefaults();
When playing my LiveStream it did not get into any code of TimedMetadataStreamHandler or TimedMetadataStreamHandlerFactory.

I want to get the 0x15 data from the .TS file (from a LiveStream).
Coordinator
May 5, 2014 at 3:08 PM
Edited May 5, 2014 at 3:35 PM
Ah, okay, I see.. TsDump overrides all the stream handlers with it's own "PesStreamCopyHandler"; it is what I use when debugging the TS demuxing code, so I want the raw demuxed streams without any interference by stream handlers. (This default is overridden by the last argument to the TsMediaParser ctor inside the MediaDump ctor.)

If you change the PesStreamCopyHandler constructor as I suggested above and run TsDump on your stream, then you will probably find three new files in same directory as TsDump.exe. The one you would probably find of most interest would be called something along the lines of: "TS_PIDnnn_15.bin". It should start with an ID3 tag.
May 6, 2014 at 11:27 AM
I finaly found the problem, yay!

I had to adjust some stuff to make it trigger the TimedMetadataStreamHandlerFactory.

First I added a 'Metadata' to the StreamContents enum
        public enum StreamContents
        {
            Unknown = 0,
            Audio,
            Video,
            Metadata,
            Other,
            Private,
            Reserved
        }
and then I associated 0x15 with the StreamContents.Metadata
                new TsStreamType(0x15, StreamContents.Metadata, "Metadata carried in PES packets"),
finally I changed the TsMediaParser.cs > DefaultProgramStreamsHandler method to:
So that the StreamContents.Metadata wouldn't be set to '.BlockStream = true;'
        static void DefaultProgramStreamsHandler(IProgramStreams pss)
        {
            var hasAudio = false;
            var hasVideo = false;

            foreach (var stream in pss.Streams)
            {
                switch (stream.StreamType.Contents)
                {
                    case TsStreamType.StreamContents.Audio:
                        if (hasAudio)
                            stream.BlockStream = true;
                        else
                            hasAudio = true;
                        break;
                    case TsStreamType.StreamContents.Video:
                        if (hasVideo)
                            stream.BlockStream = true;
                        else
                            hasVideo = true;
                        break;
                    case TsStreamType.StreamContents.Metadata:
                        break;
                    default:
                        stream.BlockStream = true;
                        break;
                }
            }
        }
Now I only have to make some adjustments to trigger my mainpage with the byte array.

Any recommendations for doing that?
May 6, 2014 at 12:47 PM
Oke I was to fast, by not blocking the Metadata the videostream does not play anymore...
Back to work.
Coordinator
May 6, 2014 at 1:20 PM
Are you consuming the metadata packets in the StreamBuffer? If not, then the BufferingManager will eventually stop playback to prevent eating up too much memory.

I pushed some changes to the code on Saturday that keeps the configuration code from waiting for stream handlers that do not have a configurator. If you don't have those changes (or if I didn't get them right with the fake data I was using...), then the playback will wait forever for stream configuration to complete.

You can customize the stream filter without changing the default filter. See AudioTrackStreamer.cs for an example:
_mediaManagerParameters = new MediaManagerParameters
    {
        ProgramStreamsHandler =
            streams =>
            {
                var firstAudio = streams.Streams.First(x => x.StreamType.Contents == TsStreamType.StreamContents.Audio);

                var others = streams.Streams.Where(x => x.Pid != firstAudio.Pid);
                foreach (var programStream in others)
                    programStream.BlockStream = true;
            }
    };
and later in the same file,
    void InitializeMediaStream()
    {
        if (null != _mediaStreamFacade)
            return;

        _mediaStreamFacade = MediaStreamFacadeSettings.Parameters.Create(_httpClients);

        _mediaStreamFacade.SetParameter(_bufferingPolicy);

        _mediaStreamFacade.SetParameter(_mediaManagerParameters);

        _mediaStreamFacade.StateChange += TsMediaManagerOnStateChange;
    }
If you happen to know that your transport stream only contains streams you actually want, then an empty ProgramStreamsHandler should be fine (the default one permits the first video stream and the first audio streams it finds, and blocks everything else).
May 6, 2014 at 2:25 PM
I checked out the latest and greatest code on Monday.

I don't know about the streambuffer but I handle my code like this in the TimedMetadataStreamHandler
        public override void PacketHandler(TsPesPacket packet)
        {
            base.PacketHandler(packet);

            if (packet != null && packet.Buffer != null)
                MetadataHandler.Current.TriggerId3(this, packet.Buffer);

            if (null != packet)
                _pesPacketPool.FreePesPacket(packet);
        }
The MetadataHandler is a class thats basically a singleton with a event and method for safely triggering the event.
The event has custom EventArgs that contains the byte[] from the packet.
In my MainPage I add a Method to the eventhandler with some displaying code. (nothing to serious right now, pure testing)

For testing I use: http://devimages.apple.com/samplecode/adDemo/ad.m3u8

I do not get anything when the Metadata stream is not blocked.
It looks like when the TimedMetadataStreamHandler is connected then the video does nothing.
Coordinator
May 6, 2014 at 2:40 PM
That sounds like configuration is not completing. Could you try putting a breakpoint on TsMediaStreamSource.Configure() and see if the code gets there?
May 6, 2014 at 3:02 PM
It gets there.

Want the full console output?
May 6, 2014 at 3:17 PM
I send you a wetransfer with the entire solution.

For testing open the Samples.WP8 solution, then test the HlsView.WP8.
May 7, 2014 at 1:37 PM
I tried to find out why the Video and Audio does not play when the Metadata handler is hooked but I can't find anyting...

Do you know what is going wrong?
Coordinator
May 7, 2014 at 3:56 PM
There are a couple of things going on--perhaps "going wrong" would be more descriptive. The example stream has #EXT-X-DISCONTINUITIY tags, but since all the streams appear to be compatible (same audio and video formats), the timestamp filter should sort it out. Unfortunately, TsTimestamp is having trouble because the code that measures packet durations is, in turn, having trouble dealing with some odd AAC frames (some with Level = 2 and plenty of leading and trailing garbage data). I'm not sure if the AAC gunk is really in the source segments or something going wrong in phonesm, regardless, the audio handling code should not be getting confused by that kind of data.

This helps, but is not sufficient (the most important part is the "return false;" that becomes "continue;"):
 Source/Libraries/SM.Media/TsTimestamp.cs | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)

diff --git a/Source/Libraries/SM.Media/TsTimestamp.cs b/Source/Libraries/SM.Media/TsTimestamp.cs
index f56bcf2..44c4cff 100644
--- a/Source/Libraries/SM.Media/TsTimestamp.cs
+++ b/Source/Libraries/SM.Media/TsTimestamp.cs
@@ -110,9 +110,9 @@ namespace SM.Media
                 foreach (var state in _packetsStates)
                 {
                     if (state.Packets.Count <= 0)
-                        return false;
+                        continue;
 
-                    if (!state.Duration.HasValue)
+                    if (!state.Duration.HasValue || !state.PresentationTimestamp.HasValue)
                         continue;
 
                     var packet = state.Packets.First();
@@ -122,7 +122,7 @@ namespace SM.Media
 
                     var actualPts = packet.PresentationTimestamp - _timestampOffset.Value;
 
-                    var expectedPts = state.PresentationTimestamp + state.Duration.Value;
+                    var expectedPts = state.PresentationTimestamp.Value + state.Duration.Value;
 
                     var error = actualPts - expectedPts;
 
@@ -138,9 +138,6 @@ namespace SM.Media
                 }
             }
 
-            if (!_timestampOffset.HasValue)
-                return false;
-
             if (_timestampOffset != TimeSpan.Zero)
                 AdjustTimestamps(_timestampOffset.Value);
 
@@ -207,9 +204,9 @@ namespace SM.Media
             public TimeSpan? DecodeTimestamp;
             public TimeSpan? Duration;
             public Func<TsPesPacket, TimeSpan?> GetDuration;
+            public bool IsMedia;
             public ICollection<TsPesPacket> Packets;
             public TimeSpan? PresentationTimestamp;
-            public bool IsMedia;
         }
 
         #endregion
That change should get pushed in a bit, but first I need to figure out what is going on with the audio and I need to verify that TsTimestamp isn't letting the timestamps jump all over the place.

You'll also save yourself some cross-thread problems if you do this in MainPage.xaml.cs ID3 tag event handler:
   var task = Dispatcher.DispatchAsync(() => { CountBox.Text = count.ToString(CultureInfo.InvariantCulture); });

   TaskCollector.Default.Add(task, "OnI33TagTrigger");
May 8, 2014 at 9:19 AM
Thanks, this 'solves' my problem for now.

Now the Video, Audio is playing as well as getting the Metadata.

I don't think that the stream is the problem because it plays nicely when I block the Metadata stream.
My HLS Live Stream (that I can't share due to rights & stuff) had the same results as that example stream I provided to you.

Although the Video now works, when metadata gets handled the Video is stuttering even when I removed the code that triggers my MainPage. (when Metadata is disabled it's not stuttering, maybe same problem as the Video/Audio issue?)
Coordinator
May 8, 2014 at 12:31 PM
If you don't have discontinuity tags, try setting TsTimestamp.EnableDiscontinutityFilter to false. It may help with the the stuttering (I suspect TsTimestamp is getting confused by the metadata stream, causing the timestamps to jump around).
May 8, 2014 at 12:56 PM
I'll try that, thanks.

I am testing again and now it does not stutter on metadata.
Maybe it was coincidence that it was stuttering when receiving metadata.

The stuttering occasionally happens, maybe it has to do something with buffering or CPU load.
May 9, 2014 at 7:15 AM
I found out what is causing the stutter, pretty obvious but didn't notice it before.

The HLS player isn't switching between bitrates when it can't load the next video segment fast enough.
As for low-end devices it has lots of problems with the highest bitrate.

Is the above fix going to be final or just as a quick fix?
Coordinator
May 9, 2014 at 12:23 PM
You can set the bitrate before playback starts, but it will not adapt during playback (see the second point under "LImitations" on the main project page. The default implementation just picks whatever is first in the master playlist file. It can be changed by setting HlsPlaylistSegmentManagerPolicy.SelectSubProgram (see Selecting the Stream).

This will get fixed at some point, but it will require re-architecting how the processing pipeline works. (It is for the same reason that #EXT-X-DISCONTINUITY isn't handled properly.)