August 03, 2017

Converting Motion JPEG to WebRTC

By Anton Venema

One of the most common things we see people doing is creating custom audio and video sources and sinks.

Converting Motion JPEG to WebRTC

In this blog post, we are going to put together a custom video source that is fed by a live motion JPEG web stream. When we’re done, you should be able to drop the custom class into the LiveSwitch or IceLink .NET chat example and use it as the video source.

Getting started with a custom video source is simple:

  1. Create a class that extends VideoSource.
  2. Add a constructor that defines the format of the raised frames.
  3. Implement the Label property getter.
  4. Implement the DoStart method that activates the source thread.
  5. Implement the DoStop method that deactivates the source thread.

Here’s a simple skeleton implementation to get us started:

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
public class MotionJpegSource : VideoSource
{
    public MotionJpegSource()
        : base(VideoFormat.Bgr)
    { }
 
    public override string Label
    {
        get { return "Motion JPEG Source"; }
    }
 
    protected override Future<object> DoStart()
    {
        var promise = new Promise<object>();
        try
        {
            //TODO: activate the source thread
 
            promise.Resolve(null);
        }
        catch (Exception ex)
        {
            promise.Reject(ex);
        }
        return promise;
    }
 
    protected override Future<object> DoStop()
    {
        var promise = new Promise<object>();
        try
        {
            //TODO: deactivate the source thread
 
            promise.Resolve(null);
        }
        catch (Exception ex)
        {
            promise.Reject(ex);
        }
        return promise;
    }
}

 

Notice that the start and stop implementations are promise/future-based and can either run asynchronously or resolve immediately. The try/catch pattern above ensures that any exceptions encountered during activation or deactivation will reject the promise.

You could drop this class into your application as-is, but it wouldn’t do much.

We need to fill out the DoStart method so it actually does something interesting. Specifically, we want to start pulling down JPEG images from a network camera so we can raise them to the LiveSwitch or IceLink media pipeline by calling RaiseFrame.

Over at Channel 9, Brian Peek wrote an excellent class that consumes a motion JPEG stream and raises a Bitmap whenever a new frame is decoded. Rather than write our own motion JPEG processor from scratch, we’ll take a bit of a shortcut and make use of his MjpegDecoder.

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
public class MotionJpegSource : VideoSource
{
    public string Url { get; set; }
 
    private MjpegDecoder _Decoder;
 
    public MotionJpegSource(string url)
        : base(VideoFormat.Bgr)
    {
        Url = url;
    }
 
    public override string Label
    {
        get { return "Motion JPEG Source"; }
    }
 
    protected override Future<object> DoStart()
    {
        var promise = new Promise<object>();
        try
        {
            _Decoder = new MjpegDecoder();
            _Decoder.FrameReady += _Decoder_FrameReady;
            _Decoder.ParseStream(new Uri(Url));
 
            promise.Resolve(null);
        }
        catch (Exception ex)
        {
            promise.Reject(ex);
        }
        return promise;
    }
 
    protected override Future<object> DoStop()
    {
        var promise = new Promise<object>();
        try
        {
            if (_Decoder != null)
            {
                _Decoder.FrameReady -= _Decoder_FrameReady;
                _Decoder.StopStream();
                _Decoder = null;
            }
 
            promise.Resolve(null);
        }
        catch (Exception ex)
        {
            promise.Reject(ex);
        }
        return promise;
    }
 
    private void _Decoder_FrameReady(object sender, FrameReadyEventArgs e)
    {
        var buffer = ImageUtility.BitmapToBuffer(e.Bitmap);
        RaiseFrame(new VideoFrame(buffer));
        buffer.Free();
    }
}

 

In the DoStart implementation, we’re now initializing a new MjpegDecoder using the URL passed into the source constructor and attaching a handler to the FrameReady event. In the DoStop implementation, we simply detach our handler and stop the decoder.

When a frame is ready, we make use of ImageUtility to convert the Bitmap into a VideoBuffer before raising it as part of a VideoFrame. The VideoBuffer returned by ImageUtility is backed by a DataBuffer from the DataBufferPool, so after raising the frame, it is important to free (dereference) the buffer back to the pool.

That’s it! You now have a fully functioning motion JPEG source.

All that remains is to revise CreateVideoSource in your LocalMedia class to return a MotionJpegSource with the URL of your favourite motion JPEG stream:

 

1
2
3
4
protected override VideoSource CreateVideoSource()
{
    return new MotionJpegSource("http://...");
}

 

Happy coding!

LiveSwitch Whitepaper-2

Anton Venema

As Frozen Mountain’s CTO, Anton is one of the world’s foremost experts on RTC solutions, as well as the technical visionary and prime architect of our products, IceLink and WebSync, and our custom solutions. Anton is responsible for ensuring that Frozen Mountain’s products exceed the needs of today and predict the needs of tomorrow.

RSS Feed

Subscribe Now

Looking for Something in Particular?

    Recent posts