Streaming OpenCV frames 15 October 2018 on Krystian's Keep

If you’ve ever played with robotics and OpenCV you’d probably wondered how to display frames, which are processed on your robot, on your computer. Well, the solution may be to stream your video over the network.

One of the advantages of this approach is that you can annotate the video on your robot before sending it. That means you can, for example, mark targets that you are tracking, display FPS count and much more.

My choice was to send the video using a UDP socket because it is faster and more convenient than TCP as it’s connectionless. Also, it is the preferred choice for streaming real-time video. Below there is a simple example program, which shows how to send data through an UDP soket using Boost’s Asio library.

// This program sends a message
// to udp://127.0.0.1:5001
// and exits.

#include <boost/asio.hpp>
#include <iostream>

namespace ip = boost::asio::ip;

int main(int argc, char *argv[]) {

    // try changing these parameters and see what happens
    std::string addr = "127.0.0.1";
    std::string port = "5001";
    std::string message = "this message will be sent\n";

    // set up the socket
    boost::asio::io_context io_context;
    ip::udp::socket socket(io_context, ip::udp::endpoint(ip::udp::v4(), 0));
    ip::udp::resolver resolver(io_context);
    ip::udp::resolver::results_type endpoints = resolver.resolve(ip::udp::v4(), addr, port);

    // send the message
    socket.send_to(boost::asio::buffer(std::string(message)), (const ip::basic_endpoint<ip::udp> &) *endpoints.begin());

    return 0;
}

When I launched netcat to listen on port 5001 before running the program above I received the message.

❯ nc -u -l -p 5001 127.0.0.1
this message will be sent

Now that we have learned how to send data through the network, we are ready to stream video frames, right? Not so fast. Sending raw frames would be extremally inefficient. One frame of a BGR24 640x480 video takes up 3 B * 640 * 480 = 921600 B = 900 KB. However, there is a method to send lighter frames. We can encode our images as JPG and stream them, so that we are actually streaming a MJPEG video. Looking at the size of several JPG encoded frames i found that they may take up as little as 93 KB (the size varies). Thats an order of magnitude smaller than a raw one.

We can use the imencode() function from OpenCV to encode our frames.

There is one more problem. On my machine whenever i wanted to send an encoded frame I got the Message too long error. I solved this problem by splitting the data to chunks and then sending those chunks separatly. Below is the final, working code.

// TODO: annotate the video or do an algorithm

#include <boost/asio.hpp>
#include <iostream>
#include <opencv2/opencv.hpp>

namespace ip = boost::asio::ip;

int main(int argc, char *argv[]) {
    const int chunk_size = 64000;

    std::string addr = "127.0.0.1";
    std::string port = "5001";

    boost::asio::io_context io_context;
    ip::udp::socket socket(io_context, ip::udp::endpoint(ip::udp::v4(), 0));
    ip::udp::resolver resolver(io_context);
    ip::udp::resolver::results_type endpoints = resolver.resolve(ip::udp::v4(), addr, port);

    cv::Mat frame;
    cv::VideoCapture capture(0);

    while (true) {
        capture >> frame;

        std::vector<uchar> buf;
        cv::imencode(".jpg", frame, buf, std::vector<int>());

        for (auto it = buf.begin(); it < buf.end(); it += chunk_size) {
            auto end = it + chunk_size;
            if (end >= buf.end()) {
                end = buf.end();
            }
            socket.send_to(
                    boost::asio::buffer(std::string(it, end)),
                    (const ip::basic_endpoint<ip::udp> &) *endpoints.begin()
            );
        }
    }
}

There is still room for improvement though. The video could be compressed using interframes. H.264 is probably the most popular choice.

Have a comment on one of my posts? Start a discussion in my public inbox by sending an email to ~krystianch/public-inbox@lists.sr.ht [mailing list etiquette]

Articles from blogs I read

Status update, January 2024

Hi! This month has been pretty hectic due to the SourceHut network outage. We’ve all in the staff team invested a lot of time and energy to minimize the downtime as much as possible. Thankfully things have settled down now, there are still a lot of follow-up…

via emersion January 21, 2024

SourceHut network outage post-mortem

It’s been a busy couple of weeks here at SourceHut. At the time of writing, we have restored SourceHut to full service following an unprecedented 170 hour outage, and while we still have numerous kinks to sort out following an unscheduled emergency migration…

via Blogs on Sourcehut January 19, 2024

Dlaczego o tym się nie mówi? – recenzja książki Emilie Pine

Dziś polecajka – książka „O tym się nie mówi” Emilie Pine. Jest to lektura, która wzbudziła we mnie dużo emocji i wywarła na mnie bardzo dobre wrażenie. Eseje są napisane w nietuzinkowy i zaskakujący sposób oraz pomimo trudnych tematów, w lekkim stylu. Two…

via Powiedziała, co wiedziała January 4, 2024

Generated by openring