If you’ve ever played with robotics and OpenCV you’d probably wondered how to display frames, which are processed on your robot, on your computer. Well, the solution may be to stream your video over the network.
One of the advantages of this approach is that you can annotate the video on your robot before sending it. That means you can, for example, mark targets that you are tracking, display FPS count and much more.
My choice was to send the video using a UDP socket because it is faster and more convenient than TCP as it’s connectionless. Also, it is the preferred choice for streaming real-time video. Below there is a simple example program, which shows how to send data through an UDP soket using Boost’s Asio library.
// This program sends a message
// to udp://127.0.0.1:5001
// and exits.
#include <boost/asio.hpp>
#include <iostream>
namespace ip = boost::asio::ip;
int main(int argc, char *argv[]) {
// try changing these parameters and see what happens
std::string addr = "127.0.0.1";
std::string port = "5001";
std::string message = "this message will be sent\n";
// set up the socket
boost::asio::io_context io_context;
ip::udp::socket socket(io_context, ip::udp::endpoint(ip::udp::v4(), 0));
ip::udp::resolver resolver(io_context);
ip::udp::resolver::results_type endpoints = resolver.resolve(ip::udp::v4(), addr, port);
// send the message
socket.send_to(boost::asio::buffer(std::string(message)), (const ip::basic_endpoint<ip::udp> &) *endpoints.begin());
return 0;
}
When I launched netcat to listen on port 5001 before running the program above I received the message.
❯ nc -u -l -p 5001 127.0.0.1
this message will be sent
Now that we have learned how to send data through the network, we are ready to stream video frames, right? Not so fast. Sending raw frames would be extremally inefficient. One frame of a BGR24 640x480 video takes up 3 B * 640 * 480 = 921600 B = 900 KB. However, there is a method to send lighter frames. We can encode our images as JPG and stream them, so that we are actually streaming a MJPEG video. Looking at the size of several JPG encoded frames i found that they may take up as little as 93 KB (the size varies). Thats an order of magnitude smaller than a raw one.
We can use the imencode() function from OpenCV to encode our frames.
There is one more problem. On my machine whenever i wanted to send an encoded frame I got the Message too long error. I solved this problem by splitting the data to chunks and then sending those chunks separatly. Below is the final, working code.
// TODO: annotate the video or do an algorithm
#include <boost/asio.hpp>
#include <iostream>
#include <opencv2/opencv.hpp>
namespace ip = boost::asio::ip;
int main(int argc, char *argv[]) {
const int chunk_size = 64000;
std::string addr = "127.0.0.1";
std::string port = "5001";
boost::asio::io_context io_context;
ip::udp::socket socket(io_context, ip::udp::endpoint(ip::udp::v4(), 0));
ip::udp::resolver resolver(io_context);
ip::udp::resolver::results_type endpoints = resolver.resolve(ip::udp::v4(), addr, port);
cv::Mat frame;
cv::VideoCapture capture(0);
while (true) {
capture >> frame;
std::vector<uchar> buf;
cv::imencode(".jpg", frame, buf, std::vector<int>());
for (auto it = buf.begin(); it < buf.end(); it += chunk_size) {
auto end = it + chunk_size;
if (end >= buf.end()) {
end = buf.end();
}
socket.send_to(
boost::asio::buffer(std::string(it, end)),
(const ip::basic_endpoint<ip::udp> &) *endpoints.begin()
);
}
}
}
There is still room for improvement though. The video could be compressed using interframes. H.264 is probably the most popular choice.