OpenCV VideoCapture lag due to the capture buffer

Question

I am capturing video through a webcam which gives a mjpeg stream. I did the video capture in a worker thread. I start the capture like this:

const std::string videoStreamAddress = "http://192.168.1.173:80/live/0/mjpeg.jpg?x.mjpeg";
qDebug() << "start";
cap.open(videoStreamAddress);
qDebug() << "really started";
cap.set(CV_CAP_PROP_FRAME_WIDTH, 720);
cap.set(CV_CAP_PROP_FRAME_HEIGHT, 576);

the camera is feeding the stream at 20fps. But if I did the reading in 20fps like this:

if (!cap.isOpened()) return;

        Mat frame;
        cap >> frame; // get a new frame from camera
        mutex.lock();

        m_imageFrame = frame;
        mutex.unlock();

Then there is a 3+ seconds lag. The reason is that the captured video is first stored in a buffer.When I first start the camera, the buffer is accumulated but I did not read the frames out. So If I read from the buffer it always gives me the old frames. The only solutions I have now is to read the buffer at 30fps so it will clean the buffer quickly and there's no more serious lag.

Is there any other possible solution so that I could clean/flush the buffer manually each time I start the camera?

Answer 1

OpenCV Solution

According to this source, you can set the buffersize of a cv::VideoCapture object.

cv::VideoCapture cap;
cap.set(CV_CAP_PROP_BUFFERSIZE, 3); // internal buffer will now store only 3 frames

// rest of your code...

There is an important limitation however:

CV_CAP_PROP_BUFFERSIZE Amount of frames stored in internal buffer memory (note: only supported by DC1394 v 2.x backend currently)

Update from comments. In newer versions of OpenCV (3.4+), the limitation seems to be gone and the code uses scoped enumerations:

cv::VideoCapture cap;
cap.set(cv::CAP_PROP_BUFFERSIZE, 3);

---

Hackaround 1

If the solution does not work, take a look at this post that explains how to hack around the issue.

In a nutshell: the time needed to query a frame is measured; if it is too low, it means the frame was read from the buffer and can be discarded. Continue querying frames until the time measured exceeds a certain limit. When this happens, the buffer was empty and the returned frame is up to date.

(The answer on the linked post shows: returning a frame from the buffer takes about 1/8th the time of returning an up to date frame. Your mileage may vary, of course!)

---

Hackaround 2

A different solution, inspired by this post, is to create a third thread that grabs frames continuously at high speed to keep the buffer empty. This thread should use the cv::VideoCapture.grab() to avoid overhead.

You could use a simple spin-lock to synchronize reading frames between the real worker thread and the third thread.

Answer 2

Guys this is pretty stupid and nasty solution, but accepted answer didn't helped me for some reasons. (Code in python but the essence pretty clear)

# vcap.set(cv2.CAP_PROP_BUFFERSIZE, 1)
data = np.zeros((1140, 2560))
image = plt.imshow(data)

while True:
    vcap = cv2.VideoCapture("rtsp://admin:@192.168.3.231")
    ret, frame = vcap.read()
    image.set_data(frame)
    plt.pause(0.5) # any other consuming operation
    vcap.release()

Answer 3

An implementation of Hackaround 2 in Maarten's answer using Python. It starts a thread and keeps the latest frame from camera.read() as a class attribute. A similar strategy can be done in c++

import threading
import cv2

# Define the thread that will continuously pull frames from the camera
class CameraBufferCleanerThread(threading.Thread):
    def __init__(self, camera, name='camera-buffer-cleaner-thread'):
        self.camera = camera
        self.last_frame = None
        super(CameraBufferCleanerThread, self).__init__(name=name)
        self.start()

    def run(self):
        while True:
            ret, self.last_frame = self.camera.read()

# Start the camera
camera = cv2.VideoCapture(0)

# Start the cleaning thread
cam_cleaner = CameraBufferCleanerThread(camera)

# Use the frame whenever you want
while True:
    if cam_cleaner.last_frame is not None:
        cv2.imshow('The last frame', cam_cleaner.last_frame)
    cv2.waitKey(10)

Answer 4

You can make sure that grabbing the frame took a bit of time. It is quite simple to code, though a bit unreliable; potentially, this code could lead to a deadlock.

#include <chrono>
using clock = std::chrono::high_resolution_clock;
using duration_float = std::chrono::duration_cast<std::chrono::duration<float>>;
// ...
while (1) {
    TimePoint time_start = clock::now();
    camera.grab();
    if (duration_float(clock::now() - time_start).count() * camera.get(cv::CAP_PROP_FPS) > 0.5) {
        break;
    }
}
camera.retrieve(dst_image);

The code uses C++11.

Answer 5

There is an option to drop old buffers if you use a GStreamer pipeline. appsink drop=true option "Drops old buffers when the buffer queue is filled". In my particular case, there is a delay (from time to time) during the live stream processing, so it's needed to get the latest frame each VideoCapture.read call.

#include <chrono>
#include <thread>

#include <opencv4/opencv2/highgui.hpp>

static constexpr const char * const WINDOW = "1";

void video_test() {
    // It doesn't work properly without `drop=true` option
    cv::VideoCapture video("v4l2src device=/dev/video0 ! videoconvert ! videoscale ! videorate ! video/x-raw,width=640 ! appsink drop=true", cv::CAP_GSTREAMER);

    if(!video.isOpened()) {
        return;
    }

    cv::namedWindow(
        WINDOW,
        cv::WINDOW_GUI_NORMAL | cv::WINDOW_NORMAL | cv::WINDOW_KEEPRATIO
    );
    cv::resizeWindow(WINDOW, 700, 700);

    cv::Mat frame;
    const std::chrono::seconds sec(1);
    while(true) {
        if(!video.read(frame)) {
            break;
        }
        std::this_thread::sleep_for(sec);
        cv::imshow(WINDOW, frame);
        cv::waitKey(1);
    }
}

Answer 6

If you know the framerate of your camera you may use this information (i.e. 30 frames per second) to grab the frames until you got a lower frame rate. It works because if grab function become delayed (i.e. get more time to grab a frame than the standard frame rate), it means that you got every frame inside the buffer and opencv need to wait the next frame to come from camera.

while(True):
    prev_time=time.time()
    ref=vid.grab()
    if (time.time()-prev_time)>0.030:#something around 33 FPS
        break
ret,frame = vid.retrieve(ref)