usb_cam latency when using low fps

I am using the **usb_cam** driver package and **rqt_image_view** to view the image from a USB camera. The problem I'm facing is that the latency of the displayed image increases a lot when I drop the frame rate. At the default 30fps, there is no noticeable lag when viewing through rqt_image_view. But 30fps uses a lot of CPU and I only need about 5 fps. Unfortunately, when I change the framerate parameter in usb_cam to 5 fps, the latency jumps to around 1 second (eyeballed). All parameters are default values except the pixel_format which is yuyv. (I tried changing the io_method around but that had no effect.) Any thoughts on how I could lower the latency is appreciated!