CineMPC: Controlling Camera Intrinsics and Extrinsics for Autonomous Cinematography

by   Pablo Pueyo, et al.

We present CineMPC, an algorithm to autonomously control a UAV-borne video camera in a nonlinear MPC loop. CineMPC controls both the position and orientation of the camera-the camera extrinsics-as well as the lens focal length, focal distance, and aperture-the camera intrinsics. While some existing solutions autonomously control the position and orientation of the camera, no existing solutions also control the intrinsic parameters, which are essential tools for rich cinematographic expression. The intrinsic parameters control the parts of the scene that are focused or blurred, and the viewers' perception of depth in the scene. Cinematographers commonly use the camera intrinsics to direct the viewers' attention through the use of focus, to convey suspense through telephoto views, inspire awe through wide-angle views, and generally to convey an emotionally rich viewing experience. Our algorithm can use any existing approach to detect the subjects in the scene, and tracks those subjects throughout a user-specified desired camera trajectory that includes camera intrinsics. CineMPC closes the loop from camera images to UAV trajectory in order to follow the desired relative trajectory as the subjects move through the scene. The cinematographer can use CineMPC to autonomously record scenes using the full array of cinematographic tools for artistic expression.


page 1

page 5


Closed-loop Control of Catalytic Janus Microrobots

We report a closed-loop control system for paramagnetic catalytically se...

Feature-less Stitching of Cylindrical Tunnel

Traditional image stitching algorithms use transforms such as homography...

Brain over Brawn – Using a Stereo Camera to Detect, Track and Intercept a Faster UAV by Reconstructing Its Trajectory

The work presented in this paper demonstrates our approach to intercepti...

Orientation Matters: 6-DoF Autonomous Camera Movement for Minimally Invasive Surgery

We propose a new method for six-degree-of-freedom (6-DoF) autonomous cam...

Trajectory Prediction Path Planning for an Object Intercepting UAV with a Mounted Depth Camera

A novel control software architecture using ROS C++ is introduced fo...

From a Bird's Eye View to See: Joint Camera and Subject Registration without the Camera Calibration

We tackle a new problem of multi-view camera and subject registration in...

Batteries, camera, action! Learning a semantic control space for expressive robot cinematography

Aerial vehicles are revolutionizing the way film-makers can capture shot...

Please sign up or login with your details

Forgot password? Click here to reset