Code Documentation

Dataset I/O

class opensfm.dataset.DataSet(data_path)

Accessors to the main input and output data.

Data include input images, masks, and segmentation as well temporary data such as features and matches and the final reconstructions.

All data is stored inside a single folder with a specific subfolder structure.

It is possible to store data remotely or in different formats by subclassing this class and overloading its methods.

camera_models_overrides_exists()

Check if camera overrides file exists.

exif_overrides_exists()

Check if EXIF overrides file exists.

feature_type()

Return the type of local features (e.g. AKAZE, SURF, SIFT)

images()

List of file names of all images in the dataset.

load_camera_models()

Return camera models data

load_camera_models_overrides()

Load camera models overrides data.

load_combined_mask(image)

Combine binary mask with segmentation mask.

Return a mask that is non-zero only where the binary mask and the segmentation mask are non-zero.

load_detection(image)

Load image detection if it exists, otherwise return None.

load_exif(image)

Return extracted exif information, as dictionary, usually with fields:

Field	Type	Description
width	int	Width of image, in pixels
height	int	Height of image, in pixels
focal_prior	float	Focal length (real) / sensor width

Parameters:image – Image name, with extension (i.e. 123.jpg)

load_exif_overrides()

Load EXIF overrides data.

load_ground_control_points()

Load ground control points.

It uses reference_lla to convert the coordinates to topocentric reference frame.

load_image(image)

Load image pixels as numpy array.

The array is 3D, indexed by y-coord, x-coord, channel. The channels are in RGB order.

load_mask(image)

Load image mask if it exists, otherwise return None.

load_reference()

Load reference as a topocentric converter.

load_report(path)

Load a report file as a string.

load_segmentation(image)

Load image segmentation if it exists, otherwise return None.

load_segmentation_mask(image)

Build a mask from segmentation ignore values.

The mask is non-zero only for pixels with segmentation labels not in segmentation_ignore_values.

load_tracks_graph(filename=None)

Return graph (networkx data structure) of tracks

load_undistorted_combined_mask(image)

Combine undistorted binary mask with segmentation mask.

Return a mask that is non-zero only where the binary mask and the segmentation mask are non-zero.

load_undistorted_detection(image)

Load an undistorted image detection.

load_undistorted_image(image)

Load undistorted image pixels as a numpy array.

load_undistorted_mask(image)

Load undistorted mask pixels as a numpy array.

load_undistorted_segmentation(image)

Load an undistorted image segmentation.

load_undistorted_segmentation_mask(image)

Build a mask from the undistorted segmentation.

The mask is non-zero only for pixels with segmentation labels not in segmentation_ignore_values.

open_image_file(image)

Open image file and return file object.

profile_log()

Filename where to write timings.

save_camera_models(camera_models)

Save camera models data

save_camera_models_overrides(camera_models)

Save camera models overrides data

save_ply(reconstruction, filename=None, no_cameras=False, no_points=False)

Save a reconstruction in PLY format.

save_report(report_str, path)

Save report string to a file.

save_undistorted_detection(image, array)

Save the undistorted image detection.

save_undistorted_image(image, array)

Save undistorted image pixels.

save_undistorted_mask(image, array)

Save the undistorted image mask.

save_undistorted_segmentation(image, array)

Save the undistorted image segmentation.

segmentation_ignore_values(image)

List of label values to ignore.

Pixels with this labels values will be masked out and won’t be processed when extracting features or computing depthmaps.

undistorted_detection_exists(image)

Check if the undistorted detection file exists.

undistorted_mask_exists(image)

Check if the undistorted mask file exists.

undistorted_segmentation_exists(image)

Check if the undistorted segmentation file exists.

Reconstruction Types

Basic types for building a reconstruction.

class opensfm.types.BrownPerspectiveCamera

Define a perspective camera.

width

int – image width.

height

int – image height.

focal_x

real – estimated focal length for the X axis.

focal_y

real – estimated focal length for the Y axis.

c_x

real – estimated principal point X.

c_y

real – estimated principal point Y.

k1

real – estimated first radial distortion parameter.

k2

real – estimated second radial distortion parameter.

p1

real – estimated first tangential distortion parameter.

p2

real – estimated second tangential distortion parameter.

k3

real – estimated third radial distortion parameter.

focal_x_prior

real – prior focal length for the X axis.

focal_y_prior

real – prior focal length for the Y axis.

c_x_prior

real – prior principal point X.

c_y_prior

real – prior principal point Y.

k1_prior

real – prior first radial distortion parameter.

k2_prior

real – prior second radial distortion parameter.

p1_prior

real – prior first tangential distortion parameter.

p2_prior

real – prior second tangential distortion parameter.

k3_prior

real – prior third radial distortion parameter.

back_project(pixel, depth)

Project a pixel to a fronto-parallel plane at a given depth.

back_project_many(pixels, depths)

Project pixels to fronto-parallel planes at given depths.

get_K()

The calibration matrix.

get_K_in_pixel_coordinates(width=None, height=None)

The calibration matrix that maps to pixel coordinates.

Coordinates (0,0) correspond to the center of the top-left pixel, and (width - 1, height - 1) to the center of bottom-right pixel.

You can optionally pass the width and height of the image, in case you are using a resized versior of the original image.

pixel_bearing(pixel)

Unit vector pointing to the pixel viewing direction.

pixel_bearing_many(pixels)

Unit vector pointing to the pixel viewing directions.

pixel_bearings(pixels)

Deprecated: use pixel_bearing_many.

project(point)

Project a 3D point in camera coordinates to the image plane.

project_many(points)

Project 3D points in camera coordinates to the image plane.

class opensfm.types.Camera

Abstract camera class.

A camera is unique defined for its identification description (id), the projection type (projection_type) and its internal calibration parameters, which depend on the particular Camera sub-class.

id

str – camera description.

projection_type

str – projection type.

class opensfm.types.FisheyeCamera

Define a fisheye camera.

width

int – image width.

height

int – image height.

focal

real – estimated focal lenght.

k1

real – estimated first distortion parameter.

k2

real – estimated second distortion parameter.

focal_prior

real – prior focal lenght.

k1_prior

real – prior first distortion parameter.

k2_prior

real – prior second distortion parameter.

back_project(pixel, depth)

Project a pixel to a fronto-parallel plane at a given depth.

back_project_many(pixels, depths)

Project pixels to fronto-parallel planes at given depths.

get_K()

The calibration matrix.

get_K_in_pixel_coordinates(width=None, height=None)

The calibration matrix that maps to pixel coordinates.

Coordinates (0,0) correspond to the center of the top-left pixel, and (width - 1, height - 1) to the center of bottom-right pixel.

You can optionally pass the width and height of the image, in case you are using a resized versior of the original image.

pixel_bearing(pixel)

Unit vector pointing to the pixel viewing direction.

pixel_bearing_many(pixels)

Unit vector pointing to the pixel viewing directions.

pixel_bearings(pixels)

Deprecated: use pixel_bearing_many.

project(point)

Project a 3D point in camera coordinates to the image plane.

project_many(points)

Project 3D points in camera coordinates to the image plane.

class opensfm.types.GroundControlPointObservation

A ground control point observation.

lla

latitue, longitude and altitude

coordinates

x, y, z coordinates in topocentric reference frame

shot_id

the shot where the point is observed

shot_coordinates

2d coordinates of the observation

class opensfm.types.PerspectiveCamera

Define a perspective camera.

width

int – image width.

height

int – image height.

focal

real – estimated focal lenght.

k1

real – estimated first distortion parameter.

k2

real – estimated second distortion parameter.

focal_prior

real – prior focal lenght.

k1_prior

real – prior first distortion parameter.

k2_prior

real – prior second distortion parameter.

back_project(pixel, depth)

Project a pixel to a fronto-parallel plane at a given depth.

back_project_many(pixels, depths)

Project pixels to fronto-parallel planes at given depths.

get_K()

The calibration matrix.

get_K_in_pixel_coordinates(width=None, height=None)

The calibration matrix that maps to pixel coordinates.

Coordinates (0,0) correspond to the center of the top-left pixel, and (width - 1, height - 1) to the center of bottom-right pixel.

You can optionally pass the width and height of the image, in case you are using a resized versior of the original image.

pixel_bearing(pixel)

Unit vector pointing to the pixel viewing direction.

pixel_bearing_many(pixels)

Unit vectors pointing to the pixel viewing directions.

pixel_bearings(pixels)

Deprecated: use pixel_bearing_many.

project(point)

Project a 3D point in camera coordinates to the image plane.

project_many(points)

Project 3D points in camera coordinates to the image plane.

class opensfm.types.Point

Defines a 3D point.

id

int – identification number.

color

list(int) – list containing the RGB values.

coordinates

list(real) – list containing the 3D position.

reprojection_error

real – the reprojection error.

class opensfm.types.Pose(rotation=array([0., 0., 0.]), translation=array([0., 0., 0.]))

Defines the pose parameters of a camera.

The extrinsic parameters are defined by a 3x1 rotation vector which maps the camera rotation respect to the origin frame (rotation) and a 3x1 translation vector which maps the camera translation respect to the origin frame (translation).

rotation

vector – the rotation vector.

translation

vector – the rotation vector.

compose(other)

Get the composition of this pose with another.

composed = self * other

get_Rt()

Get pose as a 3x4 matrix (R|t).

get_origin()

The origin of the pose in world coordinates.

get_rotation_matrix()

Get rotation as a 3x3 matrix.

inverse()

Get the inverse of this pose.

rotation

Rotation in angle-axis format.

set_origin(origin)

Set the origin of the pose in world coordinates.

>>> pose = Pose()
>>> pose.rotation = np.array([0., 1., 2.])
>>> origin = [1., 2., 3.]
>>> pose.set_origin(origin)
>>> np.allclose(origin, pose.get_origin())
True

set_rotation_matrix(rotation_matrix, permissive=False)

Set rotation as a 3x3 matrix.

>>> pose = Pose()
>>> pose.rotation = np.array([0., 1., 2.])
>>> R = pose.get_rotation_matrix()
>>> pose.set_rotation_matrix(R)
>>> np.allclose(pose.rotation, [0., 1., 2.])
True

>>> pose.set_rotation_matrix([[3,-4, 1], [ 5, 3,-7], [-9, 2, 6]])
Traceback (most recent call last):
...
ValueError: Not orthogonal

>>> pose.set_rotation_matrix([[0, 0, 1], [-1, 0, 0], [0, 1, 0]])
Traceback (most recent call last):
...
ValueError: Determinant not 1

transform(point)

Transform a point from world to this pose coordinates.

transform_inverse(point)

Transform a point from this pose to world coordinates.

transform_inverse_many(points)

Transform points from this pose to world coordinates.

transform_many(points)

Transform points from world coordinates to this pose.

translation

Translation vector.

class opensfm.types.Reconstruction

Defines the reconstructed scene.

cameras

Dict(Camera) – List of cameras.

shots

Dict(Shot) – List of reconstructed shots.

points

Dict(Point) – List of reconstructed points.

add_camera(camera)

Add a camera in the list

Parameters:camera – The camera.

add_point(point)

Add a point in the list

Parameters:point – The point.

add_shot(shot)

Add a shot in the list

Parameters:shot – The shot.

get_camera(id)

Return a camera by id.

Returns:If exists returns the camera, otherwise None.

get_point(id)

Return a point by id.

Returns:If exists returns the point, otherwise None.

get_shot(id)

Return a shot by id.

Returns:If exists returns the shot, otherwise None.

class opensfm.types.Shot

Defines a shot in a reconstructed scene.

A shot here is refered as a unique view inside the scene defined by the image filename (id), the used camera with its refined internal parameters (camera), the fully camera pose respect to the scene origin frame (pose) and the GPS data obtained in the moment that the picture was taken (metadata).

id

str – picture filename.

camera

Camera – camera.

pose

Pose – extrinsic parameters.

metadata

ShotMetadata – GPS, compass, capture time, etc.

back_project(pixel, depth)

Project a pixel to a fronto-parallel plane at a given depth.

The plane is defined by z = depth in the shot reference frame.

back_project_many(pixels, depths)

Project pixels to fronto-parallel planes at given depths. The planes are defined by z = depth in the shot reference frame.

project(point)

Project a 3D point to the image plane.

project_many(points)

Project 3D points to the image plane.

viewing_direction()

The viewing direction of the shot.

That is the positive camera Z axis in world coordinates.

class opensfm.types.ShotMesh

Triangular mesh of points visible in a shot

vertices

(list of vectors) mesh vertices

faces

(list of triplets) triangles’ topology

class opensfm.types.ShotMetadata

Defines GPS data from a taken picture.

orientation

int – the exif orientation tag (1-8).

capture_time

real – the capture time.

gps_dop

real – the GPS dop.

gps_position

vector – the GPS position.

class opensfm.types.SphericalCamera

A spherical camera generating equirectangular projections.

width

int – image width.

height

int – image height.

pixel_bearing(pixel)

Unit vector pointing to the pixel viewing direction.

pixel_bearing_many(pixels)

Unit vector pointing to the pixel viewing directions.

pixel_bearings(pixels)

Deprecated: use pixel_bearing_many.

project(point)

Project a 3D point in camera coordinates to the image plane.

project_many(points)

Project 3D points in camera coordinates to the image plane.

Features

Tools to extract features.

opensfm.features.extract_features(color_image, config, mask=None)

Detect features in an image.

The type of feature detected is determined by the feature_type config option.

The coordinates of the detected points are returned in normalized image coordinates.

Returns:

• points: x, y, size and angle for each feature
• descriptors: the descriptor of each feature
• colors: the color of the center of each feature

Return type:tuple

opensfm.features.mask_and_normalize_features(points, desc, colors, width, height, mask=None)

Remove features outside the mask and normalize image coordinates.

opensfm.features.resized_image(image, config)

Resize image to feature_process_size.

opensfm.features.root_feature_surf(desc, l2_normalization=False, partial=False)

Experimental square root mapping of surf-like feature, only work for 64-dim surf now

Matching

opensfm.matching.match_lowe(index, f2, config)

Match features and apply Lowe’s ratio filter.

Parameters:

• index – flann index if the first image
• f2 – feature descriptors of the second image
• config – config parameters

opensfm.matching.match_lowe_bf(f1, f2, config)

Bruteforce matching and Lowe’s ratio filtering.

Parameters:

• f1 – feature descriptors of the first image
• f2 – feature descriptors of the second image
• config – config parameters

opensfm.matching.match_symmetric(fi, indexi, fj, indexj, config)

Match in both directions and keep consistent matches.

Parameters:

• fi – feature descriptors of the first image
• indexi – flann index if the first image
• fj – feature descriptors of the second image
• indexj – flann index of the second image
• config – config parameters

opensfm.matching.robust_match(p1, p2, camera1, camera2, matches, config)

Filter matches by fitting a geometric model.

If cameras are perspective without distortion, then the Fundamental matrix is used. Otherwise, we use the Essential matrix.

opensfm.matching.robust_match_calibrated(p1, p2, camera1, camera2, matches, config)

Filter matches by estimating the Essential matrix via RANSAC.

opensfm.matching.robust_match_fundamental(p1, p2, matches, config)

Filter matches by estimating the Fundamental matrix via RANSAC.

Incremental Reconstruction

Incremental reconstruction pipeline

class opensfm.reconstruction.ShouldBundle(data, reconstruction)

Helper to keep track of when to run bundle.

class opensfm.reconstruction.ShouldRetriangulate(data, reconstruction)

Helper to keep track of when to re-triangulate.

class opensfm.reconstruction.TrackTriangulator(graph, reconstruction)

Triangulate tracks in a reconstruction.

Caches shot origin and rotation matrix

triangulate(track, reproj_threshold, min_ray_angle_degrees)

Triangulate track and add point to reconstruction.

triangulate_dlt(track, reproj_threshold, min_ray_angle_degrees)

Triangulate track using DLT and add point to reconstruction.

opensfm.reconstruction.align_two_reconstruction(r1, r2, common_tracks, threshold)

Estimate similarity transform between two reconstructions.

opensfm.reconstruction.bootstrap_reconstruction(data, graph, im1, im2, p1, p2)

Start a reconstruction using two shots.

opensfm.reconstruction.bundle(graph, reconstruction, gcp, config)

Bundle adjust a reconstruction.

opensfm.reconstruction.bundle_local(graph, reconstruction, gcp, central_shot_id, config)

Bundle adjust the local neighborhood of a shot.

opensfm.reconstruction.bundle_single_view(graph, reconstruction, shot_id, config)

Bundle adjust a single camera.

opensfm.reconstruction.compute_image_pairs(track_dict, data)

All matched image pairs sorted by reconstructability.

opensfm.reconstruction.direct_shot_neighbors(graph, reconstruction, shot_ids, min_common_points, max_neighbors)

Reconstructed shots sharing reconstructed points with a shot set.

opensfm.reconstruction.get_image_metadata(data, image)

Get image metadata as a ShotMetadata object.

opensfm.reconstruction.grow_reconstruction(data, graph, reconstruction, images, gcp)

Incrementally add shots to an initial reconstruction.

opensfm.reconstruction.incremental_reconstruction(data, graph)

Run the entire incremental reconstruction pipeline.

opensfm.reconstruction.merge_reconstructions(reconstructions, config)

Greedily merge reconstructions with common tracks.

opensfm.reconstruction.merge_two_reconstructions(r1, r2, config, threshold=1)

Merge two reconstructions with common tracks IDs.

opensfm.reconstruction.paint_reconstruction(data, graph, reconstruction)

Set the color of the points from the color of the tracks.

opensfm.reconstruction.pairwise_reconstructability(common_tracks, rotation_inliers)

Likeliness of an image pair giving a good initial reconstruction.

opensfm.reconstruction.reconstructed_points_for_images(graph, reconstruction, images)

Number of reconstructed points visible on each image.

Returns:A list of (image, num_point) pairs sorted by decreasing number of points.

opensfm.reconstruction.remove_outliers(graph, reconstruction, config)

Remove points with large reprojection error.

opensfm.reconstruction.resect(graph, reconstruction, shot_id, camera, metadata, threshold, min_inliers)

Try resecting and adding a shot to the reconstruction.

Returns:True on success.

opensfm.reconstruction.retriangulate(graph, reconstruction, config)

Retrianguate all points

opensfm.reconstruction.shot_lla_and_compass(shot, reference)

Lat, lon, alt and compass of the reconstructed shot position.

opensfm.reconstruction.shot_neighborhood(graph, reconstruction, central_shot_id, radius, min_common_points, max_interior_size)

Reconstructed shots near a given shot.

Returns:

• interior: the list of shots at distance smaller than radius
• boundary: shots sharing at least on point with the interior

Return type:a tuple with interior and boundary

Central shot is at distance 0. Shots at distance n + 1 share at least min_common_points points with shots at distance n.

opensfm.reconstruction.triangulate_shot_features(graph, reconstruction, shot_id, config)

Reconstruct as many tracks seen in shot_id as possible.

opensfm.reconstruction.two_view_reconstruction(p1, p2, camera1, camera2, threshold)

Reconstruct two views using the 5-point method.

Parameters:

• p2 (p1,) – lists points in the images
• camera2 (camera1,) – Camera models
• threshold – reprojection error threshold

Returns:

rotation, translation and inlier list

opensfm.reconstruction.two_view_reconstruction_general(p1, p2, camera1, camera2, threshold)

Reconstruct two views from point correspondences.

These will try different reconstruction methods and return the results of the one with most inliers.

Parameters:

• p2 (p1,) – lists points in the images
• camera2 (camera1,) – Camera models
• threshold – reprojection error threshold

Returns:

rotation, translation and inlier list

opensfm.reconstruction.two_view_reconstruction_plane_based(p1, p2, camera1, camera2, threshold)

Reconstruct two views from point correspondences lying on a plane.

Parameters:

• p2 (p1,) – lists points in the images
• camera2 (camera1,) – Camera models
• threshold – reprojection error threshold

Returns:

rotation, translation and inlier list

opensfm.reconstruction.two_view_reconstruction_rotation_only(p1, p2, camera1, camera2, threshold)

Find rotation between two views from point correspondences.

Parameters:

• p2 (p1,) – lists points in the images
• camera2 (camera1,) – Camera models
• threshold – reprojection error threshold

Returns:

rotation and inlier list

Config

opensfm.config.default_config()

Return default configuration

opensfm.config.load_config(filepath)

Load config from a config.yaml filepath