watonomous.github.io
[ Software Division : High Level Data Fusion ]
Created by [ Rowan Dempster], last modified by [ Henry Wang] on Feb 10, 2020
High Level Data Fusion (HLDF)
We have labeled 2-D data from the cameras and 3-D data from the Lidars.
HLDF associates the two datas together to give the labeled 2-D data
depth.
IOU
Processing uses Intersection over Union (IoU) to determine which data to
associate with each other.
IoU value = Area of Overlap / Area of Union
IoU values is a range between 0 - 1
0 = No overlap
1 = Perfect overlap
[Projection Matrix]
3D bounding boxes gives position and dimensions in terms of meters.
2D bounding boxes gives position and dimensions in terms of pixels.
We use the projection matrix of the camera and lidar to convert from
meters to pixels.
The projection matrix is made up of the camera's intrinsic matrix and
the camera-lidar extrinsic matrix.
[
{.confluence-embedded-image
.confluence-external-resource}]
[X, Y, Z is the meter coordinates and u, v are the pixel coordinates (u
= x, v = y).]
[When you perform the matrix multiplication: projection_matrix *
meter_coordinates, the resulting pixel matrix has some value 's'
multiplied by it. eg) [su, sv, s]]
[So to get the pixel coordinates, divide the entire matrix by the bottom
value to get [u, v, 1].]
\
Multiple points can be converted at the same time by extending the width of the XYZ matrix and adding more points to them.
[Lidar/Camera Coordinates]
The lidar and the camera have different coordinates.
Camera's x is lidar's y, camera's y is lidar's -z, and camera's z
is lidar's -x.
For the coordinate transformation to work, they need to be in the same
coordinates.
So, when inputting lidar data to be converted, format it like this:
[
{.confluence-embedded-image
.confluence-external-resource}]
Timer
Why a timer?
Sometimes when we get a 3d bounding box, there is no 2d box to associate
it with. When that happens hldf just sends all the 3d boxes as unknown
What the timer does is if we get a 3d box, and we don't have the 2d,
wait a bit (like 0.1-0.2 sec) to see if 2d detections will come
Document generated by Confluence on Dec 10, 2021 04:02