Data Format

The timestamps of the hdf5 data are fully synchronized, so we recommend using this format of data.

The base time represents the difference between the timestamps in the hdf5 data and the timestamps in the rosbag data. It is stored in the group '/base_timestamp'.

  
  $ h5ls -r 2024-07-15-14-53-45.hdf5/base_timestamp
  /time                    Dataset {1/Inf}
  
    

Event data is stored in the groups '/prophesee/event_cam_left' and '/prophesee/event_cam_right'. We provide the distortion coefficients and models from calibration.

  
  $ h5ls -r 2024-07-15-14-53-45.hdf5/prophesee/event_cam_left
  /calib                   Group
  /calib/T_to_left_bfs     Dataset {4, 4}
  /calib/camera_model      Dataset {SCALAR}
  /calib/distortion_coeffs Dataset {4}
  /calib/distortion_model  Dataset {SCALAR}
  /calib/intrinsics        Dataset {4}
  /calib/resolution        Dataset {2}
  /ms_map_idx              Dataset {16886}
  /p                       Dataset {44005484/Inf}
  /t                       Dataset {44005484/Inf}
  /x                       Dataset {44005484/Inf}
  /y                       Dataset {44005484/Inf}
  

• 'calib':
  • 'T_to_left_bfs': Transformation from the frame of the camera to the frame of the Prophesee left event camera.
  • 'resolution': (width, height) resolution.
  • 'camera_model/instrinsics': Camera models and coefficients used in Kalibr.
  • 'distortion_model/distortion_coeffs': Distortion model and coefficients used in Kalibr.
• 'ms_map_idx': Precomputed index for the events for each ms of the dataset.
• ('x', 'y', 't', 'p'): Event array with (x, y) coordinate, timestamp, and polarity.

RGB data is stored in the groups '/blackflys/left' and '/blackflys/right'.

  
  $ h5ls -r 2024-07-15-14-53-45.hdf5/blackflys/left
  /calib                   Group
  /calib/camera_model      Dataset {SCALAR}
  /calib/distortion_coeffs Dataset {4}
  /calib/distortion_model  Dataset {SCALAR}
  /calib/intrinsics        Dataset {4}
  /calib/left_exposure_time Dataset {1}
  /calib/resolution        Dataset {2}
  /data                    Dataset {338/Inf, 1200, 1920, 3}
  /prophesee_event_cam_left_ms_map_idx Dataset {338}
  /ts                      Dataset {338/Inf}
  

• 'calib': Camera calibration information. Please refer to the Event Data section for more information.
• 'data': Image data in (n, x, y, c) where n is the image index, (x,y) are the pixel coordinates, and c is the channel (RGB).
• 'prophesee_event_cam_left_ms_map_idx': Mapping to the index of event camera data for each image.
• 'ts': Timestamp of the image.

GNSS/INS data is stored in the group '/integratedNavigation/data'.

  
  $ h5ls -r 2024-07-15-14-53-45.hdf5/integratedNavigation/data
  /attitude                Dataset {1734/Inf, 3}
  /calib                   Group
  /calib/T_to_lidar        Dataset {4, 4}
  /position                Dataset {1734/Inf, 3}
  /ts                      Dataset {1734/Inf}
  /velocity                Dataset {1734/Inf, 3}
  
    

• 'calib/T_to_lidar': Transformation from the frame of the GNSS/INS to the frame of the LiDAR.
• 'attitude': Attitude information represents the orientation of the GNSS/INS system in the world coordinate system.
• 'position': Position information represents the location of the GNSS/INS system in the world coordinate system.
• 'velocity': Velocity information represents the speed of the GNSS/INS system in the world coordinate system."
• 'ts': Timestamp of the GNSS/INS data.

LiDAR data is stored in the group '/livox/lidar'.

  
  $ h5ls -r 2024-07-15-14-53-45.hdf5/livox/lidar
  /calib                   Group
  /calib/T_to_left_cam     Dataset {4, 4}
  /calib/T_to_right_cam    Dataset {4, 4}
  /ms_map_idx              Dataset {16856}
  /points                  Group
  /points/t                Dataset {7847346/Inf}
  /points/x                Dataset {7847346/Inf}
  /points/y                Dataset {7847346/Inf}
  /points/z                Dataset {7847346/Inf}
  
    

• 'calib':
• 'T_to_left_cam': Transformation from the frame of the LiDAR to the left RGB camera.
• 'T_to_right_cam': Transformation from the frame of the LiDAR to the right RGB camera.
• ('points/x', 'points/y', 'points/z', 'points/p'): Points array with (x, y, z) coordinate and timestamp.
• ms_map_idx: Precomputed index for the points for each ms of the dataset.

IMU data is stored in the group '/livox/imu'.

  
  $ h5ls -r 2024-07-15-14-53-45.hdf5/livox/imu
  /angular_vel             Dataset {3521/Inf, 3}
  /calib                   Group
  /calib/T_to_left_cam     Dataset {4, 4}
  /calib/T_to_right_cam    Dataset {4, 4}
  /linear_accel            Dataset {3521/Inf, 3}
  /ts                      Dataset {3521/Inf}
  
    

• 'calib':
• 'T_to_left_cam': Transformation from the frame of the IMU to the left RGB camera.
• 'T_to_right_cam': Transformation from the frame of the IMU to the right RGB camera.
• 'angular_vel': Angular velocity provided by the IMU.
• 'linear_accel': Linear acceleration provided by the IMU.
• 'ts': Timestamp for the IMU samples.

The timestamps in the rosbag data from the multi-sensor platform are synchronized, except for the event data. We recommend using the unpacked hdf5 file mentioned above.

• '/event_cam_left/events' (event_camera_msgs/EventPacket): Events from the left event camera.
• '/event_cam_right/events' (event_camera_msgs/EventPacket): Events from the right event camera.
• '/stereo/left/image_color/compressed' (sensor_msgs/CompressedImage): Color images from the left RGB camera.
• '/stereo/right/image_color/compressed' (sensor_msgs/CompressedImage): Color images from the right RGB camera.
• '/integratedNavigation/data' (integrated_navigation/integratedNavigationMsgx1): Pose from GNSS/INS (UG005 X1).
• '/livox/imu' (sensor_msgs/Imu): Acceleration and angular velocity from IMU.
• '/livox/lidar' (livox_ros_driver2/CustomMsg): Point clouds from the LiDAR.

The data format of the rosbag from the small-scale testbed is as follows:

• '/capture_node/events' (dv_ros_msgs/EventArray): Events from the Inivation DVXplorer camera.
• '/image_raw' (sensor_msgs/Image): Color images from the RGB camera.

The format of the ground truth TTC (gt_ttc.csv files) is as follows:

  
  gt_ttc.csv:
  Index     timestamp (s)      distance (m)      velocity (m/s)      ttc (s)
    
  

• 'Index': The index corresponding to the TTC.
• 'timestamp': The timestamp corresponding to the TTC.
• 'distance': Relative distance between the host vehicle and the target object.
• 'velocity': Relative velocity between the host vehicle and the target object.
• 'ttc': The value of the ground truth TTC.

If you want to submit the results of your algorithm to the leaderboard, please follow the format below:

  
  results.txt:
  Index     timestamp (s)      ttc (s)     cost time(s)  
  

• 'Index': The index corresponding to the TTC.
• 'timestamp': The timestamp corresponding to the TTC.
• 'ttc': The value of the estimated TTC.
• 'cost time': The time consumed for a single TTC estimation.