Overview

The current method uses the robot to calibrate the sensors. The robot will need to hold a calibration tag and move through positions with the tag in the sensors field of view during the calibration process. The positions will need to be are set by the user. A Optionally, a calibration tag will also need to may be mounted as a stationary tag in the scene (not attached to a robot) in the field of view of the sensors in order for the system to be able to detect if the sensors have shifted out of calibration.

The cal.json file contains all the relevant information to calibrate and run the RapidSense system with multiple cameras.

Calibration Requirements

The requirements for calibration are:

• Aruco tag mounted off of the robot faceplate Static on the user created mount, attached to the robot. The details of Aruco tags can be found in this user manual under System Setup - Hardware Requirements - Calibration Tag Details.

• (Optional)Stationary Aruco tag placed in the scene which is easily clearly visible to the sensor(s)

• Calibration preset in RPC called “Calibration” with accurate TCP frame located at the center of the Aruco tag. More details in the cal.json file linked below

  • For calibrating, create a frame on the Aruco tag face centered with its X axis (Red) pointing to the right, Y (Green) pointing upwards and Z (Blue) coming out of the face. Something like this:

...

• cal.json file which has all the camera serial_numbers and it’s each sensor’s corresponding robot target positions position for calibration which have the Aruco tag on the robot in the sensor field of view

Setup

...

• markers.json file

...

• which has the Aruco marker’s ID, length, and name of the robot to which the marker was mounted.

• A home robot target position with the following naming convention: robotname_home.

• The calibration robot target positions present in cal.json should be connected on Roadmap to the home target in RPC project in order for the calibration to succeed.

Setup and Running Calibration Process Manually

RapidSense generates a default cal.json file upon running with all the detected sensors listed. It is important you have all the sensors you wish to use connected BEFORE starting RapidSense. If you want to add sensors later, you must remove your existing cal.json and restart RapidSense and reconfigure the file.

After the cal.json file has been generated, run the calibration_service.py script and go open a browser to localhost:9000/calibration to calibrate all the cameras after rapidsense and rtr_rapidsense_app and proxy applications are running . This or alternatively, you can also use the RapidSense Monitor interface UI (see below) to perform calibration. The calibration script will pull in information from the cal.json file and make move the robots move to the cal_their targets sequentially, extrinsically calibrating all corresponding cameras. The robot will then return to the home location and all the cameras will save the pose of the static marker stationary tag if placed in the scene. After a successful calibration, all the values will be automatically updated in the cal.json file.

Architecture

The diagram below gives a quick overview of the architecture of the calibration sequence:

...

The calibration ASCII command { 'topic': 'Calibrate', 'data': { 'serial_number' : serial_number, 'tcp_pose' : tcp_pose } } is used where the serial_number is the camera to be calibrated and the TCP_pose is for the associated robot with the tag mounted on it. These values are sent over the proxy to rapidsense where aruco detection algorithm finds the markers which gives us the aruco_to_camera transformation. The TCP_pose is requested when in the calibration preset so we get the world_to_aruco transform.

Using these two transforms, we can calculate the world_to_camera transform as:

World_to_camera = World_to_aruco * aruco_to_camera

After successful calibration, we get the pose of the static marker and save it in the file cal.json. We send out a status = 0 to appliance and also update the GetRapidSenseStatus as true for the is_calibrated for the corresponding sensor.

...

Further details of components can be found in the pages below:

...

Running Calibration from RapidSense Monitor UI

In the Settings page of the RapidSense Monitor UI, there is a button available which will initiate the calibration process. A warning message will pop up to warn a user to ensure the calibration tag is attached to the robot and that the robot will move once it is confirmed to calibrate.

...

Check the configuration

After calibration is completed successfully, confirm calibration succeeded by issuing a ScanScene command to show obstacles detected by RapidSense. Visually inspect that the objects in the volume match expectation. Unsuccessful calibration could result in the following.

...

If calibration fails, verify the tag(s) are properly mounted to the robots (i.e., their orientation is correct, etc).

Sample cal.json File for (3) Sensors

{
    "calibration": [
        {
            "serial_number": "xyz",
            "extrinsics": [
                1.0,
                0.0,
                0.0,
                0.0,
                0.0,
                1.0,
                0.0,
                0.0,
                0.0,
                0.0,
                1.0,
                0.0,
                0.0,
                0.0,
                0.0,
                1.0
            ],
            "last_calibrated": "2024-03-27 21:40:10.549000000",
            "marker": {
                "robot_mounted": {
                    "id": 8,
                    "cal_target": "xyz_cal"
                }
            }
        },
        {
            "serial_number": "abc",
            "extrinsics": [
                1.0,
                0.0,
                0.0,
                0.0,
                0.0,
                1.0,
                0.0,
                0.0,
                0.0,
                0.0,
                1.0,
                0.0,
                0.0,
                0.0,
                0.0,
                1.0
            ],
            "last_calibrated": "2024-03-27 21:40:15.218000000",
            "marker": {
                "robot_mounted": {
                    "id": 8,
                    "cal_target": "abc_cal"
                }
            }
        },
        {
            "serial_number": "123",
            "extrinsics": [
                1.0,
                0.0,
                0.0,
                0.0,
                0.0,
                1.0,
                0.0,
                0.0,
                0.0,
                0.0,
                1.0,
                0.0,
                0.0,
                0.0,
                0.0,
                1.0
            ],
            "last_calibrated": "2024-03-27 21:40:20.819000000",
            "marker": {
                "robot_mounted": {
                    "id": 8,
                    "cal_target": "123_cal"
                }
            }
        }
    ]
}

Sample markers.json File associated to the previous cal.json sample

{
    "markers": [
        {
            "id": 8,
            "length": 0.09,
            "robot": "Fanuc"
        }
    ]
}

Demo Video of Calibration

...