Custom rendering sensor example

[adityapande-1995]

Desired behavior

We have a custom sensor example here : https://github.com/gazebosim/gz-sim/tree/ign-gazebo6/examples/plugin/custom_sensor_system , and it would be nice to have a custom rendering sensor example as well, that people could use as a template to write their own rendering sensors.

Implementation suggestion

We could leverage #1475, and use osrf/lrauv#213 and a guide. A start could be a camera sensor modified in a trivial way, like mirroring the image.

[Myzhar]

A stereo camera example could be useful, it was available in Gazebo Classic, but there's nothing similar in the new Gazebo/Ignition.

[tejalbarnwal]

Hey, I would like to work on this issue.
I was considering building a mirror camera sensor first, with the doppler velocity log sensor and camera sensor serving as references. After that, I could create a stereo sensor as a conventional example.

[JEller98]

Hello there, I'll get to work on implementing this. Will do some poking around to get a few ideas, and hopefully I and my team will be able to get something figured out!

[azeey]

Thanks @JEller98. I'll assign this to you. @tejalbarnwal Did you ever get a chance to work on this?

[tejalbarnwal]

Hi @azeey ,
I apologize for the inconvenience, but I haven't had the chance to work on it. I became busy with another matter, and regrettably, it slipped my mind.

[azeey]

@tejalbarnwal no worries. Just checking to make sure we are not duplicating our efforts. @JEller98 please go ahead!

[JEller98]

Hello there, my teammates had a few questions regarding this issue; we're fairly new to this, so apologies if any of these seem redundant:

What are the specifications of this request?
What is the expected functionality?
Is documentation available for the examples?
Should we follow the test cases created in TestSensorSystem.cc/TestSensorSystem.hh?
Where will this live in the codebase?
What are the expected inputs?
What are the expected outputs?
Who are the users?

Any information regarding these questions would be much appreciated.

[dakejahl]

I would love a custom rendering sensor example.

I'm working on implementing an optical flow plugin for PX4. I've got it working with a custom plugin by adding a "camera" sensor to my models sdf, saving the latest frame in the OnImage callback, and then calculating/publishing flow in the Update() function of the class inheriting from Sensor. I'd like to instead have my plugin inherit directly from CameraSensor. I'm not sure if this is even the right approach though, since CameraSensor::Load(_sdf) doesn't work due to it expecting sdf::SensorType::CAMERA when in reality it's CUSTOM.

Can you advise? I would like to eventually expand this idea to a VSLAM plugin with stereo cameras.

[iche033]

I'd like to instead have my plugin inherit directly from CameraSensor. I'm not sure if this is even the right approach though, since CameraSensor::Load(_sdf) doesn't work due to it expecting sdf::SensorType::CAMERA when in reality it's CUSTOM.

For your use case, one workaround to try is override CameraSensor::Load(_sdf) method in your custom sensor class that inherits from CameraSensor, make a copy of _sdf and set the type to sdf::SensorType::CAMERA before calling CameraSensor::Load

bool CustomCameraSensor::Load(const sdf::Sensor &_sdf)
  sdf::Sensor sdfSensor sdfCopy = _sdf;
  sdfCopy.SetType(sdf::SensorType::CAMERA);
  CameraSensor::Load(sdfCopy);

  // load your other custom parameters
}

[dakejahl]

@iche033 thanks, let me know if we should move this to Discord or a separate thread.

I tried this along with setting the camera sensor

bool OpticalFlowSensor::Load(const sdf::Sensor &_sdf)
{
    sdf::Sensor sdfCopy = _sdf;
    sdfCopy.SetType(sdf::SensorType::CAMERA);
    sdf::Camera cam;
    sdfCopy.SetCameraSensor(cam);

    // Load base sensor params
    if (!CameraSensor::Load(sdfCopy)) {
        gzerr << "Failed to load camera sdf" << std::endl;
        return false;
    }

    ...

However it's still not quite right.

[Err] [CameraSensor.cc:468] Camera doesn't exist.

This is how I am creating it in my model.sdf

      <!-- Optical flow sensor that inherits from CameraSensor -->
      <sensor name="optical_flow" type="custom" gz:type="optical_flow">
          <pose>0 0 0 0 1.5707 0</pose>
          <always_on>1</always_on>
          <update_rate>50</update_rate>
          <visualize>true</visualize>
          <topic>/optical_flow</topic>

          <camera>
              <horizontal_fov>0.733038</horizontal_fov>
              <image>
                  <width>35</width>
                  <height>35</height>
                  <format>L8</format>
              </image>
              <clip>
                  <near>0.1</near>
                  <far>30</far>
              </clip>
          </camera>

          <gz:optical_flow>
              <!-- Any optical flow specific configs -->
          </gz:optical_flow>
      </sensor>

[iche033]

Looks like the camera sdf is not populated if the type is CUSTOM, and I think that's why you created an empty sdf::Camera.

You can parse the from sdf by doing:

  sdf::Sensor sdfCopy = _sdf;
  sdfCopy.SetType(sdf::SensorType::CAMERA);
  sdf::Camera cam;
  cam.Load(sdfCopy.Element()->FindElement("camera"));
  sdfCopy.SetCameraSensor(cam);
  ...
  if (!CameraSensor::Load(sdfCopy)) {

[dakejahl]

@iche033 It doesn't work either. I'm thinking that this is not a supported thing for custom sensor plugins. I've been looking through the source code and it doesn't appear to be designed for this usage.