Fish eye lens correction

[polycherry]

Hello! I am trying to label quite a few images so wanted to ask before that if there is a fish-eye correction that I need to apply before labelling.
Do I need to do it on sleap or before, on the videos?
My videos currently look like this -

[github-actions[bot]]

Thank you for starting a new discussion!

We appreciate your input and will review it soon.

Warning

A friendly reminder that this is a public forum. Please be cautious when clicking links, downloading files, or running scripts posted by others.

• Always verify the credibility of links and code.
• Avoid running scripts or installing files from untrusted sources.
• If you're unsure, ask for clarification before proceeding.

Stay safe and happy SLEAPing!

Best regards,
The Team

[talmo]

Hi @polycherry,

Good question!

First off, let me just say that that's nowhere close to the worst lens distortion that we've seen, so you're starting from a good place :)

In general, it would be ideal if you undistorted the videos themselves before annotating or training in SLEAP. The downside is that this is slow and compute-intensive since you have to apply it to every pixel on every frame and re-render the video. The upside is that you'll have improved efficiency since the SLEAP model will be better able to reuse image features that it learns across the entire image, as opposed to having to learn different variants of what a mouse ear looks like under different degrees of nonlinear distortion. Note that this will only help to some extent -- ultimately, lens distortion is mapping different numbers of photons onto different pixels, so you lose information around the periphery that simply can't be recovered (i.e., it'll look blurry when you undistort).

On the other hand, if your distortion isn't that bad (which I'd say is probably the case with your data), then you can save yourself the hassle of re-rendering the undistorted videos and instead just apply the undistortion of the keypoint coordinates themselves, which will be orders of magnitude faster.

Let us know what you end up doing or if you have any questions!

Cheers,

Talmo