HDF reordering of frames

[rupertoverall]

I have made a video from a collection of test frames and have run inference over this. I am pulling the node coordinates out of the .slp archive (by inspecting the HDF structure) and using these to annotate the frames to check how well our different labelling methods perform. In most cases though, the frame index of the predicted data do not correspond to the frames in the video—this means that my annotations do not match the hand-labelled positions in the training data or the image from the video.
I have dug into the instance/frame indexing in the HDF files but I cannot find a pattern that will lead to matching of the frames with predicted points. I had no problem doing this in a previous project (where I just annotated a video after inference). I can't see what is different this time.
I also note that the number of instances in the predicted data is different from that in the training data (and the two different .slp files from different models also have different instance numbers).
Can someone suggest where I should be looking to understand this problem and get my annotations in sync?

[github-actions[bot]]

Thank you for starting a new discussion!

We appreciate your input and will review it soon.

Warning

A friendly reminder that this is a public forum. Please be cautious when clicking links, downloading files, or running scripts posted by others.

• Always verify the credibility of links and code.
• Avoid running scripts or installing files from untrusted sources.
• If you're unsure, ask for clarification before proceeding.

Stay safe and happy SLEAPing!

Best regards,
The Team

[talmo]

Hi @rupertoverall,

Right, the SLP file, while HDF5-based, is not intended for direct reading since it uses a pretty complex format to serialize data. It's efficient for use in the GUI and most workflows, and contains lots of metadata that you probably don't need for analysis.

For these cases, we recommend exporting to an Analysis HDF5 format (from the File menu or using sleap-convert).

If your data isn't tracked and you want to work with the SLP files directly, the easiest thing to do would be to use sleap-io. This is a standalone library to work with SLP and other files that makes it very straightforward to load in the data structures from these SLP files.

One reason the indexing doesn't line up is that there may not be labeled frames (LabeledFrame objects) on every video frame. Another is that the labels may contain data from multiple videos, and they may be stored in the order that they were annotated.

Let us know if sleap-io works for your case or if you need any help!

Cheers,

Talmo

[rupertoverall]

I will need more help with this I’m afraid. I have looked at sleap.io and can extract the frames using my very rudimentary python skills, but I do not see any difference to the information I already get from picking the HDF apart in R—except that the number of frames returned varies (this could be due to frame mangling when collating the frames into a video for inference).
I put 1103 frames into the inference but am not always getting the same number out (and cannot verify that they are in the same order). I note that issue #1976 gets close, but I would need to inference on the SLP package (is that possible?) and specify videos not frames.

To step back a bit; the ultimate goal is, with a dataset of annotated frames on hand (in a SLP package), to edit a subset of the annotations -> build models using the subset with/without the edits -> run inference on the original annotated dataset using both models -> see in what way the predicted nodes from each model differ from the hand-annotated positions.
I may have been going about the whole process wrongly - so I am happy for any solution that moves our work forward.

[eberrigan]

You can use a SLP package, but you do not have to as long as the paths in the .slp file to the videos are correct relative to where you are running the command. The SLP package gives the advantage that you can embed the videos in the file and then you do not have to worry about where the videos are located. This is especially useful when running inference or training remotely.

If you want to follow a similar solution as #1976 and not use Python, you can make a directory of videos, and specify that as your --data_path arg in sleap-track. Please note that sleap-track will have more options for specifying input and output paths on our development branch (see documentation here. Follow these instructions to install sleap from source and create an environment with the latest SLEAP code that we are developing.

Please let us know if these suggestions work for you. Thanks!