Ability to reverse search a cuda library for metapackage containing it?

[matthewfeickert]

Comment:

This question comes from the context of #59 (comment), where I've found that cuda-nvtx-dev is a package I need, but I haven't been able to find the cuda-x metapackage that contains it (if there is one).

My real question is if I have a CUDA conda-forge package if it is possible to query all the metapackages to figure out which metapackage contains it?

I assume the answer is no, as this would be a search feature in conda/mamba/pixi and that would require the infrastructure tools to be able to do generic reverse search.

Follow up question: Is there a list of all the CUDA metapackages? The user guides currently list this subset

cuda-feedstock/recipe/doc/end_user_run_guide.md

Lines 64 to 69 in a4d8ef4

	For convenience, a number of additional metapackages are available:
	- `cuda-runtime`: All CUDA runtime libraries needed to run a CUDA application
	- `cuda-libraries`: All libraries required to run a CUDA application requiring libraries beyond the CUDA runtime (such as the CUDA math libraries) as well as packages needed to perform JIT compilation
	- `cuda-visual-tools`: GUIs for visualizing and profiling such as Nsight Compute
	- `cuda-command-line-tools`: Command line tools for analyzing and profiling such as cupti, cuda-gdb, and Compute Sanitizer
	- `cuda-tools`: All tools for analyzing and profiling, both GUI (includes cuda-visual-tools) and CLI (includes cuda-command-line-tools)

pixi search on cuda, cuda-libraries-dev, cuda-tools, libcublas-dev don't show cuda-nvtx-dev as a dependency.

[matthewfeickert]

Ah, I think the answer is that there is no metapackage for cuda-nvtx-dev given that https://github.com/search?q=org%3Aconda-forge+%22cuda-nvtx-dev%22&type=code doesn't show it appearing in any other CUDA/NVIDIA feedstocks (beyond its own: https://github.com/conda-forge/cuda-nvtx-feedstock).

[jakirkham]

Yeah searching the conda-forge org on GitHub can be pretty effective

These may not address the question, but here are some useful inspection tools

Another useful tool is this streamlit dashboard Jaime put together for looking at package details. We use this on the https://conda-forge.org/packages page to fill in the "Metadata" columns "Browse" links

The somewhat low-level conda package -w <file path> can be handy for figuring out what package provides a file in an environment

Probably closer to what you are looking for is conda-tree, which is a project in the incubator that lets one explore the dependency tree of a package. Using it with the cuda package would provide the full CUDA Toolkit tree

[matthewfeickert]

Wow, this is great, @jakirkham, thanks!

We use this on the https://conda-forge.org/packages page to fill in the "Metadata" columns "Browse" links

TIL that https://conda-forge.org/packages exists — this is going to be much more helpful than guessing feedstock URLs in the future! The dashboard is quite nice too.

Probably closer to what you are looking for is conda-tree, which is a project in the incubator that lets one explore the dependency tree of a package. Using it with the cuda package would provide the full CUDA Toolkit tree

Kinda. You can do this with pixi tree as well. What I more want is a dashboard app that you type in a package name and it tells you all the packages on conda-forge that depend on that package. Though after some more searching and coming across conda/conda#6670 (comment), I now see that this is provided through conda repoquery whoneeds.

$ micromamba repoquery whoneeds --channel conda-forge awkward | awk '{print $1}' | sort --unique

────────────────────────────────────────────────────────────────────────────────────────────
akimbo
awkward-numba
awkward-pandas
cabinetry
clouddrift
coffea
conda-forge/linux-64
conda-forge/noarch
cuda_histogram
dask-awkward
fastjet
Getting
km3io
km3pipe
lbapcommon
lhcbdirac
Name
pykealib
pylhe
ragged
rubin-env-extras
rubin-env-rsp
scikit-hep
tiled-base
uproot
uproot-base
uproot-browser
uproot-methods

(eh...there's some cruft in there from the stdout of the search channels, but close enough)

and

$ micromamba repoquery whoneeds --channel conda-forge cuda-nvtx-dev
Getting repodata from channels...

conda-forge/linux-64                                        Using cache
conda-forge/noarch                                          Using cache
nodefaults/linux-64                                         Using cache
nodefaults/noarch                                           Using cache
No entries matching "cuda-nvtx-dev" found
Try looking remotely with '--remote'.

shows no results, which confirms that there's no metapackage that provides it as a run requirement.

I think this addresses my question though, so I'll close this now. 👍

[jakirkham]

What I more want is a dashboard app that you type in a package name and it tells you all the packages on conda-forge that depend on that package. Though after some more searching and coming across conda/conda#6670 (comment), I now see that this is provided through conda repoquery whoneeds.

This seems like a good feature request for the conda streamlit metadata app

We may also just want to document all these search approaches somewhere that would be easier to find. The conda-forge webpage could be one choice. Though would be interested to hear where you looked for this info and where it might be more easily surfaced by folks searching in the future

[matthewfeickert]

This seems like a good feature request for the conda streamlit metadata app

@jakirkham Suggestion taken. :) Quansight-Labs/conda-metadata-app#24

We may also just want to document all these search approaches somewhere that would be easier to find. The conda-forge webpage could be one choice. Though would be interested to hear where you looked for this info and where it might be more easily surfaced by folks searching in the future.

Where I looked to find conda repoquery whoneeds? I searched around for a while and then I Googled "conda reverse search" and conda/conda#6670 came up. I originally ignored this as the Issue is focused on the removed conda search --reverse-dependency feature, but I eventually came back and scrolled to the bottom and noticed conda/conda#6670 (comment). I then searched "conda repoquery", noticed that all the results were actually for mamba, clicked the search result that linked to the mamba user guide, and then arrived at the user guide examples for mamba repoquery.

[jakirkham]

Thanks Matthew! 🙏

Nicely written. +1'd the issue

Ah interesting. That certainly feels like a roundabout way to discover this (not your fault of course). If we add this info to the conda-forge or Conda docs, would this have helped (or would this help future users)?

[matthewfeickert]

If we add this info to the conda-forge or Conda docs, would this have helped (or would this help future users)?

Maybe? I think there is a documentation issue with conda more in that though repoquery has been brought into conda from mamba

$ docker run --rm -ti ghcr.io/prefix-dev/pixi:latest 
root@90a33b8a9346:/# pixi global install conda
Installed package conda 24.11.3 py310hff52083_0 from conda-forge
These executables have been added to /root/.pixi/bin
 -  activate
 -  conda
 -  conda-env
 -  deactivate

! To use them, make sure to add /root/.pixi/bin to your PATH
root@90a33b8a9346:/# export PATH=/root/.pixi/bin:$PATH
root@90a33b8a9346:/# conda --version     
conda 24.11.3
root@90a33b8a9346:/# conda repoquery --help
usage: conda repoquery [-h] {whoneeds,depends,search} ...

Advanced search for repodata.

positional arguments:
  {whoneeds,depends,search}
    whoneeds            Show packages that depend on this package.
    depends             Show dependencies of this package.
    search              Show all available package versions.

options:
  -h, --help            Show this help message and exit.
root@90a33b8a9346:/#

it isn't listed in the conda commands section of the docs and if you search the conda docs for repoquery you actually get no results at all

and so you have to search the mamba docs to even find out about it. You also need to know to search "repoquery" in the first place, which I don't think is a CLI API term most novice users are going to intuit.

But I could see an example also landing in the conda-forge user docs "Tips & tricks" section.

[jakirkham]

Good point. Looks like upstream issue ( conda/conda#14172 ) is relevant here

Appears the command lives inside conda-libmamba-solver where it is documented. Though I don't expect the average user to find that necessarily