How do I use versioning support with binary caching?

[rkp4s]

I have been experimenting with binary caching support within vcpkg. It seems like a promising feature for a development team to share binary packages. Specifically, I have been trying to use Azure Artifacts as a shared binary cache where the packages are in the NuGet format. These packages are then consumed by Visual Studio projects. This set-up seemed to work well until I tried to couple it with versioning support. Let me try to explain my workflow:

Install sqlite3 (x64-windows) to Azure Artifacts as a NuGet package:
vcpkg install sqlite3:x64-windows --binarysource=clear;nuget,<private_nuget_server_url>,readwrite

To install a different version of sqlite3 to Azure Artifacts, I simply checkout the appropriate commit for the desired sqlite3 version and run the above command again. These package are then appear as published packages within Azure Artifacts as sqlite3_<triplet> <package_version>-vcpkg<package_abi>.

To consume these packages, I created a simple Visual Studio project. Within my local vcpkg repo, I run vcpkg integrate project. That creates a NuGet package that I can then install into my newly created project. From Package Manager Console within my Visual Studio project, I run Install-Package vcpkg.C.src.external.vcpkg -Source "<vcpkg_root>\scripts\buildsystems".

Then, I added a basic manifest file in the root of my project's solution directory that looks something like this:

{
  "name": "vcpkg-test",
  "version-string": "0.0.1",
  "port-version": 0,
  "description": "test vcpkg integration",
  "dependencies": [
    {
      "name": "sqlite3"
    }
  ]
}

Within my Visual Studio project, the "Use Vcpkg" and "Use Vcpkg Manifest" properties are set to "Yes". The following environment variables are set: VCPKG_FEATURE_FLAGS=versions,binarycaching,manifests, VCPKG_BINARY_SOURCES=clear;nuget,<private_nuget_server_url>,read;default,readwrite, and VCPKG_ROOT=<vcpkg_root>. (Visual Studio was restarted after setting the environment variables.)

When I build the project as is, everything works as expected. Hooray! vcpkg correctly downloads the appropriate version of sqlite3 based on the version referenced in my vcpkg root. Alas, binary caching fails when I introduce versioning support into my manifest:

{
  "name": "vcpkg-test",
  "version-string": "0.0.1",
  "port-version": 0,
  "description": "test vcpkg integration",
  "builtin-baseline": "6e8aaa76f89f7a99fbe2ba4333c61cf940cd8bc4",
  "dependencies": [
    {
      "name": "sqlite3",
      "version>=": "3.33.0"
    }
  ]
}

When I build my project with the manifest above, it always results in a cache miss and the package is built locally. Looking through the debug output from the vcpkg install command, it seems like the ABI versions differ for the same package version (e.g. sqlite3 x64-window 3.33.0) from the producer and consumer perspectives. I've tried specifying different builtin-baselines, but nothing seems to work.

My goal here is create a unified development environment for a small team. Packages are built via vcpkg, published to Azure Artifacts, and consumed by the team members without them having to build the binaries themselves. Ideally, this workflow would be as seamless as possible on the consumer side. Package versions could be clearly defined in the manifest. They'd have their local vcpkg instance, but it'd really just be a proxy to the binary cache.

In my testing and experimentation, production and consumption were happening on the same machine with the same vcpkg root. Between producing and consuming the packages, I'd clean my local environment to simulate the target environment as closely as possible.

I'm relatively new to vcpkg and NuGet so there's a strong possibility that I'm doing something wrong. If anyone has some guidance, it would be greatly appreciated.

Referenced docs:
https://github.com/microsoft/vcpkg/blob/master/docs/users/binarycaching.md
https://github.com/microsoft/vcpkg/blob/master/docs/users/versioning.md
https://devblogs.microsoft.com/cppblog/take-control-of-your-vcpkg-dependencies-with-versioning-support/
https://devblogs.microsoft.com/cppblog/vcpkg-accelerate-your-team-development-environment-with-binary-caching-and-manifests/

[rkp4s]

Update: The second instance of the manifest in my previous comment should have looked like this

{
  "name": "vcpkg-test",
  "version-string": "0.0.1",
  "port-version": 0,
  "description": "test vcpkg integration",
  "dependencies": [
    "sqlite3"
  ],
  "builtin-baseline": "6e8aaa76f89f7a99fbe2ba4333c61cf940cd8bc4",
  "overrides": [
    {
      "name": "sqlite3",
      "version": "3.33.0"
    }
  ]
}

That looks more like the preferred syntax for what I'm trying to achieve (pinning a specific package version). I've tried different incantations of this (e.g. different baselines), but they all seem to result in a cache miss with versioning.

[PhoebeHui]

@vicroms, could you please help take a look?

[rkp4s]

Perhaps, I'm going about this wrong. Rather than install the package directly to the Azure Artifacts/NuGet server, a more declarative approach seems to work:

vcpkg install sqlite3:x64-windows
vcpkg export sqlite3:x64-windows --nuget --nuget-version=3.35.2 --nuget-id=sqlite3
downloads\tools\nuget-5.5.1-windows\nuget.exe push -Source <private_nuget_server_url> -ApiKey <key> sqlite3.3.35.2.nupkg

This explicitly installs sqlite3:x64-windows, exports the package in NuGet format, and publishes the package to the server. Consumers can then just install the NuGet package.

So I've tried two different approaches for publishing and consuming packages: A) rely on vcpkg manifest (my original post) and B) rely on nuget (presented in this comment). There are some pros/cons to these different approaches:

Approach A:
Pro: This approach seems more declarative on the consumer side. The manifest directly states which packages, versions, etc. that it needs.
Con: Requires vcpkg on the consumer side.
Pro: Seems like it might permit packages to be bundled individually within NuGet (see second bullet for Approach B). Admittedly, I haven't tested this.
Con: Doesn't seem to work well with versioning.

Approach B:
Pro: Doesn't require/rely on vcpkg on the consumer side. This means fewer moving parts for the majority of a development team - the consumers.
Con: Requires that all dependencies be bundled together. This is briefly discussed in Issue #1808. For large projects, specifically projects with many dependencies, this becomes a bit of a pain. Any minor library change requires that the whole dependency package be republished on the producer side and redownloaded on the consumer side.
Pro: Clear workflow that seems to just work

Just wanted to share my thoughts here in case it helps anyone else. Otherwise, it seems like I have a workflow that satisfies my original goal.

[meastp]

This might be related to investigate differences: #16615