Deb Packages Rule

This is a continuation of the work started in https://github.com/bazelbuild/rules_pkg/tree/main/deb_packages

Getting started

deb_packages is a repository rule, and therefore made to be used in the WORKSPACE.

First, tell bazel to load the rule and its dependencies with a load() statement.

http_archive(
    name = "rules_deb_packages",
    sha256 = "674ce7b66c345aaa9ab898608618a0a0db857cbed8e8d0794ca46e375fd5ff76",
    urls = ["https://github.com/petermylemans/rules_deb_packages/releases/download/v0.4.0/rules_deb_packages.tar.gz"],
)

load("@rules_deb_packages//:repositories.bzl", "deb_packages_dependencies")

deb_packages_dependencies()

Next, for every source of deb packages, create a deb_packages rule. You can define additional mirrors per package source, but it is assumed that all these mirrors will serve the exact same files. Hashes are checked after downloading files.

load("@rules_deb_packages//:deb_packages.bzl", "deb_packages")

deb_packages(
    name = "debian_buster_amd64",
    arch = "amd64",
    packages = {
        "base-files": "pool/main/b/base-files/base-files_10.3+deb10u8_amd64.deb",
        "busybox": "pool/main/b/busybox/busybox_1.30.1-4_amd64.deb",
        "ca-certificates": "pool/main/c/ca-certificates/ca-certificates_20200601~deb10u2_all.deb",
        "libc6": "pool/main/g/glibc/libc6_2.28-10_amd64.deb",
        "libssl1.1": "pool/main/o/openssl/libssl1.1_1.1.1d-0+deb10u4_amd64.deb",
        "netbase": "pool/main/n/netbase/netbase_5.6_all.deb",
        "openssl": "pool/main/o/openssl/openssl_1.1.1d-0+deb10u4_amd64.deb",
        "tzdata": "pool/main/t/tzdata/tzdata_2021a-0+deb10u1_all.deb",
    },
    packages_sha256 = {
        "base-files": "eda9ec7196cae25adfa1cb91be0c9071b26904540fc90bab8529b2a93ece62b1",
        "busybox": "1e32ea742bddec4ed5a530ee2f423cdfc297c6280bfbb45c97bf12eecf5c3ec1",
        "ca-certificates": "a9e267a24088c793a9cf782455fd344db5fdced714f112a8857c5bfd07179387",
        "libc6": "6f703e27185f594f8633159d00180ea1df12d84f152261b6e88af75667195a79",
        "libssl1.1": "b02b468f0fad929b5d2b38ae05607c22c4f1ef70adc2688fb02b9d9514d6ac51",
        "netbase": "baf0872964df0ccb10e464b47d995acbba5a0d12a97afe2646d9a6bb97e8d79d",
        "openssl": "fa997ab8745f28ea3553ebc623872bb6809cc94898f6c72d81a757d9ee47dfe3",
        "tzdata": "00da63f221b9afa6bc766742807e398cf183565faba339649bafa3f93375fbcb",
    },
    sources = [
        "http://deb.debian.org/debian buster main",
        "http://deb.debian.org/debian buster-updates main",
        "http://deb.debian.org/debian-security buster/updates main",
    ],
    timestamp = "20210216T115751Z",
    urls = [
        "http://deb.debian.org/debian/$(package_path)",
        "http://deb.debian.org/debian-security/$(package_path)",
        "https://snapshot.debian.org/archive/debian/$(timestamp)/$(package_path)",  # Needed in case of supersed archive no more available on the mirrors
        "https://snapshot.debian.org/archive/debian-security/$(timestamp)/$(package_path)",  # Needed in case of supersed archive no more available on the mirrors
    ],
)

Internally .deb files referenced here will be downloaded by Bazel, renamed to their SHA256 hash (not all characters used in file names are legal in bazel names) and made available in a dictionary named the same as the deb_packages rule. This dictionary is made available in a file named deb_packages.bzl in the debs subfolder of this rule. You can find the generated and downloaded files in the ./bazel-YourWorkSpace/external/your_rule_name/debs folder after building the project if you're interested.

To actually use the .deb files in a BUILD file rule like container_image, you first have to load all dictionaries of package sources you want to use. This is done with the load("@your_rule_name//debs:deb_packages.bzl", "your_rule_name") line. Then you can use the dictionary named the same as the deb_packages rule to refer to the packages you defined in the WORKSPACE file.

load("@io_bazel_rules_docker//container:container.bzl", "container_image")
load("@debian_buster_amd64//debs:deb_packages.bzl", "debian_buster_amd64")

container_image(
    name = "base_buster",
    debs = [
        debian_buster_amd64["base-files"],
        debian_buster_amd64["netbase"],
        debian_buster_amd64["tzdata"],
        debian_buster_amd64["libc6"],
        debian_buster_amd64["libssl1.1"],
        debian_buster_amd64["busybox"],
    ],
    entrypoint = [
        "busybox",
        "sh",
    ],
    env = {"PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"},
)

Adding/updating new packages or package sources

Automatically using the update_deb_packages tool

As you saw, most of the information is already available on mirrors anyways as soon as you know the distro, exact package name, architecture and version. If you enter the correct rule name for the pgp_key field, this also means that you can do this in a verified chain of trust.

The update_deb_packages tool can help you with this.

To use it, first add an update_deb_packages rule to your workspaces top-level BUILD file.

load("@rules_deb_packages//:update_deb_packages.bzl", "update_deb_packages")

update_deb_packages(
    name = "update_deb_packages",
    bzl_files = [
        "debian_buster_amd64.bzl", # if you use macro to import in WORKSPACE otherwise optional
    ],
    pgp_keys = [
        "@buster_archive_key//file",
        "@buster_security_archive_key//file",
    ],
)

The update tool will check the signatures of the release info before updating the hashes of the packages. So you need to tell it about the trusted PGP keys typically obtained from https://ftp-master.debian.org/keys.html.

Also create a http_file rule in your WORKSPACE that references this key and make sure to include a SHA256 hash, so it won't change later:

http_file(
    name = "buster_archive_key",
    sha256 = "9c854992fc6c423efe8622c3c326a66e73268995ecbe8f685129063206a18043",
    urls = ["https://ftp-master.debian.org/keys/archive-key-10.asc"],
)

http_file(
    name = "buster_security_archive_key",
    sha256 = "4cf886d6df0fc1c185ce9fb085d1cd8d678bc460e6267d80a833d7ea507a0fbd",
    urls = ["https://ftp-master.debian.org/keys/archive-key-10-security.asc"],
)

Next create a deb_packages rule in your WORKSPACE file without any packages defined:

deb_packages(
    name = "debian_buster_amd64",
    arch = "amd64",
    packages = {
        "base-files": "",
        "busybox": "",
    },
    packages_sha256 = {
        "base-files": "",
        "busybox": "",
    },
    sources = [
        "http://deb.debian.org/debian buster main",
        "http://deb.debian.org/debian buster-updates main",
        "http://deb.debian.org/debian-security buster/updates main",
    ],
    timestamp = "20210216T115751Z",
    urls = [
        "http://deb.debian.org/debian/$(package_path)",
        "http://deb.debian.org/debian-security/$(package_path)",
        "https://snapshot.debian.org/archive/debian/$(timestamp)/$(package_path)",  # Needed in case of supersed archive no more available on the mirrors
        "https://snapshot.debian.org/archive/debian-security/$(timestamp)/$(package_path)",  # Needed in case of supersed archive no more available on the mirrors
    ],
)

Now run bazel run update_deb_packages (similar to the gazelle tool used by the golang Bazel rules) and the helper tool will fetch the relevant files from the mirror(s) and add the data for missing packages at the respective deb_packages rule.

It will also update any existing packages to either the most recent version available on the mirror or a version you specified in the package name (package=version). The string latest is also supported if you want to use version pinning.

Manually

Choose a Debian mirror that you want to use, for example http://deb.debian.org/debian.

Visit the /dists/ directory on that mirror and choose the distro you want to use, for example jessie.

Download the Release and Release.gpg files in the distro's folder (in our example: http://deb.debian.org/debian/dists/jessie/Release and http://deb.debian.org/debian/dists/jessie/Release.gpg).

Verify the file's signature: gpg --verify Release.gpg Release It must be signed with a vald signature by one of the keys on this site: https://ftp-master.debian.org/keys.html

Also create a http_file rule that references this key and make sure to include a SHA256 hash, so it won't change later:

http_file(
    name = "jessie_archive_key",
    sha256 = "e42141a829b9fde8392ea2c0e329321bb29e5c0453b0b48e33c9f88bdc4873c5",
    urls = ["https://ftp-master.debian.org/keys/archive-key-8.asc"],
)

This file contains the paths to various other files and their hashes. Scroll down to the SHA256 section and choose the path to the Packages file that you want to use (for example main/binary-amd64/Packages.xz) and also note down its hash.

Append the Packages file path to your mirror URL + /dists/yourdistro (for example http://deb.debian.org/debian/dists/jessie/main/binary-amd64/Packages.xz) and download the resulting file.

Verify the hash of the file you received (with the exception of the GPG keys site, all these downloads happen on insecure channels by design) with sha256sum: sha256sum Packages.xz

Unpack the archive (if you downloaded the Packages.gz or Packages.xz file) and now you'll have a huge text file that contains hashes and paths to all Debian packages in that repository.

Open this file and start looking for the package names you want to use in your BUILD files. You can do this for example in a text editor or using grep (the -A switch prints that many lines after each match): grep -A 25 "Package: python2.7-minimal" Packages

Now you finally have the info that you must enter in the deb_packages rule: The value at Filename is the path to the exact package to be used and the value at SHA256 is the verified hash that this file will have.

Now enter this information in the WORKSPACE file in a deb_packages rule:

deb_packages(
    name = "my_new_manual_source",
    arch = "amd64",
    packages = {
        "libpython2.7-minimal": "pool/main/p/python2.7/libpython2.7-minimal_2.7.9-2+deb8u1_amd64.deb",
    },
    packages_sha256 = {
        "libpython2.7-minimal": "916e2c541aa954239cb8da45d1d7e4ecec232b24d3af8982e76bf43d3e1758f3",
    },
    timestamp = "20210216T115751Z",
    urls = [
        "http://deb.debian.org/debian/$(package_path)",
        "http://deb.debian.org/debian-security/$(package_path)",
        "https://snapshot.debian.org/archive/debian/$(timestamp)/$(package_path)",  # Needed in case of supersed archive no more available on the mirrors
        "https://snapshot.debian.org/archive/debian-security/$(timestamp)/$(package_path)",  # Needed in case of supersed archive no more available on the mirrors
    ],
)

Reference

deb_packages

deb_packages(name, arch, mirrors, packages, packages_sha256, sources)

A workspace rule that downloads .deb packages from a Debian style repository and makes them available in the WORKSPACE.

For a deb_packages rule named foo_bar, packages can be used by loading load("@foo_bar//debs:deb_packages.bzl", "foo_bar") into your BUILD file, then referencing the package with foo_bar['packagename'].

The packagename is expected to be the exact package name as available upstream, with an optional version string appended. This is not enforced by bazel or these rules, but makes automatic parsing and updating much easier. If you use the update_deb_packages helper, version pinning with packagename=version is supported.

Every key name in the packages section must exactly match a key name in the packages_sha256 section.

Attributes
name	rule name, required
arch	the target package architecture, required Examples: amd64, arm64, s390x etc.
urls	the full url of the package repository, required We need the full url needed for accessing the required deb package files. These urls can be based on debian or ubuntu mirrors and support templating with the following variables : $(package_path) : will be replaced with the full package path, eg : pool/main/b/base-files/base-files_10.3+deb10u8_amd64.deb $(package_name) : will be replaced with the package name (after the last '/' of the package path), eg: base-files_10.3+deb10u8_amd64.deb $(timestamp) : will be replaced wit he timestamp attribute value, eg : 20210216T115751Z e.g. https://snapshot.debian.org/archive/debian-security/$(timestamp)/$(package_path)
packages	a dictionary mapping packagename to package_path, required The deb file is expected to be found at mirror + package_path Package names can optionally contain a version (packagename=1.2.3-4)
packages_sha256	a dictionary mapping packagename to package_hash, required The deb file at package_path is expected to have this sha256 hash Keys need to be the same as in the packages dictionary
sources	a list of full sources of the package repository in format similar to apt sources.list without the deb prefix e.g. 'http://deb.debian.org/debian buster main'
timestamp	timestamp of the last update (format : 'yyyyMMddTHHmmssZ'), not mandatory e.g. 20210216T115751Z

update_deb_packages

update_deb_packages(name, pgp_keys)

A rule that helps keep all deb_package repository rules up to date, by checking current version against the latest available version in the specified sources.

Attributes
name	rule name, required
bzl_files	optional list of additional bzl files (path relative to workspace) to scan for deb_packages rules the WORKSPACE file is always scanned
pgp_keys	required list of trusted PGP keys that were used to sign release information of source repositories