RFC: Build Fedora EPEL for AlmaLinux and AlmaLinux Kitten x86_64_v2

[jonathanspw]

No description provided.

[alexiri]

We'll probably need a custom disttag to differentiate these builds from the upstream ones, should we define it now?

[Conan-Kudo]

What would the custom DistTag accomplish? There's no source differences from the original ones from Fedora. If we allowed any source changes, then yes, I would agree. But the Vendor: will already be different since we're building in our build system.

[Conan-Kudo]

We should make sure that we have a BugURL defined so that these RPMs say to file bugs in the AlmaLinux bug tracker though, and probably have an EPEL tracker in our bug tracker.

[alexiri]

This was a suggestion from @carlwgeorge to reduce potential confusion when bugs inevitably get filed in the wrong place.

[carlwgeorge]

But the Vendor: will already be different since we're building in our build system.

Users rarely know to check the Vendor tag, and it's even more rare for that information to be included in bug reports. Meanwhile the bugzilla template for EPEL already asks for the package NVR, which would capture a custom disttag and make it obvious to EPEL packagers that the package wasn't provided by EPEL.

[Conan-Kudo]

But there are literally not going to be any source changes, so the delta is basically zero.

[carlwgeorge]

While the package source themselves will be the same, the inputs to the build won't be, because they will be built against Kitten 10 and Alma 10.

[carlwgeorge]

EPEL 10 will have more than two branches. Over it's lifecycle there will a total of eleven branches, with two or three active at any given time.

[Conan-Kudo]

While the package source themselves will be the same, the inputs to the build won't be, because they will be built against Kitten 10 and Alma 10.

We don't even do this for Alma core packages unless they are modified. So doing it different for unmodified EPEL packages would be a significant deviation from our existing policy. From my point of view, it's no different from people doing rebuilds in COPR or other sources, and we don't require changes there either.

[jonathanspw]

In today's ALESCo meeting it came up that we need to define what we do with old minor-version branches - ie sending them to vault.

[musicinmybrain]

I co-maintain a package, stockfish, that:

• has an EPEL10 branch
• needs to explicitly tell its build system about the minimum available CPU features because the software does not support runtime detection and dispatch
• currently configures itself for x86_64-v3 in EPEL10
• benefits significantly, performance-wise, from higher architecture levels

Assuming an x86_64-v3 builder, I expect this package would still build and pass its tests in an x86_64-v2 rebuild, but the binary would include x86_64-v3 instructions and would crash on x86_64-v2 hardware.

This package is a pretty unusual case, perhaps even a unique one, but there are legitimate reasons why it is the way it is.

Is there any way that such a package could be adapted to work as expected in the proposed EPEL-for-x86_64-v2 without giving up on x86_64-v3 specializations in EPEL10 proper? I suppose there would ideally be some way to conditionalize the spec file on the actual architectural baseline, rather than on the Fedora/EL release number.

[carlwgeorge]

We don't even do this for Alma core packages unless they are modified.

An EPEL bug filed for a package in the base of Alma will get closed outright because it's not in EPEL at all. An EPEL bug filed for a package that is in EPEL and also in the Alma rebuild of EPEL, using an identical disttag, will not get closed outright because it will appear to the maintainer to be a regular EPEL bug. If the bug is specific to the v2 rebuild, it will waste the maintainer's time and delay getting the bug in front of the Alma devs who are interested in resolving the issue.

So doing it different for unmodified EPEL packages would be a significant deviation from our existing policy.

Except they're not unmodified, they use a different buildroot, which is a modification. And that level of deviation will only grow with time as Alma does more cool and interesting things differently from CentOS/RHEL.

From my point of view, it's no different from people doing rebuilds in COPR or other sources, and we don't require changes there either.

COPR doesn't require anything, it's a wild west. I've also seen this exact situation play out with COPR, where a bug is reported for a package with an identical NVR as an EPEL package, and it wastes the maintainer's time and delays fixing the issue for the user. I'd like to avoid the same problem with this EPEL v2 rebuild. Keep in mind that when this happens it doesn't give the EPEL maintainer a positive feeling towards the COPR in question. In the same vein, I think it would be best to avoid similar reputation damage for Alma.

[michel-slm]

While the package source themselves will be the same, the inputs to the build won't be, because they will be built against Kitten 10 and Alma 10.

We don't even do this for Alma core packages unless they are modified. So doing it different for unmodified EPEL packages would be a significant deviation from our existing policy. From my point of view, it's no different from people doing rebuilds in COPR or other sources, and we don't require changes there either.

While the package source themselves will be the same, the inputs to the build won't be, because they will be built against Kitten 10 and Alma 10.

We don't even do this for Alma core packages unless they are modified. So doing it different for unmodified EPEL packages would be a significant deviation from our existing policy. From my point of view, it's no different from people doing rebuilds in COPR or other sources, and we don't require changes there either.

Perhaps Alma packages built against v2 should also use this new dist tag, since I expect Alma bug triagers would also want to know which of the build it is

As for COPR - that should be different. In many cases the packages are not in Fedora, or the packages are but the rebuild is just temporary, or the NEVRAs don't match exactly. Here these are official, supported packages - except different entities are doing the supporting. And Alma users who use EPEL would be used to filing RHBZs for EPEL and muscle memory takes a while to retrain (in the optimistic case).

[carlwgeorge]

As for COPR - that should be different. In many cases the packages are not in Fedora, or the packages are but the rebuild is just temporary, or the NEVRAs don't match exactly.

Right, and when the NVR is different that is often the first clue to a maintainer that the build is not an official EPEL build. If the bug report is for 1.2-3, and the maintainer knows they only have built up to 1.2-2, they know the package came from somewhere else. A different disttag gives the Alma v2 EPEL an always different release field.

[Conan-Kudo]

I co-maintain a package, stockfish, that:

• has an EPEL10 branch
• needs to explicitly tell its build system about the minimum available CPU features because the software does not support runtime detection and dispatch
• currently configures itself for x86_64-v3 in EPEL10
• benefits significantly, performance-wise, from higher architecture levels

Assuming an x86_64-v3 builder, I expect this package would still build and pass its tests in an x86_64-v2 rebuild, but the binary would include x86_64-v3 instructions and would crash on x86_64-v2 hardware.

This package is a pretty unusual case, perhaps even a unique one, but there are legitimate reasons why it is the way it is.

Is there any way that such a package could be adapted to work as expected in the proposed EPEL-for-x86_64-v2 without giving up on x86_64-v3 specializations in EPEL10 proper? I suppose there would ideally be some way to conditionalize the spec file on the actual architectural baseline, rather than on the Fedora/EL release number.

@musicinmybrain, the RPM architecture will be x86_64_v2, so your package can know how to pick the right one by simply using %ifarch x86_64_v2 or %if "%{_arch}" == "x86_64_v2" conditionals.

[Conan-Kudo]

And note, there is a generic x86_64 architecture macro for %ifarch: %{x86_64}. It functions the same way that %{ix86}, %{arm32} and %{arm64} macros do.

[musicinmybrain]

@musicinmybrain, the RPM architecture will be x86_64_v2, so your package can know how to pick the right one by simply using %ifarch x86_64_v2 or %if "%{_arch}" == "x86_64_v2" conditionals.

I just tried this in %build as an experiment:

echo '==== ARCH TEST ===='
echo '%%{_arch} = %{_arch}' 
%ifarch x86_64_v3  
echo 'matched x86_64_v3'
%endif
%ifarch x86_64_v2
echo 'matched x86_64_v2'
%endif
%ifarch x86_64
echo 'matched x86_64'
%endif
exit 1

For fedpkg --release $RELEASE mockbuild, I saw the same thing no matter whether $RELEASE was rawhide, epel10, epel9, or epel8:

==== ARCH TEST ====
%{_arch} = x86_64
matched x86_64

So I like your suggestion, but if it does end up working in the AlmaLinux rebuild, it looks like that will be the only place that it works, which is unfortunate.

[Conan-Kudo]

It also works in openSUSE, as openSUSE has plumbed in the micro architectures into their build system. It doesn't work in Fedora yet because Koji has not been taught about it yet, and the patches to the DNF stack need to be reworked and upstreamed.

[Conan-Kudo]

You can test this locally by using rpmbuild --target <rpmarch> -ba, as this is the only build system independent way to check it out.