Add support to release Linux aarch64 wheels

[odidev]

Problem

On aarch64, ‘pip install numcodecs’ builds the wheels from source code and then installs it. It requires the user to have a development environment installed on his system. Also, it takes some time to build the wheels than downloading and extracting the wheels from pypi.

Resolution

On aarch64, ‘pip install numcodecs’ should download the wheels from pypi

@joshmoore and Team Please let me know your interest in releasing aarch64 wheels. I can help in this.

[odidev]

I have done some modifications to build the wheel for arm64 but were getting below error -

gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DHAVE_LZ4=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ic-blosc/blosc -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/zstd-1.4.8 -Ic-blosc/internal-complibs/lz4-1.9.3 -Ic-blosc/internal-complibs/zstd-1.4.8/dictBuilder -Ic-blosc/internal-complibs/zstd-1.4.8/common -Ic-blosc/internal-complibs/zstd-1.4.8/decompress -Ic-blosc/internal-complibs/zstd-1.4.8/legacy -Ic-blosc/internal-complibs/zstd-1.4.8/dll -Ic-blosc/internal-complibs/zstd-1.4.8/deprecated -Ic-blosc/internal-complibs/zstd-1.4.8/compress -Ic-blosc/internal-complibs/zstd-1.4.8/dll/example -I/opt/_internal/cpython-3.6.12/include/python3.6m -c numcodecs/blosc.c -o build/temp.linux-aarch64-3.6/numcodecs/blosc.o -mno-sse2 -mno-avx2
    gcc: error: unrecognized command line option ‘-mno-sse2’
    gcc: error: unrecognized command line option ‘-mno-avx2’
    [numcodecs] command 'gcc' failed with exit status 1
    Traceback (most recent call last):
      File "/opt/_internal/cpython-3.6.12/lib/python3.6/distutils/unixccompiler.py", line 118, in _compile
        extra_postargs)
      File "/opt/_internal/cpython-3.6.12/lib/python3.6/distutils/ccompiler.py", line 909, in spawn
        spawn(cmd, dry_run=self.dry_run)
      File "/opt/_internal/cpython-3.6.12/lib/python3.6/distutils/spawn.py", line 36, in spawn
        _spawn_posix(cmd, search_path, dry_run=dry_run)
      File "/opt/_internal/cpython-3.6.12/lib/python3.6/distutils/spawn.py", line 159, in _spawn_posix
        % (cmd, exit_status))
    distutils.errors.DistutilsExecError: command 'gcc' failed with exit status 1

I have bypassed the flag by modifying the code and now the wheel is building successfully for arm64.

Commit Link - odidev@5a7e2b5

Github actions Link - https://github.com/odidev/numcodecs/runs/3789990435?check_suite_focus=true

If the changes looks good. Shall I procced by raising the PR?

[joshmoore]

Hi @odidev. Wow. Thanks for looking into this. (also TIL about [auto]. Nice!) From my side, I'd love to see a PR.

[dstansby]

There's a PR for this here: #315

[dstansby]

building the wheels using emulation is taking a lot of time (hours). Before we go any further or merge support for aarch64 wheels, it would be good to understand:

1. What systems/machines out in the wild use this architechture? Is it common?
2. How long does it take to build numcodecs from source on this system.

If it is fairly quick (~minutes) and these systems are rare, then I'm not sure whether this is worth pursuing. But if they are not rare, and/or builds still take forwever, we can think more about this.

[Cadair]

+1 on seeing aarch64 wheels fwiw.

[dstansby]

I'm still unclear on

What systems/machines out in the wild use this architechture? Is it common?

What aarch64 systems are you looking to support?

[Cadair]

SunPy got a specific request to support linux VMs on macos arm64 hardware, but also the pi4 and pi5 use aarch64 wheels if they have a 64bit OS installed.

[Cadair]

In answer to:

How long does it take to build numcodecs from source on this system.

It seems to be taking ~~5 mins per python version on our GitHub actions aarch64 runner.

[dstansby]

Is that a Github hosted runner that numcodecs could use too? At #563 we're using emulation to do the aarch64 wheels, which is why they take a while (20 mins...) to build.

[Cadair]

Yes, you can use GH hosted runners, but you have to pay for them. They are cheap tho.

[dstansby]

Hi all, it looks like arm64 runners are now free 🥳 I imagine this will make building the wheels run much quicker, so we'd definitely welcome a PR to build the wheels on the arm64 runners to see if they build in a reasonable time.