Skip to content

Commit

Permalink
Use GitHub actions as CI/CD (#6)
Browse files Browse the repository at this point in the history
* Only ignore directories starting with build, not all files

* workflow pulled over from cwipc_util branch issue32-github-actions

* Changes to update to the new situation with the umbrella repo. Let's see what happens.

* Checkout submodules

* For now disable OSX and Windows builds

* Playing with how to install dependent packages

* Trying to get actions to work again

* More installations

* Fixed indentation

* Sigh, pcl is no longer available with choco. Attempt to use vcpkg

* Fixed where to use sudo and where not

* Adding vcpkg as a submodule, which seems to be preferred usage

* Other way of using vcpkg. Output test failures (if there are failures)

* Attempting to find out why k4aoffline tests fails on Linux

* install boost with choco on windows (hoping this forestalls building from source with vcpkg)

* Ensure lfs content is checked out, also in submodules

* Attempting install

* Retry github action build

* Temporary do Windows builds only

* Attempting to use a vcpkg.json file, to allow caching of built dependencies in the CI/CD on windows

* Temporarily remove gitlab ci/cd. Don't merge.

* Fixed syntax error

* Don't install boost with choco on windows

* For debugging, upload vcpkg directory as arifact

* Pack vcpkg using tar

* vcpkg too big. Keep du listing only.

* vcpkg too big. Keep du listing only.

* vcpkg isnt working. Try all-in-one pcl installer

* Fixed curl command

* More hacking to try and install pcl

* More debugging

* More debugging

* More debugging

* Trying aghain, now with powershell

* Trying aghain, now with powershell

* Trying aghain, now with powershell

* Trying aghain, now with powershell

* Trying aghain, now with powershell

* Yet another attempt: download complete installed PCL directory from zipfile

* Fixed yaml syntax error

* Typo

* Pass PCL_ROOT to cmake

* Got rid of matrix, more hassle than it is worth. Use separate jobs.

* Fixed indentation

* Check why jpeg-turbo isn't found

* Check why jpeg-turbo isn't found

* Check why jpeg-turbo isn't found

* More checking of build process

* More checking of build process

* Different way to save ctest log (and build folder on Windows)

* Upload installed folder as artifact, another attempt at capturing build folder on failure

* Temporarily remove gitlab ci/cd (while debugging github actions)

* Install K4A sdks using installers and msi

* Trying to find out what happened to k4a installs

* Added opencv

* Added opencv

* Capture install logs

* Capture install logs

* Adding command from actions/runner-images#1704 in hope that will make installers run

* python-opencv doesn't exist

* Use Start-Process -Wait to run installers

* Use Start-Process -Wait to run installers

* Use Start-Process -Wait to run installers

* Use Start-Process -Wait to run installers

* Check that we have enough disk space

* More fiddling

* Giving up on getting installers to run under github actions. Prepared zipfiles for k4a and k4abt and download and unpack those.

* Allow specifying K4A_ROOT and K4ABT_ROOT on windows

* Fixing cmake options.

* Sigh, put cmake in trace mode

* We used to add BUILD_TYPE to bin/lib folder paths. Disabled, to get rid of double BUILD_TYPE on github actions (possibly due to newer cmake version)

* Check what environment variables are available

* Get rid of workaround to add CMAKE_BUILD_TYPE to output directories.

* We should setup the output directories

* Added GITHUB_REF_NAME to archive filenames

* Cater for BUILD_TYPE in windows PATH

* Get rid of vcpkg

* Install libjpeg-turbo from installer, not through choco (which has a grossly outdated version)

* Reference environment variables with env.

* Removed quotes

* Merged master into inssue16-cmake

* Merged master into inssue16-cmake

* Re-adding gitlab-ci.yml

* Use install script to install third party packages

* Removed CMAKE_BUILD_TYPE from paths for runtime/library/archive output directories. See what this does for Windows builds on gitlab and github.

* Run the windows install script, see if this fixes at least the Python tests

* Ensure cmake finds the correct Python (windows)

* Python_FIND_STRATEGY also needed

* Python_FIND_STRATEGY also needed

* More debugging option tries

* Try to fix python to use with Python3_EXECUTABLE

* Convert python executable path to Windows form

* Removed extraneous quotes

* Move thridparty install dir out of source directory

* Move thridparty install dir out of source directory

* Fixed CMake ROOT variables to have the correct names

* Fixed destination of 3rd party package unzip

* Yet another case fix for a variable

* Fixed stupid typo in kinect install path

* Attempting to find out where 3rdparty packages are installed

* Ah! DestinationPath should reference the parent of where to unzip an archive, not the destination folder. Let's see if that fixes things.

* Now PCL can't be found. Trying PCL_DIR in stead of PCL_ROOT

* Revert "Now PCL can't be found. Trying PCL_DIR in stead of PCL_ROOT"

This reverts commit f3e29c1.

* Typo in PCL pathname

* Variables to define where packages are installed were wrong case. Fixed to k4a_ROOT and k4abt_ROOT.

* Sigh. Adding quotes around cmake pathname arguments

* Sigh. Adding quotes around cmake pathname arguments differently

* No amount of quoting is going to work because of ashutoshvarma/action-cmake-build#231.

* Wild idea: let's pass the parent dir of the PCL installation.

* Install OpenCV

* Direct OpenCV search to x64/vc15

* Sigh, another try. Install OpenCV to c:/opencv

* Added cmake tracing to see if I can find opencv issue

* Opencv self-extracting zip destination should also be parent folder

* More debugging how opencv is installed

* OpenCV was installed completely empty, only directory structure. See whether choco works better.

* Fixed up some PATH additions

* Check why osx/linux uploads are not working

* Temporarily disable test on windows. Try and fix install dir.

* Fixed environment variable reference

* Run a script to find DLL loading problems

* Use github.ref_name

* Fixed path problems

* Install dlltracer before tryig to use it

* Do cmake calls from source directory (referring to build directory)

* Use PCL 1.11.1 in stead of 1.12.1. It looks like the problem _may_ be that PCL 1.12 uses instructions that are unsupported on the CI/CD machines.

* Re-enabled testsing on Windows.

* Install PCL with installer, waiting a random 120 seconds. See PointCloudLibrary/pcl#5216 for details.

* Getting started on powershell script to ainstall all dependencies.

* Moving towards running installers.

* Fixing the first few installs.

* Install most packages with original installer

* Fixed some quoting

* Disable Linux and MacOS for now

* Playing with a script to return the git version information

* Extract version numbers from version string

* Don't print developer warnings

* Fixed k4abt path

* Make pip calls more quiet

* Re-enabled linux and OSX builds

* Use zip for windows aretefact

* For reasons unknown we suddenly need to add the winsock libraries on Windows...

* Getting started on creating a release

* Fixed syntax error

* Enable creation of releases, on tagged pushes

* Generate version string from git describe during build. Export with cwipc_get_version() from C and Python APIs. Use when creating Python packages. Expose through cwipc_ scripts with --version option.

* Generate version string from git describe during build. Export with cwipc_get_version() from C and Python APIs. Use when creating Python packages. Expose through cwipc_ scripts with --version option.

* Ensure we have all tags (so version code works)

* Ensure we have all tags (so version code works)

* Getting rid of dead wood

* Debugging git describe failure on gitlab ci/cd

* Run pip installation for testing serially

* Adding GIT_DEPTH: 999 because 50 may be too small

* Isolate venv from system python, otherwise script entrypoints don't get installed for testing if the packages have been installed before running ctest

* Apparently wheel is a separate module that needs to be installed

* Sigh. See which python packages are installed

* Tweaking linux build on gitlab, to see whether this fixes test failures

* More tweaking linux build on gitlab, to see whether this fixes test failures

* Work around open3d issue on Flauwte by not running those tests.

* Updated documentation

* Updated

* Name the files to be uploaded for the release

* Adding scripts directory to installation

* Updated readme files

* It seems action-gh-release doesn't upload files. Trying marvinpinto/action-automatic-releases

* Going back to actions/create-release, even though it is unsupported. Because maybe it works, unlike the other actions...

* Increase sleep time for installing pcl (sigh)

* Forgot to add token to the creae-release step

* download-artifact wants path to parent

* Post release as draft and don't upload full release notes

* Updated CHANGELOG

* Submodules merged into master.
  • Loading branch information
jackjansen authored May 1, 2022
1 parent c429c7e commit cfdd039
Show file tree
Hide file tree
Showing 19 changed files with 540 additions and 76 deletions.
296 changes: 296 additions & 0 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,296 @@
name: Build, test and optionally deploy

on: [push]

jobs:
build-windows:
name: build-windows
runs-on: windows-latest
steps:
- name: Checkout code
uses: actions/checkout@v2
with:
submodules: 'recursive'
lfs: 'true'
fetch-depth: 0
- name: Pre-install (Windows)
shell: pwsh
run: |
$thirdpartydir="$((Get-Item ..).FullName)\3rdparty"
mkdir $thirdpartydir
- name: Install realsense SDK
shell: pwsh
run: |
choco install -y realsense-sdk2
Add-Content $env:GITHUB_PATH "C:\Program Files (x86)\Intel RealSense SDK 2.0\bin\x64"
- name: Install opencv (Windows)
shell: pwsh
run: |
$thirdpartydir="$((Get-Item ..\3rdparty).FullName)"
$installer="$thirdpartydir\opencv-installer.exe"
(New-Object System.Net.WebClient).DownloadFile("https://github.com/opencv/opencv/releases/download/4.5.5/opencv-4.5.5-vc14_vc15.exe","$installer");
Start-Process "$installer" -ArgumentList '-o"C:\" -y' -Wait
Add-Content $env:GITHUB_PATH "C:/opencv/build/bin"
- name: Install libturbojpeg (Windows)
shell: pwsh
run: |
$thirdpartydir="$((Get-Item ..\3rdparty).FullName)"
$installer="$thirdpartydir\jpegturbo-installer.exe"
(New-Object System.Net.WebClient).DownloadFile("https://sourceforge.net/projects/libjpeg-turbo/files/2.1.3/libjpeg-turbo-2.1.3-vc64.exe","$installer");
Start-Process "$installer" -ArgumentList "/S" -Wait
Add-Content $env:GITHUB_PATH "C:/libjpeg-turbo64/bin"
- name: Install K4A (Windows)
shell: pwsh
run: |
$thirdpartydir="$((Get-Item ..\3rdparty).FullName)"
$installer="$thirdpartydir\k4a-installer.exe"
(New-Object System.Net.WebClient).DownloadFile("https://download.microsoft.com/download/3/d/6/3d6d9e99-a251-4cf3-8c6a-8e108e960b4b/Azure%20Kinect%20SDK%201.4.1.exe","$installer");
Start-Process "$installer" -ArgumentList "/S" -Wait
Add-Content $env:GITHUB_PATH "C:/Program Files/Azure Kinect SDK v1.4.1/sdk/windows-desktop/amd64/release/bin"
- name: Install K4ABT (Windows)
shell: pwsh
run: |
$thirdpartydir="$((Get-Item ..\3rdparty).FullName)"
$msifile="$thirdpartydir\k4abt-installer.msi"
(New-Object System.Net.WebClient).DownloadFile("https://download.microsoft.com/download/9/d/b/9dbe0fbe-c9c3-4228-a64c-1e0a08736ec1/Azure%20Kinect%20Body%20Tracking%20SDK%201.1.1.msi","$msifile");
Start-Process "$msifile" -ArgumentList '/quiet /passive' -Wait
Add-Content $env:GITHUB_PATH "C:/Program Files/Azure Kinect Body Tracking SDK/tools"
- name: Install PCL (Windows)
shell: pwsh
run: |
$thirdpartydir="$((Get-Item ..\3rdparty).FullName)"
$installer="$thirdpartydir/pcl-installer.exe"
(New-Object System.Net.WebClient).DownloadFile("https://github.com/PointCloudLibrary/pcl/releases/download/pcl-1.11.1/PCL-1.11.1-AllInOne-msvc2019-win64.exe","$installer");
& "$installer" /S
Start-Sleep -s 300
Add-Content $env:GITHUB_PATH "C:/Program Files/PCL 1.11.1/bin"
Add-Content $env:GITHUB_PATH "C:/Program Files/PCL 1.11.1/3rdparty/VTK/bin"
Add-Content $env:GITHUB_PATH "C:/Program Files/OpenNI2/redist"
- name: Install OpenCV (Windows)
shell: pwsh
run: |
choco install opencv
Add-Content $env:GITHUB_PATH "C:/tools/opencv/build/x64/vc15/bin"
Add-Content $env:GITHUB_ENV "OPENCV_DIR=C:/tools/opencv/build"
- name: Install Python packages (Windows)
shell: bash
run: |
./scripts/install-3rdparty-win1064.sh
pythonExecutable=`which python`
pythonRoot=`python -c "import sys ; print(sys.prefix)"`
pythonExecutable=`cygpath --windows "$pythonExecutable"`
echo "Python_ROOT_DIR=$pythonRoot" >> $GITHUB_ENV
echo "Python3_EXECUTABLE=$pythonExecutable" >> $GITHUB_ENV
- name: Build & Test
uses: ashutoshvarma/action-cmake-build@master
with:
build-dir: ${{ github.workspace }}/build
configure-options: -Wno-dev -DCMAKE_INSTALL_PREFIX=${{ github.workspace }}/installed -DPython3_EXECUTABLE=${{ env.Python3_EXECUTABLE }} -DOpenCV_ROOT=${{ env.OPENCV_DIR }}
build-type: Release
run-test: true
install-build: true
- name: upload CTest output in case of failure
if: ${{ failure() }}
uses: actions/upload-artifact@v3
with:
name: windows-ctest-output
path: build/Testing/Temporary/LastTest.log
- name: Capture build folder as artifact
if: ${{ failure() }}
shell: bash
run: tar cfz build.tgz build
- name: Upload build folder
if: ${{ failure() }}
uses: actions/upload-artifact@v3
with:
name: windows-build-folder
path: build.tgz
- name: Capture installed folder as artifact
shell: bash
run: |
7z a cwipc_win1064_${{ github.ref_name }}.zip installed/
- name: Upload installed folder
uses: actions/upload-artifact@v3
with:
name: cwipc_win1064_${{ github.ref_name }}.zip
path: cwipc_win1064_${{ github.ref_name }}.zip
- name: Check what was created and installed
if: always()
shell: bash
run: |
ls -l /c
ls -l /c/tools
ls -l "/c/Program Files"
ls -l "/c/Program Files (x86)"
ls -l
ls -l ..
build-macos:
name: build-macos
runs-on: macos-latest
steps:
- name: Checkout code
uses: actions/checkout@v2
with:
submodules: 'recursive'
lfs: 'true'
fetch-depth: 0
- name: Install dependencies
shell: bash
run: ./scripts/install-3rdparty-osx1015.sh
- name: Build & Test
uses: ashutoshvarma/action-cmake-build@master
with:
build-dir: ${{ github.workspace }}/build
configure-options: -Wno-dev
run-test: true
install-build: true
install-options: --prefix ${{ github.workspace }}/installed
- name: upload CTest output in case of failure
if: ${{ failure() }}
uses: actions/upload-artifact@v3
with:
name: macos-ctest-output
path: build/Testing/Temporary/LastTest.log
- name: Capture build folder as artifact
if: ${{ failure() }}
shell: bash
run: tar cfz build.tgz build
- name: Upload build folder
if: ${{ failure() }}
uses: actions/upload-artifact@v3
with:
name: macos-build-folder
path: build.tgz
- name: Capture installed folder as artifact
shell: bash
run: |
tar -c -v -f cwipc_osx1014_${{ github.ref_name }}.tgz -z -C installed .
ls -l
- name: Upload installed folder
uses: actions/upload-artifact@v3
with:
name: cwipc_osx1014_${{ github.ref_name }}.tgz
path: cwipc_osx1014_${{ github.ref_name }}.tgz

build-linux:
name: build-linux
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v2
with:
submodules: 'recursive'
lfs: 'true'
fetch-depth: 0
- name: Install dependencies
shell: bash
run: ./scripts/install-3rdparty-ubuntu2004.sh
- name: Build & Test
uses: ashutoshvarma/action-cmake-build@master
with:
build-dir: ${{ github.workspace }}/build
configure-options: -Wno-dev
run-test: true
install-build: true
install-options: --prefix ${{ github.workspace }}/installed
- name: upload CTest output in case of failure
if: ${{ failure() }}
uses: actions/upload-artifact@v3
with:
name: linux-ctest-output
path: build/Testing/Temporary/LastTest.log
- name: Capture build folder as artifact
if: ${{ failure() }}
shell: bash
run: tar cfz build.tgz build
- name: Upload build folder
if: ${{ failure() }}
uses: actions/upload-artifact@v3
with:
name: linux-build-folder
path: build.tgz
- name: Capture installed folder as artifact
shell: bash
run: |
tar -c -v -f cwipc_ubuntu2004_${{ github.ref_name }}.tgz -z -C installed .
ls -l
- name: Upload installed folder
uses: actions/upload-artifact@v3
with:
name: cwipc_ubuntu2004_${{ github.ref_name }}.tgz
path: cwipc_ubuntu2004_${{ github.ref_name }}.tgz

create-release:
if: github.event_name == 'push' && contains(github.ref, 'refs/tags/')
name: create-release
runs-on: ubuntu-latest
needs:
- build-windows
- build-macos
- build-linux
steps:
- name: Checkout code
uses: actions/checkout@v2
with:
submodules: 'recursive'
lfs: 'true'
- name: Download Windows installed folder
uses: actions/download-artifact@v3
with:
name: cwipc_win1064_${{ github.ref_name }}.zip
path: ./
- name: Download MacOS installed folder
uses: actions/download-artifact@v3
with:
name: cwipc_osx1014_${{ github.ref_name }}.tgz
path: ./
- name: Download Linux installed folder
uses: actions/download-artifact@v3
with:
name: cwipc_ubuntu2004_${{ github.ref_name }}.tgz
path: ./
- name: Check what is there
shell: bash
run: ls -l
- name: Create Release
uses: actions/create-release@v1
id: create_release
env:
GITHUB_TOKEN: ${{ github.token }}
with:
draft: true
prerelease: false
release_name: ${{ github.ref }}
tag_name: ${{ github.ref }}
- name: Upload Windows installed folder
uses: actions/upload-release-asset@v1
env:
GITHUB_TOKEN: ${{ github.token }}
with:
upload_url: ${{ steps.create_release.outputs.upload_url }}
asset_path: cwipc_win1064_${{ github.ref_name }}.zip
asset_name: cwipc_win1064_${{ github.ref_name }}.zip
asset_content_type: application/zip
- name: Upload MacOS installed folder
uses: actions/upload-release-asset@v1
env:
GITHUB_TOKEN: ${{ github.token }}
with:
upload_url: ${{ steps.create_release.outputs.upload_url }}
asset_path: cwipc_osx1014_${{ github.ref_name }}.tgz
asset_name: cwipc_osx1014_${{ github.ref_name }}.tgz
asset_content_type: application/gzip
- name: Upload Linux installed folder
uses: actions/upload-release-asset@v1
env:
GITHUB_TOKEN: ${{ github.token }}
with:
upload_url: ${{ steps.create_release.outputs.upload_url }}
asset_path: cwipc_ubuntu2004_${{ github.ref_name }}.tgz
asset_name: cwipc_ubuntu2004_${{ github.ref_name }}.tgz
asset_content_type: application/gzip
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
installed
build*
build*/
26 changes: 18 additions & 8 deletions .gitlab-ci.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
variables:
GIT_SUBMODULE_STRATEGY: recursive
GIT_STRATEGY: clone # Only enable this line to fix problems with multiple branches in submodules

GIT_STRATEGY: clone
GIT_DEPTH: 999
stages:
- build
- test
Expand All @@ -15,28 +15,33 @@ build_linux:
- ubuntu2004
- vrtogether
before_script:
- git fetch --tags
- git describe --match 'v*'
- ./scripts/install-3rdparty-ubuntu2004.sh
script:
- rm -rf installed
- mkdir -p installed/lib
- export LD_LIBRARY_PATH="`pwd`/installed/lib:$LD_LIBRARY_PATH"
- instdir=`cd installed ; pwd`
- export LD_LIBRARY_PATH="$instdir/lib:$LD_LIBRARY_PATH"
- rm -rf build
- mkdir build
- cd build
- cmake .. -DCMAKE_INSTALL_PREFIX=$instdir
- make
- make test
- make install
- cmake --build .
# There is a problem with Python open3d on the flauwte CI/CD runner.
- ctest -E '(cwipc_tilecolor)|(cwipc_calibrate)'
- cmake --install .
- cd ..
- tar --directory installed -cvaf cwipc_ubuntu2004_$CI_COMMIT_TAG.tgz .
- touch successful.tmp
after_script:
- if [ ! -e successful.tmp ]; then cp build/Testing/Temporary/LastTest.log ctest_log_ubuntu2004.log; fi
- if [ ! -e successful.tmp ]; then tar cfz build_folder_ubuntu2004.tgz build; fi
- ./build/venv/bin/pip list
artifacts:
when: always
paths:
- cwipc_ubuntu2004_$CI_COMMIT_TAG.tgz
- ctest_log_ubuntu2004.log
- build_folder_ubuntu2004.tgz

build_osx:
stage: build
Expand All @@ -46,6 +51,8 @@ build_osx:
- osx1015
- vrtogether
before_script:
- git fetch --tags
- git describe --match 'v*'
- ./scripts/install-3rdparty-osx1015.sh
script:
- mkdir -p installed/lib
Expand All @@ -63,6 +70,7 @@ build_osx:
- touch successful.tmp
after_script:
- if [ ! -e successful.tmp ]; then cp build/Testing/Temporary/LastTest.log ctest_log_osx1015.log; fi
- ./build/venv/bin/pip list
artifacts:
when: always
paths:
Expand All @@ -79,6 +87,8 @@ build_win:
variables:
OPENGL_IS_BROKEN: "true" # See issue #19
before_script:
- git fetch --tags
- git describe --match 'v*'
- ./scripts/install-3rdparty-win1064.sh
script:
- mkdir -p installed/bin
Expand Down
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- cwipc_get_version() added to API, CWIPC_API_VERSION updated.

### Changed

- (kinect) cameraconfig is now independent of whether we use map\_color\_to\_depth or not
- (kinect) implemented function generate_point_cloud_v2 which uses a lookup table to compute the pointcloud.
- (kinect) opencv is now used for filtering the depth map: Thresholds + erosion. much faster.
- (macos) MacOS 10.15 is now the target platform: 10.14 is no longer easily supported in brew.
- Build process, CMakefiles and versioning have been streamlined.
- Github is now the primary hosting platform and Gitlab is considered a mirror.

### Removed

Expand Down
Loading

0 comments on commit cfdd039

Please sign in to comment.