Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

win: patch from OSGeo4W applied #4121

Open
wants to merge 16 commits into
base: main
Choose a base branch
from
Open

Conversation

landam
Copy link
Member

@landam landam commented Jul 31, 2024

@landam landam self-assigned this Jul 31, 2024
@landam landam marked this pull request as draft July 31, 2024 17:01
@landam landam added this to the 8.4.1 milestone Jul 31, 2024
@landam landam added the windows Microsoft Windows specific label Jul 31, 2024
@github-actions github-actions bot added the CI Continuous integration label Jul 31, 2024
@neteler
Copy link
Member

neteler commented Jul 31, 2024

A dependency yet missing?

From the CI log:

...
configure: error: *** couldn't find libpng-config
Error: Process completed with exit code 1.

@nilason
Copy link
Contributor

nilason commented Jul 31, 2024

A dependency yet missing?

From the CI log:

...
configure: error: *** couldn't find libpng-config
Error: Process completed with exit code 1.

Not a dependency, but https://github.com/OSGeo/grass/blob/main/mswindows/osgeo4w/libpng-config (see #2679).

ldesousa and others added 4 commits July 31, 2024 21:46
Add basic tests for GeoPackage and Shapefile export which use import to test the result (so they test round trip but focus on the export).
@jef-n
Copy link
Contributor

jef-n commented Aug 1, 2024

Note that build_osgeo4w.sh and osgeo4w.yml are GRASS things, that are not used in osgeo4w - they were just aligned with package.sh, which actually is.

@landam landam changed the title Patch from OSGeo4W applied win: patch from OSGeo4W applied Aug 4, 2024
@landam
Copy link
Member Author

landam commented Aug 4, 2024

After reintroducing mingw-w64-x86_64-libpng the CI pass successfully.

@jef-n Please could you elaborate why you are removing this dependency in the workflow (https://github.com/jef-n/OSGeo4W/blob/master/src/grass/osgeo4w/patch#L9)? On the other side it's installed in package.sh (https://github.com/jef-n/OSGeo4W/blob/master/src/grass/osgeo4w/package.sh#L101)

@jef-n
Copy link
Contributor

jef-n commented Aug 5, 2024

After reintroducing mingw-w64-x86_64-libpng the CI pass successfully.

@jef-n Please could you elaborate why you are removing this dependency in the workflow (https://github.com/jef-n/OSGeo4W/blob/master/src/grass/osgeo4w/patch#L9)? On the other side it's installed in package.sh (https://github.com/jef-n/OSGeo4W/blob/master/src/grass/osgeo4w/package.sh#L101)

um, osgeo4w has libpng. not sure why it was only removed from build_osgeo4w.sh. As said I don't use that. Maybe I wanted to replace msys libpng with's OSGeo4W's png, replaced it in both and failed to re-add it in build_osgeo4w.sh, when I found that it was not working. But I don't recall… In OSGeo4W it's just a patch to an unused file ;)

jef-n added a commit to jef-n/OSGeo4W that referenced this pull request Aug 5, 2024
@jef-n
Copy link
Contributor

jef-n commented Aug 5, 2024

There was a change in libpng that broke our copy of libpng-config - adapted.

@landam
Copy link
Member Author

landam commented Aug 22, 2024

Any idea why osgeo4w CI is failing on tests while compilation is successful? It seems to be unrelated to this PR...

@echoix
Copy link
Member

echoix commented Aug 22, 2024

There were a lot more failed files, like 58% of files passed, instead of 81%

@github-actions github-actions bot added GUI wxGUI related docker Docker related vector Related to vector data processing raster Related to raster data processing temporal Related to temporal data processing Python Related code is in Python labels Aug 22, 2024
@landam landam removed tests Related to Test Suite raster3d notebook labels Aug 22, 2024
@landam
Copy link
Member Author

landam commented Aug 22, 2024

Any idea why osgeo4w CI is failing on tests while compilation is successful? It seems to be unrelated to this PR...

Solved by b435503

@landam landam marked this pull request as ready for review August 22, 2024 20:52
@echoix
Copy link
Member

echoix commented Aug 22, 2024

There seems to be a bad rebase somewhere, there are some unrelated commits in the PR. (It doesn't really matter if we adjust the squashed commit message)

Was there a logic in the new order of the OSGeo4W dependencies, was it in the patch that it was in that order? It's not alphabetical neither, I don't find the logic. I'll try to compare manually which ones are changed, as word-based diff highlighting doesn't show the changes clearly.

Copy link
Contributor

@nilason nilason left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me in general. I have some thoughts on one change, which I'd like to look into though.

@@ -234,7 +234,7 @@ USE_PTHREAD = @USE_PTHREAD@
#OpenMP
OPENMP_INCPATH = @OPENMP_INCPATH@
OPENMP_LIBPATH = @OPENMP_LIBPATH@
OPENMP_LIB = @OPENMP_LIB@
OPENMP_LIB = @OPENMP_CFLAGS@ @OPENMP_LIB@
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@OPENMP_CFLAGS@ shouldn't be needed to be added to OPENMP_LIB (?!), OPENMP_CFLAGS should be added to Makefiles that need it (via EXTRA_CFLAGS).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just tested successfully locally without this change. There should be no need to add OPENMP_CFLAGS to OPENMP_LIB.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the contrary, I tried using the package.sh build on CI, more closer to OSGeo4W build, and I had some gomp errors: https://github.com/echoix/grass/actions/runs/10641753457/job/29503311365?pr=207#step:9:1864

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we document somewhere why and which DLL is needed to include in the winGRASS package? IIRC I was involved in a trial/error-way to check which of the DLLs we added finally back then.

@@ -248,7 +241,9 @@ if [ -n "$PACKAGE_PATCH" ]; then

# copy dependencies (TODO: to be reduced)
cp -uv $DLLS apps/grass/grass$POSTFIX/bin
cp -uv /mingw64/etc/fonts/fonts.conf apps/grass/grass$POSTFIX/etc

# copy R batch files
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be checked whether we include these files already in another way, e.g. in the NSIS installer-

@@ -8,6 +8,8 @@ REM Uncomment if you want to use Bash instead of Cmd
REM Note that msys package must be also installed
REM set GRASS_SH=%OSGEO4W_ROOT%\apps\msys\bin\sh.exe

set PYTHONPATH=%OSGEO4W_ROOT%\apps\grass\grass@POSTFIX@\etc\python;%PYTHONPATH%
set GRASS_COMPATIBILITY_TEST=0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does this mean?

Copy link
Member

@hellik hellik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see my inline comments for a final check

@nilason
Copy link
Contributor

nilason commented Aug 26, 2024

Was there a logic in the new order of the OSGeo4W dependencies, was it in the patch that it was in that order? It's not alphabetical neither, I don't find the logic. I'll try to compare manually which ones are changed, as word-based diff highlighting doesn't show the changes clearly.

The current changes are:

 cairo-devel
-fftw
 freetype-devel
 gdal-devel
-gdal-ecw
-gdal-mrsid
 geos-devel
+libjpeg-turbo-devel
 liblas-devel
 libpng-devel
 libpq-devel
 libtiff-devel
-libxdr
 netcdf-devel
-pdal-devel
-pdcurses
 proj-devel
-python3-matplotlib
+python3-core
 python3-numpy
 python3-ply
 python3-pywin32
+python3-six
 python3-wxpython
-regex-devel
+sqlite3-devel
 zstd-devel

@echoix
Copy link
Member

echoix commented Aug 26, 2024

Thanks @nilason! I don't have a strong opinion on the changed dependencies, apart from matplotlib that I thought it was included as a dependency in other builds, but I don't know if it is really used in our modules or just in addons...

@echoix
Copy link
Member

echoix commented Sep 21, 2024

I fixed the conflicts, and took the new deps and sorted them into the action (it would have worked without the multiline list too, but now the diff is clear)

There was one conflict in the configure flags that was recently changed to just have --with-lapack, whilst here it had an include path. Please make sure you agree with my choice

@echoix
Copy link
Member

echoix commented Sep 23, 2024

Fixed conflicts

@nilason
Copy link
Contributor

nilason commented Oct 2, 2024

@landam Let us continue the OpenMP issue from ML here.

Extracting from the logs from CI and https://wingrass.fsv.cvut.cz reveals the following:

Win CI runner (main)

checking for omp.h... yes
checking for omp_get_num_threads in -lomp... no
checking for GOMP_parallel_start in -lgomp... yes
checking for x86_64-w64-mingw32-gcc option to support OpenMP... -fopenmp

x86_64-w64-mingw32-gcc -I/c/OSGeo4W/include -pipe   -I/c/OSGeo4W/include -I/d/a/grass/grass/dist.x86_64-w64-mingw32/include -I/d/a/grass/grass/dist.x86_64-w64-mingw32/include  -D_FILE_OFFSET_BITS=64 -I/c/OSGeo4W/include -I/c/OSGeo4W/include -fopenmp -DPACKAGE=\""grasslibs"\"   -I/c/OSGeo4W/include -I/d/a/grass/grass/dist.x86_64-w64-mingw32/include -I/d/a/grass/grass/dist.x86_64-w64-mingw32/include -DRELDIR=\"lib/rst/interp_float\" -o OBJ.x86_64-w64-mingw32/segmen2d_parallel.o -c segmen2d_parallel.c
x86_64-w64-mingw32-gcc -shared -o /d/a/grass/grass/dist.x86_64-w64-mingw32/lib/libgrass_interpfl.8.5.dll -L/d/a/grass/grass/dist.x86_64-w64-mingw32/lib -L/d/a/grass/grass/dist.x86_64-w64-mingw32/lib -Wl,--export-dynamic,--enable-runtime-pseudo-reloc  -L/c/OSGeo4W/lib -L/c/OSGeo4W/bin   OBJ.x86_64-w64-mingw32/distance.o OBJ.x86_64-w64-mingw32/func2d.o OBJ.x86_64-w64-mingw32/init2d.o OBJ.x86_64-w64-mingw32/input2d.o OBJ.x86_64-w64-mingw32/interp2d.o OBJ.x86_64-w64-mingw32/matrix.o OBJ.x86_64-w64-mingw32/minmax.o OBJ.x86_64-w64-mingw32/output2d.o OBJ.x86_64-w64-mingw32/point2d.o OBJ.x86_64-w64-mingw32/point2d_parallel.o OBJ.x86_64-w64-mingw32/resout2d.o OBJ.x86_64-w64-mingw32/ressegm2d.o OBJ.x86_64-w64-mingw32/secpar2d.o OBJ.x86_64-w64-mingw32/segmen2d.o OBJ.x86_64-w64-mingw32/segmen2d_parallel.o OBJ.x86_64-w64-mingw32/vinput2d.o OBJ.x86_64-w64-mingw32/write2d.o -lgrass_gis.8.5 -lintl -lgrass_raster.8.5 -lgrass_vector.8.5 -lgrass_gmath.8.5 -lgrass_dbmiclient.8.5 -lgrass_dbmibase.8.5  -lgrass_bitmap.8.5 -lgrass_qtree.8.5 -lgrass_interpdata.8.5   -lgomp  

package.sh

checking whether to use OpenMP... "yes"
checking for location of OpenMP includes... 
checking for location of OpenMP library... 
checking for omp.h... yes
checking for omp_get_num_threads in -lomp... yes
checking for x86_64-w64-mingw32-gcc option to support OpenMP... -fopenmp

x86_64-w64-mingw32-gcc -I/c/osgeo4w/include -g -O2   -I/c/osgeo4w/include -I/usr/src/grass85/dist.x86_64-w64-mingw32/include -I/usr/src/grass85/dist.x86_64-w64-mingw32/include  -D_FILE_OFFSET_BITS=64 -I/c/osgeo4w/include -I/c/osgeo4w/include -fopenmp -DPACKAGE=\""grasslibs"\"   -I/c/osgeo4w/include -I/usr/src/grass85/dist.x86_64-w64-mingw32/include -I/usr/src/grass85/dist.x86_64-w64-mingw32/include -DRELDIR=\"lib/rst/interp_float\" -o OBJ.x86_64-w64-mingw32/segmen2d_parallel.o -c segmen2d_parallel.c
x86_64-w64-mingw32-gcc -shared -o /usr/src/grass85/dist.x86_64-w64-mingw32/lib/libgrass_interpfl.8.5.dll -L/usr/src/grass85/dist.x86_64-w64-mingw32/lib -L/usr/src/grass85/dist.x86_64-w64-mingw32/lib -Wl,--export-dynamic,--enable-runtime-pseudo-reloc  -L/c/osgeo4w/lib   OBJ.x86_64-w64-mingw32/distance.o OBJ.x86_64-w64-mingw32/func2d.o OBJ.x86_64-w64-mingw32/init2d.o OBJ.x86_64-w64-mingw32/input2d.o OBJ.x86_64-w64-mingw32/interp2d.o OBJ.x86_64-w64-mingw32/matrix.o OBJ.x86_64-w64-mingw32/minmax.o OBJ.x86_64-w64-mingw32/output2d.o OBJ.x86_64-w64-mingw32/point2d.o OBJ.x86_64-w64-mingw32/point2d_parallel.o OBJ.x86_64-w64-mingw32/resout2d.o OBJ.x86_64-w64-mingw32/ressegm2d.o OBJ.x86_64-w64-mingw32/secpar2d.o OBJ.x86_64-w64-mingw32/segmen2d.o OBJ.x86_64-w64-mingw32/segmen2d_parallel.o OBJ.x86_64-w64-mingw32/vinput2d.o OBJ.x86_64-w64-mingw32/write2d.o -lgrass_gis.8.5 -lintl -lgrass_raster.8.5 -lgrass_vector.8.5 -lgrass_gmath.8.5 -lgrass_dbmiclient.8.5 -lgrass_dbmibase.8.5  -lgrass_bitmap.8.5 -lgrass_qtree.8.5 -lgrass_interpdata.8.5   -lomp  

C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: OBJ.x86_64-w64-mingw32/segmen2d_parallel.o: in function `IL_interp_segments_2d_parallel._omp_fn.0':
C:/msys64/usr/src/grass85/lib/rst/interp_float/segmen2d_parallel.c:108:(.text+0x1b2): undefined reference to `GOMP_loop_nonmonotonic_dynamic_start'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/src/grass85/lib/rst/interp_float/segmen2d_parallel.c:144:(.text+0x5a5): undefined reference to `GOMP_loop_nonmonotonic_dynamic_next'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/src/grass85/lib/rst/interp_float/segmen2d_parallel.c:144:(.text+0x5b2): undefined reference to `GOMP_loop_end'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/src/grass85/lib/rst/interp_float/segmen2d_parallel.c:421:(.text+0xd53): undefined reference to `GOMP_critical_start'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/src/grass85/lib/rst/interp_float/segmen2d_parallel.c:421:(.text+0xe57): undefined reference to `GOMP_critical_end'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/src/grass85/lib/rst/interp_float/segmen2d_parallel.c:405:(.text+0xf83): undefined reference to `GOMP_critical_start'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/src/grass85/lib/rst/interp_float/segmen2d_parallel.c:405:(.text+0xfb5): undefined reference to `GOMP_critical_end'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: OBJ.x86_64-w64-mingw32/segmen2d_parallel.o: in function `IL_interp_segments_2d_parallel':
C:/msys64/usr/src/grass85/lib/rst/interp_float/segmen2d_parallel.c:108:(.text+0x1404): undefined reference to `GOMP_parallel'

The "package" configure finds a working libomp.dll (I suppose), but try to build and link with libgomp.dll but compiles against libgomp.dll and links with libomp.dll.

This might be the result of ddl copying:

/mingw64/bin/libgomp-1.dll

The following difference is striking:

--with-libs="${OSGEO4W_ROOT_MSYS}/lib ${OSGEO4W_ROOT_MSYS}/bin" \

--with-libs="$OSGEO4W_ROOT_MSYS/lib" \

I don't see any changes that could influence failure in 8.4.1dev (compared to the 8.4.0 release):

8.4.0...releasebranch_8_4

@nilason
Copy link
Contributor

nilason commented Oct 2, 2024

osgeo4w installs libomp.dll:
https://github.com/jef-n/OSGeo4W/blob/f8ada2fe6637b1abde0a361fa8b5b2089a82e33d/src/grass/osgeo4w/package.sh#L104

which is probably why configure hooks up on it.

Perhaps

-/mingw64/bin/libgomp-1.dll
+/mingw64/bin/libomp.dll

in

/mingw64/bin/libgomp-1.dll

could be the solution?

@nilason
Copy link
Contributor

nilason commented Nov 7, 2024

At least
OpenMP issues need to be addressed for 8.4.1, I’ll add backport label for this PR as not to miss this.

@nilason nilason added the backport to 8.4 PR needs to be backported to release branch 8.4 label Nov 7, 2024
@echoix
Copy link
Member

echoix commented Nov 7, 2024

The conflicts in the PR I might have time this weekend to address them, it's not too long, it's the sorted lines that need to be chosen with respect to the changes this PR made

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport to 8.4 PR needs to be backported to release branch 8.4 CI Continuous integration windows Microsoft Windows specific
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants