SYCL exception thrown while running on Intel(R) Data Center GPU Max 1100

[sidarth-narayanan-csi]

Hello All,

I am trying to run a simple dot product program (pasted below onemath_dot.cpp) using oneMath on Intel and NVIDIA GPUs. I run into the following error during the same.

snarayanan@intel-eagle:scratch$ ./onemath_dot
Running on: Intel(R) Data Center GPU Max 1100
SYCL exception: The program was built for 1 devices
Build program log for 'Intel(R) Data Center GPU Max 1100':

The program runs fine on a NVIDIA A100 and AMD Radeon RX 7900 GRE . It also runs fine if I set the ONEAPI_DEVICE_SELECTOR=opencl:cpu and I did enable MKLCPU and MKLGPU backends while buidling oneMath (v0.6).

Here is the cmake command used:

cmake ../../oneMath_src -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_C_COMPILER=$CC -DENABLE_MKLCPU_BACKEND=True -DENABLE_MKLGPU_BACKEND=True -DENABLE_CUBLAS_BACKEND=True -DENABLE_CUSOLVER_BACKEND=True -DENABLE_CUFFT_BACKEND=True -DENABLE_CURAND_BACKEND=True -DENABLE_CUSPARSE_BACKEND=True -DBUILD_EXAMPLES=False -DBUILD_FUNCTIONAL_TESTS=False

Here are some information about the build and run environment:

• Compiler used = ICPX 2025.0.0
• oneAPI toolkit version 2025.0 (with corresponding Codeplay plugins)
• CUDA toolkit version 12.6
• OS : Rocky Linux 9.4

Here is the program:

#include <sycl/sycl.hpp>
#include "oneapi/math.hpp"
#include <iostream>
#include <vector>

int main()
{
// Define the size of the vectors
const size_t N = 1024;

// Create SYCL queue
sycl::queue q(sycl::default_selector{});

std::cout << "Running on: " << q.get_device().get_info<sycl::info::device::name>() << "\n";

// Allocate Unified Shared Memory (USM) for input and output
float *x = sycl::malloc_shared<float>(N, q);
float *y = sycl::malloc_shared<float>(N, q);
float *result = sycl::malloc_shared<float>(1, q);

// Initialize vectors x and y
for (size_t i = 0; i < N; i++)
{
x[i] = static_cast<float>(i + 1);
y[i] = static_cast<float>(N - i);
}

// Set initial result to 0
*result = 0.0f;

try
{
// Compute the dot product using oneMath
oneapi::math::blas::column_major::dot(q, N, x, 1, y, 1, result).wait();

// Print the result
std::cout << "Dot product result: " << *result << std::endl;
}
catch (sycl::exception const &e)
{
std::cerr << "SYCL exception: " << e.what() << std::endl;
}

// Free USM memory
sycl::free(x, q);
sycl::free(y, q);
sycl::free(result, q);

return 0;
}

Here is the compile command used:

icpx -fsycl -I$ONEMATH_DIR/include/ -L$ONEMATH_DIR/lib -Wno-deprecated-declarations -fsycl-targets=spir64_gen -Xs '-device xe-hpc' -o onemath_dot onemath_dot.cpp -lonemath

Any help on this matter is highly appreciated.

Please let me know if more details are needed.

Thanks & Regards,
Sidarth Narayanan
Software Development Engineer - HPC Group
Convergent Science Inc
[email protected]

[Rbiessy]

Hello @sidarth-narayanan-csi, thank you for the question.

Can you confirm, is there no more errors after the message "Build program log for..."?
I think the issue is that you are setting -fsycl-targets=spir64_gen -Xs '-device xe-hpc'. In particular I have doubts about using -Xs '-device xe-hpc' for a GPU Max 1100, this may not be the right architecture and cause the runtime build failures. What happens if you don't set this flag?

Do you need AOT compilation? oneMath does not compile many SYCL kernels but setting -fsycl-targets=spir64_gen will not work for Nvidia or AMD devices if oneMath needs to run a SYCL kernel. The safest would be to also remove this flag.
If you really need AOT compilation and want to run on multiple devices you need to set multiple targets like -fsycl-targets=spir64_gen,nvidia_gpu_sm_80,amd_gpu_gfx90a for instance (may need different values for your devices). This is not a tested configuration so I have low confidence this would work first try.

I am able to build and run a single binary of your sample on both Nvidia A100 and Intel PVC using the same compilation flags without -fsycl-targets=spir64_gen -Xs '-device xe-hpc' and building all the oneMath domains with:

cmake -Bbuild-mkl-cuda -GNinja -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER=icpx -DBUILD_FUNCTIONAL_TESTS=OFF -DENABLE_CUBLAS_BACKEND=ON -DENABLE_CURAND_BACKEND=ON -DENABLE_CUSOLVER_BACKEND=ON -DENABLE_CUFFT_BACKEND=ON -DENABLE_CUSPARSE_BACKEND=ON -DCMAKE_INSTALL_PREFIX=/path/to/onemath/install
ninja -Cbuild-mkl-cuda install

You likely don't need to enable all the domains though.

[sidarth-narayanan-csi]

Hello @Rbiessy ,

Thank you for the response.

Can you confirm, is there no more errors after the message "Build program log for..."?

Yes, I can confirm that there is nothing below that error message.

snarayanan@intel-eagle:scratch$ ./onemath_dot
Running on: Intel(R) Data Center GPU Max 1100
SYCL exception: The program was built for 1 devices
Build program log for 'Intel(R) Data Center GPU Max 1100':

snarayanan@intel-eagle:scratch$

Do you need AOT compilation?

We do want to do AOT compile if and when possible.

What happens if you don't set this flag?

I get the same error even without setting the flag (no -fsycl-targets option passed) .

I also tried the new AOT compile flags (DPC++ Users Manual) for all three vendors and was able to compile and run successfully on NVIDIA and AMD GPUs but again failed with the same error on Intel GPU.

Compile Command:

icpx -fsycl -I$ONEMATH_DIR/include/ -L$ONEMATH_DIR/lib -Wno-deprecated-declarations -fsycl-targets=intel_gpu_pvc,amd_gpu_gfx1100,nvidia_gpu_sm_80 -o onemath_dot onemath_dot.cpp -lonemath

NVIDIA, AMD and INTEL GPU Outputs:

snarayanan@nvidia-icarus:scratch$ ./onemath_dot
Running on: NVIDIA A100-PCIE-40GB
Dot product result: 1.79482e+08


snarayanan@amd-pegasus:scratch$ ./onemath_dot
Running on: Radeon RX 7900 GRE
Dot product result: 1.79482e+08


snarayanan@intel-eagle:scratch$ ./onemath_dot
Running on: Intel(R) Data Center GPU Max 1100
SYCL exception: The program was built for 1 devices
Build program log for 'Intel(R) Data Center GPU Max 1100':

I also tried writing a simple dot product SYCL kernel and used the same compile command and that seems to work just fine.

If possible, Could you please confirm if my build procedure for oneMath seems right to you ?

Thank you for for looking into this.

-Sidarth

[Rbiessy]

Could you please confirm if my build procedure for oneMath seems right to you ?

Yes the CMake command to build oneMath look correct to me. What does the value $CXX evaluate to? It should be set to icpx (or the full path to icpx if it is not in the PATH variable).

I can't tell what could be wrong. I would suggest to build the sample and oneMath in debug mode and run it with gdb to find out what is throwing the exception. If you can catch throw and share the gdb backtrace this would give more information to understand the issue. You could also consider building a more recent version of intel/llvm, this may print more information somehow.
What is surprising is that the dot operation should not build any SYCL kernel so I don't understand how the sample can run into the exception you pasted. On Intel GPU, oneMath should end up calling Intel oneMKL directly (here).

[sidarth-narayanan-csi]

@Rbiessy We contacted Intel for a separate issue about their GPU debuggers and that led us redo the drivers from scratch which seemed to solve this issue too.

Thank you for your assistance with this matter.