Repository navigation
LAMMPS EESSI module does not use GPU on MN5-ACC #1449
Description
Hello,
I have observed that the EESSI module LAMMPS/2Aug2023_update2-foss-2023a-kokkos-CUDA-12.1.1 does not appear to utilize GPUs when running on the MN5-ACC cluster. However, when running the same simulation with a natively installed LAMMPS module on the system, the GPUs are correctly used.
When running the simulation with the EESSI module, GPU utilization remains at 0% according to nvidia-smi, and the simulation appears to run entirely on CPU. When using a locally installed LAMMPS module with similar CUDA/Kokkos support, the GPU is used correctly and the workload appears on nvidia-smi. The only difference between the runs is the LAMMPS module used. The same input and job configuration are used. I have attached both job scripts (attached native and eessi slurm job scripts) and the input used. Please let me know if additional debugging information would be helpful.
Activity
boegel commentedon Mar 10, 2026
@JonNavarroGaritano Thank you for reporting this, we'll take a closer look ASAP.
cc @ocaisa
boegel commentedon Mar 10, 2026
@JonNavarroGaritano I just gave it a quick try on MareNostrum 5 myself, with some minimal changes to the job script (to use
-q acc_debug).The job doesn't properly start because of this:
Seems like part of the input (
forcefield) is missing in the tarball you shared?I can't see anything obviously wrong: the GPU drivers seem to have been correctly exposed to EESSI, and (at least for me) LAMMPS is reporting that it's planning to use GPUs (though I'm not familiar with LAMMPS at all myself):
What I can't check if whether the exposure of the GPU drivers has been done correctly on the node where your job is being executed.
Have you tried using
acc_debuginstead ofacc_bench?Can you try including this in your job script (after calling
nvidia-cuda-mps-control, because maybe that's causing trouble somehow):When I try this in an interactive Slurm session (on
acc_debug), it seems OK:Here's the output I get from
deviceQuery:I would like to see what
deviceQueryoutputs in your case, when you run it in a job script submitted with-q acc_bench...PS: For questions like this, it's better to open an EESSI support ticket, see https://gitlab.com/eessi/support
laraPPr commentedon Mar 11, 2026
I have also tested on
acc_debugwith https://github.com/lammps/lammps/blob/develop/examples/voronoi/in.voronoi. And as you can see the GPUs were used. I tried submitting toacc_benchbut I was not able to and when I checked the documentation I could also not find it in the listed queus (https://www.bsc.es/supportkc/docs/MareNostrum5/slurm#queues-qos).I only switched the input file since it is part of the installation. and the queue the rest of the parameters were all the same.
ocaisa commentedon Mar 11, 2026
@JonNavarroGaritano I'm in the same position as @boegel , don't have the required
forcefield. I submitted my job with-q acc_ehpcand I do seebefore the failure due to the missing
forcefieldocaisa commentedon Mar 11, 2026
I managed to jump on a node before the job errored out and I see
JonNavarroGaritano commentedon Mar 11, 2026
Hi,
Thanks for your help. Sorry, I forgot to attach the forcefield file. I have attached it now. Also, I tried to run the job adding deviceQuery after the nvidia-cuda-mps-control call. However, I'm still getting the same behaviour in the nvidia-smi output. I attach the job output file, I think that detects CUDA drivers correctly. The difference between the acc_debug and acc_bench is the maximum runtime and the maximum number of processors that can be used in a job script, and I think it should not affect the behaviour.
With the input file input.in, I see 15% GPU utilization at one point, but the rest of the time it is 0%. Meanwhile, with the native lammps module, I see 55-70% GPU utilization during execution. With the file in.voronoi, execution is too short, but at one point it gave 15%, and the rest remained at 0%.
out.37670781.log
forcefield.txt
ocaisa commentedon Mar 11, 2026
I now see the same as you, I see the GPUs spike very briefly but the rest of the time they seem to be idle
ocaisa commentedon Mar 11, 2026
Just to sanity check things, I tested a GROMACS execution and the GPUs are used just fine.
I'm afraid it will be very hard to pin down this issue since LAMMPS is not complaining. I'm rebuilding that LAMMPS module myself to see if it makes a difference. Ideally I would just try a later version of LAMMPS but we don't have one yet with CUDA support in
2025.06boegel commentedon Mar 11, 2026
Should we contact upstream on this, maybe they have some pointers on how to figure this out?
ocaisa commentedon Mar 11, 2026
I would only contact upstream if you could reproduce it with a relatively recent stable release of LAMMPS. If we could have a newer version in
2025.06we could try it there.boegel commentedon Mar 11, 2026
I suspect that some critical part of LAMMPS was unintentionally built without GPU support for the installation we have in EESSI, since generally speaking things look fine.
@JonNavarroGaritano Can you provide some more details on the type of calculation you're running here, in particular which LAMMPS packages are relevant.
In addition, it would be really helpful if you can share some details on how the other LAMMPS installation that you are using was configured/built (if you have that information), that may help us to zoom into the problem...
Can you share the output of
lammps -hfor that non-EESSI installation of LAMMPS?laraPPr commentedon Mar 12, 2026
I just looked at the performance difference of one of our GPU systems with a locally installed LAMMPS and the LAMMPS module in EESSI. Compared to our local installation the EESSI seems to perform better. That seems to indicate that it is a problem with the way we build and install LAMMPS with easybuild rather than a problem with the EESSI software stack.
ocaisa commentedon Mar 12, 2026
I used another simple benchmark on
acc_ehpcand I can see 100% GPU utilisation for both the local and EESSI LAMMPS, with the EESSI one giving slightly better overall performance.in.lj.txt
log.lammps.lj.eessi.txt
log.lammps.lj.system.txt
It's something to do with how LAMMPS was built, perhaps there were patches applied to your local build? I tried to find out but I can't find any clues.
ocaisa commentedon Mar 12, 2026
I also rebuilt the EESSI version with latest EasyBuild and still saw the same behaviour.
I would like us to get to a new LAMMPS build with CUDA support in
2025.06and then try to reproduce there.JonNavarroGaritano commentedon Mar 12, 2026
Hello,
Interesting @laraPPr ! I think also that the problem could be related to LAMMPS building process. The following lines are the steps that the BSC support team carry out for the installation of the software: lammps-28Mar2023
salloc -A bsc99 -q acc_bench -c 80 --exclusive
module purge
module load openmpi/4.1.5-gcc
module load cuda
module load fftw/3.3.10-gcc-ompi
module load cmake
cd /apps/ACC/LAMMPS/SRC/lammps-28Mar2023
rm -rf build
mkdir build; cd build
cmake
-DCMAKE_BUILD_TYPE="RelWithDebInfo"
-DCMAKE_CXX_COMPILER="/apps/ACC/LAMMPS/SRC/lammps-28Mar2023/lib/kokkos/bin/nvcc_wrapper"
-DCMAKE_C_COMPILE="gcc"
-DCMAKE_Fortran_COMPILER="gfortran"
-DMPI_CXX="g++"
-DMPI_CXX_COMPILER="mpicxx"
-DCUFFT_LIBRARY="/usr/local/cuda-12.2/lib64/libcufft.so"
-DCMAKE_CXX_STANDARD=17
-DCMAKE_CXX_STANDARD_REQUIRED=ON
-DLAMMPS_MACHINE="mpi"
-DBUILD_MPI=ON
-DBUILD_OMP=ON
-DGPU_API=cuda
-DGPU_ARCH="sm_90"
-DKokkos_ENABLE_CUDA=ON
-DKokkos_ARCH_HOPPER90=ON
-DKokkos_ENABLE_SERIAL=ON
-DKokkos_ENABLE_OPENMP=ON
-DKokkos_ENABLE_DEBUG=ON
-DFFT_KOKKOS="CUFFT"
-DPKG_KOKKOS=ON
-DPKG_MOLECULE=ON
-DPKG_KSPACE=ON
-DPKG_RIGID=ON
-DPKG_MANYBODY=ON
-DPKG_GPU=ON
../cmake
cmake --build .
cmake -DCMAKE_INSTALL_PREFIX=/apps/ACC/LAMMPS/28Mar2023/GCC/OPENMPI -P cmake_install.cmake
Let me know if you need something additionally.
Thanks for your help,
laraPPr commentedon Mar 12, 2026
I think we should look at the
-DKokkos_*flags we have also updated build procedure recently. So I'm also curious what we would see with a newer lammps in 2025.06. At the moment I'm still having trouble with building on 1 ARM target (see easybuilders/easybuild-easyconfigs#25133)I'll also dump our cmake flags with which we build.
boegel commentedon Mar 12, 2026
@JonNavarroGaritano Thanks for providing those details, that should help us quite a bit.
At first sight:
-DPKG_GPUin our build, see also https://docs.lammps.org/Packages_details.html#pkg-gpu . I'm not sure why that's the case, and I don't think that explains the problem being reported here (because-sf kkis used in the LAMMPS command, not-sf gpu)DFFT_KOKKOS="CUFFT"and-DCUFFT_LIBRARY=...), which may have a significant impact on performance as well for GPU installations of LAMMPS.cuFFT is part of CUDA, so changing/fixing this should not be very difficult I think...
boegel commentedon Mar 12, 2026
The LAMMPS easyblock that is used to install LAMMPS with EasyBuild in EESSI was updated relatively recently to be aware of the
-DFFT_KOKKOSconfigure option, but we only use it for LAMMPS versions 29Aug2024 and newer:FFT_KOKKOSconfigure option easybuilders/easybuild-easyblocks#3784Maybe that was a bit too conservative, and we should relax that a bit, so it's also used for LAMMPS 2Aug2023?
A more promising avenue perhaps is to focus on getting a more recent version of LAMMPS deployed into EESSI, like
LAMMPS-22Jul2025-foss-2024a-kokkos-CUDA-12.6.0.eb.@JonNavarroGaritano Would that work for you, or do you need LAMMPS 2Aug2023 specifically for some reason?
JonNavarroGaritano commentedon Mar 13, 2026
Hi @boegel ,
Thanks for the clarification!
For our use case I don't specifically need LAMMPS 2Aug2023. What I really need is simply a functional LAMMPS build with GPU support (Kokkos/CUDA) in EESSI. I don't have any strict requirement regarding the exact version.
So if deploying a more recent version like LAMMPS-22Jul2025 with proper GPU support works better, that would also work perfectly for us. For your information, we currently have the 2023/06 (in which we have the LAMMPS/2Aug2023_update2-foss-2023a-kokkos-CUDA-12.1.1) and 2025.06 EESSI versions installed in MN5.
laraPPr commentedon Apr 7, 2026
We have recently deployed LAMMPS-22Jul2025 for CPU only and I will now start looking into getting it installed with CUDA support as well. See #1461.