Tutorials

• 1: CMake
• 2: FEBio
• 2.1: CAD Modeling Instructions for Pelvic Hip Fracture
• 2.2: Meshing Instructions for Hip Model
• 2.3: Working with Simmetrix Meshes
• 3: Hugo
• 4: Kokkos
• 4.1: Building Kokkos with CMake on SCOREC (rhel9) machines
• 4.2: Building kokkos-tutorials
• 5: MFEM
• 6: Omega_h
• 7: PCMS
• 8: Sterllerator Related Tools
• 9: TOMMS
• 10: XGC

1 - CMake

Get CMake on SCOREC machines (rhel 7)

 module unuse /opt/scorec/spack/lmod/linux-rhel7-x86_64/Core
 module use /opt/scorec/spack/v0154_2/lmod/linux-rhel7-x86_64/Core
 module load cmake/3.20.0

Learn CMake

• Professional CMake: A Practical Guide: it’s available for reading on LACES Zotero library.
• CMake Cookbook: This github repo has a lot of examples and practice on the given files follwoing the book.

2 - FEBio

2.1 - CAD Modeling Instructions for Pelvic Hip Fracture

Note: Ensure that you follow the steps faithfully to avoid issues and achieve desired results

Step 1: Opening and Preprocessing in Autodesk Meshmixer

1. Open M11_cg_bone_lpelvis.obj in Autodesk Meshmixer.
2. Select the entire geometry.
3. Navigate to Edit > Remesh to reduce the chance of overlapping during offsetting.
4. Accept the Remesh operation.
5. Continue with Edit > Extrude > Offset +1.77 mm (direction: normal).
6. Select the geometry again and perform Edit > Remesh (optional but beneficial).

Step 2: Exporting and Transferring to Siemens NX

1. Export the model as an .obj file.
2. Transfer the file to Siemens NX on SCOREC computers using WinSCP or hard drive.
3. Save and close the file to individually save convergent bodies as separate .prt files.

Step 3: Performing Subtractions in Siemens NX

1. Open one file and import the second (outer and inner).
2. Navigate to File > Start > Modeling.
3. Select the outer part and use the Subtract operation with settings:
   • Subtract Tool from Target. Keep Tool operation.*
4. Repeat the subtraction with the sacrum until a full shell model is created.

Step 4: Processing the Sacrum Shell

1. To remove the right half of the sacrum: Place a Datum CSYS in the middle of the sacrum shell using Home > Datum CSYS and position the datum such that there is left-right symmetry.
2. Trim the body using Trim Body command after selecting the appropriate plane. You may have to select the Reverse Direction option to leave the desired left half remaining.
   • It is advised to trim the cancellous and cortical parts separately.
3. Save as a .prt.

Step 5: Facet Cleanup and Hole Insertion

1. In NX, Open the cartilage (m11_cg_jnt_lsi.obj) and save it as a part file.
2. Create a new model in Siemens NX and ensure that units are in mm.
3. Import the Sacrum, Cartilage, and Pelvis parts.
4. Use Facet > Cleanup Facet Body to clean mesh.
5. Delete all minor extra parts that may appear.
6. Add a fracture to the sacrum (Note: trim body command would be more efficient, but has proven to be problematic with regard to the meshing)
   • First, duplicate the entire geometry as it currently exists and make sure its copy is superimposed upon the original
   • Create another Datum CSYS, place it where shear would occur
   • Create a sketch of a large square of arbitrary size, as long as it encompasses the geometry
   • Extrude it in one direction (right if observing from the front) such that it covers half of the sacrum and all of the pelvis
   • Use the subtract command to subtract the box from the part of the bone (choose one of the superimposed copies) which it overlaps
   • Repeat the process for the other half, this time deleting from the other superimposed half
   • This will leave you with one right side and one left side, leaving you with the geometry you had after Step 4, but with a “fracture”
7. Insert a hole:
   • Place a Datum CSYS and then Create Sketch on Plane.
   • Draw a rectangle going into/through sacrum (~100mm length, 3.66 – 3.67mm radius).
   • Revolve the rectangle
   • Select the cylinder and use Facet Body from Body to make the new cylinder convergent.
   • Use Subtract Tool to remove the cylinder from the bone model (one part at a time).
   • Repeat for the second hole, which should not be parallel to the first.

Step 6: Inserting Screw and Final Steps

1. Open the screw (screw-7-74.prt) and remove the Split Bodies and Datum Planes.
2. Import the updated screw part using Assemblies > Add Component.
3. Align the screw to the hole using Assembly constraints (Touch Align the centerlines first and then use Distance Align).
4. Save the entire model as a .prt file.
5. Export the final model as a Parasolid binary file .x_b

2.2 - Meshing Instructions for Hip Model

1. Importing Geometry:

   • Open SimModeler
   • If the file is in parasolid format, go to File → Import Geometry. For files in .obj or .stl format, choose Import Discrete Data.

2. Preparing for meshing:

   • Navigate to the Modeling tab, choose Delete Parts, and select the screws.
   • In the Meshing tab, click on Generate Mesh. Uncheck “Structured Mesh” and proceed. Upon successful generation, view the mesh by selecting Show Mesh, which creates an .sms file. (Remember to return to the .smd file for generating a new mesh if required.)

3. Refining Mesh:

   • Use Mesh Attributes located on the right-hand side of the screen.
   • The following chart represents some suggested Mesh Attributes and corresponding values:

     Type	Sub-Type	Value
     Mesh Size	Absolute	0.001
     Mesh Curvature Refinement	Absolute	0.005
     Volume Shape Metric	Aspect Ratio	3.0
     Allow Refinement For Shape	Relative	0.0001

   • After making refinements, regenerate the mesh to incorporate the changes.

4. Quality Check:

   • Assess the quality of the mesh by going to the .sms file and navigating to the Display Tab.
   • Click Mesh Stats and scrutinize the Aspect Ratio.

5. Exporting Mesh:

   • To export the mesh, navigate to File > Export Mesh.
   • Save the file if not done previously.
   • Choose the export format as Abaqus 3D (.inp).

6. Meshing Screws Separately:

   • Follow the same steps as for the bone meshing process for the screws.

2.3 - Working with Simmetrix Meshes

3 - Hugo

Install Hugo on Windows

1. Download the latest Hugo binary from here or v0.119 directly from here.

2. Extract the zip file to a folder of your choice. For example, C:\Hugo\bin.

3. Add the path to the Hugo binary to your PATH environment variable. For example, C:\Hugo\bin.

   1. Search for “Edit the system environment variables” in the Windows search bar.
   2. Click on “Environment Variables…”.
   3. Under “System variables”, select “Path” and click “Edit…”.
   4. Click “New” and add the path to the Hugo binary. For example, C:\Hugo\bin.
   5. Click “OK” to save the changes.

4. Open a new command prompt and run hugo version to verify that Hugo is installed properly.

5. You can now delete the zip file.

Install Go on Windows (it is required for Hugo)

1. Download the latest Go binary from here.
2. Run the installer.

Install Git on Windows (it is required for Hugo)

1. Download the latest Git binary from https://git-scm.com/download/win.
2. Run the installer.

Run Hugo on Windows

1. Open git bash. You can find it in the Windows search bar. Do not use the Windows command prompt or PowerShell.
2. Clone the repository: git clone your-repository-url.
3. Navigate to the repository: cd your-repository-name.
4. Run Hugo: hugo server.
5. Copy the URL from the output and paste it in your browser.

4 - Kokkos

4.1 - Building Kokkos with CMake on SCOREC (rhel9) machines

1. Clone the Kokkos Repository: Create a new directory called Kokkos and clone Kokkos repository from GitHUb in that directory. You can do this using Git:

git clone https://github.com/kokkos/kokkos.git

2. Load required modules: Kokkos requires compiler and cmake to build. Load the following modules. (Note: The modules are as per rhel9 machines. You can’t use it on rhel7 machines.)

module use /opt/scorec/spack/rhel9/v0201_4/lmod/linux-rhel9-x86_64/Core/
module load gcc/12.3.0-iil3lno mpich/4.1.1-xpoyz4t cuda/12.1.1-zxa4msk
module load cmake/3.20.0

3. Configure with CMake: Create a new file config.sh (you can name anything) in a source directory. Inside config.sh, add the following CMake instructions:

cmake -S . \
     -B build \
     -DCMAKE_CXX_COMPILER=g++ \
     -DBUILD_SHARED_LIBS=ON \
     -DCMAKE_INSTALL_PREFIX=/lore/<username>/Kokkos/Install \
     -DKokkos_ENABLE_OPENMP=ON  

4. Build and Install: Run the config.sh file with the command . config.sh and go to the build directory (cd build). Run the command make install.

5. Add in your environment variable: Add the executable to the LD_LIBRARY_PATHenvironment variable. Use the following command to add to the environment variable.

export LD_LIBRARY_PATH=/lore/<username>/Kokkos/Install/lib64:$LD_LIBRARY_PATH

4.2 - Building kokkos-tutorials

The following instructions is for building a new Kokkos library for each exercise. For more details, check Kokkos Tutorials.

1. Clone the Kokkos Tutorials Repository: The tutorials for the Kokkos C++ programming. You can clone the tutorials with the following command.

git clone https://github.com/kokkos/kokkos-tutorials.git

2. Load required modules: Kokkos tutorials requires compiler and make to build. Load the following modules. (Note: The modules are as per rhel9 machines. You can’t use it on rhel7 machines.)

module use /opt/scorec/spack/rhel9/v0201_4/lmod/linux-rhel9-x86_64/Core/
module load gcc/12.3.0-iil3lno mpich/4.1.1-xpoyz4t cuda/12.1.1-zxa4msk

3. Find the GPU architecture of your machine: To find out the GPU architecture of your machine, follow this.

4. Build with make: Inside kokkos-tutorials folder, go to Exercises folder.In each exercise folder, there is a makefile to build each exercise. Open the Makefile and make the following changes.

KOKKOS_PATH = /path/to/kokkos 
KOKKOS_DEVICES = "<your GPU language>"
KOKKOS_ARCH = "<your GPU Architecture>"

For example,

KOKKOS_PATH = /lore/<username>/Kokkos/kokkos`
KOKKOS_DEVICES = "Cuda"
KOKKOS_ARCH = "Ada89"

After making changes, run the command make -j8

5. Run the executable: Once configuring and building the exercise, run an exexutable. ```bash ./

5 - MFEM

MFEM is a free, lightweight, scalable C++ library for finite element methods.

MFEM Installation

MFEM installation information can be found here. Follow the appropriate installation instructions according to your necessity.

More detailed guide can be found in this README file. This might be needed to build MFEM with custom options for example linking with other libraries.

Installation using cmake on SCOREC machines

1. Clone the mfem repository

git clone https://github.com/mfem/mfem.git

or get the tarball from here.

2. Load the necessary modules

module unuse /opt/scorec/spack/tmod/tinux—rhet7—x86_64/Core
module use /opt/scorec/spack/v0181_1/lmod/linux—rhe17—x86_64/Core
module use /opt/scorec/modules
module load gcc/11.2.0 mpich/4.0.2
module load cmake/3.20

3. Install hypre and metis using these instructions from MFEM website. They showed the installation process for metis-4.0 but I used metis-5.1.0 and it worked fine.

4. Create a configuration file with the following content

cmake -S mfem-4.6 -B mfem-build \
  -DMETIS_DIR=metis-5.1.0 \
  -DMFEM_USE_MPI=ON \
  -DCMAKE_BUILD_TYPE=RelWithDebInfo \
  -DCMAKE_INSTALL_PREFIX=/lore/<your_username>/mFEM/install/mfem-mpi

change <your_username> to your username. It’s preferable to use the /lore directory for installation. Read more about storage management on SCOREC machines in the FAQ section.

6 - Omega_h

Omega_h is a C++14 library that implements tetrahedron and triangle mesh adaptativity, with a focus on scalable HPC performance using (optionally) MPI and OpenMP or CUDA. It is intended to provided adaptive functionality to existing simulation codes.

This is a fork of the original Omega_h repository of SANDIA Labs. The fork is maintained by the SCOREC.

Find Omega_h on SCOREC machines with CUDA and MPI enabled

1. Compiled without Gmsh: /lore/mersoj/laces-software/build/PASCAL61/omega_h/install/ PASCAL61 is the GPU architecture. Find your GPU architecture with this tutorial.
2. Compiled with Gmsh: /lore/hasanm4/Omega_H/. And the Gmsh executable is in /lore/hasanm4/Gmsh/

Remember to add the shared libraries of gmsh and omega_h to your LD_LIBRARY_PATH environment variable if you are using the compiled version of Omega_h.

Usage of Omega_h

1. Use as a library in PCMS or with any other codes. See pcms tutorial for an example.
2. Use to convert mesh files of other formats to Omega_h format. See this tutorial as a simple example.

Compile Omega_h on SCOREC machines (rhel7)

1. Load the necessary modules:

# you may also need to remove all the previously loaded modules
#module purge
module unuse /opt/scorec/spack/tmod/tinux—rhet7—x86_64/Core
module use /opt/scorec/spack/v0181_1/lmod/linux—rhe17—x86_64/Core
module use /opt/scorec/modules
module load gcc/11.2.0 mpich/4.0.2
module load fftw/3.3.10
module load cuda/11.4
module load cmake/3.20
# if you want to use Symmetrix with it
module load simmetrix-simmodsuite/2023.1-230907dev # or any other version

2. Clone the repository:

git clone https://github.com/SCOREC/omega_h.git
cd omega_h

3. Create a config file with the following content:

cmake -S . -B build \
 -DCMAKE_CXX_COMPILER=`which mpicxx` \
     -DCMAKE_C_COMPILER=`which mpicc` \
     -DCMAKE_INSTALL_PREFIX=/lore/<yourUsername>/Omega_H_OMP/ \
     -DGmsh_INCLUDE_DIRS=/lore/<yourUsername>/Gmsh/include/ \
     -DKokkos_DIR=/lore/<yourUsername>/Kokkos/kokkosInstall/lib64/cmake/Kokkos \
     -DCMAKE_BUILD_TYPE=Debug \
     -DOmega_h_DBG=OFF \
     # if you want to compile with symmetrix
     -DOmega_h_USE_SimModSuite=on \
     -DSIM_MPI=mpich4.0.2 

Important notes on the flags: Change the CMAKE_INSTALL_PREFIX to your preferred directory. If you want to use Gmsh, change the Gmsh_INCLUDE_DIRS to the directory where you have Gmsh compiled and installed. If you don’t, remove the line. If you want to use CUDA or OpenMP, change the Kokkos_DIR to the directory where you have Kokkos installed. CUDA/OpenMP will depend on the Kokkos installation. If you don’t want to use CUDA or OpenMP, remove the line. If you want to compile in release mode, change the CMAKE_BUILD_TYPE to Release or RelWithDebInfo.

4. Build and install:

. <yourConfigFile>
cd build
make -j8 install

7 - PCMS

8 - Sterllerator Related Tools

This jupyter notebook shows how to open the Beams3D and BMW output and plot the magnetic fields. It is found that both of them generates the same magnetic field.

BMW

Installation on SCOREC machines (rhel9)

Note: To find the RHEL version of the machine you are on, you can use the following command:

    cat /etc/redhat-release

This is the recommended way of BMW installation since most of the SCOREC machines are now RHEL9. This process only works on RHEL 9 machines of SCOREC and was tested on 13th June 2024. Some modules may change in the future.

1. Go to your intended installation directory with cd (/lore/<username>/ recommended, see more details about space management here) and clone the repository:

    git clone https://github.com/ORNL-Fusion/Stellarator-Tools.git

2. Enter the directory and create a build directory:

    cd Stellarator-Tools
    mkdir build
    cd build

3. Load necessary modules: (ignore the warnings)

   module use /opt/scorec/spack/rhel9/v0201_4/lmod/linux-rhel9-x86_64/Core/
   module load gcc/12.3.0-iil3lno mpich/4.1.1-xpoyz4t cuda/12.1.1-zxa4msk
   module load cmake 
   module load netcdf-c netcdf-fortran openblas netlib-scalapack

4. Run cmake with the following command:

    cmake -B ./ -S ../ -DBUILD_BMW=ON -DBUILD_MAKEGRID=ON -DCMAKE_INSTALL_PREFIX=/lore/<username>/BMW

Change the installation directory as per your requirement. 5. Build and install:

    make -j4
    make install

6. To use the installed xmbw and mgrid, the shared library paths need to be updated. Execute the following export commands before using them.

    export LD_LIBRARY_PATH=$OPENBLAS_RHEL9_ROOT/lib:$LD_LIBRARY_PATH
    export LD_LIBRARY_PATH=$NETLIB_SCALAPACK_RHEL9_ROOT/lib:$LD_LIBRARY_PATH
    export LD_LIBRARY_PATH=$NETCDF_FORTRAN_RHEL9_ROOT/lib:$LD_LIBRARY_PATH
    export LD_LIBRARY_PATH=$NETCDF_C_RHEL9_ROOT/lib:$LD_LIBRARY_PATH

Note: Every time you want to use xmbw or mgrid on a new terminal, you need to export the shared library paths after loading the necessary modules(step 3).

Installation on SCOREC machines (rhel7)

This GitHub reporsiory contains the source code and brief installation guide of BMW: BMW.

Note: Direct compilation did not work for me. I used Stellarator-Tools supplied by ORNL-Fusion.

1. Clone the repository:

    git clone https://github.com/ORNL-Fusion/Stellarator-Tools.git

2. Go the the directory and create a build directory:

    cd Stellarator-Tools
    mkdir build
    cd build

3. Load the necessary modules:

    module unuse /opt/scorec/spack/lmod/linux-rhel7-x86_64/Core
    module use /opt/scorec/spack/v0201_4/lmod/linux-rhel7-x86_64/Core
    module load gcc/11.2.0-zcqgw mpich/4.1.1-6p32n

    module load netcdf-c cmake netcdf-fortran openblas netlib-scalapack

4. Run cmake: You can also follow the ccmake method from the repository README. But I recommend the following:

     cmake -B ./build -S ./ -DBUILD_BMW=ON -DCMAKE_INSTALL_PREFIX=/lore/<yourUsername>/BMW

5. Build and install:

    make -j4 install

or

    make -j4

This will create the bmw executable in the bin directory of the installation directory.

6. Update the shared libraries (xbmw will keep printing error messages until it finds all the shared libraries): (These directories may change depending on the modules loaded for installation)

    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/scorec/spack/v0201_4/install/linux-rhel7-x86_64/gcc-11.2.0/openblas-0.3.23-4kpgzbwtvvnf4m4f6rqvyclh2khpfepb/lib
    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/scorec/spack/v0201_4/install/linux-rhel7-x86_64/gcc-11.2.0/netlib-scalapack-2.2.0-w3lmjdvbshpvqiihwxm2fygyjyzu275t/lib
    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/scorec/spack/v0201_4/install/linux-rhel7-x86_64/gcc-11.2.0/netcdf-c-4.9.2-hzgyaz36ol6aqb4o3ne3xjabccpxjlo4/lib
    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/scorec/spack/v0201_4/install/linux-rhel7-x86_64/gcc-11.2.0/netcdf-fortran-4.6.0-hwoxscrowy6gh75n5ypnlr3btfver36x/lib

Usage of BMW: Run a case from VMEC Equillibria

There can be two kinds of use cases for bmw: the coil generated magnetic filed is considered or they are not considered. If you don’t need the coil generated magnetic field, you just need the wout_...nc file and follow the steps below.

1. Go to the directory where you have the case files (e.g. vmec_equillibria/NCSX/free_boundary_from_vmec_wiki):

    cd vmec_equillibria/NCSX/free_boundary_from_vmec_wiki

You can create the wout_...nc file using vmec or use the provided wout_...nc file as the input to bmw.

2. Run bmw:

    /lore/<yourUsername>/BMW/bin/xbmw -num_r=100 -num_p=20 -num_z=100 -rmax=1.5 -rmin=0.9 -zmax=0.5 -zmin=-0.5 -woutf=wout_ncsx_c09r00_free_birth.nc -outf=bmw_ncsx_c09r00_out.nc

This will produce the bmw_ncsx_c09r00_out.nc file in the current directory.

4. To read the output file, you can use python and the netCDF4 library. Here in this Jupyter Notebook file, I have plotted the magnetic field from the output file. GitHub Gist for Analysis

Using Coil Generated Magnetic Field

1. Check if the coils file given in the case directory. Generally, it’s given and name as coils.xxxxx.
2. Change the ier_flag in the wout_...nc file to 0 before starting the bmw run. You can modify .nc files in python using the netCDF4 library.

    import netCDF4 as nc
    wout = nc.Dataset('wout_ncsx_c09r00_free_birth.nc', 'r')
    wout.variables['ier_flag'][:] = 0
    wout.close()

3. BMW only accepts coils file in mgrid format. So, convert the coils.xxxxx file to mgrid format using the following command:

    /lore/<yourUsername>/BMW/bin/mgrid < input_file.txt

To know how to create an input file for mgrid, check this STELLOPT MAKEGRID Tutorial. Note that their binary name is xgrid but in the BMW installation, it’s mgrid.

4. Run bmw with the mgrid file generated in the above step. In this case, you don’t have to specify the parameters for the grid since the grid is already provided by the mgrid file.

    /lore/<yourUsername>/BMW/bin/xbmw -woutf=wout_ncsx_c09r00_free_birth.nc -outf=bmw_ncsx_c09r00_out.nc -mgridf=mgrid_xxxx.nc

This will take a long time to run depending on the grid size. You can also do parallel run using mpirun and -para flag.

Beams3D

Installation on SCOREC Machines

I am not able to compile the STELLOPT tool on scorec yet. The installation instructions given in the STELLOPT README is not working.

If you have access to the PPPL clusters (e.g. stellar), they have compiled STELLOPT tools. You can use them directly.

    ssh stellar-intel
    module use /home/caoxiang/module
    module load stellopt/intel 

Learn more about it here

Using Docker Image

I used the docker image of STELOPT to run Beams3D where all the necessary tools are already installed. The docker image is available at Docker Hub.

1. Get the docker image:

    docker pull zhucaoxiang/stellopt

2. Run the docker image depending on your OS. SCOREC does not support docker yet. So, I used my own computer. You may also want to run the docker as root user since the image doesn’t support writing any file for non-root users.

    docker run -it -u root zhucaoxiang/stellopt

3. The NAG library is also needed to be installed but it is a proprietary software. A trial version(for 30 days) can be found here. Installation is very straightforward. Just follow the instructions given in the website.

According to their documentation, you can avoid NAG and use LSODE instead. Try changing the NAG flag to F in the make_debian.inc file as well as changing the INTEGRATOR flag to LSODE. If you succeed, please add in the documentation or create an issue.

4. After nag installation, add NAG to the environment using the following command:

    source /home/NAG/nll6i293bl/scripts/nagvars.sh int64 vendor dynamic

5. Then, follow the Beams3D tutorial to run Beams3D. I used the following command after editing the input file.

    xbeams3d -vmec ncsx_c09r00_free_birth -coil coils.c09r00 -vessel NCSX_wall_nbiport_acc.dat -field

6. It will create a .h5 file and it can be read by python. Magnetic field data is read in this Jupyter Notebook

9 - TOMMS

TOMMS is a Tokamak meshing software.

Getting TOMMS

TOMMS (previously Fusion) is a private repository. You need to be added to the repository to access it. Please contact appropriate people to get access to the repository. When you get access, you will be able to clone the repository using the following command:

git clone https://github.com/SCOREC/tomms.git

Installation and Example Usage

Complete installation instructions and usage are available in the TOMMS repository wiki. No special measures are required except changing the configuration files to point to the appropriate directories (dependencies and installation). Configuration files depend on the system/OS you are trying to install on and it is pointed out in the wiki and usage are explained in the TOMMS user-guide.

10 - XGC

XGC is a private repository. Ask Princeton Plasma Physics Laboratory (PPPL) for access to the repository. The documentation can be found here: XGC Documentation

Installation on SCOREC RHEL9

The master (latest commit 5d2cb943a) or d2neutral can be installed with spack using the following packages:

# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  # add package specs to the `specs` list
  specs:
  - kokkos@4.2.01+openmp+serial
  - cabana@0.5.0
  - fftw
  - petsc@3.15.0+fortran+metis+scalapack
  - adios2@2.8.0
  - netlib-lapack
  - googletest
  view: true
  concretizer:
    unify: true
  packages:
    python:
      externals:
      - spec: python@3.9.18
        prefix: /usr
      buildable: false

with this compiler configuration:

gcc@11.4.1:
        paths:
                cc = /usr/bin/gcc
                cxx = /usr/bin/g++
                f77 = /usr/bin/gfortran
                fc = /usr/bin/gfortran
        modules  = []
        operating system  = rhel9

Run spack concretize -f and spack install to install the packages. Now, before installing XGC, the following changes have to be made to the source code:

Following the XGC convention, a file called CMake/find_dependencies_scorecrh9-spack.cmake should be created with the following content:

find_package(FFTW3 REQUIRED)
find_package(PETSC REQUIRED)
find_package(LAPACK REQUIRED)

To properly install, these files had to be modified too and they are not added to a pull request yet.

1. In XGC_core/cpp/file_reader.hpp add the following line:

#include <iostream>

2. In libs/camtimers/CMakeLists.txt replace the following lines:

install (FILES ${CMAKE_BINARY_DIR}/perf_mod.mod DESTINATION include)
install (FILES ${CMAKE_BINARY_DIR}/perf_utils.mod DESTINATION include)

with these lines:

install(FILES ${CMAKE_BINARY_DIR}/libs/camtimers/perf_mod.mod DESTINATION include)
install (FILES ${CMAKE_BINARY_DIR}/libs/camtimers/perf_utils.mod DESTINATION include)

Now, to install XGC, activate the spack environment, load the necessary packages and run the following commands:

export XGC_PLATFORM=scorecrh9-spack

cmake -S . -B build \
  -DCMAKE_Fortran_FLAGS="-fallow-argument-mismatch" \
  -DCMAKE_BUILD_TYPE=RelwithDebInfo \
  -DCMAKE_INSTALL_PREFIX=build/install \
  -DCMAKE_CXX_COMPILER=$MPICXX \
  -DCMAKE_C_COMPILER=$MPICC \
  -DCMAKE_Fortran_COMPILER=$MPIFC