====== NIMROD 2022 ======

The method for getting NIMROD running at NERSC has changed slightly as the default version of some modules on NERSC have changed. This wiki will give instructions to help set up the most recent version of nimdevel and include some tips to help compile older versions of NIMROD. This page  will run through the steps necessary to compile the code at NERSC. 
 

===== Accessing the newest version of nimdevel  =====

The newest version of nimdevel can be accessed using the following command:

  svn co --username <username> https://ice.txcorp.com/nimsvn/nimall/trunk nimall

The user will then be asked to enter their password. This will check out the nimall trunk which contains the source code for the newest version of nimdevel as well as the dependent packages. 

If only the source code for the newest version of nimdevel is needed (no dependent packages) then the following command may also be used: 

  svn co --username <username> https://ice.txcorp.com/nimsvn/nimroot/trunk nimdevel

===== Compiling the newest version of NIMDEVEL at NERSC  =====

To compile the code at NERSC some modules are required. Every user will have some of the necessary modules already loaded as the default when logging in but some extra modules are necessary. It may be convenient to create a script that can be run to load the necessary modules. An example script that loads the necessary modules is shown below: 

  #!/bin/sh
  
  module swap PrgEnv-intel PrgEnv-gnu 
  module load gcc/11.2.0
  module load cray-petsc/3.14.5.0
  module load alps/6.6.67-7.0.3.1_3.25__gb91cd181.ari
  module load cray-hdf5-parallel/1.12.1.1
  module load cray-netcdf-hdf5parallel/4.8.1.1
  module load python
   
  echo ' Nimrod modules loaded' 
  
The above script will load the correct versions of the modules needed and echo 'Nimrod modules loaded' when complete. To run this script use the command: 

  . <script name> 

Now that the necessary modules have been loaded the code can be compiled. There are two main methods of doing this at NERSC. The first is to use the mknimall.sh script provided if nimall is checked out. Alternatively, Jake's config scripts may be used to build the code on Cori. 

====   Mknimall method  ====
 
  
 To use this method, move to the top of the nimall directory where mknimall.sh is located and enter the following: 

  ./mknimall.sh -m cray.gcc -t -N -o -E USE_MPI=mpich -i [full path to software directory] -b [full path to Build directory]

The above method will then produce a log file named mknimall.log. A value or 0 or 1 must be returned at the bottom for the build to be successful. If this is unsuccessful, the BUILD file at the top of the nimall directory will have more instructions. After a successful build, the user should move to the following directories: 
  
  build directory/nimdevel/par/bin    
  build directory/nimdevel/ser/bin 
      
and check the executables are there. /par/bin will contain the executables for the parallel build and /ser/bin will contain the executables for the serial build of Nimrod. 

====   Jake's config scripts method  ====

Alternatively, pre-existing configure scripts created by Jake King may also be used. This section will give instructions on how to copy over and use these scripts on Cori. These will also work if the nimdevel only branch is checked out, which does not contain the mknimall.sh script. 

**Step 1:** make two new directories, one for the parallel version and one for the serial version. This can be done using the following command: 
 
  mkdir nimdevel-ser 
  mkdir nimdevel-par 
  
**Step 2:** Copy over the config scripts to the corresponding directory. These config scripts are in the following directories: 

  /global/common/software/nimrod/cori/software-gcc-11.2/nimdevel/share-ser/cori.nersc.gov-nimdevel-ser-config.sh
  /global/common/software/nimrod/cori/software-gcc-11.2/nimdevel/share-par/cori.nersc.gov-nimdevel-par-config.sh

**Step 3:** Modify the config scripts. The config scripts will need modifying to correctly build the code. Two lines will need to be edited so that the source and install directories are changed to the correct locations in the user's directory. The lines to change should be modified to read : 

<code>
-DCMAKE_INSTALL_PREFIX:PATH=/ <full path to install directory> \
-DSUPRA_SEARCH_PATH:PATH='/global/common/software/nimrod/cori/software-gcc-11.2;/global/common/software/nimrod/cori/software-gcc-11.2/userdocs' /<full path to nimdevel source code directory>
</code> 

After this has been done, run the config scripts from the corresponding directory and then use the make install command. This should successfully make the serial and parallel versions of nimdevel. 
  
===== Compiling older versions at NERSC =====

Some older versions of Nimrod ( such as 3.2.4) may contain a README that instructs the user to compile the code slightly differently . When following these instructions the user may run into some errors when compiling. One common issue is getting a type mismatch error when calling the mpi_bcast routine. This is due to these version of nimrod being created and compiled using an older version of gcc. To fix this issue, add the fortran compile flag:

<code>
-fallow-argument-mismatch
</code>
 
If using cmake to compile, this can be done by adding the following line into the CMakeLists.txt file:

  set(CMAKE_Fortran_FLAGS "-fallow-argument-mismatch ${CMAKE_Fortran_FLAGS}")


=====   Modifying and recompiling the code  =====

One method of easily modifying the code one NERSC is to use Juypter notebook. Further instructions can be found here:  https://docs.nersc.gov/services/jupyter/ \\
If the source code is modified, the user does not need to repeat all of the steps listed above for compiling the code. Instead the user can move to the build directory and use the make/make install command again. This will save time instead of rebuilding the entire code again.