- Introduction to HPC clusters
- UNIX Introduction
- Condo 2017
- Classroom HPC Cluster
- File Transfers
- Cloud Back-up with Rclone
- Globus Connect
- Sample Job Scripts
- Using DDT Parallel Debugger, MAP profiler and Performance Reports
- Using Matlab Parallel Server
- Using ANSYS RSM
- Support & Contacts
- Systems & Equipment
- FAQ: Frequently Asked Questions
- Contact Us
- Cluster Access Request
Parallelism on Nova Cluster is obtained by using MPI. All accounts are set up so that MPI uses the high performance Infiniband communication network. One can use either Intel MPI library (module load intel) or OpenMPI (module load openmpi).
To compile Fortran MPI programs use mpiifort with Intel MPI or mpif90 with OpenMPI. To compile C and C++ MPI programs use mpiicc/mpiicpc with Intel MPI or mpicc with OpenMPI.
Use Slurm Job Script Generator to create a script to submit to the Slurm Workload Manager. Remember that there are 36 processors per node, so for a 32 processor job, you only need 1 node.
In the script add module commands, e.g.:
module purge module load intel
and the mpirun command, e.g.:
mpirun -np 32 ./a.out
to start 32 MPI processes. OpenMPI modules may not have mpirun command available. Instead use orterun command:
orterun -np 32 ./a.out
to start 32 MPI processes.
Make sure that the executable (a.out in the example above) resides in one of the following locations:
/home/user (where 'user' is your user name)
/work/group (where 'group' is your group name, issue 'groups' to find it out)
These locations are mounted on each of the compute nodes. Don't place the executable in the local filesystem (/tmp) as each node has its own /tmp . Files placed into /tmp on the front end node won't be available on the compute nodes, so mpirun won't be able to start processes on compute nodes.
One can use the storage on the disk drive on each of the compute nodes by reading and writing to $TMPDIR. This is temporary storage that can be used only during the execution of your program. Only processors executing on a node have access to this disk drive. Since 36 processors share this same storage, you must include the rank of the executing MPI processes when reading and writing files to $TMPDIR. The size of $TMPDIR is about 1.5 TB on the regular compute nodes. For more details see https://www.hpc.iastate.edu/guides/nova/storage .