How To Run Abinit On XSEDE:PSC/Blacklight

Although Blacklight uses the same "qsub" software as NERSC, the module function is a little less streamlined. Here is an example script on how to set up a submission file on the PSC/Blacklight system.

#!/bin/csh
#PBS -l ncpus=16
#PBS -l walltime=00:05:00
#PBS -q debug
#PBS -j oe
#PBS -N mz11396

source /usr/share/modules/init/csh
module load abinit/6.12.1
set ABINIT = /usr/local/packages/Abinit/abinit-6.12.1/bin/abinit

ja

cd $HOME/mz11396

mpirun -np 16 $ABINIT < mz11396.files >log

ja -s

Most of this will appear familiar to NERSC users.

  • ncpus sets the number of CPUs (there are 16/node on Blacklight).
  • Blacklight has two queues: debug and batch, the time and processor count limits for these are described here. For production calculations you must use the "batch" queue.
  • Note the three additional lines (starting with "source", "module", and "set"); these are needed to correctly set up the environment on Blacklight (only slightly more complicated than on NERSC).
  • the ja commands are used for Job Accounting (returning how much walltime, etc. are used by your calculation. You can omit them if you like, but this doesn't add any computational overhead, and might be useful for finding out how long your job took
  • mpirun behaves just like on NERSC:Carver, just note that we are using the $ABINIT environmental variable defined earlier in the script.
  • For more information on hwo to use Blacklight, see their help document
  • In particular, you probably do not want to run jobs out of your $HOME directory (as the quota is quite small on Blacklight, 5 GB vs. 40 GB on NERSC). Instead, run your jobs in $SCRATCH (copy the inputs and important outputs (e.g., .out file, DEN, ELF) to $HOME when you are finished). Note that files left on scratch after 21 days will be deleted! Copy files to ws* if necessary. For more description of the file system on Blacklight see this section of the documentation.
Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License