{"last_commit_sha":"caf0a1806316226032d25a660f2f5072f95fe004","persisted":true,"title":"SLURM basics","content":"[[_TOC_]] # Running a job on the cluster Jobs can be run on the cluster through the [srun](https://slurm.schedmd.com/srun.html) or [sbatch](https://slurm.schedmd.com/sbatch.html) commands. Both command supports common options, the main differences being that * `srun` will block until the needed resources are available (or time out). It will print on the standard output and can run arbitrary commands, including sbatch scripts. Please refer to the [srun manual](https://slurm.schedmd.com/srun.html) for a full description. * `sbatch` will schedule the job for later execution, when the requested resources are available. It will only work with an executable [sbatch script](#running-in-batch). Please refer to the [sbatch manual](https://slurm.schedmd.com/sbatch.html) for a full description. Here, we will only mention the most common/important options. ## Selecting a partition Before you can run anything through SLURM, you need to select a partition. A partition can identify a job profile. ### Licallo All queue have a maximum time limit of 24 hours. Other limitation may apply in term of number of concurrently running jobs or resource usage. Here are the currently available partitions on Licallo * **seq** dedicated to sequential jobs. All you sequential jobs or arrays of sequential jobs taking more than 4 hours. * **seq-short** if your sequential job takes less than 4 hours, it should go in there (to be scheduled more esealy) * **short** short jobs, both sequential and parallel. Used mostly for debugging. * **fdr** parallel jobs requiring small node can go in there. The nodes in this partition have to sockets/processors of 10 cores each and 64Go of RAM. * **x40** parallel job requiring fat node (typically hybrid job) should go in this partition. The nodes in this partition have 2 sockets/processors of 20 core each and 192Go of RAM. * **1to** the big memory partition, only contains one node, with 1To of RAM and 4 sockets of 8 core each. ### Lagrange TBD ## Running interactively Any command can be dispatched on a compute node through `srun`: ``` $ hostname pollux.cluster $ srun --partition seq --time 0:1:0 hostname p080.cluster $ srun --partition x40 --time 0:1:0 hostname x033.cluster $ ``` * The `--partition ` (`-p ` for short) is required to select the partition * The `--time ` (`-t ` for short) is required to select the expected runtime upper bound. This option is not mandatory, but he default runtime is very short. Let say you have a big compilation to run and assuming you build system support parallel build, you can easily dedicate a full node to that task: ``` $ srun -p fdr --nodes 1 --exclusive -t 0:10:0 make -j ... [100%] Built target test_isonug_dependencies [100%] Built target test_restart_rad_dependencies $ ``` * `--nodes ` (-N for short) allocate one node to the task. * `--exclusive` wont share the node with another task. * `make -j` our parallel make task ## Running in batch ### SLURM script We need to write a slurm/sbatch script. An sbatch script must start with a [shebang line](https://en.wikipedia.org/wiki/Shebang_(Unix)): ``` $ head -1 ./run.slurm #!/usr/bin/env bash $ ``` and be [executable](https://en.wikipedia.org/wiki/Modes_(Unix)): ``` $ ls -l ./run.slurm -rw-r--r-- 1 alainm sit 180 Aug 14 10:09 ./run.slurm # WRONG $ chmod a+x ./run.slurm [alainm@pollux multithread]$ ls -l ./run.slurm -rwxr-xr-x 1 alainm sit 180 Aug 14 10:09 ./run.slurm # GOOD $ ``` ### Example *You will need to load the [Intel compiler](/Licallo/Software Environment) to run this example. It is provided by one of **the intel-\*-runtime** module.* Compile the following [sequential.cpp](https://gitlab.oca.eu/DSI/HPC/blob/master/SLURM/sequential/sequential.cpp) source code: ``` $ module load intel-gnu8-runtime $ icpc hello_world.cpp -o sequential $ ``` We will submit it with the [sequential.slurm](https://gitlab.oca.eu/DSI/HPC/blob/master/SLURM/sequential/sequential.slurm) script. ``` $ sbatch ./sequential.slurm Submitted batch job 15292921 $ ``` wait for some time for the `rabit..out` to appear: ``` $ ls rabit.15292921.out sequential hello_world.cpp sequential.slurm $ more rabit.15292921.out This job was launch from pollux.cluster in /beegfs/SCRATCH/alainm/view/HPC/SLURM/sequential What's up, Doc ? $ ``` ### SBATCH directive As we can see in the [sequential.slurm](https://gitlab.oca.eu/DSI/HPC/blob/master/SLURM/sequential/sequential.slurm) script, all the option we previously passed on the command line can be set once and for all in the script using the `#SBATCH