GPU Exercises 4

GPU Exercises #4

Your own code: run it as a well-behaved GPU job, check it, and scale up.

Exercises

Checklist before you submit
Run your code
Check how it went
Scale up with a job array
No code of your own?
Wrap up

Time: the remaining time in the session. This exercise is deliberately open: bring your own code, or use one of the suggestions at the end.

Checklist before you submit

Work through this in a CPU interactive session (srun -p <CPU_PARTITION> --pty bash):

[ ] Does your code use a GPU? Look for CUDA, GPU or NVIDIA in its documentation, or for a library such as PyTorch, TensorFlow, CuPy, Numba or JAX. If it does not, the GPU will sit idle.
[ ] Do you know which modules and packages it needs, with versions? Write them down. Do not rely on default versions.
[ ] Is everything installed already? Do downloads and installs now. GPU batch jobs should not need the internet.
[ ] Does it run on a tiny input? Test on the CPU first, if your code supports it.
[ ] Where are the input and output files? Keep them on Lustre, and write results somewhere you will not lose them.
[ ] How long will it take? If you do not know, run a very small case first and estimate.

Run your code

Copy the template and open it:
```
cp byo_gpu.job my_gpu_job.job
```
Edit the lines marked <-- CHANGE: the job name, a realistic --time (keep it to 10 minutes or less today), your modules and environment, and the command that runs your code.
Keep --gres=gpu:1. One GPU per job: 6 GPUs are shared by the whole room.
Make sure your code reports whether it found a GPU. In Python, for example:
```
import torch
print("CUDA available:", torch.cuda.is_available())
```
Submit and watch:
```
sbatch my_gpu_job.job
squeue --me
```

(back to top)

Check how it went

A job that finished is not necessarily a job that used the GPU well. Check:

Did it finish cleanly?
```
sacct -j <jobid> --format=JobID,JobName,Partition,Elapsed,MaxRSS,State,ExitCode
```
State should be COMPLETED and ExitCode 0:0. TIMEOUT means it hit --time. OUT_OF_MEMORY means it ran out of CPU memory, so raise --mem.
Was the GPU busy? Add a usage log to your script, as in GPU Exercises 2:
```
nvidia-smi --query-gpu=timestamp,utilization.gpu,memory.used --format=csv -l 2 > gpu_usage-$SLURM_JOB_ID.csv &
MON_PID=$!
python my_script.py
kill $MON_PID
```
Utilisation that stays low usually means the code is waiting on the CPU or on data loading, or the input is too small for the GPU to help.
How much GPU memory did it use? If the peak is close to the total, a slightly bigger input will fail. If it is tiny, you may be able to do more work per job.
Was the time limit sensible? Compare Elapsed with your --time. A limit 2 to 3 times the real run time is comfortable. A very long limit makes the job wait longer to start.

(back to top)

Scale up with a job array

To run the same code with different inputs (seeds, parameters, data files), use a job array instead of submitting jobs by hand or in a loop. The example gpu_array.job runs four tasks, two at a time, one GPU each:

#SBATCH --array=1-4%2
#SBATCH --output=gpu_array-%A_%a.out
...
python my_script.py --seed $SLURM_ARRAY_TASK_ID

Copy gpu_array.job, fill in the partition and module, and point it at your script.
Make your script use the task number ($SLURM_ARRAY_TASK_ID) to choose its input.
Submit it. In squeue --me, how are the array tasks shown?
Etiquette: the %2 limits how many tasks run at once. On a busy shared system, always set one.

(back to top)

No code of your own?

Pick one of these:

Change the benchmark. In gpu_check.py, add a second operation (for example an element-wise torch.sin) and time it on the CPU and GPU. Which gets more benefit from the GPU, matrix multiply or element-wise maths? Why?
Try a different matrix size sweep as a job array, using $SLURM_ARRAY_TASK_ID to pick --n.
Extend the CUDA program. In vector_add.cu, time the memory copies as well as the kernel. How much of the total time is spent moving data?
Explore the framework. Use the PyTorch tutorials to train a very small model on the GPU for a few epochs (keep the run under 10 minutes), and log its speed.

(back to top)

Wrap up

Before you leave:

[ ] Copy anything you want to keep from your job folders into your own storage.
[ ] scancel any jobs you no longer need: squeue --me to check.
[ ] Remove environments you do not need: conda env remove -p <path> and conda clean --all.
[ ] Note down which of the checklist points caught you out. That is where to look first next time something fails.

Where next?

The Advanced Exercises cover installing your own software and EasyBuild.
Ask questions in the Slack channel, or raise a request through the ITS Help Desk.
Bigger GPU projects: talk to the Research Computing team about limits, partitions and access.

(back to top)