HW8 How To: CUDA Profiling

There seems to be a lot of confusion regarding how to get results with NVVP on euler.

Since you should NEVER run interactive jobs on the ME759 partition (for those of you who continue ignoring this rule, I may have to limit your access to the cluster), you need to generate the profiling data using the command line tool nvprof.

The following submit script should do the trick...

  Code:

#!/bin/bash
#SBATCH -N 1 -n 1 -t 0-00:05:00 --gres=gpu:1
#SBATCH -o /dev/null

cd $SLURM_SUBMIT_DIR
nvprof -o profiling_data.nvp%p ./my_executable


Once you've submitted and allowed this script to run, connect to euler using X11 Forwarding (for linux/mac users this is ssh -X username@euler.wacc.wisc.edu).

Open NVVP (username@euler ~$ nvvp &) and then go to File > Import and select your profiling_data.nvp.

That's it. If you have troubles running NVVP or nvprof, it might mean that you need to load the cuda environment variables. This can be accomplished with the command module load cuda which can be run on the command line, or appended to your ~/.bash_profile. If you are using a specific version of cuda with the module command (which is not what you were instructed to do in class) you may still experience problems.

[parvathi@euler ~]$
** (java:18101): WARNING **: Could not open X display
Nvvp: Cannot open display:

** (nvvp:18100): WARNING **: Could not open X display
Nvvp: Cannot open display:
Nvvp:
An error has occurred. See the log file
/home/parvathi/nvvp_workspace/.metadata/.log.

I get the above message when I try to open nvvp. I am working on a Mac and used '-X' to login to euler. I don't understand why. I have tried the 'module load cuda' as well.

Are you trying to run it from inside tmux ? I got a similar error when I tried to do that .

I also got this warning when trying to open nvvp on my mac. Do i need to be inside a linux os
for nvvp to work ? Or does my host OS need to have nvvp installed ?

[mikkelnielsen@euler ~]$
** (java:3244): WARNING **: Could not open X display
Nvvp: Cannot open display:

** (nvvp:3243): WARNING **: Could not open X display
Nvvp: Cannot open display:
Nvvp:
An error has occurred. See the log file
/home/mikkelnielsen/nvvp_workspace/.metadata/.log.

It worked for me on Linux( without downloading anything) it did not work on Mac. Did anyone else get it working on Mac ? I guess that could make sense/might need to download something to make it work on mac.

[parvathi@euler ~]$
** (java:18101): WARNING **: Could not open X display
Nvvp: Cannot open display:

** (nvvp:18100): WARNING **: Could not open X display
Nvvp: Cannot open display:
Nvvp:
An error has occurred. See the log file
/home/parvathi/nvvp_workspace/.metadata/.log.

I get the above message when I try to open nvvp. I am working on a Mac and used '-X' to login to euler. I don't understand why. I have tried the 'module load cuda' as well.

If you are experiencing this problem, it could be a couple of issues. Please ensure that X11 works on your mac. In addition, please make sure that you aren't using screen or tmux as they can prevent proper X forwarding.

There seems to be a lot of confusion regarding how to get results with NVVP on euler.

Since you should NEVER run interactive jobs on the ME759 partition (for those of you who continue ignoring this rule, I may have to limit your access to the cluster), you need to generate the profiling data using the command line tool nvprof.

The following submit script should do the trick...

  Code:

#!/bin/bash
#SBATCH -N 1 -n 1 -t 0-00:05:00 --gres=gpu:1
#SBATCH -o /dev/null

cd $SLURM_SUBMIT_DIR
nvprof -o profiling_data.nvp%p ./my_executable


Once you've submitted and allowed this script to run, connect to euler using X11 Forwarding (for linux/mac users this is ssh -X username@euler.wacc.wisc.edu).

Open NVVP (username@euler ~$ nvvp &) and then go to File > Import and select your profiling_data.nvp.

That's it. If you have troubles running NVVP or nvprof, it might mean that you need to load the cuda environment variables. This can be accomplished with the command module load cuda which can be run on the command line, or appended to your ~/.bash_profile. If you are using a specific version of cuda with the module command (which is not what you were instructed to do in class) you may still experience problems.

EDIT:

Please verify that you are submitting the above script using the option '-p slurm_me759' if you do not pass that to sbatch on the command line, or add it to the file, your account might become automatically deactivated.

(If that has happened, you need to contact your sysadmin --me-- via email, do not post it on the Forum.)