FAQ  •  Login

mpiexec/mpirun examples for Euler

Moderators: Dan Negrut, ME964 Spring 2012

<<

Andrew Seidl

Administrator
Administrator

Posts: 193

Joined: Thu Oct 28, 2010 11:54 am

Unread post Sat Mar 31, 2012 4:18 pm

mpiexec/mpirun examples for Euler

Below are some examples of how to use MPI on Euler. It is assumed that these are performed inside a Torque job (interactive or batch) with the appropriate resources requested (nodes=m:ppn=n).

Use all resources currently available within the job. Starts m*n processes.
  Code:
$ mpiexec --hostfile $PBS_NODEFILE ./integrate_mpi

Start x number of processes, where x <= m*n. See the manpage for how to control where these are run.
  Code:
$ mpiexec -n x --hostfile $PBS_NODEFILE ./integrate_mpi

Use y nodes, each with x processes, there x <= m && y <= n. Starts x*y processes.
  Code:
$ mpiexec -n $((x*y)) --npernode x --hostfile $PBS_NODEFILE ./integrate_mpi

See the mpiexec manpage (man mpiexec) for many more options.

If you are asked for a password when trying to run MPI jobs, enable passwordless logins between Euler and the compute nodes. Run the following on Euler, making sure to not set a passphrase:
  Code:
$ ssh-keygen
$ ssh-copy-id localhost


Some other notes:
  • Our current install of OpenMPI (sources provided by Mellanox) has some issues with Torque integration. Use the --hostfile as above to get around this
  • The Infiniband cards for Euler's GPU nodes are in PCIe2.0 x4 slots (no x8 slots and all the x16 slots have GPUs). As such, the theoretical maximum achievable bandwidth is 2000MB/s.
  • We will be adding 15 more CPU nodes (same specs as Euler15) to Euler during the week of April 1. [edit: added. See the hint below.]

Another helpful hint for getting slightly more consistent timings: force your jobs to be run on similar hardware by appending the :intel or :amd flags to your resource request. :intel will direct your jobs to our Intel Xeon 5520 nodes (the ones with the GPUs) while :amd will direct them to the AMD Opteron 6274 nodes (the ones with 64 cores each).
<<

S12tcpatterson

Newbie
Newbie

Posts: 10

Joined: Mon Jan 23, 2012 9:07 pm

Unread post Sun Apr 15, 2012 5:58 pm

Re: mpiexec/mpirun examples for Euler

I can't seem to find the manpage that you mention here. Here is what I get:

$ man mpiexec
No manual entry for mpiexec
<<

Andrew Seidl

Administrator
Administrator

Posts: 193

Joined: Thu Oct 28, 2010 11:54 am

Unread post Sun Apr 15, 2012 7:02 pm

Re: mpiexec/mpirun examples for Euler

Should be working on your next login. My manpaths always get set correctly, so I never noticed that the environment modules weren't setting it for other people.

Return to ME964 Spring 2012: High Performance Computing

Who is online

Users browsing this forum: No registered users and 1 guest

cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Designed by ST Software.