Matlab is installed on the HPC clusters and is licensed for academic use only.

Contents

## Setting up the environment

All required environment variables can be set by loading the Matlab module, to do this type the command:

1 2 3 |
$ module add matlab |

Matlab is now centrally licenced and managed so there is no need to set a licence server before using it.

## Running on the login nodes

Matlab can be launched by entering its name at the command prompt; i.e.:

1 2 3 |
$ matlab |

Please note that this method should not be used for long computational use.

## Running through the batch scheduler

The batch scheduler allows both interactive and batch jobs to be submitted during which users will have exclusive access to the resources they request.

### Running through an interactive shell

The following will launch Matlab interactively, displaying the full GUI:

1 2 3 |
$ qrsh -V -l h_rt=hh:mm:ss matlab -desktop -singleCompThread |

In the above command:

hh:mm:ss is the length of real-time the shell will exist for,

-V exports the the current environment.

The parameter -singleCompThread ensures that Matlab will run only on a single thread, since the default multiple threaded behaviour can cause problems.

e.g. to run Matlab for 6 hours:

1 2 3 |
$ qrsh -V -l h_rt=6:00:00 matlab -desktop -singleCompThread |

This will run Matlab within the terminal from which it was launched.

Alternatively, the following will launch an interactive session where the interaction is via a command prompt interface only:

1 2 3 |
$ qrsh -pty y -V -l h_rt=6:00:00 matlab -nodesktop -singleCompThread |

Where, hh:mm:ss is the length of real-time the shell will exist for.

### Batch Execution

To run matlab in batch-mode you must first generate a list of commands for matlab to process in a file; e.g. file cosplot.m:

*The
% symbol denotes a comment in Matlab scripts*

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
% MATLAB M-file example to approximate a sawtooth % with a truncated Fourier expansion. nterms=5; fourbypi=4.0/pi; np=100; y(1:np)=pi/2.0; x(1:np)=linspace(-2.0*pi,2*pi,np); for k=1:nterms twokm=2*k-1; y=y-fourbypi*cos(twokm*x)/twokm^2; end; plot(x,y); print -deps matlab_test_plot.ps; quit; |

Please note that the last line must be ” quit; “!

A script must then be created that will request resources from the queuing system and launch the matlab executable; script runmatlab.sh:

1 2 3 4 5 6 7 8 9 10 11 |
# Export current environment #$ -V # Set a 10 min limit #$ -l h_rt=0:10:00 # Load matlab module module add matlab # run matlab using command file # -nodisplay flag should be given to suppress graphics matlab -nodisplay < cosplot.m |

This can be submitted to the batch scheduler system using:

1 2 3 |
$ qsub runmatlab.sh |

The output file matlab_test_plot.ps can be viewed with Ghostscript using the command gs matlab_test_plot.ps.

## Parallel Matlab jobs

Recent versions of Matlab support three methods of making use of multiple CPU cores:

- Multi-threaded support in various Matlab routines and toolboxes
- Multiple “worker” support (on a single host)
- Multiple “worker” support (on multiple hosts)

### Multi-threaded support

This is the simplest method of allowing Matlab to use more than one CPU core. Many functions and toolboxes, for example fft() and various linear algebra routines, automatically make use of multiple cores where available.

To do this, we would recommend adding the flag
-l nodes=1,ppn=1 to your qsub command and job scripts. This will ask for an entire compute node. Note that this will **NOT** speed up Matlab while it is not using functions that are able to take advantage of multiple cores.

Please see the Matlab product documentation for details on what functions are multithreaded.

### Multiple “worker” support (on a single host)

Matlab can launch multiple copies of itself, called workers, that are able to communicate and share work between them. This functionality is called the Parallel Toolbox and you may need to adjust code to work properly in this fashion. For example, for loops that can be run in parallel may be distributed across all workers by use of the parfor Matlab command.

To do this, launch a parallel job, for example by adding the -pe smp 8 flag to the qrsh or qsub commands from above. This requests 8 cores on a single compute node and stores the number of cores granted in the NSLOTS environment variable. Once it has started, execute the Matlab command parpool('local', 8) or, more generally, parpool('local', str2num(getenv('NSLOTS'))), which in this case would launch 8 workers.

According to Matlab’s documentation, there may be a limit on the number of workers that can be requested in this way; however, this has been tested on ARC3 on up to 24 cores.

### Multiple “worker” support (on multiple hosts)

The multiple workers in the Parallel Toolbox can be spread across multiple hosts in order to make use of a greater number of cores and/or memory: Matlab calls this the Distributed Computing Server. Although this is very similar technology, Mathworks license it differently: at the time of writing we can only support a maximum of 32 distributed workers – please contact arc-help@lists.leeds.ac.uk if this limits your research.

The same code that can make use of the Parallel Toolbox can also make use of Distributed Computing Server; however, there are currently a few setup steps required and the method of launching it is different.

**NOTE:** these instructions were put together using Matlab 2017b.

Setup steps:

*Configure Matlab to use an appropriate MPI.*Execute the following commands:- $ mkdir ~/matlab
- $ cp $SGE_ROOT/$SGE_CELL/common/leeds/pe/matlab/mpiLibConf.m ~/matlab/

*Configure Matlab to submit batch jobs.*Launch Matlab on a login node and:- From the toolbar, select:
*HOME -> Parallel -> Manage Cluster Profiles* - Select
*Import* - Select file /services/sge_prod/default/common/leeds/pe/matlab/Gridengine.settings

- From the toolbar, select:

*(these steps should become redundant in future installations of Matlab on ARC)*

To use:

- Launch Matlab on a login node and
*NOT*from inside a job. - Launch a 32 worker distributed pool with the following Matlab command: pool = parpool('Gridengine', 32);
- This causes Matlab to submit a 32 core job. Once it has started running, you will be able to type further commands.

By default, a Gridengine parpool requests one core and 1G RAM per worker for 48 hours. Control over the type of job can be achieved by editing the cluster profile within Matlab. For example, to change the job runtime (h_rt), memory per worker (h_vmem) or add arbitrary qsub flags (qsub_args):

- From toolbar, select:
*HOME -> Parallel -> Manage Cluster Profiles* - Click on
*Gridengine*(on left hand side) - Click on
*Properties*tab (on right hand side) - Click on
*Edit*button (bottom right) - Scroll to the
*AdditionalProperties*box in the*SCHEDULER INTEGRATION*section. - Select the vale for
*h_rt*,*h_vmem*or*qsub_args*and modify accordingly. - Click on
*Done*button (bottom right)

*Hybrid multiple workers and multithreading parallelisation*. It may be useful to try combining both Matlab’s multithreading and Distributed Computing Server features (for example, your Matlab code spends all its time in a parfor loop and each trip through that loop spends all of its time in multithreaded routines like fft()): if appropriate, this would make the best use of our limited Distributed Computing Server licenses. To do this:

- Use the above method for changing
*h_rt*to edit the cluster profile but instead change*use_np_syntax*to*true*. This will caluse entire compute nodes to be allocated to the pool. - Repeat steps 1-4 again, but then change the value for
*NumThreads*(instead of modifying the*AdditionalProperties*box). This will tell Matlab to multithread and it will also cause*NumThread*cores to be allocated to each worker. - Since entire compute nodes are allocated, please select ‘sensible’ values for the number of workers and the number of cores per worker to make good use of the allocated resources. For example, on ARC3’s 24 core compute nodes, the number of workers in the pool multiplied by
*NumThreads*should be a multiple of 24.

Troubleshooting Distributed Computing Server:

*No matter how many workers I ask for, parpool reports there is only 1 worker!*MPI has probably not been configured correctly and Matlab is running multiple workers that cannot communicate with each other. Ensure you have copied mpiLibConf.m (see above) and restart Matlab.*When the pool is shutdown, the job goes into an error state (“Eqw”).*Matlab often deletes the files associated with the jobs it submits before the jobs themselves are deleted. If this happens, please qdel the job manually.

## GPU Matlab jobs

Matlab can take advantage of GPU hardware. To do this, first launch Matlab inside a job where GPU resources have been requested (see here). Within Matlab, the gpuDevices command will provide details of the available GPU(s):

e.g.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
>> gpuDevice ans = CUDADevice with properties: Name: 'Tesla P100-PCIE-12GB' Index: 1 ComputeCapability: '6.0' SupportsDouble: 1 DriverVersion: 9 ToolkitVersion: 8 MaxThreadsPerBlock: 1024 MaxShmemPerBlock: 49152 MaxThreadBlockSize: [1024 1024 64] MaxGridSize: [2.1475e+09 65535 65535] SIMDWidth: 32 TotalMemory: 1.2786e+10 AvailableMemory: 1.2401e+10 MultiprocessorCount: 56 ClockRateKHz: 1328500 ComputeMode: 'Default' GPUOverlapsTransfers: 1 KernelExecutionTimeout: 0 CanMapHostMemory: 1 DeviceSupported: 1 DeviceSelected: 1 |

Please see the main Matlab product documentation for writing code to use GPUs.

Multiple GPUs can also be taken advantage of; however, this must be in conjunction with the Parallel Toolbox feature (see the above section on parallel jobs and the Matlab website) as a single worker can only use one GPU at a time.

## Compiling Matlab

To run a large number of matlab jobs simultaneously it may be necessary to use the matlab compiler to create a stand-alone application. This will also avoid using too many license tokens. For a full description of the compiler please look at the in-program help or the Matlab compiler user’s guide.

### Code preparation

* Script *m-files cannot be compiled directly, first they have to be converted to

*m-files. In general, this is a case of wrapping the main section of code within a function. For more details look at converting script m-files section in the user guide.*

**function**### Compiling

The Matlab compiler makes use of the GNU C compilers and as the Intel compiler is loaded by default you shouldswitch to the correct version of GNU compiler via the command:

1 2 3 |
module switch intel gnu |

The compiler can be invoked from within Matlab or directly from the command line via the tool mcc . However, to get the code to compile in single threaded mode the compiler should be invoked directly from the command line. In general this would take the form:

1 2 3 |
mcc -R -singleCompThread -m your_program.m |

It is very important that you include the -R singleCompThread, otherwise matlab will run in multi-threaded mode and cause problems when running through the batch queues.

This compilation will yield two files of interest

- your_program , which is the compiled code
- runyourprogram.sh , which is the wrapper to run your compiled code.

### Running the application

Once the matlab module is loaded, you can run your stand-alone application via the wrapper and specifing the correct path to various Matlab components. This can easily be done via the environment variable $MATLAB_HOME. An example job script for submission to the batch queue would look like:

1 2 3 4 5 6 7 8 9 10 |
# Export current environment #$ -V # Set a 6 hour limit #$ -l h_rt=6:00:00 # Load matlab module module add matlab #set the path to Matlab runtime, via $MATLAB_HOME, run program via wrapper ./run_your_program.sh $MATLAB_HOME |