Debugging and Profiling Tools

Software does not always behave as intended, or with the performance expected. There are various tools available to understand what applications are doing and to help correct them; while this is primarily of use to software developers, it can be of benefit when using software other people have written.

Please note: we are experimenting with the capabilities of the various tools and envisage selecting a set of them to provide user training on in the future. If you use any of these pieces of software, it would be highly appreciated if you contact us and give feedback.

Please refer to the installed software pages for details of which machines that particular software packages are installed on.


Debuggers are generally used to influence and watch how a piece of code is running to check its correctness. Execution can be run until a chosen point in the code, known as a breakpoint, allowing variables to be inspected and updated or alternative code to be executed interactively.

The debuggers we have available are:

Tool Module Summary
Allinea DDT allinea Industry-leading parallel debugger
GNU Debugger (GDB)* N/A Text-based debugger, useful for debugging serial codes or individual MPI ranks
MUST must MPI correctness checker – report on MPI application programming errors

Although GDB is designed for debugging serial or multi-threaded programs, it can be useful for MPI programs with a little effort. See our notes for more details.


Profilers are used to take measurements of a code for later analysis, to check performance. Although some tools focus on particular types of measurement, different tools typically offer broadly similar functionality: the biggest differences can be in the interfaces to analyse the collected data.

Types of Measurement – Profiling, Hardware Counters and Tracing

Profilers are able to directly measure things to direct the programmer’s attention:

  • Time spent in different functions, subroutines or loops to find the most expensive parts of the program
  • File I/O, such as numbers of reads/writes and sizes
  • Parallel communications with other MPI ranks, !OpenMP threads or pthreads

Additionally, many of them are able to examine hardware counters which show how the CPU is performing when running the application. This is done via the PAPI library and allows the tools to calculate various things, such as how fast the code is running (e.g. in floating point instructions per second, flops) or how well the program is making use of data locality in memory (e.g. proportion of memory cache hits to misses).

Profiles provide a summary of execution. Sometimes, particularly for parallel programs, it is useful to record all the operations such as communication over the life of the application. This is typically known as a trace and can help understand how MPI deadlocks happen or why particular threads or MPI ranks are idle because they are waiting on other threads or ranks.

Simple-To-Use Profilers

Tool Module Summary
mpiP mpiP Very simple tool to discover which lines calling MPI routines are the most expensive
ompP ompP Very simple tool to discover how well the parallel regions in OpenMP codes are operating

More Powerful Profilers/Tracers

Tool Module Summary
TAU tau Profiler and MPI trace analyser for MPI, OpenMP, threaded and serial codes. Supports profiling Python 2.x programs as well as the usual C/C++/Fortran languages.
Scalasca scalasca Measure and analyse parallel behaviour of MPI and OpenMP codes written in C/C++/Fortran.
Allinea MAP allinea Profiler from the same people who wrote the DDT parallel debugger
Paraver, etc. paraver / extrae Profiler and MPI trace analyser for MPI, OpenMP, threaded and serial codes. Powerful for creating unusual 2D visualisations comparing different measurements of the program.
HPCToolkit hpctoolkit Profiler and MPI trace analyser for MPI, OpenMP, threaded and serial codes.
OpenSpeedShop openspeedshop Profiler and MPI trace analyser for MPI, OpenMP, threaded and serial codes.

File I/O Tracing

Tool Module Summary
Darshan darshan Use with MPI programs to generate I/O traces

Miscellaneous Tools

Tool Module Summary
Valgrind valgrind Analysis tools for detecting memory leaks, thread errors, cache usage, heap profiler, etc. plus kcachegrind
ntel Trace Analyzer itac Analyse MPI traces generated by the Intel MPI library
MPI program profiler* mpe2 Link against MPI programs to generate MPI traces in SLOG2 format. TAU can do the same job without needing to relink – perhaps better to use that instead.