Each ARC3 compute node contains a limited amount of dedicated storage and its capacity and performance is different for each type of node. As this storage is local to the compute node, it can offer more predictable performance than writing to the high performance filesystem, /nobackup , which is shared between all compute nodes
On our previous clusters, such storage is available for use by jobs – mostly under /scratch and via the directory name contained in the $TMPDIR environment variable, but also a small amount under /tmp . However, jobs could not reserve the local space they need and so could fail unexpectedly if another job running on the same node was also writing to local disk.
On ARC3, if a job wishes to use the compute node disk, this should be explicitly requested by the job. It is likely only to be useful if you currently use /scratch on our other clusters (if that is the case, please try to use it and report back your thoughts).
ARC2 and MARC1
Summary:
- Use of /tmp should be avoided, as it is relatively small and space usage is uncontrolled.
- /scratch is big, but space usage is uncontrolled. Files inside it are deleted 96 hours after last use, so that you have time to move the contents to somewhere else if you need to.
- $TMPDIR within a job refers to a per-job directory within the big /scratch directory, which is deleted at the end of the job.
ARC3
Summary:
- Use of /tmp should be avoided, as it is relatively small and space usage is uncontrolled.
- Use of /scratch should be avoided, as this is now the same storage as /tmp
- $TMPDIR within a job refers to a per-job directory which has been assigned dedicated storage for the job.
The specification of the compute nodes on ARC3 and their local disk characteristics:
Node Type | Number of nodes | Memory | Local Disk Capacity | Local Disk Type |
---|---|---|---|---|
Standard ( -l 24core-128G ) | 165 | 128GB | 100GB | ssd |
High Memory ( -l 24core-768G ) | 2 | 768GB | 800GB | hdd |
GPGPU | 2 – each with 2 NVIDIA K80 GPUs | 128GB | 800GB | hdd |
Notes:
- ssd refers to a solid state disk, which has excellent performance characteristics for reading and writing
- hdd refers to a mechanical hard disk with a spinning platter, which has good performance characteristics for large sequential reads or writes only.
Usage
Option | Description | Default |
---|---|---|
-l disk=<bytes> | Sets the limit of the local disk available under $TMPDIR (per slot) | 1G per slot |
-l disk_type=*|ssd|hdd | Specifies the type of local disk that files under $TMPDIR will be written to. There are 3 options
Reading a writing will be much quicker to a solid state drive and this option should be used if you have lots of read and write operations. |
* |
-l disk_out=<directory> | Specifies the directory to which the contents of $TMPDIR will be copied to at the end of a job. As $TMPDIR will be deleted at the end of the job, this could be useful if you need to keep its contents, for example if you are check pointing.
The files will be copied to directory <directory>/<job_id>.<task_id> Please note that, for distributed parallel jobs, only the $TMPDIR on the compute node where the job script runs is saved. |
No copy by default. |
-l disk_usejobname | When used in conjunction with the -l disk_out=<directory> option, the directory name that $TMPDIR is copied to at the end of the job is changed to <directory>/<job_name> , where <job_name> can be specified with the -N <job_name> option. | false |