Disc Quotas and Filesystems

Filesystem summary: ARC3

*At the present time, users have a single home directory shared between ARC1 and ARC2 so the same files and directories are available to both clusters. MARC1 has it’s own, separate home directories.

**This is a shared Lustre filesystem – files created on these are immediately visible to all nodes.

Directory Explanation Backed Up? Quota Files removed
/home/xxx/username User home directory* Yes 5GB No
/nobackup/username High speed shared parallel file system** No No, but total capacity of 836TB is shared between all users Yes, unused files are expired after 90 days
/scratch Each node has a small hard drive which is part of /nobackup (mounted as /scratch ) which can be used by your applications to read and write data. There is also local disc that can be requested for /scratch through the qsub command.

Filesystem summary: ARC2

Directory Explanation Backed Up? Quota Files removed
/home/xxx/username User home directory* Yes 5GB No
/nobackup/username High speed shared parallel file system** No No, but total capacity of 170TB is shared between all users Yes, unused files are expired after 90 days
/scratch Each node has a small hard drive (mounted as /scratch ) which can be used by your applications to read and write data.

Filesystem summary: MARC1

Directory Explanation Backed Up? Quota Files removed
/home/xxx/username User home directory* Yes 20GB No
/nobackup/username High speed shared parallel file system** No No, but total capacity of 500TB is shared between all users Yes, unused files are expired after 90 days
/scratch Each node has a small hard drive (mounted as /scratch ) which can be used by your applications to read and write data.

Note: MARC1 home directories are not shared with the other clusters

Home Directories

By default all users have a 5GB “soft” quota and a 10GB “hard” quota.

Soft quota

Users can exceed this quota, up to the hard quota limit for a limited time ~7 days.

Hard quota

Users are unable to exceed this quota; your jobs will fail if this limit is exceeded.

The quota Command

The Linux quota command will allow you to check how much of your disc quota you have used.

These home directories are hosted on the University’s central file server (storage area network (SAN)). This gives a storage location highly protected against data loss, however this is provided at the expense of capacity and performance.

As HPC users often require large storage and high performance (Large-bandwidth I/O and (IOPS)), the /nobackup disk is provided which utilises the Lustre parallel filesystem, delivering ~3.2GB/s over infiniband to a ~100 TB (usable) filesystem.

Users should only store important data they wish to have backed-up in their home directory (e.g. environment files, source code, some input data). Programs should be run from, and their data produced into the /nobackup area. Important data should be transferred from /nobackup to a more-protected storage device, once the I/O intensive work is complete.

Backups

At the current time, backups of /home directories are taken nightly and retained for about 28 days.
If you have accidentally deleted files it may be possible to retrieve them.

Files stored on /nobackup are not backed up or archived in any way. Responsibility for data stored on /nobackup lies with the user.

Please email arc-help@lists.leeds.ac.uk with as much of the following information as possible:

  • full pathname and name(s) of files
  • when the file was deleted
  • when the file was last modified
  • when the file was created

/nobackup parallel filestore

Each HPC cluster has it’s own independent parallel filestore. We do not set a user quota on the amount of data stored on /nobackup but all researchers should appreciate that this is a shared resource of finite size. If the volume of data you are storing is affecting the work of other researchers we may ask you to reduce your usage.

It is not intended as a long term store for data nor is it suitable for data storage under your project’s Research Data Management plan.

The /nobackup filestore is intended for working files and data. Any data that has not been used for a period of 90 days will be expired (deleted). You will receive two warnings that this is to happen, one two weeks before deletion and a second one week before deletion. We are not usually able to retrieve expired data.

The data stored on /nobackup is not backed up.

It is your responsibility to regularly back up or archive your important data to a secure location away from the cluster.

You will not get a storage location on /nobackup by default. To use it, create a directory with the same name as your username using the mkdir command.