• FLCComputingBackup

Data Backup in the FLC Group

Home Directories

• The contents of all home directories are stored on AFS and backed up centrally by the DESY IT group every night. (Note that the AFS group system:administrators needs read permissions to make the backup – this is included in all ACLs by default.) The most recent backup appears inside your home directory as “~/.OldFiles” – this is a read-only directory which reflects the state of your home directory at the time of the latest backup (typically in the early night) and which does not count for your AFS disk quota.

• See the DESY IT page on the Central Backup at DESY to find out how to retrieve older backups.

Local Disks

• All local FLC computers (with names currently ranging from flc01 to flc49) have a disk for local data storage that is mounted as “/scratch”. The contents of the directory “/data” (which is usually a symbolic link to “/scratch/data”) are backed up internally by the FLC administrators. This backup actually consists of three different levels:

  • A full backup (containing all data) is done approximately twice a year, or for special reasons (e. g. exchange of the disk or system upgrade).

  • A differential backup (containing all data that has changed since the latest full backup) is done every week.

  • An incremental backup (containing all data that has changed since the latest backup) is done (almost) daily.

• In order not to exceed the storage capacities of the backup system, you should mark all data that does not need to be backed up – this is typically everything that can be reproduced with a reasonable amount of work in a reasonable amount of time (Monte Carlo data, processed events, local copies of data that is kept elsewhere, local software installations, …) and all temporary data. Due to the policy of doing differential and incremental backups, you should avoid moving, renaming, or touching files and directories unnecessarily.

• To exclude files and directories from the backup, do the following:

  • Create a file named “.nobackup” in every directory that contains files or subdirectories that should not be backed up.

  • Write the names of these files or subdirectories (separated by line breaks) into the corresponding .nobackup file. Note that you may use shell wildcards (“?” for a single character, “*” for an arbitrary number of characters), but no pathnames pointing into subdirectories (create further .nobackup files in these subdirectories instead). If a directory is listed in a .nobackup file, none of its contents (including all subdirectories) will be backed up.

• You can run the script “whatsup++” (in Jörgen Samson’s public AFS area) to see which of your files and directories will be included in the backup (this may take some time). In the end, the script will also tell you how much disk space is used by these files in total. A few gigabytes of backup data will generally be okay, but if the size of your backup data reaches the order of 10 GB, one of the FLC administrators may come and see you. He or she can help you in deciding which of your data should be backed up and which not. See the section “Miscellaneous” if you have to store large amounts of data permanently.

  • shell$ ~samson/public/whatsup++ /data/${USER} | less

• Please remember that it’s your responsibility to create and maintain proper .nobackup files – not only on your own local computer, but on every local FLC computer on which you own a directory in /data. If you fail to comply with these rules and the size of your backup data exceeds the limit, your data will not be backed up at all. Therefore, keep in mind that you are the one who will eventually profit from the backup – every disk will crash, you just don’t know when.

Group Servers

• The group server flcl01 is included in the Central Backup at DESY. In contrast to that, the disk contents of the cluster servers lc3 and lc4 are not backed up. This holds in particular for the “pool” storage which is located at “lc3:/pool” and “lc4:/pool” and which should be automounted on all local FLC computers as “/nfs/flc/lc3/pool” and “/nfs/flc/lc4/pool” respectively.

• Keep in mind that lc3 and lc4 are the entry points to the FLC batch cluster, and you may well use their disk space as a temporary buffer for your jobs that are currently running on the cluster. However, if you’re using the “pool” as a medium for mass storage, you have to be aware that your data will not be backed up. See the section “Miscellaneous” for other possibilities to store large amounts of data.

• The AFS-based pool storage under “/group/pool/${USER}” offers the usual “.OldFiles” snapshot of the previous night, but there is no long-term backup. The AFS file servers use RAID storage systems, but you may not want to rely solely on this built-in mechanism if files of vital importance are concerned.

• Please use the available disk space responsibly. No matter how large the disks of lc3 and lc4 may be, space is still limited and you are not the only user. Also keep in mind that flcl01 is not meant as a large-scale file server. As a consequence, the “/data” filesystem has a per-user quota of 2 GB, including a temporary limit of 5 GB that may be used for a grace period of up to five days.

Miscellaneous

• dCache: The dCache system at DESY is able to store large amounts of data – have a look at the dCache Home Page to find out more details. Since the data in the dCache system is stored on tapes, the dCache can be regarded as a rather safe storage. Every FLC user should have their personal dCache directory under “/pnfs/desy.de/flc/users/${USER}” – contact the administrators if you don’t own a directory yet. The PNFS filesystem is mounted on the FLC group servers and on some of the local computers.

• Nirvana: Data that you do not expect to need anymore in the medium-term future (but which you do not want to delete for good, either) can be moved into the “nirvana” directory “/scratch/nirvana” on flc34 – contact the FLC administrators in case of need. All data accumulated in the “nirvana” will be written to archive tapes from time to time. After that, it will be deleted from the disk, thus freeing up space. Go and see one of the administrators if you actually need to retrieve some data from the archive again.

• Grid storage: If you have a Grid certificate and you are a member of a VO (probably “ilc” or “calice”), you can store your data on a Grid Storage Element (SE). The default Storage Element for the VOs ilc and calice is currently named “dcache-se-desy.desy.de”. Consult the gLite 3.1 User Guide to learn how to access Storage Elements and how to move your data back and forth. You can create replicas of your files on multiple Storage Elements for improved data security.

• TSM Archive: The DESY IT department provides the TSM Archive for long-term storage of your data. See the DESY IT page on the Central Backup at DESY for more information.