How to store your research data

The DFG and RUB require that you store the primary data (that means for us the input & final output files) from your research for 10 years and that they are stored in accordance with the FAIR principles (Findable, Accessible, Interoperable, Reusable, see DFG's code of conduct):

In the Quantum Chemistry Group we ensure this by the following:

1. backing up the home directories on the desktops machines,
2. storing scripts, source code, and text documents in git repositories,
3. and finally, saving after completion of the projects (usually at the end of the PhD or before someone leaves the group) the data in addition on DVDs or an external USB disc.

This requires that every group member continuously (not only at the end of your MSc or PhD):

1. Stores all final research data in the home directory for the desktop machines in separate directories than private data. In the directory trees with research data all (sub)directories have to be group-readable and group-executable for akhaet, and all files have to be group-readable for akhaet.
2. Puts all files (like *.tex, *.doc, *.docx, *.ppt, *.pptx, figures, etc.) used to present research results in a publication, talk, poster or in other form into git repositories under the folder /home/turbomole/GITREPOSITORY. Alternatively, the data for talks, posters, reportsc, etc. can have to stored in folders in your home directory in the theochem network that is included in the backup system.
3. Ensures that all research data is stored on machines or devices at RUB that are group-accessible. Any data that is generated or edited on mobile devices (laptops, tablets, USB sticks, etc. that are carried around) or on private devices has to be synchronized daily with a machine or device at RUB that is group-accessible. (This can e.g.\ be done automatically via sciebo.) The research data that needs to be stored are the input and output files for/from all calculations from which results have been used for a publication, thesis, report, presentation, poster, etc. Input files are those files that a needed to reproduce the calculation, e.g. for TURBOMOLE, the control and coord files, and if non-standard basis sets are used, the basis set files (but not automatically generated intermediate files like mos, alpha, beta, lmos, hessapprox, optinfo, forceapprox, statistics.*, *dens*, *.cao, CC*, etc.; this also includes reformatted MO data e.g.\ in molden.input format and files with similar content with other names). If data is processed further, e.g. fitted to functions, structure analysis, MO analysis, generation of plots, etc. also the input and output files for these calculations have to be stored. This can then also include scripts and program used for plotting.
4. stores her/his data on all machines in a transparent & understandable directory structure
with sufficient documentation (README files) and/or the metadata that is neccessary to implement the FAIR principles (Findable, Accessible, Interoperable, Reusable)
5. The directory tree for each project should contain a README file explaining the data structure
6. Final data used for publications, a thesis, or a report should be accompanied with information on the work flow by which it was obtained unless this is documented in the input files. In particular it has to be documented how the nuclear coordinates have been obtained. The preferred way to do this via README files.
7. Self-written programs or scripts that have been used to generate data (including input files) or post-processing data have to be stored either together with the data and in the group's git repository (unlesss special agreements apply as e.g. for the TURBOMOLE package). They should contain or be accompanied with a documentation that explains its usage.
8. keeps her/his directories cleaned up, i.e. free from scratch files, data trash from calculations that crashed, huge files that can easily be reproduced with the input files, but are difficult to store for a long time (typically any binary data, MOs, *.plt, etc.)
9. regularly (at least once every month) synchronizes the data that needs to be saved to her/his home directory on the desktop machines, the command rsync -avz sourcedir login:targetdir is recommended for this
10. Those who made changes to the TURBOMOLE program package should regularly (at least weekly) commit their changes and push them to turbogit. Whenever the work on a branch has been completed, the branch should be pushed to turbogit and deleted from the local repository with git branch --delete branch_name.
11. All other scripts and programs that could potentially be of general use should be put into a git repository below /home/turbomole/GITREPOSITORY

Some rules for structuring the data:

• Research results from computations in general: All output files that contain results that have been used for any publication, report, or thesis has to be stored together with all input files, documentation, and metadata that is needed to reproduce the calculations.
  • If input files contain references to other files or folders with additional input data the references to those files and folders should be given whenever with relative paths and point to files and folders that are kept in the directory tree for the same project or project part so that the references do not get broken when this directory tree is moved to another place for long time storage (backup) or recovered from there.
  • Where the compound/chemical system, software and version, or applied workflow is not immediately apparent from the names of the files or specified in the header of the files, it should be documented in a README file stored in the same directory tree. If the same workflow is applied with the same software to a set of chemical systems this description should preferable be given in a top level directory that is common for this set.
• TURBOMOLE calculations: There should be one directory per calculation (required control input file). It has to contain all input files that are needed to reproduce the calculation and all output files that contain results that have been used in any publication, report, or thesis.
  • Input files required to reproduce calculations are the control file and, if not included in the control file, e.g. the files with the coordinates, gradients, hessian.
  • Make sure that the control file contains only file references with relative paths, no absolute paths that get broken if a set of directories is moved or copied to a different place (e.g. for backup).
• COSMOtherm calculations: There should be one directory per set of conformers for an acid/base pair. It has to contain all input files that are needed to reproduce the calculation and all output files that contain results that have been used in any publication, report, or thesis.
  • Output files are files with the job output (*.out) with the energies and the summary files (*.tab). The directory structure has to contain a README file that describes how the cosmo files were generated.
  • Input files required to reproduce calculations are usually the *.inp files plus all cosmo files that are not taken from a database included in COSMOtherm.
  • Make sure that the input files contains only file references with relative paths, no absolute paths that get broken if a set of directories is moved or copied to a different place (e.g. for backup). The only exception are references to files in the COSMOtherm database.

Some rules for cleaning up after a calculation:

1. Immediatley after a calculation has finished it should be checked if it was successful for not. In both cases any backup directory, named BACK or similar, has to be removed (not renamed!) to a) avoid redundant storage and b) document that this calculation has been checked. Only if there are problems that require debugging or discussion with an expert such directories might be kept. Those and only those files that contain final output or input needed for reproducing the calculation should be moved into the directory from where the calculations was started. Don't copy it to a subdirectory to avoid data duplication.
2. The following files that are sometimes left behind by TURBOMOLE programs should be deleted immediately (preferably already in the run scripts for the queue):
   • diff_dft_oper, diff_dft_density, diff_densmat, diff_fockmat, errvec, oldfock, ddens, diis*, arh* (dscf, ridft scratch files)
   • dh (aoforce binary file with differentiated one-electron integrals)
   • slave*output, metastase, hp_mpi_appfile, machinefile, control.old* (MPI parallel runs)
   • CCB*, CCY*, CCH*, CCK*, CCI*, CCJ*, CCV*, CC_, CCTR*, CC3*, CCXI*, CCfe0*, RIR12* (scratch files from ricc2, ccsdf12, or pnoccsd)
   • PNO* (scratch files from pnoccsd)
   • a*mat.tmp, frho_cosmo, out.ccf, out.cosmo, and cosmo*.tmp (scratch files from COSMO)
   Note: These are only examples. Everyone has to check for his type of calculations and applications which data has to be kept and which data should be deleted. If you are not sure, please ask.
3. After successful geometry optimizations, remove job.[0-9]*, keep only job.last
4. After calculations with the scripts for numerical differentiation (NumForce, NumGrad, etc.) the directories created by these scripts should be cleaned up (delete all *cluster*err*, wherefrom, statistics* files), then archive it in a tar file, compress it with bzip2 and delete the original directory.
5. After calculations with woelfling-job the directories created by should be cleaned up (delete mos, alpha, beta, CC*, statistics*, wherefrom, hessapprox), then archive the directories in a tar file, compress it with bzip2 and delete the original directories. Also delete the files with intermediate results for the path.

Some rules for cleaning up after completing a project part:

1. Delete directories with the results from calculations that crashed or failed for technical reasons
2. Delete huge files with data that is not needed to check or reproduce the calculation as e.g. mos, alpha, beta, *.cao, CC*, *.plt, dens, exdens
3. If you worked on program development delete directories or files from calculations that were done with unfinished debug versions which can not be reproduced (such results should never enter any publication or report).
4. Make sure that the directory names are easy to understand but not too long and that the directory tree has a systematic logic structure.
5. Move the directory tree to the file system on the desktop machines which is regularly backed up.