start:fs_dm:archiving:irods:details

Technical Background Information to the ZDV iRODS System

Storage Locations

iRODS works based a kind of a virtual file system, with its own lingo:

  • directories are called collections, they may contain further subcollections
  • files are (data objects).

iRODS uses so called 'Resources' to archive the collections and data objects. The resources are organized hierarchically. The root is a replication resource, where other resources are added as children. Currently there is a compound resource consisting itself of a cache (unix filesystem) and a universal mass storage system (here: TSM) as archive. The cache has a size of 8TB, once it is fills up, the oldest data objects will be deleted on the cache. If required, they will be fetched back from the archive.

replResc:replication
├── cephfsResc:unixfilesystem
└── compResc:compound
    ├── netappResc:unixfilesystem
    └── tsmResc:univmss

Transfer Speeds

Here we show some transfer speed benchmarks:

 iRODS upload benchmark with one file of varying size.  iRODS upload benchmark with multiple tiny files.

This explains why a directory, resp. collection, consisting of many small files, should be compressed. iRODS works best for files > 5GB.

Supplied iRODS Configuration

Every user has a hidden directory ${HOME}/.irods with the file irods_environment.json our the HPC Systems home directory containing the connection information for the iRODS archive. Below you see the information template for irods_environment.json. Do not edit this file.

{
    "irods_client_server_negotiation": "request_server_negotiation",
    "irods_client_server_policy": "CS_NEG_REQUIRE",
    "irods_authentication_scheme": "KRB",
    "irods_host": "irods.zdv.uni-mainz.de",
    "irods_port": 1247,
    "irods_user_name": "<$USER>",
    "irods_zone_name": "zdv",
    "irods_encryption_key_size": 32,
    "irods_encryption_salt_size": 8,
    "irods_encryption_num_hash_rounds": 16,
    "irods_encryption_algorithm": "AES-256-CBC"
}

Please do not use iinit, it will not work and change your ${HOME}/.irods/irods_environment.json file. In case it happened, remove the directory ${HOME}/.irods and log in again, it will be restored upon login.

Security Risk

If you initiate the command iinit it might happen that an additional file .irodsA is also in the directory ${HOME}/.irods . Please remove this file! It contains your password in a decryptable form.
  • start/fs_dm/archiving/irods/details.txt
  • Last modified: 2021/04/09 14:01
  • by meesters