User Tools

Site Tools


data_management:irods

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
Next revision Both sides next revision
data_management:irods [2019/08/19 10:16]
jsteinka [Navigation]
data_management:irods [2019/11/19 08:34]
jsteinka [Publishing]
Line 5: Line 5:
   *  files are (//data objects//).   *  files are (//data objects//).
  
-All commands from the iRODS command line tools start with an '//​i//'​.+All commands from the iRODS command line tools start with an '//​i//' ​and they are installed on the mogon login nodes. 
 + 
 +iRODS uses so called '//​Resources//'​ to archive the //​collections//​ and //data objects//. The //​resources//​ are organized hierarchically. The root is a //​replication resource//, where other //​resources//​ are added as children. Currently there is a //compound resource// consisting itself of a cache (unix filesystem) and a universal mass storage system (here: TSM) as archive. The cache has a size of 8TB, once it is fills up, the oldest //data objects// will be deleted on the cache. If required, they will be fetched back from the archive. 
 + 
 +<​code>​ 
 +replResc:​replication 
 +├── cephfsResc:​unixfilesystem 
 +└── compResc:​compound 
 +    ├── netappResc:​unixfilesystem 
 +    └── tsmResc:​univmss 
 +</​code>​
  
 On request it is still possible to apply for a [[archiving:​tsm|TSM account]], to store the data on tapes. On request it is still possible to apply for a [[archiving:​tsm|TSM account]], to store the data on tapes.
Line 18: Line 28:
  
 If a directory, resp. //​collection//,​ consists of many small files, those files should be [[archiving:​preparation|compressed]]. iRODS works best for files > 5GB. If a directory, resp. //​collection//,​ consists of many small files, those files should be [[archiving:​preparation|compressed]]. iRODS works best for files > 5GB.
 +{{ :​data_management:​test-large-files.png?​direct&​400 | iRODS upload benchmark with one file of varying size.}} 
 +{{ :​data_management:​test-small-files.png?​direct&​400 | iRODS upload benchmark with multiple tiny files.}}
 ===== iRODS Account ===== ===== iRODS Account =====
  
Line 27: Line 38:
  
 Collections in the home folder of individual user will be deleted once the account gets deleted. Only the // /​zdv/​project/<​PROJECT NAME> // collections will be archived for an appropriate period (default: 10 years, as suggested by the [[https://​www.dfg.de/​foerderung/​grundlagen_rahmenbedingungen/​gwp/​|DFG,​ Leitlinie 17]]). Collections in the home folder of individual user will be deleted once the account gets deleted. Only the // /​zdv/​project/<​PROJECT NAME> // collections will be archived for an appropriate period (default: 10 years, as suggested by the [[https://​www.dfg.de/​foerderung/​grundlagen_rahmenbedingungen/​gwp/​|DFG,​ Leitlinie 17]]).
 +
 +The data saved to the above project collection is owned by the group. This means after a user leaves the project, the data can still be accessed by the other group members, as long as the ACLs are not modified.
 </​WRAP>​ </​WRAP>​
  
Line 51: Line 64:
 **Security warning** **Security warning**
  
-If you initiate the command ''​iinit''​ it might happen that an additional file '//​.irodsA//'​ is also in the folder '//​.irods//'​. Please remove this file! It contains your entered ​password in a decryptable form.+If you initiate the command ''​iinit''​ it might happen that an additional file '//​.irodsA//'​ is also in the folder '//​.irods//'​. Please remove this file! It contains your password in a decryptable form.
 </​WRAP>​ </​WRAP>​
  
Line 62: Line 75:
 As mentioned above, iRODS is a kind of a virtual filesystem. The following commands can be used to for navigation. As mentioned above, iRODS is a kind of a virtual filesystem. The following commands can be used to for navigation.
  
-^ Command ​   ^ Parameters ​ ^ Description ​                                                         +^ Command ​   ^ Parameters ​ ^ Description ​                                                        ​
-| ''​ipwd'' ​  ​| ​            | print current iRODS working directory (colection) ​                   +| ''​ipwd'' ​  ​| ​            | print current iRODS working directory (colection) ​                  ​
-| ''​ils'' ​   | -l, -L, -A  | list iRODS collection (-l: with details; -L: more details; -A: ACLs) | +| ''​ils'' ​   | -l, -L, -A  | list iRODS collection (-l: with details; -L: more details; -A: ACL) | 
-| ''​icd'' ​   | <​target> ​   | change iRODS collection ​                                             +| ''​icd'' ​   | <​target> ​   | change iRODS collection ​                                            ​
-| ''​imkdir''​ | -p <​coll> ​  | create a new collection (directory; -p: with parents) ​               |+| ''​imkdir''​ | -p <​coll> ​  | create a new collection (directory; -p: with parents) ​              ​|
  
 Each user gets his/her personal home under /​zdv/​home/​${USER} and access to the associated Mogon I/II projects under /​zdv/​project/<​PROJECT NAME>​. ​ Each user gets his/her personal home under /​zdv/​home/​${USER} and access to the associated Mogon I/II projects under /​zdv/​project/<​PROJECT NAME>​. ​
Line 72: Line 85:
 Accessible folders: Accessible folders:
   * ''/​zdv/​home/​${USER}''​ private directory   * ''/​zdv/​home/​${USER}''​ private directory
-  * ''/​zdv/​trash/​${USER}''​ private trash bin 
   * ''/​zdv/​project/<​PROJECT NAME>''​ project directory   * ''/​zdv/​project/<​PROJECT NAME>''​ project directory
   * ''/​zdv/​home/​public''​ every registered user can read/​write/​delete   * ''/​zdv/​home/​public''​ every registered user can read/​write/​delete
 +  * ''/​zdv/​trash/​home/​${USER}''​ private trash bin
 +
 +
 +<WRAP center round info 80%>
 +Only the project directory are meant for permanent archiving. We are working on a solution to prevent archiving to the **volatile** iRODS homes.
 +</​WRAP>​
  
 ==== Archiving ==== ==== Archiving ====
Line 84: Line 102:
 | ''​ichksum''​ | -r <​obj%%|%%coll>​ | Compute and store checksums (-r: recursive) ​                   | | ''​ichksum''​ | -r <​obj%%|%%coll>​ | Compute and store checksums (-r: recursive) ​                   |
  
-The checksum is calculated server side and we highly recommend to switch it on immediately on upload. Nevertheless,​ You can also do it later on with ''​ichksum -K <​filename>''​. It creates a checksum equivalent to the command ''​sha256sum <local filename>​ | cut -d " " -f 1 | xxd -r -p | base64'',​ which you can compare to ensure data integrity. The checksums can be queried with ''​ils -L''​ and ''​ichksum''​.+The checksum is calculated server side and we highly recommend to switch it on immediately on upload. Nevertheless,​ You can also do it later on with ''​ichksum -K <​filename>''​. It creates a checksum equivalent to the command ''​sha256sum <local filename>​ | cut -d " " -f 1 | xxd -r -p | base64'',​ which you can compare to ensure data integrity. The checksums can be queried with ''​ils -L''​ and ''​ichksum''​. However, if you don't do it on upload with ''​iput -k'',​ there will be no checksum for the TSM ressource.
  
 As mentioned above, several small files should be bundled. Nevertheless,​ you can still extract an uploaded tar archive on the server and index all contained files with the command ''​ibun''​. Please read its man page for further details (''​man ibun''​). As mentioned above, several small files should be bundled. Nevertheless,​ you can still extract an uploaded tar archive on the server and index all contained files with the command ''​ibun''​. Please read its man page for further details (''​man ibun''​).
Line 107: Line 125:
 | -f        | overwrite local existing files | | -f        | overwrite local existing files |
  
 +=== Example ===
 +<code bash>
 +
 +[user@login01 ~]$ kinit
 +Password for user@UNI-MAINZ.DE: ​
 +
 +[user@login01 ~]$ ipwd
 +/​zdv/​home/​user
 +
 +[user@login01 ~]$ ils /​zdv/​home/​public
 +/​zdv/​home/​public:​
 +  hello_world.txt
 +[user@login01 ~]$ ils -l /​zdv/​home/​public
 +/​zdv/​home/​public:​
 +  rods              0 replResc;​compResc;​netappResc ​          24 2019-08-19.11:​00 & hello_world.txt
 +  rods              1 replResc;​compResc;​tsmResc ​          24 2019-08-19.11:​01 & hello_world.txt
 +[user@login01 ~]$ ils -L /​zdv/​home/​public
 +/​zdv/​home/​public:​
 +  rods              0 replResc;​compResc;​netappResc ​          24 2019-08-19.11:​00 & hello_world.txt
 +        generic ​   /​fsapp/​iRODS/​Vault/​home/​public/​hello_world.txt
 +  rods              1 replResc;​compResc;​tsmResc ​          24 2019-08-19.11:​01 & hello_world.txt
 +        generic ​   /​fsapp/​iRODS/​Vault/​home/​public/​hello_world.txt
 +[user@login01 ~]$ ils -A /​zdv/​home/​public
 +/​zdv/​home/​public:​
 +        ACL - g:​public#​zdv:​own ​  
 +        Inheritance - Disabled
 +  hello_world.txt
 +        ACL - public#​zdv:​read object ​  ​rods#​zdv:​own ​  
 +[user@login01 ~]$ ils -LA /​zdv/​home/​public
 +/​zdv/​home/​public:​
 +        ACL - g:​public#​zdv:​own ​  
 +        Inheritance - Disabled
 +  rods              0 replResc;​compResc;​netappResc ​          24 2019-08-19.11:​00 & hello_world.txt
 +        generic ​   /​fsapp/​iRODS/​Vault/​home/​public/​hello_world.txt
 +        ACL - public#​zdv:​read object ​  ​rods#​zdv:​own ​  
 +  rods              1 replResc;​compResc;​tsmResc ​          24 2019-08-19.11:​01 & hello_world.txt
 +        generic ​   /​fsapp/​iRODS/​Vault/​home/​public/​hello_world.txt
 +        ACL - public#​zdv:​read object ​  ​rods#​zdv:​own ​  
 +
 +[user@login01 ~]$ iget /​zdv/​home/​public/​hello_world.txt
 +[user@login01 ~]$ ls -l hello_world.txt ​
 +-rw-r----- 1 user zdv 24 Aug 19 11:18 hello_world.txt
 +
 +[user@login01 ~]$ iput hello_world.txt ​
 +[user@login01 ~]$ ils -L
 +/​zdv/​home/​user:​
 +  user          0 replResc;​compResc;​netappResc ​          24 2019-08-19.11:​20 & hello_world.txt
 +        generic ​   /​fsapp/​iRODS/​Vault/​home/​user/​hello_world.txt
 +  user          1 replResc;​compResc;​tsmResc ​          24 2019-08-19.11:​20 & hello_world.txt
 +        generic ​   /​fsapp/​iRODS/​Vault/​home/​user/​hello_world.txt
 +
 +ichksum hello_world.txt
 +    hello_world.txt ​   sha2:​XPdR4XQP49lWUGEfPJz0Jo+kmkndGxz6rCQUzCqHteA=
 +Total checksum performed = 1, Failed checksum = 0
 +[user@login01 ~]$ ils -L
 +/​zdv/​home/​user:​
 +  user          0 replResc;​compResc;​netappResc ​          24 2019-08-19.11:​20 & hello_world.txt
 +    sha2:​XPdR4XQP49lWUGEfPJz0Jo+kmkndGxz6rCQUzCqHteA= ​   generic ​   /​fsapp/​iRODS/​Vault/​home/​user/​hello_world.txt
 +  user          1 replResc;​compResc;​tsmResc ​          24 2019-08-19.11:​20 & hello_world.txt
 +        generic ​   /​fsapp/​iRODS/​Vault/​home/​user/​hello_world.txt
 +
 +[user@login01 ~]$  sha256sum hello_world.txt | cut -d " " -f 1 | xxd -r -p | base64
 +XPdR4XQP49lWUGEfPJz0Jo+kmkndGxz6rCQUzCqHteA=
 +
 +[user@login01 ~]$ irm -f hello_world.txt ​
 +[user@login01 ~]$ iput -k hello_world.txt ​
 +[user@login01 ~]$ ils -L
 +/​zdv/​home/​user:​
 +  user          0 replResc;​compResc;​netappResc ​          24 2019-08-19.11:​27 & hello_world.txt
 +    sha2:​XPdR4XQP49lWUGEfPJz0Jo+kmkndGxz6rCQUzCqHteA= ​   generic ​   /​fsapp/​iRODS/​Vault/​home/​user/​hello_world.txt
 +  user          1 replResc;​compResc;​tsmResc ​          24 2019-08-19.11:​28 & hello_world.txt
 +    sha2:​XPdR4XQP49lWUGEfPJz0Jo+kmkndGxz6rCQUzCqHteA= ​   generic ​   /​fsapp/​iRODS/​Vault/​home/​user/​hello_world.txt
 +</​code>​
 ==== Metadata: ''​imeta''​ ==== ==== Metadata: ''​imeta''​ ====
  
Line 129: Line 220:
  
  
-The following ​two commands upload the filesystem table and query the metadata:+The following ​command lists the metadata ​automatically associated with the previously upladed file ''​hello_world.txt''​:
 <code bash> <code bash>
-iput <​path/​filename>​ +imeta ls -d hello_world.txt
-imeta ls -d <​filename>​+
 </​code>​ </​code>​
  
Line 138: Line 228:
  
 <​code>​ <​code>​
-AVUs defined for dataObj ​<​filename>​:+AVUs defined for dataObj ​hello_world.txt:
 attribute: AccessRights attribute: AccessRights
 value: closed value: closed
Line 144: Line 234:
 ---- ----
 attribute: Creator attribute: Creator
-value: ​<you>+value: ​Steinkamp, J.
 units: ​ units: ​
 ---- ----
 attribute: Date attribute: Date
-value: ​1562677318+value: ​1566206896
 units: ​ units: ​
 ---- ----
 attribute: ExpiryDate attribute: ExpiryDate
-value: ​1878901318+value: ​1882430896
 units: ​ units: ​
 ---- ----
Line 166: Line 256:
 value: Johannes Gutenberg-University value: Johannes Gutenberg-University
 units: ​ units: ​
 +[user@login01 ~]$ 
 </​code>​ </​code>​
  
 +You can now add a title, which is not created automatically:​
 <​code>​ <​code>​
-imeta set -d <​filename> ​Title "​Archive of experimental szstem from '​$(date)'"​+imeta set -d hello_world.txt ​Title "​Archive of experimental szstem from '​$(date)'"​
 </​code>​ </​code>​
  
-If you query the Attribute '​Table'​ with ''​imeta ls -d <​filename> ​Title''​ you get:+If you query the Attribute '​Table'​ with ''​imeta ls -d hello_world.txt ​Title''​ you get:
 <​code>​ <​code>​
-AVUs defined for dataObj ​fstab:+AVUs defined for dataObj ​hello_world.txt:
 attribute: Title attribute: Title
-value: Archive of experimental szstem from '<​somedate>​'+value: Archive of experimental szstem from 'Mon Aug 19 11:39:48 CEST 2019'
 units: ​ units: ​
 </​code>​ </​code>​
Line 191: Line 283:
  
 <code shell> <code shell>
-imeta mod -d <​filename> ​Title "​Archive of experimental szstem from '<​somedate>​'"​ v:"​Archive of experimental system from '<​somedate>​'"​+imeta mod -d hello_world.txt ​Title "​Archive of experimental szstem from 'Mon Aug 19 11:39:48 CEST 2019'"​ v:"​Archive of experimental system from 'Mon Aug 19 11:39:48 CEST 2019'"​
 </​code>​ </​code>​
  
Line 246: Line 338:
 ==== Searching ==== ==== Searching ====
  
 +=== for filenames: ''​ilocate''​ ===
  
-  * ''​ilocate -t "​%fstab%"''​ search for filename (pattern) +<code bash>
-  * ''​imeta qu -C Name like "​%fstab%"''​ search for metadata+
  
-==== Publishing ====+[user@login01 ~]$ ilocate -t "​hello_world.txt"​  
 +/​zdv/​home/​user/​hello_world.txt 
 +/​zdv/​home/​public/​hello_world.txt 
 +</​code>​
  
-For public access a **ticket** needs to be created ​for collections or data objectsFor example, if you use the above uploaded ​''​fstab'' ​file (here for my personal home directorywhich might not work in the future any more, see above why).+=== for metadata: ''​imeta qu''​ === 
 + 
 +You must know, if you want to search ​for data object (''​-d''​) or a collection (''​-C''​)And you can use SQL wildcards (''​%''​)if you don't know the exact pattern you are looking for. The wildcard pattern matching is also applicable for ''​ilocate''​.
  
 <code bash> <code bash>
-iticket create read fstab+[user@login01 ~]$ imeta qu -d Title like "​Archive%"​ 
 +collection: /​zdv/​home/​user 
 +dataObj: hello_world.txt
 </​code>​ </​code>​
-returns:+ 
 +=== via database query''​iquest''​ === 
 + 
 +For this complex syntax consult the [[https://​docs.irods.org/​4.2.6/​icommands/​user/#​iquest|Online help]]. 
 + 
 +==== Publishing ==== 
 + 
 +For public access a **ticket** needs to be created for collections or data objects. For example, if you use the above uploaded ''​hello_world.txt''​ file again. 
 <code bash> <code bash>
-ticket:GGkUTXdJpfK7VPi+[user@login01 ~]$ iticket create read hello_world.txt 
 +ticket:ACR2RKDyuZMBRmb
 </​code>​ </​code>​
  
-Querying ​the metadata via the provided REST-API ​returns a JSON stringThis can be viewed in the browser or as done here using curl:+With this ticket and the path everybody can query information and the content of collections and data objects vi a provided REST-APIJSON strings are returned for valid URLs. 
 + 
 +=== General information about data objects ===
  
 <code bash> <code bash>
-curl https://​irods-test.zdv.uni-mainz.de/​irods-rest/​rest/​dataObject/​zdv/​home/​jsteinka/​fstab/​metadata?ticket=GGkUTXdJpfK7VPi+[user@login01 ~]$ curl https://​irods-web.zdv.uni-mainz.de/​irods-rest/​rest/​dataObject/​zdv/​home/​jsteinka/​hello_world.txt?ticket=ACR2RKDyuZMBRmb
 </​code>​ </​code>​
-returns: 
 <code JavaScript>​ <code JavaScript>​
-{"metadataEntries": ​+{"id":1808764
-  {"​count":​1+ "​collectionId":24346
-   ​"lastResult":true+ "​dataName":"​hello_world.txt", 
-   ​"totalRecords":0, + "​collectionName":"​/​zdv/​home/​jsteinka", 
-   "attribute":"​protected", + "​dataReplicationNumber":0, 
-   ​"value":"​false", + "​dataVersion":0, 
-   ​"unit":""​},​  + "​dataTypeName":"​generic", 
-  {"​count":​2,​ + "​dataSize":24
-   "​lastResult":​true,​ + "​resourceGroupName":"",​ 
-   "​totalRecords":0, + "​resourceName":"​netappResc", 
-   "​attribute":"​AccessRights",​ + "​dataPath":"​/​fsapp/​iRODS/​Vault/​home/​jsteinka/​hello_world.txt", 
-   "​value":"​closed",​ + "​dataOwnerName":"​jsteinka", 
-   "​unit":""​},​ + "​dataOwnerZone":"​zdv", 
-  {"​count":​3,​ + "​replicationStatus":"​1", 
-   "​lastResult":​true,​ + "​dataStatus":"",​ 
-   "totalRecords":0, + "​checksum":"​sha2:XPdR4XQP49lWUGEfPJz0Jo+kmkndGxz6rCQUzCqHteA=", 
-   ​"attribute":"​Publisher", + "​expiry":"​00000000000", 
-   ​"value":"​Johannes Gutenberg-University"​+ "​dataMapId":0, 
-   ​"unit":""​}+ "​comments":"",​ 
-  {"count":4, + "​createdAt":1566206868000
-   "lastResult":true+ "​updatedAt":1566206868000
-   ​"totalRecords":0, + "​specColType":"​NORMAL", 
-   "attribute":"​Location", + "​objectPath":""​ 
-   ​"value":"​Mainz, Germany", +}
-   ​"unit":""​}+
-  {"count":5, +
-   "lastResult":true+
-   ​"totalRecords":0, +
-   "attribute":"​Creator", +
-   ​"value":"​Steinkamp, Jörg",​ +
-   "​unit"​:""}+
-  {"count":6, +
-   "lastResult":true+
-   ​"totalRecords":0, +
-   ​"attribute":"​Title", +
-   ​"value":"​Filesystem Table of '​login01.mogon'"​+
-   ​"unit":""​}+
-  {"count":7, +
-   "lastResult":true+
-   ​"totalRecords":0, +
-   "attribute":"​Date",​ +
-   "​value":"​1562679909",​ +
-   "​unit":""​},​ +
-  {"​count":​8,​ +
-   "​lastResult":​true,​ +
-   "​totalRecords":​0,​ +
-   "​attribute":"​ExpiryDate",​ +
-   "​value":"​1878903909",​ +
-   "​unit":""​}],​ +
-"​objectType":"​DATA_OBJECT",​ +
-"​uniqueNameString":"/​zdv/​home/​jsteinka/​fstab"​}+
 </​code>​ </​code>​
  
 +=== Querying the metadata ===
  
-  * Retrieve the metadata of a collection: 
 <code bash> <code bash>
-curl https://​irods-test.zdv.uni-mainz.de/​irods-rest/​rest/​collection/​zdv/​home/​jsteinka/​test/​metadata?​ticket=AKR9iYWmfSU6niN +[user@login01 ~]$ curl https://​irods-web.zdv.uni-mainz.de/​irods-rest/​rest/​dataObject/​zdv/​home/​jsteinka/​hello_world.txt/​metadata?​ticket=ACR2RKDyuZMBRmb
-{"​metadataEntries":​[ +
-        {"​count":​1,​ +
-         "​lastResult":​true,​ +
-         "​totalRecords":​0,​ +
-         "​attribute":"​Name",​ +
-         "​value":"​This is a test project","​unit":""​}],​ +
-    "​objectType":"​COLLECTION",​ +
-    "​uniqueNameString":"​zdv/​home/​jsteinka/​test"​ +
-}+
 </​code>​ </​code>​
-  * Retrieve the metadata ​of a data object:+<code JavaScript>​ 
 +{"​metadataEntries":​ [ 
 +    {"​count":​1,​ 
 +     "​lastResult":​true,​ 
 +     "​totalRecords":​0,​ 
 +     "​attribute":"​AccessRights",​ 
 +     "​value":"​closed",​ 
 +     "​unit":""​},​ 
 +    {"​count":​2,​ 
 +     "​lastResult":​true,​ 
 +     "​totalRecords":​0,​ 
 +     "​attribute":"​Publisher",​ 
 +     "​value":"​Johannes Gutenberg-University",​ 
 +     "​unit":""​},​ 
 +    {"​count":​3,​ 
 +     "​lastResult":​true,​ 
 +     "​totalRecords":​0,​ 
 +     "​attribute":"​Location",​ 
 +     "​value":"​Mainz,​ Germany",​ 
 +     "​unit":""​},​ 
 +    {"​count":​4,​ 
 +     "​lastResult":​true,​ 
 +     "​totalRecords":​0,​ 
 +     "​attribute":"​protected",​ 
 +     "​value":"​false",​ 
 +     "​unit":""​},​ 
 +    {"​count":​5,​ 
 +     "​lastResult":​true,​ 
 +     "​totalRecords":​0,​ 
 +     "​attribute":"​Creator",​ 
 +     "​value":"​Steinkamp,​ J.", 
 +     "​unit":""​},​ 
 +    {"​count":​6,​ 
 +     "​lastResult":​true,​ 
 +     "​totalRecords":​0,​ 
 +     "​attribute":"​Date",​ 
 +     "​value":"​1566206896",​ 
 +     "​unit":""​},​ 
 +    {"​count":​7,​ 
 +     "​lastResult":​true,"​ 
 +     ​totalRecords":​0,​ 
 +     "​attribute":"​ExpiryDate",​ 
 +     "​value":"​1882430896",​ 
 +     "​unit":""​},​ 
 +    {"​count":​8,​ 
 +     "​lastResult":​true,​ 
 +     "​totalRecords":​0,​ 
 +     "​attribute":"​Title",​ 
 +     "​value":"​Archive ​of experimental system from 'Mon Aug 19 11:39:48 CEST 2019'",​ 
 +     "​unit":""​}],​ 
 +"​objectType":"​DATA_OBJECT","​uniqueNameString":"/​zdv/​home/​jsteinka/​hello_world.txt"​} 
 +</​code>​ 
 + 
 +=== Downloading data === 
 +The file content can be viewed with ''​curl''​ or downloaded with ''​wget''​.
 <code bash> <code bash>
-curl https://​irods-test.zdv.uni-mainz.de/​irods-rest/​rest/​dataObject/​zdv/​home/​jsteinka/​test/​garbage_small.zero/​metadata?ticket=qM3arKFlkBM0Ue5 +curl https://​irods-web.zdv.uni-mainz.de/​irods-rest/​rest/​fileContents/​zdv/​home/​jsteinka/​hello_world.txt?ticket=ACR2RKDyuZMBRmb
-{"​metadataEntries":​[ +
-        {"​count":​1,​ +
-         "​lastResult":​true,​ +
-         "​totalRecords":​0,​ +
-         "​attribute":"​Creator",​ +
-         "​value":"​Joerg Steinkamp",​ +
-         "​unit":""​}, ​  +
-        {"​count":​2,​ +
-         "​lastResult":​true,​ +
-         "​totalRecords":​0,​ +
-         "​attribute":"​Description",​ +
-         "​value":"​only zeroes from dd if=/​dev/​zero of=garbage_small-zero bs=1024 count=24",​ +
-         "​unit":""​}],​ +
-    "​objectType":"​DATA_OBJECT",​ +
-    "​uniqueNameString":"/​zdv/​home/​jsteinka/​test/​garbage_small.zero"​ +
-}+
 </​code>​ </​code>​
-  * Download ​the data object:+ 
 +=== Retrieve ​the metadata of a collection ===
 <code bash> <code bash>
-wget  ​https://​irods-test.zdv.uni-mainz.de/​irods-rest/​rest/​fileContents/zdv/home/jsteinka/test/​garbage_small.zero?ticket=qM3arKFlkBM0Ue5+curl https://​irods-web.zdv.uni-mainz.de/​irods-rest/​rest/​collection/zdv/home/public/helloCollection?ticket=mbyAAFGm7vhUdyM
 </​code>​ </​code>​
 +<code JavaScript>​
 +{
 + "​collectionId":​1808781,​
 + "​collectionName":"/​zdv/​home/​public/​helloCollection",​
 + "​objectPath":"",​
 + "​collectionParentName":"/​zdv/​home/​public/",​
 + "​collectionOwnerName":"​rods",​
 + "​collectionOwnerZone":"​zdv",​
 + "​collectionMapId":"​0",​
 + "​collectionInheritance":"",​
 + "​comments":"",​
 + "​info1":"",​
 + "​info2":"",​
 + "​createdAt":​1566213731000,​
 + "​modifiedAt":​1566213731000,​
 + "​specColType":"​NORMAL",​
 + "​children":​[]
 +}
 +</​code>​
 +
 +=== REST-API URL ===
 +
 +the URL for the REST-API consists of:
 +  * Server (https://​irods-web.uni-mainz.de/​irods-rest/​rest)
 +  * what (collection|dataObject|fileContents)
 +  * iRODS path
 +  * optionally '​metadata'​
 +  * Ticket string (?​ticket=1234567890)
  
 +For further information,​ please read the original [[https://​github.com/​DICE-UNC/​irods-rest|IRODS-REST documentation]]
 ===== Data Policy/​Recommendation ===== ===== Data Policy/​Recommendation =====
  
 The "​Creator"​ is the responsible person in the sense of the Urheberrechtsgesetz,​ taking care that reusing of third party data is legal and in the sense of the DSGVO, that personal data is handled correctly. Even if the "​Creator"​ is not employed at the university any more. The "​Creator"​ is the responsible person in the sense of the Urheberrechtsgesetz,​ taking care that reusing of third party data is legal and in the sense of the DSGVO, that personal data is handled correctly. Even if the "​Creator"​ is not employed at the university any more.
  
- +There exists ​[[https://​doi.org/​10.5281/​zenodo.3368293|decision guide]] if data can be published, sadly only in german.
-If user leaves the university file ownership goes to the next hierarchical user.+
  
 ==== Licensing ==== ==== Licensing ====
Line 376: Line 516:
     * [[https://​www.gnu.org/​licenses/​|GPL,​ GPLv2, GPLv3]]     * [[https://​www.gnu.org/​licenses/​|GPL,​ GPLv2, GPLv3]]
     * [[https://​opensource.org/​licenses/​MIT|MIT]]     * [[https://​opensource.org/​licenses/​MIT|MIT]]
 +    * [[http://​www.linfo.org/​bsdlicense.html|BSD]]
   * Arts, Images, Text, etc.   * Arts, Images, Text, etc.
     * [[https://​creativecommons.org/​choose/​|Creative Commons (Text, Arts, Photos, ...)]]     * [[https://​creativecommons.org/​choose/​|Creative Commons (Text, Arts, Photos, ...)]]
Line 381: Line 522:
     * [[https://​opendatacommons.org/​licenses/​|Open Data Commons]]     * [[https://​opendatacommons.org/​licenses/​|Open Data Commons]]
  
-The applicability of CC-BY licenses for datasets is [[https://​ckan4rdm.wordpress.com/​2019/​06/​05/​creative-commons-lizenzen-sind-fur-forschungsdaten-ungeeignet/​|doubtful]]. ​Other licenses search at [[https://​licenses.opendefinition.org/​|Open Definition Licenses Service]]+ 
 +The applicability of CC-BY licenses ​for Software is not recommended:​ [[https://​creativecommons.org/​about/​program-areas/​software/​|CC-recommendation]] and [[https://​opensource.stackexchange.com/​questions/​1717/​why-is-cc-by-sa-discouraged-for-code|discussion]]. The same applies ​for datasets, their publication under a CC-license other than CC0 is [[https://​ckan4rdm.wordpress.com/​2019/​06/​05/​creative-commons-lizenzen-sind-fur-forschungsdaten-ungeeignet/​|doubtful]]. ​For other dataset ​licenses search at [[https://​licenses.opendefinition.org/​|Open Definition Licenses Service]]
 + 
  
 Proprietary file formats should be avoided, since you don't know if the software to open them still exists in a few years. Try to stick to open standards. Proprietary file formats should be avoided, since you don't know if the software to open them still exists in a few years. Try to stick to open standards.
data_management/irods.txt · Last modified: 2020/01/30 12:36 by jsteinka