User Tools

Site Tools


data_management:irods

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision
Previous revision
data_management:irods [2019/08/15 10:35]
meesters created
data_management:irods [2019/11/19 08:35] (current)
jsteinka [Publishing]
Line 5: Line 5:
   *  files are (//data objects//).   *  files are (//data objects//).
  
-All commands from the iRODS command line tools start with an '//​i//'​.+All commands from the iRODS command line tools start with an '//​i//' ​and they are installed on the mogon login nodes. 
 + 
 +iRODS uses so called '//​Resources//'​ to archive the //​collections//​ and //data objects//. The //​resources//​ are organized hierarchically. The root is a //​replication resource//, where other //​resources//​ are added as children. Currently there is a //compound resource// consisting itself of a cache (unix filesystem) and a universal mass storage system (here: TSM) as archive. The cache has a size of 8TB, once it is fills up, the oldest //data objects// will be deleted on the cache. If required, they will be fetched back from the archive. 
 + 
 +<​code>​ 
 +replResc:​replication 
 +├── cephfsResc:​unixfilesystem 
 +└── compResc:​compound 
 +    ├── netappResc:​unixfilesystem 
 +    └── tsmResc:​univmss 
 +</​code>​
  
 On request it is still possible to apply for a [[archiving:​tsm|TSM account]], to store the data on tapes. On request it is still possible to apply for a [[archiving:​tsm|TSM account]], to store the data on tapes.
Line 18: Line 28:
  
 If a directory, resp. //​collection//,​ consists of many small files, those files should be [[archiving:​preparation|compressed]]. iRODS works best for files > 5GB. If a directory, resp. //​collection//,​ consists of many small files, those files should be [[archiving:​preparation|compressed]]. iRODS works best for files > 5GB.
 +{{ :​data_management:​test-large-files.png?​direct&​400 | iRODS upload benchmark with one file of varying size.}} 
 +{{ :​data_management:​test-small-files.png?​direct&​400 | iRODS upload benchmark with multiple tiny files.}}
 ===== iRODS Account ===== ===== iRODS Account =====
  
Line 27: Line 38:
  
 Collections in the home folder of individual user will be deleted once the account gets deleted. Only the // /​zdv/​project/<​PROJECT NAME> // collections will be archived for an appropriate period (default: 10 years, as suggested by the [[https://​www.dfg.de/​foerderung/​grundlagen_rahmenbedingungen/​gwp/​|DFG,​ Leitlinie 17]]). Collections in the home folder of individual user will be deleted once the account gets deleted. Only the // /​zdv/​project/<​PROJECT NAME> // collections will be archived for an appropriate period (default: 10 years, as suggested by the [[https://​www.dfg.de/​foerderung/​grundlagen_rahmenbedingungen/​gwp/​|DFG,​ Leitlinie 17]]).
 +
 +The data saved to the above project collection is owned by the group. This means after a user leaves the project, the data can still be accessed by the other group members, as long as the ACLs are not modified.
 </​WRAP>​ </​WRAP>​
  
Line 51: Line 64:
 **Security warning** **Security warning**
  
-If you initiate the command ''​iinit''​ it might happen that an additional file '//​.irodsA//'​ is also in the folder '//​.irods//'​. Please remove this file! It contains your entered ​password in a decryptable form.+If you initiate the command ''​iinit''​ it might happen that an additional file '//​.irodsA//'​ is also in the folder '//​.irods//'​. Please remove this file! It contains your password in a decryptable form.
 </​WRAP>​ </​WRAP>​
  
Line 62: Line 75:
 As mentioned above, iRODS is a kind of a virtual filesystem. The following commands can be used to for navigation. As mentioned above, iRODS is a kind of a virtual filesystem. The following commands can be used to for navigation.
  
-^ Command ​   ^ Parameters ​ ^ Description ​                                               +^ Command ​   ^ Parameters ​ ^ Description ​                                                        ​
-| ''​ipwd'' ​  ​| ​            | print current iRODS working directory (colection) ​         +| ''​ipwd'' ​  ​| ​            | print current iRODS working directory (colection) ​                  ​
-| ''​ils'' ​   | -l, -L      | list iRODS collection (-l: with details; -L: more details) | +| ''​ils'' ​   | -l, -L, -A  ​| list iRODS collection (-l: with details; -L: more details; -A: ACL) | 
-| ''​icd'' ​   | <​target> ​   | change iRODS collection ​                                   +| ''​icd'' ​   | <​target> ​   | change iRODS collection ​                                            ​
-| ''​imkdir''​ | -p <​coll> ​  | create a new collection (directory; -p: with parents) ​     |+| ''​imkdir''​ | -p <​coll> ​  | create a new collection (directory; -p: with parents) ​              ​|
  
 Each user gets his/her personal home under /​zdv/​home/​${USER} and access to the associated Mogon I/II projects under /​zdv/​project/<​PROJECT NAME>​. ​ Each user gets his/her personal home under /​zdv/​home/​${USER} and access to the associated Mogon I/II projects under /​zdv/​project/<​PROJECT NAME>​. ​
Line 72: Line 85:
 Accessible folders: Accessible folders:
   * ''/​zdv/​home/​${USER}''​ private directory   * ''/​zdv/​home/​${USER}''​ private directory
-  * ''/​zdv/​trash/​${USER}''​ private trash bin 
   * ''/​zdv/​project/<​PROJECT NAME>''​ project directory   * ''/​zdv/​project/<​PROJECT NAME>''​ project directory
   * ''/​zdv/​home/​public''​ every registered user can read/​write/​delete   * ''/​zdv/​home/​public''​ every registered user can read/​write/​delete
 +  * ''/​zdv/​trash/​home/​${USER}''​ private trash bin
 +
 +
 +<WRAP center round info 80%>
 +Only the project directory are meant for permanent archiving. We are working on a solution to prevent archiving to the **volatile** iRODS homes.
 +</​WRAP>​
  
 ==== Archiving ==== ==== Archiving ====
Line 84: Line 102:
 | ''​ichksum''​ | -r <​obj%%|%%coll>​ | Compute and store checksums (-r: recursive) ​                   | | ''​ichksum''​ | -r <​obj%%|%%coll>​ | Compute and store checksums (-r: recursive) ​                   |
  
-The checksum is calculated server side and we highly recommend to switch it on immediately on upload. Nevertheless,​ You can also do it later on with ''​ichksum -K <​filename>''​. It creates a checksum equivalent to the command ''​sha256sum <local filename>​ | cut -d " " -f 1 | xxd -r -p | base64'',​ which you can compare to ensure data integrity. The checksums can be queried with ''​ils -L''​ and ''​ichksum''​.+The checksum is calculated server side and we highly recommend to switch it on immediately on upload. Nevertheless,​ You can also do it later on with ''​ichksum -K <​filename>''​. It creates a checksum equivalent to the command ''​sha256sum <local filename>​ | cut -d " " -f 1 | xxd -r -p | base64'',​ which you can compare to ensure data integrity. The checksums can be queried with ''​ils -L''​ and ''​ichksum''​. However, if you don't do it on upload with ''​iput -k'',​ there will be no checksum for the TSM ressource.
  
 As mentioned above, several small files should be bundled. Nevertheless,​ you can still extract an uploaded tar archive on the server and index all contained files with the command ''​ibun''​. Please read its man page for further details (''​man ibun''​). As mentioned above, several small files should be bundled. Nevertheless,​ you can still extract an uploaded tar archive on the server and index all contained files with the command ''​ibun''​. Please read its man page for further details (''​man ibun''​).
Line 107: Line 125:
 | -f        | overwrite local existing files | | -f        | overwrite local existing files |
  
 +=== Example ===
 +<code bash>
 +
 +[user@login01 ~]$ kinit
 +Password for user@UNI-MAINZ.DE: ​
 +
 +[user@login01 ~]$ ipwd
 +/​zdv/​home/​user
 +
 +[user@login01 ~]$ ils /​zdv/​home/​public
 +/​zdv/​home/​public:​
 +  hello_world.txt
 +[user@login01 ~]$ ils -l /​zdv/​home/​public
 +/​zdv/​home/​public:​
 +  rods              0 replResc;​compResc;​netappResc ​          24 2019-08-19.11:​00 & hello_world.txt
 +  rods              1 replResc;​compResc;​tsmResc ​          24 2019-08-19.11:​01 & hello_world.txt
 +[user@login01 ~]$ ils -L /​zdv/​home/​public
 +/​zdv/​home/​public:​
 +  rods              0 replResc;​compResc;​netappResc ​          24 2019-08-19.11:​00 & hello_world.txt
 +        generic ​   /​fsapp/​iRODS/​Vault/​home/​public/​hello_world.txt
 +  rods              1 replResc;​compResc;​tsmResc ​          24 2019-08-19.11:​01 & hello_world.txt
 +        generic ​   /​fsapp/​iRODS/​Vault/​home/​public/​hello_world.txt
 +[user@login01 ~]$ ils -A /​zdv/​home/​public
 +/​zdv/​home/​public:​
 +        ACL - g:​public#​zdv:​own ​  
 +        Inheritance - Disabled
 +  hello_world.txt
 +        ACL - public#​zdv:​read object ​  ​rods#​zdv:​own ​  
 +[user@login01 ~]$ ils -LA /​zdv/​home/​public
 +/​zdv/​home/​public:​
 +        ACL - g:​public#​zdv:​own ​  
 +        Inheritance - Disabled
 +  rods              0 replResc;​compResc;​netappResc ​          24 2019-08-19.11:​00 & hello_world.txt
 +        generic ​   /​fsapp/​iRODS/​Vault/​home/​public/​hello_world.txt
 +        ACL - public#​zdv:​read object ​  ​rods#​zdv:​own ​  
 +  rods              1 replResc;​compResc;​tsmResc ​          24 2019-08-19.11:​01 & hello_world.txt
 +        generic ​   /​fsapp/​iRODS/​Vault/​home/​public/​hello_world.txt
 +        ACL - public#​zdv:​read object ​  ​rods#​zdv:​own ​  
 +
 +[user@login01 ~]$ iget /​zdv/​home/​public/​hello_world.txt
 +[user@login01 ~]$ ls -l hello_world.txt ​
 +-rw-r----- 1 user zdv 24 Aug 19 11:18 hello_world.txt
 +
 +[user@login01 ~]$ iput hello_world.txt ​
 +[user@login01 ~]$ ils -L
 +/​zdv/​home/​user:​
 +  user          0 replResc;​compResc;​netappResc ​          24 2019-08-19.11:​20 & hello_world.txt
 +        generic ​   /​fsapp/​iRODS/​Vault/​home/​user/​hello_world.txt
 +  user          1 replResc;​compResc;​tsmResc ​          24 2019-08-19.11:​20 & hello_world.txt
 +        generic ​   /​fsapp/​iRODS/​Vault/​home/​user/​hello_world.txt
 +
 +ichksum hello_world.txt
 +    hello_world.txt ​   sha2:​XPdR4XQP49lWUGEfPJz0Jo+kmkndGxz6rCQUzCqHteA=
 +Total checksum performed = 1, Failed checksum = 0
 +[user@login01 ~]$ ils -L
 +/​zdv/​home/​user:​
 +  user          0 replResc;​compResc;​netappResc ​          24 2019-08-19.11:​20 & hello_world.txt
 +    sha2:​XPdR4XQP49lWUGEfPJz0Jo+kmkndGxz6rCQUzCqHteA= ​   generic ​   /​fsapp/​iRODS/​Vault/​home/​user/​hello_world.txt
 +  user          1 replResc;​compResc;​tsmResc ​          24 2019-08-19.11:​20 & hello_world.txt
 +        generic ​   /​fsapp/​iRODS/​Vault/​home/​user/​hello_world.txt
 +
 +[user@login01 ~]$  sha256sum hello_world.txt | cut -d " " -f 1 | xxd -r -p | base64
 +XPdR4XQP49lWUGEfPJz0Jo+kmkndGxz6rCQUzCqHteA=
 +
 +[user@login01 ~]$ irm -f hello_world.txt ​
 +[user@login01 ~]$ iput -k hello_world.txt ​
 +[user@login01 ~]$ ils -L
 +/​zdv/​home/​user:​
 +  user          0 replResc;​compResc;​netappResc ​          24 2019-08-19.11:​27 & hello_world.txt
 +    sha2:​XPdR4XQP49lWUGEfPJz0Jo+kmkndGxz6rCQUzCqHteA= ​   generic ​   /​fsapp/​iRODS/​Vault/​home/​user/​hello_world.txt
 +  user          1 replResc;​compResc;​tsmResc ​          24 2019-08-19.11:​28 & hello_world.txt
 +    sha2:​XPdR4XQP49lWUGEfPJz0Jo+kmkndGxz6rCQUzCqHteA= ​   generic ​   /​fsapp/​iRODS/​Vault/​home/​user/​hello_world.txt
 +</​code>​
 ==== Metadata: ''​imeta''​ ==== ==== Metadata: ''​imeta''​ ====
  
Line 129: Line 220:
  
  
-The following ​two commands upload the filesystem table and query the metadata:+The following ​command lists the metadata ​automatically associated with the previously upladed file ''​hello_world.txt''​:
 <code bash> <code bash>
-iput <​path/​filename>​ +imeta ls -d hello_world.txt
-imeta ls -d <​filename>​+
 </​code>​ </​code>​
  
Line 138: Line 228:
  
 <​code>​ <​code>​
-AVUs defined for dataObj ​<​filename>​:+AVUs defined for dataObj ​hello_world.txt:
 attribute: AccessRights attribute: AccessRights
 value: closed value: closed
Line 144: Line 234:
 ---- ----
 attribute: Creator attribute: Creator
-value: ​<you>+value: ​Steinkamp, J.
 units: ​ units: ​
 ---- ----
 attribute: Date attribute: Date
-value: ​1562677318+value: ​1566206896
 units: ​ units: ​
 ---- ----
 attribute: ExpiryDate attribute: ExpiryDate
-value: ​1878901318+value: ​1882430896
 units: ​ units: ​
 ---- ----
Line 166: Line 256:
 value: Johannes Gutenberg-University value: Johannes Gutenberg-University
 units: ​ units: ​
 +[user@login01 ~]$ 
 </​code>​ </​code>​
  
 +You can now add a title, which is not created automatically:​
 <​code>​ <​code>​
-imeta set -d <​filename> ​Title "​Archive of experimental szstem from '​$(date)'"​+imeta set -d hello_world.txt ​Title "​Archive of experimental szstem from '​$(date)'"​
 </​code>​ </​code>​
  
-If you query the Attribute '​Table'​ with ''​imeta ls -d <​filename> ​Title''​ you get:+If you query the Attribute '​Table'​ with ''​imeta ls -d hello_world.txt ​Title''​ you get:
 <​code>​ <​code>​
-AVUs defined for dataObj ​fstab:+AVUs defined for dataObj ​hello_world.txt:
 attribute: Title attribute: Title
-value: Archive of experimental szstem from '<​somedate>​'+value: Archive of experimental szstem from 'Mon Aug 19 11:39:48 CEST 2019'
 units: ​ units: ​
 </​code>​ </​code>​
Line 191: Line 283:
  
 <code shell> <code shell>
-imeta mod -d <​filename> ​Title "​Archive of experimental szstem from '<​somedate>​'"​ v:"​Archive of experimental system from '<​somedate>​'"​+imeta mod -d hello_world.txt ​Title "​Archive of experimental szstem from 'Mon Aug 19 11:39:48 CEST 2019'"​ v:"​Archive of experimental system from 'Mon Aug 19 11:39:48 CEST 2019'"​
 </​code>​ </​code>​
  
Line 246: Line 338:
 ==== Searching ==== ==== Searching ====
  
 +=== for filenames: ''​ilocate''​ ===
  
-  * ''​ilocate -t "​%fstab%"''​ search for filename (pattern) +<code bash>
-  * ''​imeta qu -C Name like "​%fstab%"''​ search for metadata+
  
-==== Publishing ====+[user@login01 ~]$ ilocate -t "​hello_world.txt"​  
 +/​zdv/​home/​user/​hello_world.txt 
 +/​zdv/​home/​public/​hello_world.txt 
 +</​code>​
  
-For public access a **ticket** needs to be created ​for collections or data objectsFor example, if you use the above uploaded ​''​fstab'' ​file (here for my personal home directorywhich might not work in the future any more, see above why).+=== for metadata: ''​imeta qu''​ === 
 + 
 +You must know, if you want to search ​for data object (''​-d''​) or a collection (''​-C''​)And you can use SQL wildcards (''​%''​)if you don't know the exact pattern you are looking for. The wildcard pattern matching is also applicable for ''​ilocate''​.
  
 <code bash> <code bash>
-iticket create read fstab+[user@login01 ~]$ imeta qu -d Title like "​Archive%"​ 
 +collection: /​zdv/​home/​user 
 +dataObj: hello_world.txt
 </​code>​ </​code>​
-returns:+ 
 +=== via database query''​iquest''​ === 
 + 
 +For this complex syntax consult the [[https://​docs.irods.org/​4.2.6/​icommands/​user/#​iquest|Online help]]. 
 + 
 +==== Publishing ==== 
 + 
 +For public access a **ticket** needs to be created for collections or data objects. For example, if you use the above uploaded ''​hello_world.txt''​ file again. 
 <code bash> <code bash>
-ticket:GGkUTXdJpfK7VPi+[user@login01 ~]$ iticket create read hello_world.txt 
 +ticket:ACR2RKDyuZMBRmb
 </​code>​ </​code>​
  
-Querying ​the metadata via the provided REST-API ​returns a JSON stringThis can be viewed in the browser or as done here using curl:+With this ticket and the path everybody can query information and the content of collections and data objects vi a provided REST-APIJSON strings are returned for valid URLs. 
 + 
 +=== General information about data objects ===
  
 <code bash> <code bash>
-curl https://​irods-test.zdv.uni-mainz.de/​irods-rest/​rest/​dataObject/​zdv/​home/​jsteinka/​fstab/​metadata?ticket=GGkUTXdJpfK7VPi+[user@login01 ~]$ curl https://​irods-web.zdv.uni-mainz.de/​irods-rest/​rest/​dataObject/​zdv/​home/​jsteinka/​hello_world.txt?ticket=ACR2RKDyuZMBRmb
 </​code>​ </​code>​
-returns: 
 <code JavaScript>​ <code JavaScript>​
-{"metadataEntries": ​+{"id":1808764
-  {"​count":​1+ "​collectionId":24346
-   ​"lastResult":true+ "​dataName":"​hello_world.txt", 
-   ​"totalRecords":0, + "​collectionName":"​/​zdv/​home/​jsteinka", 
-   "attribute":"​protected", + "​dataReplicationNumber":0, 
-   ​"value":"​false", + "​dataVersion":0, 
-   ​"unit":""​},​  + "​dataTypeName":"​generic", 
-  {"​count":​2,​ + "​dataSize":24
-   "​lastResult":​true,​ + "​resourceGroupName":"",​ 
-   "​totalRecords":0, + "​resourceName":"​netappResc", 
-   "​attribute":"​AccessRights",​ + "​dataPath":"​/​fsapp/​iRODS/​Vault/​home/​jsteinka/​hello_world.txt", 
-   "​value":"​closed",​ + "​dataOwnerName":"​jsteinka", 
-   "​unit":""​},​ + "​dataOwnerZone":"​zdv", 
-  {"​count":​3,​ + "​replicationStatus":"​1", 
-   "​lastResult":​true,​ + "​dataStatus":"",​ 
-   "totalRecords":0, + "​checksum":"​sha2:XPdR4XQP49lWUGEfPJz0Jo+kmkndGxz6rCQUzCqHteA=", 
-   ​"attribute":"​Publisher", + "​expiry":"​00000000000", 
-   ​"value":"​Johannes Gutenberg-University"​+ "​dataMapId":0, 
-   ​"unit":""​}+ "​comments":"",​ 
-  {"count":4, + "​createdAt":1566206868000
-   "lastResult":true+ "​updatedAt":1566206868000
-   ​"totalRecords":0, + "​specColType":"​NORMAL", 
-   "attribute":"​Location", + "​objectPath":""​ 
-   ​"value":"​Mainz, Germany", +}
-   ​"unit":""​}+
-  {"count":5, +
-   "lastResult":true+
-   ​"totalRecords":0, +
-   "attribute":"​Creator", +
-   ​"value":"​Steinkamp, Jörg",​ +
-   "​unit"​:""}+
-  {"count":6, +
-   "lastResult":true+
-   ​"totalRecords":0, +
-   ​"attribute":"​Title", +
-   ​"value":"​Filesystem Table of '​login01.mogon'"​+
-   ​"unit":""​}+
-  {"count":7, +
-   "lastResult":true+
-   ​"totalRecords":0, +
-   "attribute":"​Date",​ +
-   "​value":"​1562679909",​ +
-   "​unit":""​},​ +
-  {"​count":​8,​ +
-   "​lastResult":​true,​ +
-   "​totalRecords":​0,​ +
-   "​attribute":"​ExpiryDate",​ +
-   "​value":"​1878903909",​ +
-   "​unit":""​}],​ +
-"​objectType":"​DATA_OBJECT",​ +
-"​uniqueNameString":"/​zdv/​home/​jsteinka/​fstab"​}+
 </​code>​ </​code>​
  
 +=== Querying the metadata ===
  
-  * Retrieve the metadata of a collection: 
 <code bash> <code bash>
-curl https://​irods-test.zdv.uni-mainz.de/​irods-rest/​rest/​collection/​zdv/​home/​jsteinka/​test/​metadata?​ticket=AKR9iYWmfSU6niN +[user@login01 ~]$ curl https://​irods-web.zdv.uni-mainz.de/​irods-rest/​rest/​dataObject/​zdv/​home/​jsteinka/​hello_world.txt/​metadata?​ticket=ACR2RKDyuZMBRmb
-{"​metadataEntries":​[ +
-        {"​count":​1,​ +
-         "​lastResult":​true,​ +
-         "​totalRecords":​0,​ +
-         "​attribute":"​Name",​ +
-         "​value":"​This is a test project","​unit":""​}],​ +
-    "​objectType":"​COLLECTION",​ +
-    "​uniqueNameString":"​zdv/​home/​jsteinka/​test"​ +
-}+
 </​code>​ </​code>​
-  * Retrieve the metadata ​of a data object:+<code JavaScript>​ 
 +{"​metadataEntries":​ [ 
 +    {"​count":​1,​ 
 +     "​lastResult":​true,​ 
 +     "​totalRecords":​0,​ 
 +     "​attribute":"​AccessRights",​ 
 +     "​value":"​closed",​ 
 +     "​unit":""​},​ 
 +    {"​count":​2,​ 
 +     "​lastResult":​true,​ 
 +     "​totalRecords":​0,​ 
 +     "​attribute":"​Publisher",​ 
 +     "​value":"​Johannes Gutenberg-University",​ 
 +     "​unit":""​},​ 
 +    {"​count":​3,​ 
 +     "​lastResult":​true,​ 
 +     "​totalRecords":​0,​ 
 +     "​attribute":"​Location",​ 
 +     "​value":"​Mainz,​ Germany",​ 
 +     "​unit":""​},​ 
 +    {"​count":​4,​ 
 +     "​lastResult":​true,​ 
 +     "​totalRecords":​0,​ 
 +     "​attribute":"​protected",​ 
 +     "​value":"​false",​ 
 +     "​unit":""​},​ 
 +    {"​count":​5,​ 
 +     "​lastResult":​true,​ 
 +     "​totalRecords":​0,​ 
 +     "​attribute":"​Creator",​ 
 +     "​value":"​Steinkamp,​ J.", 
 +     "​unit":""​},​ 
 +    {"​count":​6,​ 
 +     "​lastResult":​true,​ 
 +     "​totalRecords":​0,​ 
 +     "​attribute":"​Date",​ 
 +     "​value":"​1566206896",​ 
 +     "​unit":""​},​ 
 +    {"​count":​7,​ 
 +     "​lastResult":​true,"​ 
 +     ​totalRecords":​0,​ 
 +     "​attribute":"​ExpiryDate",​ 
 +     "​value":"​1882430896",​ 
 +     "​unit":""​},​ 
 +    {"​count":​8,​ 
 +     "​lastResult":​true,​ 
 +     "​totalRecords":​0,​ 
 +     "​attribute":"​Title",​ 
 +     "​value":"​Archive ​of experimental system from 'Mon Aug 19 11:39:48 CEST 2019'",​ 
 +     "​unit":""​}],​ 
 +"​objectType":"​DATA_OBJECT","​uniqueNameString":"/​zdv/​home/​jsteinka/​hello_world.txt"​} 
 +</​code>​ 
 + 
 +=== Downloading data === 
 +The file content can be viewed with ''​curl''​ or downloaded with ''​wget''​.
 <code bash> <code bash>
-curl https://​irods-test.zdv.uni-mainz.de/​irods-rest/​rest/​dataObject/​zdv/​home/​jsteinka/​test/​garbage_small.zero/​metadata?ticket=qM3arKFlkBM0Ue5 +curl https://​irods-web.zdv.uni-mainz.de/​irods-rest/​rest/​fileContents/​zdv/​home/​jsteinka/​hello_world.txt?ticket=ACR2RKDyuZMBRmb
-{"​metadataEntries":​[ +
-        {"​count":​1,​ +
-         "​lastResult":​true,​ +
-         "​totalRecords":​0,​ +
-         "​attribute":"​Creator",​ +
-         "​value":"​Joerg Steinkamp",​ +
-         "​unit":""​}, ​  +
-        {"​count":​2,​ +
-         "​lastResult":​true,​ +
-         "​totalRecords":​0,​ +
-         "​attribute":"​Description",​ +
-         "​value":"​only zeroes from dd if=/​dev/​zero of=garbage_small-zero bs=1024 count=24",​ +
-         "​unit":""​}],​ +
-    "​objectType":"​DATA_OBJECT",​ +
-    "​uniqueNameString":"/​zdv/​home/​jsteinka/​test/​garbage_small.zero"​ +
-}+
 </​code>​ </​code>​
-  * Download ​the data object:+ 
 +=== Retrieve ​the metadata of a collection ===
 <code bash> <code bash>
-wget  ​https://​irods-test.zdv.uni-mainz.de/​irods-rest/​rest/​fileContents/zdv/home/jsteinka/test/​garbage_small.zero?ticket=qM3arKFlkBM0Ue5+curl https://​irods-web.zdv.uni-mainz.de/​irods-rest/​rest/​collection/zdv/home/public/helloCollection?ticket=mbyAAFGm7vhUdyM
 </​code>​ </​code>​
 +<code JavaScript>​
 +{
 + "​collectionId":​1808781,​
 + "​collectionName":"/​zdv/​home/​public/​helloCollection",​
 + "​objectPath":"",​
 + "​collectionParentName":"/​zdv/​home/​public/",​
 + "​collectionOwnerName":"​rods",​
 + "​collectionOwnerZone":"​zdv",​
 + "​collectionMapId":"​0",​
 + "​collectionInheritance":"",​
 + "​comments":"",​
 + "​info1":"",​
 + "​info2":"",​
 + "​createdAt":​1566213731000,​
 + "​modifiedAt":​1566213731000,​
 + "​specColType":"​NORMAL",​
 + "​children":​[]
 +}
 +</​code>​
 +
 +=== REST-API URL ===
 +
 +the URL for the REST-API consists of:
 +  * Server (https://​irods-web.zdv.uni-mainz.de/​irods-rest/​rest)
 +  * what (collection|dataObject|fileContents)
 +  * iRODS path
 +  * optionally '​metadata'​
 +  * Ticket string (?​ticket=1234567890)
  
 +For further information,​ please read the original [[https://​github.com/​DICE-UNC/​irods-rest|IRODS-REST documentation]]
 ===== Data Policy/​Recommendation ===== ===== Data Policy/​Recommendation =====
  
 The "​Creator"​ is the responsible person in the sense of the Urheberrechtsgesetz,​ taking care that reusing of third party data is legal and in the sense of the DSGVO, that personal data is handled correctly. Even if the "​Creator"​ is not employed at the university any more. The "​Creator"​ is the responsible person in the sense of the Urheberrechtsgesetz,​ taking care that reusing of third party data is legal and in the sense of the DSGVO, that personal data is handled correctly. Even if the "​Creator"​ is not employed at the university any more.
  
- +There exists ​[[https://​doi.org/​10.5281/​zenodo.3368293|decision guide]] if data can be published, sadly only in german.
-If user leaves the university file ownership goes to the next hierarchical user.+
  
 ==== Licensing ==== ==== Licensing ====
Line 376: Line 516:
     * [[https://​www.gnu.org/​licenses/​|GPL,​ GPLv2, GPLv3]]     * [[https://​www.gnu.org/​licenses/​|GPL,​ GPLv2, GPLv3]]
     * [[https://​opensource.org/​licenses/​MIT|MIT]]     * [[https://​opensource.org/​licenses/​MIT|MIT]]
 +    * [[http://​www.linfo.org/​bsdlicense.html|BSD]]
   * Arts, Images, Text, etc.   * Arts, Images, Text, etc.
     * [[https://​creativecommons.org/​choose/​|Creative Commons (Text, Arts, Photos, ...)]]     * [[https://​creativecommons.org/​choose/​|Creative Commons (Text, Arts, Photos, ...)]]
Line 381: Line 522:
     * [[https://​opendatacommons.org/​licenses/​|Open Data Commons]]     * [[https://​opendatacommons.org/​licenses/​|Open Data Commons]]
  
-The applicability of CC-BY licenses for datasets is [[https://​ckan4rdm.wordpress.com/​2019/​06/​05/​creative-commons-lizenzen-sind-fur-forschungsdaten-ungeeignet/​|doubtful]]. ​Other licenses search at [[https://​licenses.opendefinition.org/​|Open Definition Licenses Service]]+ 
 +The applicability of CC-BY licenses ​for Software is not recommended:​ [[https://​creativecommons.org/​about/​program-areas/​software/​|CC-recommendation]] and [[https://​opensource.stackexchange.com/​questions/​1717/​why-is-cc-by-sa-discouraged-for-code|discussion]]. The same applies ​for datasets, their publication under a CC-license other than CC0 is [[https://​ckan4rdm.wordpress.com/​2019/​06/​05/​creative-commons-lizenzen-sind-fur-forschungsdaten-ungeeignet/​|doubtful]]. ​For other dataset ​licenses search at [[https://​licenses.opendefinition.org/​|Open Definition Licenses Service]]
 + 
  
 Proprietary file formats should be avoided, since you don't know if the software to open them still exists in a few years. Try to stick to open standards. Proprietary file formats should be avoided, since you don't know if the software to open them still exists in a few years. Try to stick to open standards.
data_management/irods.1565858101.txt.gz · Last modified: 2019/08/15 10:35 by meesters