User Tools

Site Tools


data_management:irods

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
data_management:irods [2019/08/22 13:08]
jsteinka [Searching]
data_management:irods [2020/01/23 22:02] (current)
meesters
Line 1: Line 1:
 +<WRAP center round important 60%>
 +Please note, that this service is still in its beta-phase. Do not hesitate to [[training_and_outreach:​ticket_system|contact us]] in order to discuss your data management plan.
 +</​WRAP>​
 +
 ====== Archiving with iRODS ====== ====== Archiving with iRODS ======
  
Line 11: Line 15:
 <​code>​ <​code>​
 replResc:​replication replResc:​replication
 +├── cephfsResc:​unixfilesystem
 └── compResc:​compound └── compResc:​compound
     ├── netappResc:​unixfilesystem     ├── netappResc:​unixfilesystem
Line 27: Line 32:
  
 If a directory, resp. //​collection//,​ consists of many small files, those files should be [[archiving:​preparation|compressed]]. iRODS works best for files > 5GB. If a directory, resp. //​collection//,​ consists of many small files, those files should be [[archiving:​preparation|compressed]]. iRODS works best for files > 5GB.
 +{{ :​data_management:​test-large-files.png?​direct&​400 | iRODS upload benchmark with one file of varying size.}} 
 +{{ :​data_management:​test-small-files.png?​direct&​400 | iRODS upload benchmark with multiple tiny files.}}
 ===== iRODS Account ===== ===== iRODS Account =====
  
Line 86: Line 92:
   * ''/​zdv/​home/​public''​ every registered user can read/​write/​delete   * ''/​zdv/​home/​public''​ every registered user can read/​write/​delete
   * ''/​zdv/​trash/​home/​${USER}''​ private trash bin   * ''/​zdv/​trash/​home/​${USER}''​ private trash bin
 +
 +
 +<WRAP center round info 80%>
 +Only the project directory are meant for permanent archiving. We are working on a solution to prevent archiving to the **volatile** iRODS homes.
 +</​WRAP>​
  
 ==== Archiving ==== ==== Archiving ====
Line 368: Line 379:
  
 <code bash> <code bash>
-[user@login01 ~]$ curl https://​irods-test.zdv.uni-mainz.de/​irods-rest/​rest/​dataObject/​zdv/​home/​jsteinka/​hello_world.txt?​ticket=ACR2RKDyuZMBRmb+[user@login01 ~]$ curl https://​irods-web.zdv.uni-mainz.de/​irods-rest/​rest/​dataObject/​zdv/​home/​jsteinka/​hello_world.txt?​ticket=ACR2RKDyuZMBRmb
 </​code>​ </​code>​
 <code JavaScript>​ <code JavaScript>​
Line 400: Line 411:
  
 <code bash> <code bash>
-[user@login01 ~]$ curl https://​irods-test.zdv.uni-mainz.de/​irods-rest/​rest/​dataObject/​zdv/​home/​jsteinka/​hello_world.txt/​metadata?​ticket=ACR2RKDyuZMBRmb+[user@login01 ~]$ curl https://​irods-web.zdv.uni-mainz.de/​irods-rest/​rest/​dataObject/​zdv/​home/​jsteinka/​hello_world.txt/​metadata?​ticket=ACR2RKDyuZMBRmb
 </​code>​ </​code>​
 <code JavaScript>​ <code JavaScript>​
Line 458: Line 469:
 The file content can be viewed with ''​curl''​ or downloaded with ''​wget''​. The file content can be viewed with ''​curl''​ or downloaded with ''​wget''​.
 <code bash> <code bash>
-curl https://​irods-test-01.zdv.uni-mainz.de/​irods-rest/​rest/​fileContents/​zdv/​home/​jsteinka/​hello_world.txt?​ticket=ACR2RKDyuZMBRmb+curl https://​irods-web.zdv.uni-mainz.de/​irods-rest/​rest/​fileContents/​zdv/​home/​jsteinka/​hello_world.txt?​ticket=ACR2RKDyuZMBRmb
 </​code>​ </​code>​
  
 === Retrieve the metadata of a collection === === Retrieve the metadata of a collection ===
 <code bash> <code bash>
-curl https://​irods-test.zdv.uni-mainz.de/​irods-rest/​rest/​collection/​zdv/​home/​public/​helloCollection?​ticket=mbyAAFGm7vhUdyM+curl https://​irods-web.zdv.uni-mainz.de/​irods-rest/​rest/​collection/​zdv/​home/​public/​helloCollection?​ticket=mbyAAFGm7vhUdyM
 </​code>​ </​code>​
 <code JavaScript>​ <code JavaScript>​
Line 488: Line 499:
  
 the URL for the REST-API consists of: the URL for the REST-API consists of:
-  * Server (https://​irods-test.uni-mainz.de/​irods-rest/​rest)+  * Server (https://​irods-web.zdv.uni-mainz.de/​irods-rest/​rest)
   * what (collection|dataObject|fileContents)   * what (collection|dataObject|fileContents)
   * iRODS path   * iRODS path
Line 509: Line 520:
     * [[https://​www.gnu.org/​licenses/​|GPL,​ GPLv2, GPLv3]]     * [[https://​www.gnu.org/​licenses/​|GPL,​ GPLv2, GPLv3]]
     * [[https://​opensource.org/​licenses/​MIT|MIT]]     * [[https://​opensource.org/​licenses/​MIT|MIT]]
 +    * [[http://​www.linfo.org/​bsdlicense.html|BSD]]
   * Arts, Images, Text, etc.   * Arts, Images, Text, etc.
     * [[https://​creativecommons.org/​choose/​|Creative Commons (Text, Arts, Photos, ...)]]     * [[https://​creativecommons.org/​choose/​|Creative Commons (Text, Arts, Photos, ...)]]
Line 514: Line 526:
     * [[https://​opendatacommons.org/​licenses/​|Open Data Commons]]     * [[https://​opendatacommons.org/​licenses/​|Open Data Commons]]
  
-The applicability of CC-BY licenses for datasets is [[https://​ckan4rdm.wordpress.com/​2019/​06/​05/​creative-commons-lizenzen-sind-fur-forschungsdaten-ungeeignet/​|doubtful]]. ​Other licenses search at [[https://​licenses.opendefinition.org/​|Open Definition Licenses Service]]+ 
 +The applicability of CC-BY licenses ​for Software is not recommended:​ [[https://​creativecommons.org/​about/​program-areas/​software/​|CC-recommendation]] and [[https://​opensource.stackexchange.com/​questions/​1717/​why-is-cc-by-sa-discouraged-for-code|discussion]]. The same applies ​for datasets, their publication under a CC-license other than CC0 is [[https://​ckan4rdm.wordpress.com/​2019/​06/​05/​creative-commons-lizenzen-sind-fur-forschungsdaten-ungeeignet/​|doubtful]]. ​For other dataset ​licenses search at [[https://​licenses.opendefinition.org/​|Open Definition Licenses Service]]
 + 
  
 Proprietary file formats should be avoided, since you don't know if the software to open them still exists in a few years. Try to stick to open standards. Proprietary file formats should be avoided, since you don't know if the software to open them still exists in a few years. Try to stick to open standards.
data_management/irods.1566472103.txt.gz · Last modified: 2019/08/22 13:08 by jsteinka