2. Usage¶
2.1. General help¶
In order to get help for any cernopendata-client command, use the
--help option:
$ cernopendata-client --help
Usage: cernopendata-client [OPTIONS] COMMAND [ARGS]...
Options:
--help Show this message and exit.
Commands:
download-files Download data files belonging to a record.
get-file-locations Get a list of data file locations of a record.
get-metadata Get metadata content of a record.
list-directory List contents of a EOSPUBLIC Open Data directory.
verify-files Verify downloaded data file integrity.
version Return cernopendata-client version.
2.2. Selecting records¶
The data published on the CERN Open Data portal are organised in bibliographic
records. Each record is uniquely identified by a numerical record
ID, for example record 1. Moreover, some records are
minted with a Digital Object Identifier (DOI), for example
10.7483/OPENDATA.CMS.A342.9982. Each of these
identifiers can be used in various cernopendata-client commands to
select record one is interested at. For example:
$ cernopendata-client <command> --recid 1
$ cernopendata-client <command> --doi 10.7483/OPENDATA.CMS.A342.9982
Various available commands are shown below.
2.3. Getting metadata¶
In order to get metadata information about a record, please use the get-metadata command:
$ cernopendata-client get-metadata --recid 1
{
"$schema": "http://opendata.cern.ch/schema/records/record-v1.0.0.json",
"abstract": {
"description": "<p>BTau primary dataset in AOD format from RunB of 2010</p> <p>This dataset contains all runs from 2010 RunB. The list of validated runs, which must be applied to all analyses, can be found in</p>",
"links": [
{
"recid": "1000"
}
]
},
"accelerator": "CERN-LHC",
"collaboration": {
"name": "CMS collaboration",
"recid": "450"
},
...
This will output a JSON containing all the record metadata, such as title, authors, keywords, collision energy, etc. The JSON may also contain interesting physics information describing the dataset.
If you would like to extract parts of metadata, for example to extract only the dataset title, or only the Global Tag information for CMS datasets, you can use –output-value command-line option:
$ cernopendata-client get-metadata --recid 1 --output-value title
/BTau/Run2010B-Apr21ReReco-v1/AOD
$ cernopendata-client get-metadata --recid 1 --output-value system_details.global_tag
FT_R_42_V10A::All
If the output field produces a list of values, you may want to filter only certain field values of interest. For example, you may want to ask not to output all authors of a dataset record, but only the authors matching certain particular author name. You can use the –filter command-line option to achieve this:
$ cernopendata-client get-metadata --recid 329 --output-value authors.orcid
0000-0002-2634-4958
0000-0001-8363-9827
0000-0002-9764-9783
0000-0002-9266-1783
0000-0001-5610-1060
0000-0001-7613-8063
$ cernopendata-client get-metadata --recid 329 --output-value authors.orcid --filter name='Rousseau, David'
0000-0001-7613-8063
2.4. Listing available data files¶
In order to get a list of data files belonging to a record, please use the get-file-locations command:
HTTP protocol
$ cernopendata-client get-file-locations --recid 5500
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/BuildFile.xml
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/HiggsDemoAnalyzer.cc
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/List_indexfile.txt
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/M4Lnormdatall.cc
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/M4Lnormdatall_lvl3.cc
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/demoanalyzer_cfg_level3MC.py
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/demoanalyzer_cfg_level3data.py
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/demoanalyzer_cfg_level4MC.py
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/demoanalyzer_cfg_level4data.py
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/mass4l_combine.pdf
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/mass4l_combine.png
This command will output URIs for all the files associated with the record ID
5500, using the HTTP protocol. Note that you can specify --server
https://opendata.cern.ch if you would like to use the HTTPS protocol instead.
XRootD protocol
Note that you can use --protocol xrootd command-line option if you
would rather see the equivalent XRootD endpoints for the files:
$ cernopendata-client get-file-locations --recid 5500 --protocol xrootd
root://eospublic.cern.ch//eos/opendata/cms/software/HiggsExample20112012/BuildFile.xml
root://eospublic.cern.ch//eos/opendata/cms/software/HiggsExample20112012/HiggsDemoAnalyzer.cc
root://eospublic.cern.ch//eos/opendata/cms/software/HiggsExample20112012/List_indexfile.txt
root://eospublic.cern.ch//eos/opendata/cms/software/HiggsExample20112012/M4Lnormdatall.cc
root://eospublic.cern.ch//eos/opendata/cms/software/HiggsExample20112012/M4Lnormdatall_lvl3.cc
root://eospublic.cern.ch//eos/opendata/cms/software/HiggsExample20112012/demoanalyzer_cfg_level3MC.py
root://eospublic.cern.ch//eos/opendata/cms/software/HiggsExample20112012/demoanalyzer_cfg_level3data.py
root://eospublic.cern.ch//eos/opendata/cms/software/HiggsExample20112012/demoanalyzer_cfg_level4MC.py
root://eospublic.cern.ch//eos/opendata/cms/software/HiggsExample20112012/demoanalyzer_cfg_level4data.py
root://eospublic.cern.ch//eos/opendata/cms/software/HiggsExample20112012/mass4l_combine.pdf
root://eospublic.cern.ch//eos/opendata/cms/software/HiggsExample20112012/mass4l_combine.png
The data files can be downloaded via XRootD protocol using the xrdcp command.
File sizes and checksums
If you would like to know in advance the file sizes and checksums, you can use
--verbose option:
$ cernopendata-client get-file-locations --recid 5500 --verbose
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/BuildFile.xml 305 adler32:ff63668a
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/HiggsDemoAnalyzer.cc 83761 adler32:f205f068
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/List_indexfile.txt 1669 adler32:46a907fc
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/M4Lnormdatall.cc 14943 adler32:af301992
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/M4Lnormdatall_lvl3.cc 15805 adler32:9d9b2126
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/demoanalyzer_cfg_level3MC.py 3741 adler32:cc943381
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/demoanalyzer_cfg_level3data.py 3689 adler32:1d3e2a43
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/demoanalyzer_cfg_level4MC.py 3874 adler32:9cbd53a3
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/demoanalyzer_cfg_level4data.py 3821 adler32:177b49c0
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/mass4l_combine.pdf 18170 adler32:19c6a6a2
http://opendata.cern.ch/eos/opendata/cms/software/HiggsExample20112012/mass4l_combine.png 93152 adler32:62e0c299
2.5. Downloading data files¶
In order to download data files belonging to a record, please use the download-files command. The command can download files over HTTP, HTTPS or XRootD protocols and verify the file checksums.
HTTP protocol
By default the download-files command uses HTTP protocol:
$ cernopendata-client download-files --recid 5500
==> Downloading file 1 of 11
-> File: ./5500/BuildFile.xml
-> Progress: 0/0 KiB (100%)
==> Downloading file 2 of 11
-> File: ./5500/HiggsDemoAnalyzer.cc
-> Progress: 81/81 KiB (100%)
==> Downloading file 3 of 11
-> File: ./5500/List_indexfile.txt
-> Progress: 1/1 KiB (100%)
==> Downloading file 4 of 11
-> File: ./5500/M4Lnormdatall.cc
-> Progress: 14/14 KiB (100%)
==> Downloading file 5 of 11
-> File: ./5500/M4Lnormdatall_lvl3.cc
-> Progress: 15/15 KiB (100%)
==> Downloading file 6 of 11
-> File: ./5500/demoanalyzer_cfg_level3MC.py
-> Progress: 3/3 KiB (100%)
==> Downloading file 7 of 11
-> File: ./5500/demoanalyzer_cfg_level3data.py
-> Progress: 3/3 KiB (100%)
==> Downloading file 8 of 11
-> File: ./5500/demoanalyzer_cfg_level4MC.py
-> Progress: 3/3 KiB (100%)
==> Downloading file 9 of 11
-> File: ./5500/demoanalyzer_cfg_level4data.py
-> Progress: 3/3 KiB (100%)
==> Downloading file 10 of 11
-> File: ./5500/mass4l_combine.pdf
-> Progress: 17/17 KiB (100%)
==> Downloading file 11 of 11
-> File: ./5500/mass4l_combine.png
-> Progress: 90/90 KiB (100%)
==> Success!
The command will download files into a local directory called 5500 after
the record ID input parameter.
By default the download will be carried out over HTTP protocol. If you would
like to use the HTTPS protocol instead , please specify --server
https://opendata.cern.ch.
Note that you can also download files from another server, for example from our
Quality Assurance server, by using --server http://opendata-qa.cern.ch.
XRootD protocol
If you have installed client with XRootD support, you can use --protocol
xrootd command-line option to use that protocol instead of HTTP/HTTPS:
$ cernopendata-client download-files --recid 5500 --protocol xrootd
==> Downloading file 1 of 11
-> File: ./5500/BuildFile.xml
==> Downloading file 2 of 11
-> File: ./5500/HiggsDemoAnalyzer.cc
==> Downloading file 3 of 11
-> File: ./5500/List_indexfile.txt
==> Downloading file 4 of 11
-> File: ./5500/M4Lnormdatall.cc
==> Downloading file 5 of 11
-> File: ./5500/M4Lnormdatall_lvl3.cc
==> Downloading file 6 of 11
-> File: ./5500/demoanalyzer_cfg_level3MC.py
==> Downloading file 7 of 11
-> File: ./5500/demoanalyzer_cfg_level3data.py
==> Downloading file 8 of 11
-> File: ./5500/demoanalyzer_cfg_level4MC.py
==> Downloading file 9 of 11
-> File: ./5500/demoanalyzer_cfg_level4data.py
==> Downloading file 10 of 11
-> File: ./5500/mass4l_combine.pdf
==> Downloading file 11 of 11
-> File: ./5500/mass4l_combine.png
==> Success!
Select download engine
You can specify the download engine with --download-engine option.
requestsandpycurlare two supported download engines for HTTP protocol.xrootdis the only supported download engine for XRootD protocol.
$ cernopendata-client download-files --recid 5500 --filter-name BuildFile.xml --download-engine pycurl
==> Downloading file 1 of 1
-> File: ./5500/BuildFile.xml
-> Progress: 0/0 KiB (100%)
==> Success!
Filter by name
A dataset may consist of thousands of files. You can use powerful filtering options to download only certain files matching your criteria.
For example, you can download only files matching exactly a given file name using the --filter-name option:
$ cernopendata-client download-files --recid 5500 --filter-name BuildFile.xml
==> Downloading file 1 of 1
-> File: ./5500/BuildFile.xml
-> Progress: 0/0 KiB (100%)
==> Success!
$ cernopendata-client download-files --recid 5500 --filter-name BuildFile.xml,List_indexfile.txt
==> Downloading file 1 of 2
-> File: ./5500/BuildFile.xml
-> Progress: 0/0 KiB (100%)
==> Downloading file 2 of 2
-> File: ./5500/List_indexfile.txt
-> Progress: 1/1 KiB (100%)
==> Success!
Filter by regular expression
You can download all files matching a certain regular expression using the --filter-regexp option:
$ cernopendata-client download-files --recid 5500 --filter-regexp py$
==> Downloading file 1 of 4
-> File: ./5500/demoanalyzer_cfg_level3MC.py
-> Progress: 3/3 KiB (100%)
==> Downloading file 2 of 4
-> File: ./5500/demoanalyzer_cfg_level3data.py
-> Progress: 3/3 KiB (100%)
==> Downloading file 3 of 4
-> File: ./5500/demoanalyzer_cfg_level4MC.py
-> Progress: 3/3 KiB (100%)
==> Downloading file 4 of 4
-> File: ./5500/demoanalyzer_cfg_level4data.py
-> Progress: 3/3 KiB (100%)
==> Success!
Filter by range
You can also download files from a specified range (i-j) using the --filter-range option:
$ cernopendata-client download-files --recid 5500 --filter-range 1-4
==> Downloading file 1 of 4
-> File: ./5500/BuildFile.xml
-> Progress: 0/0 KiB (100%)
==> Downloading file 2 of 4
-> File: ./5500/HiggsDemoAnalyzer.cc
-> Progress: 81/81 KiB (100%)
==> Downloading file 3 of 4
-> File: ./5500/List_indexfile.txt
-> Progress: 1/1 KiB (100%)
==> Downloading file 4 of 4
-> File: ./5500/M4Lnormdatall.cc
-> Progress: 14/14 KiB (100%)
==> Success!
$ cernopendata-client download-files --recid 5500 --filter-range 1-2,5-7
==> Downloading file 1 of 5
-> File: ./5500/BuildFile.xml
==> Downloading file 2 of 5
-> File: ./5500/HiggsDemoAnalyzer.cc
==> Downloading file 3 of 5
-> File: ./5500/M4Lnormdatall_lvl3.cc
==> Downloading file 4 of 5
-> File: ./5500/demoanalyzer_cfg_level3MC.py
==> Downloading file 5 of 5
-> File: ./5500/demoanalyzer_cfg_level3data.py
==> Success!
Filter by combining multiple selectors
You can combine multiple filters in the same download command. Here are several examples:
$ cernopendata-client download-files --recid 5500 --filter-regexp py --filter-range 1-2
==> Downloading file 1 of 2
-> File: ./5500/demoanalyzer_cfg_level3MC.py
-> Progress: 3/3 KiB (100%)
==> Downloading file 2 of 2
-> File: ./5500/demoanalyzer_cfg_level3data.py
-> Progress: 3/3 KiB (100%)
==> Success!
$ cernopendata-client download-files --recid 5500 --filter-regexp py --filter-range 1-2,4-4
==> Downloading file 1 of 3
-> File: ./5500/demoanalyzer_cfg_level3MC.py
==> Downloading file 2 of 3
-> File: ./5500/demoanalyzer_cfg_level3data.py
==> Downloading file 3 of 3
-> File: ./5500/demoanalyzer_cfg_level4data.py
==> Success!
2.6. Verifying files¶
If you have downloaded the data files for a certain record, and you would like to verify their integrity and check whether there haven’t been some critical updates on the CERN Open Data portal side, you can use the verify-files command:
$ cernopendata-client verify-files --recid 5500
==> Verifying number of files for record 5500...
-> Expected 11, found 11
==> Verifying file BuildFile.xml...
-> Expected size 305, found 305
-> Expected checksum adler32:ff63668a, found adler32:ff63668a
==> Verifying file HiggsDemoAnalyzer.cc...
-> Expected size 83761, found 83761
-> Expected checksum adler32:f205f068, found adler32:f205f068
==> Verifying file List_indexfile.txt...
-> Expected size 1669, found 1669
-> Expected checksum adler32:46a907fc, found adler32:46a907fc
==> Verifying file M4Lnormdatall.cc...
-> Expected size 14943, found 14943
-> Expected checksum adler32:af301992, found adler32:af301992
==> Verifying file M4Lnormdatall_lvl3.cc...
-> Expected size 15805, found 15805
-> Expected checksum adler32:9d9b2126, found adler32:9d9b2126
==> Verifying file demoanalyzer_cfg_level3MC.py...
-> Expected size 3741, found 3741
-> Expected checksum adler32:cc943381, found adler32:cc943381
==> Verifying file demoanalyzer_cfg_level3data.py...
-> Expected size 3689, found 3689
-> Expected checksum adler32:1d3e2a43, found adler32:1d3e2a43
==> Verifying file demoanalyzer_cfg_level4MC.py...
-> Expected size 3874, found 3874
-> Expected checksum adler32:9cbd53a3, found adler32:9cbd53a3
==> Verifying file demoanalyzer_cfg_level4data.py...
-> Expected size 3821, found 3821
-> Expected checksum adler32:177b49c0, found adler32:177b49c0
==> Verifying file mass4l_combine.pdf...
-> Expected size 18170, found 18170
-> Expected checksum adler32:19c6a6a2, found adler32:19c6a6a2
==> Verifying file mass4l_combine.png...
-> Expected size 93152, found 93152
-> Expected checksum adler32:62e0c299, found adler32:62e0c299
==> Success!
Note that you can verify each file “just in time” as it is being downloaded as well:
$ cernopendata-client download-files --recid 5500 --filter-range 1-4 --verify
==> Downloading file 1 of 4
-> File: ./5500/BuildFile.xml
==> Verifying file BuildFile.xml...
-> Expected size 305, found 305
-> Expected checksum adler32:ff63668a, found adler32:ff63668a
==> Downloading file 2 of 4
-> File: ./5500/HiggsDemoAnalyzer.cc
==> Verifying file HiggsDemoAnalyzer.cc...
-> Expected size 83761, found 83761
-> Expected checksum adler32:f205f068, found adler32:f205f068
==> Downloading file 3 of 4
-> File: ./5500/List_indexfile.txt
==> Verifying file List_indexfile.txt...
-> Expected size 1669, found 1669
-> Expected checksum adler32:46a907fc, found adler32:46a907fc
==> Downloading file 4 of 4
-> File: ./5500/M4Lnormdatall.cc
==> Verifying file M4Lnormdatall.cc...
-> Expected size 14943, found 14943
-> Expected checksum adler32:af301992, found adler32:af301992
==> Success!
2.7. Listing directories¶
The CERN Open Data files are hosted on the EOSPUBLIC data storage service. In order to get a list of files belonging to a certain EOSPUBLIC directory, please use the list-directory command:
$ cernopendata-client list-directory /eos/opendata/cms/validated-runs/Commissioning10
Commissioning10-May19ReReco_7TeV.json
Commissioning10-May19ReReco_900GeV.json
The list-directory command uses XRootD protocol to list data files and hence it is available only when you install the XRootD flavour. Please see the Installation documentation for more details.
Iterate recursively
Note that you can use --recursive command-line option if you would like to
iterate also through all the subdirectories of the given directory:
$ cernopendata-client list-directory /eos/opendata/cms/validated-runs --recursive
Commissioning10-May19ReReco_7TeV.json
Commissioning10-May19ReReco_900GeV.json
Cert_190456-208686_8TeV_22Jan2013ReReco_Collisions12_JSON.txt
Cert_160404-180252_7TeV_ReRecoNov08_Collisions11_JSON.txt
Cert_136033-149442_7TeV_Apr21ReReco_Collisions10_JSON_v2.txt
Iterate recursively with timeout
If you would like to list a directory that contains a large amount of files,
you can specify --timeout option in order to exit after a certain amount of
time. The default timeout is 60 seconds.
$ cernopendata-client list-directory /eos/opendata/cms/Run2010B/BTau/AOD --recursive --timeout 30
CMS_Run2010B_BTau_AOD_Apr21ReReco-v1_0000_file_index.json
CMS_Run2010B_BTau_AOD_Apr21ReReco-v1_0000_file_index.txt
CMS_Run2010B_BTau_AOD_Apr21ReReco-v1_0001_file_index.json
CMS_Run2010B_BTau_AOD_Apr21ReReco-v1_0001_file_index.txt
CMS_Run2010B_BTau_AOD_Apr21ReReco-v1_0002_file_index.json
CMS_Run2010B_BTau_AOD_Apr21ReReco-v1_0002_file_index.txt
CMS_Run2010B_BTau_AOD_Apr21ReReco-v1_0003_file_index.json
CMS_Run2010B_BTau_AOD_Apr21ReReco-v1_0003_file_index.txt
CMS_Run2010B_BTau_AOD_Apr21ReReco-v1_0004_file_index.json
CMS_Run2010B_BTau_AOD_Apr21ReReco-v1_0004_file_index.txt
CMS_Run2010B_BTau_AOD_Apr21ReReco-v1_0005_file_index.json
CMS_Run2010B_BTau_AOD_Apr21ReReco-v1_0005_file_index.txt
..
2.8. More information¶
For more information about all the available cernopendata-client commands
and options, please see CLI API.
