CCI Tagger

Overview

This package provides a command line tool moles_esgf_tag to generate dataset tags for both MOLES and ESGF.

Installation

Create a Python virtual environment:

virtualenv tagger
source tagger/bin/activate

Find the latest release of the code and install it in the virtual environment, i.e. for version 1.0.4:

pip install https://github.com/cedadev/cci-tagger/archive/v1.0.4.tar.gz

Usage

moles_esgf_tag [-h] (-d DATASET | -f FILE | -s) [-m] [-u] [--file_count FILE_COUNT] [-t DEFAULT_TERMS_FILE]
               [--no_check_sum] [-v]

You can tag an individual dataset, or tag all the datasets listed in a file. By default a check sum will be produces for each file.

Arguments:

-h, --help            show help message and exit

-d DATASET, --dataset DATASET
                      the full path to the dataset that is to be tagged. This option is used to tag a single
                      dataset.

-f FILE, --file FILE  the name of the file containing a list of datasets to process. This option is used for
                      tagging one or more datasets.

-s, --show_mappings   show the local vocabulary mappings

-m, --use_mappings    use the local vocabulary mappings. This will map a number of non compliant terms to
                      allowed terms.

-u, --update_moles    update the MOLEs catalogue directly rather than produce a csv file.

-t DEFAULT_TERMS_FILE, --default_terms_file DEFAULT_TERMS_FILE
                      the name of the file containing a list of default vocabulary terms to associate with
                      a dataset. Property values for 'institution', 'platform', 'sensor' and 
                      'time_coverage_resolution' may be comma separated lists.

--file_count FILE_COUNT
                      how many .nt files to look at per dataset

--no_check_sum        do not produce a check sum for each file

-v, --verbose         increase output verbosity

Output

A number of files are produced as output:

• esgf_drs.json contains a list of DRS and associated files and check sums
• moles_tags.csv contains a list of dataset paths and vocabulary URLs
• moles_esgf_mapping.csv contains mappings between dataset paths and DRS
• error.txt contains a list of errors

Examples

moles_esgf_tag -d /neodc/esacci/cloud/data/L3C/avhrr_noaa-16 -v
moles_esgf_tag -f datapath --file_count 2 --no_check_sum -m -v
moles_esgf_tag -s

DEFAULT_TERMS_FILE

This file should have the format of:

<property name>=<vocabulary term>

When property name is 'institution', 'platform', 'sensor' or 'time_coverage_resolution' then vocabulary term may be a comma separated list

For example:

ecv=soil moisture
processing_level=Level 4
sensor=modis,airs