2023-05-30 12:49:08,211 UTC [357476] INFO esmvalcore._main:495 ______________________________________________________________________ _____ ____ __ ____ __ _ _____ _ | ____/ ___|| \/ \ \ / /_ _| |_ _|__ ___ | | | _| \___ \| |\/| |\ \ / / _` | | | |/ _ \ / _ \| | | |___ ___) | | | | \ V / (_| | | | | (_) | (_) | | |_____|____/|_| |_| \_/ \__,_|_| |_|\___/ \___/|_| ______________________________________________________________________ ESMValTool - Earth System Model Evaluation Tool. http://www.esmvaltool.org CORE DEVELOPMENT TEAM AND CONTACTS: Birgit Hassler (Co-PI; DLR, Germany - birgit.hassler@dlr.de) Alistair Sellar (Co-PI; Met Office, UK - alistair.sellar@metoffice.gov.uk) Bouwe Andela (Netherlands eScience Center, The Netherlands - b.andela@esciencecenter.nl) Lee de Mora (PML, UK - ledm@pml.ac.uk) Niels Drost (Netherlands eScience Center, The Netherlands - n.drost@esciencecenter.nl) Veronika Eyring (DLR, Germany - veronika.eyring@dlr.de) Bettina Gier (UBremen, Germany - gier@uni-bremen.de) Remi Kazeroni (DLR, Germany - remi.kazeroni@dlr.de) Nikolay Koldunov (AWI, Germany - nikolay.koldunov@awi.de) Axel Lauer (DLR, Germany - axel.lauer@dlr.de) Saskia Loosveldt-Tomas (BSC, Spain - saskia.loosveldt@bsc.es) Ruth Lorenz (ETH Zurich, Switzerland - ruth.lorenz@env.ethz.ch) Benjamin Mueller (LMU, Germany - b.mueller@iggf.geo.uni-muenchen.de) Valeriu Predoi (URead, UK - valeriu.predoi@ncas.ac.uk) Mattia Righi (DLR, Germany - mattia.righi@dlr.de) Manuel Schlund (DLR, Germany - manuel.schlund@dlr.de) Breixo Solino Fernandez (DLR, Germany - breixo.solinofernandez@dlr.de) Javier Vegas-Regidor (BSC, Spain - javier.vegas@bsc.es) Klaus Zimmermann (SMHI, Sweden - klaus.zimmermann@smhi.se) For further help, please read the documentation at http://docs.esmvaltool.org. Have fun! 2023-05-30 12:49:08,211 UTC [357476] INFO esmvalcore._main:496 Package versions 2023-05-30 12:49:08,211 UTC [357476] INFO esmvalcore._main:497 ---------------- 2023-05-30 12:49:08,211 UTC [357476] INFO esmvalcore._main:498 ESMValCore: 2.8.0 2023-05-30 12:49:08,211 UTC [357476] INFO esmvalcore._main:500 ESMValTool: 2.8.0 2023-05-30 12:49:08,211 UTC [357476] INFO esmvalcore._main:501 ---------------- 2023-05-30 12:49:08,211 UTC [357476] INFO esmvalcore._main:502 Using config file /nird/home/adagj/.esmvaltool/config-user.yml 2023-05-30 12:49:08,211 UTC [357476] INFO esmvalcore._main:503 Writing program log files to: /projects/NS9560K/www/diagnostics/esmvaltool/adagj/tmp/recipe_validation_CMIP6_20230530_124908/run/main_log.txt /projects/NS9560K/www/diagnostics/esmvaltool/adagj/tmp/recipe_validation_CMIP6_20230530_124908/run/main_log_debug.txt 2023-05-30 12:49:08,324 UTC [357476] INFO esmvalcore.config._esgf_pyclient:116 Using default ESGF configuration, configuration file /nird/home/adagj/.esmvaltool/esgf-pyclient.yml not present. 2023-05-30 12:49:08,324 UTC [357476] WARNING esmvalcore.config._esgf_pyclient:181 ESGF credentials missing, only data that is accessible without logging in will be available. See https://esgf.github.io/esgf-user-support/user_guide.html for instructions on how to create an account if you do not have one yet. Next, configure your system so esmvaltool can use your credentials. This can be done using the keyring package, or you can just enter them in /nird/home/adagj/.esmvaltool/esgf-pyclient.yml. keyring ======= First install the keyring package (requires a supported backend, see https://pypi.org/project/keyring/): $ pip install keyring Next, set your username and password by running the commands: $ keyring set ESGF hostname $ keyring set ESGF username $ keyring set ESGF password To check that you entered your credentials correctly, run: $ keyring get ESGF hostname $ keyring get ESGF username $ keyring get ESGF password configuration file ================== You can store the hostname, username, and password or your OpenID account in a plain text in the file /nird/home/adagj/.esmvaltool/esgf-pyclient.yml like this: logon: hostname: "your-hostname" username: "your-username" password: "your-password" or your can configure an interactive log in: logon: interactive: true Note that storing your password in plain text in the configuration file is less secure. On shared systems, make sure the permissions of the file are set so only you can read it, i.e. $ ls -l /nird/home/adagj/.esmvaltool/esgf-pyclient.yml shows permissions -rw-------. 2023-05-30 12:49:08,885 UTC [357476] INFO esmvalcore._main:85 Starting the Earth System Model Evaluation Tool at time: 2023-05-30 12:49:08 UTC 2023-05-30 12:49:08,885 UTC [357476] INFO esmvalcore._main:89 ---------------------------------------------------------------------- 2023-05-30 12:49:08,885 UTC [357476] INFO esmvalcore._main:90 RECIPE = /nird/home/adagj/esmvaltool_workshop/recipe_validation_CMIP6.yml 2023-05-30 12:49:08,885 UTC [357476] INFO esmvalcore._main:91 RUNDIR = /projects/NS9560K/www/diagnostics/esmvaltool/adagj/tmp/recipe_validation_CMIP6_20230530_124908/run 2023-05-30 12:49:08,885 UTC [357476] INFO esmvalcore._main:92 WORKDIR = /projects/NS9560K/www/diagnostics/esmvaltool/adagj/tmp/recipe_validation_CMIP6_20230530_124908/work 2023-05-30 12:49:08,885 UTC [357476] INFO esmvalcore._main:93 PREPROCDIR = /projects/NS9560K/www/diagnostics/esmvaltool/adagj/tmp/recipe_validation_CMIP6_20230530_124908/preproc 2023-05-30 12:49:08,886 UTC [357476] INFO esmvalcore._main:94 PLOTDIR = /projects/NS9560K/www/diagnostics/esmvaltool/adagj/tmp/recipe_validation_CMIP6_20230530_124908/plots 2023-05-30 12:49:08,886 UTC [357476] INFO esmvalcore._main:95 ---------------------------------------------------------------------- 2023-05-30 12:49:08,886 UTC [357476] INFO esmvalcore._main:98 Running tasks using at most 2 processes 2023-05-30 12:49:08,886 UTC [357476] INFO esmvalcore._main:100 If your system hangs during execution, it may not have enough memory for keeping this number of tasks in memory. 2023-05-30 12:49:08,886 UTC [357476] INFO esmvalcore._main:103 If you experience memory problems, try reducing 'max_parallel_tasks' in your user configuration file. 2023-05-30 12:49:08,887 UTC [357476] DEBUG esmvalcore._recipe.check:52 Checking recipe against schema /diagnostics/esmvaltool/2.8.0/lib/python3.10/site-packages/esmvalcore/_recipe/recipe_schema.yml 2023-05-30 12:49:08,914 UTC [357476] DEBUG esmvalcore._recipe.to_datasets:344 Populating list of datasets for variable tas in diagnostic validation_with_ERA-Interim 2023-05-30 12:49:08,914 UTC [357476] DEBUG esmvalcore._recipe.to_datasets:368 Found Dataset: tas, Amon, CMIP6, IPSL-CM6A-LR, historical, r1i1p1f1, gr 2023-05-30 12:49:08,914 UTC [357476] DEBUG esmvalcore._recipe.to_datasets:368 Found Dataset: tas, Amon, CMIP6, UKESM1-0-LL, historical, r1i1p1f2, gn 2023-05-30 12:49:08,914 UTC [357476] DEBUG esmvalcore._recipe.to_datasets:368 Found Dataset: tas, Amon, OBS6, ERA-Interim, 1 2023-05-30 12:49:08,915 UTC [357476] DEBUG esmvalcore._recipe.recipe:1014 Retrieving diagnostics from recipe 2023-05-30 12:49:08,915 UTC [357476] DEBUG esmvalcore._recipe.recipe:1042 Setting script for diagnostic validation_with_ERA-Interim 2023-05-30 12:49:08,915 UTC [357476] INFO esmvalcore._recipe.recipe:1232 Creating tasks from recipe 2023-05-30 12:49:08,916 UTC [357476] INFO esmvalcore._recipe.recipe:1241 Creating tasks for diagnostic validation_with_ERA-Interim 2023-05-30 12:49:08,916 UTC [357476] INFO esmvalcore._recipe.recipe:1163 Creating diagnostic task validation_with_ERA-Interim/lat_lon 2023-05-30 12:49:08,916 UTC [357476] INFO esmvalcore._recipe.recipe:1163 Creating diagnostic task validation_with_ERA-Interim/zonal_mean 2023-05-30 12:49:08,916 UTC [357476] INFO esmvalcore._recipe.recipe:1163 Creating diagnostic task validation_with_ERA-Interim/meridional_mean 2023-05-30 12:49:08,916 UTC [357476] INFO esmvalcore._recipe.recipe:1216 Creating preprocessor task validation_with_ERA-Interim/tas 2023-05-30 12:49:08,916 UTC [357476] INFO esmvalcore._recipe.recipe:849 Creating preprocessor 'pp_rad' task for variable 'tas' 2023-05-30 12:49:08,917 UTC [357476] WARNING esmvalcore.local:426 'CMIP6' rootpaths '/scratch/adagj/ESGF/CMIP6' set in config-user.yml do not exist 2023-05-30 12:49:08,918 UTC [357476] DEBUG esmvalcore.local:458 Looking for files matching: /projects/NS9034K/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/historical/r1i1p1f1/Amon/tas/gr/*/tas_Amon_IPSL-CM6A-LR_historical_r1i1p1f1_gr*.nc /projects/NS9560K-datalake/ESGF/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/historical/r1i1p1f1/Amon/tas/gr/*/tas_Amon_IPSL-CM6A-LR_historical_r1i1p1f1_gr*.nc /scratch/adagj/ESGF/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/historical/r1i1p1f1/Amon/tas/gr/*/tas_Amon_IPSL-CM6A-LR_historical_r1i1p1f1_gr*.nc 2023-05-30 12:49:08,936 UTC [357476] DEBUG esmvalcore._recipe.recipe:446 Using input files for variable tas of dataset CMIP6 IPSL-CM6A-LR: /projects/NS9560K-datalake/ESGF/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/historical/r1i1p1f1/Amon/tas/gr/v20180803/tas_Amon_IPSL-CM6A-LR_historical_r1i1p1f1_gr_185001-201412.nc 2023-05-30 12:49:08,936 UTC [357476] INFO esmvalcore._recipe.recipe:717 Found input files for Dataset: tas, Amon, CMIP6, IPSL-CM6A-LR, CMIP, historical, r1i1p1f1, gr, v20180803 2023-05-30 12:49:08,937 UTC [357476] DEBUG esmvalcore.local:458 Looking for files matching: /projects/NS9034K/CMIP6/CMIP/MOHC/UKESM1-0-LL/historical/r1i1p1f2/Amon/tas/gn/*/tas_Amon_UKESM1-0-LL_historical_r1i1p1f2_gn*.nc /projects/NS9034K/CMIP6/CMIP/NERC/UKESM1-0-LL/historical/r1i1p1f2/Amon/tas/gn/*/tas_Amon_UKESM1-0-LL_historical_r1i1p1f2_gn*.nc /projects/NS9034K/CMIP6/CMIP/NIMS-KMA/UKESM1-0-LL/historical/r1i1p1f2/Amon/tas/gn/*/tas_Amon_UKESM1-0-LL_historical_r1i1p1f2_gn*.nc /projects/NS9034K/CMIP6/CMIP/NIWA/UKESM1-0-LL/historical/r1i1p1f2/Amon/tas/gn/*/tas_Amon_UKESM1-0-LL_historical_r1i1p1f2_gn*.nc /projects/NS9560K-datalake/ESGF/CMIP6/CMIP/MOHC/UKESM1-0-LL/historical/r1i1p1f2/Amon/tas/gn/*/tas_Amon_UKESM1-0-LL_historical_r1i1p1f2_gn*.nc /projects/NS9560K-datalake/ESGF/CMIP6/CMIP/NERC/UKESM1-0-LL/historical/r1i1p1f2/Amon/tas/gn/*/tas_Amon_UKESM1-0-LL_historical_r1i1p1f2_gn*.nc /projects/NS9560K-datalake/ESGF/CMIP6/CMIP/NIMS-KMA/UKESM1-0-LL/historical/r1i1p1f2/Amon/tas/gn/*/tas_Amon_UKESM1-0-LL_historical_r1i1p1f2_gn*.nc /projects/NS9560K-datalake/ESGF/CMIP6/CMIP/NIWA/UKESM1-0-LL/historical/r1i1p1f2/Amon/tas/gn/*/tas_Amon_UKESM1-0-LL_historical_r1i1p1f2_gn*.nc /scratch/adagj/ESGF/CMIP6/CMIP/MOHC/UKESM1-0-LL/historical/r1i1p1f2/Amon/tas/gn/*/tas_Amon_UKESM1-0-LL_historical_r1i1p1f2_gn*.nc /scratch/adagj/ESGF/CMIP6/CMIP/NERC/UKESM1-0-LL/historical/r1i1p1f2/Amon/tas/gn/*/tas_Amon_UKESM1-0-LL_historical_r1i1p1f2_gn*.nc /scratch/adagj/ESGF/CMIP6/CMIP/NIMS-KMA/UKESM1-0-LL/historical/r1i1p1f2/Amon/tas/gn/*/tas_Amon_UKESM1-0-LL_historical_r1i1p1f2_gn*.nc /scratch/adagj/ESGF/CMIP6/CMIP/NIWA/UKESM1-0-LL/historical/r1i1p1f2/Amon/tas/gn/*/tas_Amon_UKESM1-0-LL_historical_r1i1p1f2_gn*.nc 2023-05-30 12:49:08,941 UTC [357476] DEBUG esmvalcore._recipe.recipe:446 Using input files for variable tas of dataset CMIP6 UKESM1-0-LL: /projects/NS9560K-datalake/ESGF/CMIP6/CMIP/MOHC/UKESM1-0-LL/historical/r1i1p1f2/Amon/tas/gn/v20190406/tas_Amon_UKESM1-0-LL_historical_r1i1p1f2_gn_185001-194912.nc /projects/NS9560K-datalake/ESGF/CMIP6/CMIP/MOHC/UKESM1-0-LL/historical/r1i1p1f2/Amon/tas/gn/v20190406/tas_Amon_UKESM1-0-LL_historical_r1i1p1f2_gn_195001-201412.nc 2023-05-30 12:49:08,941 UTC [357476] INFO esmvalcore._recipe.recipe:717 Found input files for Dataset: tas, Amon, CMIP6, UKESM1-0-LL, CMIP, historical, r1i1p1f2, gn, v20190406 2023-05-30 12:49:08,941 UTC [357476] WARNING esmvalcore.local:426 'OBS6' rootpaths '/scratch/adagj/ESGF/obsdata' set in config-user.yml do not exist 2023-05-30 12:49:08,941 UTC [357476] DEBUG esmvalcore.local:458 Looking for files matching: /projects/NS9560K-datalake/ESGF/Tier3/ERA-Interim/OBS6_ERA-Interim_reanaly_1_Amon_tas[_.]*nc /scratch/adagj/ESGF/obsdata/Tier3/ERA-Interim/OBS6_ERA-Interim_reanaly_1_Amon_tas[_.]*nc 2023-05-30 12:49:08,942 UTC [357476] ERROR esmvalcore._recipe.check:118 No input files found for Dataset: {'diagnostic': 'validation_with_ERA-Interim', 'variable_group': 'tas', 'dataset': 'ERA-Interim', 'project': 'OBS6', 'mip': 'Amon', 'short_name': 'tas', 'alias': 'OBS6', 'frequency': 'mon', 'long_name': 'Near-Surface Air Temperature', 'modeling_realm': ['atmos'], 'original_short_name': 'tas', 'preprocessor': 'pp_rad', 'recipe_dataset_index': 2, 'standard_name': 'air_temperature', 'tier': 3, 'timerange': '1980/2000', 'type': 'reanaly', 'units': 'K', 'version': 1} session: 'recipe_validation_CMIP6_20230530_124908' 2023-05-30 12:49:08,942 UTC [357476] ERROR esmvalcore._recipe.check:124 Looked for files matching /projects/NS9560K-datalake/ESGF/Tier3/ERA-Interim/OBS6_ERA-Interim_reanaly_1_Amon_tas[_.]*nc /scratch/adagj/ESGF/obsdata/Tier3/ERA-Interim/OBS6_ERA-Interim_reanaly_1_Amon_tas[_.]*nc 2023-05-30 12:49:08,942 UTC [357476] ERROR esmvalcore._recipe.check:125 Set 'log_level' to 'debug' to get more information 2023-05-30 12:49:08,943 UTC [357476] ERROR esmvalcore._recipe.recipe:960 Could not create all tasks 2023-05-30 12:49:08,943 UTC [357476] ERROR esmvalcore._recipe.recipe:962 Missing data for preprocessor validation_with_ERA-Interim/tas: - Missing data for Dataset: tas, Amon, OBS6, ERA-Interim, 1 2023-05-30 12:49:09,335 UTC [357476] INFO esmvalcore._task:126 Maximum memory used (estimate): 0.0 GB 2023-05-30 12:49:09,336 UTC [357476] INFO esmvalcore._task:128 Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur. 2023-05-30 12:49:09,336 UTC [357476] ERROR esmvalcore._main:526 Could not create all tasks 2023-05-30 12:49:09,337 UTC [357476] DEBUG esmvalcore._main:527 Stack trace for debugging: Traceback (most recent call last): File "/diagnostics/esmvaltool/2.8.0/lib/python3.10/site-packages/esmvalcore/_main.py", line 521, in run fire.Fire(ESMValTool()) File "/diagnostics/esmvaltool/2.8.0/lib/python3.10/site-packages/fire/core.py", line 141, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/diagnostics/esmvaltool/2.8.0/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire component, remaining_args = _CallAndUpdateTrace( File "/diagnostics/esmvaltool/2.8.0/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) File "/diagnostics/esmvaltool/2.8.0/lib/python3.10/site-packages/esmvalcore/_main.py", line 408, in run self._run(recipe, session) File "/diagnostics/esmvaltool/2.8.0/lib/python3.10/site-packages/esmvalcore/_main.py", line 450, in _run process_recipe(recipe_file=recipe, session=session) File "/diagnostics/esmvaltool/2.8.0/lib/python3.10/site-packages/esmvalcore/_main.py", line 119, in process_recipe recipe = read_recipe_file(recipe_file, session) File "/diagnostics/esmvaltool/2.8.0/lib/python3.10/site-packages/esmvalcore/_recipe/recipe.py", line 82, in read_recipe_file return Recipe(raw_recipe, session, recipe_file=filename) File "/diagnostics/esmvaltool/2.8.0/lib/python3.10/site-packages/esmvalcore/_recipe/recipe.py", line 917, in __init__ self.tasks = self.initialize_tasks() File "/diagnostics/esmvaltool/2.8.0/lib/python3.10/site-packages/esmvalcore/_recipe/recipe.py", line 1279, in initialize_tasks tasks = self._create_tasks() File "/diagnostics/esmvaltool/2.8.0/lib/python3.10/site-packages/esmvalcore/_recipe/recipe.py", line 1267, in _create_tasks raise recipe_error esmvalcore.exceptions.RecipeError: Could not create all tasks