Site and global datasets with the executable file

This notebook shows how to run the STEMMUS_SCOPE model uisng the Executable file, including preprocessing and postprocessing. See the Different ways to run the model for different ways e.g. Executable file, Matlab, Octave.

Also, check the Getting started page on documentation of STEMMUS_SCOPE and pystemmusscope.

For postprocesisng, the file required_netcdf_variables.csv can be found at https://github.com/EcoExtreML/STEMMUS_SCOPE/blob/main/utils/csv_to_nc/required_netcdf_variables.csv.

Which datasets are available?¶

The config file shows how to set the paths to different data needed by the model. Check the data source on Snelius and CRIB and modify the config file accordingly:

On CRIB¶

• Site data:
  • path to data: /data/shared/EcoExtreML/STEMMUS_SCOPEv1.0.0/input,
  • config file template: https://github.com/EcoExtreML/STEMMUS_SCOPE/blob/main/config_file_crib.txt
• Global data:
  • path to data /data/shared/EcoExtreML/STEMMUS_SCOPEv1.0.0/global_data,
  • config file template: https://github.com/EcoExtreML/STEMMUS_SCOPE/blob/main/run_model_on_snellius/config_file_crib_global.txt

On Snellius¶

• Site data:
  • path to data: /projects/0/ttse0619/,
  • config file template: https://github.com/EcoExtreML/STEMMUS_SCOPE/blob/main/run_model_on_snellius/config_file_snellius.txt
• Global data:
  • path to data: /projects/0/ttse0619/global_data,
  • config file template: https://github.com/EcoExtreML/STEMMUS_SCOPE/blob/main/run_model_on_snellius/config_file_snellius_global.txt

Choose how do you want to run the model:

Using site data

Using global data

Using site data¶

In [1]:

import os
from PyStemmusScope import StemmusScope
from PyStemmusScope import save

import os from PyStemmusScope import StemmusScope from PyStemmusScope import save

In [2]:

# user must provide the correct path
path_to_config_file = "./my_config_template.txt"
path_to_exe_file = "./STEMMUS_SCOPE"

# user must provide the correct path path_to_config_file = "./my_config_template.txt" path_to_exe_file = "./STEMMUS_SCOPE"

If you run the model on your own machine, make sure LD_LIBRARY_PATH is set correctly. To do this, uncomment the cell below and run it:

In [3]:

# # Set LD_LIBRARY_PATH
# matlab_path = !whereis MATLAB
# matlab_path = matlab_path.s.split(": ")[1]
# os.environ['LD_LIBRARY_PATH'] = (
#     f"{matlab_path}/MATLAB_Runtime/R2023a/runtime/glnxa64:"
#     f"{matlab_path}/MATLAB_Runtime/R2023a/bin/glnxa64:"
#     f"{matlab_path}/MATLAB_Runtime/R2023a/sys/os/glnxa64:"
#     f"{matlab_path}/MATLAB_Runtime/R2023a/extern/bin/glnxa64:"
#     f"{matlab_path}/MATLAB_Runtime/R2023a/sys/opengl/lib/glnxa64")
# print(os.environ['LD_LIBRARY_PATH'])

# # Set LD_LIBRARY_PATH # matlab_path = !whereis MATLAB # matlab_path = matlab_path.s.split(": ")[1] # os.environ['LD_LIBRARY_PATH'] = ( # f"{matlab_path}/MATLAB_Runtime/R2023a/runtime/glnxa64:" # f"{matlab_path}/MATLAB_Runtime/R2023a/bin/glnxa64:" # f"{matlab_path}/MATLAB_Runtime/R2023a/sys/os/glnxa64:" # f"{matlab_path}/MATLAB_Runtime/R2023a/extern/bin/glnxa64:" # f"{matlab_path}/MATLAB_Runtime/R2023a/sys/opengl/lib/glnxa64") # print(os.environ['LD_LIBRARY_PATH'])

/usr/local/MATLAB/MATLAB_Runtime/R2023a/runtime/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/R2023a/bin/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/R2023a/sys/os/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/R2023a/extern/bin/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/R2023a/sys/opengl/lib/glnxa64

In [4]:

# create an an instance of the model
model = StemmusScope(config_file=path_to_config_file, model_src_path=path_to_exe_file)

# create an an instance of the model model = StemmusScope(config_file=path_to_config_file, model_src_path=path_to_exe_file)

In [5]:

# setup the model
# here you can change the location and start & end time
config_path = model.setup(
    Location="ZA-Kru",
    StartTime="2001-01-01T00:00",
    EndTime="2001-01-02T01:30",
)

# new config file genertaed to run the model
print(f"New config file {config_path}")

# see input and output paths generated by the model
print(f'Model input dir {model.config["InputPath"]}')
print(f'Model output dir {model.config["OutputPath"]}')

# setup the model # here you can change the location and start & end time config_path = model.setup( Location="ZA-Kru", StartTime="2001-01-01T00:00", EndTime="2001-01-02T01:30", ) # new config file genertaed to run the model print(f"New config file {config_path}") # see input and output paths generated by the model print(f'Model input dir {model.config["InputPath"]}') print(f'Model output dir {model.config["OutputPath"]}')

New config file /home/sarah/temp/ecoextreml/test/input/ZA-Kru_2024-01-24-1125/ZA-Kru_2024-01-24-1125_config.txt
Model input dir /home/sarah/temp/ecoextreml/test/input/ZA-Kru_2024-01-24-1125/
Model output dir /home/sarah/temp/ecoextreml/test/output/ZA-Kru_2024-01-24-1125/

In [6]:

# run the model
result = model.run()
result

# run the model result = model.run() result

Out[6]:

'Reading config from /home/sarah/temp/ecoextreml/test/input/ZA-Kru_2024-01-24-1125/ZA-Kru_2024-01-24-1125_config.txt\nFinished model initialization\nThe calculations start now\nFinalizing STEMMUS_SCOPE\n'

In [7]:

# save output in netcdf format
required_netcdf_variables = "./required_netcdf_variables.csv"
nc_file_name = save.to_netcdf(config_path, required_netcdf_variables)
print(nc_file_name)

# save output in netcdf format required_netcdf_variables = "./required_netcdf_variables.csv" nc_file_name = save.to_netcdf(config_path, required_netcdf_variables) print(nc_file_name)

/home/sarah/temp/ecoextreml/test/output/ZA-Kru_2024-01-24-1125/ZA-Kru_2024-01-24-1125_STEMMUS_SCOPE.nc

Using global data¶

In [6]:

import os
from PyStemmusScope import StemmusScope
from PyStemmusScope import save

import os from PyStemmusScope import StemmusScope from PyStemmusScope import save

In [7]:

# user must provide the correct path
path_to_config_file = "./my_config_template_global.txt"
path_to_exe_file = "./STEMMUS_SCOPE"

# user must provide the correct path path_to_config_file = "./my_config_template_global.txt" path_to_exe_file = "./STEMMUS_SCOPE"

If you run the model on your own machine, make sure LD_LIBRARY_PATH is set correctly. To do this, uncomment the cell below and run it:

In [8]:

# # Set LD_LIBRARY_PATH
# matlab_path = !whereis MATLAB
# matlab_path = matlab_path.s.split(": ")[1]
# os.environ['LD_LIBRARY_PATH'] = (
#     f"{matlab_path}/MATLAB_Runtime/R2023a/runtime/glnxa64:"
#     f"{matlab_path}/MATLAB_Runtime/R2023a/bin/glnxa64:"
#     f"{matlab_path}/MATLAB_Runtime/R2023a/sys/os/glnxa64:"
#     f"{matlab_path}/MATLAB_Runtime/R2023a/extern/bin/glnxa64:"
#     f"{matlab_path}/MATLAB_Runtime/R2023a/sys/opengl/lib/glnxa64")
# print(os.environ['LD_LIBRARY_PATH'])

# # Set LD_LIBRARY_PATH # matlab_path = !whereis MATLAB # matlab_path = matlab_path.s.split(": ")[1] # os.environ['LD_LIBRARY_PATH'] = ( # f"{matlab_path}/MATLAB_Runtime/R2023a/runtime/glnxa64:" # f"{matlab_path}/MATLAB_Runtime/R2023a/bin/glnxa64:" # f"{matlab_path}/MATLAB_Runtime/R2023a/sys/os/glnxa64:" # f"{matlab_path}/MATLAB_Runtime/R2023a/extern/bin/glnxa64:" # f"{matlab_path}/MATLAB_Runtime/R2023a/sys/opengl/lib/glnxa64") # print(os.environ['LD_LIBRARY_PATH'])

/usr/local/MATLAB/MATLAB_Runtime/R2023a/runtime/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/R2023a/bin/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/R2023a/sys/os/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/R2023a/extern/bin/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/R2023a/sys/opengl/lib/glnxa64

In [9]:

# create an an instance of the model
model = StemmusScope(config_file=path_to_config_file, model_src_path=path_to_exe_file)

# create an an instance of the model model = StemmusScope(config_file=path_to_config_file, model_src_path=path_to_exe_file)

In [10]:

# setup the model
config_path = model.setup(
    Location="(61.8474, 24.2948)",
    StartTime="2014-01-10T00:00",
    EndTime="2014-01-15T00:00",
)

# new config file genertaed to run the model
print(f"New config file {config_path}")

# see input and output paths generated by the model
print(f'Model input dir {model.config["InputPath"]}')
print(f'Model output dir {model.config["OutputPath"]}')

# setup the model config_path = model.setup( Location="(61.8474, 24.2948)", StartTime="2014-01-10T00:00", EndTime="2014-01-15T00:00", ) # new config file genertaed to run the model print(f"New config file {config_path}") # see input and output paths generated by the model print(f'Model input dir {model.config["InputPath"]}') print(f'Model output dir {model.config["OutputPath"]}')

/home/sarah/miniconda3/envs/pystemmusscope/lib/python3.10/site-packages/dask/array/core.py:4836: PerformanceWarning: Increasing number of chunks by factor of 16
  result = blockwise(
/home/sarah/miniconda3/envs/pystemmusscope/lib/python3.10/site-packages/dask/array/core.py:4836: PerformanceWarning: Increasing number of chunks by factor of 16
  result = blockwise(
/home/sarah/miniconda3/envs/pystemmusscope/lib/python3.10/site-packages/dask/array/core.py:4836: PerformanceWarning: Increasing number of chunks by factor of 16
  result = blockwise(
/home/sarah/miniconda3/envs/pystemmusscope/lib/python3.10/site-packages/dask/array/core.py:4836: PerformanceWarning: Increasing number of chunks by factor of 16
  result = blockwise(

New config file /home/sarah/temp/ecoextreml/test/input/global_N61-847_E24-295_2024-02-02-1620/global_2024-02-02-1620_config.txt
Model input dir /home/sarah/temp/ecoextreml/test/input/global_N61-847_E24-295_2024-02-02-1620/
Model output dir /home/sarah/temp/ecoextreml/test/output/global_N61-847_E24-295_2024-02-02-1620/

In [ ]:

# run the model
result = model.run()
result

# run the model result = model.run() result

In [ ]:

# save output in netcdf format
required_netcdf_variables = "./required_netcdf_variables.csv"
nc_file_name = save.to_netcdf(config_path, required_netcdf_variables)
print(nc_file_name)

# save output in netcdf format required_netcdf_variables = "./required_netcdf_variables.csv" nc_file_name = save.to_netcdf(config_path, required_netcdf_variables) print(nc_file_name)

In [ ]: