{ "cells": [ { "cell_type": "markdown", "id": "fa561492", "metadata": { "tags": [] }, "source": [ "
\n", "\n", "

ICON Training - Hands-on Session

\n", "\n", "# Exercise 6.3: ICON ComIn, Part III\n", "\n", "---\n", "The **ICON Community Interface (ComIn)** enables the integration of **plugins** which is a flexible way to extend the code with your own routines and variables at run-time. \n", "\n", "* **Step P3**: In the final step, we learn how to use the coupling library [YAC](https://dkrz-sw.gitlab-pages.dkrz.de/yac/) in a plugin to interpolate fields from the ICON unstructured grid onto a custom lat-lon grid, and then visualize the interpolated fields.\n" ] }, { "cell_type": "markdown", "id": "e61c91f4-91e2-4e24-a168-049ee2687e33", "metadata": {}, "source": [ "
\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "e45dc60e-c089-4d64-9bc5-d23a62ce1c90", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "## Step P3: Using YAC in ComIn plugin for interpolation" ] }, { "cell_type": "markdown", "id": "2823986c-061e-47f7-955a-0f5b4dfdaf8f", "metadata": {}, "source": [ "\n", " \n", " \n", " \n", " \n", "
\n", "
\n", " Exercise: \n", " In this exercise, we will develop the plugin step by step. Some parts will be provided, and you will be asked to implement the necessary missing components:

\n", "

\n", " Access the surface pressure (pres_sfc) variable from ICON in your plugin. This time, use the YAC instance to interpolate the variable from the ICON unstructured grid into a custom lat-lon grid with 0.05 degree resolution on rank 0. Then, visualize the interpolated output every one hour.\n", "

\n", "
\n", " 		\n", " \n", "
\n" ] }, { "cell_type": "markdown", "id": "90a392f7-4c59-4ac5-b242-228e734655ee", "metadata": {}, "source": [ "### Step-by-step writing ComIn plugin " ] }, { "cell_type": "markdown", "id": "cd7df0ff-de80-447c-89c1-d0bbed5fd51b", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "In this notebook, we will append Jupyter notebook code cells to the initially empty file scripts/comin_plugin_P3.py, step by step. This will happen automatically as we run the cells. To do this, we'll define a custom Jupyter cell magic command in append_magic.py." ] }, { "cell_type": "code", "execution_count": null, "id": "6dc53ee8-efc4-4d7a-b225-88cf85dfd658", "metadata": {}, "outputs": [], "source": [ "%load_ext append_magic" ] }, { "cell_type": "markdown", "id": "1f7b7404-b1a1-4b92-9371-212f873bb544", "metadata": {}, "source": [ "First steps:\n", "\n", "- We need to import the necessary packages, most importantly, `yac` and `comin`.\n", "- We get the neccessary descriptive data.\n", "- We query the YAC instance ID. This is the predefined identifier used to reference the YAC instance for defining components, grids, and times in the coupling framework." ] }, { "cell_type": "code", "execution_count": null, "id": "54e0951d-bf3c-442a-be61-fc73a40a2a11", "metadata": {}, "outputs": [], "source": [ "%%append_to_script scripts/comin_plugin_P3.py --reset\n", "\n", "from yac import (\n", " YAC, Reg2dGrid, Location, Field, TimeUnit, Reduction,\n", " InterpolationStack, NNNReductionType, UnstructuredGrid\n", ")\n", "import comin\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "\n", " \n", "# === GET DESCRIPTIVE DATA STRUCTURE ===\n", "\n", "domain = comin.descrdata_get_domain(1) # DOMAIN 1\n", "glob = comin.descrdata_get_global()\n", "rank = comin.parallel_get_host_mpi_rank()\n", "\n", "assert glob.yac_instance_id != -1, \"The host-model is not configured with YAC\"\n", "yac = YAC.from_id(glob.yac_instance_id)" ] }, { "cell_type": "markdown", "id": "b389fd43-82ef-489c-896e-683cf9290899", "metadata": {}, "source": [ "We then define the source grid and component. **YAC components** are the individual Earth system model components - such as atmosphere, ocean, or an output writer - that are coupled together to exchange data fields through the YAC library. \n", "\n", "The source component will be based on ICON's computational grid and the following code snippets show how to define such an unstructured mesh with the YAC coupler." ] }, { "cell_type": "code", "execution_count": null, "id": "a79926b9-05be-4615-ab82-7b24d459e110", "metadata": {}, "outputs": [], "source": [ "%%append_to_script scripts/comin_plugin_P3.py\n", "\n", "# === DEFINE SOURCE GRID AND COMPONENT ===\n", "\n", "source_component = yac.predef_comp(\"comin_example_source\")\n", "\n", "ncells = domain.cells.ncells\n", "nverts = domain.verts.nverts\n", "\n", "# the \"connectivity\" field contains the three vertices for each cell:\n", "connectivity = (np.asarray(domain.cells.vertex_blk) - 1) * glob.nproma + (\n", " np.asarray(domain.cells.vertex_idx) - 1\n", ")\n", "\n", "# We also provide the geographical coordinates for each vertex.\n", "# Please note that we need to specify the 'order=F' argument\n", "# because Fortran (ICON) uses column-major ordering.\n", "icon_grid = UnstructuredGrid(\n", " \"icon_grid\",\n", " np.ones(domain.cells.ncells) * 3,\n", " np.ravel(domain.verts.vlon, order='F')[: domain.verts.nverts],\n", " np.ravel(domain.verts.vlat, order='F')[: domain.verts.nverts],\n", " np.ravel(np.swapaxes(connectivity, 0, 1))[: 3 * domain.cells.ncells],\n", ")\n", "\n", "# \"def_points\": This method is used to specify the locations (such as corners,\n", "# centers, or edges) of grid points for coupling fields between components.\n", "icon_cell_centers = icon_grid.def_points(\n", " Location.CELL,\n", " np.ravel(domain.cells.clon)[:ncells],\n", " np.ravel(domain.cells.clat)[:ncells],\n", ")" ] }, { "cell_type": "markdown", "id": "590b8e57-863f-4e64-a568-3e52cb60e0d3", "metadata": {}, "source": [ "
\n", "Exercise: In this part, define the target grid and component. Define the target grid only on rank = 0.
\n", " Some hints:\n", " \n", "
" ] }, { "cell_type": "code", "execution_count": null, "id": "da8f9a59-62e4-4ce0-879f-3ce24f3814ca", "metadata": {}, "outputs": [], "source": [ "%%append_to_script scripts/comin_plugin_P3.py\n", "\n", "#write your code here" ] }, { "cell_type": "markdown", "id": "57d79a1b-d9b2-4ab7-ac2f-406146a53e95", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [ "hide-input" ] }, "source": [ "
\n", "Solution
\n", "\n", "```python\n", "if rank == 0:\n", " target_component = yac.predef_comp(\"comin_example_target\")\n", "\n", " vlon = np.deg2rad(np.arange(1.0, 16.6 + 0.001, 0.05))\n", " vlat = np.deg2rad(np.arange(44.1, 57.5 + 0.001, 0.05))\n", " grid = Reg2dGrid(\"example_grid\", vlon, vlat, cyclic=(True, False))\n", " cell_centers = grid.def_points(\n", " Location.CELL,\n", " vlon + 0.5 * (vlon[1] - vlon[0]),\n", " vlat[:-1] + 0.5 * (vlat[1] - vlat[0]),\n", " )\n", "```\n", "
\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "6705ec3e-af3c-4fd1-820f-14e40a206258", "metadata": {}, "source": [ "Now, we access the variable `pres_sfc` in the secondary constructor, in a similar way to the previous exercises P1 and P2." ] }, { "cell_type": "code", "execution_count": null, "id": "ce741580-a2a8-4380-a4a2-fffdf4020e27", "metadata": {}, "outputs": [], "source": [ "%%append_to_script scripts/comin_plugin_P3.py\n", "\n", "@comin.EP_SECONDARY_CONSTRUCTOR\n", "def secondary_constructor():\n", " global comin_pres_sfc\n", " comin_pres_sfc = comin.var_get(\n", " [comin.EP_ATM_WRITE_OUTPUT_BEFORE], \n", " (\"pres_sfc\", 1), \n", " comin.COMIN_FLAG_READ\n", " )" ] }, { "cell_type": "markdown", "id": "105c1379-17b4-4585-a1f7-b5c065c9dbdc", "metadata": {}, "source": [ "
\n", "Exercise:\n", "In the next step, we need to define the coupling fields. This part is partially provided. Define the target field in the cell below. See the YAC library documentation here for more information.\n", "
\n" ] }, { "cell_type": "code", "execution_count": null, "id": "a4f609df-2d05-415d-a5b8-354e2153eaa5", "metadata": {}, "outputs": [], "source": [ "%%append_to_script scripts/comin_plugin_P3.py\n", "\n", "@comin.EP_ATM_YAC_DEFCOMP_AFTER\n", "def define_fields():\n", " global yac_pres_sfc_target, yac_pres_sfc_source\n", "\n", " step = str(int(comin.descrdata_get_timesteplength(1)))\n", " yac_pres_sfc_source = Field.create(\n", " \"pres_sfc\", source_component, icon_cell_centers, 1, step, TimeUnit.SECOND,\n", " )\n", " if rank == 0:\n", "\n", " # >>>>\n", " # TODO: Define the target field here\n", " # <<<<\n", " \n", " nnn = InterpolationStack()\n", " nnn.add_nnn(NNNReductionType.AVG, n=1, scale=1.0, max_search_distance=0.0)\n", " yac.def_couple(\n", " \"comin_example_source\",\n", " \"icon_grid\",\n", " \"pres_sfc\",\n", " #ToDo,\n", " #ToDo,\n", " #ToDo,\n", " step,\n", " TimeUnit.SECOND,\n", " Reduction.TIME_NONE,\n", " nnn,\n", " )" ] }, { "cell_type": "markdown", "id": "0dba87be-fbe5-40d6-b700-683128090b00", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [ "hide-input" ] }, "source": [ "
\n", "Solution
\n", "\n", "```python\n", " if rank == 0:\n", " yac_pres_sfc_target = Field.create(\n", " \"pres_sfc\", target_component, cell_centers, 1, step, TimeUnit.SECOND,\n", " )\n", " nnn = InterpolationStack()\n", " nnn.add_nnn(NNNReductionType.AVG, n=1, scale=1.0, max_search_distance=0.0)\n", " yac.def_couple(\n", " \"comin_example_source\",\n", " \"icon_grid\",\n", " \"pres_sfc\",\n", " \"comin_example_target\",\n", " \"example_grid\",\n", " \"pres_sfc\",\n", " step,\n", " TimeUnit.SECOND,\n", " Reduction.TIME_NONE,\n", " nnn,\n", " )\n", "```\n", "
\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "e0a531ad-748b-4139-92bb-cc36d883dd38", "metadata": {}, "source": [ "Finally, we **put** the source field on the source grid and **get** the target field on the target grid.\n", "\n", "
\n", "Exercise:\n", " Complete the code below. Furthermore, use the provided helper function to plot the interpolated `pres_sfc` field at one-hour intervals.\n", "
" ] }, { "cell_type": "code", "execution_count": null, "id": "65d0cf1b-de92-4fc5-8cfd-1705d48fb246", "metadata": {}, "outputs": [], "source": [ "%%append_to_script scripts/comin_plugin_P3.py\n", "\n", "#=== HELPER FUNCTION ====\n", "import isodate\n", "\n", "def is_time_to_plot(interval_iso):\n", " \"\"\"Check if the current time is at the start of a given interval.\"\"\"\n", " interval = comin.descrdata_get_simulation_interval()\n", " exp_start = np.datetime64(interval.exp_start)\n", " current_time = np.datetime64(comin.current_get_datetime())\n", " interval_ns = int(isodate.parse_duration(interval_iso).total_seconds() * 1e9)\n", " elapsed_ns = (current_time - exp_start).astype('timedelta64[ns]').astype(int)\n", " return (elapsed_ns % interval_ns) == 0\n", "\n", "\n", "## === PUT and GET fields (Partially Given) ===\n", "\n", "@comin.EP_ATM_TIMELOOP_END\n", "def process():\n", " yac_pres_sfc_source.put(np.ravel(comin_pres_sfc)[: domain.cells.ncells])\n", " if rank == 0:\n", "\n", " # >>>>\n", " # TODO: get the field via a call to the YAC library\n", " # <<<<\n", " \n", " # Plot the interpolated field every 1 hour\n", " if is_time_to_plot(\"PT1H\"):\n", " plt.imshow(np.reshape(data, [vlat.shape[0]-1, vlon.shape[0]]))\n", " filename = f\"pres_sfc_{comin.current_get_datetime()}.png\"\n", " print(filename)\n", " plt.savefig(filename)" ] }, { "cell_type": "markdown", "id": "910265d6-53aa-4a85-b2fb-9df8746898d3", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [ "hide-input" ] }, "source": [ "
\n", "Solution
\n", "\n", "```python\n", "data, info = yac_pres_sfc_target.get()\n", "```\n", "
\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "e8961d8a-7a5a-4dd8-a7cb-f9787a2ab344", "metadata": {}, "source": [ "**At this point, you have completed your plugin. Now, let's run the ICON!**" ] }, { "cell_type": "markdown", "id": "3f5c4d44-49bc-43ba-962d-901eb51d745a", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "### Running the ICON model" ] }, { "cell_type": "code", "execution_count": null, "id": "3811200e-b8c6-4f0c-8936-33f933cf0cf4", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [], "source": [ "import os\n", "import subprocess\n", "\n", "user = os.environ['USER']\n", "home = os.environ['HOME']\n", "scratchdir = f\"/scratch/{user[0]}/{user}\"\n", "icondir = f\"/pool/data/ICON/ICON_training/icon\"\n", "expdir = f\"{scratchdir}/exercise_comin/P3\"\n", "%env ICONDIR={icondir}\n", "%env EXPDIR={expdir}\n", "%env SCRATCHDIR={scratchdir}" ] }, { "cell_type": "markdown", "id": "aa3fe6ff-4290-4fb4-b931-86fd217339af", "metadata": {}, "source": [ "Run the setup script which generates the namelists, output directory etc.:" ] }, { "cell_type": "code", "execution_count": null, "id": "bee55ef8-7b81-4401-b6c6-06879da9135f", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [], "source": [ "!bash $HOME/icon-training-scripts/exercise_comin/prepared/prepare_icon_run.sh" ] }, { "cell_type": "markdown", "id": "6f240cb2-0878-47d2-a0b0-c5d64d772a2a", "metadata": {}, "source": [ "As in the previous exercises, we have to append the plugin block to the ICON namelist settings:" ] }, { "cell_type": "code", "execution_count": null, "id": "3973a495-7ac1-4dc2-8b78-4013e7818d0f", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [], "source": [ "with open(f\"{expdir}/NAMELIST_ICON\", 'a') as f:\n", " f.write(f\"\"\"&comin_nml\n", " plugin_list(1)%name = \"comin_plugin\"\n", " plugin_list(1)%plugin_library = \"{icondir}/build/externals/comin/build/plugins/python_adapter/libpython_adapter.so\"\n", " plugin_list(1)%options = \"{home}/icon-training-scripts/exercise_comin/scripts/comin_plugin_P3.py\"\n", "/\n", "\"\"\")" ] }, { "cell_type": "markdown", "id": "83337293-fb02-4358-b3e7-f22ffe3fbc52", "metadata": {}, "source": [ "Submit the job. The job will run for approximately 2:30 minutes.\n", "\n", "
\n", "Important note: We recommend rerunning the entire notebook up to this cell before doing so, to clean up the plugin script and remove any erroneously appended errors.\n", "
" ] }, { "cell_type": "code", "execution_count": null, "id": "f64c37b5-2b5a-47b7-9f7f-029c3f9b6c50", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [], "source": [ "!cd $EXPDIR && sbatch --account=$SLURM_JOB_ACCOUNT --export=ICONDIR $EXPDIR/icon-lam.sbatch" ] }, { "cell_type": "code", "execution_count": null, "id": "990d279f-63a5-43f8-936b-d82db4941e14", "metadata": {}, "outputs": [], "source": [ "!squeue -u $USER" ] }, { "cell_type": "markdown", "id": "fe473a83-2452-49d3-a6c2-3c430cea9ab2", "metadata": {}, "source": [ "Display one of the generated output images:" ] }, { "cell_type": "code", "execution_count": null, "id": "84d54ae0-95a2-4eef-a019-3374d36b1b44", "metadata": {}, "outputs": [], "source": [ "from IPython.display import Image, display\n", "path = f\"{expdir}/pres_sfc_2021-07-14T12:00:00.000.png\"\n", "display(Image(filename=path))" ] }, { "cell_type": "markdown", "id": "fdd32fb4-2435-4299-901c-d5a232afe687", "metadata": {}, "source": [ "
\n", "Solution
\n", "\n", " The resulting plot pres_sfc_2021-07-14T12:00:00.000.png looks like this:
\n", " \n", "
\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "3099312b-470c-4f48-93e8-f00a235f59dc", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "id": "55ab88a8-1a48-4b83-a995-b46066877388", "metadata": { "tags": [] }, "source": [ "

Congratulations! You have successfully completed Exercise 6, Part III.

\n", "\n", "By the way: Further examples of the ICON Community Interface (ComIn) in Jupyter can be found at https://gitlab.dkrz.de/icon-comin/comin-training-exercise. ComIn also comes with several example plugins that demonstrate its usage with plugins written in Fortran and C/C++, which can be found at https://icon-comin.gitlab-pages.dkrz.de/comin/." ] }, { "cell_type": "markdown", "id": "1f5d5129", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "id": "96e76730-628c-48a2-bfd1-295dc121a395", "metadata": {}, "source": [ "## Further Reading and Resources" ] }, { "cell_type": "markdown", "id": "ede1b65e-3c04-4839-a0f8-0cfe9b6ea4ff", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "id": "7923d75d", "metadata": {}, "source": [ "---\n", "\n", "*Author info: Deutscher Wetterdienst (DWD) 2025 :: icon@dwd.de. For a full list of contributors, see CONTRIBUTING in the root directory. License info: see LICENSE file.*" ] } ], "metadata": { "kernelspec": { "display_name": "1 Python 3 (based on the module python3/2023.01)", "language": "python", "name": "python3_2023_01" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.10" } }, "nbformat": 4, "nbformat_minor": 5 }