{ "cells": [ { "cell_type": "markdown", "id": "fa561492", "metadata": { "tags": [] }, "source": [ "
\n", "\n", "

ICON Training - Hands-on Session

\n", "\n", "# Exercise 6.2: ICON ComIn, Part II\n", "\n", "---\n", "\n", "The **ICON Community Interface (ComIn)** enables the integration of **plugins** which is a flexible way to extend the code with your own routines and variables at run-time.\n", "This part continues the practical exercise from the last section: more advanced, more complex features are now added to the Python plugin from “Step 1”:\n", "\n", "* **Step P2**: We learn how to perform data MPI gathering.\n" ] }, { "cell_type": "markdown", "id": "e61c91f4-91e2-4e24-a168-049ee2687e33", "metadata": {}, "source": [ "
\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "001c6eb4-bd00-4d7b-93e8-b62ec7a3703a", "metadata": { "tags": [] }, "source": [ "## Step P2: MPI Gathering - Explanation of the Algorithm\n", "\n", "Using the MPI library for parallel communication is rather complex for newcomers. It requires dealing with the partitioning of data arrays into processes, which means translating between global and process-wise indices. For this reason, rather than explaining this mechanism directly in the ComIn plugin, we begin with a \"dry run\" in this Python-based Jupyter notebook." ] }, { "cell_type": "markdown", "id": "10ae47c2-6f67-4829-91c0-f0b9a6a1a5f1", "metadata": {}, "source": [ "
\n", " Exercise: We simulate how distributed data across MPI processes can be gathered back to a single array. We will do so without using MPI and ComIn. This helps build an intuition for MPI gathering operation in parallel computing.\n", "

To complete this task, please follow the steps outlined in the next cells below.

\n", "
\n" ] }, { "cell_type": "markdown", "id": "5bcfbc6f-11a5-4a83-9d03-6b00b72c52bf", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "id": "59c6ab9f-8c44-4ca4-a2c9-9779c495ac99", "metadata": {}, "source": [ "Consider the following global array:" ] }, { "cell_type": "code", "execution_count": null, "id": "92d6225d-c5a9-43f6-a0b3-5cb364b53768", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "my_array_global = np.array([10, 20, 30, 40, 50, 60, 70, 80, 90, 100])" ] }, { "cell_type": "markdown", "id": "19067d2a-9294-4c7e-a893-1eaef775c06d", "metadata": {}, "source": [ "Assume it's distributed across **2** processes. Therefore we have a local array on each proccess." ] }, { "cell_type": "code", "execution_count": null, "id": "70cab254-0471-4adb-8322-feefa6cfd528", "metadata": {}, "outputs": [], "source": [ "nranks = 2\n", "local_arrays = np.array_split(my_array_global, nranks)\n", "ncells_glb = len(my_array_global)" ] }, { "cell_type": "markdown", "id": "df2dfc7a-f98b-4a63-965f-fc663c08f1d0", "metadata": {}, "source": [ "**Exercise A**: Use an appropriate Python command to determine the global indices that each process owns." ] }, { "cell_type": "code", "execution_count": null, "id": "fd892e1c-e411-4efa-b494-5eeb2d128942", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "2008de14-233e-4ef9-ad62-d3e2da6b3b9e", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [ "hide-input" ] }, "source": [ "
\n", "Solution
\n", "\n", "The global indices \"owned\" by the two processes would be\n", "- `[10, 20, 30, 40, 50]` for process #0\n", "- `[60, 70, 80, 90, 100]` for process #1\n", "\n", "\n", "```python\n", "global_idx = np.array_split(np.arange(ncells_glb), nranks)\n", "```\n", "
\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "7ba8109c-867d-422e-b671-efd979a12e95", "metadata": {}, "source": [ "**Exercise B**: Take the perspective of process 0 and verify that the data it received corresponds to the correct portion of the original global array." ] }, { "cell_type": "code", "execution_count": null, "id": "8c15afb3-bdec-4cc5-88e9-c421ed70bd15", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "0ecfd001-cfaf-4773-bf9c-0ef776a886bb", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [ "hide-input" ] }, "source": [ "
\n", "Solution
\n", "\n", "```python\n", "assert np.all(my_array_global[global_idx[0]] == local_arrays[0]), \"Mismatch for rank 0\"\n", "```\n", "
\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "0c4d5864-ed6d-4a69-9d67-710c9f3a97df", "metadata": {}, "source": [ "**Exercise C**: \"Simulate\" the gather operation with a simple `for i in range(nranks) ...` and confirm that the result matches the original global array." ] }, { "cell_type": "code", "execution_count": null, "id": "d6f10713-7c7a-4889-a9aa-13eee450e9fc", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "382bcff2-777e-4fdf-8f25-a73e2c6f9cf3", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [ "hide-input" ] }, "source": [ "
\n", "Solution
\n", "\n", "```python\n", "gathered_data = [(global_idx[i], local_arrays[i]) for i in range(nranks)]\n", "idx, val = zip(*gathered_data)\n", "\n", "result = np.empty(ncells_glb)\n", "result[np.concatenate(idx)] = np.concatenate(val)\n", "\n", "print(f\"{result=}\")\n", "```\n", "
\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "26da9669-259a-456a-80d9-124bd920031e", "metadata": {}, "source": [ "## Performing an MPI gathering operation in the ComIn plugin\n", "\n", "Let us now apply this gather operation to the actual ComIn Python script, which is attached to the ICON model." ] }, { "cell_type": "markdown", "id": "4b971172-1648-494a-9eb1-67b515e7d68e", "metadata": {}, "source": [ "
\n", " Exercise: \n", "When working with ComIn plugins in ICON, you access variables locally, meaning each MPI process only sees its own subset of data.\n", "To reconstruct a global array, you need to perform a gather operation - similar to what you did in the previous exercise.\n", "\n", "Instead of implementing the gather functionality manually, it's better (and simpler) to use the gather functionality provided by the MPI library.\n", "\n", "In this task, you will run a pre-written ComIn plugin scripts/comin_plugin_P2.py which demonstrates the same concept you explored above, now in an actual MPI-based plugin. All parts of this script are already given:\n", "\n", "- It accesses the variable pres_sfc from ICON.\n", "\n", "- It retrieves the host MPI communicator using the mpi4py Python package.\n", "\n", "- It gathers the distributed data array on rank 0.\n", "
\n" ] }, { "cell_type": "markdown", "id": "78382d84-3f42-4119-881f-f18d052e0455", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "### Running the ICON model" ] }, { "cell_type": "code", "execution_count": null, "id": "b1f32033-ee31-4d2f-b960-923e02b7328c", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [], "source": [ "import os\n", "import subprocess\n", "\n", "user = os.environ['USER']\n", "home = os.environ['HOME']\n", "scratchdir = f\"/scratch/{user[0]}/{user}\"\n", "icondir = f\"/pool/data/ICON/ICON_training/icon\"\n", "expdir = f\"{scratchdir}/exercise_comin/P2\"\n", "\n", "%env ICONDIR={icondir}\n", "%env EXPDIR={expdir}\n", "%env SCRATCHDIR={scratchdir}" ] }, { "cell_type": "markdown", "id": "bbff685b-b019-4e90-923c-28ca1fe0eaca", "metadata": {}, "source": [ "Run setup script" ] }, { "cell_type": "code", "execution_count": null, "id": "ee967cbc-91ef-4de7-b1a0-67fdc42c8b10", "metadata": {}, "outputs": [], "source": [ "!bash $HOME/icon-training-scripts/exercise_comin/prepared/prepare_icon_run.sh" ] }, { "cell_type": "code", "execution_count": null, "id": "056d6c1f-47df-4c49-829b-f6c730556735", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [], "source": [ "# Append plugin block\n", "with open(f\"{expdir}/NAMELIST_ICON\", 'a') as f:\n", " f.write(f\"\"\"&comin_nml\n", " plugin_list(1)%name = \"comin_plugin\"\n", " plugin_list(1)%plugin_library = \"{icondir}/build/externals/comin/build/plugins/python_adapter/libpython_adapter.so\"\n", " plugin_list(1)%options = \"{home}/icon-training-scripts/exercise_comin/scripts/comin_plugin_P2.py\"\n", "/\n", "\"\"\")" ] }, { "cell_type": "code", "execution_count": null, "id": "7ee49579-5b38-45b9-98d7-ef5a1a059320", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [], "source": [ "# Submit job\n", "!cd $EXPDIR && sbatch --account=$SLURM_JOB_ACCOUNT --export=ICONDIR $EXPDIR/icon-lam.sbatch" ] }, { "cell_type": "markdown", "id": "26b02bdd-3407-455d-81e3-6fb6ab21496a", "metadata": {}, "source": [ "---" ] }, { "cell_type": "code", "execution_count": null, "id": "1eb17cdc-abef-4b98-9545-4792c0051b11", "metadata": {}, "outputs": [], "source": [ "!squeue -u $USER" ] }, { "cell_type": "markdown", "id": "3099312b-470c-4f48-93e8-f00a235f59dc", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "id": "55ab88a8-1a48-4b83-a995-b46066877388", "metadata": { "tags": [] }, "source": [ "

Congratulations! You have successfully completed Exercise 6, Part II.

" ] }, { "cell_type": "markdown", "id": "1f5d5129", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "id": "96e76730-628c-48a2-bfd1-295dc121a395", "metadata": {}, "source": [ "## Further Reading and Resources" ] }, { "cell_type": "markdown", "id": "ede1b65e-3c04-4839-a0f8-0cfe9b6ea4ff", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "id": "7923d75d", "metadata": {}, "source": [ "---\n", "\n", "*Author info: Deutscher Wetterdienst (DWD) 2025 :: icon@dwd.de. For a full list of contributors, see CONTRIBUTING in the root directory. License info: see LICENSE file.*" ] } ], "metadata": { "kernelspec": { "display_name": "0 Python 3 (based on the module python3/unstable", "language": "python", "name": "python3_unstable" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.10" } }, "nbformat": 4, "nbformat_minor": 5 }