Skip to content

Backend overview

Backend work is split into multiple repositories, each having their quirks to work with. In particular,

  • NDSL - Pulls everything together and exposes "the DSL". In particular, orchestration code lives in this repository.
  • GT4Py - The component that defines the frontend, has all the intermediate representations (IR) and dispatches to multiple backends.
  • DaCe - Data-driven optimization framework used in GT4Py as performance backend. Full program optimizer driving orchestration.

NDSL is then used in the following repositories:

  • PyFV3 NDSL port of the FV3 dynamical core.
  • pace combines the PyFV3 port with PySHiELD physics, a DSL port of the SHiELD physics.

Backend workflows

There's two main workflows for the backend depending on which part model code is targeted for transpiling. The backend can either work

  • on a stencil level, known as GridTools or stencil workflow
  • or on the full program, known as orchestration workflow.

The orchestration workflow wraps the stencil workflow and extends transpiling to the python code in-between the stencils. This allows for more analysis (e.g. in the DaCe backends) and lowers that launch overhead for (small) stencils.

Stencil workflow diagram stencil workflow

Orchestration workflow diagram orchestration workflow

GT4Py / DaCe bridge

DaCe provides two high-performance backends in the GT4Py backend; one for CPU, and one for GPU. The code that "converts" GT4Py's "Optimization IR" (OIR) to DaCe SDFGs is commonly know as "the bridge", see GT4Py/Dace bridge.

Schedule tree

The schedule tree is the code representation used to do macro-level optimization like loop re-ordering and merging. Further reading:

Orchestration

System that bring stencils and python together to open up whole program optimization. Required frontend/backend system to enable the most potent wide-context optimization: thoughts, issues and details.