Distributed Dynamic Data-structures for Parallel Hierarchical AMR

• Transparent access to Scalable Distributed Dynamic Arrays/Grids/Grid-Hierarchies

• Multi-grid/Line-Multi-grid support within AMR

• Shadow grid hierarchy for on-the-fly error estimation

• Automatic dynamic partitioning and load distribution

• Locality, Locality, Locality !!!

• Scalability, Portability, Performance

Programming Interface

• Application Objects => Abstract Data Types

• Coarse grained Single Program Multiple Data (SPMD) form of data parallelism

• C++ driver

• declares and defines computational domain and application variables in terms of GrACE programming abstractions


• defines overall structure of the AMR algorithms

• FORTRAN/FORTRAN 90/C computational kernels

• defined on regular arrays

• Grid Hierarchy Abstraction

• Template for the distributed adaptive grid hierarchy

• Grid Function Abstraction

• Application fields defined on the adaptive grid hierarchy

• Grid Geometry Abstraction

• High-level tools  for addressing regions in the computational domain

• Data-parallel forall operator

• Parallel operation for all grid components at a particular time step & level
  forall (gf, time, level, component)
     Call FORTRAN Subroutine...
  end_forall

• Refining and regriding

• Encapsulates: Generation of refined grids, Redistribution, Load-balancing, Data-transfers, Interaction schedules
  Refine(GH, Level, BBoxList)
  RecomposeHierarchy(GH)

• Prolongations and Restrictions

• User defined prolong/restrict routines for each GridFunction
  foreachGF(GH, GF, DIM, GFType)
     SetProlongFunction(GF, Pfunc);
     SetRestrictFunction(GF, Rfunc);
  end_forallGF

Checkpoint/Restart/Rollback

• Each Grid Function can be individually selected or deselected for check-pointing

• Checkpoint files independent of number of processors

Integrate IO/Visualization

• Allocates a hierarchy of IO/Viz servers

• User-defined IO routine

• Existing Interfaces for HDF/IEEEIO

• 1-D Slices via Xgraph

Multi-grid Support

• Allow multi-grid cycling at each level of the AMR hierarchy