Distributed Dynamic Data-Management

Manish Parashar & James C. Browne
parashar@caip.rutgers.edu; browne@cs.utexas.edu

Overview

This research addressed management of distributed data-structures that can change in their size, information content, and interactions with other distributed data-structures during their lifetime. Such distributed, dynamic data-structures underly (among other application areas) adaptive scientific and engineering simulations, multiresolution databases, and interactive scientific visualization systems. A key requirement for the efficiency of these applications is the preservation of application locality requirements in-spite of its dynamics. The research has lead to the development of the Hierarchical Distributed Dynamic Array (HDDA). HDDA is a data-management system that provides pure array access semantics to hierarchical, dynamic and physically distributed data. It is a hierarchical array in that each element of the HDDA can also be an HDDA; it is a dynamic array in that it can grow and shrink dynamically. HDDA objects encapsulate distribution, dynamic load balancing, communications, and consistency management. The key component the HDDA is a hierarchical global index space which is a one-dimensional, multilevel index set that can expand and contract dynamically. The index space is derived from the application domain using Space-filling Mappings and encodes application locality requirements. Further, it has the ability to preserve this locality through expansion and contraction. This index space is used by the HDDA for application domain partitioning, as a global name space for name resolution and for communication scheduling. HDDA storage is implemented as an Extendible Hash Table that uses the index space as its hash-key space. HDDA is being successfully used for data-management in the different application domains.

Application Domains

  1. Adaptive Simulation: HDDA has been used to provide dynamic data-management support for parallel adaptive techniques for scientific and engineering simulations. These include parallel hierarchical adaptive mesh-refinement and multigrid techniques, parallel hp-adaptive finite-element techniques, and adaptive particle methods. Using HDDA abstractions has typically reduced the coding complexity of the dynamically adaptive solution methods by several orders of magnitude; while the efficiency of the resulting implementation is comparable, and frequently superior to implementations coded directly in C or Fortran.
  2. Multiresolution Databases: HDDA technology is currently being used as a basis for a multiresolution database for managing satellite data. The requirement of the database is the ability to store, retrieve and transform distributed satellite data at desired resolutions.
  3. Interactive Visualization & Computation Steering: HDDA data-objects are being used to drive interactive visualization and experimentation. The idea is to extend HDDA base objects with appropriate visualization and interaction capabilities.
Manish Parashar, parashar@caip.rutgers.edu
CAIP Center & Department of Electrical & Computer Engineering
Rutgers, The State University of New Jersey