Pawn & IPARS

Structure of the Files

obtained using `tree -d PAWN | sed -e '/CVS/d'

PAWN
|-- AppServer
|   `-- discover
|-- Client
|-- IparsFactory
|   |-- ipars
|   `-- ptolemy
|       `-- plot
|-- OptimizationPeer
|-- advertisements
|-- messages
|-- misc
|-- services
|   |-- appex
|   |-- chat
|   |-- locking
|   |-- monsteer
|   |-- optimization
|   `-- presence
`-- utilities

The IparsFactory contains the IparsErrorEstimation function in C. It uses JNI to invoke the error estimation from Java. The IparsFactory directory therefore contains a libiparserror.so.

The Optimization functionality, in this case the Very Fast Simulated Annealing (VFSA) are in the PAWN/services/optimization directory. It uses the Fortran gvfsa.f routine, wrapped in the VFSA.c and in turn wrapped in VFSA.java for invocation by the OptimizationPeer which implements the OptimizationListener (using the OptimizationService).

The executable for IPARS is invoked in the class IparsFactoryMain by the IparsFactoryService.

It is invoked using a full path to the executable, so the IparsFactory has to reside on the same machine as the executables of IPARS.

The files needed by IPARS to run are:

INPUT FILE :		gv
PERMEABILITY FILE : spe9_2700
WELL FILE :			mywell.inc

POROSITY FILE :

From ipars.f we read:

IF THE RESPONSE FILE IPARS.IN EXISTS IN THE CURRENT DIRECTORY,
FILE NAMES WILL BE READ FROM THAT FILE RATHER THAN THE TERMINAL.
THIS OPTION PROVIDES A BATCH MODE FOR THE SIMULATOR.

How to start the application and optimization process

Start the peers on the same or different hosts on the LAN.
For example:

A- ant runappserver  ON grid1.rutgers.edu
B- ant runipars      ON frea.rutgers.edu
C- ant runvfsa       ON asfaloth.rutgers.edu
D- ant runclient     ON bach.rutgers.edu

A- The AppServer starts the Discover server to forward all application object requests and responses to the clients.

B- The IparsFactory has to be started on a host that has access to the filesystem where the iaprs binary resides and define the full path to this binary.

C- The VFSA optimization has to start on a host capablie of running the VFSA binary.

D- The client can be started on any other peer. While all other components have to be started on Linux machines, the client can be started on a host running Windows.

The picture above shows the entire process from a high-level view. Each arrow is an interaction among the peers shown. The order is from an overall application optimization perspective than an actual sequence of deployment.

Structure of the Files obtained

The IPARS source files can be obtained from CVS (/caip/u2/cvsroot)

cvs co IPARS-DEMO-2002

This will consist in a LARGE directory of over 80 MB including a redundant directory that once removed brings the total directory size to 54MB. the structure of this directory is

IPARS-DEMO-2002
|--DIOS
|--IPARSv2
|--MALGO-GVFSA
|--VFSA
|--VFSA_Vincent

DIOS contains the source of the DIOS library created by Rajeev Muralidhar (rajeevdm@caip) and Manish Parashar (parashar@caip).

DIOS stands for Distributed Interactive Object Substrate. It is an application toolkit used to wrap application objects and export them to a Middleware server for Expert web-based collaborations. In other words, it translates Object or Non-Object Oriented scientific application functionalities into an intermediary format understood by the DISCOVER middleware for web-based collaborations. Currently DIOS is successfully used with IPARS, RM3D and the Wave Transport simulation application.

Although DIOS does not change the application itself, it sits a layer above of the application and requires that the application exports the definition of the objects that it wants controlled online, as well as open its input/output controls for dynamic request and response monitoring and steering. This incurrs a slight application modification that makes the main source tree of an application slightly different from the main trunk.

That is why the version of IPARS present here, that includes the DIOS wrapper controls, may be different from future version of IPARS as under development at the University of Texas at Austin. In order to keep the versions in synch, the DIOS wrapped version has to be regularly updated.

The MALGO-GVFSA directory contains code provided by Malgoretsza Pescenzka, regarding the Optimization algorithm based on the Very Fast Simulated Annealing Algorithm designed by Mrinal Sen at UT Austin. This algorithm optimizes IPARS and therefore requires input and output capabilities between IPARS and the VFSA algorithm. This is achieved when IPARS returns from a computation for specific input well parameters, feeding the optimization algorithm with its return value, that VFSA will optimize and return an optimized input parameters to IPARS for a new run. This process is repeated until the overall optimization is achieved. In our case, the optimization involved the minimization of the Economic Value (cost of oil extraction). The results from IPARS are normalized using an error calculation routine.

The other VFSA and VFSA_Vincent are similar to the above directory.
In the initial version of VFSA provided by Malgo, the VFSA fortran code was the "Master" application, meaning that it defined the main sequence of operations, invoking the IPARS application, normalizing its return, optimizing the revenue, and starting new applications with optimal parameters. In the PAWN version, this process is left to the dynamic discovery of peers. The two entities are no more bound by a tight sequence, and there is no "Master" anymore. Each application, IPARS and the Optimization are separate and upon discovering each other will start the optimization process.

IPARSv2/
|-- MACE
|   |-- arch_makes
|   |-- bin
|   |-- include
|   |-- lib
|   |-- src
|   |   |-- macesrc
|   |   `-- mblibsrc
|   `-- utilities
|       |-- iparsdiscover
|       |   `-- Save
|       |-- mblk_mace
|       `-- mblk_make
|-- MBLib
|   |-- lib
|   `-- src
|-- air
|-- airv2
|-- black
|-- blackn
|-- data
|   `-- dgimpesdata
|       |-- Example1
|       `-- Example2
|-- dgimpes
|   `-- dg3d
|       |-- lib
|       `-- src
|           |-- dgb_solver_iter
|           |-- dgf_fem
|           |-- dgi_io
|           |-- dgm_mesh_mod
|           |-- dgp_conv_diff
|           |-- dgu_utilities
|           `-- include
|-- dgmisci
|   `-- dgsrc
|-- doc
|   `-- models
|       |-- air
|       |   `-- figs
|       |-- black
|       |-- hydroe
|       `-- trchem
|-- drive
|-- exec
|-- hydro
|-- hydroe
|-- hydros
|-- input
|-- java
|   `-- save
|-- lapack
|   `-- blas
|-- make
|   `-- modular
|-- manishwork
|-- mblk
|-- memman
|-- mmodel
|-- parall
|-- print
|-- single
|-- singlei
|-- size
|   `-- modular
|-- solve
|   |-- gmres
|   |-- lsor
|   |-- pcg
|   |-- util
|   `-- ygmres
|       `-- mlilu.IPARS
|-- trchem
|-- util
|-- virajwork
|-- visual
|   |-- scripts
|   `-- tecplot
|-- wells
`-- work
    `-- Run

Modifications to IPARSv2 from Hector Klie October 8th 2003

To make this version of IPARSv2 DIOS enabled the following modifications needed to be made

+--IPARSv2/
Import the java directory

+--IPARSv2/MACE/utilities
Import the iparsdiscover directory
Import the discover.mak file

+--IPARSv2/work/Makefile
ADD include ../MACE/utilities/discover.mak
ADD include ../MACE/utilities/visual.mak
Add DISCOVER_OBJ and DISCOVER_LIB to the list of OBJS and LIBS

+--IPARSv2/work/ipars.siz
Modifiy DISCOVER "C" to DISCOVER "" to enable DISCOVER
Add JAVAVIS "" to enable visualisation of Java plots

+--IPARSv2/make/modular/visual.mak
ADD
javah.h: $(SORCJ)javah.h
        $(COPYIT)

This version of IPARS did not Link when using GCC 3.2.20, Rolling back
to 2.96 worked.
This needs to be changed in the files:

IPARS-DEMO-2002/IPARSv2/make/modular/lnx.mak
IPARS-DEMO-2002/DIOS/arch_makes/make.defn.linux

To change the Discover server address modify the file 

IPARS-DEMO-2002/MACE/utilities/iparsdiscover/discovermain.dcpp

to point the servername to the appropriate server

Setup as of October 21st 2003

There are currently two different version of the PawnIpars application. One is used for development, the Main trunk in CVS of this version only compiles with JXTA 2.1. The ~/PAWN directory is currently checked out from the CVS Branch jxta-2-0 and compiles with JXTA-2.0 libraries. The other version resides under ~/WORK/PAWN and has modifications from the main cvs trunk that have not been comitted, it is not part of a branch and is used for the DEMO of the AORO process. It compiles using the JXTA libraries release 65e. This version is becoming more and more different from the production one, and merging time is nearing.

Main Differences between PAWN and WORK/PAWN

Among the most important differences are the message queue and transmission handling, where the new version makes strong use of the utilities classes PMessageSender,PMessageReceiver in IparsFactoryMain and OptimizationPeer, the old version is still using the utilities class PipeListener that only parses the incoming message by class type for proper handling.
There is a MessageHandler that is a precursor of the PMessageSender and PMessageReceiver that handles the outgoing messages enqueued in a messagequeue defined in the IparsFactoryMain and OptimizationGUI classes.
DONE on 10/22/03 <= Other minor differences are the location setting of the frames in WORK/PAWN that is meant to be demo'd. These need to be merged with the new version.

DONE on 10/22/03 <= Also, the IparsFactory frame has additional fields and buttons to set the directory and executable of the IPARS simulation to avoid recompiling every time this location has to change, making it easier to debug.

The work for Decadence, the JXTA_MPI project is not available in the WORK/PAWN repositoy.
The new version of PAWN contains a ConfigurePlatform class that is used to bypass the graphical configuration screen of JXTA. THis was used to deploy peers during the evaluation of Decadence. The Distributed Shell Binaries were modified to allow for a delayed startup of the command on all the nodes. To disable graphics an argument has to be passed in the build.xml file

<jvmarg value="-Dnographics=true"/>

Similarly an argument allows selecting the services to load dynamically Where the sequence of bit is Master,Worker,Resource Manager,Result Collector, Work Dispatcher, Partitionner,
[M][W][RM][RC][WD][P]

<jvmarg value="-Dservices=101001"/>

For more information on Decadence visit http://www.caip.rutgers.edu/~vincentm/DecadenceProject/decadnce/Decadence.html

Starting the JacORB, Globus Middleware

cd DiscoverPortal/WEB-INF/classes
$JABCORB_BIN/trader Trader.ior -d db/
$TOMCAT_HOME/startup.sh
grid1.rutgers.edu:8080/DISCOG/Portal/index.hml

Setup as of February 2004

The two versions of Pawn_Ipars (JXTA 2.0 and JXTA 2.2) have been merged and need some testing to validate the proper functioning of the whole process.
The files are in the Main CVS branch PAWN in u2
They compile with JXTA 2.2.

Wolfgang's visit (July 5th to July 7th 2004)

July 4th

The PAWN_IPARS source code for this work is retrieved from the Main branch of CVS, and compiles with JXTA 2.2. Porting it to JXTA 2.3 will require significant updates and can be conducted later.

The plan is to focus on deploying the potential of this system to run IPARS simulations with different optimizations in Parallel.
This setup should show significant performance improvement over running IPARS simulations in batch processing mode.
In this process the issues that need to be addressed are

Create an Optimizer interface that all optimization algorithms *have* to comply to
Establish unique end-to-end communication links between the IPARSFactory and every IPARS run that it starts
Create a GUI option for selecting between available Optimizers
Every communication link has to be uniquely identified
- Between IparsFactory and Each Optimizer
- Between IparsFactory and Each IPARS run

The communication model implemented in the system is depicted on the figure below:

The system contains:

Collection of Optimizers
Collection of IPARS instances
ONE OptimizationService
ONE IPARSFactory

Each Optimizer is uniquely linked to the IPARSFactory through a virtual link that goes through the Optimization service. This link has a unique Identifier that is determined at the optimizer creation time. The Optimization Service and IPARS Factory need to act as Message Dispatchers for each individual link.

Communications:
IPARS instances connect to the DISCOVER server through AppServer/discover/Daemon --> DiscoverApp.connectNotify(appcount)
connectNotify creates a PipeCommServer Bidirectional pipe for communication with the IPARS Factory

07/05/2004- Modified Design to enable multiple Optimization-Driven applications

The key actors in this new design are:

OptimizationService:
- Send Guess
- Notifies Optimizers when an IPARS completes a run (total revenue)
OptimizerInstance:
- Handles the optimization process
OptimizatonGUI:
- Displays GUI for selecting a new Optimizer
- and setting Optimizer Parameters
IparsFactoryMain:
- starts an IPARS instance
- checks if an IPARS return for a specific input had already been computed
AppConnection:
- Notifies optimization service when IPARS terminates

In this new design, the IPARS Factory has the unique task of starting IPARS Instances (previously checking if a run with a certain input had been generated).
The return is then passed on to the OptimizationService which is responsible of notifying all Optimizers of a new revenue for a specific input.
The optimizer instances that match that return can then produce a new guess and generate a new IPARS run through the IPARS Factory

Pawn Javadocs available here

07/08/2004-PAWN version 2

This new version of PAWN has the Optimizer Instances control the request for IPARS evaluations.
Every Optimizer instance is composed of a unique Evaluator and an optimizer algorithm

Evaluator handles communication with the IPARSFactory. It specifies the input that should be evaluated
Optimizer handles receiving returns from the Evaluator and generating new evaluation requests

Where previously the OptimizerInstance was performing the task of both, the Evaluator and Optimizer, hooking into the algorithm at every return from IPARS, the control is now in the hands of the Evaluator that can send as many requests as the optimizer algorithm needs.
This new version is in my CVS under the tag PAWNv2

07/29/2004

IPARSFactory's JobManager is now in place and can execute parallel runs of IPARS on multiple nodes (see JobManager, Jobs, Resources).
This works as follows:

Upon receiving an input from the OptimizationService, IPARSFactory adds this input as a new JobUnit to the JobManager's queue of pending_jobs
The JobManager pops a JobUnit from its queue and searches for an available resource on which to run the Job. If none is found, the job is put on hold until a resource becomes available.
The Job is removed when an IPARS run terminates (IparsFactoryMain::receivedRevenue), the resource on which the job was running is made available for running a new Job

The number of Jobs that can run on a resource is configurable, so is the list of available resources.

Every query from the OptimizationService is now identified using a unique identifier Key rather than the input, this decouples the Job definition from the Job identification, allowing jobs to be as complex as Job Description Documents.

The Optimization service sends the input along with the unique identifier to the IparsFactory.

When the computation ends, the IparsFactory sends back the results with the associated Key to the OptimizationService.

The Optimizationservice then notifies the Evaluator, that in turn notifies the optimizer of the result.

Using a key also enables aggregating Job requests, as identical inputs are not computed every time but only once. Every interested peer has a unique key along with the input that allows it to be notified upon receiving the response to the computation for a specific input. This aggregation is performed in the JobManager of the IparsFactory.