Meteor

This document is lab notes that succintly describe the experimental setup to run Meteor on a 64 node linux cluster.
Running multiple peers per node, up to ~300 peers
This is not a step-by-step document on how to get Meteor out of CVS and compile it into a Jar.
Make sure that you already have a working Meteor.jar before trying all this out.

Environment

Meteor is written in Java (J2SDK1.4.1) and uses the JXTA 2.2 libraries.

It uses ANT as a build and execution environment.

PREPARE THE Meteor.jar FILE BEFORE SETTING UP THE EXPERIMENT

All files mentioned here can be obtained from the CVS repository for Meteor

Setup

Tools

  • VNC client to connect to a cluster using X-window
  • ANT to compile, jar, run, and structure directories
  • Jxta 2.2 Libraries to write the application
  • Distributed Shell to run peers on a cluster

Scripts

Once the Meteor application is packaged into a single Jar, the first thing to do is to prepare the experiment directory where all peers will start from. Remember that Jxta requires every peer to have a space of its own to save configuration files and cache advertisements (the .jxta directory). So each peer needs a directory of its own.
Simple bash scripts can be written that can help in the process of deploying the structure and make the experiments repeatable. In the example below, I briefly describe the major steps to get such a directory structure up :
  • Obtain the list of cluster nodes (say in a file named clusterNodes)
  • Create directories for each peer one each cluster node
    for i in `cat clusterNodes`
      do
    	 mkdir XPERIMENTS/$i
    	 mkdir XPERIMENTS/results # where Squid stores its results
    	 for j in 2 3 4 5      #these are for 4 additional peers per node
    	   do
    	      mkdir XPERIMENTS/$i/$j
    	      mkdir XPERIMENTS/$i/$j/results
    	   done
      done
    
  • Link the build.xml files (prepared beforehand) to each directory
    for i in `cat clusterNodes` 
      do 
         ln -s build1.xml XPERIMENTS/$i
         ln -s build2.xml XPERIMENTS/$i/2  #Each of these point to different RVs
         ln -s build3.xml XPERIMENTS/$i/3
         ln -s build4.xml XPERIMENTS/$i/4
         ln -s build5.xml XPERIMENTS/$i/5
      done
    
    $i is the name of the node on the cluster.
    Each buildX.xml might differ from the other by passing different command-line arguments to the peers. In my runs,the differences were essentially, the Rendezvous Peers to connect to, the setting of a peer being a rendezvous as opposed to not being a rendezvous, and the setting of graphics or non-graphics mode.
  • You should get a structure similar to the example below for one peer:
    XPERIMENTS/
      |-- celepharn
      |   |-- 2
      |   |   |-- build.xml -> ~METEOR/build2.xml
      |   |-- 3
      |   |   |-- build.xml -> ~METEOR/build3.xml
      |   |-- 4
      |   |   |-- build.xml -> ~METEOR/build4.xml
      |   |-- 5
      |   |   |-- build.xml -> ~METEOR/build5.xml
      |   |   
      |   |-- build.xml -> ~METEOR/build1.xml
    ...
    
  • Setup the rendezvous peers
  • Write a script (say runNode.sh) to start a node
    xp=`basename $HOSTNAME .rutgers.edu`
    cd XPERIMENTS/$xp; ant run &   # Starts the first peer
    sleep 20; cd XPERIMENTS/$xp/2/; ant run &; # Starts the second peer
    sleep 20; cd XPERIMENTS/$xp/3/; ant run &; # Starts the third peer
    sleep 20; cd XPERIMENTS/$xp/4/; ant run &; # Starts the fourth peer
    sleep 20; cd XPERIMENTS/$xp/5/; ant run &; # Starts the fifth peer
    
  • dsh -g $1 -c -d 80 runNode.sh
    
    Here -g is the group file with names of the cluster nodes
    -c is for concurrent starts, -d is the delay in seconds

  • Once experiments are done all processes have to be terminated and the cache diretory for each peer deleted.
    for i in `cat clusterNodes`
      do
         rsh $i killall java  
         rm -rf XPERIMENTS/$i/.jxta/cm
         for j in 2 3 4 5
           do
             rm -rf XPERIMENTS/$i/$j/.jxta/cm
           done
      done
    

Sequence of operations

  • Start the Bootstrap node, that should have a shell interface and a graphical configuration
  • Start all subsequent Rendezvous peers that are pointed to in the buil[2-5].xml files (to lighten the load on the Main bootstrap node).
  • Run the dsh script on as many nodes as needed using an appropriate (for example .dsh/group/clusterNodesH) filename.
  • Wait for the peers to stabilize (sending messages, etc...)
  • Type commnads at the jxtashell Prompt line (status, sendmessage, querys, etc...)
  • Clean up cache directories and kill all running processes

Command-Line arguments

Meteor accepts runtime command-line arguments to avoid recompiling for some parameters these are:

CommnadOperationDefault
sizeSets the size of the dimension of the ring36
isrvSets whether this peer is a Rendezvous Peerfalse
localportSets the local port on which this peer listens on9711
bootDefines the time a node waits for other nodes to respond to its discovery message before declaring that it is alone in the network3000 ms
rvWhen bypassing the graphics configuration window, rv sets the address of the rendezvous that this node will use to connect to--
nographicsSets whether this node should run the Shell GUI window or not true
queueingUse Queueing for messages sent and received, to avoid missing some messages while processing othersfalse
randomidChoosing a randomized identifier for the node rather than a deterministic onefalse
chordonlySelects the services to load and start, when true, is only used to verify that the chord is accuratefalse
net.jxta.tls.principalThe user name to use for starting this peer--
net.jxta.tls.passwordThe password to use for this peer--

Deployment setup

The table below summarizes the command-line parameters set for each different peer type.
These are set in corresponding build.xml files.

Peerisrvchordonlysizelocalportrandomidqueueingnographicsbootrv
BootStraptruefalse-9711truefalsefalse5000its IP
Rendezvoustruefalse-9711truefalsefalse5000BootStraps' IP
Peer in XPERIMENTS directoryfalsefalse-9711truefalsetrue5000BootStraps' IP
Peer in XPERIMENTS/2-5 directoriesfalsefalse-9712-5truefalsetrue5000A Rendezvous IP

Size is not set because when the Squid layer is enabled, the size of the ring is read from a configuration file part of Squid (conf/squid.txt) When using only Chord, the size defaults to 36.

Meteor Shell Commands

Once the nodes start, the graphics-enabled peers can execute some jxtashell commands to interact with the system
statusprint the routing table
sendmessageprint the routing table of a specific identifier
lookupfind the peer that holds information for the identifier passed as argument
postPost a message to the destination id specified using Squid
querysPost a query on the ring
publishPublishes information on a particular node identifier
getGet information from a peer identifier

Packaging Meteor using Java Network Launching Protocol (JNLP)

Java Web Start is a great tool to launch Java applications from the web.
Here are the steps that I followed and the problems I encountered on the way
  • Have a working Jar version of the application
  • Create a .jnlp file for the application
  • Transform all parts of the code that need a resource (config file, property etc...) to use the ClassLoader.getResource. This is because JNLP uses Jars only so all additional resources have to be bundled inside the Jar.
  • Make sure that the JXTA URLs are created using the IDFactory.jxtaURL and not "new URL"
  • Sign every Jar
  • Add the JNLP application mime type to the Web server
Here is a sample JNLP file for METEOR
<?xml version="1.0" encoding="utf-8"?> 
<jnlp spec="1.0+" codebase="file:///c:/mydocu~1/jxtalib/"
     > 
<information> 
	<title>Meteor App</title> 
	<vendor>Sun Microsystems, Inc.</vendor> 
	<description>DHT Application</description> 
	<description kind="short">
	   </description> 
	<offline-allowed/> 
</information> 

<security> 
   <all-permissions/> 
</security> 

<resources> 
	<j2se version="1.4+"/> 
	<jar href="Meteor.jar" download="eager"/> 
	<jar href="cryptix-asn1.jar" download="eager"/>
	<jar href="cryptix32.jar" download="eager"/>
	<jar href="javax.servlet.jar" download="lazy"/>
	<jar href="jaxen.jar" download="eager"/>
	<jar href="jdom.jar" download="eager"/>
	<jar href="jxta.jar" download="eager"/>
	<jar href="jxtaext.jar" download="lazy"/>
	<jar href="jxtaptls.jar" download="eager"/>
	<jar href="jxtasecurity.jar" download="eager"/>
	<jar href="jxtashell.jar" download="eager"/>
	<jar href="log4j.jar" download="eager"/>
	<jar href="org.mortbay.jetty.jar" download="lazy"/>
   <jar href="bcprov-jdk14.jar" download="eager"/>	
	<jar href="saxpath.jar" download="eager"/>
	<jar href="xerces.jar" download="eager"/>
</resources> 

<application-desc main-class="meteor.lookup.Loader"/> 
	
</jnlp> 
To access a config file using getResource can be done in the following way:
InputStream is = this.getClass().getClassLoader().getResourceAsStream("conf.properties");
BufferedReader lr = new BufferedReader(new InputStreamReader(is));
To load log4j.properties from the JAR can be done like this
PropertyConfigurator.configure(meteor.lookup.Loader.class.getClassLoader().getResource("log4j.properties"));
Use IDFactory.jxtaURL instead of new URL like this
 IDFactory.fromURL(IDFactory.jxtaURL(refPeerGroupSpec)));
 
To sign a jar you need to first create a Key
 keytool -genkey -keystore myKeystore -alias myself 
Create a self-signed certificate
keytool -selfcert -alias myself -keystore myKeystore 
Sign the jar with the certificate
jarsigner -keystore myKeystore test.jar myself 
For Jars that are already signed, like bcprov-jdk14.jar, I extracted the files and deleted the META-INF and re-Jarred it, and then signed it myself.


Try it here


...more to come

All JXTA shell commands can also be used, for example to monitor all peers that have yet connected to the Rendezvous the shell command peers -r sends a discovery request to all peers, and peers prints out a list of peers in the group