Meteor
This document is lab notes that succintly describe the experimental setup to run Meteor on a 64 node linux cluster.
Running multiple peers per node, up to ~300 peers
This is not a step-by-step document on how to get Meteor out of CVS and compile it into a Jar.
Make sure that you already have a working Meteor.jar before trying all this out.
Environment
Meteor is written in Java (J2SDK1.4.1) and uses the JXTA 2.2 libraries.
It uses ANT as a build and execution environment.
PREPARE THE Meteor.jar FILE BEFORE SETTING UP THE EXPERIMENT
All files mentioned here can be obtained from the CVS repository for Meteor
Setup
Tools
- VNC client to connect to a cluster using X-window
- ANT to compile, jar, run, and structure directories
- Jxta 2.2 Libraries to write the application
- Distributed Shell to run peers on a cluster
Scripts
Once the Meteor application is packaged into a single Jar, the first thing to do is to prepare the experiment directory where all peers will start from.
Remember that Jxta requires every peer to have a space of its own to save configuration files and cache advertisements (the .jxta directory).
So each peer needs a directory of its own.
Simple bash scripts can be written that can help in the process of deploying the structure and make the experiments repeatable.
In the example below, I briefly describe the major steps to get such a directory structure up :
- Obtain the list of cluster nodes (say in a file named clusterNodes)
- Create directories for each peer one each cluster node
for i in `cat clusterNodes`
do
mkdir XPERIMENTS/$i
mkdir XPERIMENTS/results # where Squid stores its results
for j in 2 3 4 5 #these are for 4 additional peers per node
do
mkdir XPERIMENTS/$i/$j
mkdir XPERIMENTS/$i/$j/results
done
done
- Link the build.xml files (prepared beforehand) to each directory
for i in `cat clusterNodes`
do
ln -s build1.xml XPERIMENTS/$i
ln -s build2.xml XPERIMENTS/$i/2 #Each of these point to different RVs
ln -s build3.xml XPERIMENTS/$i/3
ln -s build4.xml XPERIMENTS/$i/4
ln -s build5.xml XPERIMENTS/$i/5
done
$i is the name of the node on the cluster.
Each buildX.xml might differ from the other by passing different command-line arguments to the peers. In my runs,the differences were essentially, the Rendezvous Peers to connect to, the setting of a peer being a rendezvous as opposed to not being a rendezvous, and the setting of graphics or non-graphics mode.
- You should get a structure similar to the example below for one peer:
XPERIMENTS/
|-- celepharn
| |-- 2
| | |-- build.xml -> ~METEOR/build2.xml
| |-- 3
| | |-- build.xml -> ~METEOR/build3.xml
| |-- 4
| | |-- build.xml -> ~METEOR/build4.xml
| |-- 5
| | |-- build.xml -> ~METEOR/build5.xml
| |
| |-- build.xml -> ~METEOR/build1.xml
...
- Setup the rendezvous peers
- Write a script (say runNode.sh) to start a node
xp=`basename $HOSTNAME .rutgers.edu`
cd XPERIMENTS/$xp; ant run & # Starts the first peer
sleep 20; cd XPERIMENTS/$xp/2/; ant run &; # Starts the second peer
sleep 20; cd XPERIMENTS/$xp/3/; ant run &; # Starts the third peer
sleep 20; cd XPERIMENTS/$xp/4/; ant run &; # Starts the fourth peer
sleep 20; cd XPERIMENTS/$xp/5/; ant run &; # Starts the fifth peer
-
dsh -g $1 -c -d 80 runNode.sh
Here -g is the group file with names of the cluster nodes
-c is for concurrent starts, -d is the delay in seconds
- Once experiments are done all processes have to be terminated and the cache diretory for each peer deleted.
for i in `cat clusterNodes`
do
rsh $i killall java
rm -rf XPERIMENTS/$i/.jxta/cm
for j in 2 3 4 5
do
rm -rf XPERIMENTS/$i/$j/.jxta/cm
done
done
Sequence of operations
- Start the Bootstrap node, that should have a shell interface and a graphical configuration
- Start all subsequent Rendezvous peers that are pointed to in the buil[2-5].xml files (to lighten the load on the Main bootstrap node).
- Run the dsh script on as many nodes as needed using an appropriate (for example .dsh/group/clusterNodesH) filename.
- Wait for the peers to stabilize (sending messages, etc...)
- Type commnads at the jxtashell Prompt line (status, sendmessage, querys, etc...)
- Clean up cache directories and kill all running processes
Command-Line arguments
Meteor accepts runtime command-line arguments to avoid recompiling for some parameters these are:
| Commnad | Operation | Default |
| size | Sets the size of the dimension of the ring | 36 |
| isrv | Sets whether this peer is a Rendezvous Peer | false |
| localport | Sets the local port on which this peer listens on | 9711 |
| boot | Defines the time a node waits for other nodes to respond to its discovery message before declaring that it is alone in the network | 3000 ms |
| rv | When bypassing the graphics configuration window, rv sets the address of the rendezvous that this node will use to connect to | -- |
| nographics | Sets whether this node should run the Shell GUI window or not | true |
| queueing | Use Queueing for messages sent and received, to avoid missing some messages while processing others | false |
| randomid | Choosing a randomized identifier for the node rather than a deterministic one | false |
| chordonly | Selects the services to load and start, when true, is only used to verify that the chord is accurate | false |
| net.jxta.tls.principal | The user name to use for starting this peer | -- |
| net.jxta.tls.password | The password to use for this peer | -- |
Deployment setup
The table below summarizes the command-line parameters set for each different peer type.
These are set in corresponding build.xml files.
| Peer | isrv | chordonly | size | localport | randomid | queueing | nographics | boot | rv |
| BootStrap | true | false | - | 9711 | true | false | false | 5000 | its IP |
| Rendezvous | true | false | - | 9711 | true | false | false | 5000 | BootStraps' IP |
| Peer in XPERIMENTS directory | false | false | - | 9711 | true | false | true | 5000 | BootStraps' IP |
| Peer in XPERIMENTS/2-5 directories | false | false | - | 9712-5 | true | false | true | 5000 | A Rendezvous IP |
Size is not set because when the Squid layer is enabled, the size of the ring
is read from a configuration file part of Squid (conf/squid.txt)
When using only Chord, the size defaults to 36.
Meteor Shell Commands
Once the nodes start, the graphics-enabled peers can execute some jxtashell commands to interact with the system
| status | print the routing table |
| sendmessage | print the routing table of a specific identifier |
| lookup | find the peer that holds information for the identifier passed as argument |
| post | Post a message to the destination id specified using Squid |
| querys | Post a query on the ring |
| publish | Publishes information on a particular node identifier |
| get | Get information from a peer identifier |
Packaging Meteor using Java Network Launching Protocol (JNLP)
Java Web Start is a great tool to launch Java applications from the web.
Here are the steps that I followed and the problems I encountered on the way
- Have a working Jar version of the application
- Create a .jnlp file for the application
- Transform all parts of the code that need a resource (config file, property etc...)
to use the ClassLoader.getResource. This is because JNLP uses Jars only so all additional
resources have to be bundled inside the Jar.
- Make sure that the JXTA URLs are created using the IDFactory.jxtaURL and not "new URL"
- Sign every Jar
- Add the JNLP application mime type to the Web server
Here is a sample JNLP file for METEOR
<?xml version="1.0" encoding="utf-8"?>
<jnlp spec="1.0+" codebase="file:///c:/mydocu~1/jxtalib/"
>
<information>
<title>Meteor App</title>
<vendor>Sun Microsystems, Inc.</vendor>
<description>DHT Application</description>
<description kind="short">
</description>
<offline-allowed/>
</information>
<security>
<all-permissions/>
</security>
<resources>
<j2se version="1.4+"/>
<jar href="Meteor.jar" download="eager"/>
<jar href="cryptix-asn1.jar" download="eager"/>
<jar href="cryptix32.jar" download="eager"/>
<jar href="javax.servlet.jar" download="lazy"/>
<jar href="jaxen.jar" download="eager"/>
<jar href="jdom.jar" download="eager"/>
<jar href="jxta.jar" download="eager"/>
<jar href="jxtaext.jar" download="lazy"/>
<jar href="jxtaptls.jar" download="eager"/>
<jar href="jxtasecurity.jar" download="eager"/>
<jar href="jxtashell.jar" download="eager"/>
<jar href="log4j.jar" download="eager"/>
<jar href="org.mortbay.jetty.jar" download="lazy"/>
<jar href="bcprov-jdk14.jar" download="eager"/>
<jar href="saxpath.jar" download="eager"/>
<jar href="xerces.jar" download="eager"/>
</resources>
<application-desc main-class="meteor.lookup.Loader"/>
</jnlp>
To access a config file using getResource can be done in the following way:
InputStream is = this.getClass().getClassLoader().getResourceAsStream("conf.properties");
BufferedReader lr = new BufferedReader(new InputStreamReader(is));
To load log4j.properties from the JAR can be done like this
PropertyConfigurator.configure(meteor.lookup.Loader.class.getClassLoader().getResource("log4j.properties"));
Use IDFactory.jxtaURL instead of new URL like this
IDFactory.fromURL(IDFactory.jxtaURL(refPeerGroupSpec)));
To sign a jar you need to first create a Key
keytool -genkey -keystore myKeystore -alias myself
Create a self-signed certificate
keytool -selfcert -alias myself -keystore myKeystore
Sign the jar with the certificate
jarsigner -keystore myKeystore test.jar myself
For Jars that are already signed, like bcprov-jdk14.jar, I extracted the files and deleted the META-INF
and re-Jarred it, and then signed it myself.
Try it here
...more to come
All JXTA shell commands can also be used, for example to monitor all peers that have yet connected to the Rendezvous the shell command peers -r sends a discovery request to all peers, and peers prints out a list of peers in the group
|