Grid Weka on EC2

Weka is a data mining and data discovery tool. We have installed Weka as standalone software on a Amazon EC2 node already. Refer to this links for the past articles on Weka on EC2.

Weka Data Mining on EC2 - install
Weka Data Mining on EC2 - testing

In the posting roadmap I mentioned we would looking at some Grid or Web aware versions of Weka. Originally Weka was developed for consumption within a closed group. Individual researchers could run their data mining either on their own workstation or on a server.

Outline:

Grid Weka was developed out of the University College, in Dublin, Ireland. It has additional java code to allow Weka to offload various processing steps to both local and remote servers.

Install:

The Grid Weka HOWTO guide was good. It assumes that you know how to install Java and setup the Java environment correctly though.

1. Install Java. Get the latest JDK 1.5 or higher from java.sun.com
2. Download Grid Weka
3. Place the weka.jar file in an appropriate location.
   
4. Make sure your JAVA_HOME environment variable is set.
5. Create and edit a file called .weka-parallel and place in users home directory.
   
6. Run the GridWeka Servers using command: java -classpath /yourpath/weka.jar weka.core.DistributedServer yourport &
7. To access the remote servers you will need to open the port in your firewall.

Results:

Grid Weka works as expected. I haven't tested the real potential benefit of using remote servers to run multiple classifications in parallel. I will and post the results in a future article.

The duration of the classifications were longer using the remote servers than running on a single dedicated process on a single server. Most of the time spent though was in network traffic.
Once again the poor network performance between EC2 nodes is a killer for network intensive applications.

I found a Java execution/library, JFS which uses the idea of Parallel streams. I was able to reduce the network time by around 20%.

This suggests that the Grid Weka would benefit from being compression aware, allowing for the stream of data to be compressed on the fly, effectively doubling the network bandwidth at the expense of CPU. With the increasing CPU performance, follow Google's lead and use the spare CPU time to compress everything.

There is a java misc IOstream class which utilizes gzip
http://java.sun.com/developer/technicalArticles/Streams/ProgIOStreams/

I reviewed the code and adding the Gzip wrapper around the IOstream is easy and there are plenty of examples of code out on the lazy net. More on that later...

Full install and results:

[root@ip-10-251-71-99 ~]# id weka
uid=502(weka) gid=503(weka) groups=503(weka)
[root@ip-10-251-71-99 ~]# su - weka
[weka@ip-10-251-71-99 tutorial]$ env|grep JAVA
JAVA_HOME=/usr/local/java

[weka@ip-10-251-71-99 tutorial]$ java -version
java version "1.5.0_12"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_12-b04)
Java HotSpot(TM) Client VM (build 1.5.0_12-b04, mixed mode, sharing)
[weka@ip-10-251-71-99 tutorial]$ ls
J48-data.model          J48-segment-data.out                  Leukemia-ALLAML.NaiveBayes.J48.pred
J48-data.out            Leukemia-ALLAML.NaiveBayes.J48.model  Leukemia-ALLAML.tree.J48.model
J48-segment-data.model  Leukemia-ALLAML.NaiveBayes.J48.out    Leukemia-ALLAML.tree.J48.out
[weka@ip-10-251-71-99 tutorial]$ cd ..
[weka@ip-10-251-71-99 ~]$ ls
tutorial  weka-3-4-11.zip
[weka@ip-10-251-71-99 ~]$ mkdir gridweka
[weka@ip-10-251-71-99 ~]$ cd gridweka/
[weka@ip-10-251-71-99 gridweka]$ wget http://cssa.ucd.ie/xin/weka/weka.jar
--19:04:04--  http://cssa.ucd.ie/xin/weka/weka.jar
         => `weka.jar'
Resolving cssa.ucd.ie... 193.1.132.54
Connecting to cssa.ucd.ie|193.1.132.54|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1,926,948 (1.8M) [application/octet-stream]

100%[============================================================================>] 1,926,948     91.54K/s    ETA 00:00

19:04:25 (90.80 KB/s) - `weka.jar' saved [1926948/1926948]

 Starting two weka servers 

[weka@ip-10-251-71-99 gridweka]$ pwd
/home/weka/gridweka
[weka@ip-10-251-71-99 gridweka]$ java -classpath /home/weka/gridweka/weka.jar weka.core.DistributedServer 8001 &
[1] 2735
[weka@ip-10-251-71-99 gridweka]$ Thu Feb 21 19:11:37 EST 2008: Server started on port 8001
Thu Feb 21 19:11:37 EST 2008: Waiting for connections...

[weka@ip-10-251-71-99 gridweka]$ java -classpath /home/weka/gridweka/weka.jar weka.core.DistributedServer 8002 &
[2] 2745
[weka@ip-10-251-71-99 gridweka]$ Thu Feb 21 19:15:37 EST 2008: Server started on port 8002
Thu Feb 21 19:15:37 EST 2008: Waiting for connections...

 On client machine which happens to be the same box. 

$ cat .weka-parallel
PORT=8001
ec2-72-44-33-131.compute-1.amazonaws.com
2
1024

 Results 

[weka@ip-10-251-71-99 gridweka]$ java  -classpath /home/weka/gridweka/weka.jar weka.classifiers.trees.J48 \
-t $WEKAHOME/data/segment-challenge.arff -d segment.model -a
---Judgement--- server 1: Memory free : 524288000 --> Passed!
---Judgement--- server 2: Memory free : 524288000 --> Passed!
Thu Feb 21 19:38:54 EST 2008: Connection job 0 with localhost closed.
Thu Feb 21 19:38:55 EST 2008: Processed job 0 request from localhost
Thu Feb 21 19:38:55 EST 2008: Connection job 0 with localhost closed.
---Judgement--- server 1: Memory free : 524288000 --> Passed!
Using 1st Server(1) to do crossValidate.
---Judgement--- server 2: Memory free : 524288000 --> Passed!
Thu Feb 21 19:38:57 EST 2008: Connection job 0 with localhost closed.
Thu Feb 21 19:38:57 EST 2008: Processed job 1 request from localhost
server 1: index 0
--------- Refresh the Rank ---------
--> connectInfo 0: rank 2 time 0
--> connectInfo 1: rank 1 time 339
--> connectInfo 2: rank 3 time 0
------------------------------------
****************  Checking servers' status ****************
--------- Refresh the Rank ---------
--> connectInfo 0: rank 2 time 0
--> connectInfo 1: rank 1 time 339
--> connectInfo 2: rank 3 time 0
------------------------------------
server 1: index 1
server 1: index 2
server 1: index 3
server 1: index 4
server 1: index 5
server 1: index 6
server 1: index 7
server 1: index 8
****************  Checking servers' status ****************
--------- Refresh the Rank ---------
--> connectInfo 0: rank 2 time 0
--> connectInfo 1: rank 1 time 339
--> connectInfo 2: rank 3 time 0
------------------------------------
server 1: index 9
server 1: index 10
server 1: index 11
server 1: index 12
server 1: index 13
server 1: index 14
server 1: index 15
server 1: index 16
server 1: index 17
****************  Checking servers' status ****************
--------- Refresh the Rank ---------
--> connectInfo 0: rank 2 time 0
--> connectInfo 1: rank 1 time 339
--> connectInfo 2: rank 3 time 0
------------------------------------
server 1: index 18
server 1: index 19
server 1: index 20
server 1: index 21
server 1: index 22
server 1: index 23
server 1: index 24
server 1: index 25
****************  Checking servers' status ****************
--------- Refresh the Rank ---------
--> connectInfo 0: rank 2 time 0
--> connectInfo 1: rank 1 time 339
--> connectInfo 2: rank 3 time 0
------------------------------------
server 1: index 26
server 1: index 27
server 1: index 28
server 1: index 29
server 1: index 30
server 1: index 31
server 1: index 32
server 1: index 33
server 1: index 34
****************  Checking servers' status ****************
--------- Refresh the Rank ---------
--> connectInfo 0: rank 2 time 0
--> connectInfo 1: rank 1 time 339
--> connectInfo 2: rank 3 time 0
------------------------------------
server 1: index 35
server 1: index 36
server 1: index 37
server 1: index 38
server 1: index 39
server 1: index 40
server 1: index 41
server 1: index 42
server 1: index 43
****************  Checking servers' status ****************
--------- Refresh the Rank ---------
--> connectInfo 0: rank 2 time 0
--> connectInfo 1: rank 1 time 339
--> connectInfo 2: rank 3 time 0
------------------------------------
server 1: index 44
server 1: index 45
server 1: index 46
server 1: index 47
server 1: index 48
server 1: index 49
J48 pruned tree
------------------

region-centroid-row <= 155 |   value-mean <= 91.4444 |   |   rawred-mean <= 24.6667 |   |   |   hue-mean <= -1.89048 |   |   |   |   hue-mean <= -2.22266 |   |   |   |   |   region-centroid-row <= 146: foliage (102.0/1.0) |   |   |   |   |   region-centroid-row > 146: cement (3.0)
|   |   |   |   hue-mean > -2.22266
|   |   |   |   |   rawred-mean <= 2.55556 |   |   |   |   |   |   hue-mean <= -2.09121 |   |   |   |   |   |   |   region-centroid-row <= 129: foliage (50.0) |   |   |   |   |   |   |   region-centroid-row > 129
|   |   |   |   |   |   |   |   region-centroid-col <= 128 |   |   |   |   |   |   |   |   |   rawred-mean <= 0.666667: foliage (30.0/4.0) |   |   |   |   |   |   |   |   |   rawred-mean > 0.666667: window (5.0)
|   |   |   |   |   |   |   |   region-centroid-col > 128
|   |   |   |   |   |   |   |   |   vedge-mean <= 0.333334: window (11.0) |   |   |   |   |   |   |   |   |   vedge-mean > 0.333334
|   |   |   |   |   |   |   |   |   |   region-centroid-col <= 216: window (3.0) |   |   |   |   |   |   |   |   |   |   region-centroid-col > 216: foliage (2.0)
|   |   |   |   |   |   hue-mean > -2.09121: window (38.0/1.0)
|   |   |   |   |   rawred-mean > 2.55556
|   |   |   |   |   |   region-centroid-row <= 121 |   |   |   |   |   |   |   exgreen-mean <= -15.4444: brickface (2.0/1.0) |   |   |   |   |   |   |   exgreen-mean > -15.4444
|   |   |   |   |   |   |   |   vedge-mean <= 2.94444: window (75.0) |   |   |   |   |   |   |   |   vedge-mean > 2.94444
|   |   |   |   |   |   |   |   |   region-centroid-col <= 134: cement (2.0) |   |   |   |   |   |   |   |   |   region-centroid-col > 134: window (8.0)
|   |   |   |   |   |   region-centroid-row > 121
|   |   |   |   |   |   |   rawred-mean <= 7.88889 |   |   |   |   |   |   |   |   region-centroid-col <= 43: brickface (2.0) |   |   |   |   |   |   |   |   region-centroid-col > 43: window (13.0/2.0)
|   |   |   |   |   |   |   rawred-mean > 7.88889
|   |   |   |   |   |   |   |   saturation-mean <= 0.492526: cement (15.0) |   |   |   |   |   |   |   |   saturation-mean > 0.492526
|   |   |   |   |   |   |   |   |   region-centroid-col <= 82: foliage (2.0) |   |   |   |   |   |   |   |   |   region-centroid-col > 82: cement (4.0/1.0)
|   |   |   hue-mean > -1.89048
|   |   |   |   exgreen-mean <= -4.77778 |   |   |   |   |   vedge-mean <= 2.77778: brickface (198.0/2.0) |   |   |   |   |   vedge-mean > 2.77778
|   |   |   |   |   |   region-centroid-row <= 115: brickface (4.0) |   |   |   |   |   |   region-centroid-row > 115: foliage (3.0/1.0)
|   |   |   |   exgreen-mean > -4.77778
|   |   |   |   |   hedge-mean <= 0.833335 |   |   |   |   |   |   region-centroid-col <= 115: foliage (4.0) |   |   |   |   |   |   region-centroid-col > 115: window (42.0)
|   |   |   |   |   hedge-mean > 0.833335: grass (2.0)
|   |   rawred-mean > 24.6667
|   |   |   hue-mean <= -2.17742 |   |   |   |   vedge-mean <= 5: window (4.0/1.0) |   |   |   |   vedge-mean > 5: foliage (18.0)
|   |   |   hue-mean > -2.17742
|   |   |   |   rawgreen-mean <= 24.4444: brickface (3.0/1.0) |   |   |   |   rawgreen-mean > 24.4444: cement (180.0)
|   value-mean > 91.4444: sky (220.0)
region-centroid-row > 155
|   exgreen-mean <= -2 |   |   saturation-mean <= 0.385555 |   |   |   region-centroid-row <= 159 |   |   |   |   region-centroid-col <= 208: cement (3.0) |   |   |   |   region-centroid-col > 208: path (2.0)
|   |   |   region-centroid-row > 159: path (234.0)
|   |   saturation-mean > 0.385555: cement (11.0)
|   exgreen-mean > -2: grass (205.0)

Number of Leaves  :     34

Size of the tree :      67


Time taken to build model: 4.19 seconds
Time taken to test model on training data: 0.03 seconds

=== Error on training data ===

Correctly Classified Instances        1485               99      %
Incorrectly Classified Instances        15                1      %
Kappa statistic                          0.9883
Mean absolute error                      0.0029
Root mean squared error                  0.0535
Relative absolute error                  1.1672 %
Root relative squared error             15.2785 %
Total Number of Instances             1500


=== Confusion Matrix ===

 a   b   c   d   e   f   g   <-- classified as  205   0   0   0   0   0   0 |   a = brickface    0 220   0   0   0   0   0 |   b = sky    1   0 205   0   2   0   0 |   c = foliage    1   0   0 217   2   0   0 |   d = cement    2   0   6   1 195   0   0 |   e = window    0   0   0   0   0 236   0 |   f = path    0   0   0   0   0   0 207 |   g = grass    === Stratified cross-validation ===  Correctly Classified Instances        1450               96.6667 % Incorrectly Classified Instances        50                3.3333 % Kappa statistic                          0.9611 Mean absolute error                      0.0095 Root mean squared error                  0.0976 Relative absolute error                  3.8904 % Root relative squared error             27.8938 % Total Number of Instances             1500  Cross-validation ran in parallel using this computer and the following machines: localhost/127.0.0.1   === Confusion Matrix ===     a   b   c   d   e   f   g   <-- classified as  199   0   1   2   3   0   0 |   a = brickface    0 220   0   0   0   0   0 |   b = sky    0   1 196   3   8   0   0 |   c = foliage    0   0   6 209   5   0   0 |   d = cement    2   0   9   6 187   0   0 |   e = window    0   0   0   2   0 234   0 |   f = path    0   0   0   0   2   0 205 |   g = grass   Thu Feb 21 19:39:10 EST 2008: Connection job 1 with localhost closed.   Test remote client. Use telnet first to check the port 

telnet ec2-72-44-33-131.compute-1.amazonaws.com 8001

Escape Character is 'CTRL+]'

Telnet> quit

 On the server 

[weka@ip-10-251-71-99 gridweka]$ Thu Feb 21 19:48:57 EST 2008: Connection job 0 with 203-214-155-114.dyn.iinet.net.au closed.

 Issues with configuration file location 


 Continuing running client on Linux specifying -C option to use 2 parallel servers. 

java  -classpath /home/weka/gridweka/weka.jar weka.classifiers.trees.J48 \
-T $WEKAHOME/data/segment-test.arff -l segment.model -a -C 2

Thu Feb 21 20:31:32 EST 2008: Processed job 0 request from localhost
Thu Feb 21 20:31:32 EST 2008: Connection job 0 with localhost closed.
Thu Feb 21 20:31:32 EST 2008: Processed job 5 request from localhost
Thu Feb 21 20:31:32 EST 2008: Connection job 5 with localhost closed.

J48 pruned tree
------------------

region-centroid-row <= 155 |   value-mean <= 91.4444 |   |   rawred-mean <= 24.6667 |   |   |   hue-mean <= -1.89048 |   |   |   |   hue-mean <= -2.22266 |   |   |   |   |   region-centroid-row <= 146: foliage (102.0/1.0) |   |   |   |   |   region-centroid-row > 146: cement (3.0)
|   |   |   |   hue-mean > -2.22266
|   |   |   |   |   rawred-mean <= 2.55556 |   |   |   |   |   |   hue-mean <= -2.09121 |   |   |   |   |   |   |   region-centroid-row <= 129: foliage (50.0) |   |   |   |   |   |   |   region-centroid-row > 129
|   |   |   |   |   |   |   |   region-centroid-col <= 128 |   |   |   |   |   |   |   |   |   rawred-mean <= 0.666667: foliage (30.0/4.0) |   |   |   |   |   |   |   |   |   rawred-mean > 0.666667: window (5.0)
|   |   |   |   |   |   |   |   region-centroid-col > 128
|   |   |   |   |   |   |   |   |   vedge-mean <= 0.333334: window (11.0) |   |   |   |   |   |   |   |   |   vedge-mean > 0.333334
|   |   |   |   |   |   |   |   |   |   region-centroid-col <= 216: window (3.0) |   |   |   |   |   |   |   |   |   |   region-centroid-col > 216: foliage (2.0)
|   |   |   |   |   |   hue-mean > -2.09121: window (38.0/1.0)
|   |   |   |   |   rawred-mean > 2.55556
|   |   |   |   |   |   region-centroid-row <= 121 |   |   |   |   |   |   |   exgreen-mean <= -15.4444: brickface (2.0/1.0) |   |   |   |   |   |   |   exgreen-mean > -15.4444
|   |   |   |   |   |   |   |   vedge-mean <= 2.94444: window (75.0) |   |   |   |   |   |   |   |   vedge-mean > 2.94444
|   |   |   |   |   |   |   |   |   region-centroid-col <= 134: cement (2.0) |   |   |   |   |   |   |   |   |   region-centroid-col > 134: window (8.0)
|   |   |   |   |   |   region-centroid-row > 121
|   |   |   |   |   |   |   rawred-mean <= 7.88889 |   |   |   |   |   |   |   |   region-centroid-col <= 43: brickface (2.0) |   |   |   |   |   |   |   |   region-centroid-col > 43: window (13.0/2.0)
|   |   |   |   |   |   |   rawred-mean > 7.88889
|   |   |   |   |   |   |   |   saturation-mean <= 0.492526: cement (15.0) |   |   |   |   |   |   |   |   saturation-mean > 0.492526
|   |   |   |   |   |   |   |   |   region-centroid-col <= 82: foliage (2.0) |   |   |   |   |   |   |   |   |   region-centroid-col > 82: cement (4.0/1.0)
|   |   |   hue-mean > -1.89048
|   |   |   |   exgreen-mean <= -4.77778 |   |   |   |   |   vedge-mean <= 2.77778: brickface (198.0/2.0) |   |   |   |   |   vedge-mean > 2.77778
|   |   |   |   |   |   region-centroid-row <= 115: brickface (4.0) |   |   |   |   |   |   region-centroid-row > 115: foliage (3.0/1.0)
|   |   |   |   exgreen-mean > -4.77778
|   |   |   |   |   hedge-mean <= 0.833335 |   |   |   |   |   |   region-centroid-col <= 115: foliage (4.0) |   |   |   |   |   |   region-centroid-col > 115: window (42.0)
|   |   |   |   |   hedge-mean > 0.833335: grass (2.0)
|   |   rawred-mean > 24.6667
|   |   |   hue-mean <= -2.17742 |   |   |   |   vedge-mean <= 5: window (4.0/1.0) |   |   |   |   vedge-mean > 5: foliage (18.0)
|   |   |   hue-mean > -2.17742
|   |   |   |   rawgreen-mean <= 24.4444: brickface (3.0/1.0) |   |   |   |   rawgreen-mean > 24.4444: cement (180.0)
|   value-mean > 91.4444: sky (220.0)
region-centroid-row > 155
|   exgreen-mean <= -2 |   |   saturation-mean <= 0.385555 |   |   |   region-centroid-row <= 159 |   |   |   |   region-centroid-col <= 208: cement (3.0) |   |   |   |   region-centroid-col > 208: path (2.0)
|   |   |   region-centroid-row > 159: path (234.0)
|   |   saturation-mean > 0.385555: cement (11.0)
|   exgreen-mean > -2: grass (205.0)

Number of Leaves  :     34

Size of the tree :      67



=== Error on test data ===

Correctly Classified Instances         779               96.1728 %
Incorrectly Classified Instances        31                3.8272 %
Kappa statistic                          0.9553
Mean absolute error                      0.0109
Root mean squared error                  0.1046
Relative absolute error                  4.4715 %
Root relative squared error             29.905  %
Total Number of Instances              810


=== Confusion Matrix ===

 a   b   c   d   e   f   g   <-- classified as  124   0   0   0   1   0   0 |   a = brickface    0 110   0   0   0   0   0 |   b = sky    1   0 119   0   2   0   0 |   c = foliage    1   0   0 107   2   0   0 |   d = cement    1   0  12   7 105   0   1 |   e = window    0   0   0   0   0  94   0 |   f = path    0   0   1   0   0   2 120 |   g = grass   Running parallel cross-validation on 2 servers 

java  -classpath /home/weka/gridweka/weka.jar weka.classifiers.trees.J48 -t \
$WEKAHOME/data/segment-challenge.arff -d segment.model -x 10 -a -C 2
---Judgement--- server 1: Memory free : 524288000 --> Passed!
---Judgement--- server 2: Memory free : 524288000 --> Passed!
Thu Feb 21 20:34:09 EST 2008: Connection job 0 with localhost closed.
Thu Feb 21 20:34:09 EST 2008: Processed job 6 request from localhost
Thu Feb 21 20:34:10 EST 2008: Connection job 6 with localhost closed.
---Judgement--- server 1: Memory free : 524288000 --> Passed!
Using 1st Server(1) to do crossValidate.
---Judgement--- server 2: Memory free : 524288000 --> Passed!
Using 2st Server(2) to do crossValidate.
Thu Feb 21 20:34:13 EST 2008: Processed job 1 request from localhost
server 2: index 0
--------- Refresh the Rank ---------
--> connectInfo 0: rank 2 time 0
--> connectInfo 1: rank 3 time 0
--> connectInfo 2: rank 1 time 571
------------------------------------
server 2: index 1
server 2: index 2
server 2: index 3
server 2: index 4
server 2: index 5
server 2: index 6
server 2: index 7
server 2: index 8
server 2: index 9

J48 pruned tree
------------------

region-centroid-row <= 155 |   value-mean <= 91.4444 |   |   rawred-mean <= 24.6667 |   |   |   hue-mean <= -1.89048 |   |   |   |   hue-mean <= -2.22266 |   |   |   |   |   region-centroid-row <= 146: foliage (102.0/1.0) |   |   |   |   |   region-centroid-row > 146: cement (3.0)
|   |   |   |   hue-mean > -2.22266
|   |   |   |   |   rawred-mean <= 2.55556 |   |   |   |   |   |   hue-mean <= -2.09121 |   |   |   |   |   |   |   region-centroid-row <= 129: foliage (50.0) |   |   |   |   |   |   |   region-centroid-row > 129
|   |   |   |   |   |   |   |   region-centroid-col <= 128 |   |   |   |   |   |   |   |   |   rawred-mean <= 0.666667: foliage (30.0/4.0) |   |   |   |   |   |   |   |   |   rawred-mean > 0.666667: window (5.0)
|   |   |   |   |   |   |   |   region-centroid-col > 128
|   |   |   |   |   |   |   |   |   vedge-mean <= 0.333334: window (11.0) |   |   |   |   |   |   |   |   |   vedge-mean > 0.333334
|   |   |   |   |   |   |   |   |   |   region-centroid-col <= 216: window (3.0) |   |   |   |   |   |   |   |   |   |   region-centroid-col > 216: foliage (2.0)
|   |   |   |   |   |   hue-mean > -2.09121: window (38.0/1.0)
|   |   |   |   |   rawred-mean > 2.55556
|   |   |   |   |   |   region-centroid-row <= 121 |   |   |   |   |   |   |   exgreen-mean <= -15.4444: brickface (2.0/1.0) |   |   |   |   |   |   |   exgreen-mean > -15.4444
|   |   |   |   |   |   |   |   vedge-mean <= 2.94444: window (75.0) |   |   |   |   |   |   |   |   vedge-mean > 2.94444
|   |   |   |   |   |   |   |   |   region-centroid-col <= 134: cement (2.0) |   |   |   |   |   |   |   |   |   region-centroid-col > 134: window (8.0)
|   |   |   |   |   |   region-centroid-row > 121
|   |   |   |   |   |   |   rawred-mean <= 7.88889 |   |   |   |   |   |   |   |   region-centroid-col <= 43: brickface (2.0) |   |   |   |   |   |   |   |   region-centroid-col > 43: window (13.0/2.0)
|   |   |   |   |   |   |   rawred-mean > 7.88889
|   |   |   |   |   |   |   |   saturation-mean <= 0.492526: cement (15.0) |   |   |   |   |   |   |   |   saturation-mean > 0.492526
|   |   |   |   |   |   |   |   |   region-centroid-col <= 82: foliage (2.0) |   |   |   |   |   |   |   |   |   region-centroid-col > 82: cement (4.0/1.0)
|   |   |   hue-mean > -1.89048
|   |   |   |   exgreen-mean <= -4.77778 |   |   |   |   |   vedge-mean <= 2.77778: brickface (198.0/2.0) |   |   |   |   |   vedge-mean > 2.77778
|   |   |   |   |   |   region-centroid-row <= 115: brickface (4.0) |   |   |   |   |   |   region-centroid-row > 115: foliage (3.0/1.0)
|   |   |   |   exgreen-mean > -4.77778
|   |   |   |   |   hedge-mean <= 0.833335 |   |   |   |   |   |   region-centroid-col <= 115: foliage (4.0) |   |   |   |   |   |   region-centroid-col > 115: window (42.0)
|   |   |   |   |   hedge-mean > 0.833335: grass (2.0)
|   |   rawred-mean > 24.6667
|   |   |   hue-mean <= -2.17742 |   |   |   |   vedge-mean <= 5: window (4.0/1.0) |   |   |   |   vedge-mean > 5: foliage (18.0)
|   |   |   hue-mean > -2.17742
|   |   |   |   rawgreen-mean <= 24.4444: brickface (3.0/1.0) |   |   |   |   rawgreen-mean > 24.4444: cement (180.0)
|   value-mean > 91.4444: sky (220.0)
region-centroid-row > 155
|   exgreen-mean <= -2 |   |   saturation-mean <= 0.385555 |   |   |   region-centroid-row <= 159 |   |   |   |   region-centroid-col <= 208: cement (3.0) |   |   |   |   region-centroid-col > 208: path (2.0)
|   |   |   region-centroid-row > 159: path (234.0)
|   |   saturation-mean > 0.385555: cement (11.0)
|   exgreen-mean > -2: grass (205.0)

Number of Leaves  :     34

Size of the tree :      67


Time taken to build model: 0.96 seconds
Time taken to test model on training data: 0.09 seconds

=== Error on training data ===

Correctly Classified Instances        1485               99      %
Incorrectly Classified Instances        15                1      %
Kappa statistic                          0.9883
Mean absolute error                      0.0029
Root mean squared error                  0.0535
Relative absolute error                  1.1672 %
Root relative squared error             15.2785 %
Total Number of Instances             1500


=== Confusion Matrix ===

 a   b   c   d   e   f   g   <-- classified as  205   0   0   0   0   0   0 |   a = brickface    0 220   0   0   0   0   0 |   b = sky    1   0 205   0   2   0   0 |   c = foliage    1   0   0 217   2   0   0 |   d = cement    2   0   6   1 195   0   0 |   e = window    0   0   0   0   0 236   0 |   f = path    0   0   0   0   0   0 207 |   g = grass    === Stratified cross-validation ===  Correctly Classified Instances        1436               95.7333 % Incorrectly Classified Instances        64                4.2667 % Kappa statistic                          0.9502 Mean absolute error                      0.0122 Root mean squared error                  0.1104 Relative absolute error                  4.9799 % Root relative squared error             31.5589 % Total Number of Instances             1500  Cross-validation ran in parallel using this computer and the following machines: localhost/127.0.0.1   === Confusion Matrix ===     a   b   c   d   e   f   g   <-- classified as  196   0   3   1   5   0   0 |   a = brickface    0 220   0   0   0   0   0 |   b = sky    0   1 196   2   9   0   0 |   c = foliage    2   0   4 207   6   1   0 |   d = cement    3   0  16   6 179   0   0 |   e = window    0   0   0   3   0 233   0 |   f = path    0   0   0   0   2   0 205 |   g = grass  Thu Feb 21 20:34:18 EST 2008: Connection job 1 with localhost closed. Thu Feb 21 20:34:18 EST 2008: Connection job 0 with localhost closed.   Ok starting another server with two more GridWeka servers 

[weka@ip-10-251-69-175 gridweka]$ java -classpath /home/weka/gridweka/weka.jar weka.core.DistributedServer 8001 &
[1] 2694
[weka@ip-10-251-69-175 gridweka]$ Thu Feb 21 20:56:32 EST 2008: Server started on port 8001
Thu Feb 21 20:56:32 EST 2008: Waiting for connections...

[weka@ip-10-251-69-175 gridweka]$ java -classpath /home/weka/gridweka/weka.jar weka.core.DistributedServer 8002 &
[2] 2704
[weka@ip-10-251-69-175 gridweka]$ Thu Feb 21 20:56:40 EST 2008: Server started on port 8002
Thu Feb 21 20:56:40 EST 2008: Waiting for connections...

[weka@ip-10-251-69-175 gridweka]$ hostname
ip-10-251-69-175

 Updating the .weka-parallel file 

[weka@ip-10-251-71-99 gridweka]$ vi /home/weka/.weka-parallel
[weka@ip-10-251-71-99 gridweka]$ cat /home/weka/.weka-parallel
PORT=8001
localhost
2
1024
ip-10-251-69-175
2
1024

 Ok running the previous classification with cross validation. This time on 2 servers each running 2 GridWeka
servers


[weka@ip-10-251-71-99 gridweka]$ java  -classpath /home/weka/gridweka/weka.jar \
weka.classifiers.trees.J48 -t $WEKAHOME/data/segment-challenge.arff \
-d segment.model -x 10 -a -C 4
---Judgement--- server 1: Memory free : 524288000 --> Passed!
---Judgement--- server 2: Memory free : 524288000 --> Passed!
Thu Feb 21 21:02:51 EST 2008: Connection job 0 with localhost closed.
Thu Feb 21 21:02:51 EST 2008: Processed job 9 request from localhost
Thu Feb 21 21:02:51 EST 2008: Connection job 9 with localhost closed.
---Judgement--- server 1: Memory free : 524288000 --> Passed!
Using 1st Server(1) to do crossValidate.
---Judgement--- server 2: Memory free : 524288000 --> Passed!
Using 2st Server(2) to do crossValidate.
---Judgement--- server 3: Memory free : 524288000 --> Passed!
Using 3st Server(3) to do crossValidate.
---Judgement--- server 4: Memory free : 524288000 --> Passed!
Using 4st Server(4) to do crossValidate.
server 3: index 0
--------- Refresh the Rank ---------
--> connectInfo 0: rank 2 time 0
--> connectInfo 1: rank 3 time 0
--> connectInfo 2: rank 4 time 0
--> connectInfo 3: rank 1 time 585
--> connectInfo 4: rank 5 time 0
------------------------------------
server 3: index 1
****************  Checking servers' status ****************Thu Feb 21 21:03:04 EST 2008: Processed job 3 request from localhost

--------- Refresh the Rank ---------
--> connectInfo 0: rank 2 time 0
--> connectInfo 1: rank 3 time 0
--> connectInfo 2: rank 4 time 0
--> connectInfo 3: rank 1 time 585
--> connectInfo 4: rank 5 time 0
------------------------------------
server 3: index 3
server 3: index 4
server 2: index 2
--------- Refresh the Rank ---------
--> connectInfo 0: rank 3 time 0
--> connectInfo 1: rank 4 time 0
--> connectInfo 2: rank 2 time 608
--> connectInfo 3: rank 1 time 585
--> connectInfo 4: rank 5 time 0
------------------------------------
Thu Feb 21 21:03:04 EST 2008: Processed job 10 request from localhost
server 2: index 7
server 3: index 8
server 2: index 9
server 3: index 5
server 1: index 6
--------- Refresh the Rank ---------
--> connectInfo 0: rank 4 time 0
--> connectInfo 1: rank 3 time 1063
--> connectInfo 2: rank 2 time 608
--> connectInfo 3: rank 1 time 585
--> connectInfo 4: rank 5 time 0
------------------------------------
server 2: index 5
server 4: index 5
--------- Refresh the Rank ---------
--> connectInfo 0: rank 5 time 0
--> connectInfo 1: rank 4 time 1063
--> connectInfo 2: rank 2 time 608
--> connectInfo 3: rank 1 time 585
--> connectInfo 4: rank 3 time 973
------------------------------------
Thu Feb 21 21:03:06 EST 2008: Connection job 10 with localhost closed.

J48 pruned tree
------------------

region-centroid-row <= 155 |   value-mean <= 91.4444 |   |   rawred-mean <= 24.6667 |   |   |   hue-mean <= -1.89048 |   |   |   |   hue-mean <= -2.22266 |   |   |   |   |   region-centroid-row <= 146: foliage (102.0/1.0) |   |   |   |   |   region-centroid-row > 146: cement (3.0)
|   |   |   |   hue-mean > -2.22266
|   |   |   |   |   rawred-mean <= 2.55556 |   |   |   |   |   |   hue-mean <= -2.09121 |   |   |   |   |   |   |   region-centroid-row <= 129: foliage (50.0) |   |   |   |   |   |   |   region-centroid-row > 129
|   |   |   |   |   |   |   |   region-centroid-col <= 128 |   |   |   |   |   |   |   |   |   rawred-mean <= 0.666667: foliage (30.0/4.0) |   |   |   |   |   |   |   |   |   rawred-mean > 0.666667: window (5.0)
|   |   |   |   |   |   |   |   region-centroid-col > 128
|   |   |   |   |   |   |   |   |   vedge-mean <= 0.333334: window (11.0) |   |   |   |   |   |   |   |   |   vedge-mean > 0.333334
|   |   |   |   |   |   |   |   |   |   region-centroid-col <= 216: window (3.0) |   |   |   |   |   |   |   |   |   |   region-centroid-col > 216: foliage (2.0)
|   |   |   |   |   |   hue-mean > -2.09121: window (38.0/1.0)
|   |   |   |   |   rawred-mean > 2.55556
|   |   |   |   |   |   region-centroid-row <= 121 |   |   |   |   |   |   |   exgreen-mean <= -15.4444: brickface (2.0/1.0) |   |   |   |   |   |   |   exgreen-mean > -15.4444
|   |   |   |   |   |   |   |   vedge-mean <= 2.94444: window (75.0) |   |   |   |   |   |   |   |   vedge-mean > 2.94444
|   |   |   |   |   |   |   |   |   region-centroid-col <= 134: cement (2.0) |   |   |   |   |   |   |   |   |   region-centroid-col > 134: window (8.0)
|   |   |   |   |   |   region-centroid-row > 121
|   |   |   |   |   |   |   rawred-mean <= 7.88889 |   |   |   |   |   |   |   |   region-centroid-col <= 43: brickface (2.0) |   |   |   |   |   |   |   |   region-centroid-col > 43: window (13.0/2.0)
|   |   |   |   |   |   |   rawred-mean > 7.88889
|   |   |   |   |   |   |   |   saturation-mean <= 0.492526: cement (15.0) |   |   |   |   |   |   |   |   saturation-mean > 0.492526
|   |   |   |   |   |   |   |   |   region-centroid-col <= 82: foliage (2.0) |   |   |   |   |   |   |   |   |   region-centroid-col > 82: cement (4.0/1.0)
|   |   |   hue-mean > -1.89048
|   |   |   |   exgreen-mean <= -4.77778 |   |   |   |   |   vedge-mean <= 2.77778: brickface (198.0/2.0) |   |   |   |   |   vedge-mean > 2.77778
|   |   |   |   |   |   region-centroid-row <= 115: brickface (4.0) |   |   |   |   |   |   region-centroid-row > 115: foliage (3.0/1.0)
|   |   |   |   exgreen-mean > -4.77778
|   |   |   |   |   hedge-mean <= 0.833335 |   |   |   |   |   |   region-centroid-col <= 115: foliage (4.0) |   |   |   |   |   |   region-centroid-col > 115: window (42.0)
|   |   |   |   |   hedge-mean > 0.833335: grass (2.0)
|   |   rawred-mean > 24.6667
|   |   |   hue-mean <= -2.17742 |   |   |   |   vedge-mean <= 5: window (4.0/1.0) |   |   |   |   vedge-mean > 5: foliage (18.0)
|   |   |   hue-mean > -2.17742
|   |   |   |   rawgreen-mean <= 24.4444: brickface (3.0/1.0) |   |   |   |   rawgreen-mean > 24.4444: cement (180.0)
|   value-mean > 91.4444: sky (220.0)
region-centroid-row > 155
|   exgreen-mean <= -2 |   |   saturation-mean <= 0.385555 |   |   |   region-centroid-row <= 159 |   |   |   |   region-centroid-col <= 208: cement (3.0) |   |   |   |   region-centroid-col > 208: path (2.0)
|   |   |   region-centroid-row > 159: path (234.0)
|   |   saturation-mean > 0.385555: cement (11.0)
|   exgreen-mean > -2: grass (205.0)

Number of Leaves  :     34

Size of the tree :      67


Time taken to build model: 5.06 seconds
Time taken to test model on training data: 0.09 seconds

=== Error on training data ===

Correctly Classified Instances        1485               99      %
Incorrectly Classified Instances        15                1      %
Kappa statistic                          0.9883
Mean absolute error                      0.0029
Root mean squared error                  0.0535
Relative absolute error                  1.1672 %
Root relative squared error             15.2785 %
Total Number of Instances             1500


=== Confusion Matrix ===

 a   b   c   d   e   f   g   <-- classified as  205   0   0   0   0   0   0 |   a = brickface    0 220   0   0   0   0   0 |   b = sky    1   0 205   0   2   0   0 |   c = foliage    1   0   0 217   2   0   0 |   d = cement    2   0   6   1 195   0   0 |   e = window    0   0   0   0   0 236   0 |   f = path    0   0   0   0   0   0 207 |   g = grass    === Stratified cross-validation ===  Correctly Classified Instances        1436               95.7333 % Incorrectly Classified Instances        64                4.2667 % Kappa statistic                          0.9502 Mean absolute error                      0.0122 Root mean squared error                  0.1104 Relative absolute error                  4.9799 % Root relative squared error             31.5589 % Total Number of Instances             1500  Cross-validation ran in parallel using this computer and the following machines: ip-10-251-69-175/10.251.69.175 localhost/127.0.0.1 localhost/127.0.0.1 ip-10-251-69-175/10.251.69.175   === Confusion Matrix ===     a   b   c   d   e   f   g   <-- classified as  196   0   3   1   5   0   0 |   a = brickface    0 220   0   0   0   0   0 |   b = sky    0   1 196   2   9   0   0 |   c = foliage    2   0   4 207   6   1   0 |   d = cement    3   0  16   6 179   0   0 |   e = window    0   0   0   3   0 233   0 |   f = path    0   0   0   0   2   0 205 |   g = grass  Thu Feb 21 21:03:07 EST 2008: Connection job 3 with localhost closed.   Final demo, using the Leukemia-ALLAML data 

time java  -classpath /home/weka/gridweka/weka.jar weka.classifiers.trees.J48 \
-t $WEKAHOME/data/ALL-AML_train.arff -d Leukemia-ALLAML.tree.J48.model \
-i -x 10 -a -C 4

---Judgement--- server 1: Memory free : 524288000 --> Passed!
---Judgement--- server 2: Memory free : 524288000 --> Passed!
Thu Feb 21 21:10:03 EST 2008: Connection job 0 with localhost closed.
Thu Feb 21 21:10:03 EST 2008: Processed job 13 request from localhost
Thu Feb 21 21:10:05 EST 2008: Connection job 13 with localhost closed.
---Judgement--- server 1: Memory free : 524288000 --> Passed!
Using 1st Server(1) to do crossValidate.
---Judgement--- server 2: Memory free : 524288000 --> Passed!
Using 2st Server(2) to do crossValidate.
---Judgement--- server 3: Memory free : 524288000 --> Passed!
Using 3st Server(3) to do crossValidate.
---Judgement--- server 4: Memory free : 524288000 --> Passed!
Using 4st Server(4) to do crossValidate.
Thu Feb 21 21:10:57 EST 2008: Processed job 5 request from localhost
Thu Feb 21 21:11:03 EST 2008: Processed job 14 request from localhost
server 4: index 2
--------- Refresh the Rank ---------
--> connectInfo 0: rank 2 time 0
--> connectInfo 1: rank 3 time 0
--> connectInfo 2: rank 4 time 0
--> connectInfo 3: rank 5 time 0
--> connectInfo 4: rank 1 time 676
------------------------------------
server 4: index 4
****************  Checking servers' status ****************
--------- Refresh the Rank ---------
--> connectInfo 0: rank 2 time 0
--> connectInfo 1: rank 3 time 0
--> connectInfo 2: rank 4 time 0
--> connectInfo 3: rank 5 time 0
--> connectInfo 4: rank 1 time 676
------------------------------------
server 4: index 5
server 1: index 3
--------- Refresh the Rank ---------
--> connectInfo 0: rank 3 time 0
--> connectInfo 1: rank 1 time 606
--> connectInfo 2: rank 4 time 0
--> connectInfo 3: rank 5 time 0
--> connectInfo 4: rank 2 time 676
------------------------------------
server 1: index 6
server 4: index 7
server 1: index 8
server 4: index 9
server 1: index 0
server 4: index 0
****************  Checking servers' status ****************
--------- Refresh the Rank ---------
--> connectInfo 0: rank 3 time 0
--> connectInfo 1: rank 1 time 606
--> connectInfo 2: rank 4 time 0
--> connectInfo 3: rank 5 time 0
--> connectInfo 4: rank 2 time 676
------------------------------------
server 1: index 1

J48 pruned tree
------------------

attribute4847 <= 938: ALL (27.0) attribute4847 > 938: AML (11.0)

Number of Leaves  :     2

Size of the tree :      3


Time taken to build model: 5.07 seconds
Time taken to test model on training data: 0.01 seconds

=== Error on training data ===

Correctly Classified Instances          38              100      %
Incorrectly Classified Instances         0                0      %
Kappa statistic                          1
Mean absolute error                      0
Root mean squared error                  0
Relative absolute error                  0      %
Root relative squared error              0      %
Total Number of Instances               38


=== Detailed Accuracy By Class ===

TP Rate   FP Rate   Precision   Recall  F-Measure   Class
1         0          1         1         1        ALL
1         0          1         1         1        AML


=== Confusion Matrix ===

a  b   <-- classified as  27  0 |  a = ALL   0 11 |  b = AML    === Stratified cross-validation ===  Correctly Classified Instances          32               84.2105 % Incorrectly Classified Instances         6               15.7895 % Kappa statistic                          0.6358 Mean absolute error                      0.1579 Root mean squared error                  0.3974 Relative absolute error                 37.8015 % Root relative squared error             87.2867 % Total Number of Instances               38  Cross-validation ran in parallel using this computer and the following machines: ip-10-251-69-175/10.251.69.175 localhost/127.0.0.1   === Detailed Accuracy By Class ===  TP Rate   FP Rate   Precision   Recall  F-Measure   Class   0.852     0.182      0.92      0.852     0.885    ALL   0.818     0.148      0.692     0.818     0.75     AML   === Confusion Matrix ===    a  b   <-- classified as  23  4 |  a = ALL   2  9 |  b = AML  Thu Feb 21 21:11:10 EST 2008: Connection job 14 with localhost closed. server 3: index 0 --------- Refresh the Rank --------- --> connectInfo 0: rank 4 time 0
--> connectInfo 1: rank 1 time 606
--> connectInfo 2: rank 5 time 0
--> connectInfo 3: rank 3 time 699
--> connectInfo 4: rank 2 time 676
------------------------------------
server 2: index 1
--------- Refresh the Rank ---------
--> connectInfo 0: rank 5 time 0
--> connectInfo 1: rank 1 time 606
--> connectInfo 2: rank 2 time 637
--> connectInfo 3: rank 4 time 699
--> connectInfo 4: rank 3 time 676
------------------------------------
Thu Feb 21 21:11:21 EST 2008: Connection job 5 with localhost closed.

real    1m19.614s
user    0m13.140s
sys     0m12.660s

 rerunning with java option -Xprof to profile to CPU 

Flat profile of 39.70 secs (113 total ticks): Thread-5

Interpreted + native   Method
49.5%     0  +    53    java.net.SocketInputStream.socketRead0
8.4%     0  +     9    java.net.SocketOutputStream.socketWrite0
57.9%     0  +    62    Total interpreted

   Compiled + native   Method
0.9%     0  +     1    java.lang.String.
0.9%     0  +     1    Total compiled

       Stub + native   Method
32.7%     0  +    35    java.io.FileInputStream.read
6.5%     0  +     7    java.net.SocketOutputStream.socketWrite0
1.9%     0  +     2    java.lang.System.identityHashCode
41.1%     0  +    44    Total stub

Thread-local ticks:
5.3%     6             Blocked (of total)


Flat profile of 34.95 secs (111 total ticks): Thread-7

   Compiled + native   Method
1.8%     1  +     1    weka.classifiers.EvaluationClient.determineIndex
1.8%     1  +     1    Total compiled

Thread-local ticks:
98.2%   109             Compilation


Flat profile of 54.81 secs (473 total ticks): main

Interpreted + native   Method
0.6%     0  +     2    java.net.Inet4AddressImpl.lookupAllHostAddr
0.3%     0  +     1    java.lang.System.currentTimeMillis
0.3%     0  +     1    java.io.FileInputStream.read
0.3%     0  +     1    java.lang.Thread.start0
0.3%     0  +     1    java.util.zip.ZipFile.getEntry
0.3%     0  +     1    weka.classifiers.trees.J48.main
0.3%     1  +     0    java.net.InetAddress.getCachedAddress
0.3%     0  +     1    weka.classifiers.Evaluation.evaluateModel
2.5%     1  +     8    Total interpreted

   Compiled + native   Method
22.5%    81  +     0    weka.classifiers.BuildModelClient.start
1.4%     5  +     0    java.io.DataInputStream.readLine
1.1%     4  +     0    weka.classifiers.EvaluationClient.start
1.1%     4  +     0    sun.misc.FloatingDecimal.readJavaFormatString
0.3%     1  +     0    java.lang.StringBuffer.toString
0.3%     1  +     0    java.lang.AbstractStringBuilder.append
0.3%     1  +     0    java.io.ObjectOutputStream.defaultWriteFields
0.3%     1  +     0    java.lang.String.regionMatches
0.3%     1  +     0    sun.nio.cs.US_ASCII$Decoder.decodeArrayLoop
0.3%     1  +     0    weka.core.Instances.getInstanceFull
0.3%     1  +     0    java.io.ObjectOutputStream.writeObject0
28.1%   101  +     0    Total compiled

       Stub + native   Method
60.8%     0  +   219    java.io.FileInputStream.read
6.1%     0  +    22    java.io.FileOutputStream.writeBytes
0.3%     0  +     1    java.lang.Class.isArray
0.3%     0  +     1    java.lang.Float.floatToIntBits
67.5%     0  +   243    Total stub

Thread-local ticks:
23.9%   113             Blocked (of total)
1.9%     7             Compilation


Flat profile of 35.26 secs (126 total ticks): Thread-6

Interpreted + native   Method
66.1%     0  +    82    java.net.SocketInputStream.socketRead0
7.3%     0  +     9    java.net.SocketOutputStream.socketWrite0
0.8%     0  +     1    java.net.PlainSocketImpl.socketConnect
74.2%     0  +    92    Total interpreted

   Compiled + native   Method
2.4%     0  +     3    java.io.ObjectStreamClass.lookup
2.4%     0  +     3    Total compiled

       Stub + native   Method
20.2%     0  +    25    java.io.FileInputStream.read
20.2%     0  +    25    Total stub

Thread-local ticks:
1.6%     2             Blocked (of total)
3.2%     4             Compilation


Global summary of 55.12 seconds:
100.0%   504             Received ticks
1.8%     9             Received GC ticks
1.4%     7             Compilation
0.2%     1             Unknown code

real    0m55.151s
user    0m8.470s
sys     0m12.520s

 Rerun the JFS Parallel Access to Network the time was about 20% faster 

http://jfs.des.udc.es/docs/jfs.html

time jfsrun weka.classifiers.trees.J48 -t $WEKAHOME/data/ALL-AML_train.arff \
-d Leukemia-ALLAML.tree.J48.model -x 10 -a -C 4


real    0m41.736s
user    0m6.990s
sys     0m9.180s

Installing Kettle on EC2

As I mentioned in the roadmap, I am going to run through installing Kettle or Pentaho Data Integration (PDI) on EC2.

For starters I am just using the small instances on EC2. However we can start pushing and benchmarking later. Given some of the disappointment to the lack of network bandwidth, at least as the applications currently use it, on the larger instances, running an Kettle Master/Slave Cluster is still going to be limited by the amount of traffic to maintain and manage the cluster.

On with the show, I had a full blown Pentaho demo Amazon Machine Image (AMI) already for a previous post of Pentaho BI Suite. However I wanted to just install the Kettle portion only. Here I am concentrating on the ETL portion of Pentaho.

Install:

1. Install Java (JDK 1.5 or better)
2. Install MySQL 5.0 or better.
3. Download Kettle
   
4. mkdir /usr/local/kettle
5. unzip Kettle-3.0.1.zip -d /usr/local/kettle
6. chmod +x /usr/local/kettle/*.sh
7. export PATH=$PATH:/usr/local/kettle/

Tests:

Simple tests to make sure it is running ok and there are no java classpath issues, just run runSamples.sh

cd /usr/local/kettle
./runSamples.sh

EXECUTING TRANSFORMATION [samples/transformations/Add sequence - specify a common counter.ktr]
INFO  27-01 20:34:13,704 (LogWriter.java:println:403)  -Pan - Logging is at level : Minimal logging
INFO  27-01 20:34:13,707 (LogWriter.java:println:403)  -Pan - Start of run.
2008/01/27 20:34:16:700 EST [INFO] DefaultFileReplicator - Using "/tmp/vfs_cache" as temporary files store.
INFO  27-01 20:34:17,111 (LogWriter.java:println:403)  -Trans - Dispatching started for filename [samples/transformations/Add sequence - specify a common counter.ktr]
INFO  27-01 20:34:17,477 (LogWriter.java:println:403)  -Trans - Transformation ended.
INFO  27-01 20:34:17,483 (LogWriter.java:println:403)  -Pan - Finished!
INFO  27-01 20:34:17,484 (LogWriter.java:println:403)  -Pan - Start=2008/01/27 20:34:16.954, Stop=2008/01/27 20:34:17.483
INFO  27-01 20:34:17,484 (LogWriter.java:println:403)  -Pan - Processing ended after 0 seconds.
EXECUTING TRANSFORMATION [samples/transformations/Aggregate - basics.ktr]
INFO  27-01 20:34:18,221 (LogWriter.java:println:403)  -Pan - Logging is at level : Minimal logging
INFO  27-01 20:34:18,223 (LogWriter.java:println:403)  -Pan - Start of run.
2008/01/27 20:34:21:225 EST [INFO] DefaultFileReplicator - Using "/tmp/vfs_cache" as temporary files store.
INFO  27-01 20:34:21,896 (LogWriter.java:println:403)  -Trans - Dispatching started for filename [samples/transformations/Aggregate - basics.ktr]
INFO  27-01 20:34:23,525 (LogWriter.java:println:403)  -Trans - Transformation ended.
INFO  27-01 20:34:23,527 (LogWriter.java:println:403)  -Pan - Finished!
INFO  27-01 20:34:23,528 (LogWriter.java:println:403)  -Pan - Start=2008/01/27 20:34:21.410, Stop=2008/01/27 20:34:23.528
INFO  27-01 20:34:23,528 (LogWriter.java:println:403)  -Pan - Processing ended after 2 seconds.
EXECUTING TRANSFORMATION [samples/transformations/Calculator - Substract constant value one from a number.ktr]
INFO  27-01 20:34:24,273 (LogWriter.java:println:403)  -Pan - Logging is at level : Minimal logging
INFO  27-01 20:34:24,276 (LogWriter.java:println:403)  -Pan - Start of run.
2008/01/27 20:34:27:296 EST [INFO] DefaultFileReplicator - Using "/tmp/vfs_cache" as temporary files store.
INFO  27-01 20:34:27,720 (LogWriter.java:println:403)  -Trans - Dispatching started for filename [samples/transformations/Calculator - Substract constant value one from a number.ktr]
INFO  27-01 20:34:27,875 (LogWriter.java:println:403)  -Trans - Transformation ended.
INFO  27-01 20:34:27,878 (LogWriter.java:println:403)  -Pan - Finished!
INFO  27-01 20:34:27,879 (LogWriter.java:println:403)  -Pan - Start=2008/01/27 20:34:27.522, Stop=2008/01/27 20:34:27.878
...

Setup repository db (using MySQL 5.1 db on EC2)

export PASSWD=yourpasswordhere
mysql -u root -p$PASSWD

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.1.20-beta-log MySQL Community Server (GPL)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> create database kettle_repos;
Query OK, 1 row affected (0.00 sec)

mysql> grant all on kettle_repos.* to paulm@'myhost identified by 'xxxx';
Query OK, 0 rows affected (0.00 sec)

Test connectivity first using mysql client, you may to need to allow your machine to connect via any firewall and/or grant permission in your EC2 security group.
Just add your ip address and the port 3306 only.

mysql -u paulm -p$PASSWD --host=ec2host -D kettle_repos --protocol=tcp

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 5 to server version: 5.1.20-beta-log

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> show tables;
--------------
show tables
--------------

Empty set (0.24 sec)

mysql> exit

Now create a new database connection in Pentaho Spoon. The Kettle GUI

Results:

Connection to database [MySQL51_repos] is OK.
Hostname       : ec2-67-202-33-238.compute-1.amazonaws.com
Port           : 3306
Database name  : kettle_repos

Now create a new repository, choose the create/upgrade option in the connection dialog

Start a new transformation.

I understand this is a simple step, if this was it, I would use LOAD DATA INFILE to load the data. The idea is the build on the simple tasks, learning new stuff and then we can combine the simple steps into complex analysis and truly push Kettle ETL engine.

I am using the KDD Cup 1999 dataset, just a 1000 rows of csv to start

zcat kddcup.data_10_percent.gz|head -1000 > kddcup.data.1000.csv

The headers are in a separate kddcup.names file however I mucked around with sed and got the file into a csv header row at this row to the CSV file to make the file input step and field names easier to edit.

cat kddcup.names |awk -F ":" ' { print $1"," }' > kddcup.name.header
tr -d '\n' < kddcup.name.header > kddcup.name.header.clean
cat kddcup.name.header.clean

type,duration,protocol_type,service,flag,src_bytes,dst_bytes,land,wrong_fragment,urgent,hot,num_failed_logins,logged_in,num_comp
_shell,su_attempted,num_root,num_file_creations,num_shells,num_access_files,num_outbound_cmds,is_host_login,is_guest_login,count
error_rate,srv_serror_rate,rerror_rate,srv_rerror_rate,same_srv_rate,diff_srv_rate,srv_diff_host_rate,dst_host_count,dst_host_sr
host_same_srv_rate,dst_host_diff_srv_rate,dst_host_same_src_port_rate,dst_host_srv_diff_host_rate,dst_host_serror_rate,dst_host_
ate,dst_host_rerror_rate,dst_host_srv_rerror_rate

Create a database to hold the data and grant access.

mysql -u root -p$PASSWD
mysql> create database KDD99;
Query OK, 1 row affected (0.03 sec)

mysql> grant all on KDD99.* to 'paulm'@'myhost' identified by 'xxxx';
Query OK, 0 rows affected (0.00 sec)

Sample CREATE TABLE, generated from table output step

CREATE TABLE KDD99.KDD99
(
type INT
, duration VARCHAR(3)
, protocol_type VARCHAR(8)
, service VARCHAR(8)
, flag INT
, src_bytes INT
, dst_bytes INT
, land INT
, wrong_fragment INT
, urgent INT
, hot INT
, num_failed_logins INT
, logged_in INT
, num_compromised INT
, root_shell INT
, su_attempted INT
, num_root INT
, num_file_creations INT
, num_shells INT
, num_access_files INT
, num_outbound_cmds INT
, is_host_login INT
, is_guest_login INT
, count INT
, srv_count INT
, serror_rate INT
, srv_serror_rate INT
, rerror_rate INT
, srv_rerror_rate INT
, same_srv_rate INT
, diff_srv_rate FLOAT
, srv_diff_host_rate INT
, dst_host_count INT
, dst_host_srv_count INT
, dst_host_same_srv_rate INT
, dst_host_diff_srv_rate FLOAT
, dst_host_same_src_port_rate FLOAT
, dst_host_srv_diff_host_rate INT
, dst_host_serror_rate INT
, dst_host_srv_serror_rate INT
, dst_host_rerror_rate INT
, dst_host_srv_rerror_rate VARCHAR(25)
)
;

Run the transformation.
The most fun of any data loads is finding that your datatypes and lengths are
not always long enough.

2008/01/28 14:14:00 - Spoon - Transformation opened.
2008/01/28 14:14:00 - Spoon - Launching transformation [Transformation 1]...
2008/01/28 14:14:00 - Spoon - Started the transformation execution.
2008/01/28 14:14:00 - Transformation 1 - Dispatching started for transformation [Transformation 1]
2008/01/28 14:14:00 - Transformation 1 - Nr of arguments detected:0
2008/01/28 14:14:00 - Transformation 1 - This is not a replay transformation
2008/01/28 14:14:00 - Transformation 1 - This transformation can be replayed with replay date: 2008/01/28 14:14:00
2008/01/28 14:14:00 - Transformation 1 - Initialising 2 steps...
2008/01/28 14:14:02 - Table output.0 - Connected to database [MySQL51_data] (commit=1000)
2008/01/28 14:14:03 - CSV file input.0 - Starting to run...
2008/01/28 14:14:03 - Table output.0 - Starting to run...
2008/01/28 14:14:03 - CSV file input.0 - Finished processing (I=1001, O=0, R=0, W=1000, U=0, E=0)
2008/01/28 14:18:07 - Table output.0 - ERROR (version 3.0.1, build 534 from 2007/12/12 12:28:23) : Because of an error, this step can't continue:
2008/01/28 14:18:07 - Table output.0 - ERROR (version 3.0.1, build 534 from 2007/12/12 12:28:23) : Error batch inserting rows into table [KDD99].
2008/01/28 14:18:07 - Table output.0 - ERROR (version 3.0.1, build 534 from 2007/12/12 12:28:23) : Errors encountered (first 10):
2008/01/28 14:18:07 - Table output.0 - ERROR (version 3.0.1, build 534 from 2007/12/12 12:28:23) : Data truncation: Data too long for column 'duration' at row 1
2008/01/28 14:18:07 - Table output.0 - ERROR (version 3.0.1, build 534 from 2007/12/12 12:28:23) :
2008/01/28 14:18:07 - Table output.0 - ERROR (version 3.0.1, build 534 from 2007/12/12 12:28:23) :
2008/01/28 14:18:07 - Table output.0 - ERROR (version 3.0.1, build 534 from 2007/12/12 12:28:23) : Error updating batch
2008/01/28 14:18:07 - Table output.0 - ERROR (version 3.0.1, build 534 from 2007/12/12 12:28:23) : Data truncation: Data too long for column 'duration' at row 1
2008/01/28 14:18:07 - Spoon - The transformation has finished!!
2008/01/28 14:18:08 - Table output.0 - Finished processing (I=0, O=999, R=1000, W=0, U=0, E=1)

Virtual Datamine Roadmap Q1 2008

I thought I would update what is coming in terms of subject areas and posts.

Pentaho Clusters on EC2:

1. Installing JBoss AS
   
2. Combining all the parts to form the single compute node
   
3. Installing software for Cluster master

Kettle on EC2:

1. Installing Kettle on EC2.
2. Running Kettle Slaves on EC2.
3. Demo of remote transformation using EC2 Kettle nodes.
   

Weka on EC2:

1. Install and demo of GridWeka
2. Install and demo of WekaAsWS (Weka as a Webservice)
3. Using Weka to mine clickstreams

Globus Cluster Software on EC2:

1. Install a master node.
2. Install of grid/slave/client node.

That should keep me busy until at least April.

Once the services are stable, the next stage is to start presenting them for consumption.

I am looking at a combination of free and various levels of paid services using a similar
model to telco cell/mobile phone credits.

Have Fun

JGroups Cluster on EC2 large 64bit instances

Introduction:

In this series of articles I have been covering getting the proposed Pentaho Cluster running on EC2.
http://blog.vmdatamine.com/2007/09/pentaho-business-suite-cluster-research.html
http://blog.vmdatamine.com/2007/09/pentaho-cluster-installing-jgroups.html
http://blog.vmdatamine.com/2007/11/pentaho-cluster-installing-jgroups-on.html

In the last article, I ran a JGroups cluster test and found the results disappointing compared to the test results published by JBoss.

So given there are new larger instances available now with better performance I decided to see how JGroups would perform on those instances.

The specification of the large instances can be found in Amazon's announcement.

The only change from the last test from an upgrade to Java (JDK 6 Release 3) and running on Amazon public image Fedora 64 bit OS.

I tried both the large and extra large instances in a 4 node cluster setup running TCP.

Comments:

1. The network bandwidth is still the limiting factor.
2. I had to modify the tcp.xml settings to enable queues to stop the test hanging sporadically.
3. The larger 64 bit instances have more throughput vs the small nodes. This could be due to settings or CPU is an underlining factor after all.
4. You are not going to reach the JBoss performance results without a faster network.

Results:

Two nodes: 2 senders:

-- results:

10.252.93.220:7800 (myself):
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=3211ms, msgs/sec=6228.59, throughput=6.23MB

10.252.99.47:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=3236ms, msgs/sec=6180.47, throughput=6.18MB

combined: 6204.53 msgs/sec averaged over all receivers (throughput=6.2MB/sec)

Two nodes: 1 sender, 1 receiver

-- results:

10.252.93.220:7800 (myself):
num_msgs_expected=10000, num_msgs_received=10000 (loss rate=0.0%), received=10MB, time=2607ms, msgs/sec=3835.83, throughput=3.84MB

10.252.99.47:7800:
num_msgs_expected=10000, num_msgs_received=10000 (loss rate=0.0%), received=10MB, time=2515ms, msgs/sec=3976.14, throughput=3.98MB

combined: 3905.98 msgs/sec averaged over all receivers
(throughput=3.9MB/sec)

4 nodes: 2 senders, 2 receivers

-- results:

10.252.23.15:7800:

num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=4640ms, msgs/sec=4310.34, throughput=4.31MB
10.252.98.208:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=4687ms, msgs/sec=4267.12, throughput=4.27MB
10.252.79.0:7800 (myself):
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=4642ms, msgs/sec=4308.49, throughput=4.31MB
10.252.93.203:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=4654ms, msgs/sec=4297.38, throughput=4.3MB

combined: 4295.83 msgs/sec averaged over all receivers (throughput=4.3MB/sec)


4 nodes: 2 senders, 2 receivers 100k messages test

-- results:

10.252.23.15:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=19828ms, msgs/sec=10086.75, throughput=10.09MB

10.252.98.208:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=19893ms, msgs/sec=10053.79, throughput=10.05MB

10.252.79.0:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=19828ms, msgs/sec=10086.75, throughput=10.09MB

10.252.93.203:7800 (myself):
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=19917ms, msgs/sec=10041.67, throughput=10.04MB

combined: 10067.24 msgs/sec averaged over all receivers (throughput=10.07MB/sec)

2nd run:

-- results:

10.252.23.15:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=20605ms, msgs/sec=9706.38, throughput=9.71MB

10.252.98.208:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=20629ms, msgs/sec=9695.09, throughput=9.7MB

10.252.79.0:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=20590ms, msgs/sec=9713.45, throughput=9.71MB

10.252.93.203:7800 (myself):
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=20636ms, msgs/sec=9691.8, throughput=9.69MB

combined: 9701.68 msgs/sec averaged over all receivers (throughput=9.7MB/sec)

Extra large 4 nodes : 2 senders, 2 receivers

-- results:

10.252.106.3:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=17857ms, msgs/sec=11200.09, throughput=11.2MB

10.252.15.79:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=17831ms, msgs/sec=11216.42, throughput=11.22MB

10.252.10.223:7800 (myself):
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=17837ms, msgs/sec=11212.65, throughput=11.21MB

10.252.6.223:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=17837ms, msgs/sec=11212.65, throughput=11.21MB

combined: 11210.45 msgs/sec averaged over all receivers (throughput=11.21MB/sec)


-- results:

10.252.106.3:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=15585ms, msgs/sec=12832.85, throughput=12.83MB

10.252.15.79:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=15564ms, msgs/sec=12850.17, throughput=12.85MB

10.252.10.223:7800 (myself):
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=15563ms, msgs/sec=12850.99, throughput=12.85MB

10.252.6.223:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=15563ms, msgs/sec=12850.99, throughput=12.85MB

combined: 12846.25 msgs/sec averaged over all receivers (throughput=12.85MB/sec)

Extra large 4 nodes : 4 senders and TCP queues

-- results:

10.252.106.3:7800:
num_msgs_expected=400000, num_msgs_received=400000 (loss rate=0.0%), received=400MB, time=27784ms, msgs/sec=14396.78, throughput=14.4MB

10.252.15.79:7800:
num_msgs_expected=400000, num_msgs_received=400000 (loss rate=0.0%), received=400MB, time=27744ms, msgs/sec=14417.53, throughput=14.42MB

10.252.10.223:7800 (myself):
num_msgs_expected=400000, num_msgs_received=400000 (loss rate=0.0%), received=400MB, time=27761ms, msgs/sec=14408.7, throughput=14.41MB

10.252.6.223:7800:
num_msgs_expected=400000, num_msgs_received=400000 (loss rate=0.0%), received=400MB, time=27755ms, msgs/sec=14411.82, throughput=14.41MB

combined: 14408.71 msgs/sec averaged over all receivers (throughput=14.41MB/sec)

with TCP queue_max_size set to 1000

10.252.106.3:7800:
num_msgs_expected=400000, num_msgs_received=400000 (loss rate=0.0%), received=400MB, time=26667ms, msgs/sec=14999.81, throughput=15MB

10.252.15.79:7800:
num_msgs_expected=400000, num_msgs_received=400000 (loss rate=0.0%), received=400MB, time=26709ms, msgs/sec=14976.23, throughput=14.98MB

10.252.10.223:7800 (myself):
num_msgs_expected=400000, num_msgs_received=400000 (loss rate=0.0%), received=400MB, time=26676ms, msgs/sec=14994.75, throughput=14.99MB

10.252.6.223:7800:
num_msgs_expected=400000, num_msgs_received=400000 (loss rate=0.0%), received=400MB, time=26710ms, msgs/sec=14975.66, throughput=14.98MB

combined: 14986.61 msgs/sec averaged over all receivers (throughput=14.99MB/sec)

Example of TCP.XML (note &gt and &lt to handle HTML)

<>
   < TCP start_port="7800"
        loopback="false"
        discard_incompatible_packets="true"
        max_bundle_size="64000"
        max_bundle_timeout="30"
        use_incoming_packet_handler="true"
        enable_bundling="true"
        use_send_queues="true"
        sock_conn_timeout="300"
        skip_suspected_members="true"

        use_concurrent_stack="true"

        thread_pool.enabled="true"
        thread_pool.min_threads="8"
        thread_pool.max_threads="40"
        thread_pool.keep_alive_time="5000"
        thread_pool.queue_enabled="true"
        thread_pool.queue_max_size="100"
        thread_pool.rejection_policy="run"

        oob_thread_pool.enabled="true"
        oob_thread_pool.min_threads="8"
        oob_thread_pool.max_threads="20"
        oob_thread_pool.keep_alive_time="5000"
        oob_thread_pool.queue_enabled="true"
        oob_thread_pool.queue_max_size="100"
        oob_thread_pool.rejection_policy="run"/ >

< TCPPING timeout="3000"
            initial_hosts="${jgroups.tcpping.initial_hosts:10.252.106.3[7800],
10.252.15.79[7800],10.252.10.223[7800],10.252.6.223[7800]}"
            port_range="1"
            num_initial_members="2"/ >

Pentaho Cluster : Installing JGroups on EC2

Overview:

Wondering why I hadn't updated my progress with installing JGroups on EC2?
It was because I had three false starts and got nowhere.

Finally however I found some more documentation and was able to get it running.

I found this report about a JGroups Performance test and the associated JBoss wiki Perftests.

That was enough information to understand how to get it working. It also helped that the more recent version JGroups 2.5.1 came with some sample configuration files.

Comments:

1. The network bandwidth between EC2 nodes is the limiting factor.
2. For 2 nodes: 4183.65 msgs/sec averaged over all receivers (throughput=4.18MB/sec) vs 60783.12 msgs/sec averaged over all receivers (throughput=60.78MB/sec)
3. For 4 nodes: 3852.11 msgs/sec averaged over all receivers (throughput=3.85MB/sec)
   vs 60783.12 msgs/sec averaged over all receivers (throughput=60.78MB/sec)

The JGroups on EC2 performance versus the JGroups Performance report was very bad. Without changing any settings it was 15 times slower. This is the benefit of having a 1 Gigabit LAN versus 100 Megabit LAN.
The next step would be to test on the larger instances. If the network performance, rated as better for the larger instances versus the default is true, it will show up in the results.


Install:

1. wget http://easynews.dl.sourceforge.net/sourceforge/javagroups/JGroups-2.5.1.bin.zip
2. unzip JGroups-2.5.1.bin.zip -d YourJavaLibDirectory
3. cd YourJavaLibDirectory.
4. nslookup `hostname` to get your servers IP address.
5. edit the JGroups-2.5.1.bin/config.txt and JGroups-2.5.1.bin/tcp.xml to add the hosts. See the sample files at the bottom of this post.
   
6. java -cp JGroups-2.5.1.bin/concurrent.jar:JGroups-2.5.1.bin/jgroups-all.jar:JGroups-2.5.1.bin/commons-logging.jar org.jgroups.tests.perf.Test -receiver -config JGroups-2.5.1.bin/config.txt -props JGroups-2.5.1.bin/tcp.xml
7. java -cp JGroups-2.5.1.bin/concurrent.jar:JGroups-2.5.1.bin/jgroups-all.jar:JGroups-2.5.1.bin/commons-logging.jar org.jgroups.tests.perf.Test -sender -config JGroups-2.5.1.bin/config.txt -props JGroups-2.5.1.bin/tcp.xml
8. If you have the hosts correct it should run the test.

Results:

2 nodes
-- results:

10.255.23.160:7800 (myself):
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=4664ms, msgs/sec=4288.16, throughput=4.29MB

10.255.26.143:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=4903ms, msgs/sec=4079.14, throughput=4.08MB

combined: 4183.65 msgs/sec averaged over all receivers (throughput=4.18MB/sec)

4 nodes (2 senders, 2 receivers):

-- results:

10.253.15.95:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5212ms, msgs/sec=3837.3, throughput=3.84MB

10.255.23.160:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5174ms, msgs/sec=3865.48, throughput=3.87MB

10.255.26.143:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5192ms, msgs/sec=3852.08, throughput=3.85MB

10.253.83.143:7800 (myself):
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5190ms, msgs/sec=3853.56, throughput=3.85MB

combined: 3852.11 msgs/sec averaged over all receivers (throughput=3.85MB/sec)


Sample Output:

2 Nodes:

----------------------- TEST -----------------------
Date: Mon Nov 05 04:30:54 EST 2007
Run by: root

mcast_port: 7500
log_interval: 1000
sender: true
props: JGroups-2.5.1.bin/tcp.xml
jmx: false
bind_addr: localhost
num_members: 2
msg_size: 1000
dump_transport_stats: false
start_port: 7800
topic: topic/testTopic
num_senders: 2
cluster: 10.255.23.160:7800,10.255.26.143:7801
num_msgs: 10000
transport: org.jgroups.tests.perf.transports.JGroupsTransport
config: JGroups-2.5.1.bin/config.txt
processing_delay: 0
mcast_addr: 228.1.2.3
JGroups version: 2.5.1

Nov 5, 2007 4:30:54 AM org.jgroups.JChannel init
INFO: JGroups version: 2.5.1

-------------------------------------------------------
GMS: address is 10.255.26.143:7800
-------------------------------------------------------
-- 10.255.26.143:7800 joined
-- waiting for 2 members to join
-- 10.255.23.160:7800 joined
-- READY (2 acks)

-- sending 10000 1KB messages
-- received 1000 messages
-- received 2000 messages
++ sent 1000
-- received 3000 messages
++ sent 2000
-- received 4000 messages
-- received 5000 messages
++ sent 3000
-- received 6000 messages
++ sent 4000
-- received 7000 messages
-- received 8000 messages
++ sent 5000
-- received 9000 messages
-- received 10000 messages
-- received 11000 messages
++ sent 6000
-- received 12000 messages
++ sent 7000
-- received 13000 messages
-- received 14000 messages
-- received 15000 messages
++ sent 8000
-- received 16000 messages
-- received 17000 messages
++ sent 9000
-- received 18000 messages
-- received 19000 messages
++ sent 10000
-- received 20000 messages

-- results:

10.255.23.160:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=4664ms, msgs/sec=4288.16, throughput=4.29MB

10.255.26.143:7800 (myself):
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=4903ms, msgs/sec=4079.14, throughput=4.08MB

combined: 4183.65 msgs/sec averaged over all receivers (throughput=4.18MB/sec)


4 Nodes:

Sender Node Output


----------------------- TEST -----------------------
Date: Mon Nov 05 04:30:54 EST 2007
Run by: root

mcast_port: 7500
log_interval: 1000
sender: true
props: JGroups-2.5.1.bin/tcp.xml
jmx: false
bind_addr: localhost
num_members: 2
msg_size: 1000
dump_transport_stats: false
start_port: 7800
topic: topic/testTopic
num_senders: 2
cluster: 10.255.23.160:7800,10.255.26.143:7801
num_msgs: 10000
transport: org.jgroups.tests.perf.transports.JGroupsTransport
config: JGroups-2.5.1.bin/config.txt
processing_delay: 0
mcast_addr: 228.1.2.3
JGroups version: 2.5.1

Nov 5, 2007 4:30:54 AM org.jgroups.JChannel init
INFO: JGroups version: 2.5.1

-------------------------------------------------------
GMS: address is 10.255.26.143:7800
-------------------------------------------------------
-- 10.255.26.143:7800 joined
-- waiting for 2 members to join
-- 10.255.23.160:7800 joined
-- READY (2 acks)

-- sending 10000 1KB messages
-- received 1000 messages
-- received 2000 messages
++ sent 1000
-- received 3000 messages
++ sent 2000
-- received 4000 messages
-- received 5000 messages
++ sent 3000
-- received 6000 messages
++ sent 4000
-- received 7000 messages
-- received 8000 messages
++ sent 5000
-- received 9000 messages
-- received 10000 messages
-- received 11000 messages
++ sent 6000
-- received 12000 messages
++ sent 7000
-- received 13000 messages
-- received 14000 messages
-- received 15000 messages
++ sent 8000
-- received 16000 messages
-- received 17000 messages
++ sent 9000
-- received 18000 messages
-- received 19000 messages
++ sent 10000
-- received 20000 messages

-- results:

10.255.23.160:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=4664ms, msgs/sec=4288.16, throughput=4.29MB

10.255.26.143:7800 (myself):
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=4903ms, msgs/sec=4079.14, throughput=4.08MB

combined: 4183.65 msgs/sec averaged over all receivers (throughput=4.18MB/sec)


Receiver Node Output

----------------------- TEST -----------------------
Date: Mon Nov 05 04:46:12 EST 2007
Run by: root

mcast_port: 7500
log_interval: 1000
sender: false
props: JGroups-2.5.1.bin/tcp.xml
jmx: false
bind_addr: localhost
num_members: 4
msg_size: 1000
dump_transport_stats: false
start_port: 7800
topic: topic/testTopic
num_senders: 2
cluster: 10.255.23.160:7800,10.255.26.143:7801,10.253.83.143:7802,10.253.15.95:7803
num_msgs: 10000
transport: org.jgroups.tests.perf.transports.JGroupsTransport
config: JGroups-2.5.1.bin/config.txt
processing_delay: 0
mcast_addr: 228.1.2.3
JGroups version: 2.5.1

Nov 5, 2007 4:46:12 AM org.jgroups.JChannel init
INFO: JGroups version: 2.5.1

-------------------------------------------------------
GMS: address is 10.253.83.143:7800
-------------------------------------------------------
-- 10.253.15.95:7800 joined
-- 10.253.83.143:7800 joined
-- waiting for 4 members to join
-- 10.255.23.160:7800 joined
-- 10.255.26.143:7800 joined
-- READY (4 acks)

-- received 1000 messages
-- received 2000 messages
-- received 3000 messages
-- received 4000 messages
-- received 5000 messages
-- received 6000 messages
-- received 7000 messages
-- received 8000 messages
-- received 9000 messages
-- received 10000 messages
-- received 11000 messages
-- received 12000 messages
-- received 13000 messages
-- received 14000 messages
-- received 15000 messages
-- received 16000 messages
-- received 17000 messages
-- received 18000 messages
-- received 19000 messages
-- received 20000 messages

-- local results:
sender: 10.255.23.160:7800: num_msgs_expected=10000, num_msgs_received=10000 (loss rate=0.0%), received=10MB, time=5180ms,

msgs/sec=1930.5, throughput=1.93MB
sender: 10.253.15.95:7800: num_msgs_expected=10000, num_msgs_received=10000 (loss rate=0.0%), received=10MB, time=4832ms,

msgs/sec=2069.54, throughput=2.07MB


-- results:

10.253.15.95:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5212ms, msgs/sec=3837.3, throughput=3.84MB

10.255.23.160:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5174ms, msgs/sec=3865.48, throughput=3.87MB

10.255.26.143:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5192ms, msgs/sec=3852.08, throughput=3.85MB

10.253.83.143:7800 (myself):
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5190ms, msgs/sec=3853.56, throughput=3.85MB

combined: 3852.11 msgs/sec averaged over all receivers (throughput=3.85MB/sec)

-- results:

10.253.15.95:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5212ms, msgs/sec=3837.3, throughput=3.84MB

10.255.23.160:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5174ms, msgs/sec=3865.48, throughput=3.87MB

10.255.26.143:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5192ms, msgs/sec=3852.08, throughput=3.85MB

10.253.83.143:7800 (myself):
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5190ms, msgs/sec=3853.56, throughput=3.85MB

combined: 3852.11 msgs/sec averaged over all receivers (throughput=3.85MB/sec)


Sample config.txt file

############################
# only used by TCP Transport
############################

# List of hosts in the cluster. Since we don't specify ports, you cannot run multiple TcpTransports
# on the same machine: each ember has to be run on a separate machine (this may be changed in a next version)
#cluster=127.0.0.1:7800,127.0.0.1:7801
# 2nodes # cluster=10.255.23.160:7800,10.255.26.143:7801
cluster=10.255.23.160:7800,10.255.26.143:7801,10.253.83.143:7802,10.253.15.95:7803


Sample hosts line in tcp.xml



initial_hosts="${jgroups.tcpping.initial_hosts:10.255.23.160[7800],10.255.26.143[7801],10.253.83.143[7802],10.253.15.95[7803]}"

IOzone benchmark on EC2

I have run off another IO benchmark on EC2 using IOZone.

The OS is Centos 4.4 OS, running on Amazon Machine Image (AMI), which is a Zen based Virtual Machine (VM).

There seem to be a couple of sweet spots identified by the benchmark.

1. To stay in CPU Cache keep your file size smaller than under 256KB and read and write in 64 Byte chunks
2. If you must read from larger files, the size of the cache is the maximum size to remain in memory
3. Reading is best done between 64 Byte to 512 Byte chunks.
4. At least on EC2 stay away from reading 16 Kbyte files in 128 byte, 1K and 8K chunks.
   

The results and larger graphs can be found here

http://s3.amazonaws.com/dbadojo_benchmark/iozone_ec2_write.GIF
http://s3.amazonaws.com/dbadojo_benchmark/iozone_ec2_read.GIF
http://s3.amazonaws.com/dbadojo_benchmark/iozone_ec2_random_read.GIF
http://s3.amazonaws.com/dbadojo_benchmark/iozone_ec2_random_write.GIF
http://s3.amazonaws.com/dbadojo_benchmark/iozone_benchmark_ec2.zip (Right Click SAVE AS)

Have Fun

Paul

Bonnie IO benchmark on EC2

I thought it might be useful to cross-link to an article I posted about running the bonnie and bonnie++ IO benchmark tool against EC2.

http://blog.dbadojo.com/2007/10/bonnie-io-benchmark-vs-ec2.html

Now back to finalizing testing JGroups on 2 separate nodes.

Have Fun

Paul