VM Datamining

Saturday, January 19, 2008

Virtual Datamine Roadmap Q1 2008

I thought I would update what is coming in terms of subject areas and posts.

Pentaho Clusters on EC2:

Installing JBoss AS
Combining all the parts to form the single compute node
Installing software for Cluster master

Kettle on EC2:

Installing Kettle on EC2.
Running Kettle Slaves on EC2.
Demo of remote transformation using EC2 Kettle nodes.

Weka on EC2:

Install and demo of GridWeka
Install and demo of WekaAsWS (Weka as a Webservice)
Using Weka to mine clickstreams

Globus Cluster Software on EC2:

Install a master node.
Install of grid/slave/client node.

That should keep me busy until at least April.

Once the services are stable, the next stage is to start presenting them for consumption.

I am looking at a combination of free and various levels of paid services using a similar
model to telco cell/mobile phone credits.

Have Fun

Saturday, December 8, 2007

JGroups Cluster on EC2 large 64bit instances

Introduction:

In this series of articles I have been covering getting the proposed Pentaho Cluster running on EC2.
http://blog.vmdatamine.com/2007/09/pentaho-business-suite-cluster-research.html
http://blog.vmdatamine.com/2007/09/pentaho-cluster-installing-jgroups.html
http://blog.vmdatamine.com/2007/11/pentaho-cluster-installing-jgroups-on.html

In the last article, I ran a JGroups cluster test and found the results disappointing compared to the test results published by JBoss.

So given there are new larger instances available now with better performance I decided to see how JGroups would perform on those instances.

The specification of the large instances can be found in Amazon's announcement.

The only change from the last test from an upgrade to Java (JDK 6 Release 3) and running on Amazon public image Fedora 64 bit OS.

I tried both the large and extra large instances in a 4 node cluster setup running TCP.

Comments:

The network bandwidth is still the limiting factor.
I had to modify the tcp.xml settings to enable queues to stop the test hanging sporadically.
The larger 64 bit instances have more throughput vs the small nodes. This could be due to settings or CPU is an underlining factor after all.
You are not going to reach the JBoss performance results without a faster network.

Results:

Two nodes: 2 senders:

-- results:

10.252.93.220:7800 (myself):
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=3211ms, msgs/sec=6228.59, throughput=6.23MB

10.252.99.47:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=3236ms, msgs/sec=6180.47, throughput=6.18MB

combined: 6204.53 msgs/sec averaged over all receivers (throughput=6.2MB/sec)

Two nodes: 1 sender, 1 receiver

-- results:

10.252.93.220:7800 (myself):
num_msgs_expected=10000, num_msgs_received=10000 (loss rate=0.0%), received=10MB, time=2607ms, msgs/sec=3835.83, throughput=3.84MB

10.252.99.47:7800:
num_msgs_expected=10000, num_msgs_received=10000 (loss rate=0.0%), received=10MB, time=2515ms, msgs/sec=3976.14, throughput=3.98MB

combined: 3905.98 msgs/sec averaged over all receivers
(throughput=3.9MB/sec)

4 nodes: 2 senders, 2 receivers

-- results:

10.252.23.15:7800:

num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=4640ms, msgs/sec=4310.34, throughput=4.31MB
10.252.98.208:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=4687ms, msgs/sec=4267.12, throughput=4.27MB
10.252.79.0:7800 (myself):
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=4642ms, msgs/sec=4308.49, throughput=4.31MB
10.252.93.203:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=4654ms, msgs/sec=4297.38, throughput=4.3MB

combined: 4295.83 msgs/sec averaged over all receivers (throughput=4.3MB/sec)

4 nodes: 2 senders, 2 receivers 100k messages test

-- results:

10.252.23.15:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=19828ms, msgs/sec=10086.75, throughput=10.09MB

10.252.98.208:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=19893ms, msgs/sec=10053.79, throughput=10.05MB

10.252.79.0:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=19828ms, msgs/sec=10086.75, throughput=10.09MB

10.252.93.203:7800 (myself):
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=19917ms, msgs/sec=10041.67, throughput=10.04MB

combined: 10067.24 msgs/sec averaged over all receivers (throughput=10.07MB/sec)

2nd run:

-- results:

10.252.23.15:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=20605ms, msgs/sec=9706.38, throughput=9.71MB

10.252.98.208:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=20629ms, msgs/sec=9695.09, throughput=9.7MB

10.252.79.0:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=20590ms, msgs/sec=9713.45, throughput=9.71MB

10.252.93.203:7800 (myself):
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=20636ms, msgs/sec=9691.8, throughput=9.69MB

combined: 9701.68 msgs/sec averaged over all receivers (throughput=9.7MB/sec)

Extra large 4 nodes : 2 senders, 2 receivers

-- results:

10.252.106.3:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=17857ms, msgs/sec=11200.09, throughput=11.2MB

10.252.15.79:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=17831ms, msgs/sec=11216.42, throughput=11.22MB

10.252.10.223:7800 (myself):
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=17837ms, msgs/sec=11212.65, throughput=11.21MB

10.252.6.223:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=17837ms, msgs/sec=11212.65, throughput=11.21MB

combined: 11210.45 msgs/sec averaged over all receivers (throughput=11.21MB/sec)

-- results:

10.252.106.3:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=15585ms, msgs/sec=12832.85, throughput=12.83MB

10.252.15.79:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=15564ms, msgs/sec=12850.17, throughput=12.85MB

10.252.10.223:7800 (myself):
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=15563ms, msgs/sec=12850.99, throughput=12.85MB

10.252.6.223:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=15563ms, msgs/sec=12850.99, throughput=12.85MB

combined: 12846.25 msgs/sec averaged over all receivers (throughput=12.85MB/sec)

Extra large 4 nodes : 4 senders and TCP queues

-- results:

10.252.106.3:7800:
num_msgs_expected=400000, num_msgs_received=400000 (loss rate=0.0%), received=400MB, time=27784ms, msgs/sec=14396.78, throughput=14.4MB

10.252.15.79:7800:
num_msgs_expected=400000, num_msgs_received=400000 (loss rate=0.0%), received=400MB, time=27744ms, msgs/sec=14417.53, throughput=14.42MB

10.252.10.223:7800 (myself):
num_msgs_expected=400000, num_msgs_received=400000 (loss rate=0.0%), received=400MB, time=27761ms, msgs/sec=14408.7, throughput=14.41MB

10.252.6.223:7800:
num_msgs_expected=400000, num_msgs_received=400000 (loss rate=0.0%), received=400MB, time=27755ms, msgs/sec=14411.82, throughput=14.41MB

combined: 14408.71 msgs/sec averaged over all receivers (throughput=14.41MB/sec)

with TCP queue_max_size set to 1000

10.252.106.3:7800:
num_msgs_expected=400000, num_msgs_received=400000 (loss rate=0.0%), received=400MB, time=26667ms, msgs/sec=14999.81, throughput=15MB

10.252.15.79:7800:
num_msgs_expected=400000, num_msgs_received=400000 (loss rate=0.0%), received=400MB, time=26709ms, msgs/sec=14976.23, throughput=14.98MB

10.252.10.223:7800 (myself):
num_msgs_expected=400000, num_msgs_received=400000 (loss rate=0.0%), received=400MB, time=26676ms, msgs/sec=14994.75, throughput=14.99MB

10.252.6.223:7800:
num_msgs_expected=400000, num_msgs_received=400000 (loss rate=0.0%), received=400MB, time=26710ms, msgs/sec=14975.66, throughput=14.98MB

combined: 14986.61 msgs/sec averaged over all receivers (throughput=14.99MB/sec)

Example of TCP.XML (note &gt and &lt to handle HTML)


<>
   < TCP start_port="7800"
        loopback="false"
        discard_incompatible_packets="true"
        max_bundle_size="64000"
        max_bundle_timeout="30"
        use_incoming_packet_handler="true"
        enable_bundling="true"
        use_send_queues="true"
        sock_conn_timeout="300"
        skip_suspected_members="true"

        use_concurrent_stack="true"

        thread_pool.enabled="true"
        thread_pool.min_threads="8"
        thread_pool.max_threads="40"
        thread_pool.keep_alive_time="5000"
        thread_pool.queue_enabled="true"
        thread_pool.queue_max_size="100"
        thread_pool.rejection_policy="run"

        oob_thread_pool.enabled="true"
        oob_thread_pool.min_threads="8"
        oob_thread_pool.max_threads="20"
        oob_thread_pool.keep_alive_time="5000"
        oob_thread_pool.queue_enabled="true"
        oob_thread_pool.queue_max_size="100"
        oob_thread_pool.rejection_policy="run"/ >

< TCPPING timeout="3000"
            initial_hosts="${jgroups.tcpping.initial_hosts:10.252.106.3[7800],
10.252.15.79[7800],10.252.10.223[7800],10.252.6.223[7800]}"
            port_range="1"
            num_initial_members="2"/ >

Monday, November 5, 2007

Pentaho Cluster : Installing JGroups on EC2

Overview:

Wondering why I hadn't updated my progress with installing JGroups on EC2?
It was because I had three false starts and got nowhere.

Finally however I found some more documentation and was able to get it running.

I found this report about a JGroups Performance test and the associated JBoss wiki Perftests.

That was enough information to understand how to get it working. It also helped that the more recent version JGroups 2.5.1 came with some sample configuration files.

Comments:

The network bandwidth between EC2 nodes is the limiting factor.
For 2 nodes: 4183.65 msgs/sec averaged over all receivers (throughput=4.18MB/sec) vs 60783.12 msgs/sec averaged over all receivers (throughput=60.78MB/sec)
For 4 nodes: 3852.11 msgs/sec averaged over all receivers (throughput=3.85MB/sec)
vs 60783.12 msgs/sec averaged over all receivers (throughput=60.78MB/sec)

The JGroups on EC2 performance versus the JGroups Performance report was very bad. Without changing any settings it was 15 times slower. This is the benefit of having a 1 Gigabit LAN versus 100 Megabit LAN.
The next step would be to test on the larger instances. If the network performance, rated as better for the larger instances versus the default is true, it will show up in the results.

Install:

wget http://easynews.dl.sourceforge.net/sourceforge/javagroups/JGroups-2.5.1.bin.zip
unzip JGroups-2.5.1.bin.zip -d YourJavaLibDirectory
cd YourJavaLibDirectory.
nslookup `hostname` to get your servers IP address.
edit the JGroups-2.5.1.bin/config.txt and JGroups-2.5.1.bin/tcp.xml to add the hosts. See the sample files at the bottom of this post.
java -cp JGroups-2.5.1.bin/concurrent.jar:JGroups-2.5.1.bin/jgroups-all.jar:JGroups-2.5.1.bin/commons-logging.jar org.jgroups.tests.perf.Test -receiver -config JGroups-2.5.1.bin/config.txt -props JGroups-2.5.1.bin/tcp.xml
java -cp JGroups-2.5.1.bin/concurrent.jar:JGroups-2.5.1.bin/jgroups-all.jar:JGroups-2.5.1.bin/commons-logging.jar org.jgroups.tests.perf.Test -sender -config JGroups-2.5.1.bin/config.txt -props JGroups-2.5.1.bin/tcp.xml
If you have the hosts correct it should run the test.

Results:

2 nodes
-- results:

10.255.23.160:7800 (myself):
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=4664ms, msgs/sec=4288.16, throughput=4.29MB

10.255.26.143:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=4903ms, msgs/sec=4079.14, throughput=4.08MB

combined: 4183.65 msgs/sec averaged over all receivers (throughput=4.18MB/sec)

4 nodes (2 senders, 2 receivers):

-- results:

10.253.15.95:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5212ms, msgs/sec=3837.3, throughput=3.84MB

10.255.23.160:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5174ms, msgs/sec=3865.48, throughput=3.87MB

10.255.26.143:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5192ms, msgs/sec=3852.08, throughput=3.85MB

10.253.83.143:7800 (myself):
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5190ms, msgs/sec=3853.56, throughput=3.85MB

combined: 3852.11 msgs/sec averaged over all receivers (throughput=3.85MB/sec)

Sample Output:

2 Nodes:

----------------------- TEST -----------------------
Date: Mon Nov 05 04:30:54 EST 2007
Run by: root

mcast_port: 7500
log_interval: 1000
sender: true
props: JGroups-2.5.1.bin/tcp.xml
jmx: false
bind_addr: localhost
num_members: 2
msg_size: 1000
dump_transport_stats: false
start_port: 7800
topic: topic/testTopic
num_senders: 2
cluster: 10.255.23.160:7800,10.255.26.143:7801
num_msgs: 10000
transport: org.jgroups.tests.perf.transports.JGroupsTransport
config: JGroups-2.5.1.bin/config.txt
processing_delay: 0
mcast_addr: 228.1.2.3
JGroups version: 2.5.1

Nov 5, 2007 4:30:54 AM org.jgroups.JChannel init
INFO: JGroups version: 2.5.1

-------------------------------------------------------
GMS: address is 10.255.26.143:7800
-------------------------------------------------------
-- 10.255.26.143:7800 joined
-- waiting for 2 members to join
-- 10.255.23.160:7800 joined
-- READY (2 acks)

-- sending 10000 1KB messages
-- received 1000 messages
-- received 2000 messages
++ sent 1000
-- received 3000 messages
++ sent 2000
-- received 4000 messages
-- received 5000 messages
++ sent 3000
-- received 6000 messages
++ sent 4000
-- received 7000 messages
-- received 8000 messages
++ sent 5000
-- received 9000 messages
-- received 10000 messages
-- received 11000 messages
++ sent 6000
-- received 12000 messages
++ sent 7000
-- received 13000 messages
-- received 14000 messages
-- received 15000 messages
++ sent 8000
-- received 16000 messages
-- received 17000 messages
++ sent 9000
-- received 18000 messages
-- received 19000 messages
++ sent 10000
-- received 20000 messages

-- results:

10.255.23.160:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=4664ms, msgs/sec=4288.16, throughput=4.29MB

10.255.26.143:7800 (myself):
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=4903ms, msgs/sec=4079.14, throughput=4.08MB

combined: 4183.65 msgs/sec averaged over all receivers (throughput=4.18MB/sec)

4 Nodes:

Sender Node Output

----------------------- TEST -----------------------
Date: Mon Nov 05 04:30:54 EST 2007
Run by: root

mcast_port: 7500
log_interval: 1000
sender: true
props: JGroups-2.5.1.bin/tcp.xml
jmx: false
bind_addr: localhost
num_members: 2
msg_size: 1000
dump_transport_stats: false
start_port: 7800
topic: topic/testTopic
num_senders: 2
cluster: 10.255.23.160:7800,10.255.26.143:7801
num_msgs: 10000
transport: org.jgroups.tests.perf.transports.JGroupsTransport
config: JGroups-2.5.1.bin/config.txt
processing_delay: 0
mcast_addr: 228.1.2.3
JGroups version: 2.5.1

Nov 5, 2007 4:30:54 AM org.jgroups.JChannel init
INFO: JGroups version: 2.5.1

-------------------------------------------------------
GMS: address is 10.255.26.143:7800
-------------------------------------------------------
-- 10.255.26.143:7800 joined
-- waiting for 2 members to join
-- 10.255.23.160:7800 joined
-- READY (2 acks)

-- sending 10000 1KB messages
-- received 1000 messages
-- received 2000 messages
++ sent 1000
-- received 3000 messages
++ sent 2000
-- received 4000 messages
-- received 5000 messages
++ sent 3000
-- received 6000 messages
++ sent 4000
-- received 7000 messages
-- received 8000 messages
++ sent 5000
-- received 9000 messages
-- received 10000 messages
-- received 11000 messages
++ sent 6000
-- received 12000 messages
++ sent 7000
-- received 13000 messages
-- received 14000 messages
-- received 15000 messages
++ sent 8000
-- received 16000 messages
-- received 17000 messages
++ sent 9000
-- received 18000 messages
-- received 19000 messages
++ sent 10000
-- received 20000 messages

-- results:

10.255.23.160:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=4664ms, msgs/sec=4288.16, throughput=4.29MB

10.255.26.143:7800 (myself):
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=4903ms, msgs/sec=4079.14, throughput=4.08MB

combined: 4183.65 msgs/sec averaged over all receivers (throughput=4.18MB/sec)

Receiver Node Output

----------------------- TEST -----------------------
Date: Mon Nov 05 04:46:12 EST 2007
Run by: root

mcast_port: 7500
log_interval: 1000
sender: false
props: JGroups-2.5.1.bin/tcp.xml
jmx: false
bind_addr: localhost
num_members: 4
msg_size: 1000
dump_transport_stats: false
start_port: 7800
topic: topic/testTopic
num_senders: 2
cluster: 10.255.23.160:7800,10.255.26.143:7801,10.253.83.143:7802,10.253.15.95:7803
num_msgs: 10000
transport: org.jgroups.tests.perf.transports.JGroupsTransport
config: JGroups-2.5.1.bin/config.txt
processing_delay: 0
mcast_addr: 228.1.2.3
JGroups version: 2.5.1

Nov 5, 2007 4:46:12 AM org.jgroups.JChannel init
INFO: JGroups version: 2.5.1

-------------------------------------------------------
GMS: address is 10.253.83.143:7800
-------------------------------------------------------
-- 10.253.15.95:7800 joined
-- 10.253.83.143:7800 joined
-- waiting for 4 members to join
-- 10.255.23.160:7800 joined
-- 10.255.26.143:7800 joined
-- READY (4 acks)

-- received 1000 messages
-- received 2000 messages
-- received 3000 messages
-- received 4000 messages
-- received 5000 messages
-- received 6000 messages
-- received 7000 messages
-- received 8000 messages
-- received 9000 messages
-- received 10000 messages
-- received 11000 messages
-- received 12000 messages
-- received 13000 messages
-- received 14000 messages
-- received 15000 messages
-- received 16000 messages
-- received 17000 messages
-- received 18000 messages
-- received 19000 messages
-- received 20000 messages

-- local results:
sender: 10.255.23.160:7800: num_msgs_expected=10000, num_msgs_received=10000 (loss rate=0.0%), received=10MB, time=5180ms,

msgs/sec=1930.5, throughput=1.93MB
sender: 10.253.15.95:7800: num_msgs_expected=10000, num_msgs_received=10000 (loss rate=0.0%), received=10MB, time=4832ms,

msgs/sec=2069.54, throughput=2.07MB

-- results:

10.253.15.95:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5212ms, msgs/sec=3837.3, throughput=3.84MB

10.255.23.160:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5174ms, msgs/sec=3865.48, throughput=3.87MB

10.255.26.143:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5192ms, msgs/sec=3852.08, throughput=3.85MB

10.253.83.143:7800 (myself):
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5190ms, msgs/sec=3853.56, throughput=3.85MB

combined: 3852.11 msgs/sec averaged over all receivers (throughput=3.85MB/sec)

-- results:

10.253.15.95:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5212ms, msgs/sec=3837.3, throughput=3.84MB

10.255.23.160:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5174ms, msgs/sec=3865.48, throughput=3.87MB

10.255.26.143:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5192ms, msgs/sec=3852.08, throughput=3.85MB

10.253.83.143:7800 (myself):
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5190ms, msgs/sec=3853.56, throughput=3.85MB

combined: 3852.11 msgs/sec averaged over all receivers (throughput=3.85MB/sec)

Sample config.txt file

############################
# only used by TCP Transport
############################

# List of hosts in the cluster. Since we don't specify ports, you cannot run multiple TcpTransports
# on the same machine: each ember has to be run on a separate machine (this may be changed in a next version)
#cluster=127.0.0.1:7800,127.0.0.1:7801
# 2nodes # cluster=10.255.23.160:7800,10.255.26.143:7801
cluster=10.255.23.160:7800,10.255.26.143:7801,10.253.83.143:7802,10.253.15.95:7803

Sample hosts line in tcp.xml

initial_hosts="${jgroups.tcpping.initial_hosts:10.255.23.160[7800],10.255.26.143[7801],10.253.83.143[7802],10.253.15.95[7803]}"

Thursday, October 18, 2007

IOzone benchmark on EC2

I have run off another IO benchmark on EC2 using IOZone.

The OS is Centos 4.4 OS, running on Amazon Machine Image (AMI), which is a Zen based Virtual Machine (VM).

There seem to be a couple of sweet spots identified by the benchmark.

To stay in CPU Cache keep your file size smaller than under 256KB and read and write in 64 Byte chunks
If you must read from larger files, the size of the cache is the maximum size to remain in memory
Reading is best done between 64 Byte to 512 Byte chunks.
At least on EC2 stay away from reading 16 Kbyte files in 128 byte, 1K and 8K chunks.

The results and larger graphs can be found here

http://s3.amazonaws.com/dbadojo_benchmark/iozone_ec2_write.GIF
http://s3.amazonaws.com/dbadojo_benchmark/iozone_ec2_read.GIF
http://s3.amazonaws.com/dbadojo_benchmark/iozone_ec2_random_read.GIF
http://s3.amazonaws.com/dbadojo_benchmark/iozone_ec2_random_write.GIF
http://s3.amazonaws.com/dbadojo_benchmark/iozone_benchmark_ec2.zip (Right Click SAVE AS)

Have Fun

Paul

Thursday, October 4, 2007

Bonnie IO benchmark on EC2

I thought it might be useful to cross-link to an article I posted about running the bonnie and bonnie++ IO benchmark tool against EC2.

http://blog.dbadojo.com/2007/10/bonnie-io-benchmark-vs-ec2.html

Now back to finalizing testing JGroups on 2 separate nodes.

Have Fun

Paul

Thursday, September 20, 2007

Pentaho Cluster: Installing JGroups

As I mentioned in the previous post I am going to attempt to make a Pentaho Cluster, based on this presentation(PDF) and outlined in this preparation post.

So the fundamental building block for this cluster is the JGroups Java libraries, so first cab off the rank is to download and install the jar files and get demo to work.

After some mishaps with my CLASSPATH I got that sorted and tested the demo using two JGroups instances running on the same node. See the picture. Drawing in one window was immediately reflected in the second window on the right.

The install.html which is part of the zipped download file was good and explained the procedure well.

So the next thing is try the JGroups clustering on two separate EC2 nodes. That is next...

Have Fun

Paul

Installing JGroups onto a EC2 node with Java already installed.

Check Java is installed

java -version


java version "1.5.0_12"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_12-b04)
Java HotSpot(TM) Client VM (build 1.5.0_12-b04, mixed mode, sharing)

Download JGroups

http://labs.jboss.com/jgroups/downloads
wget http://easynews.dl.sourceforge.net/sourceforge/javagroups/JGroups-2.3.bin.zip

Unzip and copy files to /usr/local

unzip JGroups-2.3.bin.zip -d /usr/local

Run the checker to make sure you have installed correctly

java -jar JGroups-2.3.bin/jgroups-all.jar


Version:        2.3
CVS:            $Id: Version.java,v 1.35 2006/06/11 19:15:23 belaban Exp $
History:        (see doc/history.txt for details)

Start X-window to EC2

ssh -i id_rsa-gsg-keypair -X root@yourEC2host

Check the display is set correctly

echo $DISPLAY

localhost:10.0

Run the demo twice, use & to background each command from command line.

cd /usr/local
java -cp JGroups-2.3.bin/concurrent.jar:JGroups-2.3.bin/jgroups-all.jar:JGroups-2.3.bin/commons-logging.jar org.jgroups.demos.Draw


Sep 20, 2007 7:34:29 AM org.jgroups.protocols.UDP createSockets
INFO: sockets will use interface 10.253.22.176
Sep 20, 2007 7:34:29 AM org.jgroups.protocols.UDP createSockets
INFO: socket information:
local_addr=10.253.22.176:32772, mcast_addr=228.8.8.8:45566, bind_addr=/10.253.22.176, ttl=32
sock: bound to 10.253.22.176:32772, receive buffer size=64000, send buffer size=32000
mcast_recv_sock: bound to 10.253.22.176:45566, send buffer size=64000, receive buffer size=64000
mcast_send_sock: bound to 10.253.22.176:32773, send buffer size=64000, receive buffer size=64000

-------------------------------------------------------
GMS: address is 10.253.22.176:32772
-------------------------------------------------------
** View=[10.253.22.176:32769|1] [10.253.22.176:32769, 10.253.22.176:32772]
** View=[10.253.22.176:32769|1] [10.253.22.176:32769, 10.253.22.176:32772]

Monday, September 10, 2007

Pentaho Business Suite Cluster : Research and preparation

As I mentioned in this article on running the Pentaho Demo on EC2, according to a recent presentation, Pentaho has the ability to be clustered using JBoss Clustering (JBoss JGroups), JBoss AS and Apache.

This post is to outline the background documentation I am using to give this cluster a go on EC2.
There are some posts indicating that the lack of multicasting with EC2 is a issue, however comments and the doco suggest that it will revert to TCP, although that is slower.

Given I already have Pentaho installed and saved as an AMI, I am going to build on that to make the single node. The documentation and wikis confident that it should auto discover and be a piece of cake, however we shall see.

Have Fun

Paul

Here is the specification of the cluster node (from that presentation)

Single Node

Single CPU
2 GB RAM
JBoss AS 4.0.3
JBoss JGroups?
Pentaho 1.1.5

Cluster Master:

Apache HTTP server 2.0.58 with mod_jk module version 1.2.15
JGroups cluster master
JMS / Web Services for Operational BI

Doco (Documentation)

http://docs.jboss.org/jbossas/jboss4guide/r4/html/jbosscache.chapt.html
http://docs.jboss.org/jbossas/jboss4guide/r4/html/cluster.chapt.html
http://www.onjava.com/pub/a/onjava/2002/07/10/jboss.html
http://www.jboss.org/wiki/Wiki.jsp?page=JBossHA
http://www.jboss.org/developers/projects/jboss/clustering
http://clusterstore.demo.jboss.com/

Forums and Blogs:

http://www.jgroups.org/javagroupsnew/docs/Perftest.html
http://blog.decaresystems.ie/index.php/2007/01/29/amazon-web-services-the-future-of-datacenter-computing-part-1
http://blog.decaresystems.ie/index.php/2007/02/12/amazon-web-services-the-future-of-datacenter-computing-part-2

Apache
http://jakarta.apache.org/tomcat/tomcat-3.3-doc/mod_jk-howto.html

Downloads

http://labs.jboss.com/jbossas/downloads
http://labs.jboss.com/jgroups/downloads
http://labs.jboss.com/jbosscache/download/index.html