JGroups Cluster on EC2 large 64bit instances

Introduction:

In this series of articles I have been covering getting the proposed Pentaho Cluster running on EC2.
http://blog.vmdatamine.com/2007/09/pentaho-business-suite-cluster-research.html
http://blog.vmdatamine.com/2007/09/pentaho-cluster-installing-jgroups.html
http://blog.vmdatamine.com/2007/11/pentaho-cluster-installing-jgroups-on.html

In the last article, I ran a JGroups cluster test and found the results disappointing compared to the test results published by JBoss.

So given there are new larger instances available now with better performance I decided to see how JGroups would perform on those instances.

The specification of the large instances can be found in Amazon's announcement.

The only change from the last test from an upgrade to Java (JDK 6 Release 3) and running on Amazon public image Fedora 64 bit OS.

I tried both the large and extra large instances in a 4 node cluster setup running TCP.

Comments:

1. The network bandwidth is still the limiting factor.
2. I had to modify the tcp.xml settings to enable queues to stop the test hanging sporadically.
3. The larger 64 bit instances have more throughput vs the small nodes. This could be due to settings or CPU is an underlining factor after all.
4. You are not going to reach the JBoss performance results without a faster network.

Results:

Two nodes: 2 senders:

-- results:

10.252.93.220:7800 (myself):
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=3211ms, msgs/sec=6228.59, throughput=6.23MB

10.252.99.47:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=3236ms, msgs/sec=6180.47, throughput=6.18MB

combined: 6204.53 msgs/sec averaged over all receivers (throughput=6.2MB/sec)

Two nodes: 1 sender, 1 receiver

-- results:

10.252.93.220:7800 (myself):
num_msgs_expected=10000, num_msgs_received=10000 (loss rate=0.0%), received=10MB, time=2607ms, msgs/sec=3835.83, throughput=3.84MB

10.252.99.47:7800:
num_msgs_expected=10000, num_msgs_received=10000 (loss rate=0.0%), received=10MB, time=2515ms, msgs/sec=3976.14, throughput=3.98MB

combined: 3905.98 msgs/sec averaged over all receivers
(throughput=3.9MB/sec)

4 nodes: 2 senders, 2 receivers

-- results:

10.252.23.15:7800:

num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=4640ms, msgs/sec=4310.34, throughput=4.31MB
10.252.98.208:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=4687ms, msgs/sec=4267.12, throughput=4.27MB
10.252.79.0:7800 (myself):
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=4642ms, msgs/sec=4308.49, throughput=4.31MB
10.252.93.203:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=4654ms, msgs/sec=4297.38, throughput=4.3MB

combined: 4295.83 msgs/sec averaged over all receivers (throughput=4.3MB/sec)


4 nodes: 2 senders, 2 receivers 100k messages test

-- results:

10.252.23.15:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=19828ms, msgs/sec=10086.75, throughput=10.09MB

10.252.98.208:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=19893ms, msgs/sec=10053.79, throughput=10.05MB

10.252.79.0:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=19828ms, msgs/sec=10086.75, throughput=10.09MB

10.252.93.203:7800 (myself):
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=19917ms, msgs/sec=10041.67, throughput=10.04MB

combined: 10067.24 msgs/sec averaged over all receivers (throughput=10.07MB/sec)

2nd run:

-- results:

10.252.23.15:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=20605ms, msgs/sec=9706.38, throughput=9.71MB

10.252.98.208:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=20629ms, msgs/sec=9695.09, throughput=9.7MB

10.252.79.0:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=20590ms, msgs/sec=9713.45, throughput=9.71MB

10.252.93.203:7800 (myself):
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=20636ms, msgs/sec=9691.8, throughput=9.69MB

combined: 9701.68 msgs/sec averaged over all receivers (throughput=9.7MB/sec)

Extra large 4 nodes : 2 senders, 2 receivers

-- results:

10.252.106.3:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=17857ms, msgs/sec=11200.09, throughput=11.2MB

10.252.15.79:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=17831ms, msgs/sec=11216.42, throughput=11.22MB

10.252.10.223:7800 (myself):
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=17837ms, msgs/sec=11212.65, throughput=11.21MB

10.252.6.223:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=17837ms, msgs/sec=11212.65, throughput=11.21MB

combined: 11210.45 msgs/sec averaged over all receivers (throughput=11.21MB/sec)


-- results:

10.252.106.3:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=15585ms, msgs/sec=12832.85, throughput=12.83MB

10.252.15.79:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=15564ms, msgs/sec=12850.17, throughput=12.85MB

10.252.10.223:7800 (myself):
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=15563ms, msgs/sec=12850.99, throughput=12.85MB

10.252.6.223:7800:
num_msgs_expected=200000, num_msgs_received=200000 (loss rate=0.0%), received=200MB, time=15563ms, msgs/sec=12850.99, throughput=12.85MB

combined: 12846.25 msgs/sec averaged over all receivers (throughput=12.85MB/sec)

Extra large 4 nodes : 4 senders and TCP queues

-- results:

10.252.106.3:7800:
num_msgs_expected=400000, num_msgs_received=400000 (loss rate=0.0%), received=400MB, time=27784ms, msgs/sec=14396.78, throughput=14.4MB

10.252.15.79:7800:
num_msgs_expected=400000, num_msgs_received=400000 (loss rate=0.0%), received=400MB, time=27744ms, msgs/sec=14417.53, throughput=14.42MB

10.252.10.223:7800 (myself):
num_msgs_expected=400000, num_msgs_received=400000 (loss rate=0.0%), received=400MB, time=27761ms, msgs/sec=14408.7, throughput=14.41MB

10.252.6.223:7800:
num_msgs_expected=400000, num_msgs_received=400000 (loss rate=0.0%), received=400MB, time=27755ms, msgs/sec=14411.82, throughput=14.41MB

combined: 14408.71 msgs/sec averaged over all receivers (throughput=14.41MB/sec)

with TCP queue_max_size set to 1000

10.252.106.3:7800:
num_msgs_expected=400000, num_msgs_received=400000 (loss rate=0.0%), received=400MB, time=26667ms, msgs/sec=14999.81, throughput=15MB

10.252.15.79:7800:
num_msgs_expected=400000, num_msgs_received=400000 (loss rate=0.0%), received=400MB, time=26709ms, msgs/sec=14976.23, throughput=14.98MB

10.252.10.223:7800 (myself):
num_msgs_expected=400000, num_msgs_received=400000 (loss rate=0.0%), received=400MB, time=26676ms, msgs/sec=14994.75, throughput=14.99MB

10.252.6.223:7800:
num_msgs_expected=400000, num_msgs_received=400000 (loss rate=0.0%), received=400MB, time=26710ms, msgs/sec=14975.66, throughput=14.98MB

combined: 14986.61 msgs/sec averaged over all receivers (throughput=14.99MB/sec)

Example of TCP.XML (note &gt and &lt to handle HTML)

<>
   < TCP start_port="7800"
        loopback="false"
        discard_incompatible_packets="true"
        max_bundle_size="64000"
        max_bundle_timeout="30"
        use_incoming_packet_handler="true"
        enable_bundling="true"
        use_send_queues="true"
        sock_conn_timeout="300"
        skip_suspected_members="true"

        use_concurrent_stack="true"

        thread_pool.enabled="true"
        thread_pool.min_threads="8"
        thread_pool.max_threads="40"
        thread_pool.keep_alive_time="5000"
        thread_pool.queue_enabled="true"
        thread_pool.queue_max_size="100"
        thread_pool.rejection_policy="run"

        oob_thread_pool.enabled="true"
        oob_thread_pool.min_threads="8"
        oob_thread_pool.max_threads="20"
        oob_thread_pool.keep_alive_time="5000"
        oob_thread_pool.queue_enabled="true"
        oob_thread_pool.queue_max_size="100"
        oob_thread_pool.rejection_policy="run"/ >

< TCPPING timeout="3000"
            initial_hosts="${jgroups.tcpping.initial_hosts:10.252.106.3[7800],
10.252.15.79[7800],10.252.10.223[7800],10.252.6.223[7800]}"
            port_range="1"
            num_initial_members="2"/ >

Pentaho Cluster : Installing JGroups on EC2

Overview:

Wondering why I hadn't updated my progress with installing JGroups on EC2?
It was because I had three false starts and got nowhere.

Finally however I found some more documentation and was able to get it running.

I found this report about a JGroups Performance test and the associated JBoss wiki Perftests.

That was enough information to understand how to get it working. It also helped that the more recent version JGroups 2.5.1 came with some sample configuration files.

Comments:

1. The network bandwidth between EC2 nodes is the limiting factor.
2. For 2 nodes: 4183.65 msgs/sec averaged over all receivers (throughput=4.18MB/sec) vs 60783.12 msgs/sec averaged over all receivers (throughput=60.78MB/sec)
3. For 4 nodes: 3852.11 msgs/sec averaged over all receivers (throughput=3.85MB/sec)
   vs 60783.12 msgs/sec averaged over all receivers (throughput=60.78MB/sec)

The JGroups on EC2 performance versus the JGroups Performance report was very bad. Without changing any settings it was 15 times slower. This is the benefit of having a 1 Gigabit LAN versus 100 Megabit LAN.
The next step would be to test on the larger instances. If the network performance, rated as better for the larger instances versus the default is true, it will show up in the results.


Install:

1. wget http://easynews.dl.sourceforge.net/sourceforge/javagroups/JGroups-2.5.1.bin.zip
2. unzip JGroups-2.5.1.bin.zip -d YourJavaLibDirectory
3. cd YourJavaLibDirectory.
4. nslookup `hostname` to get your servers IP address.
5. edit the JGroups-2.5.1.bin/config.txt and JGroups-2.5.1.bin/tcp.xml to add the hosts. See the sample files at the bottom of this post.
   
6. java -cp JGroups-2.5.1.bin/concurrent.jar:JGroups-2.5.1.bin/jgroups-all.jar:JGroups-2.5.1.bin/commons-logging.jar org.jgroups.tests.perf.Test -receiver -config JGroups-2.5.1.bin/config.txt -props JGroups-2.5.1.bin/tcp.xml
7. java -cp JGroups-2.5.1.bin/concurrent.jar:JGroups-2.5.1.bin/jgroups-all.jar:JGroups-2.5.1.bin/commons-logging.jar org.jgroups.tests.perf.Test -sender -config JGroups-2.5.1.bin/config.txt -props JGroups-2.5.1.bin/tcp.xml
8. If you have the hosts correct it should run the test.

Results:

2 nodes
-- results:

10.255.23.160:7800 (myself):
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=4664ms, msgs/sec=4288.16, throughput=4.29MB

10.255.26.143:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=4903ms, msgs/sec=4079.14, throughput=4.08MB

combined: 4183.65 msgs/sec averaged over all receivers (throughput=4.18MB/sec)

4 nodes (2 senders, 2 receivers):

-- results:

10.253.15.95:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5212ms, msgs/sec=3837.3, throughput=3.84MB

10.255.23.160:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5174ms, msgs/sec=3865.48, throughput=3.87MB

10.255.26.143:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5192ms, msgs/sec=3852.08, throughput=3.85MB

10.253.83.143:7800 (myself):
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5190ms, msgs/sec=3853.56, throughput=3.85MB

combined: 3852.11 msgs/sec averaged over all receivers (throughput=3.85MB/sec)


Sample Output:

2 Nodes:

----------------------- TEST -----------------------
Date: Mon Nov 05 04:30:54 EST 2007
Run by: root

mcast_port: 7500
log_interval: 1000
sender: true
props: JGroups-2.5.1.bin/tcp.xml
jmx: false
bind_addr: localhost
num_members: 2
msg_size: 1000
dump_transport_stats: false
start_port: 7800
topic: topic/testTopic
num_senders: 2
cluster: 10.255.23.160:7800,10.255.26.143:7801
num_msgs: 10000
transport: org.jgroups.tests.perf.transports.JGroupsTransport
config: JGroups-2.5.1.bin/config.txt
processing_delay: 0
mcast_addr: 228.1.2.3
JGroups version: 2.5.1

Nov 5, 2007 4:30:54 AM org.jgroups.JChannel init
INFO: JGroups version: 2.5.1

-------------------------------------------------------
GMS: address is 10.255.26.143:7800
-------------------------------------------------------
-- 10.255.26.143:7800 joined
-- waiting for 2 members to join
-- 10.255.23.160:7800 joined
-- READY (2 acks)

-- sending 10000 1KB messages
-- received 1000 messages
-- received 2000 messages
++ sent 1000
-- received 3000 messages
++ sent 2000
-- received 4000 messages
-- received 5000 messages
++ sent 3000
-- received 6000 messages
++ sent 4000
-- received 7000 messages
-- received 8000 messages
++ sent 5000
-- received 9000 messages
-- received 10000 messages
-- received 11000 messages
++ sent 6000
-- received 12000 messages
++ sent 7000
-- received 13000 messages
-- received 14000 messages
-- received 15000 messages
++ sent 8000
-- received 16000 messages
-- received 17000 messages
++ sent 9000
-- received 18000 messages
-- received 19000 messages
++ sent 10000
-- received 20000 messages

-- results:

10.255.23.160:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=4664ms, msgs/sec=4288.16, throughput=4.29MB

10.255.26.143:7800 (myself):
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=4903ms, msgs/sec=4079.14, throughput=4.08MB

combined: 4183.65 msgs/sec averaged over all receivers (throughput=4.18MB/sec)


4 Nodes:

Sender Node Output


----------------------- TEST -----------------------
Date: Mon Nov 05 04:30:54 EST 2007
Run by: root

mcast_port: 7500
log_interval: 1000
sender: true
props: JGroups-2.5.1.bin/tcp.xml
jmx: false
bind_addr: localhost
num_members: 2
msg_size: 1000
dump_transport_stats: false
start_port: 7800
topic: topic/testTopic
num_senders: 2
cluster: 10.255.23.160:7800,10.255.26.143:7801
num_msgs: 10000
transport: org.jgroups.tests.perf.transports.JGroupsTransport
config: JGroups-2.5.1.bin/config.txt
processing_delay: 0
mcast_addr: 228.1.2.3
JGroups version: 2.5.1

Nov 5, 2007 4:30:54 AM org.jgroups.JChannel init
INFO: JGroups version: 2.5.1

-------------------------------------------------------
GMS: address is 10.255.26.143:7800
-------------------------------------------------------
-- 10.255.26.143:7800 joined
-- waiting for 2 members to join
-- 10.255.23.160:7800 joined
-- READY (2 acks)

-- sending 10000 1KB messages
-- received 1000 messages
-- received 2000 messages
++ sent 1000
-- received 3000 messages
++ sent 2000
-- received 4000 messages
-- received 5000 messages
++ sent 3000
-- received 6000 messages
++ sent 4000
-- received 7000 messages
-- received 8000 messages
++ sent 5000
-- received 9000 messages
-- received 10000 messages
-- received 11000 messages
++ sent 6000
-- received 12000 messages
++ sent 7000
-- received 13000 messages
-- received 14000 messages
-- received 15000 messages
++ sent 8000
-- received 16000 messages
-- received 17000 messages
++ sent 9000
-- received 18000 messages
-- received 19000 messages
++ sent 10000
-- received 20000 messages

-- results:

10.255.23.160:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=4664ms, msgs/sec=4288.16, throughput=4.29MB

10.255.26.143:7800 (myself):
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=4903ms, msgs/sec=4079.14, throughput=4.08MB

combined: 4183.65 msgs/sec averaged over all receivers (throughput=4.18MB/sec)


Receiver Node Output

----------------------- TEST -----------------------
Date: Mon Nov 05 04:46:12 EST 2007
Run by: root

mcast_port: 7500
log_interval: 1000
sender: false
props: JGroups-2.5.1.bin/tcp.xml
jmx: false
bind_addr: localhost
num_members: 4
msg_size: 1000
dump_transport_stats: false
start_port: 7800
topic: topic/testTopic
num_senders: 2
cluster: 10.255.23.160:7800,10.255.26.143:7801,10.253.83.143:7802,10.253.15.95:7803
num_msgs: 10000
transport: org.jgroups.tests.perf.transports.JGroupsTransport
config: JGroups-2.5.1.bin/config.txt
processing_delay: 0
mcast_addr: 228.1.2.3
JGroups version: 2.5.1

Nov 5, 2007 4:46:12 AM org.jgroups.JChannel init
INFO: JGroups version: 2.5.1

-------------------------------------------------------
GMS: address is 10.253.83.143:7800
-------------------------------------------------------
-- 10.253.15.95:7800 joined
-- 10.253.83.143:7800 joined
-- waiting for 4 members to join
-- 10.255.23.160:7800 joined
-- 10.255.26.143:7800 joined
-- READY (4 acks)

-- received 1000 messages
-- received 2000 messages
-- received 3000 messages
-- received 4000 messages
-- received 5000 messages
-- received 6000 messages
-- received 7000 messages
-- received 8000 messages
-- received 9000 messages
-- received 10000 messages
-- received 11000 messages
-- received 12000 messages
-- received 13000 messages
-- received 14000 messages
-- received 15000 messages
-- received 16000 messages
-- received 17000 messages
-- received 18000 messages
-- received 19000 messages
-- received 20000 messages

-- local results:
sender: 10.255.23.160:7800: num_msgs_expected=10000, num_msgs_received=10000 (loss rate=0.0%), received=10MB, time=5180ms,

msgs/sec=1930.5, throughput=1.93MB
sender: 10.253.15.95:7800: num_msgs_expected=10000, num_msgs_received=10000 (loss rate=0.0%), received=10MB, time=4832ms,

msgs/sec=2069.54, throughput=2.07MB


-- results:

10.253.15.95:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5212ms, msgs/sec=3837.3, throughput=3.84MB

10.255.23.160:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5174ms, msgs/sec=3865.48, throughput=3.87MB

10.255.26.143:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5192ms, msgs/sec=3852.08, throughput=3.85MB

10.253.83.143:7800 (myself):
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5190ms, msgs/sec=3853.56, throughput=3.85MB

combined: 3852.11 msgs/sec averaged over all receivers (throughput=3.85MB/sec)

-- results:

10.253.15.95:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5212ms, msgs/sec=3837.3, throughput=3.84MB

10.255.23.160:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5174ms, msgs/sec=3865.48, throughput=3.87MB

10.255.26.143:7800:
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5192ms, msgs/sec=3852.08, throughput=3.85MB

10.253.83.143:7800 (myself):
num_msgs_expected=20000, num_msgs_received=20000 (loss rate=0.0%), received=20MB, time=5190ms, msgs/sec=3853.56, throughput=3.85MB

combined: 3852.11 msgs/sec averaged over all receivers (throughput=3.85MB/sec)


Sample config.txt file

############################
# only used by TCP Transport
############################

# List of hosts in the cluster. Since we don't specify ports, you cannot run multiple TcpTransports
# on the same machine: each ember has to be run on a separate machine (this may be changed in a next version)
#cluster=127.0.0.1:7800,127.0.0.1:7801
# 2nodes # cluster=10.255.23.160:7800,10.255.26.143:7801
cluster=10.255.23.160:7800,10.255.26.143:7801,10.253.83.143:7802,10.253.15.95:7803


Sample hosts line in tcp.xml



initial_hosts="${jgroups.tcpping.initial_hosts:10.255.23.160[7800],10.255.26.143[7801],10.253.83.143[7802],10.253.15.95[7803]}"

IOzone benchmark on EC2

I have run off another IO benchmark on EC2 using IOZone.

The OS is Centos 4.4 OS, running on Amazon Machine Image (AMI), which is a Zen based Virtual Machine (VM).

There seem to be a couple of sweet spots identified by the benchmark.

1. To stay in CPU Cache keep your file size smaller than under 256KB and read and write in 64 Byte chunks
2. If you must read from larger files, the size of the cache is the maximum size to remain in memory
3. Reading is best done between 64 Byte to 512 Byte chunks.
4. At least on EC2 stay away from reading 16 Kbyte files in 128 byte, 1K and 8K chunks.
   

The results and larger graphs can be found here

http://s3.amazonaws.com/dbadojo_benchmark/iozone_ec2_write.GIF
http://s3.amazonaws.com/dbadojo_benchmark/iozone_ec2_read.GIF
http://s3.amazonaws.com/dbadojo_benchmark/iozone_ec2_random_read.GIF
http://s3.amazonaws.com/dbadojo_benchmark/iozone_ec2_random_write.GIF
http://s3.amazonaws.com/dbadojo_benchmark/iozone_benchmark_ec2.zip (Right Click SAVE AS)

Have Fun

Paul

Bonnie IO benchmark on EC2

I thought it might be useful to cross-link to an article I posted about running the bonnie and bonnie++ IO benchmark tool against EC2.

http://blog.dbadojo.com/2007/10/bonnie-io-benchmark-vs-ec2.html

Now back to finalizing testing JGroups on 2 separate nodes.

Have Fun

Paul

Pentaho Cluster: Installing JGroups

As I mentioned in the previous post I am going to attempt to make a Pentaho Cluster, based on this presentation(PDF) and outlined in this preparation post.

So the fundamental building block for this cluster is the JGroups Java libraries, so first cab off the rank is to download and install the jar files and get demo to work.

After some mishaps with my CLASSPATH I got that sorted and tested the demo using two JGroups instances running on the same node. See the picture. Drawing in one window was immediately reflected in the second window on the right.

The install.html which is part of the zipped download file was good and explained the procedure well.

So the next thing is try the JGroups clustering on two separate EC2 nodes. That is next...

Have Fun

Paul

Installing JGroups onto a EC2 node with Java already installed.

Check Java is installed

java -version

java version "1.5.0_12"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_12-b04)
Java HotSpot(TM) Client VM (build 1.5.0_12-b04, mixed mode, sharing)

Download JGroups

http://labs.jboss.com/jgroups/downloads
wget http://easynews.dl.sourceforge.net/sourceforge/javagroups/JGroups-2.3.bin.zip

Unzip and copy files to /usr/local

unzip JGroups-2.3.bin.zip -d /usr/local

Run the checker to make sure you have installed correctly

java -jar JGroups-2.3.bin/jgroups-all.jar

Version:        2.3
CVS:            $Id: Version.java,v 1.35 2006/06/11 19:15:23 belaban Exp $
History:        (see doc/history.txt for details)

Start X-window to EC2

ssh -i id_rsa-gsg-keypair -X root@yourEC2host

Check the display is set correctly

echo $DISPLAY

localhost:10.0

Run the demo twice, use & to background each command from command line.

cd /usr/local
java -cp JGroups-2.3.bin/concurrent.jar:JGroups-2.3.bin/jgroups-all.jar:JGroups-2.3.bin/commons-logging.jar org.jgroups.demos.Draw

Sep 20, 2007 7:34:29 AM org.jgroups.protocols.UDP createSockets
INFO: sockets will use interface 10.253.22.176
Sep 20, 2007 7:34:29 AM org.jgroups.protocols.UDP createSockets
INFO: socket information:
local_addr=10.253.22.176:32772, mcast_addr=228.8.8.8:45566, bind_addr=/10.253.22.176, ttl=32
sock: bound to 10.253.22.176:32772, receive buffer size=64000, send buffer size=32000
mcast_recv_sock: bound to 10.253.22.176:45566, send buffer size=64000, receive buffer size=64000
mcast_send_sock: bound to 10.253.22.176:32773, send buffer size=64000, receive buffer size=64000

-------------------------------------------------------
GMS: address is 10.253.22.176:32772
-------------------------------------------------------
** View=[10.253.22.176:32769|1] [10.253.22.176:32769, 10.253.22.176:32772]
** View=[10.253.22.176:32769|1] [10.253.22.176:32769, 10.253.22.176:32772]

Pentaho Business Suite Cluster : Research and preparation

As I mentioned in this article on running the Pentaho Demo on EC2, according to a recent presentation, Pentaho has the ability to be clustered using JBoss Clustering (JBoss JGroups), JBoss AS and Apache.

This post is to outline the background documentation I am using to give this cluster a go on EC2.
There are some posts indicating that the lack of multicasting with EC2 is a issue, however comments and the doco suggest that it will revert to TCP, although that is slower.

Given I already have Pentaho installed and saved as an AMI, I am going to build on that to make the single node. The documentation and wikis confident that it should auto discover and be a piece of cake, however we shall see.

Have Fun

Paul

Here is the specification of the cluster node (from that presentation)

Single Node

Single CPU
2 GB RAM
JBoss AS 4.0.3
JBoss JGroups?
Pentaho 1.1.5

Cluster Master:

Apache HTTP server 2.0.58 with mod_jk module version 1.2.15
JGroups cluster master
JMS / Web Services for Operational BI

Doco (Documentation)

http://docs.jboss.org/jbossas/jboss4guide/r4/html/jbosscache.chapt.html
http://docs.jboss.org/jbossas/jboss4guide/r4/html/cluster.chapt.html
http://www.onjava.com/pub/a/onjava/2002/07/10/jboss.html
http://www.jboss.org/wiki/Wiki.jsp?page=JBossHA
http://www.jboss.org/developers/projects/jboss/clustering
http://clusterstore.demo.jboss.com/

Forums and Blogs:

http://www.jgroups.org/javagroupsnew/docs/Perftest.html
http://blog.decaresystems.ie/index.php/2007/01/29/amazon-web-services-the-future-of-datacenter-computing-part-1
http://blog.decaresystems.ie/index.php/2007/02/12/amazon-web-services-the-future-of-datacenter-computing-part-2

Apache
http://jakarta.apache.org/tomcat/tomcat-3.3-doc/mod_jk-howto.html

Downloads

http://labs.jboss.com/jbossas/downloads
http://labs.jboss.com/jgroups/downloads
http://labs.jboss.com/jbosscache/download/index.html

Mondrian OLAP on MySQL EC2 Part 1

If you are wondering why there has been no recent postings, it is due to mainly to struggling to get the Mondrian installed. It had nothing to do with the environment and mostly due to documentation missing vital examples or explanation.

It is with relief I can say I have passed the test and can pass over to the fair lands of using Mondrian OLAP and running the demo.

You can download Mondrian from the Pentaho website or directly from sourceforge.
Use this installation document as a guide however it glosses over some of the nice little details which will make the demo work or not.

Comments:

1. Installing the demo was way too hard, just bundle tomcat or build a include everything binary for MySQL, PostgreSQL, Oracle or whatever. You want people to try the program, not many people will persist like I did to get it to work.
   
2. Having to hand edit every file which connects to the database or is used as a tomcat configuration file sucks, ever heard of storing that stuff in a single file or in the database.
3. Provide a simple SQL script to create schema objects and load data. Running the Foodmart loader did not prove you could run the demo! use this SQL script otherwise from Gizzar
   
4. Some kind of example of either a tomcat or other verification tool would be good, rather than having to lather, rinse and repeat over java, JDBC, Tomcat and red herring errors.

Many thanks to some other blogs and sites which helped solved various issues which cropped up
Gizzar article on Mondrian Open Source OLAP with MySQL
University of Vienna (Wien) for Tomcat FAQ solving tomcat shutdown on missing X
Mondrian Forum

Have Fun

Paul

Install:
Note: This really requires a HOWTO document. I will work on that based on this. Hopefully you can use this in conjunction with installation guide and the various other blogs who had fun with this.

1. Install some linux packages: yum install gcc autoconf
   
2. Download the last release of Mondrian non-embedded files from SourceForge.
3. Download and install Java 1.5 or better from Sun Java Downloads.
   
4. Download and install MySQL 5.0 from MySQL Downloads.
5. Download and unzip the MySQL JDBC driver
   
6. Download and install Apache Tomcat 5.0.28.
7. Verify that Java, MySQL and Tomcat are working by doing the following:
   
8. java -version
9. mysql -V
10. /usr/local/tomcat/bin/startup.sh
11. Point your browser at http://yourhostname:8080, if it works, tomcat is working.
    
12. unzip Mondrian.zip -b /usr/local/mondrian
13. Create the MySQL database: mysqladmin create foodmart -u root -p
14. Create the Foodmart User: create user 'foodmart'@'yourhostname' identified by 'foodmart';
15. Grant permissions: grant all privileges on *.* to 'foodmart'@'yourhostname' identified by 'foodmart';
16. Explode the mondrian.war into /usr/local/tomcat/webapps/mondrian: jar -xvf mondrian.war
17. Locate the 4 jar files eigenbase-properties.jar,eigenbase-resgen.jar,eigenbase-xom.jar and log4j-1.2.9.jar
    
18. Run Mondrian.FoodmartLoader to create tables and load data, pass in the full path to all the required JAR files, otherwise it will fail with a Class notFound error. This is an example:
    



java -cp "lib/mondrian.jar: /usr/local/mysql-5.0.45-linux-i686/mysql-connector-java-5.0.7/src/lib/log4j-1.2.9.jar: 
lib/eigenbase-xom.jar:lib/eigenbase-resgen.jar:lib/eigenbase-properties.jar:
/usr/local/mysql/mysql-connector-java-5.0.7/mysql-connector-java-5.0.7-bin.jar" \
mondrian.test.loader.MondrianFoodMartLoader \
-verbose -tables -data -indexes \
-jdbcDrivers=com.mysql.jdbc.Driver \
-inputFile=demo/FoodMartCreateData.sql \
-outputJdbcURL="jdbc:mysql://localhost/foodmart?user=foodmart&password=foodmart"



19. cp /usr/local/jakarta-tomcat-5.0.28/webapps/mondrian/WEB-INF/lib/xalan.jar /usr/local/jakarta-tomcat-5.0.28/common/endorsed/
20. Modify the query files, web.xml, datasource.xml and mondrian.properties file to replace localhost with your hostname



sed -i`date +%y%m%d` -e "s/localhost/yourhostname/" fourhier.jsp
sed -i`date +%y%m%d` -e "s/localhost/yourhostname/" mondrian.jsp
sed -i`date +%y%m%d` -e "s/localhost/yourhostname/" colors.jsp
sed -i`date +%y%m%d` -e "s/localhost/yourhostname/" arrows.jsp
sed -i`date +%y%m%d` -e "s/localhost/yourhostname/" web.xml
sed -i`date +%y%m%d` -e "s/localhost/yourhostname/" datasource.xml
sed -i`date +%y%m%d` -e "s/localhost/yourhostname/" mondrian.properties



21. Modify the connection string in each file as per install guide, including the #38 stuff it is not a browser character issue.
22. Test connectivity for mysql user foodmart@yourhostname. Connection errors can cause this error



Mondrian Error:Internal error: Error while creating SQL dialect



23. export CATALINA_OPTS='-Djava.awt.headless=true' or add to your shell profile, this stops tomcat dying when X-windows is not found!! Thanks to this link for solving that.