Datamining in the cloud

Amazon EC2 and other on-demand or cloud computing vendors are providing a way to scale up and scale down your computing and storage requirements.

Over the last 18 months or so I have reviewed and demo'ed a bunch of Data mining and Business intelligence (BI) software and suites. This was both for my benefit, so I can check out what is out there is this software space and also to see how effective each is at providing a way to scale up and scale down your BI computing needs.

Many people want to use these compute clouds like dedicated hosting providers which is the wrong use, at least until the prices drop to be comparable to dedicated hosting.
What they are for is for peak demand resourcing, short term and mid term computing requirements. This is especially true for tasks where the work you are trying to do is constrained by CPU and which scale well.

Let's look at the short term computing example :

You have a task which will take one machine with 4 cores (2x Dual Core) 16 hours to complete or 4 hours per core. However the task scales, so you if you throw 16 cores at it, it will be completed in 1 hour.

I see this as the core of providing any kind of on-demand service. The main selling point is you are saving people time. They can take that time and reinvest it (if you like) and do more analysis or more in-depth analysis. They can broaden the techniques they use to analyze their data or use the same techniques in more detail.

One example of this type of broadening on analysis is using Weka and running many different learning algorithms on the same training and testing set of data to find the best one. This is perfect use of on-demand computing.

So how to you provide this type of service under a cloud computing model?

You have introduce a queue.
Your base load is done by a dedicated hosting provider and your peak load by starting and stopping (ramping up and down) the compute resources as required.

Why stop with one queue, many service providers have many channels to accessing their services. So work which must be done immediately where time is a premium, would pay some premium for the immediacy required. Work which isn't required until the next business day can be completed when spare resources are freed up.

Another way to approach this is to let a market form for the compute resources. The best mechanism for controlling the use of resources is a price signal.

Have Fun

Note about the site

If you have visited the site in the past, you would have noticed I have changed the template and added a bunch of RSS and email subscription information on the sidebar.

I hope this makes it easier to consume the content as it appears if you use a RSS feed reader.

As to the jobs link, I am always interested in new and interesting widgets, I don't expect anything much to happen with it.

The main reason for changing templates was so the code and screen dumps I put into posts are not truncated off for being too wide.

Have Fun

Paul

Streams real time data manipulation links

As I mentioned I would put up a more detailed article on the background to streams.

It is a reasonably hot topic as the broker boys and quants in driving research due to the potential to make megabucks, tying together stream technology with algorithmic trading.
So like old mate Wayne Gretzky "Skate to where the puck is going to be", I am thinking about where the puck (or buck if you like) is going to be, current research is where the puck is at now.

Note, this is not the same as Oracle streams which is a different technology. Oracle streams mines Oracle redo transaction logs to enable real-time application to other databases either as a form of replication (for redundancy) or reporting or both.

Here are the URLs I used to start to get a feel for streams.

http://en.wikipedia.org/wiki/StreamSQL

Stanford University Links:

http://infolab.stanford.edu/stream/

Brown University Links:

http://www.cs.brown.edu/research/aurora/
http://www.cs.brown.edu/research/aurora/publications.html
http://www.cs.brown.edu/research/aurora/aurora_1_2.tar.gz
http://www.cs.brown.edu/research/borealis/public/

Offshoots from this research:

http://www.Streambase.com
http://www.streamsql.org

If you have others free feel to post them in the comments and I will add to the article.

What is VM datamine

I have been playing with Amazon EC2 for a couple of months now.
I have a blog dedicated to creating various databases (commercial and otherwise). At the moment the aim is to get Oracle RAC running on EC2.

So I have been creating various Amazon Machine Images (AMI), basically EC2 runs off the back of Xen virtualization software.

Rather than write a generalised blog, I am trying to be more specific with content. So this blog will cover various EC2 builds testing out the many and varied datamining/Business Intelligence (BI) software out there.

I was currently reviewing the basis of bizgres (an extension of PostgreSQL) with its commercial arm of Greenplum.

In my websurfing over the last 2 or so days since the seed of this idea formed I have also looked at the streamsql, which is a real-time streaming extension to standard SQL. It also has a commercial arm in Streambase. More about Streams shortly.

So what is the idea?

Why not use Amazon as a testing ground to providing a web service based on the above technology and more.

For example:
Instead of requiring a dedicated server for checking clickstream, why not store or upload your web access logs (even real time clicks) to Amazon S3 or an equivalent web storage service and then combining that with specific virtual instances (processing nodes) which can fire up and shutdown as required.
The process nodes morph from a streams collection tool, to a dataminer based on Bizgres, then again morphing to producing a output viewable in something like Yale.

Amazon also have a message queue service so there is nothing to stop writing a director node which spawns and reaps the above service nodes as required, storing the dataset being processed like a car on a production line.

Interested in helping out?

Either post a comment or email me at paulmoen at gmail dot com

Have Fun

Paul