Using Innostore with Riak

February 22, 2010 at 11:00 AM | categories: Riak, Erlang, Innostore, Database

Innostore is an Erlang application that provides an API for storing and retrieving key/value data using the InnoDB storage system. This storage system is the same one used by MySQL for reliable, transactional data storage. It’s a proven, fast system and perfect for use with Riak if you have a large amount of data to store. Let’s take a look at how you can use Innostore as a backend for Riak.

(Note: I assume that you have successfully built an instance of Riak for your platform. If you built Riak from source in ~/riak, then set $RIAK to ~/riak/rel/riak.”)

We first get started by grabbing a stable release of Innostore. You’ll need to download the source for a release from: http://bitbucket.org/basho/innostore/downloads/

Looking in the “Tags & snapshots” section, you should download the source for the highest available RELEASE_* tag. In my case, RELEASE_4 is the most recent release, so I’ll grab the bz2 file associated with it:

http://bitbucket.org/basho/innostore/get/RELEASE_4.tar.bz2

Once I have the source code, it’s time to unpack it and build:

$ tar -xjf innostore-RELEASE_4.tar.bz2
$ cd innostore
$ make

Depending on the speed of the machine you are building on, this may take a few minutes to complete. At the end, you should see a series of unit tests run, with the output ending:

=======================================================
All 7 tests passed.
100222 7:43:58 InnoDB: Shutdown completed; log sequence number 90283
Cover analysis: /Users/dizzyd/src/public/innostore/.eunit/index.html

Now that we have successfully built Innostore, it’s time to install it into the Riak distribution:

$ ./rebar install target=$RIAK/lib

If you look in the $RIAK/lib directory now, you should see the innostore-4 directory alongside a bunch of .ez files and other directories which compose the Riak release.

Now, we need to tell Riak to use the Innostore driver as a backend. Make sure Riak is not running. Edit $RIAK/etc/app.config, setting the value for “storage_backend” as follows:

{storage_backend, innostore_riak},

In addition, append the configuration for the Innostore application after the SASL section:

{sasl, [ ....
]}, %% < -- make sure you add a comma here!!

{innostore, [
{data_home_dir, "data/innodb"}, %% Where data files go
{log_group_home_dir, "data/innodb"}, %% Where log files go
{buffer_pool_size, 2147483648} %% 2G in-memory buffer in bytes
]}

You may need to adjust the directories for your data_home_dir and log_group_home_dirs to match where you want the inno data and log files to be stored. If possible, make sure that the data and log dirs are on separate disks -- this can yield much better performance.

Once you've completed the changes to $RIAK/etc/app.config, you're ready to start Riak:

$ $RIAK/bin/riak console

As it starts up, you should see messages from Inno that end with something like:

100220 16:36:58 InnoDB: highest supported file format is Barracuda.
100220 16:36:58 Embedded InnoDB 1.0.3.5325 started; log sequence number 45764

That’s it! You’re ready to start using Riak for storing truly massive amounts of data.

Enjoy,

Dave Smith

Basho Podcast Three - An Introduction To Innostore

February 02, 2010 at 03:45 PM | categories: Riak, InnoDB, Erlang, Innostore

You may remember that Basho recently open-sourced Innostore, our standalone Erlang application that provides a simple interface to embedded InnoDB...

In this podcast, Dave "Dizzy" Smith and Justin Sheehy discuss the release of Innostore, why we built it, how we use it in Riak, and why it might be useful for other Erlang projects. The discussion focuses on the stability and predictability of InnoDB, especially under load and as compared with other storage backends like DETS.

And of course, go download Innostore when you are done with the podcast.

Enjoy!

Mark



If you are having problems getting the podcast to play, click here to play in new window or right click to download the podcast.

Dave Smith Gives a General Overview of Rebar

December 21, 2009 at 02:00 PM | categories: Riak, Screencast, Erlang, Rebar

As a follow up to my first screencast on using Rebar for embedded Riak nodes, this video gives a more general overview of Rebar using ibrowse, an existing Erlang application, to show functionality and intended uses.

Enjoy,

Dave

(View on Vimeo)

If You Happen to Find Yourself in Stockholm...

November 09, 2009 at 12:22 PM | categories: Riak, Erlang, Nitrogen, NoSQL

Then make sure to check out Basho's very own Rusty Klophaus, who will be opening up the Erlang User Conference, slated to kick-off November 12th.

Rusty will be giving a brief overview of both Nitrogen and Riak, and then plans to describe common patterns and practices of Nitrogen and Riak development against the background of a sample application that allows a presenter to share and control a slideshow over the internet. In short, his presentation is not to be missed. (Official abstract can be found here.)

So, if you find yourself in Sweden this Thursday, make sure to show your face at the Astoria on Nybrogatan in Stockholm, and show your support for Erlang, Riak, Nitrogen and, most-importantly, Rusty.

Riak On,

Mark Phillips