The Release of the Riak Wiki and the Fourth Basho Podcast

We are moving at warp speed here at Basho and today we are releasing what we feel is a very important enhancement to Riak: a wiki.

You can find it here:

http://wiki.basho.com

Documentation and resources are a main priority right now for Basho, and a well maintained and up-to-date wiki is something we see as critical. Our goal is to make Riak simple and intuitive to download, build, program against, and build apps on. So, you should expect a lot more from us in this regard. Also, we still have much to add to the Riak Wiki, so if you think we are missing a resource or some documentation that makes Riak easier to use and learn about, please tell us.

Secondly, we had the chance to record the fourth installment of the Basho Riak podcast (below), and it was a good one. We hooked up with Tim Anglade, CTO of GemKitty and a growing authority on the NoSQL space. On the heels of his presentation at NoSQL Live from Boston, we picked his brain a bit about where he thinks the industry is going and what needs to change for the current iteration of NoSQL to go from being a fad and curiosity to a full fledged industry.

According to Tim, "We have an image problem right now with NoSQL as a brand," and "NoSQL is over-hyped and the projects behind it are under-hyped."

We also took a few minutes to talk about the Riak 0.9.1 release. Highlights include binary builds, as well as several new client libraries that expose all of Riak's advanced features.

In short, if you are at all interested in the future of the NoSQL space, you're not going to want to miss this.

Lastly, if you haven't already done so, go download the latest version of Riak.

Enjoy!

Mark

Link Walking By Example

Riak has a notion of “links” as part of the metadata of its objects. We talk about traversing, or “walking”, links, but what do the queries for doing so actually look like?

Let's put four objects in riak:

hb/first will link to hb/second and hb/third hb/second will link to hb/fourth hb/third will also link to hb/fourth hb/fouth doesn't link anywhere

$ curl -X PUT -H "content-type:text/plain" \
  -H "Link: </raw/hb/second>; riaktag=\"foo\", </raw/hb/third>; riaktag=\"bar\"" \
  http://localhost:8098/raw/hb/first --data "hello"

$ curl -X PUT -H "content-type: text/plain" \
  -H "Link:</raw/hb/fourth>; riaktag=\"foo\"" \
  http://localhost:8098/raw/hb/second --data "the second"

$ curl -X PUT -H "content-type: text/plain" \
  -H "Link:</raw/hb/fourth>; riaktag=\"foo\"" \
  http://localhost:8098/raw/hb/third --data "the third"

$ curl -X PUT -H "content-type: text/plain" \
  http://localhost:8098/raw/hb/fourth --data "the fourth"

Now, say we wanted to start at hb/first, and follow all of its outbound links. The easiest way to do this is with the link-walker URL syntax:

$ curl http://localhost:8098/raw/hb/first/_,_,_

The response will be a multipart/mixed body with two parts: the hb/second object in one, and the hb/third object in the other:

--N2gzGP3AY8wpwdQY0jio62L9nJm
Content-Type: multipart/mixed; boundary=3ai6VRl4aLli3dKw8tG9unUeznT

--3ai6VRl4aLli3dKw8tG9unUeznT
X-Riak-Vclock: a85hYGBgzGDKBVIsTKLLozOYEhnzWBn+H/h5hC8LAA==
Location: /raw/hb/third
Content-Type: text/plain
Link: </raw/hb>; rel="up", </raw/hb/fourth>; riaktag="foo"
Etag: 5Fs0VskZWx7Y25tf1oQsvS
Last-Modified: Wed, 24 Feb 2010 15:25:51 GMT

the third
--3ai6VRl4aLli3dKw8tG9unUeznT
X-Riak-Vclock: a85hYGBgzGDKBVIsLEHbN2YwJTLmsTLMPvDzCF8WAA==
Location: /raw/hb/second
Content-Type: text/plain
Link: </raw/hb>; rel="up", </raw/hb/fourth>; riaktag="foo"
Etag: 2ZKEJ2gaT57NT7xhLDPCQz
Last-Modified: Wed, 24 Feb 2010 15:24:11 GMT

the second
--3ai6VRl4aLli3dKw8tG9unUeznT--

--N2gzGP3AY8wpwdQY0jio62L9nJm--

It's also possible to express the same query in map-reduce, directly:

$ curl -X POST -H "content-type:application/json" \
  http://localhost:8098/mapred --data @-
{"inputs":[["hb","first"]],"query":[{"link":{}},{"map":{"language":"javascript","source":"function(v)
{ return [v]; }"}}]}
^D

That's the exact same query. The content type of the response is different. It's now a JSON array with two elements: a JSON encoding of the hb/second object, and a JSON encoding of the hb/third object. (Pretty-printed here, for clarity.)

[
    {
        "bucket": "hb",
        "key": "second",
        "vclock": "a85hYGBgzGDKBVIsLEHbN2YwJTLmsTLMPvDzCF8WAA==",
        "values": [
            {
                "metadata": {
                    "Links": [
                        ["hb","fourth","foo"]
                    ],
                    "X-Riak-VTag": "2ZKEJ2gaT57NT7xhLDPCQz",
                    "content-type": "text/plain",
                    "X-Riak-Last-Modified": "Wed, 24 Feb 2010 15:24:11 GMT",
                    "X-Riak-Meta": []
                },
                "data": "the second"
            }
        ]
    },
    {
        "bucket": "hb",
        "key": "third",
        "vclock": "a85hYGBgzGDKBVIsTKLLozOYEhnzWBn+H/h5hC8LAA==",
        "values": [
            {
                "metadata": {
                    "Links": [
                        ["hb","fourth","foo"]
                    ],
                    "X-Riak-VTag": "5Fs0VskZWx7Y25tf1oQsvS",
                    "content-type": "text/plain",
                    "X-Riak-Last-Modified": "Wed, 24 Feb 2010 15:25:51 GMT",
                    "X-Riak-Meta": []
                },
                "data": "the third"
            }
        ]
    }
]

Another interesting query is “follow only links that are tagged foo.” For that, just add a tag field to the link phase spec:

$ curl -X POST -H "content-type:application/json" \
  http://localhost:8098/mapred --data @-
{"inputs":[["hb","first"]],"query":[{"link":{"tag":"foo"}},{"map":{"language":"javascript","source":"function(v)
{ return [v]; }"}}]}
^D

Here you should get a JSON array with one element: a JSON encoding of the hb/second object. The link to the hb/third object was tagged bar, so that link was not followed. The equivalent URL syntax is:

$ curl http://localhost:8098/raw/hb/first/_,foo,_

It's also possible to filter links by bucket by adding a bucket field to the link phase spec, or by replacing the first underscore with a bucket name in the URL format. But, all of our example links point to the same bucket, so hb is the only interesting setting here.

Link phases may also be chained together (or put after other phases if those phases produce bucket/key lists). For example, we could follow the links all the way from hb/first to hb/fourth with:

$ curl -X POST -H "content-type:application/json" \
  http://localhost:8098/mapred --data @-
{"inputs":[["hb","first"]],"query":[{"link":{}},{"link":{}},{"map":{"language":"javascript","source":"function(v)
{ return [v]; }"}}]}
^D

(Notice the added link phase.) If you run that, you'll find that you get two copies of the hb/fourth object in the response. This is because we didn't bother uniquifying the results of the link extraction, and both hb/second and hb/third link to hb/fourth. A reduce phase is fairly easy to add:

$ curl -X POST -H "content-type:application/json" \
  http://localhost:8098/mapred --data @-
{"inputs":[["hb","first"]],"query":[{"link":{}},{"link":{}},{"reduce":{"language":"erlang","module":"riak_mapreduce","function":"reduce_set_union"}},{"map":{"language":"javascript","source":"function(v)
{ return [v]; }"}}]}
^D

The resource handling the URL link-walking format does just this:

$ curl http://localhost:8098/raw/hb/first/_,_,_/_,_,_

That should get you just one copy of the hb/fourth object.

So why choose either map/reduce or URL-syntax? The advantage of URL syntax is that if you're starting from just one object, and just want to get the objects at the ends of the links, and you can handle multipart/mixed encoding, then URL syntax is much simpler and more compact. Map/reduce with link phases should be your choice if you want to start from multiple objects at once, or you want to get some processed or aggregated form of the objects, or you want the result to be JSON-encoded.

Riak version 0.8 note: In Riak 0.8, the format of the result of 'link' map/reduce phases was not able to be transformed into JSON. This meant both that it was not possible to put a Javascript reduce phase right after a link phase, and also that it was not possible to end an HTTP map/reduce query with a link phase. Those issues have been resolved in the tip of the source repository, and will be part of the 0.9 release.

Riak version 0.9 note: In Riak 0.9, the URL for Riak's HTTP interface will change from /raw to /riak. You may need to adjust the above examples accordingly.

-Bryan

Using Innostore with Riak

Innostore is an Erlang application that provides an API for storing and retrieving key/value data using the InnoDB storage system. This storage system is the same one used by MySQL for reliable, transactional data storage. It’s a proven, fast system and perfect for use with Riak if you have a large amount of data to store. Let’s take a look at how you can use Innostore as a backend for Riak.

(Note: I assume that you have successfully built an instance of Riak for your platform. If you built Riak from source in ~/riak, then set $RIAK to ~/riak/rel/riak.”)

We first get started by grabbing a stable release of Innostore. You’ll need to download the source for a release from: http://bitbucket.org/basho/innostore/downloads/

Looking in the “Tags & snapshots” section, you should download the source for the highest available RELEASE_* tag. In my case, RELEASE_4 is the most recent release, so I’ll grab the bz2 file associated with it:

http://bitbucket.org/basho/innostore/get/RELEASE_4.tar.bz2


Once I have the source code, it’s time to unpack it and build:

$ tar -xjf innostore-RELEASE_4.tar.bz2
$ cd innostore
$ make


Depending on the speed of the machine you are building on, this may take a few minutes to complete. At the end, you should see a series of unit tests run, with the output ending:

=======================================================
All 7 tests passed.
100222 7:43:58 InnoDB: Shutdown completed; log sequence number 90283
Cover analysis: /Users/dizzyd/src/public/innostore/.eunit/index.html


Now that we have successfully built Innostore, it’s time to install it into the Riak distribution:

$ ./rebar install target=$RIAK/lib

If you look in the $RIAK/lib directory now, you should see the innostore-4 directory alongside a bunch of .ez files and other directories which compose the Riak release.

Now, we need to tell Riak to use the Innostore driver as a backend. Make sure Riak is not running. Edit $RIAK/etc/app.config, setting the value for “storage_backend” as follows:

{storage_backend, innostore_riak},

In addition, append the configuration for the Innostore application after the SASL section:

{sasl, [ ....
]}, %% < -- make sure you add a comma here!!

{innostore, [
{data_home_dir, "data/innodb"}, %% Where data files go
{log_group_home_dir, "data/innodb"}, %% Where log files go
{buffer_pool_size, 2147483648} %% 2G in-memory buffer in bytes
]}

You may need to adjust the directories for your data_home_dir and log_group_home_dirs to match where you want the inno data and log files to be stored. If possible, make sure that the data and log dirs are on separate disks -- this can yield much better performance.

Once you've completed the changes to $RIAK/etc/app.config, you're ready to start Riak:

$ $RIAK/bin/riak console

As it starts up, you should see messages from Inno that end with something like:

100220 16:36:58 InnoDB: highest supported file format is Barracuda.
100220 16:36:58 Embedded InnoDB 1.0.3.5325 started; log sequence number 45764

That’s it! You’re ready to start using Riak for storing truly massive amounts of data.

Enjoy,

Dave Smith

Calling all Rubyists - Ripple has Arrived!

The Basho Dev. Team has been very excited about working with the Ruby community for some time. The only problem was we were heads down on so many other projects that it was hard to make any progress. But, even with all that work on our plate, we were committed to showing some love to Rubyists and their frameworks.

Enter Sean Cribbs. As Sean details in his latest blog post, "You Got your Riak in my Ruby," Basho and the stellar team at Sonian made it possible for him to hack on Ripple, a one-of-a-kind client library and object mapper for Riak. The full feature set for Ripple can be found on Sean's blog, but highlights include a DataMapper-like API, an easy builder-style interface to Map/Reduce, and near-term integration with Rails 3.

And, in case you need any convincing that you should consider Riak as the primary datastore for your next Rails app, check out Sean's earlier post, "Why Riak should power your next Rails app."

So, if you've read enough and want to get your hands on Ripple, go check it out on GitHub.

If you don't have Riak downloaded and built yet, get on it.

Lastly, you are going to be seeing a lot more Riak in your Ruby. So stay tuned because we have some big plans.

Best,

The Release Riak 0.8 and JavaScript Map/Reduce

We are happy to announce the release of Riak 0.8 available for download immediately. Riak 0.8 features a number of enhancements to the core map/reduce machinery that will make Riak more accessible to a wider audience. The biggest enhancement is the ability to write map/reduce queries in JavaScript. We're using our erlang_js project to integrate Mozilla's Spidermonkey engine directly into Riak to keep overhead to a minimum.

We've also built a spiffy REST API for submitting map/reduce queries. Queries are described in JSON and POST-ed to the Riak server. Results are sent back as JSON for your processing pleasure. And, the REST interface supports streaming results for large result sets, too.

To kick it all off, we've put together a short screencast demonstrating how to use Riak's flashy new features. You can watch it below, or view it on Vimeo. There's also a slew of bug fixes and optimizations included in Riak 0.8. See the release notes for all the juicy details.

Download and enjoy!

Kevin

View on Vimeo

Basho Podcast Three - An Introduction To Innostore

You may remember that Basho recently open-sourced Innostore, our standalone Erlang application that provides a simple interface to embedded InnoDB...

In this podcast, Dave "Dizzy" Smith and Justin Sheehy discuss the release of Innostore, why we built it, how we use it in Riak, and why it might be useful for other Erlang projects. The discussion focuses on the stability and predictability of InnoDB, especially under load and as compared with other storage backends like DETS.

And of course, go download Innostore when you are done with the podcast.

Enjoy!

Mark

If you are having problems getting the podcast to play, click here to play in new window or right click to download the podcast.

Why Vector Clocks are Easy

date = Wednesday
vclock = Alice:1

date = Tuesday
vclock = Alice:1, Ben:1

date = Tuesday
vclock = Alice:1, Ben:1, Dave:1

date = Thursday
vclock = Alice:1, Cathy:1

date = Tuesday
vclock = Alice:1, Ben:1, Dave:1

date = Thursday
vclock = Alice:1, Cathy:1

date = Thursday
vclock = Alice:1, Ben:1, Cathy:1, Dave:2

date = Tuesday
vclock = Alice:1, Ben:1, Dave:1

date = Thursday
vclock = Alice:1, Ben:1, Cathy:1, Dave:2

curl -X PUT -H "X-Riak-ClientId: Alice" -H "content-type: text/plain" \
http://localhost:8098/raw/plans/dinner --data "Wednesday"

curl -i http://localhost:8098/raw/plans/dinner
HTTP/1.1 200 OK
X-Riak-Vclock: a85hYGBgzGDKBVIsrLnh3BlMiYx5rAzLJpw7wpcFAA==
Content-Type: text/plain
Content-Length: 9

Wednesday

curl -X PUT -H "X-Riak-ClientId: Ben" -H "content-type: text/plain" \
-H "X-Riak-Vclock: a85hYGBgzGDKBVIsrLnh3BlMiYx5rAzLJpw7wpcFAA==" \
http://localhost:8098/raw/plans/dinner --data "Tuesday"

curl -i http://localhost:8098/raw/plans/dinner
...
X-Riak-Vclock: a85hYGBgymDKBVIsrLnh3BlMiYx5rAymfeeO8EGFWRLl30GF/00ACmcBAA==
...
curl -X PUT -H "X-Riak-ClientId: Dave" -H "content-type: text/plain" \
-H "X-Riak-Vclock: a85hYGBgymDKBVIsrLnh3BlMiYx5rAymfeeO8EGFWRLl30GF/00ACmcBAA==" \
http://localhost:8098/raw/plans/dinner --data "Tuesday"

curl -X PUT -H "X-Riak-ClientId: Cathy" -H "content-type: text/plain" \
-H "X-Riak-Vclock: a85hYGBgzGDKBVIsrLnh3BlMiYx5rAzLJpw7wpcFAA==" \
http://localhost:8098/raw/plans/dinner --data "Thursday"

curl -i -H "Accept: multipart/mixed" http://localhost:8098/raw/plans/dinner
HTTP/1.1 300 Multiple Choices
X-Riak-Vclock: a85hYGBgzWDKBVIsrLnh3BlMiYx5rAymfeeO8EGFWRLl30GF1fsRwsypF59BhT0mIoTZ/1SYQIUrEcJszUksu9R6kCWyAA==
Content-Type: multipart/mixed; boundary=ZZ3eyjUllBi7GXRRMJsUublFxjn
Content-Length: 368


--ZZ3eyjUllBi7GXRRMJsUublFxjn
Content-Type: text/plain

Tuesday
--ZZ3eyjUllBi7GXRRMJsUublFxjn
Content-Type: text/plain

Thursday
--ZZ3eyjUllBi7GXRRMJsUublFxjn--

curl -X PUT -H "X-Riak-ClientId: Dave" -H "content-type: text/plain" \
-H "X-Riak-Vclock: a85hYGBgzWDKBVIsrLnh3BlMiYx5rAymfeeO8EGFWRLl30GF1fsRwsypF59BhT0mIoTZ/1SYQIUrEcJszUksu9R6kCWyAA==" \
http://localhost:8098/raw/plans/dinner --data "Thursday"

curl -i http://localhost:8098/raw/plans/dinner
HTTP/1.1 200 OK
X-Riak-Vclock: a85hYGBgzWDKBVIsrLnh3BlMiYx5rAymfeeO8EGFWRLl30GF1fvhwmzNSSy71HqgEpUTEerZ/1SYYBFmTr34DCjMBBTOnQwUzgIA
Content-Type: text/plain
Content-Length: 7

Thursday

Innostore -- connecting Erlang to InnoDB

Riak has pluggable storage engines, and so we're always on the lookout for better ways that users can store their data locally. Recent experiences with some Basho customers managing some large datasets led us to believe that InnoDB might work out very well for them.

To answer that question and fill that need, Innostore was written. It is a standalone Erlang application that provides a simple interface to Embedded InnoDB. So far its performance has been quite good, though InnoDB (with or without the Innostore API) is highly dependent on tuning the local configuration to match the local hardware. Luckily, Dizzy -- the author of Innostore -- has some heavy-duty experience doing that kind of tuning and as a result we've been able to help people meet their performance goals using Innostore.

-Justin

Latest blog posts

• The Release of the Riak Wiki and the Fourth Basho Podcast
• Link Walking By Example
• Using Innostore with Riak
• Calling all Rubyists - Ripple has Arrived!
• The Release Riak 0.8 and JavaScript Map/Reduce

Categories

• Beer (1)
• Developers (1)
• Operations (1)
• Nitrogen (1)
• InnoDB (2)
• Scaling (2)
• erlang_js (1)
• (1)
• Ops (1)
• Analytics (1)
• Scalability (2)
• Fault Tolerance (2)
• Resources (1)
• Distributed Systems (3)
• Search (3)
• Startups (2)
• Rebar (3)
• Map/Reduce (4)
• Screencast (4)
• JavaScript (2)
• Ruby (1)
• Links (1)
• Database (9)
• Innostore (2)
• Erlang (4)
• Riak (21)
• NoSQL (10)
• Podcast (3)

Archives

• March 2010  (1)
• February 2010  (5)
• January 2010  (4)
• December 2009  (8)
• November 2009  (3)
• October 2009  (1)

Other Recommended Reading

• The RESTful Way
• The Beerriot Blog
• Rusty Klophaus
• Gradual Epiphany

Contact

Basho Technologies

196 Broadway
Cambridge, MA 02139

617.714.1700

info@basho.com