NoSQL: CouchDB with CloudBees and Cloudant

(WARNING: this blog is a bit of a tutorial, so feel free to let your eyes glaze over if it gets boring - it will show how easy it is to use partner services to build applications, in this case a NoSQL database.)

In this tutorial I will build a CouchDB based url shortener service (http://cloudbe.es) using CloudBees and the services/add-on platform with Cloudant.

CouchDB is a popular NoSQL database - specifically you could called it a “document database” - i.e. it is all about storing documents which consist of sets of fields in a JSON document against a key. There is no schema - a document can consist of any JSON structure - there is no requirement that the same fields be present in each document unlike rows in a RDBMS. Of course CouchDB has a lot of the other benefits of NoSQL databases - high scalability and ease of use.

Cloudant provides an excellent hosted CouchDB service. This takes care of *all* the management of CouchDB you would normally need to do - compaction, backups, and more.

Building a URL shortener

A url shortener is just that - takes a long messy URL and generates a token/shorter URL you can pass around.

Step 1: Get a cloudbees.com account, signup to a free RUN@cloud service.

Step 2: Go to grandcentral.cloudbees.com/subscriptions - and choose Cloudant.

 
 

Step 3: Create a new web app skeleton.

For this I will use the Play! Framework - a simple framework for building RESTful web apps. I can use the command

play new urlshort —with scala

This will create an empty project using scala. To deploy to cloudbees - a handy way is to add cloudbees 0.2.1 to the dependencies file (don’t worry - source repo will be available to look at).

Step 4: Create an empty database.

From grandcentral.cloudbees.com/services - you can navigate to your Cloudant/CouchDB console. Any time you need to get back to the CouchDB console you can navigate from grandcentral.

 

This shows the “Futon” web interface to browse your data, quite a handy tool. Here you can create a new database - I called mine “urly”. Once you have a database, at the top left corner you can see an icon which will take you to the Cloudant management screen - click through to the newly created database, and you will be able to set permissions, and generate an api key:

 

The important bit is to note the api key details (a user name and password that you will use in your app). The user just generated should also get read/write access to your database (set it with the checkboxes).

Step 5: Set up credentials in your application config.

In this case, as I am using the Play! framework - the configuration is in application.conf. The keys I set up were:

#CloudBees details
bees.api.key= key for deploying
bees.api.secret= secret for deploying
bees.api.domain=michaelvideo
bees.api.name=urly

#Cloudant details
couch.user=api key user
couch.password=api key password

I put the CloudBees details in there for easy deployment but technically you don’t have to (you can pass them to command line if you like).

Step 6: Talking to CouchDB

One of the really nice things about CouchDB is that you don’t need any drivers or dependencies to talk to it - https is a “native” api for it. So all that is needed is a simple http/REST/WS client.

Urls are how you address content - you can store a document in a nominated key for example by PUT’ing a JSON body to https://michaelvideo.cloudbees.cloudant.com/urly/ (urly is the name of the database). Updates to data require you to specify the current version tag of the data (field in the JSON body) as a form of optimistic locking.

Getting the data back is a simple GET ! (obviously there is a lot more you can read about if you need). In this case - the keys are the randomly generated “slugs” that form the short URL.

Step 7: Wire up the application.

Descending from the mountain top, a simple codebase which generates random url “slugs” and then does appropriate http redirects was carved in stone (also here here https://github.com/michaelneale/URL-shortener):

 

Authentication with CouchDB is using http basic over SSL (using the api keys mentioned earlier).

Step 8: Deploy.

 
(This will use the Play! CloudBees plugin - thanks Ivan !)

I then set up a domain and a record pointing to the host for the app, running at http://cloudbe.es/

Note that there was no need to migrate any data, or setup a schema (just create the database by name - which I could have also done programmatically).

The data is safely stored in CouchDB (below is browsing it with Futon):

Map Reduce and views

A further enhancement is to create views. CouchDB uses a Map Reduce model to provide views and aggregate calculations.

A view is similar to what relational databases offer and only requires a map function to be provided. Reducing is a more complicated topic and I won’t show it, but it could be used, for example, to calculate aggregates (sum or average over some range of documents). The advantage of a Map Reduce model is that the calculation can be distributed and run in parallel with the data - good for very large sets of data.

 

The Map function above simply takes a document and emits another document which is made up of the slug and the agent string (each browser provides an agent string). The key in this case is the url we stored - this means we can (if we want) lookup slugs based on URL (the reverse of what we were doing before). Note that as the fields in each document are optional - even if there is no agent field the above view will still work (just will have no agent field in the view output).

The functions are written in javascript - and are themselves stored as documents (with a special naming convention). The results of these views are accessed the same as the normal documents you write data to.

Hopefully from this you get a taste of the convenience and power of CouchDB and Cloudant.

You can sign up for both CloudBees (if you don’t have an account) and the Cloudant service here: http://cloudbees.com/platform-service-cloudant.cb

 

— Michael Neale, Elite Developer and Architect

 

 

Blog Categories: 

Add new comment