Todos 06 - Building with CouchDB


This page will show and explain how to build DataStore to work with CouchDB as backend database server.

 

Backend

 

What is CouchDB?

CouchDB is a document-oriented database built by Apache and is written using the Erlang language. The best (and only) way to interact with CouchDB is using HTTP, which makes it great fit for web applications. You can read more and learn about CouchDB in this online book.

 

Setup the database

There are couple of methods you use to set up a CouchDB database for the purposes of this tutorial: run local instance, host at couchone.com, and host at CloudAnt.com.  NOTE: the CoucbDB project also getting branded CouchOne as of 2010-11-27.  Please check around if links break.

 

1. Run local instance

Scroll a little at couchone.com until you see Download CouchOne and select the download for your favorite OS.

TODO: explain how to run locally

 

2. Register for couchone.com hosting

Enter 4 bits of information at couchone.com registration page and you instantly have your very own online CouchDB database.

Since CouchDB is design for web use it have administration console called Futon to let you administer your database.

 

3. Register to CloudAnt.com

Go to CloudAnt.com Oxygen registration page and in second you will have your own CouchDB database. CloudAnt have their own administration console as well as Futon.

 

Create the schema

You don't have to; that is one of the great things about CouchDB. It does not require you to define a schema for your data. Put your document in the database and it will be stored. Because CouchDB is not a relational database, it does not index your data nor query your data the way SQL Server or Oracle does.

 

Create the database

Go into Futon on which ever installation you chose during the setup process. On the bottom left corner of the screen you will either see Welcome to Admin Party! Eveone is admin" or you will see Signup or Login. For the former, you don't have to do anything; Admin Party means everybody has administrator-level access and no login is required. For the later, you will need to login with your admin account.

 

At the top left corner you will see "Create database" link, click on it.

Enter "todos" as database name and click Create. That's it.

 

Play with CouchDB

Open Futon and go into your todos database.

 

Create new document

On the top left corner you will see "New Document" link, press on it.

The document edit screen will show table contain single row. The row Field will say "_id" and the value (which is now editable) will have a long UUID which compose of hex-decimal and letters. Clear the text box and write doc1 in it.

On the top left corner press the "Save Document" link.

A second row will be added to the table. This row is _rev with value of 1-XXX (XXX will be different on each machine).

You have just created a new document in your CouchDB. This is the simplest document you can create. As you see each document must have at least two fields: _id and _rev. The _id field allow you to reference this document while _rev allow CouchDB to keep track on documents as they are changing. Each time you save a document its _rev field will be change by CouchDB. _rev is built from two parts - revision number and hash code on the document content. Read about it in more detail in the CouchDB book.

 

Change a document

Click the Add Field button, it will create a new row in the table and put the focus on the text box in Field column. write description in the text box and press Tab. The focus is now on the Value text box, write Hello CouchDB and press Enter. Futon will automatically surround your text with parentesess to make it string.

Click the Save Document link. Notice the _id field stay the same but the _rev field change to 2-YYY. Also you just add a new field to your document without the need to define column in table nor type of the field.

Click Add Field again and enter isDone in Field text box and true on the Value. Notice Futon does not surround true with parentesess because it is boolean value and not a string.

Click Save Document link and see the _rev change once more.

 

See the raw document

On the right upper corner click the Source tab. Futon will change the view to show you the raw view of the document. Now you see that a document is nothing more than JSON. You can also double click on the Source content to edit the document as JSON.

Because documents are JSON they are not limited to flat name/value pairs, they can contain hierarchical structures as complex as you need.

 

Delete a document

On the middle up of the window click the Delete Document... link. A dialog will appear ask you if you sure you want to delete the document. Click Delete and the document will be delete from the database.

 

CRUD with HTTP

We will now repeat the create, update and delete operation and you'll see how they are communicated to CouchDB.

Open Terminal to use curl to interact with CouchDB.

 

I will now show each command and how it change CouchDB. I am using localhost:5984 as my server, change it if you use any other address.

 

curl http://localhost:5984/todos -H "Content-Type:application/json" -d '{"description":"HTTP"}' -X POST 

POST will create new document in todos database with the content we supply, that is "description":"HTTP". CouchDB will assign _id and _rev to our new document.

As you can see CouchDB has reply to us:

{"ok":true,"id":"34ead35bfc32ab858b7b5446aa002c09","rev":"1-786e63f02916b93805d01132b917305b"}

The ok field signal everything went OK. The id field contain the id of our new document and the rev contain the revision of our new document.

 

curl http://localhost:5984/todos/34ead35bfc32ab858b7b5446aa002c09 -H "Content-Type:application/json" -d '{"isDone":false,"_rev":"1-786e63f02916b93805d01132b917305b"}' -X PUT

PUT will update exist document. First thing to notice is that we POST this request not to /todos but specifically to the address of our new document. Each document can be reached using its id field. Also we add new field isDone to our document and we also have to send the _rev field in the document with the rev return from our POST because CouchDB must know we have the latest version of this document.

Here again CouchDB reply to us:

{"ok":true,"id":"34ead35bfc32ab858b7b5446aa002c09","rev":"2-e84e08f018b870b7bdaffc0f6848e86f"} 

The ok field signal that everything is ok. the id contain the id of the document we changed and the rev contain the new revision assign the our document after the change we request has been done.

 

curl http://localhost:5984/todos/34ead35bfc32ab858b7b5446aa002c09?rev=2-e84e08f018b870b7bdaffc0f6848e86f -X DELETE

DELETE, as wired as it may sound, delete a document. Here like POST we send the request to our document but because DELETE does not have body we carry the revision information as query parameter. If you try to delete a document without including the rev query parameter you will get this reply {"error":"conflict","reason":"Document update conflict."} which means CouchDB could not verify you have the latest version and so has reject your changes due to possible conflict.

The reply we get might look strange:

{"ok":true,"id":"34ead35bfc32ab858b7b5446aa002c09","rev":"3-6347d5414facc350d74dbb26c187246d"}

We get the same reply as in POST and PUT. The ok still here and the id also here but notice rev has been incremented even though we have delete the document. You can read more about how CouchDB store the document internally but just know that when you delete a document it is not actually been delete right away. CouchDB store it (with the new revision) with field called _delete and value true. 

 

The reason it is important to play like this with CouchDB is to understand how SproutCore app will interact with it.

 

 

Design Documents

Design document are special type of document CouchDB can store. A design document is identify by have _design/<name> as its id.

I will not go into detail about design document but you can read more in this chapter of CouchDB book.

 

Views

CouchDB does not use SQL to query data. Instead CouchDB use JavaScript to define map-reduce function.

You can read the chapter about views in the CouchDB book, I will show you how to create one view here.

 

allTasks view

We need to create view which will return all of the documents in our database that represent tasks.

In Futon open todos database. On the right upper corner you will see Views drop-down list that currently say All Documents. This means Futon now show all the documents you have in your database. Open it and select Temporary view..., this will open screen for edit and test views.

As you can see view is define from two parts: map and reduce functions. For now we will only use the map function.

Enter the following text into Map Function text box:

function(doc) {
  if (!(doc.description==null || doc.isDone==null)) {
    emit(doc._id, doc);
  }
}

This function will run once for every document we have in our database. CouchDB will pass to it a single document. The function check if the document has description and isDone properties, if so it emit (select) the document. View function output is key/value pair, in our case the key is the _id field of the document and the value is the document itself.

Because we have no documents yet we cannot test the view so we will just save it for later. Click the Save As... button and enter app in the Design Document text box and enter allTasks in the View name text box. Click Save button.

 

Frontend

 

Because CouchDB has requirement that are not impose by other backends we need to implement specific DataSource. Specifically we need to add code to handle the revision changes of each document.

 

You can see a complete implementation of Todos application with CouchDB DataStore at http://github.com/ido-ran/CouchTodos.

 

DataSource

 

As any other DataSource implementation we start by extending SC.DataSource

 

Todos.TaskDataSource = SC.DataSource.extend(


In order to support different database name we keep the name of the database in private property

 

_dbpath: 'todos',


These two helper function format url for document and view access.

 

getServerPath: function(resourceName) {
     var path = '/' + this._dbpath + "//" + resourceName;
     return path;
},
     getServerView: function(viewName) {
     var path = '/' + this._dbpath + "/_design/app/_view/" + viewName;
     return path;
},

 

Fetching Records

 

 

Fetch handle retrieve of multiple records at one round trip.
Currently we are only supporting fetch of all tasks.
We start by sending a GET HTTP request for view called allTasks we created earlier. We ask SC to call didFetchTasks when response or failure occur.

  fetch: function(store, query) {
     if (query === Todos.TASKS_QUERY) {
          SC.Request.getUrl(this.getServerView('allTasks')).json()
                         .header('Accept', 'application/json')
                         .notify(this, 'didFetchTasks', store, query)
                         .send();
          return YES;
     }
    return NO ; // return YES if you handled the query
  },

Once we get response we first check if the response is success. If so we extract the response which look something like this:
{
    "total_rows": 6,
 
  "offset": 0<span style="color: #000000;" _mce_style="color: #000000;">,</span>
  "rows":[ ... ]
All CouchDB response for view request look this way. We only need the rows we extract them from the JSON structure.
Note that all rows have both _id and _rev fields in them which is important for later interaction with CouchDB.
We notify the store we got the response and pass the received records (that is documents).
didFetchTasks: function(response, store, query) {
  if(SC.ok(response)) {
    var body = response.get('encodedBody');
    var couchResponse = SC.json.decode(body);
    var records = couchResponse.rows.getEach('value');
    store.loadRecords(Todos.Task, records);
    store.dataSourceDidFetchQuery(query);
 } else {
    store.dataSourceDidErrorQuery(query, response);
 }
 },

Single Record

We have not implement the retrieveRecord function because Todos app never use it.

 

  retrieveRecord: function(store, storeKey) {
      return NO ; // return YES if you handled the storeKey
  },
  /**

     Process response from CouchDB of create, update, delete operations. @returns id,rev for success, null for failure. */

     processResponse: function(response) {
          if (SC.ok(response)) {
               var body = response.get('encodedBody');
               var couchResponse = SC.json.decode(body);
               var ok = couchResponse.ok;
               if (ok != YES) return {"error":true, "response":couchResponse};
                    var id = couchResponse.id;
                    var rev = couchResponse.rev;
                    return {"ok":true, "id": id, "rev": rev};
               } else {
                    return {"error":true, "response":response};
               }
       },
  /**

Get the latest revision of the document. For docs which were fetch from the server we use _rev field, and for docs that were modified we use the local _docsRev dictionary. */

  getDocRev: function(doc) {
return doc._rev;
  },
  

Creating Records

 

Create new task is pretty easy to do, all we need is to send POST request to CouchDB with the information we want to store in the document represent our Task.

To do so we send POST request to the root url of our database and pass the data stored in the record in our local DataStore.

CouchDB will assign unique ID to our document as well as revision, but we will only get those values once the response will return to us so we ask SproutCore to call didCreateTask for that.

 

createRecord: function(store, storeKey) {
     if (SC.kindOf(store.recordTypeFor(storeKey), Todos.Task)) {
          SC.Request.postUrl(this.getServerPath('/')).json()
                            .header('Accept', 'application/json')
                            .notify(this, this.didCreateTask, store, storeKey)
                            .send(store.readDataHash(storeKey));
          return YES;
    } 
    return NO ; // return YES if you handled the storeKey
  },
Once a response is returned to us we pass it to processResponse for processing. If all went well we will get back from processReponse an object with three fields: ok, id and rev.
We need to store those bits of information along with our local document for further interaction with CouchDB because CouchDB expect the revision be part of the document to ensure its optimistic concurrency. 
didCreateTask: function(response, store, storeKey) {
     var couchRes = this.processResponse(response);
     if (couchRes.ok) {
          // Add _id and _rev to the local document for further server interaction.
          var localDoc = store.readDataHash(storeKey);
          localDoc._id = couchRes.id;
          localDoc._rev = couchRes.rev;
          store.dataSourceDidComplete(storeKey, localDoc, couchRes.id);
     } else {
        store.dataSourceDidError(storeKey, response);
     }
},


Updating Records

 

When a task is changed locally we need to send it as PUT request to CouchDB in order to update the database.

The PUT request need to be address to the document url we want to update.

The payload of the request is the enter local document which include the latest known _rev field of the document.

If for some reason someone else has update the document on the server after we have request it this PUT request will fail with 409 response which means a potential conflict. In order to correct this situation we need to re-request the document to get the latest version of it.

 

updateRecord: function(store, storeKey) {    
  if (SC.kindOf(store.recordTypeFor(storeKey), Todos.Task)) {
     var id = store.idFor(storeKey);
     var dataHash = store.readDataHash(storeKey);
     SC.Request.putUrl(this.getServerPath(id)).json()
                       .header('Accept', 'application/json')
                      .notify(this, this.didUpdateTask, store, storeKey)
                        .send(dataHash);
     return YES;
   }
   return NO;
},


After the response return we yet again get the document id and the new revision of the document.

We update the _rev field of our local document.

 

didUpdateTask: function(response, store, storeKey) {
   var couchRes = this.processResponse(response);
   if (couchRes.ok) {
     // Update the local _rev of this document.
     var localDoc = store.readDataHash(storeKey);
     localDoc._rev = couchRes.rev;
     store.dataSourceDidComplete(storeKey, localDoc) ;
   } else {
     store.dataSourceDidError(storeKey);
   }
},
  

Destroying Records

 

Once the user decide to delete a task we need to send DELETE request in order for the document representing the Task to be deleted.

Here like in the update function we address the request to the document url.

Because DELETE request should not carry any payload we deliver the revision of the document as query parameter.

 

destroyRecord: function(store, storeKey) {
    if (SC.kindOf(store.recordTypeFor(storeKey), Todos.Task)) {
          var id = store.idFor(storeKey);
          //var rev = this._docsRev[id];
          var dataHash = store.readDataHash(storeKey);
          var rev = this.getDocRev(dataHash);
          SC.Request.deleteUrl(this.getServerPath(id + "?rev=" + rev)).json()
                            .header('Accept', 'application/json')
                            .notify(this, this.didDeleteTask, store, storeKey)
                            .send();
          return YES;
     } 
     return NO ; // return YES if you handled the storeKey
  },

Once a response is return we process it like any other operation above but if all succeeded we notify the DataStore that the record has been destroyed.

  didDeleteTask: function(response, store, storeKey) {
     var couchRes = this.processResponse(response);
     if (couchRes.ok) {
          store.dataSourceDidDestroy(storeKey);
     } else {
          store.dataSourceDidError(response);
     }
  }


The end

}) ;

 

 

 

Settings up proxy

 

Because we are currently running our SproutCore application using sc-server which usually use port 4020 and we run CouchDB as online database which run in different url or local database which run in different port we need to setup a proxy for sc-server to use.

 

Open Buildfile in your root directory of the Todos application and add this line:

proxy '/todos', :to => 'localhost:5984'

 

Fusion backend with frontend (aka CouchApp)

 

As you already know after you complete to develop SproutCore application all you end up deploying is static files like HTML, CSS, JavaScript.

What you might not know yet is that CouchDB know to store files as well as JSON documents. By files I mean any static file you can store on a disk you can also store inside CouchDB. Also after storing the file in CouchDB you can access it via HTTP like you access static file in any other web server.

If you combine those two things together you get a CouchApp which is according to CouchDB docs is just a JavaScript and HTML5 app that can be served directly to the browser from CouchDB. 

The good thing about this architecture is:

 

  1. You simplify deployment because you use the same backend as database as well as web-server.
  2. You don't need any proxy because the same URL (that is host and port) you use to get you app is the URL you goto the fetch and update data so keep the Same Origin Policy.