validates_presence_of :purpose : Tag couchdb, everything about couchdb

Thesis is done - thanks for your support!

Thu, 05 Aug 2010 10:41:00 +0000

I’m done!! Three weeks ago I handed in my diplom thesis. The practical part was the design and implementation of a distributed, replicatable outliner with the NoSQL database CouchDB. As theoretical work I evaluated Distributed Systems, CouchDB, various web technologies and programming strategies. I still have to pass the oral exam, but my degree is finally very close - soon I can call myself a graduated computer scientist. Yay!

As soon as academic procedures allow it, I’ll publish the text and the source code of the thesis here. Until then I’d like to say thank you to a few people. As the foreword of thesis won’t get as much attention as this blog, I’ll do it here.

First first first of all, my appreciation goes to my boys at home and my parents for backing me up by spending loads of time with my child. Not only my professional life is based upon their support. THANKS.

Further thanks go to:

Upstream for generously sponsoring a part of the time I spent researching - especially Alex Lang for his support and coming up with the initial idea. Without them I probably would have done some usability study for the drawer, or I would have had lots of fun in some enterprisey Java company. Also I felt I wanted to strive hard to write good code in my thesis to match Upstream’s high standards for code quality and test coverage - not a bad thing.

Allround guru Jan Lehnardt - he not only knew and happily provided the answers to all questions I had about CouchDB, databases in general, JavaScript, LaTeX, typography and virtually everything else; he was also constantly encouraging and guiding me to contribute to open source projects, trusting in my skills and doing awesome things. One day I’ll also submit a talk to one of your conferences, word :-)

Jan Lehnardt, myself and Alex Lang on the couch with thesis.

My professor Stefan Edlich for doing a great job in mentoring me. He is also the university professor with the most phenomenal email response times I have ever heard of.

Urs, Andy and Ischa advised me on the structuring and helped to make the result look quite scientific. They patiently explained to me how to go about writing such a long academic text. Ischa also is one of the very few people who actually read the whole thing in one piece.

Several lovely people who helped at certain points during the research or the programming, listened with interest and/or assisted figuring out specific problems: Pat Allan (getting started was the hardest part); Urs again (many small and big issues all along the way); Frank Prößdorf and Lukas Rieder (figuring out with me what a Couchapp actually is); Till Klampäckel (skilled advice about Lounge and EC2); Kristina Schneider (tricky CSS shit); Thilo Utke and Sven Fuchs (die HTMLUnit bug, die); Ian White (this one bug in the code you had no idea what it was for but figured it out anyway); and various people from the Ruby and Javascript communities who helped me to shape my ideas by asking basic or challenging questions.

Some more people who wrote great open source programs, did great support and were always open to ideas: my thanks go to a good part of the CouchDB community here, and also to Aaron Quint and TJ Holowaychuk.

Finally my army of proofreaders! I split the text up in very small parts so that also people with a limited time budget had no excuse - I can really recommend this approach ;-). The list includes almost everyone mentioned above, and also: Adrian, Anne, Birgit, Bumi, Diana, Eike, Flo, Grushi, Helge, Jan S., Micha, Neels, Ole and Patrick.

The very last one goes to my sweetheart for brightening my life in those last stressful months.

Thank you everyone! I’m glad to know you all!!

Security in CouchDB: changing the authentication db

Thu, 29 Apr 2010 10:40:00 +0000

I recently wrote a testsuite for CouchDB’s Javascript HTTP API. Among the things I tested were the security methods. That’s those that deal with authentication - like signup, login, logout and so on. The challenge was to make sure my tests wouldn’t interfere with the CouchDB setup of the person running the tests.

Background

CouchDB stores the information about the database users in a special database, the authentication_db. The default authentication_db is called “_users”. So this is where your user data is saved when you sign up without taking care about the details, e.g. when you use the signup method provided by the JQuery HTTP API. Read more about the ways to interact with the authentication_db in the HTTP API documentation (JQuery methods), and read more about general CouchDB security in the CouchDB Wiki or in The Definitive Guide.

How to set the authentication_db

When you e.g. create database users from within your tests, how can you make sure they don’t mess up the actual authentication_db, the one the unsuspecting user might use in production? Of course you have to create a custom authentication_db for storing the user information, and make CouchDB use that one. And after your tests are finished, you want CouchDB to use the original authentication_db again, so that only your testsuite uses the custom users database.

So there is one function you call right at the beginning of the part of the testsuite that’s concerned with security features:

function useTestUserDb(){
  test_users_db = new CouchDB("test_users_db");
  var xhr = CouchDB.request("PUT", "/_config/couch_httpd_auth/authentication_db", {
    body: JSON.stringify("test_users_db")
  });
  if(typeof(old_auth_db) == 'undefined'){
    old_auth_db = xhr.responseText.replace(/\n/,'').replace(/"/g,'');
  }
}

We create a custom authentication_db (“test_users_db”) and write its name into the CouchDB configuration. If we don’t have the name of the old authentication_db stored yet (the one that’s called “_users” per default, but might be called differently), we strip this name from special characters and store it in a variable, so we can restore it later. This is the function we call at the very end of the testsuite:

function useOldUserDb(){
  CouchDB.request("PUT", "/_config/couch_httpd_auth/authentication_db", {
    body: JSON.stringify(old_auth_db)
  });
  test_users_db.deleteDb();
}

In the last line, we delete our custom authentication_db. Everything is back to normal, there is no trace of our tests left. You can have a look at how it looks like in production.

How to work with the authentication_db

You can create a new user like this:

var userDoc = CouchDB.prepareUserDoc({name: "Gaius Baltar", roles: ["president"]}, "secretpassword");
test_users_db.save(userDoc);

Then you are logged in and can do whatever you want. About working with users and roles, read the chapter in The Definitive Guide. When you’re done, delete the userDoc from the authentication_db, as the authentication_db has to be empty before it can be deleted:

test_users_db.deleteDoc({_id : userDoc.id, _rev : userDoc.rev})

Writing a testsuite for the CouchDB API

Wed, 28 Apr 2010 12:07:00 +0000

CouchDB is a database that I quite like. In fact, I haven’t done much else this winter, as the programming part of my thesis involved diving into many of CouchDB’s features. I had just finished my little program and was desperately looking for an excuse not to start writing the thesis. Mercifully Jan from Couchio came along and asked if I wouldn’t like to write a testsuite for CouchDB’s Javascript HTTP API, and the documentation with it. I wanted!

What is this HTTP API?

If you have worked with CouchDB from a Javascript program before (eg Sammy.js is a good companion for a simple web application), you have probably used the methods a file named couch.js provides. With this interface you can easily access the most common CouchDB functionality without having to handle the XMLHttpRequest yourself. It allows you to create a “database object” and then perform things on this object, for example save or delete documents:

var db = new CouchDB("test_db");
db.createDb();

var doc = {"_id" : "Starbuck", "name" : "Kara Thrace"};
var saved_doc = db.save(doc);

// do something
db.deleteDoc({_id : "123", _rev : saved_doc.rev});

If you’re using JQuery, you can use the JQuery interface in jquery.couch.js. It covers basically the same things as couch.js, but it uses, you guessed it, JQuery. That means you can send your requests asynchronously and handle the response in the callback. Same example:

var db = $.couch.db('spec_db');
db.create();

var doc = {"_id" : "Starbuck", "name" : "Kara Thrace"};
db.saveDoc(doc, {
  success: function(response){
    // do something
    db.removeDoc({_id : "123", _rev : response.rev});
  }
});

The code looks a bit more complicated, but with this API you have the full power of asynchronicity in your head, where spelling the word asynchronicity is the worst part.

Writing the testsuite

I was free to choose which testing framework I wanted to use. The tests in CouchDB use a kind of home grown testing framework, which does its job, but is not really the state of the art of Unit Testing. The API tests aren’t supposed to run within the normal CouchDB testsuite anyway. That is the one you always run in Futon when you installed CouchDB. So it was possible to use something different … like JSpec. (I have also blogged and talked about it in the past.)

How to run JSpec tests? If you are familiar with JSpec tests, you know you define your tests in Javascript files, reference those from a HTML file and open that in a browser. When you want your tests to perform calls to the database, this won’t work - you get the error message “Access to restricted URI denied”. Shortly this is because AJAX only allows connections to the same resource it is called from. Therefore you have to run the tests from within the database. CouchDB has a folder share/www, which gets linked to the “/_utils/” path of your local installation. In this folder you find Futon. That’s where you have to put your tests and all the helper scripts. Then you can run the tests directly from the couch by browsing to /_utils/spec/run.html.

Writing the testsuite was pretty straightforward. Along the way I found some bugs in the code (no surprise as it was written untested). One of the problems I had to solve I’ve explained in another blogpost: How to overwrite the authentication_db so the tests don’t interfere with the CouchDB setup.

There’s one thing I’m not completely happy with: The tests do really need kind of long to run, up to 3 minutes, depends a lot on the browser and CPU load. The other CouchDB Javascript tests are a lot faster. I think the new ones are a bit more comprehensive. And I preferred a very clean setup to how it’s done in the old tests: I have a database created before every single assertion and have it dropped afterwards. So you know exactly in which state your database is during the test. I guess that’s worth the longer runtime.

So. If you want to see a lot of JSpec code in action, and want to learn more about CouchDB, have a look at the code. At the moment it’s in my fork only, the community (someone - you?) has yet to integrate it into Futon. For all the details, read my rundown and left TODOs.

Documentation

I also wrote API Docs for all the methods. It can be found here. Later you will find it nicely formatted in the CouchDB Wiki and on the Couchio website.

If you find anything is missing or you don’t understand it, feel free to drop me a line. I’m happy to clarify it as long as it’s all freshly in my brain.

PS - I consider example data in tests important, so I took all the data from the SciFi series Battlestar Galactica, currently on my playlist and I luv it. And it gave me an excuse to read in the Battlestar Wiki every now and then when I was looking for correct spelling :)

Bulk deletion of documents in CouchDB

Tue, 22 Dec 2009 17:37:00 +0000

For the couchapp I'm writing, I recently wanted to do a bulk delete operation with CouchDB, but found very little about it on the web. I don't know if my approach is the best way to go, but well, it works - if you know a better way, I'm happy to learn about it.

In the file jquery.couch.js that comes with CouchDB there are a couple of convenience methods to help you dealing with the database, documents and views. There is a method for deletion too (removeDoc), but it only can handle one document at a time. In the style of bulkSave, I added a bulkRemove method:

bulkRemove: function(docs, options){
  docs.docs = $.each(docs.docs, function(i, doc){doc._deleted = true});
  $.extend(options, {successStatus: 201});
  ajax({
      type: "POST",
      contentType: "application/json",
      dataType: "json", 
      data: toJSON(docs),
      url: this.uri + "_bulk_docs" + encodeOptions(options)
    },
    options,
    "The documents could not be deleted"
  );
},

In here I add the attribute "_deleted = true" to every document I want to delete, and then I do a bulk update with the documents. That's it.

This is one possible use case: In the delete action of my controller I first fetch an array with the post and its comments from the couch (how to do this see below), and then I call the bulkRemove action on that array.

couchapp.design.view('post_with_comments', {
  startkey: [params['id']],
  endkey: [params['id'], {}],
  success: function(json) {
    if (json.rows.length > 0) { 
      var post_and_comments = json.rows.map(function(row) {return row.value}); 
      couchapp.db.bulkRemove({docs: post_and_comments}, {
        success: function() {
          flash = {message: 'Post deleted.', type: 'notice'}
          redirect('#/outlines', flash);
        },
        error: function(response_code, msg) {
          flash = {message: 'Error deleting post: ' + msg, type: 'error'};
        }
      });
    }      
  }
});

Note that the "flash" object I'm using is my own Sammy.js implementation of a Rails-like flash to show a message after a redirect. Maybe I'll share that in another blog post.

As a bonus, this is the view I'm using to retrieve a post and its comments in one request. I'm following the pattern Christopher Lenz is recommending in his article on CouchDB Joins.

function(doc) {
  if (doc.type == "Post") {
    emit([doc._id, 0], doc);
  } else if (doc.type == "Comment") {
    emit([doc.post_id, 1, Date.parse(doc.created_at)], doc);
  }
}

When you call this view with

startkey: [params['id']],
endkey: [params['id'], {}]

you get an array with the post as the first element, and then all the comments that have the post's ID as post_id.

This can of course also be used for things other than deleting: to show all the comments for a post, I get the array's first element and extract the title and content. Then I remove it from the array with

json.rows.splice(0,1);

You now have an array with only the post's comments and you can display them somehow.

Couchapp / Culerity glitches, part 1

Wed, 04 Nov 2009 10:07:00 +0000

Do you know what's behind these buzzwords: CouchDB / Couchapp / Sammy.js? Not really? Then please look at this article now: Sammy.js, CouchDB, and the new web architecture. The title says it all.

This setup happens to be the field of my diploma thesis, friendly sponsored and supported by Upstream. During the next months I'll develop a super duper thing with these technologies, and then I'll write 80 scientific pages about it. Yees, I'm not that much looking forward to the second part! Until then, I'm planning to semi-regularly blog about my findings.

I'm not allowed to share the code before I have the diploma in my hands. But, because my workmate Frank Proessdorf is so jealous of what I do, he and I started working on an app with a similar setup on our Upstream Research Fridays, to try things out. Here it is. Don't expect anything yet.

Today I'm going to tell you about some details of my testing setup. My *cough* supervising tutor Alexander Lang already wrote down all there is to know yet about Test-Driven Development with Cucumber-Culerity-Celerity and Couchapps.

But there are a few things that will make your life hard, when the almost perfect Rails testing world has spoiled you as much as me. One thing is that we have to deal with a lot of asynchrony. Sammy renders the page, loads the data from the couch, and whenever the data is there it gets rendered into the page. The best way to do this is via callbacks. I'm going to write another blog post about that soon.

In the integration test the problem is that celerity doesn't behave as it claims it does:

$browser.wait

is supposed to make the test wait until everything is rendered completely. But for whatever reason it doesn't. The result is that your test fails because it searches for your expressions in an unfinished page.

Because of that, in culerity we need something like:

sleep 0.4

after every follow, press and go-to-path step. When using culerity in a Rails environment, this usually is enough. Sammy behaves a bit more unpredictable - it sometimes takes more than a whole second until everything is loaded.

Solution? We can either increase the sleep time a lot, but that's not very nice, the features need long enough to run as it is. Fortunately we can use another method that's build into celerity: wait_while.

I created a div "spinner" somewhere in index.html:

 <div id="spinner"></div>

You can also add an actual spinner gif here. Just take care to place the div outside of your sammy element selector ("#main" by default), because this div gets replaced all the time.

We need to show that div before every route that gets run. In application.js:

before(function() {
  $('#spinner').show();
});

And we need to hide it after your rendering is completed. I haven't found an ideal solution where to do that, it depends on the application you are writing. As a quick and dirty fix, you can add it as a last line in sammy's partial function, in sammy.js:

partial: function(path, data, callback) {
  //...
  $('#spinner').hide();
},

I prefer to have more control about when I hide it, as my partial calls sometimes have nested callbacks. So I put it in application.js, resp. in my controllers, after each render call or in the partial callbacks. I admit this still sucks a bit.

Finally, amend the cucumber step:

When /I wait for the AJAX call to finish/ do
  sleep 0.2
  puts 'Waiting for page to load ...' if $browser.div(:id, 'spinner').visible?
  $browser.wait_while { $browser.div(:id, 'spinner').visible?} 
end

The browser still needs to sleep a bit before it actually finds the spinner div. But with this step in my common_culerity_steps.rb, the features finally run smoothly.