validates_presence_of :purpose :

Refactoring asynchronous code

Mon, 15 Feb 2010 16:17:00 +0000

How to refactor a long chunk of asynchronous code is one thing I learned during my Javascript & CouchDB project. It’s not a difficult thing, but I thought it was before I figured out how to do it, so I guess it might be interesting for Javascript newbies.

Asynchronous code

In the first couple of weeks I struggled with this new style of coding before I got the hang of it. If you’re fairly new to Javascript, you’re probably used to just assign a variable in one line and use it in the next line. In the asynchronous world you query for a value and do stuff with it in the callback function of the query.

To illustrate - “normal” code looks like this:

var result = db.query("select * from T");
// use result

Althoug writing web applications just screams for doing it like this instead:


db.query("select..", function (result) {
// use result
});

(Examples stolen from Ryan Dahl’s Presentation about Node.js.)

The Problem

I have a method that checks if I have to display a notification message. For this it makes a lot of requests to the database, the received data has to be compared with other data, there’s lots of if/else clauses to consider all kinds of conditions and edge cases. It started simple, but then more and more conditions came in, unless I had the much feared diagonal closing bracket line on my screen.

You don’t need to know the details, just look at the picture and you’ll get the impression.

Refactoring

By way of illustration I made up a much shorter example:


checkForConflicts: function(){
  //set arguments
  openDoc(id, function(result){
    // do stuff with arguments
    if(result.bar == whatever){
      // do more stuff with result
      openAnotherDoc(id2, function(result2){
        if(result2.baz == true){
          // show notification message with result2
        }
      });
    }
  });
}

It is time to refactor after you are sure this actually is the way to go and the code basically works (“make it work, then make it pretty”). First you group the code into methods: You split the lines by what they do, and then you name each section. I think around 8 LOC is a good length for a method.

If this was synchronous code, you could just call these methods after each other. But this doesn’t work here, because result and result2 are not available outside the function that retrieves them.

My first approach was to call the functions from within each other. This works - but to see what checkForConflicts does at its very heart, you have to scroll down through all the methods - not much gained:

checkForConflicts: function(){
  //set arguments
  doSomething(arguments);
}

doSomething(arguments){
  openDoc(id, function(result){
    // do stuff with arguments
    if(result.bar = whatever){
      doSomethingElse(result);
    }
  });
}

doSomethingElse(result){
  // do more stuff with result
  openAnotherDoc(id2, function(result2){
    if(result2.baz == true){
      showMessage(result2);
    }
  });
}

showMessage = function(result2){
  // show notification message with result2
}

The way to do it is to use callbacks. Each method gets an anonymous function as (additional) argument: in the method declaration this function has the name callback. This callback is called if all the conditions within the method are met. The callback gets its arguments from within the method that calls it. You have to specify these arguments again in checkForConflicts to have a valid function definition.

checkForConflicts: function(){
  //set arguments
  doSomething(arguments, function(result){
    doSomethingElse(result, function(result2){
      showMessage(result2); 
    });
  });
}

doSomething(arguments, callback){
  openDoc(id, function(result){
    // do stuff with arguments
    if(result.bar = whatever){
      callback(result);
    }
  });
}

doSomethingElse(result, callback){
  // do more stuff with result
  openAnotherDoc(id2, function(result2){
    if(result2.baz == true){
      callback(result2);
    }
  });
}

showMessage = function(result2){
  // show notification message with result2
}

When you look at checkForConflicts, you now immediately see what it does, without being bothered by the details. As a side effect, deciding which code needs access to which arguments helps you to understand your code better. In my case I was able to optimize the use of passed variables a lot.

You might argue that the code is much longer now. First, longer is not worse if it is more readable. Second, this is only an example - if there was code instead of //do stuff the few extra lines wouldn’t carry weight.

Finally here is a screenshot of my refactored code. All the methods don’t fit on one screen, but you can see what checkForConflicts does very quickly.

Original article written by Lena and published on validates_presence_of :purpose | direct link to this article

Testing PUT requests with Culerity

Sun, 31 Jan 2010 21:22:00 +0000

One major pain at testing with culerity so far was a HtmlUnit bug that causes a ScriptException with PUT AJAX requests in jQuery. When you tried to test an update you got a “java.lang.IllegalArgumentException: The content cannot be null”. Fortunately, this bug is fixed now. With latest HtmlUnit build culerity should run fine, and I tell you how.

I assume you have culerity installed already. If not, follow these instructions for testing Rails apps or these for testing Couchapps.

1. Make sure you have up-to-date versions of celerity and culerity:
$ jruby -S gem install celerity #(gives you celerity-0.7.8)
$ gem install culerity –source http://gemcutter.org #(gives you culerity-0.2.7)

2. Get the latest HtmlUnit nightly build:
$ wget http://build.canoo.com/htmlunit/artifacts/htmlunit-2.7-SNAPSHOT-with-dependencies.zip
$ unzip htmlunit-2.7-SNAPSHOT-with-dependencies.zip
$ cd htmlunit-2.7-SNAPSHOT-with-dependencies/lib
These are the jar files you need to copy into your celerity gem.

3. Find your celerity gem and copy the files:
$ jruby -S gem which celerity
$ cp * PATH_OF_JRUBY_INSTALLATION/lib/ruby/gems/1.8/gems/celerity-0.7.7/lib/celerity/htmlunit

4. The new HtmlUnit needs JQuery 1.4, so you need to upgrade if you haven’t done already. There’s a backwards compatibility plugin for 1.3 if this gives you trouble.

Have fun!

Original article written by Lena and published on validates_presence_of :purpose | direct link to this article

JSpec - JavaScript Unit testing how it should be

Mon, 04 Jan 2010 12:07:00 +0000

Selecting a unit test framework for a JavaScript-only application - not a task with an obvious answer! Coming from the Rails world, where most people work with either TestUnit or RSpec, I was a bit lost when I started researching. This table on wikipedia was quite helpful for a first overview. But: JsUnit, JSTest, jsUnitTest, jsUnity - these are actually different frameworks, and there’s even many more …

I narrowed the selection down to those being currently developed, or are in use in “big” projects. JsUnit seems to be one of the major players, but I found almost no documentation, and the code base hasn’t changed a lot in the last years. That’s not a negative thing in itself, but having an active community around is a good thing. QUnit is used for testing JQuery, it looked like you could work with it as well, but its syntax and setup didn’t make me go “wow” exactly. Same goes for JQUnit, about which I had read good stuff in several blog posts. It’s also what Sammy.js is tested with.

I like RSpec, so when I had my first look at ScrewUnit, I was delighted to find there’s BDD syntax for JavaScript too. ScrewUnit is also a part of Blue Ridge, which is sadly still only available as a Rails plugin. So, ScrewUnit looked nice.

And THEN … I found JSpec. I was thrilled! It’s a JavaScript framework, but it doesn’t look like JavaScript at all: no curly brackets, no semicolons, just plain domain language. It just looks like RSpec. Here is a random snippet from my test suite, just have a look:

describe 'NoteCollection'
  before_each
    note3 = new Note({id: "8c8", text: 'three'});
    note1 = new Note({id: "ae9", text: 'one', next_id: "107"});
    note2 = new Note({id: "107", text: 'two', next_id: "8c8"});
      
    notes = new NoteCollection([note3, note1, note2]);
  end
  
  describe 'firstNote'
    it 'should return a note'
      notes.firstNote().should.be_an_instance_of Note
    end
    
    it 'should return the first note'
      notes.firstNote().id.should.eql note1.id
    end

    it 'should throw an error if there is more than one note that could be the first note'
      note4 = new Note({id: "fff", rev: "2-420"});
      notes.notes.push(note4);
      -{notes.firstNote()}.should.throw_error 'More than one first note found'
    end  
  end
end

And that are just 3 out of over 45 core matchers. There are also matchers for JQuery, see the next example, where I put the DOM I wanted to test in a fixture called outline.html. In Rails I don’t use fixtures, but if you want to test an application where all the markup is generated out of many different sources, it makes sense to test little pieces of the DOM in isolation. In ScrewUnit you have only one file for your all your mocked DOM.

describe 'NoteElement'
  before_each
    outline = elements(fixture('outline'))
    notes = outline.find('li')
  
    parent_note = new NoteElement($(notes.get(0)))
    child_note = new NoteElement($(notes.get(1)))
  end
  
  describe 'focusNextTextarea'
    it 'should focus the first child note if there is one'
      parent_note.focusNextTextarea();
      parent_note.noteLi().should.not.have_attr("data-focus")
      child_note.noteLi().attr("data-focus").should.eql 'true'
    end
  end

  describe 'insertUpdateNotePointers'    
    it 'should call setNextPointerToNewlyInsertedNote'
      inserted_note = {id:'4', text: 'inserted!'};
      parent_note.should.receive('setNextPointerToNewlyInsertedNote').with_args(inserted_note)
      parent_note.insertUpdateNotePointers(inserted_note);
    end
  end
end

Note, if you don’t like the RSpec-like Syntax, you can also skip the JSpec DSL and just use plain Javascript syntax.

More JSpec goodness: You can either run the tests in the browser, or in the terminal, so you can include it in your CI. In the terminal you can specify the browsers it should be tested with: The browsers will be opened in the background, but you won’t see your stuff being run like in Selenium. You can mock timers to test asynchronous calls. You can stub methods, mock AJAX requests, specify shared behaviours, hook into JSpec to specify your own matchers or do whatever else, and … just see for yourself.

Installation is easy too: just add JSpec.js and JSpec.css to your application. If you need them, add the files for jquery, timers or xhr support too. In spec/index.html you include your application files and specify which tests you want to run. Then open it in the browser, wait a few milliseconds, and see the result. If you want to run JSpec tests in the terminal, just install the JSpec Ruby gem. For this you also have to install Rhino (a Java-based JavaScript interpreter) - when you are on OS X, this should get you started.

What I don’t like about JSpec … the BlueRidge browser output is much more clear and pretty. I thought really hard about more things but that’s it.

Make sure to read the documentation on the JSpec website as well as the github Readme. Both ressources look similar, but at the time of writing there were different things on the two pages.

Original article written by Lena and published on validates_presence_of :purpose | direct link to this article

Bulk deletion of documents in CouchDB

Tue, 22 Dec 2009 17:37:00 +0000

For the couchapp I'm writing, I recently wanted to do a bulk delete operation with CouchDB, but found very little about it on the web. I don't know if my approach is the best way to go, but well, it works - if you know a better way, I'm happy to learn about it.

In the file jquery.couch.js that comes with CouchDB there are a couple of convenience methods to help you dealing with the database, documents and views. There is a method for deletion too (removeDoc), but it only can handle one document at a time. In the style of bulkSave, I added a bulkRemove method:

bulkRemove: function(docs, options){
  docs.docs = $.each(docs.docs, function(i, doc){doc._deleted = true});
  $.extend(options, {successStatus: 201});
  ajax({
      type: "POST",
      contentType: "application/json",
      dataType: "json", 
      data: toJSON(docs),
      url: this.uri + "_bulk_docs" + encodeOptions(options)
    },
    options,
    "The documents could not be deleted"
  );
},

In here I add the attribute "_deleted = true" to every document I want to delete, and then I do a bulk update with the documents. That's it.

This is one possible use case: In the delete action of my controller I first fetch an array with the post and its comments from the couch (how to do this see below), and then I call the bulkRemove action on that array.

couchapp.design.view('post_with_comments', {
  startkey: [params['id']],
  endkey: [params['id'], {}],
  success: function(json) {
    if (json.rows.length > 0) { 
      var post_and_comments = json.rows.map(function(row) {return row.value}); 
      couchapp.db.bulkRemove({docs: post_and_comments}, {
        success: function() {
          flash = {message: 'Post deleted.', type: 'notice'}
          redirect('#/outlines', flash);
        },
        error: function(response_code, msg) {
          flash = {message: 'Error deleting post: ' + msg, type: 'error'};
        }
      });
    }      
  }
});

Note that the "flash" object I'm using is my own Sammy.js implementation of a Rails-like flash to show a message after a redirect. Maybe I'll share that in another blog post.

As a bonus, this is the view I'm using to retrieve a post and its comments in one request. I'm following the pattern Christopher Lenz is recommending in his article on CouchDB Joins.

function(doc) {
  if (doc.type == "Post") {
    emit([doc._id, 0], doc);
  } else if (doc.type == "Comment") {
    emit([doc.post_id, 1, Date.parse(doc.created_at)], doc);
  }
}

When you call this view with

startkey: [params['id']],
endkey: [params['id'], {}]

you get an array with the post as the first element, and then all the comments that have the post's ID as post_id.

This can of course also be used for things other than deleting: to show all the comments for a post, I get the array's first element and extract the title and content. Then I remove it from the array with

json.rows.splice(0,1);

You now have an array with only the post's comments and you can display them somehow.

Original article written by Lena and published on validates_presence_of :purpose | direct link to this article

Testing key events with Culerity

Sun, 22 Nov 2009 20:43:00 +0000

The application I'm writing depends heavily on key events. When you press enter in a specific text field, some other field opens, when you press tab, the text fields moves, etc. Of course I need to cover this in my cucumber integration tests. The Celerity API is a bit thin for this purpose: You can fire a, let's say, keydown event - but it doesn't pass through the specific key code.

Actually it's easy when you know how. You can execute any javascript directly on the Culerity RemoteBrowserProxy object. In order to fire a specific key event, you have to create a KeyboardEvent, initialize it with whatever you want, and dispatch it on the element.

Read up the details, especially the meaning of initKeyEvent's parameters in this Javascript API.

I ended up with this cucumber step:

When /^I hit "([^\"]*)" in a text_field with id "([^\"]*)"$/ do |key, id|
  key_code = case key
    when 'enter'
      13
    when 'up'
      38
    when 'down'
      40
    else
      0
    end
  $browser.execute_script('event = document.createEvent("KeyboardEvent");')
  $browser.execute_script('event.initKeyEvent("keydown", true, false, document.window, false, false, false, false, ' + key_code.to_s + ', 0)')
  $browser.execute_script("document.getElementById('#{id}').dispatchEvent(event)")
  When 'I wait for the AJAX call to finish'
end

Note that this is meant to work for Firefox only. For IE you can do a similar thing with createEventObject, see here how.

Original article written by Lena and published on validates_presence_of :purpose | direct link to this article

Couchapp / Culerity glitches, part 1

Wed, 04 Nov 2009 10:07:00 +0000

Do you know what's behind these buzzwords: CouchDB / Couchapp / Sammy.js? Not really? Then please look at this article now: Sammy.js, CouchDB, and the new web architecture. The title says it all.

This setup happens to be the field of my diploma thesis, friendly sponsored and supported by Upstream. During the next months I'll develop a super duper thing with these technologies, and then I'll write 80 scientific pages about it. Yees, I'm not that much looking forward to the second part! Until then, I'm planning to semi-regularly blog about my findings.

I'm not allowed to share the code before I have the diploma in my hands. But, because my workmate Frank Proessdorf is so jealous of what I do, he and I started working on an app with a similar setup on our Upstream Research Fridays, to try things out. Here it is. Don't expect anything yet.

Today I'm going to tell you about some details of my testing setup. My *cough* supervising tutor Alexander Lang already wrote down all there is to know yet about Test-Driven Development with Cucumber-Culerity-Celerity and Couchapps.

But there are a few things that will make your life hard, when the almost perfect Rails testing world has spoiled you as much as me. One thing is that we have to deal with a lot of asynchrony. Sammy renders the page, loads the data from the couch, and whenever the data is there it gets rendered into the page. The best way to do this is via callbacks. I'm going to write another blog post about that soon.

In the integration test the problem is that celerity doesn't behave as it claims it does:

$browser.wait

is supposed to make the test wait until everything is rendered completely. But for whatever reason it doesn't. The result is that your test fails because it searches for your expressions in an unfinished page.

Because of that, in culerity we need something like:

sleep 0.4

after every follow, press and go-to-path step. When using culerity in a Rails environment, this usually is enough. Sammy behaves a bit more unpredictable - it sometimes takes more than a whole second until everything is loaded.

Solution? We can either increase the sleep time a lot, but that's not very nice, the features need long enough to run as it is. Fortunately we can use another method that's build into celerity: wait_while.

I created a div "spinner" somewhere in index.html:

 <div id="spinner"></div>

You can also add an actual spinner gif here. Just take care to place the div outside of your sammy element selector ("#main" by default), because this div gets replaced all the time.

We need to show that div before every route that gets run. In application.js:

before(function() {
  $('#spinner').show();
});

And we need to hide it after your rendering is completed. I haven't found an ideal solution where to do that, it depends on the application you are writing. As a quick and dirty fix, you can add it as a last line in sammy's partial function, in sammy.js:

partial: function(path, data, callback) {
  //...
  $('#spinner').hide();
},

I prefer to have more control about when I hide it, as my partial calls sometimes have nested callbacks. So I put it in application.js, resp. in my controllers, after each render call or in the partial callbacks. I admit this still sucks a bit.

Finally, amend the cucumber step:

When /I wait for the AJAX call to finish/ do
  sleep 0.2
  puts 'Waiting for page to load ...' if $browser.div(:id, 'spinner').visible?
  $browser.wait_while { $browser.div(:id, 'spinner').visible?} 
end

The browser still needs to sleep a bit before it actually finds the spinner div. But with this step in my common_culerity_steps.rb, the features finally run smoothly.

Original article written by Lena and published on validates_presence_of :purpose | direct link to this article

Comatose bug when creating pages in migrations

Tue, 27 Oct 2009 22:27:00 +0000

Comatose is a micro CMS, easy to embed in your Rails app, ideal when the customer wants to update her FAQ once in a while. I work on a few projects where comatose is integrated, and it's been the second time now that one specific problem has given headaches to me and my colleagues. Time to document the solution.

Comatose has a hierarchical structure. All the pages are children of the comatose root page, which is usually not in use. This root page is created by the migration AddComatoseSupport. When we add comatose to a project, we usually create another migration, where some child pages are created and populated (because the customer usually already knows what pages she wants). This can look like this:

root = ComatosePage.find_by_title('Home Page')
faq = ComatosePage.create(:title => 'FAQ', :author => 'System', :body => "Text goes here.", :parent_id => root.id

All will be fine if you run the AddComatoseSupport migration first, and then the other one. If you run both of them in one go, all will look correctly first, too. But then - try to access the application and you get "a nil object when you didn't expect it!" A look in the database will tell you that your root page and your FAQ page are both there - but the slugs (needed to access the pages) are nil. Why is that?

The answer: The class ComatosePage gets overwritten in the AddComatoseSupport migration, to make sure the comatose root page can be created (they are invalid without parent_id). But if you have another migration following this one, where you create a couple of comatose pages, they still are instances of that overwritten class. Especially the before/after filters from the original ComatosePage aren't evaluated. In those filters the slug fields are meant to be populated, so they are nil.

First soluton: Define a class ComatoseRootPage instead, so you won't override anything.

Second and much easier solution: Remove the class redefinition altogether, for the root page just do

ComatosePage.new(...).save(false)

See on github.

Unfortunately the comatose project is not updated very often, so this bug is still unsolved in the generated migrations.

Original article written by Lena and published on validates_presence_of :purpose | direct link to this article

Snippets for cucumber

Sun, 25 Oct 2009 21:53:00 +0000

In every project I use cucumber (that means, always) there are a few things that I keep reinventing, or I spend a lot of time searching for them in other projects. Here are some of them. They aren't all written by me, but maybe my coworkers like to use this as a reference, too?

When you upgrade to the current cucumber (0.4.2) from an older version some steps will fail, because cucumber doesn't examine the whole html anymore. You can still check the raw html with these steps:

Then /^I should see "([^\"]*)" in the source$/ do |text|
  response.body.should match(/#{text}/)
end

Then /^I should not see "([^\"]*)" in the source$/ do |text|
  response.body.should_not match(/#{text}/)
end

Want to test sorting in one step?

Then /^I should see "([^\"]*)" before "([^\"]*)"$/ do |first, second|
  response.body.should match(/#{first}.*#{second}/im)
end

Sometimes you want to click a link in a specific div:

When /^I buy "([^\"]*)"$/ do |title|
  product = Product.find_by_title(title)
  field_by_xpath("//div[@id='buy_product_dialog_product_#{product.id}']//input").click
end

Or you want to test what happens when you delete a specific object:

When /^I press "Delete" for comment "([^\"]*)"$/ do |comment|
  tag = Comment.find_by_title(comment)
  field_by_xpath("//form[@id='delete_comment_#{comment.id}']//input[@type='submit']").click
end

With this you can check for a string that's within a tag with a specific id:

Then /^I should see "([^\"]+)" in "([^\"]+)"$/ do |content, css_id|
  response.body.should have_tag("##{css_id}", content)
end

Use this regex to check for something within a given tag:

Then /^I should see "([^\"]*)" within a (\w+) tag$/ do |text, tag|
  response.body.should match(/<#{tag}[^>]*>[^<]*#{text}[^<]*<\/#{tag}>/)
end

Cucumber 0.4.2 auto-generates webrat_steps.rb including a step to save the current page and open it in the browser. Just in case not everyone knows this step by now:

Then /^show me the page$/ do
  save_and_open_page
end

Original article written by Lena and published on validates_presence_of :purpose | direct link to this article

Things I learned recently

Sat, 24 Oct 2009 21:12:00 +0000

Doing client work for Upstream, I recently had the pleasure of pair programming with fabulous Pat Allan for a month. Here are a few of the things I learned.

Pair Programming is a good thing (not that I haven't known that before). Here is yet another good article about pairing if you aren't convinced yet. Next to improved concentration, keeping each other from getting stuck in stuff, I like the sharing of best practices, Textmate short cuts, syntax tricks etc.
Instead of writing "puts" in, after and before blocks that don't act as expected, use ruby-debug. It's really easy, and gives you so much more information about your situation. Just do:
```
sudo gem install ruby-debug
```
And in your code, at the place you want to examine:
```
require 'ruby-debug'
debugger
```
Then restart script/server with the --debug option. It stops when the line where you put 'debugger' is reached. You can directly call variables to explore their values. The basic commands are:
- n - next
- s - step into method
- c - continue to breakpoint
Read the source code of the gems you're using. If you are not sure how to do something with the gem, the actual code is often more informative than documentation. Lachie Cox describes in his blog how to set up a Textmate shortcut with which you can access all your gems without any typing.
Rubular is a nice app to test regular expressions in your browser.
I have never used RCov before. Of course it doesn't mean too much, but having a test coverage of 97,7% feels good, I can tell you. A Code to Test Ratio of 1:2.7 is not bad either.
The RCov gem has moved to Relevance, read here how to set it up for an RSpec/Cucumber environment.
And finally, I got convinced that it's good to have a blog ...

Here we are!

Original article written by Lena and published on validates_presence_of :purpose | direct link to this article