Moves On Rails: Category Ruby

Rails log analyzer

bart.tenbrinke@movesonrails.com (Bart ten Brinke) — Thu, 14 Aug 2008 22:58:00 +0200

You've probably all been there: your application is running slow, but why? What views or actions are clogging up the mongrels? Or are the mongrels just waiting for the database?

Rails log analyzer is a simple but very powerful command-line analysis tool to quickly determine what is taking time, on all kinds of different levels. At the moment it can tell you the following statistics:

Top 10 most requested actions
Top 10 actions by time - cumulative
Top 10 actions by time - per request mean
Top 10 worst DB offenders - cumulative time
Top 10 worst DB offenders - mean time
Mongrel process blockers (> 1.0 seconds) - frequency
Requests graph - requests per hour

For an example run, or the analyzer take a look at the github. http://github.com/wvanbergen/rails-log-analyzer/

Optimizing math

andre.foeken@movesonrails.com (Andre Foeken) — Thu, 10 Apr 2008 22:43:00 +0200

As you might have guessed from Bart's previous article we've been looking at ways to speed up our Rails app. We've been profiling and query optimizing but at some point we reached a dead end.

Our app needs to calculate a lot of distances between geo locations. Until now we've been happy using a home-grown Ruby method to calculate these distances but our profiling showed that it was (as one may suspect) horribly slow.

We now have several options:

We could try to built a faster Ruby method (but that would be hard since it's pure math and not really a lot can be done here)
We could use the database (mysql in our case) to calculate our distances (A lot more db connections in our case, since we need distances between lots of points. Not the standard stuff that gems like acts_as_mappable can handle)
Use RubyInline to create a faster C based method

We decided to look at RubyInline. A gem that enabled C code to be used right inside a Ruby script. We rewrote the method in C. A simple benchmark proved that our inline C method was 2.3 times faster!

require 'inline'
inline do |builder|
    builder.include '<math.h>'
    builder.c "double calc_distance_between(...) { ... }"
end

Although this result is very good, it does complicate your app and makes it less readable. These inline methods have to be used with care. But in our simple (and very localized) case we decided to keep the C method in favor of the pure Ruby call.

attr_accessor_with_default

andre.foeken@movesonrails.com (Andre Foeken) — Wed, 05 Mar 2008 09:37:00 +0100

What a marvelous feature, but be wary! It can cause some unexpected behavior if you don't know what you are doing.

This morning we found a rather suspicious bug that lead to unexpected things. Here is an example:

class Person ; attr_accessor_with_default :things, {} ; end john = Person.new john.things[:table] = true ... jim = Person.new jim.things => {:table => true} # huh??

As it would seem attr_accessor_with_default has some problems with collections. Since we are sharing the instance over the entire class. Peter Williams noted this problem several month ago in his article (which we didn't read until it was too late), however the solution he provided still left us with some very undesireable behaviour.

class Person ; attr_accessor_with_default :things, {{}} ; end # note the extra brackets! john = Person.new john.things[:table] = true ... jim = Person.new jim.things => {} # okay! john.things => {} # uhm... not okay!

The variable would only stick if we actually assigned it.

john.things = {:table => true} john.things => {:table => true} # yay!

But doing this every time is not only a pain but also introduces very hard to debug errors, since the assignment does not fail...it just doesn't work!

We solved it by going old-school. Back to the normal accessor for collections.

class Person attr_accessor :things def initialize attributes=nil super self.things = {} end end

Now the code works as expected and we can use all operators (like >>, []) from the get go.

Update: It seems this had no effect on ActiveRecord objects that were created using finders so here is the fix for any those:

class Person < ActiveRecord::Base attr_accessor :things def after_initialize self.things = {} end end

Javascript testing problems

Steven van der Vegt — Tue, 04 Mar 2008 17:02:00 +0100

A few months ago I got the assignment to set up and build javascripts-tests for the scheduling view of moves. At first I had to select a testing-framework where the test would be written in. The framework had to be able to run both unit and integrations tests. Also it would be very nice if we can run these test automatically through the command-line instead of clicking around in the browser (autotest).

After reviewing some frameworks, I chose the Crosscheck framework (http://www.thefrontside.net/crosscheck). Crosscheck is a javascript-testing-framework written in java. It is crossplatform and you can control it from the command-line. It is even able to emulate the behavior of multiple popular browsers like ie6, Firefox 1.0 and Firefox 1.5. Sounds like the ultimate testing framework!

So after my decision I started playing around and implementing tests. However, when the tests became more complex I ran into trouble. Some basic browser features like document.write() (ie6) and the Option object where missing. I was able to work around these problems, but the real trouble began with the integration tests. As our application relies heavily on ajax through the prototype framework, testing this functionality is crucial. However, I was not able to do this. Performing one hack after another, I finally gave up.

The crosscheck framework was clearly not mature enough to satisfy my needs. My conclusion is: Using crosscheck for unit-testing is doable, but the framework is not mature enough for the use of integration tests. So what are the problems with crosscheck? As mentioned earlier, it is written in Java, so it tries to emulate the browser behavior based on it's specifications. The advantages of this approach is that you can emulate more than one browser, the disadvantages are that the emulation is an approach, so you will never really get the real behavior of the browser.

If a browser changes its implementation, the framework is always outdated. You don't get the browsers quirks, so if it your tests pass in the test framework, there is no guarantee it will work in the real-browser world. Another problem with crosscheck is that is seems to be abandoned. The last changes are from end-2007 and as far as I can see none of the reported bugs have been fixed yet. Firefox 3.x and ie7 are becoming standard, but these browsers are not available as a test-browser in crosscheck (and there are no clues they will be soon).

What can we expect from the continuity of crosscheck? If you, like us, want to be able to test a long-term project, then it is important that your test-framework is long-term too. So what does my ideal javascript-test-framework look like?

It should:

Run from the command line
Represent the browser realistically
Emulate ajax responses
Mock objects
Run implementation tests

As far as I know, no such framework exists. However, in my view, it shouldn't be too hard to realize. We could embed the mozilla tree in an application. That way we always have the latest version of the mozilla engine and an exact copy of the behavior including it's quirks. Through the Mozilla API we can access the DOM and other functions. The framework should be able to easily load tests and run them (for this part it's important to take a good look at the x-unit patterns).

At this moment I don't have a clue how easy it is to emulate the ajax responses through mozilla API calls. Furthermore the framework should be give a rich toolkit for testing including abilities to mock objects. You start the framework via the command-line for easy integration with test-runners like autotest. Of course this application should be released under the GPL-licence. Now we only need someone to implement this for us!

Steven van der Vegt (s.vandervegt TA student.utwente.nl)

Ps we are offering an internship for the development of the described plugin at Nedap healthcare. Interrested? bart.tenbrinke@movesonrails.com International students welcome!

Autotest 100% CPU solution

bart.tenbrinke@movesonrails.com (Bart ten Brinke) — Wed, 06 Feb 2008 09:42:00 +0100

Autotest is great, but when it is waiting for test, my MacBook turns into a whirlwind as autotest takes 100% CPU. After looking at autotest.rb, we easily find the waiting function:

def wait_for_changes
  hook :waiting
  Kernel.sleep self.sleep until find_files_to_test
end

As self.sleep is defaulted to 1 it means that my laptop does not sleep at all. Changing this is quite easy, as you can just add this to your .autotest file in your home directory:

Autotest.add_hook :initialize do |at|
  at.sleep = 5
end

Experiment with the amount to determine what works for you. Offcourse autotest will now react slower to you changes, but hey: you can't have everything. Enjoy the silence!

The new programmers excuse for slacking of

bart.tenbrinke@movesonrails.com (Bart ten Brinke) — Thu, 31 Jan 2008 14:39:00 +0100

Original by XKCD.

Cha-Ching and dutch banks

andre.foeken@movesonrails.com (Andre Foeken) — Wed, 23 Jan 2008 20:22:00 +0100

Like lots of you I bought the incredible MacHeist app bundle last week. One of the appz was Cha-Ching, a finance program to keep track of your money.

One of the most useful features is the import. You can simply import all of your bank's records and be done with it. Unfortunately the dutch banks (or at least the Rabobank and Postbank) don't support the same formats Cha-Ching supports (like OFX/QIF).

I wrote a small ruby script to convert the banks CSV files to OFX files. There are two separate files below (one for each bank):

CSV convertor - Postbank (Ruby)

CSV convertor - Rabobank (Ruby)

Before you can run the files you need two additional gems (if you haven't got them already).

sudo gem install extensions
sudo gem install builder

After this you can use the scripts like this:

ruby csv_convert.rb [CSV FILE] > export.ofx
ruby csv_rabo_convert.rb [CSV FILE] > export.ofx

Update: Now supports Tiger using PHP version of script and (optional) automator

For the next files you don't need anything else than a vanilla Tiger install of OSX.

CSV convertor - Postbank (PHP)

CSV convertor - Rabobank (PHP)

You can use them like this:

php csv_convert.php [CSV FILE] > export.ofx
php csv_rabo_convert.php [CSV FILE] > export.ofx

For the simple version, use the automator zip file! Read the README to get it to work.

CSV convertor - Postbank/Rabobank (PHP/Automator)

Rspec plain text stories

andre.foeken@movesonrails.com (Andre Foeken) — Tue, 06 Nov 2007 15:50:00 +0100

Simply said. They are great. We've been using them since they came out and have been following the trunk ever since.

They are so easy and elegant, they just needed a little TextMate love. So here you go: A language grammer and an adapted Vibrant Ink theme. (Download here)

P.S If anyone can tell me how to include stories in autotest runs I would be very happy!

Moving to 2.0

andre.foeken@movesonrails.com (Andre Foeken) — Thu, 18 Oct 2007 15:15:00 +0200

Issues found so far:

Lots of small routing changes. Thank god for grep! : addresses_path(@employee) => employee_addresses_path(@employee) . (Don't forget the *_url methods!)
Several plugins failed due to extract_options_from_args!. This method has been replaced with the nicer: args.extract_options! (i.e acts_as_paranoid, acts_as_mappable, paginating_find)
acts_as_paranoid had minor issues: fix by replacing construct_count_with_legacy_args to construct_count_options_from_args
ActiveResource is now part of rails core. Be sure to freeze edge twice if you are upgrading from 1.2.3 or lower. All of our libXML additions had to be redone. (this time through /patches, tnx fngtps)
Different behavior of the render method. Before you could call render "addresses/show" if you wanted to, this has been changed to render :template => "addresses/show". This affects several plugins too (like rspec_on_rails)
We no longer need the mysql_tasks plugin, since this functionality is now build right in!
redhillonrails_core has some issues with connection adapters. Apparently rails 2.0 no longer loads adapters it doesn't need. This creates some issues with the redhills plugin since it tries to include stuff in those adapters. Adding a begin/rescue block around each include solves the issue.
The development environment no longer needs the config.breakpoint_server = true setting.
Polymorphic models are now saved with the base class as type in external objects.
Autocomplete textfields are now a plugin: ./script/plugin install auto_complete
If you have overridden the to_json methods, be sure to change that to to_json options={}, else they might fail.
Migrations no longer working? Try removing duplicate names. We have two migrations with the same name (but different id) and it just skipped the first one (!)

Benefits so far:

Speed! We did expect some speed increase, but this is major! Our pdf generating stuff uses a lot of ActiveRecord and we saw decreases of more than 50% request time.

ActiveResource: REST, WSDL, XSD?

andre.foeken@movesonrails.com (Andre Foeken) — Mon, 24 Sep 2007 15:57:00 +0200

We love REST. It's simple and clean, and combined with ActiveResource it is certainly the best app to app bridge we've worked with so far. But...

What if you want to have more freedom? Say we want to build a database based on a REST webservice. Now imagine we don't want any info about the service in the ruby program that actually builds the db.

We are facing two major problems:

We don't know which resources are available.
We don't know the fields and types of the resource in advance.

The first problem will eventually be solved with WSDL 2.0 or if you need a solution right now: by a default listing resource.

 # Our default object is called a Resource, on the server we have 
 # a Resource object that just returns each resource we have in 
 # a string array.

 resources = Resource.find(:all)
   => ["address","person","country"]

 # Now we can do all kinds of crazy stuff :)
 resources.each do |resource|
   name = resource.capitalize.to_sym
   new_resource = Object.const_set(name, Class.new(ActiveResource::Base))
   new_resource.site = Resource.site
 end

After this little piece of code we have a full dynamic set of ActiveResource classes ready to be used :)

The second problem we face is much more interesting. Normally some kind of resource definition would be applied like XSD. But native ruby XSD support is kind of lacking (it sucks) and more importantly it doesn't feel like a rails solution.

We wanted a cleaner, simpler and more elegant (more RESTy) solution. How about Person.schema?

It returns an ActiveResourceSchema object:

Person.schema.fields
   => { :name => FixNum, :date_of_birth => DateTime, 
:parents => [{ :id => FixNum }] }

What happens is actually quite simple. When the Resource objects receives the schema method call it calls find(:first, :params => { :schema => true }).

The server responds to this param with a sample record, with just the field names and types. We build an ActiveResourceSchema object to wrap those and return a nice array of fields :)

Note: These code snippets are just examples, we are currently building this. If there's enough interest we might submit it as a plugin/patch for ARes.