In Rails, you can run rake test:units, rake test:functionals, and rake test:integration which run the tests in ./test/unit, ./test/functional, and ./test/integration, respectively. You can also run rake test, which runs all three.

But rake test runs all three one after another—I’ve always wondered why—so you pay the cost of loading the test suite (and Rails) three times.

It’s not hard to run unit, functional, and integration tests all at once (although I had a couple of false starts). Here’s a test:all task that does the trick:

require 'rbconfig'
require 'rake/testtask'

namespace :test do
  Rake::TestTask.new(:all => "test:prepare") do |t|
    t.libs << "test"
    t.test_files = %w{unit functional integration}.map { |dir| Dir.glob("test/#{dir}/*_test.rb") }.flatten
  end
end

I have a program that lists all of my company’s projects and various statistics about them. Between little APIs, background services, and large applications, we’re getting quite a collection of Rails projects. I decided to add a column to show which version of Rails each was running—a column that would be automatically updated.

It turns out, it’s pretty easy to do with grit and bundler.

require 'grit'
require 'bundler'

repo = Grit::Repo.new(PROJECT_PATH)
lockfile = repo.tree/'Gemfile.lock'
locked_gems = Bundler::LockfileParser.new(lockfile.data)
rails = locked_gems.specs.find { |spec| spec.name == "rails" }
rails.version

In the past you may have been told that the key to economic success is efficiency…[or] conformance to plan… I would suggest that both of these popular beliefs are fundamentally wrong. The key to economic success is making good economic choices with the freshest possible information. (p235, emphasis mine)

Armed with the wrong definition of economic success, we would naturally measure—and optimize—the wrong things. At best this wastes effort, at worst it invites failure. Reinertsen’s goal in The Principles of Product Development Flow is to identify flaws in the conventional wisdom and to equip the reader with modern insights into product development. In support of his arguments, he draws on fields as diverse as lean manufacturing, economic theory, queueing theory, telecommunications, operating system design, and maneuver warfare.

The Principles of Product Development Flow is organized into 9 chapters and 175 brief principles.

Making Good Economic Choices

Perhaps the single most important weakness of the current orthodoxy is its failure to correctly quantify economics. (p4)

Product developers measure things like cycle time, value-add time, and other “proxy variables”. What we don’t measure is how these variables interact. As a result, we can’t answer questions like Which is less expensive: a 1-month delay or a $5,000 increase in expense? or Is this feature worth pursuing if it will take twice as long to implement?

To answer these questions we must express metrics in a common unit, lifecycle profits. Converting everything to dollars also reveals a critical insight: the best business plan is seldom the optimization of a single proxy variable. (It’s seldom the most efficient or quickest-to-market plan.) The best plan tends to be a “u-curve optimization” that optimizes the trade-offs between single variables.

The Freshest Possible Information

To manage product development effectively, we must recognize that valuable new information is constantly arriving throughout the development cycle. (p38)

Reinertsen likens product development to warfare. There are significant unknowns; new threats and opportunities appear continually; small advantages or disadvantages can yield disproportionate results. Payoff changes constantly: endeavors become more or less costly and more or less profitable, continually altering the optimal strategy.

When the best-laid plans have short shelf lives, learning becomes a vital competence. Faster feedback, earlier validation, and shorter cycles all fuel learning and enable product developers to make good economic choices with fresh information.

Queues

Optimizing trade-offs and maximizing learning are insights basic to Reinertsen’s view of product development, but they are not the focus of his book. Reinertsen spends more time developing the powerful concept of queues.

Today we realize that inventory is the biggest source of waste in manufacturing. Work product lying idle on the factory floor destroys quality, efficiency, and cycle time. (p34)

Queues are invisible inventory. Long queues of work-in-progress destroy quality, efficiency, and cycle time. They have many detrimental effects, but because intellectual work-in-progress is difficult to see, queues often go unmeasured.

Queue size is an invaluable metric, but like any proxy variable we do best to express it in economic terms. Estimating the cost of delay for a feature or product enables us to manage queues from economic facts.

Recommendations

The Principles of Product Development Flow contains 175 principles. Many of them cluster around a few important themes:

  • Make queues visible. A Kanban can be used to show work-in-progress in real time while Continuous Flow Diagrams give many insights into queues over time.
  • Reduce batch size. Iterating quickly minimizes the impact of queues without adding resources. It also accelerates feedback and learning. Focus on reducing transaction cost—the cost of releasing a batch—to make smaller batches more affordable. Unit Testing and Continuous Integration are strategies for lowering transaction cost in software development.
  • Plan in advance for flexibility. Flexibility enables you to react to emerging queues. Reinertsen’s most frequent example of planned flexibility is cross-training employees. Hiring reserves is another.
  • Prize initiative. Those closest to a problem can react quickest—and with the freshest possible information. Empower them to seize opportunities while coordinating a team’s efforts by establishing a shared vision and setting parameters for decisions.
  • Co-locate workers. The best way to shrink transaction cost in communication and to foster a shared vision within a team is to put teammates in the same room.

Good Design

In the first chapter of The Design of Everyday Things, Donald Norman pokes fun at bad designs. He mocks smooth, beautiful doors that give no indication of how to open them and complex office phones with more functions than buttons. But users rarely see these as failures of design. People are more prone to blame themselves for being unable to figure out how poorly designed things work.

When a person uses a device, she builds a mental model of it: a conjecture, an imaginary explanation of how it works. To have success with—and to enjoy using—the device, that mental model must be consistent with the way the system really works. Design, says Norman, is “an act of communication” between designers and users. In this conversation, “all communication takes place through the system image.” The “system image” is the whole appearance and operation of a device including:

  • How it indicates what state it is in
  • How it indicates what actions are available (and which are not)
  • How it indicates the results of a potential action (before and after the action is taken)

A device that clearly communicates its state, available actions, and their consequences “disappears,” says Norman; and the user is able to “work directly on the problem at hand without having to be aware of the device.”

Tools for Design

How can a device so effectively communicate its image? Norman enumerates several tools: Visibility, Natural Mappings, Affordances, Constraints, Feedback.

Visibility
A device’s functions should be visible. Less frequently used functions may be hidden to reduce the apparent complexity of the device; but at all times, it should be apparent to the user what actions are available and how to perform them.
Natural Mappings
As much as possible, controls should bear a metaphorical relationship to the real world: they should have a “natural mapping”. (Norman draws several examples from the layouts of light switches and compares them to the positions of the lamps they control.) Any design that requires labels, diagrams, or instructions, he says, is probably faulty: unnaturally mapped.
Affordances
Affordances, like natural mappings, are clues (often visual) that suggest how a control may be manipulated. Handles on doors afford grasping and extruded buttons afford pushing. Affordances help users to discover how to evoke the actions a device has made visible.
Constraints
“Affordances suggest the range of possibilities, constraints limit the number of alternatives.” Constraints make it harder to commit errors. The mechanisms that make it difficult for you to lock your keys in your car are a kind of constraint. (In chapter four, Norman discusses four categories of constraints: physical, semantic, cultural, and logical.)
Feedback
There are two aspects of action: execution and evaluation. After executing an action, we watch for its effects, see that it accomplished what we expected, and adjust our expectations or intentions. Rich feedback about actions gives users confidence about a device and emboldens them to explore it.

The Process of Design

Much of The Design of Everyday Things is dedicated to the tools listed above; but its last two chapters reflect on the difficulties of the design process.

The gap between designer and user makes it just as difficult for the designer to understand the user as for the user to understand the device. Norman widens that gap with two observations:

  1. First, there is a difference in how designers and users approach a device: “designers often become experts with the device itself,” whereas “users are experts at the task they are trying to perform with the device.”
  2. A second difficulty is that “most designers…are separated from the end users by multiple layers of corporate bureaucracy, marketing, customer services, etc. These people believe they know what customers want and feedback from the real world is limited by the filters they impose.”

How can the gap be managed? To some extent it may be closed. “Much good design evolves.” Wherever possible, design and release in smaller increments. When design iterations and user feedback happen in smaller cycles, the conversation between designer and user approaches real-time.

For avoiding pitfalls in the gap between designers and users, Norman offers a few bits of advice:

  • Problems dominate users’ perspective; design for the problem case. Norman devotes an entire—excellent—chapter to thinking about errors.
  • Resist the temptation to design primarily for aesthetics. The primary goal of a device is rarely to “wow” an audience, to win an award, or to please a client. Keep its purpose in terms of its users in focus.
  • Fight creeping featurism. “Feature creep” has become a familiar byword, but Norman offers an insightful solution to it: modularize. This is similar to the single purpose rule in programming: each function should do just one thing and each class should represent a single concept. Higher up the software model, the best applications solve a single problem—and cooperate with applications that solve related problems.

Handlebars for Ruby?

For a current project I needed to choose a templating language. For this project, “logic-less” is best; and I love the syntax of Handlebars.js. But its templates need to be rendered on the server.

Charles Lowell wrote a Ruby library that runs Handlebars. Handlebars.rb is a tiny library that binds Handlebars.js to Ruby. The heavy lifting is done by Lowell’s Ruby Racer which runs on V8 and magic.

Is Handlebars.rb ready for production? Would it be buggy or dreadfully slow?

Handlebars vs Liquid vs ERB

Here are the results of a benchmark:

  PostWithHelpers Benchmark:
                                  user     system      total        real
  erb:                        0.340000   0.000000   0.340000 (  0.340879)
  liquid:                     0.820000   0.000000   0.820000 (  0.823544)
  handlebars:                 0.590000   0.020000   0.610000 (  0.600626)
  liquid (precompiled):       0.500000   0.000000   0.500000 (  0.499670)
  handlebars (precompiled):   0.120000   0.000000   0.120000 (  0.117546)
  
  
  SimplePost Benchmark:
                                  user     system      total        real
  erb:                        0.250000   0.000000   0.250000 (  0.254124)
  liquid:                     0.690000   0.000000   0.690000 (  0.687987)
  handlebars:                 0.480000   0.000000   0.480000 (  0.492988)
  liquid (precompiled):       0.470000   0.000000   0.470000 (  0.464284)
  handlebars (precompiled):   0.090000   0.000000   0.090000 (  0.096169)

Each templating engine rendered each test 1200 times. The SimplePost test renders a very simple template that creates an ordered list of posts. The PostWithHelpers test tests registering and utilizing a helper with each engine.

Verifying my results

I published my testing framework to GitHub (boblail/liquid_handlebars). To run the tests simply do:

  git clone git://github.com/boblail/liquid_handlebars.git
  cd liquid_handlebars
  bundle install
  rake test

Notes

I updated the version of Handlebars.js that Handlebars.rb was referencing so that I could take advantage of performance improvements and the new Handlebars.registerHelper API (boblail/handlebars.rb).

There is a problem with how The Ruby Racer executes function calls: Handlebars.compile returns a function that is bound to an internally used “class”, JavaScriptGenerator. To call this function from Ruby, you use the call method of V8::Function; but this method binds the template function to the global namespace. As a result, it doesn’t have access to helpers registered with Handlebars.registerHelper.

I worked around this problem with a hack:

  json = MultiJson.encode(data)
  @context['template'] = @template
  @context.eval "template(#{json})"

What composed_of is for

(Skip to the problem)

Chances are you’ve needed to serialize a complex value to a database—a mailing address, for example. An address is composed of several strings, but it is taken all together as a single value.

Active Record’s composed_of macro allows you to control how such a value is serialized to your database. Your database could have fields named address_street, address_city, address_state, and address_zip, for example; but composed_of allows ActiveRecord to load them as an instance of Address into an attribute named address. So you can write

household = Household.new
household.address = Address.new(:street => '1600 Pennsylvania Avenue NW', :city => 'Washington', :state => 'DC', :zip => '20500')

rather than

household = Household.new
household.address_street = '1600 Pennsylvania Avenue NW'
household.address_city = 'Washington'
household.address_state = 'DC'
household.address_zip = '20500'

And, of course, now you can extract your address-specific logic into a reusable class. (Hooray!)

One bit of logic that seems natural to add to our Address class is validation. ActiveModel was designed for just this purpose. We can add ActiveRecord-like validations with a single include:

class Address
  include ActiveModel::Validations
  
  validates :street, :presence => true
  validates :city,   :presence => true
  validates :state,  :presence => true
  validates :zip,    :presence => true, :format => {:with => /\A\d{5}(-\d{4})?\Z/}

And then you can validate an Address along with its Household like this:

validates_associated :address

And everything will look shiny until you ran your tests.

Why composed_of is broken

If you run:

household = Household.new
household.address = Address.new(:street => '1600 Pennsylvania Avenue NW', :city => 'Washington', :state => 'DC', :zip => '20500')
household.valid?

you get TypeError: can't modify frozen object because ActiveRecord is freezing your Address as soon as it is assigned. Here’s a test that demonstrates this:

household = Household.new
address = Address.new(:street => '1600 Pennsylvania Avenue NW', :city => 'Washington', :state => 'DC', :zip => '20500')
address.frozen? # returns false
household.address = address
address.frozen? # returns true

valid? throws the exception because it tries to create two new instance variables in the frozen object. One is @validation_context; the other is @errors, which is instantiated by valid? if it doesn’t already exist.

The following isn’t a pretty work-around; but it is better than monkey-patching Rails or installing evil_ruby to unfreeze objects during validation.

The work-around

  1. Override validation_context to disable that feature.

    def validation_context=(value); end

  2. Make sure that @errors is instantiated before the object is assigned to an ActiveRecord attribute.

    def initialize(*args)
  errors
  # other initialization...
end

Lighthouse ticket 5646 describes this problem.

After far too much time, this worked for me:

  1. Download libiconv and install using: 
    ./configure
make
sudo make install
  2. Download glib2 and install using: 
    CPPFLAGS="-I/usr/local/include" LDFLAGS="-L/usr/local/lib -lintl" \
./configure --with-libiconv=gnu
make
sudo make install
  3. Make sure pkg-config can find out about glib by typing: 
    export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
  4. Download mdb-tools 0.5 (I encoutered several bugs with 6.1pre) and install using: 
    ./configure --disable-shared
    make
    sudo make install

Configuring Rails Applications

This week I was configuring an application to run different instances depending on the subdomain requested. This is really quite easy: here’s the basic theory DHH used in Basecamp: http://gist.github.com/45076.

Easy enough. But, now, I want to use this only in some environments. In development I want http://localhost:3000 to just be the development instance (@current_account = Account.first).

I’d love to solve this by adding something like the following to config/environments/development.rb

config.get_instance_from_subdomain = false

Googling, I found a lot of advice about creating custom configuration systems and several plugins that would do it for me. But, IMHO, all of these involved writing too much software. We’ve already got Rails::Configuration, why reinvent the wheel? In my case, at least, there was no need.

Extending the Wheel

Since it’s quite easy in Ruby to extend classes, I added this block to config/environment.rb:

Rails::Initializer.run do |config|
 
  # extend Rails::Configuration
  config.class.class_eval do
   
    def get_instance_from_subdomain
      @get_instance_from_subdomain ||= true
    end
    def get_instance_from_subdomain=(value)
      @get_instance_from_subdomain = value
    end
  
  end
  
  # ...

end

Voila!

Rails::OrderedOptions

But it wouldn’t be a lot of fun to have to define accessors for every property you need to configure if you have several of them. It would be nicer to add configuration options on the fly – as easily as adding values to a hash. Turns out, the Rails team didn’t want to waste keystrokes writing accessors either. Groups of configuration options (like config.action_mailer) are actually instances of Rails::OrderedOptions – an object that lets you add properties at run-time. You can create your own groups of flexible configuration options like this:

Rails::Initializer.run do |config|
 
  # extend Rails::Configuration
  config.class.class_eval do
   
    def application_mailer
      @application_mailer ||= Rails::OrderedOptions.new
    end
  
  end
  
  # ...

end

Now you can define namespaced configuration just by assigning values to properties of config.application_mailer:

config.application_mailer.property1 = false
config.application_mailer.property2 = :awesome

Lots of Javascript

In more than one of my projects, I have javascript or stylesheet files with several hundred lines. Also, I rotate between several active projects and forget exactly where I put things. :( What I want to do is break these files into smaller ones by responsibility but be able to concatenate them for production to reduce requests.

Rails has an excellent way of accomplishing the concatenation: the :cache option that works for both javascript_include_tag and stylesheet_link_tag. With it, you can write:

javascript_include_tag “effects”, “controls”, “builder”, “scriptaculous”, :cache => “scriptaculous_cached”

Rails will even let you shorten this by using register_javascript_expansion so that you can write:

javascript_include_tag :scriptaculous, :cache => “scriptaculous_cached”

That’s pretty awesome; but not as awesome as

helper :all

Automagic

I love how the macro for ActionController scours my helpers directory and includes whatever I place there. If I add a helper or refactor my helpers, I don’t have to remember update a list of helpers that lives somewhere else. That’s what I’d like to see from javascript_include_tag. So here’s what I came up with:


def include_javascript_dir(path)
  javascripts = Dir.glob_in(File.join(RAILS_ROOT, “public/javascripts”, path), “*.js”)
  javascripts = javascripts.map{|file| File.join(“/javascripts”, path, file)}
  javascript_include_tag *(javascripts + [{:cache => “#{path}/all”}])
end

(NB: The above method uses a customization of Ruby’s Dir class, glob_in which executes glob in a relative path.)

This method includes every file with the extension .js in a given directory. Thanks to Rails, they’re each included separately in the development environment but are concatenated into the file all.js in production. Sweet!

Order

We’re not done yet, though. If I move all of the Scriptaculous files to /public/javascripts/scriptaculous and then put include_javascript_dir "scriptaculous" in my layouts, the files will get loaded: but there’s no guarantee that “effects” will be loaded before “controls” and JavaScript errors will be thrown.

We need to supply more information to include_javascript_dir; but I don’t want to have to spell out the order the files should be in or there won’t be any point to having this method discover them automatically. Here’s what I settled on:

((order & javascripts) + (javascripts - order))

Here order is a list of script file names in the order you care about and javascripts is a list of real javascript files that Ruby has discovered. The intersection of order and javascripts yields files-that-really-exist in the order you care about; and javascripts - order gives you all the rest of the real files.

I added another parameter to let you specify the relative path to your javascript files. (One of my projects uses “/public/js” and another “/public/javascripts”) Here’s what I’m using now:


def include_javascript_dir( path, options={} )
  js_path = options[:path] || “javascripts”
  javascripts = Dir.glob_in(File.join(RAILS_ROOT, “public”, js_path, path), “*.js”)
  if (order=options[:order])
    order = order.map{|s| “#{s}.js”}
    javascripts = ((order & javascripts) + (javascripts - order))
  end
  javascripts = javascripts.map{|file| File.join(“/#{js_path}”, path, file)}
  javascript_include_tag *(javascripts + [{:cache => “#{path}/concat”}])
end

I’ve wrapped it up with the helper methods I use in all my Rails applications: ttp://github.com/boblail/lail_extensions

One of the pieces of code I find myself writing over and over again is a banner for flash messages.

<div class="error" id="error">
  <%=h flash[:error] %>
</div>
<div class="notice" id="notice">
  <%=h flash[:notice] %>
</div>

That already is six lines of code that I will never care about again. And, to date, I’ve written then 5 times. They get worse, though, if I want those divs to be hidden when there is no message to display. I could wrap them like this:

<% if flash[:notice] %>
  ...
<% end %>

And if there is no message, the block will be absent from the page. But inevitably, I want to show those messages on an XHR, so my layouts get a little more complex:

<div class="error" id="error" <%= "style=\"display:none;\"" if flash[:error] %>

This is a perfect candidate for refactoring. My final product looks like this:

def flash_message(key, options={})
  message = flash[key] || ""
  options.reverse_merge!(:class => "flash #{key}", :id => "flash_#{key}")
  options.merge!(:style => "display:none;") if message.empty?
  content_tag :div, message, options
end

It replaces the six lines above with these two:

<%= flash_message :notice %>
<%= flash_message :error %>

I’ve tucked it into a plugin of other private conventions and will drop it into every new Rails app I write.