Local Reactivity in Meteor Web Applications

Every once in a while I stumble into something that changes the universe for me, allowing me rethink how I approach problems and develop solutions. Stumble because, while I can't necessarily plan on things like this happening, if I try to listen to people and keep my eyes open, I figure I'll eventually get lucky. When Meteor buzzed in a Cincinnati Javascript User Group meeting this Spring, I was happily surprised. And now just a few months later, it's become my new go-to web application framework.

Rather than try to introduce you to Meteor myself, I'll simply defer to the meteor web site www.meteor.com and let them do the talking. The site includes a high-level description of the framework, detailed documentation, example code and links to videos. The quality of the description and documentation alone merit your attention - and the capability of Meteor deserves your respect. This stuff is brilliant.

Of all the things Meteor brings to the table, Reactivity is the game changer for me. Updates to data sources cause dependent computations to be re-run, and the application shows updates immediately. The burden of carefully choreographing a dance between the server and the client to display pleasing changes in a timely way has been lifted. The dance is already beautiful and the performers are well-rehearsed. I can focus on building great user experience rather than on things like scratching my head over problems with data structures or working the weird bugs out of ajax callbacks.

Recently, when developing a web app I found myself wanting to use this strategy locally as well, building reactivity to be used only within a client. Though Meteor gives you session-level reactivity that propagates changes made to session variables locally, this wasn't quite what I wanted. Session variables are global to the client; I wanted variables that were at least scoped so I could keep them more closely associated with the functions and objects that use them. I've been conditioned to believe that global variables smell bad and I'm not ready to give up that sense yet.

Instead, I opted to adapt a notion that was mentioned in the Meteor documentation. I associated a variable and a dependency in an object and called it an injective. The idea is that I can inject client values into the reactive flow. In Coffeescript,

    Deps.injective = (init, options) ->
        _value: init ? 0
        _dep: new Deps.Dependency
        _force: !!(options && options.force) ? false
        set: (value) ->
            if (@_value != value) || @_force
                @_value = value
                @changed()
            @
        get: ->
            @depend()
            @_value
        depend: ->
            @_dep.depend()
            @
        changed: ->
            @_dep.changed()
            @
        force: (f) ->
            @_force = !!f
            @

The internal _value and _dep are managed through the get and set functions. Calling get returns the value while making the function asking for it dependent on the injective. Updating a value using set re-computes all functions that depend upon it.

A quick example perhaps, again in Coffeescript.

    App.innerWidth = Deps.injective window.innerWidth
    $(window).resize -> App.innerWidth.set window.innerWidth

This code creates an injective that tracks the inner width of the browser window. When the window resizes, the injective's value is set to the new inner width. While this might not sound like much, anything that has been computed based on the getting the value of innerwidth will be automatically recomputed when it changes. In the context of this example, that means that you don't have to know about the 23 layout decisions you made based on this value and how they'll need to propagate, and then manage it all yourself. Instead, you just grab the corner of the window and resize it, and watch the magic happen.

The depend and changed functions provide access to the dependency mechanism and are used by get and set. When depend is called, the injective adds the calling function to its list of dependents. An update to the injective using set will re-run these dependent functions. The changed function is what triggers the dependents to be re-run.

Finally, force is a way to force changes to objects and arrays values to be propagated without having to write any special comparitors. When the value being set is a scalar, == works: if the value is different it is copied and changed is called. If the value is an object (or an array) and the change is made within the object itself, == won't report the object has changed; a more complex comparison must be written to do the test. But that really isn't needed if you know about the complexity already. The force function (or constructor argument) provides a simple shortcut: if _force has been set to true then the set function trusts that the incoming value is different; set just goes ahead and updates the value and propagates the change. If something more complex is needed, values in the object can be adjusted and the injective's changed function can be called directly.

I've submitted injective (a Javascript version) to the meteor github repository. Maybe they'll integrate it, maybe not; I wasn't fixing a bug, just adding an enhancement so it doesn't have any sort of priority. However, it's a clean way to inject values into the reactivity stream that's useful for me. If it looks like it may be useful for you too, then enjoy!

Simple Drone Flying Algorithms

The AR Parrot Drone seems to be all the rage these days, especially here at the Neo Cincinnati office. Two officemates have drones, and they've taken to the office space and the hallways during lunchtime. The drone comes with piloting apps that control it in real time - but they can also be controlled programmatically. Work on drone drivers is progressing (one in Ruby, the other in Clojure) that can send commands to the drone.

A drome command is specified as a 5-tuple: (roll, pitch, yaw, altitude, duration) all relative to it's hover state. Once a command is complete, the drone returns to a hover state at its current location. From this it's relatively straightforward to make a drone fly in a simple pattern. Tilt and pitch define the forward direction, and yaw is used to change it.

To actually plan a pattern however, the command's values must be understood in relation to each other. Each is value specified in [-1, 1] in a drone command, and translated to an angle. Before programmed maneuvers can be created, scaling constants between angles must be determined. For example, consider flying one turn around a circle: the drone must be tilted forward (some combination of pitch and roll) and spun (a yaw value) over the time needed to complete a revolution.

(r_c, p_c, y_c, 0, t_c)

For a given tilt, spinning too fast spirals inward, while spinning too slow spirals outward. Too long or short a time will orbit more or less than one turn. Empirical studies of drone flight and timing must be done to determine these values. Assuming these values have been appropriately factored in then,

figure	roll	pitch	yaw	altitude	duration	notes
Circle	r	p	y	0	t	p&r = y
Spiral in	r	p	y	0	t	p&r < y
Spiral out	r	p	y	0	t	p&r > y
Figure 8	r	p	y	0	t	p&r = y t = one revolution
	r	p	-y	0	t
Tilted circle	r	p	y	a	t	p&r = y t = half revolution
	r	p	y	-a	t
3-leafed Clover	r	p	y1	0	t1	p&r = y1 t1 = two-thirds revolution t2 = half a revolution
	0	0	y2	0	t2
	r	p	y1	0	t1
	0	0	y2	0	t2
	r	p	y1	0	t1

Compound curves are defined as a series of simple pieces.

This is all fine, but to watch a drone move around a room in these sorts of algebraic patterns doesn't feel very aerobatic to me. The flying seems stunted and formulaic... more natural flight is a more complex dance of values. My best reference is a bird flying or fish swimming - when they maneuver, combinations of roll, pitch and yaw are smooth and complex. Changes in roll, pitch and yaw affect altitude. Coordinated moves are made to convert velocity into height. Soaring, diving, swooping - all feel like they should be aerobatic primitives.

Of course, these all will still be formulaic. But they'll look more natural. Somewhere in there is the beginning of the algebra of nature and what constitutes enough of this essence to feel birdish or fishish.

I'll be looking at this as our drones continue to learn to fly.

Laissez les bons temps rouler!

Starting on the day before Mardi Gras, I will be joining EdgeCase working in their Cincinnati office. To me, a much bigger party than they'll be having in the French Quarter!

This is a phenomenal opportunity, working alongside the genius of Jim Weirich and Scott Barron every day, and collocated with the Gaslight Software crew. (Let me tell you, this is a really smart room.)

Many thanks to Joe, Ken, Adam and Leon for taking the time to meet with me and make sure I'd fit there, and to everyone who's supporting me in this transition. To be sure, I'll miss the folks I've been working with, but I'm ready to get cracking on new stuff!

Prototype and Witch's Brew

When things work differently in one part of a package than they do in another - for no apparently good reason - programming indigestion may occur. When you beat your head against the code for too long trying to figure out why things are screwing up, big headaches are sure to come. The cause of my current indigestion and headache? Element.insert in prototype.

I have some javascript code that renders a string containing some HTML to be added to the bottom of the contents of a some target element. I do the necessary incantation,

Element.insert('target',htmlString)

Which quite promptly does nothing. WTF?

It turns out that the htmlString I'm sending either needs a javascript toElement method defined that will convert it into a DOM object or it must be a DOM object already. The first alternative is unpalatable - I just want to create html, not a tree of DOM cruft. The second is just as bad, unless I can get the work done for me. Fortunately, I can.

In the body of my document, I declare a special, invisible div,

<div id="_cauldron_" style="display:none;"/>

and I use it to magically transform my html string into an object that I can insert into the target,

Element.update("_cauldron_",htmlString)
Element.insert(target,$("_cauldron_").firstChild)

The cauldron is where the html must brew to make the magic happen.

You may be asking, "Why does the conversion happen differently in the update? Why doesn't the insert work the same way?" Good question. I have no good answer though. To my way of thinking, The conversion should happen exactly the same way - otherwise kludges like this are forced. Suffice to say that it's all a moving target and everything's always changing. I'm sure this one will get fixed in a future Prototype release (or maybe it is already, my version is not the latest) but it does get frustrating.

Though the indigestion lingers, at least the headache is a little better now.

A little Hash goodness

Some quick refactoring this weekend had me throwing together some tidbits.

module Enumerable

  # return Hash of enumeration to yielded values
  def collect_hash hash={}
    inject(hash) {|h,e| h[e] = block_given? ? yield(e) : nil; h }
  end

  # return Hash of enumeration to non-nil yielded values
  def select_hash hash={}, &block
    collect_hash(hash,&block).compact
  end

end


class Hash

  # return Hash with nil values removed
  def compact
    delete_if {|k,v| !v }
  end
  
  # array-style push of key-values
  def <<(hash={})
    merge! hash
  end

end

Yes, I know they've been done before, but they're quick one-liners and let me eliminate a lot of code. And as we all know, the easy code to maintain is the code you delete.

The Missing Link in RJS Chaining

One of the things that make Ruby and Rails my respective current language and web framework of choice is the collective ingenuity that has made building things easy. These folks have thought of everything, gosh darn it, and I get to use it. A good day is when I get something working, and with Ruby and Rails I have a lot of those days.

Imagine my dismay however, when on a recent project I had the need for something that wasn't already baked into Rails! The problem occurred when chaining partials together - one partial calling another, calling another, and so on down the line. I find myself doing it a lot these days, updating many parts on a page based upon seemingly simple model changes. Chaining partials together is easy, you just set up some context and render, and repeat.

Using Ajax and RJS is now a way of life; I got my baptism in the Javascript waters a long while back, a must for doing esoteric, dynamic rendering. RJS certainly isn't what makes such rendering possible, but it sure makes it easier. Some may think that there are better mechanisms - and maybe in special situations there are - but the simplicity and versatility of generating pieces of web pages using RJS is unparalleled once you understand the subtleties of building Javascript that will be executed in the context of a page.

Context is the issue. Let us say my foo partial renders my bar partial. The context is established as a hash of variables to values that is built by foo and passed into bar as the value of the :locals key in the render call. When bar is rendered, the hash is unraveled behind the scenes and the variables become available.

All except one, that is: the meta-variable that is the locals hash itself!

Unless I'm missing something, when you want foo to pass whatever locals it had received to bar, Rails comes up short! There's no way that foo can know all the variables that got passed in because this information is lost. The hash has been absorbed and is no longer part of the context! The variables in the locals hash that were set up by foo have been unpacked and established as the context of bar, but the containment itself is no longer available.

An argument might be made that making the locals hash available would be wrong; if the code knew it had the locals context, partial-coupling might be considered to be too tight. Perhaps so, but I'd say this really isn't true - assuming context in the form of injected variables is really at the same level of coupling. After all, what Rails is doing for me is just saving me some coding by allowing me to access a variable directly instead of as locals[:variable] if the incoming locals hash were exposed. The way I see it is that permitting a partial to access the incoming locals hash is actually promotes less coupling. A partial could just pass, augment and pass, or create and pass a new hash to the partials it calls - not necessarily knowing to what use the information in the hash might be put, or which partial down the line might be using it.

Chaining is about delegating responsibility - entities at intermediate levels really shouldn't have to be concerned with the details of what's happening above and below them. They should be able to pass information along with the understanding that if they don't need it, that doesn't mean that something further along the chain won't. If there's any doubt about this perspective, consider that partials are simply about rendering a page - the page is the real context and many complex renderings of application state changes may be made upon receipt of a seemingly simple event. That's what RJS is really good for - making multiple coordinated changes to a page. Allowing information to be passed down from on high so chained partials can use it according to need in the context of rendering different parts of a page is exactly why RJS is so sweet. Forcing intermediate partials to know what their subordinates need makes them difficult to write, hard to test, and downright painful to change.

Of course, just because this isn't automatic doesn't mean I can't do it myself. Passing foo a reference to the :locals hash as the value of the :local_assigns variable in the :locals hash it receives itself, for instance, allows me to pass it as the :locals hash to bar. Not a great solution, and it smells a little. Yes, it does mean I have to do some extra work, something Rails has helped me to otherwise avoid, but I can put up with this. Sigh. I sure wish the locals hash was already exposed - it is the missing link for chaining.

Bottom line: not having access to the incoming locals hash in a partial is a bummer, but it isn't the end of the world.

Why Can't We All Just Get Along?

While building a browser-based graphical editor in Rails for a client, I ran across an cross-technology compatibility problem. The reason was not immediately apparent and caused me quite a bit of head-scratching. After sleeping on it for a night, I finally figured it out and found a workaround.

In the graphical editing app that I'm building, one of the things a user does is create selections of items. When selecting, shift-clicking on an item toggles it in or out of a selection. I use SVG to represent the items being edited in the browser and manipulate them using Javascript.

Building composite graphical elements is made easier by using the SVG use element since this can encapsulate each part, setting up display and event handling for each in a compartmentalized way, and the code for representing the parts can be shared across items. An item may reference many use elements in its representation. These elements are then added to the document and displayed automagically in the browser. Javascript functions handle the events targeted at the items.

Everything's good so far. Simple SVG elements can be manipulated easily by the Javascript, and AJAX calls are sent to the controller to update the model. Shift-clicks toggle items in and out of the selection as expected. But shift-clicking an element that includes a use element in its definition (in my case, a path element is being referenced by another element) causes a new graphical editing window to be opened in addition to the toggling.

What the heck?! Debugging the Javascript in Firebug shows that all of the expected things are happening correctly and the code is running cleanly. Yet as soon as the code finishes, up pops a new window. I'm certainly not firing a request to do this. What's going on?

It turns out that Firefox implements shift-click-on-a-link functionality that will open a linked reference in a new tab or window. This is extremely useful - I do it all the time when I'm browsing. But it didn't occur to me that this behaviour transcends normal event handling! I need different behavior in my editor!

It turns out that the way that SVG support is built in, shift-clicking on an SVG element that's drawn with the use element is misinterpretted by Firefox as a request by the user to follow the link embedded in the use element. But this reference is meant to be invisible - SVG uses these reference to display graphics, not to expose clickable links.

This wouldn't be so bad if there were a way to tell the browser that the event has been handled and not to do the default shift-click-on-a-link functionality. Normally such things are done by halting the propagation of events. Alas, for the shift-click, this has no effect. It appears that it is not possible to stop this behavior through Javascript. There may be a way, but I couldn't find it.

It's not a bug in the browser. It's not a bug in SVG. It's a bug that emerges when the two technologies come together. If the browser didn't hide access to its shift-click behavior, all would be well. Or if SVG didn't use the html linkage mechanism to do element referencing, all would also be well. But shift-click is inescapable in the browser and the element referencing is as it is in SVG. The bug is in the space where they overlap.

The workaround is not to use the use element in SVG if such elements can be shift-clicked. This causes the size of scene representations to balloon when the items are complex, but at least they can respond appropriately to events. The fix is to make Firefox (and whatever other browsers exhibit this problem) understand the difference between internal and external referencing via linkage in SVG.

Emergent bugs like these are insidious. It is the clash between multiple context-free systems that occupy the same technological space that lead to such problems. I predict they will appear more frequently as we move forward and continue building embeddable software. The best we can do for now is to fix or workaround the problems as they are found as I have here.

The bottom line is that we must recognize that in the areas where context-free systems that are used together overlap there will be ambiguity. The composite behavior of these systems must be considered and dealt with appropriately, either through code or policy. It's not clear that we'll ever be able to find all of these overlaps prior to encountering the problems they cause; I just hope the scenarios in which the problems occur aren't too dangerous.

Users, Roles, Rights and Sights

Chad Fowler's Rails Recipes book lays out authorization as the interrogation of the many-to-many connections between users and roles, and between roles and rights, a right being a named controller-action pair. The many-to-many relationships are established using roles_users and rights_roles tables in the database.

This indirection makes bulk assignment of rights easy, simply by assigning roles to a user, and authorizing using nested detection:

class ApplicationController
  before_filter :check_authorization

def check_authorization
    unless @operator.roles.detect{|role|
      role.rights.detect{|right|
        right.action == action_name &&
        right.controller == self.class.controller_path } }
      flash[:notice] =
        "You are not authorized to view the requested page"
      request.env["HTTP_REFERER"] ?
        (redirect_to :back) : (redirect_to home_url)
    end
  end
end

Here, @operator is the user that is operating the application.

This works fine, but the interrogation of rights seems a little too removed from the user. I prefer asking if a user has a particular right directly:

class User
  def has_right?(controller,action)
    rights =
      User.find_by_sql [
        "select * FROM rights ri, rights_roles rr, roles_users ru where"+
        " ru.user_id = ? and ri.controller = ? and ri.action = ?"+
        " and rr.role_id = ru.role_id and ri.id = rr.right_id",
        id, controller, action ]
    rights.size > 0
  end
end

I also took the liberty of dropping the in-Ruby detection since a single sql query is faster than the multiple smaller queries that detection requires (look at the log files - one query for each role.)

So this is a bit faster and is a drop-in replacement for the detection in the original code:

class ApplicationController
  def check_authorization
    unless @operator.has_right?(self.class.controller_path,action_name)
      flash[:notice] =
        "You are not authorized to view the requested page"
      request.env["HTTP_REFERER"] ?
        (redirect_to :back) : (redirect_to home_url)
    end
  end
end

Besides simplifying the authorization, since has_right? is a method of a User, any user's rights can be simply interrogated, which useful in setting up rights administration for an application.

Formulating the right-checking in this way also leads to the notion of sight-checking, that is checking if a user has the right to see something. For instance, when building out a page if there is a question as to whether or not a user is allowed to see something, a sight can be established that allows it to be seen. The absence of a requested sight in the rights table for that user's roles implies that the user should not see the sight.

Sights don't necessarily depend on controller-actions; typically they're just checking to see that the user has a right with a particular name. The code to check sights is a simple as that of checking rights:

class User
  def has_sight?(name)
    sights =
      User.find_by_sql [
        "select * FROM rights ri, rights_roles rr, roles_users ru where"+
        " ru.user_id = ? and ri.name = ?"+
        " and rr.role_id = ru.role_id and ri.id = rr.right_id",
        id, name ]
    sights.size > 0
  end
end

Checking for sights looks for a Right's name, while checking for rights looks for a Right's controller and action. Moving the interrogation of rights to the user and adding the notion of sights allows pages to be constructed more simply based on what a particular user is allowed to do.

Enhancements to Railroad

A picture is worth a thousand words, and is certainly easier to talk about than reading a bunch of code. At least for me.

On a recent Rails project two dozen models and sixty associations were needed to drive the application. A medium sized effort. However, it's tough to talk users through the complexity when the need arises, even when you're just dealing with a small chunk. After drawing circles and arrows on whiteboards too many times, I decided to mechanize.

I'd used Railroad (a great rubygem written by Javier Smaldone) a few times before to document smaller projects. It loads models and controllers from a rails project and renders everything in 'dot' format that can be processed by graphviz, an open source graph rendering framework. I railroaded the app and quickly had diagrams in hand. Happiness all around.

But there were a few things I noticed as I talked around the pictures. I needed colors. And labels. And fewer circles and arrows. I decided to take a dive into the railroad code and add some features.

What I eventually ended up with was a considerable set of changes.

Changes to Allow Subgraphing

Often I didn't want to see the whole graph at once. What I wanted was the ability to include only the models I listed, or exclude a set of models from the whole. After implementing this, I decided I also wanted the ability to focus on a set of nodes, including them and any other directly-connected nodes.

	I added support to only include model classes in a diagram by name specifying -I class1[,classN] on the command line.
	I added support to exclude model classes from a diagram by name using -E class1[,classN].
	I added the capability to focus on a set of model classes. By specifying -F class1[,classN] on the command line, only these nodes, the nodes they connect to, and the associations between them are displayed. However, this is also subject to any additional model class exclusions (-E) or inclusions (-I) that are specified.

By using -F, -E and -I in combination, a logical subset of the model can be displayed fairly easily.

Model Node Content Display Changes

I found that I sometimes needed more or less information in a displayed node. Brief mode was already present (for just displaying the class name in a node) but I wanted a little more content control.

	I changed the display of magic fields in model nodes to be off by default. Before, you'd hide the magic fields by specifying --hide-magic on the command line. I added a --show-magic option to turn them on instead.
	I changed the display of association fields in the node to be off by default, shown only by specifying --show-assoc-fields on the command line, and label them with the name of the associated model.
	I added a -B option to display all nodes as brief except focused nodes (those specified with -F).

Using -B with -F turned out to be a great way to present information. I could see what was being focused upon in context, without extraneous detail.

Changes to Association Display

I found I needed to be able to show different associations differently. Conventional Rails associations were fine, left black and unlabeled. But polymorphic associations, many-to-many through relationships, and unconventionally named associations needed different labels and colors. By default, I decided to always display these and provide methods to hide them.

	Unconventional associations and their labels are blue; navy blue if only one side of the relationship is unconventional. The labels are hidden by specifying --hide-uaslab on the command line.
	Polymorphic associations and labels are red. The labels are hidden with --hide-paslab.
	Through associations and labels are dark green. The labels are hidden with --hide-taslab.
	For convenience, all association labels may be hidden with --hide-aslab.

Multiple Diagrams at Once

Once the rest was done and I'd used it all for a while, I decided that automatically generating a focused diagram for each model would save me a lot of time.

I added support to create a focused diagram for each model class in its own dot file using -O FILE. The output file name for each model is created as FILE.dot, and FILE may include directory separators.

A simple rake task completed the automation:

@railroad_command = Config::CONFIG["target_vendor"] == 'pc' ?
  'railroad.bat' : 'railroad'

task :graphs do
  FileUtils.mkdir_p 'graphs'
  `#{@railroad_command} -M -b -o graphs/_overview.dot`
  `#{@railroad_command} -M -B -O graphs/`
  FileUtils.cd 'graphs' do
    FileList['*.dot'].each do |f|
      `dot -Tpng #{f} -o #{f.gsub(/dot$/,"png")}`
    end
  end
end

Finally, I refactored the completed code, DRYing out the iterative changes I'd made.

What I now have is a nice, simple way to produce diagrams of the database and associations between tables for discussion and documentation. I submitted the patch to Javier and hopefully it will be integrated into the railroad trunk fairly soon. You can grab railroad-0.5.0 and the patch and play with it if you want by downloading it from rubyforge.

Installing Rails Plugins from Local Sources

I'm usually reluctant to go change "big things", almost always prefering to work around any issues I encounter. On the rare occaision though, I find I really must make changes or I'd forget what I did and struggle with it again later.

The script/plugin install command in Rails is a wonderful thing. It fetches code from repositories (like subversion or github) and installs it into your Rails application. Plugins generally extend Rails' capabilities, much in the way that Ruby is extended through Rubygems. However, the plugin installer assumes a repository, or at least the web. If there is a local copy of the plugin in your environment that you'd like to use as the source, you're out of luck. You have to do the install by hand, and remember to set up some environment specification. There's just no provision for local installs.

Necessity being the mother of invention, when I found myself needing to install a plugin from local sources, I tried to find a workaround. When I could find nothing satisfying, I did some editing. A very slight amount of editing. In ruby/lib/ruby/gems/1.8/gems/rails-2.1.0/lib/commands/plugin.rb

class Plugin

  def file_url?
    @uri =~ /^\//
  end

  def install(method=nil, options = {})
    method ||= rails_env.best_install_method?
    if :http == method
      method = :export if svn_url?
      method = :clone  if git_url?
      method = :file   if file_url?
    end

    uninstall if installed? and options[:force]

    unless installed?
      send("install_using_#{method}", options)
      run_install_hook
    else
      puts "already installed: #{name} (#{uri}).  pass --force to reinstall"
    end
  end

  private

    def install_using_file(options = {})
      root = rails_env.root
      mkdir_p "#{root}/vendor/plugins"
      Dir.chdir "#{root}/vendor/plugins" do
        cp_r @uri, @name
      end
    end

end

That is, if the specified plugin being installed starts with a slash, assume it's an absolute path to the directory that contains the plugin and fetch it by copying. Then the install can go along it's merry way.

Now I can just fire off a

   ruby script/plugin install /Downloads/rails/plugins/foo

and it will install the foo plugin in the specified directory into my Rails project, just as if it were out on the net.

This certainly may not be perfect, and if I were polishing it I'd add a check for a "file:" protocol and include relative paths, but this was just enough for my needs.

Running Fast and the Demise of Dynamic Scaffolding in Rails

Sometimes things just get away from you. You get distracted, turn your head, and then when you look back everything's changed.

"In this place it takes all the running you can do, to keep in the same place." - Red Queen to Alice in Lewis Carroll's Through the Looking Glass

Admittedly I haven't been on the Rails edge for a while. My day job keeps me in Java, and my own work has been more focused on interfacing with FXRuby than through a browser. When I did use Rails, I used my older version and got along just fine.

But over the last few weeks I decided to pull in the latest and greatest. A few minutes of Rails 2.1 Gem downloading, another minute to create an app framework, lay in a db design as a migration, create a controller and add the requisite scaffold call into the code, and give it a try...

Dynamic scaffolding is no longer part of Rails. It was removed because it was considered detrimental - since scaffolding is intended to be temporary, it should be able to be easily removed. If the guts of scaffolding are dynamic, only being instantiated at runtime, you can't replace the scaffolding incrementally. It's all or nothing, which is hardly agile.

Of course, in the upcoming third edition of Agile Development with Rails this all explained. In the old days (sic) of dynamic scaffolding, a change to the database table underlying an interface changed the interface. Though this is no longer the case, the book includes examples of how to change the code files manually. You add a few lines of code to a few different files and everything just works.

This is great when you're changing polished code, but a pain when you're prototyping. When you're building the initial cut of an app and data model, you want to play before you do any polishing. Work on the big picture first! Adding code to a couple of different files is a burden at this point. Especially since every character you have to type has a relatively large potential to be wrong and slow you down.

While I certainly don't expect to get dynamic scaffolding back (although I could use some alternatives - Streamlined or ActiveScaffold, for instance,) internally Rails does know how to produce the scaffolding code. It makes sense to be able to use Rails to generate the textual pieces of any new scaffolding so that they can be quickly cut and pasted into place.

I don't really care how it's done - output to a file, or the screen, or grafted to the end of the migration as a comment, or whatever - but it should be done by Rails. I really don't want any specialized capability for producing missing scaffolding in an editor macro or template. Editors should be general purpose, requiring little to no maintenance. Putting the intelligence into macros and templates would force them to have to track the next changes in Rails. Cut and paste seems an optimal strategy in the face of losing dynamic scaffolding.

Don't get me wrong. Removing dynamic scaffolding is clearly a good thing for all the reasons it was done. But in the spirit of writing less code, Rails should facilitate the development process by producing intended scaffolding on demand to be copied into place by the developer. This would make it quicker to "get things right" and keep any new scaffolding style-synchronized with the initially generated Rails scaffolding.

Leastways, it couldn't hurt.

Creating Alternate Ruby Objects

One of the things I've found slightly frustrating about Ruby is the new method. Nearly twenty years ago I was writing code in Objective-C on NeXT, which had explicit allocation and initialization:

   Object foo = [[Foo alloc] init]

Using this invocation, foo could be any sort of Object, and init could return any sort of Object. Ruby (and most other OO languages, for that matter) combine the alloc-init into a 'new' method:

   foo = Foo.new

The problem is that the object returned is a Foo, at least unless a class supplied its own specialized new.

In a short paper I've written, Creating Alternate Ruby Objects, I discuss a generalized mechanism that allows object remapping in new. This enables a class' new method to return a different object than the one allocated, according to decisions made in the initialize method.

The mechanism has been added to the release 1.0.4 of my eymiha rubygem in my Chunks of Ruby Infrastructure Project on RubyForge.

The Joy of Testing with EasyLog

In a previous entry I talked about my EasyLogging infrastructure. I'm even happier about it now. I ended up writing some tests and code over the weekend and found myself wanting to track values during my development iterations. With my log_debug statements in the code I simply needed to establish the logging context in the test case to see what my value flows looked like.

At the beginning of the test file, test class, or even interspersed in the test method itself I do a

  start_logging STDERR

when I want logging to begin. At that point any messages that are logged will be written while the test proceeds. When I'm no longer interested, I can do a

  finish_logging

to stop the logging output.

I started to use it consistently in my test cases, putting finish_logging in the teardown method, and then at the top of the test case logging to STDERR if I wanted output or to nil if I didn't. Coupled with the logging in the classes I'm writing (which are subjected to thresholding and directed to files in normal operation) this makes quick work of determining where things have gone awry when tests fail in subtle ways.

Fail in subtle ways? Isn't that an indicator that more tests need to be written? Yes, of course! But in the real world, not all the tests that should get written actually do get written, despite our good intentions. The logging is especially useful in these situations to help identify untested conditions. If I turn on logging output, and then look at the code and a few lines of logged values, I can figure out what's going on and whip together a test a lot more quickly than just wondering why the heck my code isn't doing what I though it should.

Another Ruby Inconsistency

Maybe it's just me, but I keep finding interpreter issues underlying my code. The latest has to do with accessing the file system. In the context of unit testing,

require 'test/unit'
require 'fileutils'

class TC_File_Access < Test::Unit::TestCase

  def setup
    rm_r "temp" if File.exist? "temp"
  end

  def test_non_exisitent_intermediate
    mkdir "temp"
    touch "temp/foo"
    assert_raise(Errno::ENOTDIR) { File.new "temp/foo/bar", "w" }
    rm_r "temp"
  end

end

runs fine on my Mac, bar cannot be created under temp/foo since it's not a directory. Running it under Windows also errors, but the raised error is Errno::ENOENT instead. Of course this plays havoc with tests until the context of the environment is added

class TC_File_Access

  def test_non_exisitent_intermediate
    mkdir "temp"
    touch "temp/foo"
    if Config::CONFIG["target_vendor"] == 'pc'
      assert_raise(Errno::ENOENT)  { File.new "temp/foo/bar", "w" }
    else
      assert_raise(Errno::ENOTDIR) { File.new "temp/foo/bar", "w" }
    end
    rm_r "temp"
  end

end

I hate having to add this sort of code, but the underlying OS is pushing these errors to Ruby and its doing the 'right' thing, pushing up what it gets. I guess my complaint is that I wish Ruby could hide these sorts of details and remap the error to a consistent (most correct) value. It'd be a little more code, but it's all about making life easier for the lowly developer, right?

Imperfect worlds are such a pain.

Easy Logging For Ruby

In the spirit of invisible infrastructure, I built some framework around the Ruby Logger class and made it more succinct, more ubiquitous, and easier to use. The results are discussed in Easy Logging For Ruby, a paper describing the motivation and code that allows a developer to add logging to Ruby code with virtually no effort.

The code is available Chunks of Ruby Infrastructure project as part of the eymiha_util-1.0.1 (or later) rubygem.

How Accurate Is Your Clock?

I bumped my head against another one. While Ruby is not equal in all environments, I at least want to control the difference wherever possible.

When you do a Time.now, you'll get something like

  Tue May 20 17:08:21 -400 2008

Lets say that you're formating that time for a log message though...

  Time.now.strftime "%m/%d/%y %H:%M:%S"

which gives

  05/20/08 17:08:21

Great, except that you'd like subsecond precision.

  (time = Time.now).strftime "%m/%d/%y %H:%M:%S." <<
     ("%06d" % time.usec)

Well, my iMac gives me six digits of microseconds,

  05/20/08 17:08:21.943286

but alas, my PC only three,

  05/20/08 17:08:21.943000

While you may say, "So what?" I must reply, "Yuck." I just don't want those empty zeros hanging out there. The purist in me wants to lop them off. What I want is

  (time = Time.now).strftime "%m/%d/%y %H:%M:%S." <<
     ("%06d" % time.usec)[0,@usecs]

where @usecs is the subsecond precision of the clock.

Now, I could just use the Config to get the platform I'm on and assign the correct value, but in this instance I'd like to be more proactive. I can figure out the right value empirically. I start by getting an Array of sample microseconds that are slightly spread out in time.

  t = (1..5).collect { sleep 0.001001; "#{Time.now.usec}" }

Then I figure out what digit contains the last non-zero digit

  nz = t.collect { |s| s.length - (/[1-9]/ =~ s.reverse) }

And finally, I just take the max

  @usecs = nz.max

Why the multiple samples? Because there's still a one in ten chance that a zero will occur naturally in the real non-zero digit position. Or one in one hundred for two zeros, or one in a thousand for three. By running multiple samples and taking the max, we won't be likely fooled, statistically speaking. Why five samples? I just figure that those odds are pretty darned good.

Of course, we can collapse this all nicely,

  class Time

    @@subsecond_precision = nil

    def self.subsecond_precision
      @@subsecond_precision ||=
        (1..5).collect {
          sleep 0.001001
          s = "#{Time.now.usec}"
          s.length - (/[1-9]/ =~ s.reverse)
        }.max
    end

  end

Now I can just use Time.subsecond_precision in place of @usecs above. I don't have to worry about using system-dependent assignments. I can just do it once when I need it and move forward.

In the Spirit of Brevity, class_attr Methods

Ruby is the language for coding brevity with elegance. The more you "get" Ruby, the less code you end up writing. One of the ways you do this is to just metacode the things you find yourself doing a lot and become more productive. And that's a beautiful thing!

Recently, I've found myself writing and rewriting class attribute accessors, like

class Foo

  def self.bar
    @@bar
  end

  def self.bar=(bar)
    @@bar = bar
  end

end

Being from the old school, while I'm not adverse to just using the @@ directly inside the class, I prefer some encapsulation. And when you want to expose the class' innards, you need to write these methods anyway. This if fine, but it gets a bit too verbose for me.

Instance attributes are exposed easily with the attr methods

class Mumble

  attr_accessor :barfle

end

which creates the barfle and barfle= instance methods. What I want is an analogous

class Foo

  class_attr_accessor :bar

end

to create my self.bar and self.bar= methods. Metaprogramming to the rescue!

class Module

  def class_attr_reader(*kattrs)
    kattrs.each { |kattr|
      ka = kattr.to_s
      reader = <<EOS
def #{ka}
  @@ka
end
EOS
      self.module_eval reader
    }
  end

  def class_attr_writer(*kattrs)
    kattrs.each { |kattr|
      ka = kattr.to_s
      writer = <<EOS
def self.#{ka}=(#{ka})
  @@#{ka} = #{ka}
end
EOS
      self.module_eval writer
    }
  end

  def class_attr_accessor(*kattrs)
    class_attr_reader(*kattrs)
    class_attr_writer(*kattrs)
  end

end

Sweetness. I just love Ruby!

Life in the Windows-Bizarro Universe

You know what my biggest coding-rant about Windows is? Drive designators. They just suck.

Back in the pre-PC days when Unix was the new kid on the block, the notion of path specifications became popular. It was a big change from Dataset declarations in JCL; directories as files was a concept that made sense. It caught on quickly and everyone wanted to do it the Unix way. (One of my first projects at Bell Labs in the early 80s was finishing a port of Unix's ed command in the mainframe environment - the directory-structure logic was as close to Unix as I could make it and made the mainframers quite happy.)

Then PCs came along. Rooted in CP/M, with many influences affecting its evolution, the paths became much like Unix's paths with three main differences: the filenames were limited to 8.3 format, the separators within a path were backslashes, and the a drive designator could be specified.

While modern windows has eliminated the 8.3 limitations and underlying filesystem libraries treat both the slash and backslash with ease, the drive designator lives on as a pain in the ass. I understand the pervasiveness of the feature; the allure of a simple top-level shortcut is strong. But it really has crufted up portability of software from one platform to another for the last thirty years.

Take, for example, the home directory. This is the directory that a user first "lands in" after logging into a system. From inside Ruby code, if you need to get that home directory, you can just

  ENV['HOME']

But on a PC, you have to

  "#{ENV['HOMEDRIVE']}#{ENV['HOMEPATH']}"

This may not seem too painful perhaps, but you need to jump through this hoop whenever you access the filesystem. I've been gradually building up a set of covers for this sort of behavior, such as

  class File
    def self.home_directory
      if Config::CONFIG['target_vendor'] == 'pc'
        "#{ENV['HOMEDRIVE'}#{ENV['HOMEPATH']}"
      else
        ENV['HOME']
      end
    end
  end

for getting the home directory, but there's still no good way to hide the problem once and for all. You just need to be aware of drive designators when you're building paths. It's a fact of life.

But it still sucks.

Opening a URL in a Viewer from Ruby

While web apps are all the rage, what with the browser being ubiquitous and everything, there are still some features that don't scale - usually stuff that requires a lot of high-bandwidth data getting manipulated and displayed. Sorry, but interactive data-heavy apps just don't yet run well enough on the web to make the jump. For other things, yes; but not for these.

Unfortunately, applications based on these sorts of features are the ones I've spent a lot of time writing. They want to be web-enabled, not web-based. It's one thing to use the web as a communications medium, but another to use it as an interaction medium.

So, I got Lyle Johnson's book (FXRuby: Create Lean and Mean GUIs with Ruby) I've been messing with FXRuby lately. Quite a nice little package. I'm moving some big-application stuff into it and should have more to say about it in the upcoming weeks and months. One of the parts I just moved in deals with opening html documents and URLs. A long time ago I decided to write all of my help documents in html and use the browser for displaying them. When the user clicks a help button in an app (or something like that) I need to bring up the requested page in a browser.

Opening a URL in a browser seems like it should be almost trivial, but from a Ruby app there is a little art to it... Different vendors have different mechanisms to do it, and you have to make it work right so the users of your apps don't get flustered. So I hide the vendor-specific logic under a general method:

def open url
  send "open_#{Config::CONFIG['target_vendor']}".to_sym, url
end

This dispatcher will call the open for the specific vendor. I make the separation here, so I can handle each vendor cleanly no matter what sort of crazy stuff they may be doing.

Some vendors make it easy. Take Apple, for instance:

def open_apple url
  system "open #{url}"
end

Apple's made opening a file in your viewer of choice a basic function of the OS. By hiding the details, it's less work and higher productivity for me.

In Windows it takes more effort. Because the viewer of choice is buried in the registry, I have to dig it out:

require 'win32/registry'

def windows_browser
  Win32::Registry::HKEY_CLASSES_ROOT.
       open('htmlfile\shell\open\command') { |reg|
            reg_type, reg_value = reg.read('')
              return reg_value
  }
end

Now I can do the open:

def open_pc url
  system "#{windows_browser} #{url}"
end

On other systems (say Linux, for instance) there's no direct solution. Because the viewer-of-choice is buried in desktop preferences that may be different for different window managers, there's no definitive way to know what it is and how to pull it out. So I opt for flexibility - I depend on an outsider to inject the name of the URL viewer into the mechanism prior to opening the url.

attr_accessor :url_viewer

def method_missing(method,*args)
  if method.to_s =~ /^open_/
    if @url_viewer
      system "#{@url_viewer} #{*args}"
    else
      raise DocumentException,
            "no URL Viewer was designated to open the URL."
    end
  end
end

This isn't enough, however. The Kernel's system method will block until the application opening the URL returns. I don't want the application to stop and wait! In order for the app to stay in control, the open has to run in it's own thread.

def open_document url
  Thread.new {
    send "open_#{Config::CONFIG['target_vendor']}".to_sym, url
  }
end

But there's a problem. I'd like to catch the raised exceptions from the dispatched opens if something unexpected happens - but since the exceptions come from a new thread that I'm not waiting for, they'll just go into the aether. I need to do this using another mechanism. So instead of raising, I'll hypothecate an exception handling mechanism that the thread can use to notify the app that there was a problem during the open.

attr_accessor :exception_handler

def open_document url
  Thread.new {
    begin
      send "open_#{Config::CONFIG['target_vendor']}".to_sym, url
    rescue Exception => exception
      @exception_handler.handle exception
    end
  }
end

and we'll let the caller pre-designate the exception handler. Though this isn't quite as nice as rescuing in the caller, I'm not as concerned since the limited set of things that can fail when opening a url really come down to configuration issues or the absence of the URL's target.

While this will open a URL in a viewer from a Ruby app, there's a little more work needed. I want to ensure the URL is properly-formed enough not to choke the viewer. I'll do this by normalizing before I do the open.

def open_document url
  Thread.new {
    begin
      send "open_#{Config::CONFIG['target_vendor']}".to_sym,
           normalize(url)
    rescue Exception => exception
      @exception_handler.handle exception
    end
  }
end

The normalizing is just a bit funky, but trivial in concept - just return a String containing the normalized URL. My top-level logic is: if it's a file resource, then normalize it as a file; otherwise, validate it as a URI. Since Ruby already comes with a URI class, I just use it.

@@using_pc_filesystem =
     Config::CONFIG['target_vendor'] == "pc"

def normalize(url)
  (file_url? url) ? nomalize_file(url) : URI.parse(url).to_s
end

def file_url?
  (url =~ /^file:/) or
  (url =~ /^\//) or
  ((url =~ /^[A-Za-z]:/) and @@using_pc_filesystem) or
  !(url =~ /:/)
end

It's a URL file resource if it starts with file:, a slash or a drive designator (on a pc) or it doesn't have a colon in it.

Normalizing a file amounts to giving back the normalized file name with file:// prepended to it.

def normalize_file file_url
  path = normalize_file_path(
       (file_url =~ /file:\/\//) ? $' : file_url)
  "file://#{path}"
end

def normalize_file_path file_url
  if absolute_file_path? file_url
    file_url
  elsif @relative_base != nil
    "#{relative_base}#{file_url}"
  else
    raise UrlException, "no relative file base was configured"
  end
end

A file path is absolute if it starts with a drive designator and it's on a PC, or a slash (with no drive designator) if it isn't.

def absolute_file_path? file_url
  @@using_pc_filesystem ? (file_url =~ /^[A-Za-z]:\//) :
      (!(file_url =~ /^[A-Za-z]:/) and (file_url =~ /^\//))
end

Finaly, a relative base is prepended to relative file paths. It fits between a drive designator and the relative path or a PC, or otherwise just sits at the front of the path. When I assign it, I make sure it looks like it'll work.

def relative_base=(relative_base)
  if valid_relative_base? relative_base
    @relative_base = relative_base
  else
    raise UrlException, "Invalid relative base '#{relative_base}'
  end
end

def valid_relative_base? relative_base
  ((@@using_pc_filesystem and (relative_base =~ /^[A-Za-z]:\//)) or
   (!@@using_pc_filesystem and (relative_base =~ /^\//))) and
  (relative_base =~ /\/$/)
end

This wraps everything up nicely. Nice enough that I wrapped it up into a gem I can pull into any of my apps. I added it to my cori project (Chunks Of Ruby Infrastructure) on rubyforge in the eymiha_url rubygem.

Having done this once, I can now get on with the meat of writing my interactive-but-data-heavy Ruby applications, waiting for enough bandwidth on the Internet to someday move them to the web.

Exploratory Surgery and Unit Testing

Test Driven Development is one of those milestones in your development career that once reached, seldom causes you to look back. When writing unit tests before you write the code to pass them becomes second nature, and re-running the unit tests on each change becomes part of the development process, quality simply mushrooms.

But the Church of TDD has several orthodoxies. One sect insists that only the public parts of a unit should be tested. Another says that everything needs to be tested, public or private. I'm a proud member of a third group, whose credo is expounded by Cedric Beust: When it comes down to testing, I follow a simple rule: "if it can break, test it." This pragmatic position seems perfect for the agile developer. Still, as much as possible, I try to keep my private methods lightweight and stick to testing the visible public methods, exercising them extensively enough to make sure the underlying private ones work.

However, I was recently putting together some code that served as a dispatcher: a public method would dispatch to one of a number of private methods depending on some set of conditions. Normally I'd just set up each condition and make sure the returned value was what was expected. But in this case, the dispatching was done inside a new Thread, and no reasonable surface value was returned to check. I needed to get under its skin and test the private methods.

Ruby is kind to testers. I'm a big fan of extending a class within the unit test code itself that will let me do a more complete job. Exposing an attribute or mocking a return value in the testing code is all part of the game, as long as I'm not changing the code I'm releasing. If I have to slip something into the code under test, then it's too dirty. It shouldn't have any inkling being tested, otherwise the system breaks down, governments collapse and fire starts raining from the sky. No thank you.

Instead, one needs to do exporatory operations with surgical instruments. Probe the body, touch a nerve, see what twitches. And for god's sake, don't change anything! In this sense, the private method is the nerve you're touching - what is needed is the probe that lets you get at it.

As everyone knows, testing a public method, say bar, is easy:

   def test_bar
     Foo foo = Foo.new
     assert_equal foo.bar, expected_bar,
                  "all is not well in Foo's bar."
   end

but if we have a private method, frapp, you need to get at it surgically. I opt for a variation in a mechanism proposed by Jay Fields that offers less exposure:

class Class
  def publicize_private_instance_method(method)
    needs_publicity =
         self.private_instance_methods.include? method.to_s
    public method if needs_publicity
    yield
    private method if needs_publicity
  end
end

This allows the simple test:

   def test_frapp
     Foo foo = Foo.new
     Foo.publicize_private_instance_method :frapp do
       assert_equal foo.frapp, expected_frapp,
                    "all is not well in Foo's frapp."
   end

By only exposing the one private method during the course of the test, I'm minimizing the possibility of any unintentional effects of its publicity, and thus am bit more confident than changing the character of the entire class and all of the classes from which it inherits.

I've found this to be a quite a nice surgical probe in my quest for better unit testing.