thinair

Boulder, Colorado

elevation 5400 feet

your guide: Eric Dobbs

Heraclitus, Lau Tzu, Gautama Buddha, and Pink Floyd

Tuesday 04 February 2014 at 08:45

Time itself is an infinite scroll.

Help me Visualize the Invisible in Design

Tuesday 18 December 2012 at 19:45

Hey design friends:

What images do you think of when I ask you about the invisible part of design... the lines and shapes and proportions that make your design hang together in a coherent way?

I'm writing an article to teach computer geeks about design and to explain why CSS sucks as a language for designers. I need some visual support to explain the invisible. Words aren't going cut it. Although beautiful images about typeface design would work nicely. :-)

I've got a few examples here: two from architecture, one a study for a figure drawing. These are in the right direction, but I'd lovee images from many other design disciplines.

Cultural Bias, HTML, CSS, and Conway's Law

Monday 17 December 2012 at 03:10

Overheard: "I'm just a web designer. I don't program or anything."

Here a web designer adopts the cultural bias which values programming above design. But the bias cuts both ways. Designers are not to be trusted with code and coders are not to be trusted with design.

HTML and CSS are unfortunate consequences of this bias. In the ideal world, HTML can be purely semantic and the look-and-feel can be done completely with the CSS. Except that world doesn't really exist and HTML gets littered with extra <divs> to prop up the design needs. And CSS gets littered with duplication of paddings and margins (at the very least) to adjust and control the positions of elements on the page.

And so we have grown templating languages on the server side to try to manage the deficiencies in HTML and CSS in various ways. The menagerie of HTML templating languages is beyond imagination. For CSS we now have SASS and LESS and SCSS: basically templating languages for CSS.

What the server-side languages have in common is introducing turing completeness for languages that are not themselves turing complete. When one language doesn't do what you want, invent another language which can be compiled into the lesser language. This is how C begat C++ begat Java and C# which... never mind, I've gone too far already.

You can see Conway's Law at work here. The programmers and designers are on separate teams and speak different languages. So architectural interfaces are created between the teams. Code goes on this side. Design goes on that side. Over time the architectural boundary between the teams accumulates a lot of kludge on either side to accommodate the inability for the teams to really communicate. And that boundary becomes a point of friction that slows down development and growth.

CSS is especially unfortunate. It is intended for design and it completely misses the mark right from the outset. Seriously. The heart of CSS from a design point of view is the box model. Let me say that again just so you really get the complete and painful irony. The language designed for designers jams all web design into a BOX model. Designers by nature want to think non-linearly and outside-the-box and the language they've been given confines them to a hierarchical tree of boxes. Seriously. So it's hobbled as a programming language and it's a cruel form of torture as a design language.

Framework Adoption Antipattern or MVC is not an architecture

Friday 09 November 2012 at 01:00

Allow me to introduce you to the Framework Adoption Antipattern. And with it I will share some software history that you youngin's might do well to learn.

  1. Get fed up with current tangled mess
  2. Choose The New Hotness
  3. Choose a database
  4. Spend a couple weeks integrating the New Hotness and database with IDE and test infrastructure
  5. Gradually create a New Tangled Mess

The software industry is built around cycles of new adoption. The churn creates an artificial pressure to keep up with the latest and greatest. There's always a new hotness. There's also the illusion that this time around maybe we'll get started on the right foot. Maybe this new hotness will not lead us into a tangled mess.

For those of us who've been to more than one rodeo, it's a depressing to watch history repeat itself in the new hotness, just like it did in the old and busted. The next wave is super tempting. Get ahead of the crowd and you can become the hot shot writing books or speaking at conferences. The early adopters always seem like the coolest kids on the block. Added bonus, you can ditch the tangled mess you're in and start fresh. But every revolution becomes the new establishment. Which is why we keep going in circles.

Advice for a young programmer

I know how this sounds to you. I'm just old and crotchety. I don't get it. I'm part of the establishment. This is a new world. What you're building is really going to change everything.

You're right. You are going to change everything. But you will also learn the truth in the cliche: the more things change the more they stay the same. In five or ten years you will look back at what you've created and see some depressingly familiar tangles. And there will be another new hotness. Your once revolutionary new hotness will grow up to become the new old and busted.

This story is for the long term. As an industry we still don't know how to teach what we do. The only way to learn these lessons is to join a revolution and experience the transformation to establishment. This advice-disguised-as-a-story is for programmers starting their second rodeo.

Historical background on the path to MVC architecture for web apps

Sherman, set the WAYBAC machine to 1995. It was a momentous year. Three items launched with particular fanfare: the Internet was commercialized, Sun released Java, and Netscape released LiveScript JavaScript. Perl 5 and python 1 had been released one year earlier. The first public announcements of PHP and ruby were also in 1995. And the first working draft of XML was in 1996. All of these things in their respective communities were the new hotness. All of them are now the establishment. I'll also mention that Design Patterns was published in late 1994 'cos it comes into the story later.

At the time enterprise computing was dominated by two-tier, client-server architecture: a fat Windows client connecting to a fat database. Over the next few years web applications would be dominated by Perl CGI and Cold Fusion and its copycats: ASP, JSP, and PHP. Sun, IBM, Oracle, WebLogic, BEA and others jumped on the new three-tier architecture. They were selling java middleware in hopes of breaking Microsoft's grip on desktop computing. Instead of a fat Windows client, businesses could use the web browser that's installed with the OS and move their applications onto expensive servers.

By the turn of the century, Internet Explorer had nearly won the browser wars. Netscape would be bought by AOL in a few years. On the server side, Sun and friends were facing backlash against Enterprise Java Beans (EJBs) and Microsoft started its push to move the ASP community to .NET. Sun began evangelizing the Model 2 architecture as the new hotness: separate the display of content from the business logic. It was a fashionable pitch at the time: CSS was promising similar benefits of separating design from markup.

Sun's model 2 marketing and MVC

It was right at the turn of the century when our cultural wires got crossed and we started using the MVC pattern to describe web architecture. MVC was a profound innovation in object-oriented user interface design from Smalltalk-80. That dash eighty refers to 1980 so we're clear that the pattern was already twenty years old at the time. In fact, MVC is used in Design Patterns as an example to help explain what a design pattern is. This was a rare moment when the new hotness was consciously applying lessons from software history.

In the final days of 1999, JavaWorld published Understanding JavaServer Pages Model 2 architecture. In May of 2000, Craig McClanahan from Sun, donated a reference implementation of Sun's Model 2 architecture to the Apache Software Foundation. Struts would become the de-facto standard for java web frameworks. No question it was a terrific improvement to apply the MVC pattern to web apps in contrast to the Cold-Fusion-JSP-ASP-PHP tag-soup model 1. And yet, and yet....

In Sun's marketing and the hype around Struts, Model 2 was described as an architecture. In every explanation the MVC pattern was used to explain the architecture. And so the Model 2 architecture, the MVC pattern, and the Struts framework were all conceptually muddled in the Java community.

And then Rails was the new hotness

Another half-decade later when Rails burst onto the scene, MVC was taken for granted as the de facto best practice for web application architecture. A new generation of programmers were introduced to web applications and MVC-as-architecture and The Rails Way at the same time.

What's wrong with MVC and Model 2 for web applications?

MVC originally lived in a Smalltalk image, which is sorta like that virtual machine you have up in the cloud running your modern web applications. Only it was a lot less complicated. Importantly the M and the V and the C were all living in the same image. When messages were passed between the different components in an MVC pattern in Smalltalk, the messages didn't have far to go.

Model 2 by contrast was full of network latency because it grew up when Sun was trying to sell hardware into the enterprise. There were browsers on the desktop and there was middleware running java on expensive hardware, and then a database (probably Oracle) running on another bit of expensive hardware.

Web frameworks have been contending with two key pieces of friction for the past decade or so. On the client side there's the statelessness of HTTP and on the back end there's the object-relational mapping to get data back and forth from a pile of object-oriented business logic on the server into a pile of relational algebra in the database.

MVC in Smalltalk suffered from neither of those key problems. Data were persisted within the image right along side the class definitions, and the View and Controller were in direct and very stateful communication.

Ever since the Model 2 architecture co-opted MVC, Model has come to mean some object-relational mapping, View is something from the menagerie of templating languages, and the Controller... Ahh the controller...

Controller as a term is meaningless. No, it's worse than that. Controller is actively destructive. I know exactly what a Controller is, and so do you. But my Controller is different from your Controller. We're using the same word to describe completely different things. The only common ground we have is that we know there's something between the Model and the View.

MVC is not an architecture and neither is your framework

MVC is a pattern. It's beautiful and full of wisdom. It's an exceptionally good example to teach the principle of separating concerns. But the coopting of MVC into an architectural framework effectively blinded us to the principles and left us with software dogma. And such powerful dogma that the Rails revolutionaries embraced the dogma wholesale even as their rhetoric railed against excessive ceremony and dogma in the java community.

If you looked at a typical rails app you'd think that MVC and ActiveRecord were the only design patterns you need. And as applications have grown from simple foundations in Rails into enterprise-sized beasts, we hear about developers reaching for Plain Old Ruby Objects to speed up their test suites. There's buzz about refactoring away from fat controllers and fat models. Rails apps have become most of what it originally opposed.

What's more insidious is the pervasive use of object inheritance in web frameworks. Design Patterns has been published for almost two decades and itself summarized wisdom from the previous two decades of object-oriented design. A core principle espoused therein is to prefer composition to inheritance and yet frameworks continue to recommend their developers inherit. This and a database schema is all you need: class Post < ActiveRecord::Base; end

Yep, Rails apps are a tangled mess. Let's switch to the New Hotness.

There's a lot of excitement building around javascript these days: Backbone.js, Ember.js, Node.js, and Meteor are a few examples. There's buzz around various functional languages: Scala, Erlang, Haskell, and Clojure, for example. But really, why bother with the web anymore when there's Andriod and iOS? As always there's a lot of options in the category of new hotness.

Nevermind me and my war stories. Just make the switch and start drafting talks for the surge of javascript or functional or mobile conferences. You can re-invent Rails in a new language as a fast path to join the next generation of thought leaders.

Touch, shake, tilt: touchmove, deviceorientation, devicemotion apis

Monday 07 May 2012 at 23:58

For a few weeks I've been experimenting with some new (to me) HTML5 apis for multitouch events, device orientation and device motion. I'm planning to work these into my turtle graphics implementations, but needed to understand what information these sensors provide.

A couple weeks ago I got around to looking at Tim Bray's post about sensor kenetics on Android devices. Fun and just the nudge I needed to start visualizing the inputs from these sensors.

I've only tested these on iOS, so no promises for those of you with droids. Have a look first at this expriment with javascript multitouch which includes real-time charts of deviceorientation and devicemotion apis. The charts are modeled directly on Tim's.

It was a good enough start. Fiddling around with the charts real-time helped a lot. But I'm much happier with tonight's milestone. Here is an interactive visualization of deviceorientation and devicemotion apis. The deviceorientation controls the position of three circles on the screen. The devicemotion adjusts the radius of the circles.

Bell Labs' Unsung Heros in the History of Computing

Monday 30 April 2012 at 00:42

I'm trying to fix a bug in the popular history of computing.

We know Alan Turing and his role deciphering enigma codes in WWII. But can you name anyone who secured the Allied communications? Why were the Axis unable to decipher our codes? The heros who secured Allied communications were bound to secrecy while the history of computing was being written. Ironic: their success in keeping secrets has kept their role secret too.

Over the holidays in 2001 I met Sarah's extended family for the first time. I was introduced as a computer programmer to her grandfather, Ralph Miller. "What do you know about the Internet?" he asked like the opening question in an oral exam. Over the next couple hours I remember feeling like we were modems trying various ways to handshake. He was speaking the telecom jargon of an electrical engineer who'd been retired for 20 years. I was speaking with the limited telecom knowledge left over from configuring Ascend and Cisco routers with frame relay and ISDN lines five years earlier. It was hard to find common ground.

"Grandpa's telling Eric how he invented the Internet."

"Oh good. Maybe he'll be able to explain it to the rest of us."

In retrospect, I'm profoundly lucky to have had regular conversations over the past decade with one of the pioneers of the digital age. He didn't inventing the Internet. It was more fundamental than that. Ralph worked at Bell Labs on the team which created the X System, as it was called at Bell Labs, known as SIGSALY when it was in service. The National Security Agency has heralded it as the start of the digital revolution. But it and its engineers need a promotion in the popular history.

In that first conversation with Ralph, there was one point that sticks in my mind. It was one of the few places where we had common language. He said "they brought in a hot-shot kid from MIT to try to break the code. What was his name?" It was a name I'd heard before. "Shannon. Claude Shannon." After a moment of reflection he added, "He never could break it."

I think what struck me most was his tone of voice. He completely lacked the sense of reverence I'd always heard from people talking about Claude Shannon. Here was my fiancée's grandpa describing one of the demigods of computing as a bright kid, a math wiz who'd nevertheless been beaten in by a math problem. There were other names which Ralph reveres: R. C. Mathes, R. K. Potter, and H. W. Dudley. But Claude Shannon was just a youngster in Ralph's eyes, and given too much credit as an individual for work that was created by a very high performing team.

The cypher used in SIGSALY was a one-time pad. Shannon ended up writing a proof that the one-time pad is unbreakable. Part of the reason Shannon's initial publications on cryptography and information theory were so complete is because he'd been involved in analyzing the most ground-breaking secret communications system of the day -- a system that would remain a tightly guarded military secret for another thirty years. The implementation and essential innovation came first. The groundbreaking theory came second. But the popular history of computing was written while the implementation was still under wraps.

On Friday, Talk of the Nation interviewed Jon Gertner about his new book The Idea Factory: Bell Labs and the Great Age of American Innovation. The chapter on Shannon is a perfect example of popular history missing this key part of the story. Although the rest of the world were taken by surprise by the insights in the "Communication Theory of Secrecy Systems", and "A Mathematical Theory of Communication" neither Ralph nor his colleagues were. For them Shannon had captured the common knowledge among the engineers involved in the project.

There are a few details which Gertner gets wrong about Pulse Code Modulation (PCM), by the way. On page 127 he writes "Shannon wasn't interested in helping with the complex implementation of PCM -- that was a job for the development engineers at Bell Labs, and would end up taking them more than a decade." On the contrary, a patent for PCM was filed in 1943 by Ralph and his assistant Badgley. From the outset of the project Bell Labs were looking for a way to combine the Vernam cypher, the one-time pad which had been devised for telegraph encryption, with H. W. Dudley's vocoder which could compress and synthesize speech. PCM was the inflection point. Analog to digital. Once the signal was digital it could be combined with a random key, the essential ingredient for unbreakable encryption.

Here's the other interesting part. Ralph turned 105 in March. I'll have a chance to visit with him again this summer. Got any questions for him? Imagine you could talk to someone like Turing or von Neumann or Shannon. What would you want to know?

Here's a list of inventors and their patents related to SIGSALY. These are some of the unsung heros.

Inventor U.S. Patent Filed Issued
A. A. Lundstron, L. G. Schimpf 3,897,591 8/27/42 7/29/75
A. E. Melhose 3,891,799 9/27/44 6/24/75
A. J. Busch 3,968,454 9/27/44 7/6/76
D. K. Gannett 3,893,326 9/27/44 9/28/76
D. K. Gannett 3,924,075 3/20/47 12/2/75
D. K. Gannett 3,934,078 5/1/46 1/20/76
D. K. Gannett 3,944,744 5/10/45 3/16/76
D. K. Gannett 3,944,745 5/10/45 3/16/76
D. K. Gannett 3,953,677 5/10/45 4/27/76
D. K. Gannett 3,953,678 5/10/45 4/27/76
D. K. Gannett 3,965,297 5/1/46 6/22/76
D. K. Gannett, A. C. Norwine 3,983,327 7/9/45 9/28/76
E. Peterson 3,924,074 5/19/45 12/2/75
E. Peterson 3,979,558 6/30/44 9/7/76
H. L. Barney 3,193,626 12/29/44 7/6/65
H. W. Dudley 3,470,323 6/30/44 9/30/69
H. W. Dudley 3,985,958 12/18/41 10/12/76
K. H. Davis, A. C. Norwine 3,024,321 12/29/44 3/6/62
L. G. Schimpf 3,394,314 7/17/43 7/23/68
M. E. Mohr 3,076,146 12/27/45 1/29/63
M. E. Mohr 3,188,390 12/20/43 6/8/65
N. D. Newby, H. E. Vaughan 3,373,245 8/27/42 3/12/68
O. Myers 3,937,888 7/17/43 2/10/75
R. C. Mathes 3,967,066 9/24/41 6/29/76
R. C. Mathes 3,991,273 10/4/43 11/9/76
R. H. Badgley, L. G. Schimpf 3,405,362 12/20/43 10/8/68
R. H. Badgley, R. L. Miller 3,912,868 7/17/43 10/14/75
R. K. Potter 3,340,361 7/9/45 9/5/67
R. K. Potter 3,967,067 9/24/41 6/29/76
R. L. Miller 3,887,772 6/30/44 6/3/75
R. L. Miller 3,965,296 6/30/44 8/24/76
R. L. Miller 3,976,839 6/30/44 8/24/76

Programming by Touch

Sunday 18 March 2012 at 13:52

Typing on phones is a drag. Programming requires typing. Game over for programming on touch screens, right? What if programming didn't require typing? Here's a first shot, very incomplete.

Still too early to call this programming. If you squint and turn your head to the side you can see something like a function with no parameters. There are no loops, branches, nor variables. But it does work on a touch device.

Try it on your phone: http://dobbse.net/turtle/wander

Turtle Geometry exercises

Thursday 22 December 2011 at 17:05
► History

Exercises and examples from chapter one of Turtle Geometry by Abelson and diSessa implemented in javascript. Kinda blown away by this book: only on chapter one and this page already offers a pretty fun playground for exploring geometry and rudimentary graphics programming.

Click some of the links below for examples and inspiration. You can also type commands into the form below the turtle's sandbox. Commands you type will be saved in the history below the sandbox.

Pen controls include penup, pendown, pensize pixels, and pencolor color. Any CSS color name or hex value will work for color: Wikipedia: Web Colors

Have a look at the source code for these exercises if you like: tg.js

Testing cookies in Capybara, Rack::Test, RSpec, and Rails: debugging other people's code (2 of 2)

Sunday 13 December 2011 at 23:10

A quick refresher, mainly for those who may land here first instead of on part 1. I spent all of last Friday figuring out a failing capybara test around browser cookies. First, the tests:

describe "ServiceRequestController", type: :request do
  it "redirects when credentials are missing" do
    get new_service_request_path
    response.status.should == 302
  end

  it "renders the form when credentials are found" do
    page.driver.browser.set_cookie 'username=jhendrix'
    get new_service_request_path
    response.status.should == 200
  end    
end

Second, the controller:

class ServiceRequestController < ApplicationController
  def new
    user = LegacyUser.find_by_username(request.cookies['username'])
    if !user
      redirect_to cookies_path
    end
  end
end

First test passes, second test fails. The previous post traced through line one of my failing spec: page.driver.browser.set_cookie 'username=jhendrix'.

tl;dr

If you're here just for the work-around when testing with cookies in Capybara with the Rack::Test driver and RSpec, here's the best solution I've found so far to make it work. Nevermind trying to set cookies via capybara. Just pass in the HTTP_COOKIE header with the call to get.

describe "ServiceRequestController", type: :request do
  ...
  it "renders the form when credentials are found" do
    get new_service_request_path, {}, 'HTTP_COOKIE' => 'username=jhendrix'
    response.status.should == 200
  end    
end

More tricks for debugging other people's code: aka. code caving in a maze of twisty passages

I still haven't answered my first question: "am I setting cookies correctly?" As you can guess from the tl;dr, the short answer is "No". But life is a journey, not a destination. I'll start again with the first line: page.driver.browser.set_cookie 'username=jhendrix' The first place to look is at set_cookie and see if I'm actually setting one. I'll start with the most important programming power tools find and grep and I'll add in another one: intentionally raising an exception.

find ${GEM_PATH//:/ /} -type f -exec grep -nH -e 'def set_cookie' {} /dev/null \;

Here's the results of that search, cleaned up for legibility and brevity


action_dispatch/http/response.rb:158:                   def set_cookie(key, value)
action_dispatch/middleware/session/cookie_store.rb:65:  def set_cookie(env, session_id, cookie)
rack/response.rb:58:                                    def set_cookie(key, value)
rack/session/abstract/id.rb:320:                        def set_cookie(env, headers, cookie)
rack/session/cookie.rb:121:                             def set_cookie(env, headers, cookie)
rack/mock_session.rb:23:                                def set_cookie(cookie, uri = nil)

As I mentioned in part 1, the hits in action_dispatch, rack/response and rack/session are probably all related to the real cookie code. That stuff has to be working correctly or no Rails app would be able to save cookies. It almost has to be something in rack/mock_session. But this time around I'll take one extra moment test my hypothesis. Look at the source for that method in Rack::MockSession :

def set_cookie(cookie, uri = nil)
  cookie_jar.merge(cookie, uri)
end

Where's cookie_jar come from?

def cookie_jar
  @cookie_jar ||= Rack::Test::CookieJar.new([], @default_host)
end

I'll take drastic action to confirm this is the code in question:

def cookie_jar
  @cookie_jar ||= Rack::Test::CookieJar.new([], @default_host)
  raise @cookiejar.inspect
end

Running the specs... sure enough they fail with a RuntimeError, for example:

1) new service requests redirects when credentials are missing
   Failure/Error: get new_service_request_path
   RuntimeError:
     #<Rack::Test::CookieJar:0x00000103d592c0 @default_host="www.example.com", @cookies=[]>
   # ./spec/requests/service_requests_controller_spec.rb:5:in `block (2 levels) in <top (required)>'

Now I know I'm going in the right direction. Is the cookie is getting set?

def set_cookie(cookie, uri = nil)
  puts "Rack::MockSession.set_cookie(#{cookie})"
  cookie_jar.merge(cookie, uri)
  raise @cookiejar.inspect
end

...

def cookie_jar
  @cookie_jar ||= Rack::Test::CookieJar.new([], @default_host)
end

.Rack::MockSession.set_cookie('username=jhendrix')
F

Failures:

  1) new service requests renders the form when credentials are found
     Failure/Error: page.driver.browser.set_cookie "username=jhendrix"
     RuntimeError:
       #<Rack::Test::CookieJar:0x000001015ee0f0 @default_host="www.example.com", @cookies=[#<Rack::Test::Cookie:0x000001015ee050 @default_host="www.example.com", @name_value_raw="username=jhendrix", @name="username", @value="jhendrix", @options={"domain"=>"www.example.com", "path"=>""}>]>
     # ./spec/requests/service_requests_controller_spec.rb:10:in `block (2 levels) in <top (required)>'

set_cookie works: the cookie I've set in the spec shows up in the CookieJar. Why is the spec failing? I'll change the code to print out the cookie_jar instead of raising an exception and keep searching.

def set_cookie(cookie, uri = nil)
  result = cookie_jar.merge(cookie, uri)
  puts "set_cookie('#{cookie}')\n#{@cookiejar.inspect}"
  result
end

Line 2 of the failing spec

get new_service_request_path

My trusty friends find and grep will turn up a long list. get is a really popular method name for objects of every shape and size.

find ${GEM_PATH//:/ /} -type f -exec grep -nH -e 'def get(' {} /dev/null \;

I'll draw your attention to the punctuation... I'm searching for get( and not just get so the results don't include false leads like get_coffee. Cleaned up results:

action_controller/test_case.rb:364:                   def get(action, parameters = nil, session = nil, flash = nil)
action_dispatch/routing/mapper.rb:476:                def get(*args, &block)
action_dispatch/routing/route_set.rb:106:             def get(name)
action_dispatch/testing/integration.rb:32:            def get(path, parameters = nil, headers = nil)
action_view/flows.rb:12:                              def get(key)
action_view/flows.rb:46:                              def get(key)
active_model/errors.rb:93:                            def get(key)
active_record/identity_map.rb:77:                     def get(klass, primary_key)
active_resource/connection.rb:79:                     def get(path, headers = {})
active_resource/custom_methods.rb:56:                 def get(custom_method_name, options = {})
active_resource/custom_methods.rb:89:                 def get(method_name, options = {})
bundler/vendor/thor/actions/file_manipulation.rb:72:  def get(source, *args, &block)
capybara/rack_test/driver.rb:75:                      def get(*args, &block)
ffi/struct.rb:40:                                     def get(ptr)
ffi/struct.rb:51:                                     def get(ptr)
ffi/struct.rb:66:                                     def get(ptr)
rack/mock.rb:56:                                      def get(uri, opts={})
rack/cache/appengine.rb:27:                           def get(key)
rack-cache-1.1/test/entitystore_test.rb:248:          def get(key); self[key]; end;
rack-cache-1.1/test/metastore_test.rb:318:            def get(key); self[key]; end;
rack-cache-1.1/test/spec_setup.rb:169:                def get(stem, env={}, &b)
rack/test.rb:55:                                      def get(uri, params = {}, env = {}, &block)
rake-0.9.2.2/doc/proto_rake.rdoc:70:                  def get(task_name)
rspec/matchers/operator_matcher.rb:15:                def get(klass, operator)
sass/util/subset_map.rb:73:                           def get(set)
selenium/webdriver/common/driver.rb:105:              def get(url)
selenium/webdriver/remote/bridge.rb:98:               def get(url)
selenium/webdriver/support/event_firing_bridge.rb:20: def get(url)
thor-0.14.6/lib/thor/actions/file_manipulation.rb:72: def get(source, *args, &block)

I could use the same trick by methodically editing every one of these results and raise exceptions in each and then run the specs again. But that would be tedious and I have another trick to share. It's particularly useful for any Domain Specific Language. I'll just instrument the spec itself to report which get to investigate.

it "renders the form when credentials are found" do
  page.driver.browser.set_cookie "username=jhendrix"
  puts "#{method(:get)}"
  get new_service_request_path
  response.status.should == 200
end

Running the spec...

#<Method: RSpec::Core::ExampleGroup::Nested_1(ActionDispatch::Integration::Runner)#get>

Connecting the dots between the earlier result action_dispatch/testing/integration.rb and ActionDispatch::Integration::Runner

It's not quite as easy to spot, but notice that get is forwarded to integration_session

%w(get post put head delete cookies assigns
   xml_http_request xhr get_via_redirect post_via_redirect).each do |method|
  define_method(method) do |*args|
    reset! unless integration_session
    # reset the html_document variable, but only for new get/post calls
    @html_document = nil unless method.in?(["cookies", "assigns"])
    integration_session.__send__(method, *args).tap do
      copy_session_variables!
    end
  end
end

Integration::Session is defined earlier in the same file:

class Session
  DEFAULT_HOST = "www.example.com"

  include Test::Unit::Assertions
  include TestProcess, RequestHelpers, Assertions

There's no sign of any method named get in Integration::Session, but looking through the included modules, I found it in RequestHelpers

def get(path, parameters = nil, headers = nil)
  process :get, path, parameters, headers
end

which calls process which after a bunch of setup creates a new Rack::Test::Session

def process(method, path, parameters = nil, rack_env = nil)
  ...
  session = Rack::Test::Session.new(_mock_session)

which calls _mock_session

def _mock_session
  @_mock_session ||= Rack::MockSession.new(@app, host)
end

That code looks familiar. Both the first and second lines of the spec ask Rack::MockSession for their cookies. Let's instrument that class a little further.

def cookie_jar
  @cookie_jar ||= Rack::Test::CookieJar.new([], @default_host)
  puts "    Rack::MockSession.cookie_jar #{@cookie_jar}"
  @cookie_jar
end

I also need to know when get is called. Back to ActionDispatch::Integration::RequestHelpers

def get(path, parameters = nil, headers = nil)
  puts "ActionDispatch::Integration::RequestHelpers.get #{path}"
  process :get, path, parameters, headers
end

Run the failing spec.

Rack::MockSession.cookie_jar #<Rack::Test::CookieJar:0x00000103458068>
Rack::MockSession.cookie_jar #<Rack::Test::CookieJar:0x00000103458068>
Rack::MockSession.set_cookie('username=jhendrix') #<Rack::Test::CookieJar:0x00000103458068 @default_host="www.example.com", @cookies=[#<Rack::Test::Cookie:0x00000103457e88 @default_host="www.example.com", @name_value_raw="username=jhendrix", @name="username", @value="jhendrix", @options={"domain"=>"www.example.com", "path"=>""}>]>
#<Method: RSpec::Core::ExampleGroup::Nested_1(ActionDispatch::Integration::Runner)#get>
ActionDispatch::Integration::RequestHelpers.get /service_requests/new
Rack::MockSession.cookie_jar #<Rack::Test::CookieJar:0x00000102dce008>
Rack::MockSession.cookie_jar #<Rack::Test::CookieJar:0x00000102dce008>

The source of the problem is revealed in these results but it takes careful reading to spot it. Inside MockSession, cookie_jar is called twice. Then comes the output from the call to set_cookie. Line 4 is from the spec where I identified which get was called. Line 5 is the debug output I just added. And notice that cookie_jar is again called twice within the call to process.

All reasonable enough. Where's the bug?

Look very closely at the CookieJars in use. It'll help if I put them right next to each other:

Rack::MockSession.cookie_jar #<Rack::Test::CookieJar:0x00000103458068>
Rack::MockSession.cookie_jar #<Rack::Test::CookieJar:0x00000102dce008>

Those may be the same class but they are different instances. The call to get reads its cookies from a different CookieJar than where they are set in the call to set_cookie. In fact, those two calls even have different MockSessions. Removing all the previous debug output and adding this:

def initialize(app, default_host = Rack::Test::DEFAULT_HOST)
  puts "#{self}"
  @app = app
  @after_request = []
  @default_host = default_host
  @last_request = nil
  @last_response = nil
end

gives the following output when I run the spec again.

#<Rack::MockSession:0x00000102e69ad0>
#<Rack::MockSession:0x00000102d8e9d0>

My first work-around was this:

it "renders the form when credentials are found" do
  page.driver.browser.set_cookie 'username=jhendrix'
  page.driver.get new_service_request_path
  response.status.should == 200
end

That does get those two lines of code using the same MockSession which also makes the test pass:

#<Rack::MockSession:0x000001036e9db8>
.

Finished in 8.14 seconds
1 example, 0 failures

But in the end, that smells wrong to me. And having just looked at ActionDispatch::Integration::RequestHelpers.get I had a new idea for providing cookies:

it "renders the form when credentials are found" do
  get new_service_request_path, {}, 'HTTP_COOKIE' => 'username=jhendrix'
  response.status.should == 200
end

That test passes.

recap of debugging tricks

Beginner's Mind is really important. Don't be clever. Just carefully follow the code. Take notes along the way so you know the landmarks and can find your way back.

find and grep to search in $GEM_PATH for method definitions.

Raise exceptions in possible matches to confirm you're looking in the right place. Replace the exceptions with puts statements when you need to let the software proceed.

Ruby's built-in introspection, especially Object#method can help identify methods in DSLs where you may not have an easy handle on the object.

You have to read carefully. Read the code carefully and read debug output and stack traces carefully. Even when you narrow it down to the same class in similar contexts, you may be looking at different instances of the class.

recap of this specific "bug"

It is reasonable to think that calling set_cookie should work for the next get or post call. It doesn't. page.driver.browser.set_cookie creates one instance of Rack::MockSession and get creates another. They're not the same session.

On the other hand, maybe it's not a bug in the code but in the documentation. If everyone knows to just pass in HTTP_COOKIE with the request, then there's no need for a pull request at all. I think that's where I've landed.

If for some reason you feel motivated to fix the interaction around set_cookie, there's some kind of coordination needed between Capybara's Rack::Test driver and the Rack::Test::Methods so they can share the same Rack::MockSession.

Testing cookies in Capybara, Rack::Test, RSpec, and Rails: debugging other people's code (1 of 2)

Sunday 11 December 2011 at 23:39

This morning Sarah and the kids made gingerbread cookies, thoroughly destroying all illusions of joyous family fun for the holidays. Speaking of cookies failing to deliver promises, I blew the entire day Friday figuring out a failing capybara test around browser cookies. Not a good weekend for cookies.

It's a Rails app which depends entirely on a legacy authentication and authorization system. The simplest thing that could possibly work is use cookies set by the existing authentication system. Should be simple.

describe "ServiceRequestController" do
  it "redirects when credentials are missing" do
    get new_service_request_path
    response.status.should == 302
  end
end

Test is red.

class ServiceRequestController < ApplicationController
  def new
    redirect_to cookies_path
  end
end

Test is green.

describe "ServiceRequestController" do
  ...
  it "renders the form when credentials are found" do
    page.driver.browser.set_cookie 'username=jhendrix'
    get new_service_request_path
    response.status.should == 200
  end    
end

Test is red.

class ServiceRequestController < ApplicationController
  def new
    user = LegacyUser.find_by_username(request.cookies['username'])
    if !user
      redirect_to cookies_path
    end
  end
end

Test is red. I found a work-around just before lunch, but spent the rest of the day trying to understand why that doesn't work.

tl;dr

If you're here just for the work-around when testing with cookies in Capybara and RSpec, here's my first work around (but the better way to test cookies in Capybara, Rack::Test and RSpec is in part 2 ):

describe "ServiceRequestController" do
  ...
  it "renders the form when credentials are found" do
    page.driver.browser.set_cookie 'username=jhendrix'
    page.driver.get new_service_request_path
    response.status.should == 200
  end    
end

Use the page.driver when you call get.

Debugging other people's code: aka. code caving in a maze of twisty passages

What follows are some of the most important programming skills I learned working with Rob and Paul at bivio: how to find your way around other people's code. I'll start with the failing test.

describe "ServiceRequestController" do
  ...
  it "renders form when credentials are found" do
    page.driver.browser.set_cookie 'username=jhendrix'
    get new_service_request_path
    response.status.should == 200
  end    
end

First question: Am I calling set_cookie correctly? There's a profoundly important tone in this question. Start with a beginner's mind. Don't guess. Don't be clever. Just read and follow the code. This is also one of the hardest things to do. As programmers we're all pretty intoxicated with our own cleverness. Let go and use The Source.

In RubyMine you can Command-Click on set_cookie or type Command-B. That pulls up a list of likely candidates. Without RubyMine you need some unix fu. Two of the most important, general purpose, and language independent programming tools ever are find and grep. Get to know them. They are your best friends.

find ${GEM_PATH//:/ /} -type f -exec grep -nH -e 'def set_cookie' {} /dev/null \;

Either way you search, the results are similar (though not identical). These are cleaned up from the results via find and grep:


action_dispatch/http/response.rb:158:                   def set_cookie(key, value)
action_dispatch/middleware/session/cookie_store.rb:65:  def set_cookie(env, session_id, cookie)
rack/response.rb:58:                                    def set_cookie(key, value)
rack/session/abstract/id.rb:320:                        def set_cookie(env, headers, cookie)
rack/session/cookie.rb:121:                             def set_cookie(env, headers, cookie)
rack/mock_session.rb:23:                                def set_cookie(cookie, uri = nil)

The hits in action_dispatch, rack/response and rack/session are probably all related to the real cookie code. That stuff has to be working correctly or no Rails app would be able to save cookies. It almost has to be something in rack/mock_session.

There's a certain bit of irony here. I didn't trust these search results. I was Being Clever. I took a longer path and ended up in rack/mock_session anyway. The longer path did give me the confidence of a second opinion. Trusting The Source would have produced my answer more quickly. Then again, I might have found the answer and still not trusted it. I'll share the longer path I took which is pretty much what I could learn by stepping through with a debugger.

In this call, page.driver.browser.set_code where does page come from? I know it's from Capybara, but where exactly? It'll be in here somewhere: https://github.com/jnicklas/capybara/tree/master/lib/capybara.

cucumber.rb	
dsl.rb
rails.rb	
rspec.rb	
selector.rb	
server.rb	
session.rb	
version.rb
Looking over that list dsl.rb looked most promising. And sure enough in Capybara::DSL I find

def page
  Capybara.current_session
end

current_session is defined earlier in the file:

def current_session
  session_pool["#{current_driver}:#{session_name}:#{app.object_id}"] ||= Capybara::Session.new(current_driver, app)
end

Now I know that page is a Capybara::Session using current_driver. What's current_driver? It's also defined in dsl.rb:

def current_driver
  @current_driver || default_driver
end

And default_driver?

def default_driver
  @default_driver || :rack_test
end

So page is a Capybara::Session initalized with :rack_test. The relevant code:

attr_reader :mode, :app

def initialize(mode, app=nil)
  @mode = mode
  @app = app
end

def driver
  @driver ||= begin
    unless Capybara.drivers.has_key?(mode)
      other_drivers = Capybara.drivers.keys.map { |key| key.inspect }
      raise Capybara::DriverNotFoundError, "no driver called #{mode.inspect} was found, available drivers: #{other_drivers.join(', ')}"
    end
    Capybara.drivers[mode].call(app)
  end
end

Remind me again what it was I was searching for? I've bounced around the code enough to be lost in "a maze of twisty little passages, all alike." Oh yeah: page.driver.browser.set_cookie. After all this caving, I'm finally through page and now looking for driver. The good news is, it's right here: Capybara.drivers[:rack_test].call(app). But where do I find Capybara.drivers? Well given the name, let's look first at the top level capybara module: Capybara.

def drivers
  @drivers ||= {}
end

Searching through that file, this is the only line that refers to @drivers. What about references to drivers itself? Here's the only one I could find:

def register_driver(name, &block)
  drivers[name] = block
end

And register_driver?

Capybara.register_driver :rack_test do |app|
  Capybara::RackTest::Driver.new(app)
end

Now I know the driver is a Capybara::RackTest::Driver and the top of that file invites the next question: what's the browser?

  attr_reader :app, :options

def initialize(app, options={})
  raise ArgumentError, "rack-test requires a rack application, but none was given" unless app
  @app = app
  @options = DEFAULT_OPTIONS.merge(options)
end

def browser
  @browser ||= Capybara::RackTest::Browser.new(self)
end

Are we there yet? What was I looking for again? Oh, right. It's set_cookie. But there's no sign of that method in Capybara::RackTest::Browser. But there is an important bit that's easy to miss right up top:

class Capybara::RackTest::Browser
  include ::Rack::Test::Methods

Let's take a peek in Rack::Test::Methods. set_cookie is delegated to the current_session.

METHODS = [
  :request,
  :get,
  :post,
  :put,
  :delete,
  :options,
  :head,
  :follow_redirect!,
  :header,
  :set_cookie,
  :clear_cookies,
  :authorize,
  :basic_authorize,
  :digest_authorize,
  :last_response,
  :last_request
]

def_delegators :current_session, *METHODS

current_session points to rack_test_session:


def current_session # :nodoc:
  rack_test_session(_current_session_names.last)
end

which points to build_rack_test_session:


def rack_test_session(name = :default) # :nodoc:
  return build_rack_test_session(name) unless name

  @_rack_test_sessions ||= {}
  @_rack_test_sessions[name] ||= build_rack_test_session(name)
end

which creates a Rack::Test::Session with a rack_mock_session:


def build_rack_test_session(name) # :nodoc:
  Rack::Test::Session.new(rack_mock_session(name))
end

which points to build_rack_mock_session:


def rack_mock_session(name = :default) # :nodoc:
  return build_rack_mock_session unless name

  @_rack_mock_sessions ||= {}
  @_rack_mock_sessions[name] ||= build_rack_mock_session
end

"Which will bring us back to" Rack::MockSession.


def build_rack_mock_session # :nodoc:
  Rack::MockSession.new(app)
end

And inside Rack::MockSession is where the test calls set_cookie.

def set_cookie(cookie, uri = nil)
  cookie_jar.merge(cookie, uri)
end

And what, finally, is cookie_jar?

def cookie_jar
  @cookie_jar ||= Rack::Test::CookieJar.new([], @default_host)
end

It's probably a newly minted Rack::Test::CookieJar. Now wouldn't it have been a lot easier if I'd just trusted the search results in the first place? find and grep are your best friends. You can trust them to save you a lot of time.