El chele corrects me in a comment to Citizen Power and the Political Bell Curve. I quote it here so the comment and response both make it to the RSS subscribers.
Eric - I think that part of the struggle you are having in deciding how to have an impact on the political system is that you've internalized the "popular" conception of the political structure in the US. Paraphrasing Nietzsche: "The Constitution is a modern idea, that is to say, it is a false one."
The Constitution most certainly did not "divide power into four groups"--it divided power into two groups: the Federal government and the State governments, with the balance of power subtly but firmly tipped in favor of the former. The passage of the 16th and 17th amendments and the Federal Reserve Act at the beginning of the 20th century removed the few remaining checks on Federal power. At that point the Federal government became a National government and it has not stopped growing since.
We have a National government that is out of control--and you can caucus and vote all you want and you will do nothing more than feed the system. In the end revolution--guns and bloodshed not bumperstickers and politically-conscious "Rock The Vote" warm-fuzzies--is what is headed our way.
El chele further recommends The Hologram of Liberty for more on the myth of the Constitution. The link itself is worth visiting, I'm sure the book is an interesting read.
Voting is a duty too many citizens shirk. It may only feed the system, but the more citizens voting, the more balanced the diet will be. That would be an important improvement. More importantly, abstaining from the vote does not starve the system but only exaggerates its imbalance. A tax revolt might starve the system, but not voting makes political matters worse.
El chele, your attention to States' rights compelled me to re-examine The Constitution. A number of things leap out in light of our conversation. I'm almost completely wrong about the power given to the people. "People" only appears twice in the document: once in the preamble (_We the People..._) and once in the discussion of the election of Representatives. By contrast "State" is mentioned throughout. Much of the document describes the boundaries of federal power and the division of federal power. The rights of the people are granted in the Bill of Rights but the only powers granted to the people are in the brief but sweeping afterthought of the Tenth Amendment: "The powers not delegated to the United States by the Constitution, nor prohibited by it to the states, are reserved to the states respectively, or to the people."
After reconsidering my position, I still think there's cynical genius in the Constitution in setting the various governmental powers in opposition to each other. With the power hungry occupied with each other, the People can pursue life, liberty and happiness mostly unmolested.
We're agreed that there's a revolution calling as I alluded earlier, though I am convinced there are peaceful solutions that won't require guns and bloodshed.
Coincident with your comments, I found my way to Gen. Wesley Clark's article Broken Engagement in the Washington Monthly thanks to Coyote Gulch.
The article is interesting in its own right, but in the context of our discussion about the Constitution, I want to apply Clark's reasoning to the notion of revolution. I agree that the popular understanding of the Constitution is largely mythical. The government of the people, by the people and for the people is a myth created by Lincoln in the Gettysburg Address. But as Gen. Clark observes, those myths have power -- "the power of our values".
The neoconservative goal was more ambitious than merely toppling dictators: By creating a democracy in Iraq, our success would, in the president's words, "send forth the news from Damascus to Tehran--that freedom can be the future of every nation," and Iraq's democracy would serve as a beacon that would ignite liberation movements and a "forward strategy of freedom" around the Middle East.
This rhetoric is undeniably inspiring. We should have pride in our history, confidence in our principles, and take security in the knowledge that we are at the epicenter of a 228-year revolution in the transformation of political systems. But recognizing the power of our values also means understanding their meaning. Freedom and dignity spring from within the human heart. They are not imposed. And inside the human heart is where the impetus for political change must be generated.
Clark also offers the following analysis of the collapse of the Soviet Union in the 80s. Bear with me, these quotes are relevant to the discussion.
The foreign policy consensus coalesced around containment, an idea which had been in the air since the early post-war period, when George Kennan, then a veteran American diplomat, published his seminal Foreign Affairs article "The Sources of Soviet Conduct." Kennan argued that the Soviet system contained within it "the seeds of its own decay." During the 1950s and 1960s, containment translated that observation into policy, holding the line against Soviet expansion with U.S. military buildups while quietly advancing a simultaneous program of cultural engagement with citizens and dissidents in countries under the Soviet thumb.
These subtler efforts mattered a great deal. The 1975 Helsinki Accords proved to be the crucial step in opening the way for the subsequent peaceful democratization of the Soviet bloc. The accords, signed by the Communist governments of the East, guaranteed individual human and political rights to all peoples and limited the authority of governments to act against their own citizens. However flimsy the human rights provisions seemed at the time, they provided a crucial platform for dissidents such as Russian physicist Andrei Sakharov. These dissidents, though often jailed and exiled, built organizations that publicized their governments' many violations of the accords, garnering Western attention and support and inspiring their countrymen with the knowledge that it was possible to stand up to the political powers that be.
With the rise of the Solidarity movement in Poland in the 1980s, it became clear once more that it would be the demands of native peoples, not military intervention from the West, that would extend democracy's reach eastward.
Clark goes on to apply this logic to Iraq, and beats up the White House for failing to understand it. I enjoyed that turn, but for our conversation, I think the US Federal system contains within it the seeds of its own decay. A bloody conflict will not be necessary, though we do need to contain the spread of federal power. You are quite correct that I hope to effect the US political system. My appeal is aimed at the hearts of US citizens "where the impetus for political change must be generated." I see these political entries in my blog as a "program of cultural engagement with citizens and dissidents" of the US and hope to inspire my countrymen to reclaim their own power and correct the gross imbalances in our country.
I am deeply suspicious of concentrations of power. Redistributing power back to the States would be a welcome decentralization of power. At this point, the States have surrendered too much of their own power to be able to reclaim it, for example the 17th Amendment you mention. But there is latent power in we the people if only we will reclaim it. Our cultural myth of the government of the people, by the people, and for the people is only a myth because the people aren't participating.
Politicians are necessarily swayed by the weight of public opinion, even if the votes themselves are a fairly flimsy form of power. In an election year, public opinion is measured by pollsters who count the opinions of registered voters. They probably further filter their numbers by those who have voted in the past and plan to vote again. If half the people are unregistered or not planning to vote, their opinions won't be reflected in the polls that measure the weight of public opinion. Voting and polls are necessary, if insufficient, measures to help contain the federal government.
I begin with three non-sequiturs.
Can I weave them into a coherent story?
I have an imaginary origin for Groovy, whether or not it actually happened this way in reality. First, I need to set the stage. I've been lurking around James Strachan and Bob McWhirter since meeting them through Jason van Zyl and Turbine and Maven. I've been subscribed to James' and Bob's blogs since I started blogging. In fact, James introduced me to the blogsphere, both because it was his blog I read first and because he was among the first to blogroll mine (thanks, James, on both counts).
When I met James he was developing Jelly. Around that time Jelly replaced Ant at the heart of Maven. In those very early days of Maven, I remember Bob complaining somewhat loudly about programming in XML. Also at that time Jason was evangelizing Avalon, Inversion of Control, and also Aspect-Oriented Programming. Coincidentally I was introduced to Jython at about the same time by Scott Shattuck, with whom I was working at PlanetCAD. Jython led me to python. Python and blogs lead to PyBlosxom (PyBlosxom just released version 1.0) A lambda in PyBlosxom lead me to finally read On Lisp. Through On Lisp I groked closures. With closures in JavaScript and SVG, I recreated my animation of a vanishing point and rekindled my interest in an animation language. Now the circle is complete.
Back to Groovy. I imagine that it was friction between James' interesting and useful work on Jelly and Bob's rightful distain of programming in angle brackets which spawned Groovy. Clearly there's some Jython, Python, and Ruby mixed in too as evidenced by the excellent use of closures and other functional goodness.
If Java brought C++ programmers half-way to lisp, I think Jython, Python, Ruby, and Groovy are going to take Java programmers another quarter of the way. XP and unit testing will help this along because unit tests do more for your code than the compiler and these dynamic languages make programming faster.
Moreover, Aspect Oriented Programming will take Java programmers another eighth of the way to lisp. CLOS has a notion of before, after, and around methods which I think map nicely to various things I've seen in AOP. But I made the assertion at work today that what people are trying to do with aspects in Java would probably be done primarily with macros in lisp. Of course I'm speaking primarily from my intuition having done vanishing little with either aspects or macros.
I suppose I'm just projecting my own programming growth path onto the Java programming community as a whole. I only recently groked closures. Understanding macros and continuations came rather quickly after that, though actually groking those will be a while unless I start doing real work in lisp.
In the past week at work, I've been rhetorically asking why not just throw out all this perl code and rewrite it in lisp? It's a lot of code to rewrite. No customers are asking us to program in lisp. None of our problems specifically demand it. Perl is a much larger community, therefore there are probably more useful libraries available, broader documentation. One thing new customers fear is vendor lock-in. When they don't know us yet, they don't trust us yet. If the code they pay for is in perl, it will be easier for them to find someone else who can work on it than if it were in lisp.
My other imaginary story is that the Java community (and by competitive pressure the VB and .NET communities too) will collectively learn different ways to think from Groovy and Jython and AOP. Those ways of thinking will lift the veils of mystery from lisp and make it into a popular enterprise programming language. Of course it could work out differently. Maybe Service Oriented Architectures will teach us all to think in terms of messages like Smalltalk programmers.
The presidential election has always been a choice of lesser evils.
I remember now. I remember how it started. [1]
I changed my registration from independent to democrat when I learned that participation in Colorado caucuses requires political affiliation -- you get to participate in the caucuses of your chosen party.
I thought that by getting involved in the political process earlier, I might help improve the choices. Unfortunately, I've still never participated in the caucuses. This year it became clear that the presidential choices are made well before the Colorado caucuses.
So much for that idea. Maybe I should change my registration back to independent if I'm not going to make it to the caucuses anyway. Sigh.
The balance of power in the US has definitely lost its balance. I think the main problem is that voters have checked out of the process. This has two effects I want to discuss here. First, by choosing not to vote, citizens surrender their own power to those who hold power. It makes the power-hungry more powerful than they would otherwise be. Second, I think our leadership are more extreme than the political views of the general population and I think voter apathy is the cause.
The Constitution divided power into four groups. The government was divided into Executive, Legislative, and Judicial branches, and the People were given the power to determine who would lead the Executive and Legislative branches and the power to dismiss some in the Judiciary the members of the House of Representatives. The Bill of Rights granted the people other rights and powers. The cynical genius of the Founders was in setting the power-hungry in opposition to each other. Machiavelli's (and Thucydides' and Homer's) advice is deeply honored: power corrupts, absolute power corrupts absolutely. Dividing the power sets the inevitably corrupt power-mongers against each other, thereby limiting the damage any can do to the citizenry, and limiting the damage they can do to one another. But the voters are the lest power hungry of the bunch and a critical check against abuses of power among the three branches of government. If the voters excuse themselves of their responsibilities, then perhaps the most important stabilizing influence in the system fails. The powerful assume greater power and the potential damage from the inevitable corruption increases.
The second effect makes matters worse. Political views in America almost certainly fall into a traditional bell curve. The majority of the people are in the middle and a minority are in the extremes whether Democrat or Republican. Those at the extremes are voting. The half of the country that does not vote comes from the middle. This has the effect of flattening the bell curve and thereby making the extremes considerably more important to the politicians. We end up with leadership that reflects the more extreme political views on both sides. The campaigns will play to the middle, particularly when the extremes are so evenly divided. But their actions when in office will betray their more extreme leanings. If everyone were voting the curve of the voting population would more closely align with that of the general population. The leadership would have to respect the middle in action, not just words. We would end up with leaders more closely aligned with generally more moderate views of the country as a whole. Instead what we have is a political bell curve imbalanced toward the extremes.
These two effects leave us with more extreme leaders, more powerful leaders, and by Machiavellian law, more severe corruption.
Negative campaigns tend to turn off those who might otherwise vote. This works to the advantage of the extremists, because those who abandon their right to vote are in the middle of the curve.
Please know that your vote counts. Politicians have to consider how their actions effect voters. There are all sorts of people polling voters to see how the campaigns are going. If you are not a voter then your views don't show up in the polls and the politicians will behave accordingly.
Please help to restore the balance of power in the US. Reclaim your power -- however small or irrelevant it may seem. Please educate yourself about the issues and the candidates. Compare them to your own values and vote for the lesser evil -- whichever you think that is. Abstaining from the vote is not an effective form of protest. You don't show them anything by letting the extremes of the political spectrum decide.
[1] I'm intentionally alluding to "I Remember Now" from Operation: Mindcrime by Queensryche. The album begins:
"It's ten minutes past curfew. Why are you still up? Hallo? Hallo? Perhaps you need another shot. That should do it. Sweet dreams, you bastard."
"I remember now. I remember how it started. I can't remember yesterday. I just remember doing...
what they told me. (...told me ...told me ...told me)"
For you poor souls who aren't familiar with "Operation: Mindcrime" here are a few other choice lyrics from Revolution Calling to help you understand to what I'm alluding:
...
Got no love for politicians, Or that crazy scene in D.C.
It's just a power mad town
But the time is ripe for changes, There's a growing feeling
That taking a chance on a new kind of vision is due
I used to trust the media to tell me the truth, tell us the truth
But now I've seen the payoffs everywhere I look
Who do you trust when everyone's a crook?
Revolution calling ...
...
I used to think that only America's way, way was right
But now the holy dollar rules everybody's lives
Gotta make a million doesn't matter who dies
Revolution calling ...
I've been chewing on these thoughts for many weeks. I'm enraged and outraged by our occupation of Iraq. I doubt ranting about it will help, but I know remaining silent won't.
War is an excuse to ignore our morals. We do these unspeakable things when we are at war. Our leaders publicly wring their hands and rationalize about Justice and Democracy and nevertheless deny due process of the law to "enemy combatants". The most admirable rhetoric about Iraqi liberation sent American youth out to kill or be killed.
At least in World War II it was the Japanese who attacked Pearl Harbor and the Germans who invaded Poland and Austria. In those days we were clearly defending ourselves and our allies. By contrast Iraq was never a threat to the United States and barely a threat to their neighbors. They didn't have weapons of mass destruction and had nothing whatsoever to do with September 11th.
This time we started the war. We the People of the United States invaded Iraq without provocation.
The country we attacked was enormously weakened. They had been severely beaten in the Gulf War and then endured more than a decade of international sanctions. It's like a professional heavy-weight boxer picking a fight with a malnourished junior high school bully. At least through the Cold War we were standing up to another superpower.
Did I hear you right, Mr. Bush? We had no choice? The most powerful country in the world was forced into this war by a deceptive, third world dictator? How can we be the most powerful and yet so powerless?
Saddam Hussein was probably the most successful mass murderer in history with the help of his regime. He imprisoned, tortured, or killed people on the mere suspicion that they were plotting against him -- no due process, no burden of proof, just suspicion. We rationalize "liberation of the Iraqi people was reason enough to invade Iraq." The ends of Iraqi liberation justify the means of unprovoked, unilateral military invasion and occupation.
Unfortunately there is no end, only changes. Nothing actually ends, so there is nothing to provide justification for our unsavory means. One evil action leads to another. It should come as little surprise that Americans tortured Iraqis.
More cynically, We the People of the United States showed little concern for the Iraqi people for two decades of Saddam's rule in Iraq. Suddenly in the last two years we developed a national conscience? Somehow September 11th snapped us out of the insensitive spell we were under? No. We invaded Iraq because we were scared of them. That's right. We, the professional heavy-weight boxer, were afraid of the malnourished junior high school bully. So we went in an kicked his ass. Very courageous. Very admirable. After the fact, we discover that the gun we thought he was holding wasn't really a gun. "Shoot first, ask questions later" is not the moral ground on which the United States was founded.
We took "preemptive action" in our own defense. In case it's not obvious, that is an oxymoron. You can't pick the fight and call it self-defense. You start it, you are the attacker, not the defender.
What we have done in Iraq is replace Saddam Hussein, not remove him. In the microcosm of Iraq, we have taken on the responsibility of controlling the Shi'a, Sunni and Kurds. To be sure we're kinder and gentler than the Ba'th party, but only barely. We shut down newspapers when they encourage opposition to our rule. When it suits us, we take prisoners to countries which lack legal protection against violent interrogation. We even torture and humiliate prisoners on Iraqi soil. None of these actions represent our moral standards.
We abandoned our morals in unilaterally taking over another country. In the geopolitical macrocosm, we are the ruthless dictator. The US Government executed the Iraqi Government on the suspicion that they were stockpiling weapons of mass destruction and training terrorists. We also did it to send a message to the rest of the world. Shock and awe. We hoped to scare the rest of the Middle East into submission with the implied threat of other executions.
Tell me again about the Axis of Evil, Mr. Bush.
No matter what happens next, we cannot remove the stain on our own conscience and enormous damage to our reputation for having abused our power.
The actions of our government are the actions "of the people, by the people, and for the people." The blood, abuse, and torture is on our hands. We have abused our power and violated the world's trust. Words will not repair the damage. Our actions must change.
Yippie! I get to tell you "I told you so."
About this time last year I suggested using digital identity on mail servers to cut down on spam. Yahoo! has added an excellent dimension with DomainKeys. They suggest ISPs publish their public keys via DNS since there are already control measures in place to ensure that only the domain owner can modify their DNS records. Thanks to Jeremy Zawodny via Simon Willison.
It's not fool proof, but it immediately gives Yahoo! a leg up on their competition. It will get harder for spammers to spoof Yahoo! addresses in the From: line. And competition may compel others among the big ISPs to adopt DomainKeys. Basically, Yahoo! is a big enough player that their endorsement may encourage wider adoption of the technique. I don't expect to see a drop in spam overnight, but it looks like a step in the right direction.
Thanks to Jeff for posting this info on the COMUG mailing list and thanks also to to Simon's other pointers regarding the Mac OS X help url exploit. Update: It is important that you follow this procedure for each user on your Mac -- the protocol helper preferences are stored separately for each user (more details at the bottom).
Mac OS X handles various URLs specially. By default 'help' urls are handled by Apple's Help Viewer which has the feature of allowing you to run an arbitrary script or shell command. See Simon's post for links to harmless examples of the exploit, links to more detailed discussion, and to his preferred fix.
The countermeasure is to change the application that handles 'help' urls to something other than Help Viewer. The fixes Jeff and Simon pointed to both require downloads, and from reading the more detailed discussion, I think Simon's is a more complete solution. But it turns out you don't have to download anything, though I had to swallow a little pride.
Internet Explorer allows you to change the protocol helpers for the OS.
The fix is very easy. Here's the very granular step-by-step instructions (don't be daunted by the number of steps -- I described every button click :-). I tested this with OS X 10.3.3 and IE 5.2.3 -- the steps might differ on other versions.
Simon's post also mentions creating a helper for the 'disk' protocol. The Finder will happily mount remote disk images from a URL which could be used in a two-part exploit: in one link you would mount a nasty script in a known location on the victim's computer via the 'disk' url and in the other link you would use the 'help' url to launch that nasty script. However, 'ftp' urls will also be mounted by the Finder, and I'm fairly sure that the .Mac iDisk uses a 'webdav' url. There may well be others I don't know about. Disabling the 'help' url vulnerability will at least prevent a link on a web page from launching a nasty script. Breaking those other urls is more than I'm willing to do on my own system. Having said that, expect to see some social engineering emails trying to get clicking assistance from naive Mac users.
Update: Another member of COMUG, Dan Oetting mentioned in another email that these preferences need to be set for each user. So I dug a little harder to see where they might be stored. It looks like they are here: ~/Library/Preferences/com.apple.internetconfig.plist. That might open the possibility of creating a script to change the settings if you have a lot of macs to fix. It's not a trivial change because the values of the protocol helpers are stored in some encoded format.
Quick XML rant: what is the point of using an XML file for preferences if you're going to store values using some other encoded format? If Apple had gone a little further xml-ifying the Internet Config preferences file, someone would already have written the script to automate the change for networks of many computers, or Macs with many user accounts.
Over the weekend I thought of a useful way to visualize the problem I described in the previous post. Noting it here will hopefully spare someone in the future (maybe even me) from the pain I went through getting here.
The goal is to import data from twelve years of backups of the database our software replaced. But exactly which data do I import? Data changes over time, how has this particular pile of data changed? The backups are MS Access databases. Each year's backup is a collection of tables.
Essentially I have three dimensions of change:
This three-dimensional metaphor helps me organize my approach to the data.
Is that breakdown too trivial to bother mentioning? It seems pretty obvious in retrospect. Prior to describing it in this way, I just had a big ball of mud, and my efforts to understand the mud just spread the mud around. No meaningful shape emerged in the data, just more confusion. I could answer some questions, but my questions were not getting me any closer to getting the data imported. With this three-dimensional metaphor in mind, my questions and their answers are getting somewhere. I can focus my attention on particular lines and planes.
For a given backup, the columns in the tables are constant, but the data in the rows may be inconsistent. Are the dates all in the same format? Are there missing dates? Do the boolean columns use 1/0 or Y/N? Are they consistent about 1/0? Can I uniquely identify each row, or are there redundant records? None of these questions have anything to do with the time dimension. Their answers lead me toward the code I'll need to use to massage the data into a consistent form that I can import.
Slicing another way I can ask which columns were added over time. Load the column names for a particular table in each year's backup, collect the unique column names, and track in which years they appear. Which columns exist in all the backups, and which ones don't? Those that exist in all the backups will be useful for identifying unique records over time. For those that don't, do I need a default value to apply to the older records, or can I ignore the column in the newer records?
The data are mostly consistent, but there are areas that will need human intervention before they can be imported. I'll automate as much as I can and collect the records which break my automation -- bring the exceptions to my attention so I can massage them into shape.
There are relationships between the tables. Some clearly rely on primary keys and foreign keys. Some don't, which adds another dimension to the problem. Thinking in dimensions is getting somewhere.
I call it flailing. Intuition tells me the problem on which I'm working has an easy solution, but it evades my every turn. An elusive solution teases me along dead end after dead end.
I have to import data from twelve years of backups. I don't expect that to be easy. But some steps should be. Our client's original system was built in Access. The files were archived annually with names like DB1990.mdb, DB1991.mdb, DB1992.mdb and so on through 2002. There is a Win32 laptop at work, but Access isn't installed: DBI and DBD::ODBC to the rescue. Though I haven't used DBI before, JDBC translates quite well. I work with a bunch of unix geeks, so cygwin is installed. I thankfully didn't have to re-traumatize my fingers with DOS.
This "one"-liner prints out the tables from one data source.
% perl -MDBI -e'$dbh=DBI->connect("dbi:ODBC:DB1990"); \
print join("\n", @{$dbh->tables}), "\n";'I can ignore most of the tables -- reports and summaries. But I can't ignore BROKER and BROKER90 nor POLICY and POLICY90 and so on. The other backups include tables like FOO and FOO91 and FOO and FOO92. The challenge is to weed out the redundant data from all those backups and to clean the inevitable data inconsistencies. A few more perl scripts and the mighty diff revealed some columns were added over time but the schema is mostly stable for the first 10 years with some bigger changes in the last two.
What's the simplest thing that could possibly work?
I thought about dumping the data into tab delimited tables and groveling over it with regular expressions. But it was already in a structured form and I could use SQL instead of regular expressions. Leaving the data where it was seemed like less work, so I started banging out some one-off scripts using DBI. Simple. The repetition quickly got tedious, and Rob routinely advocates once-and-only-once, so I made a slightly interactive script and started reusing.
1 while(dispatch_command());
sub dispatch_command {
print $prompt;
my $cmd = <STDIN>;
chomp($cmd);
$cmd =~ s/;\s*$//;
$cmd =~ m/^([^ ]+) /;
my $method = lc($1);
return $dispatch->{$meth}->($cmd);
}Subroutines referenced in the $dispatch table handled specific commands: connecting to a data source, displaying metadata, or an SQL select. I didn't get there immediately. I just added a simple thing here and there. I removed a little repitition. I ended up with a rudamentary perl SQL shell. That turns out not to be the simplest thing that could possibly work. But it was a chain of simplest things that lead me there. And I was kinda proud of dispatch_command.
The title of this post comes from one of Dr. Veltman's good friends.
True genius is knowing when to stop. Many people have ideas, but the genius quickly abandons the bad ones, whereas the rest of us press on long after an idea has revealed itself as a failure.
For the time I spent I should have just bought a copy of Access and installed it. I talked with Rob about it later. His first suggestion was looking for something on CPAN -- except that our internet connection was down that day (more telecom frustration). His second suggestion was also more simple: import the raw data into postgresql and use psql.
The next day I decided to abaondon the interactive approach and just see what I could get out of the first database. I thought it should be pretty easy, and by this time I needed a small success to build back some momentum. What were the differences between POLICY and POLICY90? I dumped the two tables into tab-delimited text files. perl warned me about uninitialized values in string concatination, so I replaced undefined values with --null--. But diff didn't tell me much. It compares line-by-line and each line was different -- more columns in one than the other. While I was talking with Rob about something else he asked about the --null-- things. "If you just write out perl, you can eval it later and just leave those as undef. Don't invent your own syntax." Sometimes it's the little things. Actually, um... I did that on purpose. See, I knew I'd learn another cool trick from Rob if I put --null-- in there. ;-)
Switching to Data::Dumper cut my code down dramatically. The lesson from Rob is if you just use perl you don't have to write a parser. Data::Dumper is my new friend. It also put each 'cell' on its own line. That made diff -ub much more helpful. But I was still visually inspecting the diffs. I wanted a mechanical way of confirming that the extra columns contained all null values.
I wrote an emacs macro to repeatedly search over the diff file. I thought it might stop when it couldn't find the pattern in marked region, but instead it just expanded the region. Took a while to follow that dead end. I tried using emacs diff-mode to see if it would make the macro any easier. That lead me to ediff and emerge. But the files I was diffing were 12MB each and emacs kept crashing -- "don't know my own strength". I tried various other ways of bending emacs to the task thinking it would be simpler than writing a perl script to do the same. Finally after repeated failures I asked David if he new an emacs incantation that would help. "Why not use perl to look at the table with the extra columns and just grep for nulls?"
That would be simple.
$result = do 'DB1990.dump';
print "hooray, they're all null\n"
if (int(@$result) == grep(!defined($_->[21]), @$result));How do you you learn to recognize the dead-ends before you get to the end? Is it just a matter of experience? Or does it actually require genius?