MySQL Table Exists (1050) but Doesn't Exist (1051)

Ran into an interesting issue this morning:

mysql> drop table RELATIONSHIP_TYPE;
ERROR 1051 (42S02): Unknown table 'relationship_type'

Which wouldn't be that weird except for:

mysql> CREATE TABLE `RELATIONSHIP_TYPE` (`relationshipTypeId` INT UNSIGNED AUTO_INCREMENT  NOT NULL, `name` VARCHAR(10) NOT NULL, `code` VARCHAR(2) NOT NULL, CONSTRAINT `PK_RELATIONSHIP_TYPE` PRIMARY KEY (`relationshipTypeId`)) ENGINE INNODB;
ERROR 1050 (42S01): Table 'relationship_type' already exists

After a fair amount of Google searching, I finally found an answer that implied I should try this:

$ mysqladmin flush-tables

Which seems to have solved the problem. I'm posting this in the hopes that the next person will have an easier time finding the answer.

Maven Growl

I've written up a small script for displaying Maven build notifications using Growl, and put it up on Github.

On Releasing an Open-Source Project (Moo v1.0)

Like many developers, I’ve used a large number of open-source projects over the years to get my work done. Also like many developers, I’ve worked for companies that were wary of contributing to open-source projects to any significant degree. Worse, several of these companies attempted to claim the rights to any software built while employed by them, without regard to whether this was built for them, or on their time. These attitudes are common, suspect, and should probably be the subject of another post.

I’ve returned to doing freelance software development and software consulting this year (and, hey, if you need someone to help you with software development, drop me a line). Although working for yourself has some drawbacks, it does come with a lot of freedom. That includes the freedom to work on open-source projects in unbillable time (or, if you have an understanding with your client, even potentially in billable time).

I’ve been taking advantage of that recently, by building a project to map objects to objects, called Moo. I’ve been on projects that have needed something like this a number of times, and teams I’ve been on have built something very similar to this twice before. I’m really hoping this is the last time I’ll have to write this particular piece of code, because from now on I’ll be able to adopt my own code, even if I’m working for another company.

In order to increase the odds that I’ll be able to adopt it, and to help as many other teams as I can, I’ve used the BSD license. That means you can bury Moo inside your own work and you don’t have to pay me, or adjust the license of your project. If you want to credit me in some way, that’d be cool, but mostly I just want to make sure that, if Moo is useful to you (or to me!) there aren’t any obstacles to that.

Moo v1.0 Released

I’m happy to announce the release of Moo v1.0, an open-source, BSD-licensed project for Mapping Objects to Objects.

What’s It For?

Moo allows you to map objects to objects, by copying data from one object or object graph and creating or updating another object/graph. Along the way, Moo can do translations, flattening objects, calling methods, and otherwise helping to map objects that aren’t identical copies of each other.

Moo is the sort of tool you use to work around a problem, not fix it. If you find yourself wanting something like Moo, you should probably ask yourself — am I just papering over an architectural issue or a limitation of a framework that I should be trying to resolve in some other way?

That said, I’ve needed Moo before and I’ll need it again; some problems are too much trouble to fix when there’s a reasonable workaround. I’ve used Moo for data transfer objects (DTOs) and when serializing an object/graph.

When possible, I like to use a rich domain model throughout an application, rather than using a domain-model in some kind of service layer and data transfer objects in the interface tier. But there are times when data transfer objects are more palatable. For instance, if you find yourself on a project using GWT and Hibernate together, you’ll discover that GWT serialization and Hibernate proxy objects aren’t a match made in heaven, and you might find yourself wanting some data transfer objects.

If you like to use well-connected object models, you’ll also find that the default serialization approach for GWT and lots of web service frameworks don’t have great mechanisms for tweaking what parts of the object graph you return, particularly if you want to make those decisions differently depending on the service method being invoked.

If you’re using Moo, you can have a domain model object like User, and return List from lists and queries, and return a UserDetails object from a more detailed ‘get’ query. I’ve written an example like this into the User Guide for Moo.

What Did You Learn?

Well, for starters, GitHub’s pretty cool. I considered using Google Code with Mercurial, or bitbucket, but I wanted to spend some more time with git and GitHub. I’m glad I did, I’ve been really pleased with GitHub. I’m not yet sold on its issue tracker, but for now I’m using all the pieces simply because it’s nice to have them all integrated.

And Sonatype’s Maven repository hosting for open-source projects is great, since it uses Nexus, and syncs with central.

On the other hand, I hadn’t signed Maven artifacts with the maven-gpg-plugin before, and as is typically with less-used Maven plugins, there were a couple significant bugs that I had to track down before I could deploy a GPG-signed release.

Next Up

I have a few more open-source projects planned, starting with some work I did to support concurrent testing using JUnit. I’ll be mixing up this kind of open-source work with side projects and paying gigs, so I’m not making any guarantees about timelines.

If you do end up using Moo, drop me a line, I’d love to hear about it.

Ignite Toronto 1

The first Ignite Toronto went off on Tue 25-Aug-2009 at the Drake Hotel Underground. I was able to shoot a few photos:


All in all, the ignites were enjoyable, although I do wish the ignites had started sooner. There was ample time to drink and discuss in advance, but I'd like time to see all the ignites, and then talk some more afterwards without wishing I were home, asleep.

I'm getting old, I know.

Of particular amusement was when the Creemore Springs tap decided to fail very publicly with a bit of a Beersplosion:

GuestList - Ben Vinegar - DemoCamp Toronto 21

Ben Vinegar presented Guestlist at DemoCamp Toronto 21 (Velma Rogers Theatre; 333 Bloor Street East; 28-Jul-2009):


If you'd like to see the large/HD version, click through to YouTube. I also recorded the Q&A that followed (YouTube):


Ben had a self-deprecating style which initially made me wonder if the presentation was going to bomb, but he pulled it off with aplomb, and it ended up being one of the more entertaining demos of the night.

Guestlist is one of the more polished and promising applications I've seen at DemoCamp, and I expect this won't be the last time you hear something about it. I'd say it has a relatively bright future in front of it. It's not a unique product, it's just very well executed.

Apparently the Guestlist folks mostly (all?) work at FreshBooks.

Knight's Tour in Scala

I've been doing some experimentation with Scala lately, mostly by reading the Wampler / Payne Programming Scala in O'Reilly Labs. Learning a language by reading about it is deceptive, so I've been looking for an excuse to write some non-trivial Scala code with which to better exercise and test my knowledge of the language.

After skimming the Daily WTF's Praxis for Automating the Knight's Tour, it seemed like a tractable problem that would be fun to code up in Scala, so I proceeded to do just that. It's a brute-force solution to the problem, and I'm not yet sure how idiomatic my Scala code is, but it does, at least, solve the problem:

class KnightsTour(val dimensions : Pair[int,int], val startPosition : Pair[int,int] ) {

  val relativeMoves = List( Pair(2,1), Pair(2,-1), Pair(-2,1), Pair(-2,-1), Pair(1,2), Pair(-1,2), Pair(1,-2), Pair(-1,-2) )
  val xPositions = 0 until dimensions._1
  val yPositions = 0 until dimensions._2
  val tourLength = dimensions._1 * dimensions._2;
  var closedTours = 0;
  var openTours = 0;

  def printPaths() : Unit = {
    printChildPaths( List(startPosition) )
    println( "There were " + ( openTours + closedTours ) + " tours (" + openTours + " open, " + closedTours + " closed)." );
  }

  def printChildPaths( path : List[ Pair[int,int] ] ) : Unit = {
    val moves = getMoves( path.first );

    if( path.size == tourLength ) {
      if( moves.contains(startPosition) ) {
        closedTours += 1
        println( "Found a closed tour: " + path )
      } else {
        openTours += 1
        println( "Found an open tour: " + path )
      }
    } else {
      moves.filter( move => !path.contains(move) ).foreach( move => printChildPaths( move :: path ) )
    }
  }

  def getMoves( position: Pair[int,int] ) : List[Pair[int,int]] = {
    relativeMoves.map( move => Pair(position._1 + move._1, position._2 + move._2 ) ).filter( withinDimensions )
  }

  def withinDimensions( position: Pair[int,int] ) : boolean = {
    return xPositions.contains( position._1 ) && yPositions.contains( position._2 )
  }

}

val tour = new KnightsTour( Pair(3,4), Pair(0,0) )
tour.printPaths();

All in all, I'm pleased with it, although I'm sure i'll read this again later and wince at my early Scala code. Comments welcome.

WeGoWeGo - Dan Wood & Martin Pietrzak - DemoCamp Toronto 21

Dan Wood and Martin Pietrzak presented WeGoWeGo at DemoCamp Toronto 21 (Velma Rogers Theatre, 333 Bloor Street East; 28-Jul-2009):


I also captured the Q&A session that followed:


As always, you can feel free to click through to YouTube (presentation, Q&A) to watch the larger HD stream.

They do a user-generated content site for listing things to do. They attempt to differentiate through the search model ("faceted search categorization"). I have to say, this just didn't seem like it was currently differentiated enough to grab my attention. There's a lot of web sites in this space, and I'm not convinced the search is sufficient, particularly in a world where network effects create a sort of lock-in, giving the first-movers a bit of an edge.

That said, I'm happy to be proven wrong.

Mashup Arts - Shaun MacDonald - DemoCamp Toronto 21

Shaun MacDonald presented Mashup Arts at DemoCamp Toronto 21 (July 28th, 2009; Velma Rogers Theatre; 333 Bloor Street East):



If you'd like to see the video larger in HD, feel free to click through to YouTube.

MashupArts is essentially rich media e-cards whereby you load videos and photos from the web into your card and then share it, and get others to collaborate with you on the card before sending it to the victim/recipient.

The concept seems sound, although I think there's still a fair ways to go before this is a polished, fully-featured application. There are features that I'd like to see added, and parts of the interface felt like they needed further work, but as long as it continues to improve, this seems like something that has potential to succeed.

Breeze - Alan Lynse / Cascada Mobile - DemoCamp Toronto 21

Alan Lynse of Cascada Mobile presented Breeze at DemoCamp Toronto 21 at the Velma Rogers Theatre (333 Bloor Street East) on July 28th, 2009. Breeze is a platform for building mobile applications using web technologies that run on a wide array of mobile phone platforms (iPhone, Android, etc.):


In this case, not only can you click through to YouTube, I recommend that you do, since part of this demonstration involves coding in Eclipse, which is a lot easier on your eyes if you switch to the larger HD mode.

Seems like a good way for companies to build simple applications that cross all handsets. If you want the polish of a fully native application that's tailored to the capabilities and design of a particular device, this probably won't cut it, but not everybody needs that, particularly for the first pass.

I don't see this surplanting native mobile application development any time soon, but it's a useful option to keep in mind.

HomeStars.com - Brian Sharwood - DemoCamp Toronto 21

Brian Sharwood presented HomeStars.com at DemoCamp Toronto 21 (July 28th, 2009; Velma Rogers Theatre, 333 Bloor Street East):


Feel free to click through to YouTube to see it in full-size HD.

Some good anecdotes, and the site functionality /sounds/ good, but there wasn't much in the way of a demo here. More demo, less presentation would have been good.

That said, I would check out the site if I needed some work done and wasn't going to do it myself.

Twitter Background Credits

I've recently redone my twitter background (which you can see on my twitter page), and in so doing used a few images that, while creative-commons-licensed, should be attributed. Because twitter pages can't easily hold credit links, it seemed worth doing here, rather than there.

Accordingly, you can find the images I used here:

• Coffee Stain
• Wood Grain
  

The perforated tractor-feed paper and polaroid were photoshopped together, rather than clipped; they're not perfect, but they're good enough for my purposes.

ArtAnywhere - Christine Renaud - DemoCamp Toronto 21

Christine Renaud presented ArtAnywhere at DemoCamp Toronto 21, July 28th, 2009 at the Velma Rogers theatre:


In essence, ArtAnywhere allows artists to rent their art on a monthly basis to people with an interest in art. The cost of renting is fixed, so more expensive work is simply rented for more months in total before it is "paid off." Sounds promising, although it sounded during the question period like they haven't worked out all the details with respect to how to deal with damage done to art, and so forth.

Still, it could be of benefit to all (revenue for artists, art for the general populace), and there's obviously a profit model, so it'll be interesting to see where this goes.

Zoocasa, presented by Saul Colt at DemoCamp Toronto 21

Saul Colt presented Zoocasa at DemoCamp Toronto 21 on Wednesday July 28th, 2009 at the Velma Rogers Theatre. Zoocasa is a site to search for real-estate in Canada. They argue they're complementary to other real-estate sites like Realtor.ca, and differentiate themselves by ways to search for houses:



You can click-through to YouTube to see the video in larger form; it's available in HD.

They get their listings from the public web, which means that they have some listings that MLS doesn't have, and don't have all the listings that MLS does have. This is their achilles heel; if MLS data were publicly available, Zoocasa might well be a better choice than realtor.ca, but since the real estate business keeps their listings under lock and key, there's only so far Zoocasa can go.

There was a spirited question period. In particular, an audience-member who felt that Zoocasa needed to consider alternate forms of profit-model. I wasn't capturing questions in order to save battery life, which is a position this presentation made me want to revise.

On a side note, I shot the video pretty closely cropped, which doesn't give iMovie much image stabilization to work with, so it's very shaky-cam; I'll have to shoot it a little wider in the future. A tripod would help, but would probably get in the way of catching both the slides and the video in concert.

'elmcity' by Jon Udell @ DemoCamp Toronto 21

Jon Udell was the first presenter at DemoCamp Toronto 21, covering 'elmcity', his project to allow a directory of events by linking in iCal format using delicious accounts as markers for finding the iCal files.

Despite a proliferation of event information on the web in various forms, it's not always easy to bring that together in a useful way. Although "elmcity" seems like an interesting project, I'm not sure that I see elmcity changing that directly. I agree with the problem statement, but the demo didn't make me want to stand up and ask, "How can I get access to this right away?"

Jon took a little longer than most presentations, and thus is longer than YouTube's 10-minute limit, so I've broken his presentation up into two videos:

DemoCamp Toronto 21: Introduction

DemoCamp Toronto 21 (July 28th, 2009) took place at the Velma Rogers Theatre at 333 Bloor St. Joey DeVilla (@AccordionGuy), David Crow (@davidcrow) and Jay Goldman (@jaygoldman) kicked it off with the initial introductions:



The sessions for the night were:

• Jon Udell, Microsoft, elmcity
• Saul Colt, Zoocasa
• Christine Renaud, ArtAnywhere
• Brian Sharwood, Homestars.com
• Alan Lynse, Cascada Mobile
• Shaun McDonald, Mashup Arts
• Dan Wood, WeGoWeGo
• Ben Vinegar, Guestlistapp.com
• Jason Roks, guiGOOG
  

As well, there slides showing upcoming events:

• Startup Drinks (July 29th)
• WiredWedTO (Aug 12th)
• #HoHOTo (Aug 18th)
• Ignite Toronto (Aug 25th)
• TEDxTO (Sep 10th)
  

And a few job listings:

• Unsyced - Lead Developer (Waterloo)
• FileMobile - Director, Technology & Operations
• Aoulous - Software Developer
• Sericon Technology - Facebook Application Testing
  

Midway through the presentations, there was a short break for pizza, cookies and drinks:

Although I focused on capturing the presentations, I was able to shoot a few photos:


Videos of the presentations to come.

Fixing Hessian Flex 3.2.0 References

If you're using Hessian Flex 3.2.0 and you'd like to fix the problem with references, the following one-line fix seems to do the job for me:

$ svn diff
Index: src/main/flex/hessian/io/Hessian2Input.as
===================================================================
--- src/main/flex/hessian/io/Hessian2Input.as (revision 1172)
+++ src/main/flex/hessian/io/Hessian2Input.as (working copy)
@@ -129,9 +129,10 @@
     public override function init(di:IDataInput):void
     {
       _di = di;
       _buffer = new ByteArray();
       _offset = 0;
       _length = 0;
+      _refs = null;
     }

Basically, HessianOperation hangs on to a Hessian2Input class between invocations, and calls init() to clear out the state before using the class for another invocation. The init() doesn't currently clear the reference cache. By setting _refs to null, you clear the cache and Hessian2Input will simply instantiate a new array in its place if and when it needs to do so.

Dependency Trap - Hessian, Spring, Tapestry

A project I'm doing some work with is using Hessian, Spring and Tapestry, and we've just discovered that we're deep in a dependency trap, as follows:

1. Hessian, a binary remote-invocation protocol written by Caucho, avoids cycles and reduces bandwidth by only serializing objects once per request by using object references. Unfortunately, hessian-flex 3.2.0 and below have a problem with references such that they have a tendency to point to the wrong object. (See: my initial report of a bug in references, confirmation of references bug).
2. There's a new version of hessian-flex, hessian-flex-4.0.0, which solves this reference problem, but it's only compatible with hessian-3.2.1 and above. (See: hessian-flex-4.0 posted)
   
3. Hessian-3.2.1 and above do not work with the Spring Framework's remoting framework in Spring 2.5.6; there's been some motion to resolve this in Spring 3.X, but that may not be complete. Further, hessian-3.2.1 itself has a bug. (See: summary of the current state, spring-framework bug for supporting later hessian versions, hessian null-pointer bug)
4. Tapestry-spring, which integrates the spring framework's context into Tapestry's IoC, apparently doesn't play well with Spring 3.x. (See: T5.1/Spring/Antlr issue, another issue I posted myself, but which isn't yet in the list archives).

So, we can:

1. Continue with the dependencies we have, which leave us with broken hessian references.
2. Upgrade to a newer hessian-flex, which requires a newer spring framework, possibly one that isn't even made yet, and which isn't compatible with Tapestry.
3. Fix hessian-flex-3.2.0 ourselves to avoid the reference issue.
4. Write our own hessian remoting (or patch existing) for springframework 2.5.6 to export services using hessian-3.2.1.
5. Fix tapestry-spring to work with Spring 3.x.
6. Abandon hessian in favor of Spring-BlazeDS.
7. Other equally painful choices, probably.

It's always nice to have choices.

Flex Builder 3 on Eclipse 3.5 (Gallileo) on OS X?

I spent a little time this morning trying to get FlexBuilder 3 up and running in Eclipse 3.5 (Gallileo). A little experimentation and reading implied that getting it up in the Cocoa version of Eclipse wasn't going to happen.

I then tried a few more times with Eclipse 3.5 carbon, and still wasn't able to get it working. So if you've managed to get FlexBuilder 3.X up and running on Eclipse 3.5 on OS X, I'd be happy to hear more.

Caucho's OSGi Pains

Earlier this month, I commented on Atlassian's OSGi experiences, and how it seemed that most of these experiences recount the pain of getting up-close-and-personal with OSGi. I've just noticed that Caucho has a similar tale of woe; they considered using OSGi within their application server and eventually rejected it, at least for the time being.

Now, you could of course argue that this a way for Caucho to explain the lack of OSGi support within their application server. I'm not trying to position their tale in any way. I don't have enough OSGi experience to agree or disagree with the points they raise, I'm just trying to synthesize from the experiences of others, and that seems to come down to this:

If you read detailed adventures of people's experiences with OSGi, it seems to come with a fair amount of pain, so keep that in mind when you're looking at OSGi for your own purposes.

Java for OS X Update 4, Java Preferences and JAVA_HOME

Java for Mac OS X 10.5 Update 4 is out, and if you're on Mac OS X 10.5, you should install it as soon as possible; it patches the serious vulnerabilities that existed in Java for OS X before that.

However, if you're unfortunate, installing this will also mess up your ability to run different versions of Java. On OS X, the version of Java you run by default can be controlled using the Java Preferences panel:

Unfortunately, I quickly discovered that this was no longer true on my system. Although Java 1.6 was heading up the pack, Java 1.5 seemed to be the VM of choice when compiling and running Java code. And since I'm on a project that's using Java 1.6, this was a serious problem.

Changing the value in the Java Preferences application had no effect. Neither did rebooting. I briefly considered reconfiguring the Java setup using softlinks, but it seemed like that was a last resort as it would be further breaking the Java preferences panel. I did some digging and didn't find very much at first.

Eventually I discovered this post on Apple's "java-dev" mailing list talking about changes to the way JAVA_HOME works on OS X, and started experimenting. I discovered:

• The value of JAVA_HOME was pointing to Java 1.5 on my system.
• Altering JAVA_HOME had an immediate and significant impact on what version of Java was used to run 'java' or 'javac' by the executeables that were softlinked by OS X.
• The output of the new 'java_home' command was influenced by whatever went into the Java Preferences panel.

Accordingly, I did as the mailing list suggested, and set JAVA_HOME to `/usr/libexec/java_home` and voila, everything seems to have resolved itself. If any of you run into this, hopefully you'll find success with this solution as well.

. In order to better support JAVA_HOME in OSX there have been some changes. Out of curiosity, I switched the order of the JVMs in Java Preferences and ran the new 'java_home' command and the results would change. I also tried changing the value of the JAVA_HOME environment variable, which immediately affected which version of Java would be run by the 'java' and 'javac' commands.

I'm guessing that I had a JAVA_HOME setting already, pointing to Java 1.5, and it was this value that was overriding whatever was in my Java Preferences. Adjusting JAVA_HOME to be the result of the java_home invocation seems to have solved my problems.

I'm hoping that by posting this I can save some of you the same pain.

PHP Considered Harmful

I noticed this morning (via @PeterBell) that PHP was getting a goto statement in v5.3. Some would argue that the Goto statement has been considered harmful since the late '60s, if not before.

OSGI Case Studies == Pain

Every time I see someone give both sides of their OSGi tale, it reminds me a lot of my EJB 1.x experiences, and I remind myself to stay away unless I absolutely have to.

At QCon (2007?), I saw a presentation on Spring Dynamic Modules. They pointed out some of the issues you might have with OSGi and how Spring DM might help. Mostly, it made me want to avoid OSGi.

I've just quickly skimmed Atlassian's presentation on how they've used OSGi for their plugin architecture, and while I'm sure it has provided them with benefits, and it may be worth it for their needs, my key takeaway is still, "stay away from OSGi unless you have no other choice."

Basically, just looks like it's going to come with a fair amount of pain, so you'd best know in advance that you absolutely need the benefits to offset the cost.

Lift Book Outsells All Other Web Frameworks at JavaOne

It's interesting to review the top sellers at the JavaOne bookstore. Seems to break down like this:

• Two JavaFX books, at #1 and #2. Considering how few people I know who are interested in JavaFX, this feels like a stuffed ballot box to me, but, what the hell, I'll give it the benefit of the doubt. Maybe people were impressed with JavaFX at the conference.
• Three scala books, at #5, #8, #9. Interesting that 'lift' was the only web framework whose book made the top ten, unless you want to count the Java EE book (I don't).
  
• Four core Java and Java EE books at #3, #4, #6, #7. Not really shocking for a Java conference.
• One solaris book.
  

What's New in Eclipse 3.5: Milestone 7

Milestone seven of Eclipse 3.5 is out, and it solves a long-standing annoyance for me with Eclipse, which is that you switch between editor tabs with Ctrl-PgUp and Ctrl-PgDown, but in a multi-tab editor, you switch within the editor with the same keystroke. This leads to weird behaviours like when you want to switch from a Java tab to another Java tab but pass through an XML tab with design/source view and get "trapped" because the keystroke you were using no longer lets you continue.

It also looks like support for cocoa on OS X continues to improve.

What's New in Eclipse 3.5M5?

Eclipse 3.5 M5 is out, and as this cycle winds down, the changes are getting less significant.

I like the way they're planning on highlighting matching characters in the Open Type dialog:
It's also nice that they'll be offering an Open Implementation hyperlink:

Fixing Politics from the Grassroots Up

"Politics" and "Government" are broken.

The world has changed. Politics and government have not kept up.

Without trying to start a debate about the nature of government and politics¹, I'll assert: the aim of Government should be to protect the welfare of the people² and politics is the means through which the people influence government and thus protect their collective welfare.

And, perhaps, at some point in the distant past, that was how things worked, but those times, if they ever existed, are done. Most people don't feel that politics allows them to protect their welfare, or even to influence government. As a result, many voters are disengaged and/or jaded, voter turnout is dropping³ (Canada, US), particularly for the young. Those who aren't disengaged are often trying to exploit the system for their own gain.

This can be changed. It's already changing.

We're Ready for Change
The North American economy has taken a pounding, and that will continue for months, if not years. This causes unrest, and unrest increases the appetite for change. Everyone is keenly aware that things, as they stand now, are not working. Even Microsoft, a company that has stood to gain from the status quo for many years now, is ready to speak out in favor of change.

The people of the United States of America have voted in Barack Obama, who made change a key element of his platform. And, frankly, it isn't hard to see that simply the act of voting in an african-american as a president is a welcome sort of change. You can see from the optimism that surrounded Obama's inauguration that people haven't lost all hope that Government can change their life for the better.

There are signs of change here in Toronto as well, but we'll come back to these.

Political parties have been slow to adapt to these changes. They've been stuck in old ways and old models: Politics 1.0, which is all about lawn signs and leaflets and broadcast advertising and television debates.

We Have the Tools of Change
The past fifteen years have brought remarkable changes to the world in the ways we communicate, the ways to share knowledge, the ways we reach out to each other, network and collaborate. Although the tools of change are themselves changing, we're already far more powerful than we were, particularly in the ways in which collective effort and desire can be channelled and harnessed towards a common goal.

In the past few years, political parties and organizers have been learning, changing their ways, adopting meetups and social networks, facebook, twitter, iPhone applications and user-generated content. This is progress, but it's still coming from the same mindset. These are new ways to help political parties communicate with their loyal members. They're ways to engage those loyal members in recruiting drives and fundraising. They don't and can't scale to real two-way communication. It's a start, but isn't enough.

Be The Change
It's not enough to take new technologies and apply the same political mindset. It's great that Kerry was able to take Meetups to increase grassroots fundraising, and it's interesting that political leaders like @JackLayton are finding ways to use twitter to get their message across to more people, faster, to mobilize their supporters. But these are simply ways that we can get more involved with our political parties.

If we truly wish to, as Ghandi reputedly urged, "be the change we want to see in the world," then we're aiming too low. Rather than trying to use these technologies to stay better connected with party strategy, we should be using these to enact change directly on the world.

Here in Toronto, when the tech-savvy populace banded together to create and support #HoHoTo, we donated $25,000 and lots of food to local foodbanks. We followed that up with #ChangeCamp, where politicians and the people met to talk about practical change, like opening up government data to allow the people to add innovative value through mashups and analysis. We have @mayormiller on twitter, attending #mesh to talk about open government, and open data.

It's this kind of actions that really give me hope for change, hope for a people that is passionate about and actively engaged in helping each other to not only survive, but to thrive. Public debate and direct action are far more interesting than carefully channeled political messages. I'm hoping that these kinds of events will continue and even accelerate.

So, what does this all mean?

A Manifesto for Change
It's time for change. We have the appetite for change and the tools of change. We have the will and means to get directly involved in the the governance of our society, to make a difference. We can channel this energy through open governments, social entrepreneurship, and other channels. We don't want to support, we want to participate. We don't want to hear the messages from on high, we want to make the world a better place, directly, immediately, through our actions

Accordingly, I suggest that we value:

• Action over politics. We need to focus on making a difference and not be distracted by supporting particular political parties and getting caught up in partisan messaging. Activism has its place, but I'd rather see that same energy go into changing the world directly.
  
• Open over closed. We need open data and open government, not black-box governance with private data and closed-door sessions.
  
• Grassroots action over top-down action. There are solid reasons for centralized agencies in government, but innovation happens at the fringe, and that means you. We have an engaged, civic-minded, technologically-savvy populace, so let's get out there and do something.
  

This is something we can act on now. Change the world, and, following the example of Jessica Jackley,"retire on happiness". Sounds good, doesn't it?




Notes
1 - Although the nature of Government and Politics are themselves subjects on which entire volumes can and have been written, and I'm sure some will disagree with the definitions I've chosen here.
2 - By balancing some definition of collective good with individual freedom; difficult to get right, but worth striving for.
3 - Although voter turnout is on a steady downward trend, the trend isn't as steep bad as the news would have you believe. It's not a good sign, but it's also not impending doom.

First Month as an Apple User: *%@#&$

So, on 09-Feb at 9am, I placed an order for a MacBook Pro 15".

There were a few delays in shipping the order, but it finally shipped on 14-Feb and I got it on the 17th. A little slow, but no big deal. I spent the week setting it up so that it would be ready for personal use and contract work.

On Friday (20th), the ethernet port suddenly died altogether. Wifi was still working, so I signed up for the Genius Bar Monday morning, then took it home. I tried it with some more networking equipment over the weekend to make sure the problem was the laptop, but sadly nothing worked.

On monday (23rd) the Genius Bar agreed that the ethernet port was dead, and suggested I send it back for a replacement. The joy of getting a custom-configured machine from apple (more RAM, faster HD) is that instead of doing a quick in-store exchange, they can make you wait for another few days. I returned it the next day after getting their RMA form, and they processed the return.

The order status updated to show the return processed on the 25th, and I began to wait some more. By 01-Mar, I was starting my new contract, without a good machine to do it on, which was really irritating. On 02-Mar, Apple updated their processor lineup. I checked my order status again and noticed that I was getting the 2.66GHz update, but that my hard drive was downgraded to the 5400 rpm.

After another half-hour with support, I'm back to a 7200rpm hard drive, and I've learned that the hold on the replacement laptop was never released after I returned the original, so I've been waiting for no reason. The hold is now released, and now, sometime over a month after I ordered it, I may receive a working laptop.

So far, my user experience as an Apple convert isn't working out all that well. The apple support people are nice enough, but the fact that I've had to call them several times in the first month of ownership, most of which I've spent without an actual Apple is really starting to piss me off.

This had better be the last hangup.

Generating Flex HTML Templates with Flex Mojos in Maven

Since I'd managed to reach a base level of comfort with Flex Mojos fairly quickly, we decided to see how hard it would be to extend it the rest of the way. As it turns out, not terrifically difficult.

I wasn't willing to disable the working FlexBuilder-integrated build that would take the results that FlexBuilder would generate from target/bin-release and include it into an assembly, so I took a little time to use a dual-profile approach to setting up the assembly:

  <profiles>
    <profile>
      <id>flex-mojos-assembly</id>
      <activation>
        <activeByDefault>true</activeByDefault>
      </activation>
      <build>
        <plugins>
         <plugin>
            <artifactId>maven-assembly-plugin</artifactId>
            <configuration>
              <descriptors>
                <descriptor>flex-mojos-assembly.xml</descriptor>
              </descriptors>
            </configuration>
          </plugin>
        </plugins>
      </build>
    </profile>

    <profile>
      <id>flex-builder-assembly</id>
      <build>
        <plugins>
         <plugin>
            <artifactId>maven-assembly-plugin</artifactId>
            <configuration>
              <descriptors>
                <descriptor>flex-builder-assembly.xml</descriptor>
              </descriptors>
            </configuration>
          </plugin>
        </plugins>
      </build>
    </profile>
  </profiles>

This gave me the freedom to continue working on integrating Flex Mojos approach without breaking what was already in place, particularly since we haven't thoroughly gone over the SWF that Flex Mojos generates. If I want to build an assembly with the SWF that flex-mojos generates, I simply run the build. If I want to use the release build from FlexBuilder, I run the same build but turn on the flex-builder-assembly profile (which automatically takes the other profile out).

Now, this approach doesn't take Flex Mojos out entirely; it just ignores the results of the flexmojo compilation process if it isn't being used. This is a little wasteful, so I may come back and revisit that approach later. For now, it's a lot less complex than trying to remove the flexmojos integration entirely, which is done through a parent POM approach.

After that, it was a matter of generating the HTML using the local custom html template by adding this to the flex-mojos-assembly profile:

          <plugin> 
            <groupId>info.flex-mojos</groupId> 
            <artifactId>html-wrapper-mojo</artifactId> 
            <executions>
              <execution>
                <goals>
                  <goal>wrapper</goal>
                </goals>
                <configuration>
                  <targetPlayer>9.0.124</targetPlayer>
                  <templateURI>folder:html-template</templateURI>
                  <outputDirectory>${project.build.directory}/html-template</outputDirectory>
                  <parameters>
                    <bgcolor>#ffffff</bgcolor>
                    <!-- Defaults Follow -->
                    <!--
                    Version comes from 'targetPlayer' or '9.0.0' if no targetPlayer.
                 <version_major>9</version_major>
                 <version_minor>0</version_minor>
                 <version_revision>0</version_revision>
                 <swf>${project.build.finalName}</swf>
                 <width>100%</height>
                 <height>100%</height>
                 <application>${project.artifactId}</application>
                 <bgcolor>#869ca7</bgcolor>
                    -->
                  </parameters>
                </configuration>
              </execution>
            </executions>
          </plugin>

And setting up an assembly descriptor to pull in what I needed from the project structure and the results of the generation:

<assembly>
  <id>flex</id>
  <formats>
    <format>zip</format>
  </formats>
  <includeBaseDirectory>false</includeBaseDirectory>
  <files>
    <!-- SWF -->
   <file>
     <source>${project.build.directory}/${project.build.finalName}.swf</source>
   </file>
  </files>
  <fileSets>
    <!-- HTML Template -->
    <fileSet>
      <directory>target/html-template</directory>
      <outputDirectory />
      <includes>
        <include>**/*</include>
      </includes>
    </fileSet>
    <!-- Project Resources -->
    <fileSet>
      <directory>src/main/flex</directory>
      <outputDirectory />
      <includes>
        <include>assets/**/*</include>
        <include>images/**/*</include>
        <include>*.html</include>
        <include>*.mp3</include>
      </includes>
    </fileSet>
  </fileSets>
</assembly>

This gives me a final assembly that looks a lot like the assembly the project was already generating. Now to find out if it works the same way ...

Building Flex with Maven and Flex Mojos

I've been getting to know a project that uses Maven and Flex.  The flex portion of the project is currently built using FlexBuilder, and it seemed worth investigating whether or not buildng the Flex using Maven were feasible.

I took a look around at the options, and it seems like the Flex-Mojos project is the one with the most momentum.  It's relatively current, gets updated regularly, and the team seems to be working with Sonatype, well-known in the Maven community.

The Flex Mojos project is still pretty rough in some areas.  In order to get things done, you need to search the blog, the google group, the google code home and the newer wiki and source code repository hosted at Sonatype, as well as the plugin documentation.  I'm hoping that with a little more time, the documentation will start to coalesce in a single location and get better, although sparse documentation is par for the course when it comes to Maven plugins.

With a little work, I was able to get the project to build an SWF file:

 <parent>
   <groupid>info.flex-mojos</groupid>
   <artifactid>flex-super-pom</artifactid>
   <version>2.0</version>
 </parent>

 <repositories>
   <repository>
     <id>flex-mojos-repository</id>
     <url>http://svn.sonatype.org/flexmojos/repository</url>
     <releases><enabled>true</enabled></releases>
     <snapshots><enabled>false</enabled></snapshots>
   </repository>
 </repositories>

  <build>
   <plugins>
  
     <plugin>
       <groupid>info.flex-mojos</groupid>
       <artifactid>flex-compiler-mojo</artifactid>
       <configuration>
         <sourcefile>CrystalQ2.mxml</sourcefile>
       </configuration>
     </plugin>

   </plugins>
 </build>

I also needed to declare a dependency on the SWC library on which the project depended:

 <dependencies>
  <dependency>
    <groupid>com.caucho</groupid>
    <artifactid>hessian-flex</artifactid>
    <version>3.2.0</version>
    <type>swc</type>
  </dependency>
 </dependencies>

There's still other things I'd need to do to mimic what Flex Builder offers, such as generate an HTML Wrapper and copy assets needed by the final application but not included in the SWF.

That said, I'd say that the basics are already in place and I'm starting to feel comfortable that there are options for putting Flex and Maven together, should you so desire.

Maven Resource-Filtering Escapes Backslashes

Having moved up to Maven Resources Plugin 2.3 in order to stop Maven from attempting to filter some binary files for us, and in so doing, destroying them, we've discovered that 2.3 has its own problem -- it escapes backslashes and colons in files where escaping is totally unwarranted, which does a surprisingly good job of destroying an XML file we need for Selenium and Java Web Start to play nicely together (a mimetypes.rdf that tells Firefox how to deal with a JNLP file).

Three cheers for Maven, an ecosystem of tiny plugins, each of which has its own set of bugs and idiosyncrasies, which collaborate to ensure that there's no valid way to build your project.  Whee.

Are We Ready for Bespin?

It's interesting to finally see what's going on in Mozilla Labs around Bespin.  It's an interesting idea, making a web-based development environment.  It's been tried before, but I'm curious to see it develop and see how far they can take it.  

What they have already seems promising, but it's a long way from replacing a desktop application for me at this stage.  Still, with enough persistence and enough time, there are undoubtedly some things a web-based application could do that would be difficult for a desktop application to mimic.  We'll see what happens.

I'm not sure developers are ready to move their development environment into a browser.  More specifically, I'm not sure that it will be possible to create a compelling, seamless enough web development environment.  But if it can be done, these are some of the people I'd like to see working on it, so I'll keep my fingers crossed and keep my eye on Bespin as it evolves.

Metro Toronto Fires All breast Writers

Well, last night I read (twitter, Globe and Mail) that Metro Toronto had fired all its staff writers and was going to be using unpaid interns.

This morning, reading a review of Friday the 13th, I found this paragraph, emphasis mine:

There's no mistaking Friday's intentions -- this is definitely for Jason Voorhees fans.  Not only does it overlap the thirteen killings (very clever!) that occur over the film's 97 minutes, but they're ultimately underwhelming.  breasts.  In fact, with very little suspense to speak of, Friday the 13th isn't scary at all ... unless of course, you're a film critic.

I'm wondering -- is this what happens when unpaid interns write your copy, or is an act of vengance in the last words penned by a departing writer?

Productivity and Pair Programming

I'm with Raganwald on the productivity issue. I don't feel like I can measure my own productivity or that of other developers, and I don't think anyone's made a really conclusive argument either direction about the productivity of pair programming, although I will say that there are times when I've found it very useful and times when I've found it a little irritating.

And, frankly, I think a good chunk of productivity studies are bunk anyway. The idea that we can measure and suggest that we're 36% more productive than people X years ago seems to be making claims about precision that aren't supported by any kind of useful data.

But what I will say is that despite the lack of a good productivity measure, I suspect that both a single developer and a developer-pair are more productive than any group of developers who are arguing about productivity instead of writing code.

On that note, I think I'll go write some code.

Bean Validation Public Draft

Sounds like Bean Validation has been voted through the public draft stage. I've just finally managed to get my voluminous feedback into the appropriate hibernate forum after wading through the spec.

Mostly, it looks reasonable, although I'd prefer to skip the bootstrapping entirely, or alter it pretty significantly so that it doesn't remind me of Why I Hate Frameworks.

I'm also not sold that we really NEED a JSR for this until there are multiple competing and incompatible implementations that take on validation seriously, which isn't currently true, IMO. Although I've used Hibernate Validator before, I'm not sure one implementation justifies a JSR.

But anyway.

Releasing Multi-Module Maven Projects

When you're doing a release using the Maven release plugin, Maven likes to ask you what version you'd like to release and what the version should be after the release.  For final releases, the defaults are often sensible, and you can accept them by turning on batch mode.  For interim releases, they usually aren't.

When you're doing this with a single project or a project with a very small number of modules, answering those questions for each module isn't such a big deal.  When you're working with a project with a larger number of modules, this gets very tedious very quickly.  The batch mode example  suggests that you can pre-specify the release version for each module, but it fails to highlight the property that a friend of mine just pointed out: autoVersionSubmodules.  That'll save me a lot of pain in the future.

Google App Engine Roadmap

The Google AppEngine roadmap shows a lot of promise; it'll significantly increase the sophistication of the applications that can be delivered using AppEngine. It's still true that it's hard to justify building a real application in an environment you don't control and can't easily duplicate, but it's nice to see solution gaining maturity.

If Google offered an open-source production environment -- not necessarily anything on Google's scale, but something that someone could deploy to at least a single server, or possibly a cluster of their own, of reasonable production quality, that would reduce the lock-in to the point that someone could conceive of building their own applications on AppEngine.

Eclipse 3.5 M4: Never Join Lines

As of Eclipse 3.5 M4, the formatter has an option to "Never Join Lines" or preserve user line breaks.

Finally.

I've gotta say, that's been one of my most consistent irritations with the Eclipse formatter ever since I started using the damn thing, something that IntelliJ IDEA did farther back than I can remember. I'm really looking forward to that.

While I'm on the subject:

• Eclipse 3.5 M5
• The ability to quickly paste a site in to install rather than having to manage your sites seems like a good addition, in keeping with how people actually want to use the system.
• SWT/Cocoa leaving experimental stage, that's good news for Apple fans.
  
• Eclipse 3.5 M4
• Preserving line breaks.
  
• Eclipse 3.5 M3
• The equals/hashcode compiler warning seems like a good idea.
• The extension point for clean-ups seems like something that could result in some useful additions given time.
  
• Eclipse 3.5.M2
• Eclipse 3.5.M1
• Support for ${inheritDoc} in the Javadoc view is great.
  
• Comparing word documents is the sort of feature I hope I don't need to use, but will probably end up using at some point anyway.
• A warning for comparing identical expressions seems valuable.

All in all, there's nothing here that seems critical, but lots of useful polish, and the formatter one alone is enough to warrant the ugprade. I'll have to take a look at the larger release train as well to see what else is coming.

That said, I have to say, I'm disappointed with 3.4 release -- I've had more problems with it than with previous versions, it feels like the quality's dropped off a little.

Apple should ...

Many people who try to give Apple advice seem to ignore that which has made Apple strong. For instance, when The Apple Blog suggests:

To make big strides in the mobile handset market, Apple should aim for the Pearl and Curve market space, either by introducing a simplified, lower-cost iPhone Nano, or by releasing a brand new iPhone model, but keeping the 3G around at a discounted price.

They should do nothing of the sort. RIM introduced the Pearl and the Curve basically because they'd failed to appeal sufficiently to the consumer market, because they had failed to put a great camera and good media support into the blackberries. In essence, they admitted that they were unable or unwilling to put together a single device to meet the needs of both markets.

Apple shouldn't do that, for more than one reason. First of all, model proliferation is where madness lies, particularly for application development. If you have to write an iPhone app and you're going to constantly ask yourself, "What's the resolution? Does this model have a camera?", the experience is significantly more complicated, and that comes out in your application design and choices.

Secondly, Apple is known for its design and its simplicity -- at this point, I don't think Apple needs to admit another model is necessary. At best, something like an iPhone nano, as suggested in the pre-macworld coverage: the same basic device in a smaller package. But, frankly, I think they should focus on building the best damn one model they can for several years to come yet.

Unreliable Java -version:XXX Command-Line

In order to help the operations team administer a web application, we'd built them some command-line tools in Java.  To launch those tools, we included a helpful shell script.  Since the application required Java 1.6+, it seemed prudent to use the Java command-line option that exists for that purpose.  Accordingly, we put "-version:1.6+" in the command-line options for Java.

In testing that script, we discovered that while it seemed to work reasonably well on Windows machines, it didn't work on any of the Linux installations we tried, from Red Hat to CentOS to Unbutu, some with manually-installed Java, and others with the package install direct from the operating-system repository.

At this point, I'd have to recommend that you treat the -version:XXX option in the Java command-line as unreliable.  If you control the environment in which it's being used throughly enough to know that -version:XXX will work, then you can probably also control the version of Java being used, thus making it moot.

What does Java 6 have against image/png?

In mapping from filenames to mime types, I've typically used Java Activation Framework's MimetypeFileTypeMap.  Leaving aside the fact that the code for this class (and possibly most of JAF) is pretty old and ugly, it mostly works.  Today, however, we discovered that it was giving us application/octet-stream for a ".png" file, and I decided to dig a little deeper.

I started by reading the docs, and checking the mimetypes.default included in activation-1.1.jar/META-INF, which seems to include png and PNG.  I turned on just enough logging for JAF to spit out what files it was loading and what information it got from each:

    @Test
    public void testPng() {
        System.setProperty( "javax.activation.debug", "true" );
        Logger logger = Logger.getLogger( "javax.activation" );
        logger.setLevel( Level.FINEST );
        logger.addHandler( new ConsoleHandler() );
        MimetypesFileTypeMap map = new MimetypesFileTypeMap();
        Assert.assertEquals( "image/png", map.getContentType( "myImage.png" ) );
    }

That told me that JAF could only find the mimetypes.default, but that it believed png wasn't in the file:

MimetypesFileTypeMap: load HOME
MimetypesFileTypeMap: load SYS
MimetypesFileTypeMap: load JAR
MimetypesFileTypeMap: !anyLoaded
MimetypesFileTypeMap: not loading mime types file: /META-INF/mime.types
MimetypesFileTypeMap: load DEF
Added: MIMETypeEntry: text/html, html
Added: MIMETypeEntry: text/html, htm
Added: MIMETypeEntry: text/html, HTML
Added: MIMETypeEntry: text/html, HTM
Added: MIMETypeEntry: text/plain, txt
Added: MIMETypeEntry: text/plain, text
Added: MIMETypeEntry: text/plain, TXT
Added: MIMETypeEntry: text/plain, TEXT
Added: MIMETypeEntry: image/gif, gif
Added: MIMETypeEntry: image/gif, GIF
Added: MIMETypeEntry: image/ief, ief
Added: MIMETypeEntry: image/jpeg, jpeg
Added: MIMETypeEntry: image/jpeg, jpg
Added: MIMETypeEntry: image/jpeg, jpe
Added: MIMETypeEntry: image/jpeg, JPG
Added: MIMETypeEntry: image/tiff, tiff
Added: MIMETypeEntry: image/tiff, tif
Added: MIMETypeEntry: image/x-xwindowdump, xwd
Added: MIMETypeEntry: application/postscript, ai
Added: MIMETypeEntry: application/postscript, eps
Added: MIMETypeEntry: application/postscript, ps
Added: MIMETypeEntry: application/rtf, rtf
Added: MIMETypeEntry: application/x-tex, tex
Added: MIMETypeEntry: application/x-texinfo, texinfo
Added: MIMETypeEntry: application/x-texinfo, texi
Added: MIMETypeEntry: application/x-troff, t
Added: MIMETypeEntry: application/x-troff, tr
Added: MIMETypeEntry: application/x-troff, roff
Added: MIMETypeEntry: audio/basic, au
Added: MIMETypeEntry: audio/midi, midi
Added: MIMETypeEntry: audio/midi, mid
Added: MIMETypeEntry: audio/x-aifc, aifc
Added: MIMETypeEntry: audio/x-aiff, aif
Added: MIMETypeEntry: audio/x-aiff, aiff
Added: MIMETypeEntry: audio/x-mpeg, mpeg
Added: MIMETypeEntry: audio/x-mpeg, mpg
Added: MIMETypeEntry: audio/x-wav, wav
Added: MIMETypeEntry: video/mpeg, mpeg
Added: MIMETypeEntry: video/mpeg, mpg
Added: MIMETypeEntry: video/mpeg, mpe
Added: MIMETypeEntry: video/quicktime, qt
Added: MIMETypeEntry: video/quicktime, mov
Added: MIMETypeEntry: video/x-msvideo, avi
MimetypesFileTypeMap: successfully loaded mime types file: /META-INF/mimetypes.default

That implied to me that there was another mimetypes.default available, which a quick classloader test shows to be true:

    @Test
    public void testDefaultResource() throws IOException {
        System.out.println( "Displaying default resources: " );
        int index = 1;
        Enumeration resources = Thread.currentThread().getContextClassLoader().getResources( "META-INF/mimetypes.default" );
        while( resources.hasMoreElements() ) {
            URL item = resources.nextElement();
            System.out.printf( "\t%d: %s\n", index++, item );
        }
        System.out.println( "Done printing resources." );
    }

Turns out, the other mimetypes.default is coming from Java 6 (update 4) resources.jar, which inexplicably doesn't include an image/png content type:

Displaying default resources: 
 1: jar:file:.../jre1.6.0_04/lib/resources.jar!/META-INF/mimetypes.default
 2: jar:file:.../m2_repo/javax/activation/activation/1.1/activation-1.1.jar!/META-INF/mimetypes.default
Done printing resources.

Since activation 1.1's been out for a long time, I can't think of a good reason why image/png would be missing from a relatively modern JRE.

As always, in order to hedge against running into this again and help anyone else who might hit the same issue, thought it was worth writing down.

Story Granularity, Complexity and Planning

Obie Fernandez wrote about some problems he's been running into with respect to story granularity, and Bill de hÓra picked up the thread.  This is a subject I've struggled with a number of times, so I'm going to give myself a little free therapy and try and work through some of it here.

Story Granularity

Whenever possible, it's good to break down the desired features of an agile project into stories that are pretty granular.  What "pretty granular" means to your team and company varies somewhat, but something in the range from a day to a week would be relatively common in many of the projects I've worked on.  

Once the story gets significantly beyond that threshold, most teams will do their best to try and break it down into smaller pieces.  But there are times where that's difficult, or possibly even undesirable.

Minimal Marketable Feature

One of the ways to look at story granularity is that you want to assemble the minimal marketable feature.  You want the story to represent exactly the minimum amount of work that you could do and still "market" or "sell" the feature to the desired userbase.  If anything were removed from the story, the story could no longer be sold, it won't be credible or cohesive.  I find this perspective is often useful to help me "right-size" stories.  Stories that are larger than the minimal marketable feature are gold-plated, and stories that are smaller than the minimal marketable feature don't deliver meaningful user value.

Unfortunately, it often seems to be true that the minimal marketable feature is larger than your desired story size, and that creates a subtle tension between project and product management in an agile environment.

There are, of course, ways to try and work around this problem, and de hÓra talks about some of these.  Fundamentally, what tends to happen is that you subdivide the minimal marketable feature.  When you do this, it tends to create some other interesting problems.

How can you subdivide the minimal marketable feature?

Sometimes, this isn't very hard.  There are natural lines in the feature that allow you to implement it in several phases, each of which makes a cohesive piece of functionality, just not functionality that is individually marketable.

Often, however, the functionality cannot be broken into cohesive chunks.  If you end up having to deliver it in pieces that might individually break parts of the system, it's difficult to find the right division lines.  It gets even more complicated when in order to subdivide the feature and estimate the pieces, you need to pick an implementation approach.  Selecting a detailed implementation approach is a decision that many agile teams would like to defer past the initial planning stage, but if you can't easily finish your release and iteration planning without dividing the story and you can't divide the story without talking about implementation, then you're somewhat stuck.  

In these cases, you could take a 'theme' or an 'epic' past your planning stage, knowing that it's a risk and an unknown, but preferring that to talking about implementation concerns too soon.  Alternately, you could buckle down and do some up-front design and hope that whatever plan you come up with still makes sense when it's time to do the detailed implementation.  There are tradeoffs to be made here, and I can't make them for you.

What happens when the minimal marketable feature spans several iterations or releases?
If the subdivision is necessary for tracking progress, but all the work will be done within an iteration, then you've dodged another bullet.  If, on the other hand, the pieces that make up the minimal marketable feature will be split across two or more iterations, you're facing a new problem.  Even if those iterations are within the same release right now you may be reducing your ability to release, to adapt, to change.  

This is only a small problem if the subdivisions each deliver cohesive functionality that are simply too small to be individually valuable.  It's a much bigger problem if the elements aren't individually cohesive, where the result of a particular story might leave your project in an inconsistent state.  In these cases, you may need approaches that allow you to deliver code without linking it directly to the application, or the ability to deliver features that are disabled.  If, like some organizations, these approaches are already necessary (e.g. to work with multi-tenant SaaS solutions), this may not be incredibly painful.

Are large stories a cause of problems or a symptom?

Sometimes, the presence of large stories isn't the real source of the problems you're facing.  The problem is that the story shouldn't really be as large as it is, and has become so for a few reasons.

First, you want to be sure that the minimal marketable feature really is minimal.  This can happen when the product owner / manager is relatively new to agile planning and hasn't yet experienced the joy of discovering that your users can get by with a lot less than they think they need.  It can also happen when clients are unreasonable, when features are designed by committees, or filtered through a decision-making structure where each layer needs to 'innovate'.  It can happen when someone is trying to avoid past mistakes by ensuring that new features are robust and flexible, a sort of anti-YAGNI tendency.  If this is happening, you need to understand why it's happening so that you can try and find a solution that doesn't involve building extra-large stories where a medium-sized story will do.

You also need to come to terms with overall system complexity.  I've often seen systems that grow and evolve, add special cases and new features that interact with each other to create a surprising amount of complexity.  When this happens, adding even simple features can require a great deal of work.  "Sure, we can add that 'customer profile' box on the home page, but we'll need to create a customer profile display that is aware of the multi-tenant additional fields and meets the role-based security constraints.  We'll need to add several new service methods to customer web service API, and a new component to the user interface layer.  Unfortunately, three of those fields aren't available to this system, so we're going to need to make a JMS call to the CMS in order to get those pieces. We'll need to be able to turn the component on and off on a per-client and per-interface basis, and possibly on a per-role basis in order to work with existing clients, not all of whom are going to want to display this kind of information, and we may also  need to add field-specific display configurations."  

I might be exaggerating a little, but as systems grow, they tend to accumulate complexity, and many organizations don't do a strong job of keeping complexity down, either because they don't understand the heavy price they're paying for the complexity, or because they simply feel that the complexity is necessary.

In Summary

It's easy to say stories should be small.  It's not always easy to make it happen.  The devil's in the details.