One well-known issue with Open Source is the so-called "Dependency Hell" problem.

Background

This situation arises when one package, let's call it "A", depends on a number of other packages, "B" and "C", that have their own dependencies in turn (B depends on D and E).  When trying to get A installed correctly, you first have to find and install all of it's dependencies.  In this case, you need to install B first.  But wait, that means you have to install D and E before B can work. 

The tree looks like this:

        A
       /  \
     B     C
    / \
  D    E

The Problem 

Here's where the real fun begins.

It turns out, according to your research, that D requires a special tool, compiler, or operating system library called "X" before it can do its thing.  And C requires another thing "Y".  That's all well and good, but lo and behold, X and Y are found to be incompatible.  For whatever technical reason, X and Y either can't be installed simultaneously, are different versions of the same tool, or simply cause another chain of problems when installed together.

This means it's time to start jumping through some hoops.  One option is that you can install X, build D, remove X, and finally put on Y to build C.  Whew, that was hard!  But wait, there's more.  What if both X and Y are required at runtime to work properly?  Now you're really in trouble.

Unfortunately, the scenario I just described is not simply fiction, it's an every day occurrence in the open source world.  Many projects, especially when they're based on the C programming language, get into these other circular dependency situations that make programmers pull their hair out and punch their neighbors.

The Solution

It might be hard to hear for some die-hard C fans, but I think the solution is either A) don't depend on shared libraries and other operating system resources, or B) use Java or another language that makes it easy to include all necessary run-time dependencies with your code.  Yes, both of these solutions can lead to dramatically increased disk space usage, but hey, when's the last time your new computer came with a 30 megabyte hard drive?  Laptops now ship with fast 160GB+ drives; it's time to move on.

How Operating System Vendors Help (Or Don't Help)

Okay, here's where I'm going with all of this.  Linux and most of the thousands of open source projects in its ecosystem are written in C.  Vast numbers of these projects suffer from the problems I've described above, so what are the Linux vendors doing about all of this?  Well, it looks like they're saying to developers and users, "Look at all these cool lego blocks we've given you! Have fun building things with them!"  In other words, they're doing nothing.  Actually, that's not quite right.  They're doing worse than nothing.  They're continuing to ship new versions of their operating systems with updates to all these shared libraries and other tools that quickly break fragile dependency trees.

But perhaps I'm not being fair to the OS distributors.  It's not like they're the ones creating the fragile dependency trees in the first place, right?  They're not the ones forcing people to use the outmoded tools and libraries they package up along with the core OS.

Hmm… maybe we're onto something here.  Why is it that the OS vendor ships this stuff in the first place?  I mean, they're just a bunch of unrelated add-on packages, applications, tools, and utilities that go uncertified, untested, and unsupported.  Why do they come with the operating system any way?  Shouldn't the operating system vendor spend all its time trying to make the operating system (and the accompanying desktop environment) as robust, stable, and security-hole-free as possible?  Or maybe they like this situation because large software vendors like Oracle can pay them to certify a particular application on each new release of the operating system?

Conclusion

I think what's needed to solve these nasty problems is for the open source community in general to move away from low-level operating system dependencies and other C-based tools that expect pre-existing shared resources in their run-time environment.  It's 2007; let's concentrate on virtual machine technology for both hardware and software purposes.  That is to say, let's continue the migration to Java, Python, Ruby, and other languages that shield us from these painful problems of yesteryear.  Let's move away from depending on operating system vendors to provide our above-the-os world.

High Mobley nicely outlined the business case for open source last week, and not surprisingly his words generated some pretty heated comments.  The open source vs. proprietary issue always brings out passionate debate on both sides.

I think the business cases Mobley covered are valid, but as some of the comments suggested there’s more to technology decisions than the fees associated with a particular software license.  Technology investments always come with issues like training, ongoing maintenance, interoperability, and ease of configuration and integration.  These decisions aren’t made in a vacuum where the up-front licensing costs are the only factors considered.

Sometimes open source software isn’t the best solution for a particular problem, business environment, or the personnel involved.  But when open source offers the best solution for a problem—and enterprises are increasingly choosing open source over commercial software—the key is to go with a commercial open source vendor that can fill in the gaps around the issues mentioned above.  Harper Mann perfectly summed up the situation in a separate blog:

…the solution is not to avoid open source, but rather to engage commercial open source. That is, work with companies that can connect and enhance what open source tools can do. Companies that serve as immediate liaisons for both the open source community and the business community. Ultimately, companies that can provide what end users need.

In short, that’s what we do here at OpenLogic.  We offer the services that enterprises want (certification, support, training, updates, indemnification, and management tools) for hundreds of the most popular open source packages.  This puts open source on a level playing field with proprietary software and allows companies to choose the best business solution for each situation.

As we come up on the one year anniversary of the Open Logic Expert Community (OXC), I've been thinking about what's going well and what we can improve on.  The program itself is going really well.  What's not going well is the misconception some people have that we are only giving away trinkets!

We've signed up committers and contributors for many, many open source software projects – we currently have over 250 open source packages in our library!  These committers, OXC members, have proven that they can not only solve our enterprise customers' needs but they can do it in a very timely and professional manner.  As a strong advocate of the open source community, I'm proud to say that they've lived up to all my expectations and gone beyond.  Around the globe, around the clock.  We often have difficulty figuring out who to credit for an issue because several people will jump on it at once!  

As for the misconceptions … people seem to think we aren't paying cold, hard cash for this work.  Well, we are.  (At least I suppose PayPal, wire transfers and checks count as cold, hard cash in today's economy!)  When we started this program we talked to a lot of open source community members and got their feedback.  90% of them said they'd rather have things like XBoxes than cash.  The other 10% said they'd rather give their money to open source organizations like Apache.  So we said, ok we'll give you an incentive like an XBox to sign up and then for each issue you solve you'll get points that you can trade in for cash (1:1), merchandise or you can donate it.  So far everyone except for one person has opted for the cash.  Another notable point is that the community members thought $20/issue would be fair … we thought about it and decided $100/issue would be a better starting point.  

The only other "issue" we've had is that we haven't had a bug to fix in every project.  It's disappointing to line up people to support projects and then find out they've done such a good job on their project that it doesn't have any bugs.  What a problem to have!

So, as we come up on the one year anniversary of the OpenLogic Expert Community, I'm happy to report the program is successful in achieving our goal of uniting enterprises with the open source community in a way that enables enterprises to use open source software and supports the open source community at the same time. 

Thank you to all of our Expert Community members and to the rest of the open source community for all their hard work developing and supporting some awesome software!  

Red Hat announced its Red Hat Exchange program last week.  (Coincidentally, the abbreviation for the program—RHX—reminds me of the OXC abbreviation used for our OpenLogic Expert Community program.)   They've already signed up a number of open source vendors and are offering those vendors’ open source applications as part of the Red Hat Network.  By consolidating sourcing and support for the Linux operating system and a number of applications on top of it, Red Hat is hoping to make it that much easier for Global 2000 companies to use open source software in a production environment.

While this looks like a step in the right direction, my concern is that the Red Hat Network doesn't go far enough in providing the breadth and choice that customers want.

As we work with Global 2000 companies, we find that they are using a broad range of open source packages.  On the average, companies that we talk to report using 75 different open source packages.  (In fact, this number probably underestimates actual usage, since many companies do not have an easy way to inventory open source.) 

First, when large enterprises look for support for open source, they typically want coverage for the whole breadth of open source products they use.  This includes products like Spring, Struts, Eclipse, JSF, Ant, Maven, CVS and Subversion, which go beyond the open source applications that have been announced as part of RHX.

Second, companies look for choice within any product category.  They want support for open source products like Tomcat and Geronimo that might compete with Red Hat’s products.

Lastly, customers are deploying open source across multiple platforms — including RHEL, SUSE, Solaris and even Windows.  They want open source support that covers all of the platforms they deploy on.  Whether RHX will ever support multiple platforms remains a question.

As Dave Rosenberg commented on Matt Asay’s blog about RHX, “The thing I don’t love about rhx is that it’s not an open market.”  With Red Hat controlling vendor participation in RHX, competition and choice—at least for some types of applications—will be limited.

At OpenLogic, we have chosen to offer support for hundreds of open source packages that give customers options across a variety of product categories and a variety of platforms.  We believe that this provides the breadth AND choice that companies want.

Red Hat will undoubtedly continue to fill out their offering, and perhaps someday it may be considered a one-stop-shop for open source.  But it’s my belief that customers have needs that go beyond what Red Hat is willing to offer.  At the end of the day enterprises will pick the solutions best meet their needs, whether or not those solutions all come from RHX. 

Several disparate activities have conspired to produce this post.

1.) I've been listening to Phil McKinney's Killer Innovations podcast a lot lately on my 140 mile round trip commute to OpenLogic. 2.) I've been reading the book The Evolution of Useful Things by Henry Petroski. (not while driving :-) 3.) I've been frustrated by the unnecessary complexity and bloat of numerous mainstream open source projects 4.) Commercial software is every bit as bloated except there's no opportunity to do anything about it. (While these are all related motivations for me, #3 and #4 are a topics of future posts…this post I'll devote to Petroski's book and how it's relevant to open source innovation.)

Engineer As Critic

Petroski contends it is frustration with existing products or processes, rather than necessity, that is the real mother of invention. He studies the characteristics of historic inventors and concludes it's the "engineer as critic" that drives invention. It's not a vacuum or lack of a product which can address a particular problem, but how imperfect current solutions are at satisfying the need. Petroski's example of the evolution of the zipper, from straight pins and buttons to the modern zipper in the early 1900's, is a great lesson from which open source developers can learn. From the time work began on the zipper, it took about 30 years and numerous failed companies before it was finally patented and in common use. In the meantime, people didn't walk around naked, but their garments were definitely more drafty. The draftiest of all might have been the early adopters of the zipper who found their newfangled gadget was wedged to such a degree it had to be cut out out of the fabric in order to extract the occupant from the pant or skirt. At least the occupant could get out – I can remember more than a few wedged Windows '98 users who weren't so lucky. There are huge insights for the open source community in Petroski's book. First, the impetus for most open source projects is exactly his "engineer as critic." This is a good thing – the open source community really started this way. It's axiomatic in open source culture that if you don't like something, you roll up your sleeves and fix it or create something to fix it. You literally and figuratively have the license to do that. It's this cultural DNA of open source which gives me unrestrained confidence that most of the "useful everyday things" we'll see in technology over the next 30 years will be derived in some way from open source efforts.

Open Source as Innovative Process

With that insight, it's not a stretch to see that open source is an entire system and process for innovation. Open source is more an engine for innovation than it is a specific software package or money saving device. Open source by definition is structured to incorporate the Petroski "engineer as critic" as the fundamental driver of change. As the whole process develops, the actor(s) driving the change will evolve, but for right now, engineer-as-critic is where most innovations begin because engineers are typically on the front line of doing something about it. With those glowing remarks behind us, I think there are a lot of open source projects, even some mainstream open source packages, that look a lot like the early zipper when it got wedged at inopportune moments. Without putting too fine a point on it, the open source analog can leave early adopters with their zipper down and no way to zip it up. The growing recognition of that fact has created the impetus for certification, indemnification, and emphasizes the critical need for community support around an open source project. Again, imagine if you were an early adopter of the zipper and couldn't get zipped up one night in the wash room during intermission at the opera. You get that sinking feeling that you're more than a few degrees away from choking the inventor through your LinkedIn network. You can't really roll up your sleeves and fix the problem as would be prescribed in open source culture. There's no community of zipper technicians available to help resolve your embarrassing problem. The obvious analog with commercial software is that unless you're a huge company, and possibly unless you're a government entity, you have no leverage with Oracle, Microsoft, IBM or any of the other large tech company. Even with purchased, commercial support, you could still have some embarrassing software "draft." The obvious analog in open source is if the OSS project isn't mature, or actively developed, and if there's not a vibrant, helpful community behind it, you can have the open source equivalent of commercial software "draft." In that respect there's not much difference between commercial and open source software – it still comes down to how well it's supported and how responsive the people behind it are. Despite the technical similarities between commercial software and open source software, I submit that your chances of affecting a positive outcome are greater with open source than most commercial software.

Is Source Code Access Really the Issue?

Consider these questions: (I've got to pick on open source a little) The last time your Linux box failed to behave correctly after a distro's kernel update, was your first thought: a) "I better roll up my sleeves and dust off my kernel debugger and figure this thing out…I'm going to get to the bottom of this, I have the tools and source code" b) "I'll bet I can find a Bugzilla entry somewhere and patch it up" c) none of the above Now it's Windows turn in the barrel: The last time you had a problem with your Windows operating system, was your first thought: a) "I should let Microsoft know about this, I'm sure they'll want to improve their product and my feedback will be important to them" b) "I can fix this, but I have no source code." c) none of the above No doubt the answer for 99.9% of those reading this post is c) in both cases.

The Impetus for Open Source is Changing

This leads to an inescapable conclusion that while the engineer-as-critic is the impetus for open source and technical innovation in general, it's not the end of the story. As the questions above attempt to demonstrate, whether you have source code access or not, the likelihood that Joe Average computer user will dive into the internals of any project of moderate complexity and start banging away on source code like one of the million monkeys trying to write Shakespeare, is next to zero. Even if you have the raw skills and the desire, you might not have time to learn everything that needs to be learned in order to fix it and submit the fix. Again, the value of the support and community begins to shine, doesn't it? That is the essential rub for any software product to take its next leap in subscribers. Any product, whether it's a zipper or a web application server or database, has to work well and be well supported by technically savvy people for it to be a success. There have to be other innovations in the delivery that go with the necessary, but not sufficient, Engineer-as-critic innovation system. We need Accountant-as-critic, Tech-writer-as-critic, Marketer-as-critic, Architect-as-critic, Graphic-artist-as-critic, User-experience-critic. We need the whole lot of them to take open source beyond the early adopter zipper phase it's in now. It's time to innovate in all areas, not just engineering, to accelerate the adoption rate of open source.