Blog

Something that resonated with me

*Simmons:* [H]ow did you learn to...write so well?

*Gladwell:* ...I'm not sure I can answer that...except to say that I really love writing, in a totally uncomplicated way. When I was in high school, I ran track and in the beginning I thought of training as a kind of necessary evil on the way to racing. But then, the more I ran, the more I realized that what I loved was running, and it didn't much matter to me whether it came in the training form or the racing form. I feel the same way about writing... They say that Wayne Gretzky, as a 2-year-old, would cry when the Saturday night hockey game on TV was over, because it seemed to him at that age unbearably sad that something he loved so much had to come to end, and I've always thought that was the simplest explanation for why Gretzky was Gretzky...

---snip---

*Gladwell:* This is actually a question I'm obsessed with: Why don't people work hard when it's in their best interest to do so? Why does Eddy Curry come to camp every year overweight?

The (short) answer is that it's really risky to work hard, because then if you fail you can no longer say that you failed because you didn't work hard. It's a form of self-protection. I swear that's why Mickelson has that almost absurdly calm demeanor. If he loses, he can always say: Well, I could have practiced more, and maybe next year I will and I'll win then. When Tiger loses, what does he tell himself? He worked as hard as he possibly could. He prepared like no one else in the game and he still lost. That has to be devastating, and dealing with that kind of conclusion takes a very special and rare kind of resilience. Most of the psychological research on this is focused on why some kids don't study for tests -- which is a much more serious version of the same problem. If you get drunk the night before an exam instead of studying and you fail, then the problem is that you got drunk. If you do study and you fail, the problem is that you're stupid -- and stupid, for a student, is a death sentence. The point is that it is far more psychologically dangerous and difficult to prepare for a task than not to prepare. People think that Tiger is tougher than Mickelson because he works harder. Wrong: Tiger is tougher than Mickelson and because of that he works harder.

Continued..

Web services are no silver bullet

John Stahl recently gave me some heat for my assertion that Elgg should be built on top of Drupal.

I think that the next few years are going to bring tremendous challenges for applications that do not easily communicate with other applications that are “outside their platform” i.e are written using a different language/framework, run on a different server, etc....The days of monolithic application stacks that try to do everything are fading fast. A new “network-centric” software ecosystem is starting to bloom.

This is wishful thinking. I've spent much of the past few years puzzling over this exact question. While I am personally very much a proponent of web standards and web services I am pessimistic as to how much immediate impact they will have in the evolving marketplace of non-profit/ngo & advocay web technology services.

Backstory

I didn't always think this way. If you told me a year and a half ago that I would be hawking CivicSpace/Drupal as the über-platform that could meet virtually every need of any size organization I would have told you you were nuts....

After the Dean campaign ended and we started work on CivicSpace our assumptions were:

  • Drupal wouldn't be able to "scale" to meet the needs of large organizations
  • We would not be able to develop the platform quick enough to meet all the core needs of organizations any time soon
  • We would integrate third party service providers to fill the functionality gap organizations required

Then some interesting and unexpected things happened:

  • It turned out Drupal could scale
  • CiviCRM showed up and filled the functionality gap
  • Lots of vendors established quickly growing businesses servicing the technology. The top half dozen firms now employ ~ 50 people between them and each are looking to hire.

This reshaped my thinking on the future of the web-technology marketplace quite a bit. We have an immediate opportunity to commodify the core web-technology organizations need to a single integrated and scalable open-source application stack and this is a very good thing for the marketplace. Over the next few years I believe the advantages afforded by this stack of technology (CS/Drupal/CiviCRM) will far outweigh the benifits realized by the integration of applications accross web-services in terms of costs saved and passed on to technology owners and innovations in technology and services.

Integration accross web-services and web-standards is relatively costly

  • Web applications that are not built from the ground up to be integrated will never play nicely with one another. Let me repeat: if an open-source application is not built to be integrated with your web-application it probably isn't worth the effort to try to integrate it. This renders 95%+ of open-source web applications useless for those looking to leverage the work of other communities.
  • If an application has a strong API it is still tricky to integrate. Even integrating with CiviCRM, an application that was built from the ground up to interface with CMS's and shares the same web-app environment as Drupal, it costs us 2-10X more to integrate with it than it does to integrate with a standard Drupal module.
  • The most cutting edge private sector web-service and web-standards are not that advanced. We still don't have workable single sign-on solution, the cutting edge of semantic information interchange across the web is to embed it in XHTML, the hottest API's from SalesForce and Flickr haven't made much of a dent in the marketplace, and please don't get me started on the semantic web.

The market currently prefers a single integrated stack

  • The majority of the market for web-technology services is owned by companies like Kintera, GetActive, and Convio that serve as one-stop shops for virtualy all an organizations web-technology needs. They have trained technology owners to make purchasing decisions under the assumption that they can cut a check to a single entity that will provide and support a complete "solution". I.E. there is not much point to shopping around for "best of breed" services to integrate, "just give us your business and all your technology problems will go away". This creates an uphill battle in the marketplace for vendors selling what will be seen as "piecemail" solutions created by amassing various web-technology providers products across web-services.
  • There are huge unanswered usability concerns created by integrating together two different applications. This is espcially felt in functionality that is presented to an organization's contituents. Technology owners demand a seamless user-experience across their user facing application space (CMS, event tools, community tools, etc). In the future as more and more functionality moves out of the back-office and onto the web these concerns will only increase.
  • Open-Source vendors eat costs when integrating third party services, mantaining integration, and licensing software, so they have economic incentives to service a complete pre-integrated stack of technology instead of servicing a suite of other providers products integrated over web-services.

The future of the application stack and the role I think webservices will play

I hope and expect that in the next year the CS/CiviCRM/Drupal stack will evolve to a point where it can compete head on with the likes of Kintera, GetActive, and Convio. When this happens open-source vendors will grow into full blown ASP's that will be able to sell services that undercut the current market of proprietary service providers and will be able to grow downmarket to smaller organizations and horizontally to for-profits with overlapping technology needs. With so many organizations and vendors based on the same codebase it will create a very efficient marketplace that supports application development and services. It will also open up the marketplace to anyone wishing to specialize their services towards a vertical, sell data services, or offer 'best of breed' applications. Since the majority of the vendors business will be based entirely around customization and hosting / support and not licensing fees to support product development and sales, they will be much more likely to partner with 3rd party providers or create specialized services themselves. Over time as web-standards evolve the third party services and specialization will grow increasingly important in the marketplace. And then we can all live happily ever after....

Drupal Camp

Heads up! Noel Hidalgo has started to organize the first "Drupal Camp", a multi-day intensive Drupal developer training program. (The CivicActions folks are very keen on making this happen as well.)

Why?
Every Drupal shop I know of is in a crunch of Drupal development talent. The limit on their ability to take on more and bigger projects is not from lack of sales leads or lack of management capability, it is simply there are not enough seasoned developers on hand to pay to work on Drupal development projects. An experienced and talented web-application developer will take a bit of time to learn Drupal inside and out and get up to speed developing complex websites, so it is not simply a matter of bringing in fresh talent, we need fresh Drupal talent. The Drupal economy is rocketing skyward and this problem will only become more intense in the forseable future. This is an immediate and growing problem that we need to start solving now.

Who?

  1. Drupal newbies sponsored by their parent companies (CivicActions, Trellon, etc)
  2. Drupal engineering experts to lead the class. Anyone up for this? :) (Matt Westgate and/or Crunchywelch provided he doesn't drown? Anyone else interested?).
  3. Other Drupal folks in the neighborhood also might want to get together for a code sprint or to discuss Drupal projects.

When?
At Vancouver Josh and I floated out the idea of doing it in conjunction with the Personal Democracy Forum on May 15th. This dates works with me, does it sound reasonable to you?

Where?
If we do it in NYC in conjunction with PDF, I can think of two possiblities off the top of my head:

I will follow up with my contacts at both of these venues tomorrow. Does anyone know of another possible venue?

Curriculum

We should make sure that whatever is taugh and learned at "Drupal Camp" is recorded and synthesized for posterity. We want to have many more of these in the future so we should aim for publish-able curriculum that can be later posted on Drupal.org.

Next Steps?

  1. Gregory Heller is organizing a consultant conference call to talk about this and other issues shortly on the consultant mailinglist.
  2. David Geilhufe proposed we look into the former Ars Digita training camps. I'll sign up to follow up and do some research and try to talk to folks involved in organizing them.
  3. We need to figure out a data / location. Unless someone can come up with a better data / location than PDF on May 15th I say let's go with that. I will start looking seriously into venues tomorrow.
  4. We need to line up the teaching staff and get firm commitments from vendors to sponsor newbies. I can take this on starting Friday.

Thoughts?

Update: Some folks on the consultant mailinglist are motioning to do a training sooner (next 2-3 weeks) and somewhere less expensive than NYC.

YouTube, "the flickr of video"

To put this in perspective:

Why?

  1. Many more people are interested in watching crazy videos than looking at pretty pictures on the web.
  2. Flickr caters to the "boutiquie" web too-oh crowd while YouTube caters to the MySpace generation.
  3. You-tube had little competition out of the gate while Flickr had to deal with: yahoo photos, snapfish, kodack gallery etc.

Elgg and Drupal: Benefits to competition?

Bill Fitzgerald from the Elgg project recently responded to my previous post, "Moodle, Elgg, CivicSpace, CiviCRM and Drupal should join forces".

"All in all, it's an interesting idea, but not one I'd get behind. The different apps -- particularly Drupal and Elgg -- have some overlapping functionality, that indicate against joining the projects..... Variety is a good thing. To create a loose metaphor, I see it as akin to genetic variability. Differences within the open source community make us stronger."

Ben Werdmuller, the technical lead on Elgg followed up by saying:

"We're big on integration through open standards and would like to encourage that wherever possible; we're not so big on merging and creating bloated, monolithic systems."

In my experience with CivicSpace, Drupal, and now CiviCRM I have found the opposite of what Bill and Ben have said to be true.

Tight interlocking of open-source projects make us stronger, not differences

  • I can garuntee you that if we had chosen to build CivicSpace as a stand-alone app like Elgg instead of on top of Drupal the project would have failed.
  • At the recent Drupal Con in Vancouver there were well over 100 attendees, many of whom make a living around the platform. By my estimates about nearly a third of those people came to Drupal via the CivicSpace project. While our core code contributions to the Drupal project so far have been pretty modest our evangalizing and community support efforts (like DrupalDocs) have had a notable impact to the Drupal ecology.

Building on the same web-app stack makes for a cleaner implementation

  • If two web applications were not built from the ground up to work together they will not be easily integrated. We've seen numerous efforts in the Drupal community to try to integrate external user / content management applications such as PHPList and Gallery but even though the applications provide unique functionality, the integration difficulties are so intense it is yet to be seen if the effort is worth it in the end.
  • Drupal's succesfull Ecommerce module package is a full blown commerce system that handles all product point of sales, billing, and shipping, and tracking inside Drupal. It is a great example of the avantages afforded by building on the Drupal application framework rather than integrating an external specialized application like OSCommerce.
  • Even with an application like CiviCRM that was built from the ground up to be integrated into CMS's like Drupal it is still very costly to integrate.

Building Elgg independently from Drupal will lead to unnecessary competition instead of collaboration between the two projects

  • Not a single functionality point of Elgg is unique. Drupal + contributed modules can be configured to power a site that does _basically_ what Elgg is designed to do. This means virtually all engineering effort going into Elgg is in duplicate to work done in the Drupal community.
  • Decision makers in Universities IT departments will be forced to choose between adopting Elgg or Drupal instead of being able to invest in a solution that captures the best of both projects.
  • The KDE vs. GNOME battle is a great example of the negative effects of open-source project competition vs. the dramatic recent success of the Ubuntu project built on top of Debian.

Digging into OpenBRR of Moodle and Sakai

"Business Readiness Ratingâ„¢ (BRR) is being proposed as a new standard model for rating open source software. It is intended to enable the entire community (enterprise adopters and developers) to rate software in an open and standardized way....sponsored by Carnegie Mellon West Center for Open Source Investigation, O'Reilly CodeZoo, SpikeSource and Intel...BRR will give companies a trusted, unbiased source for determining whether the open source software they are considering is mature enough to adopt.... The ultimate goal of BRR is to give companies a trusted, unbiased source for determining whether the open source software they are considering is mature enough to adopt. It will help adopters assess which open source software is best suited to their needs and enable them to share findings with the community."

Functionality (25%)
Moodle: 5
Sakai: 3

Usability (20%)
Moodle: 4
Sakai: 4

Documentation (15%)
Moodle: 3.1
Sakai: 3.1

Community (12%)
Moodle: 4.2
Sakai: 3.2

Security (10%)
Moodle: 4.2
Sakai: 3.4

Support (10%)
Moodle: 4
Sakai: 1.5

Adoption (8%)
Moodle: 4.4
Sakai: 4.2

(Note: the Y axis scale is 4.1 - 4.4, not 0-5. Sorry, couldn't figure out MS Word)

Total Weighted Score
Moodle: 4.19
Sakai: 3.23

More

Oracle is trying to buy open-source

Read the story here

  • Jboss is asking for $400M
  • Zend is asking for $200M

Why? This is pure defense, not an open-source embrace. Anyone who thinks Larry Elison loves the penguin is smoking crack. Oracle makes it's billions on licensing, Jboss and Zend make their millions on extensions and support. The open-source stack commodifies Oracle cash cow and sucks it's margin's bone dry. How many people who work inside Oracles big black tower read slashdot and play with LAMP or Ruby-on-Rails in their free time? Enough to know closed source ERP is fucked. Enough to freak out Elison. Enough to spend a billion or so squashing some peons. Zend is the ringleader of PHP and Jboss is the darling of open-source Java and Oracle is trying to take them out.

Higher-ed LMS market penetration: Moodle vs. Blackboard+WebCT vs. Sakai

  1. Moodle: 2,981 deployments / 54% market share
    8,772 X (348 / 1024)
  2. Blackboard + WebCT: 2,500 deployments / 45% market share
    3,700 - 1,200 K-12
  3. Sakai: 35 deployments / .63% market share
  4. Taken from the Sakai partners page.

Related Posts:
Sakai vs. Moodle
Digging into OpenBRR Rating of Sakai and Moodle

Costs of forking

A lot of full time Drupal/CS consultants I've been talking to here in Vancouver have been sharing concerns about how to meet the needs of their clients while still being able to contribute code they development back to the Drupal community. The general assumption is that most clients are oblivious to the mechanics of open-source communities and assume that they are paying simply for an engineer to develop new functionality and ignore all the time required to work with the Drupal community to make sure contributions and patches are accepted and merged back.

I think the simplest solution is to figure out a way to asses costs to forking code that can be communicated to customers upfront when contracts are negotiated.

How about this estimate?

  1. Costs to mantain your own forked module: 25 developer hours X $100 an hour X 2 times a year = $5,000 a year per module.
  2. Open-Source development resources lost: 5 hours per patch X $100 an hour = $500 per patch.

Using these estimates I would think that consultants should be able to justify adding in a line item to their contracts up front to cover their costs to contribute their code back to the community. Do you think this will work?

Supporting Drupal platform development

I am posting from the basement of 800 Robson Center Vancouver BC at the Open Source CMS Summit / DrupalCon 2006. This is without a doubt the most fun work-travel event I've taken part in (I've done more than twenty in the past year). One year ago at the DrupalCon in Vancouver there were about a dozen of us. In Portland six months ago there were thirty or fourty. Today there are well over one hundred Drupaler's in attendance hanging out, learning, hacking, discussing, drinking beer, and scheming for Drupal world domination.

The Drupal software, the user base, and espescially the economy around the platform have seen an explosive growth in the last year. When we started CivicSpace a year and a half ago there were only a few of us who made a living in and around Drupal. Today there are well over one hundred people employed to hack on Drupal between the contract firms and the user-developer base. With a little napkin math I estimate that in a year the Drupal economy has grown from ~$500K a year (10 X $50K) to well over $5M a year!

Despite this phenominal growth I have some concerns about what the next year holds in store for the Drupal platform and the community around it. As I see it, most of the value created by the Drupal community that vendors and consultants rely on to sell as services to customers come from unpaid core contributions. Examples of this are the majority of 4.7 improvements: multiple block regions, new forms API, installer work - these were all unpaid contributions to the community by volunteers or pro-bono work from Drupal firms. At the same time, many of the seasoned Drupal core hackers are getting job offers left and right to come and do Drupal consulting full time. This is creating a scenario in which while many more people are paying their bills through Drupal hacking and more and more business are built around business value created by the Drupal community, it is not getting any easier to make sure the heavy lifting on core work is getting done. For example: I know of no funding availiable to support key Drupal core projects such as continued forms API work, actions & workflows development, CCK, and most importantly; usability improvements.

In the next year I am expecting the economy around Drupal to continue on it's fantastic pace. I would not be surprised if by 2007 the economy was well over $20M annually. My big question here in Vancouver is: What can we do now to make sure that some of the money coming into the community over the next year is put back in to supporting the core work that we as a community are relying on?

Syndicate content