Home › Blog › Zacker's blog

Web services are no silver bullet

John Stahl recently gave me some heat for my assertion that Elgg should be built on top of Drupal.

I think that the next few years are going to bring tremendous challenges for applications that do not easily communicate with other applications that are “outside their platform” i.e are written using a different language/framework, run on a different server, etc....The days of monolithic application stacks that try to do everything are fading fast. A new “network-centric” software ecosystem is starting to bloom.

This is wishful thinking. I've spent much of the past few years puzzling over this exact question. While I am personally very much a proponent of web standards and web services I am pessimistic as to how much immediate impact they will have in the evolving marketplace of non-profit/ngo & advocay web technology services.

Backstory

I didn't always think this way. If you told me a year and a half ago that I would be hawking CivicSpace/Drupal as the über-platform that could meet virtually every need of any size organization I would have told you you were nuts....

After the Dean campaign ended and we started work on CivicSpace our assumptions were:

  • Drupal wouldn't be able to "scale" to meet the needs of large organizations
  • We would not be able to develop the platform quick enough to meet all the core needs of organizations any time soon
  • We would integrate third party service providers to fill the functionality gap organizations required

Then some interesting and unexpected things happened:

  • It turned out Drupal could scale
  • CiviCRM showed up and filled the functionality gap
  • Lots of vendors established quickly growing businesses servicing the technology. The top half dozen firms now employ ~ 50 people between them and each are looking to hire.

This reshaped my thinking on the future of the web-technology marketplace quite a bit. We have an immediate opportunity to commodify the core web-technology organizations need to a single integrated and scalable open-source application stack and this is a very good thing for the marketplace. Over the next few years I believe the advantages afforded by this stack of technology (CS/Drupal/CiviCRM) will far outweigh the benifits realized by the integration of applications accross web-services in terms of costs saved and passed on to technology owners and innovations in technology and services.

Integration accross web-services and web-standards is relatively costly

  • Web applications that are not built from the ground up to be integrated will never play nicely with one another. Let me repeat: if an open-source application is not built to be integrated with your web-application it probably isn't worth the effort to try to integrate it. This renders 95%+ of open-source web applications useless for those looking to leverage the work of other communities.
  • If an application has a strong API it is still tricky to integrate. Even integrating with CiviCRM, an application that was built from the ground up to interface with CMS's and shares the same web-app environment as Drupal, it costs us 2-10X more to integrate with it than it does to integrate with a standard Drupal module.
  • The most cutting edge private sector web-service and web-standards are not that advanced. We still don't have workable single sign-on solution, the cutting edge of semantic information interchange across the web is to embed it in XHTML, the hottest API's from SalesForce and Flickr haven't made much of a dent in the marketplace, and please don't get me started on the semantic web.

The market currently prefers a single integrated stack

  • The majority of the market for web-technology services is owned by companies like Kintera, GetActive, and Convio that serve as one-stop shops for virtualy all an organizations web-technology needs. They have trained technology owners to make purchasing decisions under the assumption that they can cut a check to a single entity that will provide and support a complete "solution". I.E. there is not much point to shopping around for "best of breed" services to integrate, "just give us your business and all your technology problems will go away". This creates an uphill battle in the marketplace for vendors selling what will be seen as "piecemail" solutions created by amassing various web-technology providers products across web-services.
  • There are huge unanswered usability concerns created by integrating together two different applications. This is espcially felt in functionality that is presented to an organization's contituents. Technology owners demand a seamless user-experience across their user facing application space (CMS, event tools, community tools, etc). In the future as more and more functionality moves out of the back-office and onto the web these concerns will only increase.
  • Open-Source vendors eat costs when integrating third party services, mantaining integration, and licensing software, so they have economic incentives to service a complete pre-integrated stack of technology instead of servicing a suite of other providers products integrated over web-services.

The future of the application stack and the role I think webservices will play

I hope and expect that in the next year the CS/CiviCRM/Drupal stack will evolve to a point where it can compete head on with the likes of Kintera, GetActive, and Convio. When this happens open-source vendors will grow into full blown ASP's that will be able to sell services that undercut the current market of proprietary service providers and will be able to grow downmarket to smaller organizations and horizontally to for-profits with overlapping technology needs. With so many organizations and vendors based on the same codebase it will create a very efficient marketplace that supports application development and services. It will also open up the marketplace to anyone wishing to specialize their services towards a vertical, sell data services, or offer 'best of breed' applications. Since the majority of the vendors business will be based entirely around customization and hosting / support and not licensing fees to support product development and sales, they will be much more likely to partner with 3rd party providers or create specialized services themselves. Over time as web-standards evolve the third party services and specialization will grow increasingly important in the marketplace. And then we can all live happily ever after....

Comments

Thu, 02/03/2006 - 12:30pm — David Geilhufe (not verified)

I love important conversations :)

Steve made a very important comment, IMHO.

But we in the nonprofit world shouldn't be building the enterprise apps with the full blown API.

Depending on the day I agree with him. If it's a day that nonprofit needs basically match a for profit business processes, I'm right there with him.

If it's a day when I'm trying to figure out what social change on the web is going to look like in the future, I diverge.

(1) Is social change is going to be defined by the needs of the needs of corporations who are generating sales via communication with buyers and consumers (i.e. where Salesforce's customers are)? This makes me nervous.

(2) The costs of an enterprise application is falling like a rock. Why give into the nonprofit culture of scarcity when an enterprise application with a full blown API developed within an open source ecosystem might have a price tag of a few million at _most_?

My practical side is very much in agreement with Steve-- lets build on what is already there. The side of me driven by core values is nervous about putting the tools and technology of social change into the hands of corporate America.

Wed, 31/12/1969 - 4:00pm — Zacker

Well..

Corporations have very little code in the CS/Drupal/CiviCRM stack and I would happilly put it up against the "best of breed" tools in the "enterprise" market today. And at the current rate of improvement I don't see how any closed-source providers will be able to keep up into the future. Should we be trying to build our own open-source version of the flickr service? No, we're probably better off in most cases integrating their service via API's. But should we be designing our own CRM system (CiviCRM) and building on top of "our own" open-source CMS system (Drupal)? Absolutely. We'd be sitting around waiting for corporate america to come build us the tools we need for far to long and what they would deliver would be a kludge to implement for the non-profit/advocacy sector. Besides, it is going to cost us and our users far less to build and mantain our own stack than it would cost to pay their licensing fees over time.

Wed, 01/03/2006 - 9:48am — Steve (not verified)

transition time

I think we're in a transition point between monolithic software and software as a service architecture. Youc an tell we're at a transition point because so many companies are saying, "Yeah, sure, we have an API," when they really don't. You can also tell we're at a transition point because there are a few companied out there that are clearly ahead of the rest of the world: Google, Amazon, and Salesforce.

There are billions of dollars being made using enterprise-grade web services--look at Amazon's developer program. You say Salesforce.com's API hasn't made much of a dent in the marketplace--I disagree. 15,000+ companies and nonprofits are doing their CRM with them. 50% of all the traffic to the Salesforce.com app is via the API. 50%!

They also have many partners who have or are currently developing software that adds functionality by interfacing solely with their web services API.

I think Salesforce.com is a bellwether for where software development is going. Get your core competency right. Make it a great app that's highly usable. Start with the API and build your app on top of that.

Is it an expensive way to build an app? More expensive than hacking it out in a scripting language. But we in the nonprofit world shouldn't be building the enterprise apps with the full blown API. We should be leveraging those platforms in novel interconnections. We should be building small applications that can plug and play APIs to do specific functions that are necessary to nonprofits.

There will always be a place for software that doesn't have an API, that tries to build all the functionality users might need directly into it's own code base. But that software will continue to lose ground to API-based products and novel combinations of those products.

This is a fun and important discussion to be having!

Wed, 31/12/1969 - 4:00pm — Zacker

not quite

I think we're in a transition point between monolithic software and software as a service architecture. Youc an tell we're at a transition point because so many companies are saying, "Yeah, sure, we have an API," when they really don't. You can also tell we're at a transition point because there are a few companied out there that are clearly ahead of the rest of the world: Google, Amazon, and Salesforce.

There are billions of dollars being made using enterprise-grade web services--look at Amazon's developer program. You say Salesforce.com's API hasn't made much of a dent in the marketplace--I disagree. 15,000+ companies and nonprofits are doing their CRM with them. 50% of all the traffic to the Salesforce.com app is via the API. 50%!

They also have many partners who have or are currently developing software that adds functionality by interfacing solely with their web services API.

I think Salesforce.com is a bellwether for where software development is going. Get your core competency right. Make it a great app that's highly usable. Start with the API and build your app on top of that.

I definately think API's are enabling old shcool web businesses like Amazon and Ebay to leverage more from their core competancy by opening up their platform to innovation, but the API's have not fundementally changed how they do business. Amazon, Ebay, and Salesforce's API's support a host of 'add-on' services and products but they clearly didn't roll out their API's "around the assumption that external applications are first-class citizens of their ecosystems". After all, why would they? That would have under cut their business model. The buy in that public API's and open-source is getting by the established players in the marketplace are mostly a result of defensive positioning agaisnt subversions of their business models, not because the businesses "see the light".

Is it an expensive way to build an app? More expensive than hacking it out in a scripting language. But we in the nonprofit world shouldn't be building the enterprise apps with the full blown API. We should be leveraging those platforms in novel interconnections. We should be building small applications that can plug and play APIs to do specific functions that are necessary to nonprofits.

There will always be a place for software that doesn't have an API, that tries to build all the functionality users might need directly into it's own code base. But that software will continue to lose ground to API-based products and novel combinations of those products.

Any web-application these days not build with a strong internal API is probably not going to make it. But I am still very dubious of the notion that it makes more sense to focus on integrating external applications in the short term rather than integrating directly onto our stack. It simply costs way less to integrate directly and affords the users and vendors much more. But sure, let's experiment with external API's and see how many third party services we can integrate over time.

Wed, 01/03/2006 - 5:18am — Nick Lewis (not verified)

Word. In my mind, you pretty much summed it all up. Especially your comments on the hellish costs of integrating outside webservices to a cs/crm/drupal platform.

Not to mention the majority of "APIs" provided by ASPs are for show, much like hiflautin titles which non-profits throw around to their ranks-- I think I'm going to ask for "Senior Executive Director of Web Based Technologies and Initatives" at mine. I feel such a title would make me sound pretty durn important.

Wed, 31/12/1969 - 4:00pm — Zacker

:)

Nick, you're the funniest hifalutin Senior Executive Director of Web Based Technologies and Initatives I know.