The Long Tail

First there was his article in Wired, then there was his blog (i.e. ideas beta test bed), and now there is Chris Anderson’s book, which I recently finished reading and highly recommend: The Long Tail: Why the Future of Business is Selling Less of More.

The Long Tail

The Long Tail” comes from a type of statistics phenomenon known as a “long-tailed distribution” wherein the tail of the curve is very long relative to its head.

I see that Nicolas Carr is trying to further “the debate about the extent of the Long Tail phenomenon” by capturing some of the recent dialog between Chris Anderson and Lee Gomes of Wall Street Journal. Chris responds to Lee afterward here. Before that exchange, Tim Wu of Slate penned a review of The Long Tail entitled “The Wrong Tail: How to Turn a Powerful Idea into a Dubious Theory of Everything.”

While I don’t think that long-tailed distributions are under every rock in the quarries of culture and commerce, I can’t help but wonder what the implications of The Long Tail are when considering ECM and SOA.

I sketched the following on my office white board while discussing my impressions of this book with some of my colleagues:

White boarding TLT

Example music purveyors above are listed as examples of the following economies: physical goods (atoms) in physical stores (Tower Records), digital catalogs of physical goods with warehouses help to mitigate some of the zero-sum shelf space game)(, and digital catalogs of digitals goods (bits) with digital distribution that is ideally end-to-end (Apple’s iTunes).

Is there a long tail for content and/or a long tail for content services and solutions? Common content management (CM) needs (necessities vs. hits) are at the head (i.e. what Gartner calls “Basic Content Services”), and specialized CM needs (niches) occupy the (substantial?) tail.

Chris’s remark–“For too long, we’ve been suffering the tyranny of least common denominator fare”–resonates with me where most mainstream ECM functionality is concerned. Checkin, checkout, versioning, branching, lifecycle, workflow, etc. are powerful atoms but they’re meant to be a means to a wide variety of perhaps domain-specific ends. They’re the words that should make up the sentences and even chapters of ECM.

Applying to ECM enables platform vendors like EMC to provide a range of high-value CM services to its diverse customer base. “Clumps of services” can be bundled and sold according to the target domain or industry. Customer-driven composition, instead of code-based customization, can take effect, in part, because services are free from specific user experience and because their bundling is designed to address the task-centric workplace and the need to do more with less (e.g. a bundle of ten services should yield more than ten solutions–some with user experience and others solely focused on back end automation).

It seems clear, too, that vendors engaged in this pursuit must provide more than just a “veneer of variety.” By this, I’m referring to the book’s discussion about some of the perceived paradoxes surrounding increasing choice. The author contends that the process of choosing rather than the number of choices has more to do with subsequent motivation to buy or a sense of confusion and even oppression. Is the service consumer debilitated or liberated? Are the choices ordered in ways benefiting the consumer? Is informed choice enabled?

And what about the nature of service discovery in today’s SOAs? Public UDDI-based registries haven’t panned out as originally planned, and even within enterprises, UDDI alone often isn’t a sufficient basis for discovery. If the notion of filters as presented in this book (i.e. recommendations, rankings, search, taxonomies, etc.) has application in SOA, how will standards such as UDDI evolve accordingly? Given that traditional telephone company yellow pages are essentially service provider-based ads, is “UDDI as the yellow pages for web services” enough for informed discovery within the longer tail of services, CM-related or otherwise? (Ads shouldn’t be confused with recommendations.)

The book talks about the fractal nature of The Long Tail–tails within tails and “self-similarity at multiple scales.” ECM has evolved to address a particular set of broad requirements (e.g. web content management (WCM), digital asset management (DAM), collaboration, compliance, archiving, imaging, reporting, search, etc.). Each of these industry segments represents its own tail. Each vertical served by a particular set of product offerings represents another tail. Each of these tails should be served by a set of tailored filters to increase customer confidence that specific business needs will be met. Out of this fractal environment ontologies emerge (e.g. ways to organize ECM services)–both on the part of the provider and on the parts of consumers with robust business intelligence and analytics in place to reveal discrepancies and opportunities in near real-time.

A couple of ideas hit me while I was reading: (1) Playlist sharing and OPML sharing are increasing in popularity as ways to recommend music and content to others. Why not provide the ability to share content subscriptions within ECM repositories, too, perhaps leveraging something like OPML? Consider the typical project team whose membership may change over time based on its development phase. Being able to pass such information easily to new team members could greatly decrease ramp-up time. (2) What about realizing a standard way to auto-discover OPML much like RSS/Atom (feeds) can be auto-discovered today? This realization could enable OPML-based community formation (i.e. my OPML references a set of feeds; if one of these feeds has OPML associated with it then continue to fan out; …).

In unfinished form, here some more of the impressions this books has left with me. All quotes are from the book’s author unless attributed otherwise:

  • Three forces of The Long Tail: Make it – democratize the tools of production; get it out there – democratize the tools of distribution (e.g. cut costs of consumption; bits vs. atoms); help me find it – connect supply and demand
  • Identify niches and serve them authentically
  • Affinity-based and massively parallel culture
  • Mass culture versus micro-cultures massively parallel – how to avoid divisiveness, factions, walls, etc.?
  • “These days our water coolers are increasingly virtual – there are many different ones, and the people who gather around them are self-selected. We are turning from a mass market back into a niche nation, defined now not by our geography but by our interests.”
  • Redefining connectivity, community, commerce, business, culture, allegiance, affinity
  • Greater choice involves solving a specific need/want WELL
  • Choice = competition, distraction, confusion; therefore, increase quality and experience
  • “Market of multitudes” and “mass of niches”
  • Consumer market of non-compliance/non-conformance versus business compliance as a potential challenge (friction) and opportunity
  • Physical versus online (virtual) boundaries; enterprise versus consumer boundaries
  • Professional < -> Prosumer < -> Consumer
  • Consumer becomes producer: “It is when the tools of production are transparent that we are inspired to create.”
  • The book’s discussion about content reuse in the form of house music makes me think that house software or house apps could be additional labels for today’s mashups.
  • Availability at reasonable cost both to the producer and the consumer
  • [The Long Tail as] “the true shape of demand in our culture, unfiltered by the economics of scarcity”
  • “Our growing affluence has allowed us to shift from being bargain shoppers buying branded (or even unbranded) commodities to become mini-connoisseurs, flexing our taste with a 1000 little indulgences that set us apart from others.”
  • How will consumers behave in markets of infinite choice?
  • Recall my first impression of Rhapsody and later of Pandora (blogged both herein)
  • “The biggest money is in the smallest sales.” -Kevin Laws
  • “Popularity is no longer a monopoly on profitability.”
  • What “norms” today are “forced” and poised to break apart? What choices, boundaries, etc. are artificial? What advantage can be gained by exposing them as such (artificial)? For example, the author suggests that 30-minute TV shows (i.e. 22 minute shows with 8 minutes of ads) are artificial: “Demand [for TV] will shift to shorter content for convenience and entertainment, and longer content for substance and satisfaction. But the arbitrary middle will not hold.”
  • Re-evaluate what receives my attention? Even if it’s still deserving of my attention, should it continue to consume the same amount of my time (e.g. TiVo or your choice of DVR/PVR for TV-without-ads-and-on-your-schedule, 1.4x playback of audio, etc.)? Greater choice can raise the bar of what is worthy and what constitutes waste. It’s all about control (e.g. my control over news and the newsworthy, not someone else’s dictate that such programming occurs at 6pm or any other particular time).
  • Human beings are curious; enabling exploration is compelling.
  • “In a world of infinite choice, context–not content–is king.” -Rob Reid, (Rhapsody)
  • Filters – consider how to promote and encourage context via submission, especially as context would otherwise fall off and become harder to ascertain (subtleties and nuances) (e.g. consider what Pandora provides in its recommendation UI)
  • Continue feeding navigation and explanation (i.e. discovery > acquisition) but understand when the destination take over the journey, too.
  • Contextual pivoting (context pivots) – enabling different perspective to a given context (or jumping from one context to another)
  • Google as an example of providing different contextual views (e.g. text, pictures, email, video, etc.)
  • The Information Age is becoming the Recommendation Age (also the Participation Age) … recommendation = respected opinion … respect comes by reputation … reputation through interaction and engagement
  • “The motives to create are not the same in the head as they are in the tail.” (aka Reputation Economy)
  • “Niches operate by different economics than the mainstream.” So if you apply “scarcity thinking” to long tail content, much of it can be counterintuitive.
  • Difference between push and pull markets (e.g. shrink-wrapped software vs. “)
  • “One size fits one. Many sizes fit many.”
  • WAS: dozens of markets of millions; IS: millions of markets of dozens
  • “Transparency can build trust at no cost.”
  • “Don’t predict; measure and respond.”

Finally, the author states: “Although the decline of mainstream cultural institutions may result in some people turning to echo chambers of like-minded views, I suspect that over time the power of human curiosity combined with near-infinite access to information will tend to make most people more open-minded, not less.” While I hope that he is right about open-mindedness, I wonder what the expression of this will become. Will the masses become more engaged in and committed to the process of discovering truth among opinions, or will we suffer mental atrophy and apathy by taking the easy way out and settling for merely something aligned with personal or collective bias? This is the challenge and opportunity of probabilistic systems where certainty and likelihood are concerned: you enter cone of uncertainty and are able to leave before absolute truth is acquired. Will we experience an erosion of critical thinking?

“With great power comes great responsibility,” and this applies more than ever to identifying truth in a relativistic society. Don’t confuse opinion with truth; don’t confuse judgment with discernment.

Update 12/1/2008: For more of my book reviews and to see what else is in my book library (i.e. just the business-related or software-related non-fiction therein), please visit my Books page.