information overload

As I’ve mentioned before, blogs and RSS are eating up a lot of my time. Scanning some 200-400 posts a day is grueling, but it’s a requirement of the breakneck speed at which the space is developing. Unfortunately, the explosive growth of the blogosphere is proving difficult to tame from a user’s perspective.

I would attribute part of the problem to the cyclic nature of the blogosphere. First, someone posts an interesting story – if you’re particularly unlucky, you’ll be subscribed directly to that blog and see the story the first time. Then a bunch more people will link to it – and again, if you’re subscribed to those blogs, you’ll see the story for a second time. At this point, the story will start to bubble up through the ecosystem of aggregator sites until it shows up on the radar of sites like Popdex, Blogdex, and Technorati. And then the mainstream media gets a hold of the story, and we go for another twirl on the information overload merry-go-round.

To extend my earlier thoughts on the need for a better user interface for RSS aggregators: an aggregator should not only have the ability to group related posts, but should have memory. By “memory” I mean that if I delete a group of posts on a particular topic, I should be able to make them go away. Forever. Now, some of this may be me wishing for a computer to read my mind, determine what I’d like to read about, and spit it out to me (I seem to recall something like this in Arthur C. Clarke’s novel Cradle). The idea would be that if I no longer care about the most untranslatable word, or how required registration is a dumb idea, then I don’t have to keep seeing posts related to those stories in my aggregator.

Part of the solution to this problem requires RSS (or Atom) to incorporate a mechanism to tell the aggregator about the “root” story URL. But what is the true “root” story – heck, even smart guys like Joi have to pause to consider who to credit as the source for a post. What chance does a piece of software have?

Nevertheless, it would appear that unless we start thinking about how to address this phenomenon, because it’s only going to get worse. So, step one: name the enemy. What do we call the problem of stories endlessly ricocheting around the blogosphere? The Blog Bounce? The Blog Echo? Hmm. Not snappy enough. Any thoughts out there?

I’m embarrassed to say I only bothered to install an RSS aggregator last week, after I became painfully aware that the bookmarks folder in my browser is the web equivalent of a roach motel: web sites check in, but they don’t ever get checked out. Using RSS to do the heavy lifting seemed like a good idea to keep me informed – hey, if even the Canadian government is hip to RSS, what have I got to lose?

I’ll tell you what: a giant pile of my spare time, that’s what.

It’s not that RSS itself is a bad idea – serving out content in an XML format that can be easily parsed and aggregated is a great idea. I, like any self-respecting, cutting-edge geek, want the newest information and I want it milliseconds after it’s been captured by sensor-studded bloggers on the scene to capture the moment. The problem with RSS is you start drowning in information, and most of that information is essentially identical.

Everyone starts assembling their list of RSS feeds by hitting the big sites first – Slashdot, Wired, CNet, Scripting News, Boing Boing, and anyone else that appears in the top ten of Technorati. And then the updates come streaming in…and in…and in…and in. Who would have thought that 6 billion people could generate so much information? Suddenly I’m dealing with the problem of cleaning out my RSS aggregator in a computer-age equivalent of beat the clock.

But it gets worse.

With RSS, every one of those sites you valued for aggregating news suddenly looks remarkably similar. For example, last week every one of these sites broadcast the latest story about Gmail, or a variant thereof – and there I was, drowning in a mass of RSS posts from different sites on exactly the same topic.

Whoopee, isn’t this magical.

Someone needs to take RSS aggregators/readers to the next level. What I’d like to see is an RSS reader that examines the links in various RSS feeds and assembles a hierarchy of feeds on a particular topic. So instead of seeing a zillion posts, I should see one – the root post to which all the other RSS posts point either directly or indirectly through other intermediate posts. Of course, this would require some work on the part of bloggers to identify the source of their information to enable readers to create such a hierarchy of related posts. For all I know, the solution already exists, but I haven’t heard about it because I ceased to surf the web sometime last week to delete posts in my RSS reader instead.

Grouping related topics in RSS would be one simple way to help people triage the deluge of new information available each and every day. If RSS readers grouped related items, I could delete a “tree” of RSS posts on a topic if I decided I didn’t care about the topic. On the other hand, if I did care to see the commentary, I could “zoom in” to the related RSS posts to view related information and commentary. That way, I could not only get all the latest information using RSS, but also actually find the right information from sites that aggregate news.

We need to start finding smarter ways to present information – all our efforts to enable intelligent access to information still feel really ham-fisted. Clever, but ham-fisted. We have more access to data than ever, but I fear our access to knowledge remains either unchanged or increasingly impaired. Though RSS gives you access to all the information you can eat, our stomach for information is finite. Nobody needs every piece of available information – they need the ability to spot the high points on the sea of information and navigate towards the shores of those information landmarks they actually care about. To do otherwise is to leave the user treading water.