Information Overload: Can you see what's coming?

As I've embarked upon a new adventure and am in the formulation stage by necessity -- and of interest since it helps me get even better at connecting the dots -- I've been investing in research time. It's key to the path I've chosen to be even more on top of the changes and thought leadership surrounding blogging, podcasting, video blogging, RSS, microformats (and all the other enabling technologies and combinations thereof). These developments are allowing us to be better able to tap in to the global conciousness of humankind like never before in history -- if there are methods at-our-fingertips to do so.

The river of content is flowing faster and faster. This river of content available on the internet is reaching flood stage and is in a variey of media types. As newspapers, magazines, radio and television lose eyeballs to the internet and become ever more desperate to cling to their advertisers, they are finding increasingly garish and dumbed down methods of getting the attention of the eyeball owners back (which, in my view, will only push people away faster). As broadband continues its adoption and more people get on the internet and attempt to connect their own dots, it's becoming exponentially more difficult to see or tap in to the collective consciousness and stay on top of changes in an industry, area of interest, or even to stay relevant in the workplace. Primarily it's more difficult to understand change and to see disruptive technologies or business models coming...and having time to act.

How can you answer the questions or understand the answers over time: What's going to happen to your industry or your job in the next five years?  To your children's future? To your health, old age, or social and political developments?  How will you know and/or see what's coming? Most importantly, how will you be able to stay on top of what Brad Pitt and Angelina Jolie are up to?  ;-)

Much of my research time has been invested in aggregating (using RSS in a news aggregator) and consuming the "river of content" that is flowing -- in what I now know is an increasingly difficult thing to stay on top of daily and keep my head above water as the water rises.

In discussions or debates with people, I used to be able to easily cite sources and statistics, "Well, The New York Times on Wednesday said..." or "...in the Wall Street Journal this morning the emphasized..." or "...and on NBC Nightly News last night..." but no more. I now am skimming and reading articles on dozens of news sites, technology journals, clicking on links of sources linked to by a blogger, and a whole lot more. The river of content is turning in to a flood and my instinct is to get to higher ground.

But where or what is the higher ground?

Newsgator currently is my aggregation friend. I like their user interface and, most importantly, that I can set it up to automatically remove articles as I read them and the it refreshes itself...as well as enabling me to leave one computer, go to another, and pick up where I left off (this is huge for me as I leave my main computer, use my portable, or want to show someone an article I've "clipped" quite frequently which helps remembering and able to cite my source).

Like all aggregators, I can easily add RSS feeds from any source in Newsgator that I decide is worthwhile...or take advantage of ones they recommend (Note: they, like many other aggregators, also have "smart feeds" that allow keywords to be used as filters by you so as to snag content from sources relevant to you that you may not have aggregated already).

I'm also enjoying "social smart aggregation*" sites that aggregate and identify "conversations" that are going on in the blogosphere (current favorites are tech.memeorandum and Topix (a site that also follows top stories in various categories like news, health, travel, etc.).

I've also been enjoying the onrush of sites like the social-promotion-of-articles site Digg which states as their tech-news-focused mission as, "Digg is a technology news website that employs non-hierarchical editorial control. With digg, users submit stories for review, but rather than allowing an editor to decide which stories go on the homepage, the users do." Now there is even an aggregator of technology aggregators called Diggdot.us which brings together Digg, Slashdot (the old hottest site of links to great tech articles), as well as the social bookmarking site del.icio.us (a social bookmarking site allows people to bookmark Web-based content they find worthy of bookmarking publically. They were acquired by Yahoo recently who obviously saw the power in their approach).

The problem for me is that what *other* people find is hot is not necessarily what I find hot or relevant (though I'm keenly interested in what the collective consciousness of people *do* find compelling making these social smart aggregation sites useful). To illustrate why I focus on what's upstream as far as possible, let me explain why I'm not very interested when goods get in to the mass market and are available in Target, WalMart or Best Buy. Using technology as an example, I get bored when I go to Best Buy since I'm aware of what is happening with components (e.g., Intel's chip roadmap showing which chips will eventually get in to computing-centric devices and thus finished goods), read Engadget, Gizmodo and DPReview among other gadget sites, so by the time it's on Best Buy's shelves, I know what I'll purchase, already purchased it online, or decided it's not for me.

What strikes me is that this river of content that I'm in is causing me to gasp for air as I am increasingly bobbing under water. Some times I don't where I am since the river banks are unfamiliar and the river is slowly moving faster. Too much content, too little time, too much happening on too many fronts. Fortunately I'm always in seeking mode and therefore am able to stay on top of things better than most, but I'm quickly getting in to information overload that even I am finding uncomfortable and unwieldly.

For example, I learn tons from podcast interviews with thought leaders. I now have 36 hours of quite good weekly podcasts to listen to with 10 hours per week (if I'm lucky!) available to listen. Sometimes I'll go through blogs and sites aggregated in Newsgator and find that, when I'm done after an hour of reading, hundreds of new articles appear when Newsgator refreshes! It's stunning how quickly this this has become a problem for me and one that I believe only algorithims and substantially smarter services could provide.

This content deluge -- and identifying trends and the nuggets of useful information -- is a huge problem for me and for you regardless of what industry you're in or job you have. It will become an even bigger issue when content producers around the world are increasingly accessible (with automatic translation getting more robust and accurate) and people interested in a topic or category are then compelled to *widen* the scope of information they want to consume, analyze and understand.

SOLUTIONS

1. Aggregators: there are a lot of them (web, email, desktop, and built-into-browser ones) and they're mainly focused today on the human made decisions on what 'feeds' to be aggregated. Aggregation companies clearly see this and are accelerating their ability to offer smarter and smarter service capability, by industry, and so on
2. Smart Aggregators: this name is my own made up one, but my definition is aggregator services who've made decisions on how to filter RSS feeds and/or have already put together those smarter and smarter capabilities (i.e., algorithims that parse feeds and categorize their contents based on some preset criteria) and, most importantly, are seeking ways to perform more robust pattern matching that will deliver increasingly more relevant and targeted content. In addition to tech.memeorandum and Topix described above, this category also includes:

• *Social smart aggregation. These aggregators use a concept of promoting articles aggregated based on the number of people who click any given article or link to it (see Digg, Diggdot.us and Slashdot descriptions above)
• HyperSuper
• NewsIsFree: In particular I like how they've created "channels" which they categorize and decide within which channel a feed should reside
• Newsburst from CNet: In my view, no company has better Web design than CNet. Take a peek at this list of their recommended sources. How easy is that to choose 'em? Their filtering isn't cutting edge, though, but that's probably OK since most people don't yet have a clue about aggregation.
• UPDATE: PubSub is a service I overlooked in this original post. "PubSub is a matching service that instantly notifies you when new content is created that matches your subscription. Using a proprietary Matching Engine, PubSub is able to read millions of data sources on your behalf and notify you instantly whenever a match is made."

Newspapers and magazines are becoming increasingly irrelevant (e.g., by the time I pick up my newspaper, I've already read virtually all the important articles online), so you can see how useful aggregators could be to you too. These aggregators will become ever more capable of defining trends and disruptors for every industry. In addition to just bringing together content from sites, blogs and sources that happen to share an affinity for some given topic.

FINAL THOUGHTS

How useful and cool would it be to have domain experts have tools at their disposable to create a smart aggregation for their own respective area of expertise and either engage their affinity group or publish their set of aggregated feeds?  There are millions of content producers and bloggers with a perspective and point of view and an expertise in some given area. Anyone looking to innovate, be plugged in or to understand....share with me a strong desire to stay on top of things and not drown in the exponentially increasing river of content and information affecting us and what we do.

My recommendation? Pick an aggregator and get signed up (most are free or have modest charges for more features). Choose from their pre-defined categories or recommendations. Add a few of your daily reads (most blogs have RSS built-in and you can copy the feed URL and paste it in to the aggregator thus adding it to your list of aggregated feeds). You'll soon see the power at-your-fingertips and understand what I'm talking about with "smart aggregators".

CAUTION ABOUT AGGREGATORS: If you decide that aggregators are for you -- and especially if you use a web based aggregator like I do and invest a lot of time inputting all your desired feeds -- there's no easy way to export your feeds and switch to another aggregator!  Though I understand a news aggregator's reluctance to lower switching costs, it still bothers me, for example, that Newsgator's OPML export file (the XML file that describes all my subscriptions to RSS feeds) doesn't validate.

When I tested an export from Newsgator and an upload in to Google Reader as though I was going to switch, the subscriptions and articles didn't appear since the OPML apparently isn't valid). Though I edited the OPML file in a text editor (still wasn't successful), who the hell is gonna do that!?!

 

Comments

[Scott Quick]

Hi Steve,

Invite you to learn more about Attensa at:

http://www.attensa.com/
and...
http://craigslemonade.typepad.com/weblog/

Feel free to contact me if you have additional questions.

Scott

[Laurence Timms]

I struggled with the same problem when I was trying to read something like 400 feeds a day. I realised that the only reason I was reading around 80% was *just in case something interesting came up* - and 80% of the time, it didn't.

So with 80% of the feeds being irrelevant 80% of the time, I tried using smart feeds from PubSub and Technorati. These guys asked for tags/keywords to define my smart feeds. But I didn't know what those tags were. There's no tag that defines 'interesting stuff'. Or even one that defines 'interesting tech stuff'.

I decided (like Larry and Sergei) that interesting stuff is usually defined by reference; the more people that link to something, the more interesting it is. To add another angle, if people are linking to something *now*, or near-now, then it's even more interesting.

This is the basis I've used to create http://chuquet.com. It scans a rich set of high-value feeds and distils the most 'interesting' stuff out of them.

At the moment it just does this across the board without any particular focus (it doesn't distinguish between tech and politics for example). The next step is to allow people to add a degree of personalisation to describe to chuquet what *kind* of things they find interesting. I see this as a long term relationship. It's not just a case of enter tags, get feed. It's a case of enter tags & the url of your favourite blog & your opml, get feed, tune feed over time. Chuquet will watch the links you click and tune the feed to give you more of what you like over time.

The reason I haven't publicised chuquet.com is because I feel that until you can get a degree of personalisation from your 'interesting' feed, it's no more than an automatic link referral service. Good, but not great. When the personalisation stuff starts to go live over the next two weeks then I'll start pushing chuquet.com a lot more.

I guess my take on things is a lot more simplistic than yours, but I appreciate your comments and feedback.

[Anne Z.]

"The problem for me is that what *other* people find is hot is not necessarily what I find hot or relevant"

I definitely have that problem. I'm interested in what the mom blogs are talking about as well as anything about women in tech and then lately I'm into RSS, OPML, attention.xml, reading lists, etc. I need a way to find the nourishment in this watery soup of information. I'm subscribed to over 100 feeds right now and that's about as many as I can handle with Bloglines simplistic approach to aggregation.

I really enjoyed your post. Love the idea of trying to get to higher ground.

[Lisa Williams]

Wow! Great post.

[Lisa Williams]

Wow! Great post.

[Chris - Touchstone Gadget]

Heya Steve!

This is an awesome post. I think everyone is facing this problem to one degree or another - and I think you have articulated better than most!

We have been working to solve it by approaching the solution with an understanding that:

1. People have other things to do - so there should be a way to get a 'heads up display' of information while you work

2. The heads-up-display should only interrupt you when something immediately important happens

3. Information comes from places other than RSS - so you need to support multiple sources through adapters.

So we are trying to do all this with our 'all-purpose heads-up-display gadget' called 'Touchstone'.

You can check out more info at www.touchstonegadget.com.

It's not designed to replace the aggregators you spoke of - merely extend them into your periphery vision while you work on other things.

This short circuits the idea of 30 hours of read in a 24 hour day!