Shifting Search from Static to Real-time

By: John Battelle

I've been mulling something that keeps tugging at my mind as a Big Idea for some time now, and I may as well Think Out Loud about it and see what comes up.

To summarize, I think Search is about to undergo an important evolution. It remains to be seen if this is punctuated equilibrium or a slow, constant process (it sort of feels like both), but the end result strikes me as extremely important: Very soon, we will be able to ask Search a very basic and extraordinarily important question that I can best summarize as this: What are people saying about (my query) right now?

When it first hit critical mass, it seemed Google answered this question. For the first time, you could ask a question in your native tongue, and get an answer. It felt immediate, but save for the speed with which the search results were rendered, it was not. Instead, it was archival - Google was the ultimate interface for stuff that had already been said - a while ago. When you queried Google, you got the popular wisdom - but only after it was uttered, edited into HTML format, published on the web, and then crawled and stored by Google's technology. True, that has sped up - Google indexes a lot of sites more than once a day now - but as it nears the event horizon, this approach to search won't scale.

In short, Google represents a remarkable achievement: the ability to query the static web. But it remains to be seen if it can shift into a new phase: querying the realtime web.

It's inarguable that the web is shifting into a new time axis. Blogging was the first real indication of this, but blogging, while much faster than the traditional HTML-driven web, is, in the end, still the HTML-driven web. To its credit, Technorati saw blogging as the vanguard of a shift to real time, and tried to become the first search engine for "the live web". It failed to gain critical mass, but I think the main reason was that the web was not yet "alive".

That is changing, rapidly. Yes, I'm thinking about Twitter, of course, which is quickly gaining critical mass as a conversation hub answering the question "what are you doing?" But I'm also thinking about ambient data more broadly, in particular as described by John Markoff's article (posted here). All of us are creating fountains of ambient data, from our phones, our web surfing, our offline purchasing, our interactions with tollbooths, you name it. Combine that ambient data (the imprint we leave on the digital world from our actions) with declarative data (what we proactively say we are doing right now) and you've got a major, delicious, wonderful, massive search problem, er, opportunity.

And with that search challenge comes an equally exciting monetization opportunity.

Imagine AdSense, Live. Here's my scenario:

Let's say you are in the market to buy something - anything. Just to keep things simple, I'll use the age old digital camera example. Say you are considering a Canon EOS. Before the shift to the live web (for most mortals, that'd be now), chances are good you'd start by Googling it.

Given the way we humans work, the results are far from helpful, to my mind. You get a list of top pages for "Canon EOS", and you are off on a major research project, trying to make sense of what folks have written about Canon, or what Canon has to say about its products, comparison shopping engines on price...it's not a very good experience. Wouldn't you feel better if you could just start by asking a trusted friend? But one that has the scale of Google?

So imagine a service that feels just like Google, but instead of gathering static web results, it gathers liveweb results - what people are saying, right now (or some approximation of now - say the past few hours or so), about the Canon EOS? And/or, you could post your query to that engine, and you could get realtime results that were created - by other humans - directly in response to you? Well, you can get a taste of what such an engine might look like on search.twitter.com, but that's just a taste. Add in your social graph (what your friends, and your friend's friends are saying), far more sophisticated algorithms and - most importantly - a critical mass of real time data - and those results could be truly game changing.

Now, imagine what AdSense might look like next to those results? Of course Canon will want to be there, pitching why its EOS line is the best, and of course, so will all of its competitors. Just like AdSense now. But instead of static text ads, these ads would be the beginning of true conversations between those brands and yourself.

And that's what's got me so excited. It's coming, quickly, and the game is truly afoot.

Comments

[katheryn]

How long till the next book john?

[Kimbal Musk]

Great post John.

When you get a minute, check out OneRiot.com. We just launched and I believe we're taking a piece of the problem you've outlined above. OneRiot is a search engine that let's you find the pulse of the web. The content people are talking about today.

We do that by having over 2 million people share their activity data with us. We process their activity in real-time and create the first real-time index. The web as it is today.

So if you search for Barack Obama, you find what content people are talking about today, vs. barackobama.com or his wikipedia page.

We're tackling news first. We'll do videos and products next. Within a few months we might even have an answer to your Digital Camera example. :)

Thanks again for another insightful post on the future of search.

Kimbal

[John Battelle]

That depends on how big a book it is. I keep making it bigger. That is the problem!

[Terry Jones]

Hi John.

One of these days I hope we'll end up talking. I agree with your sense that search will undergo an important evolution. I think I see one way that might happen. Maybe of interest (see the video links at end):

http://twitter.com/Scobleizer/status/1037537924
http://twitter.com/timoreilly/status/1038946294
http://www.fluidinfo.com/terry/2008/12/04/twittendipity-a-chance-interview-with-robert-scoble/

Terry

[Gregory Banse]

Do you think live feeds be treated similar to how newswires were 30 years ago - considered a pay-for service?

[William Mougayar]

Great article. In the last 2 paragraphs:
- you've described how Twitter could start making money (via its search)
- made me think of the possibility of Google buying Twitter

How different from Twitter search should Google's indexing of Twitter be? Their blog search is dismal because they're searching good with junk.

[Brian Zisk]

Hi John,

Think you're really going to like what we're building over at StanzIQ.

Hope all is well. It's been a while.

All the best, and thanks for this posting.

Brian Zisk

[WannaDevelop.com]

Good write up, John.

Re-read the article twice just to make sure I understand what you are saying correctly... It does make sense --- but if it is that good and useful, I wonder why hasn't Google been experimenting with it? They are the "innovator" in every sense of the word when it comes to search and I am sure they have something similar coming... Nothing from Yahoo or MSFT either... Hmmm..

I think Twitter is flying high with all it's popularity and buzz and should take advantage of this and run away with some of the opportunities. Another major player in the search field would be welcomed --- although Google and MSFT have plenty of cash and could acquire them for a nice sum :)

I look forward to a follow up on this --- good stuff.

Mike

[Sameer]

Great post John!

[John Battelle]

Thanks all. Am reading and grokking comments, back at you soon...

[Brian Despain]

Look at those twitter results, I am wondering exactly what utility they actually bring? I mean what value to the user? To be frank I care less what my friends think about the Canon EOS than what the opinions of professional photographers. In that regard there need to some method for improving authority. My social graph is my social graph -> it's of dubious value to me for making buying decisions.

[Lee Zukor]

Great post, continues to generate lots of discussion in our office. The point you raise about what this feels like to users is especially near to me -- it's one thing to bring back real time results, and another thing entirely to present them in dynamic, useful ways.

[Cure Dream]

I'm not all that concerned about what twit twittered what in the last 24 hours, and I think that most of the people that do are twits.

For instance, if I was researching a camera or a car, I'd be interested in the best stuff written about it in the last year or so, not in the last five minutes.

Sure, a public relations flack might want to keep track of bad things people say on twitter so they can have their lawyers send them nastygrams, but for ordinary people, it's just a waste of time. Entertaining maybe, but a waste of time.

[Liora Blum]

As a graphic designer putting out numerous print publications, I see that Twitter generates credibility by being at the forefront of what is being said, whether or not older information is available.

[Abhishek]

Right on. Its not just realtime search, there's a lot more that can cash in on this (and provide great user experience in the process). I talked about it at Barcamp3 at Singapore (http://xploretech.blogspot.com/2009/03/barcamp-singapore-feb-28.html )

[Viagra Online]

hello friend excellent blog about Shifting Search from Static to Real-time thanks for sharing!!!!

[the canadian pharmacy]

Because Viagra's success in treating ED is mainly due to interaction with the cardiovascular and hormonal system, men find Viagra comes with lots of disadvantages. Viagra works to maintain naturally occurring chemicals within the body.

[Acai Extreme]

awesome thanks for the post.. My leadership has surely improved.

[Buy Viagra]

thanks for sharing this important information Shifting Search from Static to Real-time this information is very professional

[buy viagra]

nice info thanks

[Buy Viagra]

Hi, great post. thank you!

[Buy Generic Viagra]

The best on the world.Thanks!Buy Viagra online

[Buy Cheap Viagra]

The best website in the world.

[Buy Viagra]

Hi. Great post. my site

[buy levitra online]

Thanks for another insightful post on the future of search!