Rants About Software

There's been a lot of talk lately about Real-Time Search. During the past few months, Google has added a real-time area to it's search results, Facebook and FriendFeed now retrieve real-time results, and new search engines devoted to real-time, such as Collecta, have sprung up.

Real-Time Search is Social Media Search

In it's current state, real-time search is little more than social media search. Google's real-time capabilities seem to be primarily a listing of recent Twitter updates. This is certainly a logical starting point. In fact, many people claim that Twitter IS real-time search. To an extent, I agree.

What's a popular news topic right now? Go on Twitter and search for it. What do you get? You get a list of people posting thoughts, comments, links, images, and videos all pertaining to the topic. This isn't a typical search result list of week-old news articles. This is what people are thinking and saying RIGHT NOW!

Until fairly recently, real-time search would not have been feasible because we lacked the two main components that make it possible: Social Media and Smart Phones.

Real-time search simply cannot work without these two things. Social media outlets are necessary because that is where the content is being stored and pulled from. Smart phones are necessary because that is how people are able to post this information as it's happening. Picture this scenario...

You go to a concert. You record a video of the band. You post the video on YouTube. You mention the video on Twitter. You post a link to the video on Digg. You do this all on your iPhone... while you're still at the show. Within minutes, people from all of the World can watch your video, leave comments, share links, etc. Now suppose there is someone else who is interested in the band or that specific concert. They can now do a search and get a feed of all of this content as it's happening!

Clearly social media services, most notably Twitter, are the key to aggregated real-time search results. But what does this mean for search engines?

A New Problem for Search Engines

So we now have people constantly submitting content that can potentially be sucked up by some type of real-time aggregator/search engine. But how does a search engine determine what results are relevant? The first, and somewhat trivial, step is to merely search the major social media outlets for the given keywords where the time stamp is somewhat recent. But here's the problem.

How do you distinguish the "good" content from the "bad"? These aren't just normal web pages that we're searching for. We're not choosing search results based on their PageRank. We need some way of determining what content is substantial and what is useless. Consider the concert scenario again...

We have people at the show who are posting pics, videos, and comments. The majority of this would most likely be considered "good" content. But let's suppose that at the same time, there is someone else who is NOT at the show, but they are posting a picture online of the same band. Is this content relevant to those who are looking for information about the concert? What if somebody posts a message saying that they wish they could go to the concert? Is that worthy of appearing in the search results? If not, then how can search engines determine that it's not?

You could come up with a thousand different scenarios and debate whether or not they should be included in the search results, and that's where the problem lays. There's no clear specification for what content would be considered good. Perhaps we could come up with some basic guidelines such as the popularity of the person posting, the location of activity, or the type and size of media being posted. But these types of guidelines won't be able to accurately pull the best content for every scenario.

The search engine experience has remained relatively unchanged since it's inception. Sure, the algorithms behind the scenes have improved, but the user experience hasn't. Real-time search could very well be the next step in the user's search engine experience.

If search engines are going to take real-time search to the next step, then they are going to have to face the problems of ranking social media as quickly as it's being created. That being said, if any search engines out there want to give me a job, I'll gladly devote myself to making this happen.