Finding the Needle in the Twitter Haystack

With millions of Twitter messages floating through the Twittersphere each day, you can use the search tool at  http://search.twitter.com to find references to ANY word that gets uttered there.

So a search for the word “dog” will find every tweet that contains the word dog, and so on.  You can even search for your own twittername and see any time your name is referenced online.  Many companies now use this search feature to find out whenever anyone mentions their products or services on Twitter.

The search tool for Twitter is really quite powerful, and can also be used to generate RSS feeds that can then be embedded into other pages and services.  There is some awesome potential there.

However, Twitter’s ability to search for words being mentioned out there becomes less useful when you search for a really common word, since the search results will invariably turn up lots of stuff you probably don’t want.

When you’re attending a conference for example, you could find every mention that people make about the event by searching for the conference name.  However, it wouldn’t be all that helpful just to do a search on the term “conference” since it would catch all the other possible mentions of the word “conference” from a bunch of other conferences you don’t want. Using the full name of the conference would probably work, but because Twitter limits you to only 140 characters, it would be silly to devote so many of them to including the conference name… there would be little room left for the actual message!

To get around this problem, Twitter users came up with the idea of using a hashtag.. by adding a # in front of a search term. it’s a way to trick Twitter Search into avoiding any results that might contain the keyword but don’t have the hash in front of them.

For conferences, there will generally be a designated hashtag containing a # symbol and an abbreviation for the event. People attending and Twittering from the event can include this short code at the end of each tweet, and then a search (and also an RSS feed) can be created to grab a feed of all the tweets that contain the hashtag, regardless of who they come from. This let’s people follow the conference Tweets in a single stream.

What if the conference has an unusual name already?  A search for a conference abbreviated to “educonf” would probably find most of the references to it fairly easily, since educonf is a kind of “made up” word already.  In this case, a search for the generic term “educonf” or the properly hashtagged “#educonf” would probably turn up pretty much the exact same results.

The real need for the hashtag arises when you have search terms based on regular English words that are ambiguous to the search.  The added # to the front of them makes them unique and helps them stand out from the generic non-hashed word and stops the generic words from getting caught up in the hashtagged feed.  It also carries the added bonus that many 3rd party Twitter clients such as Tweetdeck, Tweetie or Nambu can identify the hashtags and use them to create saved searches, making it much easier to follow the stream based on that tag.

Interestingly, the search feature was never a part of Twitter’s original functionality.  Twitter search was done with a third-party tool created by a company called Summize, but the huge potential (and possibilities for future monetization of Twitter) became immediately obvious and Summize was acquired by Twitter for about $15M almost a year ago.  Now the built-in search functionality is a key part of the Twitter experience, and hashtags play an important role in making that experience even more powerful.

CC Image: ‘Haystack Owl
www.flickr.com/photos/14829735@N00/360683898

Creative Commons License
This work, unless otherwise expressly stated, is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

One thought on “Finding the Needle in the Twitter Haystack

  1. Pingback: Dear Twitter... Help! | Betchablog

That's all well and good, but what do YOU think?