Mining for Meaning with ThinkUp

Judging from the blog comments and the @replies and the RTs on Twitter, it seems that my last blog post really resonated with a few people.  Apparently I’m not the only one who sees both the enormous value of Twitter as a social networking tool, and also a level of frustration with people who dismiss it too quickly for all the wrong reasons.

Here’s another tip about Twitter than can exponentially improve its usefulness. But be warned it’s a little geeky, and if you want to set it up for yourself it will require a certain level of technical know-how. But if you can work it out, it opens up lots of possibilities for Twitter power users and anyone who just loves playing with data.

The tool is called ThinkUp, and is an open source project led by Gina Trapani (who some of you might recognise as the founder of LifeHacker, and a regular guest on the TWiT network’s This Week in Google). Among other things, Gina is a web developer who’s been working on building tools to help mine the wealth of information that flows through the tweetstream.

To install ThinkUp you need access to the back end of a webserver and the ability to set up and connect PHP pages with a MySQL database. You’ll need to be able to manage the database access and FTP files onto the server, so this is probably not the sort of thing you should try unless you run your own (or a hosted) server. But if you can get it set up, it’s pretty interesting.

ThinkUp connects to your Twitter account using all the usual APIs but then tracks, monitors and archives a copy of your Twitter activity into your own database. If you take Twitter seriously as a tool, this is important because you normally have no way to permanently archive and capture your own Twitter activity. It normally resides on Twitter’s servers and is fairly ephemeral – once a Tweet is gone it’s gone, or at least it’s usually too hard to retrieve.  Finding a Tweet that you wrote, or the responses you might have received to it, can be difficult once the moment has passed. Trying to dig up a Twitter conversation from six months ago can be near impossible.

What ThinkUp does is to archive and organise your Tweets in your own database and provide you with tools to make more sense of them. One of the most useful things is to aggregate the replies to a question I may have asked on Twitter. Here’s an example… I recently sat in a meeting at school where we were talking about the challenge we were facing with kids keeping their laptops charged, and someone asked “I wonder how other schools handle this issue?”  While sitting in the meeting I Tweeted out the question and within minutes the replies started to come back in from all over the world.

The next day I had about 20 replies to the question, and while those replies were useful, the way Twitter normally works makes those replies hard to see in aggregate form – they were scattered all over the previous 24 hours of the public Tweetstream. With ThinkUp however, the replies to the original question can be brought together in one place, providing an easy to read Q&A type page with it’s own URL. Thanks to Twitter I was able to get a broad range of responses to the question, but because of ThinkUp I was able to get a clearer insight into what that response thread was actually trying to tell me. It also gave me an easy way to share that insight with my colleagues via a single URL.

From your ThinkUp homepage you can browse a range of interesting views of your Twitter data, such as…

  • The week’s most replied to posts
  • The week’s most retweeted posts
  • Graphs to show your follower count trends
  • Graphs to show where and how you tweet
  • Inquiries (a collated list of all your Tweets that contained a question mark)
  • Most replied-to Tweets
  • Most retweeted Tweets
  • Mentions (anything that contains an @ reply to you)
  • Conversations (and exchange between you an another user)
  • Chatterboxes (people you follow who post a lot)
  • Deadbeats (people you follow who hardly ever post)
  • All your favourite tweets in one place
  • Tweets that contained URLs
  • Tweets that contained pictures

and much more.

Next to each tweet are numbers to indicate how many times it has been replied to or retweeted. Clicking on the number takes you to a page listing all the details. On some pages you can even see a Google map that show the geographic locations of where each tweet originated (I’m having a bit of trouble making this bit work on mine but it’s a neat idea!) Update: It’s mostly working now!

As you can see, over time you’ll end up with an enormous amount of data in there. But because it’ all now in your own database, that YOU manage, you have full control over it. As you should… after all, it’s YOUR data. But what makes it all so much more useful is the way you can mine this data for meaning, because it’s in the mining for meaning that real insight about the data starts to emerge.

You can find my own ThinkUp page at http://chrisbetcher.com/thinkup. If you want to try ThinkUp for yourself but can’t install it on your own server, just make yourself an account on mine (by clicking the Log In link then click Register.)  I’ll leave the site registrable for a few days for anyone who wants to try it out.