SiftLinks: Links sifted out of Twitter and into an RSS feed for you.
Posted: March 9th, 2010 | Author: James | Filed under: Web Development | Tags: codeigniter, mongodb, php, twitter | 7 Comments »There’s a couple of things I’ve been wanting to play with recently – the Twitter Auth protocol (and subsequently the Twitter API) and MongoDB. There’s also been something bugging me about Twitter recently. I follow a few interesting folk who post a lot of interesting links. But sometimes I disappear for a few days for work or for fun and despite seeing these links on my iPhone, I don’t really get the chance to properly read the articles.
What i needed was a way to strip out Tweets with a link in them and pick those links up in my feed reader when I get the chance to catch up with whats going on in the world.
And so I combined the aforementioned desire to experiment with new stuff and the need for an rss of my tweets with links in them to build SiftLinks. It’s still in the testing phase and I’m making various optimisations as I go along – but it works. If you have a need to extract links from twitter then give it a go and let me know how you get on.
SiftLinks looks useful, but I’m having some problems with it :
- After allowing access to siftlinks, the redirect back to the application seems to stall. If I use the “click here” link, I get an error (pasted here: http://pastebin.ca/1875744 )
- I’ve managed to get a list of links once, but received an error from Google Reader when trying to add the feed (Sorry, an unexpected condition has occurred which is preventing Google Reader from fulfilling the request.)
And a feature request – could you unshorten links in the feed? If there’s no character limit, it’s nice to see where the links are going.
It seems that it’s just really, really slow. I get the impression that Google Reader is timing out.
This is fabulous!
I’ve been pretty frustrated with Readtwit and I hope that SiftLinks will not crash and disappear after a while!
I am recommending it to all my followers and friends!
Hey James,
I’ve been prototyping a similar application for about a month or so now and it is interesting to see Siftlinks which seems to do it much better than what I am able to manage at the moment.
What caught my eye was the MongoID being used in the feed URLs. I was using KO3 and Mongo for doing the same.
How is it scaling for you? The problem I had was with crawling all links for the actual titles.
@Jonathan: I got a lot more users a lot quicker than I expected and my small Rackspace Cloud instance quickly reached saturation point. I’ve made some architecture changes and given the server more RAM and it seems to be making a big difference.
@dgende: Thanks for the feedback – glad you’re enjoying it.
@Shyam: I’ll write a detailed post about how Mongo is working out for me soon rather than hide it down here in the comments.
James,
Did not realize when I’d posted earlier that you don’t actually crawl the links, which makes it a lot more easier to scale.
Will look out for the post on Mongo
Hi James,
My Siftlinks is not working anymore! The last feed I got was on July 23rd.
Is there a reason it’s not working?
I have re-allowed it but it still gives the same 07/23 feed.
Many thanks!!