7 ms·
Show HN: weekend project, Readstream
- revorad 16y agoThis is really nicely done. Can you tell us how you built it?
- dshipper 16y agoFirst we pull as much of your Twitter timeline as we can, and find the ones that have links in them. We used an open source project called Goose to grab the article text from those links and store it in our database and then push it to the user. It's all PHP/MySQL/Javascript/Java running on a VPS. It took us about 3 days of work to go from idea to completion (in between class as we're college students.) Feel free to email us or comment if you have any more questions :)
- daniel1980fl 16y agowhat do you mean by "store it in our database" ? AFAIK Twitter terms of usage (API) does not allow you to STORE anything on your machine(s)... only to access it in real time. This fact alone would stop me from spending time on building a tool that obviously violates Twitter TOS.
- ra 16y agoCan I ask you you need permission to update my twitter stream? That put me off trying it.
- wesleyzhao 16y agoWe wanted to implement a Twitter comment system on our site much like Facebook has - so we could foster a community around the articles. To do that, we wanted to be able to create a box that lets you enter in your possible 'Tweet' and click 'Tweet' to send it out. Also we wanted to let you RT from our app!
- joelhaasnoot 16y agoAbout those comments: they need work, I'm seeing a lot of straight retweets in there with no changed/added content. I would definitely filter those out, they're just retweets. Also, "Click here to read more" needs to be on the next line, it's very ugly this way.
- ajaymehta 16y agoBoth very good points, and we'll work on adding them in soon.
- wesleyzhao 16y agoYeah the commenting system the way it works right now, goes through the json array once to see if there are any @replies, and to find which were RT's were a hassle because we would have to do a bunch of string comparisons (since Twitter's search API doesn't return the relevant data). So we were thinking right now to both keep the data loading as fast as possible and keep our API calls limited, we would leave them in but throw them at the bottom of the comments.
- revorad 16y agoYou could use Backtype for getting the replies. They've probably already done a lot of work to find the most relevant tweets.
- wesleyzhao 16y ago@revorad nice! We use chartbeat so we should have thought of using backtype (their Twitter stream app of choice). Definitely will see how well that fits into our next iteration.
- decklin 16y agoIt doesn't say that on the page. I bounced because I assumed that any app that wants permission to post to my timeline is going to spam some kind of advertisement for itself -- unless I have a reason to trust it. In this case, I've never heard of the app and I don't know who you are, so I don't. I'm just explaining why I immediately left; I don't know if this happens frequently enough for you to care.
- dshipper 16y agoPro tip: You can navigate through the articles using the arrow keys on your keyboard. Do you guys have any suggestions as to how to make this more clear? (We already say to do so on the loading screen but feel not many people pick up on it)
- revorad 16y agoJust keep displaying a small "use arrow keys to navigate" on top of the article. When a user first logs in, show some interesting article already instead of just the "your content is loading" message.
- dshipper 16y agoInteresting. We were thinking about popping up a lightbox for a new user to remind them if they didn't use the keys. Do you think that would be too intrusive? We definitely considered doing that, the biggest question that we ran into was how to select the article. We thought we could get something random from the database, calculate it based on what your friends are interested in, or calculate it based on what's trending overall. Because we have been coding for a long time (it's 7 AM and we've been up all night for the 3rd straight night) we just decided to leave it with some instructions and worry about how to do that later. We'll definitely get on it right now, it's a great suggestion.
- revorad 16y agoI hate lightboxes. I think dialog boxes are the worst part of desktop apps that we have imported to web apps. How to select an article? Just choose one you like. An old one from the best of HN? Or something from here - http://longreads.com http://longreads.com. Don't overengineer.
- dshipper 16y agoAwesome advice. Will keep that in mind thanks.
- Bogdanp 16y agoIt would be great if you could also bind the 'h' and 'l' keys to navigation (for those of us who dislike moving our hands off the home row).
- arunkk 16y agoGood tool to use. I like it
- dshipper 16y agoThanks!
- Void_ 16y ago> Watch out, you're about to be addicted. This headline almost made me close the website. Who needs more addiction to time-wasting than we already have?
- wesleyzhao 16y ago"Almost"? Nice! We are glad you gave us a chance to prove ourselves. We will definitely think about changing it.
- ajaymehta 16y agoNot everything that's addictive is necessarily time-wasting! In reasonable doses, I've been learning a ton of interesting stuff just browsing through my linked articles.
- tehwalrus 16y agoThis sounds awesome, but it would be good if I could try it without giving your app unlimited access to my twitter account forever. (much as @ra has pointed out.) Is there scope/resource for a demo version without the API features (i.e. just grabbing my timeline and showing me the links inside your UI, with retweet buttons greyed out)? I appreciate that is a fairly convoluted feature to add in 3 days though - sounds like you guys did a good job anyway! :)
- wesleyzhao 16y agoHey! So the Twitter API actually does not let you 'mimic' that you are viewing someone else's timeline other than the 'authenticated user'. When we found out about this we were pretty bummed too. However, we did like the idea of maybe not being logged on and just surfing through Nytimes feed, or ESPN, or whomever. This IS possible and would be cool/useful we think.
- mootothemax 16y agoMaybe a quick hack would be a link to view what one of your developer's accounts sees when he or she logs in? That'll give an example of what's addictive without requiring a full user signin :-)
- ajaymehta 16y agoGreat idea - a simulation would be the best way of understanding the service without any privacy concerns on the user's part.
- wesleyzhao 16y agoOh very cool point! Definitely possible - we keep a many-many db table of views (which includes the link to the user) so we could potential do like a simulation..hmm interesting thanks!
- follower 16y agoYou might want to have a demo that doesn't require signing into Twitter. I didn't go any further.
- dshipper 16y agoThanks for your feedback, I'm sorry you didn't give the app a try. Unfortunately we can't grab any of the headlines from your Twitter feed without having you sign into your account so that would be a tough obstacle for us to overcome. Eventually we may be may be able to add a feature that shows a front page based on what is popular among all of our other users, but for now if you would like to try it out without using your Twitter account we have a test account that you can login with. If you're interested just send me an email, I'd love to show you our app!
- zwadia 16y agowe have a similar problem @ https://shh.sh https://shh.sh I believe there is merit in the advice from HN here about atleast making a concerted attempt to gain user trust before asking for a key rental to the car.
- jasonkester 16y agoWhy are you grabbing things from my Twitter stream? Why would I have anything there at all for you to grab? I didn't make it past the "let us use your Twitter account" step, but my assumption from reading the homepage was that you'd put stuff into Twitter on my behalf.
- wesleyzhao 16y agoWe definitely will add some copy to better help the user understand why we need the oauth and also the specifics of what we will do with your account (which is never spam).
- Shanewho 16y agoYou should have two options, one to use my account, one to use a sample account. People would probably view the sample first, and then if they were interested then they might sign in themselves.
- sammville 16y agoNice work guys.. Consider showing the headlines in reverse order, with the newest ones being at the top and older ones at the bottom so i don't have to scroll to get the latest headlines.
- dshipper 16y agoHey sammville thanks for your feedback! Right now we're showing the tweets in reverse chronological order by tweet time, but that doesn't take into account when the article was actually published. We'll definitely take a look and see if we can take publishing time into account, but that might be difficult given the host of different places that we are pulling content from.
- cao825 16y agoDoesn't work for me. Site keeps getting connection resets.
- arturadib 16y agoHow do you guys sort articles? By date? Popularity?
- ajaymehta 16y agoAs of right now the articles are just sorted in reverse chronological order by tweet time. We definitely want to make the sorting more personalized and relevant, so we're 100% open to ideas on how to do that. Have any good ones? :)
- deno 16y agoSo, content creators convert their RSS feeds to Twitter feeds and Readstream then converts them back again? Seems pointless. Why not just use the full feed in the first place (if that's what you want, instead of the <140 chars synopsis)? I don't mean to sound cynical, but surely, I must have missed something.
- medwezys 16y agowell if you follow RSS on twitter it is your problem. If you follow people who share links, than it is a way better than RSS, because human filter is applied and you are more likely to get an interesting content.
- deno 16y agoMy friends and I did that in 2005 with Google Reader. But fair enough, the project looks nice. I'd only suggest to implement caching better: 1) Make /get_user_feed.php return only list of stories. E.g. [http://.../story1 http://.../story1, http://.../story2 http://.../story2] 2) Fetch stories one-by-one using unique links, set proper caching headers You could also use pro.embed.ly
- wesleyzhao 16y agoNice suggestion - definitely (if implemented) will really help with our SEO. The thing I guess that may be a barrier is actually switching the page and keeping a javascript array or knowing where the user is (guess we can do that via the DB though).
- ajaymehta 16y agoI think this is an amazing point and the main reason we created this. Human curation results in infinitely better content than just reading an RSS feed of the New York Times - and many people use Twitter exactly that way. Readstream just aims to streamline that same content-delivery process.
- MatthewPhillips 16y agoThat's not how it (appears to) work. This pulls from links tweeted by people you follow. It then uses a content extraction library (similar to Readability's I presume) to grab the content and display. Might be a copyright issue there. So the flow is Content -> Curators (Tweeters) -> Extraction Layer -> Display.
- acrum 16y agoNo matter how nice the landing page, there is probably a sizable number of people who will just close it without either 1) an interactive demo page showing an example user's tweets, or 2) a video demo. Video demos are pretty easy to make with tools like screenr. Without some idea of exactly how it's used and how it will benefit me, I'm at least a little hesitant to sign into it using my Twitter account.
- dhimes 16y agoOne comment on UX: After clicking "get started" I'm presented with a way to enter my email and password. It may save people a moment's confusion if you say something like "sign in to twitter: twitter username or email twitter password" Although it's clear what you are doing once you see the URL and if you read carefully- but in that first millisecond I'm thinking you want me to sign into an account for your service (which I don't have). Of course, you could always have the user create an account with you and have the wiring to their twitter account on the backend, but that would require more trust on the user's part.
- ajaymehta 16y agoI see the concern here - as the app is pretty simple right now, we're just using Twitter. We plan to bring in other sources and more personalization, so we'll definitely have Readstream accounts in the near future. But for now, Twitter makes it easy :)
- andr 16y agoAs a techie, I'd be more convinced if I read a few words on how you determine relevancy. You don't need to reveal your secret sauce, but you need to convince me it's good enough so that I don't need to look at Twitter anymore.
- ajaymehta 16y agoOur premise is that Twitter is an amazing place to get human-filtered, socially-relevant content but there's a lot of noise. As of right now, our articles are just listed in reverse chronological order by tweet time, so there's not too much of a relevancy engine in place. However, we want the app to be as relevant as possible. Basically, we don't have much relevancy secret sauce yet - but we're looking to develop some soon. :)
- yolesaber 16y agoAlthough I can't see it as a total replacement for my twitter stream, it definitely is a neat app and I will be using it. However, it would be really cool if you also could parse the links in the articles. For instance, this News Desk piece from the New Yorker (http://www.newyorker.com/online/blogs/newsdesk/2011/04/to-do-list-winners-and-warnings.html http://www.newyorker.com/online/blogs/newsdesk/2011/04/to-do...) has a lot of links and references, but they do not appear in the read stream.
- kpanghmc 16y agoInteresting project. I agree with the others who say you definitely need some sort of demo to convince users to sign up. Question though: how did you guys get around the copyright issues? It sounds like you're grabbing the content from these links and displaying them in their entirety on your page. Won't the content providers complain about this?
- kloncks 16y agoMany of the sources I saw being pulled out were freely available on the sources' RSS feeds
- kpanghmc 16y agoRight, but content providers at least have some control over their RSS feeds (e.g. adding advertising or truncating it to the first X characters of the article to force users to click to read more).
- ajaymehta 16y agoThat's a great question and one that other apps in the same space (Flipboard, Zite) have publicly struggled with. As this is just a small project for now, we're not too worried about content providers getting mad. But it's definitely something to keep an eye on and any advice you have would be greatly appreciated.
- EGF 16y agoThis is great. I had a feeling what it would do from the description and the name and it lived out to that promise. I think its a great start for a "weekend" project and would love to see where it goes and give some product feedback if you are up for it. Will try to contact you directly.
- ajaymehta 16y agoHey, thanks so much! We had a personal need for a product like this, and we thought others would too. I'm glad you're finding it to be useful. :)
- glesperance 16y agoIts nice to see that other teams are currently working to solve the noise vs signal problem by using people to curate content in a more networked way. Also, pardon my curiosity, but is that the idea with which you applied to YC or did you come up with it for the interview ? In any cases, good luck w/ your interview ; there is sure a lot of future in that type of apps!
- wesleyzhao 16y agoThanks for the compliment! This is actually not the idea we applied with. This is something we hope will WOW the panel (hopefully one of them is reading this right now...)
- sathyabhat 16y agoCan you guys consider adding a bug tracker or any place where you can accept feedback (uservoice?). Just wanted to give you folks a heads up about escape characters coming near quotes. http://i.imgur.com/pKWEK.png http://i.imgur.com/pKWEK.png Congrats on the project, love it.
- wesleyzhao 16y agoGR! Dang addslashes().... We should have something set up that does allow bug submissions... but until then feel free to email me if you see anything else wesley.zhao@gmail.com
- joshbaptiste 16y agoSeen this post late, currently receiving a "Fatal error: Uncaught exception 'EpiOAuthException'" with a PHP stack trace, really should hide errors like this from being displayed on the webpage.