8 ms·

Open Sourced: Newscombinator Front End

71 points by tomw1808 12y ago

tomw1808 12y ago
What happens when you are on the HN top site (beside you open a beer and enjoy that little tiny moment after months of hard work): a live snapshot http://i.imgur.com/0KbPjcx.png http://i.imgur.com/0KbPjcx.png
- tomw1808 12y ago
  Thank you everyone. Hard to describe the feelings, somewhere between moonwalk and break-dance. another snapshot: http://i.imgur.com/CBYRvV3.png http://i.imgur.com/CBYRvV3.png server load peaking @2.00 seems to be stable
  - tomw1808 12y ago
    After a short break, still on the HN/top. Seems like america is sleeping now and europe is not yet quite awake :) another snapshot: http://i.imgur.com/LcgXdFy.png http://i.imgur.com/LcgXdFy.png still everything running stable.
PeterWhittaker 12y ago
Cool, I like it. One request: Start all stories collapsed. Right now, the Ellen Pao story is at the top, with about two dozen related headlines beneath it - that takes up a lot of screen space (every time I refresh I have to collapse that space). It would also be good to be able to collapse/expand by clicking on the headline count. I'm not sure how I feel about the continuous scroll. Often I like that, but here I am unsure, but it may be because of my particular use case: I use LinkVisitor to mark all links as visited, so that next time I know which I've already read or decided to ignore. Having a fixed number of headlines makes it easier for me to recognize that it is time to hit cmd-fn-f2. If others get a lot of value from the continuous scroll, don't worry about it, mine is likely quite the edge case.
- tomw1808 12y ago
  Hey there, thanks for the feedback. Its kind of a funny thing here with that Ellen Pao case. I didn't see similar stories for quite a while on the very top and exactly now there is one. Shouldn't be an excuse, rather something I literally just learned about my app. And if everything is collapsed at the very beginning, it also looks kind of odd (for me at least). Thanks for the great feedback. Considering its an angular app, it shouldn't be that hard to have a switch to choose between endless scroll / normal pagination. But its a long way to go. I'll put it on the list :).
deleted 12y ago
[deleted]
gabrielgiordano 12y ago
Cool, thanks a lot :)
zo1 12y ago
Btw, your "Btw, JSON anyone?" link is broken: { "error": { "msg": "undefined field: \"log\"", "code": 400 } } Otherwise, great work! Looks very nice!
- tomw1808 12y ago
  Interesting, as it works normally for me. Which browser you're on, if I may ask? It should just open a new tab with the same request as sent to the REST interface anyways in the background... Like that one http://goo.gl/eLiUT1 http://goo.gl/eLiUT1
charlieegan3 12y ago
The related stories feature's cool. I made my own tool for reading HN (and some other sites) it's kind of different as it pulls the stories into a linear feed. I'll need to have a go at adding related stories. My site's http://www.serializer.io http://www.serializer.io if you're interested.
- tomw1808 12y ago
  I saw it today in the HN Notifications Thread (I think) and was already impressed, its great work, imho.
  - charlieegan3 12y ago
    Cheers! Nice to know I'm not the only one that thinks it's cool.
kriro 12y ago
Seems really cool, it gets the great honour of being a pinned tab in my browser (ńext to HN). I think a little icon that indicates the source next to the title (on the left) would be really useful to quickly scan what media ranks where and maybe only select the top stuff from your most trusted source etc.
- tomw1808 12y ago
  That is, indeed, a great honor. I didn't expect my Saturday ending at home, monitoring the servers - in a good way. The icon is a great idea, now the problem: What happens if a link was on 8 sites (2 x reddit, 4 x ycombinator, datatau, makernews). What would you expect next to the title?
_RPM 12y ago
why is it written in angular?
- tomw1808 12y ago
  why not?
  - nijiko 12y ago
    overt complexity for no real gains
  - tomw1808 12y ago
    Let me comment more on my short "why not" - here is why yes: First off: I don't choose a language because its 2014, 1999 or 1985. Or because its hip, cool or just "in". I choose a language for me, because it can help me achieve what I want to do in a reasonable amount of time. If the outcome is great, even the language is shit, its still great outcome. If the outcome is aweful, no matter how great the language is, the outcome is still not good. And I think the outcome is okay in this case and the framework is in my opinion pretty cool (considering I started JS first on IE5.5), so I cannot be blamed for being happy with a framework like this :). And, based on my opinion, I choose angular, because I have learned it first and to love it last year. Great community and good documentation imho. I think its the wrong move to change the unterlying API with the new verison. If that really happens, then I maybe switch, learn something new, better - maybe. React is awesome, plain Javascript too imho. Thoughts?
- enrmarc 12y ago
  I'm really intrigued. Why not Angular?
  - homarp 12y ago
    Angular is so 2014... now everything should be React based.
karbon 12y ago
I also have been exploring what it may mean to automagically categorize news content.. I took an approach on http://www.rivyr.com http://www.rivyr.com of following RSS feeds.. then reverse searching the titles against my product database to attempt to autoclassify news by product. Wish I could see your backend as that's where the secret sauce is but definitely understand not doing that ;)
- tomw1808 12y ago
  That is super interesting. I'm actually working more closely with Apache OpenNLP for feature extraction and LibLINEAR to classify the news according to their category. Its not too far from coming out of alpha to an open beta version. Why not subscribe to the mailinglist and I'll shoot you more infos over in the upcoming weeks?
serkanh 12y ago
This was something i was planning to build soon for myself. Great job on the UI.One thing kinda bugged me was when you select the source from the sidepanel it collapses for each selection not allowing to select another source without collapsing. I am on OSX Yosemite/Chrome-Version 41.0.2272.104. Also is the backend Elasticsearch? Is the backend elasticsearch?
- tomw1808 12y ago
  Hi. First: thank you for the kind words. Second: I'm going to fix that, I know, it already annoyed me today and I just implemented the collapsed sources last week for better readability. The backend is Solr in combination with a custom scraper in java and a lamp stack. The REST interface is wrapped around a php for better auth handling - and mainly because I'm a big fan of the Zend Framework and Pimcore [1] - one of the best CMS I've come across in my life (I'm not working for them). Although I cannot give you access to the full source of the scraper/backend (at the moment), why not get in touch and/or follow on twitter/mailinglist. I'm going to send out updates what problems I was struggling on and how I solved it from time to time. [1] https://www.pimcore.org/ https://www.pimcore.org/
serkanh 12y ago
This was something i was planning to build soon for myself. Great job on the UI.One thing kinda bugged me was when you select the source from the sidepanel it collapses for each selection not allowing to select another source without collapsing. I am on OSX Yosemite/Chrome-Version 41.0.2272.104. Also is the backend Elasticsearch?
sandGorgon 12y ago
this is interesting - could you talk a bit about your solr schema ? In a very naive way I would be simply storing it all as a indexable text blob. Did you do anything more, just so that you can avoid a database ?
- tomw1808 12y ago
  Sure. Well, basically Solr is just a flat index, although you can store now mutliple-values in one field and even make joins. Here is the clou: Everything is preprocessed and stored in a mysql-db and solr just "caches" everything. Its some kind of the flattened mysql schema. So I have a combination of a relational db and all the benefits with the huge bonus of the speed and reliability of solr in between. I started of with just a MySQL db, but real-time scoring is really a pain in the "back" with mysql and I couldn't figure a nice way to do it. On the other hand, if you expand a story, you see that the comments/upvotes from the sources are fetched, because joins like that are no possible in solr (read: I couldn't find a way to do it). So, all the information is preprocessed at index time and solr is basically a text-blob spread across multiple fields, yes.

71 points by tomw1808 12y ago

What happens when you are on the HN top site (beside you open a beer and enjoy that little tiny moment after months of hard work): a live snapshot http://i.imgur.com/0KbPjcx.png http://i.imgur.com/0KbPjcx.png

tomw1808 12y ago
Thank you everyone. Hard to describe the feelings, somewhere between moonwalk and break-dance. another snapshot: http://i.imgur.com/CBYRvV3.png http://i.imgur.com/CBYRvV3.png server load peaking @2.00 seems to be stable
- tomw1808 12y ago
  After a short break, still on the HN/top. Seems like america is sleeping now and europe is not yet quite awake :) another snapshot: http://i.imgur.com/LcgXdFy.png http://i.imgur.com/LcgXdFy.png still everything running stable.

PeterWhittaker 12y ago

Cool, I like it. One request: Start all stories collapsed. Right now, the Ellen Pao story is at the top, with about two dozen related headlines beneath it - that takes up a lot of screen space (every time I refresh I have to collapse that space). It would also be good to be able to collapse/expand by clicking on the headline count. I'm not sure how I feel about the continuous scroll. Often I like that, but here I am unsure, but it may be because of my particular use case: I use LinkVisitor to mark all links as visited, so that next time I know which I've already read or decided to ignore. Having a fixed number of headlines makes it easier for me to recognize that it is time to hit cmd-fn-f2. If others get a lot of value from the continuous scroll, don't worry about it, mine is likely quite the edge case.

tomw1808 12y ago
Hey there, thanks for the feedback. Its kind of a funny thing here with that Ellen Pao case. I didn't see similar stories for quite a while on the very top and exactly now there is one. Shouldn't be an excuse, rather something I literally just learned about my app. And if everything is collapsed at the very beginning, it also looks kind of odd (for me at least). Thanks for the great feedback. Considering its an angular app, it shouldn't be that hard to have a switch to choose between endless scroll / normal pagination. But its a long way to go. I'll put it on the list :).

deleted 12y ago

[deleted]

gabrielgiordano 12y ago

Cool, thanks a lot :)

zo1 12y ago

Btw, your "Btw, JSON anyone?" link is broken: { "error": { "msg": "undefined field: \"log\"", "code": 400 } } Otherwise, great work! Looks very nice!

tomw1808 12y ago
Interesting, as it works normally for me. Which browser you're on, if I may ask? It should just open a new tab with the same request as sent to the REST interface anyways in the background... Like that one http://goo.gl/eLiUT1 http://goo.gl/eLiUT1

charlieegan3 12y ago

The related stories feature's cool. I made my own tool for reading HN (and some other sites) it's kind of different as it pulls the stories into a linear feed. I'll need to have a go at adding related stories. My site's http://www.serializer.io http://www.serializer.io if you're interested.

tomw1808 12y ago
I saw it today in the HN Notifications Thread (I think) and was already impressed, its great work, imho.
- charlieegan3 12y ago
  Cheers! Nice to know I'm not the only one that thinks it's cool.

kriro 12y ago

Seems really cool, it gets the great honour of being a pinned tab in my browser (ńext to HN). I think a little icon that indicates the source next to the title (on the left) would be really useful to quickly scan what media ranks where and maybe only select the top stuff from your most trusted source etc.

tomw1808 12y ago
That is, indeed, a great honor. I didn't expect my Saturday ending at home, monitoring the servers - in a good way. The icon is a great idea, now the problem: What happens if a link was on 8 sites (2 x reddit, 4 x ycombinator, datatau, makernews). What would you expect next to the title?

_RPM 12y ago

why is it written in angular?

tomw1808 12y ago
why not?
- nijiko 12y ago
  overt complexity for no real gains
- tomw1808 12y ago
  Let me comment more on my short "why not" - here is why yes: First off: I don't choose a language because its 2014, 1999 or 1985. Or because its hip, cool or just "in". I choose a language for me, because it can help me achieve what I want to do in a reasonable amount of time. If the outcome is great, even the language is shit, its still great outcome. If the outcome is aweful, no matter how great the language is, the outcome is still not good. And I think the outcome is okay in this case and the framework is in my opinion pretty cool (considering I started JS first on IE5.5), so I cannot be blamed for being happy with a framework like this :). And, based on my opinion, I choose angular, because I have learned it first and to love it last year. Great community and good documentation imho. I think its the wrong move to change the unterlying API with the new verison. If that really happens, then I maybe switch, learn something new, better - maybe. React is awesome, plain Javascript too imho. Thoughts?
enrmarc 12y ago
I'm really intrigued. Why not Angular?
- homarp 12y ago
  Angular is so 2014... now everything should be React based.

karbon 12y ago

I also have been exploring what it may mean to automagically categorize news content.. I took an approach on http://www.rivyr.com http://www.rivyr.com of following RSS feeds.. then reverse searching the titles against my product database to attempt to autoclassify news by product. Wish I could see your backend as that's where the secret sauce is but definitely understand not doing that ;)

tomw1808 12y ago
That is super interesting. I'm actually working more closely with Apache OpenNLP for feature extraction and LibLINEAR to classify the news according to their category. Its not too far from coming out of alpha to an open beta version. Why not subscribe to the mailinglist and I'll shoot you more infos over in the upcoming weeks?

serkanh 12y ago

This was something i was planning to build soon for myself. Great job on the UI.One thing kinda bugged me was when you select the source from the sidepanel it collapses for each selection not allowing to select another source without collapsing. I am on OSX Yosemite/Chrome-Version 41.0.2272.104. Also is the backend Elasticsearch? Is the backend elasticsearch?

tomw1808 12y ago
Hi. First: thank you for the kind words. Second: I'm going to fix that, I know, it already annoyed me today and I just implemented the collapsed sources last week for better readability. The backend is Solr in combination with a custom scraper in java and a lamp stack. The REST interface is wrapped around a php for better auth handling - and mainly because I'm a big fan of the Zend Framework and Pimcore [1] - one of the best CMS I've come across in my life (I'm not working for them). Although I cannot give you access to the full source of the scraper/backend (at the moment), why not get in touch and/or follow on twitter/mailinglist. I'm going to send out updates what problems I was struggling on and how I solved it from time to time. [1] https://www.pimcore.org/ https://www.pimcore.org/

serkanh 12y ago

sandGorgon 12y ago

this is interesting - could you talk a bit about your solr schema ? In a very naive way I would be simply storing it all as a indexable text blob. Did you do anything more, just so that you can avoid a database ?

tomw1808 12y ago
Sure. Well, basically Solr is just a flat index, although you can store now mutliple-values in one field and even make joins. Here is the clou: Everything is preprocessed and stored in a mysql-db and solr just "caches" everything. Its some kind of the flattened mysql schema. So I have a combination of a relational db and all the benefits with the huge bonus of the speed and reliability of solr in between. I started of with just a MySQL db, but real-time scoring is really a pain in the "back" with mysql and I couldn't figure a nice way to do it. On the other hand, if you expand a story, you see that the comments/upvotes from the sources are fetched, because joins like that are no possible in solr (read: I couldn't find a way to do it). So, all the information is preprocessed at index time and solr is basically a text-blob spread across multiple fields, yes.