5 ms·

I'm actually very near to launching an automated news aggregator and I'm wondering if this is going to affect me. My suspicion is that sites like Techmeme / HN

by garply 18y ago

I'm actually very near to launching an automated news aggregator and I'm wondering if this is going to affect me. My suspicion is that sites like Techmeme / HN or smaller will probably be left alone. What do you guys think?

brandnewlow 18y ago
I think Techmeme is the sort of site they want to shut down. Hacker News and Digg would be much, much harder. The difference is that while Techmeme and Google news are only displaying the headlines and summaries, they are indexing the FULL article and making use of it to power their algorithms. If they determine it that way, I don't see what the big deal is. If you want to use the Twitter API, you have to pay past a certain point. Maybe a year from now, if you want to index full news articles for your aggregator, you have to pay past a certain point as well. That wouldn't be too bad at all and would leave that act of linking safe and untouched.
- garply 18y ago
  That's a very interesting way to look at the problem - algorithms like mine and Techmeme's do indeed digest the full article whereas HN does not. I had thought the primary issue would be whether or not the site provided a summary / thumbnail (as I see WindyCitizen does), not whether or not it scanned the source's bits. Having to pay for my algorithm to access this data would be a big deal to me - I'm operating on a shoestring budget and I don't want to do that.
  - brandnewlow 18y ago
    Sure, it would make things hard for you, but again, if you were using any other sort of data, there'd be usage costs. The news folks are just figuring that part out now, while every tech-first company has that build into their business from the start.
    - silentOpen 18y ago
      Except for the fact that transmission/distribution costs are almost zero and 'news' is a broadly consumed information resource. It doesn't matter that there would have been usage costs in the past -- they would be silly now.
      - brandnewlow 18y ago
        If every other API charges for usage, why shouldn't news sites? If you want to index their stories, you can do 500 queries per day for free. After that, you pay, just like any other API.
        deleted 18y ago
        [deleted]
        silentOpen 18y ago
        But the API is HTTP and the news stories are syndicated across a hundred different sites. How do you limit the crawlers under this scheme? It seems like any serious attempt to limit crawling will require major software redeployment, cooperation of crawlers, widespread authentication, or some combination of these. Is there actually a feasible way to do this without breaking the web?
        brandnewlow 18y ago
        These are all good points. I don't have answers to any of them. Feasibility is a whole other issue. My point is that if you look at online newspapers as online services, then they should be able to charge people for programmatic access to their service, just like any other tech service does through its API. If I want to build an app on the back of Yahoo BOSS, I have to pay Yahoo. If I want to build an app on the back of the New York Times, maybe I should have to pay the New York Times.
        deleted 18y ago
        [deleted]
pclark 18y ago
whats the name of your aggregator? we're doing something similar. Wanna chat? peter [\at] omgponi.es

by garply 18y ago

brandnewlow 18y ago

I think Techmeme is the sort of site they want to shut down. Hacker News and Digg would be much, much harder. The difference is that while Techmeme and Google news are only displaying the headlines and summaries, they are indexing the FULL article and making use of it to power their algorithms. If they determine it that way, I don't see what the big deal is. If you want to use the Twitter API, you have to pay past a certain point. Maybe a year from now, if you want to index full news articles for your aggregator, you have to pay past a certain point as well. That wouldn't be too bad at all and would leave that act of linking safe and untouched.

garply 18y ago
That's a very interesting way to look at the problem - algorithms like mine and Techmeme's do indeed digest the full article whereas HN does not. I had thought the primary issue would be whether or not the site provided a summary / thumbnail (as I see WindyCitizen does), not whether or not it scanned the source's bits. Having to pay for my algorithm to access this data would be a big deal to me - I'm operating on a shoestring budget and I don't want to do that.
- brandnewlow 18y ago
  Sure, it would make things hard for you, but again, if you were using any other sort of data, there'd be usage costs. The news folks are just figuring that part out now, while every tech-first company has that build into their business from the start.
  - silentOpen 18y ago
    Except for the fact that transmission/distribution costs are almost zero and 'news' is a broadly consumed information resource. It doesn't matter that there would have been usage costs in the past -- they would be silly now.
    - brandnewlow 18y ago
      If every other API charges for usage, why shouldn't news sites? If you want to index their stories, you can do 500 queries per day for free. After that, you pay, just like any other API.
      - deleted 18y ago
        [deleted]
      - silentOpen 18y ago
        But the API is HTTP and the news stories are syndicated across a hundred different sites. How do you limit the crawlers under this scheme? It seems like any serious attempt to limit crawling will require major software redeployment, cooperation of crawlers, widespread authentication, or some combination of these. Is there actually a feasible way to do this without breaking the web?
        brandnewlow 18y ago
        These are all good points. I don't have answers to any of them. Feasibility is a whole other issue. My point is that if you look at online newspapers as online services, then they should be able to charge people for programmatic access to their service, just like any other tech service does through its API. If I want to build an app on the back of Yahoo BOSS, I have to pay Yahoo. If I want to build an app on the back of the New York Times, maybe I should have to pay the New York Times.
        deleted 18y ago
        [deleted]

pclark 18y ago

whats the name of your aggregator? we're doing something similar. Wanna chat? peter [\at] omgponi.es