3 ms·

This doesn't make any sense. You talk about open data but yours is the opposite. You're just another commercial data hoarder, please don't act like you're not.

la6470 8y ago
You are a fool ... it takes money to run servers pay for electricity and cooling and salaries for the people who work for it. Don’t talk like a child.
sytelus 8y ago
You are mistaking between free and open. You can be open without being free. Maintaining web index is extremely expensive. Imagine storing most of the web on your own servers and serving it. Someone has to pay bills for all those disk space and bandwidth. I don’t think web index would ever be free (unless storage, compute and bandwidth were free) but having at reasonably priced is a very good thing. I would hope these indices are available on AWS, Azure etc where people can just use it with cloud compute and pay per use.
- 2stop 8y ago
  You are mistaking what an "index" is. You don't need to host any of "the internet" on your servers. You just need to host where it is.
- profalseidol 8y ago
  > I don’t think web index would ever be free Yet the company first mentioned does it for free, lol: https://commoncrawl.org/ https://commoncrawl.org/ I've checked Datastreamer.io for 5 seconds, I don't see any link to their repo. If not "open source" then what does "open" mean?
  - ksahin 8y ago
    Commoncrawl is not a company, it's a non-profit. Open means you can access the data, there is no assumption about the data being free or not.
    - warent 8y ago
      What? It's a nonprofit organization engaging in nonprofit business. Any organization that engages in business is a "company." Common Crawl is a company. Your comment isn't accurate and it doesn't address the parent's comment.
- jonathanstrange 8y ago
  Easy to test, though. If they were open, you could download their entire data set under some permissive license. If you can't then they are not open.

la6470 8y ago

You are a fool ... it takes money to run servers pay for electricity and cooling and salaries for the people who work for it. Don’t talk like a child.

sytelus 8y ago

You are mistaking between free and open. You can be open without being free. Maintaining web index is extremely expensive. Imagine storing most of the web on your own servers and serving it. Someone has to pay bills for all those disk space and bandwidth. I don’t think web index would ever be free (unless storage, compute and bandwidth were free) but having at reasonably priced is a very good thing. I would hope these indices are available on AWS, Azure etc where people can just use it with cloud compute and pay per use.

2stop 8y ago
You are mistaking what an "index" is. You don't need to host any of "the internet" on your servers. You just need to host where it is.
profalseidol 8y ago
> I don’t think web index would ever be free Yet the company first mentioned does it for free, lol: https://commoncrawl.org/ https://commoncrawl.org/ I've checked Datastreamer.io for 5 seconds, I don't see any link to their repo. If not "open source" then what does "open" mean?
- ksahin 8y ago
  Commoncrawl is not a company, it's a non-profit. Open means you can access the data, there is no assumption about the data being free or not.
  - warent 8y ago
    What? It's a nonprofit organization engaging in nonprofit business. Any organization that engages in business is a "company." Common Crawl is a company. Your comment isn't accurate and it doesn't address the parent's comment.
jonathanstrange 8y ago
Easy to test, though. If they were open, you could download their entire data set under some permissive license. If you can't then they are not open.