4 ms·

Is Ontolo the fastest crawler, parser, indexer?

12 points by ppsscc 10y ago

web007 10y ago
No, Google is.
- ijamj 10y ago
  https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...
howlingfantods 10y ago
Why phrase it as a question? The linked article is a PR piece that obviously argues that Ontolo is the fastest crawler, parser, indexer. The actual article title "400,000,000+ Prospects a Day: Behind the Scenes of ontolo" seems less click baity.
- joekrill 10y ago
  I didn't even bother clicking through because I saw the domain name (ontolo.com) and assumed it was a PR piece. Also, Betteridge's law of headlines gave me my answer already (spoiler: it's "no").
  - benwills 10y ago
    Hi Joe. I just saw this post here on HN. I'm the sole author of ontolo. Your comment about the ontolo domain and assuming it was a PR piece, is that because you know of ontolo as a marketing tool, or because it wasn't a more generalized site like Medium? Also, if you know of any other crawlers, etc, that are faster than ontolo (in the public domain, or public claims), let me know. There aren't a lot of folks I know who are working on this sort of thing and not also working at search engines. It'd be neat to meet some other folks who are interested in the problem.
    - joekrill 10y ago
      Hey Ben. It's because it was immediately apparent it would be a totally biased piece. When companies write about their own products, they usually do so from a completely biased perspective, in their favor, because they want whoever is reading it to favor their product. Microsoft.com isn't going to feature an article proclaiming OSX is the better operating system, for example -- so anything comparing the two on that site is going to be heavily biased towards Windows. If I'm looking for something that's actually informative, whatever I find there isn't going to be much use to me.
      - benwills 10y ago
        That makes sense. I don't know if this factors in, but I didn't post the link here. And if I ever did post a link here to ontolo, the HN post title would be very different. And I agree that few, if any, folks that talk about their own stuff are able to do so objectively.
Cyph0n 10y ago
Interesting read. I'm assuming throughput is a key requirement for the system, but I wonder how much more difficult writing and maintaining the codebase is compared to a "safer" but slower language such as Java, Scala, or Go.
- benwills 10y ago
  I just saw this post. I'm the sole author of ontolo. As for requirements, they're very low. The machines running the crawler+parser+indexer, which achieve those speeds, are dual Xeon 5639s with 72GB ram. When the indexer isn't running, the crawler and parser use 6 cores, but only "need" 4 of the 24 total on the box. The indexer generally uses very little CPU. All of those cores are actually needed only for queries. So, the limitation on throughput, at this point, is the 1gbps connection to the box. I say 'box' singular, as the 400,000,000/day is per box. As for maintaining the codebase, it's about 10k lines, plus another 10k of data that's referenced throughout. As I've written everything in the crawler and parser by hand. There are no magic boxes or libraries that it goes into, so debugging is working around only the logic I've constructed, as opposed to someone else's logic. That said, I've designed it to use as few system resources as possible, unless it would run faster using a uint32_t vs a bit field, for example. If someone else were to begin working with my code, I'm unsure of what that would be like for them. I'd like to believe it's generally straightforward, but I can't say until that happens. That said, I've also designed it to be easily updated as the way that HTML is written continues to evolve. I know that's a bit hand-wavey, but it's as much as I'm comfortable describing how it works behind the scenes, as no one else I know of (other than common search engines) parses HTML like ontolo does. If you have any other questions, just let me know. I'll be monitoring this post today. - Ben
  - Cyph0n 10y ago
    Thanks for the reply! That's truly impressive - around 4.6k rps, including all of the additional post-processing involved - on a single machine! I'm also quite surprised by the lines of code: ~20k is quite small for a project of this complexity written in C.
    - 10y ago

12 points by ppsscc 10y ago

web007 10y ago

No, Google is.

ijamj 10y ago
https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...

howlingfantods 10y ago

Why phrase it as a question? The linked article is a PR piece that obviously argues that Ontolo is the fastest crawler, parser, indexer. The actual article title "400,000,000+ Prospects a Day: Behind the Scenes of ontolo" seems less click baity.

joekrill 10y ago
I didn't even bother clicking through because I saw the domain name (ontolo.com) and assumed it was a PR piece. Also, Betteridge's law of headlines gave me my answer already (spoiler: it's "no").
- benwills 10y ago
  Hi Joe. I just saw this post here on HN. I'm the sole author of ontolo. Your comment about the ontolo domain and assuming it was a PR piece, is that because you know of ontolo as a marketing tool, or because it wasn't a more generalized site like Medium? Also, if you know of any other crawlers, etc, that are faster than ontolo (in the public domain, or public claims), let me know. There aren't a lot of folks I know who are working on this sort of thing and not also working at search engines. It'd be neat to meet some other folks who are interested in the problem.
  - joekrill 10y ago
    Hey Ben. It's because it was immediately apparent it would be a totally biased piece. When companies write about their own products, they usually do so from a completely biased perspective, in their favor, because they want whoever is reading it to favor their product. Microsoft.com isn't going to feature an article proclaiming OSX is the better operating system, for example -- so anything comparing the two on that site is going to be heavily biased towards Windows. If I'm looking for something that's actually informative, whatever I find there isn't going to be much use to me.
    - benwills 10y ago
      That makes sense. I don't know if this factors in, but I didn't post the link here. And if I ever did post a link here to ontolo, the HN post title would be very different. And I agree that few, if any, folks that talk about their own stuff are able to do so objectively.

Cyph0n 10y ago

Interesting read. I'm assuming throughput is a key requirement for the system, but I wonder how much more difficult writing and maintaining the codebase is compared to a "safer" but slower language such as Java, Scala, or Go.

benwills 10y ago
I just saw this post. I'm the sole author of ontolo. As for requirements, they're very low. The machines running the crawler+parser+indexer, which achieve those speeds, are dual Xeon 5639s with 72GB ram. When the indexer isn't running, the crawler and parser use 6 cores, but only "need" 4 of the 24 total on the box. The indexer generally uses very little CPU. All of those cores are actually needed only for queries. So, the limitation on throughput, at this point, is the 1gbps connection to the box. I say 'box' singular, as the 400,000,000/day is per box. As for maintaining the codebase, it's about 10k lines, plus another 10k of data that's referenced throughout. As I've written everything in the crawler and parser by hand. There are no magic boxes or libraries that it goes into, so debugging is working around only the logic I've constructed, as opposed to someone else's logic. That said, I've designed it to use as few system resources as possible, unless it would run faster using a uint32_t vs a bit field, for example. If someone else were to begin working with my code, I'm unsure of what that would be like for them. I'd like to believe it's generally straightforward, but I can't say until that happens. That said, I've also designed it to be easily updated as the way that HTML is written continues to evolve. I know that's a bit hand-wavey, but it's as much as I'm comfortable describing how it works behind the scenes, as no one else I know of (other than common search engines) parses HTML like ontolo does. If you have any other questions, just let me know. I'll be monitoring this post today. - Ben
- Cyph0n 10y ago
  Thanks for the reply! That's truly impressive - around 4.6k rps, including all of the additional post-processing involved - on a single machine! I'm also quite surprised by the lines of code: ~20k is quite small for a project of this complexity written in C.
  - 10y ago