4 ms·

I think it's astounding how "easy" we can get away with stuff these days. But how does this scale to larger datasets? Say, capturing all signalling traffic for

I think it's astounding how "easy" we can get away with stuff these days. But how does this scale to larger datasets? Say, capturing all signalling traffic for a telco. It's not hard to have 100s of GB a day - that's only a couple 10s of Mbps. Suddenly 20GB/sec means 5-10s queries just for that day. They can also leverage the bulk speed to reduce the size of posting lists. Just index each term to a chunk of, say, 1-5GB. And don't even index full terms, just hash them - a 64-bit hash gives you enough room for around 10 billion unique terms without causing much problems with collisions. New data in the last hour can be brute forced while the index is being generated. OTOH they seem successful so I suppose it's working quite fine.

biturd 10y ago
If I read correctly, this was staged out on a single core user machine, then pushed out to an Amazon 16 core machine. Not the most powerful either, and no mention of any tuning. They mentioned looking to hit max scale out by simply adding more machines. Which is why they mention going from 20MG -> 100 -> 1000 not being an issue. In theory, wire speed is their limitation before they have to look at other methods. At 1000MB/s you could start chunking your data into result sets in advance, taking the time it takes a human to read the results visually as always being the bottleneck, and using that time to your advantage, to make the user think the app returns answers instantly all the time. I can't imagine 1000MB/s ever being too slow when you factor in their current dashboard and what it is returning. More complex dashboards and things certainly will need reworking.
busterarm 10y ago
Having worked at a telco where our tools would take a minute or more for results to come back, 5-10 second queries that we only do a few (or even a few dozen) times a day is _fine_.
- MichaelGG 10y ago
  I was thinking more along the lines of SIP tracing for customers, which happens much more frequently for tech support roles, at least in my experience. With a quad core and a single 1TB drive I was able to service 100s of GB a day of SIP with ~1s response times. Use a mix of brute force (for temporary log) then index in background. The interactivity shifts quite a bit when you get close to "instant". Though I know some companies that are content with even 30s query times, but that's gotta limit flexibility.
  - windowsworkstoo 10y ago
    I guess its an interesting difference right - what say a developer would consider "instant" (lets say <10ms) and what the end user is satisfied with as instant (maybe <1min)...it's interesting the tradeoffs you can make when you loosen your latency requirements.
    - adrianN 10y ago
      Thinking like this leads to products like Rational Doors, a requirements database where searching is dog slow and you need to know where to look if you want to find anything.
      - windowsworkstoo 10y ago
        Really? I've no idea what that product is but I hardly think it's a "thinking" problem, more a managing expectation or shit requirements gathering problem. If it was necessary to search in a certian way over a certain size in a certain time, you could make the appropriate trade off. Sounds like you just wanted to bitch out about a particular product pal.
        adrianN 10y ago
        I admit I like bitching about slow software. But if you don't put speed as one of your top requirements, over time your software will become slower and slower. Having one operation that takes 100ms instead of 10ms is fine until two years later when somebody else who doesn't bother to read your code uses it to implement something that now takes one second instead of 100ms, which is incidentally used a hundred times to complete some query that now takes minutes instead of seconds. Now imagine you're a company like IBM or SAP that works on the same code base for decades and the cruft keeps layering and layering until even finding out which part of the code needs to be tuned becomes too expensive to explain to your customers. There is http://accidentallyquadratic.tumblr.com http://accidentallyquadratic.tumblr.com with a nice collection of "it was fast enough at the time" code.
mSparks 10y ago
I do something similar with the stuff for http://lifecode.co http://lifecode.co Couple of terrabytes processing easy but yeah limited to only a few queries per day. A lot of the effort goes into caching so disk reads and swap file use is minimised. Really that's where all the overhead is. All the public stuff for that is at http://theborgmatrix.com http://theborgmatrix.com

biturd 10y ago

If I read correctly, this was staged out on a single core user machine, then pushed out to an Amazon 16 core machine. Not the most powerful either, and no mention of any tuning. They mentioned looking to hit max scale out by simply adding more machines. Which is why they mention going from 20MG -> 100 -> 1000 not being an issue. In theory, wire speed is their limitation before they have to look at other methods. At 1000MB/s you could start chunking your data into result sets in advance, taking the time it takes a human to read the results visually as always being the bottleneck, and using that time to your advantage, to make the user think the app returns answers instantly all the time. I can't imagine 1000MB/s ever being too slow when you factor in their current dashboard and what it is returning. More complex dashboards and things certainly will need reworking.

busterarm 10y ago

Having worked at a telco where our tools would take a minute or more for results to come back, 5-10 second queries that we only do a few (or even a few dozen) times a day is _fine_.

MichaelGG 10y ago
I was thinking more along the lines of SIP tracing for customers, which happens much more frequently for tech support roles, at least in my experience. With a quad core and a single 1TB drive I was able to service 100s of GB a day of SIP with ~1s response times. Use a mix of brute force (for temporary log) then index in background. The interactivity shifts quite a bit when you get close to "instant". Though I know some companies that are content with even 30s query times, but that's gotta limit flexibility.
- windowsworkstoo 10y ago
  I guess its an interesting difference right - what say a developer would consider "instant" (lets say <10ms) and what the end user is satisfied with as instant (maybe <1min)...it's interesting the tradeoffs you can make when you loosen your latency requirements.
  - adrianN 10y ago
    Thinking like this leads to products like Rational Doors, a requirements database where searching is dog slow and you need to know where to look if you want to find anything.
    - windowsworkstoo 10y ago
      Really? I've no idea what that product is but I hardly think it's a "thinking" problem, more a managing expectation or shit requirements gathering problem. If it was necessary to search in a certian way over a certain size in a certain time, you could make the appropriate trade off. Sounds like you just wanted to bitch out about a particular product pal.
      - adrianN 10y ago
        I admit I like bitching about slow software. But if you don't put speed as one of your top requirements, over time your software will become slower and slower. Having one operation that takes 100ms instead of 10ms is fine until two years later when somebody else who doesn't bother to read your code uses it to implement something that now takes one second instead of 100ms, which is incidentally used a hundred times to complete some query that now takes minutes instead of seconds. Now imagine you're a company like IBM or SAP that works on the same code base for decades and the cruft keeps layering and layering until even finding out which part of the code needs to be tuned becomes too expensive to explain to your customers. There is http://accidentallyquadratic.tumblr.com http://accidentallyquadratic.tumblr.com with a nice collection of "it was fast enough at the time" code.

mSparks 10y ago

I do something similar with the stuff for http://lifecode.co http://lifecode.co Couple of terrabytes processing easy but yeah limited to only a few queries per day. A lot of the effort goes into caching so disk reads and swap file use is minimised. Really that's where all the overhead is. All the public stuff for that is at http://theborgmatrix.com http://theborgmatrix.com