10 ms·

You had problems with management of a cloud based api and executive visibility… so you bought a set of data centers to handle 500mio req per month? The visibil

You had problems with management of a cloud based api and executive visibility… so you bought a set of data centers to handle 500mio req per month? The visibility you will get after the capex when there’s a truly disastrous outage will be interesting.

Damogran6 5y ago
As a security guy I HATE the loss of visibility in going to the cloud. Can you duplicate it? Sure. Still not as easily as spanning a trunk and you still have to trust what you’re seeing to an extent.
- nostrebored 5y ago
  The visibility I was mentioning in the parent comment was visibility from executives in your business, but I can see how it would be confusing. There are tradeoffs — cloud removes much of the physical security risks and gives you tools to help automated incident detection. Things like serverless functions let you build out security scaffolding pretty easily. But in exchange you do have to give some trust. And I totally understand resistance there.
  - justinclift 5y ago
    > cloud removes much of the physical security risks Doesn't cloud increase the physical security risks, rather than decrease/remove?
NavinF 5y ago
Hmm that’s only 190Hz on average, but we don’t know what kind of search engine it is. For example if he’s doing ML inference for every query, it would make perfect sense to get a few cabinets at a data center. I’ve done so for a much smaller project that only needs 4 GPUs and saved a ton of money.
- fxtentacle 5y ago
  Nah, it's text-only requests returning JSON arrays of which newspaper article URLs mention which influencer or brand name keyword. The biggest hardware price point is that you need insane amounts of RAM so that you can mmap the bloom hash for the mapping from word_id to document_ids.
  - winrid 5y ago
    You could have used a sharded database like Mongo. Just throw up 10 shards, use "source" (influencer or brand name) as shard key?
    - fxtentacle 5y ago
      Yes, I could have used Mongo, but it would have been 100x to 1000x slower than an mmap-ed look up table.
      - joshuamorton 5y ago
        But you don't actually need that level of performance? You've made this system more complex and expensive to achieve a requirement that doesn't matter?
        shoo 5y ago
        you seem to have a deeper knowledge of the business & organisational context that dictate the true requirements than someone working there. please share these details so we can all learn!
        joshuamorton 5y ago
        Sure: the network request time of a person making a request over the open internet is going to be an order of magnitude longer than a DB lookup (in the right style, with a reverse-index) on the scale of data this person is describing. So making the lookup 10x faster saves you...1% of the request latency. And at the qps they've described, it's not a throughput issue either. So I'm pretty confident in saying that this is a case of premature optimization. And at some point the increase in parallelization of scans dominates mmap speed, unless you're redundantly sharding your mmaped hash table across multiple machines. And there are cases where network bandwidth is the bottleneck before disk bandwidth, though probably not this case. But yeah basically, the answer is something like "if this is the optimal choice, it probably didnt matter that much".
fxtentacle 5y ago
You might be surprised. The performance equivalent of $100k monthly in EC2 spend fits into a 16m2 cage with 52HU racks.
- benjamir 5y ago
  Which costs you more than $100k monthly to operate with the same level of manageability and reliability. We don't use AWS, because our use cases don't require that level of reliability and we simply cannot afford it, but if I needed a company to depend on IT that generates enough revenue... I probably wouldn't argue about the AWS bill. So long, prepaid at hetzner + in-house works good enough, but I know what I cannot offer with the click of a button to my user!
  - Spooky23 5y ago
    This is a religious debate among many. The IT/engineering nerd stuff doesn’t matter at all. Cloud migration decisions are always made by accounting and tax factors. I run two critical apps, one on-prem and one cloud. There is no difference in people cost, and the cloud service costs about 20% more on the infrastructure side. We went cloud because customer uptake was unknown and making capital investments didn’t make sense. I’ve had a few scenarios where we’ve moved workloads from cloud to on-prem and reverse. These things are tools and it doesn’t pay to be dogmatic.
    - sdoering 5y ago
      > These things are tools and it doesn’t pay to be dogmatic. I wish I would hear this line more often. So many things today are (pseudo-) religious now. The right frsmework/language, cloud or on prem, x vs not x. Especially bad imho when somebody tries to tell you how you could do better with 'not x' instead of x you are currently using without even trying to understand the context this decision resides in. [Edit] typo
      - qorrect 5y ago
        > So many things today are (pseudo-) religious now. The right frsmework/language, cloud or on prem, x vs not x. Might have always been that way? We just have so many more tools to argue over now.

Damogran6 5y ago

As a security guy I HATE the loss of visibility in going to the cloud. Can you duplicate it? Sure. Still not as easily as spanning a trunk and you still have to trust what you’re seeing to an extent.

nostrebored 5y ago
The visibility I was mentioning in the parent comment was visibility from executives in your business, but I can see how it would be confusing. There are tradeoffs — cloud removes much of the physical security risks and gives you tools to help automated incident detection. Things like serverless functions let you build out security scaffolding pretty easily. But in exchange you do have to give some trust. And I totally understand resistance there.
- justinclift 5y ago
  > cloud removes much of the physical security risks Doesn't cloud increase the physical security risks, rather than decrease/remove?

NavinF 5y ago

Hmm that’s only 190Hz on average, but we don’t know what kind of search engine it is. For example if he’s doing ML inference for every query, it would make perfect sense to get a few cabinets at a data center. I’ve done so for a much smaller project that only needs 4 GPUs and saved a ton of money.

fxtentacle 5y ago
Nah, it's text-only requests returning JSON arrays of which newspaper article URLs mention which influencer or brand name keyword. The biggest hardware price point is that you need insane amounts of RAM so that you can mmap the bloom hash for the mapping from word_id to document_ids.
- winrid 5y ago
  You could have used a sharded database like Mongo. Just throw up 10 shards, use "source" (influencer or brand name) as shard key?
  - fxtentacle 5y ago
    Yes, I could have used Mongo, but it would have been 100x to 1000x slower than an mmap-ed look up table.
    - joshuamorton 5y ago
      But you don't actually need that level of performance? You've made this system more complex and expensive to achieve a requirement that doesn't matter?
      - shoo 5y ago
        you seem to have a deeper knowledge of the business & organisational context that dictate the true requirements than someone working there. please share these details so we can all learn!
        joshuamorton 5y ago
        Sure: the network request time of a person making a request over the open internet is going to be an order of magnitude longer than a DB lookup (in the right style, with a reverse-index) on the scale of data this person is describing. So making the lookup 10x faster saves you...1% of the request latency. And at the qps they've described, it's not a throughput issue either. So I'm pretty confident in saying that this is a case of premature optimization. And at some point the increase in parallelization of scans dominates mmap speed, unless you're redundantly sharding your mmaped hash table across multiple machines. And there are cases where network bandwidth is the bottleneck before disk bandwidth, though probably not this case. But yeah basically, the answer is something like "if this is the optimal choice, it probably didnt matter that much".

fxtentacle 5y ago

You might be surprised. The performance equivalent of $100k monthly in EC2 spend fits into a 16m2 cage with 52HU racks.

benjamir 5y ago
Which costs you more than $100k monthly to operate with the same level of manageability and reliability. We don't use AWS, because our use cases don't require that level of reliability and we simply cannot afford it, but if I needed a company to depend on IT that generates enough revenue... I probably wouldn't argue about the AWS bill. So long, prepaid at hetzner + in-house works good enough, but I know what I cannot offer with the click of a button to my user!
- Spooky23 5y ago
  This is a religious debate among many. The IT/engineering nerd stuff doesn’t matter at all. Cloud migration decisions are always made by accounting and tax factors. I run two critical apps, one on-prem and one cloud. There is no difference in people cost, and the cloud service costs about 20% more on the infrastructure side. We went cloud because customer uptake was unknown and making capital investments didn’t make sense. I’ve had a few scenarios where we’ve moved workloads from cloud to on-prem and reverse. These things are tools and it doesn’t pay to be dogmatic.
  - sdoering 5y ago
    > These things are tools and it doesn’t pay to be dogmatic. I wish I would hear this line more often. So many things today are (pseudo-) religious now. The right frsmework/language, cloud or on prem, x vs not x. Especially bad imho when somebody tries to tell you how you could do better with 'not x' instead of x you are currently using without even trying to understand the context this decision resides in. [Edit] typo
    - qorrect 5y ago
      > So many things today are (pseudo-) religious now. The right frsmework/language, cloud or on prem, x vs not x. Might have always been that way? We just have so many more tools to argue over now.