7 ms·

Yes, I used to, but No, I fixed it :) Among other things, I am team lead for a private search engine whose partner-accessible API handles roughly 500 mio requ

Yes, I used to, but No, I fixed it :) Among other things, I am team lead for a private search engine whose partner-accessible API handles roughly 500 mio requests per month. I used to feel powerless and stressed out by the complexity and the scale, because whenever stuff broke (and it always does at this scale), I had to start playing politics, asking for favors, or threatening people on the phone to get it fixed. Higher management would hold me accountable for the downtime even when the whole S3 AZ was offline and there was clearly nothing I could do except for hoping that we'll somehow reach one of their support engineers. But over time, management's "stand on the shoulders of giants" brainwashing wore off so that they actually started to read all the "AWS outage XY" information that we forwarded to them. They started to actually believe us when we said "Nothing we can do, call Amazon!". And then, I found a struggling hosting company with almost compatible tooling and we purchased them. And I moved all of our systems off the public cloud and onto our private cloud hosting service. Nowadays, people still hold me (at least emotionally) accountable for any issue or downtime, but I feel much better about it :) Because now it actually is withing my circle of power. I have root on all relevant servers, so if shit hits the fan, I can fix things or delegate to my team. Your situation sounds like you will constantly take the blame for other people's fault. I would imagine that to be disheartening and extremely exhausting.

flyinglizard 5y ago
Care to share uptime metrics on AWS vs your own servers?
- fxtentacle 5y ago
  That wouldn't be much help because the AWS and Heroku metrics are always green, no matter what. If you can't push updates to production, they count that as a developer-only outage and do not deduct it from their reported uptime. For me, the most important metric would be time that me and my team spent fixing issues. And that went down significantly. After a year of everyone feeling burned out, now people can take extended vacations again. One big issue for example was the connectivity between EC2 servers degrading, so that instead of the usual 1gbit/s they would only get 10mbit/s. It's not quite an outage, but it makes things painfully slow and that sluggishness is visible for end users. Getting reliable network speeds is much easier if all the servers are in the same physical room.
ddorian43 5y ago
What are you using for aws alternatives? Example for S3?
- ckdarby 5y ago
  >What are you using for aws alternatives? Example for S3? Not OP but they're probably using Rook/Minio
  - deleted 5y ago
    [deleted]
- fxtentacle 5y ago
  docker + self-developed image management + CEPH
nostrebored 5y ago
You had problems with management of a cloud based api and executive visibility… so you bought a set of data centers to handle 500mio req per month? The visibility you will get after the capex when there’s a truly disastrous outage will be interesting.
- Damogran6 5y ago
  As a security guy I HATE the loss of visibility in going to the cloud. Can you duplicate it? Sure. Still not as easily as spanning a trunk and you still have to trust what you’re seeing to an extent.
  - nostrebored 5y ago
    The visibility I was mentioning in the parent comment was visibility from executives in your business, but I can see how it would be confusing. There are tradeoffs — cloud removes much of the physical security risks and gives you tools to help automated incident detection. Things like serverless functions let you build out security scaffolding pretty easily. But in exchange you do have to give some trust. And I totally understand resistance there.
    - justinclift 5y ago
      > cloud removes much of the physical security risks Doesn't cloud increase the physical security risks, rather than decrease/remove?
- NavinF 5y ago
  Hmm that’s only 190Hz on average, but we don’t know what kind of search engine it is. For example if he’s doing ML inference for every query, it would make perfect sense to get a few cabinets at a data center. I’ve done so for a much smaller project that only needs 4 GPUs and saved a ton of money.
  - fxtentacle 5y ago
    Nah, it's text-only requests returning JSON arrays of which newspaper article URLs mention which influencer or brand name keyword. The biggest hardware price point is that you need insane amounts of RAM so that you can mmap the bloom hash for the mapping from word_id to document_ids.
ckdarby 5y ago
>I used to feel powerless and stressed out by the complexity and the scale, because whenever stuff broke (and it always does at this scale), I had to start playing politics, asking for favors, or threatening people on the phone to get it fixed. Higher management would hold me accountable for the downtime even when the whole S3 AZ was offline and there was clearly nothing I could do except for hoping that we'll somehow reach one of their support engineers. If the business can't afford to have downtime then they should be paying for enterprise support. You'll be able to connect to someone in < 10 mins and have dedicated individuals you can reach out to.
- jerjerjer 5y ago
  You never hosted on AWS, did you?
  - phillu 5y ago
    Enterprise Support never disappointed me so far. Maybe not <10 minute response time, but we never felt left alone during an outage. But I guess this is also highly region/geo dependent.
  - ckdarby 5y ago
    >You never hosted on AWS, did you? Previously 2k employee company, with the entire advertising back office on AWS. Currently >$1M YR at AWS, you can get the idea of scale & what is running, here: https://www.youtube.com/playlist?list=PLf-67McbxkT6iduMWoUsh3iHaZQl3jgvq https://www.youtube.com/playlist?list=PLf-67McbxkT6iduMWoUsh...
  - 0x445442 5y ago
    In the two years I worked on serverless AWS I filed four support tickets. Three out of those four I came up with the solution or fix on my own before support could find a solution. The other ticket was still open when I left the company. But the best part is when support wanted to know how I resolved the issues. I always asked how much they were going to pay me for that information.
- FpUser 5y ago
aeyes 5y ago
I feel that your problems aren't even remotely related to my problems with large distributed systems. My problems are all about convincing the company that I need 200 engineers to work on extremely large software projects before we hit a scalability wall. That wall might be 2 years in the future so usually it is next to impossible to convince anyone to take engineers out of product development. Even more so because working on this changes absolutely nothing for the end user, it is usually some internal system related to data storage or processing which can't cope anymore. Imagine that you are Amazon and for some scalability reason you have to rewrite the storage layer of your product catalog. Immediately you have a million problems like data migration, reporting, data ingestion, making it work with all the related systems like search, recommendations, reviews and so on. And even if you get the ball rolling you have to work across dozens of different teams which can be hard because naturally people resist change. Why do large sites like Facebook, Amazon, Twitter and Instagram all essentially look the same after 10 years but some of them now have 10x the amount of engineers? I think they have so much data and so many dependencies between parts of the system that any fundamental change is extremely hard to pull off. They even cut back on features like API access. But I am pretty sure that most of them have rewritten the whole thing at least 3 times.
- andai 5y ago
  Heh, I wish they still looked the same. They added an order of magnitude of HTML and JS bloat while removing functionality.
- ClumsyPilot 5y ago
  "That wall might be 2 years in the future so usually it is next to impossible to convince anyone to take engineers out of product development. Even more so because working on this changes absolutely nothing for the end user" It seems to be the same story in fiels of Infrastructure maintenance, Aircraft design (boeing Max), and mortgage CDOs (2008). Was it always like this or the new management doesn not care untill something explodes?
  - imachine1980_ 5y ago
    a manufacturing company is designed the ground up to works whit machine but isn't the same whit software, is hard to understand that triple data isn't only triple server but a totaly different software stack, and exponentially more complex is not only put more factories like textile.
throwaway219732 5y ago
briandilley 5y ago
What i read here was "Cloud is hard, so I took on even more responsibility"
- fxtentacle 5y ago
  What you should read is: At the monthly spend of a mid-sized company, it is impossible to get phone support from any public cloud provider.
jqgatsby 5y ago
@fxtentacle, I was curious which private search engine this is for. Is the system you are describing ImageRights.com?
- fxtentacle 5y ago
  No, ImageRights is much more requests and mostly images. Also, at ImageRights I don't have management above me that I would need to convince :) This one is text-only and used by influencers and brands to check which newspapers report about their events. As I said, it's internally used by a few partner companies who buy the API from my client and sell news alerts to their clients. BTW, I'm hoping to one day build something similar as an open source search engine where people pay for the data generation and then effectively run their own ad-free Google clone, but so far interest has been _very_ low: https://news.ycombinator.com/item?id=30374611 https://news.ycombinator.com/item?id=30374611 (1 upvote) https://news.ycombinator.com/item?id=30361385 https://news.ycombinator.com/item?id=30361385 (5 upvotes) EDIT: Out of curiosity I just checked and found my intuition wrong. The ImageRights API averages 316rps = 819mio requests per month. So it's not that much bigger.
mmcnl 5y ago
If you rely on public cloud infrastructure, you should understand both the advantages and disadvantages. Seems like your company forgot about the disadvantages.

flyinglizard 5y ago

Care to share uptime metrics on AWS vs your own servers?

fxtentacle 5y ago
That wouldn't be much help because the AWS and Heroku metrics are always green, no matter what. If you can't push updates to production, they count that as a developer-only outage and do not deduct it from their reported uptime. For me, the most important metric would be time that me and my team spent fixing issues. And that went down significantly. After a year of everyone feeling burned out, now people can take extended vacations again. One big issue for example was the connectivity between EC2 servers degrading, so that instead of the usual 1gbit/s they would only get 10mbit/s. It's not quite an outage, but it makes things painfully slow and that sluggishness is visible for end users. Getting reliable network speeds is much easier if all the servers are in the same physical room.

ddorian43 5y ago

What are you using for aws alternatives? Example for S3?

ckdarby 5y ago
>What are you using for aws alternatives? Example for S3? Not OP but they're probably using Rook/Minio
- deleted 5y ago
  [deleted]
fxtentacle 5y ago
docker + self-developed image management + CEPH

nostrebored 5y ago

You had problems with management of a cloud based api and executive visibility… so you bought a set of data centers to handle 500mio req per month? The visibility you will get after the capex when there’s a truly disastrous outage will be interesting.

Damogran6 5y ago
As a security guy I HATE the loss of visibility in going to the cloud. Can you duplicate it? Sure. Still not as easily as spanning a trunk and you still have to trust what you’re seeing to an extent.
- nostrebored 5y ago
  The visibility I was mentioning in the parent comment was visibility from executives in your business, but I can see how it would be confusing. There are tradeoffs — cloud removes much of the physical security risks and gives you tools to help automated incident detection. Things like serverless functions let you build out security scaffolding pretty easily. But in exchange you do have to give some trust. And I totally understand resistance there.
  - justinclift 5y ago
    > cloud removes much of the physical security risks Doesn't cloud increase the physical security risks, rather than decrease/remove?
NavinF 5y ago
Hmm that’s only 190Hz on average, but we don’t know what kind of search engine it is. For example if he’s doing ML inference for every query, it would make perfect sense to get a few cabinets at a data center. I’ve done so for a much smaller project that only needs 4 GPUs and saved a ton of money.
- fxtentacle 5y ago
  Nah, it's text-only requests returning JSON arrays of which newspaper article URLs mention which influencer or brand name keyword. The biggest hardware price point is that you need insane amounts of RAM so that you can mmap the bloom hash for the mapping from word_id to document_ids.

ckdarby 5y ago

>I used to feel powerless and stressed out by the complexity and the scale, because whenever stuff broke (and it always does at this scale), I had to start playing politics, asking for favors, or threatening people on the phone to get it fixed. Higher management would hold me accountable for the downtime even when the whole S3 AZ was offline and there was clearly nothing I could do except for hoping that we'll somehow reach one of their support engineers. If the business can't afford to have downtime then they should be paying for enterprise support. You'll be able to connect to someone in < 10 mins and have dedicated individuals you can reach out to.

jerjerjer 5y ago
You never hosted on AWS, did you?
- phillu 5y ago
  Enterprise Support never disappointed me so far. Maybe not <10 minute response time, but we never felt left alone during an outage. But I guess this is also highly region/geo dependent.
- ckdarby 5y ago
  >You never hosted on AWS, did you? Previously 2k employee company, with the entire advertising back office on AWS. Currently >$1M YR at AWS, you can get the idea of scale & what is running, here: https://www.youtube.com/playlist?list=PLf-67McbxkT6iduMWoUsh3iHaZQl3jgvq https://www.youtube.com/playlist?list=PLf-67McbxkT6iduMWoUsh...
- 0x445442 5y ago
  In the two years I worked on serverless AWS I filed four support tickets. Three out of those four I came up with the solution or fix on my own before support could find a solution. The other ticket was still open when I left the company. But the best part is when support wanted to know how I resolved the issues. I always asked how much they were going to pay me for that information.
FpUser 5y ago

aeyes 5y ago

I feel that your problems aren't even remotely related to my problems with large distributed systems. My problems are all about convincing the company that I need 200 engineers to work on extremely large software projects before we hit a scalability wall. That wall might be 2 years in the future so usually it is next to impossible to convince anyone to take engineers out of product development. Even more so because working on this changes absolutely nothing for the end user, it is usually some internal system related to data storage or processing which can't cope anymore. Imagine that you are Amazon and for some scalability reason you have to rewrite the storage layer of your product catalog. Immediately you have a million problems like data migration, reporting, data ingestion, making it work with all the related systems like search, recommendations, reviews and so on. And even if you get the ball rolling you have to work across dozens of different teams which can be hard because naturally people resist change. Why do large sites like Facebook, Amazon, Twitter and Instagram all essentially look the same after 10 years but some of them now have 10x the amount of engineers? I think they have so much data and so many dependencies between parts of the system that any fundamental change is extremely hard to pull off. They even cut back on features like API access. But I am pretty sure that most of them have rewritten the whole thing at least 3 times.

andai 5y ago
Heh, I wish they still looked the same. They added an order of magnitude of HTML and JS bloat while removing functionality.
ClumsyPilot 5y ago
"That wall might be 2 years in the future so usually it is next to impossible to convince anyone to take engineers out of product development. Even more so because working on this changes absolutely nothing for the end user" It seems to be the same story in fiels of Infrastructure maintenance, Aircraft design (boeing Max), and mortgage CDOs (2008). Was it always like this or the new management doesn not care untill something explodes?
- imachine1980_ 5y ago
  a manufacturing company is designed the ground up to works whit machine but isn't the same whit software, is hard to understand that triple data isn't only triple server but a totaly different software stack, and exponentially more complex is not only put more factories like textile.

throwaway219732 5y ago

briandilley 5y ago

What i read here was "Cloud is hard, so I took on even more responsibility"

fxtentacle 5y ago
What you should read is: At the monthly spend of a mid-sized company, it is impossible to get phone support from any public cloud provider.

jqgatsby 5y ago

@fxtentacle, I was curious which private search engine this is for. Is the system you are describing ImageRights.com?

fxtentacle 5y ago
No, ImageRights is much more requests and mostly images. Also, at ImageRights I don't have management above me that I would need to convince :) This one is text-only and used by influencers and brands to check which newspapers report about their events. As I said, it's internally used by a few partner companies who buy the API from my client and sell news alerts to their clients. BTW, I'm hoping to one day build something similar as an open source search engine where people pay for the data generation and then effectively run their own ad-free Google clone, but so far interest has been _very_ low: https://news.ycombinator.com/item?id=30374611 https://news.ycombinator.com/item?id=30374611 (1 upvote) https://news.ycombinator.com/item?id=30361385 https://news.ycombinator.com/item?id=30361385 (5 upvotes) EDIT: Out of curiosity I just checked and found my intuition wrong. The ImageRights API averages 316rps = 819mio requests per month. So it's not that much bigger.

mmcnl 5y ago

If you rely on public cloud infrastructure, you should understand both the advantages and disadvantages. Seems like your company forgot about the disadvantages.