4 ms·

Funny story time. I was once part of an acquisition from a much larger corporate entity. The new parent company was in the middle of a huge cloud migration, an

Funny story time. I was once part of an acquisition from a much larger corporate entity. The new parent company was in the middle of a huge cloud migration, and as part of our integration into their org, we were required to migrate our services to the cloud. Our calculations said it would cost 3x as much to run our infra on the cloud. We pushed back, and were greenlit on creating a hybrid architecture that allowed us to launch machines both on-prem and in the cloud (via a direct link to the cloud datacenter). This gave us the benefit of autoscaling our volatile services, while maintaining our predictable services on the cheap. After I left, apparently my former team was strong-armed into migrating everything to the cloud. A few years go by, and guess who reaches out on LinkedIn? The parent org was curious how we built the hybrid infra, and wanted us to come back to do it again. I didn't go back.

smitty1e 3y ago
My funny story is built on the idea that AWS is Hotel California for your data. A customer had an interest in merging the data from an older account into a new one, just to simplify matters. Enterprise data. Going back years. Not even leaving the region. The AWS rep in the meeting kinda pauses, says: "We'll get back to you on the cost to do that." The sticker shock was enough that the customer simply inherited the old account, rather than making things tidy.
- hhsectech 3y ago
  Eh? I've never had a problem moving data out of AWS. Have people lost the ability to write export and backup scripts?
  - LadyCailin 3y ago
    It’s the cost of data egress, which isn’t free.
    - mciancia 3y ago
      But there is no paid egress when we are moving data between account within one region, rigth?
      - storyinmemo 3y ago
        There is. You pay a price for any cross-VPC traffic.
        CubsFan1060 3y ago
        This isn't true, at least not anymore. You can peer two vpc's and as long as you are transferring within the same (real) AZ, it's free: https://aws.amazon.com/about-aws/whats-new/2021/05/amazon-vpc-announces-pricing-change-for-vpc-peering/ https://aws.amazon.com/about-aws/whats-new/2021/05/amazon-vp... Even peered VPC's only pay "normal" prices: https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer "Data transferred "in" to and "out" from Amazon EC2, Amazon RDS, Amazon Redshift, Amazon DynamoDB Accelerator (DAX), and Amazon ElastiCache instances, Elastic Network Interfaces or VPC Peering connections across Availability Zones in the same AWS Region is charged at $0.01/GB in each direction."
  - Draiken 3y ago
    The ingress/egress cost is ridiculously high. Some companies don't care, but it is there and I've seen it catch people off guard multiple times.
    - varjag 3y ago
      Oh come on from the description both accounts could be sitting on the same datacenter LAN.
  - interroboink 3y ago
    My (peripheral) experience is that it is much cheaper to get data in than to get data out. When you have the amount of data being discussed — "Enterprise data. Going back years." — that can get very costly. It's the amount of data where it makes more sense to put hard drives on a truck and drive across the country rather than send it over a network, where this becomes an issue (actually, probably a bit before then).
    - fcarraldo 3y ago
      AWS actually has a service for this - Snowmobile, a storage datacenter inside of a shipping container, which is driven to you on a semi truck. https://aws.amazon.com/snowmobile/ https://aws.amazon.com/snowmobile/
      - xmcqdpt2 3y ago
        They do not! > Q: Can I export data from AWS with Snowmobile? > > Snowmobile does not support data export. It is designed to let you quickly, easily, and more securely migrate exabytes of data to AWS. When you need to export data from AWS, you can use AWS Snowball Edge to quickly export up to 100TB per appliance and run multiple export jobs in parallel as necessary. Visit the Snowball Edge FAQs to learn more. https://aws.amazon.com/snowmobile/faqs/?nc2=h_mo-lang https://aws.amazon.com/snowmobile/faqs/?nc2=h_mo-lang Why would they make it convenient to leave?
        fcarraldo 3y ago
        Oh, TIL! Thanks for correcting me.
      - brickteacup 3y ago
        That's only for data into AWS though, not data out
  - Shorel 3y ago
    Just in network costs, there's a huge asymmetry. Uploading data to AWS is free. Downloading data from them, you have to pay. When you have enough data, that cost is quite significant.
  - mijoharas 3y ago
    There's a cost for data egress (but not ingress)
- banku_brougham 3y ago
  Is R2 a sensible option for hosting data? I understand egress is chesp.
  - stickfigure 3y ago
    R2 is great. Our GCS bill (almost all egress) jumped from a few hundred dollars a month to a couple thousand dollars a month last year due to a usage spike. We rush-migrated to R2 and now that part of the bill is $0. I've heard some people here on HN say that it's slow, but I haven't noticed a difference. We're mainly dealing with multi-megabyte image files, so YMMV if you have a different workload.
    - banku_brougham 3y ago
      awesome. I remember reading about this a while ago, but never tried. Since it has the same API i can imagine its not daunting as a multi-cloud infrastructure. I guess permissions might be more complex, as in EC2 instance profiles wouldnt grant access, etc.
      - stickfigure 3y ago
        Just to make sure nobody is confused by this - R2 has the same API as S3, not GCS. We had to build a simple abstraction around GCS/S3 to perform the migration. But if you're migrating from S3, it's pretty much drop-in. We even use the AWS-provided S3 Java library (especially convenient for making signed URLs).
hhsectech 3y ago
There are two possible scenarios here. Firstly, they can't find the talent to support what you implemented...or more likely, your docs suck! I've made a career out of inheriting other peoples whacky setups and supporting them (as well as fixing them) and almost always its documentation that has prevented the client getting anywhere. I personally dont care if the docs are crap because usually the first thing I do is update / actually write the docs to make them usable. For a lot of techs though crap documentation is a deal breaker. Crap docs aren't always the fault of the guys implementing though, sometimes there are time constraints that prevent proper docs being written. Quite frequently though its outsourced development agencies that refuse to write it because its "out of scope" and a "billable extra". Which I think is an egregious stance...doxs Should be part and parcel of the project. Mandatory.
- smokel 3y ago
  I agree that bad documentation is a serious problem in many cases. So much so that your suggestion to write the documentation after the fact can become quite impossible. If there is only one thing that juniors should learn about writing documentation (be it comments or design documents), it is this: document why something is there. If resources are limited, you can safely skip comments that describe how something works, because that information is also available in code. (It might help to describe what is available, especially if code is spread out over multiple repositories, libraries, teams, etc.) (Also, I suppose the comment I'm responding to could've been slightly more forgiving to GP, but that's another story.)
- lazyasciiart 3y ago
  Unfortunately it’s also possible that e.g the company switched from share point to confluence and lost half the entire knowledge base because it wasn’t labeled the way they thought it was. Or that the docs were all purged because they were part of an abandoned project.
- adrianmsmith 3y ago
  > Quite frequently though its outsourced development agencies that refuse to write it It's also completely against their interest to write docs as it makes their replacement easier. That's why you need someone competent on the buying side to insist on the docs. A lot of companies outsource because they don't have this competency themselves. So it's inevitable that this sort of thing happens and companies get locked in and can't replace their contractors, because they don't have any docs.
nyc_data_geek 3y ago
Yes, I do believe autoscaling is actually a good use case for public cloud. If you have bursty load that requires a lot of resources at peak which would sit idle most of the time, probably doesn't make sense to own what you need for those peaks.

smitty1e 3y ago

My funny story is built on the idea that AWS is Hotel California for your data. A customer had an interest in merging the data from an older account into a new one, just to simplify matters. Enterprise data. Going back years. Not even leaving the region. The AWS rep in the meeting kinda pauses, says: "We'll get back to you on the cost to do that." The sticker shock was enough that the customer simply inherited the old account, rather than making things tidy.

hhsectech 3y ago
Eh? I've never had a problem moving data out of AWS. Have people lost the ability to write export and backup scripts?
- LadyCailin 3y ago
  It’s the cost of data egress, which isn’t free.
  - mciancia 3y ago
    But there is no paid egress when we are moving data between account within one region, rigth?
    - storyinmemo 3y ago
      There is. You pay a price for any cross-VPC traffic.
      - CubsFan1060 3y ago
        This isn't true, at least not anymore. You can peer two vpc's and as long as you are transferring within the same (real) AZ, it's free: https://aws.amazon.com/about-aws/whats-new/2021/05/amazon-vpc-announces-pricing-change-for-vpc-peering/ https://aws.amazon.com/about-aws/whats-new/2021/05/amazon-vp... Even peered VPC's only pay "normal" prices: https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer "Data transferred "in" to and "out" from Amazon EC2, Amazon RDS, Amazon Redshift, Amazon DynamoDB Accelerator (DAX), and Amazon ElastiCache instances, Elastic Network Interfaces or VPC Peering connections across Availability Zones in the same AWS Region is charged at $0.01/GB in each direction."
- Draiken 3y ago
  The ingress/egress cost is ridiculously high. Some companies don't care, but it is there and I've seen it catch people off guard multiple times.
  - varjag 3y ago
    Oh come on from the description both accounts could be sitting on the same datacenter LAN.
- interroboink 3y ago
  My (peripheral) experience is that it is much cheaper to get data in than to get data out. When you have the amount of data being discussed — "Enterprise data. Going back years." — that can get very costly. It's the amount of data where it makes more sense to put hard drives on a truck and drive across the country rather than send it over a network, where this becomes an issue (actually, probably a bit before then).
  - fcarraldo 3y ago
    AWS actually has a service for this - Snowmobile, a storage datacenter inside of a shipping container, which is driven to you on a semi truck. https://aws.amazon.com/snowmobile/ https://aws.amazon.com/snowmobile/
    - xmcqdpt2 3y ago
      They do not! > Q: Can I export data from AWS with Snowmobile? > > Snowmobile does not support data export. It is designed to let you quickly, easily, and more securely migrate exabytes of data to AWS. When you need to export data from AWS, you can use AWS Snowball Edge to quickly export up to 100TB per appliance and run multiple export jobs in parallel as necessary. Visit the Snowball Edge FAQs to learn more. https://aws.amazon.com/snowmobile/faqs/?nc2=h_mo-lang https://aws.amazon.com/snowmobile/faqs/?nc2=h_mo-lang Why would they make it convenient to leave?
      - fcarraldo 3y ago
        Oh, TIL! Thanks for correcting me.
    - brickteacup 3y ago
      That's only for data into AWS though, not data out
- Shorel 3y ago
  Just in network costs, there's a huge asymmetry. Uploading data to AWS is free. Downloading data from them, you have to pay. When you have enough data, that cost is quite significant.
- mijoharas 3y ago
  There's a cost for data egress (but not ingress)
banku_brougham 3y ago
Is R2 a sensible option for hosting data? I understand egress is chesp.
- stickfigure 3y ago
  R2 is great. Our GCS bill (almost all egress) jumped from a few hundred dollars a month to a couple thousand dollars a month last year due to a usage spike. We rush-migrated to R2 and now that part of the bill is $0. I've heard some people here on HN say that it's slow, but I haven't noticed a difference. We're mainly dealing with multi-megabyte image files, so YMMV if you have a different workload.
  - banku_brougham 3y ago
    awesome. I remember reading about this a while ago, but never tried. Since it has the same API i can imagine its not daunting as a multi-cloud infrastructure. I guess permissions might be more complex, as in EC2 instance profiles wouldnt grant access, etc.
    - stickfigure 3y ago
      Just to make sure nobody is confused by this - R2 has the same API as S3, not GCS. We had to build a simple abstraction around GCS/S3 to perform the migration. But if you're migrating from S3, it's pretty much drop-in. We even use the AWS-provided S3 Java library (especially convenient for making signed URLs).

hhsectech 3y ago

There are two possible scenarios here. Firstly, they can't find the talent to support what you implemented...or more likely, your docs suck! I've made a career out of inheriting other peoples whacky setups and supporting them (as well as fixing them) and almost always its documentation that has prevented the client getting anywhere. I personally dont care if the docs are crap because usually the first thing I do is update / actually write the docs to make them usable. For a lot of techs though crap documentation is a deal breaker. Crap docs aren't always the fault of the guys implementing though, sometimes there are time constraints that prevent proper docs being written. Quite frequently though its outsourced development agencies that refuse to write it because its "out of scope" and a "billable extra". Which I think is an egregious stance...doxs Should be part and parcel of the project. Mandatory.

smokel 3y ago
I agree that bad documentation is a serious problem in many cases. So much so that your suggestion to write the documentation after the fact can become quite impossible. If there is only one thing that juniors should learn about writing documentation (be it comments or design documents), it is this: document why something is there. If resources are limited, you can safely skip comments that describe how something works, because that information is also available in code. (It might help to describe what is available, especially if code is spread out over multiple repositories, libraries, teams, etc.) (Also, I suppose the comment I'm responding to could've been slightly more forgiving to GP, but that's another story.)
lazyasciiart 3y ago
Unfortunately it’s also possible that e.g the company switched from share point to confluence and lost half the entire knowledge base because it wasn’t labeled the way they thought it was. Or that the docs were all purged because they were part of an abandoned project.
adrianmsmith 3y ago
> Quite frequently though its outsourced development agencies that refuse to write it It's also completely against their interest to write docs as it makes their replacement easier. That's why you need someone competent on the buying side to insist on the docs. A lot of companies outsource because they don't have this competency themselves. So it's inevitable that this sort of thing happens and companies get locked in and can't replace their contractors, because they don't have any docs.

nyc_data_geek 3y ago

Yes, I do believe autoscaling is actually a good use case for public cloud. If you have bursty load that requires a lot of resources at peak which would sit idle most of the time, probably doesn't make sense to own what you need for those peaks.