7 ms·

I like Unsplash. They and Pixabay are what I use for apps I have developed in the past. But simply, this floors me. I looked at their costs and with some dev

I like Unsplash. They and Pixabay are what I use for apps I have developed in the past. But simply, this floors me. I looked at their costs and with some developer muscle, you could find savings such as: - Move Fastly to Cloudflare. They don't expressly say what the cost is for that. But moving to CF would eliminate it. - Move Heroku to Digital Ocean. It's not difficult to create a fully redundant solution. - Move from Imgix to having Golang micro services which handle the resizing of images and use something like Belugacdn for the CDN. Beluga is $5k a month for a PB. (Or some other cheaper CDN if you don't like Beluga, but damn... imgix pricing) I'm pretty sure that a savvy CTO could save at least $50k a month with a well designed project that does this over many months and achieves the same result and keeps the redundancy concerns for the small team. I do realise however, why the team have done this. In the same position (with very few resources) I would probably have done the same. But damn, when a cost of a service gets up to a years salary ($120k) for a good developer. Time to seek alternatives.

qes 7y ago
They say they spend $12,700/month on Fastly. I guess I don't know offhand the difference between Fastly and Imgix (which they also say is their CDN), but the levels of bandwidth they're talking about are not going to be free at Cloudflare. We serve significantly more bandwidth and requests than their 2016 numbers at under $15k compared to their $17.5k, and we're not particularly lean on infra costs being a .Net shop with Win servers and MS SQL on Azure, and I can still bring our monthly a lot closer to $10k if I can find time for some projects. But the reality is that even thousands of dollars per month saved can take a good while to ROI against the costs of development, or more so the opportunity costs.
- judge2020 7y ago
  > the levels of bandwidth they're talking about are not going to be free at Cloudflare Correct. Cloudflare's terms state that you can't serve a disproportionate/substantial amount of non-HTML content, so they would need to sign up for Enterprise with a full contract to get image serving via CF. Enterprise is generally $5k+/month, but CF contracts can offer a subset of enterprise features from $500-$5k/month, so I would guess ~3k/month just to serve multiple petabytes of images. This doesn't eliminate the bandwidth costs so CF may not be the single solution for the bandwidth costs.
  - odensc 7y ago
    They specifically say they use Imgix's CDN for the images, which can be confirmed by looking at the response headers. Fastly is used for their actual website, judging by the ASN info. Meaning they wouldn't be serving petabytes of images from CloudFlare, just HTML/CSS/JS, which should be perfectly fine.
marcinzm 7y ago
$50k/month comes out to $600k/year which is the cost around 2-3 engineers counting overhead. To manage infrastructure at that scale themselves they'd need someone who knows DevOps. To write services at that scale they'd need someone who knows backend engineering and micro services. Then they'd need to be able to have 24/7 on call rotations for when things break. And of course, like any rewrite it seems like a "couple month" project until you get into the details and it becomes a two year project. Soon you've got a full 6+ person team and are paying them a lot more than $600k/year.
- manigandham 7y ago
  And some projects really are just simple. Scale makes no difference to their services, it's a basic OLTP + search with resizing image proxy. This doesn't have any SLAs nor is it critical software either. 24/7 oncall is completely unnecessary.
  - tpetry 7y ago
    Don‘t underestimate the massive cpu power needed for scaling images at their request rate
    - nine_k 7y ago
      Maybe some caching would help.
    - abhiyerra 7y ago
      Wonder if they could use S3 bucket with a lambda to resize. I’m sure their images are a long tail so certain images likely have way more traffic than others. At a previous company where we were also doing a lot of imgix traffic we did something like this but with a go service but now I’d just offload to s3 and use lambda to avoid Devops..
      - scarface74 7y ago
        I’ve done exactly this. But not at any scale. But if they are specializing in this and with low margins, AWS is not the answer. The compute cost would be relatively cheap and possibly even the storage cost. I would think that the bandwidth costs would be expensive.
puzzle 7y ago
Resizing images in Golang is not as trivial as you might imagine at first. You need to be able to handle all sorts of formats and color spaces. You'd be surprised by the kind of weird garbage your users will upload. Then you need to use SIMD, not pure Golang solutions, or performance will suffer. So you end up adopting a wrapper around libvips or similar, at which point you will start to ponder if you should have stuck with C/C++ in the first place. (It all depends on how much of Go's features you use or if it's just a nicer, safer C for you.)

qes 7y ago

They say they spend $12,700/month on Fastly. I guess I don't know offhand the difference between Fastly and Imgix (which they also say is their CDN), but the levels of bandwidth they're talking about are not going to be free at Cloudflare. We serve significantly more bandwidth and requests than their 2016 numbers at under $15k compared to their $17.5k, and we're not particularly lean on infra costs being a .Net shop with Win servers and MS SQL on Azure, and I can still bring our monthly a lot closer to $10k if I can find time for some projects. But the reality is that even thousands of dollars per month saved can take a good while to ROI against the costs of development, or more so the opportunity costs.

judge2020 7y ago
> the levels of bandwidth they're talking about are not going to be free at Cloudflare Correct. Cloudflare's terms state that you can't serve a disproportionate/substantial amount of non-HTML content, so they would need to sign up for Enterprise with a full contract to get image serving via CF. Enterprise is generally $5k+/month, but CF contracts can offer a subset of enterprise features from $500-$5k/month, so I would guess ~3k/month just to serve multiple petabytes of images. This doesn't eliminate the bandwidth costs so CF may not be the single solution for the bandwidth costs.
- odensc 7y ago
  They specifically say they use Imgix's CDN for the images, which can be confirmed by looking at the response headers. Fastly is used for their actual website, judging by the ASN info. Meaning they wouldn't be serving petabytes of images from CloudFlare, just HTML/CSS/JS, which should be perfectly fine.

marcinzm 7y ago

$50k/month comes out to $600k/year which is the cost around 2-3 engineers counting overhead. To manage infrastructure at that scale themselves they'd need someone who knows DevOps. To write services at that scale they'd need someone who knows backend engineering and micro services. Then they'd need to be able to have 24/7 on call rotations for when things break. And of course, like any rewrite it seems like a "couple month" project until you get into the details and it becomes a two year project. Soon you've got a full 6+ person team and are paying them a lot more than $600k/year.

manigandham 7y ago
And some projects really are just simple. Scale makes no difference to their services, it's a basic OLTP + search with resizing image proxy. This doesn't have any SLAs nor is it critical software either. 24/7 oncall is completely unnecessary.
- tpetry 7y ago
  Don‘t underestimate the massive cpu power needed for scaling images at their request rate
  - nine_k 7y ago
    Maybe some caching would help.
  - abhiyerra 7y ago
    Wonder if they could use S3 bucket with a lambda to resize. I’m sure their images are a long tail so certain images likely have way more traffic than others. At a previous company where we were also doing a lot of imgix traffic we did something like this but with a go service but now I’d just offload to s3 and use lambda to avoid Devops..
    - scarface74 7y ago
      I’ve done exactly this. But not at any scale. But if they are specializing in this and with low margins, AWS is not the answer. The compute cost would be relatively cheap and possibly even the storage cost. I would think that the bandwidth costs would be expensive.

puzzle 7y ago

Resizing images in Golang is not as trivial as you might imagine at first. You need to be able to handle all sorts of formats and color spaces. You'd be surprised by the kind of weird garbage your users will upload. Then you need to use SIMD, not pure Golang solutions, or performance will suffer. So you end up adopting a wrapper around libvips or similar, at which point you will start to ponder if you should have stuck with C/C++ in the first place. (It all depends on how much of Go's features you use or if it's just a nicer, safer C for you.)