3 ms·

Interesting I wonder if its seems easy because it’s explained simply or if it really is simple to put in place. I want to make a clone now, just to try. At leas

by 0wis 3y ago

Interesting I wonder if its seems easy because it’s explained simply or if it really is simple to put in place. I want to make a clone now, just to try. At least for this inspiration, this article was well worth the read. Thanks !

rmetzler 3y ago
Instagram engineers found some remarkable simple solutions to some hard problems. It's not easy to come up with these solutions. Designing the IDs for example is no small feat, but since this is now common knowledge it's probably not too hard to build a similar system. To get traction from users is the real challenge.
- koonsolo 3y ago
  > To get traction from users is the real challenge Unfortunately yes. Scaling is a problem I would love to have :D.
- BoorishBears 3y ago
  Which makes things like Vercel so depressing to me. You've got a company paying off influential people in a space where people are looking for guidance, convincing them that they too have hard problems that cannot settle for simple solutions. Selling the narrative that developers need to be all in on the most irrelevant aspects of building a product, and ignoring the fact that if you instead focus on building simple, easy to maintain software, the fact your LCP isn't hyper optimized by some newly invented mental model for app development won't matter: Google (or any search engine for that matter) will not ignore the fact people just actually want your content. They do not care how great your web core vitals are if you waste a bunch of time bending over for some irrelevant bullshit problem instead of talking to users and iterating.
  - wild_egg 3y ago
    The fun part is often all that bullshit hyper optimized complexity can still be beaten on all the web core vitals by a humble LAMP server So not only are you wasting your time on all the wrong things, you're also gaining nearly no benefit from it I'm pretty convinced that all these shiny new hotness tools and frameworks of the past decade are actually just meant to sabotage competition and small companies
    - bornfreddy 3y ago
      I agree - for most of them. One exception would be React/Vue/other frontend frameworks that split updating the state and visualizing it. It sounds like a small thing, but it makes a world of difference in non-trivial projects, compared to native js / jquery. Then again, it's the idea itself that matters, overoptimizing the implementation is not beneficial. I.e., React classes are just as good (or better) as hooks as far as I'm concerned, it would be better if they would let it be at that stage.
      - BoorishBears 3y ago
        Vercel and co managed to mangle the one benefit you mentioned. They introduced the concept of a "server component": one that is rendered once and cannot update state. And instead of making that opt-in, they made it the default. — That is to say, the default of React is to no longer allow updating state in components. No value proposition is safe from the forces of financially incentivized thought leaders.
        leerob 3y ago
        The Meta team introduced Server Components: https://legacy.reactjs.org/blog/2020/12/21/data-fetching-with-react-server-components.html https://legacy.reactjs.org/blog/2020/12/21/data-fetching-wit...
        BoorishBears 3y ago
        The RFC was driven by Vercel. Vercel announced RSC landing in experimental on stage before the React team even updated their docs I'm not even going to get into how ridiculous it is that the default config chosen was to break people's builds for a feature that isn't mature enough to have anything more than a single reference implementation from Vercel.
- jakjak123 3y ago
  Yeah, we took some of their ideas about ID design and have applied it in a way that we could improve our storage.
jalk 3y ago
It looks fairly standard tbh. A lot of the heavy lifting is done by AWS. They likely didn’t start out with this exact architecture in mind, but adapted along the way. I.e. something like: The timestamped key format improved poor performance. Redis was introduced when Postgres was getting swamped. Didn’t replace the existing memcached with Redis as it worked as it should. And ofc there surely were a ton of oddities/ issues with their AWS setup that they spent time fixing. The nginx description is a bit vague, but could be some “hack” to work around some ELB scaling behavior that wasn’t to their liking.
engineercodex 3y ago
Author here. Comments like this make my day, so thank you! I’m trying to find old software engineering gems and explain them as simply as possible, so I’m glad you found it simple to understand. Also, it’s definitely possible to make a clone, but the hard part is getting the users :)
- 0wis 3y ago
  Of course, I do not envision getting users. Only for fun and training - I think having examples of simple solutions to seemingly hard problems in mind makes it easier to come up with one in the future.
- 0xFF0123 3y ago
  Do you have a link to Fabric? My Google Fu is failing me, looks like it might be dead?
  - thruflo 3y ago
    https://www.fabfile.org https://www.fabfile.org
    - JimDabell 3y ago
      Fabric is still an option but nowadays Ansible is probably the closest popular choice for things like that.
  - seedie 3y ago
    I was wrong thruflo pasted the correct link You are probably right. The Instagram Engineering blog [0] points to the Fabric documentation on Read the Docs [1] which is empty. "Read the docs" links to the GitHub repo [2] but it's 404 and even the GitHub organization zwsyff888 is gone. [0] https://instagram-engineering.com/what-powers-instagram-hundreds-of-instances-dozens-of-technologies-adf2e22da2ad https://instagram-engineering.com/what-powers-instagram-hund... [1] https://fabric.readthedocs.io https://fabric.readthedocs.io [2] https://github.com/zwsyff888/fabric https://github.com/zwsyff888/fabric
adamauckland 3y ago
It's pretty simple if your goal is using boring, stable technologies. Django/Postgres/Memcached/Redis/Nginx are all really stable. I've built quite a few projects using almost exactly the same stack over the last 15 years. Almost all are still running, those that aren't are for business reasons not technical. The problem is, they're not exciting The Next Thing real-time javascript somethings, so a lot of devs won't want to use it.

by 0wis 3y ago

rmetzler 3y ago

Instagram engineers found some remarkable simple solutions to some hard problems. It's not easy to come up with these solutions. Designing the IDs for example is no small feat, but since this is now common knowledge it's probably not too hard to build a similar system. To get traction from users is the real challenge.

koonsolo 3y ago
> To get traction from users is the real challenge Unfortunately yes. Scaling is a problem I would love to have :D.
BoorishBears 3y ago
Which makes things like Vercel so depressing to me. You've got a company paying off influential people in a space where people are looking for guidance, convincing them that they too have hard problems that cannot settle for simple solutions. Selling the narrative that developers need to be all in on the most irrelevant aspects of building a product, and ignoring the fact that if you instead focus on building simple, easy to maintain software, the fact your LCP isn't hyper optimized by some newly invented mental model for app development won't matter: Google (or any search engine for that matter) will not ignore the fact people just actually want your content. They do not care how great your web core vitals are if you waste a bunch of time bending over for some irrelevant bullshit problem instead of talking to users and iterating.
- wild_egg 3y ago
  The fun part is often all that bullshit hyper optimized complexity can still be beaten on all the web core vitals by a humble LAMP server So not only are you wasting your time on all the wrong things, you're also gaining nearly no benefit from it I'm pretty convinced that all these shiny new hotness tools and frameworks of the past decade are actually just meant to sabotage competition and small companies
  - bornfreddy 3y ago
    I agree - for most of them. One exception would be React/Vue/other frontend frameworks that split updating the state and visualizing it. It sounds like a small thing, but it makes a world of difference in non-trivial projects, compared to native js / jquery. Then again, it's the idea itself that matters, overoptimizing the implementation is not beneficial. I.e., React classes are just as good (or better) as hooks as far as I'm concerned, it would be better if they would let it be at that stage.
    - BoorishBears 3y ago
      Vercel and co managed to mangle the one benefit you mentioned. They introduced the concept of a "server component": one that is rendered once and cannot update state. And instead of making that opt-in, they made it the default. — That is to say, the default of React is to no longer allow updating state in components. No value proposition is safe from the forces of financially incentivized thought leaders.
      - leerob 3y ago
        The Meta team introduced Server Components: https://legacy.reactjs.org/blog/2020/12/21/data-fetching-with-react-server-components.html https://legacy.reactjs.org/blog/2020/12/21/data-fetching-wit...
        BoorishBears 3y ago
        The RFC was driven by Vercel. Vercel announced RSC landing in experimental on stage before the React team even updated their docs I'm not even going to get into how ridiculous it is that the default config chosen was to break people's builds for a feature that isn't mature enough to have anything more than a single reference implementation from Vercel.
jakjak123 3y ago
Yeah, we took some of their ideas about ID design and have applied it in a way that we could improve our storage.

jalk 3y ago

It looks fairly standard tbh. A lot of the heavy lifting is done by AWS. They likely didn’t start out with this exact architecture in mind, but adapted along the way. I.e. something like: The timestamped key format improved poor performance. Redis was introduced when Postgres was getting swamped. Didn’t replace the existing memcached with Redis as it worked as it should. And ofc there surely were a ton of oddities/ issues with their AWS setup that they spent time fixing. The nginx description is a bit vague, but could be some “hack” to work around some ELB scaling behavior that wasn’t to their liking.

engineercodex 3y ago

Author here. Comments like this make my day, so thank you! I’m trying to find old software engineering gems and explain them as simply as possible, so I’m glad you found it simple to understand. Also, it’s definitely possible to make a clone, but the hard part is getting the users :)

0wis 3y ago
Of course, I do not envision getting users. Only for fun and training - I think having examples of simple solutions to seemingly hard problems in mind makes it easier to come up with one in the future.
0xFF0123 3y ago
Do you have a link to Fabric? My Google Fu is failing me, looks like it might be dead?
- thruflo 3y ago
  https://www.fabfile.org https://www.fabfile.org
  - JimDabell 3y ago
    Fabric is still an option but nowadays Ansible is probably the closest popular choice for things like that.
- seedie 3y ago
  I was wrong thruflo pasted the correct link You are probably right. The Instagram Engineering blog [0] points to the Fabric documentation on Read the Docs [1] which is empty. "Read the docs" links to the GitHub repo [2] but it's 404 and even the GitHub organization zwsyff888 is gone. [0] https://instagram-engineering.com/what-powers-instagram-hundreds-of-instances-dozens-of-technologies-adf2e22da2ad https://instagram-engineering.com/what-powers-instagram-hund... [1] https://fabric.readthedocs.io https://fabric.readthedocs.io [2] https://github.com/zwsyff888/fabric https://github.com/zwsyff888/fabric

adamauckland 3y ago

It's pretty simple if your goal is using boring, stable technologies. Django/Postgres/Memcached/Redis/Nginx are all really stable. I've built quite a few projects using almost exactly the same stack over the last 15 years. Almost all are still running, those that aren't are for business reasons not technical. The problem is, they're not exciting The Next Thing real-time javascript somethings, so a lot of devs won't want to use it.