3 ms·

Instagram engineers themself wrote a bit about their backend infrastructure. One of the more important topics was how they shard the data [0] and this is also l

• Sparkyte 3y ago
  If it wasn't just an image site the potential for hotspotting would be insane! Size isn't a bad thing anymore since price has dropped exponentially since the inception of Instagram. I am positive they would use another modern technology today if it was present in the past. Fantastic read though.
  • _3u10 3y ago
    The potential for hotspotting decreases with the number of inserts per second. Like if you only did 1 insert per second and timed it right you could put all of those inserts on one server, but this would likely not overload the server. It's virtually impossible for anyone to hotspot in a meaningful way with this system.
    • Sparkyte 3y ago
      Nah this will totally be an issue. You can be on the extreme end of replication or the extreme end of sharding and experience performance problems. Sharding is more likely to hotspot depending on where hot data is consistent. The solution in most cases is a simple database that acts as a pointer database user db -> user's db. That is generated on the creation of a user. From here you create some simple cold storage models ( if user isn't active ) and some warm models which will scale out the db if the user's db grows it shards and replicates for more read access. But the last thing you want is to slow replication or have one DB that can't move to balance resource utilization. There are some new DB tech that does this without even sweating the deets.