4 ms·
It’s fascinating to see companies solve the same problems over and over again. I hope in the near future these types of problems are “definitively solved” and
by endisneigh 5y ago
It’s fascinating to see companies solve the same problems over and over again.
I hope in the near future these types of problems are “definitively solved” and more effort can be spent engineering novel features.
There’s no way to do this, but I’d be interested to see if there’s a company that just has an extremely simple stack that is very performant and popular.
I’m talking: their entire stack is Citus sharded over a few locations running two beefy servers per continent they have users in, that’s it. No dedicated search, logging or anything. Just dump everything in Postgres. If logs are lost then so be it.
A lot of the problems at scale are problems of our own creation. For example is strong consistency really necessary for tweets? Do you really need to see a followers latest tweet instantly?
Uncheck some boxes and the engineering becomes a lot simpler, very fast.
- mdoms 5y agoI'd suggest you look at Stack Overflow engineering posts. Their architecture is about as simple as you can get for a service their size. A bunch of web servers running plain old ASP.NET MVC code (recently upgraded to use ASP.NET Core). A handful of SQL server machines for the database, simple sharding strategies. Some Redis machines for caching.
- kevinmchugh 5y agoThe previous POTUS made important policy announcements on Twitter first. It would be very difficult to understand what was going on without fast, consistent tweets.
- egypturnash 5y agoWith slow, unreliable tweets he might have had to be filtered through normal channels, which would have meant they came out at more predictable times like “press conferences” after being discussed by the people involved in that whole process, instead of “I watched Fox and Friends and now I’m pissed off about whatever they wanted me to be angry about today”. It would have been much easier to understand things, that shit was constant chaos.
- bckr 5y agoPlease give an example of a circumstance in which this was true.
- jacobolus 5y agoTrump announced a ban on transgender soldiers via tweet like 1 day after giving a heads up to the Defense Secretary who was on vacation at the time. Nobody else in the military was prepared for the policy change and they were left scrambling. Multiple times he fired/hired senior members of his administration via twitter, including defense secretary, secretary of state, national security advisor, chief of staff, ...
- 542458 5y agoOne problem is that at a certain point you can’t just lose logs - for example, somebody might pay you per ad impression or click. Losing logs can mean losing evidence that those impressions/clicks happened. Logs aren’t just a debugging tool, they can be a critical legal CYA. I’m not saying that people don’t overcomplicate projects (they do) but a lot of simplifying assumptions can break down at scale.
- sateesh 5y agoBut why would one use log as source of truth ? If the info logged is important for the business, probably it should be stored in a DB (say).
- endisneigh 5y agoLogs are stored in a database, just not usually a relational one.
- sateesh 5y agoTrue, logs do needs to be stored somewhere. What l meant was: if the info is so important for the business, that detail should be stored in a system that provides a longer durability than just logging it.
- endisneigh 5y agoThough I agree, I think the parents point is that you don’t necessarily know if it’s important so you have to have some general record keeping with respect to “activity”
- alex_sf 5y ago> just logging it. This is the misunderstanding here. With a sufficiently complex system, they are never 'just logs'.
- Aeolun 5y ago
- deleted 5y ago[deleted]
- teraflop 5y ago> For example is strong consistency really necessary for tweets? Do you really need to see a followers latest tweet instantly? The problem is that "strong consistency" basically means your system behaves in predictable ways that you can easily reason about. If you give it up, you can see all kinds of failure modes that go beyond just seeing stale data. Suppose a user first sets their profile to "private", meaning only their trusted friends can see their posts. And then, having seen that that operation succeeded, they make a post that contains personal, sensitive information. You had better be damn sure that no user-facing components of your system can ever observe those two events in the wrong order.
- endisneigh 5y agoI don’t disagree at all with what you’re saying, but it’s all about expectation management.
- pcmoney 5y agoPeople should expect that when they mark their account as protected it is indeed protected. Things need to do what they say on the box
- RhodesianHunter 5y agoIf you threw strong consistency out the window and kept low latency you're still getting the "make profile private" event well before the user has navigated to the post page and composed a tweet.
- deleted 5y ago[deleted]
- saryant 5y agoThe problem is ensuring that event has been propagated all around the world so you aren’t displaying now-private tweets. That’s why we need strong consistency. I have spent much of my time at this company on this exact problem.
- obviouslynotme 5y agoAs you enter larger corporations, you see more Resume Driven Development. Overengineering isn't something to be avoided. It is the purpose of your 3 to 5 year stint there. Management rewards it so everyone participates in the circus together until one day the board fires 80% of engineering and hires contractors.
- lurker616 5y agoHow to prevent this though? From a game theoretic perspective?
- deleted 5y ago[deleted]
- pcmoney 5y agoTwitter’s real time aspect is a core value proposition compared to other social media platforms. Nothing tells you what is happening right this second as well as Twitter. It is faster than earthquake shock waves. And people rely on that speed for multi hundreds of millions of dollars in revenue.
- taeric 5y agoThis feels heavily overstated. Especially when you consider that the vast majority of "influencers" and such actually schedule their tweets. I agree that current tweets area important, of course. But I also acknowledge that it is just as often that i will be scrolling through the "In case you missed it" section of tweets. And, I have no idea how rapidly those actually made it to my timeline.
- saagarjha 5y agoIs it? Especially when there is an xkcd about this exact phenomenon? https://xkcd.com/723/ https://xkcd.com/723/
- taeric 5y agoIs it overstated? Since my timeline constantly shifts under my scrolling, often showing me it of date crap, yes. Worse, many posts are ads or trending items that.. no, you would not know if there were seconds of lag in them getting processed. You could almost argue they the small like counts have to be strong consistency. But, I often get notification of a like well before it shows on my timeline. So it is clearly not strongly consistent.
- pcmoney 5y agoYou have to take into account that the people that Twitter makes the most money off of for real time functionality are not integrated through the web interface.
- rabuse 5y ago
- bitcharmer 5y ago> Uncheck some boxes and the engineering becomes a lot simpler I think you're oversimplifying to the point of nonsense. In my domain reliable logging is an absolute must due to regulations and the fact that we deal with people's money. Pretty sure we're not the only industry that can't afford approaching logging as you painted it.
- endisneigh 5y agoSure, but most industries aren’t regulated to the extent that you would need logs.