4 ms·
I just want to serve 5 terabytes [video]
- jonatron 2y agoText to speech has improved a lot since then.
- fifticon 2y agobut the process to serve 5 terabytes hasn't.
- flymasterv 2y agoIt’s so bad, still.
- alexey-salmin 2y agoWhat do you mean? 5 terabytes is so little that I forgot how to count that low
- vitus 2y agoEh, it has, though. Half of it is about setting up a bigtable for hosting the data. I don't know of any team that's set up a new bigtable in the past 5 years; if you're really going all-in on the cloud ecosystem (per AppEngine) you'll stick it in Google Cloud Storage (where you'll note the file size limit is... 5 TiB) and call it a day. (Also: PCR zones are basically dead.) The other half is about setting up monitoring, where the mentioned choices are Diplomat (which I don't think anyone's used in about a decade) and Borgmon (which is barely staffed and strongly, strongly discouraged for any new use cases). Borgmon readability hasn't been enforced in years. And again, if you're spinning this up in GCP, just set up some cloud monitoring.
- Nihilartikel 2y agoBorgmon readability was my... least favorite readability. Is 'monarch' still a thing? It was newish around the time that I left.
- vitus 2y agoMonarch is very much still the recommended monitoring backend.
- kccqzy 2y agoBut Monarch is the backend, so people rarely interact with it. As of the time I left Google, the UI is automon, the language to write alert rules in is gmon, the language to do interactive exploration in is mash, and the CLI to interact with retentions and metrics etc is monarch_tool. It didn't matter if the backend was Monarch or not.
- vitus 2y agoSure. Most engineers will probably poke around on the UI to look at graphs, and write any special monitoring configs in GMon (a DSL in Python, which is then transpiled to mash). Directly interacting with monarch_tool should be rare, as monitoring integration with rollout automation is also provided by default. Further, the entire point of automon is to automatically generate common monitoring dashboards, which you should expect to be sufficient if you're creating a bog-standard setup. > It didn't matter if the backend was Monarch or not. It totally does. Borgmon has a totally different data model, a custom query language, its own UI, and various quirks along the way. To add insult to injury, it was a very real thing where if you wanted to set up new monitoring, you needed to get someone with Borgmon readability to approve your change (that requirement has since been lifted). Meanwhile, today, you don't need anyone with Python readability to ever look at your GMon code. You can have Automon graphs that fetch from Borgmon under the hood, but everything else that you've described is 100% Monarch-specific.
- new_user_final 2y agoThe video is from 2010
- JKCalhoun 2y agoWe love the OG presentation.
- redm 2y agoSomehow this has only gotten funnier since it was originally posted..
- ur-whale 2y ago> Somehow this has only gotten funnier since it was originally posted.. Especially the "only if you think your users are scum" part. Back in the days, even the asshat in green seem to have tried to uphold some kind of guiding principles.
- medwards666 2y agoBut ... but ... users _are_ scum, aren't they?
- staplung 2y agoThat’s no asshat; that’s Broccoli Man! Not the hero we needed but the hero we deserved.
- JKCalhoun 2y agoYeah, er, or sadder.
- leoh 2y agoThis tracks with my experiences both at Google and [Google] X.
- deleted 2y ago[deleted]
- probably_wrong 2y agoFor those who were too young in 2012, here is some background: https://news.ycombinator.com/item?id=29082014 https://news.ycombinator.com/item?id=29082014
- cjs_ac 2y ago> Rule 3. Fancy algorithms are slow when n is small, and n is usually small. Fancy algorithms have big constants. Until you know that n is frequently going to be big, don't get fancy. (Even if n does get big, use Rule 2 first.) For example, binary trees are always faster than splay trees for workaday problems. -- Rob Pike, Notes on Programming in C, 1989[0] Generally speaking, I feel that the bureaucracy involved in a programming project should be proportional to the scale of the project itself. If the 'getting started' tutorial for your programming language demands that I choose a package name for my Hello World program, you fucked up. [0] https://doc.cat-v.org/bell_labs/pikestyle https://doc.cat-v.org/bell_labs/pikestyle
- wepple 2y agoI mostly agree, because most proof-of-concepts don’t make it very far, and LLM assisted dev is really strong for that kind of low-barriers experimentation. The problem is that if you build quick n dirty, but then end up reaching success; you’re in for a bad time or a rewrite. This is where the above example of a single algorithm departs from a whole stack. Further, one bad “hello world” on a google/apple/amazon domain can make the front page of the NYTimes
- bee_rider 2y agoI dunno, if the package name is just some local thing that could be nice. Is “all software is part of a package” ever a nice simplifying assumption?
- stickfigure 2y agoNobody writes hello world for a living. This is a useless metric.
- netsharc 2y agoRecently I went on a trip with a group. Lots of videos and drone footage. Gigabytes of data. The group leader got a 100GB Google Drive space. But if I upload my stuff into it, my stuff doesn't take his 100 GB, but counts against my quota, unless I modify the files to say that $GROUP_LEADER now owns those files. Well, whatever. I try to download his videos from the web UI. Select all files in directory, wait about 20 seconds, download zip, wait another 20 seconds, finally the file download dialog shows up. It gives me a 2.3GB zip file. I open it, it's just a few files and not the complete directory. It doesn't give me the contents of the whole directory, just files until it reaches 2.3GB, and then silently fails. Great jaaab, Google!
- H8crilA 2y agoThey didn't have Borgmon readability.
- vachina 2y agoGdrive’s download as zip never worked reliably for me, and would like you, fail silently with a happy download completed. For large binaries better to use Google Drive for Desktop (their official client), which is kinda like rsync mount but bulletproof.
- ivanmontillam 2y agoYeah, except for office files (documents, spreadsheets, slides). They become cloud shortcuts, they don't exist on your disk "physically." Don't rely on that for backups. Google Takeout converts them to actual, solid files.
- netsharc 2y agoI don't see "files shared with me" on the Google Drive for Desktop, do I need to RTFM or watch a YouTube tutorial, because of course it's completely obvious, just like the 100 GB quota issue I described above?
- adithyassekhar 2y agoDid you check your browser for google drive site settings? Google drive has a zip size limit of around 2GB while downloading folders. Usually when you try to download a folder larger than 2GB, it will split these into multiple zip files which all start downloading parallely. Most browsers block these behind this permission prompt near the address bar "Allow site to download multiple files". If you blink you'll miss it.
- alexey-salmin 2y agoShould've just served it directly from MongoDB. It's web scale.
- kburman 2y agoI wish this were a joke, but I’ve actually worked on a production application designed like this. They use a single table with columns named PK1, PK2, PK3, PK4, and so on, dumping all kinds of data into it. users, orders, addresses, and even blob data. For example, PK1 might be the user ID if the row is a user entry, or PK2 could be the address ID if it’s an address object. And if PK1 happens to contain a #, that might mean it’s a user profile object because apparently that’s how they chose to distinguish it. As more entities were added, the table just kept morphing into this unstructured, unreadable mess. There’s absolutely no good reason for this kind of design. It’s just a lack of understanding of how NoSQL systems are meant to be used. The team was told to “utilize NoSQL solutions,” and this is the chaos they produced. I wish I could unsee it, but the damage is already done. It still gives me nightmares.
- cruffle_duffle 2y agoThe real fun as you get more experience isn’t the immediate horror show itself but witnessing the process and internal politics that made such a thing happen in the first place. How did somebody so clearly incompetent get so much political capital that such a system could not only be created but perpetuated and even praised as a good idea. And then you start to wonder… do the people who praise it secretly know it’s hot garbage or do they genuinely believe their praise? And the answer to that invites even more questions. And then you get to watch everything else because such a monstrosity doesn’t get built in a vacuum. It’s probably the tip of the iceberg in terms of organizational dysfunction. Who are they hiring (and what does that say about you?). What process enables such things? Why do people tolerate it? Do they know any better? Is it fixable (probably not!) and if so, is it worth it (also no!) …and of course how the fuck do I get out before I start drinking the koolaid too?
- seeknotfind 2y agoGoogle makes the impossible, hard, and it makes the easy, hard.
- deleted 2y ago[deleted]