9 ms·
Parler’s epic fail: A crash course on running your own servers
- 542458 6y agoI can’t help but find the introductory quote a bit much, given that Parler’s moderation scheme (judgement by other Parler users) typically resulted in the removal of all opinions other than those held by the majority of Parler users. In any case, Parler’s problem (well, one problem of many) was that they had MASSIVE hardware requirements that dramatically cut down on the number of places that could practically host them. It also seems that despite assurances and good sense, Parler had deeply tied itself to Amazon’s APIs, making migration off AWS slow even once a host was found.
- treeman79 6y agoMy initial thought was it can’t be that hard to migrate off quickly. Then I thought about all the major sites I’ve worked on or built over the past decade. Oh crap. There is zero chance I could do it quickly. So a turn around from 20 years ago when I would setup LAMP applications on new new hardware in an afternoon.
- john2010 6y agoVery likely many isps whitelist speedtest.net to show amazing speeds.
- 542458 6y agoThat’s why you also check Fast.com - it’s on Netflix’s servers, so you can’t whitelist Fast without also whitelisting Netflix.
- rhencke 6y agoIt really was such a brilliant move on Netflix's part.
- OldHand2018 6y agoMy rinky-dink little ISP has peering connections with Netflix, Apple and AWS. The operate in 2 midwestern cities with maybe 50,000 total customers, so it seems like the Netflix vs ISP issue only exists with the Comcasts and Xfinities of the world. Everyone else is really happy to work with Netflix; it costs them nearly nothing. https://openconnect.netflix.com/en/peering/ https://openconnect.netflix.com/en/peering/
- TheRealSteel 6y agoSurely it'd be trivial to increase the user's speed to Netflix servers if they've visited Fast.com in the last three minutes or similar?
- 542458 6y agoThanks to TLS, Netflix and Fast are indistinguishable. All your ISP knows is the IP address you’re talking to. They’d only know that you visited Fast in particular if you were using your ISP’s DNS, which you shouldn’t be using anyways :)
- cortesoft 6y agoExcept SNI will leak the domain name of the host you are connecting to.
- Lammy 6y agoPersonally I trust my ISP DNS a hell of a lot more than Google/Cloudflare. Why would I want to give them even more data about me?
- nyx_ 6y agoI've had ISP DNS servers that redirect NXDOMAIN responses to spammy "search" pages full of sponsored crap and banner ads. There's always OpenNIC, DNS.watch, or Quad9 if you're after something that isn't operated by a creepy megacorp.
- jen20 6y agoWow, where do you live?! Across the US and UK, I’ve consistently found home-ISP-supplied DNS to be slow, spammy and unreliable.
- tomatotomato37 6y agoIt's indistinguishable from an protocol perspective, not a data analysis perspective. Or to put it more practically, if a large data stream from Netflix lasts more than 10 seconds, it's video.
- rhencke 6y agoI grew up in an era where running your own servers was just generally accepted practice. It's been fascinating watching how dramatically that viewpoint has shifted over the years to the point where it is now a novel idea to do so.
- 542458 6y agoI switched my personal website to a Raspberry Pi 4 in my basement a while ago, with cloudflare caching and roll-your-own DDNS - I’ve been pleasantly surprised by how easy the whole thing was to do (although my requirements are far from exotic).
- Balgair 6y agoAny good resources to share?
- devwastaken 6y agoNot the poster but I assume an easy setup is simply ubuntu server, nginx, get a domain and put it on cloudflare free plan, setup a script that updates your home IP to cloudflare dns records. Then do letsencrypt for the https cert, setup automatic renewal, and you're probably good.
- ping_pong 6y agoI worked at a startup that managed its own bare metal. It was very hard and time consuming. And when things went down it was on you. You needed a really good datacenter partner to make sure that they were on top of things and could drop things at a moment's notice. But you don't know how responsive they will be until you're actually experiencing an outage. The level of convenience that cloud providers give is just orders of magnitude more efficient and easier.
- endisneigh 6y agoDoes there exist a de-platform proof way of hosting something on the internet? Even if you host your own server on your own premises they could forbid you from using the location, shut off your electricity, etc. Is it possible to host a website entirely on bittorrent? I suppose there's also IPFS but I'm not entirely familiar. Maybe once solar is cheap enough you could launch some sort of array of powered drones into the sky that follow the sun that send files to people via P2P - solar mesh network if you will.
- madamelic 6y agoYes, there are. IPFS, I believe, has a solution. The one I am most familiar with is Dat Browser.
- swebs 6y ago4chan has survived the wannabe-stasi deplatform mob all these years simply by hosting the servers in the admin's basement.
- rvz 6y agoLike 4chan, Gab also survived the de-platforming witch hunt and are self-hosting almost everything.
- rvz 6y agoTo Downvoters: So the above statement is somehow 'false' and 4chan and Gab are NOT self-hosting and are in the same situation like Parler which is still down? Does anyone have concrete evidence that contradicts the parent AND the grandparent statements? Or is it just the emotional reaction to two websites that one just felt the need to downvote a fact without reason or counter evidence? Let's not get any form of irrationalities in the way of the actual facts and the truth. Explanation? (With evidence of course)
- generj 6y agoI think people are responding to language like “wannabe-stasi deplatform mob” and “de-platforming witch hunt” more than the actual facts or websites involved. That language is not neutral, so the response isn’t surprising.
- tpmx 6y agoIt's kinda funny that apart from the headline this post is a typical circa 2005 tutorial on how to run your own large volume website cheaply.
- gerikson 6y agoThat's when I stopped learning how to admin systems so I am very down with this...
- tpmx 6y agoWell, it works. So much money is wasted with AWS. (Look at at their margins.) It's useful in development/testing. It's irresponsible to use managed services that don't have an easy migration path to something open.
- rvz 6y agoIt works for comma.ai as they are self-hosting their deep-learning system in house and not in the cloud with these huge costs. Deep-Learning in the cloud really is a scam and a complete waste of money.
- lightlyused 6y agoI wouldn't say it is a scam. There are a lot of benefits to running load in the cloud, understanding what works for your business is the hard part.
- ralusek 6y agoThem having margins in no way means it's a waste of money. That's not how things work. It would cost just companies so much more money for the devops team needed to achieve parity with the services they provide. Do you look at a car manufacturer, see that they're making a profit, and assume "I could clearly build a comparable car for less money?"
- 6y ago
- TameAntelope 6y ago> freedom minded > tech oligarchs ...here we go. I really struggle to understand how some folks didn’t realize they were using other people’s things until recently. Maybe the fact that they weren’t physically in a space was hiding the proverbial threshold crossing activity that takes place a dozen or so times as you travel to twitter.com. It is free speech to kick someone off your platform. Freedom minded individuals seem to think freedom only goes in one direction. I will defend DJTs right to tweet stupid shit all day and every day, just as I’ll defend Twitter’s right to kick him off the platform.
- ClumsyPilot 6y agoIs it freedom of speech to cut off your power supply and running water if I don't like your face? What about mobile connectivity and internet? Can we refuse to sell you groceries? Can you live without 'using other people's stuff' except like a hermit in a cave?
- SAI_Peregrinus 6y agoPower supply and water are utilities. Mobile connectivity and internet are not. Groceries are not. Someone else's speech isn't a protected class, except for speech due to religion. Discrimination based on things that aren't a property of being in a protected class is legal for any non-utility. Personally I'd say that internet service providers should be utilities and regulated as such, but currently they aren't so the ISPs are allowed to decide what speech to carry.
- TechnoTimeStop 6y agoAuthenticity plays a huge role here. Why do we let shit by the metric ton over the firewall attack free liberties and social discourse online? China literally has payed 3 million + people for years to do this, that's two agents for every social group in the free world. Most of the bullshit our western democracies have experienced in the last years online should be archived and studied for the crimes our Russian Chinese friends have committed against our free democracy.
- 13415 6y agoI doubt running your own server is always the right solution.There are pros and cons for and against hosting yourself. A small company without dedicated security team may get hacked and all customer data exposed, for example. Contrary to what this article insinuates, the vast majority of companies do not have to worry about violating the terms of services of cloud providers and other external services, because they don't offer services designed to violate besaid terms of services. As for Parler, of course they should have seen that coming. It's pathetic that they didn't. They should have looked at sites like the Piratebay for how to do it and prepared a bit more.
- ClumsyPilot 6y ago"the vast majority of companies do not have to worry about violating the terms of services of cloud providers" https://blog.checklyhq.com/why-the-recent-digital-ocean-killed-my-company-incident-scares-the-hell-out-of-me/ https://blog.checklyhq.com/why-the-recent-digital-ocean-kill...
- 13415 6y agoA personal anecdote why some blog poster is scared and worried about ToS of cloud providers? Sure, go ahead and host your website in your mom's basement. That's exactly my point, a company who has reason to believe they will violate the ToS of business partners should not make business with those business partners. It's kind of trivial. That's why Parler was such an epic fail, not because they didn't host everything themselves. They were acting as if they'd care about free speech - unless it wasn't totally compatible with their personal views, in case of which they'd ban instantly - and did not prepare in the slightest for contingencies and were caught in the cold. All of that in the light of plenty of precedents, ranging from Napster over Piratebay to ISIS propaganda websites. It baffles my mind how a company could be so unprepared.
- GekkePrutser 6y agoThe same happens with poor management of cloud services. So many companies got 'hacked' simply by having open S3 buckets with customer data hanging around. A cloud service doesn't shield against incompetence.
- helsinkiandrew 6y agoI’m not sure running their own hardware and relying on an ISP or colocationg at a datacentre would help Parler compared with renting services/VMs from someone like Amazon. Just about all ISP have terms and conditions that prohibit use that they find offensive. Data Foundry, mentioned in the article, acceptable use policy is below and I’m sure could be used to kick out Parler https://www.datafoundry.com/legal/aup https://www.datafoundry.com/legal/aup
- rbanffy 6y agoIt's possible to hide one's presence from the datacenter by a shell company and a CDN.
- duhast2020 6y agoHow? It would take one minute for a dude with some sniffer on his wifi to find the DNS entries and CDN used. Go to the press with "Parler is being hosted out of domain Foo w/ CDNfront" and viola.
- rbanffy 6y agoAnd, say, your CDN would not comment on their clients.
- helsinkiandrew 6y agoUltimately you can't hide IP addresses - an app or website will be making requests to or served from an IP address, which is physically traceable to the IP/data center. Then either a subpoena or publicity against the ISP will force them to do disclose who the customer is and/or remove them.
- rbanffy 6y agoYou'll be hitting the CDN, not the actual server. As for the subpoena, you can make it harder to find you by spreading your addresses across multiple jurisdictions. Put the company that owns the IP in one country and make it rent them to another company in a different country who owns the servers and the website and rents the domain name from a third company in yet another jurisdiction. Eventually they'll find you (there aren't many ways to escape a government forever), but this game can buy you enough time to move someplace where extradition is unlikely.
- ping_pong 6y agoThe "reason" given why Parler failed is awfully glib and one dimensional. You can't just serve content to millions of people per day and think you can just up and move to another provider. Had Parler just moderated their posts like they were asked to do, they would still be around. You can still have right wing, even extreme right wing views without calling for violence or organizing insurrections. The fact they essentially refused to moderate and let calls for violence fester on their platform is why they were shut off.
- rbanffy 6y agoThat and their hardware footprint seemed ridiculously overprovisioned. You can serve content to millions of people per day with fewer boxes than they were using. I always try to design a "degraded mode" read-only into my apps that covers maintenance windows and unscheduled outages. That is usually easier to lift and shift than the full app and can even be hosted someplace else if you don't mind it lagging behind the live website.
- alkonaut 6y agoOne can only assume that 90% of their hardware was doing something much heavier than “microblogging”. I’m guessing e.g tons and tons of media processing like video transcoding. Would be interesting to hear how much it costs Twitter to have video features. If I was Parler right now I’d just switch off media and throw up a very lean MVP version with just text. Even with millions of users Parler has the luxury of almost no consistency problems at all. It’s all eventual consistency and simple replication. Shouldn’t need a lot of hardware at all.
- rbanffy 6y agoI'm thinking more about sentiment analysis and profile enrichment. Remember they are (or at least were) funded by the Cambridge Analytica folks.
- _-david-_ 6y ago>You can still have right wing, even extreme right wing views without calling for violence or organizing insurrections The insurrectionists did not actually use Parler. They used Twitter and Facebook.
- amluto 6y ago> Most datacenters only lease by the rack. These racks can hold up to 42 servers and are far too expensive. Hurricane Electric will happily lease you an entire rack for $400/mo with a 1Gbps connection. I would argue that, in comparison to the prices the OP mentions, Hurricane Electric's price is quite good. Also, there is one fewer middleman between you and the Internet (and power, and rack space, etc, and the front door, etc). (I believe that $400 number does not include vast amounts of power, so the actual price tag for filling that rack with conventional hardware may be rather higher. On the other hand, depending on your use case, fitting in a small power budget may be straightforward.)
- dilyevsky 6y agoThey have some power included in that $400
- dd36 6y agoThis made me realize that one reason many of us use cloud hosting is that our home internet is lacking, whether through caps or the lack of fiber deployment. It was easy for Congress to railroad the rights of private property owners and cities for 5G but they can’t seem to do it for fiber... Maybe this can get Conservatives to realize that unfettered broadband access lessens the power of the hosts.
- rShergold 6y agoThe physical network itself evolved away from peer -> peer back to client -> server. With IPv6 I'd hoped we would today be living in a world where I could send my friend the url fe80::1ff:fe23:4567:890a/cat.mov and they could download a video directly from my phone. No matter where my phone happens to be in the world. People could host "micro services" on their personal devices and the whole world could access them. If someone wants to know if I'm free on a date they could connect to my phone's calendar app directly and ask. There are so may possibilities of truly peer to peer applications. But the network itself prevents this. Bittorrent has to jump though multiple NAT busting hoops to allow two internet users to talk to each other directly. limited IPv4 addresses mean home internet connections have dynamic IP addresses which means no one can realistically host at home. Because no one hosts at home they consume far more than they upload. Because of this the physical network was built with a much higher download bandwidth than upload. It's a vicious cycle that wasn't intentional it just sort of evolved that way.
- granshaw 6y agoThis so much. People don’t realize how much ubiquitous NAT has eroded individuals’ online capabilities and freedoms and shifted the internet from what it was meant to be (much more bidirectional than it is now) I shudder when tech folks themselves insist on NAT as a core security tenet - sure it might be a convenient extra layer but it’s so sad what’s been sacrificed in its name
- spinlock 6y agoConservatives are why we don't have fiber. AT&T was on track to install fiber in ever every home in America by the year 2000 before Bush I shut them down.
- yannoninator 6y agothis is all good except when people in a different country try access your service, they start to complain about your service being slow due to massive latency and lagging. and then you wished you would have went to the cloud after all. now you've got another problem.
- zxcvbn4038 6y agoThis article is kind of depressing, when did hosting your own server become a lost art? One of parler’s problems was they didn’t scrub headers which made it very obvious they were using AWS and which AWS services they were using. AWS is just a bunch of APIs - they could have colo’d their front end or put a bunch of varnish servers in front if it at a colo and nobody would have been the wiser they were using AWS on the backend (assuming their AWS account didn’t have Parler Inc in the payment details). From there getting taken down from one colo would just be an issue of spinning up a front end at another colo. Front that setup with a couple CDNs and now you have layers where you have to coordinate between half a dozen companies to bring the site down completely, meanwhile your data is safe in AWS because nobody knows your using AWS or even if they do it’s hard to pick you out from the millions of other AWS users.
- nacs 6y agoYep, the amount of incompetence in Parler's case is just mind-boggling. Their failure to make it cloud agnostic being 1 (even though they claimed before they went down that their app could run without AWS without issue). Their failure to be incapable (still) to get it running again on non-cloud hardware.. And of course their inability to, as you suggest, move the frontend routing to a CDN/Cloudflare/DDosguard type service and have all the heavy lifting continue to be done via AWS/Azure/Google till you can run 100% on your own hardware.
- paul_f 6y agoMaybe they didn't think they needed to. Before a few months ago, you didn't yank someone off your network because you didn't like their politics. (yea yea, terms of service something or other)
- 4cao 6y ago> Before a few months ago, you didn't yank someone off your network because you didn't like their politics. AWS dropped WikiLeaks as a customer back in 2010 for political reasons, and apparently this was announced by a US senator even before AWS did so themselves: http://edition.cnn.com/2010/US/12/01/wikileaks.amazon/index.html http://edition.cnn.com/2010/US/12/01/wikileaks.amazon/index....
- HenryBemis 6y agoThe photos though.. If you are reading this remove the server or the rug/carpet!!!! All the fluff from the carpet and the dust from the floor invading your box and will be chocking the fans!!!
- julienfr112 6y agoNice post and inspiring, but one small point bother me: "I feel GREAT not living in FEAR". Well, you just traded the fear of being shut down by big corpo for the fear of having your hard disk or memory or fan or whatever fail.
- nacs 6y agoMost rackmount servers double up on everything like power-supplies. Harddrives should be on RAID so a single drive failure shouldn't be a problem and you would have some time to goto the colo and switch out the hard drive. CPU and RAM last a very long time so it shouldn't be a problem (I've had literally 1 RAM stick failure in over a decade of hosting and the datacenter swapped out the bad memory stick in less than 30 minutes -- it also didn't bring down the server either, the kernel log started showing a bunch of ECC issues and a quick memory check pointed to the stick that had the problem). And of course you can just get double or triple the amount of servers with IP failover and such to be even more resilient to hardware failures. This article is just showing how to get started. Throw more servers at it for more redundancy and scaling as needed.
- ampdepolymerase 6y agoRunning on a shoestring budget is not very interesting. What would be interesting is running a well funded production level data center in a completely hostile political environment with every other commercial entity trying to refuse business with you.
- doggydogs94 6y agoOn the other hand, ThePirateBay seems to be able to stay up.
- joezydeco 6y agoPeter Sunde Kolmisoppi (@brokep) 1/10/21, 03:05 "The pirate bay, the most censored website in the world, started by kids, run by people with problems with alcohol, drugs and money, still is up after almost 2 decades. Parlor and gab etc have all the money around but no skills or mindset. Embarrassing. The most ironic thing is that TPBs enemies include not just the US government but also many European and the Russian one. Compared to gab/parlor which is supported by the current president of the US and probably liked by the Russian one too. First time tpb got shut down was because a Mexican gang wanted some cash. Took a day to move. Second time was when the USG forced Sweden to send 50 cops and even arrest the lawyer. Took 3 days to get back online, 1.5 days was spent to get drunk and party."
- jette_au_loin 6y agoIt's probably easier to host a website serving a bunch of .torrent files. There's dozens of those around.
- skynet-9000 6y agoOr just magnet links.
- im3w1l 6y agoI'm picking up a weird sentiment from these articles about parler. Something like... shoving someone so they fall and break their teeth, and then trying to blame them and saying, "it's your own fault for having such poor balance". A coping strategy for guilt? I just meant to harm you a little bit?
- probably_wrong 6y agoI think your analogy breaks because it represents the people "shoving" Parler and the people talking about this afterwards as one, which is not the case. A better analogy would be: Parler likes to hang out with the school bully, even though everyone told them not to. One day, the bully shoved Parker and it broke it's teeth. Should the bully's conduct have been allowed? Almost certainly not, and that's a problem. But at the same time Parler was warned about the bully and yet they disregarded all warnings, believing that it wouldn't happen to them. Parler can make a stand, saying "it is the school's job to keep bullies under control" (which is true) and live it's life as if there were no bullies because that's what life should be. Some people do precisely that for worthy causes, knowing full well what the consequences are (see: Rosa Parks). Or they can do what most of us did in school and stay away from the bully. Less principled? Sure. But our dentist bills are a lot cheaper as a result.
- im3w1l 6y agoThey are not the same people, but are they defending "their side"?
- probably_wrong 6y agoI don't think people are defending Amazon's side. Instead they are (somewhat reluctantly) admiting that what Amazon did is legal, which is not the same.
- devwastaken 6y ago"I'm eager to give big social media sites like Facebook and Twitter a run for their money." Hosting servers isn't the hard part, and it isnt the same as running a social media platform. This blog is riddled with neivete and a misuse of the word "fear" to get clicks. Don't even get me started on trying to use the "they didn't come for me" quote when relating to online posts. Government != Private companies If you want to compete you will require moderation, period. Illigal and disturbing content will have to be removed and sectioned off from your users. Or do really you want the average user seeing a trending video of gore and targeted harassment of people? Even 4chan moderates. You will have to serve hundreds of petabytes of image and video data - you will have to use cloudflare or some other cache mechanism like everybody else. Cloudflare was the reason 4chan was kept alive btw. You need lawyers on retainer that understand law pertaining to your business. If not then you're left open to civil liabilities anyone can exploit. You need money. You will require algorithms for promotion of content which requires user tracking which requires databases that contains peoples habitual information. You've now created facebook and twitter or you're running on investor money. Eventually you will have to sell out your users. People forget that the "big social media" all came from the same small places when it was a lot harder to do ( do you know how much more expensive bandwidth used to be?). The difference is they got popular and smelled the money. It takes a very special community to not do the same.
- amachefe 6y agoParler problem was not all the things you listed, it was simple hosting and the blog provided very clear steps to go about it
- listenallyall 6y agoThis article has ZERO information about Parler or it's tech stack. Total clickbait. Yet it still has 100+ upvotes and has generated 100+ comments.
- amachefe 6y agoVery good one, and timely It's been an obsession of mine for a long while, and i am overjoyed it has been written down I have not been comfortable with the Cloud or nothing and the fact that it looks like it no longer possible. This video also helps for people who want to go into self hosting https://www.youtube.com/watch?v=Qx_vGdnBqeE https://www.youtube.com/watch?v=Qx_vGdnBqeE