6 ms·

I know effectively zero about Parler's infrastructure, but I would say it's not that uncommon to build a product tied specifically to AWS. What really surprised

I know effectively zero about Parler's infrastructure, but I would say it's not that uncommon to build a product tied specifically to AWS. What really surprised me though is that it took them days to get even a static homepage up.

duskwuff 6y ago
My understanding is that Parler actively avoided making technical decisions which would tie them to AWS. Their problems coming back online have primarily been because most major hosting providers have refused to take their business (and possibly also as a result of their ridiculous hardware requirements, cf. https://twitter.com/th3j35t3r/status/1350612426115452935 https://twitter.com/th3j35t3r/status/1350612426115452935).
- WJW 6y ago
  Those are... rather onerous HW requirements. Do you really need 20k cores to run a small-medium size social network? Asking because I have personal running a 40-50 million monthly users file sharing site on less than a tenth of that and most of it was done with Rails (itself not the most minimalist of frameworks).
  - snoshy 6y ago
    Yeah, those requirements are quite steep for what Parler does. That being said, it is actually incredibly hard to scale any product that sees a sudden massive influx of users, media attention, and conflict like this. I mean this from a purely technical standpoint. If your customer base is doubling every few weeks, that's simply not enough time to hire more engineers, onboard them, and get their resources focused on optimizing the stack to improve efficiency. It's also way more realistic to scale up your infrastructure in the short run until you know you've stabilized at a point where you can purchase and operate dedicated hardware to offset the steep margins cloud providers charge. On top of it all, their product itself seems off-putting to a large portion of tech workers, which would make hiring talented individuals even harder.
    - deleted 6y ago
      [deleted]
    - ashtonkem 6y ago
      At this point, we actually do know how to scale a social media site though, and if you’re making a new one you should anticipate such exponential growth. It seems they did not set a sufficient foundation for such a possibility, and got caught unprepared.
      - snoshy 6y ago
        For sure. I shared this in another comment earlier, but they literally were using a free trial of Okta [1] to secure prod. This is why all of their user data got leaked. Their code was written to fail open, so if Okta didn't respond for some reason to auth requests, access was granted by default. [1] https://twitter.com/okta/status/1348191370528256002 https://twitter.com/okta/status/1348191370528256002
        duskwuff 6y ago
        My understanding is that that was a myth. The data which was exfiltrated from Parler was retrieved through their mobile API, which allowed for easy enumeration of content and didn't implement some expected access controls (like blocking access to deleted content).
        ashtonkem 6y ago
        Them using a trial version is not a myth; that is from the official Okta Twitter account. My understanding of the scraped data is that some APIs used both sequential keys, and ignored both permissions and their soft delete flag. It’s not clear to me whether or not that API was permanently unauthenticated, or if it failed open once Okta locked them out.
        duskwuff 6y ago
        That API was effectively public by design. It's the same API that anyone using the Parler mobile app would have been accessing.
- PaulDavisThe1st 6y ago
  why are they not looking to colocate instead?
  - snoshy 6y ago
    Probably because their workloads haven't stabilized at any meaningful threshold. You can only justify the capital expenditure if you know you can amortize it over time. Hard to do for a startup whose existence is being threatened. They are also a political hot potato, and would need the colo provider to agree to take them on and withstand the public pressure, which I doubt most would want.
    - PaulDavisThe1st 6y ago
      Their proposed current hardware costs are a tiny capital outlay. Future current hardware costs are a tiny capital outlay. They will pay far more for bandwidth than they ever will for hardware.
      - snoshy 6y ago
        I don't necessarily disagree with the hardware capex vs bandwidth tradeoff, but I don't think either of us are aware of their balance sheet to reasonably claim if this is an insurmountable cost for the business today or not. Paying for several thousand servers, getting them up and running, and tuning them isn't exactly a trivial effort... and then you have to deal with bandwidth.
        PaulDavisThe1st 6y ago
        From the tweet, looks like 540 servers max at this point. And sure, not trivial but you shouldn't be planning to run a large chat system without the technical staff to pull something like that off, whether on AWS or not.
      - ashtonkem 6y ago
        They probably don’t have the personnel who know how to order, provision, and physically manage that kind of hardware. They’re a small startup, and going with AWS means that they probably don’t have any of the necessary skills to do that.
- vkou 6y ago
  Peruse https://www.lastweekinaws.com/blog/parlers-new-serverless-architecture/ https://www.lastweekinaws.com/blog/parlers-new-serverless-ar... Unless you are building your webservice like it's 1998 (And webservices in 1998 sucked), you can't really disentangle from hosted solutions.
  - tptacek 6y ago
    Sure you can, and lots of people do.
    - snoshy 6y ago
      Agreed, I would wager that this is where a good majority of startups begin until they take a good stock of the situation at hand.
    - zaphar 6y ago
      Heck, I do it in my current job. It's not that different from running in AWS.
ashtonkem 6y ago
Parler appears to have had an inefficient architecture that required an absurd amount of compute power to run compared to their customer base. Apparently they were running one hundred metal instances for their database alone; speculation on Twitter was that they’re struggling to find someone who can give them that much capacity in a hurry.

duskwuff 6y ago

My understanding is that Parler actively avoided making technical decisions which would tie them to AWS. Their problems coming back online have primarily been because most major hosting providers have refused to take their business (and possibly also as a result of their ridiculous hardware requirements, cf. https://twitter.com/th3j35t3r/status/1350612426115452935 https://twitter.com/th3j35t3r/status/1350612426115452935).

WJW 6y ago
Those are... rather onerous HW requirements. Do you really need 20k cores to run a small-medium size social network? Asking because I have personal running a 40-50 million monthly users file sharing site on less than a tenth of that and most of it was done with Rails (itself not the most minimalist of frameworks).
- snoshy 6y ago
  Yeah, those requirements are quite steep for what Parler does. That being said, it is actually incredibly hard to scale any product that sees a sudden massive influx of users, media attention, and conflict like this. I mean this from a purely technical standpoint. If your customer base is doubling every few weeks, that's simply not enough time to hire more engineers, onboard them, and get their resources focused on optimizing the stack to improve efficiency. It's also way more realistic to scale up your infrastructure in the short run until you know you've stabilized at a point where you can purchase and operate dedicated hardware to offset the steep margins cloud providers charge. On top of it all, their product itself seems off-putting to a large portion of tech workers, which would make hiring talented individuals even harder.
  - deleted 6y ago
    [deleted]
  - ashtonkem 6y ago
    At this point, we actually do know how to scale a social media site though, and if you’re making a new one you should anticipate such exponential growth. It seems they did not set a sufficient foundation for such a possibility, and got caught unprepared.
    - snoshy 6y ago
      For sure. I shared this in another comment earlier, but they literally were using a free trial of Okta [1] to secure prod. This is why all of their user data got leaked. Their code was written to fail open, so if Okta didn't respond for some reason to auth requests, access was granted by default. [1] https://twitter.com/okta/status/1348191370528256002 https://twitter.com/okta/status/1348191370528256002
      - duskwuff 6y ago
        My understanding is that that was a myth. The data which was exfiltrated from Parler was retrieved through their mobile API, which allowed for easy enumeration of content and didn't implement some expected access controls (like blocking access to deleted content).
        ashtonkem 6y ago
        Them using a trial version is not a myth; that is from the official Okta Twitter account. My understanding of the scraped data is that some APIs used both sequential keys, and ignored both permissions and their soft delete flag. It’s not clear to me whether or not that API was permanently unauthenticated, or if it failed open once Okta locked them out.
        duskwuff 6y ago
        That API was effectively public by design. It's the same API that anyone using the Parler mobile app would have been accessing.
PaulDavisThe1st 6y ago
why are they not looking to colocate instead?
- snoshy 6y ago
  Probably because their workloads haven't stabilized at any meaningful threshold. You can only justify the capital expenditure if you know you can amortize it over time. Hard to do for a startup whose existence is being threatened. They are also a political hot potato, and would need the colo provider to agree to take them on and withstand the public pressure, which I doubt most would want.
  - PaulDavisThe1st 6y ago
    Their proposed current hardware costs are a tiny capital outlay. Future current hardware costs are a tiny capital outlay. They will pay far more for bandwidth than they ever will for hardware.
    - snoshy 6y ago
      I don't necessarily disagree with the hardware capex vs bandwidth tradeoff, but I don't think either of us are aware of their balance sheet to reasonably claim if this is an insurmountable cost for the business today or not. Paying for several thousand servers, getting them up and running, and tuning them isn't exactly a trivial effort... and then you have to deal with bandwidth.
      - PaulDavisThe1st 6y ago
        From the tweet, looks like 540 servers max at this point. And sure, not trivial but you shouldn't be planning to run a large chat system without the technical staff to pull something like that off, whether on AWS or not.
    - ashtonkem 6y ago
      They probably don’t have the personnel who know how to order, provision, and physically manage that kind of hardware. They’re a small startup, and going with AWS means that they probably don’t have any of the necessary skills to do that.
vkou 6y ago
Peruse https://www.lastweekinaws.com/blog/parlers-new-serverless-architecture/ https://www.lastweekinaws.com/blog/parlers-new-serverless-ar... Unless you are building your webservice like it's 1998 (And webservices in 1998 sucked), you can't really disentangle from hosted solutions.
- tptacek 6y ago
  Sure you can, and lots of people do.
  - snoshy 6y ago
    Agreed, I would wager that this is where a good majority of startups begin until they take a good stock of the situation at hand.
  - zaphar 6y ago
    Heck, I do it in my current job. It's not that different from running in AWS.

ashtonkem 6y ago

Parler appears to have had an inefficient architecture that required an absurd amount of compute power to run compared to their customer base. Apparently they were running one hundred metal instances for their database alone; speculation on Twitter was that they’re struggling to find someone who can give them that much capacity in a hurry.