8 ms·

Chaos in the Cloudflare Lisbon Office

• Havoc 2y ago
  How much of this is actually needed for security vs just cool?
  • tdeck 2y ago
    You mean how much of this is theater so you won't think about how this company MITMs a meaningful percentage of the internet's encrypted traffic?
    • koakuma-chan 2y ago
      You can probably say the same about AWS.
• WindyColors 2y ago
  [flagged]
• umvi 2y ago
  Is this just for PR/show and the real entropy generator is standardized across sites to be something boring in some back room? Or if a terrorist came in and cut power to a chaos wall would it really cause an extended outage?
  • linsomniac 2y ago
    >if a terrorist came in and cut power to a chaos wall That's an interesting way of saying "if the wall loses power". Your name wouldn't be Hans Gruber, would it?
    • 6stringmerc 2y ago
      “I’m a thief and an excellent one at that, and considering I’ve just promoted myself to kidnapper, you should show some respect!”
      • gunian 2y ago
        meh hollywood needs to upgrade its villains lol their narrative is just boring and trite
        • deleted 2y ago
          [deleted]
  • jgrahamc 2y ago
    Details of how we use this are here: https://blog.cloudflare.com/harnessing-office-chaos/ https://blog.cloudflare.com/harnessing-office-chaos/
  • brookst 2y ago
    It's an odd world when a hypothetical terrorist is looking to reduce entropy.
    • thinkingtoilet 2y ago
      If your goal is to weaken encryption, reducing entropy could be a step along that path.
      • brookst 2y ago
        Sure, but it is amusing to imagine terrorist cells plotting to reduce entropy. There's a SF story to be had here: the global superintelligence uses bits of litter and fallen leaves and stuff to generate unbreakable encryption; the terrorists wage a global campaign to clean up litter, prune trees, get everything neat and orderly in order to hack the system...
        • thinkingtoilet 2y ago
          Agreed! Your comment did give me a chuckle.
        • rzzzt 2y ago
          Lava lamp factories are subverted so that the wax mixture flows in a straight line without any turbulence whatsoever. Dice land on a 3 each time they are rolled, anywhere in the world. Casinos disintegrate swiftly accompanied by a bright flash.
          • brookst 2y ago
            Playing card companies are infiltrated and produce decks with nothing but the four of hearts. Coins are melted down and recast with both sides tails. Chaos. No, wait, order.
    • deadbabe 2y ago
      It’s a form of eco-terrorism.
  • pbnjay 2y ago
    Given the redundancy of multiple locations, I think even cutting power it would be difficult to cause an extended outage.
  • jgrahamc 2y ago
    That sort of happened on "NCIS", Season 16, Episode 1: https://blog.cloudflare.com/statement-concerning-events-at-glowbeam-technologies/ https://blog.cloudflare.com/statement-concerning-events-at-g...
  • crazygringo 2y ago
    It just injects additional randomness into randomness that is already perfectly fine. It's basically for PR/show, but it is actually used. But without it, things work fine too.
    • thih9 2y ago
      If it silently crashed and started to output a static number, would this affect any systems negatively?
      • luuio 2y ago
        a random number (existing entropy) XOR with a static number (the crashed wall) is still a random number, me think
      • crazygringo 2y ago
        Surely not. If you're seeding a PRNG from multiple sources of entropy, you generally concatenate them. Or if you were limited in bytes you'd XOR them. This is why, in an app, you might seed with timestamp and process ID and /dev/urandom, in case any of them happen to be non-unique or unsupported.
      • ycombinatrix 2y ago
        Probably not, unless it is their only source of entropy.
      • jedberg 2y ago
        I asked this last time about the one in London, and was told that one of the checks is that the image has changed since the last run. Otherwise the data isn't used. Prevents not only technical issues but attacks like someone blocking the camera or putting a static photo in front of the camera.
        • thih9 2y ago
          Interesting - sounds like it would have some negative effect then. Thanks for sharing. Now I wonder about some periodic offsets. E.g. if the lights are off at night, or if the skies are overcast in winter, does it skew the results in some significant way. I seriously doubt that though.
  • eastdakota 2y ago
    Great thing about entropy is that adding more never hurts. This is one of many sources — both more conventional as well as unconventional — that we use. If it were to go offline, or somehow be corrupted, it wouldn’t hurt our ability to generate entropy across the Cloudflare network. What I love about this, the lava lamp wall in San Francisco, and the double pendulums in London, is that it takes something very abstract and makes it tangible for our team and our customers.
    • AdamN 2y ago
      These are my favorite types of marketing - what I'll just call 'part of the actual stack'. Would be great if there was a Berlin office so I could join!
    • jstasiak 2y ago
      > Great thing about entropy is that adding more never hurts. I used to think the same but here's a counter-example of a (hypothetical) attack based on a malicious entropy source being able to manipulate the hash/PRNG output: https://blog.cr.yp.to/20140205-entropy.html https://blog.cr.yp.to/20140205-entropy.html Now, it's not necessarily the most likely attack to materialize, as already pointed out downthread: https://news.ycombinator.com/item?id=43391377 https://news.ycombinator.com/item?id=43391377.
    • AStonesThrow 2y ago
      Ah, pendulums. They had some cool ones in the museums we used to visit. One was 2+ stories high and it would knock down dominoes to reckon the time of day as the Earth perturbed its motions. Another much smaller pendulum in the hands-on science exhibits, you scooped sand into it and then set it swinging freely across a square black surface. It would trace out amazing patterns as it spilled sand hourglass-style. So then some bully would rock up next to me and smack the pendulum, stop it from swinging, and spill a big blob on the formerly-geometric pattern. And they invariably said "just to see what would happen". AStonesThrow would have a small meltdown or become rather indigant. I suppose their empirical science is just as valid as kicking down sandcastles on the beach. And that's how I came to prefer single-player games...
  • gunian 2y ago
    man terrorists are so basic these days they should not be considered entropy more like extremely predictable load tests conducted by powers that be lmao
  • tptacek 2y ago
    It is 1000x for PR/show. There is no practical security benefit to stuff like this. There's also no meaningful risks to doing it. It just lets them write fun blog posts.
    • dadrian 2y ago
      If they actually integrate this into randomness on their TLS servers, the only risk is that the system for getting the entropy from the lamps and waves somehow screws up, fails to parse an HTTP request or something, and accidentally seeds the whole system with no entropy. Whereas doing literally nothing and just letting Linux boot correctly on metal would be perfectly secure.
      • tptacek 2y ago
        Right, but there's no way Cloudflare is making that kind of mistake. If it was a random person on HN talking about how they'd hooked up a bespoke hardware RNG to their TLS stack I'd write some tut-tutting thing about what could go wrong, but here the security of their system collapses down to the LRNG just like every else's.
        • blibble 2y ago
          yeah, it's not as if they've had terrible bugs like one that sprayed supposedly protected cleartext all over the internet, is it?
        • dadrian 2y ago
          Maybe, but they've taken something that was effectively risk-free and added risk for absolutely no reason.
    • swyx 2y ago
      i mean its also nice office decoration and also lets them rank on HN. many companies spend far more on employer branding with far less results.
      • 2y ago
  • ivirshup 2y ago
    I think there are backup systems. I sometimes walk by the Cloudflare SF offices on weekends and it looks like the lava lamps are off.
    • eastdakota 2y ago
      They should never all be off at the same time. We do cycle through each of them turning off for a period of the day. But, even if they were all off, there are lots of other sources of entropy we use, most of which are for more traditional if far less visually interesting.
  • theandrewbailey 2y ago
    Data centers are filled with loud fans that scream 24/7. I would think plugging in a microphone to servers and using that to seed entropy would be a more scalable and portable solution than cameras looking at lava lamps or water in the office miles away. There has to be some code that already does that.
  • coppsilgold 2y ago
    The image sensor noise will produce more bits of randomness than the chaotic display. There is also dedicated "TRNG" hardware which will measure random thermal noise. Some will even get fancy with quantum effects. Any source of randomness will do, you just feed it into a hash function and extract uniform randomness you can use in cryptography. For example, if you have an image sensor that takes an image (and does no post-processing) and you feed that image into SHA256 you get 256 bits which you can use for cryptography. As long as the image is never saved there is no practical way to recreate the input and in fact the input will contain more entropy (degrees of freedom) than the output, so no one would even want to try. Most of the degrees of freedom in the image would come from sensor noise and not the scene, so you don't even need to take off the cap from the camera. In practice, multiple sources are combined. The Linux kernel does this for /dev/[u]random though it doesn't use the camera. There is a potential risk with such combination: one of the inputs may come from a source which is able to interrogate all the other sources, it would then be able to adversarially choose its contribution to skew RNG results. This is a somewhat obscure and unlikely threat model.
  • calmbonsai 2y ago
    Yes. It's just PR. One can just reverse-bias some Zener diodes to get the same thing, but it is beautiful PR.
  • bob1029 2y ago
    Technically speaking, we can generate and store an unlimited amount of information/entropy for future use. It doesn't really go bad unless you disseminate the material to inappropriate parties. You could store terabytes of it in S3 buckets for an emergency.
• Mistletoe 2y ago
  I wish there was more about the camera that images them. Is it that little security camera in the pod you can see in the video? Does it even have enough resolution for the minute water effects across the room to matter/be seen? An interesting question I would have is do you get just as much randomness without the fluid at all?
  • Palomides 2y ago
    you'd get a similar amount of randomness with a lens cap over the camera; it's a neat gimmick, but still a gimmick
    • eastdakota 2y ago
      This is correct.
  • mackenly 2y ago
    The goal isn't to capture all of the randomness of the water but instead to capture enough noise to produce a seed. All of the changes in the room get averaged into a pixel. The water just makes the room change more.
• _vufv 2y ago
  Name suggestion for the wall: "Hydrobit generator"
  • kmoser 2y ago
    They should periodically change the name at random, and let the name contribute to the entropy. Of course, this won't work if everybody votes to name it, say, "Wavy McWaveface."
• orliesaurus 2y ago
  that's kinda interesting and kinda a nice PR stunt at the same time
• callumprentice 2y ago
  I have a real soft spot for Lisbon and Nazare - hoping to live there one day after I retire. Lots of similarities with the last city I lived in - San Francisco: Big red suspension bridge that spans the gateway to the ocean (same vendor), cable cars running on impossibly hilly streets, cosmopolitan, diverse LGBTQ+ friendly people, amazing food, nearby vineyards, blossoming tech scene....
  • mykowebhn 2y ago
    I lived in both San Francisco and Lisbon, and the only glaring similarities are the two bridges and the many hills. The Lisbon I love is much more charming, less money-oriented, less pretentious, and safer. The sad thing is that it's becoming much more like America. The more it tries to become like San Francisco, the sadder I become.
    • gusmally 2y ago
      >The sad thing is that it's becoming much more like America It cannot help that more and more Americans are moving there.
• xpe 2y ago
  Who wants to bet that Steve Thompson [1], creator of Prime Target on Apple TV, might incorporate lava-lamp based entropy into season two, if we're unlucky enough to witness its creation? I've been wondering how much worse the show could be, and sadly I can envision numerous plotlines involving bizarre sources of entropy and their subversion. [1] https://www.imdb.com/name/nm1839162 https://www.imdb.com/name/nm1839162
  • rafram 2y ago
    Huh?
    • xpe 2y ago
      Prime Target gets horrible reviews for botching cryptography in cringe-worthy ways, not to mention horrid acting. Normally I would avoid such a sarcastic comment about a video series, but given the Apple TV platform, it gets a lot of eyes and affects how many people get introduced to cryptography, so it is good to be aware of it.
• joduplessis 2y ago
  Not really related to this post specifically, but my opinion on Cloudflare is slowly shifting from "really like them" to "they're actually hostile to indie devs". Almost a month ago now my account was incorrectly charged for a pro plan I didn't have --- aside from the week+ long waits between replies from support, they refuse to refund me until the engineering problem is fixed. I'm still waiting for a reply to my follow-up mail from last week. Yea, $25 is not much - but as an indie dev this makes me HUGELY nervous if that amount had even 1 more zero after it. But hey, cool office guys.
  • jgrahamc 2y ago
    Email me details of this.
    • Etheryte 2y ago
      If your users need to resort to HN comments (or Twitter or Reddit or etc) to get a support issue fixed, something is definitely wrong with the pipeline. How many of your users out there have issues, but do not happen to see a random comment thread where someone from your team is participating.
      • eastdakota 2y ago
        Mistakes inevitably happen at scale. Sometimes they’re not caught by traditional channels. What I try and encourage is our leaders to take responsibility and fix them wherever they see them. Which is what John did above.
        • joduplessis 2y ago
          Which is appreciated - emailed!
          • jgrahamc 2y ago
            I already passed it on to the team. Sorry you've had this difficulty.
• kylecazar 2y ago
  It's a cool concept, but I just came to compliment the incredible view from that Lisbon office!
  • brabel 2y ago
    The whole city of Lisbon is amazing. It's very underrated but worth a visit for sure.
    • lopis 2y ago
      How is Lisbon underrated? It's completely, chocked full of tourists and wealthy expats, and practically devoid of permanent residents because no one else can afford to live there any more.
      • regularjack 2y ago
        I agree with the sentiment, but it's very far from being devoid of permanent residents. That's maybe true for a couple of downtown neighborhoods, but definitely not true for the city as a whole.
        • andrewl-hn 2y ago
          Plus, many "wealthy expats" don't live in the city, preferring neighboring suburbs like Cascais or Sintra. The Lisbon proper is very dense, lacks greenery, and chokes in traffic, an interesting place to visit, but not to live in. Meanwhile, every single real estate developer / agent sets their prices incredibly high hoping to sell or rent out the property to those mythical "wealthy expats". I saw a stat somewhere that less than 0.2% of all real estate transactions in the country each year involve foreigners, and yet everyone blames them for high real estate prices.
          • mykowebhn 2y ago
            > lacks greenery Seriously? How about: - Avenida da Liberdade - The Botanical Garden in Principe Real - Parque Eduardo - The beautiful gardens around the Gulbenkian Museum - Jardim da Estrela - Tapada das Necessidades - Jardim do Principe Real and one of the prettiest urban neighborhood green spaces I've ever been to: Jardim Fialho de Almeida Edit: formatting
• aerostable_slug 2y ago
  Long Live Lavarand
• acedTrex 2y ago
  I have heard Cloudflare is currently in the throes of an offshoring to India debacle. How the mighty have fallen.
  • mvdtnz 2y ago
    Source?
    • acedTrex 2y ago
      Just murmurs through the grapevine from people internal over there. I trust what I'm hearing but its nothing official or concrete.
  • andrewl-hn 2y ago
    They keep hiring actively in Portugal, too, mostly because developer salaries there are much lower than in North, West, or East Europe.
    • rcarmo 2y ago
      Not that much, really.
    • eastdakota 2y ago
      Actually, it’s because we believe innovation is easier the further you are from HQ. And Portugal is wonderful. I’m spending about 4 months of the year there with the team who hail from around the world.
      • FirmwareBurner 2y ago
        Would like to work for you. Any chance to seek innovation even further away, like in Austria? That's lovely too. :)
        • eastdakota 2y ago
          We’re in Munich, pretty close, and Lisbon, which is lovely. Unlikely Austria any time soon.
• nthingtohide 2y ago
  Reminds me of this Turkish artist Refik Anadol https://refikanadol.com/works/bosphorus/ https://refikanadol.com/works/bosphorus/ Bosphorus is a data sculpture inspired by high frequency radar data collections of Marmara Sea provided by Turkish State Meteorological Service in every 30 minutes intervals. The data collection of 30 days long sea surface activity transformed into a poetic experience and visualized on a 12 meters by 3 meters long LED media wall. The art work exhibited at Pilevneli Gallery on 11th of December, 2018 – 27th of January, 2019 in Istanbul, Turkey.
  • emot 2y ago
    that's a cool experience. it's really mesmerizing and calming watch one or two of these machines, because of the way the waves move, similar to ocean waves. there's this video that shows it at the end: https://youtu.be/mYLvRaMmfho https://youtu.be/mYLvRaMmfho
  • 4gotunameagain 2y ago
    It's amazing, but data sculpture ? Really ? Why do people feel the need to appeal to authority for the things they made. This has nothing to do with a sculpture. But it doesn't take anything away from its value.
    • lnrd 2y ago
      > sculpture, an artistic form in which hard or plastic materials are worked into three-dimensional art objects. - Encyclopaedia Britannica
• ColinDabritz 2y ago
  Adding a single element from each other office would be a neat way to tie them together. For example, the new wave wall with a single lavalamp, hanging rainbow, and dual pendulum.
  • jgrahamc 2y ago
    There's custom wallpaper in the office which features elements of the other offices, but this is a fun idea. We do have a double pendulum and lava lamp elsewhere in the office, just not on the wave wall.
• thiht 2y ago
  > It’s exciting to see waves in Portugal now playing a role in keeping the Internet secure, especially given Portugal’s deep maritime history. I hope we should not read too much in the hanging of rainbows in Austin, Texas
• bohdanb 2y ago
  How does Cloudflare ensure that the wave machines in their Lisbon office generate truly unpredictable entropy for securing the Internet?
  • tH4812378 2y ago
    [flagged]
• h1fra 2y ago
  Not an expert in the field, I would assume one box is not that random since we can easily simulate fluid but the wall in its entirety is secure, right?
  • pornel 2y ago
    The fluid can't be "easily" simulated. It's a chaotic system (turbulent flow is chaotic). Even tiniest differences between the real and simulated state will add up and amplify over time. Fluid simulation is a notoriously hard problem. We don't have a solution to Navier-Stokes equations. Practical implementations have limited resolution in time and space, and plenty of simplifying assumptions.
• srinathkrishna 2y ago
  I've been to Cloudflare's SF office many times for meetups and even for an interview and their lava lamp wall is one of the coolest stuff I've seen in an office. So cool to see that they've built something similar in their Portugal office.
• lysace 2y ago
  Neat and fun PR stunt. Tech needs more of these! SGI did this almost three decades ago: https://web.archive.org/web/19971210213248/http://lavarand.sgi.com/ https://web.archive.org/web/19971210213248/http://lavarand.s... ...harnessing the power of Lava Lite® lamps to generate truly random numbers since 1996. According to https://www.lavarand.org/news/lavadiff.html https://www.lavarand.org/news/lavadiff.html: Seed production rate was about 8000 bits of seed per second on a 200 MHz SGI O2 under IRIX 6.5. The patent has since expired: https://patents.google.com/patent/US5732138A https://patents.google.com/patent/US5732138A (And Cloudflare re-implemented it, seemingly starting just after the patent expired in 2016.)
  • eastdakota 2y ago
    Had no idea there was a patent. Even if we had, think we’d have risked it.
    • lysace 2y ago
      Perhaps someone in the org knew. CEOs don't get all of the details ;). Nevertheless, it's a great tradition to carry forward and I'm happy you guys are doing it.
• mdhb 2y ago
  Instead of this PR nonsense I feel the cloudflare news we should be discussing today is the fact that they just admitted to logging the usernames and passwords of users for websites which use their service. https://benjojo.co.uk/u/benjojo/h/cR4dJWj3KZltPv3rqX https://benjojo.co.uk/u/benjojo/h/cR4dJWj3KZltPv3rqX
  • jsnell 2y ago
    The link does not provide any supporting evidence for your claim about them logging passwords. Nor does the Cloudflare blog post that's the source. It seems pretty good that we aren't discussing something that you've just made up.
    • mdhb 2y ago
      This is literally how TLS works.
      • jsnell 2y ago
        The claim you made was "they just admitted to logging the usernames and passwords". The fact is that they did not admit to that. If you think otherwise, please quote the actual admission. I don't know how you think "this is how TLS works" is relevant here. Just e.g. terminating a TLS connection certainly doesn't mean that you're going to be logging all the unencrypted data. And you claim, again, was about logging.
        • deleted 2y ago
          [deleted]
        • mdhb 2y ago
          How exactly do you suppose they are able to talk in great detail and specificity what passwords people were using in this magical world where they also aren’t inspecting everyone’s passwords as they pass through their servers?
          • 2y ago
• nektro 2y ago
  > set to the tune of an AI-generated song. way to completely ruin an otherwise lovely article
• outer_web 2y ago
  Came expecting wrestlemania, left wishing for a wrestlemania entropy source.
• npodbielski 2y ago
  OK looks cool but otherwise can be achieved much easier with few zener diodes and amplifiers. But I know... Does not looks that cool.
• nis0s 2y ago
  I wonder how random it is truly given that the environment something resides in shapes and changes it. For instance, imagine if there is some sort of environmental variable (temperature, humidity, light, or more) which produces an indiscernible, yet impactful, change in the behavior of the wall over time. In that way, there may not be true randomness, but a complex pattern which can be studied and acquired over time.
  • eastdakota 2y ago
    If you figure out how to model this fluid dynamics accurately over any reasonable period of time, call me. Lots and lots of more valuable things you could do with that, e.g., accurately predicting the weather.
    • nis0s 2y ago
      I was speaking more to something which happens with some consistency even in a random system. For instance, waves hitting the same piece of rocks over and over till the rocks take on a certain shape.
• bomewish 2y ago
  What’s the second level analysis here? We know it’s not really necessary or helpful for the ostensible reason (far cheaper/more reliable ways of capturing entropy) — so we conclude it’s a marketing gimmick. Yet for the gimmick to work they have to pretend it’s useful. They’re not fooling themselves or anyone else, though. So what’s really going on? Is it: - it IS somehow a good return on investment?? - marketing had a budget and didn’t know how else to spend it, and no one wanted to be the unpleasant person and say how it’s all a silly waste of money? - they are making a tonne of money and no one really cares, so we’ll just spend it on fun cool stuff as long as there’s a plausibleish story to go with it? - fits with a broader global company branding concept that leadership seems to like, so there’s just the momentum to keep it going (and see points above)? I can’t figure it out. I agree it’s cool! Just the make believe puzzles me a little. I’ve not worked at a big corp like this and just have to understand what’s actually happening.
  • babelfish 2y ago
    It is a gimmick, but targeted at recruiting, not sales. Cloudflare is known to be “blog driven engineering”
  • eastdakota 2y ago
    If it were merely marketing spend for customer acquisition, I bet the ROI on the lava lamp wall in SF has been 100,000x. This isn’t hard to figure out.
  • everybodyknows 2y ago
    Not #3: Trailing 12 month P/E is negative. https://www.cnbc.com/quotes/NET?qsearchterm=net https://www.cnbc.com/quotes/NET?qsearchterm=net
• bravura 2y ago
  Subtext: In case a) you want to move to Europe, b) cannot tolerate weather that differs from California, c) like Golden Gate bridges, are you aware that Cloudflare has a Lisbon office? Source: I live in Portugal and notice all the discussion about it from my Bay Area friends who want to expatriate. To here, specifically.
  • collingreen 2y ago
    Visited Lisbon once, live in SF. Can confirm I loved it and want to move there.
  • euroderf 2y ago
    Is there much I.T. in Porto ? I've heard it's a great town.
  • gusmally 2y ago
    Expatriation (immigration?) to Portugal from America furthers Portuguese wealth inequality and housing crisis. Lisbon is already basically a playground for foreigners and if you do decide to move there please try to assimilate by learning the language, etc. Source: Portuguese/US citizen. Lived there for a year with a "good" Portuguese salary of 2000 euros a month; much different lifestyle than what the typical US tech worker is accustomed to. And not doable long term unless you plan to retire in Portugal and not do much travelling.
  • mykowebhn 2y ago
    These type of comments make me cringe. Nowhere does the OP describe what Portugal is like in itself. It's very often compared to something else, like California. Portugal is Portugal.
    • bravura 2y ago
      My comment was about "why would cloudflare make a fluffy blog post like this?" And I believe its to attract people to Lisbon. I never said that I support this oversimplification of Portugal. Which is why I live in a small town, socialize with Portuguese, and study Portuguese.
• holografix 2y ago
  What a beautiful office! Pity salaries in Lisbon are so low, otherwise I’d be applying for sure.
• sgheemen 2y ago
  [dead]
• SomebodySmart 2y ago
  One day at a beach I watched a wave crash in and saw the sand washing up in a blur. I scooped up some sand and examined it closely and it dawned on me: Close-up bitmaps of sand are a perfect source of randomness. It's easy to shuffle the sand and take another picture and by doing this repeatedly you can capture enormous random key for secure encryption. One machine does the work of composing the message and saved it onto a thumb drive. You take that out and put it into a machine that only encrypts. Then you take the cryptext onto another medium and load it into a machine that is connected to the internet and transmit. Hackers cannot reach the machine that does the work.