13 ms·

Google could have killed Facebook with the flick of a switch

193 points by msh 6y ago

arthurcolle 6y ago
That so many smart and talented people choose to work on AdTech or systems that support AdTech is such a shame. That being said, I love these little anecdotes, they are super fun to read.
- dvfjsdhgfv 6y ago
  There are so many aspects of adtech that are ethically neutral. Just as advertising itself - there are even ads created by great creators that are works of modern art. It's only when it's abused adtech becomes evil.
  - arthurcolle 6y ago
    Wow, I bet I'll miss out on so much important culture by not getting visual pollution all over the place unless its regulated out of existence. No thanks, I'll continue to block it everywhere that I can, and even going further and inspecting elements to get rid of obnoxious paywalls as well! Wow, am I a hypocrite? Maybe develop real, sustainable revenue models that aren't dependent on clickbait and being a lobbyist for corporations in the form of selling eyeballs to the highest bidder. Just yuck. I'll be inclined to buy into $WHATEVER if you just let me use it for a bit, if I find it of my own means. I promise you I'm not going to buy a Swatch because of your fucking banner ad. Seriously, I promise. If I just bought a book I promise you I'm not going to buy it again if I see it appear in my side banner. 100%. Or maybe I'll just grow less dependent on external services every time I have to consider another subscription... wow! We'll see!
  - croes 6y ago
    Have you checked the art vs bullshit ratio?
    - arthurcolle 6y ago
      That probably costs extra
  - beefield 6y ago
    It may be "many" but it does not mean that the defining nature of advertising is ethically neutral. I think I just quote Banksy/Tejaratchi: "People are taking the piss out of you everyday. They butt into your life, take a cheap shot at you and then disappear. They leer at you from tall buildings and make you feel small. They make flippant comments from buses that imply you're not sexy enough and that all the fun is happening somewhere else. They are on TV making your girlfriend feel inadequate. They have access to the most sophisticated technology the world has ever seen and they bully you with it. They are The Advertisers and they are laughing at you. You, however, are forbidden to touch them. Trademarks, intellectual property rights and copyright law mean advertisers can say what they like wherever they like with total impunity. Fuck that. Any advert in a public space that gives you no choice whether you see it or not is yours. It's yours to take, re-arrange and re-use. You can do whatever you like with it. Asking for permission is like asking to keep a rock someone just threw at your head. You owe the companies nothing. Less than nothing, you especially don't owe them any courtesy. They owe you. They have re-arranged the world to put themselves in front of you. They never asked for your permission, don't even start asking for theirs." Edit: Credits also to Tejaratchi
    - arthurcolle 6y ago
      This is such a great quote. I've checked out a couple books on Banksy but haven't ever run into it before. Thanks so much for sharing!
    - Teever 6y ago
      One time I saw a fascinating article about how scientists were working to grow skin complete with natural looking hair from stem cells for burn victims and such. I thought the article was really neat and I was in a hurry so I emailed a the link to myself. The next time I logged into my gmail to read it I happened to be at a machine without adblock and I saw that google was serving me ads for garbage treatments for male pattern baldness. Fuck you Google and your shitty advertising. There's nothing wrong with going bald and I don't need you to try and make me feel insecure about my hairline when I'm trying to read about scientists helping burn victims.
      - 6y ago
  - arthurcolle 6y ago
    > so many Please name 5
    - dvfjsdhgfv 6y ago
      OK. Let's separate the ad industry from ad technology for now. So as for the latter, have a look at the Revive ad server [0]. It allows you to deliver ads with intricate rules as to when they should be served, on which websites and so on. It allows you to be less dependent on Google/FB duopoly. You can host the ads either on your own websites or contact the owners of other popular websites and arrange an ad deal that doesn't involve user tracking. Yes, this is the opposite of what modern ad networks are doing. In any case, advertising without user tracking is, in my opinion, one of the most ethically neutral forms of making your product/service known, in contrast to morally dubious methods such as advertising with user tracking, product placement, advertorials, advertising disguised as articles/videos etc. [0] https://www.revive-adserver.com/features/ https://www.revive-adserver.com/features/
  - llarsson 6y ago
    There is a difference between the medium (ads) and the targeting and distribution technology (adtech). A great ad evokes some kind of feeling, can live with you for a long time, and even bring back memories that you cherish. Yes, I agree with you, it can be a form of art in that regard. Adtech, however, is an intentionally invasive technology that exploits any means necessary and technologically feasible to more narrowly target potential consumers. You can have ads without adtech. But you can't have (commercially successful) adtech without invasion of privacy.
    - dvfjsdhgfv 6y ago
      > But you can't have (commercially successful) adtech without invasion of privacy. I think we (as a society, not just tech people) specifically need to take care of this point. Removing tracking will make it less effective, so what. We've had advertising without tracking since 19th century.
- deleted 6y ago
  [deleted]
- wwww4all 6y ago
  There are more positives in Adtech than negatives. Adtech allows people and businesses to find products and services they want and need. Gives consumers choice in discovery and businesses reach and targeting customers in need. At fundamental level, Adtech is there to help people. That and the many business opportunities are why so many talented people work in Adtech.
  - saiojd 6y ago
    Keep telling yourself that if it helps. ;) While at some level ads probably do help consumers (indirectly), my impression so far is that at a fundamental level they are about companies competing for money, in a way which wastes everybody's time. Since your point of view is so vastly different, I'm curious in which ways you feel like they give consumers choice?
  - bigphishy 6y ago
    "At fundamental level, Adtech is there to help people." Haha. Adtech is not there to "help people". Back in the day, prior to television and radio, ads were there to provide information about a product, but those days are long since past. Ads now-a-days appeal to emotional and irrational wants/desires, this is common knowledge. Ads attempt to generate needless sales by marketing to irrational and emotional desires. Adtech on the other hand, is there to drive irrational and unneeded consumerism and consumption just as well, it just happens to have some additional evil sprinkled on top. For example, adtech violates human privacy, adtech marginalizes the open web, and turns it into a creepy surveillance nightmare. To say that "At fundamental level, Adtech is there to help people." is delusional and unintentionally hilarious. I am going to swing in the dark and say the GP must be an employee of facebook
  - hesk 6y ago
    If only Google had a search engine to help people and businesses find products and services they want and need.
  - edmundsauto 6y ago
    I share many concerns about tracking people, but let's acknowledge that adtech democratizes access to services that most people (worldwide) would not be able to afford. "Free" is the only price that works for the 36% of the world's population that is in poverty.
elcomet 6y ago
Very interesting story! The title (Google being able to kill facebook) seems a bit exagerated though. If this app was so important (25 to 50% of all facebook revenues), and so important to customers (most powerful editor to manage ads), I'm sure if Google had disabled websql, they would have forked some older chrome / electron with websql support and bundle it with the app.
- arthurcolle 6y ago
  that would have made builds massive, unless you're referring to just the desktop experience...
  - elcomet 6y ago
    I'm saying they would have find a way to deliver the app to customers, whether in desktop or web form. And customers would have downloaded the app if needed, as it seems to be the only powerful way to manage ads.
- aequitas 6y ago
  Indeed, people will always find a way, especially if money is involved. And running outdated browsers for legacy support, while being a huge security risk, is not uncommon for a lot of use cases, especially back then.
- mcv 6y ago
  A hit of 25-50% of revenue would certainly have hurt, but probably not killed Facebook. That people will find a workaround is absolutely true. I once worked on a system to order mobile phone subscription from a website that only worked on IE6. So we had a website that accepted orders from customers, created jobs that would be sent to one of 10 VMs that ran IE6, where a watir script would steer IE6 to enter the order into the old website. It's the most horrible hack I ever worked on, but it worked.
  - Pelic4n 6y ago
    Absolute cringe, thanks you for this.
supermatt 6y ago
TLDR: Facebook built the app responsible for their entire ad platform on a chrome-only tech (WebSQL), and when it was deprecated, instead of dropping in a widely-used polyfill, they decided to do a major 3yr rewrite risking 25% of their revenue in the process. I guess this is the stupid shit that being "too big to fail" permits.
- asplake 6y ago
  I didn’t know that such a polyfill existed but even so, that was pretty much my reaction. Facebook would be quite capable of developing one, and potentially a route there to something more service-based once the panic is dealt with.
- kps 6y ago
  Another option would have been to use a chromium build with WebSQL still in place.
  - PeterisP 6y ago
    They would have to get all their advertiser customers to use that custom chromium build, which would be possible but would not be good for their ad sales.
    - protomyth 6y ago
      Given how long some of us had to hang on to Internet Explorer 6, I'm pretty sure the advertisers would have required their IT staff's to comply and install the "Facebook Browser".
      - ADefenestrator 6y ago
        Wouldn't have necessarily had to be nearly as bad as IE6 for security, either. Just lock the special browser to facebook.com only, or even just the ads API, and there's not a lot of room for exploit.
- joe_the_user 6y ago
Osiris 6y ago
If the product was responsible for 25% of the company revenue, why was only half of one engineer assigned to it? And after the vulnerability was discovered, they still only had 5 people assigned to it while it increased in importance to 50% of revenue?
- simonh 6y ago
  I've seen this sort of thing happen many times. What usually happens is the original team that built it leave the company or move on to other teams, and the one engineer left actually has a different job as well now but is stuck supporting it part time as a favour. It's not that you couldn't assign other engineers to help support it, but they wouldn't have any idea how to do so because there's probably no documentation and only that half an engineer around to show them how it works.
  - disgruntledphd2 6y ago
    It was because Facebook practiced open allocation for engineers at the time. Essentially, you had a choice of teams and the manager would need to convince the engineer to work on the project. At the time, ads was not a high status part of FB engineering, and as such, it was very difficult to get people to work on ads products.
    - ForHackernews 6y ago
      It's funny that people want to work at Facebook or Google but don't want to work on ads. In other companies, everyone wants to work near the profit center and avoid cost centers.
      - rvba 6y ago
        Because those projects are hard. It is much easier to make a greenfield project that will get 70% done and cancelled before anyone even asks to fix bugs.
        6y ago
spankalee 6y ago
That is not how Chrome removes features. It's based on usage. WebSQL is still supported today, in fact: https://caniuse.com/sql-storage https://caniuse.com/sql-storage
closetnerd 6y ago
Quick note: it's an unfortunate fact of life that some engineers make a career of exaggerating things.
- disgruntledphd2 6y ago
  This is not that, though. I heard this story from people involved at the time, and it's not exaggerated in any way, shape or form.
  - hoten 6y ago
    But isn't it? I can believe people inside Facebook really thought it was a critical problem, but I don't see how it could have become any more than an inconvenience. - There's a websql polyfill. I don't have experience with it, but it looks like a drop in replacement. This is such a strong counterpoint, the rest almost doesn't matter. - Features tend to be disabled in Chrome with some advanced warning. Not overnight. - websql is still enabled in chrome today - if the api were disabled, ad customers could be directed to flip the feature switch in Chrome. If money is on the line, seems like an easy enough task.
    - yyyk 6y ago
      - Safari supported websql until safari 13 (september 2019), long after this issue stopped being relevant. How many people in an organization need to use PE? Probably only a few in marketing. Surely these few people could have just bought a macbook. A few macbooks must be a very small expense compared to facebook advertising. - Or IT could have arranged a VM with an old version of Chrome.
      - disgruntledphd2 6y ago
        So this was an application used by extremely non-technical customers (i.e. marketers) spending money on behalf of many, many brands. Relying on their IT to do this just wouldn't have worked and was definitely not scalable. For context, if Power Editor disappeared along with its revenue, Facebook would have needed to report this to the market.
acd10j 6y ago
Facebook's real moat is it's massive user base not it's ad revenue, even losing 25% of it's revenue would not have effected or killed Facebook, Only building better viral social network could have killed it. Ad revenue is just a side effect of this massive moat. Just like Google can survive a temporary year or 2 of 25% drop in revenue if their search dominance continues or increases but not opposite.
CodesInChaos 6y ago
Couldn't they have compiled sqlite to javascript using emscripten as a websql replacement? sql.js seems to have been started in 2012, and even if it might not have been production ready at the time, I'd expect facebook to be able to pull off such a project by themselves.
exolymph 6y ago
If Google could have killed Facebook, it would have.
29athrowaway 6y ago
Google+ was very poorly designed.
- mcv 6y ago
  Google+ was initially very well designed, and then Google proceeded to run it into the ground. Google+ when it was about a year old, was still the best social network ever. But then Google tried to force Youtube integration, killed off Events and Hangouts, not to mention the questionable "no unusual name" policy.
jorams 6y ago
I've heard, and keep hearing, so many stories about frustrations with Facebook's tools for advertisers that the situation described doesn't surprise me at all. Somehow this massive organization with >50 000 employees and billions of users can't seem to produce quality software that allows people to pay them for access to those users. Small tasks turn into multi-day endeavors just because the tools keep crashing over and over again, losing all state. Managed advertising accounts just vanish from interfaces. Random buttons turn into "try again" roulette. It's a good thing their ads are (apparently) effective.
ai_ja_nai 6y ago
Ehehe nice story. I remember when we were to be shut down by AWS because the bill was 2 months due and the administrative contact was a inbox nobody looked at. We discovered it by chance, doing a nightly maintenance: services started to disappear one after another (including the Redis that powered the login, we weren't able to login into the product for half an hour!). We immediately phoned AWS support and asked, no, begged them to turn them up again. They did and we paid everything the morning after. Hadn't we been in front of the PCs doing the maintenance and witnessed as it was happening, we might have woken up the day after without an online bank (yes) because the production environment would have ceased to exist...
mutatio 6y ago
It raises interesting questions of whether Google (advertising company) should be the guardian of the web's primary browser; strategically it could deprecate features direct competitors depend on (let's assume FB is now also a ad company). The fact that web's primary ad platform also owns the web's primary browser sure is "interesting". The inverse of that is that it might not have incentive to deprecate features - which Google the company / partners depend on or might not be able to deprecate due to legal consequences of doing so (shutting down 25% of FB's revenue might raise eyebrows in the gov).
- jasonvorhe 6y ago
  iirc it was Mozilla who put the nail in the coffin of WebSQL and if you read this post, it actually mentions that Chrome was the last browser to even still support the feature after it had been abandoned by Firefox and Opera already. What does this have to do with Google owning Chrome? Of course, the question itself is valid but I don't see how it relates to the topic.
baybal2 6y ago
Google actually had a successful social network project, Orkut, they just continuously marginalised it for it to be too NIH for US headquarters.
- chaz6 6y ago
  As far as I can tell, Orkut was the original incarnation of a modern social network, but they failed to capitalize on it.
  - lupire 6y ago
    Orkut was a clone of The social network created by the namesake's previous company. It was launched about the same time as Facebook and after Friendster.
- daniellarusso 6y ago
  Wasn’t it very popular in Brazil for some time?
  - nojokes 6y ago
    Yes it was. Source - old Orkut user but not from Brazil.
    - hrishi 6y ago
      I'm guessing India - it's where I grew up and it was the network of choice for teens until we moved to Facebook
- bemmu 6y ago
  Also tried to band everyone (MySpace among them) together with OpenSocial, but too late.
- GuB-42 6y ago
  I don't know how they are doing it but Google manage to fail at anything social. By that I mean communication platforms, social networks and instant messengers. They have promising starts but they fuck it up later. Their only successes I'd say are GMail and YouTube. I'd say for GMail, it is just an email client they started out excellent, thanks in part to its huge storage space and good search features. And they didn't break it enough to make it unusable. As for YouTube, it is great for hosting video, but its social features are terrible. There are communities tied to popular channels, but it stays in a producer to consumer model, with no interaction between community members. The comment section is made in such a way that dialogue is almost impossible. Maybe they should hire a few people who understand human beings, because right now, it looks like the company is run by robots.
yholio 6y ago
Today it's called Business Manager and it remains a user hostile pile of JavaScript with dubious design and riddled with bugs. The only reason anybody puts up with it is because there's no other way to access powerful targeting tools, and we are forced to depend on Facebook. More to the point of the original story, no, Google couldn't have killed Facebook by flipping that switch. The Power Editor users would have done whatever it takes to access the interface, installed an insecure older browser, ran commands in the console, build a relay computer in their back yard, and so on. They will do whatever it takes because they are not choosing that platform for its features, they are professionals doing a job, and that job is reaching the user base of Facebook with their ads. If they can't do that they will lose their jobs and clients.
- cm2012 6y ago
  I find business manager has a more sensible design than AdWords, having used both a lot. TikTok is funny because they basically copied the FB ads dashboard pixel by pixel.
  - dillondoyle 6y ago
    I agree but only the actual /adsmanager/manage section and the subsequent tools (events/audiences). Basically power editor 2.0. But it seems less powerful than the old one for some reason might just be in my head though. It has a pretty low ad limit per ad group I think 100. Though some of the 'combo' ads where you can put 5 headlines, 5 copy etc help up that number. BUT I do not like the business manager which houses all the pages, users, audience assets. I find the 'line of business' really confusing and hard to see how they want it organized. Like what's the model supposed to be? Agency line of business -> client subs. But then you need a specific line of business for certain offline attribution etc so having it agency parent doesn't make sense to me?!
    - etempleton 6y ago
      I think it feels less powerful because it is a buggy mess and takes forever to use. I loath setting up Facebook ads now because I know it is going to take me forever to do it properly. I pine for the days of PowerEditor. It had a confusing interface, but like Google Ads, once you had learned it you could get what you wanted to get done quickly. I don’t mind the interface of Business Manager or the Ads Manager platform at all, but I find the ads platform to be riddled with weird bugs. It has perhaps improved slightly in the past year or so, but only slightly. They also constantly change how it works, so you can’t just copy the campaign or ad set of an ad and build out new ads from a year ago, it will almost always have to be built out from scratch because something changed, but they don’t tell you that before you copy the campaign. Why even let me copy it if it isn’t going to work?
neodymiumphish 6y ago
> This became a closely held secret in Facebook Ads leadership. We didn’t want to take any chance that word our this vulnerability could get back to Google. Considering the tracking Google does with Chrome, I suspect they recognized the traffic still occurring over WebSQL and left it active for exactly this reason. It's highly unlikely that nobody at Google was able to recognize this. Alternatively, maybe Google didn't know about Facebook's reliance on it, but was aware of other services/corporations that still relied on it for some time, and waited until they had an alternative.
- esprehn 6y ago
  At the time this story takes place Chrome didn't have a way to collect web feature usage metrics broken down by site. It was aggregated across all sites to protect user privacy. That came years later with Rappor: https://www.chromium.org/developers/design-documents/rappor https://www.chromium.org/developers/design-documents/rappor WebSQL usage was too high at the time to remove though, so while the story is fun to read Chrome never would have actually turned it off.
  - shaneos 6y ago
    [author] Hard to say. If Google leadership had heard of this vulnerability, right as they were launching Google+, there would at least have been a serious conversation about it. They spent $Bs to build Google+, changed their whole company structure to support it, and were losing hundreds of people to FB every month. I doubt bending one little rule about removing a deprecated technology would have bothered them too much.
Hnshadowbansvd 6y ago
But.. They end up to be same censuring machine as twitter and Facebook
deleted 6y ago
[deleted]
anonygler 6y ago
Today's episode in "Read Your Deprecation Warnings"
fbelzile 6y ago
This is a good story, but Google can still literally kill any business that relies on the Play Store or Chrome Store with a flick of a switch today (like mine). No human even has to get involved. At any moment, some code running on a Google server can decide to flip a bit and disable access to an app or extension in their stores.
- kokx 6y ago
  That's not entirely true. They can kill any small and medium business easily. Taking out larger competitors in such a way will likely invoke lots of unwanted antitrust attention. Over the entire world. But they can easily kill any small competitor.
wwww4all 6y ago
I laugh whenever the tech futurists talk about AI, machine learning, automation, singularity coming in the near future. Most of the important software, that run Trillions of dollars in online economy, have dependencies in ancient tech. No one wants to work in that jumble, and updating has failed in one way or another. Google ACH and ask why it’s takes several days to transfer money from bank to bank.
- _-___________-_ 6y ago
  > Google ACH and ask why it’s takes several days to transfer money from bank to bank. Given that there’s approximately one country left in the world where that’s the case, I suspect the root cause is not a technical issue.
esprehn 6y ago
This story is a fun read, but the "Chrome could ruin us at any moment" angle is more story telling hyperbole than reality. At that time on Chrome we were just figuring out a process to remove features and were pretty conservative. We debated a lot about the right way to remove things without causing damage to the web ecosystem. In an early attempt we had failed trying to remove a legacy feature and needed to revert it. It had broken a big company who had then contacted Google directly about the business impact and it taught us a lesson about being careful. IIRC WebSQL had very high usage and we didn't seriously considering removing it. It's also pretty self contained so the cost on the engine isn't that high. Chrome still supports WebSQL today because the usage remains quite high at 1%: https://chromestatus.com/metrics/feature/popularity#V8SQLTransaction_ExecuteSql_Method https://chromestatus.com/metrics/feature/popularity#V8SQLTra...
- grecy 6y ago
  .. and even if Chrome had removed WebSQL back then, couldn't that have just kept using whatever version of Chrom e they were already using?
  - nosefrog 6y ago
    My understanding is that clients would have to use an outdated version of Chrome, which is probably an untenable ask.
  - esprehn 6y ago
    I suspect a lot of FB customers are using a browser in a business setting where IT manages the Chrome updates. FB could have used a polyfill though, or used Safari, or shipped a desktop wapper around an older Chrome temporarily. Electron was brand new at the time so it probably wasn't a great option, but FB certainly had a lot of emergency options.
    - shaneos 6y ago
      [author] There were some contingency plans, but we had no time to get them ready in advance. If Google had actually pulled the plug, we wouldn't been down for everyone for many weeks, and given that the vast majority of our large customers are non-tech-savvy (low paid employees at ad agencies), a large percentage would have remained with a broken experience forever. They'd have just given up and moved their money to Google.
waterside81 6y ago
Anybody have insight into why WebSQL was chosen in the first place? I don't think I've even heard of it prior to this article.
- shaneos 6y ago
  [author] Power Editor was originally written by a brilliant engineer named Vladimir Kolesnikov (yes, all by one engineer), and he was solving a very specific problem of letting customers scale their ad spend. So he chose a technology that let them download all of their ad account locally to disk, do all their work, then upload it again. WebSQL was flexible, fast and easy to work with. It's a real shame it was deprecated to be honest
simon_weber 6y ago
Interestingly, Google later wrote their own relational db in javascript called lovefield, which was motivated primarily by the deprecation of WebSQL: https://github.com/google/lovefield/blob/master/docs/dd/00_intro.md#01-motivation https://github.com/google/lovefield/blob/master/docs/dd/00_i...
utopcell 6y ago
extreme urgency.. facebook would shut down.. new ads features blocked.. urgent creation of a team to replace product.. 3 years later it was ready (!) Best example of leading by FUD if I ever heard one.
- _carbyau_ 6y ago
  I'd call it "managing upwards". :-P
RachelF 6y ago
In the early days of FB, they relied on Google Docs and Gmail. More importantly, they used pre-built social networks by letting you import your GMail contacts.
whalesalad 6y ago
Could have killed you with the flick of a switch - but you took three years and a team of 5 people to perform the rewrite?
- shaneos 6y ago
  [author] Unfortunately yes. It was not the only thing happening at the time. FB had just gone IPO and this put a huge amount of pressure to grow revenue. When I joined the Ads org in 2012, it was about 100 people I think, with just a few products. When I left in 2017 it was well over 1000, and my job had changed to managing relationships with 55 other teams, all of whom depended on my teams product to ship their individual features (think Video ads, Carousel ads, hundreds of others). So, while doing the rewrite, we had to transition from being a product org, where a few engineers built all features for a bunch of back end teams, to an infra+product team, where we built all sorts of support for all the new teams that seemed to spring up weekly. If we had done only the technical rewrite, and not enabled all these other teams to ship their products on the UI surface that our customers actually used, FB's revenue would not have gone from $4B when I started to around $30B when I left. These things are never that simple. I wish....
_carbyau_ 6y ago
To kill FB I don't understand why Google doesn't have a full time team of people working to extend/fork Signal to add a doomscrolling "blog posts" feature with required group/circles/whatever-bullshit. This isn't about Google capturing the FB space. Simply denying it to FB. Honestly: - no ads. (Seriously Google, no ads. But you know they can't which is why this is a pipe dream....) - could sort your own timeline by timestamp, number of comments, whatever - can't be shutdown due to content. Bonus points for: - Events and adding Events into your local phone calendar.(Please don't give me another calendar to cross reference.) Google twist the knife effort: - Free public facing web hosting for those who actually want their "Wall" public. IE companies, businesses, "personalities" etc. They did this for Android to compete with Apple. They can do it with Signal to compete with FB. If nothing else, could they "buy" the features from the Signal team in a similar fashion to Blender? This'd be my preference so that Signal maintains independance at least somewhat.(Money always talks I know.)

193 points by msh 6y ago

That so many smart and talented people choose to work on AdTech or systems that support AdTech is such a shame. That being said, I love these little anecdotes, they are super fun to read.

dvfjsdhgfv 6y ago
There are so many aspects of adtech that are ethically neutral. Just as advertising itself - there are even ads created by great creators that are works of modern art. It's only when it's abused adtech becomes evil.
- arthurcolle 6y ago
  Wow, I bet I'll miss out on so much important culture by not getting visual pollution all over the place unless its regulated out of existence. No thanks, I'll continue to block it everywhere that I can, and even going further and inspecting elements to get rid of obnoxious paywalls as well! Wow, am I a hypocrite? Maybe develop real, sustainable revenue models that aren't dependent on clickbait and being a lobbyist for corporations in the form of selling eyeballs to the highest bidder. Just yuck. I'll be inclined to buy into $WHATEVER if you just let me use it for a bit, if I find it of my own means. I promise you I'm not going to buy a Swatch because of your fucking banner ad. Seriously, I promise. If I just bought a book I promise you I'm not going to buy it again if I see it appear in my side banner. 100%. Or maybe I'll just grow less dependent on external services every time I have to consider another subscription... wow! We'll see!
- croes 6y ago
  Have you checked the art vs bullshit ratio?
  - arthurcolle 6y ago
    That probably costs extra
- beefield 6y ago
  It may be "many" but it does not mean that the defining nature of advertising is ethically neutral. I think I just quote Banksy/Tejaratchi: "People are taking the piss out of you everyday. They butt into your life, take a cheap shot at you and then disappear. They leer at you from tall buildings and make you feel small. They make flippant comments from buses that imply you're not sexy enough and that all the fun is happening somewhere else. They are on TV making your girlfriend feel inadequate. They have access to the most sophisticated technology the world has ever seen and they bully you with it. They are The Advertisers and they are laughing at you. You, however, are forbidden to touch them. Trademarks, intellectual property rights and copyright law mean advertisers can say what they like wherever they like with total impunity. Fuck that. Any advert in a public space that gives you no choice whether you see it or not is yours. It's yours to take, re-arrange and re-use. You can do whatever you like with it. Asking for permission is like asking to keep a rock someone just threw at your head. You owe the companies nothing. Less than nothing, you especially don't owe them any courtesy. They owe you. They have re-arranged the world to put themselves in front of you. They never asked for your permission, don't even start asking for theirs." Edit: Credits also to Tejaratchi
  - arthurcolle 6y ago
    This is such a great quote. I've checked out a couple books on Banksy but haven't ever run into it before. Thanks so much for sharing!
  - Teever 6y ago
    One time I saw a fascinating article about how scientists were working to grow skin complete with natural looking hair from stem cells for burn victims and such. I thought the article was really neat and I was in a hurry so I emailed a the link to myself. The next time I logged into my gmail to read it I happened to be at a machine without adblock and I saw that google was serving me ads for garbage treatments for male pattern baldness. Fuck you Google and your shitty advertising. There's nothing wrong with going bald and I don't need you to try and make me feel insecure about my hairline when I'm trying to read about scientists helping burn victims.
    - 6y ago
- arthurcolle 6y ago
  > so many Please name 5
  - dvfjsdhgfv 6y ago
    OK. Let's separate the ad industry from ad technology for now. So as for the latter, have a look at the Revive ad server [0]. It allows you to deliver ads with intricate rules as to when they should be served, on which websites and so on. It allows you to be less dependent on Google/FB duopoly. You can host the ads either on your own websites or contact the owners of other popular websites and arrange an ad deal that doesn't involve user tracking. Yes, this is the opposite of what modern ad networks are doing. In any case, advertising without user tracking is, in my opinion, one of the most ethically neutral forms of making your product/service known, in contrast to morally dubious methods such as advertising with user tracking, product placement, advertorials, advertising disguised as articles/videos etc. [0] https://www.revive-adserver.com/features/ https://www.revive-adserver.com/features/
- llarsson 6y ago
  There is a difference between the medium (ads) and the targeting and distribution technology (adtech). A great ad evokes some kind of feeling, can live with you for a long time, and even bring back memories that you cherish. Yes, I agree with you, it can be a form of art in that regard. Adtech, however, is an intentionally invasive technology that exploits any means necessary and technologically feasible to more narrowly target potential consumers. You can have ads without adtech. But you can't have (commercially successful) adtech without invasion of privacy.
  - dvfjsdhgfv 6y ago
    > But you can't have (commercially successful) adtech without invasion of privacy. I think we (as a society, not just tech people) specifically need to take care of this point. Removing tracking will make it less effective, so what. We've had advertising without tracking since 19th century.
deleted 6y ago
[deleted]
wwww4all 6y ago
There are more positives in Adtech than negatives. Adtech allows people and businesses to find products and services they want and need. Gives consumers choice in discovery and businesses reach and targeting customers in need. At fundamental level, Adtech is there to help people. That and the many business opportunities are why so many talented people work in Adtech.
- saiojd 6y ago
  Keep telling yourself that if it helps. ;) While at some level ads probably do help consumers (indirectly), my impression so far is that at a fundamental level they are about companies competing for money, in a way which wastes everybody's time. Since your point of view is so vastly different, I'm curious in which ways you feel like they give consumers choice?
- bigphishy 6y ago
  "At fundamental level, Adtech is there to help people." Haha. Adtech is not there to "help people". Back in the day, prior to television and radio, ads were there to provide information about a product, but those days are long since past. Ads now-a-days appeal to emotional and irrational wants/desires, this is common knowledge. Ads attempt to generate needless sales by marketing to irrational and emotional desires. Adtech on the other hand, is there to drive irrational and unneeded consumerism and consumption just as well, it just happens to have some additional evil sprinkled on top. For example, adtech violates human privacy, adtech marginalizes the open web, and turns it into a creepy surveillance nightmare. To say that "At fundamental level, Adtech is there to help people." is delusional and unintentionally hilarious. I am going to swing in the dark and say the GP must be an employee of facebook
- hesk 6y ago
  If only Google had a search engine to help people and businesses find products and services they want and need.
- edmundsauto 6y ago
  I share many concerns about tracking people, but let's acknowledge that adtech democratizes access to services that most people (worldwide) would not be able to afford. "Free" is the only price that works for the 36% of the world's population that is in poverty.

elcomet 6y ago

Very interesting story! The title (Google being able to kill facebook) seems a bit exagerated though. If this app was so important (25 to 50% of all facebook revenues), and so important to customers (most powerful editor to manage ads), I'm sure if Google had disabled websql, they would have forked some older chrome / electron with websql support and bundle it with the app.

arthurcolle 6y ago
that would have made builds massive, unless you're referring to just the desktop experience...
- elcomet 6y ago
  I'm saying they would have find a way to deliver the app to customers, whether in desktop or web form. And customers would have downloaded the app if needed, as it seems to be the only powerful way to manage ads.
aequitas 6y ago
Indeed, people will always find a way, especially if money is involved. And running outdated browsers for legacy support, while being a huge security risk, is not uncommon for a lot of use cases, especially back then.
mcv 6y ago
A hit of 25-50% of revenue would certainly have hurt, but probably not killed Facebook. That people will find a workaround is absolutely true. I once worked on a system to order mobile phone subscription from a website that only worked on IE6. So we had a website that accepted orders from customers, created jobs that would be sent to one of 10 VMs that ran IE6, where a watir script would steer IE6 to enter the order into the old website. It's the most horrible hack I ever worked on, but it worked.
- Pelic4n 6y ago
  Absolute cringe, thanks you for this.

supermatt 6y ago

TLDR: Facebook built the app responsible for their entire ad platform on a chrome-only tech (WebSQL), and when it was deprecated, instead of dropping in a widely-used polyfill, they decided to do a major 3yr rewrite risking 25% of their revenue in the process. I guess this is the stupid shit that being "too big to fail" permits.

asplake 6y ago
I didn’t know that such a polyfill existed but even so, that was pretty much my reaction. Facebook would be quite capable of developing one, and potentially a route there to something more service-based once the panic is dealt with.
kps 6y ago
Another option would have been to use a chromium build with WebSQL still in place.
- PeterisP 6y ago
  They would have to get all their advertiser customers to use that custom chromium build, which would be possible but would not be good for their ad sales.
  - protomyth 6y ago
    Given how long some of us had to hang on to Internet Explorer 6, I'm pretty sure the advertisers would have required their IT staff's to comply and install the "Facebook Browser".
    - ADefenestrator 6y ago
      Wouldn't have necessarily had to be nearly as bad as IE6 for security, either. Just lock the special browser to facebook.com only, or even just the ads API, and there's not a lot of room for exploit.
joe_the_user 6y ago

Osiris 6y ago

If the product was responsible for 25% of the company revenue, why was only half of one engineer assigned to it? And after the vulnerability was discovered, they still only had 5 people assigned to it while it increased in importance to 50% of revenue?

simonh 6y ago
I've seen this sort of thing happen many times. What usually happens is the original team that built it leave the company or move on to other teams, and the one engineer left actually has a different job as well now but is stuck supporting it part time as a favour. It's not that you couldn't assign other engineers to help support it, but they wouldn't have any idea how to do so because there's probably no documentation and only that half an engineer around to show them how it works.
- disgruntledphd2 6y ago
  It was because Facebook practiced open allocation for engineers at the time. Essentially, you had a choice of teams and the manager would need to convince the engineer to work on the project. At the time, ads was not a high status part of FB engineering, and as such, it was very difficult to get people to work on ads products.
  - ForHackernews 6y ago
    It's funny that people want to work at Facebook or Google but don't want to work on ads. In other companies, everyone wants to work near the profit center and avoid cost centers.
    - rvba 6y ago
      Because those projects are hard. It is much easier to make a greenfield project that will get 70% done and cancelled before anyone even asks to fix bugs.
      - 6y ago

spankalee 6y ago

That is not how Chrome removes features. It's based on usage. WebSQL is still supported today, in fact: https://caniuse.com/sql-storage https://caniuse.com/sql-storage

closetnerd 6y ago

Quick note: it's an unfortunate fact of life that some engineers make a career of exaggerating things.

disgruntledphd2 6y ago
This is not that, though. I heard this story from people involved at the time, and it's not exaggerated in any way, shape or form.
- hoten 6y ago
  But isn't it? I can believe people inside Facebook really thought it was a critical problem, but I don't see how it could have become any more than an inconvenience. - There's a websql polyfill. I don't have experience with it, but it looks like a drop in replacement. This is such a strong counterpoint, the rest almost doesn't matter. - Features tend to be disabled in Chrome with some advanced warning. Not overnight. - websql is still enabled in chrome today - if the api were disabled, ad customers could be directed to flip the feature switch in Chrome. If money is on the line, seems like an easy enough task.
  - yyyk 6y ago
    - Safari supported websql until safari 13 (september 2019), long after this issue stopped being relevant. How many people in an organization need to use PE? Probably only a few in marketing. Surely these few people could have just bought a macbook. A few macbooks must be a very small expense compared to facebook advertising. - Or IT could have arranged a VM with an old version of Chrome.
    - disgruntledphd2 6y ago
      So this was an application used by extremely non-technical customers (i.e. marketers) spending money on behalf of many, many brands. Relying on their IT to do this just wouldn't have worked and was definitely not scalable. For context, if Power Editor disappeared along with its revenue, Facebook would have needed to report this to the market.

acd10j 6y ago

Facebook's real moat is it's massive user base not it's ad revenue, even losing 25% of it's revenue would not have effected or killed Facebook, Only building better viral social network could have killed it. Ad revenue is just a side effect of this massive moat. Just like Google can survive a temporary year or 2 of 25% drop in revenue if their search dominance continues or increases but not opposite.

CodesInChaos 6y ago

Couldn't they have compiled sqlite to javascript using emscripten as a websql replacement? sql.js seems to have been started in 2012, and even if it might not have been production ready at the time, I'd expect facebook to be able to pull off such a project by themselves.

exolymph 6y ago

If Google could have killed Facebook, it would have.

29athrowaway 6y ago

Google+ was very poorly designed.

mcv 6y ago
Google+ was initially very well designed, and then Google proceeded to run it into the ground. Google+ when it was about a year old, was still the best social network ever. But then Google tried to force Youtube integration, killed off Events and Hangouts, not to mention the questionable "no unusual name" policy.

jorams 6y ago

I've heard, and keep hearing, so many stories about frustrations with Facebook's tools for advertisers that the situation described doesn't surprise me at all. Somehow this massive organization with >50 000 employees and billions of users can't seem to produce quality software that allows people to pay them for access to those users. Small tasks turn into multi-day endeavors just because the tools keep crashing over and over again, losing all state. Managed advertising accounts just vanish from interfaces. Random buttons turn into "try again" roulette. It's a good thing their ads are (apparently) effective.

ai_ja_nai 6y ago

Ehehe nice story. I remember when we were to be shut down by AWS because the bill was 2 months due and the administrative contact was a inbox nobody looked at. We discovered it by chance, doing a nightly maintenance: services started to disappear one after another (including the Redis that powered the login, we weren't able to login into the product for half an hour!). We immediately phoned AWS support and asked, no, begged them to turn them up again. They did and we paid everything the morning after. Hadn't we been in front of the PCs doing the maintenance and witnessed as it was happening, we might have woken up the day after without an online bank (yes) because the production environment would have ceased to exist...

mutatio 6y ago

It raises interesting questions of whether Google (advertising company) should be the guardian of the web's primary browser; strategically it could deprecate features direct competitors depend on (let's assume FB is now also a ad company). The fact that web's primary ad platform also owns the web's primary browser sure is "interesting". The inverse of that is that it might not have incentive to deprecate features - which Google the company / partners depend on or might not be able to deprecate due to legal consequences of doing so (shutting down 25% of FB's revenue might raise eyebrows in the gov).

jasonvorhe 6y ago
iirc it was Mozilla who put the nail in the coffin of WebSQL and if you read this post, it actually mentions that Chrome was the last browser to even still support the feature after it had been abandoned by Firefox and Opera already. What does this have to do with Google owning Chrome? Of course, the question itself is valid but I don't see how it relates to the topic.

baybal2 6y ago

Google actually had a successful social network project, Orkut, they just continuously marginalised it for it to be too NIH for US headquarters.

chaz6 6y ago
As far as I can tell, Orkut was the original incarnation of a modern social network, but they failed to capitalize on it.
- lupire 6y ago
  Orkut was a clone of The social network created by the namesake's previous company. It was launched about the same time as Facebook and after Friendster.
daniellarusso 6y ago
Wasn’t it very popular in Brazil for some time?
- nojokes 6y ago
  Yes it was. Source - old Orkut user but not from Brazil.
  - hrishi 6y ago
    I'm guessing India - it's where I grew up and it was the network of choice for teens until we moved to Facebook
bemmu 6y ago
Also tried to band everyone (MySpace among them) together with OpenSocial, but too late.
GuB-42 6y ago
I don't know how they are doing it but Google manage to fail at anything social. By that I mean communication platforms, social networks and instant messengers. They have promising starts but they fuck it up later. Their only successes I'd say are GMail and YouTube. I'd say for GMail, it is just an email client they started out excellent, thanks in part to its huge storage space and good search features. And they didn't break it enough to make it unusable. As for YouTube, it is great for hosting video, but its social features are terrible. There are communities tied to popular channels, but it stays in a producer to consumer model, with no interaction between community members. The comment section is made in such a way that dialogue is almost impossible. Maybe they should hire a few people who understand human beings, because right now, it looks like the company is run by robots.

yholio 6y ago

Today it's called Business Manager and it remains a user hostile pile of JavaScript with dubious design and riddled with bugs. The only reason anybody puts up with it is because there's no other way to access powerful targeting tools, and we are forced to depend on Facebook. More to the point of the original story, no, Google couldn't have killed Facebook by flipping that switch. The Power Editor users would have done whatever it takes to access the interface, installed an insecure older browser, ran commands in the console, build a relay computer in their back yard, and so on. They will do whatever it takes because they are not choosing that platform for its features, they are professionals doing a job, and that job is reaching the user base of Facebook with their ads. If they can't do that they will lose their jobs and clients.

cm2012 6y ago
I find business manager has a more sensible design than AdWords, having used both a lot. TikTok is funny because they basically copied the FB ads dashboard pixel by pixel.
- dillondoyle 6y ago
  I agree but only the actual /adsmanager/manage section and the subsequent tools (events/audiences). Basically power editor 2.0. But it seems less powerful than the old one for some reason might just be in my head though. It has a pretty low ad limit per ad group I think 100. Though some of the 'combo' ads where you can put 5 headlines, 5 copy etc help up that number. BUT I do not like the business manager which houses all the pages, users, audience assets. I find the 'line of business' really confusing and hard to see how they want it organized. Like what's the model supposed to be? Agency line of business -> client subs. But then you need a specific line of business for certain offline attribution etc so having it agency parent doesn't make sense to me?!
  - etempleton 6y ago
    I think it feels less powerful because it is a buggy mess and takes forever to use. I loath setting up Facebook ads now because I know it is going to take me forever to do it properly. I pine for the days of PowerEditor. It had a confusing interface, but like Google Ads, once you had learned it you could get what you wanted to get done quickly. I don’t mind the interface of Business Manager or the Ads Manager platform at all, but I find the ads platform to be riddled with weird bugs. It has perhaps improved slightly in the past year or so, but only slightly. They also constantly change how it works, so you can’t just copy the campaign or ad set of an ad and build out new ads from a year ago, it will almost always have to be built out from scratch because something changed, but they don’t tell you that before you copy the campaign. Why even let me copy it if it isn’t going to work?

neodymiumphish 6y ago

> This became a closely held secret in Facebook Ads leadership. We didn’t want to take any chance that word our this vulnerability could get back to Google. Considering the tracking Google does with Chrome, I suspect they recognized the traffic still occurring over WebSQL and left it active for exactly this reason. It's highly unlikely that nobody at Google was able to recognize this. Alternatively, maybe Google didn't know about Facebook's reliance on it, but was aware of other services/corporations that still relied on it for some time, and waited until they had an alternative.

esprehn 6y ago
At the time this story takes place Chrome didn't have a way to collect web feature usage metrics broken down by site. It was aggregated across all sites to protect user privacy. That came years later with Rappor: https://www.chromium.org/developers/design-documents/rappor https://www.chromium.org/developers/design-documents/rappor WebSQL usage was too high at the time to remove though, so while the story is fun to read Chrome never would have actually turned it off.
- shaneos 6y ago
  [author] Hard to say. If Google leadership had heard of this vulnerability, right as they were launching Google+, there would at least have been a serious conversation about it. They spent $Bs to build Google+, changed their whole company structure to support it, and were losing hundreds of people to FB every month. I doubt bending one little rule about removing a deprecated technology would have bothered them too much.

Hnshadowbansvd 6y ago

But.. They end up to be same censuring machine as twitter and Facebook

deleted 6y ago

[deleted]

anonygler 6y ago

Today's episode in "Read Your Deprecation Warnings"

fbelzile 6y ago

This is a good story, but Google can still literally kill any business that relies on the Play Store or Chrome Store with a flick of a switch today (like mine). No human even has to get involved. At any moment, some code running on a Google server can decide to flip a bit and disable access to an app or extension in their stores.

kokx 6y ago
That's not entirely true. They can kill any small and medium business easily. Taking out larger competitors in such a way will likely invoke lots of unwanted antitrust attention. Over the entire world. But they can easily kill any small competitor.

wwww4all 6y ago

I laugh whenever the tech futurists talk about AI, machine learning, automation, singularity coming in the near future. Most of the important software, that run Trillions of dollars in online economy, have dependencies in ancient tech. No one wants to work in that jumble, and updating has failed in one way or another. Google ACH and ask why it’s takes several days to transfer money from bank to bank.

_-___________-_ 6y ago
> Google ACH and ask why it’s takes several days to transfer money from bank to bank. Given that there’s approximately one country left in the world where that’s the case, I suspect the root cause is not a technical issue.

esprehn 6y ago

This story is a fun read, but the "Chrome could ruin us at any moment" angle is more story telling hyperbole than reality. At that time on Chrome we were just figuring out a process to remove features and were pretty conservative. We debated a lot about the right way to remove things without causing damage to the web ecosystem. In an early attempt we had failed trying to remove a legacy feature and needed to revert it. It had broken a big company who had then contacted Google directly about the business impact and it taught us a lesson about being careful. IIRC WebSQL had very high usage and we didn't seriously considering removing it. It's also pretty self contained so the cost on the engine isn't that high. Chrome still supports WebSQL today because the usage remains quite high at 1%: https://chromestatus.com/metrics/feature/popularity#V8SQLTransaction_ExecuteSql_Method https://chromestatus.com/metrics/feature/popularity#V8SQLTra...

grecy 6y ago
.. and even if Chrome had removed WebSQL back then, couldn't that have just kept using whatever version of Chrom e they were already using?
- nosefrog 6y ago
  My understanding is that clients would have to use an outdated version of Chrome, which is probably an untenable ask.
- esprehn 6y ago
  I suspect a lot of FB customers are using a browser in a business setting where IT manages the Chrome updates. FB could have used a polyfill though, or used Safari, or shipped a desktop wapper around an older Chrome temporarily. Electron was brand new at the time so it probably wasn't a great option, but FB certainly had a lot of emergency options.
  - shaneos 6y ago
    [author] There were some contingency plans, but we had no time to get them ready in advance. If Google had actually pulled the plug, we wouldn't been down for everyone for many weeks, and given that the vast majority of our large customers are non-tech-savvy (low paid employees at ad agencies), a large percentage would have remained with a broken experience forever. They'd have just given up and moved their money to Google.

waterside81 6y ago

Anybody have insight into why WebSQL was chosen in the first place? I don't think I've even heard of it prior to this article.

shaneos 6y ago
[author] Power Editor was originally written by a brilliant engineer named Vladimir Kolesnikov (yes, all by one engineer), and he was solving a very specific problem of letting customers scale their ad spend. So he chose a technology that let them download all of their ad account locally to disk, do all their work, then upload it again. WebSQL was flexible, fast and easy to work with. It's a real shame it was deprecated to be honest

simon_weber 6y ago

Interestingly, Google later wrote their own relational db in javascript called lovefield, which was motivated primarily by the deprecation of WebSQL: https://github.com/google/lovefield/blob/master/docs/dd/00_intro.md#01-motivation https://github.com/google/lovefield/blob/master/docs/dd/00_i...

utopcell 6y ago

extreme urgency.. facebook would shut down.. new ads features blocked.. urgent creation of a team to replace product.. 3 years later it was ready (!) Best example of leading by FUD if I ever heard one.

_carbyau_ 6y ago
I'd call it "managing upwards". :-P

RachelF 6y ago

In the early days of FB, they relied on Google Docs and Gmail. More importantly, they used pre-built social networks by letting you import your GMail contacts.

whalesalad 6y ago

Could have killed you with the flick of a switch - but you took three years and a team of 5 people to perform the rewrite?

shaneos 6y ago
[author] Unfortunately yes. It was not the only thing happening at the time. FB had just gone IPO and this put a huge amount of pressure to grow revenue. When I joined the Ads org in 2012, it was about 100 people I think, with just a few products. When I left in 2017 it was well over 1000, and my job had changed to managing relationships with 55 other teams, all of whom depended on my teams product to ship their individual features (think Video ads, Carousel ads, hundreds of others). So, while doing the rewrite, we had to transition from being a product org, where a few engineers built all features for a bunch of back end teams, to an infra+product team, where we built all sorts of support for all the new teams that seemed to spring up weekly. If we had done only the technical rewrite, and not enabled all these other teams to ship their products on the UI surface that our customers actually used, FB's revenue would not have gone from $4B when I started to around $30B when I left. These things are never that simple. I wish....

_carbyau_ 6y ago

To kill FB I don't understand why Google doesn't have a full time team of people working to extend/fork Signal to add a doomscrolling "blog posts" feature with required group/circles/whatever-bullshit. This isn't about Google capturing the FB space. Simply denying it to FB. Honestly: - no ads. (Seriously Google, no ads. But you know they can't which is why this is a pipe dream....) - could sort your own timeline by timestamp, number of comments, whatever - can't be shutdown due to content. Bonus points for: - Events and adding Events into your local phone calendar.(Please don't give me another calendar to cross reference.) Google twist the knife effort: - Free public facing web hosting for those who actually want their "Wall" public. IE companies, businesses, "personalities" etc. They did this for Android to compete with Apple. They can do it with Signal to compete with FB. If nothing else, could they "buy" the features from the Signal team in a similar fashion to Blender? This'd be my preference so that Signal maintains independance at least somewhat.(Money always talks I know.)