6 ms·

> The most important deciding factor for me was whether the company has some kind of technological edge years ahead of its competitors. A friend on Google’s log

> The most important deciding factor for me was whether the company has some kind of technological edge years ahead of its competitors. A friend on Google’s logging team tells me he’s not interested in smaller companies because they are so technologically far behind Google’s planetary-scale infra that they haven’t even begun to fathom the problems that Google is solving now, much less finish solving the problems that Google already worked on a decade ago. ^ this is increasingly a choice in your career -- 'where can you go to solve big problems', and 'big problems are increasingly complex' scale is real, and tools matter. you can spend your whole project burn at the wrong company building something that you could buy somewhere else, or which already exists at a competitor slight grain of salt here is that G's logging system, from my perspective as a gcp user, is slow as balls and the UX is the incarnation of scroll jank. and also this (very good) article led to the outcome of the author building soft hands happy-ending robots

oneoff786 4y ago
On the other hand, there’s a lot of real problems that real people actually deal with that just need a logistic regression to save million bucks here and there. I like that space more.
- nerdponx 4y ago
  This space isn't sexy to write about, there's no fame and glory in it.
  - tomrod 4y ago
    Far sexier and larger than most care to admit.
  - deleted 4y ago
    [deleted]
  - Terry_Roll 4y ago
    The trappings are... https://www.youtube.com/watch?v=4VapnDFoR2U https://www.youtube.com/watch?v=4VapnDFoR2U
- axg11 4y ago
  This describes 95% of machine learning at FAANG+, unfortunately nobody likes to talk about. Context: I work at a FAANG.
  - 21723 4y ago
    Probably 99%. The people doing next-level stuff are largely being humored so the FAANGs can hire more early 20s grunts who think they're going to be working on ML (they won't be) or even given the time of day by leads on such projects (ditto). You give some famous professor a high-6 or low-7 figure salary in a position where his job is to publish papers--or not, because no one who matters in the company will ever read them--but you get incrementally more effort out of the youngsters who think they've got a chance (and they don't) of working on something more exciting than Jira tickets. I don't think the FAANGs will ever come up with AGI and I am glad for that. If the private sector gets there first, I am going to "accidentally" die of hypothermia on a hiking trip because I will do anything not to live in their world.
  - thoi4234234 4y ago
    God, and it's such a snooze-fest. Writing TFX code at Google is like having your soul-sucked through your rear-end! Imagine TF1 with all the broken APIs, but now it's all distributed! Fun.
    - DesiLurker 4y ago
      > ... having your soul-sucked through your rear-end! I laugh-snorted reading that! I am stealing that phrase.
    - winrid 4y ago
      Probably not the best analogy. Sounds like a good time.
- difflens 4y ago
  If you have a few minutes, can you list a few of these problems? Just curious here!
  - oneoff786 4y ago
    I’m a consultant. I do not work in tech. Typical opportunities look at a decision that gets made many many times. This includes systems where everything gets treated the same despite some 80/20 kind of situation, which is a lot of them. Lots of older businesses have these kinds of setups where stuff is run on gut instinct or decent enough but risk adverse rules. Don’t think crazy neural net image recognition whatever. Really just look at what the business spends a lot of money on and think “could they do that smarter?” A common thing I do is say company X has a fleet of Y assets. They repair them every N years. A good solution would be to predict which ones need repairs. Do those ones more often. Do the healthy ones less often. Pay more attention to the ones that are valuable. Better outcomes, millions less spending. Probably don’t even need a live model in prod. Just a semi annual manual export run to excel for some planner guy who’s been keeping the schedule for 2 decades
- geebee 4y ago
  I've heard about this, in different contexts. What it mainly comes down to is that incremental improvements can have massive impacts when you can apply them at a scale available at FAANG. I first read about this outside the context of machine learning, but it certainly would apply here. For those of us who don't work at such scale, can you (maybe with a little fuzziness to avoid telling too much about an internal project) give a few examples of the kind of projects where a fairly simple model can have a 1M+ impact?
  - hansonw 4y ago
    Here's a ~5 minute talk with 5 such examples (where relatively simple ML models made a 1M+ impact at a FAANG) :) https://youtu.be/zyOEOd1HkSY?t=946 https://youtu.be/zyOEOd1HkSY?t=946 Happy to talk about more details if you message me through my profile!
    - geebee 4y ago
      Thank you for the link! They were all interesting, and yes, all the result of having a high scale. For anyone curious and thinking about watching the video (I recommend watching it), the topics were 1) should you immediately re-run a failed ad payment (getting paid vs transaction costs/flagged for repeated billing), 2) should you send an IM immediately after a login failure (cost of text message vs possibility user will give up and not reset password), 3) should you fetch data for pre-loading in a web page (higher engagement with page vs cost of unnecessary loading), 4) video upload quality, 5) taking screen real estate for less commonly used UI features. Interesting examples, and yes, they're all the kind of thing that might not justify the effort for an ML model (and might not have enough data to train) for a small website or operation, but can easily justify the cost and effort when you have a huge number of transactions. On another note, this is why I often like lightening talks. So many people think that what they're doing falls below the threshold for what is an interesting presentation, when in fact it's the most relevant thing a lot of people will see at a conference.
      - hoseja 4y ago
        Wow, of those five I'd only call 3) not evil, maybe 1). (based on the video, where the twisted reasons for them are explained.)
- savant_penguin 4y ago
  Would you kindly tell what types of projects are such nice successes?
  - oneoff786 4y ago
    See adjacent responses
altdataseller 4y ago
You are talking about 2 commodities that can easily be upgraded: logging and UX. Hell, I wouldn’t even waste too many precious resources on improving those things past a point. Google has massive data and scaling advantages that can never be duplicated or fixed by smaller companies.
deleted 4y ago
[deleted]
htrp 4y ago
>G's logging system, from my perspective as a gcp user, is slow as balls and the UX is the incarnation of scroll jank does GCP use G's internal tools? Inquiring minds would like to know
adam_arthur 4y ago
Not every problem is one of scale. And with the advent of serverless, problems relating to scalability will be largely abstracted from 99% of developers in the future and more of a niche knowledge domain. Just as the inner workings of the OS are largely not well understood by most developers Obviously the principles and theory behind scalability is still important for properly structuring your app, but there won't be many novel problems to solve, and increasingly obvious architecture choices as time goes on
- Agingcoder 4y ago
  Well, I'd say they won't fully be abstracted away because devs will still have to pay for scalable systems (hw + sw) with dollars. Granted, they won't have to think about designing a solution, but they still won't have the computational power which can be afforded by larger companies (and cloud computing is ridiculously expensive), unless they have a lot of money .
rychco 4y ago
This has become increasingly important to me too. I am employed by a small (2-4 engineers at any time) company and I'm often disappointed because we're just so far behind in manpower & technical expertise that we have to dramatically reduce the scope of any problem we want to tackle. On the other hand, I also worry about getting sucked into the bureaucracy of FAANG sized companies & not having any accountability or agency over what I work on. (I realize this is a sweeping generalization of FAANG, but some of my peers have had this experience even a few years into their jobs)
- fxtentacle 4y ago
  I'm surprised. Apart from DALLE, I haven't seen any AI approach that's off limits for 4 highly motivated people with 3090 GPUs. At that compute level, you should be able to at least replicate SOTA in optical flow, structure from motion, speech recognition, text to speech, translation, text summary, sentiment analysis, image classifications, image segmentation, and of course playing video games or optimizing processes with reinforcement learning. I mean thanks to KiCad even custom sensor hardware is cheap these days. Can you give more details about what you tried to do and why that wasn't possible?
  - sigmoid10 4y ago
    Pretty much everything in the high end large language model area is off limits to people without access to a supercomputer (we're talking hundreds of A100s or several $100k in cloud computing equivalent). Open Source efforts like BigScience may open up downstream tasks for normal people, but the forefront of this research is no longer accessible to individuals.
    - fxtentacle 4y ago
      You might be surprised to hear that the KenLM language models that are used for speech recognition are actually trained on-disk using CPU. With a €149 monthly bare metal server, I could train my own LM on OSCAR DE and EN. Where I do agree with you is that transformer-style text generation models in the billion parameter range are off-limits for hobbyists. But that's only a tiny part of the useful applications of AI. And you can train them with gradient checkpointing, it's just 100x slower than what Google can do.
hintymad 4y ago
I think this reasoning is flawed. Joining Google does not mean you get to solve interesting problems because you will most likely contribute incrementally to a vast body of work. You want to build the next-gen database? You have to come up with something way better than BigTable and Spanner. You want to build a queue service? You've got to come up with something way better than Google's PubSub, which optimizes itself all the way down to the level of GBP protocols. You want to build a machine learning framework? Have you checked out TensorFlow and JAX in particular their ecosystem? You want to build a file system? Are you sure you can overcome all the organizational inertia that Google has built on Colossus over the years? You want to build a 10X more productive framework for data processing? Are you sure you can beat DataFlow or Flume4J-ecosystem in Google? The point is, Google is a mature company. Mortals like most of us don't get to break ground in a technically advanced but mature company like Google. Instead, we find fast growing new problems to solve, to hone our skill, and to get to scale. P.S., I personally know a number of prominent professors used to work on the Borg projects just to optimize for a few percent of gains. It's deep and interesting work, but nonetheless hard for mortals like me to get much out of. That is, it's a more sure bet to work for a baby Google than to work for a middle-aged Google.
- anonymousDan 4y ago
  Yes or work on things that are fundamentally at odds with Google's business model, or on systems that in some way are difficult for Google to do because they have company-spe ific constraints or legacy systems
- bricemo 4y ago
  It’s definitely a tradeoff. I personally have gone through alternate phases where I wanted to be responsible for a huge piece of a small pie (startup) versus a tiny tiny piece of a huge pie (big company). There are pros and cons to both. The big slice of small pie feels cooler and more satisfying to my ego, but in the absolute sense the tiny piece of the big pie probably is actually making a bigger difference. 1% improvement at some of these companies means north of billions of dollars and improving hundreds of millions of peoples lives.
  - 4y ago
civilized 4y ago
Yes, Google has a quadrillion internet search queries, but that only helps if your interested in solving problems that require a quadrillion internet search queries.
yowlingcat 4y ago
I see this argument a lot and it always feels like an over-literal interpretation. Let me ask you a question -- what proves your ability to work on scale as an engineer? Is it the scale of the problem, or the scale of your solution? Outside of "Google scale" logging is a nearly trivial, solved problem. If the only thing that creates the "challenges of scale" is your data cardinality, guess what. You're only solving problems of scale in the most literal (and maybe trivial) sense. Working on that at Google isn't going to prepare you for how to architect a platform for a startup that scales from 0 to 1 million active users over night without breaking a sweat. I am biased in that I care about the latter kind of scale and effectively couldn't care less about the former, because the latter is generally an existential problem to have, while the former is a nice problem to have.
albrewer 4y ago
> you can spend your whole project burn at the wrong company building something that you could buy somewhere else, or which already exists at a competitor Oh look it's me. I'm sitting here building a worse version of TestComplete and Ranorex for automated testing of a Windows desktop application. Looks good on my resume though.

oneoff786 4y ago

On the other hand, there’s a lot of real problems that real people actually deal with that just need a logistic regression to save million bucks here and there. I like that space more.

nerdponx 4y ago
This space isn't sexy to write about, there's no fame and glory in it.
- tomrod 4y ago
  Far sexier and larger than most care to admit.
- deleted 4y ago
  [deleted]
- Terry_Roll 4y ago
  The trappings are... https://www.youtube.com/watch?v=4VapnDFoR2U https://www.youtube.com/watch?v=4VapnDFoR2U
axg11 4y ago
This describes 95% of machine learning at FAANG+, unfortunately nobody likes to talk about. Context: I work at a FAANG.
- 21723 4y ago
  Probably 99%. The people doing next-level stuff are largely being humored so the FAANGs can hire more early 20s grunts who think they're going to be working on ML (they won't be) or even given the time of day by leads on such projects (ditto). You give some famous professor a high-6 or low-7 figure salary in a position where his job is to publish papers--or not, because no one who matters in the company will ever read them--but you get incrementally more effort out of the youngsters who think they've got a chance (and they don't) of working on something more exciting than Jira tickets. I don't think the FAANGs will ever come up with AGI and I am glad for that. If the private sector gets there first, I am going to "accidentally" die of hypothermia on a hiking trip because I will do anything not to live in their world.
- thoi4234234 4y ago
  God, and it's such a snooze-fest. Writing TFX code at Google is like having your soul-sucked through your rear-end! Imagine TF1 with all the broken APIs, but now it's all distributed! Fun.
  - DesiLurker 4y ago
    > ... having your soul-sucked through your rear-end! I laugh-snorted reading that! I am stealing that phrase.
  - winrid 4y ago
    Probably not the best analogy. Sounds like a good time.
difflens 4y ago
If you have a few minutes, can you list a few of these problems? Just curious here!
- oneoff786 4y ago
  I’m a consultant. I do not work in tech. Typical opportunities look at a decision that gets made many many times. This includes systems where everything gets treated the same despite some 80/20 kind of situation, which is a lot of them. Lots of older businesses have these kinds of setups where stuff is run on gut instinct or decent enough but risk adverse rules. Don’t think crazy neural net image recognition whatever. Really just look at what the business spends a lot of money on and think “could they do that smarter?” A common thing I do is say company X has a fleet of Y assets. They repair them every N years. A good solution would be to predict which ones need repairs. Do those ones more often. Do the healthy ones less often. Pay more attention to the ones that are valuable. Better outcomes, millions less spending. Probably don’t even need a live model in prod. Just a semi annual manual export run to excel for some planner guy who’s been keeping the schedule for 2 decades
geebee 4y ago
I've heard about this, in different contexts. What it mainly comes down to is that incremental improvements can have massive impacts when you can apply them at a scale available at FAANG. I first read about this outside the context of machine learning, but it certainly would apply here. For those of us who don't work at such scale, can you (maybe with a little fuzziness to avoid telling too much about an internal project) give a few examples of the kind of projects where a fairly simple model can have a 1M+ impact?
- hansonw 4y ago
  Here's a ~5 minute talk with 5 such examples (where relatively simple ML models made a 1M+ impact at a FAANG) :) https://youtu.be/zyOEOd1HkSY?t=946 https://youtu.be/zyOEOd1HkSY?t=946 Happy to talk about more details if you message me through my profile!
  - geebee 4y ago
    Thank you for the link! They were all interesting, and yes, all the result of having a high scale. For anyone curious and thinking about watching the video (I recommend watching it), the topics were 1) should you immediately re-run a failed ad payment (getting paid vs transaction costs/flagged for repeated billing), 2) should you send an IM immediately after a login failure (cost of text message vs possibility user will give up and not reset password), 3) should you fetch data for pre-loading in a web page (higher engagement with page vs cost of unnecessary loading), 4) video upload quality, 5) taking screen real estate for less commonly used UI features. Interesting examples, and yes, they're all the kind of thing that might not justify the effort for an ML model (and might not have enough data to train) for a small website or operation, but can easily justify the cost and effort when you have a huge number of transactions. On another note, this is why I often like lightening talks. So many people think that what they're doing falls below the threshold for what is an interesting presentation, when in fact it's the most relevant thing a lot of people will see at a conference.
    - hoseja 4y ago
      Wow, of those five I'd only call 3) not evil, maybe 1). (based on the video, where the twisted reasons for them are explained.)
savant_penguin 4y ago
Would you kindly tell what types of projects are such nice successes?
- oneoff786 4y ago
  See adjacent responses

altdataseller 4y ago

You are talking about 2 commodities that can easily be upgraded: logging and UX. Hell, I wouldn’t even waste too many precious resources on improving those things past a point. Google has massive data and scaling advantages that can never be duplicated or fixed by smaller companies.

deleted 4y ago

[deleted]

htrp 4y ago

>G's logging system, from my perspective as a gcp user, is slow as balls and the UX is the incarnation of scroll jank does GCP use G's internal tools? Inquiring minds would like to know

adam_arthur 4y ago

Not every problem is one of scale. And with the advent of serverless, problems relating to scalability will be largely abstracted from 99% of developers in the future and more of a niche knowledge domain. Just as the inner workings of the OS are largely not well understood by most developers Obviously the principles and theory behind scalability is still important for properly structuring your app, but there won't be many novel problems to solve, and increasingly obvious architecture choices as time goes on

Agingcoder 4y ago
Well, I'd say they won't fully be abstracted away because devs will still have to pay for scalable systems (hw + sw) with dollars. Granted, they won't have to think about designing a solution, but they still won't have the computational power which can be afforded by larger companies (and cloud computing is ridiculously expensive), unless they have a lot of money .

rychco 4y ago

This has become increasingly important to me too. I am employed by a small (2-4 engineers at any time) company and I'm often disappointed because we're just so far behind in manpower & technical expertise that we have to dramatically reduce the scope of any problem we want to tackle. On the other hand, I also worry about getting sucked into the bureaucracy of FAANG sized companies & not having any accountability or agency over what I work on. (I realize this is a sweeping generalization of FAANG, but some of my peers have had this experience even a few years into their jobs)

fxtentacle 4y ago
I'm surprised. Apart from DALLE, I haven't seen any AI approach that's off limits for 4 highly motivated people with 3090 GPUs. At that compute level, you should be able to at least replicate SOTA in optical flow, structure from motion, speech recognition, text to speech, translation, text summary, sentiment analysis, image classifications, image segmentation, and of course playing video games or optimizing processes with reinforcement learning. I mean thanks to KiCad even custom sensor hardware is cheap these days. Can you give more details about what you tried to do and why that wasn't possible?
- sigmoid10 4y ago
  Pretty much everything in the high end large language model area is off limits to people without access to a supercomputer (we're talking hundreds of A100s or several $100k in cloud computing equivalent). Open Source efforts like BigScience may open up downstream tasks for normal people, but the forefront of this research is no longer accessible to individuals.
  - fxtentacle 4y ago
    You might be surprised to hear that the KenLM language models that are used for speech recognition are actually trained on-disk using CPU. With a €149 monthly bare metal server, I could train my own LM on OSCAR DE and EN. Where I do agree with you is that transformer-style text generation models in the billion parameter range are off-limits for hobbyists. But that's only a tiny part of the useful applications of AI. And you can train them with gradient checkpointing, it's just 100x slower than what Google can do.

hintymad 4y ago

I think this reasoning is flawed. Joining Google does not mean you get to solve interesting problems because you will most likely contribute incrementally to a vast body of work. You want to build the next-gen database? You have to come up with something way better than BigTable and Spanner. You want to build a queue service? You've got to come up with something way better than Google's PubSub, which optimizes itself all the way down to the level of GBP protocols. You want to build a machine learning framework? Have you checked out TensorFlow and JAX in particular their ecosystem? You want to build a file system? Are you sure you can overcome all the organizational inertia that Google has built on Colossus over the years? You want to build a 10X more productive framework for data processing? Are you sure you can beat DataFlow or Flume4J-ecosystem in Google? The point is, Google is a mature company. Mortals like most of us don't get to break ground in a technically advanced but mature company like Google. Instead, we find fast growing new problems to solve, to hone our skill, and to get to scale. P.S., I personally know a number of prominent professors used to work on the Borg projects just to optimize for a few percent of gains. It's deep and interesting work, but nonetheless hard for mortals like me to get much out of. That is, it's a more sure bet to work for a baby Google than to work for a middle-aged Google.

anonymousDan 4y ago
Yes or work on things that are fundamentally at odds with Google's business model, or on systems that in some way are difficult for Google to do because they have company-spe ific constraints or legacy systems
bricemo 4y ago
It’s definitely a tradeoff. I personally have gone through alternate phases where I wanted to be responsible for a huge piece of a small pie (startup) versus a tiny tiny piece of a huge pie (big company). There are pros and cons to both. The big slice of small pie feels cooler and more satisfying to my ego, but in the absolute sense the tiny piece of the big pie probably is actually making a bigger difference. 1% improvement at some of these companies means north of billions of dollars and improving hundreds of millions of peoples lives.
- 4y ago

civilized 4y ago

Yes, Google has a quadrillion internet search queries, but that only helps if your interested in solving problems that require a quadrillion internet search queries.

yowlingcat 4y ago

I see this argument a lot and it always feels like an over-literal interpretation. Let me ask you a question -- what proves your ability to work on scale as an engineer? Is it the scale of the problem, or the scale of your solution? Outside of "Google scale" logging is a nearly trivial, solved problem. If the only thing that creates the "challenges of scale" is your data cardinality, guess what. You're only solving problems of scale in the most literal (and maybe trivial) sense. Working on that at Google isn't going to prepare you for how to architect a platform for a startup that scales from 0 to 1 million active users over night without breaking a sweat. I am biased in that I care about the latter kind of scale and effectively couldn't care less about the former, because the latter is generally an existential problem to have, while the former is a nice problem to have.

albrewer 4y ago

> you can spend your whole project burn at the wrong company building something that you could buy somewhere else, or which already exists at a competitor Oh look it's me. I'm sitting here building a worse version of TestComplete and Ranorex for automated testing of a Windows desktop application. Looks good on my resume though.