4 ms·

The problem is that when you’re on the cusp of a new thing, unless you’re super lucky, the result will necessarily be near the noise floor. Real science is like

The problem is that when you’re on the cusp of a new thing, unless you’re super lucky, the result will necessarily be near the noise floor. Real science is like that. But I definitely agree it’d be nice to go back and show something is true to p=.0001 or whatever. Overwhelmingly solid evidence is truly a wonderful thing, and as you say, it’s really the only way to build a solid foundation. When you engineer stuff, it needs to work 99.99-99.999% of the time or more. Otherwise you’re severely limited to how far your machine can go (in terms of complexity, levels of abstraction and organization) before it spends most of its time in a broken state. I’ve been thinking about this while playing Factorio: so much of our discussion and mental modeling of automation works under the assumption of perfect reliability. If you had SLIGHTLY below 100% reliability in Factorio, the game would be a terrible grind limited to small factories. Likewise with mathematical proofs or computer transistors or self driving cars or any other kind of automation. The reliability needs to be insanely good. You need to add a bunch of nines to whatever you’re making. A counterpoint to this is when you’re in an emergency and inaction means people die. In that case, you need to accept some uncertainty early on.

twoslide 5y ago
> when you’re on the cusp of a new thing, unless you’re super lucky, the result will necessarily be near the noise floor. Real science is like that. That's not necessarily true in social sciences. When you're working with large survey datasets, many variables are significantly related. That doesn't mean these relationships are meaningful or causal, they could be due to underlying common causes, etc. (Maybe social sciences weren't included in "real science" - but there's where a lot of stats discussions focus)
mumblemumble 5y ago
Fine. Do it like the experimental physicists do: if you think you're on to something, refine and repeat the experiment in order to get a more robust, repeatable result. The original sin of the medical and social sciences is failing to recognize a distinction between exploratory research and confirmatory research and behave accordingly.
- deleted 5y ago
  [deleted]
- Robotbeat 5y ago
  The problem is that it’s really hard to get good data, ethically, in medical sciences. Something that improves outcomes by 5-10% can be really important, but trying to get a study big enough to prove it can be super expensive already.
  - TameAntelope 5y ago
    Nobody likes being in the control group of the first working anti-aging serum...
    - q-big 5y ago
      > Nobody likes being in the control group of the first working anti-aging serum... You only know whether it works when the study has been completed. You also only know whether the drug has (potentially) disastrous consequences when the study has been completed. Thus, I am not completely sure whether your claim holds.
      - bluGill 5y ago
        You missed the working part. Success was a prerequisite to their after the fact feelings. At least some of the control group will be in old age but still alive when we know it woris. They might not know if it is infinite life (and side effects may turn it into die at 85, so some control may outlive the intervention group after the study ), but they will know on average they did worse
        q-big 5y ago
        > At least some of the control group will be in old age but still alive when we know it woris. The anti-aging serum could work (i.e. make you older), but have strong negative side effects.
mercurywells 5y ago
> I’ve been thinking about this while playing Factorio: so much of our discussion and mental modeling of automation works under the assumption of perfect reliability. If you had SLIGHTLY below 100% reliability in Factorio, the game would be a terrible grind limited to small factories. So I'm making a guess here that you play with few monsters or non-aggressive monsters?
- Robotbeat 5y ago
  Currently playing a game to minimize pollution to try to totally avoid biter attention. Surrounded by trees, now almost entirely solar with efficiency modules.
- reificator 5y ago
  > So I'm making a guess here that you play with few monsters or non-aggressive monsters? Aggressively building turret walls, defensive train lines, and so on very quickly pays dividends here. Particularly if you claim as much territory as you can each time you expand instead of simply defending what you've built out. If done this way building/improving defenses and managing enemies becomes a task you maintain every so often and doesn't spill over into the reliability of your base.
MaulingMonkey 5y ago
> If you had SLIGHTLY below 100% reliability in Factorio, the game would be a terrible grind limited to small factories. I'd argue you do have <100% reliability in Factorio, and much of the game is in increasing the 9s. Biters can wreck havok on your base. Miners contaminate your belts with the wrong types of ore, if you weren't paying enough attention near overlapping fields. Misplaced inserters may mis-feed your assemblers, reducing efficiency or leaving outright nonfunctional buildings. Misclicks can cripple large swaths of your previously working factory, ruining plenty of speedruns if they go uncaught. For later game megabase situations, you must deal with limited lifetimes as mining locations dry up, requiring you to overhaul existing systems with new routes of resources into them. As inputs are split and redirected, existing manufacturing can choke and sputter when they end up starved of resources. Letting your power plants starve of fuel can result in a small crisis! Electric miners mining coal, refineries turning oil into solid fuel, electric inserters fueling the boilers, water pumps providing the water to said boilers - these things all take power, and jump starting these after a power outage takes time you might not have if under active attack if your laser turrets are all offline as well. But you have means of remediating much of this unreliability. Emergency fuel and water stockpiles, configuring priorities such that fuel for power is prioritized ahead of your fancy new iron smelting setup, programmable alerts for when input stockpiles run low, ammo-turrets that work without power, burner inserters for your power production's critical path will bootstrap themselves after an outage, roboports that replace biter-attacked defenses. Your first smelting setup in Factorio will likely be a hand-fed burner miner and furnace, taking at most 50 coal. This will run out of power in minutes. Then you might use inserters to add a coal buffer. Then a belt of coal, so you don't need to constantly refill the coal buffer. Then a rail station, so you don't need to constantly hand-route entirely new coal and ore mining patches. Then you'll use blueprints and bots to automate much of constructing your new inputs. If you're really crazy, you'll experiment with automating the usage of those blueprints to build self-expanding bases...
- reilly3000 5y ago
  I really considered getting into Factorio but your comment is exactly why I can’t touch it. I have certain demands upon my time that would inevitably go unmet as I fuss with factory.

twoslide 5y ago

> when you’re on the cusp of a new thing, unless you’re super lucky, the result will necessarily be near the noise floor. Real science is like that. That's not necessarily true in social sciences. When you're working with large survey datasets, many variables are significantly related. That doesn't mean these relationships are meaningful or causal, they could be due to underlying common causes, etc. (Maybe social sciences weren't included in "real science" - but there's where a lot of stats discussions focus)

mumblemumble 5y ago

Fine. Do it like the experimental physicists do: if you think you're on to something, refine and repeat the experiment in order to get a more robust, repeatable result. The original sin of the medical and social sciences is failing to recognize a distinction between exploratory research and confirmatory research and behave accordingly.

deleted 5y ago
[deleted]
Robotbeat 5y ago
The problem is that it’s really hard to get good data, ethically, in medical sciences. Something that improves outcomes by 5-10% can be really important, but trying to get a study big enough to prove it can be super expensive already.
- TameAntelope 5y ago
  Nobody likes being in the control group of the first working anti-aging serum...
  - q-big 5y ago
    > Nobody likes being in the control group of the first working anti-aging serum... You only know whether it works when the study has been completed. You also only know whether the drug has (potentially) disastrous consequences when the study has been completed. Thus, I am not completely sure whether your claim holds.
    - bluGill 5y ago
      You missed the working part. Success was a prerequisite to their after the fact feelings. At least some of the control group will be in old age but still alive when we know it woris. They might not know if it is infinite life (and side effects may turn it into die at 85, so some control may outlive the intervention group after the study ), but they will know on average they did worse
      - q-big 5y ago
        > At least some of the control group will be in old age but still alive when we know it woris. The anti-aging serum could work (i.e. make you older), but have strong negative side effects.

mercurywells 5y ago

> I’ve been thinking about this while playing Factorio: so much of our discussion and mental modeling of automation works under the assumption of perfect reliability. If you had SLIGHTLY below 100% reliability in Factorio, the game would be a terrible grind limited to small factories. So I'm making a guess here that you play with few monsters or non-aggressive monsters?

Robotbeat 5y ago
Currently playing a game to minimize pollution to try to totally avoid biter attention. Surrounded by trees, now almost entirely solar with efficiency modules.
reificator 5y ago
> So I'm making a guess here that you play with few monsters or non-aggressive monsters? Aggressively building turret walls, defensive train lines, and so on very quickly pays dividends here. Particularly if you claim as much territory as you can each time you expand instead of simply defending what you've built out. If done this way building/improving defenses and managing enemies becomes a task you maintain every so often and doesn't spill over into the reliability of your base.

MaulingMonkey 5y ago

> If you had SLIGHTLY below 100% reliability in Factorio, the game would be a terrible grind limited to small factories. I'd argue you do have <100% reliability in Factorio, and much of the game is in increasing the 9s. Biters can wreck havok on your base. Miners contaminate your belts with the wrong types of ore, if you weren't paying enough attention near overlapping fields. Misplaced inserters may mis-feed your assemblers, reducing efficiency or leaving outright nonfunctional buildings. Misclicks can cripple large swaths of your previously working factory, ruining plenty of speedruns if they go uncaught. For later game megabase situations, you must deal with limited lifetimes as mining locations dry up, requiring you to overhaul existing systems with new routes of resources into them. As inputs are split and redirected, existing manufacturing can choke and sputter when they end up starved of resources. Letting your power plants starve of fuel can result in a small crisis! Electric miners mining coal, refineries turning oil into solid fuel, electric inserters fueling the boilers, water pumps providing the water to said boilers - these things all take power, and jump starting these after a power outage takes time you might not have if under active attack if your laser turrets are all offline as well. But you have means of remediating much of this unreliability. Emergency fuel and water stockpiles, configuring priorities such that fuel for power is prioritized ahead of your fancy new iron smelting setup, programmable alerts for when input stockpiles run low, ammo-turrets that work without power, burner inserters for your power production's critical path will bootstrap themselves after an outage, roboports that replace biter-attacked defenses. Your first smelting setup in Factorio will likely be a hand-fed burner miner and furnace, taking at most 50 coal. This will run out of power in minutes. Then you might use inserters to add a coal buffer. Then a belt of coal, so you don't need to constantly refill the coal buffer. Then a rail station, so you don't need to constantly hand-route entirely new coal and ore mining patches. Then you'll use blueprints and bots to automate much of constructing your new inputs. If you're really crazy, you'll experiment with automating the usage of those blueprints to build self-expanding bases...

reilly3000 5y ago
I really considered getting into Factorio but your comment is exactly why I can’t touch it. I have certain demands upon my time that would inevitably go unmet as I fuss with factory.