5 ms·

Some years ago I worked at a startup that was doing OCR on paper receipts. As part of my application to the company, I wrote a synthetic training data generator

• nomel 5y ago
  When developing anything in the real world, you don't wait for edge cases to happen naturally. You force them to happen, and look at the response. This is the rigor of engineering. Has that been lost?
  • snek_case 5y ago
    There's this dogmatic idea in the machine learning field that only real world data is valuable.
    • cinntaile 5y ago
      Data augmentation is a thing within machine learning (deep learning) so that's a big generalization.
      • snek_case 5y ago
        Data augmentation is a big thing in applied machine learning. Deep learning practitioners use data augmentation because it works incredibly well. However, deep learning researchers tend to view data augmentation as some kind of dirty trick that wouldn't be necessary if you just had a bigger dataset.
    • a_bonobo 5y ago
      I guess the field is constantly making a decision between 'do we want to outperform humans, but lose interpretability' and 'we should always be interpretable' Human-generated artificial data will always contain the human's assumptions. But this data might not contain the 'superhuman' element that leads to the ML-system outperforming humans, because we don't know what that is (yet). Receipts are a good example for artificial data, because they're human-made. But a lot of what we do with ML systems is using data that doesn't come from humans, images of wildlife, satellite images, biological data, etc
    • tsbischof 5y ago
      It's also about the issue that people will bias against generating the "difficult" data, even if only subconsciously. Real-world validation is essential to reveal whether you have actually worked through the full phase space.
      • snek_case 5y ago
        Depends. As others have said, you can be intentionally adversarial (looking for corner cases) when testing your system, in a way that real data isn't.
    • amcoastal 5y ago
      This is simply not true. Synthetic data is a huge research arm of ML work.
  • Frost1x 5y ago
    I often here "edge case" as a defense as to why something hasn't been developed or handled. There's an ongoing trend in software to avoid high hanging fruit and focus on the low hanging big and fast returns. I just accept it now although it bothers me everytime I see it happen. The beauty and allure of science and engineering, to me, is understanding something so well you can predict its future behavior and then predicting it so well you can prevent undesired behaviors and create or improve desired behaviors, within some set of bounds of course.
    • nomel 5y ago
      > I just accept it now although it bothers me everytime I see it happen. That's all fine and dandy if it has no real consequences, which is usually the case. The use cases of ML, increasingly, have very real consequences. I think this current Wild West mentality is great for greasing innovation, but not so great when a self driving car start slamming into poles at sunset, whenever there's a blinking red "Open" sign behind them.
  • zwaps 5y ago
    Well, to test the system sure. That's normal. However, the issue not quite as simple when training a model. For instance, the increasingly popular Transformer architecture pushes around internal representations when fed new training samples. These representations are not "causal parameters" - they do not represent unchanging mechanisms in real life. Instead, they depend on data. The question thus becomes: How prominent should edge cases be in the sample? A ML model trained on many edge cases may be more robust for edge cases we feed it, but it may be more unstable on real-world data (in essence, the model may have new edge cases when new data comes in that would not have been edge cases before). All of this comes from the fact that the parameter of the model are not "causal" for the underlying latent DGP. ML, in constrast to stats, does not yet have much theory regarding how to think about parameters relative to the sample of observations one trains on. The question of "causal" identification in stats is easier to tackle, because one can usually define a "scientific" or "structural" model where parameters have meaning, and so we can reason whether we capture "real, causal parameters" (which would be compatible with edge cases) or "reduced form parameters" (which would generally depend on the sample we have). ML will likely get there. Right now, people think in terms of "data shift" (which they shouldn't) and generalizing (missing the point a bit). But we will get there. By the way, this issue arose because ML systems where treated like engineering systems instead of statistical models. For better or worse.
  • Fricken 5y ago
    In autonomous vehicle development 'edge cases' are the things that engineers would never conjure up via thought experiment. They need to be discovered through real world testing. Once an edge case has been discovered, they can then artificially generate myriad subtle variations on the edge case, which they use to train their systems.
    • nomel 5y ago
      I think the point is you can augment/synthesize a video with all sorts of random lighting conditions, occlusions, etc, that could take an impossible number of miles to happen naturally. Not filling in some of the search space seems silly, to me, especially if you're instead filling it in with real life data, containing near 100% mundane near ideal conditions.
• sockpuppet69 5y ago