5 ms·

"Anybody who wants more complex and subtle filesystem interfaces is just crazy. Not only will they never get used, they'll definitely not be stable." I think

• sillysaurusx 7y ago
  I am currently trying to wring maximum performance from Cloud TPUs. This comment really resonated with me, because Cloud TPUs are complex, subtle, overly complicated, and have twenty opaque ways that you can ruin your performance. Compare that with a GPU. It took decades, but (at least for nvidia GPUs) they have finally reached the point where you can be at least reasonably certain that high-performance is the default. Sure, you can ruin your performance by doing some stupid things, but it's at least harder to do those stupid things. The defaults are high-performance. It's exactly the opposite for Cloud TPUs. By default, it only uses one core. You have to use their TPUEstimator API, which is a byzantine mess. Since two weeks ago, I've spent roughly 3 solid working days of effort solely trying to read and understand (a) what is the TPUEstimator actually doing? (b) why is it supposedly so much faster? I have some half-hearted justifications -- the answer seems to be "you have to colocate your gradients with the device; you have to scope your tensorflow graph to a specific device; and each TPU core is a separate device." But there are unanswered questions. For example, you're supposed to pass your tensorflow computation to tpu.rewrite(). Yet I've never done that. The defaults seem to just work. So does that mean it does the rewriting for me automatically? Is it being emulated in software, and I'm ruining my performance? Tensorflow, why don't you just crash instead of being so damn slow on TPUs? That would at least let me aim my optimizations! If only I had low level access to the actual TPU operations, I could just write a compiler that specifically emits instructions to give me the performance I need. But this tensorflow graph abstraction makes everything "easy" yet exponentially more complicated. Anyway. Yes. More of your mindset, please. Simplicity is such a lovely metric seldom optimized for. (Anyone who's curious can see a dramatic 11x difference in performance on GPUs vs TPUs in Colab: https://twitter.com/theshawwn/status/1196593451174891520 https://twitter.com/theshawwn/status/1196593451174891520 ... this was very surprising, since Google's marketing would have you believe that TPUs are the bee's knees. Yet by default a TPUv2 is 11x slower than a K80 GPU for some basic matrix multiplications.)
  • usmannk 7y ago
    Anecdotally, I have seen 50-100x (seriously) increases when moving from a V100 to TPUv3. To be fair, some of that is batch size increase. This was using the models in https://github.com/tensorflow/tpu/tree/master/models/official/ https://github.com/tensorflow/tpu/tree/master/models/officia.... On the other hand, a lot of those models are broken in some way and need fixing before running, caveat emptor.
    • sillysaurusx 7y ago
      Can you be more specific? I have seen many such claims, and every time I try to reproduce the results, there always seems to be some catch. We're trying to use TPUs to fine tune GPT-2 1.5B. The model takes up 5.8GB memory, which is well over half of a TPUv2 core (8GB). It always OOMs when I try to do a training step, due to the gradient calculations requiring memory. It even OOMs on a TPUv3, which has 16GB per core. I've tried using bfloat16 (which ought to cut memory usage in half) and using Ada optimizer (which should be no more expensive than plain old SGD). Yet if I colocate the gradients to the same core as the model, I always OOM. (Colocation just means "don't use any memory except the memory physically on this one core.") With colocation off, I don't OOM, and I do see some speed gains using all 8 cores. But it's no more than a factor of 2x, and in fact closer to like 1.15x (i.e. it's roughly equivalent to just using larger batch sizes on a single core). And I don't understand why I'd OOM in the TPUv3 case; even with float32, the model is only using 5.8GB out of 16GB. Are gradient computations really taking up more than 10GB for the optimizer? (That leads to https://github.com/cybertronai/gradient-checkpointing https://github.com/cybertronai/gradient-checkpointing and such, but I haven't tried it yet.) If I try the same experiment with a much smaller model (117M, or about 13x smaller), I can successfully colocate the gradients onto the same core as the model. And when I use all eight cores, I'm able to get 1225 tokens/sec (roughly 1 example per second, since 1 example = 1024 tokens for GPT-2), vs the standard case of around 400 tokens/sec when using only one core. But that's still "only" a 3x speedup. So when I see "50-100x increases," alarm bells start going off. I'm missing something fundamental here. Either you are getting 100x speedups, or I am somehow missing something fundamental. People have even started asking me for answers regarding the TPU case, and I'm forced to be like "Yeah! I expected TPUs to be so much faster too. Everyone says they're getting 100x speed gains. Yet we're 11x slower than the GPU case, and here's a notebook showing a 11x slowdown." https://github.com/shawwn/gpt-2/issues/5 https://github.com/shawwn/gpt-2/issues/5 I'm suspecting that memory bandwidth is the bottleneck here for large models. This paper even pretty much says "GPUs are more flexible and faster when memory bandwidth is an issue": https://twitter.com/mosicr/status/1196749286815481856 https://twitter.com/mosicr/status/1196749286815481856 The closest I've come to finding an actual example of a speedup to aim for is this: https://github.com/imcaspar/gpt2-ml https://github.com/imcaspar/gpt2-ml They used a TPUv3-512 pod to train a GPT-2 1.5B model to 99k steps in 50 hours. If you work out the math, that's about 1 example per second. We're getting about 0.08 examples per second on a single TPUv2 core. So yes, it's a big speedup (12.5x) but certainly not 50-100x. Yet it has 64x the cores as my TPUv2; why isn't it 64x faster? And we're only using 1 core; why not 512x faster? I have also tried this on TPUv3-8, and we're getting about the same examples/sec, further increasing the plausibility of the theory that memory bandwidth is the bottleneck.
• shrimpx 7y ago
  I like your description because it (accidentally?) likens the state of mind of wanting to write "cool" and "magical" code with being high on drugs. In my career I've made that connection many times. This team is not interested in some engineer having an "awesome trip" through arcane abstractions, we're interested in shipping stuff that works.
• baby 7y ago
  And that’s why reading Golang is much more straightforward than Rust which is way more verbose/flexible.
• pizzaparty2 7y ago
  You really like leaving in commented code? I used to prefer that too but then at work we had a policy of taking out unused code. I actually think it's cleaner so I've been doing it in my own code and rarely regret deleting something. But when code (my own or someone else's) is more chaotic I do find it to be useful to leave bits and pieces lying around.
  • jrumbut 7y ago
    I think this is about the rare cases where you have multiple competing implementations that become better or worse depending on small changes elsewhere or when a simple initial implementation can serve as documentation for a carefully optimized subsequent implementation. I tend to think there is always a better way, say creating a module that contains both implementations as alternatives, but if this is the worst wart on the codebase it is probably a good codebase. Just leaving giant blocks of commented out, trivial to recreate code as the system evolves is annoying. But I do think that the rule of not leaving old code in as comments is one where the bar to breaking it shouldn't be too terribly high. Personally I've gone overboard following it in the past, and all it did was hurt productivity. A few comments blocks aren't the end of the world.
    • blt 7y ago
      I would go for a comment like "a simpler naive implementation is in commit A0348C. It ran approx. 2.5x slower in 2019."
      • jlokier 7y ago
        I would do similar but include the commit timestamp and/or words from the subject to help locate the code if the commit id can't be retrieved. Rebasing and other means of moving patches (email etc, adding signed-off-by) can cause the original commit id to become invalid, and eventually unretrievable.
• eyelidlessness 7y ago
  I agree completely don't do magic, keep it clean and simple. I disagree with leave the thing you replaced commented. It's not just that you can find it "in the repo", it's that things drift over time. Prose comments have that reputation but code in comments that isn't being maintained drift even further. No one even has any intention of maintaining commented code except if it happens to hit find and replace, and even then good luck.
  • zer00eyz 7y ago
    > code in comments that isn't being maintained drift even further. This is also true of "its in the repo" and that doesn't get hit by "find and replace". I don't think it should be done for EVERY case, rather the cases where clarity is replaced with something that is ambiguous but meets another need. Performance hacks are notorious for being "ugly" and "magical" and having a readable companion piece would make sense. To clarify I don't think it replaces "prose" but should exist along side it, and be mixed into it.
    • eyelidlessness 7y ago
      > This is also true of "its in the repo" and that doesn't get hit by "find and replace". It's very much not! The thing "in the repo" is of exactly the same vintage as the thing in the next commit. The thing "in the comments" is preserved in amber next to a growing thing that no longer matches it. Inevitably to understand the thing "in the comments" you have to go back to commits of the same age.
    • eyelidlessness 7y ago
      To add to my earlier response: if I needed to achieve the performance critical cleverness required solution and felt it needed a human readable code sidekick, I would include both as real running code alongside each other, both of which running the same set of tests, and both of which required to be run by whatever CI/automated tests are used by the project. Comments are for humans to read, not machines.
      • 7y ago
• gugagore 7y ago
  The clean and readable version can be kept, but not in comments. Why not keep it as a reference implementation, and test the new thing against it?
• Gene_Parmesan 7y ago
  The more time I spend as a dev, the more I realize that writing clear, simple, straight forward code is actually the greater challenge. When I started progressing beyond learning the basics, the sort of projects I was building were quite simple, so writing the simple code for them came to feel boring. I would read complex codebases and see all the fascinating tricks they employed and wished I was writing code like that. It felt like, "Those are the smart programmers. I should emulate them." As the complexity of the projects I work on has increased (especially in the time since I became a part of a professional dev team), I've come to realize that most of that tricky, complicated-looking code I had read years earlier was actually kind of the easy way out. When you are just trying to get stuff done, the complexity of your code mirrors the complexity of the project. There's no effort put in (and often no time to do so) to create a smoother interface to shelter the code from the complexity of the task at hand. It's much more interesting to me now to be faced with a complex task and to figure out how to make the code simple and clear. Elegant, well-designed libraries/APIs belie the challenge in writing them. The code looks so simple that it feels immediately obvious as you're reading it. I've come to realize that reading a code base that seems dead simple -- so simple anyone could instantly understand it -- is actually often inversely related to the difficulty in creating it. Yes, necessary complexity does exist, and sometimes there's no way around doing something fairly nasty in your code. But as an ideal to strive for, I find 'simple' to be fascinating.
  • Const-me 7y ago
    > It felt like, "Those are the smart programmers. I should emulate them." That’s probably the main reason why so many C++ codebases are unnecessarily complicated. People look at what standard library developers did, and emulating it. The complexity of the standard library is actually justified: containers need to scale from 1 to 1E9 items and more, algorithms need to work with whatever broken classes users throw at them, the whole standard library needs to be extremely generic. None of these requirements apply to code of most applications, yet I have seen many C++ projects where people designed their code the same way the standard library is designed, without thinking about the reasons. The result is template heavy code which takes long time to compile, and hard to modify or debug.
  • sojmq 7y ago
• smolder 7y ago
  I don't think you can judge code just by how readable it is, and you can't frown on what seems cool and magical all the time. Sometimes "clean and readable" means that complexity is pushed outwards onto dependencies or you've simply created an inelegant solution that doesn't consider the whole problem domain. A perfect C++ program that outperforms a simple looking python program which does the same thing isn't worse because it's harder to read or to write, or better because it's faster, it's just different. If you need to optimize for readability, so be it. If you need to optimize for performance, so be it. If you write code executed trillions of times in a trillion places, where power use and the expense of hardware is dependent on the performance of the software, I think it's okay for it to be as complex as it needs to be to be very optimal, at the expense of readability and other things. It should be well annotated, sure. That's always a reasonable thing to expect except in rare, extreme cases where you don't even have the time to type out the comments. It's important to weigh the relative importance of simplicity vs speed vs memory frugality vs portability vs maintenance difficulty (dependencies and tool choice factor in here) and so on to determine the best approach. Dogma about how best to code in a general sense is only helpful if you don't want to think or talk about what's the best fit.
• ken 7y ago
  The problem is that everybody agrees on that "universal truism" but nobody can agree on what it means. "Clear, simple, straight forward, plain as day" is up to the reader. They're also a moving target. To some people, C is simple. To other people, C is full of "complex and subtle" sources of pain. To some people, a VM with a GC and a JIT is "complex and subtle". To other people, using a HLL with those features enables them to write programs which are much simpler and clearer. These positions also change over time. A JIT is a fairly standard technique in 2019, but in 1973 it was still very much in the research phase. Even the idea of writing a kernel in such a HLL as C was once revolutionary. I have an SDET friend who calls these types of universal truisms "apple pie", as in, you hold a meeting and say "Apple pie is good, right?" and everybody nods and says "Yes!" and then goes back to work and absolutely nothing was learned or decided.