4 ms·

I think it's a question of supply and demand. If professionals need throughput, they will buy a couple of hard drives and array them together. Latency is gener

I think it's a question of supply and demand. If professionals need throughput, they will buy a couple of hard drives and array them together. Latency is generally not requested.

notacoward 8y ago
> a couple of hard drives > Latency is generally not requested. That might be your experience, but it's not mine. I'm on a team that runs one of the three or four largest storage infrastructures in the world. In aggregate, those few account for a significant fraction of all the disk drives sold - enough that we have to account for possible market distortions in our planning. So I think our experience is relevant too. In my world, the professionals very much do care about latency. Yes, the systems are built for massive throughput, but that throughput has to be within a certain latency. If latency goes too high, we get calls in the middle of the night. To keep that system-wide latency low, we deploy many racks' worth of equipment at a time, or shift load between similarly sized parts of the system. For us and our peers, this capacity/performance gap is a huge issue. Has been for years, and it keeps getting worse. Those concerns might not apply to everyone, notably you, but they are quite real.
- bunnycorn 8y ago
  And you lack reading comprehension. I didn't say professionals don't care about latency. I said that professionals don't ask for latency for hard drives, they go straight to SSDs when latency is important.
  - notacoward 8y ago
    Reading comprehension doesn't cover you retconning what you said. You didn't mention SSDs at all in your previous comment. What you actually said did imply that professionals don't care about latency. Full stop. Perhaps that's not what you meant to say, but that has more to do with your writing than anyone else's reading. More to the point, your gaslight version is still wrong. Professionals do have latency expectations even with hard disks. Those requirements might not be as stringent for hard disks as for flash (or for distributed vs. local) but they're very much still there. If a significant fraction of users' requests are taking too long, they do complain. Loudly. Data scientists don't care if the system is simultaneously delivering dozens of gigabytes per second to other users, and doing ten kinds of background maintenance stuff besides. They care that their job is slow. When you're dealing with truly large amounts of data being processed by thousands of machines, and ad hoc queries none of which are likely to be repeated, "just use flash" isn't an answer. Maybe you're dealing with a different kind of professional than I am. That's fine, but you shouldn't keep making these super-general statements that are wrong for one of the largest classes of data professionals. "Couple of hard drives" was so far off the mark I literally laughed out loud.
    - bunnycorn 8y ago
      No, I didn't. It was your problem you misinterpreted what I said. Nobody else misunderstood me but you.
      - notacoward 8y ago
        > Nobody else misunderstood me but you. I think it would be more correct to say that nobody at all misunderstood you, and nobody else bothered to correct your claims. I do see there were some downvotes, though, which suggests what others thought of them.
        bunnycorn 8y ago
        There was one downvote: yours.
        notacoward 8y ago
        Nope, wasn't me.

I think it's a question of supply and demand. If professionals need throughput, they will buy a couple of hard drives and array them together. Latency is generally not requested.

notacoward 8y ago

> a couple of hard drives > Latency is generally not requested. That might be your experience, but it's not mine. I'm on a team that runs one of the three or four largest storage infrastructures in the world. In aggregate, those few account for a significant fraction of all the disk drives sold - enough that we have to account for possible market distortions in our planning. So I think our experience is relevant too. In my world, the professionals very much do care about latency. Yes, the systems are built for massive throughput, but that throughput has to be within a certain latency. If latency goes too high, we get calls in the middle of the night. To keep that system-wide latency low, we deploy many racks' worth of equipment at a time, or shift load between similarly sized parts of the system. For us and our peers, this capacity/performance gap is a huge issue. Has been for years, and it keeps getting worse. Those concerns might not apply to everyone, notably you, but they are quite real.

bunnycorn 8y ago
And you lack reading comprehension. I didn't say professionals don't care about latency. I said that professionals don't ask for latency for hard drives, they go straight to SSDs when latency is important.
- notacoward 8y ago
  Reading comprehension doesn't cover you retconning what you said. You didn't mention SSDs at all in your previous comment. What you actually said did imply that professionals don't care about latency. Full stop. Perhaps that's not what you meant to say, but that has more to do with your writing than anyone else's reading. More to the point, your gaslight version is still wrong. Professionals do have latency expectations even with hard disks. Those requirements might not be as stringent for hard disks as for flash (or for distributed vs. local) but they're very much still there. If a significant fraction of users' requests are taking too long, they do complain. Loudly. Data scientists don't care if the system is simultaneously delivering dozens of gigabytes per second to other users, and doing ten kinds of background maintenance stuff besides. They care that their job is slow. When you're dealing with truly large amounts of data being processed by thousands of machines, and ad hoc queries none of which are likely to be repeated, "just use flash" isn't an answer. Maybe you're dealing with a different kind of professional than I am. That's fine, but you shouldn't keep making these super-general statements that are wrong for one of the largest classes of data professionals. "Couple of hard drives" was so far off the mark I literally laughed out loud.
  - bunnycorn 8y ago
    No, I didn't. It was your problem you misinterpreted what I said. Nobody else misunderstood me but you.
    - notacoward 8y ago
      > Nobody else misunderstood me but you. I think it would be more correct to say that nobody at all misunderstood you, and nobody else bothered to correct your claims. I do see there were some downvotes, though, which suggests what others thought of them.
      - bunnycorn 8y ago
        There was one downvote: yours.
        notacoward 8y ago
        Nope, wasn't me.