3 ms·

Where's the option to start firing up a Kubernetes cluster?

by kungfufrog 4y ago

analog31 4y ago
Also, how long would it actually take to sort a number of records consistent with the largest workforce today?
- 082349872349872 4y ago
  3e6 employees all of whom were born after 1768? <1s?
- bobsmooth 4y ago
  A quick Google search says Walmart has 2.3 million employees. Using python to generate 2.3 million random age and name pairs and then sort them took about 8 minutes. And I'm pretty sure most of that time was spent creating the array.
  - thrown_22 4y ago
    x = [(randint(20,80), "John Smith") for x in range(2_300_000)] x.sort(key=lambda x: x[0]) x[2_300_000//2] Takes under a second. You can expand it to generate random names rather than just ages, if you feel like it, but the end result wouldn't be much different. In short: this is something you shouldn't even think about in modern computers since it's that fast.
    - vlovich123 4y ago
      On my laptop generating it takes ~1s and sorting takes ~280ms. As always, dependent on the power of the machine running things.
      - thrown_22 4y ago
        After thinking about it for a while, the reason why it was a hard problem back then is that the payroll wouldn't fit in memory and you'd need to do a sort on disk. In this day and age it would be like trying to sort 100 TB of data. Not a hard problem, but not exactly something you do daily either. And still dreadfully slow since you need to use disk memory while copying everything over. A more modern version of the question would be: >You have a list of integers, in random order, that is 20 times larger than the hard drive you have on your work station and 1,000 times larger than your work station ram....
        vlovich123 4y ago
        Naw. Sorting is a total waste here. I can read off an "infinite" amount of records of the wire with O(1) space usage and O(n) result. Even back then. Look at my radix sort solution: https://news.ycombinator.com/item?id=32684017 https://news.ycombinator.com/item?id=32684017 The intent behind the problem in the textbook is fine - think non-linearly about the problem. Just the example used has a trivial solution not mentioned (i.e. option 5).
        thrown_22 4y ago
        Yes, that's why I changed it to having a list of arbitrary integers. You're also assuming integer ages when a more natural comparison would have date of birth. Which has 365 times the number of buckets. Something that was again problematic to fit into a 1980s memory.
    - bobsmooth 4y ago
      Haha, removing the library I used to generate random names took my code down to a few seconds.
- xelxebar 4y ago
  Dyalog APL is fast: ¯1↑5e6↑{(⊂⍋⍵)⌷⍵}10e6?⍤⍴100 This finds the median of 10 million "ages" in just under 50 milliseconds. Granted, with this randomly generated data it's probably doing something smart to avoid a full 10MB allocation, so on a static dataset the performance might be a bit different.
  - mlochbaum 4y ago
    (Former Dyalog implementor) I have 9ms for the actual median calculation, which is much faster than generating the ages. It's a counting sort on 1-byte values: since the counts are so large it can just write results with memset() or an equivalent which is vectorized. You could also get the median from the counts directly, although I'm not sure APL has a way to do those counts that's faster than just sorting.
    - pcblues 4y ago
      Like any Sandra Bullock movie, I can't tell if I have witnessed the funniest or most achingly sad thing I have ever seen. I like both.
pjscott 4y ago
Since this is from the 1980s, would a VAX cluster suffice?
- retcore 4y ago
  A high volume transaction concurrency capable routine scenario would make a good update for contemporary financial services. Delta median and tick moving average strategies eg for thought. Also, how to recalculate for cancellation, hot fill or kill and other execution strategies. The original question can get you thinking very close to contemporary problems, with only modest variations. We should have more longevity built into our textbooks.
- zeristor 4y ago
  Surely the best of both worlds would be to run Vax on Kubernetes?
  - gonzus 4y ago
    You certainly meant "run Kubernetes on Vaxen", right?

Where's the option to start firing up a Kubernetes cluster?

analog31 4y ago

Also, how long would it actually take to sort a number of records consistent with the largest workforce today?

082349872349872 4y ago
3e6 employees all of whom were born after 1768? <1s?
bobsmooth 4y ago
A quick Google search says Walmart has 2.3 million employees. Using python to generate 2.3 million random age and name pairs and then sort them took about 8 minutes. And I'm pretty sure most of that time was spent creating the array.
- thrown_22 4y ago
  x = [(randint(20,80), "John Smith") for x in range(2_300_000)] x.sort(key=lambda x: x[0]) x[2_300_000//2] Takes under a second. You can expand it to generate random names rather than just ages, if you feel like it, but the end result wouldn't be much different. In short: this is something you shouldn't even think about in modern computers since it's that fast.
  - vlovich123 4y ago
    On my laptop generating it takes ~1s and sorting takes ~280ms. As always, dependent on the power of the machine running things.
    - thrown_22 4y ago
      After thinking about it for a while, the reason why it was a hard problem back then is that the payroll wouldn't fit in memory and you'd need to do a sort on disk. In this day and age it would be like trying to sort 100 TB of data. Not a hard problem, but not exactly something you do daily either. And still dreadfully slow since you need to use disk memory while copying everything over. A more modern version of the question would be: >You have a list of integers, in random order, that is 20 times larger than the hard drive you have on your work station and 1,000 times larger than your work station ram....
      - vlovich123 4y ago
        Naw. Sorting is a total waste here. I can read off an "infinite" amount of records of the wire with O(1) space usage and O(n) result. Even back then. Look at my radix sort solution: https://news.ycombinator.com/item?id=32684017 https://news.ycombinator.com/item?id=32684017 The intent behind the problem in the textbook is fine - think non-linearly about the problem. Just the example used has a trivial solution not mentioned (i.e. option 5).
        thrown_22 4y ago
        Yes, that's why I changed it to having a list of arbitrary integers. You're also assuming integer ages when a more natural comparison would have date of birth. Which has 365 times the number of buckets. Something that was again problematic to fit into a 1980s memory.
  - bobsmooth 4y ago
    Haha, removing the library I used to generate random names took my code down to a few seconds.
xelxebar 4y ago
Dyalog APL is fast: ¯1↑5e6↑{(⊂⍋⍵)⌷⍵}10e6?⍤⍴100 This finds the median of 10 million "ages" in just under 50 milliseconds. Granted, with this randomly generated data it's probably doing something smart to avoid a full 10MB allocation, so on a static dataset the performance might be a bit different.
- mlochbaum 4y ago
  (Former Dyalog implementor) I have 9ms for the actual median calculation, which is much faster than generating the ages. It's a counting sort on 1-byte values: since the counts are so large it can just write results with memset() or an equivalent which is vectorized. You could also get the median from the counts directly, although I'm not sure APL has a way to do those counts that's faster than just sorting.
  - pcblues 4y ago
    Like any Sandra Bullock movie, I can't tell if I have witnessed the funniest or most achingly sad thing I have ever seen. I like both.

pjscott 4y ago

Since this is from the 1980s, would a VAX cluster suffice?

retcore 4y ago
A high volume transaction concurrency capable routine scenario would make a good update for contemporary financial services. Delta median and tick moving average strategies eg for thought. Also, how to recalculate for cancellation, hot fill or kill and other execution strategies. The original question can get you thinking very close to contemporary problems, with only modest variations. We should have more longevity built into our textbooks.
zeristor 4y ago
Surely the best of both worlds would be to run Vax on Kubernetes?
- gonzus 4y ago
  You certainly meant "run Kubernetes on Vaxen", right?