5 ms·

Don't overcomplicate it. Xargs and curl is often enough to drive big, ad hoc jobs.

kristopolous 6y ago
These things only have a handful of customers in the world. Datasets that are tens of gigabytes, or maybe 100mil records or so...this really covers most things. And for every 1 thing it doesn't, there's 20 more claimed that a single machine using simple tools could handle just fine. Being able to detect when things have been processed, have a way to set dirty flags, prioritize things, have regions of interest, be able to have re-entrant processing, caching parts of the pipeline and having nuanced rules for invalidating it, these in my mind are kinda basic things here. When they aren't done, sure, someone will need giant resources because they're doing foolish things. But that's literally the only reason. Substituting money for sense is an old hack.
- logicchains 6y ago
  A nice paper about that idea: https://www.usenix.org/system/files/conference/hotos15/hotos15-paper-mcsherry.pdf https://www.usenix.org/system/files/conference/hotos15/hotos...
  - kristopolous 6y ago
    This is like an infoworld quality article in LaTeX. I don't know why this is beneficial. The LaTeX format scares me away most of the time. I think "oh god, I don't have all afternoon to study an academic document!" This thing is a 3 minute read
- necovek 6y ago
  Opening your mirrorless camera SD card with images and the image thumbnailer takes... Forever? Doing a facial search on it? Matching a rhythm picked up by the mic to your local music collection? Hashing and/or encryption of data. There's plenty of desktop-like use cases that would benefit from massively parallel computation, but network (or even IO) bandwidth is currently going to be the limiting factor. IOW, we are not there yet! Currently, we can parallelize tasks which are low on data and high on computation. So how can we expand the IO bandwidth for everyone, even desktop or mobile users?
  - tlarkworthy 6y ago
    Why is your image and music collection not in the cloud? Definitely an xargs + curl to serverless job in a modern setup (if auth was easier)
    - necovek 6y ago
      Privacy and control. My email is not in the cloud either. Still, as you note yourself with "if auth was easier", we'd need custom applications even for the cloud — it's just that you'd hope they provide unbounded bandwidth for each user, but I am not even sure that's the case for the biggest of players (dropbox, google drive...).

These things only have a handful of customers in the world. Datasets that are tens of gigabytes, or maybe 100mil records or so...this really covers most things. And for every 1 thing it doesn't, there's 20 more claimed that a single machine using simple tools could handle just fine. Being able to detect when things have been processed, have a way to set dirty flags, prioritize things, have regions of interest, be able to have re-entrant processing, caching parts of the pipeline and having nuanced rules for invalidating it, these in my mind are kinda basic things here. When they aren't done, sure, someone will need giant resources because they're doing foolish things. But that's literally the only reason. Substituting money for sense is an old hack.

logicchains 6y ago
A nice paper about that idea: https://www.usenix.org/system/files/conference/hotos15/hotos15-paper-mcsherry.pdf https://www.usenix.org/system/files/conference/hotos15/hotos...
- kristopolous 6y ago
  This is like an infoworld quality article in LaTeX. I don't know why this is beneficial. The LaTeX format scares me away most of the time. I think "oh god, I don't have all afternoon to study an academic document!" This thing is a 3 minute read
necovek 6y ago
Opening your mirrorless camera SD card with images and the image thumbnailer takes... Forever? Doing a facial search on it? Matching a rhythm picked up by the mic to your local music collection? Hashing and/or encryption of data. There's plenty of desktop-like use cases that would benefit from massively parallel computation, but network (or even IO) bandwidth is currently going to be the limiting factor. IOW, we are not there yet! Currently, we can parallelize tasks which are low on data and high on computation. So how can we expand the IO bandwidth for everyone, even desktop or mobile users?
- tlarkworthy 6y ago
  Why is your image and music collection not in the cloud? Definitely an xargs + curl to serverless job in a modern setup (if auth was easier)
  - necovek 6y ago
    Privacy and control. My email is not in the cloud either. Still, as you note yourself with "if auth was easier", we'd need custom applications even for the cloud — it's just that you'd hope they provide unbounded bandwidth for each user, but I am not even sure that's the case for the biggest of players (dropbox, google drive...).