3 ms·

Its a huge deal. I'm biased though because my own takes on how things should evolve were very similar. I was however completely unsuccessful in getting those id

Its a huge deal. I'm biased though because my own takes on how things should evolve were very similar. I was however completely unsuccessful in getting those ideas into production! And that, that is a huge deal. Through out my career it has been interesting to meet people with great ideas and then they are unable to get them into production, and when the idea does come into production everyone feels like "Wow, this is so obvious why didn't we do it sooner?" and some folks are banging their head against the wall :-). One of the more interesting discussions I had during my tenure at Google was about the "size" of the unit of clusters. If you toured Google you got the whole "millions of cheap replaceable computers" mantra. Sitting in Building 42 was a "rack" which had cheap PC motherboards on "pizza dishes" without all that superfluous sheet metal. Bunches of these in a rack and a festoon of network cables. What are the "first class" elements of these machines? Compute? Networking? Storage? Did you replace components? Or a whole "pizza slice" (which Google called an 'index' at the time). Really a great systems analysis problem. FWIW I'm more of a "chunk" guy (which is the direction 0xide went) and less of a "cluster" guy (which is the way Google organized their infrastructure). A lot of people associated with 0xide are folks I worked with at Sun in the early days and during that period the first hints of "beowulf" clusters vs "super computers", was memory one thing (UMA) or did it vary from place to place (NUMA). I have a paper I wrote from that time about "compute viscosity" where the effective compute rate (which at the time largely focused on transactional databases) scaled up with resource (more memory more transactions/sec for example) and scaled down with viscosity (higher latencies to get to state meant fewer transactions/sec) Sun was invested heavily in the TPC-C benchmarks at the time but they were just one load pattern one could optimize for. These guys have capitalized on all that history and it is fucking amazing! I just hope they don't get killed by acquisition[1]. [1] KbA is a technique where people who are invested in the status quo and have resources available use those resources to force the investors in a disruptive technology to sell to them and then they quietly bury the disruptive technology.

mgh95 3y ago
Can you clarify a bit on what you mean by "chunk" guy? Are you alluding to the ability to distribute work by an isolation mechanism such as cgroups vs machine a-la borg/google?
- ChuckMcM 3y ago
  More on infrastructure composition software is an abstraction above that. Is the unit of composition a rack (chunk), a server (smaller chunk), or a blade (smallest chunk)? In what I think of as classic systems architecture you've got a 'store' (storage), 'state' (memory), 'threads' (computation), and 'interconnect' (fabrics). In the 90's a lot of folks focused on fabrics (Cray, Connection Machine, Sun, etc) somewhat on threads (compute blades), and state came along for the ride. How these systems were composed was always a big thing, then along came the first Beowulf clusters that used off the shelf motherboards (a "chunk" of threads/state/store) with a generic fabric (Ethernet). Originally NASA showed that you could do some highly parallel processing on these sorts of systems and Larry and Sergei at Stanford applied it to the process of internet search. Collectively you have a 'system resource' and with software you can make it look like anything you want. When you do compute with it, its performance becomes a function of its systems balance and the demands of the workload. Its all computer sciencey and yes there is a calculus to it. This isn't something that most people dive into (or are even interested in[1]) but it was one of the things that captured my imagination early on as an engineer. I was consumed with questions like what was the difference between a microprocessor, a mini-computer, a workstation, and a mainframe? Why do they each exist? What does one do that they other can't? Things like that. [1] At Google I worked in what they called 'Platforms' early on and clearly most of the company didn't really care about the ins and outs of the systems bigtable/gfs/spanner/etc ran on, they just wanted APIs to call. But they also didn't care about utilization or costs. By the time I left some folks had just figured out (and one guy was building his career on) the fact that utilization directly affected operational costs. They still hadn't started thinking about non-uniform rack configurations for different workloads.
TheRealDunkirk 3y ago
> I just hope they don't get killed by acquisition. Private equity in the US has collectively determined that no company shall exist outside of investment ownership. I don't know what the ownership structure looks like, but generally speaking, it seems that nearly everyone has a "fuck you" number. Now that Oxide is venturing into Dell and HP's turf, I worry someone will get a fix on what Brian's number is.

mgh95 3y ago

Can you clarify a bit on what you mean by "chunk" guy? Are you alluding to the ability to distribute work by an isolation mechanism such as cgroups vs machine a-la borg/google?

ChuckMcM 3y ago
More on infrastructure composition software is an abstraction above that. Is the unit of composition a rack (chunk), a server (smaller chunk), or a blade (smallest chunk)? In what I think of as classic systems architecture you've got a 'store' (storage), 'state' (memory), 'threads' (computation), and 'interconnect' (fabrics). In the 90's a lot of folks focused on fabrics (Cray, Connection Machine, Sun, etc) somewhat on threads (compute blades), and state came along for the ride. How these systems were composed was always a big thing, then along came the first Beowulf clusters that used off the shelf motherboards (a "chunk" of threads/state/store) with a generic fabric (Ethernet). Originally NASA showed that you could do some highly parallel processing on these sorts of systems and Larry and Sergei at Stanford applied it to the process of internet search. Collectively you have a 'system resource' and with software you can make it look like anything you want. When you do compute with it, its performance becomes a function of its systems balance and the demands of the workload. Its all computer sciencey and yes there is a calculus to it. This isn't something that most people dive into (or are even interested in[1]) but it was one of the things that captured my imagination early on as an engineer. I was consumed with questions like what was the difference between a microprocessor, a mini-computer, a workstation, and a mainframe? Why do they each exist? What does one do that they other can't? Things like that. [1] At Google I worked in what they called 'Platforms' early on and clearly most of the company didn't really care about the ins and outs of the systems bigtable/gfs/spanner/etc ran on, they just wanted APIs to call. But they also didn't care about utilization or costs. By the time I left some folks had just figured out (and one guy was building his career on) the fact that utilization directly affected operational costs. They still hadn't started thinking about non-uniform rack configurations for different workloads.

TheRealDunkirk 3y ago

> I just hope they don't get killed by acquisition. Private equity in the US has collectively determined that no company shall exist outside of investment ownership. I don't know what the ownership structure looks like, but generally speaking, it seems that nearly everyone has a "fuck you" number. Now that Oxide is venturing into Dell and HP's turf, I worry someone will get a fix on what Brian's number is.