12 ms·

Chip advances require fresh ideas, speakers say

91 points by bcaulfield 9y ago

baybal2 9y ago
My prognosis as bit of an insider popping in and out of Shenzhen. All guns are pointed at _memory_ Memory is an uncompetitive industry, a cash cow unseen in history, comparable only to oil. The SEL empire is built not on top of galaxy notes, but on a pile of memory chips. The easiest way to get an order of magnitude improvement right away is to put more memory on die and closer to execution units and eliminate the I/O bottleneck, but no mem co. will sell you the memory secret sauce. Not only that memory is made on proprietary equipment, but decades of research were made entirely behind closed doors of Hynix/SEL/Micron triopoly hydra, unlike in the wider semi community where even Intel's process gets leaks out a bit in their research papers. SEL makes a lot of money not only selling you the well known rectangular pieces, but also effectively forces all top tier players buying their fancy interface IP if they want to jump on the bandwagon of the next DDR generation earlier than others: https://www.design-reuse.com/samsung/ddr-phy-c-342/ https://www.design-reuse.com/samsung/ddr-phy-c-342/ . This makes them want to keep the memory chip a separate piece even more. Many companies tried to break the cabal, or workaround them, but with no results. Even Apple's only way around this was just to put a whopping 13 megs of SRAM on die. Changing the classical Von Neuman style CPU for GPU or the trendy neural net streaming processor changes little when it comes to _hardware getting progressively worse_ at running synchronous algorithms because of memory starvation. You see, the first gen Google TPU is rumored to have the severest memory starvation problem, as do embedded GPUs without steroid pumped memory busses of gaming grade hardware. When PS3 came out, outstanding benchmark results on typical PC benchmark tasks were wrongly attributed to it having 8 dsp cores, while they were not used in any way. It was all due to it reverting back to more skinny, and synchronous operation friendlier memory. The amazing SPU performance was all thanks to that too. DSP style loads benefitted enormously from nearly synchronous memory behaviour.
- conradev 9y ago
  Yeah! Here is one example of a company working on that: > The REX Neo architecture gains its performance and efficiency improvements with a reexamining of the on chip memory system, but retains general programmability with breakthrough software tools. https://insidehpc.com/2017/02/rex-neo-energy-efficient-new-processor-architecture/ https://insidehpc.com/2017/02/rex-neo-energy-efficient-new-p... (check out the linked video)
  - deepnotderp 9y ago
    We at Vathys are as well for deep learning, check out our Stanford EE380 talk for more: http://web.stanford.edu/class/ee380/Abstracts/171206.html http://web.stanford.edu/class/ee380/Abstracts/171206.html
- pkaye 9y ago
  What do you mean the secret sauce for memory? And who is prevent whom from using it? Certainly Intel, AMD, and Nvidia should have access to the technology. Also I'm not an expert but there are complexities for mixing DRAM and CPU design in the same die.
  - slededit 9y ago
    DRAM processes do not lend themselves well to high speed. But FPGAs show that low speed with wide busses can still accomplish a lot. However if you have a great idea for logic and dram on the same die you are unlikely to be able to do so economically since you can't get the dram IP.
  - deepnotderp 9y ago
    DRAM processes use slower, higher capacitance, lower leakage RCAT transistors. Not good for logic, but who cares when logic is only ~20% of your die? The other mammoth problem however, is scaling the deep trench capacitors.
- jacksmith21006 9y ago
  Can you provide a source on the memory starvation for the TPUs? Plus are we talking generation 1 or 2 or both? Did not seem to be an issue in the paper with the first generation. https://cloud.google.com/blog/big-data/2017/05/an-in-depth-look-at-googles-first-tensor-processing-unit-tpu https://cloud.google.com/blog/big-data/2017/05/an-in-depth-l... An in-depth look at Google's first Tensor Processing Unit (TPU ...
  - baybal2 9y ago
    This is what was intensively discussed by first trial users, and later admitted by google tpu team themselves on hot chips 29 Even with monsterous HBM2 memory, they still have it. It is probably hard to predict what matrix set to prefetch when you deal with a neural net. So you have cache misses there too
  - pjscott 9y ago
    In the paper on the first-generation TPU, in section 7. They estimate that there would have been impressive gains in speed, both absolute and per-watt, if they'd had enough design time to give it more memory bandwidth: https://drive.google.com/file/d/0Bx4hafXDDq2EMzRNcy1vSUxtcEk/view https://drive.google.com/file/d/0Bx4hafXDDq2EMzRNcy1vSUxtcEk...
- milcron 9y ago
  What is SEL?
  - breck 9y ago
    Samsung
  - Nition 9y ago
    With the mention of the Galaxy Note I'm assuming Samsung Electronics.
    - milcron 9y ago
      Ah, EL from ELectronics. Search was turning up nothing.
- matt4077 9y ago
  Your arguments mixes up memory as in DRAM, where the market structure is as described, and memory as in CPU cache, which is something entirely different. If Intel wants to add more cache, they simply paste the template again. What's limiting on-die caches is the competition for space. Chip yields sink roughly proportional to die size.
  - hoopladler 9y ago
    What's the reason why they can't make bigger chips? More costly?
    - matt4077 9y ago
      Yields of functioning chips quickly drop off when you increase the size, because each single transistor has an independent risk of error. Individual error rates are extremely small, but so the number of possibly failing components is large.
      - gizmo686 9y ago
        You don't need every transistor to work though. If you can detect when a module has a broken transistor, you can simply disable that module and sell the rest of the chip. Divide the cache into multiple small modules, and it is not a big deal if you have to deactivate one because of a broken transistor. You would probably be deactivating it anyway for market segmentation.
- Nokinside 9y ago
  David Patterson already figured out that separation of the memory and the processor was the real bottleneck in 1996-97. His IRAM designs for general-purpose computer systems that integrate a processor and DRAM onto a single chip were done in The Berkeley Intelligent RAM (IRAM) Project http://iram.cs.berkeley.edu/ http://iram.cs.berkeley.edu/ Probably didn't work out for the reasons you mention.
  - pretty_dumm_guy 9y ago
    Can you please tell me what is stopping big companies like INTEL and AMD from implementing a processor like IRAM project does? I am really curious to know more about this.
    - Nokinside 9y ago
      Completely different process technology. When Intel or AMD add memory into their chips it's static RAM (SRAM). SRAM cells are large, expansive and take lots of space. DRAM cells are small and dense (only one-transistor, one-capacitor). Next question: What prevents Intel or AMD combining logic process and DRAM process or Samsung and others combining logic in their DRAM ships? Integrating CMOS and DRAM might be impossible to do so that the price/speed is less than using separate chips. Combining two processes increases the price. CPU/GPU makers don't have the latest DRAM knowledge. Reverse is also true. Partnership is required. Then there are technological problems: There are yield differences in different processes. CPU/GPU's operate in high temperatures. DRAM's technology needs to adjust or there needs to be halfway solution. It's possible that in some time in the future new technology called STT-MRAM could replace replace low-density DRAM and SRAM and it could be integrated into logic because it can use existing CMOS manufacturing techniques and processes. It will take time. (STT = Spin-Transfer Torque)
- nickpsecurity 9y ago
  Far as bringing CPU's and memory together, here was one of my favorite attempts that used DRAM processes: https://news.ycombinator.com/item?id=16350560 https://news.ycombinator.com/item?id=16350560
jacksmith21006 9y ago
Thought one of the more interesting ideas is the Jeff Dean paper on using the TPU for more traditional CS operations like an index lookup. Like to see this progress and get more data on the difference is power required using a CPU and traditional algorithms versus using a TPU and new algorithms like Jeff outlined. https://arxiv.org/abs/1712.01208 https://arxiv.org/abs/1712.01208 The Case for Learned Index Structures
narrator 9y ago
I think computing technology stagnating will probably one of the unexpected stories of the next decade. Look at Intel. Canon Lake (10nm) was supposed to be released in 2016 and now it's pushed back to mid-2018. Will even be able to get the yields to ship in volume at that time? A weird consequence of this will be the rest of the world will catch up in semi-conductor fab technology. Also, we are starting to see weird stuff happen with GPU/processor and memory prices going up as inflationary pressures are able to outrun chip technology advances. We might start to see commodity computer products hoarded as a store of value.
- sehugg 9y ago
  Also Meltdown/Spectre dragging down most of the incremental performance gain in the latest chip generation.
- matt4077 9y ago
  CPU performance stagnation has already been the (somewhat unexpected) story of the last decade: Computer have simply become fast enough for almost all users, so progress has mostly been invested into power saving. Advances have mostly been made in storage speed (SSDs), battery performance, and display quality (resolution, colours, and lately refresh rate), plus vector processing for machine learning.
  - InclinedPlane 9y ago
    I'd add GPUs and RAM sizes to that last. There are a huge number of computing tasks which fall into either the "throw more FPU at it" or the "throw more IO at it" categories. A lot of various techniques have been developed to optimize performance around certain bottlenecks but in the last 10 years we've gotten to a place where FPU cores cost a few cents each and thousands of them can be crammed into a single system, and where hundreds of gigabytes up to several terabytes of RAM are generally affordable and can be equipped into a single system. Going from a system that is hugely IO bound due to data living on spinning disks to a system where a huge multi-gigabyte database can just live in RAM 100% of the time and all other data is stored on SSDs is a several orders of magnitude speedup. And going from "you get one or two FPU ops per clock cycle" to "here's thousands of FPUs per clock cycle" has also translated into orders of magnitude improvements as well. Additionally, software has gotten better. Nginx is just plain much more streamlined than Apache, and simple caching techniques have really increased the amount of boom for your buck you get with hardware these days, at least in the server space.
anonytrary 9y ago
The chips of the future may be designed nothing like the chips of the past. This will call for more than just basement circuit rehashing. The future of chips will rest on novel solid-state and condensed-matter research.
gambler 9y ago
No shit. We're using a computer architecture envisioned in the late 50s. Our main processing units are made by a near-monopoly. The situation is so bad that to do any real computing we need to put another computer (GPU) inside of our computer. If you think about it, it's patently absurd. As far as grand new ideas: fleet architecture by Ivan Sutherland. As far as master-of-the-obvious ideas: make a hybrid unit that contains flash, RAM and an FPGa-based processor. Persistent by default. Connect a lot of them and get rid of disks, caches and memory bottlenecks. Do something similar to what XMOS does for peripherals (simulate them) to simplify hardware even further. You could even combine the two ideas above. Make a computer where programming isn't just about instructions, but about re-configuring hardware and information routing within the system.
blackrock 9y ago
Maybe it's time to fuse the CPU and memory together? Fuse an i7 processor with 32 GB of RAM. This should be far sufficient for all normal consumer needs, at this time. You can still allow for additional memory, but this would function as a level 2 memory which is slower to access.
- tejasmanohar 9y ago
  I'm also curious what ever happened to this. I know it's not a new idea [0] but haven't seen it in practice. [0]: http://www.theregister.co.uk/2013/11/21/intel_converging_memory_and_cpus/ http://www.theregister.co.uk/2013/11/21/intel_converging_mem...
- theandrewbailey 9y ago
  AMD has been shipping GPUs with HBM (on package RAM) for years. Recently, AMD and Intel have struck a deal to put that GPU+HBM in a CPU package. https://wccftech.com/intel-kaby-lake-g-amd-4gb-hbm-gpu-pictured-vega-11/ https://wccftech.com/intel-kaby-lake-g-amd-4gb-hbm-gpu-pictu... When HBM first came out, I mused at how much faster/more efficient a CPU+HBM chip could be. I wonder what's stopping it from happening.
tejasmanohar 9y ago
Friendly reminder that "The Free Lunch Is Over: A Fundamental Turn Toward Concurrency in Software" [0] was published in 2005 (~13 years ago). Thinking aloud here-- Is the lull in chip design due to the shift in focus to software over hardware due to the boom of internet companies circa 2000. Just compare entry level EE vs. CS available jobs or pay for example. Or, maybe it's just that, like markets, R&D also operates in cycles? Regardless, it would be interesting to see a comparison of $$$ invested in R&D on each side over the last n years. [0]: http://www.gotw.ca/publications/concurrency-ddj.htm http://www.gotw.ca/publications/concurrency-ddj.htm
mhkl 9y ago
I suggest to look at Mill Computing. They are designing a new CPU which has the power of Intel, the security of a mainframe and the power usage of Raspberry Pi.
- boznz 9y ago
  Interesting but I Don't see anything about physical hardware or timelines in the articles, can someone do a TLDR summary?
OhDagny 9y ago
Let's face it, all the efficiency gains is removing all the software bloat accumulated in the last 20 years of Moore's law free lunches. Layer upon layer upon layer of abstractions and bad code. That's the remaining low hanging fruit.

91 points by bcaulfield 9y ago

My prognosis as bit of an insider popping in and out of Shenzhen. All guns are pointed at _memory_ Memory is an uncompetitive industry, a cash cow unseen in history, comparable only to oil. The SEL empire is built not on top of galaxy notes, but on a pile of memory chips. The easiest way to get an order of magnitude improvement right away is to put more memory on die and closer to execution units and eliminate the I/O bottleneck, but no mem co. will sell you the memory secret sauce. Not only that memory is made on proprietary equipment, but decades of research were made entirely behind closed doors of Hynix/SEL/Micron triopoly hydra, unlike in the wider semi community where even Intel's process gets leaks out a bit in their research papers. SEL makes a lot of money not only selling you the well known rectangular pieces, but also effectively forces all top tier players buying their fancy interface IP if they want to jump on the bandwagon of the next DDR generation earlier than others: https://www.design-reuse.com/samsung/ddr-phy-c-342/ https://www.design-reuse.com/samsung/ddr-phy-c-342/ . This makes them want to keep the memory chip a separate piece even more. Many companies tried to break the cabal, or workaround them, but with no results. Even Apple's only way around this was just to put a whopping 13 megs of SRAM on die. Changing the classical Von Neuman style CPU for GPU or the trendy neural net streaming processor changes little when it comes to _hardware getting progressively worse_ at running synchronous algorithms because of memory starvation. You see, the first gen Google TPU is rumored to have the severest memory starvation problem, as do embedded GPUs without steroid pumped memory busses of gaming grade hardware. When PS3 came out, outstanding benchmark results on typical PC benchmark tasks were wrongly attributed to it having 8 dsp cores, while they were not used in any way. It was all due to it reverting back to more skinny, and synchronous operation friendlier memory. The amazing SPU performance was all thanks to that too. DSP style loads benefitted enormously from nearly synchronous memory behaviour.

conradev 9y ago
Yeah! Here is one example of a company working on that: > The REX Neo architecture gains its performance and efficiency improvements with a reexamining of the on chip memory system, but retains general programmability with breakthrough software tools. https://insidehpc.com/2017/02/rex-neo-energy-efficient-new-processor-architecture/ https://insidehpc.com/2017/02/rex-neo-energy-efficient-new-p... (check out the linked video)
- deepnotderp 9y ago
  We at Vathys are as well for deep learning, check out our Stanford EE380 talk for more: http://web.stanford.edu/class/ee380/Abstracts/171206.html http://web.stanford.edu/class/ee380/Abstracts/171206.html
pkaye 9y ago
What do you mean the secret sauce for memory? And who is prevent whom from using it? Certainly Intel, AMD, and Nvidia should have access to the technology. Also I'm not an expert but there are complexities for mixing DRAM and CPU design in the same die.
- slededit 9y ago
  DRAM processes do not lend themselves well to high speed. But FPGAs show that low speed with wide busses can still accomplish a lot. However if you have a great idea for logic and dram on the same die you are unlikely to be able to do so economically since you can't get the dram IP.
- deepnotderp 9y ago
  DRAM processes use slower, higher capacitance, lower leakage RCAT transistors. Not good for logic, but who cares when logic is only ~20% of your die? The other mammoth problem however, is scaling the deep trench capacitors.
jacksmith21006 9y ago
Can you provide a source on the memory starvation for the TPUs? Plus are we talking generation 1 or 2 or both? Did not seem to be an issue in the paper with the first generation. https://cloud.google.com/blog/big-data/2017/05/an-in-depth-look-at-googles-first-tensor-processing-unit-tpu https://cloud.google.com/blog/big-data/2017/05/an-in-depth-l... An in-depth look at Google's first Tensor Processing Unit (TPU ...
- baybal2 9y ago
  This is what was intensively discussed by first trial users, and later admitted by google tpu team themselves on hot chips 29 Even with monsterous HBM2 memory, they still have it. It is probably hard to predict what matrix set to prefetch when you deal with a neural net. So you have cache misses there too
- pjscott 9y ago
  In the paper on the first-generation TPU, in section 7. They estimate that there would have been impressive gains in speed, both absolute and per-watt, if they'd had enough design time to give it more memory bandwidth: https://drive.google.com/file/d/0Bx4hafXDDq2EMzRNcy1vSUxtcEk/view https://drive.google.com/file/d/0Bx4hafXDDq2EMzRNcy1vSUxtcEk...
milcron 9y ago
What is SEL?
- breck 9y ago
  Samsung
- Nition 9y ago
  With the mention of the Galaxy Note I'm assuming Samsung Electronics.
  - milcron 9y ago
    Ah, EL from ELectronics. Search was turning up nothing.
matt4077 9y ago
Your arguments mixes up memory as in DRAM, where the market structure is as described, and memory as in CPU cache, which is something entirely different. If Intel wants to add more cache, they simply paste the template again. What's limiting on-die caches is the competition for space. Chip yields sink roughly proportional to die size.
- hoopladler 9y ago
  What's the reason why they can't make bigger chips? More costly?
  - matt4077 9y ago
    Yields of functioning chips quickly drop off when you increase the size, because each single transistor has an independent risk of error. Individual error rates are extremely small, but so the number of possibly failing components is large.
    - gizmo686 9y ago
      You don't need every transistor to work though. If you can detect when a module has a broken transistor, you can simply disable that module and sell the rest of the chip. Divide the cache into multiple small modules, and it is not a big deal if you have to deactivate one because of a broken transistor. You would probably be deactivating it anyway for market segmentation.
Nokinside 9y ago
David Patterson already figured out that separation of the memory and the processor was the real bottleneck in 1996-97. His IRAM designs for general-purpose computer systems that integrate a processor and DRAM onto a single chip were done in The Berkeley Intelligent RAM (IRAM) Project http://iram.cs.berkeley.edu/ http://iram.cs.berkeley.edu/ Probably didn't work out for the reasons you mention.
- pretty_dumm_guy 9y ago
  Can you please tell me what is stopping big companies like INTEL and AMD from implementing a processor like IRAM project does? I am really curious to know more about this.
  - Nokinside 9y ago
    Completely different process technology. When Intel or AMD add memory into their chips it's static RAM (SRAM). SRAM cells are large, expansive and take lots of space. DRAM cells are small and dense (only one-transistor, one-capacitor). Next question: What prevents Intel or AMD combining logic process and DRAM process or Samsung and others combining logic in their DRAM ships? Integrating CMOS and DRAM might be impossible to do so that the price/speed is less than using separate chips. Combining two processes increases the price. CPU/GPU makers don't have the latest DRAM knowledge. Reverse is also true. Partnership is required. Then there are technological problems: There are yield differences in different processes. CPU/GPU's operate in high temperatures. DRAM's technology needs to adjust or there needs to be halfway solution. It's possible that in some time in the future new technology called STT-MRAM could replace replace low-density DRAM and SRAM and it could be integrated into logic because it can use existing CMOS manufacturing techniques and processes. It will take time. (STT = Spin-Transfer Torque)
nickpsecurity 9y ago
Far as bringing CPU's and memory together, here was one of my favorite attempts that used DRAM processes: https://news.ycombinator.com/item?id=16350560 https://news.ycombinator.com/item?id=16350560

jacksmith21006 9y ago

Thought one of the more interesting ideas is the Jeff Dean paper on using the TPU for more traditional CS operations like an index lookup. Like to see this progress and get more data on the difference is power required using a CPU and traditional algorithms versus using a TPU and new algorithms like Jeff outlined. https://arxiv.org/abs/1712.01208 https://arxiv.org/abs/1712.01208 The Case for Learned Index Structures

narrator 9y ago

I think computing technology stagnating will probably one of the unexpected stories of the next decade. Look at Intel. Canon Lake (10nm) was supposed to be released in 2016 and now it's pushed back to mid-2018. Will even be able to get the yields to ship in volume at that time? A weird consequence of this will be the rest of the world will catch up in semi-conductor fab technology. Also, we are starting to see weird stuff happen with GPU/processor and memory prices going up as inflationary pressures are able to outrun chip technology advances. We might start to see commodity computer products hoarded as a store of value.

sehugg 9y ago
Also Meltdown/Spectre dragging down most of the incremental performance gain in the latest chip generation.
matt4077 9y ago
CPU performance stagnation has already been the (somewhat unexpected) story of the last decade: Computer have simply become fast enough for almost all users, so progress has mostly been invested into power saving. Advances have mostly been made in storage speed (SSDs), battery performance, and display quality (resolution, colours, and lately refresh rate), plus vector processing for machine learning.
- InclinedPlane 9y ago
  I'd add GPUs and RAM sizes to that last. There are a huge number of computing tasks which fall into either the "throw more FPU at it" or the "throw more IO at it" categories. A lot of various techniques have been developed to optimize performance around certain bottlenecks but in the last 10 years we've gotten to a place where FPU cores cost a few cents each and thousands of them can be crammed into a single system, and where hundreds of gigabytes up to several terabytes of RAM are generally affordable and can be equipped into a single system. Going from a system that is hugely IO bound due to data living on spinning disks to a system where a huge multi-gigabyte database can just live in RAM 100% of the time and all other data is stored on SSDs is a several orders of magnitude speedup. And going from "you get one or two FPU ops per clock cycle" to "here's thousands of FPUs per clock cycle" has also translated into orders of magnitude improvements as well. Additionally, software has gotten better. Nginx is just plain much more streamlined than Apache, and simple caching techniques have really increased the amount of boom for your buck you get with hardware these days, at least in the server space.

anonytrary 9y ago

The chips of the future may be designed nothing like the chips of the past. This will call for more than just basement circuit rehashing. The future of chips will rest on novel solid-state and condensed-matter research.

gambler 9y ago

No shit. We're using a computer architecture envisioned in the late 50s. Our main processing units are made by a near-monopoly. The situation is so bad that to do any real computing we need to put another computer (GPU) inside of our computer. If you think about it, it's patently absurd. As far as grand new ideas: fleet architecture by Ivan Sutherland. As far as master-of-the-obvious ideas: make a hybrid unit that contains flash, RAM and an FPGa-based processor. Persistent by default. Connect a lot of them and get rid of disks, caches and memory bottlenecks. Do something similar to what XMOS does for peripherals (simulate them) to simplify hardware even further. You could even combine the two ideas above. Make a computer where programming isn't just about instructions, but about re-configuring hardware and information routing within the system.

blackrock 9y ago

Maybe it's time to fuse the CPU and memory together? Fuse an i7 processor with 32 GB of RAM. This should be far sufficient for all normal consumer needs, at this time. You can still allow for additional memory, but this would function as a level 2 memory which is slower to access.

tejasmanohar 9y ago
I'm also curious what ever happened to this. I know it's not a new idea [0] but haven't seen it in practice. [0]: http://www.theregister.co.uk/2013/11/21/intel_converging_memory_and_cpus/ http://www.theregister.co.uk/2013/11/21/intel_converging_mem...
theandrewbailey 9y ago
AMD has been shipping GPUs with HBM (on package RAM) for years. Recently, AMD and Intel have struck a deal to put that GPU+HBM in a CPU package. https://wccftech.com/intel-kaby-lake-g-amd-4gb-hbm-gpu-pictured-vega-11/ https://wccftech.com/intel-kaby-lake-g-amd-4gb-hbm-gpu-pictu... When HBM first came out, I mused at how much faster/more efficient a CPU+HBM chip could be. I wonder what's stopping it from happening.

tejasmanohar 9y ago

Friendly reminder that "The Free Lunch Is Over: A Fundamental Turn Toward Concurrency in Software" [0] was published in 2005 (~13 years ago). Thinking aloud here-- Is the lull in chip design due to the shift in focus to software over hardware due to the boom of internet companies circa 2000. Just compare entry level EE vs. CS available jobs or pay for example. Or, maybe it's just that, like markets, R&D also operates in cycles? Regardless, it would be interesting to see a comparison of $$$ invested in R&D on each side over the last n years. [0]: http://www.gotw.ca/publications/concurrency-ddj.htm http://www.gotw.ca/publications/concurrency-ddj.htm

mhkl 9y ago

I suggest to look at Mill Computing. They are designing a new CPU which has the power of Intel, the security of a mainframe and the power usage of Raspberry Pi.

boznz 9y ago
Interesting but I Don't see anything about physical hardware or timelines in the articles, can someone do a TLDR summary?

OhDagny 9y ago

Let's face it, all the efficiency gains is removing all the software bloat accumulated in the last 20 years of Moore's law free lunches. Layer upon layer upon layer of abstractions and bad code. That's the remaining low hanging fruit.