4 ms·

The Prospects for 128 Bit Processors ( John Mashey SGI 1995)

44 points by Alien1Being 2mo ago

A1kmm 2mo ago
So reflecting on that, I think the core assumption that didn't pan out is that the memory / CPU ratio will grow because we'll need more memory, hence requiring CPUs to address more than 16 EiB of data (16 EiB = 16384 PiB = 16777216 TiB of data, what 64 bits can address). But in practice, we've produced a lot more compute. The memory / CPU ratio has increased, but most growth has been from more CPUs (and generally not shared-memory ones, but ones with their own completely separate memory space). IPv6 is 128 bit, so we do have 128 bit ways of addressing computers, but I'd say we're still a long way from a CPU needing to address that much memory as a common case. The speed of light limits how far away memory can be from the CPU for good performance, so short of some drastically new memory technology, it seems unlikely we'll need it soon for any ordinary type of computing device.
- jsLavaGoat 2mo ago
  the cases where its useful, it's in vectorized instruction sets, etc.
  - kelnos 2mo ago
    You're conflating 128-bit registers with 128-bit memory addressing. This article is about the latter.
    - flohofwoe 2mo ago
      Also arguably we already have 256 or 512 bit CPUs, what matters most for memory throughput is not the register width but essentialy the L1 cache line width (eg what in old CPUs was the databus width). As for address width, we're not even close to get full 64 bit pointers from CPUs anyway, more like 48 or 52 bits. Wide pointers (eg 128 bit general registers) would make sense for carrying capabilities for memory safety though I guess.
- dist-epoch 2mo ago
  Because of LLMs we are back to memory being king (KV cache). But there was another thing - clusters of thousands/millions of machine instead of one big iron with all the memory.
deleted 2mo ago
[deleted]
Aardwolf 2mo ago
Personally I'm disappointed 128-bit floating point (quadruple precision) never properly made it into CPU's (sure it appeared in some niche ones here and there, but not in what we actually use today). After all, in the 1980's they had 80-bit ones, it's not even that far off, and they had millions times less transistors then. While probably niche and applications that need higher precision using their own custom types anyway, they'd allow cool stuff like easy to program fractals with much higher detail than now. But also, less precision loss in many applications.
- flohofwoe 2mo ago
  That's the thing, bigger datatypes means less effective memory throughput. From that perspective 16-bit or even 8-bit floats are often more useful than 80 or 128 bit floats. Same problem with 64 bit pointers and why it's often better to store narrower indices instead of full pointers, data can be packed more tightly and accessed more efficiently.
  - TheOtherHobbes 2mo ago
    The point about large word lengths is you get higher data throughput, and faster processing, because everything is going through fat pipes into a big parallel machine with multiple levels of cache and vectorisation. The issue is the utility of floats at different precisions. 128-bit floats have some benefits for high-end scientific applications, but the extra cost and complexity over 64-bit hardware would only make sense for specialised scientific supercomputing. So far it just hasn't been worth it.
- silvestrov 2mo ago
  precision loss in floating point is often exponential. If 64 bit isn't enough, then very quickly 128 is also not enough. If precision is important then you will very often want systems that represent every number as an interval [a, b] meaning that the true value is between those 2 numbers. This makes you able to detect loss of precision due to e.g. d = a / (b - c) where b-c can result in a number close to zero which makes uncertainty grow. If you use this formula iteratively then precision is lost completely no matter how many bits there are in your floating point variables.
kev009 2mo ago
One interesting thing with wider addresses isn't necessarily increasing itself, but features built upon the expansion. CHERI is one example of that.
- justincormack 2mo ago
  Yeah, or cryptographically secure non guessable addresses.
bArray 2mo ago
That exchange is quite a good prediction all things considered. > Note that "minor" implementation issues like die space, routing, and gate delays, especially of 128-bit adders & shifters are non-trivial, so people aren't going to rush out and build 128-bitters for fun, just as people matched timing dates of their 64-bitters to their expected markets. I think we're stuck with 64 bit for quite a while. The circuit size jump from 64 bit to 128 bit is significant. There's no fixed scalar or apples to apples comparison (that I'm aware of). Just for adders though, 2 bits requires 2 full adders, 4 bits requires 4 adders, 8 requires 8, etc. Another way to look at this is that 16 bit gives addressable memory up to 65k, and quite a few programs had to deal with paging in architectures like the 8086/8088. 32 bits gave us up to 4GB addressable memory, and not it's now not uncommon that a program such as a web browser exceeds this. 64 bits would give us up to 18 exabytes of addressable memory. I'm not aware of any common programs breaking into the terabyte category (even in most research), let alone petabyte and then exabyte. Even iterating over that many numbers becomes a large computational task. Just a quick test program: // gcc -O3 count.c -o count #include <stdint.h> int main(){ uint64_t z = 0; for(uint64_t i = 0; i < UINT64_MAX; i++) z += i; return (int)(z % 2); } Using uint32_t and UINT32_MAX, it returns almost instantly. For uint64_t and UINT64_MAX you will be waiting a long time. 128 bit values? Even longer. Maybe many many cores could break that memory up, but then it makes sense to have a 64 bit system with some kind of ability to occasionally change page.
- kalleboo 2mo ago
  > I'm not aware of any common programs breaking into the terabyte category (even in most research) Wouldn't that be the commercial LLMs? ChatGPT, Claude etc are estimated to be in the 2-10 TB range, and a large portion of the population of developed countries are using those apps commonly. Not on their own systems, but if we're in the 1995 academic perspective, multiuser systems are assumed. I imagine the strongest reason we won't need 128-bit memory addressing is because horizontal scaling is easier. If OpenAI had needed a single addressing plane to cover all their users, 128-bit might be required. Similar to how the internet hobbles along fine with 32-bit addressing by just adding a layer of indirection to the internet with NAT. I guess the other example of 128-bit addressing is ZFS. What are the biggest ZFS file systems? And who has more data? S3? Once you get bigger than 64-bit you want to scale out horizontally anyway and not just put it all into one flat addressable plane.
sillywalk 2mo ago
IBM's System/38 / AS/400 / iSeries / i5 / i has had 128bit pointers for its Technology Independent Machine Interface. The System/38 was released in 1980.
miga 2mo ago
[dead]

44 points by Alien1Being 2mo ago

A1kmm 2mo ago

So reflecting on that, I think the core assumption that didn't pan out is that the memory / CPU ratio will grow because we'll need more memory, hence requiring CPUs to address more than 16 EiB of data (16 EiB = 16384 PiB = 16777216 TiB of data, what 64 bits can address). But in practice, we've produced a lot more compute. The memory / CPU ratio has increased, but most growth has been from more CPUs (and generally not shared-memory ones, but ones with their own completely separate memory space). IPv6 is 128 bit, so we do have 128 bit ways of addressing computers, but I'd say we're still a long way from a CPU needing to address that much memory as a common case. The speed of light limits how far away memory can be from the CPU for good performance, so short of some drastically new memory technology, it seems unlikely we'll need it soon for any ordinary type of computing device.

jsLavaGoat 2mo ago
the cases where its useful, it's in vectorized instruction sets, etc.
- kelnos 2mo ago
  You're conflating 128-bit registers with 128-bit memory addressing. This article is about the latter.
  - flohofwoe 2mo ago
    Also arguably we already have 256 or 512 bit CPUs, what matters most for memory throughput is not the register width but essentialy the L1 cache line width (eg what in old CPUs was the databus width). As for address width, we're not even close to get full 64 bit pointers from CPUs anyway, more like 48 or 52 bits. Wide pointers (eg 128 bit general registers) would make sense for carrying capabilities for memory safety though I guess.
dist-epoch 2mo ago
Because of LLMs we are back to memory being king (KV cache). But there was another thing - clusters of thousands/millions of machine instead of one big iron with all the memory.

deleted 2mo ago

[deleted]

Aardwolf 2mo ago

Personally I'm disappointed 128-bit floating point (quadruple precision) never properly made it into CPU's (sure it appeared in some niche ones here and there, but not in what we actually use today). After all, in the 1980's they had 80-bit ones, it's not even that far off, and they had millions times less transistors then. While probably niche and applications that need higher precision using their own custom types anyway, they'd allow cool stuff like easy to program fractals with much higher detail than now. But also, less precision loss in many applications.

flohofwoe 2mo ago
That's the thing, bigger datatypes means less effective memory throughput. From that perspective 16-bit or even 8-bit floats are often more useful than 80 or 128 bit floats. Same problem with 64 bit pointers and why it's often better to store narrower indices instead of full pointers, data can be packed more tightly and accessed more efficiently.
- TheOtherHobbes 2mo ago
  The point about large word lengths is you get higher data throughput, and faster processing, because everything is going through fat pipes into a big parallel machine with multiple levels of cache and vectorisation. The issue is the utility of floats at different precisions. 128-bit floats have some benefits for high-end scientific applications, but the extra cost and complexity over 64-bit hardware would only make sense for specialised scientific supercomputing. So far it just hasn't been worth it.
silvestrov 2mo ago
precision loss in floating point is often exponential. If 64 bit isn't enough, then very quickly 128 is also not enough. If precision is important then you will very often want systems that represent every number as an interval [a, b] meaning that the true value is between those 2 numbers. This makes you able to detect loss of precision due to e.g. d = a / (b - c) where b-c can result in a number close to zero which makes uncertainty grow. If you use this formula iteratively then precision is lost completely no matter how many bits there are in your floating point variables.

kev009 2mo ago

One interesting thing with wider addresses isn't necessarily increasing itself, but features built upon the expansion. CHERI is one example of that.

justincormack 2mo ago
Yeah, or cryptographically secure non guessable addresses.

bArray 2mo ago

That exchange is quite a good prediction all things considered. > Note that "minor" implementation issues like die space, routing, and gate delays, especially of 128-bit adders & shifters are non-trivial, so people aren't going to rush out and build 128-bitters for fun, just as people matched timing dates of their 64-bitters to their expected markets. I think we're stuck with 64 bit for quite a while. The circuit size jump from 64 bit to 128 bit is significant. There's no fixed scalar or apples to apples comparison (that I'm aware of). Just for adders though, 2 bits requires 2 full adders, 4 bits requires 4 adders, 8 requires 8, etc. Another way to look at this is that 16 bit gives addressable memory up to 65k, and quite a few programs had to deal with paging in architectures like the 8086/8088. 32 bits gave us up to 4GB addressable memory, and not it's now not uncommon that a program such as a web browser exceeds this. 64 bits would give us up to 18 exabytes of addressable memory. I'm not aware of any common programs breaking into the terabyte category (even in most research), let alone petabyte and then exabyte. Even iterating over that many numbers becomes a large computational task. Just a quick test program: // gcc -O3 count.c -o count #include <stdint.h> int main(){ uint64_t z = 0; for(uint64_t i = 0; i < UINT64_MAX; i++) z += i; return (int)(z % 2); } Using uint32_t and UINT32_MAX, it returns almost instantly. For uint64_t and UINT64_MAX you will be waiting a long time. 128 bit values? Even longer. Maybe many many cores could break that memory up, but then it makes sense to have a 64 bit system with some kind of ability to occasionally change page.

kalleboo 2mo ago
> I'm not aware of any common programs breaking into the terabyte category (even in most research) Wouldn't that be the commercial LLMs? ChatGPT, Claude etc are estimated to be in the 2-10 TB range, and a large portion of the population of developed countries are using those apps commonly. Not on their own systems, but if we're in the 1995 academic perspective, multiuser systems are assumed. I imagine the strongest reason we won't need 128-bit memory addressing is because horizontal scaling is easier. If OpenAI had needed a single addressing plane to cover all their users, 128-bit might be required. Similar to how the internet hobbles along fine with 32-bit addressing by just adding a layer of indirection to the internet with NAT. I guess the other example of 128-bit addressing is ZFS. What are the biggest ZFS file systems? And who has more data? S3? Once you get bigger than 64-bit you want to scale out horizontally anyway and not just put it all into one flat addressable plane.

sillywalk 2mo ago

IBM's System/38 / AS/400 / iSeries / i5 / i has had 128bit pointers for its Technology Independent Machine Interface. The System/38 was released in 1980.

miga 2mo ago

[dead]