5 ms·

I worked with Domino extensively and some of the core technology was great. But with a 64GB database size limit it was really only suitable for small applicatio

by nradov 2y ago

I worked with Domino extensively and some of the core technology was great. But with a 64GB database size limit it was really only suitable for small applications. We tried to work around that limit with sharding but it wasn't very practical.

cube00 2y ago
Assuming the space was being consumed by attachments, DAOS can help with this. There's reports [1] of people getting up to 2TB once you get the attachments out of the NSF. [1]: https://ds_infolib.hcltechsw.com/ldd/nd85forum.nsf/DateAllFlatweb/FDA9C3207FAB52D48525757B003E7762?OpenDocument https://ds_infolib.hcltechsw.com/ldd/nd85forum.nsf/DateAllFl...
- nradov 2y ago
  Ya, that didn't help. We were dealing with high volumes of healthcare structured text data, not attachments. I never understood why IBM couldn't just fix the NSF size limit. It was so stupid.
  - dcrazy 2y ago
    I have no specific knowledge of the Domino database format, but extrapolating from my knowledge of database storage formats in general: the database file almost certainly contained data structures of fixed size. Many of the fields in these data structures likely contained offsets to other data structures, such as an index that contains the offsets of each item. Fixing the size limit would require migrating to a new file format with different data structures. Migrations are always hard to sell to customers and a ripe opportunity for data corruption, and horizontal sharding can solve the same problem while also addressing other scalability and reliability issues.
    - nradov 2y ago
      I never claimed that changing database file formats was easy, but IBM had already increased the size limit once before from 4GB to 64GB. So obviously increasing it further could have been done. Unfortunately IBM leadership never saw Domino as anything more than a platform for small internal corporate IT applications so they didn't care about fixing the limitations; it was a total failure of vision. IBM didn't provide any tools for horizontal sharding either. We had to roll our own, which was a huge expense and made every part of the application far more complex. And it actually made reliability worse; because displaying incomplete data was not acceptable to customers, if any one database got corrupted then everything failed.
codingdave 2y ago
Yeah, and that was not the only odd limit. 64K limit on fieldnames in the entire app was my favorite weird one. At one point there were docs online of all the various limits in Domino, and you had to work around them all. This is also why I say there are few people who know Domino well enough to run it today - even in its heydey there was some mad science to do large apps on it, knowing not only all the quirks but how to work around them.

by nradov 2y ago

cube00 2y ago

Assuming the space was being consumed by attachments, DAOS can help with this. There's reports [1] of people getting up to 2TB once you get the attachments out of the NSF. [1]: https://ds_infolib.hcltechsw.com/ldd/nd85forum.nsf/DateAllFlatweb/FDA9C3207FAB52D48525757B003E7762?OpenDocument https://ds_infolib.hcltechsw.com/ldd/nd85forum.nsf/DateAllFl...

nradov 2y ago
Ya, that didn't help. We were dealing with high volumes of healthcare structured text data, not attachments. I never understood why IBM couldn't just fix the NSF size limit. It was so stupid.
- dcrazy 2y ago
  I have no specific knowledge of the Domino database format, but extrapolating from my knowledge of database storage formats in general: the database file almost certainly contained data structures of fixed size. Many of the fields in these data structures likely contained offsets to other data structures, such as an index that contains the offsets of each item. Fixing the size limit would require migrating to a new file format with different data structures. Migrations are always hard to sell to customers and a ripe opportunity for data corruption, and horizontal sharding can solve the same problem while also addressing other scalability and reliability issues.
  - nradov 2y ago
    I never claimed that changing database file formats was easy, but IBM had already increased the size limit once before from 4GB to 64GB. So obviously increasing it further could have been done. Unfortunately IBM leadership never saw Domino as anything more than a platform for small internal corporate IT applications so they didn't care about fixing the limitations; it was a total failure of vision. IBM didn't provide any tools for horizontal sharding either. We had to roll our own, which was a huge expense and made every part of the application far more complex. And it actually made reliability worse; because displaying incomplete data was not acceptable to customers, if any one database got corrupted then everything failed.

codingdave 2y ago

Yeah, and that was not the only odd limit. 64K limit on fieldnames in the entire app was my favorite weird one. At one point there were docs online of all the various limits in Domino, and you had to work around them all. This is also why I say there are few people who know Domino well enough to run it today - even in its heydey there was some mad science to do large apps on it, knowing not only all the quirks but how to work around them.