4 ms·

Could you provide some references for this work on caching and schedulers? I’m also curious what application domains are storing close to petabytes on a single

by moab 7y ago

Could you provide some references for this work on caching and schedulers? I’m also curious what application domains are storing close to petabytes on a single machine with only a few terabytes of dram.

jandrewrogers 7y ago
Ultra-dense database engines are typically used for data models created from measuring entities or places in the physical world, sometimes in real-time. This includes telemetry and sensors from mobile phones, automobiles, satellites, features extracted from video, network packets, the web, etc and sometimes all of the above. These are the largest data models that I know of; most medium-sized non-tech companies I work with have 10-100 PB data models now. The size doubles quickly even if you regularly truncate old data. The extremely high storage density saves both hardware and software engineering costs by massively reducing the required cluster size. This has become a great emerging market for new database tech, tremendous amounts of money are being spent on these data models and traditional data infrastructures are very poor for this. The size and velocity of the data implies that you need a single database instance that can run ingestion workloads concurrent with analysis workloads. The research on caching algorithms for ultra-dense storage is new, it is literally just starting to be worked into production designs, so I wouldn't expect references. Most advanced database research is done outside academia, so a significant fraction is published long after implementation or not formally published at all. Publication is not the objective per se and there is little time or incentive to do that work. Database research has a discovery problem, I hear about most interesting new research via informal community channels and much of that never shows up in literature.
- moab 7y ago
  Thanks for the reply. I would be very interested if you could point me towards any private companies providing these kind of database solutions. I am particularly interested in their story around concurrent ingestion + analytics workloads. This is a tricky problem that most existing works solve by either doing stupid things a la. having a single-version system, or by using lots of synchronization + complex version-friendly concurrent data structures. Solutions I have seen in academia are usually tested well below the 1TB of data range, and I don't imagine they scale to the kind of data sizes you mention. The fact that the research is being done outside of academia and not being published is pretty sad imo. Hopefully those in the know decide to publish at least tech-reports/whitepapers once they have a share of the market.
  - ignoramous 7y ago
    I'm not sure if shakti-db [0] fits the bill of what jandrewrogers is talking about, but the same folks behind it built one of the fastest time-series databases in-use today, the kdb+ [1]. [0] https://shakti.com https://shakti.com [1] https://news.ycombinator.com/item?id=13481824 https://news.ycombinator.com/item?id=13481824

by moab 7y ago

jandrewrogers 7y ago

Ultra-dense database engines are typically used for data models created from measuring entities or places in the physical world, sometimes in real-time. This includes telemetry and sensors from mobile phones, automobiles, satellites, features extracted from video, network packets, the web, etc and sometimes all of the above. These are the largest data models that I know of; most medium-sized non-tech companies I work with have 10-100 PB data models now. The size doubles quickly even if you regularly truncate old data. The extremely high storage density saves both hardware and software engineering costs by massively reducing the required cluster size. This has become a great emerging market for new database tech, tremendous amounts of money are being spent on these data models and traditional data infrastructures are very poor for this. The size and velocity of the data implies that you need a single database instance that can run ingestion workloads concurrent with analysis workloads. The research on caching algorithms for ultra-dense storage is new, it is literally just starting to be worked into production designs, so I wouldn't expect references. Most advanced database research is done outside academia, so a significant fraction is published long after implementation or not formally published at all. Publication is not the objective per se and there is little time or incentive to do that work. Database research has a discovery problem, I hear about most interesting new research via informal community channels and much of that never shows up in literature.

moab 7y ago
Thanks for the reply. I would be very interested if you could point me towards any private companies providing these kind of database solutions. I am particularly interested in their story around concurrent ingestion + analytics workloads. This is a tricky problem that most existing works solve by either doing stupid things a la. having a single-version system, or by using lots of synchronization + complex version-friendly concurrent data structures. Solutions I have seen in academia are usually tested well below the 1TB of data range, and I don't imagine they scale to the kind of data sizes you mention. The fact that the research is being done outside of academia and not being published is pretty sad imo. Hopefully those in the know decide to publish at least tech-reports/whitepapers once they have a share of the market.
- ignoramous 7y ago
  I'm not sure if shakti-db [0] fits the bill of what jandrewrogers is talking about, but the same folks behind it built one of the fastest time-series databases in-use today, the kdb+ [1]. [0] https://shakti.com https://shakti.com [1] https://news.ycombinator.com/item?id=13481824 https://news.ycombinator.com/item?id=13481824