6 ms·
I wonder if 1.7 petabytes of data (1T indexed records) could fit on a single (very) beefy baremetal server for under a couple thousand dollars a month, served b
by _xnmw 2y ago
I wonder if 1.7 petabytes of data (1T indexed records) could fit on a single (very) beefy baremetal server for under a couple thousand dollars a month, served by SQLite.
Like this: https://use.expensify.com/blog/scaling-sqlite-to-4m-qps-on-a-single-server https://use.expensify.com/blog/scaling-sqlite-to-4m-qps-on-a...
- sanderjd 2y agoSometimes things just aren't nails, even when you have a really good hammer.
- kondro 2y agoGiven 30.7TB SSD’s are about $5500 each and you’d need 56 to to get to 1.7PB (with no redundancy). Not to mention that SQLite’s maximum DB size is 140TB. I don’t think you’d be able to fit this much storage into a single machine, especially not for a few thousand a month and SQLite wouldn’t be appropriate for this use-case.
- bayindirh 2y agoIf you install a RAID controller and a couple of disk boxes, it's possible with 1:1 replication, or with backups. 60 disk 3.5" units already exist, so 2.5" SSD racks. It won't be cheap, but will be resilient and fast. Bloody fast if you have the budget.
- zaphirplane 2y ago> or with backups take a while to restore a PB and a way to take a hot backup without impacting the service that by itself is a task or snapshots which is more disk > 1:1 replication Depending on the amount of writes could be a ton of extra disk and a bucket for network cost
- bayindirh 2y agoThese systems support zero-downtime snapshots. You tell it to snapshot, it instantly snapshots, you can run a differential/incremental backup at great speeds. Your RAID controller is already caching the hot data, so the impact is minimal. Except network cost there's no extra disk required. It's just broadcasted writes consumed on the other hand. These boxes are not dumb JBODS. They support their own replication/backup subsystems, so everything is transparent.
- Closi 2y agoResilient and fast from a disk perspective, but in practice massively bottlenecked by the fact that Sqlite can only have 1 writer at a time.
- Neil44 2y agoAt the moment they're just paying someone else to buy $5000 SSD's and run a database on them at many X markup.
- omeid2 2y agoThere is no upper bounds to economy of scales. Maybe there is for the cents per GB of raw storage, but power usage, security, rent, and everything else scales too, and few of them have upper bounds on economy of scales.
- Retric 2y agoEconomies of scale generally have upper limits. Often when you approach the largest scale the existing market will supply you essentially need to become your own supplier which then runs into span of control issues. The organization needs to become competitive in that new market or their costs increase. Keep scaling and eventually vertical integration ends up looking like a Soviet style planned economy. Your remote mining town needs some way for people to get soap etc so you open a store with it’s own supply chain etc etc.
- omeid2 2y agoYes, there is an upper limit, and that is the entirety of global economy. But that assertion is as useless as it is true.
- Retric 2y agoHardly the global economy, even approaching a global monopoly for something tends to hit such limits. Few companies deal with this today because most markets are fragmented. But even at the 30% marketshare the iPhone has been dealing with these issues for a while. They just can’t buy 200 million volume buttons or whatever off the shelf. Now imagine what happens if one of their suppliers would fail days before the phone launches. They have such tight integration not just with the manufacturer’s process but also their finances because Apple simply can’t get replacements at scale on short notice. And remember that’s at ~30% market share, it just gets worse after that.
- ndriscoll 2y agoThere are 61.44 TB NVMe drives (best price I've seen right now is ~6200. They were ~4800 earlier this year). You can have a 1U server with 32 E1.L slots so you should be able to fit ~1.9PB raw storage into 1U for a little over $200k. Don't know how business financing works, but at 8% interest with a 5 year amortization, that's a bit over $4k/month.
- mobilemidget 2y agoDo you have any good recommendation for such 1U server with 32 slots? Thanks
- perryh2 2y agoSupermicro https://www.supermicro.com/en/products/nvme?pro=formfactor%3D1U%26drive%3D32 https://www.supermicro.com/en/products/nvme?pro=formfactor%3...
- hallway_monitor 2y agoShows 2U in order to fit 32 drives, but hopefully you can find a 2U slot for this absolute monster of a storage box if you have that need.
- ndriscoll 2y agoThis one is 1U and supports dual socket CPU and 6 TB RAM: https://www.supermicro.com/en/products/system/1U/1029/SSG-1029P-NEL32R.cfm https://www.supermicro.com/en/products/system/1U/1029/SSG-10... I've never done procurement so can't really speak to recommendations. I just see that the products apparently exist. There's also 32 drive 1U JBOF enclosures for more expansion: https://www.supermicro.com/en/products/system/1u/136/ssg-136r-nel32jbf.cfm https://www.supermicro.com/en/products/system/1u/136/ssg-136...
- jakjak123 2y agoOur ops team actually wanted to do this, but we on the project have nightmares from putting 1PB of database on a single host ><
- choppaface 2y agoStorageReview plays with 2PB flash machines all the time https://www.youtube.com/watch?v=UQMKtlIjeuk https://www.youtube.com/watch?v=UQMKtlIjeuk 1PB in a rack with spinning rust + flash buffer has been easy for years now.
- BlackLotus89 2y agoNo it won't. sqlite "only" works with up to 281TB [0] [1] [0] https://www.sqlite.org/releaselog/3_33_0.html https://www.sqlite.org/releaselog/3_33_0.html [1] https://www.sqlite.org/limits.html https://www.sqlite.org/limits.html (#12)
- sgt 2y agoYou can split up into 10 SQLite DB's on this individual server.
- cdchn 2y agoYou've now implemented sharding on top of SQLite. Eventually all programs will be able to read email.
- mrbungie 2y agoAny non-trivial complexity codebase eventually implements a mediocre SQL/Lisp/etc.
- Closi 2y agoExactly! Why take a system not designed for this sort of scale and force it to scale, rather than use systems which are designed and tested for this scale and volume? All you will do is hackily re-invent all the other things that the other databases had to do to scale to this extent. Plus size is only one limit, you would be limited to 1 write every few milliseconds. My napkin maths estimate is that there are at least 1-2m writes per hour going into this thing, so probably 300-600 writes / second (Average) and maybe over 1k writes/second peak. We are going to fall over here! Not sure why some people seem to have a viwe of "There is no scaling problem that can't be solved with a sufficient enough number of SQLite databases".
- zaphirplane 2y ago> You can split up into 10 SQLite DB's on this individual server. 1 is a scalable, managed, highly available service, with economies of scale the other is a fixed size, capital expenditure with fixed performance, limited DR, requiring a couple of SRE/DevOps and colo There is also the will it always work question
- tinyspacewizard 2y agoAlso a bit scary to have a system without a scaling mechnism built-in in the path of customer traffic. At some point you may be racing to upgrade it.
- sgt 2y agoHow would you replicate that SQLite DB onto other hosts to achieve redundancy?
- thangngoc89 2y agoOne could use Litestream [1] [1]: https://litestream.io https://litestream.io
- szundi 2y agoWhat if a continuous replication system has a bug one day, and you realize you are just a bit corrupted and have to rerun? Or is it the same with cloud tools?
- thangngoc89 2y agoThat's why you always test your backup. I backup the full sqlite.db every day and test the litestream replication every week. So far litestream have been solid.
- zaphirplane 2y agoBy the time the TB is restored, time to start the next test How do you detect restored but bit flipped data ?
- thangngoc89 2y agoI do this in backup testing: sqlite3 /path/to/db sqlite> PRAGMA integrity_check; See SQLite3 documentation: https://www.sqlite.org/pragma.html#pragma_integrity_check https://www.sqlite.org/pragma.html#pragma_integrity_check
- zaphirplane 2y agoSounds like it will take awhile for TB and it checks db integrity not data integrity
- riku_iki 2y agoit will take forever to create that index. Link describes 10B rows dataset.
- khaki54 2y agoThe value proposition of commercial loud isn't cost savings unless you manage to quantify all of the ancillary and extrinsic factors such as security risk, HVAC, datacenter personnel, and hardware lifecycle. Any well capitalized and organized company could build their own cloud much more cheaply, but really a significant portion of the calculation is outsourcing the risk components.
- _xnmw 2y agoThe problem with outsourcing the risk components is that you don't know for sure whether they are properly taken care of. Major cloud providers have been caught "oopsing" your data, and bam it is gone. Furthermore, they have no incentive to be more efficient about it, they could easily be using 10x the amount of resources necessary, and you wouldn't even have a clue, you're just paying for evermore expensive crap that becomes less reliable over time.
- ddorian43 2y agoBut the cloud providers compete with other other! Look at the efficient market in display in their bandwidth pricing!
- PretzelPirate 2y agoFor very large customers, the cloud providers do compete with each other on cost. They often pay different prices than are advertised.
- ownagefool 2y ago> Any well capitalized and organized company could build their own cloud much more cheaply Lots of orgs fail to turn money into talent and then talent into products. It just takes one bad hire at senior level and suddenly your cloud is a vmware install where all machines are boot off network disk, and contention makes the entire thing fall over.
- pclmulqdq 2y agoYou wouldn't want to do 1T records on one server even if you could. At that scale, you would prefer to be somewhat distributed for availability and scalability. Also, SQLite has issues at large scale. A reasonable number for one server is about 32-128 TB, and 1.7 petabytes with some redundancy fits nicely in ~30 servers with a decent distributed database.
- klysm 2y agoSure but then you get a whole new set of costs and folks you have to hire to maintain that hardware.
- siva7 2y agoMaybe it could and now you got 99 new Problems. That's why more experienced decision makers won't allow this to happen.
- Closi 2y ago1.7 petabytes on Sqlite? Sqlite's own advice: > If your data will grow to a size that you are uncomfortable or unable to fit into a single disk file, then you should select a solution other than SQLite. SQLite supports databases up to 281 terabytes in size, assuming you can find a disk drive and filesystem that will support 281-terabyte files. > Even so, when the size of the content looks like it might creep into the terabyte range, it would be good to consider a centralized client/server database [over SQLite].
- cheeze 2y agoThis is the worry IMO. It's fine to dump it on a server with SQLite, but once you start hitting scaling limits, you're in for a potentially rough migration.
- deleted 2y ago[deleted]
- callalex 2y agoOne of the main reasons you put up with the annoyances of tuple-based storage like DynamoDB is because you want extremely high availability that simply cannot be provided by one computer in one physical location.