9 ms·
Updating a 50 terabyte PostgreSQL database (2018)
- Cieplak 6y agoTerms like ”master / slaves” carry racial overtones so many prefer terms like ”primary / replicas”. Similarly, ”allowlist / blocklist” rather than ”whitelist / blacklist”.
- lurker3901 6y agoyeah, nah. I still use master / slave and whitelist / blacklist, I'll continue to do so as long as this Orwellian policing of language continues.
- oars 6y agoI am now using main branch instead of master branch for all my new projects.
- atonse 6y agoI actually sort of agree with retiring master and slave because they were just not good descriptive names. Primary and secondary are more accurate descriptions. But I am still struggling to see an issue with blacklist and whitelist as they actually don’t seem to have anything to do with race, nor have they ever. They have more to do with darkness and light, or death, etc. Like something being a black box just has no relation whatsoever to race. And if we start to change words that demonstrably have nothing to do with race, then anything is fair game. And that’s just a very unnerving place to be.
- Tostino 6y agoThis is where I'm at too. Totally on board for retiring the use of master/slave for primary/replica. But like you said, whitelist/blacklist didn't have anything to do with race. I just don't see why it's relevant at all to the conversation when we are talking about terms or names that may be racially insensitive. It's on you to bring up evidence for why a phrase is insensitive. I've yet to see any.
- anonfunction 6y agoThe argument is that the underlying meaning of whitelist being “ok” and blacklist being “not ok” implies some bias based on color. It wasn’t long ago that there were “whites only” and “no blacks allowed” signs throughout the US. For foreign speakers using a more descriptive name like allowlist and denylist is a benefit as well.
- Tostino 6y agoI mean I am not going to fight kicking and screaming as language evolves. I am also for descriptive names like allowlist/denylist. I just don't see the argument for changing that phrase it in the same light as I see the argument against master/slave.
- atonse 6y agoThese references have everything to do with presence and absence of light, not race. We are diurnal so we mostly see light as a positive (daylight, when we are awake), and darkness as a negative (night when we are asleep, and have evolved to be more wary of a situation where we can’t see well). My argument for all this is, instead of actually vilifying the colors themselves which are meaningless, why don’t we stop referring to humans as white and black? Where’s that debate? Because those words are meaningless. Neither are white nor black. They’re brown. We are all different shades of brown.
- ithkuil 6y agoQuantum supremacy, command execution, whitespace, race condition, Red-black trees, blackboard, n-gram, Domain Specific Language, Domain, Chained iterators, locks. Jokes aside, what matters is what you mean by words and how do you feel when you realize that what you're saying genuinely offends somebody. Imagine you don't really think about slavery when you say "sell down the river". Somebody points it out to you what it actually means. You can say "oh, shit" and try to avoid the phrase. Or you can say "it's my inalienable right to keep using this sentence". Or you can say "well, yes in this case it makes sense, this sentence is racist, but if we start to police words, where does it stop?" Let's see; let's replace that sentence with any of the above. Is "whitespace" really a space for white people or is it because space between words is ink-free? Is a "race condition" a condition of a race of people or a competition of speed? The words are spelled and pronounced the same but they are not the same word. One comes from romance languages and the other from Norse. But it's not only etymology. Contrast "master" in "master-slave" where it clearly means the owner/controller of the slave and it stems from shameful practices actually happened in our past by and to our ancestors. Contrast this with "master copy", "to master an art", etc. where the word retains its original meaning of "teacher" (magister). Be smart, be curious, don't stop at the surface.
- dang 6y ago"Eschew flamebait. Don't introduce flamewar topics unless you have something genuinely new to say. Avoid unrelated controversies and generic tangents." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html Please don't take HN threads on generic ideological tangents. This point has been argued to death a thousand times here already, and no interesting discussion is going to be had by changing the subject to it. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sort=byDate&type=comment&query=generic%20ideolog%20by:dang https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor... Moreover this article is from 2018, so bringing it up in this context is an anachronism.
- Cieplak 6y agoAck. I read the article before the 2018 suffix was appended to the link so was initially under the impression that it was a recent article. Was trying to be helpful given they concluded the article by saying they were hiring, and that some minor language adjustments might help them towards that objective. My apologies, definitely not trying to incite controversy.
- scrollaway 6y agoAdyen is a dutch company FYI. And whether their post is from 2021 or 2018, this is not a helpful reply. I won't delve into why to avoid the exact flamebait dang was talking about, but suffice to say that if your comment were rude when saying it face to face to someone giving a talk, it's rude here too.
- fastest963 6y agoThey're halting database traffic for 15-30 minutes. I'm confused how this won't mean they have downtime? The application can queue transactions but how is the application handling reads after those queued trsnactions and what if 2 queued transactions conflict?
- mjthompson 6y agoAlso very confused by this. I was waiting for a part that explained how data was being accessed and managed during that window. I wouldn't really want a different, likely subpar system responsible for queuing and serving requests during that window. I can only think they are directing it to a different cluster.
- denysvitali 6y agoMaybe their concept of downtime is purely related on the transactions log side. Assuming they're like my bank: they do maintenance like these every X months, and during that time frame ATMs and E-Banking are offline, but transactions keep on working without interruptions. Not sure if this is the case though
- zbendefy 6y agoThey mentioned this: > One other detail to note is that we built our software architecture in such a way that we can stop traffic to our PostgreSQL databases, queue the transactions,and run a PostgreSQL update without affecting payments acceptance.
- julienfr112 6y agoAnd how can you at a bank run a transaction if you don't know if there is money or not in the account ???
- mrkeen 6y ago1) Risk management. Accept some losses if necessary. 2) Overdraft. Take the money, putting the account below zero, and fine the customer for letting their account go below zero.
- asguy 6y ago(2018)
- dang 6y agoAdded above. Thanks!
- hardwaresofton 6y ago> As PostgreSQL options were not suitable for the next upgrade, in parallel we considered other possibilities. Our storage devices were able to make instant snapshots and also make them available on remote storage devices over the network, within a much smaller timeframe ZFS?
- znpy 6y agoIt might be NetApp.
- hardwaresofton 6y agoAh thanks -- I was wondering why they wouldn't mention it if it was ZFS, would have made a good addition to the article. I guess they didn't say because it was something proprietary?
- znpy 6y agoNo idea, I actually came to the comments for the same reason: looking for details about the storage. In my opinion it's either zfs or netapp. Zfs can replicate datasets via zfs send, netapp has a snapmirror functionality that does basically the same. Also, iirc, netapp is contributor to freebsd, so it might be zfs anyway underneath. I wouldn't be surprised. Last time I had the pleasure to create a snapshot for a volume in a netapp it kinda felt like creating a zfs snapshot, in term of speed and ease. I thought of netapp since they're running fancy 768gb boxes. If they spend money well on their hardware, they probably spend money well on their storage too.
- hardwaresofton 6y agoYeah zfs send (and a bunch of other features/architecture choices) have been one of the biggest reasons I'm interested in the zfs ecosystem and kind of keep a lookout for it/finding ways to work it into infrastructure that I build. I know that Hardware RAID is what the big boys use, but assuming the performance hit is OK, the feature set of ZFS with a properly tuned database installation like a match made in heaven, was hoping to have them comment...
- gdgdgdgbgfff 6y agoVertical scaling like this will have quite a few physical limits, eg max cores (is it 64?). I would think that they may be fast approaching these limits?. For a an scalable enterprise solution would it not be better, and also more cost-efficient, to horizontally scale. Shard the tables over many nodes.
- jnwatson 6y agoYou can buy boxes with 30x more RAM without going super boutique. You can buy quad-socket server motherboards, so you can have 64x4 cores. For the vast majority of businesses, vertical scaling is quite feasible.
- noir_lord 6y ago128x4 (EPYC 7000 series go up to 128) assuming you can get a 4 socket board.
- gdgdgdgbgfff 6y agoI thought 64 cores was a limit coming from Postgres http://rhaas.blogspot.com/2012/04/did-i-say-32-cores-how-about-64.html http://rhaas.blogspot.com/2012/04/did-i-say-32-cores-how-abo...
- darksaints 6y agoI would think it is a bit higher now, as this benchmark is for a pretty old version that's not even supported anymore. In particular, I'm pretty sure that shared memory throughput is a major factor for scaling the number of cores, and there have been quite a few changes to the shared memory subsystem since then. But even then, there probably are some significant overheads that are reached before you get to the limits of single node systems that are available today.
- kasey_junk 6y agoThat’s a 10 year old restriction. At least as of Postgres 9.2 you could run more than 64 cores (though Postgres didn’t always use them effectively).
- z3t4 6y agoI would probably implement an "archive" system that stores static data that is mostly accessed by "sum" functions - In order to keep the main database relatively small. If the data is immutable/static replication and caching becomes much easier.
- KptMarchewa 6y agoI'm not sure what you mean by sum functions, but Postgres natively support foreign tables, and I'm aware of one bank that uses this for older, immutable data.
- z3t4 6y agoWith "sum" functions, I mean that for money transactions the data will likely be stored in columns and the most common operation will likely be to sum each column. So you could store an "archive" with all the columns/transactions, then store just the sum of each column in the "hot" database. If the data is static/immutable you have a lot of options when it comes to caching and optimizing different queries. Another strategy would also be to "shard" the database... I guess storing everything in the same database is the most simple solution, but problems will arise when you have to replicate/recover (an arbitrary) 100+ TB of data. Just copying it over a 100Gbit link will take 3 hours.
- janandonly 6y agoAm I the only one who is shocked to discover that payments (even very small sums) are stored for years on end, maybe indefinite?? Simple deleting a row that is 366 days old is not an option to keep the PostgreSQL DB relatively small?
- cabraca 6y agoThats not shocking. Its probably due to regulatory laws that transactions are saved that long.
- xmaayy 6y agoBut they don't need to be in the active DB right? Auditing and reporting can be done on an immutable database, while transactions are processed on a smaller active one.
- wdb 6y agoIt’s a regulatory requirement keep it for 5-7 years e.g for disputes etc
- KptMarchewa 6y agoJust to add to this: GDPR does not apply here.
- PeterisP 6y agoIt's worth noting that it's not necessarily specific to bank payments, depending on your jurisdiction it may be that general accounting laws that require every company to store payment history for at least 5 years, for example. To add on that, for some types of transaction the regulatory requirements are much, much longer - I was working in a jurisdiction where data of housing loan repayments had to be stored, by law, for 70 years after the end of the loan; so for a 30-year mortgage you'd have to be prepared to store every repayment for 100 years; and information on salaries calculated and paid has to be stored for 75 years (IMHO to resolve retirement-related disputes where it matters where you worked decades ago), passing on to national archive if the company is dissolved.
- pupdogg 6y agoBased on their numbers, they should be archiving their historical data in parquet format partitioned by YYYYMMDD onto something like Clickhouse. This way, they can run a lean Postgres instance(s) at all times yet still get benefits of real-time reporting. Based on their use case, they can retain up-to 30 days of data in Postgres and offload the rest onto Clickhouse.
- KptMarchewa 6y agoYou assume too much. For example, you assume that analytics aren't already run on a separate data warehouse. Ease of accessing older data is an important aspect of database holding transaction (not meaning transactional db). Wouldn't you want to check your transactions on bank page that are older than 30 days?
- manigandham 6y agoAll modern columnnar/analytical databases have no problem in pulling back older data; in fact, that's the point of running large scale analytics.
- deleted 6y ago[deleted]
- KptMarchewa 6y agoI agree, but analytics is orthogonal topic. Unless you want to use columnar database to retrieve 15 transaction that particular user did month ago...
- manigandham 6y agoThere is nothing wrong with that use case. It works just fine. It may not have the same latency as an OLTP database but there are many factors (indexing, sort/partition keys, compression, etc) that can return similar performance for those queries.