18 ms·
AWS Babelfish: The Elephant in the PostgreSQL Room?
- eternalban 6y agojuicy little bit: "Apparently AWS has around 200 open job positions (!!) for developers working on Elasticsearch"
- aynyc 6y agoHa! I guess hardware resources aren't the only thing Elasticsearch suck up!
- itg 6y agoAccording to my friends who are in AWS, the attrition rate is really high with many people leaving within less than 2 years.
- JulianRaphael 6y agoI've been working for AWS across different teams for some time now and can't confirm that. There are a LOT of folks with +5 years tenure. In the US, the average tenure of workers aged 25-34 is 2.8 years, so you'll obviously see some turnover and some teams might see more than others - but the attrition rate is not really high across the company. In my opinion, we're hiring mainly because we are still growing really fast!
- eternalban 6y agoThat was my read of it as well. Amazon is dead serious about the fork.
- jorblumesea 6y agoYou might be on a solid team, but AWS is widely regarded as being a complete meat grinder. We've hired so many devs from AWS in the past 18 months (even pre covid) and they all have the same story. 12-14 hour days at a minimum, toxic leadership and managers, incredible pressure, tons of tech debt and mostly doing ops work. I'd say the trend for AWS is leaning "do not join". Anecdotal? Of course, but with enough stories it becomes a statistic and not an anomaly.
- JulianRaphael 6y agoHm. I will obviously appear to be biased, but isn't the collection of anecdotes that you've gathered a manifestation of some kind of inverse survivorship bias? Within the total population of people who work or have worked at AWS, the largest group of people with a negative experience will be in the subset of the population that has left the company? I really believe that companies the size of Amazon will inevitably have a certain percentage of people who had negative experiences - just like the company has negative experiences with individuals. Like I said in my other comment in this thread, I'm sure that there are some teams that don't have a good culture but it also comes down to the individual in many cases. I guess it comes down to individual preferences, some folks really don't want to work at Google, others dislike Facebook, some don't like AWS, etc.. But at the end of the day, all of these companies still attract a ton of incredibly talented people and I personally don't see that trend ending soon.
- jorblumesea 6y agoFew companies have entire websites dedicated to how poor their experience was at said company. Amazon is the only company I've ever heard of in tech where people have devoted their spare time and money building resources to warn others about how bad and toxic their experience was. Microsoft doesn't have this. Facebook or Google don't have this. It's actually mind blowing when you think about it. Imo, Amazon is a special level of toxic and abusive and it shows by their reputation in the industry and how high their turnover is. That's what I mean by anomaly vs statistic. When thousands (tens of thousands?) of your former employees have horror stories, it starts to build a strong narrative than a few bad teams or a bad manager. Or you start to question if the ratio of bad teams is that high, what is wrong the the company as a whole? Does it mean your experience at Amazon is wrong or incorrect? Absolutely not. But your experience isn't the norm.
- JulianRaphael 6y agoOut of curiosity, I just did a few searches along the lines of "is MSFT/FB/GOOG bad place to work" and for each of them there were several results/websites about how each of these companies is the worst place to work. If you look at the largest publicly available data set, Glassdoor, AWS does quite well - better than Microsoft and as well as Google. I have not worked for Amazon outside of AWS, so I can't really comment on that. However, as Amazon has more than 1,000,000 employees (400,000 of which were hired in 2020), I'd argue that you get more negative stories simply because of the larger population? Comparing Amazon to MSFT (163,000), FB (50,000) or GOOG (135,000) without taking its ginormous scale into account seems to be comparing apples to oranges. In terms of scale and typology, it would probably be better to compare AWS to these companies and I believe that the publicly available data paints a different picture than what you describe. I'm not trying to argue with you here, I've just had a quite few discussions in recent times with people who had very strong negative opinions about MSFT being a place for the lazy, Google being a place for the special snowflakes, Facebook being a place for the cultish, AWS being a place for those looking for a stepping stone in their career, etc....and I think that all of these companies actually are great places to work but there will always be a few percent who absolutely hated their time there.
- myWindoonn 6y agoMm, but you're in management. All the Amazonians I've talked to have indicated that it's a very rough environment and they were glad to get out. Maybe you get a distorted view due to the power imbalance between you and your peers?
- JulianRaphael 6y agoI'm an individual contributor in product management, so not in people management. Amazon / AWS is a challenging work environment, just like most other tech companies. In my opinion, that is also why so many people want to work for FAANG & Co, as you generally get to work on really interesting but also challenging things. I'd say that bursts of intense working periods are par for the course in FAANG & Co, but it also depends on your own ability to handle our self-service culture (which I find empowering). The culture is certainly peculiar but each large tech company has their own culture that works really well for some and doesn't jive with others. Just speaking for myself here, but I have learned a ton professionally and personally, work(ed) with inspiring people and had the best work-life balance of my career so far. Don't want to sound like a shill here - I'm certainly aware that in a company the size of Amazon there will be good and bad pockets, I just happened to work with good teams so far.
- myWindoonn 6y agoYou unfortunately not only sound like a shill, but like a shill who has read Amazon's literature. I have a few questions that are worth reflecting on; I also asked them of myself when I worked at a FAANG: * How many hours/week do you spend at your work machine? * How often is a sprint crunched; how often does the sprint manager exhort sprinters to complete the board? * How many incidents/quarter does your team have? How often does an incident route through you? * How many hours/quarter are you oncall/onduty? * How often do you have anxiety on Sunday night about going to work on Monday? The answers should be at most 40, rarely, 12, 60, and rarely. Ideally, they would be 20, never, 3, 120, and never. Note that we don't require less oncall; we require less stressful oncall.
- wiremine 6y ago> The trends for the last 8 years are clear: Oracle and SQL Server are constantly declining in popularity; MySQL is slightly declining; and PostgreSQL is clearly growing in popularity. But while PostgreSQL almost tripled in popularity in these eight years, it is still far behind the other three. It's great to see Postgres growing, but I often wonder why it isn't growing faster, especially when compared to MySQL. Without getting into a religious flame war, I'm curious what the HN thinks about that?
- sl1ck731 6y agoWordpress is probably the elephant in the room.
- xmodem 6y agoAt my current gig, we've been burned pretty badly by scalability problems with MySQL, and the prevailing wisdom is that new functionality and services that have to deal with data at scale should be built on Cassandra. My cries of "we wouldn't have this problem with Postgres" have mostly fallen on death ears.
- pizza234 6y agoCan you elaborate on the specifics of the scalability problems, and how the different databases do differently on them? It's an interesting subject. We're having scalability problems with our RDBMS, but it's something RDBMSs in general don't solve.
- xmodem 6y agoWe have a few large (not large in the scheme of things, but large for MySQL) tables that we can't add columns to without locking the table for an extended period. In Postgres this operation is instant no matter the size of the table. This not the only problem, but it is the most pernicious
- Thaxll 6y ago
- api 6y agoMore freedom as long as you run on AWS. Or did they release this?
- severino 6y agoThey plan to release it under the Apache License this year, according to: https://babelfish-for-postgresql.github.io/babelfish-for-postgresql/ https://babelfish-for-postgresql.github.io/babelfish-for-pos...
- sradman 6y agoThe elephant in the room is that AWS Babelfish could be a viable alternative to PostgreSQL. Babelfish is currently a fork of PostgreSQL that is compatible with Microsoft SQL Server’s TDS wire protocol and T-SQL language. The article proposes adding hooks for wire protocol extensions eliminating the need for separate forks. Adding new languages is already supported (EDIT: incorrect, see mattashii's comment).
- mattashii 6y ago> Adding new languages is already supported. This is true only to a degree: Adding new procedural languages is supported, but support for new query dialects (i.e. the main contact surface of most applications with an SQL RDBMS) is not currently a supported feature. As mentioned in the psql-hackers thread from the article, currently the postgresql-specific parsed syntax tree is passed through many components, and to replace the syntax you'd have to alter and recompile the core application.
- ahachete 6y agoOP. One of my fears. That unless we, as the PostgreSQL Community, do something about it, and embrace the efforts that AWS started doing and I hope will continue, it would effectively end up being a fork. And a fork with a collaboration on GitHub, which many would prefer to contribute to rather than current's Postgres contribution mechanism. And if that's the case, I wonder if it could become the MariaDB, or the LibreOffice, of Postgres. I hope we will be able to do the best thing and have the contribution that Postgres is receiving integrated and merged into Postgres.
- _msw_ 6y agoDisclosure: I work for Amazon where I build AWS infrastructure, but these opinions are mine alone. Personally, I think that the word "fork" has a lot of historical meaning in the world of Free and Open Source Software. To me, "fork" means that there is a "vote of no confidence" in the direction of the people who are making decisions about open source software. That clearly is not the case here. The PostgreSQL core team has a very good reputation of being collaborative, and thoughtful, in their work to make PostgreSQL the best database it can be for the community at large, and not build it to benefit any particular commercial interest. Often software development happens on branches. These are experiments, not forks. Over time, good ideas are propagated, and grafted, between the branches. These are not "forks" to me.
- jjjeii3 6y agoI worked with SQL Server, Oracle, MySQL and many other DBMSes. Nothing can beat PostgreSQL. Oracle's SQL Developer is the worst IDE for databases I ever seen. Oracle DB doesn't support "TOP N" or "LIMIT N" notations and many other useful features... I like SQL Server more than Oracle and Microsoft provides 2 good IDEs for it (SQL Management Studio and Azure Data Studio). But it has some bad limitations - you can't define ON ... CASCADE on more than one path. You also can't create CTEs with inserts/updates. PostgreSQL doesn't have a good IDE out of the box. I heard that Azure Data Studio now support it, but didn't try it yet, because I already paid for a commercial IDE that supports PostgreSQL. PostgreSQL also doesn't have a good high availability tooling, but all major cloud providers do this for you. Considering that you get it for free, it is a much better choice than Oracle or SQL Server.
- NDizzle 6y agoDevart makes good SQL IDEs. They came out with one for Postgres ... recently? In the last year or two. If it's similar quality to their mysql ide, I'd recommend it. https://www.devart.com/dbforge/postgresql/studio/ https://www.devart.com/dbforge/postgresql/studio/
- deleted 6y ago[deleted]
- bdcravens 6y agoWorking with the big 3 on a regular basis, DataGrip is the best tool I've found. (It lacks some management features, but it's been years since I needed those)
- pimeys 6y agoI'm working with many SQL Server, MySQL, PostgreSQL, SQLite and soon Mongo versions daily, and I have to say Datagrip is worth it's price tag easily. It is the only non-open tool I work with because it really makes work with multiple databases a lot easier.
- katnegermis 6y ago
- gonational 6y agoEmbrace, extend, extinguish, Amazon edition. In my opinion, this is the inevitable conclusion of Amazon's sudden increase in direct involvement in OSS projects. With cases like elastic search close in the rearview mirror (I know that there were reasons for this, but it was a start), It's hard not to come to this conclusion. Once the Amazon tentacles have reached down into every major staple of the OSS economy, it's difficult to predict the end result, but one thing is for certain, it will benefit Amazon more than it benefits anyone else, lest Amazon breaks its promise to its shareholders. Amazon has demonstrated a pattern of controversial vertical integration, by using internal analytics to determine which of their own merchants' products they should undermine. To believe they would not do the same thing in their largest and most important division would require tragic ignorance. This is all just my opinion
- _msw_ 6y agoDisclosure: I work for Amazon, where I build AWS infrastructure. But these are my own opinions and experiences at work, and I am not speaking for my employer. "Embrace, extend, extinguish" has never been uttered in any conversation I've been part of at Amazon. It simply is not part of the philosophy, or culture. We obsess about customers, and customers do not like the things that they depend on, like open source software that they've built their business on, to be damaged. Rather, they want to do more with open source software all the time. And many want to find ways to get off of software that is constraining them, or causing unplanned stress or surprises when their costs suddenly change due to arbitrary license changes. The promise here is to customers, and customers tell us that community-led open source software development is better for them. And this is no surprise. As a software developer, and a long time FOSS advocate, I believe it too. By listening attentively to customers, I believe we will continue to build a business that we, and our shareholders, will be proud to be a part of, and will provide long term growth.
- gonational 6y agoWith no disrespect to you, this comes across like a pre-canned response from the Amazon Anti-Anti-Trust legal team... I mean, your second and third paragraph could be the introduction to an internal presentation about "why we at Amazon should embrace, extend, and extinguish OSS". The things you listed are precisely why Amazon would want to destroy OSS, Because OSS is a threat to anyone offering proprietary software, and anyone who has built their empire on OSS will inevitably need to shift to proprietary software to retain relevance in a world where OSS software "eats the world" by, for instance, commoditizing the infrastructure layer... In other words, just in time for AWS to become less relevant due to the enormous gains in open source infrastructure as a service tools, Amazon will swoop in and help to destroy the most useful parts of the open source economy. It's easy to look at this with a naïve optimism and think "but these are the exact kinds of things that helped AWS grow to begin with", To which the obvious reply is "yes, at the bottom end, but Amazon has been focused on enterprise more and more over the past five years, and licensing agreements with other big tech companies have changed the offering and the prospects of future offerings". All you have to do is look at YouTube to see another example of this. YouTube built its user base on small contributors, but as soon as it reached a plateau it began to focus on the old guard, main stream media, Hollywood, etc., unfortunately for the little guys. This is the same pattern every company takes once they get big enough to play competitively with the big boys. This is all, of course, only my opinion.
- andred14 6y agoTime to get off AWS or prepare to. The destruction of Parler, a company that did nothing wrong, should be alarming and motivating for you to ditch AWS and all big tech censorship companies.
- thinkx 6y agoPostgres has a core design philosophy of being “extensible”. This should be embraced..
- lukaseder 6y agoFor those who can't wait, there's https://www.jooq.org/translate https://www.jooq.org/translate
- notenoughbeans 6y agoHi Lukas! We use JOOQ at our company and it has been a great experience. Thanks for all your hard work.
- lukaseder 6y agoYou're welcome :)
- saganus 6y agoI love JOOQ. Thanks for the great work!
- lukaseder 6y agoCheers! :)
- specialist 6y agoTangentially. From the OC: > I don’t have an MBA, but this to me is a clear case of The Innovator’s Dilemma. PostgreSQL protocol v3 is the incumbent, and protocol v4 and/or other protocols like HTTP are the potential disruptive innovators. I have no idea what Christensen means by "disruptive technology". https://en.wikipedia.org/wiki/Disruptive_innovation https://en.wikipedia.org/wiki/Disruptive_innovation The fifth qualifier "New firm's business model differs significantly from incumbent" seems important. The rest feel like Just So stories. Especially "Originate in low-end (less demanding customers) or new market (where none existed) footholds". I think time has shown Peter Drucker to be more right. First target affluent customers and then ride the cost curve down towards market success. I just reread Innovator's Dilemma, and searched the rest. Nothing about cost curves. I've been trying to think of how "disruption" would apply to Tesla vehicles and Apple Silicon. Neither products came in as "toys". Rather, I think both initially targeted very high end niches (novel use cases), and the underlying enabling technology was just starting their cost trajectories, whereas the incumbent mass market products were at the end of their trajectories. Plough metric tons of cash into R&D and wait for the new to overtake the old. -- Back to AWS Babelfish. AWS adds TDS and T-SQL compatibility to steal business from Microsoft Azure. Just some more bullet list items for middle management purchase bots to check off. It'll surely improve, but switchover costs for non-trivial projects would still be pretty high. Back when I was warehousing medical records on both MSSQL and Oracle, all the hard work was tweaking the physical design, eg codesigning the indices and queries, to make stuff performant. I'm sure that's still true. I think adding these features to Postgres is good for everyone but Microsoft. -- I guess I object to the common use of "disruption". As though there's an element of surprise. I'd suggest the OP framed AWS Babelfish in terms of competitive threats (aka insurmountable opportunities). And maybe predict some unforeseen consequences. On the positive side: The meaning of "disruptive innovation" wonderfully Agile. Perfect for consultants, pundits, and polywogs.
- andrewmcwatters 6y ago[deleted]