12 ms·

Does the Database Community Have an Identity Crisis?

93 points by luu 11y ago

formula1 11y ago
Very cool article. Tbh, I think instruction is not nearly as helpful as action. We sll can complain about what other people should do but wjat can we do to show how what we do. Nonetheless, I think its an important articke since databases come and go with few dticking
throwaway_exer 11y ago
I read the link. As a working DBA, it's breathless drivel. Unless you're an academic trying to get a grant or tenure, of course.
- koverstreet 11y ago
  What a singularly useless response. Disagreeing is fine, but "breathless drivel"? Come on, this isn't reddit.
  - deleted 11y ago
    [deleted]
- sqldba 11y ago
  I couldn't put it better myself and it's sad others are so shortsighted you got instantly shut down simply for not elucidating the blaringly obvious. We have a researcher saying hey you DBAs who are busy building using existing systems should be making your own systems; and in the same breathe acknowledging that takes 25+ people and a never ending bucket of money. Riiiiight. Boo hoo SQL + DocumentDB isn't enough? I never saw any justification for the premise.
genericpseudo 11y ago
The central tension alluded to here is "database people" vs "big data people". Which is a real dichotomy – as a "big data person" I've got a lot of leverage out of going "c'mon, we don't need Hadoop, we just need a relational database here, let's use Postgres", which is an option which gets culturally dismissed rather than on technical grounds. Me, I'm happy that I get to do less work and look clever while I'm doing it, but... Marketing and cultural positioning matter. That really isn't news. But when people have identity invested in denying that, on both sides, it's difficult to overcome.
- sgt101 11y ago
  In almost all applications using Postgres (or virtually any other decent RDBMS system) is the right technical choice. But (and this may be cultural) wearing my "where are we in 2 years" hat makes me push for Hadoop. The why is this - data and access to data is now trumping applications for business value. Where as I can run almost any of my applications on Postgres (almost) and many of them can co-habit on one server relatively happily, I cannot run all of them on it. This means that I will end up moving data from one of them to the others, but which? Alternatively I will shift the data from the RDBMS to Hadoop (or another data warehouse solution if I am working in oil-and-gas-three-years-ago or a hedge-fund-or-something now) where the insight that I am being kicked for daily can be extracted by combining the data product of 20 systems. Or we can use Hadoop as the core of the data system and feed RDBMS slaves (or use Impala sometimes). This has the benefit that the data is managed and clean (ho ho ho) on the core system and we can move at the speed of security checking and data understanding rather than the speed of budgets for writing and running data extracts.
  - jldugger 11y ago
    > But (and this may be cultural) wearing my "where are we in 2 years" hat makes me push for Hadoop. Google's F1 paper makes it clear that they were running Adwords, their most (maybe only) important product, on MySQL on Jan 1 2012. GFS and Big Table were not enough, and until Hadoop provides a robust ACID compliant data store, you should probably keep your pgsql books around.
    - sgt101 11y ago
      Agree - there's no doubt that where transactions are needed you need something that can manage transactions (in the particular way you need them managed). I actually think that transactions on Hadoop should be strictly limited to data import a-la HIVE but my arguement is that the enterprise data master is best on Hadoop rather in the slave systems.
  - IndianAstronaut 11y ago
    >But (and this may be cultural) wearing my "where are we in 2 years" hat makes me push for Hadoop This has been my experience as well. Worked on a data warehousing project that eventually had to get moved to Hadoop because management underestimated the scale of the data. We could have saved a lot of pain if we just started on Hadoop.
escherize 11y ago
I was hoping this would be about PLace Oriented Programming (PLOP) which confuses identity with state. It's a problem for OOP in general, where the identity of objects is predicated on their particular state. The talk http://www.infoq.com/presentations/Are-We-There-Yet-Rich-Hickey http://www.infoq.com/presentations/Are-We-There-Yet-Rich-Hic... goes over the idea in more detail.
- seanmcdirmid 11y ago
  Huh? The identity of an object allows for mutable state, it doesn't change with its state. I am the same person I was yesterday even if I'm in a different place or I cut my hair. State is predicated on identity, not the other way around (and given identity, you can have any kind of mutable state even in a language that doesn't support mutation, since an identity can be used as a key in an immutable map, changing the map changes the state where identity provides for a constant frame of reference).
  - aarpmcgee 11y ago
    "I am the same person I was yesterday even if I'm in a different place or I cut my hair." Think so?
    - fleitz 11y ago
      lol, can't stop laughing, it's too bad the comments that reveal truths about the world are so often the ones that get down voted. I'm sure eventually they will run through enough yesterdays to realize the error of their assertion.
    - amelius 11y ago
      The only constant is that we are slaves of our future selves.
    - seanmcdirmid 11y ago
      My identity is constant even if my state isn't.
  - 11y ago
filereaper 11y ago
'The article titled “Architecture of Database System” should be considered harmful' Not sure why that article is considered harmful, I read through all of it, it was a fantastic database resource.
- makomk 11y ago
  Traditional database design is considered unfashionable these days. You're meant to use more modern designs that eschew old-fashioned ideas like SQL and not causing massive data loss.
gfody 11y ago
RDBMSs are as powerful as ever and are still improving rapidly. If you look at "what's new" in Oracle 12c, SQL Server 2016, VoltDB and Vertica; you can see they're not maintenance mode. I think calling them fossils that should be burned down to fuel the next thing is way off. IMO the main thing RDBMSs have failed to do is to capture the new generation of developers who don't have the desire or the patience to learn SQL, relational concepts, various modeling techniques, and whatever sharding or shared nothing architectural fundamentals just to scale out. This generation wants to "npm install magic-database", write everything in javascript, and push to the cloud without ever having to think about scaleout. This isn't necessarily a problem with RDBMSs but RDBMSs could address it. [1] https://docs.oracle.com/database/121/NEWFT/chapter12102.htm https://docs.oracle.com/database/121/NEWFT/chapter12102.htm [2] https://msdn.microsoft.com/en-us/library/bb510411.aspx https://msdn.microsoft.com/en-us/library/bb510411.aspx [3] https://docs.voltdb.com/ReleaseNotes https://docs.voltdb.com/ReleaseNotes [4] https://community.dev.hpe.com/t5/Vertica-Blog/What-s-New-in-Vertica-7-2-2/ba-p/235496 https://community.dev.hpe.com/t5/Vertica-Blog/What-s-New-in-...
- Retra 11y ago
  >IMO the main thing RDBMSs have failed to do is to capture the new generation of developers who don't have the desire or the patience to learn [...] As one of this 'new generation of developers' (who happens to also work on industry database internals), I would say that I've been enormously unimpressed by the amount of specialized knowledge needed to implement an optimized, portable, flexible, scalable database. Do you know what tables in your DB need indexes and why? Do you know how to optimally set up your partitions? Do you know why your one query runs fast for a week and then suddenly runs slow, and what you can do about it? The database API is just too low-level for most people's comfort zone. Most people who want to use a database are just trying to stuff organized bits somewhere and be able to get them out quickly. Databases are smart, but the admin still has to learn and learn and learn -- all this esoteric, domain specific knowledge -- just to store some damn bits. Learning to use a database feels like learning machine code in that if you suck at it, you're going to pay the price of not having the best performance. Databases market themselves as data storage applications, but they are complex programming APIs at heart. People turn to databases to quickly solve data storage problems, not because they want to interface with some unsexy, cryptic API.
mwhite 11y ago
It seems like NoSQL could be left behind except for the .001% of use cases that actually require it and can't be easily replaced with extensions or (hopefully) automatable configurations of Postgres, but it would require application-level abstractions, and the database community doesn't value those enough, as evidenced by SQLAlchemy [1] not being highlighted on the homepage of every RDBMS project because of the awesome power and flexibility it gives the developer. Specifically, a JSON column should be used to store everything other than primary keys and foreign keys, and views and indexes should be automatically created based on the schema defined in the application (i.e., get the schema from the ORM at deploy time and post the data to a schema/migration management system) using something like https://github.com/mwhite/JSONAlchemy https://github.com/mwhite/JSONAlchemy It is entirely possible to implement the CouchDB or MongoDB API on top of Postgres JSON, for instance. [1] http://www.sqlalchemy.org/ http://www.sqlalchemy.org/
- oneweekwonder 11y ago
  I have been interested in a couchdb api for postgres. But thus far could not find anything. One option is PouchDB, with levelDOWN* to push it to levelUP* to store it in postgres. But the level of abstraction feels just to much. I also found another project that basicly keeps a copy in couchdb and sync it over to pg, but stuff like attachments does not work. * I'm not 100% sure about the projects and how to accomplish it.
- ljw1001 11y ago
  >> It is entirely possible to implement the CouchDB or MongoDB API on top of Postgres JSON, for instance. Sure, but that's just because someone has gone to the trouble of building a Mongo-equivalent as a Postgres module.
sgt101 11y ago
I have a counter argument. There is now too much data system innovation - there are scores of projects each of which implements a particular idea and very few of which have a large enough community to move beyond version 0.3 We can't put our pipelines onto these because the cost of understanding and adopting them overwhelms the benefit of ditching our internal code. We are in the same situation as the NPM / Javascript folks, but our data is an enterprise asset and we have to consider the impact of a project just going away. The vendors have held us to ransom for years and we've had to break out because it had got to an industry destroying level - really the bills are material to the stock price and time and again it turns out that despite handing over $100m the features and machines you need are not included and you need to send just another $5m over. The money stopped being used for R&D when the consolidation of the 00's happened. The vendors set themselves up as vertically integrated solution providers with the technology as a lock in factor rather than a competitive differentiator. This would have worked if nothing in the economy ever changed again but oddly it turns out that our needs are radically different when competing / collaborating with Facebook & Google vs "old corp who has gone bust now" who were our traditional enemy. So we're in a fork. The old route of using a trusted technology partner has gone, they have all betrayed their customers every quarter for the last 20 years, not only can't we trust them but we can absolutely predict when and how they will screw us. We flag it for every project to our execs, it's a built in assumption. On the other hand the big hope of opensource alternatives is not arriving in the way that Linux arrived, or maybe it's arriving in the way that Linux on the desktop arrived. The solution is that the middle tier of corporates have to get more real about opensource. At the moment there's a handful of companies in the Forbes 500 who are significantly involved. CIO's and CFO's don't see the benefit, there is no case at the moment, but if there were 500 >$1m opensource programs running in the corporate world rather than 20 things would be different. There needs to be a standarization effort and a co-ordination layer. We also need to think through the value chain as well. At the moment leaving it to the market and "revenues from professional services" are not working so well, and I am very cautious about the trend to paid for bits and bobs on top of open source. The worst outcome would be to go through all this and find that we are back to vendor hell but chained up in a different basement wondering where the bastards in suits have gone. Instead we're going to be guessing that the guy in a cool t-shirt and designer jeans who's fiddling with a blow-torch is going to want something soon. It's up to us to work out a new model, but what?
Dowwie 11y ago
"And it works on real problems, like combating human trafficking" well since he put it that way, why not?!
kempe 11y ago
RDBMS are great for a lot of situations. Alternatives like NoSql which stands for Not Only Sql, can be handy for other situations like how easy it is there to recursively find relations of x^n that are way more annoying to write in t-sql. Identity crises might be because RDBMS is not the only approach anymore?

93 points by luu 11y ago

Very cool article. Tbh, I think instruction is not nearly as helpful as action. We sll can complain about what other people should do but wjat can we do to show how what we do. Nonetheless, I think its an important articke since databases come and go with few dticking

throwaway_exer 11y ago

I read the link. As a working DBA, it's breathless drivel. Unless you're an academic trying to get a grant or tenure, of course.

koverstreet 11y ago
What a singularly useless response. Disagreeing is fine, but "breathless drivel"? Come on, this isn't reddit.
- deleted 11y ago
  [deleted]
sqldba 11y ago
I couldn't put it better myself and it's sad others are so shortsighted you got instantly shut down simply for not elucidating the blaringly obvious. We have a researcher saying hey you DBAs who are busy building using existing systems should be making your own systems; and in the same breathe acknowledging that takes 25+ people and a never ending bucket of money. Riiiiight. Boo hoo SQL + DocumentDB isn't enough? I never saw any justification for the premise.

genericpseudo 11y ago

The central tension alluded to here is "database people" vs "big data people". Which is a real dichotomy – as a "big data person" I've got a lot of leverage out of going "c'mon, we don't need Hadoop, we just need a relational database here, let's use Postgres", which is an option which gets culturally dismissed rather than on technical grounds. Me, I'm happy that I get to do less work and look clever while I'm doing it, but... Marketing and cultural positioning matter. That really isn't news. But when people have identity invested in denying that, on both sides, it's difficult to overcome.

sgt101 11y ago
In almost all applications using Postgres (or virtually any other decent RDBMS system) is the right technical choice. But (and this may be cultural) wearing my "where are we in 2 years" hat makes me push for Hadoop. The why is this - data and access to data is now trumping applications for business value. Where as I can run almost any of my applications on Postgres (almost) and many of them can co-habit on one server relatively happily, I cannot run all of them on it. This means that I will end up moving data from one of them to the others, but which? Alternatively I will shift the data from the RDBMS to Hadoop (or another data warehouse solution if I am working in oil-and-gas-three-years-ago or a hedge-fund-or-something now) where the insight that I am being kicked for daily can be extracted by combining the data product of 20 systems. Or we can use Hadoop as the core of the data system and feed RDBMS slaves (or use Impala sometimes). This has the benefit that the data is managed and clean (ho ho ho) on the core system and we can move at the speed of security checking and data understanding rather than the speed of budgets for writing and running data extracts.
- jldugger 11y ago
  > But (and this may be cultural) wearing my "where are we in 2 years" hat makes me push for Hadoop. Google's F1 paper makes it clear that they were running Adwords, their most (maybe only) important product, on MySQL on Jan 1 2012. GFS and Big Table were not enough, and until Hadoop provides a robust ACID compliant data store, you should probably keep your pgsql books around.
  - sgt101 11y ago
    Agree - there's no doubt that where transactions are needed you need something that can manage transactions (in the particular way you need them managed). I actually think that transactions on Hadoop should be strictly limited to data import a-la HIVE but my arguement is that the enterprise data master is best on Hadoop rather in the slave systems.
- IndianAstronaut 11y ago
  >But (and this may be cultural) wearing my "where are we in 2 years" hat makes me push for Hadoop This has been my experience as well. Worked on a data warehousing project that eventually had to get moved to Hadoop because management underestimated the scale of the data. We could have saved a lot of pain if we just started on Hadoop.

escherize 11y ago

I was hoping this would be about PLace Oriented Programming (PLOP) which confuses identity with state. It's a problem for OOP in general, where the identity of objects is predicated on their particular state. The talk http://www.infoq.com/presentations/Are-We-There-Yet-Rich-Hickey http://www.infoq.com/presentations/Are-We-There-Yet-Rich-Hic... goes over the idea in more detail.

seanmcdirmid 11y ago
Huh? The identity of an object allows for mutable state, it doesn't change with its state. I am the same person I was yesterday even if I'm in a different place or I cut my hair. State is predicated on identity, not the other way around (and given identity, you can have any kind of mutable state even in a language that doesn't support mutation, since an identity can be used as a key in an immutable map, changing the map changes the state where identity provides for a constant frame of reference).
- aarpmcgee 11y ago
  "I am the same person I was yesterday even if I'm in a different place or I cut my hair." Think so?
  - fleitz 11y ago
    lol, can't stop laughing, it's too bad the comments that reveal truths about the world are so often the ones that get down voted. I'm sure eventually they will run through enough yesterdays to realize the error of their assertion.
  - amelius 11y ago
    The only constant is that we are slaves of our future selves.
  - seanmcdirmid 11y ago
    My identity is constant even if my state isn't.
- 11y ago

filereaper 11y ago

'The article titled “Architecture of Database System” should be considered harmful' Not sure why that article is considered harmful, I read through all of it, it was a fantastic database resource.

makomk 11y ago
Traditional database design is considered unfashionable these days. You're meant to use more modern designs that eschew old-fashioned ideas like SQL and not causing massive data loss.

gfody 11y ago

RDBMSs are as powerful as ever and are still improving rapidly. If you look at "what's new" in Oracle 12c, SQL Server 2016, VoltDB and Vertica; you can see they're not maintenance mode. I think calling them fossils that should be burned down to fuel the next thing is way off. IMO the main thing RDBMSs have failed to do is to capture the new generation of developers who don't have the desire or the patience to learn SQL, relational concepts, various modeling techniques, and whatever sharding or shared nothing architectural fundamentals just to scale out. This generation wants to "npm install magic-database", write everything in javascript, and push to the cloud without ever having to think about scaleout. This isn't necessarily a problem with RDBMSs but RDBMSs could address it. [1] https://docs.oracle.com/database/121/NEWFT/chapter12102.htm https://docs.oracle.com/database/121/NEWFT/chapter12102.htm [2] https://msdn.microsoft.com/en-us/library/bb510411.aspx https://msdn.microsoft.com/en-us/library/bb510411.aspx [3] https://docs.voltdb.com/ReleaseNotes https://docs.voltdb.com/ReleaseNotes [4] https://community.dev.hpe.com/t5/Vertica-Blog/What-s-New-in-Vertica-7-2-2/ba-p/235496 https://community.dev.hpe.com/t5/Vertica-Blog/What-s-New-in-...

Retra 11y ago
>IMO the main thing RDBMSs have failed to do is to capture the new generation of developers who don't have the desire or the patience to learn [...] As one of this 'new generation of developers' (who happens to also work on industry database internals), I would say that I've been enormously unimpressed by the amount of specialized knowledge needed to implement an optimized, portable, flexible, scalable database. Do you know what tables in your DB need indexes and why? Do you know how to optimally set up your partitions? Do you know why your one query runs fast for a week and then suddenly runs slow, and what you can do about it? The database API is just too low-level for most people's comfort zone. Most people who want to use a database are just trying to stuff organized bits somewhere and be able to get them out quickly. Databases are smart, but the admin still has to learn and learn and learn -- all this esoteric, domain specific knowledge -- just to store some damn bits. Learning to use a database feels like learning machine code in that if you suck at it, you're going to pay the price of not having the best performance. Databases market themselves as data storage applications, but they are complex programming APIs at heart. People turn to databases to quickly solve data storage problems, not because they want to interface with some unsexy, cryptic API.

mwhite 11y ago

It seems like NoSQL could be left behind except for the .001% of use cases that actually require it and can't be easily replaced with extensions or (hopefully) automatable configurations of Postgres, but it would require application-level abstractions, and the database community doesn't value those enough, as evidenced by SQLAlchemy [1] not being highlighted on the homepage of every RDBMS project because of the awesome power and flexibility it gives the developer. Specifically, a JSON column should be used to store everything other than primary keys and foreign keys, and views and indexes should be automatically created based on the schema defined in the application (i.e., get the schema from the ORM at deploy time and post the data to a schema/migration management system) using something like https://github.com/mwhite/JSONAlchemy https://github.com/mwhite/JSONAlchemy It is entirely possible to implement the CouchDB or MongoDB API on top of Postgres JSON, for instance. [1] http://www.sqlalchemy.org/ http://www.sqlalchemy.org/

oneweekwonder 11y ago
I have been interested in a couchdb api for postgres. But thus far could not find anything. One option is PouchDB, with levelDOWN* to push it to levelUP* to store it in postgres. But the level of abstraction feels just to much. I also found another project that basicly keeps a copy in couchdb and sync it over to pg, but stuff like attachments does not work. * I'm not 100% sure about the projects and how to accomplish it.
ljw1001 11y ago
>> It is entirely possible to implement the CouchDB or MongoDB API on top of Postgres JSON, for instance. Sure, but that's just because someone has gone to the trouble of building a Mongo-equivalent as a Postgres module.

sgt101 11y ago

I have a counter argument. There is now too much data system innovation - there are scores of projects each of which implements a particular idea and very few of which have a large enough community to move beyond version 0.3 We can't put our pipelines onto these because the cost of understanding and adopting them overwhelms the benefit of ditching our internal code. We are in the same situation as the NPM / Javascript folks, but our data is an enterprise asset and we have to consider the impact of a project just going away. The vendors have held us to ransom for years and we've had to break out because it had got to an industry destroying level - really the bills are material to the stock price and time and again it turns out that despite handing over $100m the features and machines you need are not included and you need to send just another $5m over. The money stopped being used for R&D when the consolidation of the 00's happened. The vendors set themselves up as vertically integrated solution providers with the technology as a lock in factor rather than a competitive differentiator. This would have worked if nothing in the economy ever changed again but oddly it turns out that our needs are radically different when competing / collaborating with Facebook & Google vs "old corp who has gone bust now" who were our traditional enemy. So we're in a fork. The old route of using a trusted technology partner has gone, they have all betrayed their customers every quarter for the last 20 years, not only can't we trust them but we can absolutely predict when and how they will screw us. We flag it for every project to our execs, it's a built in assumption. On the other hand the big hope of opensource alternatives is not arriving in the way that Linux arrived, or maybe it's arriving in the way that Linux on the desktop arrived. The solution is that the middle tier of corporates have to get more real about opensource. At the moment there's a handful of companies in the Forbes 500 who are significantly involved. CIO's and CFO's don't see the benefit, there is no case at the moment, but if there were 500 >$1m opensource programs running in the corporate world rather than 20 things would be different. There needs to be a standarization effort and a co-ordination layer. We also need to think through the value chain as well. At the moment leaving it to the market and "revenues from professional services" are not working so well, and I am very cautious about the trend to paid for bits and bobs on top of open source. The worst outcome would be to go through all this and find that we are back to vendor hell but chained up in a different basement wondering where the bastards in suits have gone. Instead we're going to be guessing that the guy in a cool t-shirt and designer jeans who's fiddling with a blow-torch is going to want something soon. It's up to us to work out a new model, but what?

Dowwie 11y ago

"And it works on real problems, like combating human trafficking" well since he put it that way, why not?!

kempe 11y ago

RDBMS are great for a lot of situations. Alternatives like NoSql which stands for Not Only Sql, can be handy for other situations like how easy it is there to recursively find relations of x^n that are way more annoying to write in t-sql. Identity crises might be because RDBMS is not the only approach anymore?