3 ms·

A lot of people probably run into problems from poor design, poor programming, or just inability to problem solve. For example (two real examples I've faced):

A lot of people probably run into problems from poor design, poor programming, or just inability to problem solve. For example (two real examples I've faced): (1) Company xyz is having problems generating reports that pull from an sql database. They take too long, IT says it's because the data is too large and their hardware is old. Obviously this is big data! Wrong, re-writing the sql queries and indexing some fields solved the problem. Like, beginner-level stuff. Nobody had even looked at the code or structure of the database when initially trying to solve the problem. or (2) Company xyz can't load a dataset into memory to perform some computations on it, it's too large. Needs better hardware or big data solution obviously. With millions of rows and many columns, ok. Only 6 of those columns were even used by the processing job, and a fraction of the rows were relevant. There was no need to fit the entire dataset into memory in the first place, problem solved. I could go on. Lots of companies just lack the skills and knowledge to do things right, and have no idea what's going on under the hood or how things work. So naturally, they are drawn to the marketing buzz surrounding big data, cloud computing, data warehousing, machine learning, etc etc etc.

polsoul 11y ago
Being a dba for a decade already, resolving issues similar to the ones you described, are the sweatest I've experienced in my professional career. Unfortunately, I don't believe that fixing the problem in the database by understanding and resolving it in a logical manner is the way things will be in the near future. You know, RDBMSs aren't sexy ... developers developers developers , web developers web developers web developers :) the newest API and/or language are much more important nowadays for the newcomers. I cannot believe it, it's out of my mind, but that's the reality. Noone wants to utilize the database at its fullness anymore, everything needs to be done at the gazilion of app servers by using java/.net or the trendy 3rd/4th generation language of the day. My only hope is that the data is the thing that is not going away, and this forces us to think in a logical way about it(eventually devs/web devs will start doing the data maniplation and etc the right way, I guess you're pretty familiar with the approach in mind). Thanks
- nostrademons 11y ago
  I remember c. 2002-2005, it was RDBMSes that were the overused, overhyped solution. Basically every webapp had to be backed by a database. Everybody was moving to 3-tier architectures (even for things as simple as a TODO list), frameworks like Struts and Rails and Symphony and Django all sprung up to manage the complexity of talking to the DB, you'd see all these tips on blogs (themselves among the first database-backed webapps, even though they worked perfectly well as flat files) about how to normalize your schema and shard your data for scale and pick ID spaces and make your joins perform well. Meanwhile, a small minority of people were like "Guys? GUYS! You can just use flat files and in-memory data structures for that. Why bother with a database when 3 hashtables and a list will do the same thing several orders of magnitude faster." They had outsized accomplishments relative to their shrillness, though: Yahoo, Google, PlentyOfFish, Mailinator, ViaWeb, Hacker News were all built primarily with in-memory data structures and flat files. These things run in cycles. The meta-lesson is to let your problem dictate your technology choice instead of having your technology choice dictate your problem, and build for the problem you have now rather than worry about the problems that other people have. For many apps, a hashtable is still the right solution to get a v0.1 off the ground, and it will scale to hundreds of thousands of users.
  - polsoul 11y ago
    Fully agree with you - there are problems and problems, and 'use the right tool for the right job' should always be the way to go. hashtalbe or flat files,huh, hard to believe this is going to be useful for a non-trivial app as 'data storage engine'(so to say). You will have to implement everything yourself: - concurrency - several users updating one and the same entry - data consistency - to return the data as of the start of the 'select' for the user,what about others who updated the entry but still didn't commit. - security - fine grain access to specific data entries - what can be extracted by whom, what about auditing(who did what at what time and etc.) - backup/recovery and etc. More code most of the time means more bugs. Not everyone is Google , Amazon or Facebook to have the resources to create good custom solutions and support, improve them. For most cases, a simple rdbms database will provide you with enough 'standard' functionality to not reinvent the wheel(think security,concurrency,consistency and etc.) = to code these on your own from scratch. But in the end, it's again the nature of the problem(whether at all you'll need security, concurrency or consistency and etc.). It's nice to know what the database as a tool gives you , and I don't think this is the case nowadays - too many java/.net, and many other developers , have no idea what they can get out of a database. And I really hate when one spent several weeks to write something that could be done in several hours by using a specific database feature.

polsoul 11y ago

Being a dba for a decade already, resolving issues similar to the ones you described, are the sweatest I've experienced in my professional career. Unfortunately, I don't believe that fixing the problem in the database by understanding and resolving it in a logical manner is the way things will be in the near future. You know, RDBMSs aren't sexy ... developers developers developers , web developers web developers web developers :) the newest API and/or language are much more important nowadays for the newcomers. I cannot believe it, it's out of my mind, but that's the reality. Noone wants to utilize the database at its fullness anymore, everything needs to be done at the gazilion of app servers by using java/.net or the trendy 3rd/4th generation language of the day. My only hope is that the data is the thing that is not going away, and this forces us to think in a logical way about it(eventually devs/web devs will start doing the data maniplation and etc the right way, I guess you're pretty familiar with the approach in mind). Thanks

nostrademons 11y ago
I remember c. 2002-2005, it was RDBMSes that were the overused, overhyped solution. Basically every webapp had to be backed by a database. Everybody was moving to 3-tier architectures (even for things as simple as a TODO list), frameworks like Struts and Rails and Symphony and Django all sprung up to manage the complexity of talking to the DB, you'd see all these tips on blogs (themselves among the first database-backed webapps, even though they worked perfectly well as flat files) about how to normalize your schema and shard your data for scale and pick ID spaces and make your joins perform well. Meanwhile, a small minority of people were like "Guys? GUYS! You can just use flat files and in-memory data structures for that. Why bother with a database when 3 hashtables and a list will do the same thing several orders of magnitude faster." They had outsized accomplishments relative to their shrillness, though: Yahoo, Google, PlentyOfFish, Mailinator, ViaWeb, Hacker News were all built primarily with in-memory data structures and flat files. These things run in cycles. The meta-lesson is to let your problem dictate your technology choice instead of having your technology choice dictate your problem, and build for the problem you have now rather than worry about the problems that other people have. For many apps, a hashtable is still the right solution to get a v0.1 off the ground, and it will scale to hundreds of thousands of users.
- polsoul 11y ago
  Fully agree with you - there are problems and problems, and 'use the right tool for the right job' should always be the way to go. hashtalbe or flat files,huh, hard to believe this is going to be useful for a non-trivial app as 'data storage engine'(so to say). You will have to implement everything yourself: - concurrency - several users updating one and the same entry - data consistency - to return the data as of the start of the 'select' for the user,what about others who updated the entry but still didn't commit. - security - fine grain access to specific data entries - what can be extracted by whom, what about auditing(who did what at what time and etc.) - backup/recovery and etc. More code most of the time means more bugs. Not everyone is Google , Amazon or Facebook to have the resources to create good custom solutions and support, improve them. For most cases, a simple rdbms database will provide you with enough 'standard' functionality to not reinvent the wheel(think security,concurrency,consistency and etc.) = to code these on your own from scratch. But in the end, it's again the nature of the problem(whether at all you'll need security, concurrency or consistency and etc.). It's nice to know what the database as a tool gives you , and I don't think this is the case nowadays - too many java/.net, and many other developers , have no idea what they can get out of a database. And I really hate when one spent several weeks to write something that could be done in several hours by using a specific database feature.