6 ms·

Spreadsheets are not databases... but everyone insists on using them as such, so this happens.

by jarfil 8y ago

deleted 8y ago
[deleted]
hk__2 8y ago
> Spreadsheets are not databases... but everyone insists on using them as such, so this happens. They are, by Wikipedia’s definition: "A database is an organized collection of data, stored and accessed electronically".
- johnchristopher 8y ago
  So is a bunch of files in a folder then.
  - olooney 8y ago
    That's right. Many databases are literally just folders of files once you look under the hood - for example, here is PostgreSQL's layout: https://www.postgresql.org/docs/9.4/static/storage-file-layout.html https://www.postgresql.org/docs/9.4/static/storage-file-layo... Note in particular the comment "Each table and index is stored in a separate file" which shows that the file layout is not trivial. SQLite is another database, but it also is "just files in a folder:" the main difference is that there is a single file for all the tables and indexes but it uses several other files in the same directory as well: https://www.sqlite.org/tempfiles.html https://www.sqlite.org/tempfiles.html Note that MS SQL Server uses much the same strategy as SQLite, although you have the option of splitting it across multiple OS files if you need more control over physical storage: https://docs.microsoft.com/en-us/sql/relational-databases/databases/database-files-and-filegroups?view=sql-server-2017 https://docs.microsoft.com/en-us/sql/relational-databases/da... But if SQLite is a database, how is .xlsx or .ini or .json fundamentally any different? I don't think there's anywhere you can draw a bright line that will include all well known databases but exclude things that "merely" file formats. If we draw the line at "transactions," for example, we'll find that MongoDB and ElasticSearch no longer count. If we draw the line at "user access control" then SQLite and redis are not databases. Older RDBMSes like Oracle may eschew OS files in an attempt to gain more control but they can't escape the fact that they too are are fundamentally just "organized collection of data, stored and accessed electronically." And then there are in-memory databases - which do not use the disk at all, but are plainly still considered "databases" - it's right there in the name! These varied examples show that the word "database" is used very broadly and any definition narrower than "A database is an organized collection of data, stored and accessed electronically" will certainly fail to include many instances of things which are commonly referred to as databases.
- systoll 8y ago
  > "A spreadsheet is an interactive computer application for organization, analysis and storage of data in tabular form." By wikipedia's definition, the application Excel is the spreadsheet. I wouldn't call Excel.exe an organised collection of data. On the other hand, the relevant articles also state " A spreadsheet may also refer to one such electronic document" and "the term "database" is often used casually to refer to both a database and the DBMS used to manipulate it." I think we both know the parent's point -- Spreadsheet applications aren't general purpose database management systems; and most databases can't be cleanly implemented as spreadsheet documents.
deleted 8y ago
[deleted]
dwaltrip 8y ago
If everyone uses something as an "x", doesn't that make it an "x"? The meaning of a word is determined by its usage. Perhaps we should ask why people use it that way, and what can be done to improve the situation, instead of arguing that the whole world is doing it wrong.
johnchristopher 8y ago
Still better than a colleague of mine who insists on using word to display tables because she knows where the sort button is.
mr_toad 8y ago
Because it takes 6 months to get approval to have a new database created, two weeks to actually get access to it after it is created, and then every single database query has to go through code review and change control. Oh, and everything must be a stored procedure, no exceptions.
- lostlogin 8y ago
  And it’ll be billed out at some ridiculous fee because there are 9 layers of ticket clickers, each of which filter any request made such that the actually contractor gets told a story very different from what was intended. Academia is a strange land.
- flukus 8y ago
  Or they could use sqlite or even access, both of which are single files and should take about the same admin overhead of a new text file. Speaking of text files, they might be a better option here too. If it's being handled by excel already then sqlite would be overkill.
Nasrudith 8y ago
I think that it is a result of training or lack of it. Spreadsheets don't scale well and have validation issues but are accessible. That is its own skillset unfortunately to do properly. I have seen scientific code that is frankly painful but their flaws are understandable as I would probably make a hash out of their subject.
- michaericalribo 8y ago
  I think you really hit on the crux of the problem here. The training needed to even put together a genetics paper, much less published and read by others in the field, is /easily/ 7 years of graduate work—NONE of which is spent learning to code! There ARE better tools, but the incentives aren’t aligned for genomics researchers to learn them. Which is why bioinformatics is such a great field! Biologists are aware of the problem, and better late than never...if you’re interested in being part of the solution to this problem, there has never been a better time to get started!
  - mbreese 8y ago
    It's not that incentives aren't aligned for people to start using better tools, it's that Excel is largely good enough. So we all deal with stupid Excel hashing of specific genes (because we all know which genes are this issue by this point). This isn't exactly a new phenomenon. When I was in grad school (early 2000s), using Excel for gene names was already a well known issue. Some researchers are never going to use anything other than Excel for this even when people like me harp on them to stop treating Excel as a database. It's "good-enough", so it gets used more often than it should.

Spreadsheets are not databases... but everyone insists on using them as such, so this happens.

by jarfil 8y ago

Spreadsheets are not databases... but everyone insists on using them as such, so this happens.

deleted 8y ago

[deleted]

hk__2 8y ago

> Spreadsheets are not databases... but everyone insists on using them as such, so this happens. They are, by Wikipedia’s definition: "A database is an organized collection of data, stored and accessed electronically".

johnchristopher 8y ago
So is a bunch of files in a folder then.
- olooney 8y ago
  That's right. Many databases are literally just folders of files once you look under the hood - for example, here is PostgreSQL's layout: https://www.postgresql.org/docs/9.4/static/storage-file-layout.html https://www.postgresql.org/docs/9.4/static/storage-file-layo... Note in particular the comment "Each table and index is stored in a separate file" which shows that the file layout is not trivial. SQLite is another database, but it also is "just files in a folder:" the main difference is that there is a single file for all the tables and indexes but it uses several other files in the same directory as well: https://www.sqlite.org/tempfiles.html https://www.sqlite.org/tempfiles.html Note that MS SQL Server uses much the same strategy as SQLite, although you have the option of splitting it across multiple OS files if you need more control over physical storage: https://docs.microsoft.com/en-us/sql/relational-databases/databases/database-files-and-filegroups?view=sql-server-2017 https://docs.microsoft.com/en-us/sql/relational-databases/da... But if SQLite is a database, how is .xlsx or .ini or .json fundamentally any different? I don't think there's anywhere you can draw a bright line that will include all well known databases but exclude things that "merely" file formats. If we draw the line at "transactions," for example, we'll find that MongoDB and ElasticSearch no longer count. If we draw the line at "user access control" then SQLite and redis are not databases. Older RDBMSes like Oracle may eschew OS files in an attempt to gain more control but they can't escape the fact that they too are are fundamentally just "organized collection of data, stored and accessed electronically." And then there are in-memory databases - which do not use the disk at all, but are plainly still considered "databases" - it's right there in the name! These varied examples show that the word "database" is used very broadly and any definition narrower than "A database is an organized collection of data, stored and accessed electronically" will certainly fail to include many instances of things which are commonly referred to as databases.
systoll 8y ago
> "A spreadsheet is an interactive computer application for organization, analysis and storage of data in tabular form." By wikipedia's definition, the application Excel is the spreadsheet. I wouldn't call Excel.exe an organised collection of data. On the other hand, the relevant articles also state " A spreadsheet may also refer to one such electronic document" and "the term "database" is often used casually to refer to both a database and the DBMS used to manipulate it." I think we both know the parent's point -- Spreadsheet applications aren't general purpose database management systems; and most databases can't be cleanly implemented as spreadsheet documents.

deleted 8y ago

[deleted]

dwaltrip 8y ago

If everyone uses something as an "x", doesn't that make it an "x"? The meaning of a word is determined by its usage. Perhaps we should ask why people use it that way, and what can be done to improve the situation, instead of arguing that the whole world is doing it wrong.

johnchristopher 8y ago

Still better than a colleague of mine who insists on using word to display tables because she knows where the sort button is.

mr_toad 8y ago

Because it takes 6 months to get approval to have a new database created, two weeks to actually get access to it after it is created, and then every single database query has to go through code review and change control. Oh, and everything must be a stored procedure, no exceptions.

lostlogin 8y ago
And it’ll be billed out at some ridiculous fee because there are 9 layers of ticket clickers, each of which filter any request made such that the actually contractor gets told a story very different from what was intended. Academia is a strange land.
flukus 8y ago
Or they could use sqlite or even access, both of which are single files and should take about the same admin overhead of a new text file. Speaking of text files, they might be a better option here too. If it's being handled by excel already then sqlite would be overkill.

Nasrudith 8y ago

I think that it is a result of training or lack of it. Spreadsheets don't scale well and have validation issues but are accessible. That is its own skillset unfortunately to do properly. I have seen scientific code that is frankly painful but their flaws are understandable as I would probably make a hash out of their subject.

michaericalribo 8y ago
I think you really hit on the crux of the problem here. The training needed to even put together a genetics paper, much less published and read by others in the field, is /easily/ 7 years of graduate work—NONE of which is spent learning to code! There ARE better tools, but the incentives aren’t aligned for genomics researchers to learn them. Which is why bioinformatics is such a great field! Biologists are aware of the problem, and better late than never...if you’re interested in being part of the solution to this problem, there has never been a better time to get started!
- mbreese 8y ago
  It's not that incentives aren't aligned for people to start using better tools, it's that Excel is largely good enough. So we all deal with stupid Excel hashing of specific genes (because we all know which genes are this issue by this point). This isn't exactly a new phenomenon. When I was in grad school (early 2000s), using Excel for gene names was already a well known issue. Some researchers are never going to use anything other than Excel for this even when people like me harp on them to stop treating Excel as a database. It's "good-enough", so it gets used more often than it should.