6 ms·

With a new tool, spreadsheet users can construct custom database interfaces

89 points by renafowler 10y ago

Marionetic 10y ago
really cool, easy start to any project
zyxley 10y ago
So... they've reinvented Access?
- adzicg 10y ago
  seems to be more geared towards reporting, so reinvented 'oracle forms' probably.
  - icebraining 10y ago
    Or Filemaker, though the new versions seem more complex.
  - thorin 10y ago
    Weird comment as oracle reports as their reporting tool. Forums was great for crud only business applications, actually the majority of business applications but it went downhill when it moved from client server to Web using applets... Apex is a good replacement.
- minimaxir 10y ago
  > The tool’s scores put it at the 52nd percentile in the category of business software, which isn’t bad for an academic research project. But the scores for Microsoft’s Access database program are much worse — around the sixth percentile It is apparently more useable than Access, which is damning by faint praise.
- Pamar 10y ago
  In my experience (which includes large companies from sectors like automotive and utilities) in most cases there is a large, RDBMS backed application doing something really critical... And then when you talk with actual users you discover a swarm of Excel files that either support or often second guess the corporate system. And it is not just reporting. In some cases inputting large quantities of tabular data is just done by writing Excel macros that create CSV files to upload to the system (and sometimes, horrifingly, the data are just retyped...).
  - mtone 10y ago
    It's a scenario I've witnessed a few times too many now. Often an unsupported feature X is becoming critical to the team, so users reproduce large chunks of the corporate system in a separate spreadsheet in order to support the feature themselves, then build upon that. Without the ability to react quickly that smaller businesses have, I noticed that some form of resistance from IT is usually a big factor: lack of resources, divergent priorities or visions, unclear benefits from hypothetical feature, limited architecture, end-of-life maintenance states that end up lasting years, etc. Reality changes, systems fall behind, users adapt and spreadsheets are born. One root cause is that it's comparatively hard to add new data in a RDBMS - it usually requires a full development life cycle. On the other hand, spreadsheets are free, provide immediate benefits, allow users to define and refine requirements as they go, and can be abandoned at the drop of a hat (low risk). Quite the conundrum! From the article: > At present, Bakke’s tool enables query construction on an existing database, but it doesn’t enable the direct entry or modification of data. He expects to begin adding that functionality over the next six months Querying data is a problem with many good solutions nowadays, the bigger question is where and how to write that new data the users want to play with (other than in a spreadsheet).
    - EGreg 10y ago
      So why can't they use something like Irmin? https://github.com/mirage/irmin https://github.com/mirage/irmin
    - intrasight 10y ago
      >it's comparatively hard to add new data in a RDBMS I assume you meant to day "to add new schema". Adding new data is easy and in most enterprise environments is happening automatically every day. As far as "add new schema" goes, I'll claim that the database is just another software component, and if you pursuing an agile strategy, that you should be modifying that schema with every sprint. My client, for whom I do data warehouse consulting, updates the schema every two weeks. Because we are responsive to the business, they never revert to spreadsheet solutions. Of course they are free to create there own custom reports in spreadsheets - we'd never try to take that away. In fact we make it as easy as possible to run business curated queries and dump the data to Excel. Our newest feature for these power users is to allow them to give back to us the Excel reports they built and we will "push and deliver" - put the current data into the raw data sheets and deliver the report - usually to Sharepoint. Getting good business software solutions in place merely requires that the business hire some good developers. Why more don't do so remains a mystery. My best guess, like all things "corporate" is fear and laziness.
- emmelaich 10y ago
  or Query By Example (1975). But I'm sure much more. https://en.wikipedia.org/wiki/Query_by_Example https://en.wikipedia.org/wiki/Query_by_Example
- GlennS 10y ago
  If someone actually did reinvent Access without the concurrency data-loss problems (but still as a single file per database), they'd probably make a lot of money.
fspeech 10y ago
Very cool. The right side of the article provides links to a video presentation of the tool and a SIGMOD paper reporting on the details. It is specifically compared against Access in a study on usability. Link to the project page: http://people.csail.mit.edu/ebakke/sieuferd/ http://people.csail.mit.edu/ebakke/sieuferd/
alirobe 10y ago
So... Pivot Tables? Excel has done this for ages. Excel 2016 even has in-memory BI.
- LoSboccacc 10y ago
  Access had a database gui wizard since when? 97?
dhruvkar 10y ago
http://www.fieldbook.com http://www.fieldbook.com I haven't used this in production, but messed around with it for a while. Seems to do what this article is suggesting.
- nickpsecurity 10y ago
  Yeah, I reposted that one in a few places. Reason is getting people outta spreadsheets into better DB will need replacement to be just like spreadsheets. Anything too different in UI or workflow will get resistance and workarounds. Fieldbook was clever.
philprx 10y ago
Am i missing something but... ... Where is the tool ? Source code or it doesn't exist ;)
wx196 10y ago
"when you have something extremely industry-specific is, you have to hire a programmer who spends about a year of work to build a user interface for your particular domain" — with tools like Oracle APEX it can be done much faster, so I think nothing new.
- jlg23 10y ago
  As long as one needs a programmer the year is a good rough estimate, the hard part is not the building but the gathering of requirements. Take the programmer out of the equation and thus the communication overhead between programmer and domain specialist and you save a lot of development time.
  - wx196 10y ago
    You still need someone to think out relations in your database and how to retain it faster with large data sets. If this work can be done by domain specialist only, then it can be done the same way using APEX, FileMaker etc. Without live example of the new technology it is difficult to say how it differs.
  - sqldba 10y ago
    The thing is that having a tool like this does not make the requirement gathering go away. Dude I can build a house. I've only used a drill once in my life, I've never mixed concrete, or bricks, but I can. I will lay stuff on top of each other and use superglue and whatever it takes. But nobody will ever want to live in and it would not be safe. That's why you get a professional to do it. A user might somehow be able to magically skip gathering requirements and just dive gung ho into creating a structure for what they want. Great. And you end up with the legacy Excel monstrosity turds that we've been trying to eradicate from the Enterprise for the past two dozen years.
ww520 10y ago
Microsoft Access?
- ComodoHacker 10y ago
  Screenshots[1] look more like Power Pivot. 1. http://people.csail.mit.edu/ebakke/sieuferd/ http://people.csail.mit.edu/ebakke/sieuferd/
kfk 10y ago
OK... but where are decent user-friendly tools to manipulate, manage and share data and insights? Really, the problem is that to this day data in Corporations is a mess. The moment you step out of the standard warehouse systems you find yourself navigating a mess of spreadsheets, word docs, power points, all showing sometimes the same data in different formats. You duplicate data every time you send an Excel file by email... this data needs to be in sync and when not people get screamed at or fired. That means a lot of work goes into manually maintaining thousands of different files to make sure they tie to the overall picture. It's a mess. The bad part is that nobody is looking into this, everybody is focusing on analytics or reporting, which by now are the easy part. Then you have ancient tools like HFM or BPC that try to "enhance" spreadsheets and spreadsheets consolidations. They don't work. They are barely reliable and they cost a lot of money for no reason (they are simple SQL databases, they can't even be compared to much more complex software like salesforce or similar). Then Corporations are now being sold this all "big data" thing, which is old by now for tech audiences, but it's just new for most of the big companies nowadays. Unfortunately, while big data has a lot of potential, it further moves away investments on good old small data where probably there is much more ROI to grab simply because solving this problem is not that difficult and not that expensive if you want to actually fix it. Then now everybody wants to do Predictive, but Predictive won't beat saving hundreds of employees thousands of working hours by improving the efficiency of how we handle data and data insights. You can literally create a new workforce out of the many hours you would save with better efficiency in this area. Without even considering that on top of efficiency, you get much better, more fact based, decisions driven by the overall increase in transparency which you lose when insights are spread around thousands and thousands of files. There is some light at the end of the tunnel. Software solutions like Tableau go in the right direction, but they do not provide the much (more) needed tools to properly manipulate and manage data and, especially, consolidations and integration. The only way to get out of this mess is to control the data flow but especially to have 1 and only 1 data flow. That means once your insights are approved and locked, every other view should read this data, there should be no manual intervention any more. If the locked data is wrong, it will be wrong for everybody, which is a good thing compared to having multiple copies of the same and then people fighting on which one is the right one.
- sqldba 10y ago
johnmaguire2013 10y ago
This sounds a lot like AirTable[1] to me. AirTable has quickly become my favorite Spreadsheets replacement because it has a nice REST API, ways of linking data from multiple tables, data integrity principles, and pretty solid mobile phone data entry, while still maintaining the simplicity of a spreadsheet. [1] https://airtable.com/ https://airtable.com/
- krmmalik 10y ago
  I do like Airtable, and i can see the comparison but as far as i understand you cant do the complex querying that the article is suggesting?
based2 10y ago
http://people.csail.mit.edu/ebakke/ http://people.csail.mit.edu/ebakke/
infinite8s 10y ago
I'm not sure how this is any easier than using Tableau, which can construct most of these types of business analytic queries fairly easily. More complicated queries that pull in data from multiple tables usually requires an understanding of the organization of data across these tables, but that is true of this tool as well (at least from the video on the author's website - http://people.csail.mit.edu/ebakke/sieuferd/index.html http://people.csail.mit.edu/ebakke/sieuferd/index.html)

89 points by renafowler 10y ago

Marionetic 10y ago

really cool, easy start to any project

zyxley 10y ago

So... they've reinvented Access?

adzicg 10y ago
seems to be more geared towards reporting, so reinvented 'oracle forms' probably.
- icebraining 10y ago
  Or Filemaker, though the new versions seem more complex.
- thorin 10y ago
  Weird comment as oracle reports as their reporting tool. Forums was great for crud only business applications, actually the majority of business applications but it went downhill when it moved from client server to Web using applets... Apex is a good replacement.
minimaxir 10y ago
> The tool’s scores put it at the 52nd percentile in the category of business software, which isn’t bad for an academic research project. But the scores for Microsoft’s Access database program are much worse — around the sixth percentile It is apparently more useable than Access, which is damning by faint praise.
Pamar 10y ago
In my experience (which includes large companies from sectors like automotive and utilities) in most cases there is a large, RDBMS backed application doing something really critical... And then when you talk with actual users you discover a swarm of Excel files that either support or often second guess the corporate system. And it is not just reporting. In some cases inputting large quantities of tabular data is just done by writing Excel macros that create CSV files to upload to the system (and sometimes, horrifingly, the data are just retyped...).
- mtone 10y ago
  It's a scenario I've witnessed a few times too many now. Often an unsupported feature X is becoming critical to the team, so users reproduce large chunks of the corporate system in a separate spreadsheet in order to support the feature themselves, then build upon that. Without the ability to react quickly that smaller businesses have, I noticed that some form of resistance from IT is usually a big factor: lack of resources, divergent priorities or visions, unclear benefits from hypothetical feature, limited architecture, end-of-life maintenance states that end up lasting years, etc. Reality changes, systems fall behind, users adapt and spreadsheets are born. One root cause is that it's comparatively hard to add new data in a RDBMS - it usually requires a full development life cycle. On the other hand, spreadsheets are free, provide immediate benefits, allow users to define and refine requirements as they go, and can be abandoned at the drop of a hat (low risk). Quite the conundrum! From the article: > At present, Bakke’s tool enables query construction on an existing database, but it doesn’t enable the direct entry or modification of data. He expects to begin adding that functionality over the next six months Querying data is a problem with many good solutions nowadays, the bigger question is where and how to write that new data the users want to play with (other than in a spreadsheet).
  - EGreg 10y ago
    So why can't they use something like Irmin? https://github.com/mirage/irmin https://github.com/mirage/irmin
  - intrasight 10y ago
    >it's comparatively hard to add new data in a RDBMS I assume you meant to day "to add new schema". Adding new data is easy and in most enterprise environments is happening automatically every day. As far as "add new schema" goes, I'll claim that the database is just another software component, and if you pursuing an agile strategy, that you should be modifying that schema with every sprint. My client, for whom I do data warehouse consulting, updates the schema every two weeks. Because we are responsive to the business, they never revert to spreadsheet solutions. Of course they are free to create there own custom reports in spreadsheets - we'd never try to take that away. In fact we make it as easy as possible to run business curated queries and dump the data to Excel. Our newest feature for these power users is to allow them to give back to us the Excel reports they built and we will "push and deliver" - put the current data into the raw data sheets and deliver the report - usually to Sharepoint. Getting good business software solutions in place merely requires that the business hire some good developers. Why more don't do so remains a mystery. My best guess, like all things "corporate" is fear and laziness.
emmelaich 10y ago
or Query By Example (1975). But I'm sure much more. https://en.wikipedia.org/wiki/Query_by_Example https://en.wikipedia.org/wiki/Query_by_Example
GlennS 10y ago
If someone actually did reinvent Access without the concurrency data-loss problems (but still as a single file per database), they'd probably make a lot of money.

fspeech 10y ago

Very cool. The right side of the article provides links to a video presentation of the tool and a SIGMOD paper reporting on the details. It is specifically compared against Access in a study on usability. Link to the project page: http://people.csail.mit.edu/ebakke/sieuferd/ http://people.csail.mit.edu/ebakke/sieuferd/

alirobe 10y ago

So... Pivot Tables? Excel has done this for ages. Excel 2016 even has in-memory BI.

LoSboccacc 10y ago
Access had a database gui wizard since when? 97?

dhruvkar 10y ago

http://www.fieldbook.com http://www.fieldbook.com I haven't used this in production, but messed around with it for a while. Seems to do what this article is suggesting.

nickpsecurity 10y ago
Yeah, I reposted that one in a few places. Reason is getting people outta spreadsheets into better DB will need replacement to be just like spreadsheets. Anything too different in UI or workflow will get resistance and workarounds. Fieldbook was clever.

philprx 10y ago

Am i missing something but... ... Where is the tool ? Source code or it doesn't exist ;)

wx196 10y ago

"when you have something extremely industry-specific is, you have to hire a programmer who spends about a year of work to build a user interface for your particular domain" — with tools like Oracle APEX it can be done much faster, so I think nothing new.

jlg23 10y ago
As long as one needs a programmer the year is a good rough estimate, the hard part is not the building but the gathering of requirements. Take the programmer out of the equation and thus the communication overhead between programmer and domain specialist and you save a lot of development time.
- wx196 10y ago
  You still need someone to think out relations in your database and how to retain it faster with large data sets. If this work can be done by domain specialist only, then it can be done the same way using APEX, FileMaker etc. Without live example of the new technology it is difficult to say how it differs.
- sqldba 10y ago
  The thing is that having a tool like this does not make the requirement gathering go away. Dude I can build a house. I've only used a drill once in my life, I've never mixed concrete, or bricks, but I can. I will lay stuff on top of each other and use superglue and whatever it takes. But nobody will ever want to live in and it would not be safe. That's why you get a professional to do it. A user might somehow be able to magically skip gathering requirements and just dive gung ho into creating a structure for what they want. Great. And you end up with the legacy Excel monstrosity turds that we've been trying to eradicate from the Enterprise for the past two dozen years.

ww520 10y ago

Microsoft Access?

ComodoHacker 10y ago
Screenshots[1] look more like Power Pivot. 1. http://people.csail.mit.edu/ebakke/sieuferd/ http://people.csail.mit.edu/ebakke/sieuferd/

kfk 10y ago

OK... but where are decent user-friendly tools to manipulate, manage and share data and insights? Really, the problem is that to this day data in Corporations is a mess. The moment you step out of the standard warehouse systems you find yourself navigating a mess of spreadsheets, word docs, power points, all showing sometimes the same data in different formats. You duplicate data every time you send an Excel file by email... this data needs to be in sync and when not people get screamed at or fired. That means a lot of work goes into manually maintaining thousands of different files to make sure they tie to the overall picture. It's a mess. The bad part is that nobody is looking into this, everybody is focusing on analytics or reporting, which by now are the easy part. Then you have ancient tools like HFM or BPC that try to "enhance" spreadsheets and spreadsheets consolidations. They don't work. They are barely reliable and they cost a lot of money for no reason (they are simple SQL databases, they can't even be compared to much more complex software like salesforce or similar). Then Corporations are now being sold this all "big data" thing, which is old by now for tech audiences, but it's just new for most of the big companies nowadays. Unfortunately, while big data has a lot of potential, it further moves away investments on good old small data where probably there is much more ROI to grab simply because solving this problem is not that difficult and not that expensive if you want to actually fix it. Then now everybody wants to do Predictive, but Predictive won't beat saving hundreds of employees thousands of working hours by improving the efficiency of how we handle data and data insights. You can literally create a new workforce out of the many hours you would save with better efficiency in this area. Without even considering that on top of efficiency, you get much better, more fact based, decisions driven by the overall increase in transparency which you lose when insights are spread around thousands and thousands of files. There is some light at the end of the tunnel. Software solutions like Tableau go in the right direction, but they do not provide the much (more) needed tools to properly manipulate and manage data and, especially, consolidations and integration. The only way to get out of this mess is to control the data flow but especially to have 1 and only 1 data flow. That means once your insights are approved and locked, every other view should read this data, there should be no manual intervention any more. If the locked data is wrong, it will be wrong for everybody, which is a good thing compared to having multiple copies of the same and then people fighting on which one is the right one.

sqldba 10y ago

johnmaguire2013 10y ago

This sounds a lot like AirTable[1] to me. AirTable has quickly become my favorite Spreadsheets replacement because it has a nice REST API, ways of linking data from multiple tables, data integrity principles, and pretty solid mobile phone data entry, while still maintaining the simplicity of a spreadsheet. [1] https://airtable.com/ https://airtable.com/

krmmalik 10y ago
I do like Airtable, and i can see the comparison but as far as i understand you cant do the complex querying that the article is suggesting?

based2 10y ago

http://people.csail.mit.edu/ebakke/ http://people.csail.mit.edu/ebakke/

infinite8s 10y ago

I'm not sure how this is any easier than using Tableau, which can construct most of these types of business analytic queries fairly easily. More complicated queries that pull in data from multiple tables usually requires an understanding of the organization of data across these tables, but that is true of this tool as well (at least from the video on the author's website - http://people.csail.mit.edu/ebakke/sieuferd/index.html http://people.csail.mit.edu/ebakke/sieuferd/index.html)