5 ms·

The big data successor of the spreadsheet

61 points by misterdata 12y ago

PaulHoule 12y ago
I think the problem now is not a better excel but a better access; spreadsheets are pretty good at what they do but biz people use them when they should use a database because they find databases less intuitive.
- smacktoward 12y ago
  Yeah - my reaction to the idea of "spreadsheets for big data" is that it kind of misses the point of why spreadsheets are so popular, namely that most data is small data and a sheet of rows and columns is a super easy way to wrap your mind around a set of small data. So the big challenge is coming up with a way to work with small data that gives you the benefits a database brings over a spreadsheet (relations, querying, etc.) without losing that immediate "oh, I get this" feeling.
  - joshuahutt 12y ago
    Excel already has relations, albeit simple ones, as well as simplistic forms of querying, though filters and pivot tables. I would argue that the major benefits that a database offers over Excel is data integrity and consistency. I think the lack of that sort of constraint is a major aspect of what drives 'biz' people to spreadsheets.
  - hammerandtongs 12y ago
    The core problem of spreadsheets is that they should have been organized around tables with fixed data meanings from the beginning of the history of spreadsheets. Instead the sheet is both layout tool and data "meaning" in one glob of garbage. Apple tried to solve the issue with "Numbers" but ymmv on how well.
    - teamonkey 12y ago
      I disagree. The SUCCESS of spreadsheets is that they are general-purpose tools that deal with information laid out on a grid and they would be nowhere near as useful or powerful or indeed popular if it were otherwise (c.f. Numbers). "Data laid out on a grid" is a paradigm that applies to a wide range of problems and spreadsheets are pretty much the only tool around that can deal with that entire paradigm. The user isn't stupid. The user has a use case that isn't being catered to.
      - hammerandtongs 12y ago
        If users had a clean and straightforward way to define the permanent header row, and viewed it as just what you did for a spreadsheet, the whole industry would be better off. This is probably a glass half full/empty at the end of the day. Fact is spreadsheets don't scale because they are sloppy and they don't provide a clean coherent path to getting the user to scale themselves into a sql like solution. Oh well.
        teamonkey 12y ago
        But doing that would limit spreadsheets to a tiny subset of what they're used for in the real world. If you have a permanent header row then you limit yourself to one table of data per page, which you often don't want. Excel's tables are probably similar what you mean, but Microsoft rightly allow them as elements on each sheet rather than basing the sheet around one table.
        hammerandtongs 12y ago
        """Apple tried to solve the issue with "Numbers" but ymmv on how well.""" Your "tiny subset" is your confusion at not seeing numbers I suspect. Wikipedia actually does a decent job of trying to describe the philosophy.
        teamonkey 12y ago
        Numbers just doesn't do a good job of representing data grid information that isn't rows and columns of numerical/textual data. Its paradigm is that takes that subset of use cases and cleanly separates the presentation layer. Excel has Tables, which sort serve a similar purpose, but the sheet itself is a flexible grid with no pre-conceived purpose, which makes it much more flexible than Numbers. For example, it's relatively easy to knock up a simple Gantt chart by directly coloring cells, something that I've seen often. And you could argue that it's not the best tool for the job, and you'd be right, but it's easy and does the job well enough in many cases. And that's why it's popular and iWork isn't. Numbers concentrated on making charts and tables beautiful; Excel is a general-purpose programmable platform.
- spitfire 12y ago
  There was a really cool app on NeXT called Quantrix (Also Lotus improv had the same idea). It was a spreadsheet where rules and data were kept separate. The modern java version can access databases. A spreadsheet user could get up and running with it over the course of a week or two, then switch to a database behind the scene.
- jewel 12y ago
  I feel the same way. In the 90s you'd find small businesses (and some larger ones!) built entirely on a single Access database on a network share. At the current small business where I work, we use google sheets instead, and it leaves a lot to be desired. For the past few years an easy-to-use database interface for small businesses has been my primary idea of what I'd like to build if I founded a startup.
  - rpwilcox 12y ago
    QuickBase(.com) may be something close to what you want: while mostly aimed at > 10 people businesses. It's not great, but it does have a reasonable amount of power... especially for apps with say less than 10 tables. DabbleDB was a good one for individual users, until Twitter bought it and shut it down. Google says grovesite.com is a competitor... and I also think Dabble's founders were debating rewriting it. There's always the old FileMaker Pro, or databases of that age / time. (4D??)
    - jasoncrawford 12y ago
      @jewel, @kikidrew, @rpwilcox: We're building this: https://fieldbookapp.com https://fieldbookapp.com. We're still in private beta, but message me (jason@fieldbookapp.com) and I can get you an invite.
  - maskedunit 12y ago
    i still work at a business that is built around access and am desperately looking to move away into something html5 but as easy to use. it's insane that no one has done something that just replaces access on the web. you should definitely try to give it a go.
  - kiwidrew 12y ago
    Agree 100%. In the span of a decade or so, the goalposts for what a "small business app" have moved dramatically: multiuser by default, offline editing/sync, usable on mobile devices, modern-looking UI/UX... but the Access-like tools never made the leap. They're still stuck in the ancient past, and effectively unusable now. Why hasn't anyone been able to build a modern equivalent to Access yet? Google Sheets is laughably primitive when you try to use it as a multiuser database. It's impossible to validate input and preserve structure (giving a user edit permission, so they can insert new data, also allows them to remove cell validation and destroy the structure of the spreadsheet). Something as simple as copy-and-paste (Ctrl+C and Ctrl+V) will destroy all of your carefully placed conditional formatting and validation constraints. And sometimes a formula will just "break" for no reason at all, only to start working again a few days later... I'm honestly surprised that this is what we consider to be the start of the art. Where's my easy-to-use cloud database?
- Apofis 12y ago
  There's Libre Base.
- mamcx 12y ago
  Agree. I wok with FoxPro (like Acces, but better and powerfull) and I miss it. I enjoy python, but the DBase family is missing. I dream in build a moder relational language, but then I don't have the resources for doing it... However I still do some code/meditation about it each week...
err4nt 12y ago
Soulver is the best evolution of a calculator I've ever seen, think of it like a dynamic paper tape of calculations you can work with and alter and it keeps the results updated.
beamatronic 12y ago
The "pipeline of blocks" idea goes back a long way (Visual Smalltalk anyone?), I'm excited to see it applied in this way.
JBiserkov 12y ago
http://en.wikipedia.org/wiki/Powerpivot http://en.wikipedia.org/wiki/Powerpivot
- igrekel 12y ago
  I was about to post a comment on power pivot myself. We were building a lot of tools in R but eventually switched most of it to power pivot. My onl complaint would be that it isn't practical to refresh it with new data, it requires more manipulations than I would like.
hammerandtongs 12y ago
"""The interfaces require the end-user to know about the concept of a table, rows and columns, the concept of logical expressions, as well as aggregation, even though none of those are directly visible nor have a ‘real life’ analog.""" This sentence expresses the problem I see with designers. Instead of making that disappear into magic why not focus on actually purposely and clearly educating people about these CORE concepts? edit: Think of how you would do this if you were Khan academy...
- joe_the_user 12y ago
  Instead of making that disappear into magic why not focus on actually purposely and clearly educating people about these CORE concepts? Because "people are dumb"? Well, people aren't dumb-dumb but just as you aren't going to get everyone to code, you aren't going to get everyone to understand abstract logical relations (not generally, throughout society, though anything is possible in a single enterprise). Spreadsheet maintain their dominance in places that they logically don't belong because you will always have a person who got their position, in say a brokerage firm, through social intelligence, language intelligence or "sheer guts". And they will want to be able to manipulate data without training and spreadsheets let them. Heck, we can't replace the Excell interface with a better spreadsheet interface.
  - hammerandtongs 12y ago
    The goal isn't teaching everyone. It's to teach the very limited subset of business analysts that CARE about the answer to learn their tools well enough to ask the question properly. Generally I think we are seeing some of these things should probably be part of much earlier math education. Sets, predicate logic in 7-8th grade?
- ethanbond 12y ago
  The pragmatic reason is that it takes a lot of time. As a product designer at a company that actually tackles exactly this problem, we generally have approximately 0ms to stop a user's workflow in order to teach them a cool concept. Building intuition DURING their workflow is where things get really exciting for designers though. Edit: Should point out that I'm not quite sure if you're suggesting we teach this in school, or...? But I read this as a designer's job to teach this in-product.
sgt101 12y ago
Datameer - big data spreadsheets Scratch - best thought out block language
bigger_cheese 12y ago
At my workplace we use SAS Enterprise Guide for some of this stuff. I like it because it has a drag and drop GUI that non technical users can utilise to extract data (It executes SQL queries in background) and do simple operations (like aggregations, statistics, graphs, merging, filtering etc) At the same time it has a decent programming language underpinning it and pretty advance statistical capabilities. On the down side it still suffers from some of the problems the article references. Some non programmers, even quite technical people simple don't grasp concept of querying and joins. It is also horrendously easy for the auto generated SQL to be horribly inefficient the default appearance of the plots and other graphical output looks ugly etc, which leads to people doing things like extracting subsets of data from SAS and then pasting it into excel to manipulate it. Which is kind of self defeating.
tomlock 12y ago
I work in a big, dumb enterprise with too many spreadsheets and Access databases. What made me move away from Excel spreadsheets: 1) As in the infamous case, its too easy to miss extending a range when you add new data[1] 2) There's no good way to manage concurrent access 3) Doing aggregates of aggregates is really hard What made me then move away from Access databases: 1) Excel and Access attempt to guess the "type" of cell contents automatically, and this creates issues (Is 060E2 a product, or a scientific number? Of course, its 6000. Is 3/6/2015 a US date? Access will figure it out without you.) 2) There's still no good way to manage concurrent access 3) Storage limits still aren't that great if you want to deal with more than a few GB of data For me the ideal spreadsheet would solve all these problems :) I moved to Postgres, instead. [1]http://www.bloomberg.com/bw/articles/2013-04-18/economists-spreadsheet-error-upends-the-debt-debate http://www.bloomberg.com/bw/articles/2013-04-18/economists-s...

61 points by misterdata 12y ago

I think the problem now is not a better excel but a better access; spreadsheets are pretty good at what they do but biz people use them when they should use a database because they find databases less intuitive.

smacktoward 12y ago
Yeah - my reaction to the idea of "spreadsheets for big data" is that it kind of misses the point of why spreadsheets are so popular, namely that most data is small data and a sheet of rows and columns is a super easy way to wrap your mind around a set of small data. So the big challenge is coming up with a way to work with small data that gives you the benefits a database brings over a spreadsheet (relations, querying, etc.) without losing that immediate "oh, I get this" feeling.
- joshuahutt 12y ago
  Excel already has relations, albeit simple ones, as well as simplistic forms of querying, though filters and pivot tables. I would argue that the major benefits that a database offers over Excel is data integrity and consistency. I think the lack of that sort of constraint is a major aspect of what drives 'biz' people to spreadsheets.
- hammerandtongs 12y ago
  The core problem of spreadsheets is that they should have been organized around tables with fixed data meanings from the beginning of the history of spreadsheets. Instead the sheet is both layout tool and data "meaning" in one glob of garbage. Apple tried to solve the issue with "Numbers" but ymmv on how well.
  - teamonkey 12y ago
    I disagree. The SUCCESS of spreadsheets is that they are general-purpose tools that deal with information laid out on a grid and they would be nowhere near as useful or powerful or indeed popular if it were otherwise (c.f. Numbers). "Data laid out on a grid" is a paradigm that applies to a wide range of problems and spreadsheets are pretty much the only tool around that can deal with that entire paradigm. The user isn't stupid. The user has a use case that isn't being catered to.
    - hammerandtongs 12y ago
      If users had a clean and straightforward way to define the permanent header row, and viewed it as just what you did for a spreadsheet, the whole industry would be better off. This is probably a glass half full/empty at the end of the day. Fact is spreadsheets don't scale because they are sloppy and they don't provide a clean coherent path to getting the user to scale themselves into a sql like solution. Oh well.
      - teamonkey 12y ago
        But doing that would limit spreadsheets to a tiny subset of what they're used for in the real world. If you have a permanent header row then you limit yourself to one table of data per page, which you often don't want. Excel's tables are probably similar what you mean, but Microsoft rightly allow them as elements on each sheet rather than basing the sheet around one table.
        hammerandtongs 12y ago
        """Apple tried to solve the issue with "Numbers" but ymmv on how well.""" Your "tiny subset" is your confusion at not seeing numbers I suspect. Wikipedia actually does a decent job of trying to describe the philosophy.
        teamonkey 12y ago
        Numbers just doesn't do a good job of representing data grid information that isn't rows and columns of numerical/textual data. Its paradigm is that takes that subset of use cases and cleanly separates the presentation layer. Excel has Tables, which sort serve a similar purpose, but the sheet itself is a flexible grid with no pre-conceived purpose, which makes it much more flexible than Numbers. For example, it's relatively easy to knock up a simple Gantt chart by directly coloring cells, something that I've seen often. And you could argue that it's not the best tool for the job, and you'd be right, but it's easy and does the job well enough in many cases. And that's why it's popular and iWork isn't. Numbers concentrated on making charts and tables beautiful; Excel is a general-purpose programmable platform.
spitfire 12y ago
There was a really cool app on NeXT called Quantrix (Also Lotus improv had the same idea). It was a spreadsheet where rules and data were kept separate. The modern java version can access databases. A spreadsheet user could get up and running with it over the course of a week or two, then switch to a database behind the scene.
jewel 12y ago
I feel the same way. In the 90s you'd find small businesses (and some larger ones!) built entirely on a single Access database on a network share. At the current small business where I work, we use google sheets instead, and it leaves a lot to be desired. For the past few years an easy-to-use database interface for small businesses has been my primary idea of what I'd like to build if I founded a startup.
- rpwilcox 12y ago
  QuickBase(.com) may be something close to what you want: while mostly aimed at > 10 people businesses. It's not great, but it does have a reasonable amount of power... especially for apps with say less than 10 tables. DabbleDB was a good one for individual users, until Twitter bought it and shut it down. Google says grovesite.com is a competitor... and I also think Dabble's founders were debating rewriting it. There's always the old FileMaker Pro, or databases of that age / time. (4D??)
  - jasoncrawford 12y ago
    @jewel, @kikidrew, @rpwilcox: We're building this: https://fieldbookapp.com https://fieldbookapp.com. We're still in private beta, but message me (jason@fieldbookapp.com) and I can get you an invite.
- maskedunit 12y ago
  i still work at a business that is built around access and am desperately looking to move away into something html5 but as easy to use. it's insane that no one has done something that just replaces access on the web. you should definitely try to give it a go.
- kiwidrew 12y ago
  Agree 100%. In the span of a decade or so, the goalposts for what a "small business app" have moved dramatically: multiuser by default, offline editing/sync, usable on mobile devices, modern-looking UI/UX... but the Access-like tools never made the leap. They're still stuck in the ancient past, and effectively unusable now. Why hasn't anyone been able to build a modern equivalent to Access yet? Google Sheets is laughably primitive when you try to use it as a multiuser database. It's impossible to validate input and preserve structure (giving a user edit permission, so they can insert new data, also allows them to remove cell validation and destroy the structure of the spreadsheet). Something as simple as copy-and-paste (Ctrl+C and Ctrl+V) will destroy all of your carefully placed conditional formatting and validation constraints. And sometimes a formula will just "break" for no reason at all, only to start working again a few days later... I'm honestly surprised that this is what we consider to be the start of the art. Where's my easy-to-use cloud database?
Apofis 12y ago
There's Libre Base.
mamcx 12y ago
Agree. I wok with FoxPro (like Acces, but better and powerfull) and I miss it. I enjoy python, but the DBase family is missing. I dream in build a moder relational language, but then I don't have the resources for doing it... However I still do some code/meditation about it each week...

err4nt 12y ago

Soulver is the best evolution of a calculator I've ever seen, think of it like a dynamic paper tape of calculations you can work with and alter and it keeps the results updated.

beamatronic 12y ago

The "pipeline of blocks" idea goes back a long way (Visual Smalltalk anyone?), I'm excited to see it applied in this way.

JBiserkov 12y ago

http://en.wikipedia.org/wiki/Powerpivot http://en.wikipedia.org/wiki/Powerpivot

igrekel 12y ago
I was about to post a comment on power pivot myself. We were building a lot of tools in R but eventually switched most of it to power pivot. My onl complaint would be that it isn't practical to refresh it with new data, it requires more manipulations than I would like.

hammerandtongs 12y ago

"""The interfaces require the end-user to know about the concept of a table, rows and columns, the concept of logical expressions, as well as aggregation, even though none of those are directly visible nor have a ‘real life’ analog.""" This sentence expresses the problem I see with designers. Instead of making that disappear into magic why not focus on actually purposely and clearly educating people about these CORE concepts? edit: Think of how you would do this if you were Khan academy...

joe_the_user 12y ago
Instead of making that disappear into magic why not focus on actually purposely and clearly educating people about these CORE concepts? Because "people are dumb"? Well, people aren't dumb-dumb but just as you aren't going to get everyone to code, you aren't going to get everyone to understand abstract logical relations (not generally, throughout society, though anything is possible in a single enterprise). Spreadsheet maintain their dominance in places that they logically don't belong because you will always have a person who got their position, in say a brokerage firm, through social intelligence, language intelligence or "sheer guts". And they will want to be able to manipulate data without training and spreadsheets let them. Heck, we can't replace the Excell interface with a better spreadsheet interface.
- hammerandtongs 12y ago
  The goal isn't teaching everyone. It's to teach the very limited subset of business analysts that CARE about the answer to learn their tools well enough to ask the question properly. Generally I think we are seeing some of these things should probably be part of much earlier math education. Sets, predicate logic in 7-8th grade?
ethanbond 12y ago
The pragmatic reason is that it takes a lot of time. As a product designer at a company that actually tackles exactly this problem, we generally have approximately 0ms to stop a user's workflow in order to teach them a cool concept. Building intuition DURING their workflow is where things get really exciting for designers though. Edit: Should point out that I'm not quite sure if you're suggesting we teach this in school, or...? But I read this as a designer's job to teach this in-product.

sgt101 12y ago

Datameer - big data spreadsheets Scratch - best thought out block language

bigger_cheese 12y ago

At my workplace we use SAS Enterprise Guide for some of this stuff. I like it because it has a drag and drop GUI that non technical users can utilise to extract data (It executes SQL queries in background) and do simple operations (like aggregations, statistics, graphs, merging, filtering etc) At the same time it has a decent programming language underpinning it and pretty advance statistical capabilities. On the down side it still suffers from some of the problems the article references. Some non programmers, even quite technical people simple don't grasp concept of querying and joins. It is also horrendously easy for the auto generated SQL to be horribly inefficient the default appearance of the plots and other graphical output looks ugly etc, which leads to people doing things like extracting subsets of data from SAS and then pasting it into excel to manipulate it. Which is kind of self defeating.

tomlock 12y ago

I work in a big, dumb enterprise with too many spreadsheets and Access databases. What made me move away from Excel spreadsheets: 1) As in the infamous case, its too easy to miss extending a range when you add new data[1] 2) There's no good way to manage concurrent access 3) Doing aggregates of aggregates is really hard What made me then move away from Access databases: 1) Excel and Access attempt to guess the "type" of cell contents automatically, and this creates issues (Is 060E2 a product, or a scientific number? Of course, its 6000. Is 3/6/2015 a US date? Access will figure it out without you.) 2) There's still no good way to manage concurrent access 3) Storage limits still aren't that great if you want to deal with more than a few GB of data For me the ideal spreadsheet would solve all these problems :) I moved to Postgres, instead. [1]http://www.bloomberg.com/bw/articles/2013-04-18/economists-spreadsheet-error-upends-the-debt-debate http://www.bloomberg.com/bw/articles/2013-04-18/economists-s...