15 ms·

Spreadsheets Are Hot–and Cranking Out Complex Code

167 points by LukeEF 4y ago

datalopers 4y ago
Excel users tend to do a far better job of data storage, analysis, and forecasting than most of overpaid SaaS-reliant "modern data team" (Data Engineers, Data Analysts, Analytics Engineers, Data Scientists).
- bliteben 4y ago
  It definitely is a scary business proposition for them to improve Excel and awaken the 800lb gorilla. Microsoft is certainly capable of improving Excel in ways that might even improve the productivity of the entire world. Seems like some of these startups are likely just giving training to future Microsoft employees.
  - pete_nic 4y ago
    Agree with this. When battling Microsoft product can matter less than things like contracts and licensing. I have to believe that businesses are spending so much on Azure that they get Office for free (as an analogy - at a previous employer GSuite was nearly free because our AdSense bill was so high). Businesses are unlikely to pay extra for a few more features when they already have Excel and have no plans to move off Azure.
  - OzCrimson 4y ago
    Excel is constantly being improved. Are you aware of Power Query? And how about the 14 news functions that were released to Insiders a few weeks ago? Excel Online is far behind the desktop version of Excel but it's making fast progress.
- croes 4y ago
  I doubt that. Data hoarding, yes. Creating unsupportable spreadsheet solutions, yes.
- b-luu 4y ago
  Meh, depends on the data teams (and their chosen suite of tools)... Or on the Excel users Both sides can either hack together horrible work-arounds (a matter of "when you have a hammer, everything looks like a nail...") as well as brilliantly thought through solutions. Each tool should be used for it's best use cases, but not bent into what it wasn't designed for! IMHO spreadsheets excel at intuitively manipulating the data ON the data itself. While "modern data" tools (especially dbt) try to convert date teams to use developer best practices... At the expense of less intuitive/direct manipulation of the data. That being said, I think there are also things we could explore in that space: how to make the modern data stack more intuitive?
  - datalopers 4y ago
    > how to make the modern data stack more intuitive? I'd start with getting people to learn relational data modeling and SQL [1][2] at a deep level. Stop reaching for python and pandas/spark for every basic data manipulation task or query. Stop adding in layers of Airflow/Dagster/Prefect when a simple cron would work. Stop adding in Kubernetes/GKE/Fargate to manage the aforementioned. Stop moving data between systems constantly (meltano, airbyte, fivetran) when you already have it in a perfectly good place. Stop with the toxic positivity that's completely overflowing the modern-data world and all these bullshit VC-infused startups who are convinced they need every single element and more. 99% of business needs can be satisfied by a single Postgres/Mysql installation and a halfway-competent person armed with SQL and an understanding of normalization. Reach for Excel when you need to do more "hands-on" analysis, business modeling, charting, basic forecasting, and presentation for non-technical users. [1] https://mode.com/sql-tutorial/ https://mode.com/sql-tutorial/ [2] https://use-the-index-luke.com/ https://use-the-index-luke.com/
    - b-luu 4y ago
      Good points, but a bit extremist. I definitely agree with the over-hype making simple mundane tasks way harder than they should be. So yeah, do NOT over-engineer!! But, on the other hand, doing everything with a single Postgres and spreadsheets seems to go with the hammer-to-nail adage. And all too often, you end up with unmaintainable duck-taped hack-arounds... Which is clearly NOT better (nor necessarily worst) than the over-engineered solution. In some cases (maybe not 1% but clearly not the majority either), it does make sense to look at other tools that might be available. That being said, there are waaaay too many options to filter through, because of that darn hype bubble.
wildrhythms 4y ago
Every day I think about how Microsoft Access allowed otherwise not-so-tech-savvy users to, with just a little training and practice, build a complete relational database for their entire business, supported by a relatively sane GUI and a way to build forms and reports with very little (if any) 'code'. I have no idea what small companies are using nowadays, but I think there has to be some untapped middle ground somewhere between Microsoft Access and a dumb spreadsheet.
- mathgladiator 4y ago
  I'm structuring my UI for Adama (https://www.adama-platform.com/ https://www.adama-platform.com/) around the baby of Access and Excel, and the engine is available in early access now. I love Excel and Access, and I plan a future pivot into small business space.
- mackrevinack 4y ago
  sounds like you are describing grist https://www.getgrist.com https://www.getgrist.com
- buescher 4y ago
  Microsoft Access is, of course, still around. It would be great if there were a solid way to develop a database, forms, and reports in Access and then deploy it to the web. If you've ever had to work with someone else's Access database, it is unusual to see a reasonably normalized relational database. Most people are much more comfortable with the single flat file of Sharepoint lists.
  - maxerickson 4y ago
    For stuff at the scale of a small Access database, does the normalization (or lack of) end up being a big deal? I get that there are real advantages, I'm just wondering out loud whether they are universally applicable/important to small scale databases.
    - bliteben 4y ago
grumblepeet 4y ago
I’ve been doing this for customers for years now using Microsoft Lists (sometimes I use Excel) as the back end and PowerApps as the front end gui/data entry. Now the language from PowerApps (which is based on Excel) has been open sourced and Microsoft want to make it available in other apps too. That has questionable value but my solutions continue to be popular because the people I make them or can easily edit and make simple changes. Some of the solutions we make have been widely adopted too, with hundreds and in some cases thousands of users.
narush 4y ago
Warning: I'm a founder working in the spreadsheet space, so take the rest of my this comment with a (large) grain of salt. I've written before [1] (HN and elsewhere) about how I think spreadsheets are the most popular programming paradigm ever, we just don't talk about it much. As this article mentions, there are many ways we can push this forward. I personally think the most powerful low-code spreadsheet tools we can build are those that allow spreadsheet users to easily transition to full programming languages, if they want to. So rather than locking users into limited and proprietary product number #115 (some of them are mentioned in this article), IMO it's better if users can transition to a full programming language (like Python) very naturally. Som I've spent the past 2 years building Mito [2]. Mito is a spreadsheet extension to your JupyterLab environment. You can display any Pandas dataframe as a spreadsheet, and edit it in a very similar way to Excel. For each edit you make, it generates the corresponding Python code below for those edits. Practically, you can think about Mito as recording a macro, but instead of generating scummy-crummy VBA code, it generates Python. We're open core [3]. Feedback greatly appreciated! [1] https://naterush.io/blog/Spreadsheets-are-the-Ultimate-Progr https://naterush.io/blog/Spreadsheets-are-the-Ultimate-Progr... [2] https://trymito.io https://trymito.io [3] https://github.com/mito-ds/monorepo https://github.com/mito-ds/monorepo
- segmondy 4y ago
  Just curious, why as a Linux user shouldn't I use Libreoffice calc with python? Why bring in something so proprietary?
  - narush 4y ago
    Mito isn’t just a spreadsheet that works with Python, but a spreadsheet that allows you to generate Python code when you edit! If your just looking to work with spreadsheets with Python, I’d also reccomend checking out XLWings - I haven’t used it myself but some of our users do and love it!
  - kzrdude 4y ago
    Seemlessly integrating spreadsheets into jupyter sounds like the holy grail to me. I haven't tried it yet, though. I think their users are people like me - who work a lot in Jupyter and swear by it - in a Python data analysis/visualisation environment. To bring in the best of spreadsheets into that could be magical. To just work in libreoffice calc or Excel would be a nonstarter, it just doesn't match all the other python tools in the workflow.
wintermutestwin 4y ago
>Suddenly, the field has begun to bloom. A small cluster of startups have in the past year released spreadsheet products All of which are subscription based SAAS. Maybe I am showing my VisiCalc oldster roots, but I want my document related tools to be pay once / run locally. I long for a computing world where the free tier is run-in-a-browser-on-someone-else's-computer and the paid tier is run local and native.
- nojito 4y ago
  The best startup ideas are creating one part of the O365 system (usually Excel) and monetizing it separately.
  - wintermutestwin 4y ago
    Ironically, O365 is $7/month with max rows = 1m+ and spreadsheet.com is $9month for max rows = 10k.
- ceilingcorner 4y ago
  Sadly it feels like single purchase software is on its way out. All the momentum is toward SAAS.
  - wintermutestwin 4y ago
    As a greybeard, I have seen plenty of old ways become new again. It's hard to predict when the next shift will be or where it will go, but it will shift.
- satyrnein 4y ago
  Subscription based SaaS doesn't bother me in and of itself. Cloud based supporting multiple collaborators and multiple devices, tech support often built in, no managing upgrades or patches, etc. The problem is portability and lock-in. Which itself might be due to lack of standardization.
  - 4y ago
bdcravens 4y ago
We are a small B2B, and during the pandemic our primary source of income disappeared. Our president built out a new service, one that is VERY analytics heavy. He built it in weeks using Excel, rather than the months it would have taken using "proper development". It's a beautiful monstrosity, and eventually needs to be ported to our codebase, but it saved the company.
peterkelly 4y ago
http://www.paulgraham.com/submarine.html http://www.paulgraham.com/submarine.html
motohagiography 4y ago
I've learned products designed to replace spreadsheets have a huge hurdle because the people who use spreadsheets treat operating the sheet as their job. Replacing them removes their autonomy and control over an information process, and subsumes the value they bring to their employers - so they will resist products that threaten that. Excel is a complete management subculture. The other advice I give is if you are generating analytics, have a PowerBI connector of some kind because the people who make decisions (managers, etc) make them based on PowerBI, and not from an interface their staff is a peer at using, and likely has control over. In enterprise, they want data in metrics their staff can't see, hence a separate tool. Spreadsheets will always be with us I think. The opportunity may be in creating one that is has sufficient work-alike features with legacy ones, with new power features (python, etc) where there is a connector between the high power open development environment, and the familiar Excel ones managers use. Key thing being not asking managers or sr. employees to change.
- nerdponx 4y ago
  It's not about autonomy in an abstract sense. It's the fact that regular people can customize, hack, modify, automate, and otherwise program the spreadsheet. Data analysts and managers are more than trained monkeys; they actually need to do those kinds of things (at least sometimes) in order to do their jobs. I agree that the ideal world is one where you can connect the spreadsheets to other data sources, so you get the best of both.
  - lupire 4y ago
    ODBC is decades old
  - syshum 4y ago
    This is a double edge sword, because often time there is no one validing the results of the spread sheets... "The number looks ok" is not a good validation, and there has been some very public data errors as a result of bad spreadsheets. I often wondered if in the average business is even 10% of the spreadsheets where actually audited what would happen.... I suspect the results would be rather shocking
pete_nic 4y ago
Excel, and spreadsheets, need a re-branding campaign. They are cloud-based, low-code, ELT tools; these terms are typically associated with buzzy enterprise tech companies, not technology developed in the 1980’s.
throwaway787544 4y ago
I just learned how to use spreadsheets recently, and I love them. As a programmer, spreadsheets are exactly the kind of tech I want to use every day. User-friendly point and click, IDE features, complex functions mapping data across rows/columns/tables. All I want now is to hook it up to Postgres, and then maybe a new app to automatically design simple web pages that pull out and display data with the same functions. This would save me months of time per year versus manually coding the same things.
- kgodey 4y ago
  We're building an (open source, self hosted) tool that does the "hook it up to Postgres and pull out and display data with the same functions" portions of what'd you like. https://github.com/centerofci/mathesar https://github.com/centerofci/mathesar
jhokanson 4y ago
> Spreadsheets.com, for example, lets users dump almost anything into a cell. Drop a photo or a PDF into a cell and the product will immediately create a thumbnail, which you can then expand, as if the spreadsheet were some sort of blog content-management system. Yes please! Can I get this for Google Sheets? :/
nunb 4y ago
All this and not one mention of DabbleDB !! https://www.youtube.com/watch?v=MCVj5RZOqwY https://www.youtube.com/watch?v=MCVj5RZOqwY https://news.ycombinator.com/item?id=102 https://news.ycombinator.com/item?id=102
- recuter 4y ago
  Well it disappeared over a decade ago. Very interesting demo. Time is a flat circle I guess. Nothing actually comes to mind from the present that is quite like this, hmm. Hmm.
sys_64738 4y ago
People always want to reinvent the spreadsheet with their latest whizzbang solution without a problem. But Excel solves 80% of the common problems so nobody really cares.
- OzCrimson 4y ago
  YUP! And too often a company claiming to have something better than Excel isn't a direct equivalent. It's a way to market their whizzbang solution. Domo was claiming it could get companies off of spreadsheets. But it turns out that they're an enterprise level reporting system that's a minimum of $30,000. That's not a viable option for a 5-person non-profit, but they'll get swept up in the conversation only to find out later that no, Domo and Excel are not equivalents.
deleted 4y ago
[deleted]
leowoo91 4y ago
Of course you can do many stuff, except access control and server side validation. Spreadsheets rather shine with quick planning / analytics.
MetaMonk 4y ago
This is a ridiculous piece of native advertising.
joejohnston 4y ago
Check out Budibase. It's the leading open source low code platform and perfect for building UIs on top of spreadsheets https://github.com/Budibase/budibase https://github.com/Budibase/budibase
- cutler 4y ago
  Budibase looked perfect until I learned there is no option for hosting locally. Doesn't make sense.
  - techplex 4y ago
    Looks like they support self hosting, no? https://docs.budibase.com/docs/hosting-methods https://docs.budibase.com/docs/hosting-methods
newbie2020 4y ago
Been thinking about this problem for a long time. I use python/jupyterhub notebooks for our data analysis flows at work. I’ve become an expert at it and am a go-to person in our org… but even still when I have to try something new on “small” amounts of data, I _still_ go to excel first. It’s barrier to get the results you want os extremely low - thus-far unparalleled by any other software I’ve used in the last 20 years… and I have a hard time seeing anything that would replace that (though maybe I am just set in my ways…)
phkx 4y ago
What do are you using spreadsheets for privately? I never dig deep into their capabilities, so I mostly use them to track expenses within some particular scope, e.g. healthcare. When I recently wanted to compare several loans and estimate our financial situation several years in the future, I wrote some Python code and used a Jupyter notebook to enter parameters and make plots. Has any of you done something similar using spreadsheets? On a side note: I didn’t find a Python library for time series generation (not analysis). Something where you can build some models (e.g. loan, income, expenses) which depend on a common parameter (time) and then evaluate all your models for different values of the common parameter. Right now, I generate pandas series/dataframes and combine them afterwards, which also took some massaging of pandas (which I also usually don‘t use a lot).
- vinceguidry 4y ago
  I use Google Sheets for personal finance with TillerHQ to automate downloading bank data. It is, as far as I know, the only workable solution in this space. http://www.tillerhq.com http://www.tillerhq.com
LukeEF 4y ago
Founder of excel collaboration and versioning startup [1] here so I am a believer. There are only about 30 million programmers. There are over 1 billion Excel users. Excel is Turing complete. Excel is by far the most used programming language on the planet. It is easily 20 times more popular than the next contender. The value of Excel is that it is presenting the data, with a set of formulae that let you keep derived data up-to-date. This inferred data provides sums and computations, sometimes simple, but sometimes exquisitely complex. And through this whole range of complexity, with a billion users, virtually nobody treats Excel seriously like a programming language. We have a programming language which is essentially acting as a declarative database, and yet we don't do unit tests, we don't keep track of changes, we collaborate with Excel by sending it to our colleagues in the mail and god-forbid we should doing any serious linting of what is in the thing. Anyone who has used Excel in anger realizes why it is so brilliant. Show me another declarative constraint based, data driven inference language that I can teach to my grandmother. The problem isn't Excel. The problem is that we are treating Excel like its a word processor, and not what it is: a programming language. [1] https://versionxl.com/ https://versionxl.com/
janci 4y ago
Excel is great and versatile, yet lacks some critical features. - It's easy to load data from database or API without VBA, yet impossible to write updates back without VBA. With VBA it's still messy string concatenation of SQL queries. - VBA is security nightmare - Version control is bad. Distributing spreadsheets in emails and collecting changes back is nightmare. Sharing a file on network drive lacks fine-grained permission control. - It's hard to maintain proper normalized relational data model. It's impossible to abstract the normalized model from the user (i.e. show labels in selectboxes but store ID's) - It's locale dependent. Date formatting, column separator (comma/semicolon separated), even function names. Unusable in international data exchange. - It mixes formatting, logic and data. Impossible to make reusable blocks. New lambda functions help somewhat. And so much more.
- dfox 4y ago
  Excel is locale dependent in a weird way. The internal model (which includes the function names) is completely locale independent and only the UI is internationalized (according to OS locale) and localized (according to Excel language version). This is not a bad construction, but breaks down spectacularly when you start to use Excel as any kind of real programming or data analysis environment, because then you get to the cracks in the user-visible “API surface” where the internal/external format distinction is somewhat smeared.
- ogogmad 4y ago
  > Impossible to make reusable blocks. The click and drag down feature results in a kind-of code reuse. It bypasses the need to consciously name things. Is this generalisable? For small scale code, these concerns may be overkill.
  - somat 4y ago
    Click and drag down is the opposite of code reuse, it is effectively a one time code generator. In fact it is one of the main reason I have largely abandoned spreadsheats, the behaviour I actually want is found in sql views. Calculate all items in column with one expression vs calculate items in column with unique expressions that were copy, pasted then transformed for each and every row. The other main reason I abandoned spreadsheets is row level integrity. too many times data that goes together in a row has drifted apart(sort on column subset is main culprit), another inherent problem solved by using a relational database. the solution to having excel at every desk is to have postgres at every desk. the code will be just as bad but the data integrity will be better.
beefield 4y ago
As much as I hate to admit it, it is going to be a long time before we get rid of spreadsheets in de facto production use. And for that, I really would like to see one thing: I'd like to have a separate worksheet type for "datasheets". Looks and behaves just like an ordinary worksheet, but: - plugging in a formula applies the formula on every cell of the column. No exceptions. - you can not have different data types in cells within one column. That is, if you have dates in a column, you can't have a string in one cell of thecolumn etc. Yes, I know about powerthings in excel. No, I do not want to mess with those. Just normal spreadsheet formulas and sheets. (Okay, a couple of other things: Get rid of vba and bring in the full python ecosystem instead. And if not yet available, version control. It is luckily a while I have worked with excel, so these may be outdated comments)
- mikewarot 4y ago
  >I'd like to have a separate worksheet type for "datasheets". Looks and behaves just like an ordinary worksheet I think you're on to a good idea, but it has to be a table in the middle of a spreadsheet if you want to get acceptance. What would probably cinch it for you is if you could embed an entire table in a bounded box, with scrolling. It would break the row/column addressing for the contents, but that's always clumsy with tables in sheets anyway. Example: a data table lives in C10-M10 / C20/M20 Down in C21-M21 you could have sums, counts, etc. that work across all the data. With this arraignment, you could then sum columns, count, etc. It would be really slick if you snuck in SQL. The tradeoff in immediate addressing of cells within a table could be acceptable. A while ago, I saw something that allowed spreadsheets in spreadsheet cells, it was mostly intended for interactive use, and dispensed with cell addresses completely in a manner that seemed reasonable at the time.
KarlKemp 4y ago
Excel is by far the most successful programming language and IDE. People love to hate it (and the people using it), which is somewhat misguided: there is simply no way to change people (and they keep making new ones), so telling them that what works for them is somehow wrong is both wrong and doesn't work. Instead, the spreadsheet paradigm has the promise of being far more powerful. Jupyter notebooks are one example of adapting it to a different realm, and it also ended up being used everywhere and looked down upon by the snobs.
huijzer 4y ago
At uni, one professor used to say: "spreadsheets have all of the problems of software engineering, but none of the solutions"
foxbee 4y ago
I think 'products replacing spreadsheets' is not the right angle. Spreadsheets are great but not for everything. I think there are a lot of mis-uses of spreadsheets. When users record list type data, long-form content, relational data, etc within spreadsheets, they are probably better off using an app - they, or their IT team can build. I once seen an enterprise org process where a health and safety check was completed on a clipboard. That user would go back to their desk, push that data into excel, another user would write a script to push that excel data into MySQL. There are many use cases where spreadsheets are perfect - building financial models. For transparency - I am the cofounder of a no/low-code tool called Budibase. I am also a happy spreadsheet user. https://github.com/Budibase/budibase https://github.com/Budibase/budibase

167 points by LukeEF 4y ago

Excel users tend to do a far better job of data storage, analysis, and forecasting than most of overpaid SaaS-reliant "modern data team" (Data Engineers, Data Analysts, Analytics Engineers, Data Scientists).

bliteben 4y ago
It definitely is a scary business proposition for them to improve Excel and awaken the 800lb gorilla. Microsoft is certainly capable of improving Excel in ways that might even improve the productivity of the entire world. Seems like some of these startups are likely just giving training to future Microsoft employees.
- pete_nic 4y ago
  Agree with this. When battling Microsoft product can matter less than things like contracts and licensing. I have to believe that businesses are spending so much on Azure that they get Office for free (as an analogy - at a previous employer GSuite was nearly free because our AdSense bill was so high). Businesses are unlikely to pay extra for a few more features when they already have Excel and have no plans to move off Azure.
- OzCrimson 4y ago
  Excel is constantly being improved. Are you aware of Power Query? And how about the 14 news functions that were released to Insiders a few weeks ago? Excel Online is far behind the desktop version of Excel but it's making fast progress.
croes 4y ago
I doubt that. Data hoarding, yes. Creating unsupportable spreadsheet solutions, yes.
b-luu 4y ago
Meh, depends on the data teams (and their chosen suite of tools)... Or on the Excel users Both sides can either hack together horrible work-arounds (a matter of "when you have a hammer, everything looks like a nail...") as well as brilliantly thought through solutions. Each tool should be used for it's best use cases, but not bent into what it wasn't designed for! IMHO spreadsheets excel at intuitively manipulating the data ON the data itself. While "modern data" tools (especially dbt) try to convert date teams to use developer best practices... At the expense of less intuitive/direct manipulation of the data. That being said, I think there are also things we could explore in that space: how to make the modern data stack more intuitive?
- datalopers 4y ago
  > how to make the modern data stack more intuitive? I'd start with getting people to learn relational data modeling and SQL [1][2] at a deep level. Stop reaching for python and pandas/spark for every basic data manipulation task or query. Stop adding in layers of Airflow/Dagster/Prefect when a simple cron would work. Stop adding in Kubernetes/GKE/Fargate to manage the aforementioned. Stop moving data between systems constantly (meltano, airbyte, fivetran) when you already have it in a perfectly good place. Stop with the toxic positivity that's completely overflowing the modern-data world and all these bullshit VC-infused startups who are convinced they need every single element and more. 99% of business needs can be satisfied by a single Postgres/Mysql installation and a halfway-competent person armed with SQL and an understanding of normalization. Reach for Excel when you need to do more "hands-on" analysis, business modeling, charting, basic forecasting, and presentation for non-technical users. [1] https://mode.com/sql-tutorial/ https://mode.com/sql-tutorial/ [2] https://use-the-index-luke.com/ https://use-the-index-luke.com/
  - b-luu 4y ago
    Good points, but a bit extremist. I definitely agree with the over-hype making simple mundane tasks way harder than they should be. So yeah, do NOT over-engineer!! But, on the other hand, doing everything with a single Postgres and spreadsheets seems to go with the hammer-to-nail adage. And all too often, you end up with unmaintainable duck-taped hack-arounds... Which is clearly NOT better (nor necessarily worst) than the over-engineered solution. In some cases (maybe not 1% but clearly not the majority either), it does make sense to look at other tools that might be available. That being said, there are waaaay too many options to filter through, because of that darn hype bubble.

wildrhythms 4y ago

Every day I think about how Microsoft Access allowed otherwise not-so-tech-savvy users to, with just a little training and practice, build a complete relational database for their entire business, supported by a relatively sane GUI and a way to build forms and reports with very little (if any) 'code'. I have no idea what small companies are using nowadays, but I think there has to be some untapped middle ground somewhere between Microsoft Access and a dumb spreadsheet.

mathgladiator 4y ago
I'm structuring my UI for Adama (https://www.adama-platform.com/ https://www.adama-platform.com/) around the baby of Access and Excel, and the engine is available in early access now. I love Excel and Access, and I plan a future pivot into small business space.
mackrevinack 4y ago
sounds like you are describing grist https://www.getgrist.com https://www.getgrist.com
buescher 4y ago
Microsoft Access is, of course, still around. It would be great if there were a solid way to develop a database, forms, and reports in Access and then deploy it to the web. If you've ever had to work with someone else's Access database, it is unusual to see a reasonably normalized relational database. Most people are much more comfortable with the single flat file of Sharepoint lists.
- maxerickson 4y ago
  For stuff at the scale of a small Access database, does the normalization (or lack of) end up being a big deal? I get that there are real advantages, I'm just wondering out loud whether they are universally applicable/important to small scale databases.
  - bliteben 4y ago

grumblepeet 4y ago

I’ve been doing this for customers for years now using Microsoft Lists (sometimes I use Excel) as the back end and PowerApps as the front end gui/data entry. Now the language from PowerApps (which is based on Excel) has been open sourced and Microsoft want to make it available in other apps too. That has questionable value but my solutions continue to be popular because the people I make them or can easily edit and make simple changes. Some of the solutions we make have been widely adopted too, with hundreds and in some cases thousands of users.

narush 4y ago

Warning: I'm a founder working in the spreadsheet space, so take the rest of my this comment with a (large) grain of salt. I've written before [1] (HN and elsewhere) about how I think spreadsheets are the most popular programming paradigm ever, we just don't talk about it much. As this article mentions, there are many ways we can push this forward. I personally think the most powerful low-code spreadsheet tools we can build are those that allow spreadsheet users to easily transition to full programming languages, if they want to. So rather than locking users into limited and proprietary product number #115 (some of them are mentioned in this article), IMO it's better if users can transition to a full programming language (like Python) very naturally. Som I've spent the past 2 years building Mito [2]. Mito is a spreadsheet extension to your JupyterLab environment. You can display any Pandas dataframe as a spreadsheet, and edit it in a very similar way to Excel. For each edit you make, it generates the corresponding Python code below for those edits. Practically, you can think about Mito as recording a macro, but instead of generating scummy-crummy VBA code, it generates Python. We're open core [3]. Feedback greatly appreciated! [1] https://naterush.io/blog/Spreadsheets-are-the-Ultimate-Progr https://naterush.io/blog/Spreadsheets-are-the-Ultimate-Progr... [2] https://trymito.io https://trymito.io [3] https://github.com/mito-ds/monorepo https://github.com/mito-ds/monorepo

segmondy 4y ago
Just curious, why as a Linux user shouldn't I use Libreoffice calc with python? Why bring in something so proprietary?
- narush 4y ago
  Mito isn’t just a spreadsheet that works with Python, but a spreadsheet that allows you to generate Python code when you edit! If your just looking to work with spreadsheets with Python, I’d also reccomend checking out XLWings - I haven’t used it myself but some of our users do and love it!
- kzrdude 4y ago
  Seemlessly integrating spreadsheets into jupyter sounds like the holy grail to me. I haven't tried it yet, though. I think their users are people like me - who work a lot in Jupyter and swear by it - in a Python data analysis/visualisation environment. To bring in the best of spreadsheets into that could be magical. To just work in libreoffice calc or Excel would be a nonstarter, it just doesn't match all the other python tools in the workflow.

wintermutestwin 4y ago

>Suddenly, the field has begun to bloom. A small cluster of startups have in the past year released spreadsheet products All of which are subscription based SAAS. Maybe I am showing my VisiCalc oldster roots, but I want my document related tools to be pay once / run locally. I long for a computing world where the free tier is run-in-a-browser-on-someone-else's-computer and the paid tier is run local and native.

nojito 4y ago
The best startup ideas are creating one part of the O365 system (usually Excel) and monetizing it separately.
- wintermutestwin 4y ago
  Ironically, O365 is $7/month with max rows = 1m+ and spreadsheet.com is $9month for max rows = 10k.
ceilingcorner 4y ago
Sadly it feels like single purchase software is on its way out. All the momentum is toward SAAS.
- wintermutestwin 4y ago
  As a greybeard, I have seen plenty of old ways become new again. It's hard to predict when the next shift will be or where it will go, but it will shift.
satyrnein 4y ago
Subscription based SaaS doesn't bother me in and of itself. Cloud based supporting multiple collaborators and multiple devices, tech support often built in, no managing upgrades or patches, etc. The problem is portability and lock-in. Which itself might be due to lack of standardization.
- 4y ago

bdcravens 4y ago

We are a small B2B, and during the pandemic our primary source of income disappeared. Our president built out a new service, one that is VERY analytics heavy. He built it in weeks using Excel, rather than the months it would have taken using "proper development". It's a beautiful monstrosity, and eventually needs to be ported to our codebase, but it saved the company.

peterkelly 4y ago

http://www.paulgraham.com/submarine.html http://www.paulgraham.com/submarine.html

motohagiography 4y ago

I've learned products designed to replace spreadsheets have a huge hurdle because the people who use spreadsheets treat operating the sheet as their job. Replacing them removes their autonomy and control over an information process, and subsumes the value they bring to their employers - so they will resist products that threaten that. Excel is a complete management subculture. The other advice I give is if you are generating analytics, have a PowerBI connector of some kind because the people who make decisions (managers, etc) make them based on PowerBI, and not from an interface their staff is a peer at using, and likely has control over. In enterprise, they want data in metrics their staff can't see, hence a separate tool. Spreadsheets will always be with us I think. The opportunity may be in creating one that is has sufficient work-alike features with legacy ones, with new power features (python, etc) where there is a connector between the high power open development environment, and the familiar Excel ones managers use. Key thing being not asking managers or sr. employees to change.

nerdponx 4y ago
It's not about autonomy in an abstract sense. It's the fact that regular people can customize, hack, modify, automate, and otherwise program the spreadsheet. Data analysts and managers are more than trained monkeys; they actually need to do those kinds of things (at least sometimes) in order to do their jobs. I agree that the ideal world is one where you can connect the spreadsheets to other data sources, so you get the best of both.
- lupire 4y ago
  ODBC is decades old
- syshum 4y ago
  This is a double edge sword, because often time there is no one validing the results of the spread sheets... "The number looks ok" is not a good validation, and there has been some very public data errors as a result of bad spreadsheets. I often wondered if in the average business is even 10% of the spreadsheets where actually audited what would happen.... I suspect the results would be rather shocking

pete_nic 4y ago

Excel, and spreadsheets, need a re-branding campaign. They are cloud-based, low-code, ELT tools; these terms are typically associated with buzzy enterprise tech companies, not technology developed in the 1980’s.

throwaway787544 4y ago

I just learned how to use spreadsheets recently, and I love them. As a programmer, spreadsheets are exactly the kind of tech I want to use every day. User-friendly point and click, IDE features, complex functions mapping data across rows/columns/tables. All I want now is to hook it up to Postgres, and then maybe a new app to automatically design simple web pages that pull out and display data with the same functions. This would save me months of time per year versus manually coding the same things.

kgodey 4y ago
We're building an (open source, self hosted) tool that does the "hook it up to Postgres and pull out and display data with the same functions" portions of what'd you like. https://github.com/centerofci/mathesar https://github.com/centerofci/mathesar

jhokanson 4y ago

> Spreadsheets.com, for example, lets users dump almost anything into a cell. Drop a photo or a PDF into a cell and the product will immediately create a thumbnail, which you can then expand, as if the spreadsheet were some sort of blog content-management system. Yes please! Can I get this for Google Sheets? :/

nunb 4y ago

All this and not one mention of DabbleDB !! https://www.youtube.com/watch?v=MCVj5RZOqwY https://www.youtube.com/watch?v=MCVj5RZOqwY https://news.ycombinator.com/item?id=102 https://news.ycombinator.com/item?id=102

recuter 4y ago
Well it disappeared over a decade ago. Very interesting demo. Time is a flat circle I guess. Nothing actually comes to mind from the present that is quite like this, hmm. Hmm.

sys_64738 4y ago

People always want to reinvent the spreadsheet with their latest whizzbang solution without a problem. But Excel solves 80% of the common problems so nobody really cares.

OzCrimson 4y ago
YUP! And too often a company claiming to have something better than Excel isn't a direct equivalent. It's a way to market their whizzbang solution. Domo was claiming it could get companies off of spreadsheets. But it turns out that they're an enterprise level reporting system that's a minimum of $30,000. That's not a viable option for a 5-person non-profit, but they'll get swept up in the conversation only to find out later that no, Domo and Excel are not equivalents.

deleted 4y ago

[deleted]

leowoo91 4y ago

Of course you can do many stuff, except access control and server side validation. Spreadsheets rather shine with quick planning / analytics.

MetaMonk 4y ago

This is a ridiculous piece of native advertising.

joejohnston 4y ago

Check out Budibase. It's the leading open source low code platform and perfect for building UIs on top of spreadsheets https://github.com/Budibase/budibase https://github.com/Budibase/budibase

cutler 4y ago
Budibase looked perfect until I learned there is no option for hosting locally. Doesn't make sense.
- techplex 4y ago
  Looks like they support self hosting, no? https://docs.budibase.com/docs/hosting-methods https://docs.budibase.com/docs/hosting-methods

newbie2020 4y ago

Been thinking about this problem for a long time. I use python/jupyterhub notebooks for our data analysis flows at work. I’ve become an expert at it and am a go-to person in our org… but even still when I have to try something new on “small” amounts of data, I _still_ go to excel first. It’s barrier to get the results you want os extremely low - thus-far unparalleled by any other software I’ve used in the last 20 years… and I have a hard time seeing anything that would replace that (though maybe I am just set in my ways…)

phkx 4y ago

What do are you using spreadsheets for privately? I never dig deep into their capabilities, so I mostly use them to track expenses within some particular scope, e.g. healthcare. When I recently wanted to compare several loans and estimate our financial situation several years in the future, I wrote some Python code and used a Jupyter notebook to enter parameters and make plots. Has any of you done something similar using spreadsheets? On a side note: I didn’t find a Python library for time series generation (not analysis). Something where you can build some models (e.g. loan, income, expenses) which depend on a common parameter (time) and then evaluate all your models for different values of the common parameter. Right now, I generate pandas series/dataframes and combine them afterwards, which also took some massaging of pandas (which I also usually don‘t use a lot).

vinceguidry 4y ago
I use Google Sheets for personal finance with TillerHQ to automate downloading bank data. It is, as far as I know, the only workable solution in this space. http://www.tillerhq.com http://www.tillerhq.com

LukeEF 4y ago

Founder of excel collaboration and versioning startup [1] here so I am a believer. There are only about 30 million programmers. There are over 1 billion Excel users. Excel is Turing complete. Excel is by far the most used programming language on the planet. It is easily 20 times more popular than the next contender. The value of Excel is that it is presenting the data, with a set of formulae that let you keep derived data up-to-date. This inferred data provides sums and computations, sometimes simple, but sometimes exquisitely complex. And through this whole range of complexity, with a billion users, virtually nobody treats Excel seriously like a programming language. We have a programming language which is essentially acting as a declarative database, and yet we don't do unit tests, we don't keep track of changes, we collaborate with Excel by sending it to our colleagues in the mail and god-forbid we should doing any serious linting of what is in the thing. Anyone who has used Excel in anger realizes why it is so brilliant. Show me another declarative constraint based, data driven inference language that I can teach to my grandmother. The problem isn't Excel. The problem is that we are treating Excel like its a word processor, and not what it is: a programming language. [1] https://versionxl.com/ https://versionxl.com/

janci 4y ago

Excel is great and versatile, yet lacks some critical features. - It's easy to load data from database or API without VBA, yet impossible to write updates back without VBA. With VBA it's still messy string concatenation of SQL queries. - VBA is security nightmare - Version control is bad. Distributing spreadsheets in emails and collecting changes back is nightmare. Sharing a file on network drive lacks fine-grained permission control. - It's hard to maintain proper normalized relational data model. It's impossible to abstract the normalized model from the user (i.e. show labels in selectboxes but store ID's) - It's locale dependent. Date formatting, column separator (comma/semicolon separated), even function names. Unusable in international data exchange. - It mixes formatting, logic and data. Impossible to make reusable blocks. New lambda functions help somewhat. And so much more.

dfox 4y ago
Excel is locale dependent in a weird way. The internal model (which includes the function names) is completely locale independent and only the UI is internationalized (according to OS locale) and localized (according to Excel language version). This is not a bad construction, but breaks down spectacularly when you start to use Excel as any kind of real programming or data analysis environment, because then you get to the cracks in the user-visible “API surface” where the internal/external format distinction is somewhat smeared.
ogogmad 4y ago
> Impossible to make reusable blocks. The click and drag down feature results in a kind-of code reuse. It bypasses the need to consciously name things. Is this generalisable? For small scale code, these concerns may be overkill.
- somat 4y ago
  Click and drag down is the opposite of code reuse, it is effectively a one time code generator. In fact it is one of the main reason I have largely abandoned spreadsheats, the behaviour I actually want is found in sql views. Calculate all items in column with one expression vs calculate items in column with unique expressions that were copy, pasted then transformed for each and every row. The other main reason I abandoned spreadsheets is row level integrity. too many times data that goes together in a row has drifted apart(sort on column subset is main culprit), another inherent problem solved by using a relational database. the solution to having excel at every desk is to have postgres at every desk. the code will be just as bad but the data integrity will be better.

beefield 4y ago

As much as I hate to admit it, it is going to be a long time before we get rid of spreadsheets in de facto production use. And for that, I really would like to see one thing: I'd like to have a separate worksheet type for "datasheets". Looks and behaves just like an ordinary worksheet, but: - plugging in a formula applies the formula on every cell of the column. No exceptions. - you can not have different data types in cells within one column. That is, if you have dates in a column, you can't have a string in one cell of thecolumn etc. Yes, I know about powerthings in excel. No, I do not want to mess with those. Just normal spreadsheet formulas and sheets. (Okay, a couple of other things: Get rid of vba and bring in the full python ecosystem instead. And if not yet available, version control. It is luckily a while I have worked with excel, so these may be outdated comments)

mikewarot 4y ago
>I'd like to have a separate worksheet type for "datasheets". Looks and behaves just like an ordinary worksheet I think you're on to a good idea, but it has to be a table in the middle of a spreadsheet if you want to get acceptance. What would probably cinch it for you is if you could embed an entire table in a bounded box, with scrolling. It would break the row/column addressing for the contents, but that's always clumsy with tables in sheets anyway. Example: a data table lives in C10-M10 / C20/M20 Down in C21-M21 you could have sums, counts, etc. that work across all the data. With this arraignment, you could then sum columns, count, etc. It would be really slick if you snuck in SQL. The tradeoff in immediate addressing of cells within a table could be acceptable. A while ago, I saw something that allowed spreadsheets in spreadsheet cells, it was mostly intended for interactive use, and dispensed with cell addresses completely in a manner that seemed reasonable at the time.

KarlKemp 4y ago

Excel is by far the most successful programming language and IDE. People love to hate it (and the people using it), which is somewhat misguided: there is simply no way to change people (and they keep making new ones), so telling them that what works for them is somehow wrong is both wrong and doesn't work. Instead, the spreadsheet paradigm has the promise of being far more powerful. Jupyter notebooks are one example of adapting it to a different realm, and it also ended up being used everywhere and looked down upon by the snobs.

huijzer 4y ago

At uni, one professor used to say: "spreadsheets have all of the problems of software engineering, but none of the solutions"

foxbee 4y ago

I think 'products replacing spreadsheets' is not the right angle. Spreadsheets are great but not for everything. I think there are a lot of mis-uses of spreadsheets. When users record list type data, long-form content, relational data, etc within spreadsheets, they are probably better off using an app - they, or their IT team can build. I once seen an enterprise org process where a health and safety check was completed on a clipboard. That user would go back to their desk, push that data into excel, another user would write a script to push that excel data into MySQL. There are many use cases where spreadsheets are perfect - building financial models. For transparency - I am the cofounder of a no/low-code tool called Budibase. I am also a happy spreadsheet user. https://github.com/Budibase/budibase https://github.com/Budibase/budibase