5 ms·

I'm interested by the first pull quote: > "...analyzing customer data, which has become too large and complex for Excel. It helps me sift through massive datas

I'm interested by the first pull quote: > "...analyzing customer data, which has become too large and complex for Excel. It helps me sift through massive datasets..." The emphasis here is on the size of the data, but I think that's a red herring. Excel can certainly handle way more data than you're going to want to repeatedly pipe through an LLM. I'm betting it's the diversity of---or even lack of---schemas. Big piles of tables and unstructured data you'd normally have to normalize and join. I spent years doing data analysis slowly with Python (pandas), and it took me a long time to accept that getting better at Excel was just more efficient. Hard to accept that code isn't always a task's best interface. Automating some Excel work sounds even better.

serjester 2y ago
Can you elaborate on what you found Excel better for?
- mbforbes 2y ago
  Nothing capability-wise! Just speed and ease of manipulation. Maybe an analogy is doing image processing. One could write filters and other manipulations in code, or use something like Photoshop. It turns out Photoshop is a great interface for many of those tasks. I think the same thing is true for more data analysis than I'd realized because of my poor Excel skills. Still want to improve my spreadsheet abilities though. There's a fun video called You Suck at Excel by Joel Spolsky I'm halfway through [1]. Would love to know of more like this. [1] https://www.youtube.com/watch?v=JxBg4sMusIg https://www.youtube.com/watch?v=JxBg4sMusIg
simonw 2y ago
You don't pipe the data directly through an LLM with this tool - it writes Python code that manipulates the data. So this will handle millions of rows of data quite happily. I've used it extensively with SQLite in the past - you can upload SQLite databases to it and have it run SQL against them. You can also ask it to save data to SQLite for you to download.
- mbforbes 2y ago
  Thank you, I completely misunderstood that's how it works. I'm guessing then it feeds in the metadata and writes code based on that.
levocardia 2y ago
In the year of our lord 2024, Microsoft Excel is physically incapable of handling more than ~1 million rows or ~16,000 columns. Both are trivially easy to exceed even in very small applications or toy problems. ChatGPT does not ingest the rows of your spreadsheet as tokens, it just uploads the file and runs Python directly on top of it.
- tomnipotent 2y ago
  Excel Data Model has been around for years (2013?) and can handle ~2bln rows across a maximum of 2bln tables or 2bln columns. I don't know what that means in the context of ChatGPT, but the capability exists.

serjester 2y ago

Can you elaborate on what you found Excel better for?

mbforbes 2y ago
Nothing capability-wise! Just speed and ease of manipulation. Maybe an analogy is doing image processing. One could write filters and other manipulations in code, or use something like Photoshop. It turns out Photoshop is a great interface for many of those tasks. I think the same thing is true for more data analysis than I'd realized because of my poor Excel skills. Still want to improve my spreadsheet abilities though. There's a fun video called You Suck at Excel by Joel Spolsky I'm halfway through [1]. Would love to know of more like this. [1] https://www.youtube.com/watch?v=JxBg4sMusIg https://www.youtube.com/watch?v=JxBg4sMusIg

simonw 2y ago

You don't pipe the data directly through an LLM with this tool - it writes Python code that manipulates the data. So this will handle millions of rows of data quite happily. I've used it extensively with SQLite in the past - you can upload SQLite databases to it and have it run SQL against them. You can also ask it to save data to SQLite for you to download.

mbforbes 2y ago
Thank you, I completely misunderstood that's how it works. I'm guessing then it feeds in the metadata and writes code based on that.

levocardia 2y ago

In the year of our lord 2024, Microsoft Excel is physically incapable of handling more than ~1 million rows or ~16,000 columns. Both are trivially easy to exceed even in very small applications or toy problems. ChatGPT does not ingest the rows of your spreadsheet as tokens, it just uploads the file and runs Python directly on top of it.

tomnipotent 2y ago
Excel Data Model has been around for years (2013?) and can handle ~2bln rows across a maximum of 2bln tables or 2bln columns. I don't know what that means in the context of ChatGPT, but the capability exists.