5 ms·

Social Movements Are Pushing Google Sheets to the Breaking Point

75 points by raybb 6y ago

raybb 6y ago
Twitter link that bypasses paywall: https://t.co/WJHwemmfVl?amp=1 https://t.co/WJHwemmfVl?amp=1 Archve.is link: https://archive.is/N00i5 https://archive.is/N00i5 This got me thinking what would it take to build a decent collaborative editor that would work for hundreds of people.
- jkinudsjknds 6y ago
  It's more likely that a real time editor for spreadsheets just doesn't make sense for more than a few people. The low hanging fruit would be adding a read-only view of a static copy, because that's what people actually needed.
ENOTTY 6y ago
Sheets seems like a good way to get to a MVP quickly to prove the concept. If Sheets starts falling over due to demand, you know you've got a winner and its time to invest in something more reliable.
- arkitaip 6y ago
  Not everything is a MVP ready to disrupt. Sometimes all you have is a spreadsheet with data that lots of people have a fleeting interest in.
  - toomuchtodo 6y ago
    A spreadsheet is a database you haven't yet performed a migration on yet. I think the argument is less around better spreadsheets and more around migrating that data into real datastores once constraints are encountered (whether that's a desire to support large amounts of contributors, not run on Google, the dataset has grown too large, etc). It's hard to go from Google Sheets to a web frontend to Postgresql and a snappy shared frontend. That is the problem statement to solve for imho (interface, schema, data portability).
    - PeterCorless 6y ago
      No one has really solved the "Spreadsheet —> Database" migration painpoint. We don't have anything that is universally easy-to-use. The equivalent of a Hypercard or a Filemaker for the Web. (And no, I'm not talking about MongoDB or other NoSQL; I'm talking about the kind of thing you can hand over to a typical office worker.)
      - toomuchtodo 6y ago
        Agreed. I have a hunch that any solution to this problem will look similar to https://github.com/simonw/datasette https://github.com/simonw/datasette if it develops. Tied together with https://supabase.io/ https://supabase.io/ and you've built a versatile, powerful, open data collaboration engine.
- ISL 6y ago
  The tricky part is the timescale. For a crisis-response, an approach may reach maximum impact within days and really only be needed for a short duration (a week to a month). Migrating a system with intense user/volunteer churn (and keeping two databases synced as you transition) can be tricky. Handling permissioning at the same time is also a challenge. At FindTheMasks, we expected the government to have supply chains and epidemiological response (testing/contact tracing) together within two or three weeks. When that didn't happen, we kept bolting on new tools and strategies to double the lifetime of the endeavor, expecting the government to get its poop in a group by then. 20 weeks later, supply chains are somewhat improved, though the entire situation is compromised by the lack of overwhelming success at the epidemiological scale.
gverrilla 6y ago
This reads to me big infrastructural problems in internet.
blakesterz 6y ago
"A Google support page states that “up to 100 people with view, edit, or comment permissions can work on a Google Docs, Sheets, or Slides file at the same time” and has a section devoted to troubleshooting files that become unresponsive after being shared with many people, recognizing the common pitfall." That's a big number, but it's not as big as I'd expect from something from Google. I'd assume the average number of people working on any Sheet is 1 or 2, but I'm surprised that haven't been able to scale it up to 1000 or more by now.
- deleted 6y ago
  [deleted]
- robjan 6y ago
  I would think the document would quickly become unmanageable if more than a few dozen are editing it at the same time. Even if it's possible technically, we start to enter social problems.
- jfindley 6y ago
  I believe that it's more that enabling that scaling would likely come with tradeoffs that would have negative impacts on the more common usecases.
- pwinnski 6y ago
  Also in the piece, "Jonathan Rochelle, the lead product manager for Google’s suite of collaborative tools from 2005 to 2013, said that he and his team found that having more than 25 to 50 editors on a single document made it hard for people to collaborate effectively." The technical limitation is 2-4 times what the former product manager believed was a human limitation already. Scaling beyond that doesn't seem like a priority.
  - nitrogen 6y ago
    "View" was in that list though; is there a way to let more than 100 people view a sheet?
    - 6y ago
snowwrestler 6y ago
Not just social movements; I sat in a free career webinar and they posted a Google Sheet link in the chat for all the participants to share LinkedIn profile links. It crashed quickly and for the rest of the meeting the chat had a steady stream of comments like "it's working now", "I'm locked out, please help", "can anyone share the link to the sheet", "here's a link to another sheet we can use", etc. I sat there thinking "I can't believe there isn't a better option for this sort of thing." Kind of surprised Zoom does not have a native feature for meeting participants to share contact info with each other during the meeting.
- thebean11 6y ago
  A google form that inserted the links into a public but read-only sheet would have performed fine I'm guessing
- joshvm 6y ago
  For surveying-type requests you should probably be using a Google Form which links to a Sheet (with limited access control). You shouldn't be giving everyone simultaneous edit access to the document.
- scottlamb 6y ago
  The main problem with Google Sheets for that use case IIUC is that it's designed to be prepared for any of the users to edit arbitrary cells which can cause recomputations of other arbitrary cells, and it wants to give real-time feedback to all the other users about every edit (and cursor movement). That makes it harder to scale up to many simultaneous editors without compromises in the backend (due to the graph of recalculations) and UI (clutter, contention to claim the first blank line). A custom web app for this purpose can scale better by being designed with the knowledge each user only edits their own row and nothing else needs to be recalculated. And of course it can have a tailored UI. Unfortunately I don't know of any way for a non-developer to match that. There's no fundamental reason IMHO that there couldn't be a nice rapid, code-optional CRUD app generator, like Microsoft Access for the 21st century. Maybe this is what Microsoft Lists will do? AirTable is a general-purpose web-based RDBMS frontend, but I don't know if it scales well to many simultaneous editors either, I don't think you can tailor the UI as much as I'd like, and I hear folks complaining about database size limits. You can get Google Sheets to scale better, but it's a bit awkward: * Give folks a Google Forms link to edit their own information. This atomically claims and populates the row. * Give folks a publish link (see File | Publish To the Web) to view everyone else's information. This link scales well because it doesn't let them update anything and (more annoyingly) it doesn't show changes until they hit refresh.
aspenmayer 6y ago
I feel like Sheets should gracefully fail much better than it does under load, especially for supposedly being Google-scale. With more users, instead of not loading at all, or starting then failing to load, it would be nice to see the most recent change in read-only mode, and show an edit timeline like the Internet Archive's Wayback Machine UI shows archive occurrences on a calendar. Just a couple ideas for how Sheets could break down gracefully while retaining usability.
- reaperducer 6y ago
  With more users, instead of not loading at all, or starting then failing to load, it would be nice to see the most recent change in read-only mode I wonder if that's why the new version of macOS Numbers defaults to a read-only mode — to ease the burden when the sheets are in collaboration.
disease 6y ago
This is the second story on the front page about the weakness of modern spreadsheet software. It could be cool if someone stepped forward and showed us how to do spreadsheets using everything we now know, like what git did with source control back in the mid 2000s.
- raybb 6y ago
  Can you link to the other story? I'm not seeing it.
  - reallydontask 6y ago
    there you go https://www.theverge.com/2020/8/6/21355674/human-genes-rename-microsoft-excel-misreading-dates https://www.theverge.com/2020/8/6/21355674/human-genes-renam... edit: HN link https://news.ycombinator.com/item?id=24070385 https://news.ycombinator.com/item?id=24070385
- WorldMaker 6y ago
  What about "Don't use spreadsheets"? It's a flippant answer, sure, but spreadsheets are the index cards, sticky notes, and yarn approach to data organization. Certainly they are easy and approachable to use, but any sufficiently complex data set in general is going to be more "chaotic mind corkboard mess" than "organized". Spreadsheets are very useful data hammer, but not all data is a nail; to borrow another aphorism. That said, there's a wide gulf in the general user approachability between spreadsheet applications and more bespoke/"better" data organization options. Decades of experiences and mistakes learned from tools like DB2, FoxPro, and Access have shown us we have a hard time finding an "easy medium". AirTable thinks they have some modern ideas. Microsoft's PowerApps teams have their own modern approaches (and as almost the direct the inheritors of FoxPro/Access, presumably the most experience to learn from). Simon Willison's work with Datasette (SQLite DBs wrapped in simple view/management webpages) has had some interesting ideas. Though I don't think any one approach will solve it, it might take a whole shelf of tools, and at the end of the day maybe as a species we'll never have as clean of a user experience as the worst spreadsheet.
ISL 6y ago
With FindTheMasks, we found that tooling written around Sheets may also confront the 5 million cell limit. As you tack on extra tabs and handle more fields, it isn't too hard to approach that limit. If we do trip that threshold, we will be forced to degrade some functionality. As a bonus, I don't believe that Sheets allows anyone to see exactly how many cells are in use, so one lives with a probabilistic risk of lockup. If we were continuing to handle large volumes of intake, a switch to Airtable would be in the works. Fortunately, the PPE situation is at least somewhat better than it was in March, so we are simply keeping the lights on and wheels turning.
- spacehunt 6y ago
  Airtable has a hard limit of 50,000 records across all tables in a base though.
  - ISL 6y ago
    If those records could have an arbitrary/large number of fields, that would suffice for our foreseeable needs. We're at ~6,000 individual entries at the moment and do not expect a 10x jump at this time. Our efforts now center on updates to and removals from our public map. Airtable-wise, I'm parroting our devs on the subject. We had perpetual/nascent plans for an Airtable migration (and did so when FindTheMakers came to life) which have been tabled for now. We believe we'll make it to sunset on Sheets, but just barely.
  - arthurcolle 6y ago
    That seems like a crazy low limit compared to the traditional spreadsheets I'm used to. Is there some special meaning of "record" in their vernacular?
    - spacehunt 6y ago
      Airtable is closer to a database than a spreadsheet. Records are represented as rows in their default grid view, but there are other views available. They can also be retrieved as JSON objects through the API that Airtable automatically generate for you.
somurzakov 6y ago
that means there is an opportunity for opensoirce, lightweight, self-hosted Sheets. A product opportunity for Zoom/Slack/Github. I would love to see Elixir based solution just to see how much performance can be squeezed out of it, whatsapp seems to handle huge load just fine
lukeschlather 6y ago
I've been using Docs (and a bit of sheets) for stuff like this and haven't noticed too much issues, though I usually keep under 10 people. Honestly I hope Google doesn't try to address any of the complaints in this thread. Google Docs/Sheets/Drive function a lot like Stardew Valley multiplayer, in that pretty much anyone can do anything, and this allows really fluid communication but it simply won't work with 100 people. Sheets supports 100s of people to an extent (even more) but I wouldn't want them to optimize for this. I think if I were in charge of it I would be exploring ways that they can find UI paradigms that allow more seamless interaction and realize the dreams that I think Google had when they built Wave. * What if you could embed live video/audio in a Doc? * Sheets comments kind of suck. * Is there a way to use comments that makes them more visible without ruining the sheet view? (Maybe not.) * As this article says, people are often using a Sheet as a very hacky app dev platform. Is there a way to embed the sheet data in a Doc (or something Wave-like) that has a more seamless communication pattern? * Is there a way to build small apps on the fly without messing with a bunch of macros? Though I think a lot of this is about communication. I don't think Google should be thinking about scaling, they should be thinking about those use cases where you're talking to a small group of people and you need a little custom app that can't take more than 30 seconds to put together or just talking would be more efficient.
Yhippa 6y ago
Let's say they fix Google Sheets to be arbitrarily scalable. Now you have another problem: how can that many people edit the Sheet simultaneousy and still have it make sense?
deleted 6y ago
[deleted]
jlkuester7 6y ago
The technologically maximum number of concurrent editors that Google Sheets supports is perhaps the least important constraint when trying to crowdsource an information-base. At a certain point (well below the point where Google Sheets crashes) you run into community constraints around permissions, validation, how you organize the data, how you record changes, how you discuss changes. That being said, this article seems to completely ignore the fact that there are already well-established solutions to this problem. Off the top of my head, wikis and GitHub/Lab pages offer very nice solutions to creating a scale-able information-base with built-in features to help address some of the challenges I raised above....

75 points by raybb 6y ago

Twitter link that bypasses paywall: https://t.co/WJHwemmfVl?amp=1 https://t.co/WJHwemmfVl?amp=1 Archve.is link: https://archive.is/N00i5 https://archive.is/N00i5 This got me thinking what would it take to build a decent collaborative editor that would work for hundreds of people.

jkinudsjknds 6y ago
It's more likely that a real time editor for spreadsheets just doesn't make sense for more than a few people. The low hanging fruit would be adding a read-only view of a static copy, because that's what people actually needed.

ENOTTY 6y ago

Sheets seems like a good way to get to a MVP quickly to prove the concept. If Sheets starts falling over due to demand, you know you've got a winner and its time to invest in something more reliable.

arkitaip 6y ago
Not everything is a MVP ready to disrupt. Sometimes all you have is a spreadsheet with data that lots of people have a fleeting interest in.
- toomuchtodo 6y ago
  A spreadsheet is a database you haven't yet performed a migration on yet. I think the argument is less around better spreadsheets and more around migrating that data into real datastores once constraints are encountered (whether that's a desire to support large amounts of contributors, not run on Google, the dataset has grown too large, etc). It's hard to go from Google Sheets to a web frontend to Postgresql and a snappy shared frontend. That is the problem statement to solve for imho (interface, schema, data portability).
  - PeterCorless 6y ago
    No one has really solved the "Spreadsheet —> Database" migration painpoint. We don't have anything that is universally easy-to-use. The equivalent of a Hypercard or a Filemaker for the Web. (And no, I'm not talking about MongoDB or other NoSQL; I'm talking about the kind of thing you can hand over to a typical office worker.)
    - toomuchtodo 6y ago
      Agreed. I have a hunch that any solution to this problem will look similar to https://github.com/simonw/datasette https://github.com/simonw/datasette if it develops. Tied together with https://supabase.io/ https://supabase.io/ and you've built a versatile, powerful, open data collaboration engine.
ISL 6y ago
The tricky part is the timescale. For a crisis-response, an approach may reach maximum impact within days and really only be needed for a short duration (a week to a month). Migrating a system with intense user/volunteer churn (and keeping two databases synced as you transition) can be tricky. Handling permissioning at the same time is also a challenge. At FindTheMasks, we expected the government to have supply chains and epidemiological response (testing/contact tracing) together within two or three weeks. When that didn't happen, we kept bolting on new tools and strategies to double the lifetime of the endeavor, expecting the government to get its poop in a group by then. 20 weeks later, supply chains are somewhat improved, though the entire situation is compromised by the lack of overwhelming success at the epidemiological scale.

gverrilla 6y ago

This reads to me big infrastructural problems in internet.

blakesterz 6y ago

"A Google support page states that “up to 100 people with view, edit, or comment permissions can work on a Google Docs, Sheets, or Slides file at the same time” and has a section devoted to troubleshooting files that become unresponsive after being shared with many people, recognizing the common pitfall." That's a big number, but it's not as big as I'd expect from something from Google. I'd assume the average number of people working on any Sheet is 1 or 2, but I'm surprised that haven't been able to scale it up to 1000 or more by now.

deleted 6y ago
[deleted]
robjan 6y ago
I would think the document would quickly become unmanageable if more than a few dozen are editing it at the same time. Even if it's possible technically, we start to enter social problems.
jfindley 6y ago
I believe that it's more that enabling that scaling would likely come with tradeoffs that would have negative impacts on the more common usecases.
pwinnski 6y ago
Also in the piece, "Jonathan Rochelle, the lead product manager for Google’s suite of collaborative tools from 2005 to 2013, said that he and his team found that having more than 25 to 50 editors on a single document made it hard for people to collaborate effectively." The technical limitation is 2-4 times what the former product manager believed was a human limitation already. Scaling beyond that doesn't seem like a priority.
- nitrogen 6y ago
  "View" was in that list though; is there a way to let more than 100 people view a sheet?
  - 6y ago

snowwrestler 6y ago

Not just social movements; I sat in a free career webinar and they posted a Google Sheet link in the chat for all the participants to share LinkedIn profile links. It crashed quickly and for the rest of the meeting the chat had a steady stream of comments like "it's working now", "I'm locked out, please help", "can anyone share the link to the sheet", "here's a link to another sheet we can use", etc. I sat there thinking "I can't believe there isn't a better option for this sort of thing." Kind of surprised Zoom does not have a native feature for meeting participants to share contact info with each other during the meeting.

thebean11 6y ago
A google form that inserted the links into a public but read-only sheet would have performed fine I'm guessing
joshvm 6y ago
For surveying-type requests you should probably be using a Google Form which links to a Sheet (with limited access control). You shouldn't be giving everyone simultaneous edit access to the document.
scottlamb 6y ago
The main problem with Google Sheets for that use case IIUC is that it's designed to be prepared for any of the users to edit arbitrary cells which can cause recomputations of other arbitrary cells, and it wants to give real-time feedback to all the other users about every edit (and cursor movement). That makes it harder to scale up to many simultaneous editors without compromises in the backend (due to the graph of recalculations) and UI (clutter, contention to claim the first blank line). A custom web app for this purpose can scale better by being designed with the knowledge each user only edits their own row and nothing else needs to be recalculated. And of course it can have a tailored UI. Unfortunately I don't know of any way for a non-developer to match that. There's no fundamental reason IMHO that there couldn't be a nice rapid, code-optional CRUD app generator, like Microsoft Access for the 21st century. Maybe this is what Microsoft Lists will do? AirTable is a general-purpose web-based RDBMS frontend, but I don't know if it scales well to many simultaneous editors either, I don't think you can tailor the UI as much as I'd like, and I hear folks complaining about database size limits. You can get Google Sheets to scale better, but it's a bit awkward: * Give folks a Google Forms link to edit their own information. This atomically claims and populates the row. * Give folks a publish link (see File | Publish To the Web) to view everyone else's information. This link scales well because it doesn't let them update anything and (more annoyingly) it doesn't show changes until they hit refresh.

aspenmayer 6y ago

I feel like Sheets should gracefully fail much better than it does under load, especially for supposedly being Google-scale. With more users, instead of not loading at all, or starting then failing to load, it would be nice to see the most recent change in read-only mode, and show an edit timeline like the Internet Archive's Wayback Machine UI shows archive occurrences on a calendar. Just a couple ideas for how Sheets could break down gracefully while retaining usability.

reaperducer 6y ago
With more users, instead of not loading at all, or starting then failing to load, it would be nice to see the most recent change in read-only mode I wonder if that's why the new version of macOS Numbers defaults to a read-only mode — to ease the burden when the sheets are in collaboration.

disease 6y ago

This is the second story on the front page about the weakness of modern spreadsheet software. It could be cool if someone stepped forward and showed us how to do spreadsheets using everything we now know, like what git did with source control back in the mid 2000s.

raybb 6y ago
Can you link to the other story? I'm not seeing it.
- reallydontask 6y ago
  there you go https://www.theverge.com/2020/8/6/21355674/human-genes-rename-microsoft-excel-misreading-dates https://www.theverge.com/2020/8/6/21355674/human-genes-renam... edit: HN link https://news.ycombinator.com/item?id=24070385 https://news.ycombinator.com/item?id=24070385
WorldMaker 6y ago
What about "Don't use spreadsheets"? It's a flippant answer, sure, but spreadsheets are the index cards, sticky notes, and yarn approach to data organization. Certainly they are easy and approachable to use, but any sufficiently complex data set in general is going to be more "chaotic mind corkboard mess" than "organized". Spreadsheets are very useful data hammer, but not all data is a nail; to borrow another aphorism. That said, there's a wide gulf in the general user approachability between spreadsheet applications and more bespoke/"better" data organization options. Decades of experiences and mistakes learned from tools like DB2, FoxPro, and Access have shown us we have a hard time finding an "easy medium". AirTable thinks they have some modern ideas. Microsoft's PowerApps teams have their own modern approaches (and as almost the direct the inheritors of FoxPro/Access, presumably the most experience to learn from). Simon Willison's work with Datasette (SQLite DBs wrapped in simple view/management webpages) has had some interesting ideas. Though I don't think any one approach will solve it, it might take a whole shelf of tools, and at the end of the day maybe as a species we'll never have as clean of a user experience as the worst spreadsheet.

ISL 6y ago

With FindTheMasks, we found that tooling written around Sheets may also confront the 5 million cell limit. As you tack on extra tabs and handle more fields, it isn't too hard to approach that limit. If we do trip that threshold, we will be forced to degrade some functionality. As a bonus, I don't believe that Sheets allows anyone to see exactly how many cells are in use, so one lives with a probabilistic risk of lockup. If we were continuing to handle large volumes of intake, a switch to Airtable would be in the works. Fortunately, the PPE situation is at least somewhat better than it was in March, so we are simply keeping the lights on and wheels turning.

spacehunt 6y ago
Airtable has a hard limit of 50,000 records across all tables in a base though.
- ISL 6y ago
  If those records could have an arbitrary/large number of fields, that would suffice for our foreseeable needs. We're at ~6,000 individual entries at the moment and do not expect a 10x jump at this time. Our efforts now center on updates to and removals from our public map. Airtable-wise, I'm parroting our devs on the subject. We had perpetual/nascent plans for an Airtable migration (and did so when FindTheMakers came to life) which have been tabled for now. We believe we'll make it to sunset on Sheets, but just barely.
- arthurcolle 6y ago
  That seems like a crazy low limit compared to the traditional spreadsheets I'm used to. Is there some special meaning of "record" in their vernacular?
  - spacehunt 6y ago
    Airtable is closer to a database than a spreadsheet. Records are represented as rows in their default grid view, but there are other views available. They can also be retrieved as JSON objects through the API that Airtable automatically generate for you.

somurzakov 6y ago

that means there is an opportunity for opensoirce, lightweight, self-hosted Sheets. A product opportunity for Zoom/Slack/Github. I would love to see Elixir based solution just to see how much performance can be squeezed out of it, whatsapp seems to handle huge load just fine

lukeschlather 6y ago

I've been using Docs (and a bit of sheets) for stuff like this and haven't noticed too much issues, though I usually keep under 10 people. Honestly I hope Google doesn't try to address any of the complaints in this thread. Google Docs/Sheets/Drive function a lot like Stardew Valley multiplayer, in that pretty much anyone can do anything, and this allows really fluid communication but it simply won't work with 100 people. Sheets supports 100s of people to an extent (even more) but I wouldn't want them to optimize for this. I think if I were in charge of it I would be exploring ways that they can find UI paradigms that allow more seamless interaction and realize the dreams that I think Google had when they built Wave. * What if you could embed live video/audio in a Doc? * Sheets comments kind of suck. * Is there a way to use comments that makes them more visible without ruining the sheet view? (Maybe not.) * As this article says, people are often using a Sheet as a very hacky app dev platform. Is there a way to embed the sheet data in a Doc (or something Wave-like) that has a more seamless communication pattern? * Is there a way to build small apps on the fly without messing with a bunch of macros? Though I think a lot of this is about communication. I don't think Google should be thinking about scaling, they should be thinking about those use cases where you're talking to a small group of people and you need a little custom app that can't take more than 30 seconds to put together or just talking would be more efficient.

Yhippa 6y ago

Let's say they fix Google Sheets to be arbitrarily scalable. Now you have another problem: how can that many people edit the Sheet simultaneousy and still have it make sense?

deleted 6y ago

[deleted]

jlkuester7 6y ago

The technologically maximum number of concurrent editors that Google Sheets supports is perhaps the least important constraint when trying to crowdsource an information-base. At a certain point (well below the point where Google Sheets crashes) you run into community constraints around permissions, validation, how you organize the data, how you record changes, how you discuss changes. That being said, this article seems to completely ignore the fact that there are already well-established solutions to this problem. Off the top of my head, wikis and GitHub/Lab pages offer very nice solutions to creating a scale-able information-base with built-in features to help address some of the challenges I raised above....