9 ms·

One line of code that did cost $8k

115 points by samwillis 3y ago

elaus 3y ago
Would love to know how they resolved the problem on a technical level. After all, the apps running in the background would still try to download updates every five minutes and disabling/blocking those update requests on the server might cut those installations off from ever auto-updating again.
- yifanl 3y ago
  In another thread here, the founder states that their solution is to reduce the rate of updates from every 5 minutes to every 3 hours.
  - Clamchop 3y ago
    That and they fixed the bug, importantly. Additional feedback on HN about checking even less often, getting user consent, and writing regression tests but commentators are being dramatic and insulting about it.
robin_reala 3y ago
One of our users, who lived in a house, had their internet provider cancel their contract due to enormous traffic generated during a month. It was extremely problematic as there was no other internet provider available around. We decided to take responsibility and offer to cover all the costs related to this situation. This is how you deal with a situation like this that you’ve caused. Accidents happen, what matters is how you clean up after.
- ghaff 3y ago
  There is, although if you have one wired broadband provider for an address, and that provider isn't willing to overlook a one-time issue, there may not be a lot of options that money can resolve. (The--sometimes--availability of Starlink helps these days but a sole supplier cutting you off doesn't have a universal fix.)
tylervigen 3y ago
Is there a reason an app needs to check for updates every 5 minutes vs, say, once a day? That would have made this situation 288 times better, and reduced the processing tax on people’s devices.
- sudhirj 3y ago
  It was a mistake, a debugging line that made it to prod.
  - ndespres 3y ago
    The article says that the mistake was downloading the update repeatedly. The 5-minute check for updates was intentional.
Cenk 3y ago
Why not just use the Sparkle update framework like almost all indie Mac apps?
CaptainBern 3y ago
Why download updates at all without the user knowing? Is prompting before downloading such an awful user experience in comparison? Or, you know, prompt once per day and add a 'remind me in x days/minutes' option? The whole take-away of this article feels backwards. Instead of going 'hey maybe we should not just download stuff in the background without prompt', the author doubles down and actually proposes to add a way for the server to signal to the client that an update should be forced. I'm actually baffled by this entire thing.
- nchudleigh 3y ago
  The bug was the app kept downloading after the user agreed to download the new version.
  - CaptainBern 3y ago
    The article did not mention that as far as I can tell. >The app checks for the update every 5 minutes or when the user activates the app. >Normally, when the app detected the update - it downloaded it and stopped the 5 minutes interval until the user installed it and restarted it. To me this reads as them downloading the update unprompted (as they mention their issue with the previous updater was that it would prompt the users every 5 minutes, which would be solved by just not checking every 5 minutes). Either way, whatever the bug and its effects are, my gripe is that their conclusion is to add a way for them to remotely force-updates. As opposed to just notifying once or any of the other hundred ways to handle auto-updates that don't involve you remotely forcing an update. Edit: the founder has confirmed in the comments the app does download without user confirmation, so my original point still stands.
- michaelt 3y ago
  Well, if the user wants the update, downloading in advance is a much better experience, because you only get interrupted once, for a restart that only takes a second or two. That's in contrast to being interrupted twice (once to download, then again to restart) or a single long interruption (shutting down, then downloading, then restarting). Of course, it's a different matter if the user doesn't want the update. But we developers don't like to think about that :)
pachico 3y ago
Maybe this will encourage developers to make smaller applications too.
torstenvl 3y ago
> Screen Studio is a desktop app. It means we need some auto-updater No, it doesn't. People used desktop apps perfectly fine before the always-on phone-home era of nearly universal broadband. What's more troubling though is that this is somehow a story about monitoring cloud costs, and not a story about why your update interval was every five minutes instead of daily; or why it would auto-download instead of politely (no popups!) letting the user know there was an update; or why you failed to reconsider the fundamentals of your design in the face of the first round of negative feedback. Your users told you they felt harassed and interrupted by your prompts. That's why you were making this change in the first place. That should have told you something.
- Spivak 3y ago
  User expectations have changed in this regard. If you don't auto update users will never update. Updating your computer or apps isn't a thing anyone thinks about anymore. Don't do it like the OP clearly, and it's really sad that desktops can't provide an experience like mobile where updates are platform managed but you still gotta do it.
  - computerfriend 3y ago
    I use a package manager. It's normal, common even. My "user expectation" is that applications don't do this.
  - ghaff 3y ago
    Then they don't update. People rarely updating seems a lesser issue than forcing updates on everyone--especially at the application layer.
  - dathinab 3y ago
    yesn't there needs to be some auto updater but you don't necessary need to have some or in other words you normally want to integrate into existing update services, e.g. then package manager on linux. And if it then doesn't get updated e.g. because people don't update their system at all then that's their choice and problem IMHO. Just flag a warning/error once the API to your service changed too much or similar.
ndespres 3y ago
>> The app checks for the update every 5 minutes or when the user activates the app. This is why I use Little Snitch, an application firewall that blocks requests like this and gives me some control over which applications are phoning home all the time. There’s no reason a desktop application should check for updates every 5 minutes. Once a month ought to do it without a compelling reason for more. But none of the lessons learned in the article show any attempt to reevaluate the update check frequency.
- nchudleigh 3y ago
  Eh I'd say daily. Especially in a case like screen studio where it's under active development.
  - jacquesm 3y ago
    Why? Updates may or may not be to the benefit of the user, who should have a clear say in the time and place when their stuff is updated, nothing is more frustrating to have a job to do and then to sit there waiting for some silly update process to complete only to find out that (1) it doesn't matter to you anyway or (2) that they've borked the update or (3) that critical functionality you relied on has been removed or (4) that the marketing department thought this was a great time to try to upsell you on something. Mandatory updates should only happen for critical security issues and even those don't need to be fixed on a five minute interval, once a day at most for a check and less frequently is probably fine as well unless the app is absolutely mission critical and people will die. And even then you should probably wonder if your QA is so bad that you need to be able to intervene on a moments notice, better to design your process in such a way that you can alert the user rather than to forcibly upgrade their system, especially if there is a chance that you'll get it wrong (again). An upgrade is always a burden on the users, so you really don't want to do this too often and even if you are capable of releasing multiple updates per week it would be ridiculous to expect your users to follow that frequency.
    - pluijzer 3y ago
      In the past if there was an update for software I use I was excited and was curious about all the improvements it would bring. Now when I see some app has an update I have a feeling of dread as I am sure it will be slower, the UI more clumsy and the power features I like will surely be removed.
PurpleRamen 3y ago
This reads like there is no version-check for updates? Because why would it otherwise download the same file over and over again?
- fredoralive 3y ago
  It doesn't seem to be automatically installing the update, presumably as that would disrupt the user by restarting the app. I guess it's staging the update to run on exit or similar. So until the user actually quits the app (or clicks update etc.), it's going to be on the old version, and in this endless redownload loop.
someweirdperson 3y ago
At that update frequency, at least the fix can be deployed quickly.
dj_mc_merlin 3y ago
Not a single mention of rate limiting? This isn't 2008, if you're offering an online service and do not have some form of rate limiting that will trigger when your throughput increases by 3 orders of magnitude, that's on you.
- mschuster91 3y ago
  If you're using plain AWS S3 you can't reasonably add a rate limit - probably if you put some Lambda Edge function or Cloudfront in front of it, but that's unnecessary cost and complexity.
- 8organicbits 3y ago
  I think the approach for S3 is to have the app call a rate limted API to get the download URL, possibly as a pre-signed URL, possibly with an HTTP redirect. That doesn't help if the client caches that URL and downloads it from S3 directly many times. Alternatively, you could run your own update server, but S3 is so convenient. There's also a concern that the update is designed to check every five minutes. So a rate limit may not actually get exceeded.
  - dj_mc_merlin 3y ago
    Late reply, but > you could run your own update server, but S3 is so convenient. IMO if you're providing updates to your users you're distributing code automatically and potentially infecting their computers if anything goes wrong. You should take all proper precautions.
Ekaros 3y ago
>> Normally, when the app detected the update - it downloaded it and stopped the 5 minutes interval until the user installed it and restarted it. If I'm reading this correctly, the update mechanism stops if there is updated downloaded and waiting. So if user just doesn't install it, it will not get the versions after that? Seems like whole logic might have more issues, beyond the basic uselessness.
wslh 3y ago
The crypto world is full of single lines of code that are worths million of dollars. No pun intended. BTW, in our team we compiled a spreadsheet showing Git commits before and after security audit reports [1]. [1] https://docs.google.com/spreadsheets/d/1bqtQI05z-O6uclvYYXxHo09uDx5lclKByUEfLiZACrA/edit#gid=0 https://docs.google.com/spreadsheets/d/1bqtQI05z-O6uclvYYXxH...
ysleepy 3y ago
Why the hell does the app check every 5min for updates? Even once a day seems excessive. Thy cynic in me sees a pattern of disregard for other people's resources like with electron. Being a good software citizen is also realising that the product is not the main task running on a computer and it's not acceptable to make decisions about other peoples priorities. It feels like every tray daemon and widget assumes they are the main reason the computer is running and load a web view, 3 windows services and scheduled tasks. Admittedly the App in question is interactive, but the same principle holds. Categorical imperative hah: Construct in a way that you accept all tray icons to be constructed like.
- pbhjpbhj 3y ago
  It's more insidious than disregard, I expect. I assume it's a beacon masquerading as an update check - monitoring app usage, user locations, and such. Aside: If your users are in the same time zone you may not want to do checks/updates on startup per se in order to avoid putting all your update load into the same time slice.
  - lightbendover 3y ago
    I’m not tracking. You can perform all that monitoring and more without pretending to be a software update.
  - tough 3y ago
    > Aside: If your users are in the same time zone you may not want to do checks/updates on startup per se in order to avoid putting all your update load into the same time slice. This one bite my in the ass with a login used by IRL offices which all wanted to log in at 9:00 at the same time lol
- vintagedave 3y ago
  > The problem with the auto-updater we had was that it would prompt the user to update the app as soon as it became available. This resulted in a popup appearing while users were recording the screen, which obviously provided a bad experience as it interrupted the recording the user was making. The 5 minute bug was introduced while fixing this issue. But it was a bad one: I have so many apps "randomly" popping update notifications up all the time: Skype and Fantastical are the two worst, seemingly every day. The popups interrupt me when working, and _take keyboard focus_ meaning I lose the last half of a sentence, or sometimes even dismiss the dialog before I realise it's appeared. > the product is not the main task running on a computer and it's not acceptable to make decisions about other peoples priorities Yes! One thing I loved about early macOS (about OS X Tiger - Snow Leopard days) was that I never saw apps steal focus. On Windows at the time it happened all the time in various ways, but never on Mac. That included when apps started: if you were working in an app and started another but kept working in the first, the second started with the window in the background. It was lovely.
throwaway290 3y ago
Does it consider how much it cost users on mobile hotspots?
- ghaff 3y ago
  There are all sorts of circumstances, see also international travel, when unintentional downloads can get really expensive and it's not reasonable to expect people to police how and when they use data at a micro level. I'd much rather pester people to give permission for things than just download in the background because that's "probably what they want."
pachico 3y ago
I think this post backfired at the writer. While trying to provide transparency over a problem (we all screw up) it uncovered what I think it's a horrible idea. The issue is not the bug anymore but the update process. These comments in HN might provide much more insights than the bug itself.
Damogran6 3y ago
I hate this aspect of modern computing…Time was, you could look over at your firewall, or notice that the local network was running like crap…but Google (and for this post, Comcast/Xfinity) have GOBS of capacity. It’s easy to manage costs when your bandwidth is constrained to a relatively low rate… That super high speed 1Gb fiber connection? The one that STILL has the 1Tb/Mo cap? You can blow through it in an afternoon with one poorly constrained app (e.g. Torrenting) Same here. Google has _company ending_ amounts of bandwidth in their datacenters and you can lose everything without knowing it. (See also medical billing)
- rkangel 3y ago
  I think there is a problem here, but I don't think think that's the right problem. Not having these arbitrary technical limits is a good thing - for instance their service didn't degrade due to an unrelated problem. Regulating cost indirectly via some other metric is a very oblique way of solving the problem The right solution would be throttling of spending - easy way to set proper limits, good notifications etc. It's possible to set up all these limits but nowhere near as easy as it should be. Also, how easy would it be for Google to send you an email saying "your spend looks a bit different than normal due to this traffic here"!
- amelius 3y ago
  Bandwidth shaping/traffic control in Linux is not very mature. But yes, Google and others should implement a cap.
- _fat_santa 3y ago
  From personal experience, paying Xfinity an extra $30 or so per month to get the cap taken off was one of the first "upgrades" I made to my internet. Between me pulling node_modules for multiple projects and watching Netflix, I usually blow through the 1TB cap in the first 10 or so days.
  - Clamchop 3y ago
    I have an unlimited plan (Cox) and they've threatened to terminate me before during a period of high usage. (I was initializing an off-site backup.) Vague language about QoS for other customers in their terms of service, so at their discretion. My dislike of ISPs is evergreen.
trizoza 3y ago
Funnily, your post just made me feel a bit uncomfortable and I had the urge check my GCP spending immediately :D
jcarrano 3y ago
More than just one line: - Why check every 5 minutes? It's not like anyone will die if you wait a day to upgrade. - Why download it without user confirmation. This is the worst one. - The timer should not be stopped, or the user is forced to install the last downloaded update before they can install the latest. It seems unwarranted to go to so much trouble for a screen recorder app, when just checking for an update at startup and prompting the user to go to the website would have been enough.
Alifatisk 3y ago
> The app checks for the update every 5 minutes or when the user activates the app. Why isn't checking once a day / week enough?
lionkor 3y ago
> Write your auto-updater code very carefully. And... code review?
- Spivak 3y ago
  LGTM Relying on code review is a dangerous game because fine looking code can be riddled with bugs.
FloatArtifact 3y ago
I really appreciate the values of those like those dev who own up too their mistakes but also do their best to make it right with their end users.
- meltedcapacitor 3y ago
  they did not own to being antisocial dicks in the first place, with all the bad decisions that also apply when the thing works as intended...
philo23 3y ago
I'm glad I'm not the only one that thinks checking every 5 mins is a little bit excessive! Seeing as it's macOS only is there even any reason to roll your own update mechanism when you can use the Sparkle framework [1]? EDIT: after a bit of digging, it actually looks like it uses Squirrel instead [2] [1]: https://sparkle-project.org/ https://sparkle-project.org/ [2]: https://github.com/Squirrel/Squirrel.Mac https://github.com/Squirrel/Squirrel.Mac
AdrianB1 3y ago
$8k is a small punishment for what the application is doing. Not for the bug, but for the upgrade process.
virtualritz 3y ago
> The app checks for the update every 5 minutes or when the user activates the app. That is the actual bug.
dathinab 3y ago
Also: put limits / back-off strategies on everything
muglug 3y ago
I have fixed lines of code that cost many multiples of this in compute cost. At $previous_company, user-initiated account deletions were only soft deletes, and we were supposed to actually delete data 60 days after the request (in case people regretted their decision). Due to a bug in a SQL query we stopped deleting users (and their associated data) for over six months, racking up over half a million dollars in additional storage cost.
Cthulhu_ 3y ago
This is one reason why distribution via the official app store would be preferable, their platform, their responsibility. Granted, they will take their share so that kinda sucks.
cynicalsecurity 3y ago
Their "summary" is a disaster. They still have no idea what they are doing wrong.
pie6k 3y ago
https://screen.studio https://screen.studio founder here: Thanks for all the comments. Why 5 minutes? Well I ask this question myself as well. I think it was bad and risky idea. You create MVP and you make 20 decisions like this a day. I probably thought "why not" as it was stopping anyway and I released a few versions a day at times and wanted to get feedback and crash reports quickly. I did change it now to 3 hours. Why downloading it without user confirmation? Well - because this is how Electron auto updater works under the hood (checking for update = downloading it) and it cannot be customized without implementing a lot of the logic by myself. I'll soon modify it and do exactly that, but I was creating MVP, got traction and as solo dev I had to carefully pick things to focus on. Electron auto updater works and is battle tested so I picked it knowing this limitation and suboptimal flow. --- So I understand a lot of critique you're making and I agree with it. It's just I was making compromises, iterating quickly and making silly mistakes.
- preommr 3y ago
  > Why 5 minutes? Well I ask this question myself as well. I think it was bad and risky idea. Fair enough people make mistakes... > I did change it now to 3 hours. lol
  - pie6k 3y ago
    Shampoo
    - CyberDildonics 3y ago
      Is your response to your app still trying to update too often really just "shampoo" ?
      - pie6k 3y ago
        Not much more I have to say about crying over 500 bytes json fetch request and policing others to make it at most once a day instead of every 3 hours.
- 3y ago
ericpauley 3y ago
For comparison, this would cost $3.11 on Cloudflare R2. Data transfer fees on the traditional providers are abysmal, but also kind of a known evil at this point.
danmur 3y ago
250mb eh :). That deserves a bullet point
- __void 3y ago
  ...every now and then my brain reminds me that all of LOTR is 3.11mb, The Betrothed (by Manzoni) is 1.2mb and Inferno (by Dante) only 550 kb and then I wonder what the hell we are doing in this century...
supermatt 3y ago
Wow, 1TB of data per user. I hope they weren't on a restricted tariff or you may have also cost them a significant amount in overage fees! If you are going to prompt the user anyway, why not download AFTER they accept the install instead of before?
- 8organicbits 3y ago
  If the user is on a slow connection, the 250 MB update file could take hours to download. Doing the download first hides that slow process from tbe user.
  - deleted 3y ago
    [deleted]
  - supermatt 3y ago
    I suppose the intent there is to not keep the user waiting for those hours while the install takes place? Maybe auto-updating should work same as other apps - install automatically and use the new version on restart?
david422 3y ago
I had something similar happen to me just the other day. One of my websites was running slowly. I had run out of AWS CPU credits. Did some investigating and finally found the root cause. I had a background backup job that downloaded some resources and then zipped them up and stored them in s3. But I had changed servers, and the final step of the job on the new server was failing. So it just kept downloading resources, failing, and a short while later attempting again. I put in some new checks to notify me on failures.
luxcem 3y ago
This is bad in multiple ways, as other have said every a 5min check is ludicrous, but the article focuses on losses for the developer in a complete disregard for their users. Some people still have a limited data plan or pay according to usage (people using their mobile phone as a hotspot), in any case, downloading a 250M file without user consent is an abuse.
- cfiggers 3y ago
  Tell us you didn't read the article without telling us you didn't read the article
  - aldarisbm 3y ago
    still a valid point
  - deleted 3y ago
    [deleted]
  - nottorp 3y ago
    > in any case, downloading a 250M file without user consent is an abuse. I've read the article and saw no justification for that.
tomhallett 3y ago
Are there any saas tools which help track cloud/saas usage/charges? Or do most people build something homegrown with cloud monitors, pulling billing data from other saas apps, and making data “observability” bi dashboards. Ex: - we used to sync 1MM contacts per month to our crm, but this month it’s 2mm - I created an rds database for testing something, but forgot to turn it off (imaging a “library checkout” feature, where I get permission to create a resource for X time, then alerts if I don’t delete it)
latortuga 3y ago
My god the comments are roasting this guy, lighten up people. Like nobody has ever shipped a bug to production before? Thanks for sharing the retro OP.
deleted 3y ago
[deleted]
8organicbits 3y ago
When working in the cloud and automating systems, variable pricing scares me. Some providers offer predictable pricing, but the more cloud/variable/auto-scale/serverless you go, the harder it is to predict. Sometimes it's better to fail than to get a huge bill. If you don't have a billing alarm setup, stop reading HN and add one now.
- ghaff 3y ago
  And alarms only go so far. I understand that for revenue-producing production workloads, stop spending money vs. burn everything down isn't always an obvious decision. But, for for many personal workload cases, I absolutely want to stop spending money above a certain level no questions asked.
  - deleted 3y ago
    [deleted]
basilboli 3y ago
I've experienced broken auto-update functionality throughout my career. The only way to have a great user experience is through the App Store, which allows us to take control of our updates. This new world of frequent app updates is just a mess, and in most cases, the updates are minor or just marketing spam.
basilboli 3y ago
Ah, the good ol' days when the only way to update was inserting a floppy disk into your computer! I'm sure one day we'll return to that model and be as safe as ever.
aboringusername 3y ago
One question is why Android is so "dumb" when it comes to this sort of thing. I can set a connection as "metered", whatever that means, but no way to set a limit of max throughput (speed/data) whilst on wifi. Android/Google treats WiFi as an unlimited resource, where the OS can consume as much data as possible (so that all the ad tracker networks don't miss out). Android needs to: 1: Let you disable networking on any app (inc. system) on AOSP 2: Warn you (you know, the 'smart' element of smartphone) when an app consumes an 'unusually' high amount of data - if one app has consumed 200%+ more than any other app maybe, just maybe there should be a notification as such. This is a sad but important reminder 'you' are treated as a disposable resource on Android. You do not get afforded reasonable control measures over your own device, even on GrapheneOS (which does let you disable networking per app, but not control WiFi data consumption). Really, similar to "notification channels" there needs to be "data channels" that can be toggled on/off ("tracking", "updates", "resources") etc. And the user should be able to block at the DNS/app level (tracker detection, similar to Firefox would be nice too)
samwillis 3y ago
I'm incredibly disappointed in some of the comments here, we are all hackers and makers. We will all have made mistakes at some point, I certainly have - dropped a production DB, taken down an e-commerce site in the middle of Black Friday with a bad push, accidentally DDOSed a massive SAAS accounting package I use and crashed it globally for a couple of hours... I posted this because it's a good war story, we can all learn from it, and enjoying it at the same time. Laugh along with the OP. Point is, don't jump to conclusions that someone has made a choice for ulterior purposes, given them the benefit of the doubt. Don't tare someone down because you think they made a silly mistake, we are better than that! Screen Studio is a solo founder, making an awesome product and releasing updates to it almost daily! Adam has fixed the bug and changed the interval to 3 hours - Chrome for context has an interval or 5 hours for update checking: https://support.google.com/chrome/a/answer/6350036?hl=en#zippy=%2Chow-often-does-google-update-check-for-updates https://support.google.com/chrome/a/answer/6350036?hl=en#zip... > Google Update checks for the latest update approximately every 5 hours
- morning-coffee 3y ago
  Some of us are software engineers (not just hackers and makers) and feel compelled to point out that if you're writing code which points firehoses of internet traffic at end users of your app, you should be aware of techniques to ensure that code is functioning correctly. I would have loved to read mention of "testing" or "regression tests" in the author's summary of learnings. "Be careful" and "monitor bandwidth" don't really resonate with me as a sign the author gets it yet.
- justsomehnguy 3y ago
  > Adam has fixed the bug and changed the interval to 3 hours And should had set it to at least a week. It's not some 911/mission/missile critical program, there is absolutely no reason for checking that often. > I'm incredibly disappointed in some of the comments here Because you are the one of the people who thinks what phoning home every 5 minutes/3 hours is perfectly fine. > I posted this because it's a good war story Oh yes, it's a goooood war story how 'move fast for no fucking reason' gave trouble for everyone.
- nmeofthestate 3y ago
dan-rocks 3y ago
[dead]
dhdhhdd 3y ago
Somebody will not be happy with their Comcast over bandwidth charge. Or if they were tethering from cell phone...
johnthescott 3y ago
memory leak due to custom font added to heads up display written in Xwindows by major defense contractor. diffed code against mit base, added font free call. $35K paid after two months testing, all for < day's work.

115 points by samwillis 3y ago

Would love to know how they resolved the problem on a technical level. After all, the apps running in the background would still try to download updates every five minutes and disabling/blocking those update requests on the server might cut those installations off from ever auto-updating again.

yifanl 3y ago
In another thread here, the founder states that their solution is to reduce the rate of updates from every 5 minutes to every 3 hours.
- Clamchop 3y ago
  That and they fixed the bug, importantly. Additional feedback on HN about checking even less often, getting user consent, and writing regression tests but commentators are being dramatic and insulting about it.

robin_reala 3y ago

One of our users, who lived in a house, had their internet provider cancel their contract due to enormous traffic generated during a month. It was extremely problematic as there was no other internet provider available around. We decided to take responsibility and offer to cover all the costs related to this situation. This is how you deal with a situation like this that you’ve caused. Accidents happen, what matters is how you clean up after.

ghaff 3y ago
There is, although if you have one wired broadband provider for an address, and that provider isn't willing to overlook a one-time issue, there may not be a lot of options that money can resolve. (The--sometimes--availability of Starlink helps these days but a sole supplier cutting you off doesn't have a universal fix.)

tylervigen 3y ago

Is there a reason an app needs to check for updates every 5 minutes vs, say, once a day? That would have made this situation 288 times better, and reduced the processing tax on people’s devices.

sudhirj 3y ago
It was a mistake, a debugging line that made it to prod.
- ndespres 3y ago
  The article says that the mistake was downloading the update repeatedly. The 5-minute check for updates was intentional.

Cenk 3y ago

Why not just use the Sparkle update framework like almost all indie Mac apps?

CaptainBern 3y ago

Why download updates at all without the user knowing? Is prompting before downloading such an awful user experience in comparison? Or, you know, prompt once per day and add a 'remind me in x days/minutes' option? The whole take-away of this article feels backwards. Instead of going 'hey maybe we should not just download stuff in the background without prompt', the author doubles down and actually proposes to add a way for the server to signal to the client that an update should be forced. I'm actually baffled by this entire thing.

nchudleigh 3y ago
The bug was the app kept downloading after the user agreed to download the new version.
- CaptainBern 3y ago
  The article did not mention that as far as I can tell. >The app checks for the update every 5 minutes or when the user activates the app. >Normally, when the app detected the update - it downloaded it and stopped the 5 minutes interval until the user installed it and restarted it. To me this reads as them downloading the update unprompted (as they mention their issue with the previous updater was that it would prompt the users every 5 minutes, which would be solved by just not checking every 5 minutes). Either way, whatever the bug and its effects are, my gripe is that their conclusion is to add a way for them to remotely force-updates. As opposed to just notifying once or any of the other hundred ways to handle auto-updates that don't involve you remotely forcing an update. Edit: the founder has confirmed in the comments the app does download without user confirmation, so my original point still stands.
michaelt 3y ago
Well, if the user wants the update, downloading in advance is a much better experience, because you only get interrupted once, for a restart that only takes a second or two. That's in contrast to being interrupted twice (once to download, then again to restart) or a single long interruption (shutting down, then downloading, then restarting). Of course, it's a different matter if the user doesn't want the update. But we developers don't like to think about that :)

pachico 3y ago

Maybe this will encourage developers to make smaller applications too.

torstenvl 3y ago

> Screen Studio is a desktop app. It means we need some auto-updater No, it doesn't. People used desktop apps perfectly fine before the always-on phone-home era of nearly universal broadband. What's more troubling though is that this is somehow a story about monitoring cloud costs, and not a story about why your update interval was every five minutes instead of daily; or why it would auto-download instead of politely (no popups!) letting the user know there was an update; or why you failed to reconsider the fundamentals of your design in the face of the first round of negative feedback. Your users told you they felt harassed and interrupted by your prompts. That's why you were making this change in the first place. That should have told you something.

Spivak 3y ago
User expectations have changed in this regard. If you don't auto update users will never update. Updating your computer or apps isn't a thing anyone thinks about anymore. Don't do it like the OP clearly, and it's really sad that desktops can't provide an experience like mobile where updates are platform managed but you still gotta do it.
- computerfriend 3y ago
  I use a package manager. It's normal, common even. My "user expectation" is that applications don't do this.
- ghaff 3y ago
  Then they don't update. People rarely updating seems a lesser issue than forcing updates on everyone--especially at the application layer.
- dathinab 3y ago
  yesn't there needs to be some auto updater but you don't necessary need to have some or in other words you normally want to integrate into existing update services, e.g. then package manager on linux. And if it then doesn't get updated e.g. because people don't update their system at all then that's their choice and problem IMHO. Just flag a warning/error once the API to your service changed too much or similar.

ndespres 3y ago

>> The app checks for the update every 5 minutes or when the user activates the app. This is why I use Little Snitch, an application firewall that blocks requests like this and gives me some control over which applications are phoning home all the time. There’s no reason a desktop application should check for updates every 5 minutes. Once a month ought to do it without a compelling reason for more. But none of the lessons learned in the article show any attempt to reevaluate the update check frequency.

nchudleigh 3y ago
Eh I'd say daily. Especially in a case like screen studio where it's under active development.
- jacquesm 3y ago
  Why? Updates may or may not be to the benefit of the user, who should have a clear say in the time and place when their stuff is updated, nothing is more frustrating to have a job to do and then to sit there waiting for some silly update process to complete only to find out that (1) it doesn't matter to you anyway or (2) that they've borked the update or (3) that critical functionality you relied on has been removed or (4) that the marketing department thought this was a great time to try to upsell you on something. Mandatory updates should only happen for critical security issues and even those don't need to be fixed on a five minute interval, once a day at most for a check and less frequently is probably fine as well unless the app is absolutely mission critical and people will die. And even then you should probably wonder if your QA is so bad that you need to be able to intervene on a moments notice, better to design your process in such a way that you can alert the user rather than to forcibly upgrade their system, especially if there is a chance that you'll get it wrong (again). An upgrade is always a burden on the users, so you really don't want to do this too often and even if you are capable of releasing multiple updates per week it would be ridiculous to expect your users to follow that frequency.
  - pluijzer 3y ago
    In the past if there was an update for software I use I was excited and was curious about all the improvements it would bring. Now when I see some app has an update I have a feeling of dread as I am sure it will be slower, the UI more clumsy and the power features I like will surely be removed.

PurpleRamen 3y ago

This reads like there is no version-check for updates? Because why would it otherwise download the same file over and over again?

fredoralive 3y ago
It doesn't seem to be automatically installing the update, presumably as that would disrupt the user by restarting the app. I guess it's staging the update to run on exit or similar. So until the user actually quits the app (or clicks update etc.), it's going to be on the old version, and in this endless redownload loop.

someweirdperson 3y ago

At that update frequency, at least the fix can be deployed quickly.

dj_mc_merlin 3y ago

Not a single mention of rate limiting? This isn't 2008, if you're offering an online service and do not have some form of rate limiting that will trigger when your throughput increases by 3 orders of magnitude, that's on you.

mschuster91 3y ago
If you're using plain AWS S3 you can't reasonably add a rate limit - probably if you put some Lambda Edge function or Cloudfront in front of it, but that's unnecessary cost and complexity.
8organicbits 3y ago
I think the approach for S3 is to have the app call a rate limted API to get the download URL, possibly as a pre-signed URL, possibly with an HTTP redirect. That doesn't help if the client caches that URL and downloads it from S3 directly many times. Alternatively, you could run your own update server, but S3 is so convenient. There's also a concern that the update is designed to check every five minutes. So a rate limit may not actually get exceeded.
- dj_mc_merlin 3y ago
  Late reply, but > you could run your own update server, but S3 is so convenient. IMO if you're providing updates to your users you're distributing code automatically and potentially infecting their computers if anything goes wrong. You should take all proper precautions.

Ekaros 3y ago

>> Normally, when the app detected the update - it downloaded it and stopped the 5 minutes interval until the user installed it and restarted it. If I'm reading this correctly, the update mechanism stops if there is updated downloaded and waiting. So if user just doesn't install it, it will not get the versions after that? Seems like whole logic might have more issues, beyond the basic uselessness.

wslh 3y ago

The crypto world is full of single lines of code that are worths million of dollars. No pun intended. BTW, in our team we compiled a spreadsheet showing Git commits before and after security audit reports [1]. [1] https://docs.google.com/spreadsheets/d/1bqtQI05z-O6uclvYYXxHo09uDx5lclKByUEfLiZACrA/edit#gid=0 https://docs.google.com/spreadsheets/d/1bqtQI05z-O6uclvYYXxH...

ysleepy 3y ago

Why the hell does the app check every 5min for updates? Even once a day seems excessive. Thy cynic in me sees a pattern of disregard for other people's resources like with electron. Being a good software citizen is also realising that the product is not the main task running on a computer and it's not acceptable to make decisions about other peoples priorities. It feels like every tray daemon and widget assumes they are the main reason the computer is running and load a web view, 3 windows services and scheduled tasks. Admittedly the App in question is interactive, but the same principle holds. Categorical imperative hah: Construct in a way that you accept all tray icons to be constructed like.

pbhjpbhj 3y ago
It's more insidious than disregard, I expect. I assume it's a beacon masquerading as an update check - monitoring app usage, user locations, and such. Aside: If your users are in the same time zone you may not want to do checks/updates on startup per se in order to avoid putting all your update load into the same time slice.
- lightbendover 3y ago
  I’m not tracking. You can perform all that monitoring and more without pretending to be a software update.
- tough 3y ago
  > Aside: If your users are in the same time zone you may not want to do checks/updates on startup per se in order to avoid putting all your update load into the same time slice. This one bite my in the ass with a login used by IRL offices which all wanted to log in at 9:00 at the same time lol
vintagedave 3y ago
> The problem with the auto-updater we had was that it would prompt the user to update the app as soon as it became available. This resulted in a popup appearing while users were recording the screen, which obviously provided a bad experience as it interrupted the recording the user was making. The 5 minute bug was introduced while fixing this issue. But it was a bad one: I have so many apps "randomly" popping update notifications up all the time: Skype and Fantastical are the two worst, seemingly every day. The popups interrupt me when working, and _take keyboard focus_ meaning I lose the last half of a sentence, or sometimes even dismiss the dialog before I realise it's appeared. > the product is not the main task running on a computer and it's not acceptable to make decisions about other peoples priorities Yes! One thing I loved about early macOS (about OS X Tiger - Snow Leopard days) was that I never saw apps steal focus. On Windows at the time it happened all the time in various ways, but never on Mac. That included when apps started: if you were working in an app and started another but kept working in the first, the second started with the window in the background. It was lovely.

throwaway290 3y ago

Does it consider how much it cost users on mobile hotspots?

ghaff 3y ago
There are all sorts of circumstances, see also international travel, when unintentional downloads can get really expensive and it's not reasonable to expect people to police how and when they use data at a micro level. I'd much rather pester people to give permission for things than just download in the background because that's "probably what they want."

pachico 3y ago

I think this post backfired at the writer. While trying to provide transparency over a problem (we all screw up) it uncovered what I think it's a horrible idea. The issue is not the bug anymore but the update process. These comments in HN might provide much more insights than the bug itself.

Damogran6 3y ago

I hate this aspect of modern computing…Time was, you could look over at your firewall, or notice that the local network was running like crap…but Google (and for this post, Comcast/Xfinity) have GOBS of capacity. It’s easy to manage costs when your bandwidth is constrained to a relatively low rate… That super high speed 1Gb fiber connection? The one that STILL has the 1Tb/Mo cap? You can blow through it in an afternoon with one poorly constrained app (e.g. Torrenting) Same here. Google has _company ending_ amounts of bandwidth in their datacenters and you can lose everything without knowing it. (See also medical billing)

rkangel 3y ago
I think there is a problem here, but I don't think think that's the right problem. Not having these arbitrary technical limits is a good thing - for instance their service didn't degrade due to an unrelated problem. Regulating cost indirectly via some other metric is a very oblique way of solving the problem The right solution would be throttling of spending - easy way to set proper limits, good notifications etc. It's possible to set up all these limits but nowhere near as easy as it should be. Also, how easy would it be for Google to send you an email saying "your spend looks a bit different than normal due to this traffic here"!
amelius 3y ago
Bandwidth shaping/traffic control in Linux is not very mature. But yes, Google and others should implement a cap.
_fat_santa 3y ago
From personal experience, paying Xfinity an extra $30 or so per month to get the cap taken off was one of the first "upgrades" I made to my internet. Between me pulling node_modules for multiple projects and watching Netflix, I usually blow through the 1TB cap in the first 10 or so days.
- Clamchop 3y ago
  I have an unlimited plan (Cox) and they've threatened to terminate me before during a period of high usage. (I was initializing an off-site backup.) Vague language about QoS for other customers in their terms of service, so at their discretion. My dislike of ISPs is evergreen.

trizoza 3y ago

Funnily, your post just made me feel a bit uncomfortable and I had the urge check my GCP spending immediately :D

jcarrano 3y ago

More than just one line: - Why check every 5 minutes? It's not like anyone will die if you wait a day to upgrade. - Why download it without user confirmation. This is the worst one. - The timer should not be stopped, or the user is forced to install the last downloaded update before they can install the latest. It seems unwarranted to go to so much trouble for a screen recorder app, when just checking for an update at startup and prompting the user to go to the website would have been enough.

Alifatisk 3y ago

> The app checks for the update every 5 minutes or when the user activates the app. Why isn't checking once a day / week enough?

lionkor 3y ago

> Write your auto-updater code very carefully. And... code review?

Spivak 3y ago
LGTM Relying on code review is a dangerous game because fine looking code can be riddled with bugs.

FloatArtifact 3y ago

I really appreciate the values of those like those dev who own up too their mistakes but also do their best to make it right with their end users.

meltedcapacitor 3y ago
they did not own to being antisocial dicks in the first place, with all the bad decisions that also apply when the thing works as intended...

philo23 3y ago

I'm glad I'm not the only one that thinks checking every 5 mins is a little bit excessive! Seeing as it's macOS only is there even any reason to roll your own update mechanism when you can use the Sparkle framework [1]? EDIT: after a bit of digging, it actually looks like it uses Squirrel instead [2] [1]: https://sparkle-project.org/ https://sparkle-project.org/ [2]: https://github.com/Squirrel/Squirrel.Mac https://github.com/Squirrel/Squirrel.Mac

AdrianB1 3y ago

$8k is a small punishment for what the application is doing. Not for the bug, but for the upgrade process.

virtualritz 3y ago

> The app checks for the update every 5 minutes or when the user activates the app. That is the actual bug.

dathinab 3y ago

Also: put limits / back-off strategies on everything

muglug 3y ago

I have fixed lines of code that cost many multiples of this in compute cost. At $previous_company, user-initiated account deletions were only soft deletes, and we were supposed to actually delete data 60 days after the request (in case people regretted their decision). Due to a bug in a SQL query we stopped deleting users (and their associated data) for over six months, racking up over half a million dollars in additional storage cost.

Cthulhu_ 3y ago

This is one reason why distribution via the official app store would be preferable, their platform, their responsibility. Granted, they will take their share so that kinda sucks.

cynicalsecurity 3y ago

Their "summary" is a disaster. They still have no idea what they are doing wrong.

pie6k 3y ago

https://screen.studio https://screen.studio founder here: Thanks for all the comments. Why 5 minutes? Well I ask this question myself as well. I think it was bad and risky idea. You create MVP and you make 20 decisions like this a day. I probably thought "why not" as it was stopping anyway and I released a few versions a day at times and wanted to get feedback and crash reports quickly. I did change it now to 3 hours. Why downloading it without user confirmation? Well - because this is how Electron auto updater works under the hood (checking for update = downloading it) and it cannot be customized without implementing a lot of the logic by myself. I'll soon modify it and do exactly that, but I was creating MVP, got traction and as solo dev I had to carefully pick things to focus on. Electron auto updater works and is battle tested so I picked it knowing this limitation and suboptimal flow. --- So I understand a lot of critique you're making and I agree with it. It's just I was making compromises, iterating quickly and making silly mistakes.

preommr 3y ago
> Why 5 minutes? Well I ask this question myself as well. I think it was bad and risky idea. Fair enough people make mistakes... > I did change it now to 3 hours. lol
- pie6k 3y ago
  Shampoo
  - CyberDildonics 3y ago
    Is your response to your app still trying to update too often really just "shampoo" ?
    - pie6k 3y ago
      Not much more I have to say about crying over 500 bytes json fetch request and policing others to make it at most once a day instead of every 3 hours.
3y ago

ericpauley 3y ago

For comparison, this would cost $3.11 on Cloudflare R2. Data transfer fees on the traditional providers are abysmal, but also kind of a known evil at this point.

danmur 3y ago

250mb eh :). That deserves a bullet point

__void 3y ago
...every now and then my brain reminds me that all of LOTR is 3.11mb, The Betrothed (by Manzoni) is 1.2mb and Inferno (by Dante) only 550 kb and then I wonder what the hell we are doing in this century...

supermatt 3y ago

Wow, 1TB of data per user. I hope they weren't on a restricted tariff or you may have also cost them a significant amount in overage fees! If you are going to prompt the user anyway, why not download AFTER they accept the install instead of before?

8organicbits 3y ago
If the user is on a slow connection, the 250 MB update file could take hours to download. Doing the download first hides that slow process from tbe user.
- deleted 3y ago
  [deleted]
- supermatt 3y ago
  I suppose the intent there is to not keep the user waiting for those hours while the install takes place? Maybe auto-updating should work same as other apps - install automatically and use the new version on restart?

david422 3y ago

I had something similar happen to me just the other day. One of my websites was running slowly. I had run out of AWS CPU credits. Did some investigating and finally found the root cause. I had a background backup job that downloaded some resources and then zipped them up and stored them in s3. But I had changed servers, and the final step of the job on the new server was failing. So it just kept downloading resources, failing, and a short while later attempting again. I put in some new checks to notify me on failures.

luxcem 3y ago

This is bad in multiple ways, as other have said every a 5min check is ludicrous, but the article focuses on losses for the developer in a complete disregard for their users. Some people still have a limited data plan or pay according to usage (people using their mobile phone as a hotspot), in any case, downloading a 250M file without user consent is an abuse.

cfiggers 3y ago
Tell us you didn't read the article without telling us you didn't read the article
- aldarisbm 3y ago
  still a valid point
- deleted 3y ago
  [deleted]
- nottorp 3y ago
  > in any case, downloading a 250M file without user consent is an abuse. I've read the article and saw no justification for that.

tomhallett 3y ago

Are there any saas tools which help track cloud/saas usage/charges? Or do most people build something homegrown with cloud monitors, pulling billing data from other saas apps, and making data “observability” bi dashboards. Ex: - we used to sync 1MM contacts per month to our crm, but this month it’s 2mm - I created an rds database for testing something, but forgot to turn it off (imaging a “library checkout” feature, where I get permission to create a resource for X time, then alerts if I don’t delete it)

latortuga 3y ago

My god the comments are roasting this guy, lighten up people. Like nobody has ever shipped a bug to production before? Thanks for sharing the retro OP.

deleted 3y ago

[deleted]

8organicbits 3y ago

When working in the cloud and automating systems, variable pricing scares me. Some providers offer predictable pricing, but the more cloud/variable/auto-scale/serverless you go, the harder it is to predict. Sometimes it's better to fail than to get a huge bill. If you don't have a billing alarm setup, stop reading HN and add one now.

ghaff 3y ago
And alarms only go so far. I understand that for revenue-producing production workloads, stop spending money vs. burn everything down isn't always an obvious decision. But, for for many personal workload cases, I absolutely want to stop spending money above a certain level no questions asked.
- deleted 3y ago
  [deleted]

basilboli 3y ago

I've experienced broken auto-update functionality throughout my career. The only way to have a great user experience is through the App Store, which allows us to take control of our updates. This new world of frequent app updates is just a mess, and in most cases, the updates are minor or just marketing spam.

basilboli 3y ago

Ah, the good ol' days when the only way to update was inserting a floppy disk into your computer! I'm sure one day we'll return to that model and be as safe as ever.

aboringusername 3y ago

One question is why Android is so "dumb" when it comes to this sort of thing. I can set a connection as "metered", whatever that means, but no way to set a limit of max throughput (speed/data) whilst on wifi. Android/Google treats WiFi as an unlimited resource, where the OS can consume as much data as possible (so that all the ad tracker networks don't miss out). Android needs to: 1: Let you disable networking on any app (inc. system) on AOSP 2: Warn you (you know, the 'smart' element of smartphone) when an app consumes an 'unusually' high amount of data - if one app has consumed 200%+ more than any other app maybe, just maybe there should be a notification as such. This is a sad but important reminder 'you' are treated as a disposable resource on Android. You do not get afforded reasonable control measures over your own device, even on GrapheneOS (which does let you disable networking per app, but not control WiFi data consumption). Really, similar to "notification channels" there needs to be "data channels" that can be toggled on/off ("tracking", "updates", "resources") etc. And the user should be able to block at the DNS/app level (tracker detection, similar to Firefox would be nice too)

samwillis 3y ago

I'm incredibly disappointed in some of the comments here, we are all hackers and makers. We will all have made mistakes at some point, I certainly have - dropped a production DB, taken down an e-commerce site in the middle of Black Friday with a bad push, accidentally DDOSed a massive SAAS accounting package I use and crashed it globally for a couple of hours... I posted this because it's a good war story, we can all learn from it, and enjoying it at the same time. Laugh along with the OP. Point is, don't jump to conclusions that someone has made a choice for ulterior purposes, given them the benefit of the doubt. Don't tare someone down because you think they made a silly mistake, we are better than that! Screen Studio is a solo founder, making an awesome product and releasing updates to it almost daily! Adam has fixed the bug and changed the interval to 3 hours - Chrome for context has an interval or 5 hours for update checking: https://support.google.com/chrome/a/answer/6350036?hl=en#zippy=%2Chow-often-does-google-update-check-for-updates https://support.google.com/chrome/a/answer/6350036?hl=en#zip... > Google Update checks for the latest update approximately every 5 hours

morning-coffee 3y ago
Some of us are software engineers (not just hackers and makers) and feel compelled to point out that if you're writing code which points firehoses of internet traffic at end users of your app, you should be aware of techniques to ensure that code is functioning correctly. I would have loved to read mention of "testing" or "regression tests" in the author's summary of learnings. "Be careful" and "monitor bandwidth" don't really resonate with me as a sign the author gets it yet.
justsomehnguy 3y ago
> Adam has fixed the bug and changed the interval to 3 hours And should had set it to at least a week. It's not some 911/mission/missile critical program, there is absolutely no reason for checking that often. > I'm incredibly disappointed in some of the comments here Because you are the one of the people who thinks what phoning home every 5 minutes/3 hours is perfectly fine. > I posted this because it's a good war story Oh yes, it's a goooood war story how 'move fast for no fucking reason' gave trouble for everyone.
nmeofthestate 3y ago

dan-rocks 3y ago

[dead]

dhdhhdd 3y ago

Somebody will not be happy with their Comcast over bandwidth charge. Or if they were tethering from cell phone...

johnthescott 3y ago

memory leak due to custom font added to heads up display written in Xwindows by major defense contractor. diffed code against mit base, added font free call. $35K paid after two months testing, all for < day's work.