9 ms·
Do you debug your website? A small unimportant bug can have huge consequences.
- henning 18y agoEmbarrassing story from someone I know: if you write your own billing application, make sure you round numbers correctly, or you could cost yourself hundreds of dollars a month in under-billing clients.
- swombat 18y agoI'd say that's a nice problem to have.. if you're under-rounding by one cent enough times to cost yourself hundreds of dollars, that would mean you're making at least tens of thousands of dollars of sales...
- henning 18y agoIn the particular manifestation of this bug, he was under-charging by much more than one cent. It was thousands of dollars a month, not tens of thousands.
- nebula 18y agoI noticed a couple of utility providers making the "rounding errors" in their favor. I sensed evil intent on their part. I never thought it would have been a genuine software bug.
- LogicHoleFlaw 18y agoI write billing software for a utility provider so.... yep, it's a bug. Usually that sort of thing manifests when data moves between systems with slightly different rounding rules. There's a ton of integration points to third-party systems and they don't always agree about how these things work. Fractional things like taxes, prorating, and earned vs. deferred incomes make precision errors very easy. We've got sanity checks in the code to make sure nothing crazy is going on when we get results from third-parties but so long as any errors are within certain margins the accountants using the software are happy. Incidentally, accountants are the best users ever for bug reports. Their eye for detail and the multitude of reconciliation processes they use to analyze the results of your processing is fantastic for finding subtle bugs.
- eru 18y agoEverybody should have an accountant. They had to run somewhat buggy software (tax laws, bilancing laws etc) on unreliable hardware (humans) since a long time ago. (It's constrained based programming.)
- bemmu 18y agoThis reminds me how much I wish it would be possible to attach notes on Google Analytics graphs so I can look at them later and know what I did.
- patio11 18y agoI know the feeling. That's actually a major reason why I keep the blog up -- it is probably my best single record of what the heck I was thinking when I implemented something. I find explaining things to other people leaves better hints than my cryptic "Of course I'll know what I meant in a year! After all, I wrote this comment!" documentation.
- jrnkntl 18y agoIt looks like all the 'my fail story'-blogpostings are quite elaborate for what they're trying to say. With this story the same, it reads like a breeze but all he says is: "Hey, I didn't delete the cache everyday because of a fault I made in the crontab. That costed me pagerank and visitors." Doesn't mean that I don't like reading those stories =)
- patio11 18y agoI wish my blog would get picked up when it is NOT making me look like an idiot. sigh Edited to add: So I was trying for a less pithy comment when the phone rang from overseas. Sorry about that. Anyhow. Yep, this story really did happen to me. The good news: a one line tweak to a little file I didn't touch in something like six months probably added about 50% to my sales. The bad news: in those six months, this probably had actual economic costs of several thousand dollars. Or, to put this in relative terms, the worldwide economy just freaking collapsed and I still managed to do more damage to my net worth with a freaking typo than the total massacre in my IRA.
- swombat 18y agoI don't think this makes you look like an idiot. I submitted this article from your blog because it is insightful and shows how apparently small, unimportant "display errors" can have very real consequences in terms of traffic and revenue.
- boredguy8 18y agoI thought that was you :) Just from reading the blog. /wave
- cmars232 18y agoGetting cron right can be trickier than it looks. Even if you test your script with the right uid, are you sure cron is running with the same environment variables as the shell? I've been bitten several times by cron skipping my shell profile. I just wish my cron scripts had the same earning potential :)
- mseebach 18y agoLesson 0: Don't do premature optimization.
- tdavis 18y agoLesson N: Use a system that doesn't require you to manually remove cache files. Seriously.
- patio11 18y agoSo being that this is Rails there are a couple of ways I could have done this (assuming, again, that I cached): 1) Cache the page in memcached, which would give me access to an easy time-based expiry syntax. 2) Page cache. Instead of cleaning the cache with a one-line cron script, write a 10 line cache sweeper. Then, install a plugin to periodically execute the sweeper... or execute it with a cron script. 3) Page cache. Delete with a cronjob. Why I didn't do #1: Memcached is overkill for my needs, and keeping another process running on the VPS just gobbles its 256MB of RAM even faster, while giving me another potential security/uptime headache. Why I didn't do #2: Doesn't eliminate the dependency on cron, it just moves the actual delete operation from a single line in the crontab to more complicated Sweeper logic in the app. That is logic I'd then have to actually maintain and whatnot. (And doing #2 wouldn't have helped me if I had executed the Sweeper job as the same user I executed rm -rf as, because it would similarly have been unable to remove the file.) I might be careless but I don't make choices totally at random. ;)
- henning 18y ago"keeping another process running on the VPS just gobbles its 256MB of RAM even faster" Why not upgrade? On Slicehost, going from 256 to 512 MB is only $20 extra. If you're running a business on it, I think that's a pretty reasonable expense, especially if it can save you even a little time (within reason).
- tdavis 18y agoI never said you made it at random, just that even if it were the best option it was still a poor one. From the options you laid out, it appears none of them were particularly suited to your situation. If it were me, upon realizing that, I would have just waited to see if caching were really necessary before instituting a cron hack.
- deleted 18y ago[deleted]
- pchristensen 18y ago"Never send a commited Windows programmer to do a Unix sysadmin’s job." Classic! (says the Windows programmer learning Unix commands and tools)
- nebula 18y agoA site that is designed to serve a fresh card everyday, keeps serving the same card. How is that unimportant? small and trivial technically, yes; but unimportant?
- patio11 18y agoLong story short: I thought "a new card every day!" was an important hook when designing the site. User behavior indicates otherwise -- most don't want novelty, they either want a) the same card everyone else wants (because it is seasonal) or b) a particular card which they Googled for. Only about 8% of my users even see the "featured card of the day" page. So yeah, on the face of it, this particular feature breaking is unimportant to most of my users. However, Googlebot is MUCH more impressed with novelty than 92% of my users. Given Googlebot's outsized impact on my business, it is sort of an important stakeholder :)
- josefresco 18y agoOnly a real problem because of this: "one of the deployment steps had the side effect of nuking the cache directory." If that didn't happen he would have found the bug much earlier.
- gojomo 18y agoLittle tweaks that speed page loading can also result in surprising increases in traffic -- both visitors who stay longer and improved search-referrals. I think fast-loading pages get a ranking boost from Google, too. (They're very explicit that this is the case for AdWords landing pages.)
- jyothi 18y agoApart from being alert, cautious and the stress on testing, the takeaway for me is "this bug made a live, growing website look largely dead to the world, and Google accordingly sent most of their searchers to get their bingo cards elsewhere." Most web businesses need to understand how google search works than their own businesses.
- handelaar 18y agoThis is the second time in a row that I've been unable to read a page on my laptop because you insist on posting quotes like that. Cut that shit out, please.
- DLWormwood 18y agoHear, hear! Whatever Hacker News is using needs to force wrap pre-formatted text! I feel like I'm reading Slashdot in the late 90's because of it...
- ars 18y agoI hate that too. The solution for me is to install the 'Nuke Anything Enhanced' addon, then remove the offending lines.
- natrius 18y agoBetter solution: Install the Stylish extension and add a white-space: pre-wrap rule for this site.
- soundsop 18y agoVoting up this feature request might help: http://news.ycombinator.com/item?id=280852 http://news.ycombinator.com/item?id=280852
- jyothi 18y agoI don't see why I who commented on a valuable takeaway from the post should be downmodded for this and your comment is upmodded. Yours is the one irrelevant to the post. PG, please add a official Don'ts in formatting. It is nothing but natural to quote elsewhere on the web.
- sutro 18y agoAlthough the bug fix was only one line, it apparently necessitated a blog post that was hundreds of lines long. Perhaps the author should ask Gavin Newsom to produce a 7-hour series of videos about the bug fix?
- eru 18y agoIt's not the bug. It's the money.
- deleted 18y ago[deleted]
- lallysingh 18y ago.. reading this after spending 2 days in emerg system fixing. Only way this worked at all: 1. Unit tests 2. (Heavy) Load testing 3. Version control, and knowing how to use it effectively. What I wanted: 1. Observability of what's going on in the system If you have to attach a debugger or profiler, just to get a stack trace or know the pressure on the GC, you're not in a good place.