23 ms·
On the Use of a Life
- known 6y agoPageRank was an Academic project that got evolved into Google Corporation; https://en.wikipedia.org/wiki/Google https://en.wikipedia.org/wiki/Google "You are a product of your environment" --Clement Stone
- phonebucket 6y agoIf Colin knew that he would have proved Birch-Swinnerton-Dyer as an academic, would he have regrets about leaving academia? This is clearly an academic (no pun intended) question, but I say it to illustrate a point: it's hard to know if you made the right choices in life, since you don't have perfect information about the course of alternative paths. I have been in academia and industry. I suffer from 'grass-is-greener' syndrome right now in industry, and am pining for more research. But I don't know if I would have felt the same way if I were stuck on a postdoc-to-postdoc treadmill.
- zuhayeer 6y agoA summary in quotes: "academia is a lousy place to do novel research" "It became clear, both from my own experiences and from advice I received, that if I wanted to succeed in academia I would need to churn out incremental research papers every year — at very least until I had tenure." "In many ways, starting my own company has given me the sort of freedom which academics aspire to"
- titzer 6y agoThis comment contributes nothing to the discussion except steering people away from reading the whole thing (a whopping 5 minute read), and the author actually wrote this in the second paragraph: > I considered replying in the thread, but I think it deserves an in-depth answer — and one which will be seen by more people than would notice a reply in the middle of a 100+ comment thread. I feel like you're going directly against the long-form answer the author intended to give, so I'm downvoting you.
- tigerlily 6y agoHow does what gp wrote steer people away from reading the whole thing?
- sushshshsh 6y agoI upvoted them because I appreciate their useful and pithy insight.
- papeda 6y agoIs it pithy? The summary given is "academia sucks, be an entrepreneur". IMO the actual point of the post is "I didn't pursue an academic career, here are the various things I did instead and why I think they are meaningful".
- sushshshsh 6y agoIt sounds like you have the free time to read long form writing of potentially good/bad value, however I do not and almost all of my engagement and value seeking from this site comes from the comments section. I typically read the shortest and most down voted comments as those are clear pointers to things that I need to learn/research further, such as "do I gain anything from a piece of writing longer than a tweet?"
- titzer 6y agoThe length of this exchange is probably more than the reading time of the OP. So if you haven't already, I would suggest go ahead and reading the whole thing. I feel like the TLDR'ing of someone else's writing is a negative reinforcement loop that needs to be challenged because it leads us all into doing the same. Personally I read comments sections to find things that are not in the OP and point out weak points or blind spots--you know, actual discussion.
- sushshshsh 6y agoWhy would I read the full source code of the HashMap implementation in OpenJDK when I could read the API in 5 minutes and obtain 97% of the information I need to use it? The answer of course, is if someone can provide me a two sentence reason that explains very clearly why that extra 3% of value is so crucial.
- gowld 6y agoRecall that the original academics were not primarily paid to do it, but were wealthy-enough people who devoted their spare time to it.(Theoretical scientists needed less money to do this then experimentalists and field scientists.)
- mathgladiator 6y agoThis is a big part of life: "wealthy-enough" I feel like we do academics wrong. We should encourage a large gap between undergrad and graduate school for two reasons. First, develop empathy for the realities of shipping software (or whatever the field is). Second, build some wealth to not chase wrong grants. I take Arnold Schwarzenegger as an inspiration on this front. He did hard work and had some good fortune due to great timing (real-estate in California) to build up wealth before becoming an actor, and this manifested in the freedom to go all in on Acting. The thing that I take as inspiring is that instead of chasing the money via that path or just chilling out, he went hard on a passion.
- timsally 6y agoThis is really excellent, thanks for being willing to share Colin. It's an impactful essay and the last paragraph in particular hits very hard. I'm a big fan of your work and the path you have taken!
- cortesoft 6y agoWas trying to find the comment that triggered this, and couldn't.... anyone have a link?
- anjanb 6y agohttps://news.ycombinator.com/item?id=24533381 https://news.ycombinator.com/item?id=24533381
- deleted 6y ago[deleted]
- Abhinav2000 6y ago[Deleted]
- observr9 6y ago> I work 9-6 every day ... then sleep for a couple of hours then stay up to 3am doing self study in math and programming. Could you elaborate? You take two naps at night in lieu of 7+ solid hours? Is this sufficient? Why?
- Abhinav2000 6y agoIts not ideal, so I dont recommend it to anyone. It just so happens I am super tired after work, so I need to give my brain a rest. But because I’m sleeping so early (like 7pm - 9pm), I inevitably wake up during the late evening. Then i try not to waste the remainder of the evening and end up staying up late for math and programming. Programming is easy to stay up, math is hard to do consistently. Then I sleep 3.30am or so to 8.30am, am cranky for the whole morning, but that is okay since I can do my work in autopilot most of the time. The end result is I am definitely getting a solid few hours each day for my studies, whilst maintaining a job.
- observr9 6y agoThanks. I was just curious because I don't seem to have the mental focus at night for such endeavors, but that is the only time I have for such things.
- Abhinav2000 6y agoI struggle a lot without that small nap. Would recommend trying that, alternatively maybe wake up early and spend the first hour on your chosen personal field before work. That way, you can be happy the rest of the day, knowing you accomplished the most important thing for you (your personal endeavours), and spend the day at work more relaxingly :)
- jdsully 6y ago
- jart 6y agoI think backup system ops are super important. If an invisible college exists for security research, then the dude who invented scrypt would be pretty close to its top. He's also too modest for not using an anchor tag to link to his business, and too generous for not naming the HN user who insulted him.
- cperciva 6y agoHe's also too modest for not using an anchor tag to link to his business I figure that my my readers probably know what Tarsnap is by this point; or if they don't, they know how to use Google. and too generous for not naming the HN user who insulted him. I saw no insult. I omitted the users's name because thought the question was interesting independent of the person who asked it.
- jart 6y agoRead a different way I suppose it's actually a compliment. It's not everyday I see people voicing frustration that someone chose not to solve P vs. NP.
- namenotrequired 6y agoI didn't have to squint so hard to read "obviously a genius" as a compliment
- 5hds34dszz 6y agoAlso, we shouldn't forget his hashing algorithm, scrypt, powers a popular cryptocurrency with a market cap over 3 Billion USD. Colin has added so much beyond 'backups' and should feel quite accomplished in his technical contributions to society. I am glad he has the commercial independence to follow his dreams, rather than being trapped in academia. Colin, you are an inspiration. You have achieved so much and I can't wait to see what comes next.
- deleted 6y ago[deleted]
- protomyth 6y agoColin has added so much beyond 'backups' Frankly, working backups are good enough because his service has allowed people to recover from disaster. Hope instead of misery is a fine way to spend one's life.
- faitswulff 6y agoIndeed, Colin mentions scrypt in TFA: > In 2009, having had many users ask for passphrase-protected Tarsnap key files, and having determined that the current state of the art of password based key derivation was sorely lacking, I invented scrypt — and in the process, opened up a whole new field of cryptography. Sure, I was doing this because it was something I could do to make Tarsnap more secure; but it would be a stretch to place this under the umbrella of "spending my time working on backups".
- thamer 6y ago> scrypt, powers a popular cryptocurrency with a market cap over 3 Billion USD For those who (like me) were wondering what cryptocurrency this was referring to: Litecoin has a ~$3B market cap and uses scrypt. Dogecoin also uses it but has a much smaller market cap, closer to $350M. scrypt is an interesting choice for cryptocurrencies because it does not only need a significant amount of compute resources – which can be scaled using GPUs to perform huge amounts of work in parallel – but also requires large amounts of memory which is much more difficult to scale to the same level.
- GuB-42 6y agoI don't believe in irreplaceable geniuses that will solve every problem if given the time. Let's take the most archetypal genius: Einstein. First, he was here at the right time. Relativity was begging to be discovered, at the time we had observations good enough to invalidate the previous theories and fit the new ones. The mathematical groundwork was here, all that was needed was a spark of genius, which Einstein had, but I'm quite many others had it too. But maybe the others chose a topic other than physics, had a different education, or were born too early or too late... Furthermore, Einstein essentially took 10 years to revolutionize physics. And then, not much. By not much, I mean he was still a top class physicist, but he couldn't restart the spark of genius, or he simply didn't have fuel for it. So what if after finishing his work on general relativity, he just died, or pursued a career in management, or something like that. Many people would have probably said he would have discovered all the secrets of the universe, when in reality, it wouldn't have made much of a dent in history. So don't worry and live your life as you want to, and don't expect too much of others. Things will get discovered eventually, and just because someone did great things in the past doesn't mean they will in the future, and vice versa.
- andreilys 6y ago“ just because someone did great things in the past doesn't mean they will in the future” I would say a Picasso or a Da Vinci is much more likely to create something meaningful given the track record at the height of their career, than some random person off the street. I think you’re underestimating the amount of effort it takes to master a given field, and the subsequent dividends that are paid when you achieve mastery.
- coldtea 6y agoNot necessarily. After some point, any new Picasso was just "more of the same". The reason his later works are still valuable is mostly because of our fetishism with the products of (already) famous artists + inflation by galleries and art merchants -- not because of any unique new insight/style/proposition/innovation/inherent value they had.
- 6y ago
- barkingcat 6y agoI'm a paying Tarsnap customer and I'm proud to be sponsoring you living your life the way you want to!
- btrettel 6y ago> Sure, I have customers to assist, servers to manage (not that they need much management), and business accounting to do; but professors equally have classes to teach, students to supervise, and committees to attend. When I made this sort of comparison myself, my day job didn't seem quite so bad. I've seen a few different studies into how academics spend their time. Seems that full professors spend about 9-10 hours per week doing research, and that's working around 55 to 60 hours week in total. (The numbers vary depending on which study you look at.) https://web.archive.org/web/20180509023409/https://thebluereview.org/faculty-time-allocation/ https://web.archive.org/web/20180509023409/https://thebluere... https://www.timeshighereducation.com/blog/if-you-love-research-academia-may-not-be-you https://www.timeshighereducation.com/blog/if-you-love-resear... So, if you work 40 hours per week and then do 10 hours per week of research, in some sense you're doing better than someone who did things "right". As someone who just completed a PhD but was unable to get a research position, I'm keeping this in mind. (And, more importantly, I'm doing vauable research that would be hard to do in academia.)
- ffggvv 6y agoright but most professors have graduate students working under them essentially doing work for them. they simply direct the research
- btrettel 6y agoThe unstated assumption in my comment was that I want to do the research myself. As I have a PhD, I am of course aware that advising students is a core part of the job of most professors.
- diego 6y agoThat is a really good article. The question that prompted is very telling: the asker fails to see nuance, and reduces OP's work to "backups." He/she might be more impressed if Colin were an author of tons of papers that collectively amounted to less than scrypt.
- renewiltord 6y agoI did not realize scrypt was invented to secure Tarsnap backups. Funny. I imagined the backup problem as being a matter of just careful engineering and the pbkdf problem as requiring novel insight. Solving the latter to improve the former is pretty cool. Anyway, this guy walks the walk. But how did he do on the Putnam?
- fractionalhare 6y agoI bet he didn't even crack the top ten.
- boulos 6y agoNot that it matters, but #6: https://kskedlaya.org/putnam-archive/putnam1999results.html https://kskedlaya.org/putnam-archive/putnam1999results.html
- schoen 6y agoThis whole rhetorical question/joke was a reference to "the comeback of all time" (also involving Colin Percival): https://news.ycombinator.com/item?id=35079 https://news.ycombinator.com/item?id=35079
- cperciva 6y ago#6 alphabetically, yes. They don't announce the ranks of the top 6 because they want to just say that all the Putnam Fellows won without drawing distinctions between them.
- deleted 6y ago[deleted]
- giantg2 6y agoI waste my life implementing the stupid decisions of the business and doing chores while my wife pursues hobbies. I have no agency over my own life at work or home - I hate it. I will make no meaningful contributions to society in my life.
- soared 6y ago99 percent of humans that have ever existed had less wordly impact than you, just based on your ability to comment and post on the internet. Comparatively, you’ve had an incredible impact and driven change in the world. (I think that’s accurate)
- hsjsdhf 6y agoHave you considered hiring help to do the chores at home? You might have more time to spare for hobbies.
- giantg2 6y agoI don't really have the money for that since I also pay all the bills while my wife spends her income on hobbies.
- xupybd 6y agoSounds like some marriage counseling would go a long way. Sounds like you are not happy with the way your wife is treating you. Something will snap if you don't sort that out.
- giantg2 6y agoI've thought about it, but that would just be another expensive bill that I have to pay. I'm not sure I can swing that now with my kid's medical issues and my wife's legal issue.
- wing-_-nuts 6y ago
- davisr 6y agoRight on, man. I'm following the same path--couldn't afford college, decided to start a business, then freelance, then became a yuppie, then started living in my car to write software, started selling it, and now I'm making money (minimum wage, at-least) writing the software I want to. I've so-far released an OS and desktop sync client for this weird tablet, and now I'm writing a libre handwriting recognizer. If you have faith in yourself, and follow your spirit, you can be successful and not accountable to anyone--complete intellectual and creative freedom. However, it does require some nerve to stand against the corporate machine.
- exikyut 6y ago(The weird tablet in question probably being the ReMarkable, considering other comments)
- hobo_mark 6y agoThis reminded me of something else he wrote years ago, on people's tendency to compare themselves to others: https://news.ycombinator.com/item?id=10524346 https://news.ycombinator.com/item?id=10524346
- cperciva 6y agoHeh, I completely forgot about that. Well, I'm now 39, and still stutter when I'm nervous, but I'm slightly less socially awkward. I'm still diabetic -- no cure for that yet, sadly -- and overweight, but I've been getting more exercise and am somewhat more fit. The biggest change since I wrote that almost 5 years ago, however, is that I found a wonderful woman who somehow manages to put up with me, and we got married in June. :-)
- hobo_mark 6y agoFantastic! Congratulations to you both!
- brian_herman 6y agoWow owning your own company and a wife... Damn I wish I could be you.
- conqrr 6y agoCongrats! Any advice on dating for socially awk men who only keep getting constant rejections?
- joncrane 6y agoWrite amazing software?
- enjeyw 6y agoThank you for writing that. The "everyone's life has difficulties" is true and often said, but it's such a platitude without someone providing a concrete example.
- 6y ago
- killjoywashere 6y agoI'm in violent agreement with Colin here. I have had far more opportunity to do seriously novel work on hard hitting problems in industry than I could ever accomplish in academia. And I have an academic appointment. Which, in pandemic land, pretty much means I have a .edu email address that I also have to attend to. On the industry side, I generate and move data at scale and my teams produce industrial strength models where the SoTA won't be attempted again for years, if ever. Household brands are changing their businesses based on we've done so far, and we're just getting started. The academic system is not where you go to have staggering impact in your lifetime. $500k or $1M a year isn't how innovation scales. Eisenhower didn't pay 100 labs $5M each to come with possible ballistic missile designs. He told Bennie Schriever: I need to deliver a payload to Moscow in 30 minutes, gave him $500M, and kept everyone out of Schriever's business. And, have a read of Rickover: http://ecolo.org/documents/documents_in_english/Rickover.pdf http://ecolo.org/documents/documents_in_english/Rickover.pdf Finally, I would point to the real meat of Colin's experience: if you take on a problem, one problem, like really take it on and try to develop it, you will find 100 PhDs worth of work in short order. Mind-breakingly hard problems abound when you actually try to solve hard problems in the world.
- closeparen 6y agoOk but finding teams like this in the real world is not simple. Half the roles billed this way are cynically lying. The other half think that because they’re struggling to stay afloat, they must be swimming in the ocean, but they’re actually in backyard kiddie pools and just not very sophisticated. Maybe I am overly optimistic about academia, but my understanding is that your advisor, funding agencies, journals, etc. know the layout of the field and make sure that what you’re doing is both interesting and novel - even to a fault.
- pathseeker 6y ago>Maybe I am overly optimistic about academia, but my understanding is that your advisor, funding agencies, journals, etc. know the layout of the field and make sure that what you’re doing is both interesting and novel - even to a fault. "interesting and novel" in academia currently translates to "incremental improvements on the status quo". The article directly talks about an example of not being able to do anything too novel because there was no journal to review it.
- neckardt 6y agoWhat makes TarSnap so good? I would assume that borg + Backblaze would be equivalently secure but much cheaper.
- mips_avatar 6y agoI am still blown away by what you did with BSDiff. Recently I was benchmarking it against architecturally optimized delta generators like Courgette. Amazingly BSDiff outperformed Courgette on my payload of AMD64 binaries. You have projects with years of dev time getting outclassed by simple BSDiff.
- hardwaresofton 6y agoSo I am not familiar with those tools at all, but I feel like it's worth noting this is the difference between academia and general practitioners/laymen. laymen and general practitioners may often be the first to explore an area that academia overlooked, but with the way academia is supposed to work, the amount of deep study and essentially throwing minds at the problem is supposed to produce completely separate classes of solutions (or the understanding that a better solution can't be made). General practice is often incremental improvement, research is supposed to produce paradigm shifts. This is why PhDs are worth something to all the FAANG employers -- they can get by just hiring smart people without worrying if they can write good code or not, because it's way easier to find someone who can write you the code to implement the algorithm created by someone who's steeped their brain in a problem for 5+ years.
- hlandau 6y ago@cperciva -- I was very surprised to see you not mention bsdiff here as one of your accomplishments. Incidentally, did you ever succeed in freeing the enhanced version of bsdiff from Oxford University? Seems a shame for it to still be going to waste, even now.
- gregw2 6y agoIt's the man in the arena who counts, per Teddy Roosevelt. I really enjoyed your work on PiHex and haven't used any of your later more notable works. Loved it! But should you have done that the rest of your life? Who should you serve? Was it great progress and good for you to move from PiHex to tarsnap or a step backwards? By what standard? Do we really have as much agency as the questioner implies? Do we choose problems or do they choose us? Is there only one true joy in the world? Is there only one good work to be done? I find the original question contains false premises that I personally found very ridiculous. You responded very maturely though, and it is a credit to you. Thanks Colin. You be you.
- ImaCake 6y agoI actually went back into academia to try and maximise my utility. I am not particularly interested in forging my own path in enterprise or as the head of a laboratory. Instead I try to be a helper within other people's teams. This seems to work pretty well for me so far. Some might think I am "not the master of my own ship". However, I would argue I have chosen this pathway into high utility subservience because it does achieve my goals.
- incompatible 6y agoA life spent on a backup system is unproductive? Ha ha, if only mine had been so productive. I'd say the majority of code that I've worked on was dubious in the first place, and decommissioned by now. Thankfully I'm a nihilist and don't have any expectation that my life will be meaningful. I happily work on things that most people would consider pointless and a waste of time.
- londons_explore 6y agoAnd after an HN comment touched some nerve, he took a few hours out of the life "working on backups" to explain to the rest of us why he thinks his lifes activities were impactful. (And it seems it was!) I'm not sure HN comments (intentional or not) should ever put someone else in a position of feeling they need to explain the impact of their life.
- wendelfov 6y agoOn the one hand I agree with you: no one should be expected to justify their life. (Who would have the wisdom to judge, anyway?) But on the other hand, I do think it's useful to ask whether the limited supply of genius in our society is being well-employed. Thinkers in the finance industry worry about this a lot, for example. Is it really a "good thing" that huge pools of quantitative talent are being drawn away from math and physics towards complex derivatives trading? Is shaving a few fractions of a fraction of a fraction of a second off the duration of a securities trade a good use of their time and talents? While I agree the original comment could have been more sensitive to the real person on the other side of the screen, I still think it asked a useful question and we're getting a good discussion out of it.
- cperciva 6y agoI didn't feel that I needed to explain things. Indeed, I could easily have ignored the comment and -- buried in the middle of a large thread as it was -- very few people would have noticed it.
- jonas21 6y agoIn case anyone's curious, here's the original comment thread Colin is referring to: https://news.ycombinator.com/item?id=24532469 https://news.ycombinator.com/item?id=24532469
- nsainsbury 6y agoI'm nowhere near as smart as Colin but I'll also share here a little personal story which relates to the topic. I was also a very high achieving student in high school and university and was similarly all set for a career in academia (also studying mathematics). In my final year however, I had a full-time position doing research with CSIRO, which is a leading research organisation in Australia. I did some interesting work there - applying neural nets for classifying micro-seismic events around mine sites, and won some awards for my research. If I had wanted to, I could have stayed on and continued down that path. But I didn't. What ultimately pushed me away was everyone I bumped in to in academia was so unhappy. There was constant bickering and frustration around getting funding (a common sentiment in the division I worked in was that you had pander to big mining/oil companies and propose research topics with clear financial gain for them). It was not a happy place to be, and at the end of my time there I jumped head first in to a software job instead. I later found time to still do mathematics on my own, and have written about that journey and shared it previously on HN: https://www.neilwithdata.com/mathematics-self-learner https://www.neilwithdata.com/mathematics-self-learner and have had some other little successes in my life that make me feel like I made the right choice: https://www.neilwithdata.com/how-i-built-bbsmart https://www.neilwithdata.com/how-i-built-bbsmart Tangentially, this I think is also why I'm more open to hearing ideas from organisations like Numenta, and seeing research done outside of academia by folks like Stephen Wolfram. I think increasingly much of the most novel interesting research will be done outside of traditional academia.
- holden_nelson 6y agoLoved your SQL course.
- wigl 6y ago+1 to pandering in mining/oil. The field really felt like a fraternity (House SPE), where ideas only moved as fast as the old guard felt like it. I also felt like a necessary skill as an academic in the field was vetting ideas early for clear financial gain. Drake meme: 1. NN's for classifying micro-seismic events a la Oklahoma 2. Coming up with something new to blast shale with Not sure if the vibe is different elsewhere, but my department was similarly unhappy. At the very least, I'm glad that the experience gave me a taste of shoddy research enough to hate it and develop my own preferences.
- draw_down 6y agoIncredible reply to what I’d describe as an undignified and borderline embarrassing question. In a word: owned.
- stephc_int13 6y agoI completely agree with the author. Academia is disapointing, I had the opportunity to discover it soon enough to avoid wasting too many years. If you're attracted by deep work on novel subjects, especially in Computer Science, academia is one of the most soul crushing place to be, unfortunately.
- PascLeRasc 6y agoCould you expand on what you mean by that? I felt the total opposite when I was working in academia, it was incredibly inspiring.
- Upvoter33 6y agoI dunno. Seems to me like people can be productive in lots of different environments. Can anyone argue that people like Dick Karp were unsuccessful because they were academics? Conversely, people like Page and Brin took what they developed in an academic setting and made something amazing in industry. Others like Steve Jobs/Wozniak built something purely in industry. All things can be good and possible; there is no one path that is the right one for all.
- mathgladiator 6y ago> people can be productive in lots of different environments I think this is why sharing stories is important so wandering people can find ideas for them achieving better results.
- Upvoter33 6y agoI do agree. But this wasn't just a "sharing my story", it also had a lot of opinion about the "better way" and a lot of negative thrown in the academic direction. But yeah, it's still an interesting read.
- srtjstjsj 6y agoPage and Brin didn't do anything academically. They took a standard technique and deployed it industrially "X but on the Internet" as we describe software patents today, and scaled it up. Academy is where they made the social connections (each other and VC) needed to launch a business, before they hired DEC people to build the tech.
- mtnGoat 6y ago"Is our society structured in a way which encourages people to make less than the greatest contribution they could?" Is this a serious question? I feel like anyone with a conscious can safely answer this with a resounding "Yes!".
- prerok 6y agoI don't think it's so straightforward but I guess it depends on what societal structures we are comparing. Considering historical structures I would argue that we currently live in a world much more open to various contributions. In the past you might have been locked down in a village doing fieldwork with no one even giving you the possibility of higher education. In some parts of the world this is still true. There is no way to tell how many hidden talents we lose right now to just toiling away with daily chores. In the western society I think things are much better. The problem we are facing is that if you want and have talent to contribute to a niche is "just" the question of who will finance you. This is indeed for many a showstopper but also many have found a way to push through. Then, after you have made the contribution, the problem is to convince others of the value. How many great ideas are lost to obscurity because some powers that be are stopping them? IMHO this is the bigger problem and perhaps this is what you are referring to? I would still say that the contribution had been made by then and, with time and a bit of optimism, those powers that be also erode and a contribution can come to life. Or, if it applies, launch a startup and disrupt the powers by yourself ;)
- possibleworlds 6y agoA bit off topic, but I am major fan of single developer apps and services. They generally have a focus and lack of bloat that makes them invaluable as tools to just get things done, and because of the small business footprint they don't need to endlessly scale into oblivion to satisfy shareholders / staff wages etc. Off the top of my head I am thinking of things I use regularly like pinboard, mailmate, Name Mangler, manager.io. Even builtwith.com is (or was for a long long time) one person!
- ed25519FUUU 6y agoI think about dedicating myself to one service and main technology stack, and that just sounds dandy. The time to fix all of the bugs, incrementally improve it. Be in charge of its feature list...
- swyx 6y agocan someone offer a TL;DR on who the author is? I feel like i lack context on this Tarsnap and Colin in general, and there is no about page on the site
- deleted 6y ago[deleted]
- wenc 6y agoHere you go: https://www.tarsnap.com/about.html https://www.tarsnap.com/about.html Quote: Dr. Colin Percival studied mathematics at Simon Fraser University, entering at age 13, before going on to a doctorate in Computing at Oxford University, where his thesis covered problems in string matching, data compression, and file synchronization. In January 2004, Colin became a FreeBSD committer and a member of the FreeBSD Security Team; he became the Security Officer for FreeBSD in August 2005, a position which he held until May 2012. Aside from his work as FreeBSD Security Officer, he is probably best known in the FreeBSD community for his work on FreeBSD Update and Portsnap. In addition to direct experience handling security issues via his experience as FreeBSD Security Officer, Dr. Percival has published several research papers in areas ranging from numerical analysis to cryptography, including the first published cryptographic attack against shared caches in Intel's "HyperThreaded" processors.
- swyx 6y agoah! didnt think to look at the tarsnap about page. thank you.
- dannykwells 6y agoIn parts of academia, there is much deeper collaboration and smaller gaps in goals, incentives, etc. Both sides publish papers everyone reads, and there is general applicability. People even, gasp, go back and forth throughout a career. Pure math though, sucks and is total blackhole for talent. Definitely agree there.
- personjerry 6y ago> Is our society structured in a way which encourages people to make less than the greatest contribution they could? While he answers his question, I want to point out that posing this question implicitly ties "contribution to society" with "use of a life". The utilitarianism makes sense to me, but I don't think everyone would agree that that is the right metric (for example, average or total happiness might be alternatives).
- rdiddly 6y agoYeah. Summing it up in just 3 words: "Greatest for whom?"
- cperciva 6y agoI believe I did address the philosophical question with the fact that it's my life and I can spend it however I like. I don't think that's inconsistent with wondering how one could make the greatest contribution to society -- if anything, it makes the choice all the more meaningful.
- personjerry 6y agoI have misunderstood. I re-read the passage and it seems that in that question you were not trying to derive insights about society, but saying because society is not structured this way, therefore you went and did that. The question was left open until the end, and because of the way the original question was phrased ("society" being the subject) I thought your thesis was about optimizing your life from society's view rather than from your own point of view.
- aidenn0 6y agoIf you need to connect two servers together semi-permanently, and haven't tried spiped, stop reading and do it now. It's the single most useful tool I have on the backend. Since Colin thinks kivaloo is more useful, but I've not heard of it, in going to look at that right now... [edit] I don't really have a use for a high performance, durable, KV store right now, so not convinced on kivaloo
- cperciva 6y agoHey, I didn't say that kivaloo was useful for everyone. ;-)
- zurfer 6y agoI could not find you on https://dbdb.io https://dbdb.io. Maybe you want to submit it at least there, so people will find it. How does kivaloo compare to other databases like Redis? Why should I use it? PS: fully understand that you might have just written it for fun, but if you want it to grow, this is the kind of information I am looking for as a database user. :)
- asymmetric 6y agoWhat’s the advantage over a WireGuard point-to-point connection?
- OJFord 6y agoThe most obvious thing to say is 'it's been there for the last decade'. I don't say that to preclude it from continuing to have an advantage over WireGuard, but I haven't used it (spiped) so couldn't comment.
- aidenn0 6y agoI've been using it long before wire-guard was generally available. I would guess that it's far less configuration to just "make port X on host Y look like port Z on localhost" with spiped, but I can't be sure. Things I like about it off the top of my head: - I will often start out with a single box, then say "well the DB should really be on a different machine" and just setup an spiped between them and I'm done. The actual application configuration changes not at all since it's still connecting to localhost. - My dev machines can be configured to spiped to the non-production database and I can run tests just like I would in production, since everything internal connects to ports on localhost; I literally can't accidentally mess up the production DB in a dev environment because the only place the hostname for the DB is is in the service file that starts spiped. [edit] Totally forgot the best feature: it can proxy between unix domain socket and TCP, which is something I think it had before SSH (it certainly had it before the ssh on my machine could do it). It also will re-lookup hostnames at a configurable interval, so new connections can be sent to a different machine just by configuring DNS.
- armatav 6y agoIs entrepreneurship a trap? No; right now, it's one of the only ways to avoid being trapped. The author convinced me here - he's not "wasting his time". He's owning it.
- known 6y agoaka https://en.wiktionary.org/wiki/YMMV https://en.wiktionary.org/wiki/YMMV
- superasn 6y agoThere is one more lesson here. Sorry if this is a bit OT and I don't know Colin but the lesson is "be good at one thing and crush it." It's good that he has dedicated a big part of time to just one thing: backups (and things related to backups). And that's why he is best at it and has billion dollar companies like Stripe giving him testimonials. Had he spent his time on 20 different things, chances you wouldn't think of tarsnap when you think backups. This I feel is also a very common trap of entrepreneurship (especially for solo people).
- mikkom 6y agoI think the whole post was trying to say that he has also done other things not related to backups so you certainly are OT
- iainctduncan 6y agoThis made me chuckle because I work in consulting for private equity funds doing diligence on companies they're going to buy, and to them, encrypted backups are one of the most important things out there! They go in the category of "risks you'd better goddamned be taking care of right or we'll just keep our millions". "Just backups"???? lol
- moneywoes 6y agoDo you mind sharing your pathway in that line of work and what the Dynamics are like in tech PE?
- null000 6y agoThe academic/business split is weird. In business, you're much more likely to be unknowingly treading on known ground, but the visibility of lessons learned is, in most cases, incredibly narrow. I've worked on compilers blindly implementing features I know other competitors have worked on but can't cheat off of, and I've worked on systems programming issues that probably pushed at the state of the art, but whose lessons wouldn't go outside my team in any case. In academia, though, there's a whole host of obstacles to doing anything useful and interesting that have nothing to do with "the problem at hand". So while you can be more confident in the relative novelty of what you're doing, as well as the broad applicability of said work (since the whole point is publishing) the scope of things you actually can work on is incredibly limited until late in your career.
- jmnicolas 6y ago"On the use of a life" from a website named "daemonology" I fully expected some satanic witchcraft here! Interesting read nonetheless.
- tp5 6y agoIt's "daemonology" not "demonology" Daemon=>A computer program that runs as a background process rather than being under the direct control of the interactive user https://en.m.wikipedia.org/wiki/Daemon_(computing) https://en.m.wikipedia.org/wiki/Daemon_(computing)
- mathgladiator 6y ago> "academia is a lousy place to do novel research" This hit me. I was focused on getting a Masters in Math, and I had a 4.0 GPA and 40 hours of credit with just a test or thesis preventing me from a credential. Instead, I dropped out to focus on building products with friends. Best decision ever. At least for me since I'm 38 and on the verge of retiring. > academic institutions systemically promote exactly the sort of short-term optimization The whole publish or perish ideology is teaching people the wrong f'n skills on imaginary problems, so I gave on reading on academic papers years ago because the amount of noise to signal was intolerable. Like Colin, I do independent research but in a different space. I'm currently writing a programming language for board games ( http://www.adama-lang.org/ http://www.adama-lang.org/ ) which I believe could turn into something crazy as I'm inventing a low-cost way of doing "serverless stateful micro services" or "a key to state machine service". I don't know how I could even get a grant for such work, but it interests me. And, I've reached a nice point in my life where success is measured by whether a task brings me joy rather than external signals. In a sense, when I decide to retire, I'll have the ultimate freedom of the idealized tenure that I built myself.
- kome 6y agoBut did you learn something useful in college or the experience could have been skipped entirely?
- mathgladiator 6y agoI learned a lot of useful (and useless) stuff in college, but I do think we should encourage a gap between undergraduate and graduate studies.
- sillysaurusx 6y agoIt seems like ML is an interesting counterexample to Colin's point. Industry is now one of the best places to do ML research, but, the history is pretty fascinating. It seems like ML was a product of academia, and quite a fine product too. I think we as entrepreneur types want to dismiss academia. But there's something to be said for the institution. It's one of the only ways to "infect the world with a thought virus." Besides, which research problems has Tarsnap solved? At least, solved in a way that can be reused by the world?
- srtjstjsj 6y agoThe answer to your question is in the OP article. If you forgot, reread it.
- newbie789 6y agoWho pissed this guy off so bad? I'm not a user of Tarsnap and don't know this person, but it seems like a comment here seems to have carried a lot of weight both with him and the community (considering the fact that his rebuttal has reached #1 on this site.) I clearly don't know what's going on but is this sort of stuff normal on here?
- _-___________-_ 6y agoNothing in Colin's post came across as "pissed off" to me. It's an interesting discussion and I'm glad he decided to respond in long form.
- jl2718 6y agoIt was me, and it seemed like a fascinating person and startup story to inquire about. I didn't expect any response at all. His epic response here makes him more so. A few others responded with some bitterness, and that's okay too. I hope that they can experience enough good things in their life to see the good in others too. And yes, I believe I have seen this a few times now. Some of the most active members here are remarkably brilliant and productive in the real world too. Unfortunately for the rest of us, the causal link of the former to the latter seems to be negative for my longitudinal N=1 study. Good luck in your explorations.
- ookdatnog 6y ago> academia is a lousy place to do novel research This observation seems a bit at odds with the abundance of core technologies that originated in academia. The ENIAC (and a good chunk of the history of early computers), the internet, the world wide web (counting CERN as an academic institution here), most of the core ideas in programming language design (most programming languages we use aren't academic, but most of their ideas can be traced back to academia), most of the history of AI (including neural nets), probably most published work on algorithms and data structures, ... And that's just talking about CS. This doesn't necessarily disprove the author's point: most of the things I've listed are quite old, academia may have worsened in the meantime. But I think they glossed over the main advantage of academia over entrepreneurship: shielding the researcher from market effects. Although investors can do the same thing, as a system academia is more reliable, and for some technologies which take very long to develop or are too hard to monetize (such as algorithms) I don't think entrepreneurship offers an alternative. EDIT (small addition): I think the author is also a bit quick to generalize from "I had a lousy experience in academia" to "academia is a lousy place to do nover research".
- randomsearch 6y ago> academia may have worsened in the meantime This is, I believe, correct, and the problem. > But I think they glossed over the main advantage of academia over entrepreneurship: shielding the researcher from market effects Both I and others moved to academia for this reason. However, it is incorrect. Perhaps "market" is not the right word, but the life of an academic is dominated by competition for grant funding, and mostly focused on "salesmanship" -- more focused on selling than an entrepreneur, in fact.
- wraptile 6y ago>> academia may have worsened in the meantime > This is, I believe, correct, and the problem. I think it's a bit of the opposite: academia didn't get _that_ worse but entrepreneurship has gotten much much better. In reality it's probably a mix of both.
- theaeolist 6y ago
- Wump 6y ago> spent his time on backups instead of solving millennium problems The fundamental issue with this comment is that it assumes that solving millennium problems is obviously a high impact thing to do. It ignores the complexity and unpredictability of the world. The TarSnap developer's story is a great example of this -- how "working on backups" led to impactful work that he probably didn't predict when he started it. I often see people fall into the trap of moralizing about what "good" work is. Ironically, that prevents them from ultimately having the most impact that they could have.
- andrewjl 6y ago> In short, academic institutions systemically promote exactly the sort of short-term optimization of which, ironically, the private sector is often accused. I find it stunning at how little people understand the role of incentives in basically any sort of human-driven activity.
- nefitty 6y agoYou find it stunning that people don’t intuit a fundamental principal as regards human behavior? You’re either commenting in bad faith or you yourself don’t understand some basic principals of humanity in the 21st century itself.
- tonyarkles 6y agoIt’s interesting. I’m looking at your comment and GP’s comment and there are two equally obvious ways to interpret the GP’s comment: - academia’s bureaucracy not understanding the incentives they’re creating and the positive and negative consequences - the OP not understanding the intentional incentives and outcomes created by the academic bureaucracy
- andrewjl 6y agoYou can substitute surprising for stunning. In any case, I'm guilty of missing this as much as anyone. I chuckle at Charlie Munger's aphorism: "Well, I think I’ve been in the top 5% of my age cohort all my life in understanding the power of incentives, and all my life I’ve underestimated it. And never a year passes but I get some surprise that pushes my limit a little farther."
- atemerev 6y agoI have moved from consulting to academia with the start of the crisis, as I have thought I will bring more utility there. It is true so far, but the amount of bureaucracy is staggering. Though running a small business here in Switzerland is also impossible without a ton of bureaucratic hurdles, so there's that. I have won a "fast track" grant back in June, but the money is still not unlocked, as it requires cooperation for paperwork of 10 individuals (of which 7 are the administration) in 3 organizations.
- aborsy 6y agoIt’s unfortunate that Tarsnap uses cryptography (and technology) developed 95% in academia that seems not appreciated by its author who seems focused on the rest of the 5%. People in industry deliver a product and feel the impact. But sometimes they forget that there is several centuries of innovation in math and physics that has enabled that product. On cryptography, you need people like DJB who invent algorithms, you need thousands of pages of crypto analysis to make sense of it, you need guarantees, you need to think of quantum computers 50 years in the future, etc etc. For instance, you need Salsa20 and those hash algorithms developed by others. These components each depend on other building blocks developed earlier. I encourage you to see the broader scientific and technological enterprise. The vast majority of long term research takes place still in academia.
- falcor84 6y agoTo what part of the post are you responding? If there is any point in which the OP showed a lack of appreciation of what came before them, I missed it.
- ImprobableTruth 6y agoProbably to the "academia is a lousy place to do novel research" part. I don't know about you, but I also took this as really dismissive of the work done by people in academia.
- falcor84 6y agoInteresting. I only saw that as criticizing the institution and processes rather than the academics themselves.
- raverbashing 6y ago> but most of them can be summarized as "academia is a lousy place to do novel research" Completely agree. And the cache crypto example is a good example. Especially in computing. There's so much red tape in the academic world that doesn't make sense when you're basically thinking or writing about code (ok, I know CS is not only about code, but what I'm saying also makes sense for things like big-O analysis, crypto, etc).
- zelphirkalt 6y agoThis person, of whom I never read before reading the article, is kind of a hero in my eyes. Fully committed to their work and writing amazing software it seems. Good. More people should do so, if it makes them happy. Not everyone needs to found a family and have a 9 to 5 job. I am a big fan of society allowing for some special cases, where people can unfold their potential, instead of forcing them into some (for them) boring or unproductive jobs.
- Al-Khwarizmi 6y agoAs an academic, I agree with much of the post (although I'd say people who are able to do free, autonomous research is a minority both in academia and in industry. In industry, most people don't own the company, and even those who do often have too many restrictions imposed by clients, time, etc. to do that). But I would like to comment on one detail: "In short, academic institutions systemically promote exactly the sort of short-term optimization of which, ironically, the private sector is often accused". Not ironically. Deliberately and consciously. In recent decades, there has been a push to evaluate public services in private sector terms, i.e., basically, economic profitability (and in particular, short-term profitability). At least in my country, this is quite explicit, but I suppose it happens everywhere to a greater or lesser extent. There is a lot of public discourse that academia cannot be an ivory tower, it has to constantly be evaluated and provide proof that what it's doing is useful. It's difficult to obtain funding for basic science, you need to promise immediate practical applications to obtain grants. Papers are increasingly shunned in favor of patents, the academic that "only publishes papers that other academics read" is now a figure of parody or derision. There are even research grants whose main or sole evaluation criterion is "amount of research income generated in the last X months". So to sum up, the capitalistic mindset of looking at everything in terms of short-term profitability has harmed, and continues harming, academia. But this is not a paradox, it's something quite explicit and deliberate.
- SandB0x 6y agoThe original comment was completely unacceptable.
- ludwigvan 6y agoThe comment that is mentioned in the article is here: https://news.ycombinator.com/item?id=24533381 https://news.ycombinator.com/item?id=24533381
- jacquesm 6y agoThis is pretty close to home. A few weeks ago someone brought up the subject of my education and was surprised I didn't go to any university. In fact, I never even completed high school due to what is euphemistically called 'personal circumstances'. I went to work at 17 and had my first company by the ripe old age of 21. The alternative, some degree (likely CS) and doing computer science would seem terribly boring compared to the live that I ended up living and I feel no less productive because of missing out on that crucial secondary education. Colin is obviously about a million times smarter than I am, he would have probably moved the needle in whatever field he would have entered. I would have probably hung on just long enough to launch my own company anyway because that's just the way I'm wired. In the meantime, my one original contribution (live video on the web) has touched more lives than anything I would have done as a minor cog in some university. In the end, what others think of your life or your decisions to fill it in one way or another doesn't matter. What matters is that you feel satisfaction at what you have achieved. You are your own harshest judge and what others say about what you've done is irrelevant. Colin has received more than his fair share of flak on HN for not chasing the $ as much as he could have. I think he's done things 'just right', enough commerce to get your hands free and spend your time on the things that you think matter most. That freedom is worth far more than being beholden to investors or partners in a business. Wished there more of him and fewer wanna-be-rich-guys.
- jacquesm 6y agos/live/life/
- daniel-cussen 6y agoNot to mention he invented scrypt, which underpins billions in cryptocurrencies, including Litecoin, the second-biggest cryptocurrency. He could easily create another hash function for tens of millions.
- junon 6y ago> If I had been awarded a five-year $62,500/year grant with the sole condition of "do research", I would almost certainly have persevered in academia If that were a thing, I would have actually wanted to go to college. I was fortunate enough to have some high school teachers lay it out plainly for me given my interests and desires, and concluded much the same things as Colin has stated in the article. Not going turned into the first large risky thing I did in my life that really, _really_ paid off in later years. There was no incentive otherwise to go for the same reasons described in this article.
- lordnacho 6y agoAbout that last bit where he mentions the academic grant, I wonder if the standard of living afforded to academics has dropped over the years. If you hang around Oxford and Cambridge, you find these old buildings that look impressive. Then you study there for a bit and you find out a lot of the places seem to be divided up, as if there was once a time when each member had much more space than they did when I got there. Same goes with the common areas, they somehow seem like they were meant to be used by fewer people. Of course there are also new building there, but they're all of the cramped office variety, lending further credence to the theory.
- Fragoel2 6y agoAs an academic myself I can somehow relate with Colin sentiment, not all work that is done in academia is impactful and/or novel, and indeed the current system rewards small incremental improvements rather than working on challenging problems. However, I feel that there is academia and academia. I am lucky enough to: a) work with a supervisor that, during my PhD, encouraged me to ignore, and sometimes even challenge, the consensus view of the research community on the topic, believing that there's always more than one way to do things.; b) live in a country were universities are still mostly state-founded and so I do not need to constantly be hunting for grants. Of course, there are compromises. I do have to spend some of my time on projects/papers that I hardly believe will ever have an impact outside of the academic world but that is pretty much the same as having to manage servers and customers, as long as it leaves me enough time to work on projects/papers that I enjoy. Finally, I do believe that "small incremental improvements" is not the only way to do earn a living in academia. Being curiosity-driven, I consider myself more of a "generalist" rather than a "specialist" and I've been investigating rather diverse topics (e.g., https://news.ycombinator.com/item?id=22678471 https://news.ycombinator.com/item?id=22678471 and https://news.ycombinator.com/item?id=24361852 https://news.ycombinator.com/item?id=24361852). As I only have been doing this for a few years I have no idea whether my work will eventually be considered relevant enough to earn me a permanent position, but so far it has been quite well received by the academic community and I had no troubles in publishing it.
- jrochkind1 6y agoThe premise that "just" backups is not high on the scale of usefulness is flawed from the start. Does not nearly every other production IT project there is depend on a reliable backup infrastructure?
- erikerikson 6y agoWhile thoughtful, this missed an opportunity to write about a deeper dynamic. Most of us spend our lives chasing the creation of transactions (with employers or the market, whether grants, purchases, or contracts) and maximizing the benefit we receive from those transactions. While I appreciate that this holds us to producing in ways that others adversarially agree is valuable and can incentivize efficiency... It also holds us in competition against rather than cooperation with. It focuses us on transaction creation, not on life improvement. If something would make everyone better off but that value can't be captured it often isn't done. If something should efficiently make us all happier but costs the decision maker any money it often isn't done. These are necessarily generalizations. Instances of it being otherwise occur. Some contexts, like families and collectives allow for contradictions to my statements. These rarely scale. I think of the dynamic that children will happily help out the family and do chores. Many parents find that when compensation for chores is offered the children become less helpful. They switch from joyfully (or at least meaningfully) participating in the family endeavor that supported and supports their growth and well being (and if lucky showered them with love). They switch to demanding compensation for everything and end up doing less to help. If this describes our economic engagement, how much more could we all be happily accomplishing at greater efficiency with our time and energy? I recognize this could sound like it would depend on some idealistic prosocial world. What assumptions hold that in place? What if the little supports we give and the forgone advantages were noted to our credit? What if our game theoretic strategies for engagement influenced the rules under which we labored and the willingness of others to interact with us? How do we better optimize?
- mleonhard 6y agoIt doesn't need a prosocial world, just a small space. With some effort, we could make a few square kilometers with both democracy and socialism. We could start a town. I think the town would need these things to be successful: - All land in the town is owned by a non-profit. The non-profit's board is elected by the town. It's private property, like a university. - Businesses in the town are owned by non-profits. The non-profit's boards are elected by the town. The non-profits also provide venture capital and mentorship for new businesses. - Everyone receives the same monthly income. Everyone must work at least 10 months a year, 35-hours a week. Folks who can work but refuse to work must choose between staying in a mental clinic and leaving the town. - There are strong barriers to using outside money. For example, rent may only be paid from one's town-paid monthly income. - Health care, education, and child care are free. - Immigration to the town uses a lottery system, with buckets for various socio-economic and ethnic groups, so the people in the town represent the people in the country. The town charter specifies quarterly immigration quotas up to a target population (50% growth per year, from 50 people to 500k in 25 years). - Voting is mandatory, ranked-choice, without districts, and without propositions. Elected leaders must be women and may serve only one term. All election candidates must take a knowledge exam and their answers are published for voters to scrutinize. - Personal motor vehicles are not allowed inside the town. Everyone moves by walking, bicycles, and subway. Zoning is high-density mixed-use, like many cities in Europe and Asia. - Products are transported between buildings via underground automated systems, like airport luggage delivery systems, in standardized crates. Trash, compost, and recyclables go in sealed crates. There are multiple delivery companies with rails/tracks stacked in the under-street corridors. Large items can be delivered by small electric trucks driving slowly on the roads. Safe, quiet, and clean. Cozy. I think of this nearly every day.
- MrXOR 6y agoAcademia is the best place to do novel research, and a bad place to write great codes (like Tarsnap). You can't do great research on "Birch and Swinnerton-Dyer conjecture", if you are not an academian (like Manjul Bhargava). > In 2005, I made the first publication of the use of shared caches in multi-threaded CPUs as a cryptographic side channel... As far as I know, the first publication about cache side channel attacks was [1] by Dan Page in 2002. [1] https://eprint.iacr.org/2002/169.pdf https://eprint.iacr.org/2002/169.pdf
- cperciva 6y agoI don't think the smart-card processor cores Dan Page was considering were multithreaded.
- daxfohl 6y agoJust, thanks for posting this. Been down lately (hasn't everyone...) and wondering "what could have been" if I'd stayed in academia. This post gives me new inspiration to keep plugging away at work and spend some spare time researching things that are interesting to me, and could be just as productive as doing it in academia.