10 ms·

I might be a complete failure (after 8 years of work)

126 points by chasingsparks 16y ago

rortian 16y ago
An interesting post. I'd be interested hear what kind of background you've got. I think your remark on finance professors is quite wrong. It takes quite a lot of dedication to do a Phd. I'm not great respecter of academic prestige, but its a bit insulting to imply that people fail their way into a professorship. I'd be more than happy to share some pointers on financial papers I've read and such. Just talk about your background and what you know. If you think 'a random walk down Wall Street' is correct, it isn't. Have you seen the derivation of Black-Scholes? Do you know what most high frequency trading is based on? I'd also be interested to see what type of stuff you like to read.
- chasingsparks 16y ago
  I didn't mean it in an insulting way. I'm actually starting a Masters/Ph.D. program in computational social sciences in the fall, hoping to fully immerse myself in an environment that also finds markets fascinating. I have derived Black-Scholes; I study market micro-structure; I read as much as I can on anything related to social systems from volatility surfaces in options pricing to more generic machine learning methods such as PSO. (I'd like to respond more, but I just noticed I am 20 minutes late for my fathers birthday party.)
  - rortian 16y ago
    Cool, definitely get back to me. Kinda random but I am very interested in complex networks as well as some finance stuff.
- cjy 16y ago
  I don't think he was being insulting. He was just explaining that the incentives discourage anyone from publicly disproving the efficient market hypothesis.
  - lars512 16y ago
    This is where it gets fun though. An inefficiency being found amounts to you knowing something about pricing that others don't. You exploit it for profit, and the process of exploiting it removes or reduces the inefficiency. The only way the market is truly screwed is if consistent arbitrage opportunities become available, but the kinds of inefficiencies people find seem more transient than consistent. On another note, betting markets are (or were) apparently far less efficient than financial markets, in that the true likelihood of a team/horse/camel winning in a competition may be different to the money which punters are willing to bet on it. I know of a guy who made his fortune after many gruelling years by finally coming up with prediction models which worked in these kind of gambling scenarios.
- fernando 16y ago
  Could you elaborate on this? In which ways is "a random" walk incorrect? In what is high frequency trading based?
  - rortian 16y ago
    On the last question give me a little time to look up a paper. I'll post it. Essential it shows how the strategy of most long-short hedge funds work. I'd have to look again, but one of the major high frequency strategies uses a version of this. I'd love to read it, but the easiest way to answer your second question is to point you to 'A non random walk down wall street.' One huge assumption that Black-Scholes makes is that returns are normal distributed. This is not done for empirical reasons rather mathematical ones, namely it is so convenient. Edit: Pdf http://www.newyorkfed.org/research/conference/2007/liquidity/Khandani_Lo.pdf http://www.newyorkfed.org/research/conference/2007/liquidity... Some meta data and such if you don't feel like going with a pdf: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1015987 http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1015987
drewcrawford 16y ago
You can keep adding data points to your spline/polynomial interpolation, but that doesn't mean the model will be any better. In fact, for the purposes of predictive power, it will probably be worse. I'm not extremely well educated on the subject, so somebody correct me, but these predictive models have always struck me as just complicated splines. Add a thousand data points, and the "predictions" for old datapoints look really good. The problem isn't with the math, it's with an inappropriate application of the math based either on fundamental misunderstanding of extrapolation or on a fundamental failing of human psychology in not realizing that the model was built on the data, and not the other way around.
- mikecane 16y ago
  Long-Term Capital Management always springs to my mind. http://en.wikipedia.org/wiki/LTCM http://en.wikipedia.org/wiki/LTCM
- kiba 16y ago
  Taleb's empiricism implies resisting generalization from data and limiting the derivation of general rules from particular observations as one can be missing hidden properties. Thus he believes that scientists, economists, historians, policy makers, businessmen, and financiers are victims of an illusion of pattern. They overestimate the value of rational explanations of past data, and underestimate the prevalence of unexplainable randomness in those data. -- Wikipedia on Nassim Nicholas Taleb People project patterns onto data when in fact, they don't exists.
  - mikecane 16y ago
    "Fooled by Randomness" is a great book. But I sometimes feel about it the same way I feel about people who say lotteries are a waste of time: some people do win. And sometimes, despite everything, a transient pattern does repeat. How frustrating! Edited to add this, which is sometimes also brought up: http://en.wikipedia.org/wiki/Illusion_of_control http://en.wikipedia.org/wiki/Illusion_of_control
    - 16y ago
brianobush 16y ago
negative findings (and publishing those results) are just as useful as positive findings.
- pbhjpbhj 16y ago
  If that's true, I think it needs a lot of thought, then we at least know that negative findings are valued far less. I found that my TOE was wrong is not likely to win me a Nobel unless there are associated "positive" findings ...
  - brianobush 16y ago
    I was obviously considering only negative findings with actual thought, research plans and analysis.
stretchwithme 16y ago
The most important thing in your life is you and the brain you are carrying around in your head. If your experiences have made you smarter, you DO have something concrete to show for the years you've put in. Entire countries have had everything destroyed, but have come back to be very successful. And this happened because of what they carry around in their heads.
erikpukinskis 16y ago
If you're doing what you love, you've already won. I feel a similar way pretty often. I have a couple of visions of software I want to build that are far out of my technical reach. I've been prototyping them, on and off, for about eight years too, depending how you count. They've all failed. I start with a flash of optimism, and code until I realize I was wrong. I then stop, and rethink things until I think of another guess. This is exactly how great accomplishments happen. You may be working on an impossible problem, or you may be working on a 20 year problem. Or maybe you're lucky and it's a 10 year problem. But even if your goal is impossible, it's very likely that if you work on it for 20 years, you will solve a 20 year problem. It might be slightly askew to what you thought you were solving, but it will happen. It's almost inevitable. Regardless, I wholly support what you're doing. Keep going. And read widely... explore... make sure you're playing in the margins of your field, and exploring odd projects far outside the scope of your work. It's those places that you will make that accidental discovery... that in all these years of trying to "beat the market" you made a fundamental discovery in artificial intelligence, or network theory, or whatever. And yeah, if you're doing what you love, you've already won.
krosaen 16y ago
I find it inspiring that he's stuck with it, and wonder how's he's made a living in the meantime
- chasingsparks 16y ago
  I take short-term jobs and contract work for a few months, then quit everything for full-time research. Usually the jobs I take happen to require learning something tangential but possibly useful. It's a happy coincidence that I am perfectly happy with the absurd level of frugality required. (I live in a place that most people call the "House of Squalor. It's really not dirty, but it is semi-dilapidated.)
malloreon 16y ago
It sounds like despite your ideas of what failure is, by age 25 you've learned and applied more than most people ever will.
- singular 16y ago
  Agreed. I'm 28, soon to be 29 and I have achieved a great deal less. That kind of experience, though sour, is pretty awesome, at the very least you should be able to get yourself into a highly paid quant position at a finance firm.
jey 16y ago
Seems like it'd be a better bet to apply all your skills to creating value (wealth) instead of just trying to push it around a smidge more efficiently. In other words: find a real problem and solve it, increasing global wealth in the process, and collect the economic reward.
- HSO 16y ago
  Hm, I accidentally upvoted this so I may as well leave a comment. The dichotomy between "creating" wealth and "pushing it around" is false, in my opinion. Lots of people make it and it's intuitive, I know, but figuring out the proper value of things, trying to make forecasts about the future, even profiting from differences that shouldn't be there, all this is part of the capitalist meta-system called financial markets. One should not confuse the agency issues, borderline-legal information advantages, or even outright fraud in the markets with the basic activity that sustains them. Think about the alternative: Would you like to have to sit with a dimwit credit officer (= bureaucrat) from a big bank who can then effectively veto your plans? Or would you rather have a lively market that decides on such things? Do you want to do your IPO in a public equity market or would you prefer to try to break into an insider club of old rich people who may never come to understand your funny newfangled business model ("what's internet?!"). German sociologist Georg Simmel wrote an interesting book about money, one of the ideas I liked is that its fungibility dissolves the need for relationships, making us free to choose them, and thus effectively increasing personal freedom. The same, I'd venture, is true for capital markets. By spreading transactions around, they increase your options and freedom as entrepreneur. And for markets to work, you need people actually _in_ the market, like it or not. Figuring out value (as opposed to the going price), identifiying and exploiting inefficiencies, trying to make more intelligent forecasts of likely developments, all these _are_ "real problems" in a market economy. Having intelligent, hard-working people working on them makes the system as a whole more intelligent.
  - jey 16y ago
    I fully agree that finance is useful because it increases market efficiency. The problem is that like most fields, most of it is uncreative legwork that doesn't provide much leverage to get rich from. So a smart creative person should consider whether they're falling into the cognitive trap of "I want to make money, finance deals with piles of money, so I should apply my creativity to finance". It's just a crapshoot unless you have some real information asymmetry[1] you can exploit to make money. This could be in the form of noticing some sort of systematic inefficiency, having a unique and effective strategy of gaining market information to identify mispricings, etc. Otherwise you'd be better off finding some unmet need to creatively address and avoiding the cognitive trap that leads from "I want money" to "I should go into finance". 1. http://en.wikipedia.org/wiki/Information_asymmetry http://en.wikipedia.org/wiki/Information_asymmetry
cjy 16y ago
You've learned a lot, so you're obviously not a failure, but I think your post illustrates how draining a winner-take-all field like quantitative trading can be. You can never be sure if your wins or losses are real or just mirages caused by noise. Humans need the emotional encouragement that incremental success brings. Nothing get's me excited to work more than a good day of sales or a thoughtful comment from a customer. It takes a special type of personality to keep on persisting in the face of repeated failure. Certain start-ups also have winner-take-all properties. Anything that relies on network externalities or advertising revenue is more risky. The rewards can be big, but I prefer markets where the product's value isn't a function of the customer base and success is more incremental.
deleted 16y ago
[deleted]
edw519 16y ago
After eight years, I have nothing concrete to show for my efforts. So sad, so true. This is what happens when you play a zero-sum game. Even if you win, someone has to lose. Here's an idea. For the next 8 years, why don't you do something that helps others by making the pie bigger for everyone instead of just trying to game a bigger piece of a smaller pie for yourself. When you do for others, you always have something concrete to show for your efforts. It may not be what you expected, but it will definitely be something worthwhile, I promise you.
- aspiringsensei 16y ago
  This is a very valid way to put this, and it helped me. Thank you.
- olliesaunders 16y ago
  This is what happens when you play a zero-sum game. Even if you win, someone has to lose. I don't see the causality. He has yet to succeed because it's an incredibly difficult problem domain. There are problem domains of similar difficulty that benefit people that he could have chosen and still be in this position.
  - kristianp 16y ago
    Almost anything where you create something would be better. Then again he/she is creating software and algorithms.
    - lowkey 16y ago
      Yes, but to what end? When it comes to creating things that have value, I like objective standards and my favorite definition is Buckminster Fuller's: "Real wealth is whatever nurtures and accommodates human life." Anything else is pointless. Incidentally, a recent study claims that 60% of UK workers feel that their work is of no use to society. Financial Services, Telemarketing, Spammers, High-Frequency Traders and many other 'professions' would fit this description. Here is the link: http://www.reddit.com/r/Economics/comments/cjgx7/60_of_uk_workers_feel_their_work_is_of_no_use_to/ http://www.reddit.com/r/Economics/comments/cjgx7/60_of_uk_wo...
motters 16y ago
Many smart people have tried to "beat the markets" and failed. From what I've seen of it, this sort of activity is at best a form of numerology. Also, in trying to move money around slightly more efficiently than the next market analyst you're not really generating wealth as such, and your ideas and activities will leave no lasting value for society. You may not be a failure, since the skills acquired in analysing trends can be utilised in other areas. The process of having promising looking preliminary results, only to find that they are illusory is familiar to many scientists. Ultimately it comes down to what you really want to do with your life. Is your life only about moving money, or something else?
- Kaizyn 16y ago
  If I'm not mistaken, I believe the idea is to set up some automated trading bots that just simply make money for you on a continual basis. This constant revenue stream allows you to forget about making money and allowing you to use those resources to either fund a life of ease and luxury or other pursuits such as "making the world a better place".
narrator 16y ago
I tried to beat the market using all kinds of AI stuff off and on for a number of years. Learned a lot about AI techniques, and became a better programmer, which helped in my day job, but never got anywhere. It's a fool's game. Besides, most of these HFT systems hold positions for only a few minutes or less. There's no way someone without a huge account, to cover commissions from trading so much, and access to a lot of expensive real-time data streams could possibly make any headway at it. Not to say there aren't people who do make money at trading, but they are swing traders and/or people who've figured something out about the market that the market doesn't know yet, and thus has not revealed at all in the price movements of the stock. Since price movements are the primary input into a lot of trading algorithms, most trading algorithms are blind to these developments.
mbateman 16y ago
A lot of people are leaving comments critical of his goal of beating the market as playing a zero-sum game, just pushing wealth around, having no value for society, etc. This strikes me as dubious and high-handed, to say the least. First, as to the value of the endeavor. He is trying to, in his words, "come up with an algorithm that finds pockets of profitability in a cloud of probable randomness." If he could really do this, it would create capital flow, the ability to borrow, and so on, for people that could turn it into wealth that would not otherwise have access to that wealth. Financial markets serve a real function, and beating the efficient market hypothesis would improve their ability to fulfill that function. Second of all, he's extremely interested in the problem. He's obsessed with modeling complex systems and sees this as a sort of holy grail. Surely the fact that he's interested in it and values it counts for a lot, especially if you grant the above point, that there is a value to what he's doing. He has his reasons for being interested in it, and it's a productive endeavor. I think it's false that his lack of progress, or the reasons that he's frustrated at his lack of progress, have to do with these aspects of his goal. It's more that he's working on an extremely hard (arguably insoluble) problem where it's hard to see incremental gains. This is a problem that ambitious people in many different fields face.
- rue 16y ago
  All this is only undermined by the fact that the financial sector and the stock market in particular involves mostly fictitious money. There is no problem that it solves that could not also be solved by saying "here, I conjured you a million dollars out of thin air, act as if you had it". The market is, as it were, a MUSH. There are certainly plenty of talented people working in finance. That talent is just completely wasted.
  - barrkel 16y ago
    All money is fictional, in that it only has value to the degree that people agree and believe it has value. This is no less true for the US dollar than it is for gold, silver, oil, cattle, etc.
    - 16y ago
aspiringsensei 16y ago
My mother was down on the floor of the NYSE the other month, and she got the opportunity to corner a trader who she noticed was relying heavily on algorithms. She asked "What's the difference between a good trader and a good algorithm?" The trader said "nothing.If you're buying 1000 shares, it's probably better to simply put in a market order. A large hedge fund seeking to buy many hundreds of thousands - where the difference between paying 8.30 and 8.31 a share matters - algorithms can actually be helpful." The issue here is the definition of "trader." For this guy, trader is a person who executes large trades for an investment manager. A trading decision is not the same as an investment decision, and in that vein many trader's motivations are simply to buy a desired quantity of a financial instrument as cheaply as possible. If you're buying 1000 shares, it's probably better to simply put in a market order. A large hedge fund seeking to buy many hundreds of thousands - where the difference between paying 8.30 and 8.31 a share matters - algorithms can actually be helpful. If you're buying 1000 shares, it's probably better to simply put in a market order. A large hedge fund seeking to buy many hundreds of thousands - where the difference between paying 8.30 and 8.31 a share matters - algorithms can actually be helpful. Humans are probably better computers for figuring out which stocks to buy and sell to my mind, but there's certainly a place for algorithms if they are not too high-minded.
- cowmoo 16y ago
  You are right that for a individual investor focused on the long term, a algo-order probably wouldn't make a difference in comparison to a market order with your E-Trade/Schwab account (most likely, your order-flow won't go directly to the market anyways; it is either crossed internally, or re-routed to a broker/dealer that's paying retail brokers for the order flow such as Timberhill). However, I respectfully disagree that humans are better at making trading decisions than computers. The world of algo trading can be divided into two sides, a) high frequency traders, who through fast cancel-and-replace limit orders and colo with the market centers, try to act as virtual market-makers (or scalpers, depending on your perspective), b) buy-side institutional fund managers who want to complete their orders, without HFT predators and negative market pressure. Large block orders are spliced into small lots (i.e., VWAP) and sent to the market using intentional limit price over time to hide the movement of a huge buy or sell order on the market. Human beings might be better than machines at picking single stocks for long term investing (although most people are probably still better off investing in an ETF). But that's not how sell-side traders make money in the first place. Guys like GS/MS/Timberhill make money by having the unfair advantage of faster execution speed, more capital and specialized trading algo's against the small retail investors.
bbers 16y ago
To beat the market you HAVE to predict what trades others will make in the future. It is easy to use math to make it appear to be a numbers game but that is just abstraction. The reality is that people buy and sell stocks for 1000s of different reasons.
- chasingsparks 16y ago
  I use much less math than your average quant. I prefer simulations with a lot of agents. Think Game of Life, not partial differential equations.
naner 16y ago
This type of thing has been done before: http://en.wikipedia.org/wiki/FatKat_(investment_software) http://en.wikipedia.org/wiki/FatKat_(investment_software) http://www.idsia.ch/~juergen/finance.html http://www.idsia.ch/~juergen/finance.html http://www.technologyreview.com/blog/guest/25308/ http://www.technologyreview.com/blog/guest/25308/ I'm sure there are other examples. I think part of the problem is this guy thinks he can get rich quickly with just the right set of programming instructions.
- ruang 16y ago
  Nice quote from one of the links' links, from NY Times: But as these new techniques proliferate, some worry that promotion is outpacing reality. These techniques may be better for marketing than stock picking.
Loy 16y ago
The term "complete failure" is more revealing about your draconian idea of success. You should take a look at your own words : "they have never found their Holy Grail" [therefore, they joined the ranks of Academics as a testimony of failure]. If you're running after the Holy Grail, you have no choice but to be a hero. What kind of pressure is that ? In fact, most of the answers are in front of you, in the way you express yourself. For instance, the myths of the Holy Grail and Tantalus are ones of insoluble challenge and dilemma. Therefore, when you are wondering if the Holy Grail exist, where it is, and if you'll ever reach it... You are probably asking the wrong questions. Instead, you should ask yourself : "Who am I ?". Do you want to be rich, or do you want to "beat the markets" ? That's 2 different things, one is a mere consequence of the other. Where is your priority ? As it's not clear from your writing, I suppose it's not clear in your mind either.
- chasingsparks 16y ago
  The title was unnecessarily provocative, a reflection of my mood at the time of writing it. (It's Sunday; I'm hung-over; and I was looking at the results of a four day simulation run that were ambiguous.) My priority is in "beating the markets" because it's a fascinating challenge. I also have interests that lie beyond satisfying my intellectual curiosity but require wealth. I find the idea of using my own wealth to solve problems (e.g. avenues of cancer research) far more appealing than fund-raising or requiring public funds.
wangwei 16y ago
Even people with very high level math knowledge fails to predict the market (remember the two noble prize scientists who failed?).Since you're out of school when you were 17, I doubt you'll able develop complex schemes (unless you are a very very rare genius).
- xenophanes 16y ago
  Obviously he has the aptitude to learn things on his own. Sounds like he can learn better/faster on his own time than the large majority of students do at a school. So no need to hound him with pessimism for having an unconventional but pleasant lifestyle.
- chasingsparks 16y ago
  I was not out of school at 17. What caused you to infer that? (I'd like to correct it if it sounds that way.) I am actually starting a masters/Ph.D. program in the fall in computational social sciences. Like I mentioned in the post, I am interested in social systems in general, not only markets. (However, I do find markets to be the most interesting.)
muhfuhkuh 16y ago
Look to none other than Long Term Capital Management[1] back in the 90s to see how dangerous a game it is to either algorithmically predict or automate a thing as emotionally wrought and, at times, institutionally manipulated as the market. [1] http://en.wikipedia.org/wiki/Long-Term_Capital_Management http://en.wikipedia.org/wiki/Long-Term_Capital_Management
- jrockway 16y ago
  What? Their time-tested bond arbitrage algorithms worked great. They beat the market month after month. Only when they tried speculating on equities did they fail. And even then, their positions only took short-term losses that they would have recovered from had they not been leveraged 100-to-1. Their algorithms were correct, but they didn't have enough capital to see their positions through to the end. The LTCM story is about greed and leverage, not algorithmic trading. Plenty of money can be made trying to beat the market: it pays the salaries of the hundreds of thousands of investment bank employees around the world. The problem is, you personally can't do the same things they do, because you don't have enough capital. Raise a billion dollars and your days of commuting to work are over.
noonespecial 16y ago
Look on the bright side. Many people dedicated their lives to the search for a process to turn lead to gold. This seemingly fruitless search lead to all kinds of neat discoveries. You've learned lots of valuable things that you wouldn't have learned otherwise as well. Keep searching if you must but start benefiting from that knowledge. The real secret about turning lead to gold is that it is possible, but by the time you have mastered the skill to do so, you no longer need to or want to.
- gvb 16y ago
  Alchemy and transmutation of lead to gold is a pretty apt analogy for John Nelson's quest to turn his model of the market into gold. a) The alchemical belief in transmutation was based on a thoroughly wrong understanding of the underlying processes. John's (and other's) lack of success indicates a faulty model. b) Ironically, it transpired that, under true nuclear transmutation, it is far easier to turn gold into lead than the reverse reaction, which was the one the alchemists had ardently pursued. Nuclear experiments have successfully transmuted lead into gold, but the expense far exceeds any gain. http://en.wikipedia.org/wiki/Nuclear_transmutation http://en.wikipedia.org/wiki/Nuclear_transmutation
  - chasingsparks 16y ago
    If you look at the first sentence of the article, it is linked to the subheading of a blog post that explained my intellectual evolution -- the subtitle of which was "The Contemporary Philosopher’s Stone" which is how I refer to this problem. (http://en.wikipedia.org/wiki/Philosophers_stone http://en.wikipedia.org/wiki/Philosophers_stone) You're not the only one drawing the comparison ;)
    - khafra 16y ago
      Sounds like in that frame, most of the consoling messages here are along the lines of "our gold is not the common gold."
pvdm 16y ago
http://en.wikipedia.org/wiki/Monroe_Trout http://en.wikipedia.org/wiki/Monroe_Trout
ctingom 16y ago
Remember, no man is a failure who has friends. -- Clarence in It’s a Wonderful Life
arthurdent 16y ago
I find myself excited over the preliminary results of my increasingly sophisticated simulations This. The vast majority of the money being made in the market is being made by people with what you may consider a laughable lack of "sophistication". Start simple. Complex modeling of stock market behavior only makes sense if you think the stock market has a complex underlying structure.
- chasingsparks 16y ago
  By sophisticated, I meant refined and cultivated, not devoid of simplicity. It was a poor choice of words. At some point in my development, I started to eschew more complicated forms of AI and machine learning, in favor of simpler agent-based models with "laughably" simple assumptions. I think it's the proper direction, then again, the title I chose for the post was not arbitrary. ;)
10ren 16y ago
Question: has anyone managed to do this? (eg. do hedge funds do this, or do they combine algorithms with their own ideas and quasi-insider knowledge, plus massive amounts of timely data - inaccessible to an individual). My old supervisor (ACM fellow) comforted my failure to create AI by saying that lots of other clever people hadn't done it either. I have a feeling that you are attempting something more difficult that AI - ie, algorithmic inference without the data that your competitors have. Because of their better information, they will find a regularity (eg. 0.01% dip on Wednesdays) quicker than you - and trade it away. It would be awesome if you could do it though - forget the philosopher's stone aspect of gold-for-nothing, this would be a very cool intellectual achievement. Idea: focus on small markets/exchanges (unpopular metals, unpopular countries, even non-financial markets, like eBay) that don't attract the attention of big players. Easier to find arbitrage there; you can make some actual money, and then move onto a bigger market with your learnings, earnings, and confidence. Sort of a niche strategy. Personal Note: My AI research depressed me, and I (wisely) decided to shelve it. It's been incredibly more satisfying to work with others, to achieve doable goals, that benefit others. Fortunately, AI comes up in everything; and my current approach (of not doing AI directly) may well be the most effective way to approach the goal. It might be the same with you - both markets and inference are universal. EDIT So, Tantalus = low-hanging fruit + moving-goalposts http://en.wikipedia.org/wiki/Tantalus#Story_of_Tantalus http://en.wikipedia.org/wiki/Tantalus#Story_of_Tantalus
- pradocchia 16y ago
  Simply put, if anyone has managed to do it, you aren't likely to hear about it. That said, I know of two: First was a math prof that would play the markets every couple years when he needed a little extra cash. Second was someone who had isolated patterns of market intervention and figured out how to profit from them. But that's more like insider trading than beating the market algorithmically.
noahc 16y ago
John, 95% of people would agree with you. But they (and you) are wrong. What is your end game? Do you want to be worshiped on HN or do you want a six figure job some where? If you want either of those, you could just as well write up all you know spit it out in blog posts and free eBooks. Someone is going to recognize your skills and take it from there. At this point the worst thing you can do is keep all your failures to your self, because then no one can benefit from them.
danbmil99 16y ago
I suspect you could get a pretty good job as programmer for some hedge fund. Not only would it leverage your command of this domain, you might learn a few things that will explain why it is difficult to impossible for you to win in a vacuum. There are social engineering aspects to this game that you have not been exposed to yet. Also, you may just not be sophisticated enough, or you have pursued a dead end approach. This is a very rich field of study. If it's your passion, I say stick with it, but expand your circle of knowledge and influence. And screw all this 'zero sum' crap, capitalism is the least zero-sum game humanity has ever come up with. And speculation, market-making, betting on winners & losers is one of the most important driving forces of capitalism.
WalterBright 16y ago
Any algorithmic method of beating the market can only be successful if it is a secret and if its trades are of insignificant volume. Otherwise, knowledge of this algorithm will be factored into the prices, defeating it.
- arthurdent 16y ago
  this is really not entirely correct. when things started melting down in 2008, it was largely attributed to the fact that hedge funds basically all had the same strategies in their book. When one started liquidating, many started losing value and liquidating in a disastrous feedback loop. There are plenty of strategies that are very robust, and in hedge-fund land, its not uncommon that many players are playing very similar hands. In fact, there even exist strategies that get better as more people run them.
rjett 16y ago
"Every few months, I find myself excited over the preliminary results of my increasingly sophisticated simulations, only to be disappointed a few short weeks later to find that I was simply wrong." Isn't this why algorithmic traders "refit" their algorithms on a daily basis?
daylast 16y ago
I hate to be the bearer of bad news but, based on the comments you've made in your post, you are most likely a smart fool (as you have so aptly put). In essence, you don't even know what you do not know. I've worked at top-tier Wall St institutions and have had first hand experience working with several of the largest quant hedge funds out there. Here's my advice: if you truly enjoy markets and are sure you want to take this path, you will be much better served by spending several years working at an established quant hedge fund. This will open your eyes to the theory, processes, data, and techniques required to earn alpha consistently and in a risk efficient manner. It's a long road and that road is full of extraordinarily hard working and intelligent people that one has to constantly compete against. If you can handle this though, you will learn a bunch and at some point in the future you can strike out on your own again. At that point you will at least have a chance of success. Good luck!
Tichy 16y ago
I've met people who claimed to make money with their algorithms, which really surprised me. Your post gave me an idea, though: What if instead of running your algorithm on the whole set of stocks, you create several random subsets. One of them has a good chance of being successful, at which point you can claim that your algorithm works and sell it for $$$. (I am not entirely serious, of course - but maybe some of the successful algorithms work that way?).
soyelmango 16y ago
It sounds like a trolling headline - he himself acknowledges the value of what he has learnt. Some suggested in his comments that he should join a firm so that knowledge and experience could be shared towards finding the algorithm[s]. His reply to Henrik, "I’ve been wary about taking jobs at firms like that because of IP concerns. I always believe I am on the cusp of something great, and wouldn’t want to share my algorithms", is a factor in his progress, or lack of. By working alone, he's more likely to earn 100% of nothing, when he could collaborate and earn a small % of a very large number. That said, if he finds the algorith alone, then big congratulations and respect are in order for sticking to his convictions.
MLnick 16y ago
Only very few people have commented that, you are only 25! (I guess given the nature of the HN community :) I don't think anyone can consider themselves a failure at 25, having spent 8 years learning a lot of programming, algorithms, data mining / machine learning and complex systems modelling. It would be easier to think that if you hated what you were doing. But it sure doesn't sound like you do. So you just need to decide what you really want to be doing. If it's algo trading, I concur with another poster that says go work for a hedge fund (or bank, or prop trading shop). It is super competitive, but they have the technology infrastructure and most importantly the capital. Getting in is not easy, but simply show them all your work (it doesn't work anyway, but is indicative of your skills and way of thinking). You will learn a lot, you may hate the people and environment, or you may love it even more. And then yes after a few years of experience you will most certainly be in a better position to go off on your own again (or do something totally different, by then you will really know if you like it or not!). Many top hedge fund managers / traders only started their own thing at 30, 35, 40, even 50... I know a dentist who became a prop trader. Anything is possible. A PhD would be a great option IF it's for the right reasons. But if you want to do a startup (sounds like you might quite like the idea and you posted here on HN, so...): - you already live on ramen, so no lifestyle change there; - bootstrapping something can't cost more than losing money with trading algorithms; - you already have many of the requirements: coding / technical skills, low-cost living circumstance, love to solve tough problems and a huge amount of tenacity in the face of failure and overwhelming odds; - bonus: your interest in social systems modelling etc ties in pretty nicely with what's big right now and in the near future. So if that is what you really want to do, go for it either alone (or find a co-founder), or find a small/medium startup to work for. To make the transition a bit more natural perhaps focus on ones that are data-driven and have machine learning / modelling at the core of their business. Think recommendations, systems modelling (www.flightcaster.com) and weather (www.weatherbill.com). There are many many examples of YC and other startup companies of this nature (many focused on the social network space). Good luck in whatever you do decide to do with the next 60+ years of your life. On your deathbed you can post about whether you think you are a complete failure or not.
dunk010 16y ago
Well, he might just not be smart enough. Large financial institutions and many smaller hedge funds reap countless millions from the markets and much of this comes from proprietary trading (well, all of it in the case of hedge funds.) There's a massive brain-drain into financial institutions of many of our best and brightest, all in search of making a fortune by the time they're 30. So on the one hand you have top students from top institutions applying their considerable intellect and training into applying advanced mathematical methods against an extremely complex problem. On the other you have a guy who tinkers around himself hoping after hope that the amateur (for that is by definition what he is) attempts that he makes at beating the market are going to some day work out and make him his fortune. Unfortunately the odds are stacked against you, and those are compounded by fact that you have little or no opportunity to employ any serious leverage. "I would have probably, or at least possibly, been wealthy by other means by now". At _best_ "possibly", I think. At 25 that is an incredibly arrogant statement to make.
retube 16y ago
His approach to this - the goal of beating the market - has been totally wrong. It sounds as if he's been at this completely solo - he mentions in the comments that he worked alone as he didn't want to share his algorithms. He would have benefitted enormously from working in an institution that specialises in this kind of work (hedge funds etc). Being largely self-taught and having worked alone since he was 17 pretty much guarantees he doesn't know anything institutional trading firms won't know or have studied - employing as they do advanced mathematicians, physicists etc to work at the cutting edge of algo research and trading. Not only has he missed out academically, he's missed out operationally. "Playing-the-game" is as big a part - if not bigger part - of trading than pricing/valuation. With the resources of a large institution you can learn to play the market in a way you can't possibly by yourself. This is in fact largely how they make money - consistently, year on year. Thirdly - even if he did discover a pricing inconsistency or whatever, it's doubtful he'd be able to leverage it in as profitable a way that he would be able to as a firm.
chegra 16y ago
"He[Dan Zanger] holds the unofficial record in trading stocks by turning $11,000 into more than $18 million in 18 months in 1999-2000. He grew that to an incredible $42 million in less than two years and has the tax receipts to prove it." http://www.investopedia.com/articles/trading/04/082504.asp http://www.investopedia.com/articles/trading/04/082504.asp "Williams won the 1987 World Cup Championship of Futures Trading from the Robbins Trading Company were he turned $10,000 to over $1,100,000 (11,376%) in a 12 month competition with real money. Ten Years later his daughter Michelle won the same contest" http://en.wikipedia.org/wiki/Larry_R._Williams http://en.wikipedia.org/wiki/Larry_R._Williams Somebody has some sort of algorithm.

126 points by chasingsparks 16y ago

An interesting post. I'd be interested hear what kind of background you've got. I think your remark on finance professors is quite wrong. It takes quite a lot of dedication to do a Phd. I'm not great respecter of academic prestige, but its a bit insulting to imply that people fail their way into a professorship. I'd be more than happy to share some pointers on financial papers I've read and such. Just talk about your background and what you know. If you think 'a random walk down Wall Street' is correct, it isn't. Have you seen the derivation of Black-Scholes? Do you know what most high frequency trading is based on? I'd also be interested to see what type of stuff you like to read.

chasingsparks 16y ago
I didn't mean it in an insulting way. I'm actually starting a Masters/Ph.D. program in computational social sciences in the fall, hoping to fully immerse myself in an environment that also finds markets fascinating. I have derived Black-Scholes; I study market micro-structure; I read as much as I can on anything related to social systems from volatility surfaces in options pricing to more generic machine learning methods such as PSO. (I'd like to respond more, but I just noticed I am 20 minutes late for my fathers birthday party.)
- rortian 16y ago
  Cool, definitely get back to me. Kinda random but I am very interested in complex networks as well as some finance stuff.
cjy 16y ago
I don't think he was being insulting. He was just explaining that the incentives discourage anyone from publicly disproving the efficient market hypothesis.
- lars512 16y ago
  This is where it gets fun though. An inefficiency being found amounts to you knowing something about pricing that others don't. You exploit it for profit, and the process of exploiting it removes or reduces the inefficiency. The only way the market is truly screwed is if consistent arbitrage opportunities become available, but the kinds of inefficiencies people find seem more transient than consistent. On another note, betting markets are (or were) apparently far less efficient than financial markets, in that the true likelihood of a team/horse/camel winning in a competition may be different to the money which punters are willing to bet on it. I know of a guy who made his fortune after many gruelling years by finally coming up with prediction models which worked in these kind of gambling scenarios.
fernando 16y ago
Could you elaborate on this? In which ways is "a random" walk incorrect? In what is high frequency trading based?
- rortian 16y ago
  On the last question give me a little time to look up a paper. I'll post it. Essential it shows how the strategy of most long-short hedge funds work. I'd have to look again, but one of the major high frequency strategies uses a version of this. I'd love to read it, but the easiest way to answer your second question is to point you to 'A non random walk down wall street.' One huge assumption that Black-Scholes makes is that returns are normal distributed. This is not done for empirical reasons rather mathematical ones, namely it is so convenient. Edit: Pdf http://www.newyorkfed.org/research/conference/2007/liquidity/Khandani_Lo.pdf http://www.newyorkfed.org/research/conference/2007/liquidity... Some meta data and such if you don't feel like going with a pdf: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1015987 http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1015987

drewcrawford 16y ago

You can keep adding data points to your spline/polynomial interpolation, but that doesn't mean the model will be any better. In fact, for the purposes of predictive power, it will probably be worse. I'm not extremely well educated on the subject, so somebody correct me, but these predictive models have always struck me as just complicated splines. Add a thousand data points, and the "predictions" for old datapoints look really good. The problem isn't with the math, it's with an inappropriate application of the math based either on fundamental misunderstanding of extrapolation or on a fundamental failing of human psychology in not realizing that the model was built on the data, and not the other way around.

mikecane 16y ago
Long-Term Capital Management always springs to my mind. http://en.wikipedia.org/wiki/LTCM http://en.wikipedia.org/wiki/LTCM
kiba 16y ago
Taleb's empiricism implies resisting generalization from data and limiting the derivation of general rules from particular observations as one can be missing hidden properties. Thus he believes that scientists, economists, historians, policy makers, businessmen, and financiers are victims of an illusion of pattern. They overestimate the value of rational explanations of past data, and underestimate the prevalence of unexplainable randomness in those data. -- Wikipedia on Nassim Nicholas Taleb People project patterns onto data when in fact, they don't exists.
- mikecane 16y ago
  "Fooled by Randomness" is a great book. But I sometimes feel about it the same way I feel about people who say lotteries are a waste of time: some people do win. And sometimes, despite everything, a transient pattern does repeat. How frustrating! Edited to add this, which is sometimes also brought up: http://en.wikipedia.org/wiki/Illusion_of_control http://en.wikipedia.org/wiki/Illusion_of_control
  - 16y ago

brianobush 16y ago

negative findings (and publishing those results) are just as useful as positive findings.

pbhjpbhj 16y ago
If that's true, I think it needs a lot of thought, then we at least know that negative findings are valued far less. I found that my TOE was wrong is not likely to win me a Nobel unless there are associated "positive" findings ...
- brianobush 16y ago
  I was obviously considering only negative findings with actual thought, research plans and analysis.

stretchwithme 16y ago

The most important thing in your life is you and the brain you are carrying around in your head. If your experiences have made you smarter, you DO have something concrete to show for the years you've put in. Entire countries have had everything destroyed, but have come back to be very successful. And this happened because of what they carry around in their heads.

erikpukinskis 16y ago

If you're doing what you love, you've already won. I feel a similar way pretty often. I have a couple of visions of software I want to build that are far out of my technical reach. I've been prototyping them, on and off, for about eight years too, depending how you count. They've all failed. I start with a flash of optimism, and code until I realize I was wrong. I then stop, and rethink things until I think of another guess. This is exactly how great accomplishments happen. You may be working on an impossible problem, or you may be working on a 20 year problem. Or maybe you're lucky and it's a 10 year problem. But even if your goal is impossible, it's very likely that if you work on it for 20 years, you will solve a 20 year problem. It might be slightly askew to what you thought you were solving, but it will happen. It's almost inevitable. Regardless, I wholly support what you're doing. Keep going. And read widely... explore... make sure you're playing in the margins of your field, and exploring odd projects far outside the scope of your work. It's those places that you will make that accidental discovery... that in all these years of trying to "beat the market" you made a fundamental discovery in artificial intelligence, or network theory, or whatever. And yeah, if you're doing what you love, you've already won.

krosaen 16y ago

I find it inspiring that he's stuck with it, and wonder how's he's made a living in the meantime

chasingsparks 16y ago
I take short-term jobs and contract work for a few months, then quit everything for full-time research. Usually the jobs I take happen to require learning something tangential but possibly useful. It's a happy coincidence that I am perfectly happy with the absurd level of frugality required. (I live in a place that most people call the "House of Squalor. It's really not dirty, but it is semi-dilapidated.)

malloreon 16y ago

It sounds like despite your ideas of what failure is, by age 25 you've learned and applied more than most people ever will.

singular 16y ago
Agreed. I'm 28, soon to be 29 and I have achieved a great deal less. That kind of experience, though sour, is pretty awesome, at the very least you should be able to get yourself into a highly paid quant position at a finance firm.

jey 16y ago

Seems like it'd be a better bet to apply all your skills to creating value (wealth) instead of just trying to push it around a smidge more efficiently. In other words: find a real problem and solve it, increasing global wealth in the process, and collect the economic reward.

HSO 16y ago
Hm, I accidentally upvoted this so I may as well leave a comment. The dichotomy between "creating" wealth and "pushing it around" is false, in my opinion. Lots of people make it and it's intuitive, I know, but figuring out the proper value of things, trying to make forecasts about the future, even profiting from differences that shouldn't be there, all this is part of the capitalist meta-system called financial markets. One should not confuse the agency issues, borderline-legal information advantages, or even outright fraud in the markets with the basic activity that sustains them. Think about the alternative: Would you like to have to sit with a dimwit credit officer (= bureaucrat) from a big bank who can then effectively veto your plans? Or would you rather have a lively market that decides on such things? Do you want to do your IPO in a public equity market or would you prefer to try to break into an insider club of old rich people who may never come to understand your funny newfangled business model ("what's internet?!"). German sociologist Georg Simmel wrote an interesting book about money, one of the ideas I liked is that its fungibility dissolves the need for relationships, making us free to choose them, and thus effectively increasing personal freedom. The same, I'd venture, is true for capital markets. By spreading transactions around, they increase your options and freedom as entrepreneur. And for markets to work, you need people actually _in_ the market, like it or not. Figuring out value (as opposed to the going price), identifiying and exploiting inefficiencies, trying to make more intelligent forecasts of likely developments, all these _are_ "real problems" in a market economy. Having intelligent, hard-working people working on them makes the system as a whole more intelligent.
- jey 16y ago
  I fully agree that finance is useful because it increases market efficiency. The problem is that like most fields, most of it is uncreative legwork that doesn't provide much leverage to get rich from. So a smart creative person should consider whether they're falling into the cognitive trap of "I want to make money, finance deals with piles of money, so I should apply my creativity to finance". It's just a crapshoot unless you have some real information asymmetry[1] you can exploit to make money. This could be in the form of noticing some sort of systematic inefficiency, having a unique and effective strategy of gaining market information to identify mispricings, etc. Otherwise you'd be better off finding some unmet need to creatively address and avoiding the cognitive trap that leads from "I want money" to "I should go into finance". 1. http://en.wikipedia.org/wiki/Information_asymmetry http://en.wikipedia.org/wiki/Information_asymmetry

cjy 16y ago

You've learned a lot, so you're obviously not a failure, but I think your post illustrates how draining a winner-take-all field like quantitative trading can be. You can never be sure if your wins or losses are real or just mirages caused by noise. Humans need the emotional encouragement that incremental success brings. Nothing get's me excited to work more than a good day of sales or a thoughtful comment from a customer. It takes a special type of personality to keep on persisting in the face of repeated failure. Certain start-ups also have winner-take-all properties. Anything that relies on network externalities or advertising revenue is more risky. The rewards can be big, but I prefer markets where the product's value isn't a function of the customer base and success is more incremental.

deleted 16y ago

[deleted]

edw519 16y ago

After eight years, I have nothing concrete to show for my efforts. So sad, so true. This is what happens when you play a zero-sum game. Even if you win, someone has to lose. Here's an idea. For the next 8 years, why don't you do something that helps others by making the pie bigger for everyone instead of just trying to game a bigger piece of a smaller pie for yourself. When you do for others, you always have something concrete to show for your efforts. It may not be what you expected, but it will definitely be something worthwhile, I promise you.

aspiringsensei 16y ago
This is a very valid way to put this, and it helped me. Thank you.
olliesaunders 16y ago
This is what happens when you play a zero-sum game. Even if you win, someone has to lose. I don't see the causality. He has yet to succeed because it's an incredibly difficult problem domain. There are problem domains of similar difficulty that benefit people that he could have chosen and still be in this position.
- kristianp 16y ago
  Almost anything where you create something would be better. Then again he/she is creating software and algorithms.
  - lowkey 16y ago
    Yes, but to what end? When it comes to creating things that have value, I like objective standards and my favorite definition is Buckminster Fuller's: "Real wealth is whatever nurtures and accommodates human life." Anything else is pointless. Incidentally, a recent study claims that 60% of UK workers feel that their work is of no use to society. Financial Services, Telemarketing, Spammers, High-Frequency Traders and many other 'professions' would fit this description. Here is the link: http://www.reddit.com/r/Economics/comments/cjgx7/60_of_uk_workers_feel_their_work_is_of_no_use_to/ http://www.reddit.com/r/Economics/comments/cjgx7/60_of_uk_wo...

motters 16y ago

Many smart people have tried to "beat the markets" and failed. From what I've seen of it, this sort of activity is at best a form of numerology. Also, in trying to move money around slightly more efficiently than the next market analyst you're not really generating wealth as such, and your ideas and activities will leave no lasting value for society. You may not be a failure, since the skills acquired in analysing trends can be utilised in other areas. The process of having promising looking preliminary results, only to find that they are illusory is familiar to many scientists. Ultimately it comes down to what you really want to do with your life. Is your life only about moving money, or something else?

Kaizyn 16y ago
If I'm not mistaken, I believe the idea is to set up some automated trading bots that just simply make money for you on a continual basis. This constant revenue stream allows you to forget about making money and allowing you to use those resources to either fund a life of ease and luxury or other pursuits such as "making the world a better place".

narrator 16y ago

I tried to beat the market using all kinds of AI stuff off and on for a number of years. Learned a lot about AI techniques, and became a better programmer, which helped in my day job, but never got anywhere. It's a fool's game. Besides, most of these HFT systems hold positions for only a few minutes or less. There's no way someone without a huge account, to cover commissions from trading so much, and access to a lot of expensive real-time data streams could possibly make any headway at it. Not to say there aren't people who do make money at trading, but they are swing traders and/or people who've figured something out about the market that the market doesn't know yet, and thus has not revealed at all in the price movements of the stock. Since price movements are the primary input into a lot of trading algorithms, most trading algorithms are blind to these developments.

mbateman 16y ago

A lot of people are leaving comments critical of his goal of beating the market as playing a zero-sum game, just pushing wealth around, having no value for society, etc. This strikes me as dubious and high-handed, to say the least. First, as to the value of the endeavor. He is trying to, in his words, "come up with an algorithm that finds pockets of profitability in a cloud of probable randomness." If he could really do this, it would create capital flow, the ability to borrow, and so on, for people that could turn it into wealth that would not otherwise have access to that wealth. Financial markets serve a real function, and beating the efficient market hypothesis would improve their ability to fulfill that function. Second of all, he's extremely interested in the problem. He's obsessed with modeling complex systems and sees this as a sort of holy grail. Surely the fact that he's interested in it and values it counts for a lot, especially if you grant the above point, that there is a value to what he's doing. He has his reasons for being interested in it, and it's a productive endeavor. I think it's false that his lack of progress, or the reasons that he's frustrated at his lack of progress, have to do with these aspects of his goal. It's more that he's working on an extremely hard (arguably insoluble) problem where it's hard to see incremental gains. This is a problem that ambitious people in many different fields face.

rue 16y ago
All this is only undermined by the fact that the financial sector and the stock market in particular involves mostly fictitious money. There is no problem that it solves that could not also be solved by saying "here, I conjured you a million dollars out of thin air, act as if you had it". The market is, as it were, a MUSH. There are certainly plenty of talented people working in finance. That talent is just completely wasted.
- barrkel 16y ago
  All money is fictional, in that it only has value to the degree that people agree and believe it has value. This is no less true for the US dollar than it is for gold, silver, oil, cattle, etc.
  - 16y ago

aspiringsensei 16y ago

My mother was down on the floor of the NYSE the other month, and she got the opportunity to corner a trader who she noticed was relying heavily on algorithms. She asked "What's the difference between a good trader and a good algorithm?" The trader said "nothing.If you're buying 1000 shares, it's probably better to simply put in a market order. A large hedge fund seeking to buy many hundreds of thousands - where the difference between paying 8.30 and 8.31 a share matters - algorithms can actually be helpful." The issue here is the definition of "trader." For this guy, trader is a person who executes large trades for an investment manager. A trading decision is not the same as an investment decision, and in that vein many trader's motivations are simply to buy a desired quantity of a financial instrument as cheaply as possible. If you're buying 1000 shares, it's probably better to simply put in a market order. A large hedge fund seeking to buy many hundreds of thousands - where the difference between paying 8.30 and 8.31 a share matters - algorithms can actually be helpful. If you're buying 1000 shares, it's probably better to simply put in a market order. A large hedge fund seeking to buy many hundreds of thousands - where the difference between paying 8.30 and 8.31 a share matters - algorithms can actually be helpful. Humans are probably better computers for figuring out which stocks to buy and sell to my mind, but there's certainly a place for algorithms if they are not too high-minded.

cowmoo 16y ago
You are right that for a individual investor focused on the long term, a algo-order probably wouldn't make a difference in comparison to a market order with your E-Trade/Schwab account (most likely, your order-flow won't go directly to the market anyways; it is either crossed internally, or re-routed to a broker/dealer that's paying retail brokers for the order flow such as Timberhill). However, I respectfully disagree that humans are better at making trading decisions than computers. The world of algo trading can be divided into two sides, a) high frequency traders, who through fast cancel-and-replace limit orders and colo with the market centers, try to act as virtual market-makers (or scalpers, depending on your perspective), b) buy-side institutional fund managers who want to complete their orders, without HFT predators and negative market pressure. Large block orders are spliced into small lots (i.e., VWAP) and sent to the market using intentional limit price over time to hide the movement of a huge buy or sell order on the market. Human beings might be better than machines at picking single stocks for long term investing (although most people are probably still better off investing in an ETF). But that's not how sell-side traders make money in the first place. Guys like GS/MS/Timberhill make money by having the unfair advantage of faster execution speed, more capital and specialized trading algo's against the small retail investors.

bbers 16y ago

To beat the market you HAVE to predict what trades others will make in the future. It is easy to use math to make it appear to be a numbers game but that is just abstraction. The reality is that people buy and sell stocks for 1000s of different reasons.

chasingsparks 16y ago
I use much less math than your average quant. I prefer simulations with a lot of agents. Think Game of Life, not partial differential equations.

naner 16y ago

This type of thing has been done before: http://en.wikipedia.org/wiki/FatKat_(investment_software) http://en.wikipedia.org/wiki/FatKat_(investment_software) http://www.idsia.ch/~juergen/finance.html http://www.idsia.ch/~juergen/finance.html http://www.technologyreview.com/blog/guest/25308/ http://www.technologyreview.com/blog/guest/25308/ I'm sure there are other examples. I think part of the problem is this guy thinks he can get rich quickly with just the right set of programming instructions.

ruang 16y ago
Nice quote from one of the links' links, from NY Times: But as these new techniques proliferate, some worry that promotion is outpacing reality. These techniques may be better for marketing than stock picking.

Loy 16y ago

The term "complete failure" is more revealing about your draconian idea of success. You should take a look at your own words : "they have never found their Holy Grail" [therefore, they joined the ranks of Academics as a testimony of failure]. If you're running after the Holy Grail, you have no choice but to be a hero. What kind of pressure is that ? In fact, most of the answers are in front of you, in the way you express yourself. For instance, the myths of the Holy Grail and Tantalus are ones of insoluble challenge and dilemma. Therefore, when you are wondering if the Holy Grail exist, where it is, and if you'll ever reach it... You are probably asking the wrong questions. Instead, you should ask yourself : "Who am I ?". Do you want to be rich, or do you want to "beat the markets" ? That's 2 different things, one is a mere consequence of the other. Where is your priority ? As it's not clear from your writing, I suppose it's not clear in your mind either.

chasingsparks 16y ago
The title was unnecessarily provocative, a reflection of my mood at the time of writing it. (It's Sunday; I'm hung-over; and I was looking at the results of a four day simulation run that were ambiguous.) My priority is in "beating the markets" because it's a fascinating challenge. I also have interests that lie beyond satisfying my intellectual curiosity but require wealth. I find the idea of using my own wealth to solve problems (e.g. avenues of cancer research) far more appealing than fund-raising or requiring public funds.

wangwei 16y ago

Even people with very high level math knowledge fails to predict the market (remember the two noble prize scientists who failed?).Since you're out of school when you were 17, I doubt you'll able develop complex schemes (unless you are a very very rare genius).

xenophanes 16y ago
Obviously he has the aptitude to learn things on his own. Sounds like he can learn better/faster on his own time than the large majority of students do at a school. So no need to hound him with pessimism for having an unconventional but pleasant lifestyle.
chasingsparks 16y ago
I was not out of school at 17. What caused you to infer that? (I'd like to correct it if it sounds that way.) I am actually starting a masters/Ph.D. program in the fall in computational social sciences. Like I mentioned in the post, I am interested in social systems in general, not only markets. (However, I do find markets to be the most interesting.)

muhfuhkuh 16y ago

Look to none other than Long Term Capital Management[1] back in the 90s to see how dangerous a game it is to either algorithmically predict or automate a thing as emotionally wrought and, at times, institutionally manipulated as the market. [1] http://en.wikipedia.org/wiki/Long-Term_Capital_Management http://en.wikipedia.org/wiki/Long-Term_Capital_Management

jrockway 16y ago
What? Their time-tested bond arbitrage algorithms worked great. They beat the market month after month. Only when they tried speculating on equities did they fail. And even then, their positions only took short-term losses that they would have recovered from had they not been leveraged 100-to-1. Their algorithms were correct, but they didn't have enough capital to see their positions through to the end. The LTCM story is about greed and leverage, not algorithmic trading. Plenty of money can be made trying to beat the market: it pays the salaries of the hundreds of thousands of investment bank employees around the world. The problem is, you personally can't do the same things they do, because you don't have enough capital. Raise a billion dollars and your days of commuting to work are over.

noonespecial 16y ago

Look on the bright side. Many people dedicated their lives to the search for a process to turn lead to gold. This seemingly fruitless search lead to all kinds of neat discoveries. You've learned lots of valuable things that you wouldn't have learned otherwise as well. Keep searching if you must but start benefiting from that knowledge. The real secret about turning lead to gold is that it is possible, but by the time you have mastered the skill to do so, you no longer need to or want to.

gvb 16y ago
Alchemy and transmutation of lead to gold is a pretty apt analogy for John Nelson's quest to turn his model of the market into gold. a) The alchemical belief in transmutation was based on a thoroughly wrong understanding of the underlying processes. John's (and other's) lack of success indicates a faulty model. b) Ironically, it transpired that, under true nuclear transmutation, it is far easier to turn gold into lead than the reverse reaction, which was the one the alchemists had ardently pursued. Nuclear experiments have successfully transmuted lead into gold, but the expense far exceeds any gain. http://en.wikipedia.org/wiki/Nuclear_transmutation http://en.wikipedia.org/wiki/Nuclear_transmutation
- chasingsparks 16y ago
  If you look at the first sentence of the article, it is linked to the subheading of a blog post that explained my intellectual evolution -- the subtitle of which was "The Contemporary Philosopher’s Stone" which is how I refer to this problem. (http://en.wikipedia.org/wiki/Philosophers_stone http://en.wikipedia.org/wiki/Philosophers_stone) You're not the only one drawing the comparison ;)
  - khafra 16y ago
    Sounds like in that frame, most of the consoling messages here are along the lines of "our gold is not the common gold."

pvdm 16y ago

http://en.wikipedia.org/wiki/Monroe_Trout http://en.wikipedia.org/wiki/Monroe_Trout

ctingom 16y ago

Remember, no man is a failure who has friends. -- Clarence in It’s a Wonderful Life

arthurdent 16y ago

I find myself excited over the preliminary results of my increasingly sophisticated simulations This. The vast majority of the money being made in the market is being made by people with what you may consider a laughable lack of "sophistication". Start simple. Complex modeling of stock market behavior only makes sense if you think the stock market has a complex underlying structure.

chasingsparks 16y ago
By sophisticated, I meant refined and cultivated, not devoid of simplicity. It was a poor choice of words. At some point in my development, I started to eschew more complicated forms of AI and machine learning, in favor of simpler agent-based models with "laughably" simple assumptions. I think it's the proper direction, then again, the title I chose for the post was not arbitrary. ;)

10ren 16y ago

Question: has anyone managed to do this? (eg. do hedge funds do this, or do they combine algorithms with their own ideas and quasi-insider knowledge, plus massive amounts of timely data - inaccessible to an individual). My old supervisor (ACM fellow) comforted my failure to create AI by saying that lots of other clever people hadn't done it either. I have a feeling that you are attempting something more difficult that AI - ie, algorithmic inference without the data that your competitors have. Because of their better information, they will find a regularity (eg. 0.01% dip on Wednesdays) quicker than you - and trade it away. It would be awesome if you could do it though - forget the philosopher's stone aspect of gold-for-nothing, this would be a very cool intellectual achievement. Idea: focus on small markets/exchanges (unpopular metals, unpopular countries, even non-financial markets, like eBay) that don't attract the attention of big players. Easier to find arbitrage there; you can make some actual money, and then move onto a bigger market with your learnings, earnings, and confidence. Sort of a niche strategy. Personal Note: My AI research depressed me, and I (wisely) decided to shelve it. It's been incredibly more satisfying to work with others, to achieve doable goals, that benefit others. Fortunately, AI comes up in everything; and my current approach (of not doing AI directly) may well be the most effective way to approach the goal. It might be the same with you - both markets and inference are universal. EDIT So, Tantalus = low-hanging fruit + moving-goalposts http://en.wikipedia.org/wiki/Tantalus#Story_of_Tantalus http://en.wikipedia.org/wiki/Tantalus#Story_of_Tantalus

pradocchia 16y ago
Simply put, if anyone has managed to do it, you aren't likely to hear about it. That said, I know of two: First was a math prof that would play the markets every couple years when he needed a little extra cash. Second was someone who had isolated patterns of market intervention and figured out how to profit from them. But that's more like insider trading than beating the market algorithmically.

noahc 16y ago

John, 95% of people would agree with you. But they (and you) are wrong. What is your end game? Do you want to be worshiped on HN or do you want a six figure job some where? If you want either of those, you could just as well write up all you know spit it out in blog posts and free eBooks. Someone is going to recognize your skills and take it from there. At this point the worst thing you can do is keep all your failures to your self, because then no one can benefit from them.

danbmil99 16y ago

I suspect you could get a pretty good job as programmer for some hedge fund. Not only would it leverage your command of this domain, you might learn a few things that will explain why it is difficult to impossible for you to win in a vacuum. There are social engineering aspects to this game that you have not been exposed to yet. Also, you may just not be sophisticated enough, or you have pursued a dead end approach. This is a very rich field of study. If it's your passion, I say stick with it, but expand your circle of knowledge and influence. And screw all this 'zero sum' crap, capitalism is the least zero-sum game humanity has ever come up with. And speculation, market-making, betting on winners & losers is one of the most important driving forces of capitalism.

WalterBright 16y ago

Any algorithmic method of beating the market can only be successful if it is a secret and if its trades are of insignificant volume. Otherwise, knowledge of this algorithm will be factored into the prices, defeating it.

arthurdent 16y ago
this is really not entirely correct. when things started melting down in 2008, it was largely attributed to the fact that hedge funds basically all had the same strategies in their book. When one started liquidating, many started losing value and liquidating in a disastrous feedback loop. There are plenty of strategies that are very robust, and in hedge-fund land, its not uncommon that many players are playing very similar hands. In fact, there even exist strategies that get better as more people run them.

rjett 16y ago

"Every few months, I find myself excited over the preliminary results of my increasingly sophisticated simulations, only to be disappointed a few short weeks later to find that I was simply wrong." Isn't this why algorithmic traders "refit" their algorithms on a daily basis?

daylast 16y ago

I hate to be the bearer of bad news but, based on the comments you've made in your post, you are most likely a smart fool (as you have so aptly put). In essence, you don't even know what you do not know. I've worked at top-tier Wall St institutions and have had first hand experience working with several of the largest quant hedge funds out there. Here's my advice: if you truly enjoy markets and are sure you want to take this path, you will be much better served by spending several years working at an established quant hedge fund. This will open your eyes to the theory, processes, data, and techniques required to earn alpha consistently and in a risk efficient manner. It's a long road and that road is full of extraordinarily hard working and intelligent people that one has to constantly compete against. If you can handle this though, you will learn a bunch and at some point in the future you can strike out on your own again. At that point you will at least have a chance of success. Good luck!

Tichy 16y ago

I've met people who claimed to make money with their algorithms, which really surprised me. Your post gave me an idea, though: What if instead of running your algorithm on the whole set of stocks, you create several random subsets. One of them has a good chance of being successful, at which point you can claim that your algorithm works and sell it for $$$. (I am not entirely serious, of course - but maybe some of the successful algorithms work that way?).

soyelmango 16y ago

It sounds like a trolling headline - he himself acknowledges the value of what he has learnt. Some suggested in his comments that he should join a firm so that knowledge and experience could be shared towards finding the algorithm[s]. His reply to Henrik, "I’ve been wary about taking jobs at firms like that because of IP concerns. I always believe I am on the cusp of something great, and wouldn’t want to share my algorithms", is a factor in his progress, or lack of. By working alone, he's more likely to earn 100% of nothing, when he could collaborate and earn a small % of a very large number. That said, if he finds the algorith alone, then big congratulations and respect are in order for sticking to his convictions.

MLnick 16y ago

Only very few people have commented that, you are only 25! (I guess given the nature of the HN community :) I don't think anyone can consider themselves a failure at 25, having spent 8 years learning a lot of programming, algorithms, data mining / machine learning and complex systems modelling. It would be easier to think that if you hated what you were doing. But it sure doesn't sound like you do. So you just need to decide what you really want to be doing. If it's algo trading, I concur with another poster that says go work for a hedge fund (or bank, or prop trading shop). It is super competitive, but they have the technology infrastructure and most importantly the capital. Getting in is not easy, but simply show them all your work (it doesn't work anyway, but is indicative of your skills and way of thinking). You will learn a lot, you may hate the people and environment, or you may love it even more. And then yes after a few years of experience you will most certainly be in a better position to go off on your own again (or do something totally different, by then you will really know if you like it or not!). Many top hedge fund managers / traders only started their own thing at 30, 35, 40, even 50... I know a dentist who became a prop trader. Anything is possible. A PhD would be a great option IF it's for the right reasons. But if you want to do a startup (sounds like you might quite like the idea and you posted here on HN, so...): - you already live on ramen, so no lifestyle change there; - bootstrapping something can't cost more than losing money with trading algorithms; - you already have many of the requirements: coding / technical skills, low-cost living circumstance, love to solve tough problems and a huge amount of tenacity in the face of failure and overwhelming odds; - bonus: your interest in social systems modelling etc ties in pretty nicely with what's big right now and in the near future. So if that is what you really want to do, go for it either alone (or find a co-founder), or find a small/medium startup to work for. To make the transition a bit more natural perhaps focus on ones that are data-driven and have machine learning / modelling at the core of their business. Think recommendations, systems modelling (www.flightcaster.com) and weather (www.weatherbill.com). There are many many examples of YC and other startup companies of this nature (many focused on the social network space). Good luck in whatever you do decide to do with the next 60+ years of your life. On your deathbed you can post about whether you think you are a complete failure or not.

dunk010 16y ago

Well, he might just not be smart enough. Large financial institutions and many smaller hedge funds reap countless millions from the markets and much of this comes from proprietary trading (well, all of it in the case of hedge funds.) There's a massive brain-drain into financial institutions of many of our best and brightest, all in search of making a fortune by the time they're 30. So on the one hand you have top students from top institutions applying their considerable intellect and training into applying advanced mathematical methods against an extremely complex problem. On the other you have a guy who tinkers around himself hoping after hope that the amateur (for that is by definition what he is) attempts that he makes at beating the market are going to some day work out and make him his fortune. Unfortunately the odds are stacked against you, and those are compounded by fact that you have little or no opportunity to employ any serious leverage. "I would have probably, or at least possibly, been wealthy by other means by now". At _best_ "possibly", I think. At 25 that is an incredibly arrogant statement to make.

retube 16y ago

His approach to this - the goal of beating the market - has been totally wrong. It sounds as if he's been at this completely solo - he mentions in the comments that he worked alone as he didn't want to share his algorithms. He would have benefitted enormously from working in an institution that specialises in this kind of work (hedge funds etc). Being largely self-taught and having worked alone since he was 17 pretty much guarantees he doesn't know anything institutional trading firms won't know or have studied - employing as they do advanced mathematicians, physicists etc to work at the cutting edge of algo research and trading. Not only has he missed out academically, he's missed out operationally. "Playing-the-game" is as big a part - if not bigger part - of trading than pricing/valuation. With the resources of a large institution you can learn to play the market in a way you can't possibly by yourself. This is in fact largely how they make money - consistently, year on year. Thirdly - even if he did discover a pricing inconsistency or whatever, it's doubtful he'd be able to leverage it in as profitable a way that he would be able to as a firm.

chegra 16y ago

"He[Dan Zanger] holds the unofficial record in trading stocks by turning $11,000 into more than $18 million in 18 months in 1999-2000. He grew that to an incredible $42 million in less than two years and has the tax receipts to prove it." http://www.investopedia.com/articles/trading/04/082504.asp http://www.investopedia.com/articles/trading/04/082504.asp "Williams won the 1987 World Cup Championship of Futures Trading from the Robbins Trading Company were he turned $10,000 to over $1,100,000 (11,376%) in a 12 month competition with real money. Ten Years later his daughter Michelle won the same contest" http://en.wikipedia.org/wiki/Larry_R._Williams http://en.wikipedia.org/wiki/Larry_R._Williams Somebody has some sort of algorithm.