10 ms·
The $10 Hedge Fund Supercomputer That’s Sweeping Wall Street
- carsongross 12y agoLTCM wasn't a warning, it was prophesy.
- adultSwim 12y agoA prophesy which became manifest a few years back.
- wjnc 12y agoThe gap that firms are trying to bridge is _huge_. I work in finance (slow finance: insurance), but finance technical enough to have our own PhD-level quants building code. They are running simulations taking hours, so scaling up on # cores would seem the first step to take. But it's not that the quants don't know parallelization, it's the IT-architecture not allowing to use AWS at all. And well, many cores on your own premises is hugely expensive because they are idle most of the time. And as a sidestep, most quants are not the first you think of to woo architectures' support. (This is probably easier for firms that trade for own profit.) Point being: improvements in AI / machine learning and other statistical techniques are huge, and analytical demand exists, but 'old corporations' need cultural change before they can start working with and embedding startup-like technology in their workflows.
- mcguire 12y agoThis is pretty common outside finance as well. "We can't let our data touch any one else's network!" And, of course, we can't bother to get our IT group on the smarts wagon, so we're massively inefficient all the way around.
- Someone1234 12y agoI'm playing devil's advocate a little: There are sometimes legitimate legal hurdles to having your data hit external networks. For example, if you're a medical insurance company, then you have to get everything HIPAA legal.
- sologoub 12y agoLooks like AWS is HIPAA complaint at least for some services: "Customers may use any AWS service in an account designated as a HIPAA account, but they should only process, store and transmit PHI in the HIPAA-eligible services defined in the BAA. There are six HIPAA-eligible services today, including Amazon EC2, Amazon EBS, Amazon S3, Amazon Redshift, Amazon Glacier, and Amazon Elastic Load Balancer." http://aws.amazon.com/compliance/hipaa-compliance/ http://aws.amazon.com/compliance/hipaa-compliance/ Notably, I don't see Elastic MapReduce. It seems that if you wanted to run heavy data crunching, you'd have to run you own cluster on EC2.
- jtbigwoo 12y agoI've been on projects where we used Rackspace cloud servers or Amazon EC2 for HIPAA-compliant insurance enrollment systems. There was extra paperwork and we did a bunch of security testing, but it wasn't too bad. In both cases the projects were controlled by marketing departments who were much more concerned with cost per customer than with maintaining total control over every aspect of the system.
- mcguire 12y agoSure, sure, and some of our data is even sensitive in that way (cough, HR, cough). But I'm sure Amazon or someone else can handle that kind of requirement. We can't even open the discussion, though.
- the8472 12y ago> "We can't let our data touch any one else's network!" In some industries that can be necessary if you want to stick to the letter of the law, especially here in europe. You can outsource the processing, but then your provider has to meet fairly stringent certifications, you have the necessary "submitting to 3rd parties for purpose-specific processing" disclaimers in your privacy policies and a signed data processing agreement covering personal data compatible with national laws, which vary between countries. The microsoft ireland email case[1] also made some data protection ombudsmen suggest that the whole safe harbor deal with US companies should be considered null and void because they cannot effectively guarantee the safety of your data, even more so given the threat of national security letters + gag orders. And without that deal you can't use their cloud solutions for sensitive data. Personally identifying data has pretty high hurdles. Medical data (which might be processed by insurances?) raises even higher bars.
- sseveran 12y agoIn fact AWS will sign a HIPAA business agreement for medical data (http://aws.amazon.com/compliance/hipaa-compliance/ http://aws.amazon.com/compliance/hipaa-compliance/).
- kejaed 12y agoThey'll even help you with ITAR controls too: http://aws.amazon.com/govcloud-us/faqs/ http://aws.amazon.com/govcloud-us/faqs/
- superuser2 12y agoWhere is the line? Everyone seems to think "cloud" services threaten user privacy, but "cloud" is a marketing term. What about traditional VPSes? Managed dedicated servers? Ordinary dedicated servers? Leased servers collocated in a 3rd party datacenter? Owned servers collocated in a 3rd party datacenter? What if you use their "remote hands" service? Is there a meaningful difference between your leased office floor and your leased datacenter space? I would argue not. Do you have to own the land? What about leased vs. owned servers if they're in your office? Even if you have outright owned servers on outright owned land (unlikely b/c mortgage, but whatever), how are they managed? In many small businesses (including medical practices), they're managed by a "small business IT consulting" company which has both remote and regular on-premise access. You're trusting them to the same extent you would be trusting AWS. Is there a meaningful difference? What about the proprietary software you run? A large Electronic Medical Records package which probably contains extremely sensitive data about you is technically deployed onsite, but deployment and administration are performed remotely by the vendor's team in India. (I worked for a small biz IT consulting firm that supported the underlying hardware/Windows for one of these installations.) What about the Windows and other proprietary software with internet access that's literally everywhere? You may think you've achieved a morally superior position of trusting no one but yourself when you forego The Cloud, but I argue that's generally not the case except in the most extremely self-reliant cases (no vendor support, no contractors, no proprietary software). Those cases are not at all representative of the average business which refuses to go AWS.
- vosper 12y agoYeah. The company I work for was acquired by a big player in the industry, and one of the first things they wanted to do was move us off AWS, because the COO sees Amazon as a competitor and thinks that they'll go snooping in our databases to get a competitive advantage [1]. We managed to stave it off by coming up with a vague plan for how we'd move critical infrastructure to hardware, then shelving it indefinitely. AWS was (and is) critical to our velocity - unlike every other team in the company, we don't wait 6 weeks to start working on something while IT eventually gets around to writing a hardware purchase order and racking the gear. We don't compete for "the staging environment" because we can have as many environments as we want. [1] This is a ludicrous idea. My two main rebuttals are: - AWS takes in more revenue than our company, and they would destroy that business if it ever got out that their engineers were snooping in customer data - Netflix is a much more direct competitor than we are (they compete with Amazon Video) and also a much heavier user of (and cheerleader for, even) AWS
- nimblegorilla 12y agoIt might be time for tinfoil hats, but maybe Amazon only competes with Netflix because Netflix demonstrated how awesome Amazon's servers work for video delivery?
- bashinator 12y agoNetflix doesn't use AWS for the actual video delivery.
- throwawayaway 12y agooh no, never engineers. always maintain a layer of plausible deniability, those engineers might turn coat and rat the organisation out you see.
- hinkley 12y agoI really love it when I have to head up a project that costs the company $100-250,000 in developer time and the only reason we have to do it is that we can't get the business to buy us another $10,000 in hardware. Actually I lied. What I really love is when one customer threatens to walk if they have to buy a $3000 machine to run our new version, and the whole team spends 1-2 months doing nothing but trying to keep that account. When we could just gift them a server and save ourselves a mountain of grief and a ton of development resources for, you know, landing new accounts. If someone could start teaching basic math in business school, that would make me so happy.
- raverbashing 12y agoOf course, why have them pay a couple thousand dollars for an AWS installation when they can buy a "corporate solution" for "3 zeroes more"
- saganus 12y agoOne totally uninformed question: Do you think this could be solved with stacks of Parallela boards? Just curious, because that's kind of like the target market for the boards right? Cheap , multi-core processing?
- wjnc 12y agoTechnically: sure. But I think you are touching some different skill sets going on a continuum from (1) 'sequential simulations on regular machine' to (2) 'parallel simulation on very high end, regular machine', all the way to (3) 'massively parallel on non-regular machine'. From (1) to (2) you can manage with existing staff and limited time. From (2) to (3) is quite the change in mindset, training et cetera.
- saganus 12y agoI guess that makes sense. It does seem like a more exotic solution. Especially considering the amount of different skills that whoever is building such system would need to have. Maybe in some years we'll see if the Parallela paradigm takes off and then maybe there will be more quants out there capable of using such boards.
- sbenario 12y agoThat's one of the problems we're trying to solve with Ufora. Because it's so much more difficult to write "high performance parallel simulations", we want to make it so you can write whatever code you want, and let the platform figure out how to make it fast, parallel, and distributed.
- jacquesm 12y agoI'd look at checking if a GPU is a good match for the problem before thinking of even more exotic solutions.
- virmundi 12y agoSo could you take a page from Pixar or Seti and use company laptops while they're idle? Even if it was just on the quants laptops, it could help out. There would be no PO for the hardware. Shouldn't take to long to set up a simple system, but that could be the over optimistic American in me.
- jdboyd 12y agoWhere does Pixar say they use company laptops while they are idle?
- mryan 12y agoI've not read about Pixar doing this, but I found it interesting that IKEA does: http://www.cgsociety.org/index.php/CGSFeatures/CGSFeatureSpecial/building_3d_with_ikea http://www.cgsociety.org/index.php/CGSFeatures/CGSFeatureSpe...
- virmundi 12y agoYea, I can't find Pixar doing this. I thought I remembered an article from years ago. Other companies did it as well. https://books.google.com/books?id=QxAEAAAAMBAJ&lpg=PA51&ots=MJAXAN7zms&dq=rendering%20movie%20idle%20pcs&pg=PA51#v=onepage&q=rendering%20movie%20idle%20pcs&f=false https://books.google.com/books?id=QxAEAAAAMBAJ&lpg=PA51&ots=... But they might since RenderMan supports it, http://renderman.pixar.com/resources/current/tractor/tractor_nimby.html http://renderman.pixar.com/resources/current/tractor/tractor...
- raverbashing 12y ago> Yea, I can't find Pixar doing this Maybe because it doesn't make sense to copy a lot of model/texture data to/from a slow laptop for processing.
- njharman 12y agoThe data is small compared to processing required. Laptops aren't that slow. They are as fast as the fastest "servers" from a few/several years ago. Finally, most employees proly don't have laptops, OP misspoke.
- tdees40 12y agoConfirmed. I'm in insurance too - variable annuities specifically - and it's the most perfect problem for parallelization. In practice everyone maintains their own in-house grid (which is massively expensive and idle 90% of the time), and the technical staff lacks any serious chops to tackle GPUs. In practice the only way we've found to solve this is to go around corporate IT, so we do things like buying GPU cards and jamming them into boxes ourselves.
- ethbro 12y agoAs someone related to the field, I see a fundamental lack of quality in in-house IT when faced with problems of this magnitude. Which isn't unreasonable. IT evolved to be able to provision systems and roll out approved software. Maybe there's a development team or two to develop in-house extensions or software (though always viewed as a cost center). Therefore, the department the organization's rules are set up to funnel you through is simultaneously incapable of fulfilling this need. In practice, this leads to a ton of business specialty + devOps types from anyone who can hack Python in order to Get Stuff Done. (For those of you that don't deal with 1,000+ headcount organizations, be thankful ;) )
- tdees40 12y agoI think the big thing at the finance companies I've been at is that corporate IT was created to enable people to use Microsoft Office applications. There is no real provision for how to deal with developers writing code, and forget about hardware.
- ethbro 11y agoExactly, I think the telling question to any corporate IT team (not the head of all corporate IT) should be "When did you last identify, trial, and then make available to users a software package / process / cloud service without first receiving a request from management that you do so?" In my experience, the answer is usually never. Corporate IT exists to fulfill requirements from above, not to explore how things can be done better.
- 12y ago
- GFK_of_xmaspast 12y agoIf most of your cores are idle most of the time, you need to be doing things like searching bigger state spaces.
- mseebach 12y agoIT shops are inherently conservative, that's their job. The trick is to get the higher-ups interested - they have the power to redirect IT, and are inherently (if laggingly) interested in making employees as effective as possible. Funnily enough, a fluff-piece in Bloomberg is a pretty good way of achieving it. Tomorrow, some manager will pop into a quant's office with this article in his hand and ask "Are you doing this? Why not?" and that's how the ball gets rolling on opening up. Once upon a time, getting remote access to email was a "this will never happen", then an executive saw someone with a BlackBerry at the golf club. Once upon a time, getting email access on anything other than a BlackBerry was a "this will never happen", then an executive saw someone with an iPhone.
- Florin_Andrei 12y ago> IT-architecture not allowing to use AWS at all IPSec all traffic, including internal traffic between instances? Encrypt all volumes with keys not written anywhere within the cloud? That should do it, no?
- mariusandreiana 12y agoUnrelated: Salut Florin, cred ca ne stim de pe RLUG. Veneam pe la tine in Drumul Taberei sa facem chestii pe linux. Daca vrei sa mai keep in touch, am adresa in profil. Numai bine!
- deleted 12y ago[deleted]
- PublicEnemy111 12y agoIs it common to use prop trading strategies for trading a personal portfolio? Or do quants almost exclusively work at hedge funds
- tdees40 12y agoI've never met a quant trader trading for their own account. It requires some infrastructure that would be hard to justify unless you have lots of money to invest, and you're really good at it.
- Bostonian 12y agoIt is common for successful quant traders to be partners (have equity) in their firms, which means they are "trading for their own account" to some extent.
- dopamean 12y agoI think what he meant by "for their own account" was something smaller and personal. Not part of a larger pool containing hundreds of millions of dollars.
- ddeck 12y agoDefinitely not common, but I run my own quant strategies for my own portfolio on a fully automated basis and have done so for several years. Every other quant develper/trader I've met works at a fund or sell-side firm.
- xedarius 12y agoWould you mind detailing the platform/portal that you use?
- ddeck 12y agoNot sure what you mean by portal, but I wrote my own execution system that I run on a server located close to my broker and the exchanges. Market data is via a consolidated feed and execution via my broker to a variety of venues. I don't run any high frequency strategies so 100ms from exchange data to execution is fine for me. I use R for most research and strategy development.
- deleted 12y ago[deleted]
- bra-ket 12y agoUfora DSL: http://kb.ufora.com/index/language/documentation.html http://kb.ufora.com/index/language/documentation.html
- DominoDataLab 12y agoHi guys, I'm a founder at Domino Data Lab (one of the other companies mentioned in the article). I'm happy to answer questions anyone has. I can also corroborate the opinions in the thread here: many of our customers are in finance and insurance, and many of them don't want data leaving their networks at all (hence the need for an on-prem offering).
- walterbell 12y agoIs the on-prem offering a combination of "OSS private cloud" and a proprietary analytics / machine learning API?
- DominoDataLab 12y agoWe don't offer a proprietary ML library or API -- we run arbitrary user code, whatever languages and packages you want to use (Python, R, h2o, dato, compiled C++, etc.). The infrastructure piece is like a private cloud, yes.
- CyberDildonics 12y agoI've always been confused as to what takes a long time to calculate in finance. What are the common difficult problems that aren't getting solved?
- gourneau 12y agoSensai which is mentioned in the article is hiring Java AI/NLP folks in Palo Alto http://sensai.workable.com/jobs/19235 http://sensai.workable.com/jobs/19235 The CTO Monica Anderson also runs the bay area AI meetup group
- jchrisa 12y ago2006 called, they want their technology back.
- bd_at_rivenhill 12y agoI've met some of the guys from Ufora and seen a demo of their technology; very impressive. I've always been leery of the claim that you can create smart software to make the problems of writing parallel code magically disappear, but what they've done has made me rethink some beliefs related to working with big data at scale. Definitely a company to keep an eye on.
- sbenario 12y agoGlad to hear you like it! We think it's pretty cool ourselves :-)
- sbenario 12y agoPM @ Ufora here. Happy to answer any questions you might have!