15 ms·
A Look at How Traders and Economists Are Using the Julia Programming Language
- jefferson666 10y agojust like pandas in python, I want to give a quick impl in go lang for my backtest strategy system. now it is open sourced in github: https://github.com/qingtiandalaoye/GoDataframe https://github.com/qingtiandalaoye/GoDataframe could anyone give a performance test of it? many thanks
- VodkaHaze 10y agoAlways fun to point out is that the NY federal reserve switched their full scale model of the economy to Julia last year[1]. It's also open source, so data scientists interested in macroeconomics can feel free to play around with it. Moreover, quant-econ, an academic entity dedicated to scientific programming in the social sciences has a few notebooks in python and Julia [2] [1]https://github.com/FRBNY-DSGE/DSGE.jl https://github.com/FRBNY-DSGE/DSGE.jl [2] http://quantecon.org/notebooks.html http://quantecon.org/notebooks.html
- amelius 10y agoInteresting. What quantities do economic DSGE models measure for example? And how accurate are these models? Is there some way to play with these models? Or do you need lots of input data, which isn't freely available?
- VodkaHaze 10y agoDSGE models aim to be explanatory models of an economy. They are not the best at predicting (bayesian VARs do better forecasting, there is a tutorial on it in my quant econ link) but they turn out to do forecasting decently well, too. The reason that is is that purely statistical/predictive models of macro are subject to the Lucas Critique [1]. It says roughly, if you observe an economic relationship is happening, but you don't have an explanatory reason why it is, it's a bad idea to use it for policy prescription. The NY fed model has a whitepaper here [2], which should be accessible to the technical data scientist (technical, but not prohibitively so). Their readme points to a few posts on using it, I think the open source code comes with a csv for example input data. There should be publicly accessible macroeconomic data in a few places for you to play with it, say at FRED or the World Bank. I think Julia has a Stata-style api package for FRED data, making the data processing easier. Have fun! [1]https://en.m.wikipedia.org/wiki/Lucas_critique https://en.m.wikipedia.org/wiki/Lucas_critique [2] https://www.google.ca/url?sa=t&source=web&rct=j&url=https://www.newyorkfed.org/medialibrary/media/research/staff_reports/sr647.pdf&ved=0ahUKEwikuP3S45nQAhUl_4MKHUgPD7wQFggaMAA&usg=AFQjCNGrvKihOfAe268qPnOtRMvn3LG0vw&sig2=r8LBNM_himWcWbvLWfZ2gg https://www.google.ca/url?sa=t&source=web&rct=j&url=https://...
- economist42 10y agoThe foundation DSGE models are nothing like regular statistical or machine learning models. They are general equilibrium models, a type of mathematical model the falls under the microeconomics area of economics. DSGE try to predict how an economical system will evolve over time, given agents preferences, technologies and institutions, how much will be produced, consumed, traded, prices and how those behave over time, taking into account stochastic impacts, like idk oil price fluctuation. They are beautiful theoretically, since they are micro constructed. See, economists don't like to use aggregates only to make predictions and infer policy impacts due to something called Lucas critique ( https://en.wikipedia.org/wiki/Lucas_critique https://en.wikipedia.org/wiki/Lucas_critique ) which basically means that taking an inflation series create an ARIMA(x,y,z) to predict next weeks CPI is theoretically. invalid. The estimation however, is usually based on Bayesian hierarchical models, taking the constraints imposed by the theoretical model, I believe. Here is a good source for that https://www.newyorkfed.org/medialibrary/media/research/staff_reports/sr554.pdf https://www.newyorkfed.org/medialibrary/media/research/staff... DSGE have been criticized for their low predictive power, but it can be said that in that for mid to long term predictions they are more robust for monetary and fiscal policy use.
- conjectures 10y agoWasn't aware DSGEs were often formulated in a Bayesian framework. Learn something every day.
- dekhn 10y agoModels are just datastructures and algorithms; presumably, you mean there is an implementation of those, and that the Fed uses that implementation as the 'official' one. That's fine, but I prefer my models to be language-independent.
- gre 10y agoHow do you have language-independent data structures and algorithms that you can run on a computer?
- dekhn 10y agosorry, I should have made more clear: I want them to publish a model in a file format that can be consumed across a wide range of language. For example, YAML, protobuf, thrift, etc.
- deleted 10y ago[deleted]
- miguelrochefort 10y agoFad.
- sctb 10y agoWe've asked you already to comment civilly and substantively or not at all. Please stop posting like this.
- sebg 10y agoIf you want to learn more about Julia, here are Julia Conf videos / slides from past years: * http://juliacon.org/2014/ http://juliacon.org/2014/ * http://juliacon.org/2015/ http://juliacon.org/2015/ * Julia Con 2015 India - https://www.youtube.com/playlist?list=PL279M8GbNseuhXmZk9rWMqY8BGJr_UOxK https://www.youtube.com/playlist?list=PL279M8GbNseuhXmZk9rWM...
- optimali 10y ago2016 talks are also up - http://juliacon.org http://juliacon.org Overview - http://julialang.org/blog/2016/09/juliacon2016 http://julialang.org/blog/2016/09/juliacon2016
- e12e 10y agoSpeaking of talks on Julia, I'll again recommend: https://www.youtube.com/watch?v=dK3zRXhrFZY https://www.youtube.com/watch?v=dK3zRXhrFZY "Julia: to Lisp or not to Lisp?" from the European Lisp Symposium in May 2016. I found it was rather illuminating on Julia in particular and lisps and programming languages in general.
- agounaris 10y agoJulia is a very interesting language but big data processing is not a matter of the programming language itself, but how your platform and your architecture is put together. And nearly always your bottleneck is not the language itself..
- staticassertion 10y agoI think that if this were true, the data science libraries in Python wouldn't all be written in C under the hood, linking to Fortran, etc. Language can have a huge impact on performance with big data. Julia is just a small step forward. It still has performance problems where the JIT can't help much - like the DataFrame structure, which is effectively a black box and therefor hard to optimize.
- VodkaHaze 10y agoIf you need performance on these kind of structures, you can use structured arrays in python. The main reason I like the idea of Julia is that I don't see the problem of parallelism being addressed in python, at least not without cumbersome libraries (disclaimer: I haven't tried dask yet). If python had an equivalent to #pragma omp parallel for I wouldn't have Julia ready in mind for my next project that requires creating a high performance algorithm.
- hadsed 10y agoPython numerical libraries links to LAPACK/BLAS/etc because that's how people used to do scientific computing before Python. Also for scientific computing you're mostly CPU bound which means that no one would use Python if the libraries were slower than Fortran or C. But big data problems are different in the sense that you're typically bound by IO, so while it certainly helps to have fast code it wouldn't be the first problem you'd want to solve.
- urschrei 10y agoI would say they link to LAPACK/BLAS because there's nothing to touch the combination of speed and robustness they provide.
- arcanus 10y agoAnyone used Julia for something approaching HPC, e.g. in a distributed environment instead of MPI? I've heard plenty of buzz about it in the HPC community but haven't had a moment to play with it yet.
- KenoFischer 10y agoYes, there's been some scientific workloads that used Julia/MPI to great effect. I don't think the paper is out of review yet, so probably can't link to details, but expect news on that front very soon.
- Xcelerate 10y agoI have. I used it for all of the research I did during my PhD over the last few years. Some of my Julia code ran on Titan.
- wruza 10y agoI really liked Julia (and Juno) when tried it last year, but it was too math-aware as all [actually good] math-originated programming environments. Hope it will break that circle and get some general-purpose attention in nearest future. For me, it is very promising even without that HPC or whatever.
- s_q_b 10y agoMost financial firms need their quants to worry less about the new programming language hotness, and more about moving entire systems off unbelievably complicated Excel spreadsheets.
- markovbling 10y agoTotally - I used to work as quant for a boutique asset manager and the whole business was running on spreadsheets. Insane. Ended up putting it all in a database and developing an excel add-in to pull it from the database as array formulae. Used a great library called Excel DNA to develop the add-in using C# if anyone is interested.
- sandGorgon 10y agoJust a question-did you ever consider using Jupiter Notebooks? Or RMarkdown Notebook?
- markovbling 10y agoOnly jumped on the python train after I left finance but the reason we used an add-in is because you can build dynamic sheets with calculations that update when the inputs change (where the inputs were pulling from the database). So you could build a sheet that pulls in portfolio holdings for yesterday where yesterday updates each day and then compute performance and risk stats referencing the data cells in the sheet and it would all update. In that context it was just an easy way to build reports pulling data from a database but same applies to quickly doing one-off analysis in Excel pulling dynamic data from the database - guys in finance tend to not be programmers but they're really good at Excel. The add-in approach was really useful too because you could create function that returns the holdings of a portfolio to an array of cells (an array formula) and have a drop-down box with all portfolios that fed the input of the formula so that when you change the combo box, it changed the portfolio data and then everything recalculated off the back of that :)
- wz1000 10y agoIIRC, Standard Chartered achieved this by initially adding Haskell interop to Excel[0] and then moving to a custom GUI solution to replace Excel altogether. [0]- https://www.youtube.com/watch?v=hgOzYZDrXL0 https://www.youtube.com/watch?v=hgOzYZDrXL0
- sdegutis 10y agoWhy isn't LuaJIT instead? If they both get their speed from JIT, isn't LuaJIT faster than Julia?
- benjaminva 10y agoI have used both in the science context and found Julia much better suited for mathematical problems. Matrix/vector support built into the language is a huge advantage. On top, running code in parallel environments is better supported by Julia (Macros, scheduling, etc...). On top of it, Julia has an excellent Python interface which makes it possible to also use this huge ecosystem of libraries. Makes it easy to use matplotlib out of the box.
- cschmidt 10y agoI like Julia as well, but there is an interesting scientific library for LuaJIT: http://torch.ch/ http://torch.ch/ Anyone have experience with it?
- ulucs 10y agohaving worked with it, torch is amazing for neural network stuff (thanks to the nn rock); however it raises the same problems there are with Python. Lua is basically a wrapper around C/C++ and whenever you need to do complex stuff, you find yourself writing C/C++ rather than Lua.
- pjmlp 10y agoIsn't LuaJIT development stale?
- cryptrash 10y agoYes. Lua could have been the Julia of its time if they had decided to pursue a more advanced compilation system rather than trying to embed lua everywhere in everything. Luajit is fast because of the nature of optimizations upon the tables structure and single table architecture. Julia is a better, faster lua and I don't find myself missing being able to embed my code when I can just call into what I need FROM julia.
- williamstein 10y ago"Lovas adds that currently, the de facto standard numerical programming language when talking to banks and other finance firms is Python/NumPy."
- dx034 10y agoI doubt that. It's certainly the trend and a lot of new code is produced in Python. But most models will run on R/Matlab/C++/C#. I'd guess that most quant departments have less than 10% of their code in Python. But Python is the modern way to go, so every shop stresses that they use it and they expect every new hire to master it.
- thefastlane 10y ago... master it instead of C++ or in addition to C++?
- dx034 10y agoIn addition, although that's often not possible. You need people who can write C++ to maintain code. But it's hard to find because few people with a finance background have any experience with C++. So companies often end up developing in python and then passing it to developers who port the code to c++ for speed and to fit it in the existing framework. If they could be replaced with Julia, it would potentially save a lot of money.
- dekhn 10y agoit's almost the perfect language. The breaking issue for me was the choice to count from 1 (yes, I know there are pro arguments for counting from 1, but every other language I use doesn't do that).
- rrock 10y agoWhy limit yourself to zero or one? Julia now lets you use any start point you'd like. A fairly recent change, so you might have missed it. http://docs.julialang.org/en/latest/devdocs/offset-arrays/ http://docs.julialang.org/en/latest/devdocs/offset-arrays/
- dekhn 10y agoThe problem is that I maintain other people's code, and I don't want the language to add all those extra things. It's a big cognitive load to maintain, and leads to bugs in production software that is hard to debug. The julia docs mention that Fortran arrays can be indexed with various starting numbers. I've programmed and maintained millions of lines of Fortran over 20 years and I've never seen anybody use anything other than 0 as the starting index.
- ffriend 10y agoFortran, Mathematica, Matlab, R, Lua - all start indexing from 1 (you can find an incomplete list in [1]). For users coming from these languages (which is a large part of the Julia community) the cognitive load would be in the opposite direction. [1]: https://en.wikipedia.org/wiki/Comparison_of_programming_languages_%28array%29#Array%5Fsystem%5Fcross-reference%5Flist https://en.wikipedia.org/wiki/Comparison_of_programming_lang...
- dekhn 10y agoI suspect the number of C++/Python or Python/Fortran (where users use 0-based indexing) programmers is larger than Fortran +Math +Matlab + R + Lua. Anyway, I'm not defending my decision as a globally correct one- it only matter to me.
- hodder 10y agoAs a quant, I can't say I've heard of a single quant I know who uses Julia. It very well could be promising, but firms will be very hesitant to switch from Matlab, R, Python, C++, and VBA
- sndean 10y agoI can't find the talk that Stefan Karpinski gave where he said this, but it was something like: "We've been spending a lot of time working with firms to replace K code." Maybe the small percentage that uses K/Q/KDB+ are moving to something more readable?
- stuntprogrammer 10y agoBased on random sampling of those that contact me about such transitions, the latter is somewhat true. A couple have mentioned Julia but I wouldn't say it's the majority.
- qewbie 10y agoDo you have a link to the talk? It would not make business sense for Stefan to spend time replacing the small percentage, if it were indeed a small percentage. Actually, kdb+ is dominant everywhere on wall st, and the company mentioned in that article are retiring an old, discontinued version of k, not kdb+; perhaps that is the firm Stefan refers to? The debate about readability of k code has become boring - if you learn the language, you'll find it perfectly readable, elegant even.
- sndean 10y ago> Do you have a link to the talk? https://www.youtube.com/watch?v=QRWZBWwBVR0 https://www.youtube.com/watch?v=QRWZBWwBVR0 (I can't listen to the audio now so I'm not sure how much I misparaphrased.) And, you're right, I shouldn't have conflated KDB+ with K. > if you learn the language, you'll find it perfectly readable, elegant even. Just comparing K to something like Python.. Though, personally, it only became (somewhat) readable after playing with APL and J for a year.
- deleted 10y ago[deleted]
- Ftuuky 10y agoI'm really interested in learning more about data science. Currently attending moocs to learn R but I don't have much free time and it's a slow progress. Currently I work as analyst for an environmental consulting firm but I always wanted to work in something related to quants. I know I'm probably being naive but I want to change careers. Do you guys recommend that I keep learning R or should I take a closer look to Julia? Or both? I don't have a programming background but I do know statistics and Excel. Edit: fast, concise and useful comments in a matter of minutes, I love HN. Thanks!
- strictnein 10y agoCan't answer whether to learn R or Julia, but for a more general education on Data Science, this book might be a really good fit for you: Data Smart: Using Data Science to Transform Information into Insight https://www.amazon.com/Data-Smart-Science-Transform-Information/dp/111866146X/ https://www.amazon.com/Data-Smart-Science-Transform-Informat... All the work is done in Excel. I enjoyed the book quite a bit. Author is Chief Data Scientist for MailChimp.
- mi100hael 10y agoBecome fully confident & proficient in a single programming language before hopping around. If you get to the point where you can build a full application from scratch and answer other peoples' questions on SO or IRC with one, you'll be able to easily get up to speed with another should the need arise.
- cwmoo740 10y agoR is great for producing scientific papers, charts, and doing a vast array of "standard" statistical analyses. I have found python to be much nicer because it's more of a "real" programming language, but the numpy/scipy/pandas stack make it very easy to use for data science and financial analysis also. There's a (very) quick intro here: http://lectures.quantecon.org/py/pandas.html http://lectures.quantecon.org/py/pandas.html You can also look into quantopian for practice, but I don't recommend actually trading money. https://www.quantopian.com/ https://www.quantopian.com/
- max_ 10y ago
- mark_l_watson 10y agoI am not a quant programmer, but last summer I did a ten day job for an old customer in Julia and I had to learn the language as I worked. I thought the expressiveness and conciseness of the language was good and I liked the way the package manager used github projects; easy learning curve and REPL based development.
- edblarney 10y agoSomeone explain to me why we need a new language here? If they are trying to express ideas - wouldn't any language do just fine? They can 'compile' in an 'optimized manner' using something else ... But in terms of algorithmic expression? Unless they are writing true 'real time' stuff right for the chip ... why a new syntax?
- awm 10y ago> They can 'compile' in an 'optimized manner' using something else ... Thats actually the issue Julia attempts to solve as highlighted in the article. There are many languages that you can use to express ideas, but when you need the idea to run as fast as possible, you have to rewrite in C/Fortran/etc and then add bindings to to the language. Julia lets you write inline C, which I believe means that you can still express algorithms simply, but with some parts (trivially) optimized. (Not a huge julia user, but thats my take on it)
- edblarney 10y agoI understand that - but wouldn't a really smart compiler be able to do the same thing? And if you have to be able to write 'some parts' in C for speed ... then why not just use C? And have nice libs for whatever you are doing? Or intelligently distribute it across processes? I find it hard to believe that the 'financial industry' has performance requirements that have never been encountered in the entire rest of high tech before ...
- awm 10y agoAh, I see. I'm no expert, but I'd argue the response is "Maybe". I think this is what blaze and pypy try to do with python: they compile it so that certain things are optimized. The harder part is that you have to generally hint for the compiler to truly be the best... WRT to just why not use C, python/R/Julia are MUCH easier to prototype in than C (in my opinion, having used those languages). The "grail" is to be able to prototype quickly, and then identify the hotspots to speed up (which, in Julia's case, would be just inlining with C code).
- 10y ago
- blitzd 10y agoMy beef with Julia when I last tried it out was the lack of flexibility within the 'data munging' sphere. The assumption appeared to be that all of your data would come in to as UTF-8, whereas in my org we need strong support for Latin1/cp1252, and UTF-16LE BOM. Is this something that has changed recently, or are these traders are economists just working with more sane data?
- z92 10y agoPipe the data through a converter before feeding it into Julia.
- blitzd 10y agoWell - that's my point. I do use a converter, as it's built into python at every intersection that I need it, reading CSV, XLSX, database calls, etc. Having to stick python scripts in front of all my Julia scripts everywhere that it would perform data access doesn't really seem viable. Thus, my beef with Julia.
- jernfrost 10y agoNot a language issue per say then. It shouldn't a problem to add support for this. Julia is even designed from the start to work with Strings of different types. Something not common in most other languages I've worked with. Most string functions operate on AbstractString types. You can create whatever subtype you like implementing alternative string encodings.
- elcritch 10y agoBuilding on this comment, it's actually surprisingly straightforward to add your own encodings into the core language. Just add the appropriate types and extend the necessary base methods. In this case, it appears Julia 0.5 has a transcode method supporting UTF8/16/32. [1] 1: http://docs.julialang.org/en/release-0.5/stdlib/strings/ http://docs.julialang.org/en/release-0.5/stdlib/strings/
- blitzd 10y ago
- deleted 10y ago[deleted]
- sgt101 10y agoWhen Julia has an environment as good as Rstudio and a dashboard as good as Shiny, and a Spark interface that is as good as Sparkler and R-Spark then I'll be up for it. Writing good code in R is an effort, it's possible but the language doesn't make it easy. Julia makes me write good code off the bat and facilitates writing excellent, elegant code.
- claytonjy 10y agoThis is how I tend to feel about all non-R things; I've been using R for long enough to dislike a lot about the language, but the environment and tooling is too good to give up. Letting R be the first language I've gotten pretty good at is really spoiling me.
- pitaj 10y agoYep. In my short time working on an R project, I really enjoyed how everything was right there. I'd love to see more REPLs include package managers.
- astrochymist 10y agoRstudio is nice, no doubts there; however, I find myself avoiding language-centric IDEs. If one uses R most of the time, it's fine, but as someone who usually writes in multiple languages in a day, I find myself appreciating emacs' modes more (of which the Julia one is fine). It's also why I like the fact that the semi-official Julia IDE, Juno, isn't a standalone application (it runs in Atom and is nice, if one doesn't mind the performance of Atom...) edit: spelling
- sgt101 10y agoI liked the Julia mode in emacs, but dear god, plots, files, environment, help and viewer.. (what is viewer for?)
- IndianAstronaut 10y agoIt is the libraries which really make R. Easy decision trees, SEM, robust regressoon, etc.
- fredcy 10y ago"The importance of data and speed to information can't be understated in the capital markets ..." That says, it's less important than you could possibly state.
- rafinha 10y agodeja vu
- jefferson666 10y agojust like pandas in python, I want to give a quick impl in go lang for my backtest strategy system. now it is open sourced in github: https://github.com/qingtiandalaoye/GoDataframe https://github.com/qingtiandalaoye/GoDataframe
- farright 10y agoI've never used Julia but it reminds me of Go in that it attempts to replace Frankenstein systems that mix low level languages and high level languages/scripts, with a single modern, high level language with sufficient low level features and performance. However, in order to do so, they have to give up the generality of either language, and focus on a particular domain (numerical computation and web servers respectively). And because of this, the language never develops a big enough ecosystem. Because of this, I think Python combined with native C libraries is ultimately the better way forward and will be more successful.
- jernfrost 10y agoI really love Julia and see a bit potential outside of the scientific computing. I used to do all sorts of automations, smaller throw away programs etc in Python and Ruby, but I find Julia much more effective to work with. It also feels like a smaller language in many ways. You don't have to keep as much in your head, which makes it more effective to keep up with. The killer features for me, automating a lot of tasks is the awesome integration with the shell environment. Running and combining Unix commands is very elegant and getting hold of the output is done really smoothly. Secondly what I think saves a lot of time is that it really has ALL BATTERIES included. With Python that feels more theoretical. With Julia you just start writing the name of a function you think should exist in the REPL and it pops up. With Python you got to start guessing which package it might be located in. And that goes for stuff you use all the time like string manipulations, regular expressions, file reading and writing, process input and output etc. With Python functions often end up having unnatural names because you can't have naming conflicts. Julia handles functions with the same name but different argument types elegantly. Also there is no schizophrenia wondering if something is a function or method on an object. Everything is a function so that is easy.
- leethargo 10y agoI would also like to use Julia more for "throw away" scripts and general automation. Unfortunately, startup time (and sometimes compilation) make this a less fun endeavor than with Python etc.
- cutler 10y agoGiven Julia's pre-1.0 status, general immaturity and good Python interop wouldn't a saner approach be to use Python as your primary language and Julia for whatever needs to be fast?
- tekknik 10y agoSo how come not golang instead of Julia? I don't have anything against Julia myself but it seems like go with its maturity, easy cross compilation, static compilation and easy IPC/multithreading it would fit the bill quite well. Granted the GC is a bit slow but should be improved with the next release.
- whyrt12 10y agoGolang is far far inferior in many ways, for scientific programming. For starters, it doesn't have generics.
- deleted 10y ago[deleted]