8 ms·
Show HN: Investorsexchange.jl – parse trade-level stock market data in Julia
Backstory:
I wanted to play with intraday stock data but couldn't find a free dataset anywhere. IEXCloud [1] offers API access to 1-minute granularity intraday historical price data, but I was worried that it could get expensive or unwieldy to build up a substantial dataset via API calls. Plus, IEX gives out their raw data for free.
I probably should have just used the IEXTools python library [2] to parse IEX's raw data dumps, but I was working on a Julia project, so it felt more thematically appropriate to build a new tool from scratch.
I haven't been actively using InvestorsExchange.jl a lot lately, but it's made me the proud owner of a 50GB SQLite DB dump covering several years of trade data, and I think it would be awesome if I could help folks in the HN community more quickly build up this kind of dataset for their own curiosity or research.
Feedback is also greatly appreciated!
[1] https://www.iexcloud.io/docs/api/#historical-prices https://www.iexcloud.io/docs/api/#historical-prices
[2] https://github.com/lvfrazao/IEXTools https://github.com/lvfrazao/IEXTools
- nwah1 4y agoInvestor Sex Change?
- shafyy 4y agoYes, I'd also say that the name/domain is a bit unfortunate.
- deleted 4y ago[deleted]
- tlarkworthy 4y agoI am pretty sure it's a deliberate inside joke from expertsexchange days of yore
- reustle 4y agohttps://www.theregister.com/2001/03/16/transsexuals_drawn_to_expert_website/ https://www.theregister.com/2001/03/16/transsexuals_drawn_to... Takes me back to the days of ExpertsExchange, the biggest predecessor to Stack Overflow. They were essentially forced to change their domain to Experts-Exchange.
- MisterJoe 4y agoI am here for the comments only. :)
- seer 4y agoLike my other favorite play on words like that - the famous “Pen island”
- luismedel 4y agoI entered this post knowing I would upvote this comment.
- tonetheman 4y agoI too am here for this reason also. I would change the name. It is an issue. Put a dash in it or an underscore... just something.
- agapon 4y agoUnderscore is (was?) illegal in domain names.
- daenney 4y agoIt's not illegal in domain names. Take a look at the SRV type in the domain name system. It's underscores galore. However, for host names or for a URL's host component, you're limited to hyphens. In this case though it's just the name of a Julia package[1], which isn't bound by either the rules of domain names or host names as .jl is not a TLD. [1]: https://pkgdocs.julialang.org/v1/creating-packages/#Package-naming-guidelines https://pkgdocs.julialang.org/v1/creating-packages/#Package-...
- dougSF70 4y agoYou would be surprised by just how many investors are looking for this type of surgery...many many people.
- raffraffraff 4y agoDomain names of the old internet: - penisland.net (sells pens) - expertsexchange.com (renamed to experts-exchange.com) - therapistfinder.com (find a therapist) - whorepresents.com
- bobajeff 4y agoI remember one called penismighty.com. Which was a silly online community similar to, well nothing, but I guess the closest thing would be somethingaweful.com it even had alternative themes to the site which would lean into the different interpretations of it's name which was awesome.
- revolvingocelot 4y agoNot just the old internet! Ever considered visiting the Sierra Nevadas? Well there's a wonderful, outdoorsy place in California that's just waiting to welcome you: Lake Tahoe! Visit, uh, gotahoe.com, and keep in mind that the Nevada border, where certain things unrelated to outdoor sports are entirely legal, is right there.
- lelandfe 4y agohttps://wikipedia.org/wiki/Scunthorpe_problem https://wikipedia.org/wiki/Scunthorpe_problem
- deleted 4y ago[deleted]
- revolvingocelot 4y agoSort of? "Tahoe" isn't the problem. Sure, "hoe" is a substring of "tahoe" but it's pretty clear that "tahoe" is a placename in a string like "visitlaketahoe.com". IMO they absolutely knew what "gotahoe.com" would be parsed as, it's very clever. It'd be a Scunthorpe if the site was getting flagged for "hoe", but it's a parsing problem at the user layer, not censorship elsewhere.
- 4y ago
- tiborsaas 4y agoQuite a unique niche, I'd say go for this market.
- halotrope 4y agoNice, very nice!
- ninefathom 4y agoOh, good- I wasn't the only one.
- robofanatic 4y agohe was definitely successful in raising some eyebrows, peaking people's interests and ultimately got a lot of clicks!
- Stamp01 4y agoThey don't allow you to have jokes in here.
- canjobear 4y agoThey must use this at HRT
- matt123456789 4y agoHigh-f(r)equency Trading
- lukemerrick 4y agoHN is special, and you all here in this comment tree are the best of the best -- love the fact that everyone is having fun with this in such a civil way and historically knowledgeable way. For anyone who wants the naming backstory, InvestorsExchange.jl was originally IEXTools.jl, but Julia's package registration automatic name checks didn't like it ("Name does not meet all of the following: starts with an uppercase letter, ASCII alphanumerics only, not all letters are uppercase. Name is not at least five characters long") [1]. So to Wikipedia I went to find the non-acronym name of the IEX exchange, which is "Investors Exchange" [2]. Thank you all for helping me understand why IEX goes by IEX in all of their branding. [1] https://github.com/JuliaRegistries/General/pull/27989 https://github.com/JuliaRegistries/General/pull/27989 [2] https://en.wikipedia.org/wiki/IEX https://en.wikipedia.org/wiki/IEX
- nathias 4y agoamazing domain name
- temp_account_32 4y agoIt's not a domain name, there is no .jl TLD
- igorkraw 4y agoWhen you say they give out the raw data for free, does this mean you can download a batch of it? Or are you referring to the free API for the 5 years/30 day data on the bèage you linked?
- Jimajesty 4y agoLook at this guy, not commenting about the name.
- OJFord 4y agoReferring to these gzipped daily (two formats) dumps I think: https://iextrading.com/trading/market-data/ https://iextrading.com/trading/market-data/ API only needed for live (incl. past day) and more convenient granular access/computations it seems.
- PsySecGroup 4y ago
- 0xTJ 4y agoIt's good that the actual file is InvestorsExchange.jl, because without the E capitalized, that's definitely not how my brain parses it.
- gringo_flamingo 4y ago"Invest or sex change" is how my brain initially parsed it, lol. Was your initial reaction the same?
- freilanzer 4y agoIs that a threat?
- vineyardmike 4y agoIs it an opportunity?
- throwaway787544 4y agoI upvoted this just to keep the name on the front page longer.
- littledev 4y agoReminds me of the time the company that owns BlackBerry decided to post some listings on the new-at-the-time .jobs TLD...
- Graziano_M 4y agoI interned at RIM back in the day and every single person I went to school with asked how I liked my RIM job.
- robertlagrant 4y agoMaybe that's why they were single.
- tailspin2019 4y agoI’ll get your coat.
- Wronnay 4y agoOne file has a size of nearly 5 GB... Every day two of these files get released... So nearly 10 GB every day. So if we download the raw data every day for one year, we would have 3650 GB just for one year... It would be interesting how much reduced the size of the processed data is compared to the raw data. You say that you have 50 GB of data spanning multiple years. How many years exactly?
- malshe 4y agoYes, this type of datasets are massive. We use TAQ for research work but almost never use the raw data as is. > So if we download the raw data every day for one year, we would have 3650 GB just for one year. A small correction - the stock market is open only about 250 days a year so with your calculations the raw data size will be 2500 GB. Still massive.
- lukemerrick 4y agoAh, this is a nuanced point I totally left off the README. Each raw file is ~5GB, but the raw files are a dump of network traffic from the firehose feed that tracks not just trades, but also updates to orders that do not result in trades. If you skip all of the algorithmic bots' constant updates to their bid/offer spreads and look at just the trades that clear, you can store years of raw trades in under 100GB.
- kruxigt 4y ago
- codeaken 4y agoWhen looking at the linked archives [1] it seems that they sit at around 4GB/day. Is that really all of the data that the US stock market generates a day? I thought it would be at least 10x that. [1] https://iextrading.com/trading/market-data/ https://iextrading.com/trading/market-data/
- jabberbyte 4y agoIts only data for flows going through IEX, less than 3% market volume https://iextrading.com/stats/ https://iextrading.com/stats/
- lakomen 4y ago
- deleted 4y ago[deleted]
- crmd 4y agoI want to see this pitch deck