9 ms·

Here is the working paper [1] if anyone is more interested in the details. The gist of it is that there are two ways to get insider transaction reports from EDG

by minimax 12y ago

Here is the working paper [1] if anyone is more interested in the details. The gist of it is that there are two ways to get insider transaction reports from EDGAR. One way is to scrape the EDGAR website (polling), the other is to sign up for a push service [2] where the filings are sent to you as soon as they hit EDGAR. You have to pay for the latter service. The paper calls the scraping solution "public" even though both solutions are open to the public (though you have to pay for the push service). According to the paper, it turns out (maybe not so shockingly) that the push service is better than the scraping/polling solution about half the time. The paper is written by economists and not technologists, and so there is no mention of CDNs or web caches a.k.a. the types of things most developers would suspect as being the most obvious source of the discrepancy. 1. http://online.wsj.com/public/resources/documents/SECDissemination.pdf http://online.wsj.com/public/resources/documents/SECDissemin... 2. http://www.sec.gov/info/edgar/ednews/dissemin.htm http://www.sec.gov/info/edgar/ednews/dissemin.htm edit: There is a WSJ article suggesting the price for the push feed is around $1500 / month. http://online.wsj.com/articles/fast-traders-are-getting-data-from-sec-seconds-early-1414539997 http://online.wsj.com/articles/fast-traders-are-getting-data...

lrm242 12y ago
Ah thanks for the link to the working paper, I had been looking for it. This jumps out from the footnote on page 7: "As far as we know, this time is not available on any publicly available database. We initially obtained these times using real time “scrapes” of the SEC EDGAR site. We subsequently used a collection of these times made available by the Tier 1 subscriber. Given that this entity’s business model depends, at least in part, on obtaining and disseminating these filings in the most timely manner possible, they have strong incentives to collect accurate information about when filings become available on the SEC website." Basically, clock skew is not accounted for at all and timings are potentially done from disparate and non-common observation points. It's quite possible that the delays they observe are strictly due to noise.
- ycombobreaker 12y ago
  Noise from clock skew and timings of half a second? They make a point that their information source for timing is from a subscriber which appears to trade on the information, so timing is probably accurate within 100ms. It sounds like this information can be used for a variant of a "news" trade, which can be extremely latency-sensitive.
  - lrm242 12y ago
    Clock skew in that the clocks used by the participant may or may not have been synchronized. There are, effectively, no details about how the timing is done on the news side and there is allusion to some use of EDGAR posting timestamps which are from an entirely different clock. Even more concerning, is that they map news events to market data using non-synchronized market data timestamps. It appears to be TAQ data. The timing aspect of the entire article is very poorly described but yet is fundamental to the claim they are making.
crdoconnor 12y ago
>You have to pay for the latter service. There are no public prices and you need to email some company for more details. With most businesses this means you contact a salesperson who will play an extended game of "how much you got?" and the pricing AND the level of service provided will vary widely depending upon your negotiating power. The dearth of information about this 'product' sold by a 3rd party suggests something shady is probably going on. >The paper is written by economists and not technologists, and so there is no mention of CDNs or web caches Caching is a pretty poor excuse. Caches can be invalidated. And CDN basically means caching by another name. It makes much more sense (both economically and technically) that they simply crippled the free feed. Especially since it is provided by a private third party who makes bank on the people who purchase the premium edition.
- lrm242 12y ago
  The SEC website explicitly states: "The subscription price is set annually by Attain, LLC, using a weighted average methodology based on the number of primary feeds for subscribers at the beginning of the each year. Subscriber organizations can be invoiced annually or monthly." There are two very detailed documents on the linked website that have both business and technical details along with specifications for the service. There is nothing shady going on here.
  - crdoconnor 12y ago
    No published prices + a seemingly HIGHLY profitable portion of a government service provided by a private company = something pretty shady. Even the contracts to run private prisons don't give this much opportunity to gouge. Were this service run on an at-cost basis would maybe cost $80 or so per month. $200 at most.
    - tptacek 12y ago
      "HIGHLY profitable"? Even at the highest price quoted on this thread, this looks like a rounding error --- in the sense that if you were a startup providing this information at these prices to its total addressable market, you'd have a hard time even getting funded. The number of firms that can profit from trading off realtime access to fundamentals is not large. It's a market where viable real products need to have customer lifetime values in the many millions.
jcfrei 12y ago
I think it's very unfortunate that they offer two different data sources which provide an unfair advantage to paying subscribers. Then again I don't think there's a single serious investor out there, who attempts to gain an advantage by scraping the crummy EDGAR site. Regardless I guess this is just what you get when you outsource a public service to a private company like EDGAR Online.
- minimax 12y ago
  I don't think EDGAR is outsourced. I think EDGAR is a project run by the SEC. There appears to be a separate company called EDGAR Online which is all about taking data from the SEC's EDGAR and making it more palatable for institutional investors. The only outsourced component here is the EDGAR Dissemination System (the subscription push feed of EDGAR filings), but that is outsourced to a company called Attain, LLC.
- eli 12y ago
  Distributing a digital document in a way where everyone in the world gets it at very close to exactly the same time sounds like an incredibly difficult problem.
  - DenisM 12y ago
    Does it have to be? I'm sure we can come up with something. Off the top of my head - distribute encrypted blob ahead of time, then broadcast the key using longwave radio from the South Pole.
    - kasey_junk 12y ago
      I'm not sure that lowers the bar out of "incredibly difficult". For instance, how would you feel if you heard the SEC invested in a bespoke technology for doing this just to mitigate a document timing issue only interesting to a very small number of market participants? I'd be upset about the waste of my tax dollars personally.
      - 12y ago
malandrew 12y ago
I'm wondering if data could be distributed ahead of time in an encrypted fashion and then only a key needs to be published/pushed, which is a far more trivial thing to get out to many people simultaneously.
- Bpit 12y ago
  I like this idea.

by minimax 12y ago

lrm242 12y ago

Ah thanks for the link to the working paper, I had been looking for it. This jumps out from the footnote on page 7: "As far as we know, this time is not available on any publicly available database. We initially obtained these times using real time “scrapes” of the SEC EDGAR site. We subsequently used a collection of these times made available by the Tier 1 subscriber. Given that this entity’s business model depends, at least in part, on obtaining and disseminating these filings in the most timely manner possible, they have strong incentives to collect accurate information about when filings become available on the SEC website." Basically, clock skew is not accounted for at all and timings are potentially done from disparate and non-common observation points. It's quite possible that the delays they observe are strictly due to noise.

ycombobreaker 12y ago
Noise from clock skew and timings of half a second? They make a point that their information source for timing is from a subscriber which appears to trade on the information, so timing is probably accurate within 100ms. It sounds like this information can be used for a variant of a "news" trade, which can be extremely latency-sensitive.
- lrm242 12y ago
  Clock skew in that the clocks used by the participant may or may not have been synchronized. There are, effectively, no details about how the timing is done on the news side and there is allusion to some use of EDGAR posting timestamps which are from an entirely different clock. Even more concerning, is that they map news events to market data using non-synchronized market data timestamps. It appears to be TAQ data. The timing aspect of the entire article is very poorly described but yet is fundamental to the claim they are making.

crdoconnor 12y ago

>You have to pay for the latter service. There are no public prices and you need to email some company for more details. With most businesses this means you contact a salesperson who will play an extended game of "how much you got?" and the pricing AND the level of service provided will vary widely depending upon your negotiating power. The dearth of information about this 'product' sold by a 3rd party suggests something shady is probably going on. >The paper is written by economists and not technologists, and so there is no mention of CDNs or web caches Caching is a pretty poor excuse. Caches can be invalidated. And CDN basically means caching by another name. It makes much more sense (both economically and technically) that they simply crippled the free feed. Especially since it is provided by a private third party who makes bank on the people who purchase the premium edition.

lrm242 12y ago
The SEC website explicitly states: "The subscription price is set annually by Attain, LLC, using a weighted average methodology based on the number of primary feeds for subscribers at the beginning of the each year. Subscriber organizations can be invoiced annually or monthly." There are two very detailed documents on the linked website that have both business and technical details along with specifications for the service. There is nothing shady going on here.
- crdoconnor 12y ago
  No published prices + a seemingly HIGHLY profitable portion of a government service provided by a private company = something pretty shady. Even the contracts to run private prisons don't give this much opportunity to gouge. Were this service run on an at-cost basis would maybe cost $80 or so per month. $200 at most.
  - tptacek 12y ago
    "HIGHLY profitable"? Even at the highest price quoted on this thread, this looks like a rounding error --- in the sense that if you were a startup providing this information at these prices to its total addressable market, you'd have a hard time even getting funded. The number of firms that can profit from trading off realtime access to fundamentals is not large. It's a market where viable real products need to have customer lifetime values in the many millions.

jcfrei 12y ago

I think it's very unfortunate that they offer two different data sources which provide an unfair advantage to paying subscribers. Then again I don't think there's a single serious investor out there, who attempts to gain an advantage by scraping the crummy EDGAR site. Regardless I guess this is just what you get when you outsource a public service to a private company like EDGAR Online.

minimax 12y ago
I don't think EDGAR is outsourced. I think EDGAR is a project run by the SEC. There appears to be a separate company called EDGAR Online which is all about taking data from the SEC's EDGAR and making it more palatable for institutional investors. The only outsourced component here is the EDGAR Dissemination System (the subscription push feed of EDGAR filings), but that is outsourced to a company called Attain, LLC.
eli 12y ago
Distributing a digital document in a way where everyone in the world gets it at very close to exactly the same time sounds like an incredibly difficult problem.
- DenisM 12y ago
  Does it have to be? I'm sure we can come up with something. Off the top of my head - distribute encrypted blob ahead of time, then broadcast the key using longwave radio from the South Pole.
  - kasey_junk 12y ago
    I'm not sure that lowers the bar out of "incredibly difficult". For instance, how would you feel if you heard the SEC invested in a bespoke technology for doing this just to mitigate a document timing issue only interesting to a very small number of market participants? I'd be upset about the waste of my tax dollars personally.
    - 12y ago

malandrew 12y ago

I'm wondering if data could be distributed ahead of time in an encrypted fashion and then only a key needs to be published/pushed, which is a far more trivial thing to get out to many people simultaneously.

Bpit 12y ago
I like this idea.