7 ms·

It's easy to throw shade when Bloomberg writes an article that puts the blame on "a staffer". Having worked near some of these systems, the engineering and proc

by ynniv 4y ago

It's easy to throw shade when Bloomberg writes an article that puts the blame on "a staffer". Having worked near some of these systems, the engineering and process are actually quite good. How many companies publish their private network topology, service p99.9 in microseconds, and detailed pricing on the open web? They're in a painfully competitive global market that's ambivalent to names on buildings. In a week or so there will be a comprehensive internal post mortem, and every engineer in the company will read it because that's why they work there. "The staffer" will not be named, nor will they be fired. The process will be changed. The systems will be changed. You probably haven't heard of Pillar, but the NYSE in your head was replaced by some pretty amazing, distributed, low latency systems. The culture is to over-engineer, over-provision, plan for black swans. And test. That it works. Test that it scales. Test that backups work. Test, test, test. firmitatis, utilitatis, venustatis. This failure was due to daily testing. Sometimes things still fail. That's true anywhere. In most places your failures don't make the papers, and accidents are swept under the rug. That doesn't happen at NYSE for obvious reasons. They're not building large language models (that I know of), or self driving cars (pretty sure on this one), but they're a modern, cutting edge, "soft" real-time engineering shop. If you haven't looked already, you might find something interesting there: https://www.ice.com/careers https://www.ice.com/careers

jacquesm 4y ago
Just naming a 'staffer' though seems to already be a way to apportion blame to a segment of the employees, insulating management from what was done. Named or not doesn't really matter, clearly blame is being assigned.
- bostonsre 4y ago
  Yea, it sounds like an issue with process and automation. It shouldn't have been possible for the staffer to make a mistake that would cause this.
  - jacquesm 4y ago
    Precisely. It is never just one error. At a minimum two and if you really stare at this sort of thing long enough it isn't rare at all to discover a whole chain of them. The only difference with all the times that it went right is that this time everything was aligned 'just so'.
- ynniv 4y ago
  I think those are Bloomberg's words, or their paraphrasing of the grapevine. The high level people that I knew there weren't petty, and everyone was of the opinion that it didn't matter who clicked the button: we were all in the same boat.
ngz00 4y ago
I worked there and I can say that this is not accurate at all. It is very much a blame culture. I've seen people fired for less severe incidents. Beyond the core technology of the Pillar engine, the place is not comparable to a modern tech company in almost any way.
- throwaway122095 4y ago
  As somebody who worked with them as a client, I can confirm this. There is currently a spec-level bug with their core Pillar engine and it was essentially bounced between several different teams and ultimately ignored as nobody's problem.
  - ynniv 4y ago
    Unlike all of the "modern tech company" problems which are never ignored and only solved when someone's problem goes viral on social media. They're a big company, some groups are better than others, some customers get more attention than others.
  - mynameisvlad 4y ago
    So basically like any other medium to large company? This doesn't sound unique in the slightest.
    - uoaei 4y ago
      I would think that the company being a securities exchange would factor into the analysis. Don't you?
      - mynameisvlad 4y ago
        How does them being a securities exchange in any way affect the analysis of their software engineering practices? They're not some special snowflake, they can suffer the same software engineering and business process issues as other companies.
anonred 4y ago
Serious question: If someone is smart and capable enough to work on tangible things like AI systems or self-driving cars, why should they choose the NYSE outside of pure monetary reasons or affinity for a “modern” tech stack?
- DontchaKnowit 4y ago
  Serious Answer : There are problems in Fintech that are absolutely worth solving for altruistic reasons. One that I think is very important and might even need to incorporate AI is this : Larger financial institutions have access WAY more and WAY higher quality data surrounding stocks and options. For example, publicly available SEC filings contain extremely useful information about companies. Professional traders have access to services which provide this data accurately in programatic form (like an API). Us normal people have only the SEC filings themselves, which are enormous documents. It would be impossible to read them fast enough to ever catch up on all of them in the last say year. There are free APIs, but they are absolute dogshit and provide incomplete and inaccurate information. If someone could democratize this and provide this info for free or cheap to the public, it would be an enormous benefit to the general public.
- misja111 4y ago
  You're not making it easy to get any answers, if you cut out 2 of the main reasons for people in general to change jobs.
  - anonred 4y ago
    I consider these things table stakes when choosing a job.
- ynniv 4y ago
  I'm not recruiting for them, just sharing my experience. I included their careers link for people who might be interested because I know they're always looking for good engineers.
- hinkley 4y ago
  There’s also geography. You go far enough East and you have mostly public sector or defense jobs. A little smattering of insurance data processing. And then fintech. illinois.edu has one of the top rated CS programs, but once you graduate there are not a lot of options but to move to one of the coasts, or move back/to Chicago and try your luck there. Second City has a good deal of fintech.
BirAdam 4y ago
I worked for NYSE’s parent, ICE, and I have to agree with this. While there were many things I didn’t like about working there, the tech and the management weren’t involved in those things. A similar problem to this happened while I was working there, but it was on Endex and not NYSE. Management spoke with the responsible party, but he wasn’t fired and no punitive actions were taken against him. The blame game also wasn’t played. The team just decided to provide more eyes on the process, change the interface of the tools a bit, and move on. The company itself did face hefty fines for the screw up tho. Ultimately, the issues at ICE/NYSE are due to a highly bureaucratic structure and to onerous regulations forcing parts of that structure to exist. Given those two problems, I think ICE. does extremely well.
bob1029 4y ago
I've thought about getting into this... The stuff they work on is so incredible to me. Here's a quote from their Pillar product page: > Up to a 95% Reduction in Latency: The roundtrip latency on NYSE Pillar order entry sessions via Pillar matching engines has been reduced from ~592μs to ~32μs for FIX and from ~96μs to ~26μs for Binary, getting client orders into the market much faster. With a 92% improvement in the 99th percentile latency results, clients can also have more confidence in improved performance consistency regardless of market conditions. Reading stuff like this makes my current work feel stupid by comparison.
- Razengan 4y ago
  > Reading stuff like this makes my current work feel stupid by comparison. It makes our economic system seem stupid. Jesus, we're not calculating astrophysics or quantum mechanics. A made-up system should not require or depend upon this kind of speed or precision. Maybe we should chill. Reminds me of those pro StarCraft players who keep unnecessarily clicking the mouse to keep their APM (actions per minute) stat high.
  - DontchaKnowit 4y ago
    Absolutely agree. but "liquid markets are important" or something. rolls eyes This is just another step in the endless journey of widening the gap between your average Joe and someone with access to high level financial services.
    - bob1029 4y ago
      This is what enables products like robinhood to exist - platforms explicitly targeting the average joe investor.
- davidf18 4y ago
  [dead]
- nubb 4y ago
  wonder how they measure this and is this smart engineering from the exchange or just new fast network gear.
hackernewds 4y ago
Curious why the link to ICE.COM?
- hadlock 4y ago
  ICE owns NYSE and several other exchanges. ICE stands for Intercontinental Exchange. ICE is the IT administrator for these exchanges. I helped sell some router management software to them a while ago. ICE is fairly new, NYSE used to be independent. That changed sometime after 2008.
blibble 4y ago
do they still do UDP packet loss replay over email?

by ynniv 4y ago

jacquesm 4y ago

Just naming a 'staffer' though seems to already be a way to apportion blame to a segment of the employees, insulating management from what was done. Named or not doesn't really matter, clearly blame is being assigned.

bostonsre 4y ago
Yea, it sounds like an issue with process and automation. It shouldn't have been possible for the staffer to make a mistake that would cause this.
- jacquesm 4y ago
  Precisely. It is never just one error. At a minimum two and if you really stare at this sort of thing long enough it isn't rare at all to discover a whole chain of them. The only difference with all the times that it went right is that this time everything was aligned 'just so'.
ynniv 4y ago
I think those are Bloomberg's words, or their paraphrasing of the grapevine. The high level people that I knew there weren't petty, and everyone was of the opinion that it didn't matter who clicked the button: we were all in the same boat.

ngz00 4y ago

I worked there and I can say that this is not accurate at all. It is very much a blame culture. I've seen people fired for less severe incidents. Beyond the core technology of the Pillar engine, the place is not comparable to a modern tech company in almost any way.

throwaway122095 4y ago
As somebody who worked with them as a client, I can confirm this. There is currently a spec-level bug with their core Pillar engine and it was essentially bounced between several different teams and ultimately ignored as nobody's problem.
- ynniv 4y ago
  Unlike all of the "modern tech company" problems which are never ignored and only solved when someone's problem goes viral on social media. They're a big company, some groups are better than others, some customers get more attention than others.
- mynameisvlad 4y ago
  So basically like any other medium to large company? This doesn't sound unique in the slightest.
  - uoaei 4y ago
    I would think that the company being a securities exchange would factor into the analysis. Don't you?
    - mynameisvlad 4y ago
      How does them being a securities exchange in any way affect the analysis of their software engineering practices? They're not some special snowflake, they can suffer the same software engineering and business process issues as other companies.

anonred 4y ago

Serious question: If someone is smart and capable enough to work on tangible things like AI systems or self-driving cars, why should they choose the NYSE outside of pure monetary reasons or affinity for a “modern” tech stack?

DontchaKnowit 4y ago
Serious Answer : There are problems in Fintech that are absolutely worth solving for altruistic reasons. One that I think is very important and might even need to incorporate AI is this : Larger financial institutions have access WAY more and WAY higher quality data surrounding stocks and options. For example, publicly available SEC filings contain extremely useful information about companies. Professional traders have access to services which provide this data accurately in programatic form (like an API). Us normal people have only the SEC filings themselves, which are enormous documents. It would be impossible to read them fast enough to ever catch up on all of them in the last say year. There are free APIs, but they are absolute dogshit and provide incomplete and inaccurate information. If someone could democratize this and provide this info for free or cheap to the public, it would be an enormous benefit to the general public.
misja111 4y ago
You're not making it easy to get any answers, if you cut out 2 of the main reasons for people in general to change jobs.
- anonred 4y ago
  I consider these things table stakes when choosing a job.
ynniv 4y ago
I'm not recruiting for them, just sharing my experience. I included their careers link for people who might be interested because I know they're always looking for good engineers.
hinkley 4y ago
There’s also geography. You go far enough East and you have mostly public sector or defense jobs. A little smattering of insurance data processing. And then fintech. illinois.edu has one of the top rated CS programs, but once you graduate there are not a lot of options but to move to one of the coasts, or move back/to Chicago and try your luck there. Second City has a good deal of fintech.

BirAdam 4y ago

I worked for NYSE’s parent, ICE, and I have to agree with this. While there were many things I didn’t like about working there, the tech and the management weren’t involved in those things. A similar problem to this happened while I was working there, but it was on Endex and not NYSE. Management spoke with the responsible party, but he wasn’t fired and no punitive actions were taken against him. The blame game also wasn’t played. The team just decided to provide more eyes on the process, change the interface of the tools a bit, and move on. The company itself did face hefty fines for the screw up tho. Ultimately, the issues at ICE/NYSE are due to a highly bureaucratic structure and to onerous regulations forcing parts of that structure to exist. Given those two problems, I think ICE. does extremely well.

bob1029 4y ago

I've thought about getting into this... The stuff they work on is so incredible to me. Here's a quote from their Pillar product page: > Up to a 95% Reduction in Latency: The roundtrip latency on NYSE Pillar order entry sessions via Pillar matching engines has been reduced from ~592μs to ~32μs for FIX and from ~96μs to ~26μs for Binary, getting client orders into the market much faster. With a 92% improvement in the 99th percentile latency results, clients can also have more confidence in improved performance consistency regardless of market conditions. Reading stuff like this makes my current work feel stupid by comparison.

Razengan 4y ago
> Reading stuff like this makes my current work feel stupid by comparison. It makes our economic system seem stupid. Jesus, we're not calculating astrophysics or quantum mechanics. A made-up system should not require or depend upon this kind of speed or precision. Maybe we should chill. Reminds me of those pro StarCraft players who keep unnecessarily clicking the mouse to keep their APM (actions per minute) stat high.
- DontchaKnowit 4y ago
  Absolutely agree. but "liquid markets are important" or something. rolls eyes This is just another step in the endless journey of widening the gap between your average Joe and someone with access to high level financial services.
  - bob1029 4y ago
    This is what enables products like robinhood to exist - platforms explicitly targeting the average joe investor.
davidf18 4y ago
[dead]
nubb 4y ago
wonder how they measure this and is this smart engineering from the exchange or just new fast network gear.

hackernewds 4y ago

Curious why the link to ICE.COM?

hadlock 4y ago
ICE owns NYSE and several other exchanges. ICE stands for Intercontinental Exchange. ICE is the IT administrator for these exchanges. I helped sell some router management software to them a while ago. ICE is fairly new, NYSE used to be independent. That changed sometime after 2008.

blibble 4y ago

do they still do UDP packet loss replay over email?