5 ms·

This is unacceptable, obviously. It's a massive failure of their security testing protocols. I'm not particularly surprised that an vulnerability like this woul

This is unacceptable, obviously. It's a massive failure of their security testing protocols. I'm not particularly surprised that an vulnerability like this would get written into the code, it's an easy mistake for an inexperienced developer to make. But I'm not going to pile on the developer. We laud "separation of concerns" in our architecture, and this pattern applies to the organization of software development teams. I don't expect every developer to be aware of every vulnerability. But I do expect that a financial institution has a specialist somewhere that audits the code before it is sent for testing ("white box"), and then I expect them to have an independent audit team probe for vulnerabilities ("black box"). After the inexperienced developer has had his code rejected for various flaws, he will become quite aware of the obvious ways things like this can go wrong. Don't get me wrong, I expect that the vast majority of developers wouldn't make this mistake in the first place. But if you aren't a specialist, it is pure hubris to think that you write code that is hardened against all of the attacks out there. And if you have a vulnerability, it really doesn't matter if it's an embarrassingly simple vulnerability or one that requires sophisticated techniques to uncover and exploit. Either way, you're road kill. High-value targets like banks need two security specialists (the code audit and the penetration test) to accompany the development specialists. That's simple separation of concerns, and it works as well in team organisation as it does in code organisation.

raganwald 15y ago
Unrelated anecdote: When working with a substantial financial organisation, there were strict rules prohibiting my team from communicating with the code audit and other security teams. Going for coffee with one of the internal audit people would get you both fired. The external testing was done by a security consulting firm. Their reports were passed on to me with the identifying details stripped out so that I wouldn't even know who they were. Talk about "low coupling!"
- tptacek 15y ago
  It shouldn't matter to you who the external firm is. Stripping out their contact info, at least in the archives, is probably smart; it makes it easier for the company to keep multiple security firms working on their projects. Which, when you're a huge bank, is what you want to be doing.
- biot 15y ago
  No doubt the goal there is to prevent collusion between a developer who introduces a subtle but exploitable bug and the auditors/testers who might be interested in failing to find such a bug, for an appropriate percentage of course.
  - deleted 15y ago
    [deleted]
tptacek 15y ago
I don't know anyone at Citi, but I do know several firms that do audit & pentest work there, and I'm told it's high-volume --- in other words, it's likely every app gets tested. This is a startling miss. When you engage a software security firm to check out an app, you're trusting that their report is reasonably complete. As a practitioner, this is basically your nightmare scenario. You know in the back of your head that it's always possible you're going to miss things, but a vulnerability this trivial seems like the kind of thing you find on the first day of testing.
- joe_the_user 15y ago
  This is the kind of vulnerability that should be caught on the whiteboard before a single line of code is written. I know this isn't what you're talking about, but I hope that an institution that's protecting my money will formally specifying their protocol and not haphazardly putting their app together with "Test Driven Development". But maybe that's how they managed to be this stupid.
  - tptacek 15y ago
    Though there are no doubt very, very smart developers scattered across Citi, please bear in mind that the wiring-up-form-fields-using-J2EE jobs in large financials are among the least prestigious in our field. These are the types of apps that are among the first to get outsourced. Your idea that the input and output paths through a typical banking J2EE app were carefully whiteboarded seems mythologized, based on what I've seen at fisrv companies.
    - Robin_Message 15y ago
      Online bank websites are, at least in the UK, universally terrible. They seem to be just a very, very thin layer on top of the bank's back-end systems, giving such delightful results as being two different sorts of recurring payments and two types of non-recurring payments I can do from my online banking (actually, I can't set up or cancel every type from within online banking.) Do banks make so little from checking/current accounts that it isn't worth spending a dollar per customer to build a decent solution?
joe_the_user 15y ago
Hmm, Perhaps the problem is the "security specialists". I agree that just about any reasonably skilled developer knows to, at the absolute minimum, include a user-specific session key with every transaction. But I've also seen "security specialists" get a pass for a variety of bad behaviors. And we've heard of the complete incompetence of HBGary. My hunch is that the secrecy/obscurity that the passed for security in pre-Internet culture is exactly what allows total screw-ups like this to happen in these large organizations. The gold-plated consultants who are friends with the VP wind-up being far less competent than your average/decent developer.
- tptacek 15y ago
  There's probably a kernel of truth to your comment, but that last graf makes no sense. The "security specialists" aren't writing the code with vulnerable direct object references; the "average decent developers" are. It's true that there are consultants out there that don't improve the (poor) seaworthiness of typical developer output, but few of them are so bad as to make the situation worse. And, the notion that "good" developers turn out uniformly secure code is delusive. We're lucky to have gotten to work with some of the best run, most carefully recruited teams in the industry. You find terrifying things on all these engagements. My advice, again as a practitioner: if your security firm isn't finding terrifying things, at least on the first go-round with an app, ask very tough questions. Ask if they need an extra week or two to catch up and get real findings. Then ask which of the two of you should pay for those weeks. We are still finding code execution on a good chunk of our web gigs, even at the "good" companies.
  - joe_the_user 15y ago
    I worked at a company where we had an individual hired as a "security specialist" who wrote code and got more or less a free on his behavior and approach. I suppose that's different than a full security consultant.
    - tptacek 15y ago
      Your pentest firm should not be writing code for you. That conflict of interest is pretty obvious. If I had a consultant doing that for my company, I'd retain a rival firm to review the code, and make sure both the consultant coder and the pentester team knew about each other. If you told me someone at, say, iSec Partners (a firm I like) wrote something I was pentesting, I'd go f'ing nuts trying to find flaws in it. If you told me iSec was reviewing something I wrote, I'd stay up nights thinking of ways to shore it up.
InclinedPlane 15y ago
Sadly the problem is at a higher level than that, and not restricted to security. Today software development might as well be magic. There are no reliable objective methods for determining developer or software quality. The only method that's proven to work well is the subjective judgment of another "magician". The same applies to software security as well. This creates a boot strapping problem. If you're an organization of magicians then you have a comparatively easy time finding other good magicians. But if you don't have that hsitory and you still have the need you won't be able to know the quality of people you're hiring except on the least granular of scales (lots of failed and low quality projects). This is a big reason behind so much of the problems of software development in "the enterprise". Because people who know how to run a bank or a media company haven't the slightest clue how to run software development, and there generally aren't trusted 3rd parties that people can go to for help. Worse yet, there are perverse incentives at play. If you can't tell the difference between a good security person and a stunningly mediocre one, you are going to go with the cheapest one who you think is good enough (generally: solid job history, resume fully buzzword compliant, etc.)
- lifeisstillgood 15y ago
  >>> Today software development might as well be magic. There are no reliable objective methods for determining developer or software quality. The only method that's proven to work well is the subjective judgment of another "magician". Try replacing software development with writing english. I am going to stop banging on about this eventually, but reading and writing code is something that needs to be understood at all levels of a company before that company gets good at it. If the boss of the company and everyone who reports to him is illiterate, one cannot expect the memos and the policy documents and the amusing posters to be of any decent quality
johnconroy 15y ago
You're kidding me right? If this story is true, their "security" was retarded. Flat-out retarded. I mean, a child with a summer-camp's worth of coding experience could not make a shittier system. There is no way of justifying the people who built and maintained this system, at all. They should all be taken outside and beaten with a sack of door-knobs, then sent back to law school, or welding school, or somewhere the hell away from any kind of programmatic systems development.

raganwald 15y ago

Unrelated anecdote: When working with a substantial financial organisation, there were strict rules prohibiting my team from communicating with the code audit and other security teams. Going for coffee with one of the internal audit people would get you both fired. The external testing was done by a security consulting firm. Their reports were passed on to me with the identifying details stripped out so that I wouldn't even know who they were. Talk about "low coupling!"

tptacek 15y ago
It shouldn't matter to you who the external firm is. Stripping out their contact info, at least in the archives, is probably smart; it makes it easier for the company to keep multiple security firms working on their projects. Which, when you're a huge bank, is what you want to be doing.
biot 15y ago
No doubt the goal there is to prevent collusion between a developer who introduces a subtle but exploitable bug and the auditors/testers who might be interested in failing to find such a bug, for an appropriate percentage of course.
- deleted 15y ago
  [deleted]

tptacek 15y ago

I don't know anyone at Citi, but I do know several firms that do audit & pentest work there, and I'm told it's high-volume --- in other words, it's likely every app gets tested. This is a startling miss. When you engage a software security firm to check out an app, you're trusting that their report is reasonably complete. As a practitioner, this is basically your nightmare scenario. You know in the back of your head that it's always possible you're going to miss things, but a vulnerability this trivial seems like the kind of thing you find on the first day of testing.

joe_the_user 15y ago
This is the kind of vulnerability that should be caught on the whiteboard before a single line of code is written. I know this isn't what you're talking about, but I hope that an institution that's protecting my money will formally specifying their protocol and not haphazardly putting their app together with "Test Driven Development". But maybe that's how they managed to be this stupid.
- tptacek 15y ago
  Though there are no doubt very, very smart developers scattered across Citi, please bear in mind that the wiring-up-form-fields-using-J2EE jobs in large financials are among the least prestigious in our field. These are the types of apps that are among the first to get outsourced. Your idea that the input and output paths through a typical banking J2EE app were carefully whiteboarded seems mythologized, based on what I've seen at fisrv companies.
  - Robin_Message 15y ago
    Online bank websites are, at least in the UK, universally terrible. They seem to be just a very, very thin layer on top of the bank's back-end systems, giving such delightful results as being two different sorts of recurring payments and two types of non-recurring payments I can do from my online banking (actually, I can't set up or cancel every type from within online banking.) Do banks make so little from checking/current accounts that it isn't worth spending a dollar per customer to build a decent solution?

joe_the_user 15y ago

Hmm, Perhaps the problem is the "security specialists". I agree that just about any reasonably skilled developer knows to, at the absolute minimum, include a user-specific session key with every transaction. But I've also seen "security specialists" get a pass for a variety of bad behaviors. And we've heard of the complete incompetence of HBGary. My hunch is that the secrecy/obscurity that the passed for security in pre-Internet culture is exactly what allows total screw-ups like this to happen in these large organizations. The gold-plated consultants who are friends with the VP wind-up being far less competent than your average/decent developer.

tptacek 15y ago
There's probably a kernel of truth to your comment, but that last graf makes no sense. The "security specialists" aren't writing the code with vulnerable direct object references; the "average decent developers" are. It's true that there are consultants out there that don't improve the (poor) seaworthiness of typical developer output, but few of them are so bad as to make the situation worse. And, the notion that "good" developers turn out uniformly secure code is delusive. We're lucky to have gotten to work with some of the best run, most carefully recruited teams in the industry. You find terrifying things on all these engagements. My advice, again as a practitioner: if your security firm isn't finding terrifying things, at least on the first go-round with an app, ask very tough questions. Ask if they need an extra week or two to catch up and get real findings. Then ask which of the two of you should pay for those weeks. We are still finding code execution on a good chunk of our web gigs, even at the "good" companies.
- joe_the_user 15y ago
  I worked at a company where we had an individual hired as a "security specialist" who wrote code and got more or less a free on his behavior and approach. I suppose that's different than a full security consultant.
  - tptacek 15y ago
    Your pentest firm should not be writing code for you. That conflict of interest is pretty obvious. If I had a consultant doing that for my company, I'd retain a rival firm to review the code, and make sure both the consultant coder and the pentester team knew about each other. If you told me someone at, say, iSec Partners (a firm I like) wrote something I was pentesting, I'd go f'ing nuts trying to find flaws in it. If you told me iSec was reviewing something I wrote, I'd stay up nights thinking of ways to shore it up.

InclinedPlane 15y ago

Sadly the problem is at a higher level than that, and not restricted to security. Today software development might as well be magic. There are no reliable objective methods for determining developer or software quality. The only method that's proven to work well is the subjective judgment of another "magician". The same applies to software security as well. This creates a boot strapping problem. If you're an organization of magicians then you have a comparatively easy time finding other good magicians. But if you don't have that hsitory and you still have the need you won't be able to know the quality of people you're hiring except on the least granular of scales (lots of failed and low quality projects). This is a big reason behind so much of the problems of software development in "the enterprise". Because people who know how to run a bank or a media company haven't the slightest clue how to run software development, and there generally aren't trusted 3rd parties that people can go to for help. Worse yet, there are perverse incentives at play. If you can't tell the difference between a good security person and a stunningly mediocre one, you are going to go with the cheapest one who you think is good enough (generally: solid job history, resume fully buzzword compliant, etc.)

lifeisstillgood 15y ago
>>> Today software development might as well be magic. There are no reliable objective methods for determining developer or software quality. The only method that's proven to work well is the subjective judgment of another "magician". Try replacing software development with writing english. I am going to stop banging on about this eventually, but reading and writing code is something that needs to be understood at all levels of a company before that company gets good at it. If the boss of the company and everyone who reports to him is illiterate, one cannot expect the memos and the policy documents and the amusing posters to be of any decent quality

johnconroy 15y ago

You're kidding me right? If this story is true, their "security" was retarded. Flat-out retarded. I mean, a child with a summer-camp's worth of coding experience could not make a shittier system. There is no way of justifying the people who built and maintained this system, at all. They should all be taken outside and beaten with a sack of door-knobs, then sent back to law school, or welding school, or somewhere the hell away from any kind of programmatic systems development.