10 ms·

The Looting of ShapeShift

198 points by danielvf 11y ago

danielvf 11y ago
This is certainly the worst case scenario - your security officer installing remote access software on developers machines, stealing bitcoins from production, then selling the company source code, access credentials and access to the internal network to a Russian hacker. Building a security system to handle this level of attack is a whole level beyond stopping even determined external attackers. Are there any best practices guides on this? One thing that the article showed is the importance of external security review to deal with the threat of internal incompetence or evil.
- tlb 11y ago
  Ultimately, the founders of bitcoin companies need to understand security themselves or they won't be able to hire and manage a competent team. Despite all the theoretical advantages of blockchain over conventional banking protocols, they are not doing categorically better than banks at avoiding major losses. Although, the numbers at http://www.risk.net/operational-risk-and-regulation/opinion/2454379/bangladesh-bank-fraud-is-a-comedy-of-op-risk-errors http://www.risk.net/operational-risk-and-regulation/opinion/... are shocking.
- danielvf 11y ago
  To partially answer my own question, there's a Crypto Currency Security Standard, with Level 2 claiming on preventing solo internal attacks. https://cryptoconsortium.github.io/CCSS/ https://cryptoconsortium.github.io/CCSS/
- srdev 11y ago
  > Are there any best practices guides on this? I'm not aware of any, but we did have quite a lot of procedures meant to mitigate this sort of things when I worked in banking. For one, none of the developers had any sort of access to production machines.
- joe_the_user 11y ago
  I am not sure why those who essentially believe in utopia through an organized "war of all against all" wouldn't expect that anyone in position of trust wouldn't try to rip them off as much as possible. "Damn you, you've betrayed our trust and prevented us from building a world where we didn't have to trust people..."
- raesene9 11y ago
  Well the traditional banks have been dealing with exactly this kind of threat for a long time. There's a load of practices that are designed to stop this kind of problem in that world. Things like :- - Enforced holidays. Frauds are hard to maintain when you're not there to keep cooking the books. - Audit reviews. an independant function with the ability and authority to review key processes - split authority. Key actions need multiple people to complete (statistically fraud incidence drops a lot when you need multiple people involved) - Strong background checking. When I worked at a bank they went back 10 years of employment history and required accounting for any gaps. + credit checking + criminal record checking etc. It's all possible, just quite expensive...
- illumin8 11y ago
  It's extremely expensive. Many banks and companies in the finance industry (hedge funds) do this: Hire at least 2 or 3 people for every job. Have them watch each other whenever touching systems that connect to production or deploying code to production. Never trust any one of them with the private keys or passwords to anything - they can only get half of a secret and their co-worker gets the other half. To do this effectively, you have to build a zero trust environment, and rely on surveillance to ensure that nobody is a bad actor. It really makes CIA/NSA level security look somewhat weak. It's also very expensive, as you can imagine.
  - rodgerd 11y ago
    > Hire at least 2 or 3 people for every job. This is actually the key one - the more people who need to be corrupt, the harder it is to get away with being a crook. A surprising amount of internal fraud can be completed simply by requiring people to take solid blocks (2 weeks plus) of leave every year. That "inefficiency" the "lean 10x disruptors" congratulate themselves over does not always end well.
    - thedudemabry 11y ago
      I've kinda thought about this in the context of relatively complex technical jobs. One side of my family tree worked a bunch in the consumer banking (non-investment) industry, and I remember hearing story after story of someone being caught after their mandatory leave. But at a job where the ideal is automation, rather than manual verification and analysis, I wonder what additional obstacles must be placed in a potential attacker's way? Rotating keys based on employee schedules? I'm sure there are many well-studied ways to do this, but it's interesting to armchair analyze.
    - Terr_ 11y ago
      > solid blocks (2 weeks plus) of leave I've heard that's because a lot of those scams/tricks fall apart without constant gradual intervention, so an important part of it is that the employee is prevented from accessing most work-resources during that time.
- cortesoft 11y ago
  Step one is to properly vetting the person in charge of all your security.
oh_sigh 11y ago
What's the status of civil/criminal charges against Bob?
quanticle 11y ago
We learn some more things. Bob has prior police records in Florida, where he’s from. So they didn't even do a background check before hiring "Bob"? For a position where he would have access to systems that handled financial data? That's just grossly incompetent, in my book. I've worked for 5-man startups and Fortune 500 companies. In every case, the offer letter has stated that the offer is conditional on the successful completion of a background check, and none of the positions I've held have been remotely as sensitive as the position that Bob was hired for.
- cloudjacker 11y ago
  They don't do the background check.
  - jeebow 11y ago
    Background checks are a common requirement on insurance policies and such, so the background checks usually aren't fake. If you're counting on them being fake, expect a nasty surprise.
  - quanticle 11y ago
    They do. At least, my latest position involved submitting my employment data for the past seven years to a firm that conducted background checks. Again, this was for a position that was several layers removed from any kind of direct access to customer data or financial data.
  - randomfool 11y ago
    They do- one company found a warrant on me that I didn't know I had for an unpaid traffic ticket.
    - omarchowdhury 11y ago
      Do you know where one would go to perform an accurate background check on oneself?
tekromancr 11y ago
Facinating. More interesting reading seems to be available here: https://www.patrolx.com/wp-content/uploads/2016/04/309591980-ShapeShift-Postmortem.pdf https://www.patrolx.com/wp-content/uploads/2016/04/309591980...
jessaustin 11y ago
It seems odd to me that "Bob" hasn't been outed. It almost makes me suspect that someone isn't sure how much of "Bob's" role as portrayed in TFA is true and how much is a frame-job by an untouchable hacker [EDIT:] or wishful thinking by a frustrated executive.
- adekok 11y ago
  > It seems odd to me that "Bob" hasn't been outed. Since Bob hasn't been criminally charged with anything, "outing" him is legally "libel". He could sue, and win.
  - jessaustin 11y ago
    Why didn't you quote the next sentence, in which I directly addressed the possibility that TFA is wrong about "Bob"? Let me guess, YANAL? In USA, it ain't libel unless it's provably false. If "Bob" could prove that, why did he run?
  - ascorbic 11y ago
    It's not libel if it's true, and they'd only need to prove that to civil standard.
fasteo 11y ago
Am I the only one to think that all this narrative to blame Bob is pathetic ? This is pure and simple Mr. Voorhees (CEO) incompetency. After all, Bob is a criminal and he was just doing his "job".
- pavel_lishin 11y ago
  Are you suggesting that the criminal should not be blamed for his criminal actions or the losses that he knowingly and willingly incurred?
  - fasteo 11y ago
    Not quite. I suggest that a CEO should assume his responsibilities. He is quick to trash Bob[1] several times, but I see assumption of any responsibility at all. [1] Despite our note to all employees to come into the office urgently, Bob, our head IT guy, the one responsible for security and infrastructure, arrives at 11:30am. Soon after, Bob decides it’s time for his lunch break, and we don’t see him for an hour, during the worst incident in ShapeShift’s history
    - RangerScience 11y ago
      It's the difference between heroic responsibility and actual causality. "Of course it's my fault. There's no-one else here who could possibly be responsible for anything". The whole subtext of this was "Here's how I fucked up in leading this company", but then the actual text is causality. It is his responsibility to make sure employees are trustworth? Yes. Is he the cause of employees abusing trust? No. He then also determined that his responsibility was to let people know what had happened, so that's what he did. He told us the things they used to figure out what happened in order to attempt (and fail, and then attempt again...) to prevent it from recurring.
- danielvf 11y ago
  The article seemed pretty open about major mistakes that ShapeShift made and lessons learned. It's a good postmortem to learn from, and far more open than most would have posted.
cortesoft 11y ago
Man, calling a social security number a "social serfdom number" is really dumb and off putting. So is the continual reference to 'fiat money' constantly. I always love the irony of people so against the basic social contract are always so quick to turn to authorities when things predictably go wrong.
- mfringel 11y ago
  Indeed. Using tribal signaling when trying to reach a wider audience requires at least a small amount of skill.
- bionsuba 11y ago
  Would you call a communist a hypocrite for buying food at a grocery store?
  - tootitandbootit 11y ago
    Only if they were forced to buy food at a grocery store because their farm communes ran out of food because no one wanted to work on it.
  - fleitz 11y ago
    If they led / lived in a communist area, yes.
  - eli 11y ago
    Nope, but some people would! https://twitter.com/edroso/status/721760775530876928 https://twitter.com/edroso/status/721760775530876928
- cloudjacker 11y ago
  You might appreciate http://www.newyorker.com/humor/daily-shouts/l-p-d-libertarian-police-department http://www.newyorker.com/humor/daily-shouts/l-p-d-libertaria...
  - ikeboy 11y ago
koolba 11y ago
Not the best writing but it reads like a modern hardboiled[1]. Fun piece. [1]: https://en.wikipedia.org/wiki/Hardboiled https://en.wikipedia.org/wiki/Hardboiled
keithpeter 11y ago
"We had changed almost everything, but hadn’t scrapped our personal computers used while Bob had been part of the team. Would that have been the paranoid thing to do? Yes." At my humble and refreshingly drama-free place of work we have standard client images. Anything weird and the techies re-image the client. Assuming 'Bob' wasn't in charge of the images, would such a procedure have sorted the rdp?
- adekok 11y ago
  Maybe. The larger question was why did Bob have root access to people's individual laptops? He could have done a "snowden", grabbed their SSH keys including passphrases. That would have been much harder to detect.
  - azernik 11y ago
    That attack would have been prevented by their immediate SSH key rotation after the initial attack.
felixgallo 11y ago
I'm surprised that anyone would take this at face value. The cryptic black hat responding to e-mail with "one word: bob" is straight out of the most teenage of hacker fanfics.
a_small_island 11y ago
I thought this was a great story to read. Explained in laymans terms when needed and kept my attention throughout. Bob sucks.
floordaemon 11y ago
When things are handled professionally, you'll have a lot less drama. Why is the author being so nice to the theif over and over again? Even the last sentence! Eric brought this all on himself. Obviously 'scared' and intimidated to discuss important things WHEN they need to be discussed. Doing everything out of order -- like the background checks -- is a classic rookie move. You are basically running an Online Banking Website. You need to be aware of immense risks... ...it just goes on and on. But its clear why there is so much drama, crime and corruption and this company. Sad really. Who is to blame?
marco_salvatori 11y ago
For those interested in security engineering in the financial industry a good reading source is the banking section of: http://www.cl.cam.ac.uk/~rja14/book.html http://www.cl.cam.ac.uk/~rja14/book.html One interesting and relevant take away (of many) -- The greatest fraud threat to a financial firm are insiders. About 1% of staff across the industry is fired every year due to fraud. Within the fraud incidents, the most damaging fraud is perpetrated by senior and trusted individuals.
Nutmog 11y ago
Let's not lose sight of the fact that their cold wallet was untouched and all they lost was on the order of a hour's worth of turnover. That's more than can be said for a lot of the other bitcoin hacks.
Ontheflyflyfly 11y ago
"Very quickly, we realize he is pretty much useless. " When you hire IT people, have no clue how distinguish between a good one and a fake one, in other words have no clue, this happens. Also not enough oversight and auditing admins when money is involved is a bad sign.
Ontheflyflyfly 11y ago
When being accused of bigotry, it's never a good sign to use "social serfdom number" in your post mortem.
- tekromancr 11y ago
  That's just some shit that ancaps like to say. Also note the constant references to fiat currency. I don't think he was actually being racist with that.
  - Ontheflyflyfly 11y ago
    I agree, I don't think "serfdom" is racist, but it doesn't shine a good light on you to use, and erodes the trust people put in you by default.
    - tekromancr 11y ago
      Agreed.
buttershakes 11y ago
This reads like a case study in pure incompetance at every possible level. Lack of vetting, no third party auditing, poor segregation of customer funds. It's a total shit show. This should permanently damage their business and reputation, but the Bitcoin community has always been forgiving of people who lose their money. Fool me once...

198 points by danielvf 11y ago

This is certainly the worst case scenario - your security officer installing remote access software on developers machines, stealing bitcoins from production, then selling the company source code, access credentials and access to the internal network to a Russian hacker. Building a security system to handle this level of attack is a whole level beyond stopping even determined external attackers. Are there any best practices guides on this? One thing that the article showed is the importance of external security review to deal with the threat of internal incompetence or evil.

tlb 11y ago
Ultimately, the founders of bitcoin companies need to understand security themselves or they won't be able to hire and manage a competent team. Despite all the theoretical advantages of blockchain over conventional banking protocols, they are not doing categorically better than banks at avoiding major losses. Although, the numbers at http://www.risk.net/operational-risk-and-regulation/opinion/2454379/bangladesh-bank-fraud-is-a-comedy-of-op-risk-errors http://www.risk.net/operational-risk-and-regulation/opinion/... are shocking.
danielvf 11y ago
To partially answer my own question, there's a Crypto Currency Security Standard, with Level 2 claiming on preventing solo internal attacks. https://cryptoconsortium.github.io/CCSS/ https://cryptoconsortium.github.io/CCSS/
srdev 11y ago
> Are there any best practices guides on this? I'm not aware of any, but we did have quite a lot of procedures meant to mitigate this sort of things when I worked in banking. For one, none of the developers had any sort of access to production machines.
joe_the_user 11y ago
I am not sure why those who essentially believe in utopia through an organized "war of all against all" wouldn't expect that anyone in position of trust wouldn't try to rip them off as much as possible. "Damn you, you've betrayed our trust and prevented us from building a world where we didn't have to trust people..."
raesene9 11y ago
Well the traditional banks have been dealing with exactly this kind of threat for a long time. There's a load of practices that are designed to stop this kind of problem in that world. Things like :- - Enforced holidays. Frauds are hard to maintain when you're not there to keep cooking the books. - Audit reviews. an independant function with the ability and authority to review key processes - split authority. Key actions need multiple people to complete (statistically fraud incidence drops a lot when you need multiple people involved) - Strong background checking. When I worked at a bank they went back 10 years of employment history and required accounting for any gaps. + credit checking + criminal record checking etc. It's all possible, just quite expensive...
illumin8 11y ago
It's extremely expensive. Many banks and companies in the finance industry (hedge funds) do this: Hire at least 2 or 3 people for every job. Have them watch each other whenever touching systems that connect to production or deploying code to production. Never trust any one of them with the private keys or passwords to anything - they can only get half of a secret and their co-worker gets the other half. To do this effectively, you have to build a zero trust environment, and rely on surveillance to ensure that nobody is a bad actor. It really makes CIA/NSA level security look somewhat weak. It's also very expensive, as you can imagine.
- rodgerd 11y ago
  > Hire at least 2 or 3 people for every job. This is actually the key one - the more people who need to be corrupt, the harder it is to get away with being a crook. A surprising amount of internal fraud can be completed simply by requiring people to take solid blocks (2 weeks plus) of leave every year. That "inefficiency" the "lean 10x disruptors" congratulate themselves over does not always end well.
  - thedudemabry 11y ago
    I've kinda thought about this in the context of relatively complex technical jobs. One side of my family tree worked a bunch in the consumer banking (non-investment) industry, and I remember hearing story after story of someone being caught after their mandatory leave. But at a job where the ideal is automation, rather than manual verification and analysis, I wonder what additional obstacles must be placed in a potential attacker's way? Rotating keys based on employee schedules? I'm sure there are many well-studied ways to do this, but it's interesting to armchair analyze.
  - Terr_ 11y ago
    > solid blocks (2 weeks plus) of leave I've heard that's because a lot of those scams/tricks fall apart without constant gradual intervention, so an important part of it is that the employee is prevented from accessing most work-resources during that time.
cortesoft 11y ago
Step one is to properly vetting the person in charge of all your security.

oh_sigh 11y ago

What's the status of civil/criminal charges against Bob?

quanticle 11y ago

We learn some more things. Bob has prior police records in Florida, where he’s from. So they didn't even do a background check before hiring "Bob"? For a position where he would have access to systems that handled financial data? That's just grossly incompetent, in my book. I've worked for 5-man startups and Fortune 500 companies. In every case, the offer letter has stated that the offer is conditional on the successful completion of a background check, and none of the positions I've held have been remotely as sensitive as the position that Bob was hired for.

cloudjacker 11y ago
They don't do the background check.
- jeebow 11y ago
  Background checks are a common requirement on insurance policies and such, so the background checks usually aren't fake. If you're counting on them being fake, expect a nasty surprise.
- quanticle 11y ago
  They do. At least, my latest position involved submitting my employment data for the past seven years to a firm that conducted background checks. Again, this was for a position that was several layers removed from any kind of direct access to customer data or financial data.
- randomfool 11y ago
  They do- one company found a warrant on me that I didn't know I had for an unpaid traffic ticket.
  - omarchowdhury 11y ago
    Do you know where one would go to perform an accurate background check on oneself?

tekromancr 11y ago

Facinating. More interesting reading seems to be available here: https://www.patrolx.com/wp-content/uploads/2016/04/309591980-ShapeShift-Postmortem.pdf https://www.patrolx.com/wp-content/uploads/2016/04/309591980...

jessaustin 11y ago

It seems odd to me that "Bob" hasn't been outed. It almost makes me suspect that someone isn't sure how much of "Bob's" role as portrayed in TFA is true and how much is a frame-job by an untouchable hacker [EDIT:] or wishful thinking by a frustrated executive.

adekok 11y ago
> It seems odd to me that "Bob" hasn't been outed. Since Bob hasn't been criminally charged with anything, "outing" him is legally "libel". He could sue, and win.
- jessaustin 11y ago
  Why didn't you quote the next sentence, in which I directly addressed the possibility that TFA is wrong about "Bob"? Let me guess, YANAL? In USA, it ain't libel unless it's provably false. If "Bob" could prove that, why did he run?
- ascorbic 11y ago
  It's not libel if it's true, and they'd only need to prove that to civil standard.

fasteo 11y ago

Am I the only one to think that all this narrative to blame Bob is pathetic ? This is pure and simple Mr. Voorhees (CEO) incompetency. After all, Bob is a criminal and he was just doing his "job".

pavel_lishin 11y ago
Are you suggesting that the criminal should not be blamed for his criminal actions or the losses that he knowingly and willingly incurred?
- fasteo 11y ago
  Not quite. I suggest that a CEO should assume his responsibilities. He is quick to trash Bob[1] several times, but I see assumption of any responsibility at all. [1] Despite our note to all employees to come into the office urgently, Bob, our head IT guy, the one responsible for security and infrastructure, arrives at 11:30am. Soon after, Bob decides it’s time for his lunch break, and we don’t see him for an hour, during the worst incident in ShapeShift’s history
  - RangerScience 11y ago
    It's the difference between heroic responsibility and actual causality. "Of course it's my fault. There's no-one else here who could possibly be responsible for anything". The whole subtext of this was "Here's how I fucked up in leading this company", but then the actual text is causality. It is his responsibility to make sure employees are trustworth? Yes. Is he the cause of employees abusing trust? No. He then also determined that his responsibility was to let people know what had happened, so that's what he did. He told us the things they used to figure out what happened in order to attempt (and fail, and then attempt again...) to prevent it from recurring.
danielvf 11y ago
The article seemed pretty open about major mistakes that ShapeShift made and lessons learned. It's a good postmortem to learn from, and far more open than most would have posted.

cortesoft 11y ago

Man, calling a social security number a "social serfdom number" is really dumb and off putting. So is the continual reference to 'fiat money' constantly. I always love the irony of people so against the basic social contract are always so quick to turn to authorities when things predictably go wrong.

mfringel 11y ago
Indeed. Using tribal signaling when trying to reach a wider audience requires at least a small amount of skill.
bionsuba 11y ago
Would you call a communist a hypocrite for buying food at a grocery store?
- tootitandbootit 11y ago
  Only if they were forced to buy food at a grocery store because their farm communes ran out of food because no one wanted to work on it.
- fleitz 11y ago
  If they led / lived in a communist area, yes.
- eli 11y ago
  Nope, but some people would! https://twitter.com/edroso/status/721760775530876928 https://twitter.com/edroso/status/721760775530876928
cloudjacker 11y ago
You might appreciate http://www.newyorker.com/humor/daily-shouts/l-p-d-libertarian-police-department http://www.newyorker.com/humor/daily-shouts/l-p-d-libertaria...
- ikeboy 11y ago

koolba 11y ago

Not the best writing but it reads like a modern hardboiled[1]. Fun piece. [1]: https://en.wikipedia.org/wiki/Hardboiled https://en.wikipedia.org/wiki/Hardboiled

keithpeter 11y ago

"We had changed almost everything, but hadn’t scrapped our personal computers used while Bob had been part of the team. Would that have been the paranoid thing to do? Yes." At my humble and refreshingly drama-free place of work we have standard client images. Anything weird and the techies re-image the client. Assuming 'Bob' wasn't in charge of the images, would such a procedure have sorted the rdp?

adekok 11y ago
Maybe. The larger question was why did Bob have root access to people's individual laptops? He could have done a "snowden", grabbed their SSH keys including passphrases. That would have been much harder to detect.
- azernik 11y ago
  That attack would have been prevented by their immediate SSH key rotation after the initial attack.

felixgallo 11y ago

I'm surprised that anyone would take this at face value. The cryptic black hat responding to e-mail with "one word: bob" is straight out of the most teenage of hacker fanfics.

a_small_island 11y ago

I thought this was a great story to read. Explained in laymans terms when needed and kept my attention throughout. Bob sucks.

floordaemon 11y ago

When things are handled professionally, you'll have a lot less drama. Why is the author being so nice to the theif over and over again? Even the last sentence! Eric brought this all on himself. Obviously 'scared' and intimidated to discuss important things WHEN they need to be discussed. Doing everything out of order -- like the background checks -- is a classic rookie move. You are basically running an Online Banking Website. You need to be aware of immense risks... ...it just goes on and on. But its clear why there is so much drama, crime and corruption and this company. Sad really. Who is to blame?

marco_salvatori 11y ago

For those interested in security engineering in the financial industry a good reading source is the banking section of: http://www.cl.cam.ac.uk/~rja14/book.html http://www.cl.cam.ac.uk/~rja14/book.html One interesting and relevant take away (of many) -- The greatest fraud threat to a financial firm are insiders. About 1% of staff across the industry is fired every year due to fraud. Within the fraud incidents, the most damaging fraud is perpetrated by senior and trusted individuals.

Nutmog 11y ago

Let's not lose sight of the fact that their cold wallet was untouched and all they lost was on the order of a hour's worth of turnover. That's more than can be said for a lot of the other bitcoin hacks.

Ontheflyflyfly 11y ago

"Very quickly, we realize he is pretty much useless. " When you hire IT people, have no clue how distinguish between a good one and a fake one, in other words have no clue, this happens. Also not enough oversight and auditing admins when money is involved is a bad sign.

Ontheflyflyfly 11y ago

When being accused of bigotry, it's never a good sign to use "social serfdom number" in your post mortem.

tekromancr 11y ago
That's just some shit that ancaps like to say. Also note the constant references to fiat currency. I don't think he was actually being racist with that.
- Ontheflyflyfly 11y ago
  I agree, I don't think "serfdom" is racist, but it doesn't shine a good light on you to use, and erodes the trust people put in you by default.
  - tekromancr 11y ago
    Agreed.

buttershakes 11y ago

This reads like a case study in pure incompetance at every possible level. Lack of vetting, no third party auditing, poor segregation of customer funds. It's a total shit show. This should permanently damage their business and reputation, but the Bitcoin community has always been forgiving of people who lose their money. Fool me once...