7 ms·

Your theory seems reasonable to me, and I know you're not defending them, but ... How difficult is it to keep the right amount of bitcoin and USD in deposit? I

by Pyxl101 12y ago

Your theory seems reasonable to me, and I know you're not defending them, but ... How difficult is it to keep the right amount of bitcoin and USD in deposit? It seems pretty simple to me. Customer deposits $1.00, then I keep $1.00 in deposit in my business account. Customer deposits 1 bitcoin, I keep one bitcoin. The account for customer deposits will only ever be touched by automated systems processing these transactions. If I'm charging fees as part of depositing/withdrawing/trading currencies or BTC, then those fees are immediately swept into a separate account, the entirety of which is revenue. Kept strictly separate from deposits. I'd regularly audit both accounts to confirm that the dollar amounts of each deposit/withdrawal, as well as total, match the records of actions in my systems. Mt Gox only ever handled about 40 trades per second. Seems like you could keep everything balanced with a typical database, using transactions to implement atomic trades / deposits / withdrawals. Perhaps interfacing with money transmission systems is slightly more difficult, but once a transaction has reached the point of having probably-completed, you initiate the transaction to modify the customer's balance of bitcoin and currency. So I guess I'm agreeing with you that if they failed in this way, they must have been incompetent. P.S. I'm just making all this up. I've never worked in payments or accounting. I expect a good security or finance auditor could suggest much better controls. I'd pick up an "accounting for dummies" book and learn about the basic control principles used in accounting to prevent fraud. Perhaps I periodically download statements from my bank, print them out (or receive them in the mail) and audit those records against my own systems, by hand. Do the same electronically on a continuous basis. Set up an off-site write-only logging system to capture all transactions, and regularly audit to ensure that the system logs match bank transaction logs.

drcode 12y ago
This is just wild speculation now... ...but my best guess is their trading engine was so unstable that they ended up using one server to run a stripped-down clearance house algorithm for executing the trades as quickly as feasible, and a separate system that was the "source of truth" for how many physical assets in bitcoin/yen/USD were actually in possession by the company. I'm guessing that there was absolutely nothing "atomic" about the system Marc Karpeles had built, their storage system was just an amalgam of disconnected, duct-taped databases living in different places, that were "kinda, sorta" in sync with each other, with lots of unpredictable, undefined behavior.
wmf 12y ago
MtGox was pretty buggy and many of those bugs lost money, which could easily make them insolvent. Also, interfacing with Bitcoin at scale (e.g. managing thousands of addresses) required a lot of custom programming.
drcode 12y ago
> I'd set up a write-only logging system to be used for auditing company systems I bet Marc Karpeles had 5 people calling him at once that the trading engine was down again so THAT's where he put is efforts... He didn't get any calls from people saying "By the way, I hope you have robust auditing procedures." Like some people live paycheck to paycheck, I'm betting the MtGox guys were living from critical bug report to critical bug report, without time for "luxuries" like "atomicity" or "database auditing" (Let me say up front I am in no way intending for my comments to be a defense of the behavior of these criminals.)
- patio11 12y ago
  It has been credibly alleged [+] that Mt. Gox's general level of engineering acumen can be summed up as "No tests, no staging environment, live coding by SSHing into production." [+] Unnamed employee of Tibanne, the parent company which supplied substantially all of 0-employee Mt. Gox's staffing: http://www.pcworld.com/article/2105920/lost-in-translation-the-tangled-tale-of-mt-goxs-missing-millions.html http://www.pcworld.com/article/2105920/lost-in-translation-t...
  - gst 12y ago
    On the topic of SSH: It appears that they used their own version of SSH reimplemented in PHP: https://web.archive.org/web/20140102104411/http://blog.magicaltux.net/2010/06/27/php-can-do-anything-what-about-some-ssh https://web.archive.org/web/20140102104411/http://blog.magic...
- Pyxl101 12y ago
  True, but a system like I'm describing ought not to be terribly complicated to set up. Perhaps it's simply a remote syslog daemon under the control of a trusted exec. Preferably offsite or in a separate cage. Or if SAAS, under a distinct account. Maybe we don't block transactions on writing to the log, but whenever discrepancies are detected of more than a short time (transactions more than 60 seconds old not appearing), we page all founders and senior leaders. The trading engine stays up, but we draw much attention to the logging outage. (I'd also want controls like offline wallets and accounts that would prevent an attacker from obliterating our assets in a single withdrawal)
patio11 12y ago
If you're familiar with the Willy Report (Google it), the simplest working theory is that someone figured out a way to award themselves USD/etc balances inside of the Mt. Gox system, purchased large amounts of Bitcoins with them, then transferred them out in the ordinary course of business. This caused positive improvements in the set of metrics that Gox and the community actively monitored, and the metrics in which the discrepancy would have been obvious were apparently not actively monitored. (e.g. Comparing the sum of all USD/etc liabilities as reflected in the database with the sum of balances at Mt. Gox's single-digit number of bank accounts.) You could call it the most total failure of accounting controls in history, if one had any reason to suspect that Mt. Gox possessed meaningful accounting controls. There was no reason to suspect this, but many people were willfully blind to that, because it appeared that Bitcoiners were making money hand-over-fist. Edit to add: One reason Gox may not have done the math on the assets vs. liabilities is because they knew they were insolvent from some point in early to mid 2013, as a result of some combination of external events, including the since-reversed freezing of their US bank accounts. The clearest publicly available evidence of insolvency was persistent unwillingness to allow people to withdraw USD/etc and ludicrous excuses for why this was the case. (At one point, the CEO alleged that their daily wire volume had overwhelmed the second largest bank in Japan. This kind of claim is very useful because it is trivially falsifiable.)
- jacquesm 12y ago
  > You could call it the most total failure of accounting controls in history Enron?
  - hga 12y ago
    Nah, Enron's prominence is purely political, it was useful for various narratives. There are far worse examples, and Mt. Gox sounds like it's in the running. E.g. there's cheating on your general ledger (which at 50,000 feet might be the sort of thing Enron did), then there's not keeping a general ledger at all, which is being posited in this discussion. Even I, who's never run a company, but did watch my parents run some while I was growing up, know how essential it is to keep a general ledger. Otherwise you have no real idea what your financial state is.
Animats 12y ago
How difficult is it to keep the right amount of bitcoin and USD in deposit? ... P.S. I'm just making all this up. I've never worked in payments or accounting. Mt. Gox had a financial control problem less complex than that faced by the typical supermarket. A supermarket has cash going in and out, multiple cash registers, multiple cashiers on different shifts, credit cards, checks, benefit cards, and automated checkout lanes. That's just the front side - there's a comparable operation at the back, where merchandise comes in and is accounted for. A big supermarket has a higher transaction rate than Mt. Gox did. If a $10 bill goes missing in a supermarket, the management will know at the end of the day, and will probably be able to figure out where it went. This is routine cash control. It's a solved problem. Mt. Gox had no financial controls; we know that from press reports. None. They don't even seem to have had a general ledger. This is rare in real businesses, but it's common in fraud schemes. Fraudsters don't want accounting controls, with everything they're stealing on the record. Anyway, on Jan 3, we'll hear more. That this story came from the Tokyo Metropolitan Police is significant. The police haven't said anything until now, except that they were investigating. That silence has ended. Arrests may be announced. There's a good chance that Mark Karpeles is about to experience the standard 21-day interrogation used in Japan.
- mschuster91 12y ago
  > If a $10 bill goes missing in a supermarket, the management will know at the end of the day, and will probably be able to figure out where it went. This is routine cash control. It's a solved problem. Not necessarily, differences are a daily part of working as a supermarket cashier. Most supermarkets have agreements with their employees (sometimes mandated by law), varying from full employee liability to full coverage by employer. It's actually quite common to have differences in your register, e.g. because customers let you keep small-ish returns or give "real" tips or you as cashier in a high-frequency environment make a mistake in sorting in a bill into the right compartment - or fraudsters trick you (http://www.weser-kurier.de/region/delmenhorster-kurier_artikel,-Polizei-warnt-vor-Wechselgeldbetrug-_arid,167778.html http://www.weser-kurier.de/region/delmenhorster-kurier_artik...).
  - Animats 12y ago

by Pyxl101 12y ago

drcode 12y ago

This is just wild speculation now... ...but my best guess is their trading engine was so unstable that they ended up using one server to run a stripped-down clearance house algorithm for executing the trades as quickly as feasible, and a separate system that was the "source of truth" for how many physical assets in bitcoin/yen/USD were actually in possession by the company. I'm guessing that there was absolutely nothing "atomic" about the system Marc Karpeles had built, their storage system was just an amalgam of disconnected, duct-taped databases living in different places, that were "kinda, sorta" in sync with each other, with lots of unpredictable, undefined behavior.

wmf 12y ago

MtGox was pretty buggy and many of those bugs lost money, which could easily make them insolvent. Also, interfacing with Bitcoin at scale (e.g. managing thousands of addresses) required a lot of custom programming.

drcode 12y ago

> I'd set up a write-only logging system to be used for auditing company systems I bet Marc Karpeles had 5 people calling him at once that the trading engine was down again so THAT's where he put is efforts... He didn't get any calls from people saying "By the way, I hope you have robust auditing procedures." Like some people live paycheck to paycheck, I'm betting the MtGox guys were living from critical bug report to critical bug report, without time for "luxuries" like "atomicity" or "database auditing" (Let me say up front I am in no way intending for my comments to be a defense of the behavior of these criminals.)

patio11 12y ago
It has been credibly alleged [+] that Mt. Gox's general level of engineering acumen can be summed up as "No tests, no staging environment, live coding by SSHing into production." [+] Unnamed employee of Tibanne, the parent company which supplied substantially all of 0-employee Mt. Gox's staffing: http://www.pcworld.com/article/2105920/lost-in-translation-the-tangled-tale-of-mt-goxs-missing-millions.html http://www.pcworld.com/article/2105920/lost-in-translation-t...
- gst 12y ago
  On the topic of SSH: It appears that they used their own version of SSH reimplemented in PHP: https://web.archive.org/web/20140102104411/http://blog.magicaltux.net/2010/06/27/php-can-do-anything-what-about-some-ssh https://web.archive.org/web/20140102104411/http://blog.magic...
Pyxl101 12y ago
True, but a system like I'm describing ought not to be terribly complicated to set up. Perhaps it's simply a remote syslog daemon under the control of a trusted exec. Preferably offsite or in a separate cage. Or if SAAS, under a distinct account. Maybe we don't block transactions on writing to the log, but whenever discrepancies are detected of more than a short time (transactions more than 60 seconds old not appearing), we page all founders and senior leaders. The trading engine stays up, but we draw much attention to the logging outage. (I'd also want controls like offline wallets and accounts that would prevent an attacker from obliterating our assets in a single withdrawal)

patio11 12y ago

If you're familiar with the Willy Report (Google it), the simplest working theory is that someone figured out a way to award themselves USD/etc balances inside of the Mt. Gox system, purchased large amounts of Bitcoins with them, then transferred them out in the ordinary course of business. This caused positive improvements in the set of metrics that Gox and the community actively monitored, and the metrics in which the discrepancy would have been obvious were apparently not actively monitored. (e.g. Comparing the sum of all USD/etc liabilities as reflected in the database with the sum of balances at Mt. Gox's single-digit number of bank accounts.) You could call it the most total failure of accounting controls in history, if one had any reason to suspect that Mt. Gox possessed meaningful accounting controls. There was no reason to suspect this, but many people were willfully blind to that, because it appeared that Bitcoiners were making money hand-over-fist. Edit to add: One reason Gox may not have done the math on the assets vs. liabilities is because they knew they were insolvent from some point in early to mid 2013, as a result of some combination of external events, including the since-reversed freezing of their US bank accounts. The clearest publicly available evidence of insolvency was persistent unwillingness to allow people to withdraw USD/etc and ludicrous excuses for why this was the case. (At one point, the CEO alleged that their daily wire volume had overwhelmed the second largest bank in Japan. This kind of claim is very useful because it is trivially falsifiable.)

jacquesm 12y ago
> You could call it the most total failure of accounting controls in history Enron?
- hga 12y ago
  Nah, Enron's prominence is purely political, it was useful for various narratives. There are far worse examples, and Mt. Gox sounds like it's in the running. E.g. there's cheating on your general ledger (which at 50,000 feet might be the sort of thing Enron did), then there's not keeping a general ledger at all, which is being posited in this discussion. Even I, who's never run a company, but did watch my parents run some while I was growing up, know how essential it is to keep a general ledger. Otherwise you have no real idea what your financial state is.

Animats 12y ago

How difficult is it to keep the right amount of bitcoin and USD in deposit? ... P.S. I'm just making all this up. I've never worked in payments or accounting. Mt. Gox had a financial control problem less complex than that faced by the typical supermarket. A supermarket has cash going in and out, multiple cash registers, multiple cashiers on different shifts, credit cards, checks, benefit cards, and automated checkout lanes. That's just the front side - there's a comparable operation at the back, where merchandise comes in and is accounted for. A big supermarket has a higher transaction rate than Mt. Gox did. If a $10 bill goes missing in a supermarket, the management will know at the end of the day, and will probably be able to figure out where it went. This is routine cash control. It's a solved problem. Mt. Gox had no financial controls; we know that from press reports. None. They don't even seem to have had a general ledger. This is rare in real businesses, but it's common in fraud schemes. Fraudsters don't want accounting controls, with everything they're stealing on the record. Anyway, on Jan 3, we'll hear more. That this story came from the Tokyo Metropolitan Police is significant. The police haven't said anything until now, except that they were investigating. That silence has ended. Arrests may be announced. There's a good chance that Mark Karpeles is about to experience the standard 21-day interrogation used in Japan.

mschuster91 12y ago
> If a $10 bill goes missing in a supermarket, the management will know at the end of the day, and will probably be able to figure out where it went. This is routine cash control. It's a solved problem. Not necessarily, differences are a daily part of working as a supermarket cashier. Most supermarkets have agreements with their employees (sometimes mandated by law), varying from full employee liability to full coverage by employer. It's actually quite common to have differences in your register, e.g. because customers let you keep small-ish returns or give "real" tips or you as cashier in a high-frequency environment make a mistake in sorting in a bill into the right compartment - or fraudsters trick you (http://www.weser-kurier.de/region/delmenhorster-kurier_artikel,-Polizei-warnt-vor-Wechselgeldbetrug-_arid,167778.html http://www.weser-kurier.de/region/delmenhorster-kurier_artik...).
- Animats 12y ago