6 ms·

We are Mt. Gox: AMA

51 points by nico 14y ago

paulhauggis 14y ago
wow. After looking at all of the issues and the lack of answers from MtGox..I don't think I'm going to be investing any money in this exchange any time soon.
- SpikedCola 14y ago
  No kidding - they appear to be focusing on answering question #1 and completely avoiding #2 & #3. Does not instill confidence.
  - 3pt14159 14y ago
    You guys need to understand that these guys are super Japanese, complete with the culture of never wanting to show dishonor. If they were American I would be much more suspicious, but they have consistently fixed the problems that they have had and are desperate to hire western developers to move to Japan to help the not only build the exchange, but expand the community's trust in them.
    - illuminate 14y ago
      "You guys need to understand that these guys are super Japanese, complete with the culture of never wanting to show dishonor." Then why have a Reddit Q&A about their failures?
Mahn 14y ago
It seems they were not aware that AWS exists. They should really consider hiring engineers specialized in scaling.
- saosebastiao 14y ago
  The more fundamental problem is the lack of separation between their trading engine and their website. While a website would be perfectly fine on AWS (and likely the more significant part of their scaling problems), the trading engine belongs on a very tightly controlled platform. I don't have any confidence in their platform. It shouldn't take more than a few domain experts to build a better platform...but for some reason its not happening.
  - Mahn 14y ago
    > I don't have any confidence in their platform. Neither should anyone, as evidenced by their track record. It's just sad that it is this platform driving the price of the Bitcoin at the moment, and I'm not sure how it speaks about the future of the currency.
  - pygy_ 14y ago
    The web frontend commands the trading platform, and the BTC/money outflow. It would be irresponsible for them to host it in an environment they dont control.
- deleted 14y ago
  [deleted]
- jonpaul 14y ago
  To be fair, it's my understanding that EC2 are virtual machines. If they use EC2, they need to be 100% certain that their servers are safe and don't fall victim to the Bitcoinica Linode attack. http://bitcoinmagazine.com/the-bitcoinica-linode-theft-and-what-it-means-for-bitcoin/ http://bitcoinmagazine.com/the-bitcoinica-linode-theft-and-w...
- Wilya 14y ago
  A full month to provision new servers seems awfully long. Even if you don't want to go full cloud (which could make sense, I'm not judging that), you can easily get dedicated servers installed and configured in less than a day.
  - nasalgoat 14y ago
    Well, not if you want to do self-hosting of your own equipment. Turnaround time from Dell is minimum two weeks. Add weeks as the servers get bigger and more custom. Also, most DCs need time to set up racks, power and connectivity.
- vy8vWJlco 14y ago
  I think if any of the important parts operated on AWS, governments would be quick to take advantage of that to shut it down ("it's used int eh drugztrade~!").
- codeulike 14y ago
  I don't think that using AWS would help them that much. They need to keep very secure (know any banks that use AWS?), and the main trading engine can't be parallelised anyway (because of order sequence and the blockchain). (Also: Amazon would chuck them out as soon as the first senator reached for their phone, as with wikileaks.)
  - nwh 14y ago
    Trading on Mt Gox has absolutely nothing to do with the bitcoin blockchain.
    - codeulike 14y ago
      Ah, I didn't realise that. so they're independent of the bitcoin address system? I had read that their trading engine can't be parallelised and assumed the blockchain was the reason. I stand corrected. edit: this guy explains it: https://news.ycombinator.com/item?id=5539387 https://news.ycombinator.com/item?id=5539387
- 650REDHAIR 14y ago
  Yeah, because AWS = instant scaling. This sort of operation needs physical hardware in their own secure location. This isn't some mobile social webapp.
- fmstephe 14y ago
  One of the problems here is that an exchange buying and selling one thing doesn't benefit much from scale. The core of a matching engine relies on knowing everything that happened before it, i.e. every trade submitted which has not been matched or cancelled. This pushes the matching process down to a single thread on a single machine. I am building a matching engine now (for pleasure) at https://github.com/fmstephe/matching_engine https://github.com/fmstephe/matching_engine (Having said all that I have no idea where their performance problems are coming from - the matching engine may not be the culprit)
xoail 14y ago
It always bothered me to know that there is no other solution to DDos attacks other than shutting down the systems and wait till it gets back to normal load.
vy8vWJlco 14y ago
Isn't the idea of a central exchange (which "Magic the Gathering online exchange" is), where most of the trading occurs, a little antithetical to notion of a distributed currency?
- Mahn 14y ago
  Absolutely, but a true p2p, decentralized Bitcoin exchange service has yet to happen.
  - vy8vWJlco 14y ago
    Paper IOUs seem almost ideal, if we didn't have to rely on a central issuing authority to make them universally accepted. IOUs are easy to replicate with blind signatures (see Section 4, Examples: http://web.mit.edu/6.857/OldStuff/Fall02/handouts/L15-voting.pdf http://web.mit.edu/6.857/OldStuff/Fall02/handouts/L15-voting... ) but noone wants to hunt down as many people as they have bills, just to trade the individually-issued digital IOUs... This is what Ripple ( http://ripple.com http://ripple.com ) claims to do, though: act as a distributed clearing house to resolve and cancel debt via a social graph. I haven't had a chance to actually use Ripple though, so I am certainly not recommending it specifically, just observing that it seems to address the right problem - that money is just debt - whereas Bitcoin simply creates a finite resource to be fought over.
- minimax 14y ago
  The problem is moving the counter currency around. I can zap you some bitcoins without any problems, but it's much harder for you to zap me back some USD (or EUR, JPY, whatever).
  - vy8vWJlco 14y ago
    Fair, the ForEx role will never go away, but it seems that MtGox has also become a trading platform for the Bitcoins themselves, with the obvious downside that it is susceptible to DDoS' like any centralized service. It makes more sense for a P2P currency that this simply be done at the protocol level with at most a custom client, but one that is still a full peer. The problem I, and others, are seeing is that the blockchain is becoming huge and will only get bigger, meaning only those with the resources to maintain it will participate as full peers - hardly ideal for a distributed system since it will inevitably require a certain amount of centralization to scale.
fnordfnordfnord 14y ago
Huge opportunity here for a proper exchange / trading platform to be built. There is obviously a demand for it.
- minimax 14y ago
  1) Regulatory issues are a big question mark. 2) There isn't actually that much volume compared real electronic exchanges. Just looking at the Mt Gox page from today it looks like $30MM USD worth of bitcoins have been traded. Hard to make any real money off of that.
  - fnordfnordfnord 14y ago
    If it was a sure bet that doing it would make you rich tomorrow, don't you think there would be scores of people racing to get it done? It's hard to find fruit hanging this low, if coinlab doesn't get it, I have a feeling someone will.
  - ProblemFactory 14y ago
    > 2) There isn't actually that much volume compared real electronic exchanges. Just looking at the Mt Gox page from today it looks like $30MM USD worth of bitcoins have been traded. Hard to make any real money off of that. While it might have much less volume compared to a "real" exchange, MtGox fees are 0.6% of each transaction. $180k daily revenue is certainly "real money" for a startup.
- wmf 14y ago
  Proper trading platforms (TradeHill, CampBX, etc.) were built and then no one used them.
  - fnordfnordfnord 14y ago
    CampBX web interface is terrible, but I admit, ought to be a competitor since they at least appear to be run better than gox. TradeHill's first act ended badly. In Act II they have a min 10k deposit requirement. I keep waiting for interesting news about coinlab's taking over mtgox, but so far, not much.
jack_trades 14y ago
Talk about putting up a shingle and calling yourself an exchange. The clamor from the crowd that they need to do better PR gets me chuckling the most. They are fairly transparent about their lack of technical expertise, hinting at major issues in their understanding or seriousness of what they are doing. When one noisy person tells you how to do your job and you know they're wrong, whatever. Brush it off. When a fistful of redditors are telling you 15 different ways that you should be doing your job and it's pretty easy to show that they understand the problem domain and issues better than the team... that's a big problem. I guess people hope MtGoxs new trading platform scales better, but that is not the sort of thought/wonder to be having when you are about to press submit on thousands in trades. Bitcoin. Currency like kids and potato cannons. What could go wrong? Fire it up!
- ramblerman 14y ago
  "Bitcoin. Currency like kids and potato cannons. What could go wrong? Fire it up!" You highlight the weakness in MtGox and then by analogy suggest the whole platform is similary amateuristic. the bitcoin protocol itself is solid, tested and has some very smart people working on it.
nwh 14y ago
The best piece of all comes from their answer about new servers. > Upgrading computer systems means ordering more servers (2 weeks timeframe), setting up (1 day), load testing (2 weeks) and deployment (1 day). It's a process that can take up to one month in total.
- codeulike 14y ago
  You've got to bear in mind that they're talking to reddit. I think thats a fair description of why they can't upgrade at the drop of a hat. As they say elsewhere in the AMA: We are big in the bitcoin world, but compared to a Facebook or a Google or even a bank we are too small and don't have access to their technology What would you have them do?
  - dragonwriter 14y ago
    > Cloud solutions are not meant for large scale operations such as trading systems Arguably, hosted cloud solutions aren't ideal for large scalable operations, if you define "large scale" as in "someone with the scale of Amazon, Google, etc. that can afford to run a private cloud." But MtGox makes it pretty clear that they aren't on that scale. Their argument seems to be simultaneously that they are both too "large scale" for hosted cloud solutions and too small scale to be able to effectively address the challenges they face in-house, which, if we accept it, seems to indicate that they are at a scale that cannot effectively operate.
    - codeulike 14y ago
      Right. They couldn't effectively operate in the face of a huge trading spike and a DDOS. Hence they're upgrading. But then they get stick for taking too long to upgrade. Conversely, if they spent $$$ on a bulletproof platform, they'd probably go bust (as seems to have happened to other more technically proficient exchanges) The underlying problem is: people have too high expectations of what the bitcoin world can cobble together at this point. Its being built by hackers, not bankers.
      - 14y ago

51 points by nico 14y ago

wow. After looking at all of the issues and the lack of answers from MtGox..I don't think I'm going to be investing any money in this exchange any time soon.

SpikedCola 14y ago
No kidding - they appear to be focusing on answering question #1 and completely avoiding #2 & #3. Does not instill confidence.
- 3pt14159 14y ago
  You guys need to understand that these guys are super Japanese, complete with the culture of never wanting to show dishonor. If they were American I would be much more suspicious, but they have consistently fixed the problems that they have had and are desperate to hire western developers to move to Japan to help the not only build the exchange, but expand the community's trust in them.
  - illuminate 14y ago
    "You guys need to understand that these guys are super Japanese, complete with the culture of never wanting to show dishonor." Then why have a Reddit Q&A about their failures?

Mahn 14y ago

It seems they were not aware that AWS exists. They should really consider hiring engineers specialized in scaling.

saosebastiao 14y ago
The more fundamental problem is the lack of separation between their trading engine and their website. While a website would be perfectly fine on AWS (and likely the more significant part of their scaling problems), the trading engine belongs on a very tightly controlled platform. I don't have any confidence in their platform. It shouldn't take more than a few domain experts to build a better platform...but for some reason its not happening.
- Mahn 14y ago
  > I don't have any confidence in their platform. Neither should anyone, as evidenced by their track record. It's just sad that it is this platform driving the price of the Bitcoin at the moment, and I'm not sure how it speaks about the future of the currency.
- pygy_ 14y ago
  The web frontend commands the trading platform, and the BTC/money outflow. It would be irresponsible for them to host it in an environment they dont control.
deleted 14y ago
[deleted]
jonpaul 14y ago
To be fair, it's my understanding that EC2 are virtual machines. If they use EC2, they need to be 100% certain that their servers are safe and don't fall victim to the Bitcoinica Linode attack. http://bitcoinmagazine.com/the-bitcoinica-linode-theft-and-what-it-means-for-bitcoin/ http://bitcoinmagazine.com/the-bitcoinica-linode-theft-and-w...
Wilya 14y ago
A full month to provision new servers seems awfully long. Even if you don't want to go full cloud (which could make sense, I'm not judging that), you can easily get dedicated servers installed and configured in less than a day.
- nasalgoat 14y ago
  Well, not if you want to do self-hosting of your own equipment. Turnaround time from Dell is minimum two weeks. Add weeks as the servers get bigger and more custom. Also, most DCs need time to set up racks, power and connectivity.
vy8vWJlco 14y ago
I think if any of the important parts operated on AWS, governments would be quick to take advantage of that to shut it down ("it's used int eh drugztrade~!").
codeulike 14y ago
I don't think that using AWS would help them that much. They need to keep very secure (know any banks that use AWS?), and the main trading engine can't be parallelised anyway (because of order sequence and the blockchain). (Also: Amazon would chuck them out as soon as the first senator reached for their phone, as with wikileaks.)
- nwh 14y ago
  Trading on Mt Gox has absolutely nothing to do with the bitcoin blockchain.
  - codeulike 14y ago
    Ah, I didn't realise that. so they're independent of the bitcoin address system? I had read that their trading engine can't be parallelised and assumed the blockchain was the reason. I stand corrected. edit: this guy explains it: https://news.ycombinator.com/item?id=5539387 https://news.ycombinator.com/item?id=5539387
650REDHAIR 14y ago
Yeah, because AWS = instant scaling. This sort of operation needs physical hardware in their own secure location. This isn't some mobile social webapp.
fmstephe 14y ago
One of the problems here is that an exchange buying and selling one thing doesn't benefit much from scale. The core of a matching engine relies on knowing everything that happened before it, i.e. every trade submitted which has not been matched or cancelled. This pushes the matching process down to a single thread on a single machine. I am building a matching engine now (for pleasure) at https://github.com/fmstephe/matching_engine https://github.com/fmstephe/matching_engine (Having said all that I have no idea where their performance problems are coming from - the matching engine may not be the culprit)

xoail 14y ago

It always bothered me to know that there is no other solution to DDos attacks other than shutting down the systems and wait till it gets back to normal load.

vy8vWJlco 14y ago

Isn't the idea of a central exchange (which "Magic the Gathering online exchange" is), where most of the trading occurs, a little antithetical to notion of a distributed currency?

Mahn 14y ago
Absolutely, but a true p2p, decentralized Bitcoin exchange service has yet to happen.
- vy8vWJlco 14y ago
  Paper IOUs seem almost ideal, if we didn't have to rely on a central issuing authority to make them universally accepted. IOUs are easy to replicate with blind signatures (see Section 4, Examples: http://web.mit.edu/6.857/OldStuff/Fall02/handouts/L15-voting.pdf http://web.mit.edu/6.857/OldStuff/Fall02/handouts/L15-voting... ) but noone wants to hunt down as many people as they have bills, just to trade the individually-issued digital IOUs... This is what Ripple ( http://ripple.com http://ripple.com ) claims to do, though: act as a distributed clearing house to resolve and cancel debt via a social graph. I haven't had a chance to actually use Ripple though, so I am certainly not recommending it specifically, just observing that it seems to address the right problem - that money is just debt - whereas Bitcoin simply creates a finite resource to be fought over.
minimax 14y ago
The problem is moving the counter currency around. I can zap you some bitcoins without any problems, but it's much harder for you to zap me back some USD (or EUR, JPY, whatever).
- vy8vWJlco 14y ago
  Fair, the ForEx role will never go away, but it seems that MtGox has also become a trading platform for the Bitcoins themselves, with the obvious downside that it is susceptible to DDoS' like any centralized service. It makes more sense for a P2P currency that this simply be done at the protocol level with at most a custom client, but one that is still a full peer. The problem I, and others, are seeing is that the blockchain is becoming huge and will only get bigger, meaning only those with the resources to maintain it will participate as full peers - hardly ideal for a distributed system since it will inevitably require a certain amount of centralization to scale.

fnordfnordfnord 14y ago

Huge opportunity here for a proper exchange / trading platform to be built. There is obviously a demand for it.

minimax 14y ago
1) Regulatory issues are a big question mark. 2) There isn't actually that much volume compared real electronic exchanges. Just looking at the Mt Gox page from today it looks like $30MM USD worth of bitcoins have been traded. Hard to make any real money off of that.
- fnordfnordfnord 14y ago
  If it was a sure bet that doing it would make you rich tomorrow, don't you think there would be scores of people racing to get it done? It's hard to find fruit hanging this low, if coinlab doesn't get it, I have a feeling someone will.
- ProblemFactory 14y ago
  > 2) There isn't actually that much volume compared real electronic exchanges. Just looking at the Mt Gox page from today it looks like $30MM USD worth of bitcoins have been traded. Hard to make any real money off of that. While it might have much less volume compared to a "real" exchange, MtGox fees are 0.6% of each transaction. $180k daily revenue is certainly "real money" for a startup.
wmf 14y ago
Proper trading platforms (TradeHill, CampBX, etc.) were built and then no one used them.
- fnordfnordfnord 14y ago
  CampBX web interface is terrible, but I admit, ought to be a competitor since they at least appear to be run better than gox. TradeHill's first act ended badly. In Act II they have a min 10k deposit requirement. I keep waiting for interesting news about coinlab's taking over mtgox, but so far, not much.

jack_trades 14y ago

Talk about putting up a shingle and calling yourself an exchange. The clamor from the crowd that they need to do better PR gets me chuckling the most. They are fairly transparent about their lack of technical expertise, hinting at major issues in their understanding or seriousness of what they are doing. When one noisy person tells you how to do your job and you know they're wrong, whatever. Brush it off. When a fistful of redditors are telling you 15 different ways that you should be doing your job and it's pretty easy to show that they understand the problem domain and issues better than the team... that's a big problem. I guess people hope MtGoxs new trading platform scales better, but that is not the sort of thought/wonder to be having when you are about to press submit on thousands in trades. Bitcoin. Currency like kids and potato cannons. What could go wrong? Fire it up!

ramblerman 14y ago
"Bitcoin. Currency like kids and potato cannons. What could go wrong? Fire it up!" You highlight the weakness in MtGox and then by analogy suggest the whole platform is similary amateuristic. the bitcoin protocol itself is solid, tested and has some very smart people working on it.

nwh 14y ago

The best piece of all comes from their answer about new servers. > Upgrading computer systems means ordering more servers (2 weeks timeframe), setting up (1 day), load testing (2 weeks) and deployment (1 day). It's a process that can take up to one month in total.

codeulike 14y ago
You've got to bear in mind that they're talking to reddit. I think thats a fair description of why they can't upgrade at the drop of a hat. As they say elsewhere in the AMA: We are big in the bitcoin world, but compared to a Facebook or a Google or even a bank we are too small and don't have access to their technology What would you have them do?
- dragonwriter 14y ago
  > Cloud solutions are not meant for large scale operations such as trading systems Arguably, hosted cloud solutions aren't ideal for large scalable operations, if you define "large scale" as in "someone with the scale of Amazon, Google, etc. that can afford to run a private cloud." But MtGox makes it pretty clear that they aren't on that scale. Their argument seems to be simultaneously that they are both too "large scale" for hosted cloud solutions and too small scale to be able to effectively address the challenges they face in-house, which, if we accept it, seems to indicate that they are at a scale that cannot effectively operate.
  - codeulike 14y ago
    Right. They couldn't effectively operate in the face of a huge trading spike and a DDOS. Hence they're upgrading. But then they get stick for taking too long to upgrade. Conversely, if they spent $$$ on a bulletproof platform, they'd probably go bust (as seems to have happened to other more technically proficient exchanges) The underlying problem is: people have too high expectations of what the bitcoin world can cobble together at this point. Its being built by hackers, not bankers.
    - 14y ago