3 ms·

I can't seem to pull up any more info on this. The write up really doesn't explain why ipfs would have perfomed better in this case. Is it because the data in q

I can't seem to pull up any more info on this. The write up really doesn't explain why ipfs would have perfomed better in this case. Is it because the data in question is replicated more times? Is it because ipfs protocol is faster than HDFS? Can't seem to find the connector code either. It's a little strange to go back to map reduce when presumably YARN is available and tez or spark could be used.

Mateon1 9y ago
While I haven't looked for more info, I can assure you IPFS is a slower protocol than HDFS. The existing IPFS implementation involves a lot of overhead in memory, CPU use, and latency [and as you can see in the MapReduce bar graph, it's slower], but overall, it improves performance when bandwidth is the bottleneck. IPFS is pretty similar to BitTorrent, just more practical.
- XR0CSWV3h3kZWg 9y ago
  Why do you say bittorrent isn't practical?
  - Mateon1 9y ago
    BitTorrent isn't "not practical". It is however less practical than IPFS. With IPFS, you can effortlessly link from one tree to another, already existing tree. In BitTorrent, you have to include a .torrent file, or an infohash in some file, but no software that I know of will easily follow that link. This linking ability is an extremely useful property, that allows you to cheaply create a copy of a merkle tree, with a subset of the data replaced. The tree will operate identically to one created from scratch. You also don't need to hold all the data to "patch" the tree, which I imagine is useful in this Hadoop filesystem. Unfortunately there is no real API for operating on torrents. You could say WebTorrent is pushing in the right direction to fix this, but there doesn't seem to be much adoption for it.
TallGuyShort 9y ago
>> It's a little strange to go back to map reduce when presumably YARN is available and tez or spark could be used. YARN just refactored the scheduling / resource management out of MapReduce. Hadoop 2+ still uses the term "MapReduce" for that particular application, and that application runs on a YARN cluster, just like Tez (I think) and Spark can. I don't see anything to indicate they're NOT using YARN.
- XR0CSWV3h3kZWg 9y ago
  Yep, if they are running hadoop 2 then they have YARN and they can use MR, Tez or Spark.
- agibsonccc 9y ago
  That happened years ago. I haven't seen a hadoop 1 in the wild in a long time now. When most people think of "hadoop" now, it's more about the ecosystem of "HDFS + some addons like spark"
  - TallGuyShort 9y ago
    I'm just saying that when the article says "MapReduce", it in no way limits their workload to Hadoop 1 or 0.20.x, as OP seem to have thought.

Mateon1 9y ago

While I haven't looked for more info, I can assure you IPFS is a slower protocol than HDFS. The existing IPFS implementation involves a lot of overhead in memory, CPU use, and latency [and as you can see in the MapReduce bar graph, it's slower], but overall, it improves performance when bandwidth is the bottleneck. IPFS is pretty similar to BitTorrent, just more practical.

XR0CSWV3h3kZWg 9y ago
Why do you say bittorrent isn't practical?
- Mateon1 9y ago
  BitTorrent isn't "not practical". It is however less practical than IPFS. With IPFS, you can effortlessly link from one tree to another, already existing tree. In BitTorrent, you have to include a .torrent file, or an infohash in some file, but no software that I know of will easily follow that link. This linking ability is an extremely useful property, that allows you to cheaply create a copy of a merkle tree, with a subset of the data replaced. The tree will operate identically to one created from scratch. You also don't need to hold all the data to "patch" the tree, which I imagine is useful in this Hadoop filesystem. Unfortunately there is no real API for operating on torrents. You could say WebTorrent is pushing in the right direction to fix this, but there doesn't seem to be much adoption for it.

TallGuyShort 9y ago

>> It's a little strange to go back to map reduce when presumably YARN is available and tez or spark could be used. YARN just refactored the scheduling / resource management out of MapReduce. Hadoop 2+ still uses the term "MapReduce" for that particular application, and that application runs on a YARN cluster, just like Tez (I think) and Spark can. I don't see anything to indicate they're NOT using YARN.

XR0CSWV3h3kZWg 9y ago
Yep, if they are running hadoop 2 then they have YARN and they can use MR, Tez or Spark.
agibsonccc 9y ago
That happened years ago. I haven't seen a hadoop 1 in the wild in a long time now. When most people think of "hadoop" now, it's more about the ecosystem of "HDFS + some addons like spark"
- TallGuyShort 9y ago
  I'm just saying that when the article says "MapReduce", it in no way limits their workload to Hadoop 1 or 0.20.x, as OP seem to have thought.