7 ms·

Apache Arrow and MinIO

71 points by jtsymonds 6y ago

wesm 6y ago
(Wes here) I appreciate the Arrow shout-out but note that Apache Arrow has been a major open source community collaboration and not something I can take sole credit for.
anderspitman 6y ago
Is Arrow data commonly written to disk? I was under the impression (as someone who doesn't use it currently) it's usually converted to Parquet for this purpose.
- thundergolfer 6y ago
  I'd be interested to know any benefits of converting to Parquet if you know your reader supports Arrow.
  - lmeyerov 6y ago
    Compression. For our app, we do multilevel caches, where warm is arrow (in-memory, inflight, streaming, ...) and cold (persistent/scaleout) is parquet (ssd, s3, ...).
- andygrove 6y ago
  Arrow is a memory format and optimized for efficient vectorized processing in memory. Although it is possible to persist the Arrow format to disk, it is more common to use Parquet.
gizmodo59 6y ago
Is there any specific optimizations done in minio code for arrow or it’s just that in general reading from arrow format gives high performance in analytics?
georgewfraser 6y ago
I am truly baffled by the desire to build new data lakes. I understand why Hadoop data lakes came to be: Teradata charged a lot for storage. But now there are database systems that use object storage under-the-hood: Snowflake, BigQuery, Redshift RA3, Databricks Delta Lake. They give you the same low cost of object storage, but with a nice high-level relational interface. I am mystified by why someone would want to build their own data lake rather than just use one of these systems.
- sgt101 6y ago
  Surely those are all tools for building a data lake, and the stack described in the article is just another tool for doing the same? It's a reasonable architectural choice...
- jmngomes 6y ago
  All the tools you mention are great for their specific use cases, e.g. Snowflake BigQuery and RedShift are great for analytics over big data, the very common COUNT/SUM/AVG/PARTITION/GROUP BY analysis. But as with any other tool, they're not a good/perfect fit for diverse analysis methods of diverse data types, which I think is what a data lake aims at. Analyzing a large JSON dataset on Snowflake is possible, but either too slow or expensive when compared with more appropriate tools (e.g. Elasticsearch or a Python notebook running PySpark). Having a data lake - which I understand as a repository of raw data of diverse types, regardless of the tools - structured in a tool like S3 is very useful when you have multiple use cases over data of different kinds. For example, you could store audio files from customer calls and have them processed automatically by Spark jobs (e.g. for transcript and stats generation), structure and store call stats on a database for analytics, and do further analysis via notebooks on data science initiatives (e.g. sentiment analysis). This is akin to having a staging area for complex and diverse data types, and S3 is useful for this because of its speed, scale and management features. Teradata or Snowflake aren't a great fit for use cases like these, but they are great if the use case is to get answers to questions like "top 3 operators per team in volume of calls, by department and region, in last quarter" if the volume of calls is big. If I understood correctly, I think your comment was more focused on why use new tools when the existing are mature, but I think big data tools have had to become more specialized and targeted for specific use cases. But if the question is "why build more than one data lake", the only reason I can see is organizational: teams or different areas of an organization either need their own data lake because they have specific needs (which is rare) or won't/can't collaborate with others to have a shared asset.
hardwaresofton 6y ago
Am I the only one that just can't grok what Arrow is doing and how minio in particular made it faster? I read/skimmed this submission and I've worked out this much: - Arrow is an efficient object format that helps skip serialization - Spark works in memory - Minio makes storage of data scalable and cheaper than hard disks - (from the article) Spark jobs don't (didn't??) saturate the network - (from the article) Spark jobs needed to write the same data to many buffers to get anywhere (exporting?) - memory mapped IO is here to help (this seems like they're mixing memory mapped device IO Is the gist that because arrow does not require much serialization/deserialization, memory mapped network devices can shoot it out to another server (w/ S3 as an intermediary) at network speed, skip the disk, and get used at the destination faster? I guess the other method was program-1 -> disk -> network -> disk -> program-2 and now it's program-1 -> s3 -> program-2 ?
- gas9S9zw3P9c 6y ago
  You're not the only one. I've worked quite a bit with Minio, Arrow, and Spark, and I don't really understand the point the article is trying to make or how it's related to Minio. It's either badly explained or just a fluff piece throwing together a bunch of technologies. To me, the article says "columnar formats that don't require deserialization are good, oh, and you can use Minio to store data!"
- mindhash 6y ago
  I was involved in Arrow project for a while. The goal of the project is make transfers easier and faster (zero-copy) between dev eco-systems (java->python, python->go) and across network (to database, s3, spark server etc). And yes it has nothing to do with minio any more than it has with s3 or any cloud FS. There are easy to use writers and readers that also handle streaming. Last time I checked, > The most common usage is parquet handling. > python to JAVA IPC was in place. > The communication across systems uses protobuf and that is optimised for web-level transfers and not large scale data transfers yet. So transfers still have work to be done There was also an in-memory object sharing module. I think its no longer part of the repo now.
  - 6y ago
vikrantrathore 6y ago
Just be careful when you use minIO, as it does not support all of the S3 API's. The maintainer of the project will close the tickets for errors when using S3 libraries and ask to write custom code specific for minIO which works well in amazon S3 [1]. So if your company is ready to invest in MinIO without S3 compatibility it's a nice software, and my kudos to the team who took the efforts to build it. It's just that it's not fully S3 compatible and MinIO buckets do not behave the same as S3 buckets. [1] https://github.com/minio/minio/issues/10160 https://github.com/minio/minio/issues/10160
- polskibus 6y ago
  That's really bad. Their marketing says upfront that they are designed for S3 API. If they don't strive for maximum compatibility, then their value prop goes out of the window. I was wondering if I should use minio as a compat layer to be able to use other tools that integrate with S3, what you pointed out probably saved me lots of problem investigation time on the future.
- stingraycharles 6y ago
  We’re using minio using Amazon’s own S3 SDK, and while they don’t support the latest S3 protocol version, it’s definitely not as bad as you say it is. We had more problems with Google Cloud Storage’s S3 API than with Minio’s (I believe there was something related to listing directories with their API as well, but it’s returning a flat out error instead). The key thing to note is that it’s probably not fully compatible, but it’s definitely compatible enough for many use cases. I can also see in the issue you reported that you changed the title of the issue from "Minio is not fully S3 compatible" to "Minio is not S3 compatible", which strikes me as you having some personal beef with them.
  - vikrantrathore 6y ago
    There isn't any personal beef, I just take objections to the marekting line The defacto standard for Amazon S3 compatibility. In my view it's misleading as minIO is not a drop-in replacement for S3 API based applications. My team spend significant efforts to diagnose and find the error, as I believed in this line and asked them to retry different ways by changing code again and again, even though code was working fine with Amazon S3. I changed the title so that it can warn others, to not go through the same shooting the foot believing MinIO is a drop in replacement for Amazon S3 based applications. If you read the thread you can notice there isn't any plan to support posix like folder API in minIO which is supported in Amazon S3.
nl 6y ago
Back a few years ago (2017 I think) another dev and I took a Spark based system that was taking 3+ days to do aggregate processing over HBase-stored data. We switched to it to Parquet on HDFS and did some (ok, a lot...)performance tuning and exactly the same processing ended up at 6 seconds. TL;DR: You really really need to understand how Spark does data procesisng.
- idunno246 6y ago
  I’ve had some bad luck with hbase for similar but not tried spark - it’s better optimized for writers than readers iirc. Pinterest[1] did an interesting thing where they worked directly on the hfiles. Everywhere I’ve worked I feel like I could pick any random data processing job that took more than a few minutes and get orders of magnitude improvement, there’s just so much inefficiency [1]https://medium.com/pinterest-engineering/open-sourcing-terrapin-a-serving-system-for-batch-generated-data-7aa2f38c4472 https://medium.com/pinterest-engineering/open-sourcing-terra...
  - anshumaniax 6y ago
    What do you mean inefficiency. We use the bulk upload feature and do billions of puts in an hour and our scans can go against 3 billion rows an hour. HBase scales linearly and we are already operating it on 5 times what we had designed it for
    - idunno246 6y ago
      Ah I meant that as separate comment, not hbase specifically, but that data pipelines need updating over time. Generally since what’s worth optimizing for changes as size increase, and there’s always that years old pipeline that takes hours to run that with a few changes could be minutes
- anshumaniax 6y ago
  Really we went the exact other way for one of our use cases which required table scans and spark just lost. HBase just stores bytes in sorted order and once you know how to optimize for storage lots of wins can be achieved. I guess the use case here was aggregation so can definitely see some spark advantages.
mekster 6y ago
What is the use case of minio? Typically server disks are much more expensive and turning it into s3 compatible storage doesn't seem like a good way of utilizing it.
- markonen 6y ago
  We’re using Minio storage on our own hardware to save on S3 egress transfer fees. It wouldn’t be very attractive based on the storage fee alone, but if you have any traffic at all, the traffic will dominate your calculations. Of course, you can also get these savings with an online service like Wasabi or Backblaze B2. I just feel that using S3 as your only storage is fine, but once you move to the budget options you need an external backup. So we are using Minio and backing up to Wasabi, and still saving a lot.
  - mekster 6y ago
    I wonder but if you have access to the actual file system, why put the S3 layer unless the app is already built to use S3 in mind?
- stingraycharles 6y ago
  We use minio in our on-prem dev and CI environments, seems to be perfect for this use case.
- MogwaiAllOnYou 6y ago
  We use it as for on-prem deployments where cloud deployments use S3, also useful to spin up for local dev with Vagrant
- selfhoster11 6y ago
  Non-commercial workloads that don't need 11 nines of reliability. S3 is fairly expensive for those use cases, so Minio is a good option to replace it if you need an object-based data store (for backups, for instance).
  - mekster 6y ago
stefanha 6y ago
> Usually applications use Port mapped I/O. This is not true. On a Linux system run "cat /proc/ioports" to see the Port mapped I/O space. There are no network card or storage controller registers in Port mapped I/O space on my machine so network and disk I/O does not involve Port mapped I/O. Modern devices tend to use Memory Mapped I/O registers (see "cat /proc/iomem"). Is the author confusing MMIO vs PIO with mmap vs syscalls? They are two different things. Using mmap can be more efficient than using syscalls because the application has direct access to the page cache, no syscall overhead, and no memory copies.
nicolast 6y ago
Not an Arrow expert at all, so I may be missing something, but I fail to understand the "The Science of Reading/Writing Data" section, or rather, its relevance to the article (and Arrow). From what I could find, Arrow supports reading (writing?) data from (to?) memory-mapped files (i.e., memory regions created through mmap and friends). However, this has no relation to how the IO is being done, hence not related to access to IO devices using either IO ports, or memory mappings (DMA and such). This section seems to be mixing up two fairly distinct concepts, i.e., talking about ways to access IO devices and transfer data to/from them (among which memory mapping is an option), where the memory mapping (mmap of files) as used by Arrow is something a little different.
- formerly_proven 6y ago
  The section is completely false. Memory-mapping files maps pages from the page cache, which lives in main memory. I suppose the author confused this with memory-mapped I/O and then confused port-based I/O with applications using syscalls. I see how you can arrive in that situation when you're only viewing the system through high-level abstractions in Java. > Thanks to Wes McKinney for this brilliant innovation, its not a surprise that such an idea came from him and team, as he is well known as the creator of Pandas in Python. He calls Arrow as the future of data transfer. I assume the confusion is with the author of the blog post and not Wes mcKinney, so this callout in that context is a real disservice. > The output which displays the time, shows the power of this approach. The performance boost is tremendous, almost 50%. Keep in mind that this is reading a 2 MB file with 100k entries, which somehow manages to consume half a second of CPU time. The author compares wall time and not CPU time; both runs consume somewhere between 600 ms and over a second of wall time (again, handling 2 MB of data). I wouldn't be surprised if the first call simply takes so long because it is lazily loading a bunch of code. Later on memory consumption is measured, and one of the file format readers manages to consume -1 MB. This article has a very bad smell.

71 points by jtsymonds 6y ago

wesm 6y ago

(Wes here) I appreciate the Arrow shout-out but note that Apache Arrow has been a major open source community collaboration and not something I can take sole credit for.

anderspitman 6y ago

Is Arrow data commonly written to disk? I was under the impression (as someone who doesn't use it currently) it's usually converted to Parquet for this purpose.

thundergolfer 6y ago
I'd be interested to know any benefits of converting to Parquet if you know your reader supports Arrow.
- lmeyerov 6y ago
  Compression. For our app, we do multilevel caches, where warm is arrow (in-memory, inflight, streaming, ...) and cold (persistent/scaleout) is parquet (ssd, s3, ...).
andygrove 6y ago
Arrow is a memory format and optimized for efficient vectorized processing in memory. Although it is possible to persist the Arrow format to disk, it is more common to use Parquet.

gizmodo59 6y ago

Is there any specific optimizations done in minio code for arrow or it’s just that in general reading from arrow format gives high performance in analytics?

georgewfraser 6y ago

I am truly baffled by the desire to build new data lakes. I understand why Hadoop data lakes came to be: Teradata charged a lot for storage. But now there are database systems that use object storage under-the-hood: Snowflake, BigQuery, Redshift RA3, Databricks Delta Lake. They give you the same low cost of object storage, but with a nice high-level relational interface. I am mystified by why someone would want to build their own data lake rather than just use one of these systems.

sgt101 6y ago
Surely those are all tools for building a data lake, and the stack described in the article is just another tool for doing the same? It's a reasonable architectural choice...
jmngomes 6y ago
All the tools you mention are great for their specific use cases, e.g. Snowflake BigQuery and RedShift are great for analytics over big data, the very common COUNT/SUM/AVG/PARTITION/GROUP BY analysis. But as with any other tool, they're not a good/perfect fit for diverse analysis methods of diverse data types, which I think is what a data lake aims at. Analyzing a large JSON dataset on Snowflake is possible, but either too slow or expensive when compared with more appropriate tools (e.g. Elasticsearch or a Python notebook running PySpark). Having a data lake - which I understand as a repository of raw data of diverse types, regardless of the tools - structured in a tool like S3 is very useful when you have multiple use cases over data of different kinds. For example, you could store audio files from customer calls and have them processed automatically by Spark jobs (e.g. for transcript and stats generation), structure and store call stats on a database for analytics, and do further analysis via notebooks on data science initiatives (e.g. sentiment analysis). This is akin to having a staging area for complex and diverse data types, and S3 is useful for this because of its speed, scale and management features. Teradata or Snowflake aren't a great fit for use cases like these, but they are great if the use case is to get answers to questions like "top 3 operators per team in volume of calls, by department and region, in last quarter" if the volume of calls is big. If I understood correctly, I think your comment was more focused on why use new tools when the existing are mature, but I think big data tools have had to become more specialized and targeted for specific use cases. But if the question is "why build more than one data lake", the only reason I can see is organizational: teams or different areas of an organization either need their own data lake because they have specific needs (which is rare) or won't/can't collaborate with others to have a shared asset.

hardwaresofton 6y ago

Am I the only one that just can't grok what Arrow is doing and how minio in particular made it faster? I read/skimmed this submission and I've worked out this much: - Arrow is an efficient object format that helps skip serialization - Spark works in memory - Minio makes storage of data scalable and cheaper than hard disks - (from the article) Spark jobs don't (didn't??) saturate the network - (from the article) Spark jobs needed to write the same data to many buffers to get anywhere (exporting?) - memory mapped IO is here to help (this seems like they're mixing memory mapped device IO Is the gist that because arrow does not require much serialization/deserialization, memory mapped network devices can shoot it out to another server (w/ S3 as an intermediary) at network speed, skip the disk, and get used at the destination faster? I guess the other method was program-1 -> disk -> network -> disk -> program-2 and now it's program-1 -> s3 -> program-2 ?

gas9S9zw3P9c 6y ago
You're not the only one. I've worked quite a bit with Minio, Arrow, and Spark, and I don't really understand the point the article is trying to make or how it's related to Minio. It's either badly explained or just a fluff piece throwing together a bunch of technologies. To me, the article says "columnar formats that don't require deserialization are good, oh, and you can use Minio to store data!"
mindhash 6y ago
I was involved in Arrow project for a while. The goal of the project is make transfers easier and faster (zero-copy) between dev eco-systems (java->python, python->go) and across network (to database, s3, spark server etc). And yes it has nothing to do with minio any more than it has with s3 or any cloud FS. There are easy to use writers and readers that also handle streaming. Last time I checked, > The most common usage is parquet handling. > python to JAVA IPC was in place. > The communication across systems uses protobuf and that is optimised for web-level transfers and not large scale data transfers yet. So transfers still have work to be done There was also an in-memory object sharing module. I think its no longer part of the repo now.
- 6y ago

vikrantrathore 6y ago

Just be careful when you use minIO, as it does not support all of the S3 API's. The maintainer of the project will close the tickets for errors when using S3 libraries and ask to write custom code specific for minIO which works well in amazon S3 [1]. So if your company is ready to invest in MinIO without S3 compatibility it's a nice software, and my kudos to the team who took the efforts to build it. It's just that it's not fully S3 compatible and MinIO buckets do not behave the same as S3 buckets. [1] https://github.com/minio/minio/issues/10160 https://github.com/minio/minio/issues/10160

polskibus 6y ago
That's really bad. Their marketing says upfront that they are designed for S3 API. If they don't strive for maximum compatibility, then their value prop goes out of the window. I was wondering if I should use minio as a compat layer to be able to use other tools that integrate with S3, what you pointed out probably saved me lots of problem investigation time on the future.
stingraycharles 6y ago
We’re using minio using Amazon’s own S3 SDK, and while they don’t support the latest S3 protocol version, it’s definitely not as bad as you say it is. We had more problems with Google Cloud Storage’s S3 API than with Minio’s (I believe there was something related to listing directories with their API as well, but it’s returning a flat out error instead). The key thing to note is that it’s probably not fully compatible, but it’s definitely compatible enough for many use cases. I can also see in the issue you reported that you changed the title of the issue from "Minio is not fully S3 compatible" to "Minio is not S3 compatible", which strikes me as you having some personal beef with them.
- vikrantrathore 6y ago
  There isn't any personal beef, I just take objections to the marekting line The defacto standard for Amazon S3 compatibility. In my view it's misleading as minIO is not a drop-in replacement for S3 API based applications. My team spend significant efforts to diagnose and find the error, as I believed in this line and asked them to retry different ways by changing code again and again, even though code was working fine with Amazon S3. I changed the title so that it can warn others, to not go through the same shooting the foot believing MinIO is a drop in replacement for Amazon S3 based applications. If you read the thread you can notice there isn't any plan to support posix like folder API in minIO which is supported in Amazon S3.

nl 6y ago

Back a few years ago (2017 I think) another dev and I took a Spark based system that was taking 3+ days to do aggregate processing over HBase-stored data. We switched to it to Parquet on HDFS and did some (ok, a lot...)performance tuning and exactly the same processing ended up at 6 seconds. TL;DR: You really really need to understand how Spark does data procesisng.

idunno246 6y ago
I’ve had some bad luck with hbase for similar but not tried spark - it’s better optimized for writers than readers iirc. Pinterest[1] did an interesting thing where they worked directly on the hfiles. Everywhere I’ve worked I feel like I could pick any random data processing job that took more than a few minutes and get orders of magnitude improvement, there’s just so much inefficiency [1]https://medium.com/pinterest-engineering/open-sourcing-terrapin-a-serving-system-for-batch-generated-data-7aa2f38c4472 https://medium.com/pinterest-engineering/open-sourcing-terra...
- anshumaniax 6y ago
  What do you mean inefficiency. We use the bulk upload feature and do billions of puts in an hour and our scans can go against 3 billion rows an hour. HBase scales linearly and we are already operating it on 5 times what we had designed it for
  - idunno246 6y ago
    Ah I meant that as separate comment, not hbase specifically, but that data pipelines need updating over time. Generally since what’s worth optimizing for changes as size increase, and there’s always that years old pipeline that takes hours to run that with a few changes could be minutes
anshumaniax 6y ago
Really we went the exact other way for one of our use cases which required table scans and spark just lost. HBase just stores bytes in sorted order and once you know how to optimize for storage lots of wins can be achieved. I guess the use case here was aggregation so can definitely see some spark advantages.

mekster 6y ago

What is the use case of minio? Typically server disks are much more expensive and turning it into s3 compatible storage doesn't seem like a good way of utilizing it.

markonen 6y ago
We’re using Minio storage on our own hardware to save on S3 egress transfer fees. It wouldn’t be very attractive based on the storage fee alone, but if you have any traffic at all, the traffic will dominate your calculations. Of course, you can also get these savings with an online service like Wasabi or Backblaze B2. I just feel that using S3 as your only storage is fine, but once you move to the budget options you need an external backup. So we are using Minio and backing up to Wasabi, and still saving a lot.
- mekster 6y ago
  I wonder but if you have access to the actual file system, why put the S3 layer unless the app is already built to use S3 in mind?
stingraycharles 6y ago
We use minio in our on-prem dev and CI environments, seems to be perfect for this use case.
MogwaiAllOnYou 6y ago
We use it as for on-prem deployments where cloud deployments use S3, also useful to spin up for local dev with Vagrant
selfhoster11 6y ago
Non-commercial workloads that don't need 11 nines of reliability. S3 is fairly expensive for those use cases, so Minio is a good option to replace it if you need an object-based data store (for backups, for instance).
- mekster 6y ago

stefanha 6y ago

> Usually applications use Port mapped I/O. This is not true. On a Linux system run "cat /proc/ioports" to see the Port mapped I/O space. There are no network card or storage controller registers in Port mapped I/O space on my machine so network and disk I/O does not involve Port mapped I/O. Modern devices tend to use Memory Mapped I/O registers (see "cat /proc/iomem"). Is the author confusing MMIO vs PIO with mmap vs syscalls? They are two different things. Using mmap can be more efficient than using syscalls because the application has direct access to the page cache, no syscall overhead, and no memory copies.

nicolast 6y ago

Not an Arrow expert at all, so I may be missing something, but I fail to understand the "The Science of Reading/Writing Data" section, or rather, its relevance to the article (and Arrow). From what I could find, Arrow supports reading (writing?) data from (to?) memory-mapped files (i.e., memory regions created through mmap and friends). However, this has no relation to how the IO is being done, hence not related to access to IO devices using either IO ports, or memory mappings (DMA and such). This section seems to be mixing up two fairly distinct concepts, i.e., talking about ways to access IO devices and transfer data to/from them (among which memory mapping is an option), where the memory mapping (mmap of files) as used by Arrow is something a little different.

formerly_proven 6y ago
The section is completely false. Memory-mapping files maps pages from the page cache, which lives in main memory. I suppose the author confused this with memory-mapped I/O and then confused port-based I/O with applications using syscalls. I see how you can arrive in that situation when you're only viewing the system through high-level abstractions in Java. > Thanks to Wes McKinney for this brilliant innovation, its not a surprise that such an idea came from him and team, as he is well known as the creator of Pandas in Python. He calls Arrow as the future of data transfer. I assume the confusion is with the author of the blog post and not Wes mcKinney, so this callout in that context is a real disservice. > The output which displays the time, shows the power of this approach. The performance boost is tremendous, almost 50%. Keep in mind that this is reading a 2 MB file with 100k entries, which somehow manages to consume half a second of CPU time. The author compares wall time and not CPU time; both runs consume somewhere between 600 ms and over a second of wall time (again, handling 2 MB of data). I wouldn't be surprised if the first call simply takes so long because it is lazily loading a bunch of code. Later on memory consumption is measured, and one of the file format readers manages to consume -1 MB. This article has a very bad smell.