3 ms·

This looks great but as someone who has been doing lots of Spark lately, I feel like this will further worsen the development process. The problem with Spark i

This looks great but as someone who has been doing lots of Spark lately, I feel like this will further worsen the development process. The problem with Spark is that there's a lot of magic that happens underneath and being able to debug and figure out issues is pretty hairy. If we put this on top then there are 2 levels of magic that we will have to go through to figure out what is going on. I wish someone works on this aspect.

kvnkho 4y ago
Hi whinvik, we agree that development in Spark is hard, and that is part of the motivation of Fugue. Spark code couples the distributed orchestration and business logic together. By keeping your code in native Python or Pandas, it will be much easier to develop, debug, and maintain the business logic because your tracebacks will be in native Python. Fugue then takes it to Spark when you are ready to scale.
- whinvik 4y ago
  I appreciate your response but that is not what I was getting at. I understand that with this you only have to write Pandas and then not worry about scaling. First, I think PySpark syntax is much better than the insanity than is Pandas but if you really like Pandas then you can always use Pandas UDF which Spark supports. But let's say that writing only in Pandas is the preferred way. Now comes the magic part. How do I know that it is using the best join? Will it optimize for spills? Will there be OOM's? These are the things we need to worry about which often lead us needing to go deep inside Spark magic. Now if there's another level of magic which is Pandas to Spark transpiling as I imagine you do here, then I have even less of an idea how to tune it. Again I appreciate you are solving a specific problem in a nice way but I feel like we are actually making the problem even more complicated.
  - kvnkho 4y ago
    Ah I understand what you are saying. Not trying to aggressively make you try Fugue out, I appreciate the comments and just want to clarify some stuff. There is no transpiling, we think that is too magical. It's more of routing to the appropriate functions/method. For example, applying the Pandas function per partition. We use Pandas UDF under the hood when it makes sense. Sometimes, we use map partitions as well, it depends on what the user wants to do. So we simplify usage around these methods with minimal overhead (we benchmarked it). Fugue can be adopted as minimally as needed so if you feel the need to write native Spark code or tune Spark, we don't block that from the user. The older Fugue interface was bad in allowing this, so we reworked Fugue to have a suite of standalone functions compatible with Spark/Dask/Ray/Pandas DataFrames. Our opinion (which is totally fine if you disagree), is that most workloads don't really specific features of Spark. There are times when it makes sense to write native Spark code for sure, and for those Fugue won't be a good fit. Our job is to make sure it's not an all or nothing thing that requires the user to make compromises.
goodwanghan 4y ago
Hello whinvik, I totally agree the debugging experience of Spark is suboptimal. But why? I don't think the main reason is too much 'magic', I think it is because of the coupling of user's logic and the complicated backend logic. When there is an issue, not only you have to wait for a long time to get the error back, but also you get the error of the whole execution chain, and you have to find the real error in it, which is challenging even for experienced Spark users. Not only Spark, if you use Dask or Ray, you will find very similar experience. Can we directly make Spark Dask Ray easier to debug? I doubt. This is because the complexity of these systems are necessary. And since the problem can happen anywhere either on the user side or on the backend side, the whole stack has to be investigated. It is inevitable, otherwise, people will start to complain the real errors are not captured by the system. What can we do about it? Fugue's idea is to let you iterate on your own logic locally before using these complex systems. So you can get quicker and less noisy feedbacks. The challenge is how to unify the semantic so you don't need to change code moving between local and distributed. And how to trim down the system errors so you can focus more on your own problems. These are what Fugue solves. It's all about good practices It's common sense that if you can keep your logic agnostic to backends, and if you can iterate locally on small data, the development experience can be a lot better. But why big data developers tend not to follow these practices? Because there is too much discrepancy created by the distributed backends intentionally or unintentionally. We created such abstraction for big data developers to get more rewards than punishment when following good practices, so they are more willing to follow good practices.

kvnkho 4y ago

Hi whinvik, we agree that development in Spark is hard, and that is part of the motivation of Fugue. Spark code couples the distributed orchestration and business logic together. By keeping your code in native Python or Pandas, it will be much easier to develop, debug, and maintain the business logic because your tracebacks will be in native Python. Fugue then takes it to Spark when you are ready to scale.

whinvik 4y ago
I appreciate your response but that is not what I was getting at. I understand that with this you only have to write Pandas and then not worry about scaling. First, I think PySpark syntax is much better than the insanity than is Pandas but if you really like Pandas then you can always use Pandas UDF which Spark supports. But let's say that writing only in Pandas is the preferred way. Now comes the magic part. How do I know that it is using the best join? Will it optimize for spills? Will there be OOM's? These are the things we need to worry about which often lead us needing to go deep inside Spark magic. Now if there's another level of magic which is Pandas to Spark transpiling as I imagine you do here, then I have even less of an idea how to tune it. Again I appreciate you are solving a specific problem in a nice way but I feel like we are actually making the problem even more complicated.
- kvnkho 4y ago
  Ah I understand what you are saying. Not trying to aggressively make you try Fugue out, I appreciate the comments and just want to clarify some stuff. There is no transpiling, we think that is too magical. It's more of routing to the appropriate functions/method. For example, applying the Pandas function per partition. We use Pandas UDF under the hood when it makes sense. Sometimes, we use map partitions as well, it depends on what the user wants to do. So we simplify usage around these methods with minimal overhead (we benchmarked it). Fugue can be adopted as minimally as needed so if you feel the need to write native Spark code or tune Spark, we don't block that from the user. The older Fugue interface was bad in allowing this, so we reworked Fugue to have a suite of standalone functions compatible with Spark/Dask/Ray/Pandas DataFrames. Our opinion (which is totally fine if you disagree), is that most workloads don't really specific features of Spark. There are times when it makes sense to write native Spark code for sure, and for those Fugue won't be a good fit. Our job is to make sure it's not an all or nothing thing that requires the user to make compromises.

goodwanghan 4y ago

Hello whinvik, I totally agree the debugging experience of Spark is suboptimal. But why? I don't think the main reason is too much 'magic', I think it is because of the coupling of user's logic and the complicated backend logic. When there is an issue, not only you have to wait for a long time to get the error back, but also you get the error of the whole execution chain, and you have to find the real error in it, which is challenging even for experienced Spark users. Not only Spark, if you use Dask or Ray, you will find very similar experience. Can we directly make Spark Dask Ray easier to debug? I doubt. This is because the complexity of these systems are necessary. And since the problem can happen anywhere either on the user side or on the backend side, the whole stack has to be investigated. It is inevitable, otherwise, people will start to complain the real errors are not captured by the system. What can we do about it? Fugue's idea is to let you iterate on your own logic locally before using these complex systems. So you can get quicker and less noisy feedbacks. The challenge is how to unify the semantic so you don't need to change code moving between local and distributed. And how to trim down the system errors so you can focus more on your own problems. These are what Fugue solves. It's all about good practices It's common sense that if you can keep your logic agnostic to backends, and if you can iterate locally on small data, the development experience can be a lot better. But why big data developers tend not to follow these practices? Because there is too much discrepancy created by the distributed backends intentionally or unintentionally. We created such abstraction for big data developers to get more rewards than punishment when following good practices, so they are more willing to follow good practices.