3 ms·

I'm in the process of doing something like this internally, at a smaller scale, and it's interesting to see that many of the concepts I've been experimenting wi

I'm in the process of doing something like this internally, at a smaller scale, and it's interesting to see that many of the concepts I've been experimenting with and playing around with are formalized here in a similar manner. My "solution" doesn't build on Spark, as I just don't have enough data to necessitate it. I think the big difference is really the project's SQL first approach, which is probably going to polarize: personally, it's a decision I can't abide by, but I'm sure a lot of people will love that.

seddonm1 6y ago
Cool! Most of the time we run Spark as a single-node (i.e. --master local[*]) as now we can easily utilise large nodes like 128 core, 512GB and Spark does scale vertically well but also runs relatively well on a small node like the Docker example on the website running on a laptop. The ability to run SQL against separate storage is Spark's killer feature in my view. Arc does support the full Scala API which you can implement as a plugin (https://arc.tripl.ai/plugins/ https://arc.tripl.ai/plugins/) so for advanced teams they have full control. The reason we went for SQL-first is that we are trying to find the balance that allows Business Analysts to develop their own logic without having to learn Scala or even Python - as they probably already know SQL. Hopefully some of the ideas are relevant to what you are building.
- lordgroff 6y ago
  I'm reading the docs thoroughly, many excellent ideas, and I'm sure I'll be borrowing some concepts. I also want to commend you for putting in the time in developing proper documentation, always greatly appreciated.
  - seddonm1 6y ago
    Great! I am glad someone is reading them as it has taken a huge amount of effort. We also have a link from the Jupyter Completer (autocomplete) to the docs.

3 ms·

I'm in the process of doing something like this internally, at a smaller scale, and it's interesting to see that many of the concepts I've been experimenting wi

by lordgroff 6y ago

seddonm1 6y ago
Cool! Most of the time we run Spark as a single-node (i.e. --master local[*]) as now we can easily utilise large nodes like 128 core, 512GB and Spark does scale vertically well but also runs relatively well on a small node like the Docker example on the website running on a laptop. The ability to run SQL against separate storage is Spark's killer feature in my view. Arc does support the full Scala API which you can implement as a plugin (https://arc.tripl.ai/plugins/ https://arc.tripl.ai/plugins/) so for advanced teams they have full control. The reason we went for SQL-first is that we are trying to find the balance that allows Business Analysts to develop their own logic without having to learn Scala or even Python - as they probably already know SQL. Hopefully some of the ideas are relevant to what you are building.
- lordgroff 6y ago
  I'm reading the docs thoroughly, many excellent ideas, and I'm sure I'll be borrowing some concepts. I also want to commend you for putting in the time in developing proper documentation, always greatly appreciated.
  - seddonm1 6y ago
    Great! I am glad someone is reading them as it has taken a huge amount of effort. We also have a link from the Jupyter Completer (autocomplete) to the docs.