3 ms·

Why does every data science infrastructure provider want to be full stack? I really don't want to use an online IDE, I really don't want to be locked into one p

Why does every data science infrastructure provider want to be full stack? I really don't want to use an online IDE, I really don't want to be locked into one particular way of distributing workloads, I really don't want to move my code from GitHub. All of the individual components look great, but the all-or-nothing proposition is really off-putting.

apatil 12y ago
We definitely don't want to lock you in, just to round off the rough edges of the "on your own" experience. You can create a project from GitHub and push/pull to GitHub from Sense, and our distributed computing setup was chosen to be low-level enough to support many of the existing distributed computing tools like doRedis, but still way easier than provisioning and plumbing servers. We know that there's room for improvement on some workflows, in particular Git interop, but we have our sights on a platform that plays well with as many others as possible. Please ping us with any other thoughts you may have!
gabeiscoding 12y ago
So, what do you want? Just RStudio? Or do you want more utilities that work together and run locally? (i.e. repeatable workflows in project bundles of scripts + data + saved notebook logs etc).
choppaface 12y ago
When approaching a beachhead market, it's best to try to deliver a whole product rather than to spread one's self thin specializing for disparate consumers. Clearly users will appreciate some features of their offering more than others; shooting for full-stack at launch both gets the company more feedback and allows them the most control over the initial relationship. I totally agree that the full-stack offering can be rather off-putting, but the business side of DS still doesn't understand how to reliably make money from this market. Cloudera (and friends) have made money from a full-stack offering, so it's easy try to imitate this success. IMO the online IDE is very useful for expositions and small projects, but indeed a limiting playground for larger projects. I'd love to see SAAS/PAAS do better at co-locating their offerings so that, e.g., I could run part of the stack on my own hardware (which is where the data lives). Perhaps Docker would help facilitate this business model.

apatil 12y ago

We definitely don't want to lock you in, just to round off the rough edges of the "on your own" experience. You can create a project from GitHub and push/pull to GitHub from Sense, and our distributed computing setup was chosen to be low-level enough to support many of the existing distributed computing tools like doRedis, but still way easier than provisioning and plumbing servers. We know that there's room for improvement on some workflows, in particular Git interop, but we have our sights on a platform that plays well with as many others as possible. Please ping us with any other thoughts you may have!

gabeiscoding 12y ago

So, what do you want? Just RStudio? Or do you want more utilities that work together and run locally? (i.e. repeatable workflows in project bundles of scripts + data + saved notebook logs etc).

choppaface 12y ago

When approaching a beachhead market, it's best to try to deliver a whole product rather than to spread one's self thin specializing for disparate consumers. Clearly users will appreciate some features of their offering more than others; shooting for full-stack at launch both gets the company more feedback and allows them the most control over the initial relationship. I totally agree that the full-stack offering can be rather off-putting, but the business side of DS still doesn't understand how to reliably make money from this market. Cloudera (and friends) have made money from a full-stack offering, so it's easy try to imitate this success. IMO the online IDE is very useful for expositions and small projects, but indeed a limiting playground for larger projects. I'd love to see SAAS/PAAS do better at co-locating their offerings so that, e.g., I could run part of the stack on my own hardware (which is where the data lives). Perhaps Docker would help facilitate this business model.