15 ms·

JavaScript and the next decade of data programming (2020)

122 points by tim_sw 5y ago

flakiness 5y ago
Although I share the sentiments the author has against Python and Cloud, calling it JS a "backend" of the data science stack is a non-trivial stretch. It's obviously a front-end. Having a fast GPU computation is great for frontend as well as backend, so don't worry it to be wasted. I'd rather expect cloud vendors adopt JS/wasm as a UDF / extension for their compute offering. BigQuery does this already [1]. S3 Object lambda [2] is close. Please leave web browser as a frontend. [1] https://cloud.google.com/bigquery/docs/reference/standard-sql/user-defined-functions https://cloud.google.com/bigquery/docs/reference/standard-sq... [2] https://aws.amazon.com/blogs/aws/introducing-amazon-s3-object-lambda-use-your-code-to-process-data-as-it-is-being-retrieved-from-s3/ https://aws.amazon.com/blogs/aws/introducing-amazon-s3-objec...
- nine_k 5y ago
  It can be a "backend" the way Python is a "backend". It's the glue, the developer-facing API. The heavy lifting will likely be done by cross-compiling C++ and Fortran code to WASM, and expecting it to be JITted, or maybe rewriting portions of in in JS where JIT is already good. Whoever is first to implement a well-working GPGPU interface in the browser, wins.
  - lasagnaphil 5y ago
    WASM won’t be performant enough for heavy matrix computations because it wouldn’t be able to use SIMD (in its current state). There’s a WASM SIMD proposal in the works but it’s only for 128-bit vectors and I’m not sure it will perform the same as native code.
- rektide 5y ago
  just one example: deno has webgpu support. whatever mentality you hold now, however you think if he today, I expect almost all of us need to loosen our preconceptions rather quickly. as for what the browser ought to be, I again think you should lower your preconceptions & not be so opposed to the browser exploring we data crunching roles. especially as folk explore moving away from big cloud being near-exclusive means of computing.
  - flakiness 5y ago
    That's one attitude. I'm on the other side where having low expectation to the browser technology. People claim a lot on Web and Browser but it takes extremely wrong to get the fruit. I'm happy to be wrong eventually and will take it if it does come true. But until then, I'd stick on the pragmatic side. That's said, someone has to chase the ambition to make it happen. Maybe I was being too negative and discouraging here.
da39a3ee 5y ago
> But if you need to continually be checking random samples of a dataframe, re-running modules, and seeing if your regular expressions correctly clean a dataset, you are using a notebook interface today, even if you bundle your code into a module at some point. Notebooks like jupyter and observable are not a good advance. The primary reason is that they are leading the community of data programmers away from git-based workflows. ("git" is a shorthand for whatever version-control here due to its dominance). Although it may seem like a strange claim, the primary reason "programmers" and "software engineers" use git is for correctness/debugging: the idea of not being to switch accurately between different states of your codebase to compare program behavior is insanity to a "programmer" / "software engineer". And it should be insanity also to data programmers. But due to the rise in popularity of notebooks, it is not. These things just do not encourage the use of version control to compare different states of the code. It's going to lead to many buggy analyses, and many data programmers missing the opportunity to learn how to write code to a high level of correctness.
- jshen 5y ago
  There is nothing inherent in notebooks that prevent the use of tools like git.
  - nine_k 5y ago
    ...except the need to commit the current state to make things reproducible. And I'm afraid it's not just code in cells, but also the sequence of its execution, editing, execution again, etc, because the order of running the cells is arbitrary (whatever the user commands interactively), and the results of previous invocations form the dreaded arbitrarily mutable global state.
    - tomnipotent 5y ago
      > ...except the need to commit the current state to make things reproducible. This can definitely be a problem, but at least in the instances where I've worked with teams using Notebooks it hasn't been an issue. Usually data is either coming from networked storage (S3 etc.) as CSV/Parquet/HDF5, a database query with well-defined (enough) criterion, or API calls. > because the order of running the cells is arbitrary Most people I know have to restart their notebooks often enough to have to re-run all cells again anyway. You usually get into a good habit of making sure they're mostly "in order", or grouping common code into cells to minimize the number of cell executions and ctrl+enter's needed. > dreaded arbitrarily mutable global state It's definitely a double-edged sword, but the global state is a big part of what makes notebooks so useful. I have a repl-based environment, the ability to run arbitrary code, and global state to iteratively explore the domain of the problem I'm working on. That sword cuts me many times along the way, but they're (mostly) nicks. But it's usually worth the output, especially to the business. In all these instances, teams shared notebooks through git and most of the issues were simply understanding the context of their usage more than issues with their execution. I don't remember more than a few notebooks really being used by more than a few people anyway, so I've learned to consider them more as collaborative artifacts in an exploratory process to more mature, "packaged" code evolved from those notebooks.
da39a3ee 5y ago
> When I started tinkering with JS, I thought of it as slow; but web developers are far more obsessive about speed than any other high-level, dynamically typed language I’ve seen. Given that, isn't it odd that JS still encourages map/filter array iterations that make multiple passes over a single array? I.e. that it hasn't yet introduced some sort of comprehension or itertools-with-compilation feature that does array processing in a single pass?
- goodoldneon 5y ago
  Comprehensions are nice and I wish JS had them, but can’t you achieve the same thing with a for loop?
  - kwkelly 5y ago
    You can, but transducers would be a more consistent api
- bastawhiz 5y ago
  JavaScript has generator functions. Comprehensions are really just sugar around this functionality. You could pretty easily write your own set of generator function-powered methods for mapping and filtering.
  - brundolf 5y ago
    Which I've started doing, but it is still a real shame that the "default way" is so inefficient, especially given how many devs never even realize it. It contributes a whole lot of accidental-slowness to the ecosystem.
    - pwdisswordfish0 5y ago
      > the "default way" is so inefficient A big reason why I say screw map, filter, reduce, etc. and my "default way" never stopped being ordinary for loops.
lmeyerov 5y ago
Cool to see this :) We wrote the JS version of Arrow with some of these insights in mind. We even had a wasm + webcl backends even before building arrow and then using it to standardize some of our protocols. Going in the wild these last few years, we've had a few fundamental lessons-learned here. A lot of the above is still true, and in fact even more so. At the same time, for the data world, there are tough points like client heterogeneity and data scale. The way I like to describe it now is we've moved from a "thin <> thick" client/server model to "thick <> supercomputer" client/server. With modern latencies and bandwidth, your interactive compute session is distributed GPUs: it's a weird way to build apps and think about what a computer is, but totally commoditized and doable now. So Arrow is great for streaming MB/s to the browser. Potentially in a couple years, GB/s to v8/browser, mostly waiting for some basic issues Chrome devs don't prioritize, not fundamental ones. This gets hard when we want to interactively visually analyze say 10 GB of click trails: that can quickly get into 100 GB of in-memory representation needing streaming / out-of-core tricks, and download speed issues. Likewise, WebGPU is cool, but it's still not even what CUDA / OpenCL were exposing 10 years ago, again, mostly b/c Google WebGL people prioritize other stuff. Given that, at least for the next few years, we've been continuing to build for "1-10 GB client GPU <> 10GB-1TB server GPUs". On the plus side, "on the internet, nobody knows you're a dog", so we've been free to do GPU nodejs extenstions that WebGL standards people don't get to dictate.
- rektide 5y ago
  > Potentially in a couple years, GB/s to v8/browser, mostly waiting for some basic issues Chrome devs don't prioritize, not fundamental ones. This gets hard when we want to interactively visually analyze say 10 GB of click trails: that can quickly get into 100 GB of in-memory representation needing streaming / out-of-core tricks, and download speed issues. what issues aren't being prioritized? > Likewise, WebGPU is cool, but it's still not even what CUDA / OpenCL were exposing 10 years ago, again, mostly b/c Google WebGL people prioritize other stuff. what's missing vs say opencl? a lot of shade cast with very few contestable claims. I want to listen to your points but right now the article seems far more right to me than your ill described counterclaims. I don't see the limits you claim with webgpu sand I don't see what the browser folk need to be attending to.
fulafel 5y ago
About the #1 WebGL complaint, is it really so that you can't use integers in attributes? https://www.khronos.org/registry/webgl/specs/latest/2.0/#3.7.8 https://www.khronos.org/registry/webgl/specs/latest/2.0/#3.7... seems to say otherwise, and also https://developer.mozilla.org/en-US/docs/Web/API/WebGL2RenderingContext/vertexAttribIPointer https://developer.mozilla.org/en-US/docs/Web/API/WebGL2Rende... tells you can use integer types including gl.INT, gl.SHORT etc, and here's someone using int32 textures successfully: https://stackoverflow.com/questions/43905968/how-to-get-32-bit-integer-texture-working-in-webgl-2-0 https://stackoverflow.com/questions/43905968/how-to-get-32-b...
- kylebarron 5y ago
  Probably supported only in WebGL2
- krona 5y ago
  There is no requirement for integer arithmetic in WebGL 1; all numbers are essentially floats, even if the attribute is not float. Integer support is GLSL ES 3.0 (the shader language) i.e. WebGL2.
  - fulafel 5y ago
    Intentionally talking about about the years obsolete WebGL 1 feature set as "WebGL" while proposing WebGPU would be pretty iffy. If the issue is that WebGL 2, an incremental update to WebGL 1, doesn't have 100% user coverage after 4.5 years, WebGPU is going to be much farther off as it's not even fully specced yet. But since the article didn't say anything about webgl versions, it seems more likely that the author was just unaware.
galaxyLogic 5y ago
Summary: "Why bother with R and Python?"
bigger_cheese 5y ago
One thing I'm curious about is how do you get the data to the browser. Particularly if you are interested in doing it in a low latency way with frequent data updates? For instance what do I need to do if I want to have a chart of timeseries data that is being updated in near realtime, say every 5 seconds, appear on a webpage? In my Org data originates from Operational Historians which are essentially ring buffers for time series data and look a lot like SQL databases. How does javascript communicate with this sort of data efficiently. In "old school" Non live updating R world I would use a JDBC connection. What is javascript way of receiving the data?
- mirekrusin 5y ago
  Websockets. We do price feeds for inhouse trading platform and we don't use anything else, it works very well for browser-backend and backend-backend comms.
javierluraschi 5y ago
Couple days ago we posted a Show HN with similar arguments on why JS for data makes sense, see: https://news.ycombinator.com/item?id=27334145 https://news.ycombinator.com/item?id=27334145 -- Ben's article is much more eloquent and presents several examples, but there are a few libraries worth complementing this article: danfo.js (dataframes with tensorflow), ml5.js (a keras-like library), tidy.js (a tidy port to JS), and tensorflos.js.
tralarpa 5y ago
I am involved in a quite heavy web-based data processing and visualization application (browser regularly consumes 8GB RAM and more for a single tab). I have to say that you really can see that browsers are not made for this kind of workload, even if Javascript itself might be fast enough, down to very stupid little things (e.g. accidentially pointing your mouse in the debug window to a variable holding a large array risks to freeze the entire browser).
- fbcpck 5y ago
  Looks like something that can be solved by moving the task to background with [web workers][0]. [0]: https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers...
  - tralarpa 5y ago
    At least with Firefox, doing tasks in the background was sometimes quite painful, especially during development. For example, I do some CPU-intensive stuff in the background, close the tab because I noticed a mistake in my code -> background task still runs for several seconds eating CPU and memory until it is killed or finished (I didn't check to be honest) by the browser. Edit: As with everything related to complex web applications (and I had two or three in my career), it is obvious that the whole environment (client, protocols, languages,...) was never designed for this.
dataviz1000 5y ago
It's possible to do almost everything in JavaScript and SQL. However, the moment a developer wants to use time series forecasting models like ARIMA and ARCH for predictive forecasting there isn't much in either technologies (side cases include Oracle providing ARIMA functions in SQL). Unfortunately, it requires the server handing off computation to Python or R, somehow. Does anyone know of a tutorial how to implement python modules scipy, numpy, pandas, statsmodels, sklearn with a Docker image which can listen for incoming data streamed or otherwise from a node web server?
- tyingq 5y ago
  Doesn't address the docker part, but the syntax for this node -> python bridge looks reasonable, and there's a short comparison section for other solutions: https://github.com/Submersible/node-python-bridge https://github.com/Submersible/node-python-bridge
nojito 5y ago
So much mention of Dplyr and pandas being slow and yet not a single mention of data.table. https://h2oai.github.io/db-benchmark/ https://h2oai.github.io/db-benchmark/
lmc 5y ago
Good post. If the author is reading, the demos are still pretty intensive and a way to toggle them on and off would be much appreciated for those of us on less powerful devices.
GlennS 5y ago
I think for a significant portion of R users the jump to JavaScript is not plausible as things stand. R is one of the few languages / platforms where people who don't give a fuck about software engineering can successfully create mostly correct computer programs with reasonable speed. JavaScript is not ready for those people yet. (Although I am pretty impressed with the work the Observable people are doing in this regard.)
- blacktriangle 5y ago
  Not only is Javascript not ready for those people yet, Javascript has been moving further and further away from those people for years, both in the language spec and in the community and ecosystem around the language.
- jhap 5y ago
  Do you have any advice for R users on making the transition? Even though I am primarily an R user, these days I have been using a fair amount of JS in my work, and I'm not sure how I can get to the "next level" in my programming that your alluding to. It seems like most of the tutorials are either too rudimentary or way over my head (like the three step drawing meme). Maybe it makes sense for someone in my position to just go bite the bullet and go through the Eloquent JavaScript even though I feel like I already have a fair grasp of the content? Or learn some key principles of Software Engineering?
  - dash2 5y ago
    To be blunt, if you can seriously cope with R, javascript should be a piece of piss. R has a very, very steep learning curve... perhaps the transition is less to do with changing languages and more to do with learning principles of software engineering?

122 points by tim_sw 5y ago

Although I share the sentiments the author has against Python and Cloud, calling it JS a "backend" of the data science stack is a non-trivial stretch. It's obviously a front-end. Having a fast GPU computation is great for frontend as well as backend, so don't worry it to be wasted. I'd rather expect cloud vendors adopt JS/wasm as a UDF / extension for their compute offering. BigQuery does this already [1]. S3 Object lambda [2] is close. Please leave web browser as a frontend. [1] https://cloud.google.com/bigquery/docs/reference/standard-sql/user-defined-functions https://cloud.google.com/bigquery/docs/reference/standard-sq... [2] https://aws.amazon.com/blogs/aws/introducing-amazon-s3-object-lambda-use-your-code-to-process-data-as-it-is-being-retrieved-from-s3/ https://aws.amazon.com/blogs/aws/introducing-amazon-s3-objec...

nine_k 5y ago
It can be a "backend" the way Python is a "backend". It's the glue, the developer-facing API. The heavy lifting will likely be done by cross-compiling C++ and Fortran code to WASM, and expecting it to be JITted, or maybe rewriting portions of in in JS where JIT is already good. Whoever is first to implement a well-working GPGPU interface in the browser, wins.
- lasagnaphil 5y ago
  WASM won’t be performant enough for heavy matrix computations because it wouldn’t be able to use SIMD (in its current state). There’s a WASM SIMD proposal in the works but it’s only for 128-bit vectors and I’m not sure it will perform the same as native code.
rektide 5y ago
just one example: deno has webgpu support. whatever mentality you hold now, however you think if he today, I expect almost all of us need to loosen our preconceptions rather quickly. as for what the browser ought to be, I again think you should lower your preconceptions & not be so opposed to the browser exploring we data crunching roles. especially as folk explore moving away from big cloud being near-exclusive means of computing.
- flakiness 5y ago
  That's one attitude. I'm on the other side where having low expectation to the browser technology. People claim a lot on Web and Browser but it takes extremely wrong to get the fruit. I'm happy to be wrong eventually and will take it if it does come true. But until then, I'd stick on the pragmatic side. That's said, someone has to chase the ambition to make it happen. Maybe I was being too negative and discouraging here.

da39a3ee 5y ago

> But if you need to continually be checking random samples of a dataframe, re-running modules, and seeing if your regular expressions correctly clean a dataset, you are using a notebook interface today, even if you bundle your code into a module at some point. Notebooks like jupyter and observable are not a good advance. The primary reason is that they are leading the community of data programmers away from git-based workflows. ("git" is a shorthand for whatever version-control here due to its dominance). Although it may seem like a strange claim, the primary reason "programmers" and "software engineers" use git is for correctness/debugging: the idea of not being to switch accurately between different states of your codebase to compare program behavior is insanity to a "programmer" / "software engineer". And it should be insanity also to data programmers. But due to the rise in popularity of notebooks, it is not. These things just do not encourage the use of version control to compare different states of the code. It's going to lead to many buggy analyses, and many data programmers missing the opportunity to learn how to write code to a high level of correctness.

jshen 5y ago
There is nothing inherent in notebooks that prevent the use of tools like git.
- nine_k 5y ago
  ...except the need to commit the current state to make things reproducible. And I'm afraid it's not just code in cells, but also the sequence of its execution, editing, execution again, etc, because the order of running the cells is arbitrary (whatever the user commands interactively), and the results of previous invocations form the dreaded arbitrarily mutable global state.
  - tomnipotent 5y ago
    > ...except the need to commit the current state to make things reproducible. This can definitely be a problem, but at least in the instances where I've worked with teams using Notebooks it hasn't been an issue. Usually data is either coming from networked storage (S3 etc.) as CSV/Parquet/HDF5, a database query with well-defined (enough) criterion, or API calls. > because the order of running the cells is arbitrary Most people I know have to restart their notebooks often enough to have to re-run all cells again anyway. You usually get into a good habit of making sure they're mostly "in order", or grouping common code into cells to minimize the number of cell executions and ctrl+enter's needed. > dreaded arbitrarily mutable global state It's definitely a double-edged sword, but the global state is a big part of what makes notebooks so useful. I have a repl-based environment, the ability to run arbitrary code, and global state to iteratively explore the domain of the problem I'm working on. That sword cuts me many times along the way, but they're (mostly) nicks. But it's usually worth the output, especially to the business. In all these instances, teams shared notebooks through git and most of the issues were simply understanding the context of their usage more than issues with their execution. I don't remember more than a few notebooks really being used by more than a few people anyway, so I've learned to consider them more as collaborative artifacts in an exploratory process to more mature, "packaged" code evolved from those notebooks.

da39a3ee 5y ago

> When I started tinkering with JS, I thought of it as slow; but web developers are far more obsessive about speed than any other high-level, dynamically typed language I’ve seen. Given that, isn't it odd that JS still encourages map/filter array iterations that make multiple passes over a single array? I.e. that it hasn't yet introduced some sort of comprehension or itertools-with-compilation feature that does array processing in a single pass?

goodoldneon 5y ago
Comprehensions are nice and I wish JS had them, but can’t you achieve the same thing with a for loop?
- kwkelly 5y ago
  You can, but transducers would be a more consistent api
bastawhiz 5y ago
JavaScript has generator functions. Comprehensions are really just sugar around this functionality. You could pretty easily write your own set of generator function-powered methods for mapping and filtering.
- brundolf 5y ago
  Which I've started doing, but it is still a real shame that the "default way" is so inefficient, especially given how many devs never even realize it. It contributes a whole lot of accidental-slowness to the ecosystem.
  - pwdisswordfish0 5y ago
    > the "default way" is so inefficient A big reason why I say screw map, filter, reduce, etc. and my "default way" never stopped being ordinary for loops.

lmeyerov 5y ago

Cool to see this :) We wrote the JS version of Arrow with some of these insights in mind. We even had a wasm + webcl backends even before building arrow and then using it to standardize some of our protocols. Going in the wild these last few years, we've had a few fundamental lessons-learned here. A lot of the above is still true, and in fact even more so. At the same time, for the data world, there are tough points like client heterogeneity and data scale. The way I like to describe it now is we've moved from a "thin <> thick" client/server model to "thick <> supercomputer" client/server. With modern latencies and bandwidth, your interactive compute session is distributed GPUs: it's a weird way to build apps and think about what a computer is, but totally commoditized and doable now. So Arrow is great for streaming MB/s to the browser. Potentially in a couple years, GB/s to v8/browser, mostly waiting for some basic issues Chrome devs don't prioritize, not fundamental ones. This gets hard when we want to interactively visually analyze say 10 GB of click trails: that can quickly get into 100 GB of in-memory representation needing streaming / out-of-core tricks, and download speed issues. Likewise, WebGPU is cool, but it's still not even what CUDA / OpenCL were exposing 10 years ago, again, mostly b/c Google WebGL people prioritize other stuff. Given that, at least for the next few years, we've been continuing to build for "1-10 GB client GPU <> 10GB-1TB server GPUs". On the plus side, "on the internet, nobody knows you're a dog", so we've been free to do GPU nodejs extenstions that WebGL standards people don't get to dictate.

rektide 5y ago
> Potentially in a couple years, GB/s to v8/browser, mostly waiting for some basic issues Chrome devs don't prioritize, not fundamental ones. This gets hard when we want to interactively visually analyze say 10 GB of click trails: that can quickly get into 100 GB of in-memory representation needing streaming / out-of-core tricks, and download speed issues. what issues aren't being prioritized? > Likewise, WebGPU is cool, but it's still not even what CUDA / OpenCL were exposing 10 years ago, again, mostly b/c Google WebGL people prioritize other stuff. what's missing vs say opencl? a lot of shade cast with very few contestable claims. I want to listen to your points but right now the article seems far more right to me than your ill described counterclaims. I don't see the limits you claim with webgpu sand I don't see what the browser folk need to be attending to.

fulafel 5y ago

About the #1 WebGL complaint, is it really so that you can't use integers in attributes? https://www.khronos.org/registry/webgl/specs/latest/2.0/#3.7.8 https://www.khronos.org/registry/webgl/specs/latest/2.0/#3.7... seems to say otherwise, and also https://developer.mozilla.org/en-US/docs/Web/API/WebGL2RenderingContext/vertexAttribIPointer https://developer.mozilla.org/en-US/docs/Web/API/WebGL2Rende... tells you can use integer types including gl.INT, gl.SHORT etc, and here's someone using int32 textures successfully: https://stackoverflow.com/questions/43905968/how-to-get-32-bit-integer-texture-working-in-webgl-2-0 https://stackoverflow.com/questions/43905968/how-to-get-32-b...

kylebarron 5y ago
Probably supported only in WebGL2
krona 5y ago
There is no requirement for integer arithmetic in WebGL 1; all numbers are essentially floats, even if the attribute is not float. Integer support is GLSL ES 3.0 (the shader language) i.e. WebGL2.
- fulafel 5y ago
  Intentionally talking about about the years obsolete WebGL 1 feature set as "WebGL" while proposing WebGPU would be pretty iffy. If the issue is that WebGL 2, an incremental update to WebGL 1, doesn't have 100% user coverage after 4.5 years, WebGPU is going to be much farther off as it's not even fully specced yet. But since the article didn't say anything about webgl versions, it seems more likely that the author was just unaware.

galaxyLogic 5y ago

Summary: "Why bother with R and Python?"

bigger_cheese 5y ago

One thing I'm curious about is how do you get the data to the browser. Particularly if you are interested in doing it in a low latency way with frequent data updates? For instance what do I need to do if I want to have a chart of timeseries data that is being updated in near realtime, say every 5 seconds, appear on a webpage? In my Org data originates from Operational Historians which are essentially ring buffers for time series data and look a lot like SQL databases. How does javascript communicate with this sort of data efficiently. In "old school" Non live updating R world I would use a JDBC connection. What is javascript way of receiving the data?

mirekrusin 5y ago
Websockets. We do price feeds for inhouse trading platform and we don't use anything else, it works very well for browser-backend and backend-backend comms.

javierluraschi 5y ago

Couple days ago we posted a Show HN with similar arguments on why JS for data makes sense, see: https://news.ycombinator.com/item?id=27334145 https://news.ycombinator.com/item?id=27334145 -- Ben's article is much more eloquent and presents several examples, but there are a few libraries worth complementing this article: danfo.js (dataframes with tensorflow), ml5.js (a keras-like library), tidy.js (a tidy port to JS), and tensorflos.js.

tralarpa 5y ago

I am involved in a quite heavy web-based data processing and visualization application (browser regularly consumes 8GB RAM and more for a single tab). I have to say that you really can see that browsers are not made for this kind of workload, even if Javascript itself might be fast enough, down to very stupid little things (e.g. accidentially pointing your mouse in the debug window to a variable holding a large array risks to freeze the entire browser).

fbcpck 5y ago
Looks like something that can be solved by moving the task to background with [web workers][0]. [0]: https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers...
- tralarpa 5y ago
  At least with Firefox, doing tasks in the background was sometimes quite painful, especially during development. For example, I do some CPU-intensive stuff in the background, close the tab because I noticed a mistake in my code -> background task still runs for several seconds eating CPU and memory until it is killed or finished (I didn't check to be honest) by the browser. Edit: As with everything related to complex web applications (and I had two or three in my career), it is obvious that the whole environment (client, protocols, languages,...) was never designed for this.

dataviz1000 5y ago

It's possible to do almost everything in JavaScript and SQL. However, the moment a developer wants to use time series forecasting models like ARIMA and ARCH for predictive forecasting there isn't much in either technologies (side cases include Oracle providing ARIMA functions in SQL). Unfortunately, it requires the server handing off computation to Python or R, somehow. Does anyone know of a tutorial how to implement python modules scipy, numpy, pandas, statsmodels, sklearn with a Docker image which can listen for incoming data streamed or otherwise from a node web server?

tyingq 5y ago
Doesn't address the docker part, but the syntax for this node -> python bridge looks reasonable, and there's a short comparison section for other solutions: https://github.com/Submersible/node-python-bridge https://github.com/Submersible/node-python-bridge

nojito 5y ago

So much mention of Dplyr and pandas being slow and yet not a single mention of data.table. https://h2oai.github.io/db-benchmark/ https://h2oai.github.io/db-benchmark/

lmc 5y ago

Good post. If the author is reading, the demos are still pretty intensive and a way to toggle them on and off would be much appreciated for those of us on less powerful devices.

GlennS 5y ago

I think for a significant portion of R users the jump to JavaScript is not plausible as things stand. R is one of the few languages / platforms where people who don't give a fuck about software engineering can successfully create mostly correct computer programs with reasonable speed. JavaScript is not ready for those people yet. (Although I am pretty impressed with the work the Observable people are doing in this regard.)

blacktriangle 5y ago
Not only is Javascript not ready for those people yet, Javascript has been moving further and further away from those people for years, both in the language spec and in the community and ecosystem around the language.
jhap 5y ago
Do you have any advice for R users on making the transition? Even though I am primarily an R user, these days I have been using a fair amount of JS in my work, and I'm not sure how I can get to the "next level" in my programming that your alluding to. It seems like most of the tutorials are either too rudimentary or way over my head (like the three step drawing meme). Maybe it makes sense for someone in my position to just go bite the bullet and go through the Eloquent JavaScript even though I feel like I already have a fair grasp of the content? Or learn some key principles of Software Engineering?
- dash2 5y ago
  To be blunt, if you can seriously cope with R, javascript should be a piece of piss. R has a very, very steep learning curve... perhaps the transition is less to do with changing languages and more to do with learning principles of software engineering?