4 ms·

How they scale automatically on python (a non-functional language)? I am puzzled.

by liuliu 17y ago

m0th87 17y ago
They probably spawn several python processes
- miloshh 17y ago
  Indeed, seems like a single cloud.call will not give you any parallelism, you need many calls. So the whole parallelization challenge is left completely up to you. Still, this is a nice approach, and I'd like to see more companies in this space.
  - willbmoss 17y ago
    Agreed, a very cool idea. I believe you are right on the parallelism front, cloud.call just offloads the processing on their servers, but you'd have to call it multiple times to get any parallelism.
    - usaar333 17y ago
      PiCloud dev here. That is correct. We also offer a mapping function, cloud.map(func, arg_list), where every func(arg_list[i]) will be evaluated in parallel.
sgk284 17y ago
This is just a way to call a function and run it on a different server(s). You still have to design your program to handle parallelism.
frisco 17y ago
Python is a functional language in the sense that it has first class functions and you can pass them around.
- m0th87 17y ago
  Yes but it's not purely functional in that you cannot automagically make a script scale to n threads, as is the case with Haskell. This is what the OP is referring to.
  - usaar333 17y ago
    Correct. cloud.call, etc. are all higher-level functions. PiCloud does treat cloud.call(func) dispatches as completely functional. The function evaluated cannot have side-effects in terms of the program space; within the program, you access its output via cloud.result(). (Note: You can within the func have external side-effects - e.g. opening a connection to a database and modifying entries).
    - nico_h 17y ago
      - edit : nevermind, found the answer in the 'pitfalls' doc : yes you can create picloud call from within picloud, but they of course go against your maximum count of parallel calls - Hi I have read (quickly) your webscraper example and i still have one question : Is it possible to launch piclouds jobs from within a picloud job ? i.e. from picloud download some list of url as csvs and then start webcrapers then have the original function return the results of these scrapers ? Thanks for your answer.
  - _delirium 17y ago
    You can't really do that either in Haskell. Well, in theory you can, but in practice nobody has yet written a way to autoparallelize Haskell that actually gets you reasonable speedups, because it requires predicting which parts are worth parallelizing. Instead, the Haskell devs are mostly taking the same approach, of providing constructs for the programmer to explicitly write parallel code: http://www.haskell.org/ghc/docs/6.6/html/libraries/base/Control-Parallel.html http://www.haskell.org/ghc/docs/6.6/html/libraries/base/Cont...

How they scale automatically on python (a non-functional language)? I am puzzled.

by liuliu 17y ago

How they scale automatically on python (a non-functional language)? I am puzzled.

m0th87 17y ago

They probably spawn several python processes

miloshh 17y ago
Indeed, seems like a single cloud.call will not give you any parallelism, you need many calls. So the whole parallelization challenge is left completely up to you. Still, this is a nice approach, and I'd like to see more companies in this space.
- willbmoss 17y ago
  Agreed, a very cool idea. I believe you are right on the parallelism front, cloud.call just offloads the processing on their servers, but you'd have to call it multiple times to get any parallelism.
  - usaar333 17y ago
    PiCloud dev here. That is correct. We also offer a mapping function, cloud.map(func, arg_list), where every func(arg_list[i]) will be evaluated in parallel.

sgk284 17y ago

This is just a way to call a function and run it on a different server(s). You still have to design your program to handle parallelism.

frisco 17y ago

Python is a functional language in the sense that it has first class functions and you can pass them around.

m0th87 17y ago
Yes but it's not purely functional in that you cannot automagically make a script scale to n threads, as is the case with Haskell. This is what the OP is referring to.
- usaar333 17y ago
  Correct. cloud.call, etc. are all higher-level functions. PiCloud does treat cloud.call(func) dispatches as completely functional. The function evaluated cannot have side-effects in terms of the program space; within the program, you access its output via cloud.result(). (Note: You can within the func have external side-effects - e.g. opening a connection to a database and modifying entries).
  - nico_h 17y ago
    - edit : nevermind, found the answer in the 'pitfalls' doc : yes you can create picloud call from within picloud, but they of course go against your maximum count of parallel calls - Hi I have read (quickly) your webscraper example and i still have one question : Is it possible to launch piclouds jobs from within a picloud job ? i.e. from picloud download some list of url as csvs and then start webcrapers then have the original function return the results of these scrapers ? Thanks for your answer.
- _delirium 17y ago
  You can't really do that either in Haskell. Well, in theory you can, but in practice nobody has yet written a way to autoparallelize Haskell that actually gets you reasonable speedups, because it requires predicting which parts are worth parallelizing. Instead, the Haskell devs are mostly taking the same approach, of providing constructs for the programmer to explicitly write parallel code: http://www.haskell.org/ghc/docs/6.6/html/libraries/base/Control-Parallel.html http://www.haskell.org/ghc/docs/6.6/html/libraries/base/Cont...