5 ms·

The 25x speedup is because the gRPC python implementation is very slow [0]. [0] -- https://performance-dot-grpc-testing.appspot.com/explore?dashboard=563647026

by wdroz 7y ago

The 25x speedup is because the gRPC python implementation is very slow [0]. [0] -- https://performance-dot-grpc-testing.appspot.com/explore?dashboard=5636470266134528 https://performance-dot-grpc-testing.appspot.com/explore?das...

kbumsik 7y ago
Yeah, by skimming the article, it sound like making gRPC microservices 25x faster, not ML.
- nevi-me 7y ago
  gRPC would account for very little time relative to the k-means algorithm. I think what results in the large speedup is the "embarrassing" parallelism that an async Rust server (gRPC in this case) would give. Knowing the CPU time during the tests could confirm this. I'll try run the benchmarks on my machine this week to verify my assumption.
  - bpicolo 7y ago
    I've hit some degenerate cases on gRPC serialization. The golang grpc/proto library accounted for something like 40% of cpu time doing serialization on mapping types for an API I developed under load test. Changing that to a list of key/value pairs dropped that 40% to <1%. The gRPC python sever implementation does a lot of work using python builtins, and doesn't use recent asyncio primitives. It uses built-in python thread-pool execution, which means you also have the GIL to worry about: https://grpc.io/docs/tutorials/basic/python/#starting-the-server https://grpc.io/docs/tutorials/basic/python/#starting-the-se... Given that, I would expect its connection / request management to be pretty darn slow. gRPC has better support for Python as the client. The thread pool execution model also prevents you from doing a number of things you might want to do with gRPC servers (long-lived request streaming is out of the picture). There are a couple third party packages out there trying to make this better (e.g. https://github.com/vmagamedov/grpclib https://github.com/vmagamedov/grpclib) but I hit compatibility issues for a few cases when trying to use them for some prototypes.
- amelius 7y ago
  Of course the actual ML is not going to be faster in Rust, because the main tensor operations as used from Python are already written close to the metal. The cost of these operations is 99% of what it takes to run ML. Optimizing the remaining 1% is just not going to have a noticeable effect.
  - jstrong 7y ago
    I've spent a lot of time writing python ML code, and I think 99% (of time inside fast C/cuda code) is much higher than most programs achieve. Off hand I'd say 80% is average and 90% is good (usually comes after a performance refactor). Also, one aspect that people often overlook is that optimizing compilers can often combine or vectorize operations in ways that are an order of magnitude faster than the same operations performed consecutively in, say, pure numpy. in that scenario, even though each numpy operation is very fast for what it is, the python program that combines a bunch of them ends up running much slower.
deleted 7y ago
[deleted]

by wdroz 7y ago

kbumsik 7y ago

Yeah, by skimming the article, it sound like making gRPC microservices 25x faster, not ML.

nevi-me 7y ago
gRPC would account for very little time relative to the k-means algorithm. I think what results in the large speedup is the "embarrassing" parallelism that an async Rust server (gRPC in this case) would give. Knowing the CPU time during the tests could confirm this. I'll try run the benchmarks on my machine this week to verify my assumption.
- bpicolo 7y ago
  I've hit some degenerate cases on gRPC serialization. The golang grpc/proto library accounted for something like 40% of cpu time doing serialization on mapping types for an API I developed under load test. Changing that to a list of key/value pairs dropped that 40% to <1%. The gRPC python sever implementation does a lot of work using python builtins, and doesn't use recent asyncio primitives. It uses built-in python thread-pool execution, which means you also have the GIL to worry about: https://grpc.io/docs/tutorials/basic/python/#starting-the-server https://grpc.io/docs/tutorials/basic/python/#starting-the-se... Given that, I would expect its connection / request management to be pretty darn slow. gRPC has better support for Python as the client. The thread pool execution model also prevents you from doing a number of things you might want to do with gRPC servers (long-lived request streaming is out of the picture). There are a couple third party packages out there trying to make this better (e.g. https://github.com/vmagamedov/grpclib https://github.com/vmagamedov/grpclib) but I hit compatibility issues for a few cases when trying to use them for some prototypes.
amelius 7y ago
Of course the actual ML is not going to be faster in Rust, because the main tensor operations as used from Python are already written close to the metal. The cost of these operations is 99% of what it takes to run ML. Optimizing the remaining 1% is just not going to have a noticeable effect.
- jstrong 7y ago
  I've spent a lot of time writing python ML code, and I think 99% (of time inside fast C/cuda code) is much higher than most programs achieve. Off hand I'd say 80% is average and 90% is good (usually comes after a performance refactor). Also, one aspect that people often overlook is that optimizing compilers can often combine or vectorize operations in ways that are an order of magnitude faster than the same operations performed consecutively in, say, pure numpy. in that scenario, even though each numpy operation is very fast for what it is, the python program that combines a bunch of them ends up running much slower.

deleted 7y ago

[deleted]