3 ms·

For me, an interesting statement was "However, it took 6 days to fetch all data for all 17 million products." - in my experience of DB systems, 17 million entri

For me, an interesting statement was "However, it took 6 days to fetch all data for all 17 million products." - in my experience of DB systems, 17 million entries is significant but not particularly much, it's something that fits in the RAM of a laptop and can be imported/exported/indexed/processed in minutes (if you do batch processing, not a separate transaction per entry), perhaps hours if the architecture is lousy but certainly not days.

gtirloni 4y ago
> Any kind of operation was done through an HTTP call, which you would otherwise do with a SQL script, in ElasticSearch, you would do an HTTP request There you go.
- foomatic 4y ago
  What?
- teaearlgraycold 4y ago
  It's recommended you send ES thousands of records at a time.
fabian2k 4y ago
I think this is a very clear disadvantage of the microservice architecture they chose in this case, and the post does allude to that. To recreate this data they needed to query several different microservices that would not have been able to sustain a higher load. If I calculated this right the time they mention comes down to 30 items per second. Which is maybe not unreasonable for something that queries a whole bunch of services via HTTP, but is kinda ridiculous if you compare it to directly querying a single RDBMS. You could probably fix this by scaling everything horizontally, if that is possible. But the real solution would be as you say to have bulk processing capabilities.
- PeterisP 4y ago
  Yes, adding a "return X items" mode to the same microservices often is a way to get a significant performance boost with only minor changes, where even if your main use case needs only one item, it enables mass processing without incurring the immense overhead of a separate request per each item.
  - magicalhippo 4y ago
    This goes beyond microservices. I've done a fair share of optimization in our own code to introduce batch-oriented calls rather than singular. Even though your service is on the same LAN as the DB server, fetching thousands of rows one select at a time is very slow compared to a single, or a few, selects. In one case, the application was running on a laptop over WiFi, which increased the network latency by 10x. Suddenly a 30 sec job turned into a 5 minute job. Since one can easily implement a singular version using a batch size of 1, it's a drop-in replacement in most cases. Also, since one can easily implement a batch-style API using a singular version, you can write the API batch-oriented but implement it using the singular version if that's easier. This allows you to easily swap out the implementation if needed at a later date.
    - ikiris 4y ago
      If this is the case, you've done something horrible in your query program. threads exist, use them. If you're waiting on 500 full sequential RTTs that's your fault. network requests on local fabric can be faster than storage.
      - magicalhippo 4y ago
        Sure, threads exist, but they also make things more complicated. Simple batching can make a single thread go a lot further, was my point.
    - robertlagrant 4y ago
      Totally agree. If a frontend wants exactly one of something then implement an API that calls the same batch code, but exposes it as a singular. The thing I've found for object retrieval (as opposed to search) is that you might want to break GET semantics and have people POST in a list of IDs. Otherwise you might hit the query string size limit. Random tip.
thayne 4y ago
That kind of depends on how big each record is. And it sounds like these records are denormalized from multiple sources, so you probably have several transactions for each record. It's possible to do batching in that situation, but it definitely isn't always easy.
deleted 4y ago
[deleted]
fijiaarone 4y ago
Read his post about over-engineering a micro services architecture. A great read, and helps explain the context of this problem. https://medium.com/@hugo.oliveira.rocha/handling-eventual-consistency-11324324aec4 https://medium.com/@hugo.oliveira.rocha/handling-eventual-co...

gtirloni 4y ago

> Any kind of operation was done through an HTTP call, which you would otherwise do with a SQL script, in ElasticSearch, you would do an HTTP request There you go.

foomatic 4y ago
What?
teaearlgraycold 4y ago
It's recommended you send ES thousands of records at a time.

fabian2k 4y ago

I think this is a very clear disadvantage of the microservice architecture they chose in this case, and the post does allude to that. To recreate this data they needed to query several different microservices that would not have been able to sustain a higher load. If I calculated this right the time they mention comes down to 30 items per second. Which is maybe not unreasonable for something that queries a whole bunch of services via HTTP, but is kinda ridiculous if you compare it to directly querying a single RDBMS. You could probably fix this by scaling everything horizontally, if that is possible. But the real solution would be as you say to have bulk processing capabilities.

PeterisP 4y ago
Yes, adding a "return X items" mode to the same microservices often is a way to get a significant performance boost with only minor changes, where even if your main use case needs only one item, it enables mass processing without incurring the immense overhead of a separate request per each item.
- magicalhippo 4y ago
  This goes beyond microservices. I've done a fair share of optimization in our own code to introduce batch-oriented calls rather than singular. Even though your service is on the same LAN as the DB server, fetching thousands of rows one select at a time is very slow compared to a single, or a few, selects. In one case, the application was running on a laptop over WiFi, which increased the network latency by 10x. Suddenly a 30 sec job turned into a 5 minute job. Since one can easily implement a singular version using a batch size of 1, it's a drop-in replacement in most cases. Also, since one can easily implement a batch-style API using a singular version, you can write the API batch-oriented but implement it using the singular version if that's easier. This allows you to easily swap out the implementation if needed at a later date.
  - ikiris 4y ago
    If this is the case, you've done something horrible in your query program. threads exist, use them. If you're waiting on 500 full sequential RTTs that's your fault. network requests on local fabric can be faster than storage.
    - magicalhippo 4y ago
      Sure, threads exist, but they also make things more complicated. Simple batching can make a single thread go a lot further, was my point.
  - robertlagrant 4y ago
    Totally agree. If a frontend wants exactly one of something then implement an API that calls the same batch code, but exposes it as a singular. The thing I've found for object retrieval (as opposed to search) is that you might want to break GET semantics and have people POST in a list of IDs. Otherwise you might hit the query string size limit. Random tip.

thayne 4y ago

That kind of depends on how big each record is. And it sounds like these records are denormalized from multiple sources, so you probably have several transactions for each record. It's possible to do batching in that situation, but it definitely isn't always easy.

deleted 4y ago

[deleted]

fijiaarone 4y ago

Read his post about over-engineering a micro services architecture. A great read, and helps explain the context of this problem. https://medium.com/@hugo.oliveira.rocha/handling-eventual-consistency-11324324aec4 https://medium.com/@hugo.oliveira.rocha/handling-eventual-co...