3 ms·

Is it really a good idea to build something like this (big data, needs to crunch data fast) in Python (slow)?

quickthrower2 3y ago
This is a toy project to understand the space. He is using solr at work.
BD103 3y ago
I would argue that Python does not inhibit this project as much as you imply. There are multiple benefits to Python, such as: - A built-in dictionary type, used for indexing words - Clean and easy to read code, which is one of Python's core strengths - It's fast to draft code in, perfect for toy programs - Easy async support, which the author comments on - Plenty of libraries to do the heavy lifting of tasks not focused on by the post, such as hosting a web server, rendering template HTML, and parsing CLI arguments Yes, Python is not fast relative to C or Rust, but it's perfect for this type of project.
eichin 3y ago
Yes. Paragraph 3: > This implementation doesn’t pretend to be a production-ready search engine, just a usable toy example showing how a search engine works under the hood. The whole point is expressiveness.
vidarh 3y ago
I'm not a fan of Python, but the bigger "issue" here if this was meant to be production code (it's clearly not) is not the choice of Python, but algorithm choices. It shows the principles, but for a production search engine there are a whole lot of tricks you'd apply whether you stick with Python or not. And once you do, odds are you'd find most of the runtime spent in very small portions of code doing fairly basic composable operations on large compressed arrays, and you can get very far with just rewriting a tiny core in something faster. The number of people who need a scale where this is hard to do fast enough is small...
softwaredoug 3y ago
"Crunch data fast" is something numpy is really good at :) You can see my comment about SearchArray above, but you can do a lot of native performance comparable things if you embrace array based programming

This is a toy project to understand the space. He is using solr at work.

I would argue that Python does not inhibit this project as much as you imply. There are multiple benefits to Python, such as: - A built-in dictionary type, used for indexing words - Clean and easy to read code, which is one of Python's core strengths - It's fast to draft code in, perfect for toy programs - Easy async support, which the author comments on - Plenty of libraries to do the heavy lifting of tasks not focused on by the post, such as hosting a web server, rendering template HTML, and parsing CLI arguments Yes, Python is not fast relative to C or Rust, but it's perfect for this type of project.

eichin 3y ago

Yes. Paragraph 3: > This implementation doesn’t pretend to be a production-ready search engine, just a usable toy example showing how a search engine works under the hood. The whole point is expressiveness.

vidarh 3y ago

I'm not a fan of Python, but the bigger "issue" here if this was meant to be production code (it's clearly not) is not the choice of Python, but algorithm choices. It shows the principles, but for a production search engine there are a whole lot of tricks you'd apply whether you stick with Python or not. And once you do, odds are you'd find most of the runtime spent in very small portions of code doing fairly basic composable operations on large compressed arrays, and you can get very far with just rewriting a tiny core in something faster. The number of people who need a scale where this is hard to do fast enough is small...

softwaredoug 3y ago

"Crunch data fast" is something numpy is really good at :) You can see my comment about SearchArray above, but you can do a lot of native performance comparable things if you embrace array based programming