4 ms·
This is fairly accurate in my experience: https://twitter.com/thesmartjokes/status/684286479401652224 https://twitter.com/thesmartjokes/status/68428647940165222
by em500 8y ago
This is fairly accurate in my experience: https://twitter.com/thesmartjokes/status/684286479401652224 https://twitter.com/thesmartjokes/status/684286479401652224
Day to day, it's mostly SQL, or worse Hive queries which makes most things much slower than they should be.
- minimaxir 8y agoOne thing rarely discussed with the rise of big data is how to do efficient querying, especially at scale. I've had a ton of data science interviews which ask how to reimplement binary search from scratch (which I would never do on the job), but not anything about how to do efficient JOINs and query nesting.
- ekianjo 8y agoexactly. Optimising queries becomes really a critical part of the job when you make complex JOINS on millions of records. Just getting the data can take a huge amount of time before you can even consider models.
- deleted 8y ago[deleted]
- bioquestion 8y agoI work as a biostatistician and I've been tasked recently with querying large databases using SQL, in addition to analysing the data. However, my programming background is very limited and thus I'm sure my queries are very inefficient. Could you point me to some materials/texts about how to improve querying efficiency for SQL? If it's oriented for beginners then that would be ideal. Thanks in advance.
- collyw 8y agoThis site is great. https://use-the-index-luke.com/ https://use-the-index-luke.com/
- bioquestion 8y agoThank you! I've already started reading it and it seems to be just what I needed.
- tixocloud 8y agoI’ve been told by IT from numerous organisations that Hadoop will solve all of our team’s query inefficiencies. Also hence why we introduce new members of the team to learn how to do efficient queries and joins. And spend time upfront to structure their problems.
- tspike 8y agoDumb question: is Splunk implemented with Hive?
- Godel_unicode 8y agoNo. Splunk is it's own proprietary thing. They actually work somewhat similarly (in that they're both query engines over distributed data using a map-reduce paradigm) although a closer parallel is lucene/elasticsearch.
- eyeball 8y agoI recently found out that my company’s information security and IT ops team have licensed Splunk. They use it to analyze machine logs, firewall logs, etc. My team focuses on traditional reporting (data in oracle, manipulated with SAS, sometimes statistical modeling with SAS, resulting in reports/dashboards presented with QlikView). The IT team approached me to ask if we wanted to explore any use cases of Splunk for my team. I’ve looked around a bit and don’t understand this stuff enough to know if there are any. We generally are working with health insurance claims data, and have no problems being able to process the data we need using oracle/sas. Any ideas spring to mind that might make me want to invest time in looking at this?