4 ms·

Out if necessity I've started working with Microsoft's Kusto Query Language [1] which is used by various services in Azure (e g. their Log Analytics Workspace).

Out if necessity I've started working with Microsoft's Kusto Query Language [1] which is used by various services in Azure (e g. their Log Analytics Workspace). At first I found the language rather akward and was wondering why yet another query language. But the more I used it the more it grew on me. The thing I really like is that unlike the clauses in SQL, the order of operators isn't really fixed and it reads and feels like a pipe command in a Unix shell. One example where I find this far superior is when doing aggregations. In SQL I would have to modify both the start and the end of the query, which is quite a nuisance. [1] https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/ https://docs.microsoft.com/en-us/azure/data-explorer/kusto/q...)

antruok 4y ago
The query structure does look nice indeed! The usage of the pipe character feels odd but I suppose there are benefits in the end
- knutwannheden 4y ago
  I had the same initial reaction regarding the pipe character. But once I started thinking of the query as a pipe (like in the terminal) through which the data flows, where stuff like ORDER BY, SELECT, and GROUP BY are just operators, it started making sense.
oldmanhorton 4y ago
Kusto is used extensively within Microsoft and has been for a long time. I think it's generally really well liked and really productive, and while it tends to be quite quick, it has some similar performance pitfalls as SQL
bradford 4y ago
(disclaimer: Microsoft Employee, this is my opinion). I've been using KQL for a long time, it really is a nice language both for querying and maintaining the data. But, aside from fixing some serious language issues with SQL, I really enjoy the wide range of supported scenarios. You can use KQL to query a SQL database [1], you can use python [2], do all kinds of time-series analysis [3][4], do distinct counts on various fields without too much explicit query-authoring [5]. My main beef with Azure Data Explorer (which, as I understand it, is the engine that handles the query execution) is the price... I wish it was easy for hobbyist developers to launch and try out. [1] https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/sqlrequestplugin?pivots=azuredataexplorer https://docs.microsoft.com/en-us/azure/data-explorer/kusto/q... [2] https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/pythonplugin?pivots=azuredataexplorer https://docs.microsoft.com/en-us/azure/data-explorer/kusto/q... [3] https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/series-firfunction https://docs.microsoft.com/en-us/azure/data-explorer/kusto/q... [4] https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/series-fft-function https://docs.microsoft.com/en-us/azure/data-explorer/kusto/q... [5] https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/dcount-intersect-plugin https://docs.microsoft.com/en-us/azure/data-explorer/kusto/q...
- knutwannheden 4y ago
  There is this page about a free cluster [1]. It appears to be pretty beefy, but it seems like it is for one year only, but IMHO that isn't entirely clear. [1] https://docs.microsoft.com/en-us/azure/data-explorer/start-for-free https://docs.microsoft.com/en-us/azure/data-explorer/start-f...
  - MaKey 4y ago
    Today Microsoft announced that an emulator for Azure Data Explorer is generally available [1]. It is a Windows docker container. [1] https://aka.ms/adx.emulator.doc https://aka.ms/adx.emulator.doc
kccqzy 4y ago
Speaking of custom query languages by Big Tech companies, I've always been curious about Google's Yedalog[0]. I mean it's obviously inspired by a rich history of logic programming here, as it's trying to find a middle ground between Prolog and Datalog. I'm not sure what's come of it, and I certainly hope they eventually make it open source or at least available for BigQuery for something. [0]: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45983.pdf https://static.googleusercontent.com/media/research.google.c...

antruok 4y ago

The query structure does look nice indeed! The usage of the pipe character feels odd but I suppose there are benefits in the end

knutwannheden 4y ago
I had the same initial reaction regarding the pipe character. But once I started thinking of the query as a pipe (like in the terminal) through which the data flows, where stuff like ORDER BY, SELECT, and GROUP BY are just operators, it started making sense.

oldmanhorton 4y ago

Kusto is used extensively within Microsoft and has been for a long time. I think it's generally really well liked and really productive, and while it tends to be quite quick, it has some similar performance pitfalls as SQL

bradford 4y ago

(disclaimer: Microsoft Employee, this is my opinion). I've been using KQL for a long time, it really is a nice language both for querying and maintaining the data. But, aside from fixing some serious language issues with SQL, I really enjoy the wide range of supported scenarios. You can use KQL to query a SQL database [1], you can use python [2], do all kinds of time-series analysis [3][4], do distinct counts on various fields without too much explicit query-authoring [5]. My main beef with Azure Data Explorer (which, as I understand it, is the engine that handles the query execution) is the price... I wish it was easy for hobbyist developers to launch and try out. [1] https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/sqlrequestplugin?pivots=azuredataexplorer https://docs.microsoft.com/en-us/azure/data-explorer/kusto/q... [2] https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/pythonplugin?pivots=azuredataexplorer https://docs.microsoft.com/en-us/azure/data-explorer/kusto/q... [3] https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/series-firfunction https://docs.microsoft.com/en-us/azure/data-explorer/kusto/q... [4] https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/series-fft-function https://docs.microsoft.com/en-us/azure/data-explorer/kusto/q... [5] https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/dcount-intersect-plugin https://docs.microsoft.com/en-us/azure/data-explorer/kusto/q...

knutwannheden 4y ago
There is this page about a free cluster [1]. It appears to be pretty beefy, but it seems like it is for one year only, but IMHO that isn't entirely clear. [1] https://docs.microsoft.com/en-us/azure/data-explorer/start-for-free https://docs.microsoft.com/en-us/azure/data-explorer/start-f...
- MaKey 4y ago
  Today Microsoft announced that an emulator for Azure Data Explorer is generally available [1]. It is a Windows docker container. [1] https://aka.ms/adx.emulator.doc https://aka.ms/adx.emulator.doc

kccqzy 4y ago

Speaking of custom query languages by Big Tech companies, I've always been curious about Google's Yedalog[0]. I mean it's obviously inspired by a rich history of logic programming here, as it's trying to find a middle ground between Prolog and Datalog. I'm not sure what's come of it, and I certainly hope they eventually make it open source or at least available for BigQuery for something. [0]: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45983.pdf https://static.googleusercontent.com/media/research.google.c...