6 ms·

I'm one of the committers. The project is still early stage. At this point we're looking for feedback on the API, which we're planning on finalizing this month.

by pauldix 13y ago

I'm one of the committers. The project is still early stage. At this point we're looking for feedback on the API, which we're planning on finalizing this month. Would love to hear about anything you'd like changed or added to the API.

chubot 13y ago
I wouldn't use this bastardized SQL dialect. SQL comes from relational databases, which comes from relational algebra, which is an exceptionally poor model for time series data. It's going to end up a mess and confuse people. SQL is already a mess by itself. I would simply use functions and operators over time series or data frame types. Perhaps take a look at the R zoo library for examples of more advanced things people do with time series.
- vasilipupkin 13y ago
  Actually, it's nice to have both. you may have something already that is doing a lot of queries and this way you can swap this database in and not have to change the queries much.
- pauldix 13y ago
  We opted for the bastardized SQL because we thought it would be easier to understand. Of course, that may not be the case which is why we'd like to hear what people think of it. We based our decision on people using our API in Errplane and not understanding some parts of it without a lot of additional explanation. One of our users said about one part of it "oh, that's just like group by". I'm curious, did you find the SQL dialect readable and understandable?
  - clark-kent 13y ago
    I personally like the simplicity of the SQL dialect. It's a universal way to query data every programmer is familiar with it. Alternatively using functions and code to query data easily gets very complicated.
  - taf2 13y ago
    I really like SQL - as I was learning SQL, I thought it was terrible, but now after years of using it really makes getting to data easy, I'm very excited to see you choose SQL instead of JSON or something less query like to query...
    - beagle3 13y ago
      ... but that's sort of a stockholm syndrome; it's because you know SQL and it makes you feel comfortable, and you can think of worse options (like JSON). But SQL is really a bad option in the moodern world, especially since (I estimate) about 90% of SQL statements are built programmatically; Thus, a query language that is easier to construct from code makes a lot more sense. (And not, that's not JSON - some form of algebraic notation or LISPish notation makes much more sense). Also, SQL semantics are horrible if you have order involved (as you always do in time series).
      - taf2 13y ago
        I disagree - it's not stockholm syndrome.. for me it was realizing that SQL actually has solved a lot of common and useful ways of expressing the process of getting to the data you want in the order process you want... simply because we write code to generate SQL does not make it bad... I suspect you also think HTML is bad? Being able to say select x,y,z from table where x=1 is very simple and very clear IMO... I'm just giving you my opinion though... you clearly see things differently :D
        beagle3 13y ago
        HTML is indeed bad as a machine generated format -- which is what it is; e.g. <p>, list items and a few other things don't have a close tag, but most things do. These things (like SQL) make sense if you assume that the input is (a) written manually, and (b) by people who are not expected to do this "professionally". Neither is the case of HTML nor SQL anymore. (Seriously, SQL was originally marketed to managers with the idea that "it's just plain english so you can do it yourself, and don't need programmers!". You know how well that worked out)
- cnlwsu 13y ago
  reminds me of epl http://esper.codehaus.org/esper-4.2.0/doc/reference/en/html/epl_clauses.html http://esper.codehaus.org/esper-4.2.0/doc/reference/en/html/...
alexk 13y ago
This project looks really, really promising, thanks for working on it! How robust/scalable in your opinion the backend is at this stage? I'm just trying to set my expectations properly when checking it out. Thanks, Sasha
- pauldix 13y ago
  We're writing the clustered portion of it right now. That won't be available until December, but we'll have performance benchmarks on a variety of configurations. The single node performance at this point for writes is tens of thousands of points per second if batched, and for reads we haven't optimized yet. Queries that only have to go through a few hundred thousand points should return in < 1s. Of course, those numbers will be highly variable depending on if you're writing a series with 1 column or 30 columns. We'll be adding things to make that better. For now we're focused on creating a developer friendly API and building out clustering.
  - alexk 13y ago
    That makes total sense. Thanks for the info!
yummyfajitas 13y ago
I'll second what Chubot said that I'm not liking the SQL much. The interface I'd like would be something close to numpy (or matlab/r, if those are more familiar). Let me do vectorized operations and write functions in code. Let me load a few different timeseries into a dataframe. Most likely the easiest thing to do would just be to either embed numpy into your engine, or create simple wrappers to load data to it.
- pauldix 13y ago
  The numpy style interface might be interesting. The problem is that it would probably kill performance to marshal into a Python process. Unless there's some way to embed numpy? Supporting custom functions is definitely something we want to do.
  - yummyfajitas 13y ago
    Why would it kill performance? All you need to do is this: double* values = (double*)malloc(sizeof(double)*num_of_values); (If you are unfamiliar with numpy, it's just a python wrapper around raw blocks of memory.)
    - waterlion 13y ago
      You still have to get the bytes out of the server and into the Python process.
      - yummyfajitas 13y ago
        Assuming Python is not running in the same process as InfluxDB, amend my code snippet: double* data = shmget(key, sizeof(double)*num_data_pts, whatever_flag); But directly embedding a Python interpreter is pretty easy in C, so I imagine it should also be fairly easy in Go as well. (Of course, given that my code snippets are C, you can probably deduce I've never written any Go, so take my comments on how easy it is with a grain of salt.)
jjonte 13y ago
This looks awesome. Interestingly, I started writing something similar to this myself not too long ago. I was using DynamoDB as the backer though. I think some more aggregate functions would be useful: - Count (not listed on the Functions page, but used in the query examples?) - Sum - Standard Deviation - First - Last

by pauldix 13y ago

chubot 13y ago

I wouldn't use this bastardized SQL dialect. SQL comes from relational databases, which comes from relational algebra, which is an exceptionally poor model for time series data. It's going to end up a mess and confuse people. SQL is already a mess by itself. I would simply use functions and operators over time series or data frame types. Perhaps take a look at the R zoo library for examples of more advanced things people do with time series.

vasilipupkin 13y ago
Actually, it's nice to have both. you may have something already that is doing a lot of queries and this way you can swap this database in and not have to change the queries much.
pauldix 13y ago
We opted for the bastardized SQL because we thought it would be easier to understand. Of course, that may not be the case which is why we'd like to hear what people think of it. We based our decision on people using our API in Errplane and not understanding some parts of it without a lot of additional explanation. One of our users said about one part of it "oh, that's just like group by". I'm curious, did you find the SQL dialect readable and understandable?
- clark-kent 13y ago
  I personally like the simplicity of the SQL dialect. It's a universal way to query data every programmer is familiar with it. Alternatively using functions and code to query data easily gets very complicated.
- taf2 13y ago
  I really like SQL - as I was learning SQL, I thought it was terrible, but now after years of using it really makes getting to data easy, I'm very excited to see you choose SQL instead of JSON or something less query like to query...
  - beagle3 13y ago
    ... but that's sort of a stockholm syndrome; it's because you know SQL and it makes you feel comfortable, and you can think of worse options (like JSON). But SQL is really a bad option in the moodern world, especially since (I estimate) about 90% of SQL statements are built programmatically; Thus, a query language that is easier to construct from code makes a lot more sense. (And not, that's not JSON - some form of algebraic notation or LISPish notation makes much more sense). Also, SQL semantics are horrible if you have order involved (as you always do in time series).
    - taf2 13y ago
      I disagree - it's not stockholm syndrome.. for me it was realizing that SQL actually has solved a lot of common and useful ways of expressing the process of getting to the data you want in the order process you want... simply because we write code to generate SQL does not make it bad... I suspect you also think HTML is bad? Being able to say select x,y,z from table where x=1 is very simple and very clear IMO... I'm just giving you my opinion though... you clearly see things differently :D
      - beagle3 13y ago
        HTML is indeed bad as a machine generated format -- which is what it is; e.g. <p>, list items and a few other things don't have a close tag, but most things do. These things (like SQL) make sense if you assume that the input is (a) written manually, and (b) by people who are not expected to do this "professionally". Neither is the case of HTML nor SQL anymore. (Seriously, SQL was originally marketed to managers with the idea that "it's just plain english so you can do it yourself, and don't need programmers!". You know how well that worked out)
cnlwsu 13y ago
reminds me of epl http://esper.codehaus.org/esper-4.2.0/doc/reference/en/html/epl_clauses.html http://esper.codehaus.org/esper-4.2.0/doc/reference/en/html/...

alexk 13y ago

This project looks really, really promising, thanks for working on it! How robust/scalable in your opinion the backend is at this stage? I'm just trying to set my expectations properly when checking it out. Thanks, Sasha

pauldix 13y ago
We're writing the clustered portion of it right now. That won't be available until December, but we'll have performance benchmarks on a variety of configurations. The single node performance at this point for writes is tens of thousands of points per second if batched, and for reads we haven't optimized yet. Queries that only have to go through a few hundred thousand points should return in < 1s. Of course, those numbers will be highly variable depending on if you're writing a series with 1 column or 30 columns. We'll be adding things to make that better. For now we're focused on creating a developer friendly API and building out clustering.
- alexk 13y ago
  That makes total sense. Thanks for the info!

yummyfajitas 13y ago

I'll second what Chubot said that I'm not liking the SQL much. The interface I'd like would be something close to numpy (or matlab/r, if those are more familiar). Let me do vectorized operations and write functions in code. Let me load a few different timeseries into a dataframe. Most likely the easiest thing to do would just be to either embed numpy into your engine, or create simple wrappers to load data to it.

pauldix 13y ago
The numpy style interface might be interesting. The problem is that it would probably kill performance to marshal into a Python process. Unless there's some way to embed numpy? Supporting custom functions is definitely something we want to do.
- yummyfajitas 13y ago
  Why would it kill performance? All you need to do is this: double* values = (double*)malloc(sizeof(double)*num_of_values); (If you are unfamiliar with numpy, it's just a python wrapper around raw blocks of memory.)
  - waterlion 13y ago
    You still have to get the bytes out of the server and into the Python process.
    - yummyfajitas 13y ago
      Assuming Python is not running in the same process as InfluxDB, amend my code snippet: double* data = shmget(key, sizeof(double)*num_data_pts, whatever_flag); But directly embedding a Python interpreter is pretty easy in C, so I imagine it should also be fairly easy in Go as well. (Of course, given that my code snippets are C, you can probably deduce I've never written any Go, so take my comments on how easy it is with a grain of salt.)

jjonte 13y ago

This looks awesome. Interestingly, I started writing something similar to this myself not too long ago. I was using DynamoDB as the backer though. I think some more aggregate functions would be useful: - Count (not listed on the Functions page, but used in the query examples?) - Sum - Standard Deviation - First - Last