3 ms·

Any resource to learn SQL will do, there are minor syntax differences between databases but they are not important. And there is not difference to learn SQL for

by Kronen 7y ago

Any resource to learn SQL will do, there are minor syntax differences between databases but they are not important. And there is not difference to learn SQL for big data, machine learning or any other purpose

p1esk 7y ago
Is there a good guide on which database to choose for an ML project? Like ease of use, or speed, or maybe some particular features?
- tempguy9999 7y ago
  As no-one's answered this, I'll try though my area is DBs but certainly not ML. AFAIK ML doesn't really care about the source of data so DB is irrelevant. The conjunction of DB and ML in your question makes me wonder if you're sort of confusing the two. AFAIK they are totally independent, though some DBs come with some data mining tools, which is ML eg. MSSQL (you can prob download an eval version for nothing. It's time limited though, 6 months or so). But to repeat AFAIK ML and DBs are entirely different things. If you want to grab a DB, well I've no experience with postgres but I've heard a lot that's good, so perhaps start here.
  - p1esk 7y ago
    So far I’ve been using Excel and Pandas to maintain/pipeline my data, but I feel like a database might be a better way. I’m not sure about built in data mining tools, as I’m used to writing them myself. Also, most of my pipeline configs are in json format so I’ve been thinking about MongoDB.
    - tempguy9999 7y ago
      Sounds like you know ML then. Anything non-trivial on a spreadsheet is usually best ported to a DB, that's a pretty good rule of thumb. I don't understand about your configs being in json, nor why that makes it more suitable for mongo. If you need to store semistructured blobs of data, relational DBs can do that fine (in BLOB fields), but IIRC postgres supports json, including indexing into individual json subfields. <http://www.postgresqltutorial.com/postgresql-json/> http://www.postgresqltutorial.com/postgresql-json/> but to repeat, I've never used postgres so speak to someone else first who has. If you can do ML as it seems you are, install postgres and just get started on this tutorial stuff here - best of luck!
- tracker1 7y ago
  I would first look at the tools/libraries you are using and what they may support. PostgreSQL, mySQL/mariaDB are both well supported RDBMS in some circles and work well with Python... if you are doing map/reduce at its' core, you may do better with Hadoop or Cassandra. MongoDB may be a disconnect depending on your tooling. PostgreSQL does have a lot of options, and this includes indexing on binary serialized JSON data, that can help a lot depending on what you want to do. PLV8 is also incredibly interesting. As an aside, I tend to find that if I can do something in a Docker container (locally), I will do it in a container. This lets you spin everything up, and down and cleanup and retry without leaving remnants of stuff behind on your host environment (desktop). The only gotcha is don't use volume mounts for your db data if you're using docker desktop for windows/mac, it does not perform well.

by Kronen 7y ago

p1esk 7y ago

Is there a good guide on which database to choose for an ML project? Like ease of use, or speed, or maybe some particular features?

tempguy9999 7y ago
As no-one's answered this, I'll try though my area is DBs but certainly not ML. AFAIK ML doesn't really care about the source of data so DB is irrelevant. The conjunction of DB and ML in your question makes me wonder if you're sort of confusing the two. AFAIK they are totally independent, though some DBs come with some data mining tools, which is ML eg. MSSQL (you can prob download an eval version for nothing. It's time limited though, 6 months or so). But to repeat AFAIK ML and DBs are entirely different things. If you want to grab a DB, well I've no experience with postgres but I've heard a lot that's good, so perhaps start here.
- p1esk 7y ago
  So far I’ve been using Excel and Pandas to maintain/pipeline my data, but I feel like a database might be a better way. I’m not sure about built in data mining tools, as I’m used to writing them myself. Also, most of my pipeline configs are in json format so I’ve been thinking about MongoDB.
  - tempguy9999 7y ago
    Sounds like you know ML then. Anything non-trivial on a spreadsheet is usually best ported to a DB, that's a pretty good rule of thumb. I don't understand about your configs being in json, nor why that makes it more suitable for mongo. If you need to store semistructured blobs of data, relational DBs can do that fine (in BLOB fields), but IIRC postgres supports json, including indexing into individual json subfields. <http://www.postgresqltutorial.com/postgresql-json/> http://www.postgresqltutorial.com/postgresql-json/> but to repeat, I've never used postgres so speak to someone else first who has. If you can do ML as it seems you are, install postgres and just get started on this tutorial stuff here - best of luck!
tracker1 7y ago
I would first look at the tools/libraries you are using and what they may support. PostgreSQL, mySQL/mariaDB are both well supported RDBMS in some circles and work well with Python... if you are doing map/reduce at its' core, you may do better with Hadoop or Cassandra. MongoDB may be a disconnect depending on your tooling. PostgreSQL does have a lot of options, and this includes indexing on binary serialized JSON data, that can help a lot depending on what you want to do. PLV8 is also incredibly interesting. As an aside, I tend to find that if I can do something in a Docker container (locally), I will do it in a container. This lets you spin everything up, and down and cleanup and retry without leaving remnants of stuff behind on your host environment (desktop). The only gotcha is don't use volume mounts for your db data if you're using docker desktop for windows/mac, it does not perform well.