10 ms·

Some of my coworkers have been dealing with major Ceph issues for the past month. As part of the work, there've been a number of Ceph information dumps, RBD im

Some of my coworkers have been dealing with major Ceph issues for the past month. As part of the work, there've been a number of Ceph information dumps, RBD image information, PG information, and lists of incomplete PDs, all in text or CSV files. To help out my coworkers, I wrote up Python code to parse all of those files into a SQLite database. Why? Well, because I can ship the database to anywhere, as one file, and it can be accessed through the SQLite CLI. In another case, a client was building a list of files to consider for deletion. The list included path, size, type (file or directory), and owning user. Before doing anything else, I imported the entire thing into SQLite using the `.import` command. Which was good, because it was able to identify cases of files that included a tab in their paths!

pjmlp 6y ago
I do exactly the same with traditional RDMS. Export to single file isn't nothing new.
- zdkl 6y ago
  Every time I see a message from you it's a variant of "back in my day", "this isn't New" or some other mildly snooty point about kids these days. At this point I'm curious what even brings you to consistently comment like this. I apologise for the ad hominem but frankly the pattern is frequent and irritating? Could you at least describe your use case or tools please? Share the nostalgia or something, anything.
  - laumars 6y ago
    His posts are ridiculous. I’ve never once seen him say anything positive. It’s always condescending, presumptuous and elitist bullcrap. How he’s ended up with so much karma when his comments are consistently scraping the barrel is beyond me.
    - dvfjsdhgfv 6y ago
      > His posts are ridiculous. I went through some and it doesn't seem so. Yes, he is opinionated, but so are many HN-ers. His set of opinions is quite specific though (based on the comments I see, MS SQL Server users are a minority here, for example) so you might have that impression. > How he’s ended up with so much karma We're supposed to be adults but we still fall for this behavior engineering trick called karma. Really, who cares? What does it change in my life if I have a few points more or less? One could even argue that you can create more value for the community by lowering your karma, making comments that are alien to the collective mindhive. A downvote is almost impulsive in such cases, but your comment (before it greys out) can spark a thought, an interest, an idea, or even some important doubt that would otherwise not arise.
      - laumars 6y ago
        I’m a big advocate that incorrect comments shouldn’t be downvoted, just corrected. Incorrect comments are valuable if they are corrected with informative comments. But there is a difference between incorrect comments and comments that are broad generalisations presented in an obnoxious way. The former sparks informative conversations whereas the latter adds little informative value while pissing people off. I suspect the real value of the latter is just likeminded people patting themselves on the back for being “awesome”. Which I don’t particularly rate as a positive effect for discourse let alone the wider HN community
        Annatar 6y ago
        That is ironic because this entire web site consists of what appears to be like-minded people constantly patting each other on the back because they're so awesome; meanwhile they leave a lot to be desired, and the person you're criticizing so staunchly isn't necessarily one those persons... This website has turned into a hangout for "brogrammers", that's why I almost never participate here any more. Now I'll end up writing even less, I think.
        laumars 6y ago
  - pjmlp 6y ago
    SQL Server database export with SQL Server Management Studio, good enough for you, or want an ELi5 tutorial?
    - yread 6y ago
      Have you tried using different versions of SQL Server from Python on Linux and Windows? There are a few libraries but none of them support everything (instance names, unicode, ...). Compared to using sqlite it's a massive hassle.
      - pjmlp 6y ago
        Well, that wasn't part of the original requirements. I hardly use any Python.
  - Annatar 6y ago
    I'll take that! Now, before yinz flip out, you asked a question, and I will give you my honest answer. The way I see it. No fluff. This was supposed to be the site where the creme de la creme, the best of the best, Silicon Valley stalwarts discuss the cutting edge of computer science. Instead, n-gate.com correctly identified that it's a web site where "hacker news" with no formal education in computer science marvel like a turkey marvels its own excrement when they discover some tried and true technology. Oh, and they keep marveling at wheel reinventions. And you know what? One could say that it gets tiresome after a while, but it's far worse: it's deeply demoralizing, and makes those of us passionate about computer science deeply resent the information technology industry. And then we try to tell you how things should work, and instead of asking questions, yinz just start arguing and pushing reinvented, completely unnecessary wheels like "Kubernetes", "Docker", "Ansible", "Linux" because it's easier to mess around than warm up the chair and learn or listen to experience. The people on this website made me hate computers, because if this is the best our industry has to offer, it's devastating. So I understand where pjmlp is coming from. It's why I've all but withdrawn from here and don't bother posting anything any more.
    - laumars 6y ago
      SQLite has been around 20 years and designed to solve the kinds of problems being discussed. It’s older than git and nearly as old as SSH! So I don’t think it’s fair to brush this conversation with the same brush as those who advocate using Kubernetes for running a couple daemons.
      - Annatar 6y ago
        This was in no way meant to be a critique on SQLite; SQLite is a phenomenal, high quality software and it is extremely useful. Why, I often use it myself and it's my first go-to database for solving all kinds of problems, or even just for simulations. I've known about and used SQLite for almost two decades now. Fantastic software, and that is rare in this day and age when most software is utter garbage. It would be so wonderful if more software were of as high a quality and as simple as SQLite... What I was answering to is the completely unwarranted criticism of pjmlp from various posters here. Not only do I find it unjust, I find it appalling. The gall...
  - badsectoracula 6y ago
    pjmlp longs for the days when developers (and/or their employers) paid for their tools because he believes that this allowed them to create tools of higher overall quality than the free FLOSS stuff you find nowadays that may excel in one area which developers find interesting/fun but are awful in several others and largely proliferate not because of their quality but of their price. His comments basically follow out of this starting point so if you have that in mind they start to make sense. (and TBH i do not completely disagree with this view myself, though i do not see the point of repeating it everywhere, this train not only left the station long time ago but the station has long been converted to an Amazon warehouse - besides it isn't like the development tools of old were always that great... to be relatively on topic, there were a ton of DBMSs that simply failed and others that started nice but created stinkers so awful that killed the entire company - see dBase as a great example for that)
- CaliforniaKarl 6y ago
  That's true, I could've done exactly the same thing with MySQL, MariaDB, Postgres, etc.. But take note of the situations, and you'll notice some things which make SQLite stand out. I said I was working in Python. I was working across three environments: Two separate compute environments, and my work macOS laptop. I didn't want to have to deal with installing the appropriate DB driver, and sqlite is part of the Python standard library. Another thing to note is that I didn't already have a database server spun up, and I didn't think it was worth doing the work of spinning up a Postgres database server. Finally, I didn't intend the "single file" to be an export. I intended it to be an actual, usable database, which my co-workers can use with minimal friction. Those are a few of the reasons why I made the choice I did. And I think SQLite worked quite well!
- skissane 6y ago
  > I do exactly the same with traditional RDMS. For the use case of "I need a quick database for some ad hoc analysis", SQLite has a lot less friction than a traditional RDBMS. Creating a new DB, backing it up, sharing it with others – all these tasks are doable on a traditional RDBMS too, but generally involve additional steps or prerequisites compared to what SQLite requires. > Export to single file isn't nothing new. Most databases have some kind of export/import function based around CSV, SQL, XML, JSON, or even some proprietary binary format. But SQLite is a lot easier than that – I can just copy a single database file to another system and just open it and it works straight away. (The only proviso is make sure the DB isn't open at the time of copy, to avoid any corruption issues.) No need to run some time consuming import/export process which converts between the external format and the database's native storage format.
  - pjmlp 6y ago
    Same here, depending on the project, I always have either SQL Developer with Oracle, or SQL Server Management Studio with SQL Server installed.
    - skissane 6y ago
      Copying a database to another machine is a lot simpler with sqlite than with either of the tools you mention. So I don't agree with the "same here" part.
emmelaich 6y ago
Perhaps you should add your tables to Osquery? https://osquery.io/ https://osquery.io/
- chrisweekly 6y ago
  Thanks for the tip! Bookmarked, will look closer at OSQuery after vaca.

pjmlp 6y ago

I do exactly the same with traditional RDMS. Export to single file isn't nothing new.

zdkl 6y ago
Every time I see a message from you it's a variant of "back in my day", "this isn't New" or some other mildly snooty point about kids these days. At this point I'm curious what even brings you to consistently comment like this. I apologise for the ad hominem but frankly the pattern is frequent and irritating? Could you at least describe your use case or tools please? Share the nostalgia or something, anything.
- laumars 6y ago
  His posts are ridiculous. I’ve never once seen him say anything positive. It’s always condescending, presumptuous and elitist bullcrap. How he’s ended up with so much karma when his comments are consistently scraping the barrel is beyond me.
  - dvfjsdhgfv 6y ago
    > His posts are ridiculous. I went through some and it doesn't seem so. Yes, he is opinionated, but so are many HN-ers. His set of opinions is quite specific though (based on the comments I see, MS SQL Server users are a minority here, for example) so you might have that impression. > How he’s ended up with so much karma We're supposed to be adults but we still fall for this behavior engineering trick called karma. Really, who cares? What does it change in my life if I have a few points more or less? One could even argue that you can create more value for the community by lowering your karma, making comments that are alien to the collective mindhive. A downvote is almost impulsive in such cases, but your comment (before it greys out) can spark a thought, an interest, an idea, or even some important doubt that would otherwise not arise.
    - laumars 6y ago
      I’m a big advocate that incorrect comments shouldn’t be downvoted, just corrected. Incorrect comments are valuable if they are corrected with informative comments. But there is a difference between incorrect comments and comments that are broad generalisations presented in an obnoxious way. The former sparks informative conversations whereas the latter adds little informative value while pissing people off. I suspect the real value of the latter is just likeminded people patting themselves on the back for being “awesome”. Which I don’t particularly rate as a positive effect for discourse let alone the wider HN community
      - Annatar 6y ago
        That is ironic because this entire web site consists of what appears to be like-minded people constantly patting each other on the back because they're so awesome; meanwhile they leave a lot to be desired, and the person you're criticizing so staunchly isn't necessarily one those persons... This website has turned into a hangout for "brogrammers", that's why I almost never participate here any more. Now I'll end up writing even less, I think.
        laumars 6y ago
- pjmlp 6y ago
  SQL Server database export with SQL Server Management Studio, good enough for you, or want an ELi5 tutorial?
  - yread 6y ago
    Have you tried using different versions of SQL Server from Python on Linux and Windows? There are a few libraries but none of them support everything (instance names, unicode, ...). Compared to using sqlite it's a massive hassle.
    - pjmlp 6y ago
      Well, that wasn't part of the original requirements. I hardly use any Python.
- Annatar 6y ago
  I'll take that! Now, before yinz flip out, you asked a question, and I will give you my honest answer. The way I see it. No fluff. This was supposed to be the site where the creme de la creme, the best of the best, Silicon Valley stalwarts discuss the cutting edge of computer science. Instead, n-gate.com correctly identified that it's a web site where "hacker news" with no formal education in computer science marvel like a turkey marvels its own excrement when they discover some tried and true technology. Oh, and they keep marveling at wheel reinventions. And you know what? One could say that it gets tiresome after a while, but it's far worse: it's deeply demoralizing, and makes those of us passionate about computer science deeply resent the information technology industry. And then we try to tell you how things should work, and instead of asking questions, yinz just start arguing and pushing reinvented, completely unnecessary wheels like "Kubernetes", "Docker", "Ansible", "Linux" because it's easier to mess around than warm up the chair and learn or listen to experience. The people on this website made me hate computers, because if this is the best our industry has to offer, it's devastating. So I understand where pjmlp is coming from. It's why I've all but withdrawn from here and don't bother posting anything any more.
  - laumars 6y ago
    SQLite has been around 20 years and designed to solve the kinds of problems being discussed. It’s older than git and nearly as old as SSH! So I don’t think it’s fair to brush this conversation with the same brush as those who advocate using Kubernetes for running a couple daemons.
    - Annatar 6y ago
      This was in no way meant to be a critique on SQLite; SQLite is a phenomenal, high quality software and it is extremely useful. Why, I often use it myself and it's my first go-to database for solving all kinds of problems, or even just for simulations. I've known about and used SQLite for almost two decades now. Fantastic software, and that is rare in this day and age when most software is utter garbage. It would be so wonderful if more software were of as high a quality and as simple as SQLite... What I was answering to is the completely unwarranted criticism of pjmlp from various posters here. Not only do I find it unjust, I find it appalling. The gall...
- badsectoracula 6y ago
  pjmlp longs for the days when developers (and/or their employers) paid for their tools because he believes that this allowed them to create tools of higher overall quality than the free FLOSS stuff you find nowadays that may excel in one area which developers find interesting/fun but are awful in several others and largely proliferate not because of their quality but of their price. His comments basically follow out of this starting point so if you have that in mind they start to make sense. (and TBH i do not completely disagree with this view myself, though i do not see the point of repeating it everywhere, this train not only left the station long time ago but the station has long been converted to an Amazon warehouse - besides it isn't like the development tools of old were always that great... to be relatively on topic, there were a ton of DBMSs that simply failed and others that started nice but created stinkers so awful that killed the entire company - see dBase as a great example for that)
CaliforniaKarl 6y ago
That's true, I could've done exactly the same thing with MySQL, MariaDB, Postgres, etc.. But take note of the situations, and you'll notice some things which make SQLite stand out. I said I was working in Python. I was working across three environments: Two separate compute environments, and my work macOS laptop. I didn't want to have to deal with installing the appropriate DB driver, and sqlite is part of the Python standard library. Another thing to note is that I didn't already have a database server spun up, and I didn't think it was worth doing the work of spinning up a Postgres database server. Finally, I didn't intend the "single file" to be an export. I intended it to be an actual, usable database, which my co-workers can use with minimal friction. Those are a few of the reasons why I made the choice I did. And I think SQLite worked quite well!
skissane 6y ago
> I do exactly the same with traditional RDMS. For the use case of "I need a quick database for some ad hoc analysis", SQLite has a lot less friction than a traditional RDBMS. Creating a new DB, backing it up, sharing it with others – all these tasks are doable on a traditional RDBMS too, but generally involve additional steps or prerequisites compared to what SQLite requires. > Export to single file isn't nothing new. Most databases have some kind of export/import function based around CSV, SQL, XML, JSON, or even some proprietary binary format. But SQLite is a lot easier than that – I can just copy a single database file to another system and just open it and it works straight away. (The only proviso is make sure the DB isn't open at the time of copy, to avoid any corruption issues.) No need to run some time consuming import/export process which converts between the external format and the database's native storage format.
- pjmlp 6y ago
  Same here, depending on the project, I always have either SQL Developer with Oracle, or SQL Server Management Studio with SQL Server installed.
  - skissane 6y ago
    Copying a database to another machine is a lot simpler with sqlite than with either of the tools you mention. So I don't agree with the "same here" part.

emmelaich 6y ago

Perhaps you should add your tables to Osquery? https://osquery.io/ https://osquery.io/

chrisweekly 6y ago
Thanks for the tip! Bookmarked, will look closer at OSQuery after vaca.