7 ms·

I have an application that uses sqlite as their file format but they've cleared the header so you can't open it directly. I'd love to be able to make secondary

I have an application that uses sqlite as their file format but they've cleared the header so you can't open it directly. I'd love to be able to make secondary applications like you've described but being enterprise software they don't want to make it too easy. They obviously want to keep people locked in with their $40k per seat application! I guess the first step is figuring out the page size and other bits the other meta data you set in the header [1]. I know I just have to sit down and understand the format better and I will eventually figure it out... [1] https://www.sqlite.org/fileformat.html https://www.sqlite.org/fileformat.html

TheRealPomax 6y ago
That's mean, but you'll just have to live with that. If someone takes active steps towards preventing people from making secondary apps/utils, then they're (a) jerks, and (b) allowed to do that. Might be worth actually contacting them to ask why, if you can make the case that secondary applications will increase the value of their app, not decrease it.
- lbutler 6y ago
  I can understand why they do it, it's enterprise software so the more open you make it, then the less licences they can sell. They do have a module you can purchase to run API calls and access their files/software but as you probably guessed that's another $40k license! Most of my apps I build use this API, but for me to provide to other companies they need them to also buy the API extension. I'd love to cut out the middle man and I'll do it eventually when I reverse engineer the header!
  - mrmonkeyman 6y ago
    I don't get this mentality. You haven't created this system. You have no idea what it takes or else you would have created it yourself. You don't like the price, but you want it anyway. How is this, I mean come on, different from stealing? You even say it: cutting out the middle man. You want to remove their profits.
  - xenonite 6y ago
    Although the circumventing of security measures would be illegal in many countries. (Like removing the copy protection of a game)
- traverseda 6y ago
  >That's mean, but you'll just have to live with that I mean I know we're all on board with the idea of intellectual property actually being a thing now, but surely there are limits? I've seen people take the hard-line stance that if something is your property you should be able to dictate exactly under what situation it can be used, but there have to be limits to IP holders rights on some level, and I feel like reverse engineering a file format is a pretty reasonable place to draw that line.
  - jtbayly 6y ago
    I didn't take "you'll just have to live with that" as a claim that the user couldn't reverse engineer it. It simply means that they've made it harder for the user to do so, and so the user has to live with it being harder.
    - TheRealPomax 6y ago
      this.
  - jstanley 6y ago
    > I know we're all on board with the idea of intellectual property actually being a thing now I don't know that that's true at all! I'd say the pendulum is swinging in the opposite direction.
  - cortesoft 6y ago
    They aren't saying you have to live with the legal restriction, but rather the technical restriction of having to reverse engineer it without the headers
  - deathgrips 6y ago
    >we're all on board with the idea of intellectual property actually being a thing now No. Intellectual property is not genuine property. It is a state granted monopoly and is antithetical to free market principles.
    - 6y ago
- hamburglar 6y ago
  I'm surprised that nobody's pointed out that there are actually valid reasons other than greed to obscure your file format. It's an implementation detail, not a contract. If customers begin relying on the implementation details, you end up with angry customers when you change the implementation details. A SQLite db without the header is basically a statement saying, "we are using the obvious file format here for our convenience, not for general purpose access. Screw around in here at your own risk." If you modified their app's internal state db and screwed it up because they have designed their software with certain assumptions that aren't clear from just reading their db schema, that would be a nightmare for them to support. The easiest thing for them to do is just to try to discourage tampering with their internal state. This is especially true if there's a chance that a market for secondary apps/utils will spring up. If that's to happen and be viable, they absolutely would want to put thought into what their supported interfaces are for those apps/utils, otherwise they will end up painted into a corner and unable to change their architecture without destroying a marketplace.
  - EvanAnderson 6y ago
    I've never understood the "We can ship this but it's not contract" mentality. To my mind if you ship something it's contract. I think developers keeping that in mind would make the world full of more resilient code. You're making a contract w/ your future self, if nothing else. To the point re: modifying internal state and screwing-up the application - If you're writing anything out to persistent storage you should assume that it's untrusted data when you read it back in. If for no other reason than physics itself is a malicious (or, at best, ambivalent) actor.
    - catalogia 6y ago
      Sometimes a programmer may like to create such a contract, if only with their future self, but have reason to fear their boss will force them to break that contract in some not unlikely future.
    - hamburglar 6y ago
      If your app ships with a dynamically loaded library and I dig through the exposed functions and find something undocumented that looks useful and I figure out how to use it, that's a contract to you? That's utterly insane. Re: the concept of untrusted data, this is off in the weeds argument for argument's sake IMO. Do reasonable validations of the state data, sure, but picking nits about the nature of trust and internal application state is an infinite hole I'm not jumping into with you.
freeopinion 6y ago
At $40K/seat, that must be some special software. Simple economics invites competition. Depending on the software, it may not take too many seats to incent the funding of an alternative. Maybe this is the kind of software that requires huge development costs. But maybe it would be worth 20 seats' worth of customers joining forces to fund a team of 5 people to build you a competing app tailored to your specific needs/wants and completely under your control. Granted, that could bump your costs from $800K/year to $1.6M+/year. But only short-term. Once your software is production-ready, you drop the costs of your current software. So think of it more like going from $8M/10 years to $6-10M/10 years but having complete control to add the features you want. And perhaps having the opportunity to recoup $millions/year by licensing to others. Or, open source it and give others the same kind of control while benefiting from the features they add. Spread your development costs across more seats to further lower your $/seat. Or, look at the 100 employees your vendor currently has and lose heart, then hope somebody with deep pockets funds a competitor.
- lbutler 6y ago
  This particular software is for water utilities to model and simulate their water and waste water networks. It's mostly used so utilites can forecast growth in their areas for the next 25+ years and see the impact on their networks and feed into their capital work projects. A decently sized utility may spend up to $200M/yr on capital works so $40k isn't even a line item! There is completion in the market but consultants are forced to use what their clients pick and most utilites aren't that price sensitive. There are also open source alternatives by the EPA[1][2], and most commercial operators are just wrappers around this public domain software. I'm trying to create FOSS to help view and run these models. [1] https://en.m.wikipedia.org/wiki/EPANET https://en.m.wikipedia.org/wiki/EPANET [2] https://en.m.wikipedia.org/wiki/Storm_Water_Management_Model https://en.m.wikipedia.org/wiki/Storm_Water_Management_Model
  - divbzero 6y ago
    Thanks for sharing! Your FOSS projects [1] look awesome, wish you the best in building them out. I’m fascinated by examples of niche but critical software applications. It reminds me of Patrick MacKenzie’s thesis that there are more B2B software opportunities out there than you’d expect. [2] [1]: https://github.com/modelcreate/ https://github.com/modelcreate/ [2]: https://twitter.com/patio11/status/1213188681843892224 https://twitter.com/patio11/status/1213188681843892224
kccqzy 6y ago
I might be wrong but I thought Dropbox used to do that as well. All the sync state is stored in a sqlite file with the header changed or removed.

TheRealPomax 6y ago

That's mean, but you'll just have to live with that. If someone takes active steps towards preventing people from making secondary apps/utils, then they're (a) jerks, and (b) allowed to do that. Might be worth actually contacting them to ask why, if you can make the case that secondary applications will increase the value of their app, not decrease it.

lbutler 6y ago
I can understand why they do it, it's enterprise software so the more open you make it, then the less licences they can sell. They do have a module you can purchase to run API calls and access their files/software but as you probably guessed that's another $40k license! Most of my apps I build use this API, but for me to provide to other companies they need them to also buy the API extension. I'd love to cut out the middle man and I'll do it eventually when I reverse engineer the header!
- mrmonkeyman 6y ago
  I don't get this mentality. You haven't created this system. You have no idea what it takes or else you would have created it yourself. You don't like the price, but you want it anyway. How is this, I mean come on, different from stealing? You even say it: cutting out the middle man. You want to remove their profits.
- xenonite 6y ago
  Although the circumventing of security measures would be illegal in many countries. (Like removing the copy protection of a game)
traverseda 6y ago
>That's mean, but you'll just have to live with that I mean I know we're all on board with the idea of intellectual property actually being a thing now, but surely there are limits? I've seen people take the hard-line stance that if something is your property you should be able to dictate exactly under what situation it can be used, but there have to be limits to IP holders rights on some level, and I feel like reverse engineering a file format is a pretty reasonable place to draw that line.
- jtbayly 6y ago
  I didn't take "you'll just have to live with that" as a claim that the user couldn't reverse engineer it. It simply means that they've made it harder for the user to do so, and so the user has to live with it being harder.
  - TheRealPomax 6y ago
    this.
- jstanley 6y ago
  > I know we're all on board with the idea of intellectual property actually being a thing now I don't know that that's true at all! I'd say the pendulum is swinging in the opposite direction.
- cortesoft 6y ago
  They aren't saying you have to live with the legal restriction, but rather the technical restriction of having to reverse engineer it without the headers
- deathgrips 6y ago
  >we're all on board with the idea of intellectual property actually being a thing now No. Intellectual property is not genuine property. It is a state granted monopoly and is antithetical to free market principles.
  - 6y ago
hamburglar 6y ago
I'm surprised that nobody's pointed out that there are actually valid reasons other than greed to obscure your file format. It's an implementation detail, not a contract. If customers begin relying on the implementation details, you end up with angry customers when you change the implementation details. A SQLite db without the header is basically a statement saying, "we are using the obvious file format here for our convenience, not for general purpose access. Screw around in here at your own risk." If you modified their app's internal state db and screwed it up because they have designed their software with certain assumptions that aren't clear from just reading their db schema, that would be a nightmare for them to support. The easiest thing for them to do is just to try to discourage tampering with their internal state. This is especially true if there's a chance that a market for secondary apps/utils will spring up. If that's to happen and be viable, they absolutely would want to put thought into what their supported interfaces are for those apps/utils, otherwise they will end up painted into a corner and unable to change their architecture without destroying a marketplace.
- EvanAnderson 6y ago
  I've never understood the "We can ship this but it's not contract" mentality. To my mind if you ship something it's contract. I think developers keeping that in mind would make the world full of more resilient code. You're making a contract w/ your future self, if nothing else. To the point re: modifying internal state and screwing-up the application - If you're writing anything out to persistent storage you should assume that it's untrusted data when you read it back in. If for no other reason than physics itself is a malicious (or, at best, ambivalent) actor.
  - catalogia 6y ago
    Sometimes a programmer may like to create such a contract, if only with their future self, but have reason to fear their boss will force them to break that contract in some not unlikely future.
  - hamburglar 6y ago
    If your app ships with a dynamically loaded library and I dig through the exposed functions and find something undocumented that looks useful and I figure out how to use it, that's a contract to you? That's utterly insane. Re: the concept of untrusted data, this is off in the weeds argument for argument's sake IMO. Do reasonable validations of the state data, sure, but picking nits about the nature of trust and internal application state is an infinite hole I'm not jumping into with you.

freeopinion 6y ago

At $40K/seat, that must be some special software. Simple economics invites competition. Depending on the software, it may not take too many seats to incent the funding of an alternative. Maybe this is the kind of software that requires huge development costs. But maybe it would be worth 20 seats' worth of customers joining forces to fund a team of 5 people to build you a competing app tailored to your specific needs/wants and completely under your control. Granted, that could bump your costs from $800K/year to $1.6M+/year. But only short-term. Once your software is production-ready, you drop the costs of your current software. So think of it more like going from $8M/10 years to $6-10M/10 years but having complete control to add the features you want. And perhaps having the opportunity to recoup $millions/year by licensing to others. Or, open source it and give others the same kind of control while benefiting from the features they add. Spread your development costs across more seats to further lower your $/seat. Or, look at the 100 employees your vendor currently has and lose heart, then hope somebody with deep pockets funds a competitor.

lbutler 6y ago
This particular software is for water utilities to model and simulate their water and waste water networks. It's mostly used so utilites can forecast growth in their areas for the next 25+ years and see the impact on their networks and feed into their capital work projects. A decently sized utility may spend up to $200M/yr on capital works so $40k isn't even a line item! There is completion in the market but consultants are forced to use what their clients pick and most utilites aren't that price sensitive. There are also open source alternatives by the EPA[1][2], and most commercial operators are just wrappers around this public domain software. I'm trying to create FOSS to help view and run these models. [1] https://en.m.wikipedia.org/wiki/EPANET https://en.m.wikipedia.org/wiki/EPANET [2] https://en.m.wikipedia.org/wiki/Storm_Water_Management_Model https://en.m.wikipedia.org/wiki/Storm_Water_Management_Model
- divbzero 6y ago
  Thanks for sharing! Your FOSS projects [1] look awesome, wish you the best in building them out. I’m fascinated by examples of niche but critical software applications. It reminds me of Patrick MacKenzie’s thesis that there are more B2B software opportunities out there than you’d expect. [2] [1]: https://github.com/modelcreate/ https://github.com/modelcreate/ [2]: https://twitter.com/patio11/status/1213188681843892224 https://twitter.com/patio11/status/1213188681843892224

kccqzy 6y ago

I might be wrong but I thought Dropbox used to do that as well. All the sync state is stored in a sqlite file with the header changed or removed.