4 ms·

The abstractions you build your code on top of can be very costly, and a lot of that cost is due later. You need to carefully weight the costs of abstractions

by jmull 1y ago

The abstractions you build your code on top of can be very costly, and a lot of that cost is due later. You need to carefully weight the costs of abstractions you adopt. That goes double (or triple or quadruple) for data access, which is typically the central point of an app. When sqlite is involved, I'm really doubting an ORM is worth it in the great majority of cases. That's because sqlite is robust, mature, full-featured, very well documented, has demonstrated long-term viability, has a commitment to backwards compatibility and support, etc. I think it makes the most sense to implement an application-level data-access layer and have it use sqlite as directly as possible. What sqlite doesn't do that an app needs is dealing with Swift types. I.e., the grunt work of the sqlite3_bind_* and sqlite3_column_* calls to transform Swift values to and from sqlite3 values. (And to a lesser extent, various Swift-isms for quality of life.) But you can have that without the rather more intrusive footprint -- and therefore much higher cost -- of an ORM.

wahnfrieden 1y ago
How else will you support CloudKit with SQLite? Personally CloudKit is a requirement as my customers value having their personal data inside their own Apple account rather than storing it on my servers. I also rely on CloudKit to avoid the cost and on-call burden of operating a realtime sync service. Re: ORM, the SwiftData-like ORM part of SQLiteData is actually completely optional. You can write SQL statements directly via #sql: https://swiftpackageindex.com/pointfreeco/swift-structured-queries/~/documentation/structuredqueriescore/safesqlstrings https://swiftpackageindex.com/pointfreeco/swift-structured-q... You can wrap usage of this in your own method so that you can swap out SQLiteData in the future without lock-in. Re: Swift types, you may roll your own mapping, but performance is not trivial. Here is a benchmark from SQLiteData: Orders.fetchAll setup rampup duration SQLite (generated by Enlighter 1.4.10) 0 0.144 7.183 Lighter (1.4.10) 0 0.164 8.059 SQLiteData (1.0.0) 0 0.172 8.511 GRDB (7.4.1, manual decoding) 0 0.376 18.819 SQLite.swift (0.15.3, manual decoding) 0 0.564 27.994 SQLite.swift (0.15.3, Codable) 0 0.863 43.261 GRDB (7.4.1, Codable) 0.002 1.07 53.326 I found Enlighter and Lighter more intrusive to adopt, and the other open source solutions far slower (too slow for my needs, where my users have hundreds of thousands or millions of rows within the iOS apps)
stephencelis 1y ago
Are you suggesting SQLiteData is an ORM, or SwiftData? SwiftData (and CoreData) are certainly ORMs, but SQLiteData is not. It's simply a collection of tools on top of SQLite that provide similar functionality to SwiftData, but you always have direct access to SQLite. The things you say that SQLite doesn't do is exactly what SQLiteData provides (Swift-friendly bindings for encoding and decoding data from SQLite), and more. There's a footprint, as there is with any library, but there is no ORM level of abstraction here.
- jmull 1y ago
  Well, it has things like @Table and wants you to write SQL using Swift syntax. It's good that it lets you bypass this, but it encourages using the wrappers so I think that's going to be most of the usage. Also, if you're going to mostly bypass, then why incur the costs of adopting? The main bypass I saw was #sql, but while you can write SQL, it's a substantially different API than sqlite's C API to do the same. One problem with data access wrappers is that what's both above it and below it have strong app concerns. The developer needs to understand and control what's below it, so an intermediate abstraction gets in the way. That is, in addition to understanding the lower-level, they also have to understand the intermediate abstraction, and how it maps to the lower level. So it's really best if the API surface is minimal. What I would want is an API where the core is something like: bindParameters(pStmt, anEncodableThing) readRowColumns(pStmt, ADecodableThing.Type) -> ADecodableThing where pStmt would be as minimal a wrapper around a sqlite statement as is feasible (maybe even a pointer to a sqlite3_stmt directly?). There might be a minimal (non-existent?) wrapper around the sqlite3 connection too. (The sqlite pointer should be a public member of any wrapper, so you can call any sqlite3_ functions you want). (I'd want some convenience methods too, that combine bindParameters and readRowColumns with preparing a statement and stepping through it, like sqlite's exec.) Now, I know this doesn't address CloudKit sync at all, but I think a similarity minimal type-focused binding is best there too. It has nowhere near the 5-star API that sqlite has so there's a better argument for wrapping it, but sync tends to quickly accumulate app-level concerns when it comes to the exact details of sync conflicts, so you might as well keep that at app level. I think there maybe a set of composable convenience methods might do it, to handle the pain points without hiding the details you'll need access to. Anyway, my point is NOT that sqlitedata is bad, it's that people should be really careful about taking on the costs that it has, and consider whether it will ultimately cause more problems than it solves. Meanwhile, sqlite is world-class -- you want to wrap it as little as possible.

by jmull 1y ago

wahnfrieden 1y ago

How else will you support CloudKit with SQLite? Personally CloudKit is a requirement as my customers value having their personal data inside their own Apple account rather than storing it on my servers. I also rely on CloudKit to avoid the cost and on-call burden of operating a realtime sync service. Re: ORM, the SwiftData-like ORM part of SQLiteData is actually completely optional. You can write SQL statements directly via #sql: https://swiftpackageindex.com/pointfreeco/swift-structured-queries/~/documentation/structuredqueriescore/safesqlstrings https://swiftpackageindex.com/pointfreeco/swift-structured-q... You can wrap usage of this in your own method so that you can swap out SQLiteData in the future without lock-in. Re: Swift types, you may roll your own mapping, but performance is not trivial. Here is a benchmark from SQLiteData: Orders.fetchAll setup rampup duration SQLite (generated by Enlighter 1.4.10) 0 0.144 7.183 Lighter (1.4.10) 0 0.164 8.059 SQLiteData (1.0.0) 0 0.172 8.511 GRDB (7.4.1, manual decoding) 0 0.376 18.819 SQLite.swift (0.15.3, manual decoding) 0 0.564 27.994 SQLite.swift (0.15.3, Codable) 0 0.863 43.261 GRDB (7.4.1, Codable) 0.002 1.07 53.326 I found Enlighter and Lighter more intrusive to adopt, and the other open source solutions far slower (too slow for my needs, where my users have hundreds of thousands or millions of rows within the iOS apps)

stephencelis 1y ago

Are you suggesting SQLiteData is an ORM, or SwiftData? SwiftData (and CoreData) are certainly ORMs, but SQLiteData is not. It's simply a collection of tools on top of SQLite that provide similar functionality to SwiftData, but you always have direct access to SQLite. The things you say that SQLite doesn't do is exactly what SQLiteData provides (Swift-friendly bindings for encoding and decoding data from SQLite), and more. There's a footprint, as there is with any library, but there is no ORM level of abstraction here.

jmull 1y ago
Well, it has things like @Table and wants you to write SQL using Swift syntax. It's good that it lets you bypass this, but it encourages using the wrappers so I think that's going to be most of the usage. Also, if you're going to mostly bypass, then why incur the costs of adopting? The main bypass I saw was #sql, but while you can write SQL, it's a substantially different API than sqlite's C API to do the same. One problem with data access wrappers is that what's both above it and below it have strong app concerns. The developer needs to understand and control what's below it, so an intermediate abstraction gets in the way. That is, in addition to understanding the lower-level, they also have to understand the intermediate abstraction, and how it maps to the lower level. So it's really best if the API surface is minimal. What I would want is an API where the core is something like: bindParameters(pStmt, anEncodableThing) readRowColumns(pStmt, ADecodableThing.Type) -> ADecodableThing where pStmt would be as minimal a wrapper around a sqlite statement as is feasible (maybe even a pointer to a sqlite3_stmt directly?). There might be a minimal (non-existent?) wrapper around the sqlite3 connection too. (The sqlite pointer should be a public member of any wrapper, so you can call any sqlite3_ functions you want). (I'd want some convenience methods too, that combine bindParameters and readRowColumns with preparing a statement and stepping through it, like sqlite's exec.) Now, I know this doesn't address CloudKit sync at all, but I think a similarity minimal type-focused binding is best there too. It has nowhere near the 5-star API that sqlite has so there's a better argument for wrapping it, but sync tends to quickly accumulate app-level concerns when it comes to the exact details of sync conflicts, so you might as well keep that at app level. I think there maybe a set of composable convenience methods might do it, to handle the pain points without hiding the details you'll need access to. Anyway, my point is NOT that sqlitedata is bad, it's that people should be really careful about taking on the costs that it has, and consider whether it will ultimately cause more problems than it solves. Meanwhile, sqlite is world-class -- you want to wrap it as little as possible.