9 ms·

The author seems to assume that everybody is using SQLite, but SQLite for a production database is an extremely niche choice. Attempting to expose more popular

by luhn 5y ago

The author seems to assume that everybody is using SQLite, but SQLite for a production database is an extremely niche choice. Attempting to expose more popular options like PostgreSQL or MySQL as SQLite would be extremely difficult because SQLite only supports a subset of SQL, whereas PostgreSQL and MySQL both implement their unique superset (for the most part) of SQL. But it doesn't matter. The API doesn't matter. Web 3.0 was never about APIs, it was about data. A standardized API is only useful if it outputs standardized data. Having a bunch of bespoke SQLite tables scattered across the web gets us no closer to the ideal of Web 3.0.

jumpkick 5y ago
I thought Web 3.0 was about NFTs.
- dspillett 5y ago
  People are talking that way now. Before what is being called “Web3” ATM Web 3.0 was to be the “semantic web” (where Web 2.0 was the “interactive web” - with many things become read/write instead of read-only and greater interactivity, both in terms of social interactivity and individual interactive apps being web based, being the focus of technological enhancements). This article is talking about that earlier definition, and a way it might once again be the definition, perhaps relegating Web3 to Web 4.0 (or web we-worked-out-it-was-a-ponzi-scheme-so-just-stopped with something else being Web 4.0, if you take the more cynical view).
- tyingq 5y ago
  "Before the term was hijacked by crypto-grifters (and, admittedly, a few genuinely neat projects), web 3 (point oh) referred to Tim Berners-Lee's project to promote a standard way to expose and parse metadata on the web."
Closi 5y ago
SQLite is the most used database engine in the world, so I wouldn't call it niche. In fact, by some estimates, it is probably used more than all other database engines combined. The only difference is that it is usually run locally (compared to Postgres and your other examples), but something doesn't have to run remotely to be considered running in production :)
- luhn 5y ago
  Yes, when I said "production database" I meant a database for a web application. My iPhone running SQLite doesn't relate to Web 3.0.
  - Closi 5y ago
    > Yes, when I said "production database" I meant a database for a web application I'm not sure that's what the author of the article means though, at least in my interpretation. I assume when they say "everyone is already using it" I assumed that they meant literally everyone is using it on their phones and PCs every day, not that everyone is using it to develop production web applications (because very few people develop production web applications in the grand scheme of things!). I presume they mean that it is one of (if not the) most common databases in existence in the wild, and it's interesting that it has this property of being able to be remotely read with surprisingly little overhead (without the need to implement an entirely bespoke database to be read in this way).
    - luhn 5y ago
      I'm not sure how to parse what the author is saying besides you should be exposing your database directly, which is apparently SQLite. > In the process, it demonstrated a new kind of web app whose entire database was exposed and queryable from the outside. > The data needs to be exposed in its original form; any additional translation step will ensure that most people won't bother.
    - netcan 5y ago
      I think he does mean web apps/sites, at least in large part. He is talking about implementing of web 3.0, after all. OTOH, I suppose there's no reason why web has to apply primarily to things that are currently webstuff. You both make good points.
    - dfabulich 5y ago
      I think you misunderstood the author. "The data needs to be exposed in its original form; any additional translation step will ensure that most people won't bother. The beauty of this technique is that you are already using SQLite because it's such a powerful database; with no additional work, you can throw it on a static file server and others can easily query it over HTTP." The author believes (IMO wrongly) that there's lots of web app data that can be exposed via SQLite-over-HTTP without translating it into SQLite, because it's already in SQLite. The author is saying that since lots of web apps use SQLite for their production database, they can easily "throw" their SQLite DB onto the web. But, in that case, you're out of luck if you use Postgres, MySQL, Oracle, MS SQL Server, or any of the popular key-value datastores like Mongo, Redis, or Elasticsearch.
sekao 5y ago
My point was not that people are using SQLite in prod everywhere; read that paragraph in more of a speculative voice, not a statement of fact about the present. At any rate, i do think the range request technique makes SQLite more practical to use in database-driven apps that normally would've opted for a traditional db like postgres (though there is more work to be done to make this technique fast when doing complex queries...lots of joins are no bueno right now).
- quinnjh 5y ago
  I enjoyed the speculative bias there as its prompting me to pick up the sqlite bundle thats been sitting in my downloads while i use post-gres-like-the-rest
DangitBobby 5y ago
Well, having to write unique SQL per site is much better than having to write unique scrapers per site.
- RileyJames 5y ago
  Exactly. And there’s really no reason those queries couldn’t be made public / collaboratively maintained. You could probably take it one step further and define an OpenAPI spec which is populated via those queries. Tho that would require an intermediary / post-processing, likely with a cache. Regardless, the capability to determine how and what to consume sits with the consumer (developer) from the outset. Rather than having to scrape the data, normalise it into some form of schema, and then build an api / interface around it. And then worry about keeping it up to date.
contravariant 5y ago
Wouldn't you want to use a subset of SQL in an API? Why use a unique superset that differs per webpage?
simonw 5y ago
You may be underestimating SQLite here. It has grown a LOT of features over the past decade, many of which were directly inspired by PostgreSQL. Features you may have missed: - CTEs - the WITH statement works great. And recursive CTEs, which can do mandelbrot fractals! https://www.sqlite.org/lang_with.html https://www.sqlite.org/lang_with.html - Surprisingly good built-in full text search: https://www.sqlite.org/fts5.html https://www.sqlite.org/fts5.html - Functions for directly querying JSON data in columns: https://www.sqlite.org/json1.html https://www.sqlite.org/json1.html - Extremely comprehensive geospatial capabilities thanks to the SpatiaLite extension - this has a huge amount of functionality, which I think is better than MySQL though not yet as good as PostGIS: https://www.gaia-gis.it/fossil/libspatialite/index https://www.gaia-gis.it/fossil/libspatialite/index I'm not saying it's as "good" as PostgreSQL, but I don't thank your argument that PostgreSQL and MySQL implement a substantially larger portion of SQL holds up particularly well.
- capiki 5y ago
  As luhn said, it’s more about a standard data format than a db choice. If every client has to figure out what schema a website uses for a recipe, let’s say, then Web 3.0 is still unrealistic. Schema.org exists, but all websites adopting it seems unlikely. That being said, I can maybe see a world in which one company adopts schema.org schemas and the rest have to follow suit to be competitive in that particular domain.
  - zozbot234 5y ago
    > Schema.org exists, but all websites adopting it seems unlikely. Schema.org has the backing of major search engines and other reusers of Web-served content. It's way more likely to be adopted compared to anything else in its general domain.

by luhn 5y ago

jumpkick 5y ago

I thought Web 3.0 was about NFTs.

dspillett 5y ago
People are talking that way now. Before what is being called “Web3” ATM Web 3.0 was to be the “semantic web” (where Web 2.0 was the “interactive web” - with many things become read/write instead of read-only and greater interactivity, both in terms of social interactivity and individual interactive apps being web based, being the focus of technological enhancements). This article is talking about that earlier definition, and a way it might once again be the definition, perhaps relegating Web3 to Web 4.0 (or web we-worked-out-it-was-a-ponzi-scheme-so-just-stopped with something else being Web 4.0, if you take the more cynical view).
tyingq 5y ago
"Before the term was hijacked by crypto-grifters (and, admittedly, a few genuinely neat projects), web 3 (point oh) referred to Tim Berners-Lee's project to promote a standard way to expose and parse metadata on the web."

Closi 5y ago

SQLite is the most used database engine in the world, so I wouldn't call it niche. In fact, by some estimates, it is probably used more than all other database engines combined. The only difference is that it is usually run locally (compared to Postgres and your other examples), but something doesn't have to run remotely to be considered running in production :)

luhn 5y ago
Yes, when I said "production database" I meant a database for a web application. My iPhone running SQLite doesn't relate to Web 3.0.
- Closi 5y ago
  > Yes, when I said "production database" I meant a database for a web application I'm not sure that's what the author of the article means though, at least in my interpretation. I assume when they say "everyone is already using it" I assumed that they meant literally everyone is using it on their phones and PCs every day, not that everyone is using it to develop production web applications (because very few people develop production web applications in the grand scheme of things!). I presume they mean that it is one of (if not the) most common databases in existence in the wild, and it's interesting that it has this property of being able to be remotely read with surprisingly little overhead (without the need to implement an entirely bespoke database to be read in this way).
  - luhn 5y ago
    I'm not sure how to parse what the author is saying besides you should be exposing your database directly, which is apparently SQLite. > In the process, it demonstrated a new kind of web app whose entire database was exposed and queryable from the outside. > The data needs to be exposed in its original form; any additional translation step will ensure that most people won't bother.
  - netcan 5y ago
    I think he does mean web apps/sites, at least in large part. He is talking about implementing of web 3.0, after all. OTOH, I suppose there's no reason why web has to apply primarily to things that are currently webstuff. You both make good points.
  - dfabulich 5y ago
    I think you misunderstood the author. "The data needs to be exposed in its original form; any additional translation step will ensure that most people won't bother. The beauty of this technique is that you are already using SQLite because it's such a powerful database; with no additional work, you can throw it on a static file server and others can easily query it over HTTP." The author believes (IMO wrongly) that there's lots of web app data that can be exposed via SQLite-over-HTTP without translating it into SQLite, because it's already in SQLite. The author is saying that since lots of web apps use SQLite for their production database, they can easily "throw" their SQLite DB onto the web. But, in that case, you're out of luck if you use Postgres, MySQL, Oracle, MS SQL Server, or any of the popular key-value datastores like Mongo, Redis, or Elasticsearch.

sekao 5y ago

My point was not that people are using SQLite in prod everywhere; read that paragraph in more of a speculative voice, not a statement of fact about the present. At any rate, i do think the range request technique makes SQLite more practical to use in database-driven apps that normally would've opted for a traditional db like postgres (though there is more work to be done to make this technique fast when doing complex queries...lots of joins are no bueno right now).

quinnjh 5y ago
I enjoyed the speculative bias there as its prompting me to pick up the sqlite bundle thats been sitting in my downloads while i use post-gres-like-the-rest

DangitBobby 5y ago

Well, having to write unique SQL per site is much better than having to write unique scrapers per site.

RileyJames 5y ago
Exactly. And there’s really no reason those queries couldn’t be made public / collaboratively maintained. You could probably take it one step further and define an OpenAPI spec which is populated via those queries. Tho that would require an intermediary / post-processing, likely with a cache. Regardless, the capability to determine how and what to consume sits with the consumer (developer) from the outset. Rather than having to scrape the data, normalise it into some form of schema, and then build an api / interface around it. And then worry about keeping it up to date.

contravariant 5y ago

Wouldn't you want to use a subset of SQL in an API? Why use a unique superset that differs per webpage?

simonw 5y ago

You may be underestimating SQLite here. It has grown a LOT of features over the past decade, many of which were directly inspired by PostgreSQL. Features you may have missed: - CTEs - the WITH statement works great. And recursive CTEs, which can do mandelbrot fractals! https://www.sqlite.org/lang_with.html https://www.sqlite.org/lang_with.html - Surprisingly good built-in full text search: https://www.sqlite.org/fts5.html https://www.sqlite.org/fts5.html - Functions for directly querying JSON data in columns: https://www.sqlite.org/json1.html https://www.sqlite.org/json1.html - Extremely comprehensive geospatial capabilities thanks to the SpatiaLite extension - this has a huge amount of functionality, which I think is better than MySQL though not yet as good as PostGIS: https://www.gaia-gis.it/fossil/libspatialite/index https://www.gaia-gis.it/fossil/libspatialite/index I'm not saying it's as "good" as PostgreSQL, but I don't thank your argument that PostgreSQL and MySQL implement a substantially larger portion of SQL holds up particularly well.

capiki 5y ago
As luhn said, it’s more about a standard data format than a db choice. If every client has to figure out what schema a website uses for a recipe, let’s say, then Web 3.0 is still unrealistic. Schema.org exists, but all websites adopting it seems unlikely. That being said, I can maybe see a world in which one company adopts schema.org schemas and the rest have to follow suit to be competitive in that particular domain.
- zozbot234 5y ago
  > Schema.org exists, but all websites adopting it seems unlikely. Schema.org has the backing of major search engines and other reusers of Web-served content. It's way more likely to be adopted compared to anything else in its general domain.