14 ms·

Recently minted database technologies that I find intriguing

473 points by biggestlou 6y ago

tourist_on_road 6y ago
How does tiledb compares to something similar like milvus
- biggestlou 6y ago
  I haven't come across Milvus but I'll check it out!
hn_check 6y ago
"What I have yet to see but always secretly wanted, however, is a database that natively supports incremental updates to materialized views" SQL Server ala 10+ years ago enters the discussion.
- biggestlou 6y ago
  That's good to know! SQL Server has really never been on my radar but I think it's a good time to do some exploration.
- liprais 6y ago
  Teradata does that too
- jiggawatts 6y ago
  I've actually used materialized views in production on SQL Server, and they're great when they work, but they have far too many limitations. Most of these make sense, but some of the join limitations are internal constraints of the engine, not a fundamental limit. This prevents their use for a wide range of scenarios where they would be useful...
- garyclarke27 6y ago
  When I tried to use auto incremental materialized views in SQL Server (about 10 years ago) - they had so many limitations that they were virtually useless. I now use Postgres and this is the Number 1 feature that’s missing for me, there have been a few attempts that have unfortunately not made it into the core. I don’t understand why this critical capability is not even on the radar of the core developers - maybe this publicity from Materialize will raise its priority.
remorses 6y ago
I think that the graphql adapters like hasura and goke are also an important innovation, for small mvp projects you can create a graphql api to query your database directly from the frontend, this reduces the development time by a factor of 2 at least.
- biggestlou 6y ago
  Hmmm. I'm not finding Goke anywhere. Could you provide a link?
rst 6y ago
Materialize is neat, but there are other database systems that refresh at least some materialized views on the fly, while being smart about not rebuilding the entire view every time. See for example, Oracle, where FAST REFRESH ON COMMIT does most of what Materialize is advertised as doing, at least for views which that feature can support (restriction list here: https://stackoverflow.com/questions/49578932/materialized-view-in-oracle-with-fast-refresh-instead-of-complete-dosnt-work https://stackoverflow.com/questions/49578932/materialized-vi... ). Mind you, this comes with Oracle's extremely hefty price tag, so I'm not sure I'd recommend it to anyone who isn't already stuck with Oracle, but it is technical precedent. It would be interesting to compare notes, and see what Materialize does better.
- deleted 6y ago
  [deleted]
- willvarfar 6y ago
  Similar to the very cool TimescaleDB’s Continuous Aggregates (and InfluxDB’s not so cool Continous Queries). There’s also ksqlDB that can be used in a similar way?
- atwebb 6y ago
  Indexed views in SQL server behave similarly (though not as performant as Oracle I believe). They also have some constraints on application (schema bound, data types, nesting).
- dkhenry 6y ago
  About a year ago Vitess merged in support for what we call VReplication. ( https://vitess.io/docs/reference/features/vreplication/ https://vitess.io/docs/reference/features/vreplication/ ) The initial demo was a custom Materialized view that did calculations and rollup's on data in realish time ( it uses the MySQL binlog ). Right now the limiting factor for us calling is true Materalized View's is that configuring them is done out of band using the Vitess Control plane, but soon we will be merging in the features to allow standard DDL to configure and manage them!
- jacques_chester 6y ago
davedx 6y ago
Is tiledb useful for storing ML models?
- Shelnutt2 6y ago
  You can certainly store ML models in TileDB. I've been working on an example notebook showing how to store an opencv model in TileDB, its not quite finished yet but will be published soon. Using python as an example language, TileDB-Py offers simple functions, like "from_numpy" [1] for basic and simple models which we can write directly. For more advanced (and more common) use cases of complex models, its easy to create an array to store the model and all associated metadata. TileDB's array metadata and time travel capabilities allow for you to even see how your model changes over time as you update it. [1] https://github.com/TileDB-Inc/TileDB-Py/blob/bcaee16b194675f10235669fcb615928c0af964d/tiledb/libtiledb.pyx#L3974 https://github.com/TileDB-Inc/TileDB-Py/blob/bcaee16b194675f... Disclosure: I'm a member of the TileDB team.
  - davedx 6y ago
    Really cool, thanks for the reply.
mywacaday 6y ago
Does anybody know of a good educational resource on software/best practices that is kept up to date. Ideally something that does not include the latest bleeding edge but things that are battle hardened or getting there. Something that includes open source and commercial software would be ideal.
melvinroest 6y ago
The website didn't load for me. So here it is: https://web.archive.org/web/20200615193041/https://lucperkins.dev/blog/new-db-tech-1/ https://web.archive.org/web/20200615193041/https://lucperkin... Also, I'd like to add one database to the list (I work there for 3 weeks now): TriplyDB [0]. It is making linked data easier. Linked data is useful for when people of different organizations want a shared schema. In many commercial applications one wouldn't want this, as data is the valuable part of a company. However, scientific communities, certain government agencies and other organizations -- that I don't yet know about -- do want this. I think the coolest application of linked data is how the bio-informatics/biology community utilizes it [1, 2]. The reason I found out at all is because one person at Triply works to see if a similar thing can be achieved with psychology. It might make conducting meta-studies a bit easier. I read the HN discussions on linked data and agree with both the nay sayers (it's awkward and too idealistic [4]) and the yay sayers (it's awesome). The thing is: 1. Linked data open, open as in open source, the URI [3] is baked into its design. 2. While the 'API'/triple/RDF format can be awkward, anyone can quite easily understand it. The cool thing is: this includes non-programmers. 3. It's geared towards collaboration. In fact, when reading between the lines, I'd argue it's really good for collaboration between a big heterogeneous group of people. Disclaimer: this is my own opinion, Triply does not know I'm posting this and I don't care ;-) I simply think it's an interesting way of thinking about data. [0] triply.cc [1] A friend of mine once modeled some biochemistry part of C. Elegans from linked data into petrinets: https://www.researchgate.net/publication/263520722_Building_Executable_Biological_Pathway_Models_Automatically_from_BioPAX https://www.researchgate.net/publication/263520722_Building_... [2] https://www.google.com/search?client=safari&rls=en&q=linked+data+and+biology&ie=UTF-8&oe=UTF-8 https://www.google.com/search?client=safari&rls=en&q=linked+... -- I quickly vetted this search [3] I still don't know the difference between a URI and URL. [4] I think back in the day, linked data idealists would say that all data should be linked to interconnect all the knowledge. I'm more pragmatic and simply wonder: in which socio-technological context is linked data simply more useful than other formats? My current very tentative answer is those 3 points.
- iaabtpbtpnn 6y ago
  I recently had to deal with some RDF data expressed as N-triples. So, naturally I loaded it into a proper triplestore and embraced the whole W3C RDF universe and started teaching myself SPARQL, right? Nah, instead I just translated that shit into CSV and loaded it into Postgres, then post-processed it into a relational schema that I can actually understand and query with a language that other people at my company also know. The RDF was just a poorly specified way of communicating that schema to me, along with a data format that's no better than a 3-column CSV. Great stuff from the Semantic Web folks here, real powerful technology. Edit: Also, to answer your question, the difference between a URL (Uniform Resource Locator) and a URI (Uniform Resource Identifier) is that the URL actually points to something, an object at a particular location, and you can paste it into your web browser to view that something. A URI just uses a URL-like scheme to represent identifiers, such that your domain and directory structure provide a kind of namespace that you control. But as long as it follows the format, your URI can contain literally anything, it doesn't have to be human-readable or resolve to anything in a web browser. It might as well be mycompany_1231241542345.
petercooper 6y ago
I edit a database newsletter – https://dbweekly.com/ https://dbweekly.com/ – so tend to always have my eyes out for new releases, what's coming along, and what not. And I thought I'd share a few more things that have jumped out at me recently in case anyone's in the mood for spelunking. 1. QuestDB – https://questdb.io/ https://questdb.io/ – is a performance-focused, open-source time-series database that uses SQL. It makes heavy use of SIMD and vectorization for the performance end of things. 2. GridDB - https://griddb.net/en/ https://griddb.net/en/ - is an in-memory NoSQL time-series database (there's a theme lately with these!) out of Toshiba that was boasting doing 5m writes per second and 60m reads per second on a 20 node cluster recently. 3. MeiliSearch - https://github.com/meilisearch/MeiliSearch https://github.com/meilisearch/MeiliSearch – not exactly a database but basically an Elastic-esque search server written in Rust. Seems to have really taken off. 4. Dolt – https://github.com/liquidata-inc/dolt https://github.com/liquidata-inc/dolt – bills itself as a 'Git for data'. It's relational, speaks SQL, but has version control on everything. TerminusDB, KVRocks, and ImmuDB also get honorable mentions. InfoWorld also had an article recently about 9 'offbeat' databases to check out if you want to go even further: https://www.infoworld.com/article/3533410/9-offbeat-databases-worth-a-look.html https://www.infoworld.com/article/3533410/9-offbeat-database... Exciting times in the database space!
- rapnie 6y ago
  TerminusDB recently featured on HN. Also a git-for-data approach, time-travel, storing data as RDF and queries return JSON-LD. https://news.ycombinator.com/item?id=22867767 https://news.ycombinator.com/item?id=22867767
  - LukeEF 6y ago
    Yes, the latest version of Terminus uses JSON-LD, which is a serialisation of RDF. This enables us to store queries in a database and provides a schema documentation for our datalog query language. Release notes: https://github.com/terminusdb/terminusdb-server/blob/master/RELEASE_NOTES.md https://github.com/terminusdb/terminusdb-server/blob/master/...
- heipei 6y ago
willvarfar 6y ago
My understanding of TileDB is that it is 100% client-side. There is no server. In a sense it’s like handling orc or paraquet or even SQLite files on S3, (except tiledb are fancy r-trees) with a delta-lake-like manifest file for transactions too. I think in the future there’s going to be a sine-wave of smart-clients consuming S3 cleverly, and then smartness growing in the S3 interface so constraints and indices and things happen in the storage again, and back and forth...
- deleted 6y ago
  [deleted]
- jakebol 6y ago
  This is a good description, except that TileDB (the open source client) is not transactional but eventually consistent at least for S3 and other object stores. I like your point about consuming S3 cleverly, it's often difficult to get good out of the box performance from S3 so abstracting that to the degree possible is good for end-users. The cloud vendors though are always one or two steps ahead of companies that build upon their services. AWS Redshift for instance already can pre-index objects stored on S3 to accelerate queries at the storage layer. It's difficult as a third party vendor to compete with that.
  - biggestlou 6y ago
    This is a very interesting development that I'd like to learn more about. Whenever I've played around with writing databases (just as toy projects) I've always done so using RocksDB or something similar as a backend. This "thick client" model, though, seems to have a lot of potential benefits, most notably no need to worry about disk space or volumes (so say goodbye to a bunch of config parameters) and no need for a tiered storage setup or S3 migration tools (already accomplished!). Not ideal for most use cases but intriguing for some!
    - jakebol 6y ago
      There are a lot of issues though with S3, latency, poor performance for small reads / writes, timeouts, api rate limits, api costs, and consistency issues poorly understood by third party developers. A "thick-client" also doesn't perform well unless that client is located on a node in the same region. I think as with everything it works well in some cases and not well in others.
xchaotic 6y ago
Interesting notes but I feel like the db itself has been commoditised and the battle is elsewhere now. So anyone building a database engine today, will find out that to make it sustainable they also need an ecosystem on top of it, tooling, community, paid support, active devs, consultants (for which they may have no runway) Finally I find anything that calls itself a database and uses S3 as a backend a bit ridiculous. S3 has eventual consistency so you can’t do the operations that differentiate a database from a file system.
moonchild 6y ago
> And it’s worth noting that many have tried to do what Prisma does and failed because they. Because they what?
- biggestlou 6y ago
  Derp! Fixed that.
biggestlou 6y ago
REMOVED: I complained about the rewritten title in a way that was excessively harsh and have removed that comment.
- dang 6y ago
  The article title was linkbait: "you" and "should" are linkbait tropes, especially when combined. The site guidelines call for rewriting such titles. See https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html. When we change titles, we always try to use representative language from the article. I replaced it with a phrase from the article where it says what it's about in a neutral way. I added a hyphen to "recently minted" because it seems to me that grammar requires it. However, if we worship different grammar gods I am happy to let yours hold sway. The hyphen is now toast. This is bog-standard HN moderation. I dare say the reason you "really really hate it" is probably because you notice the cases you dislike and weight them much more heavily: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20notice%20dislike&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que.... Meanwhile the ones you don't find objectionable pass over relatively unnoticed. That is bog-standard HN perception bias. If we didn't meticulously try to keep titles linkbait-free and less-misleading (more hyphens, sorry!), HN's front page would be completely different, and since the front page is the most important thing here, the entire forum would soon be completely different. That requires editing titles all the time. petercooper, who posted elsewhere in this thread, has an HN title-change tracker (oops, I hyphened again!) which is quite nifty. It doesn't show all of them, though, and it can't tell the difference between moderator edits and submitter edits. There is a power shortage in my neighborhood and my laptop battery is about to expire, so if you don't see a link momentarily, or if I don't respond anywhere on the site for a few hours, it's because shelter-in-place (uh-oh!) prevents me from finding somewhere else to plug in. Edit: here it is: https://news.ycombinator.com/item?id=21617016 https://news.ycombinator.com/item?id=21617016. Note how we edited out the bloody pig mask.
  - biggestlou 6y ago
    I apologize for the strength of my language. There are several words I used in that comment that I should not have. It's far too easy to forget in the moment that there are real people making these decisions who may very well read what you say. I understand where you're coming from with the decision and will try to select more appropriate titles in the future. And will be much more mindful about how I register complaints.
chickenpotpie 6y ago
I’ve always felt it strange that in almost every job I’ve had databases have been one of the most important pieces of the architecture but the least debated. I’ve spent hours debating languages and frameworks, but databases always come down to whatever we have a license for/what others at the company are using. Engineering teams will always say they make sure to use the right tool for the job, but no one ever talks about if it’s right to keep using the same database for a new product.
- mywittyname 6y ago
  My experience is places with DBAs tend to keep very tight control over database design than those who do without. I've worked at several places where developers did not touch the database schema. They couldn't even propose changes, instead, you'd give the DBA team the requirements and they would give you a solution.
- xiaodai 6y ago
  bike-shedding at work. Everyone "understands" language but not too many DB systems.
- np_tedious 6y ago
  For all their faults / gotchas / overhype, I found the NoSQL wave of 5-10 years ago made these conversions a bit more common.
- mbrodersen 6y ago
  > Engineering teams will always say they make sure to use the right tool for the job Yep they sure say that. While opening a tool box with very few tools in it. Sometimes only one.
slifin 6y ago
I do wonder how databases like Datomic, and Crux are perceived (if at all) in the wider database community
- dgb23 6y ago
  I'm frankly shocked by how well these Clojure Datalog DBs work. Both the data-modelling side (datoms) and the query side are extremely expressive and well integrated into Clojure itself as well. The principles are (from my perspective): How would a database (API) look like when both data changes over time and structural flexibility were first class concepts. The result is almost mindbogglingly simple. Datalog is already such a powerful language in and of itself and I really wonder why it is still such a niche thing. Don't get me wrong. SQL is great. It is there for a reason. And it is and should be "the default" for most use-cases. But Clojure Datalog DBs have a fairly clear and large advantage, especially when data is temporal and/or graph-like.
andrewstuart 6y ago
I've found databases fascinating and tried various DB's as they come out. I always find some issue or caveat or problem and I decide in the end that Postgres gets most of the way there anyway and I return to Postgres. Whenever I get tempted by a shiny new database I remind myself "don't bet against Postgres".
- akulkarni 6y ago
  +1000 Postgres. This is exactly why we chose to build TimescaleDB on top of PostgreSQL. Reliability, JSON, mature tooling, tons of connectors.
- gen220 6y ago
  This is why I’m inherently skeptical of any database technology that isn't built on top of Postgres :) PG has mostly skipped the whole time series database trend until Timescale showed up. Still waiting waiting for the graph-db and git-db extensions! I think there’s been so much progress at the lowest level, that new SQL-databases might be kind of foolhardy to not build on top of PG at this point, especially if their main claim to fame is essentially a data type and some fancy indexing. I hope PG ends up more like Linux, and less like C++ or Java. :)
StavrosK 6y ago
This is only tangentially related, but I rediscovered an old project of mine from years ago today and am rather excited about it: https://github.com/skorokithakis/goatfish https://github.com/skorokithakis/goatfish It's basically a 200 line document database in Python that's backed by SQLite. I need to store a bunch of scraping data from a script but don't want a huge database or the hassle of making SQLite tables. Goatfish is perfect because it stores free-form objects (JSON, basically), only it lets you index by arbitrary keys in them and get fast queries that way. It's pretty simple, but a very nice compromise between the simplicity of in-memory dicts and the reliability of SQLite.
- SahAssar 6y ago
  These days I'm guessing it's better to just use sqlites builtin json support and expression indexes, right?
  - StavrosK 6y ago
    Probably so, which means I should adapt Goatfish to work as a layer over that, since its API is much nicer than raw SQL.
tehlike 6y ago
Mandatory mention: RavenDB. Probably the only LINQ native database with lots of performance optimizations squeezed in.
imglorp 6y ago
I'll just throw a note for a new product, AWS's QLDB. It's an internal managed product that combines a replicated, immutable, versioned document database with ACID transactions and an immutable, provable history of every modification. There's some streaming and subset SQL on the back end. Something this focused should have a few applications where bit level auditability matters, eg financial, chain of events, etc. Of course it comes with some tradeoffs vs a relational or kv db. I wonder if there would be room for a self-hosted clone?
lima 6y ago
ClickHouse also supports incremental streaming from Kafka into a materialized view. You can even detach and reattach the view from its backing table.
leetrout 6y ago
I miss rethinkdb. I loved their approach and their tooling.
- biggestlou 6y ago
  Yeah, very cool DB. I even considered doing a chapter on it in the next Seven Databases in Seven Weeks book but alas...
grizzles 6y ago
Software is advancing so fast. Interesting to constantly reconsider the things I consider myself ahead of the curve on vs behind the curve on. Prisma looks great so I've updated my I want functional dbs, not ORMs post: https://github.com/ericbets/erics-designs/blob/master/funcdb.md https://github.com/ericbets/erics-designs/blob/master/funcdb...
- kiwicopple 6y ago
  From your blog post and the linked article: > making a database look like an rpc api I'd recommend checking out PostgREST for this (if you're using Postgres). We used this approach in my previous startup quite successfully. We also have plans at Supabase[1] to make Postgres databases semi-ephemeral. You'll be able to spin them up/down using a restful API. The schemas/types will be accessible via a restful API: https://supabase.github.io/pg-api/ https://supabase.github.io/pg-api/. Not quite as simple as SQLite, but a good alternative. Databases are cool. [1] https://supabase.io https://supabase.io
coolleo 6y ago
What is the best source to create your own database, just for learning purposes ? Great resources will be appreciated. Thank you.
- lhdj 6y ago
  https://github.com/danistefanovic/build-your-own-x#build-your-own-database https://github.com/danistefanovic/build-your-own-x#build-you...
matlin 6y ago
I support FoundationDB's approach to databases which is basically provide a consistent, distributed, and ordered Key-Value store then you can build whatever type of database you need on top of it whether that's RDBMS, Document, Graph, etc. With that said, CouchDB 4.0 (on FDB) is going to be killer. Master-Master replication between clients and server with PouchDB is phenomenal when you remove the complicated eventual consistency on the server side. And as a plug, I'm building a multi-tenant/multi-application database on top of it.
statictype 6y ago
>What I’m really hoping for is the emergence of extremely “hackable,” resolutely non-monolithic DBs that provide a plugin interface for highly use-case-specific data types, Isn't this basically what FoundationDB is?
arauhala 6y ago
Hi Luc, What's your perspective on predictive databases like https://aito.ai https://aito.ai? I'm one of the Aito.ai founders. If you would like to hear more, I'm happy to talk one-to-one. Regards, Antti
sudhirj 6y ago
I'm working on Redis adapter for DynamoDB - Dynamo is really a distributed superset of Redis, and most of the data structures that Redis has scale effectively to the distributed hash table + B-Tree-like system that Dynamo offers. Having a well known and understood API like Redis is a boon for Dynamo, whose API is much more low level and esoteric. The Go library is in beta, working on a server that's wire compatible with Redis. https://dbproject.red https://dbproject.red https://github.com/dbProjectRED/redimo.go https://github.com/dbProjectRED/redimo.go
- tmzt 6y ago
  Will you be able to support Redis-native replication from DynamoDB?
  - sudhirj 6y ago
    It would be super complicated, but technically feasible. DynamoDB has a change log called streams that would technically work, with a lot of aggregation, even with the distributed nature. But I would think about it only as a late stage and very expensive feature.
barking 6y ago
I really wish one of the existing db technologies, Firebird, got a shot in the arm. It has both embedded and server modes which makes it unique as far as I know. Also the database is a single file which with firebirds "careful write" methodology remains consistent at all times so while you can make a backup at any time because it has MVCC, even a file copy of the database file with open transactions should not be corrupted. The installer size comes in under 10 MB. It's being actively improved, is open source with a very liberal licence but sadly it only gets a tiny fraction of the attention that SQLite, postgres etc receive
pachico 6y ago
> What I have yet to see but always secretly wanted, however, is a database that natively supports incremental updates to materialized views. Yep, that’s right: Materialize listens for changes in the data sources that you specify and updates your views as those sources change. This is precisely one of the features that make ClickHouse shine
gigatexal 6y ago
whatever became of rethinkDB -- i think it was just too far ahead of its time it; had some really interesting ideas
monksy 6y ago
I love these kinds of posts. They're targeted towards what people are finding interesting and they're highly tech related. It's a great way to find new technology.

473 points by biggestlou 6y ago

tourist_on_road 6y ago

How does tiledb compares to something similar like milvus

biggestlou 6y ago
I haven't come across Milvus but I'll check it out!

hn_check 6y ago

"What I have yet to see but always secretly wanted, however, is a database that natively supports incremental updates to materialized views" SQL Server ala 10+ years ago enters the discussion.

biggestlou 6y ago
That's good to know! SQL Server has really never been on my radar but I think it's a good time to do some exploration.
liprais 6y ago
Teradata does that too
jiggawatts 6y ago
I've actually used materialized views in production on SQL Server, and they're great when they work, but they have far too many limitations. Most of these make sense, but some of the join limitations are internal constraints of the engine, not a fundamental limit. This prevents their use for a wide range of scenarios where they would be useful...
garyclarke27 6y ago
When I tried to use auto incremental materialized views in SQL Server (about 10 years ago) - they had so many limitations that they were virtually useless. I now use Postgres and this is the Number 1 feature that’s missing for me, there have been a few attempts that have unfortunately not made it into the core. I don’t understand why this critical capability is not even on the radar of the core developers - maybe this publicity from Materialize will raise its priority.

remorses 6y ago

I think that the graphql adapters like hasura and goke are also an important innovation, for small mvp projects you can create a graphql api to query your database directly from the frontend, this reduces the development time by a factor of 2 at least.

biggestlou 6y ago
Hmmm. I'm not finding Goke anywhere. Could you provide a link?

rst 6y ago

Materialize is neat, but there are other database systems that refresh at least some materialized views on the fly, while being smart about not rebuilding the entire view every time. See for example, Oracle, where FAST REFRESH ON COMMIT does most of what Materialize is advertised as doing, at least for views which that feature can support (restriction list here: https://stackoverflow.com/questions/49578932/materialized-view-in-oracle-with-fast-refresh-instead-of-complete-dosnt-work https://stackoverflow.com/questions/49578932/materialized-vi... ). Mind you, this comes with Oracle's extremely hefty price tag, so I'm not sure I'd recommend it to anyone who isn't already stuck with Oracle, but it is technical precedent. It would be interesting to compare notes, and see what Materialize does better.

deleted 6y ago
[deleted]
willvarfar 6y ago
Similar to the very cool TimescaleDB’s Continuous Aggregates (and InfluxDB’s not so cool Continous Queries). There’s also ksqlDB that can be used in a similar way?
atwebb 6y ago
Indexed views in SQL server behave similarly (though not as performant as Oracle I believe). They also have some constraints on application (schema bound, data types, nesting).
dkhenry 6y ago
About a year ago Vitess merged in support for what we call VReplication. ( https://vitess.io/docs/reference/features/vreplication/ https://vitess.io/docs/reference/features/vreplication/ ) The initial demo was a custom Materialized view that did calculations and rollup's on data in realish time ( it uses the MySQL binlog ). Right now the limiting factor for us calling is true Materalized View's is that configuring them is done out of band using the Vitess Control plane, but soon we will be merging in the features to allow standard DDL to configure and manage them!
jacques_chester 6y ago

davedx 6y ago

Is tiledb useful for storing ML models?

Shelnutt2 6y ago
You can certainly store ML models in TileDB. I've been working on an example notebook showing how to store an opencv model in TileDB, its not quite finished yet but will be published soon. Using python as an example language, TileDB-Py offers simple functions, like "from_numpy" [1] for basic and simple models which we can write directly. For more advanced (and more common) use cases of complex models, its easy to create an array to store the model and all associated metadata. TileDB's array metadata and time travel capabilities allow for you to even see how your model changes over time as you update it. [1] https://github.com/TileDB-Inc/TileDB-Py/blob/bcaee16b194675f10235669fcb615928c0af964d/tiledb/libtiledb.pyx#L3974 https://github.com/TileDB-Inc/TileDB-Py/blob/bcaee16b194675f... Disclosure: I'm a member of the TileDB team.
- davedx 6y ago
  Really cool, thanks for the reply.

mywacaday 6y ago

Does anybody know of a good educational resource on software/best practices that is kept up to date. Ideally something that does not include the latest bleeding edge but things that are battle hardened or getting there. Something that includes open source and commercial software would be ideal.

melvinroest 6y ago

The website didn't load for me. So here it is: https://web.archive.org/web/20200615193041/https://lucperkins.dev/blog/new-db-tech-1/ https://web.archive.org/web/20200615193041/https://lucperkin... Also, I'd like to add one database to the list (I work there for 3 weeks now): TriplyDB [0]. It is making linked data easier. Linked data is useful for when people of different organizations want a shared schema. In many commercial applications one wouldn't want this, as data is the valuable part of a company. However, scientific communities, certain government agencies and other organizations -- that I don't yet know about -- do want this. I think the coolest application of linked data is how the bio-informatics/biology community utilizes it [1, 2]. The reason I found out at all is because one person at Triply works to see if a similar thing can be achieved with psychology. It might make conducting meta-studies a bit easier. I read the HN discussions on linked data and agree with both the nay sayers (it's awkward and too idealistic [4]) and the yay sayers (it's awesome). The thing is: 1. Linked data open, open as in open source, the URI [3] is baked into its design. 2. While the 'API'/triple/RDF format can be awkward, anyone can quite easily understand it. The cool thing is: this includes non-programmers. 3. It's geared towards collaboration. In fact, when reading between the lines, I'd argue it's really good for collaboration between a big heterogeneous group of people. Disclaimer: this is my own opinion, Triply does not know I'm posting this and I don't care ;-) I simply think it's an interesting way of thinking about data. [0] triply.cc [1] A friend of mine once modeled some biochemistry part of C. Elegans from linked data into petrinets: https://www.researchgate.net/publication/263520722_Building_Executable_Biological_Pathway_Models_Automatically_from_BioPAX https://www.researchgate.net/publication/263520722_Building_... [2] https://www.google.com/search?client=safari&rls=en&q=linked+data+and+biology&ie=UTF-8&oe=UTF-8 https://www.google.com/search?client=safari&rls=en&q=linked+... -- I quickly vetted this search [3] I still don't know the difference between a URI and URL. [4] I think back in the day, linked data idealists would say that all data should be linked to interconnect all the knowledge. I'm more pragmatic and simply wonder: in which socio-technological context is linked data simply more useful than other formats? My current very tentative answer is those 3 points.

iaabtpbtpnn 6y ago
I recently had to deal with some RDF data expressed as N-triples. So, naturally I loaded it into a proper triplestore and embraced the whole W3C RDF universe and started teaching myself SPARQL, right? Nah, instead I just translated that shit into CSV and loaded it into Postgres, then post-processed it into a relational schema that I can actually understand and query with a language that other people at my company also know. The RDF was just a poorly specified way of communicating that schema to me, along with a data format that's no better than a 3-column CSV. Great stuff from the Semantic Web folks here, real powerful technology. Edit: Also, to answer your question, the difference between a URL (Uniform Resource Locator) and a URI (Uniform Resource Identifier) is that the URL actually points to something, an object at a particular location, and you can paste it into your web browser to view that something. A URI just uses a URL-like scheme to represent identifiers, such that your domain and directory structure provide a kind of namespace that you control. But as long as it follows the format, your URI can contain literally anything, it doesn't have to be human-readable or resolve to anything in a web browser. It might as well be mycompany_1231241542345.

petercooper 6y ago

I edit a database newsletter – https://dbweekly.com/ https://dbweekly.com/ – so tend to always have my eyes out for new releases, what's coming along, and what not. And I thought I'd share a few more things that have jumped out at me recently in case anyone's in the mood for spelunking. 1. QuestDB – https://questdb.io/ https://questdb.io/ – is a performance-focused, open-source time-series database that uses SQL. It makes heavy use of SIMD and vectorization for the performance end of things. 2. GridDB - https://griddb.net/en/ https://griddb.net/en/ - is an in-memory NoSQL time-series database (there's a theme lately with these!) out of Toshiba that was boasting doing 5m writes per second and 60m reads per second on a 20 node cluster recently. 3. MeiliSearch - https://github.com/meilisearch/MeiliSearch https://github.com/meilisearch/MeiliSearch – not exactly a database but basically an Elastic-esque search server written in Rust. Seems to have really taken off. 4. Dolt – https://github.com/liquidata-inc/dolt https://github.com/liquidata-inc/dolt – bills itself as a 'Git for data'. It's relational, speaks SQL, but has version control on everything. TerminusDB, KVRocks, and ImmuDB also get honorable mentions. InfoWorld also had an article recently about 9 'offbeat' databases to check out if you want to go even further: https://www.infoworld.com/article/3533410/9-offbeat-databases-worth-a-look.html https://www.infoworld.com/article/3533410/9-offbeat-database... Exciting times in the database space!

rapnie 6y ago
TerminusDB recently featured on HN. Also a git-for-data approach, time-travel, storing data as RDF and queries return JSON-LD. https://news.ycombinator.com/item?id=22867767 https://news.ycombinator.com/item?id=22867767
- LukeEF 6y ago
  Yes, the latest version of Terminus uses JSON-LD, which is a serialisation of RDF. This enables us to store queries in a database and provides a schema documentation for our datalog query language. Release notes: https://github.com/terminusdb/terminusdb-server/blob/master/RELEASE_NOTES.md https://github.com/terminusdb/terminusdb-server/blob/master/...
heipei 6y ago

willvarfar 6y ago

My understanding of TileDB is that it is 100% client-side. There is no server. In a sense it’s like handling orc or paraquet or even SQLite files on S3, (except tiledb are fancy r-trees) with a delta-lake-like manifest file for transactions too. I think in the future there’s going to be a sine-wave of smart-clients consuming S3 cleverly, and then smartness growing in the S3 interface so constraints and indices and things happen in the storage again, and back and forth...

deleted 6y ago
[deleted]
jakebol 6y ago
This is a good description, except that TileDB (the open source client) is not transactional but eventually consistent at least for S3 and other object stores. I like your point about consuming S3 cleverly, it's often difficult to get good out of the box performance from S3 so abstracting that to the degree possible is good for end-users. The cloud vendors though are always one or two steps ahead of companies that build upon their services. AWS Redshift for instance already can pre-index objects stored on S3 to accelerate queries at the storage layer. It's difficult as a third party vendor to compete with that.
- biggestlou 6y ago
  This is a very interesting development that I'd like to learn more about. Whenever I've played around with writing databases (just as toy projects) I've always done so using RocksDB or something similar as a backend. This "thick client" model, though, seems to have a lot of potential benefits, most notably no need to worry about disk space or volumes (so say goodbye to a bunch of config parameters) and no need for a tiered storage setup or S3 migration tools (already accomplished!). Not ideal for most use cases but intriguing for some!
  - jakebol 6y ago
    There are a lot of issues though with S3, latency, poor performance for small reads / writes, timeouts, api rate limits, api costs, and consistency issues poorly understood by third party developers. A "thick-client" also doesn't perform well unless that client is located on a node in the same region. I think as with everything it works well in some cases and not well in others.

xchaotic 6y ago

Interesting notes but I feel like the db itself has been commoditised and the battle is elsewhere now. So anyone building a database engine today, will find out that to make it sustainable they also need an ecosystem on top of it, tooling, community, paid support, active devs, consultants (for which they may have no runway) Finally I find anything that calls itself a database and uses S3 as a backend a bit ridiculous. S3 has eventual consistency so you can’t do the operations that differentiate a database from a file system.

moonchild 6y ago

> And it’s worth noting that many have tried to do what Prisma does and failed because they. Because they what?

biggestlou 6y ago
Derp! Fixed that.

biggestlou 6y ago

REMOVED: I complained about the rewritten title in a way that was excessively harsh and have removed that comment.

dang 6y ago
The article title was linkbait: "you" and "should" are linkbait tropes, especially when combined. The site guidelines call for rewriting such titles. See https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html. When we change titles, we always try to use representative language from the article. I replaced it with a phrase from the article where it says what it's about in a neutral way. I added a hyphen to "recently minted" because it seems to me that grammar requires it. However, if we worship different grammar gods I am happy to let yours hold sway. The hyphen is now toast. This is bog-standard HN moderation. I dare say the reason you "really really hate it" is probably because you notice the cases you dislike and weight them much more heavily: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20notice%20dislike&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que.... Meanwhile the ones you don't find objectionable pass over relatively unnoticed. That is bog-standard HN perception bias. If we didn't meticulously try to keep titles linkbait-free and less-misleading (more hyphens, sorry!), HN's front page would be completely different, and since the front page is the most important thing here, the entire forum would soon be completely different. That requires editing titles all the time. petercooper, who posted elsewhere in this thread, has an HN title-change tracker (oops, I hyphened again!) which is quite nifty. It doesn't show all of them, though, and it can't tell the difference between moderator edits and submitter edits. There is a power shortage in my neighborhood and my laptop battery is about to expire, so if you don't see a link momentarily, or if I don't respond anywhere on the site for a few hours, it's because shelter-in-place (uh-oh!) prevents me from finding somewhere else to plug in. Edit: here it is: https://news.ycombinator.com/item?id=21617016 https://news.ycombinator.com/item?id=21617016. Note how we edited out the bloody pig mask.
- biggestlou 6y ago
  I apologize for the strength of my language. There are several words I used in that comment that I should not have. It's far too easy to forget in the moment that there are real people making these decisions who may very well read what you say. I understand where you're coming from with the decision and will try to select more appropriate titles in the future. And will be much more mindful about how I register complaints.

chickenpotpie 6y ago

I’ve always felt it strange that in almost every job I’ve had databases have been one of the most important pieces of the architecture but the least debated. I’ve spent hours debating languages and frameworks, but databases always come down to whatever we have a license for/what others at the company are using. Engineering teams will always say they make sure to use the right tool for the job, but no one ever talks about if it’s right to keep using the same database for a new product.

mywittyname 6y ago
My experience is places with DBAs tend to keep very tight control over database design than those who do without. I've worked at several places where developers did not touch the database schema. They couldn't even propose changes, instead, you'd give the DBA team the requirements and they would give you a solution.
xiaodai 6y ago
bike-shedding at work. Everyone "understands" language but not too many DB systems.
np_tedious 6y ago
For all their faults / gotchas / overhype, I found the NoSQL wave of 5-10 years ago made these conversions a bit more common.
mbrodersen 6y ago
> Engineering teams will always say they make sure to use the right tool for the job Yep they sure say that. While opening a tool box with very few tools in it. Sometimes only one.

slifin 6y ago

I do wonder how databases like Datomic, and Crux are perceived (if at all) in the wider database community

dgb23 6y ago
I'm frankly shocked by how well these Clojure Datalog DBs work. Both the data-modelling side (datoms) and the query side are extremely expressive and well integrated into Clojure itself as well. The principles are (from my perspective): How would a database (API) look like when both data changes over time and structural flexibility were first class concepts. The result is almost mindbogglingly simple. Datalog is already such a powerful language in and of itself and I really wonder why it is still such a niche thing. Don't get me wrong. SQL is great. It is there for a reason. And it is and should be "the default" for most use-cases. But Clojure Datalog DBs have a fairly clear and large advantage, especially when data is temporal and/or graph-like.

andrewstuart 6y ago

I've found databases fascinating and tried various DB's as they come out. I always find some issue or caveat or problem and I decide in the end that Postgres gets most of the way there anyway and I return to Postgres. Whenever I get tempted by a shiny new database I remind myself "don't bet against Postgres".

akulkarni 6y ago
+1000 Postgres. This is exactly why we chose to build TimescaleDB on top of PostgreSQL. Reliability, JSON, mature tooling, tons of connectors.
gen220 6y ago
This is why I’m inherently skeptical of any database technology that isn't built on top of Postgres :) PG has mostly skipped the whole time series database trend until Timescale showed up. Still waiting waiting for the graph-db and git-db extensions! I think there’s been so much progress at the lowest level, that new SQL-databases might be kind of foolhardy to not build on top of PG at this point, especially if their main claim to fame is essentially a data type and some fancy indexing. I hope PG ends up more like Linux, and less like C++ or Java. :)

StavrosK 6y ago

This is only tangentially related, but I rediscovered an old project of mine from years ago today and am rather excited about it: https://github.com/skorokithakis/goatfish https://github.com/skorokithakis/goatfish It's basically a 200 line document database in Python that's backed by SQLite. I need to store a bunch of scraping data from a script but don't want a huge database or the hassle of making SQLite tables. Goatfish is perfect because it stores free-form objects (JSON, basically), only it lets you index by arbitrary keys in them and get fast queries that way. It's pretty simple, but a very nice compromise between the simplicity of in-memory dicts and the reliability of SQLite.

SahAssar 6y ago
These days I'm guessing it's better to just use sqlites builtin json support and expression indexes, right?
- StavrosK 6y ago
  Probably so, which means I should adapt Goatfish to work as a layer over that, since its API is much nicer than raw SQL.

tehlike 6y ago

Mandatory mention: RavenDB. Probably the only LINQ native database with lots of performance optimizations squeezed in.

imglorp 6y ago

I'll just throw a note for a new product, AWS's QLDB. It's an internal managed product that combines a replicated, immutable, versioned document database with ACID transactions and an immutable, provable history of every modification. There's some streaming and subset SQL on the back end. Something this focused should have a few applications where bit level auditability matters, eg financial, chain of events, etc. Of course it comes with some tradeoffs vs a relational or kv db. I wonder if there would be room for a self-hosted clone?

lima 6y ago

ClickHouse also supports incremental streaming from Kafka into a materialized view. You can even detach and reattach the view from its backing table.

leetrout 6y ago

I miss rethinkdb. I loved their approach and their tooling.

biggestlou 6y ago
Yeah, very cool DB. I even considered doing a chapter on it in the next Seven Databases in Seven Weeks book but alas...

grizzles 6y ago

Software is advancing so fast. Interesting to constantly reconsider the things I consider myself ahead of the curve on vs behind the curve on. Prisma looks great so I've updated my I want functional dbs, not ORMs post: https://github.com/ericbets/erics-designs/blob/master/funcdb.md https://github.com/ericbets/erics-designs/blob/master/funcdb...

kiwicopple 6y ago
From your blog post and the linked article: > making a database look like an rpc api I'd recommend checking out PostgREST for this (if you're using Postgres). We used this approach in my previous startup quite successfully. We also have plans at Supabase[1] to make Postgres databases semi-ephemeral. You'll be able to spin them up/down using a restful API. The schemas/types will be accessible via a restful API: https://supabase.github.io/pg-api/ https://supabase.github.io/pg-api/. Not quite as simple as SQLite, but a good alternative. Databases are cool. [1] https://supabase.io https://supabase.io

coolleo 6y ago

What is the best source to create your own database, just for learning purposes ? Great resources will be appreciated. Thank you.

lhdj 6y ago
https://github.com/danistefanovic/build-your-own-x#build-your-own-database https://github.com/danistefanovic/build-your-own-x#build-you...

matlin 6y ago

I support FoundationDB's approach to databases which is basically provide a consistent, distributed, and ordered Key-Value store then you can build whatever type of database you need on top of it whether that's RDBMS, Document, Graph, etc. With that said, CouchDB 4.0 (on FDB) is going to be killer. Master-Master replication between clients and server with PouchDB is phenomenal when you remove the complicated eventual consistency on the server side. And as a plug, I'm building a multi-tenant/multi-application database on top of it.

statictype 6y ago

>What I’m really hoping for is the emergence of extremely “hackable,” resolutely non-monolithic DBs that provide a plugin interface for highly use-case-specific data types, Isn't this basically what FoundationDB is?

arauhala 6y ago

Hi Luc, What's your perspective on predictive databases like https://aito.ai https://aito.ai? I'm one of the Aito.ai founders. If you would like to hear more, I'm happy to talk one-to-one. Regards, Antti

sudhirj 6y ago

I'm working on Redis adapter for DynamoDB - Dynamo is really a distributed superset of Redis, and most of the data structures that Redis has scale effectively to the distributed hash table + B-Tree-like system that Dynamo offers. Having a well known and understood API like Redis is a boon for Dynamo, whose API is much more low level and esoteric. The Go library is in beta, working on a server that's wire compatible with Redis. https://dbproject.red https://dbproject.red https://github.com/dbProjectRED/redimo.go https://github.com/dbProjectRED/redimo.go

tmzt 6y ago
Will you be able to support Redis-native replication from DynamoDB?
- sudhirj 6y ago
  It would be super complicated, but technically feasible. DynamoDB has a change log called streams that would technically work, with a lot of aggregation, even with the distributed nature. But I would think about it only as a late stage and very expensive feature.

barking 6y ago

I really wish one of the existing db technologies, Firebird, got a shot in the arm. It has both embedded and server modes which makes it unique as far as I know. Also the database is a single file which with firebirds "careful write" methodology remains consistent at all times so while you can make a backup at any time because it has MVCC, even a file copy of the database file with open transactions should not be corrupted. The installer size comes in under 10 MB. It's being actively improved, is open source with a very liberal licence but sadly it only gets a tiny fraction of the attention that SQLite, postgres etc receive

pachico 6y ago

> What I have yet to see but always secretly wanted, however, is a database that natively supports incremental updates to materialized views. Yep, that’s right: Materialize listens for changes in the data sources that you specify and updates your views as those sources change. This is precisely one of the features that make ClickHouse shine

gigatexal 6y ago

whatever became of rethinkDB -- i think it was just too far ahead of its time it; had some really interesting ideas

monksy 6y ago

I love these kinds of posts. They're targeted towards what people are finding interesting and they're highly tech related. It's a great way to find new technology.