6 ms·

Reading about databases always makes me sad about the enterprise database I work with.

• minimaxir 10y ago
  Granted, this is in-memory computing and not intended for persistent storage. There's nothing stopping you from using Spark's SQL drivers (which support ODBC) to load things into memory, unless your database does not support it.
  • threeseed 10y ago
    They support JDBC out of the box. ODBC works via a JDBC bridge which though they are often commercial and vary in quality.
• PeCaN 10y ago
  You can read about the awesome theory behind relational algebra. Gave me a great appreciation for how awesome SQL databases actually are (and why they work the way they do).
  • 50CNT 10y ago
    Any good books on that?
    • ddispaltro 10y ago
      This java project gives you a glimpse at how it all works. It was extracted out of a now defunkt db vendor (I think?). https://calcite.apache.org/ https://calcite.apache.org/
      • Jach 10y ago
        Yeah, it came out of the Eigenbase project. (http://luciddb.sourceforge.net/ http://luciddb.sourceforge.net/ and http://www.eigenbase.org http://www.eigenbase.org). DB internals are indeed quite fun. (I worked on LucidDB(https://github.com/LucidDB/luciddb https://github.com/LucidDB/luciddb) for a time.)
    • thallian 10y ago
      I really liked 'Database in Depth' by C.J. Date.
    • rusanu 10y ago
      Transaction Processing: Concepts and Techniques [0] by Jim Gray and Andreas Reuter. Not so much relational algebra, but everything you need to know to implement a classic relational engine. [0] https://www.amazon.com/Transaction-Processing-Concepts-Techniques-Management-ebook/dp/B016W7HLX8 https://www.amazon.com/Transaction-Processing-Concepts-Techn...
    • pramodliv1 10y ago
      The original paper[0] on relational databases by Edgar Codd is quite accessible. I also loved the MOOC by Prof. Jennifer Widom[1] [0] https://www.seas.upenn.edu/~zives/03f/cis550/codd.pdf https://www.seas.upenn.edu/~zives/03f/cis550/codd.pdf [1] http://online.stanford.edu/course/databases-self-paced http://online.stanford.edu/course/databases-self-paced
    • ta0967 10y ago
      anything from C. J. Date
    • morgante 10y ago
      In addition to the great links from siblings, it might be interesting to read this Stonebraker paper on the cyclical nature of databases and how, ultimately, they all reduce to a few core paradigms. [0] [0] http://pages.cs.wisc.edu/~anhai/courses/764-sp07-anhai/datamodel.pdf http://pages.cs.wisc.edu/~anhai/courses/764-sp07-anhai/datam...
  • _RPM 10y ago
    Databases are one of the most fun computer science topics. And I don't mean _databases_ from the perspective of an admin or even a developer, but a database developer of the internals. I love trying to read the code of complex systems like databases or language compilers.
    • stingraycharles 10y ago
      Do you have some examples of interesting code ? I'm curious now!
      • dataminer 10y ago
        I find postgres source code very interesting, e.g https://github.com/postgres/postgres/tree/master/src/backend/optimizer https://github.com/postgres/postgres/tree/master/src/backend... Sqlite is another good one to read, and play around with.
      • dataminer 10y ago
        You may also find this paper interesting http://wwwlgis.informatik.uni-kl.de/archiv/wwwdvs.informatik.uni-kl.de/courses/DBSREAL/SS2005/Vorlesungsunterlagen/Implementing_Sorting.pdf http://wwwlgis.informatik.uni-kl.de/archiv/wwwdvs.informatik...
      • _RPM 10y ago
        CPython is an interesting read. The internal type system (PyObject) is pretty cool. It also uses type pruining instead of a discriminated union.
• jkot 10y ago
  Spark is not a database ;-)
  • coredog64 10y ago
    Would you please tell that to the architects I work with? We're reinventing relational databases over the top of Spark.
    • dman 10y ago
      Post their linkedin profiles, I would be happy to drop in a word :)
  • threeseed 10y ago
    But it is a SQL parser and execution engine. You could easily turn it into a database by managing a set of flat files.
• AndyNemmity 10y ago
  I work with SAP HANA as an enterprise database, and it works quite well with Spark, has code push down, query compilation, all sorts of fun stuff. Depends on the enterprise database, is my point.
  • mbesto 10y ago
    Where do you delineate workloads between HANA and Spark? (I'm just curious...used to work the internal SAP HANA dev team)