5 ms·

Show HN: A Database Written in Golang

277 points by Sahil121 2y ago

Recently created a minimal persistent relational database in Go. Main focus was on implementing & understanding working the of database, storage management & transaction handling. Use of B+ Tree for storage engine(support for indexing), managing a Free List (for reusing nodes), Support for transactions, Concurrent Reads. Still have many things to add & fix like query processing being one of the main & fixing some bugs Repo link - https://github.com/Sahilb315/AtomixDB https://github.com/Sahilb315/AtomixDB Would love to hear your thoughts

deleted 2y ago
[deleted]
pighive 2y ago
Nice, did you follow a course or any resources to build it? Please share, I have a similar goal. Thank you.
- rebelmackerel 2y ago
  I took a quick glance at the code, I believe it may build upon "Build Your Own Database From Scratch in Go" [0]. The first part of the book is available for free on the author's website, along with information on how to purchase the full book (which includes source code). I share the same goal and am working through the material after working through Codecrafters' "Build your own SQLite" [1]. Good luck! I apologize in advance for mistakes (formatting, et cetera). I just registered this account to point you toward resources I found helpful. [0] https://build-your-own.org/database/ https://build-your-own.org/database/ [1] https://app.codecrafters.io/courses/sqlite/overview https://app.codecrafters.io/courses/sqlite/overview
  - croskcool 2y ago
    Does something like this exist for java, don't want to take the burden of a new language.
    - rebelmackerel 2y ago
      I don't think these things really fit the bill, but this is closest I could find. All the codecrafters stuff is focused on reading from a sqlite DB. A great start, but you're not implementing B+ trees. Hope this gives you some options. https://github.com/Arminas42/build_your_own_database https://github.com/Arminas42/build_your_own_database https://github.com/search?q=codecrafters+sqlite-java+language%3AJava&type=repositories https://github.com/search?q=codecrafters+sqlite-java+languag...
    - lionkor 2y ago
      it's not that hard, you should try it :)
whalesalad 2y ago
How do you connect to it and actually use it? Does it behave like a SQL database? Row-based? Column-based? Good for analytical workloads? Document store? Redis/memcached database? What are it's strengths or weaknesses? Cool accomplishment in and of itself but hard for anyone here to really give you any criticism or feedback without understanding where it excels and how to work with it.
- zsoltkacsandi 2y ago
  > Main focus was on implementing & understanding working the of database I think this clearly describes what was the goal.
  - whalesalad 2y ago
    Yep... but there are dozens of different types of databases out there. There is no way to look at this codebase and give any kind of feedback one way or the other without that understanding, particularly with zero usage examples. Which is what the OP is asking about.
    - jasonlotito 2y ago
      > There is no way to look at this codebase and give any kind of feedback Less than a minute and I know how to use it. It's not complicated. The source code is available and fairly easy. If you can't figure out how to use it in a trivial amount of time, you aren't going to be able to offer anything of value. When did HN go from being about interesting stuff to making bold, ignorant statements like "There is no way to look at this codebase and give any kind of feedback one way or the other without that understanding." You should know better.
      - whalesalad 2y ago
        [flagged]
        2y ago
esafak 2y ago
[flagged]
- anonzzzies 2y ago
  "Main focus was on implementing & understanding working the of database, storage management & transaction handling." Obviously OP wrote it to understand database theory & implementation.
  - deleted 2y ago
    [deleted]
- ajsnigrutin 2y ago
  It's hackernews, basically every old thing has to be reinvented in the "language of the week", be it go, rust, ruby, or whatever was before that. From a learning perspective, nice project for OP, for 'advertising' it, i'd prefer the "what's better than the alternatives" instead of focusing on the language.
  - johnisgood 2y ago
    I want more Ada / SPARK and Factor. :D
- pjmlp 2y ago
  That a database written in an AOT compiled managed language is possible, for example. Specially relevant, because it used to be that writing database engines used to be considered systems programming, and we all know managed languages cannot be possible used for such tasks. /s
- deleted 2y ago
  [deleted]
mbreese 2y ago
This looks like a good exploratory project! One thing I’d add to the readme is an example of how you’d use the database in an example application. From the docs, it’s clear that this isn’t a sql database (yet?), so it would be good to have an example to see how to use the database. It might also be nice to have a description of what happens when you insert or get a record, so others can learn from the code too. Or can you comment here about what your favorite part of the code is? What did you figure out that you didn’t know before? If you’re using the project to learn about databases, what have you learned so far?
- Sahil121 2y ago
  thanks man yes currently it is not a fully sql database (have to add query support) My fav part of code is the data retrieval code because had some issues there & got to know lot about it Learnt about B+ Trees, transactions, managing concurrent reads & data presistence
littlemerman 2y ago
You might be interested in this golang db from mit’s database systems course: https://github.com/MIT-DB-Class/go-db-2024 https://github.com/MIT-DB-Class/go-db-2024 https://dsg.csail.mit.edu/6.5830/ https://dsg.csail.mit.edu/6.5830/
- Sahil121 2y ago
  will surely checkout thanks man
eatonphil 2y ago
If folks would like to see more examples of databases built to teach oneself, they get shared on the /r/databasedevelopment subreddit not infrequently. Some recent ones: https://www.reddit.com/r/databasedevelopment/comments/1hyig8k/my_very_own_toy_database/ https://www.reddit.com/r/databasedevelopment/comments/1hyig8... https://www.reddit.com/r/databasedevelopment/comments/1ha5ccb/building_a_database_from_scratch_simpledb/ https://www.reddit.com/r/databasedevelopment/comments/1ha5cc... https://www.reddit.com/r/databasedevelopment/comments/1dqgmsa/simpledb_an_educational_rdbms_implemented_in_go/ https://www.reddit.com/r/databasedevelopment/comments/1dqgms... https://www.reddit.com/r/databasedevelopment/comments/1ix5dzz/built_a_database_from_scratch_in_go/ https://www.reddit.com/r/databasedevelopment/comments/1ix5dz... https://www.reddit.com/r/databasedevelopment/comments/18knodn/i_made_a_lsmbased_kv_storage_engine_in_rust_help/ https://www.reddit.com/r/databasedevelopment/comments/18knod... https://www.reddit.com/r/databasedevelopment/comments/1h3w70l/tidesdb_high_performance_transactional_durable/ https://www.reddit.com/r/databasedevelopment/comments/1h3w70... https://www.reddit.com/r/databasedevelopment/comments/1gk18nd/k4_opensource_highperformance_transactional_and/ https://www.reddit.com/r/databasedevelopment/comments/1gk18n... https://www.reddit.com/r/databasedevelopment/comments/1bemf9w/new_toy_database_to_learn_and_play_with/ https://www.reddit.com/r/databasedevelopment/comments/1bemf9... https://www.reddit.com/r/databasedevelopment/comments/1iw6cxs/canopydb_transactional_kv_store_yet_another_in/ https://www.reddit.com/r/databasedevelopment/comments/1iw6cx...
- danhau 2y ago
  Nice, thanks! I didn‘t know there was a subreddit for that.
- deleted 2y ago
  [deleted]
- seafoamteal 2y ago
  Hi Phil, I thought I'd find you here! Love your blog!
  - eatonphil 2y ago
    Glad to hear it. :)
faizhalde 2y ago
excellent, would love to hear more about resources you used while implementing
- Sahil121 2y ago
  followed this book - https://build-your-own.org/database/ https://build-your-own.org/database/
twalla 2y ago
Someone asked about resources and I ran into this while evaluating embedded db options for a golang project - it's a collection of db components implemented in golang: https://github.com/thomasjungblut/go-sstables https://github.com/thomasjungblut/go-sstables
- tjungblut 2y ago
  thank you for referring, feel free to ask any questions you may have
voodooEntity 2y ago
I didnt look into it yet but i already wanted to say - cuz of curiosity i build my own graph database in golang :) and i learned alot. so i absolutly understand why you did it and what experiences you probably made on the way :D congratz !
- Sahil121 2y ago
  thanks man
- varelaseb 2y ago
  Is this published anywhere?
  - voodooEntity 2y ago
    Kinda. So first i went for a standalone application back than to use it as storage/database. This part is archived now. it was released under: https://github.com/voodooEntity/slingshotdb https://github.com/voodooEntity/slingshotdb At this point i think its important to mention that its nativly an in-memory graph database/storage. At some point (4-5 years ago) i decided to change the way i handle it. Since im using the storage in many private projects, and there was no real traction (while also maintaining a database is quite some work) i decided i gonne change the way i handle it away from a standalone database to a library that works as an in-memory graph storage. Also i added a custom query language including builder (the queries are 100% json compatible so you can in theory build queries from whatever language and transmit it via json. The library is released under : https://github.com/voodooEntity/gits https://github.com/voodooEntity/gits Currently im completly reworking the query parser but since this is a privat project and im the only maintainer it may take some to to finish it. The update will takle some smaller bugs in the query parser that can occur on very complex nested queries and also will optimize query speeds. I cant provide a date when the update will occure i only can say it will (because i need it myself). Finally said: The strength of gits is to be very fast (hence in memory handling) and easy to use from golang. Its threadsafe by default.
bob1029 2y ago
If you are wanting proper SQL command support, you could copy the SQLite parser approach. Properly parsing all valid command texts is not a problem that I would find compelling unless I was being compensated for it. https://www.sqlite.org/lemon.html https://www.sqlite.org/lemon.html You could probably use something like participle, but you'd have to translate the grammar. https://github.com/alecthomas/participle https://github.com/alecthomas/participle
- Sahil121 2y ago
  yes man have to work on adding the sql cmd support thanks for the links
- qaq 2y ago
  You can use https://github.com/cockroachdb/cockroachdb-parser https://github.com/cockroachdb/cockroachdb-parser which is basically PGSQL compatible parser implemented in golang
- mingodad 2y ago
  I do have a big collection of LALR(1) grammars to test/study/develop/document here https://mingodad.github.io/parsertl-playground/playground/ https://mingodad.github.io/parsertl-playground/playground/ including sqlite, tidb, vites, postgresql, mysql, ...
alecco 2y ago
Cool! You watched CMU's intro to Database sytems, right? It's really good and thorough. It will save you of some common pitfalls and can help you navigate the trade-offs. https://www.youtube.com/watch?v=otE2WvX3XdQ&list=PLSE8ODhjZXjYDBpQnSymaectKjxCy6BYq https://www.youtube.com/watch?v=otE2WvX3XdQ&list=PLSE8ODhjZX...
- Sahil121 2y ago
  sure man
- hobs 2y ago
  Looks fun, but the intro requirement is "Knows C++" - that's kinda a non starter right?
  - alecco 2y ago
    I'm sure that's for the homework.
  - sieabahlpark 2y ago
    [dead]
- BiraIgnacio 2y ago
  Oh thanks for the link, this is really great!
- Tostino 2y ago
  I really have liked the current Optimizer series. So much amazing info. I hope some company that heavily relies on Postgres sponsors a project to replace the optimizer with a more modern implementation. I feel like for any large applications with olap style queries (even if on the OLTP database) will benefit incredibly from that work. SQL Servers optimizer was light-years ahead of Postgres when I worked with it last over a decade ago.
  - alecco 2y ago
    Sadly, most of those corporations would rather spend millions in proprietary solutions than give back 100k to the community. I've seen it first hand many times. They only give money if they absolutely have to.
fipar 2y ago
Congratulations on a great exploratory project! It takes me back to my school years. I never got as far as you (not by any long stretch actually), but I did enjoy creating the storage layer of a database from scratch. To actually have to deal with, instead of just think of, all the edge cases, is quite the transformative experience. As a humble suggestion, since it seems your goal is to understand how relational databases work and not necessarily to write a new database that will compete with others, maybe don't make it an SQL one? We've got enough of those, and not enough of the others. Would be nice to have a new relational DB using Tutorial D as its language for example. Keep hacking!
- Sahil121 2y ago
  thanks for the suggestion man, will definitely think about this
snissn 2y ago
to also share a reference - here's an on disk hashmap that uses mmap that I made in golang: https://github.com/snissn/gomap https://github.com/snissn/gomap
tapirl 2y ago
Cool! btw, README says Go 1.17+ is required, but go.mod says 1.23+.
- cbluth 2y ago
  You should trust that the readme is up to date
  - tapirl 2y ago
    With Go toolchain [1.17, 1.23.1], it fails to build.
hsnice16 2y ago
This is so nice! I created an email verifier project in Go to check MX records, SMTP server status, and existence in the disposable list. Project - https://github.com/hsnice16/email-verifier https://github.com/hsnice16/email-verifier
curtisszmania 2y ago
[dead]
mdwhatcott 2y ago
Best part: a go.mod with 0 dependencies!
- OccamsMirror 2y ago
  Now that is impressive. Gotta love zero dependency projects.
dveeden2 2y ago
Other databases written in Go: - TiDB by PingCAP - Vitess by PlanetScale Both are basically only the SQL part as TiDB uses TiKV (written in Rust) and Vitess uses MySQL. For those who want to implement a database in Go but without having to implement a network protocol there is go-mysql, which allows you to do this: https://github.com/go-mysql-org/go-mysql/blob/master/cmd/go-mysqlserver/main.go https://github.com/go-mysql-org/go-mysql/blob/master/cmd/go-... As demonstration I created a networked SQLite: https://github.com/dveeden/go-mysql-examples/blob/main/go-mysqlite/main.go https://github.com/dveeden/go-mysql-examples/blob/main/go-my... Both TiDB and Vitess have parsers that can be used outside standalone. So if you only want to implement your own on disk format, this can help. Note that I'm working for PingCAP on TiDB and I'm also a co-maintainer for go-mysql.
- raggi 2y ago
  Vitess came from YouTube
  - CBLT 2y ago
    Didn't the Vitess team found planetscale?
    - harshitgangal 2y ago
      Yes! The founders of PlanetScale were the co-creators of Vitess at YouTube, where it was built to handle MySQL scalability. PlanetScale builds on Vitess but offers a managed, developer-friendly experience.
- aqueueaqueue 2y ago
  We use Tidb at work at scale. Great product! Was looking at the source today to understand an error code.
0x3331 2y ago
Looks like a start. Keep it up, keep learning. Definitely check out https://www.youtube.com/c/CMUDatabaseGroup https://www.youtube.com/c/CMUDatabaseGroup Great lectures on relational databases and more.
bosky101 2y ago
Maybe a good place to ask: good sdks or libraries for reading parquet?

277 points by Sahil121 2y ago

deleted 2y ago

[deleted]

pighive 2y ago

Nice, did you follow a course or any resources to build it? Please share, I have a similar goal. Thank you.

rebelmackerel 2y ago
I took a quick glance at the code, I believe it may build upon "Build Your Own Database From Scratch in Go" [0]. The first part of the book is available for free on the author's website, along with information on how to purchase the full book (which includes source code). I share the same goal and am working through the material after working through Codecrafters' "Build your own SQLite" [1]. Good luck! I apologize in advance for mistakes (formatting, et cetera). I just registered this account to point you toward resources I found helpful. [0] https://build-your-own.org/database/ https://build-your-own.org/database/ [1] https://app.codecrafters.io/courses/sqlite/overview https://app.codecrafters.io/courses/sqlite/overview
- croskcool 2y ago
  Does something like this exist for java, don't want to take the burden of a new language.
  - rebelmackerel 2y ago
    I don't think these things really fit the bill, but this is closest I could find. All the codecrafters stuff is focused on reading from a sqlite DB. A great start, but you're not implementing B+ trees. Hope this gives you some options. https://github.com/Arminas42/build_your_own_database https://github.com/Arminas42/build_your_own_database https://github.com/search?q=codecrafters+sqlite-java+language%3AJava&type=repositories https://github.com/search?q=codecrafters+sqlite-java+languag...
  - lionkor 2y ago
    it's not that hard, you should try it :)

whalesalad 2y ago

How do you connect to it and actually use it? Does it behave like a SQL database? Row-based? Column-based? Good for analytical workloads? Document store? Redis/memcached database? What are it's strengths or weaknesses? Cool accomplishment in and of itself but hard for anyone here to really give you any criticism or feedback without understanding where it excels and how to work with it.

zsoltkacsandi 2y ago
> Main focus was on implementing & understanding working the of database I think this clearly describes what was the goal.
- whalesalad 2y ago
  Yep... but there are dozens of different types of databases out there. There is no way to look at this codebase and give any kind of feedback one way or the other without that understanding, particularly with zero usage examples. Which is what the OP is asking about.
  - jasonlotito 2y ago
    > There is no way to look at this codebase and give any kind of feedback Less than a minute and I know how to use it. It's not complicated. The source code is available and fairly easy. If you can't figure out how to use it in a trivial amount of time, you aren't going to be able to offer anything of value. When did HN go from being about interesting stuff to making bold, ignorant statements like "There is no way to look at this codebase and give any kind of feedback one way or the other without that understanding." You should know better.
    - whalesalad 2y ago
      [flagged]
      - 2y ago

esafak 2y ago

[flagged]

anonzzzies 2y ago
"Main focus was on implementing & understanding working the of database, storage management & transaction handling." Obviously OP wrote it to understand database theory & implementation.
- deleted 2y ago
  [deleted]
ajsnigrutin 2y ago
It's hackernews, basically every old thing has to be reinvented in the "language of the week", be it go, rust, ruby, or whatever was before that. From a learning perspective, nice project for OP, for 'advertising' it, i'd prefer the "what's better than the alternatives" instead of focusing on the language.
- johnisgood 2y ago
  I want more Ada / SPARK and Factor. :D
pjmlp 2y ago
That a database written in an AOT compiled managed language is possible, for example. Specially relevant, because it used to be that writing database engines used to be considered systems programming, and we all know managed languages cannot be possible used for such tasks. /s
deleted 2y ago
[deleted]

mbreese 2y ago

This looks like a good exploratory project! One thing I’d add to the readme is an example of how you’d use the database in an example application. From the docs, it’s clear that this isn’t a sql database (yet?), so it would be good to have an example to see how to use the database. It might also be nice to have a description of what happens when you insert or get a record, so others can learn from the code too. Or can you comment here about what your favorite part of the code is? What did you figure out that you didn’t know before? If you’re using the project to learn about databases, what have you learned so far?

Sahil121 2y ago
thanks man yes currently it is not a fully sql database (have to add query support) My fav part of code is the data retrieval code because had some issues there & got to know lot about it Learnt about B+ Trees, transactions, managing concurrent reads & data presistence

littlemerman 2y ago

You might be interested in this golang db from mit’s database systems course: https://github.com/MIT-DB-Class/go-db-2024 https://github.com/MIT-DB-Class/go-db-2024 https://dsg.csail.mit.edu/6.5830/ https://dsg.csail.mit.edu/6.5830/

Sahil121 2y ago
will surely checkout thanks man

eatonphil 2y ago

If folks would like to see more examples of databases built to teach oneself, they get shared on the /r/databasedevelopment subreddit not infrequently. Some recent ones: https://www.reddit.com/r/databasedevelopment/comments/1hyig8k/my_very_own_toy_database/ https://www.reddit.com/r/databasedevelopment/comments/1hyig8... https://www.reddit.com/r/databasedevelopment/comments/1ha5ccb/building_a_database_from_scratch_simpledb/ https://www.reddit.com/r/databasedevelopment/comments/1ha5cc... https://www.reddit.com/r/databasedevelopment/comments/1dqgmsa/simpledb_an_educational_rdbms_implemented_in_go/ https://www.reddit.com/r/databasedevelopment/comments/1dqgms... https://www.reddit.com/r/databasedevelopment/comments/1ix5dzz/built_a_database_from_scratch_in_go/ https://www.reddit.com/r/databasedevelopment/comments/1ix5dz... https://www.reddit.com/r/databasedevelopment/comments/18knodn/i_made_a_lsmbased_kv_storage_engine_in_rust_help/ https://www.reddit.com/r/databasedevelopment/comments/18knod... https://www.reddit.com/r/databasedevelopment/comments/1h3w70l/tidesdb_high_performance_transactional_durable/ https://www.reddit.com/r/databasedevelopment/comments/1h3w70... https://www.reddit.com/r/databasedevelopment/comments/1gk18nd/k4_opensource_highperformance_transactional_and/ https://www.reddit.com/r/databasedevelopment/comments/1gk18n... https://www.reddit.com/r/databasedevelopment/comments/1bemf9w/new_toy_database_to_learn_and_play_with/ https://www.reddit.com/r/databasedevelopment/comments/1bemf9... https://www.reddit.com/r/databasedevelopment/comments/1iw6cxs/canopydb_transactional_kv_store_yet_another_in/ https://www.reddit.com/r/databasedevelopment/comments/1iw6cx...

danhau 2y ago
Nice, thanks! I didn‘t know there was a subreddit for that.
deleted 2y ago
[deleted]
seafoamteal 2y ago
Hi Phil, I thought I'd find you here! Love your blog!
- eatonphil 2y ago
  Glad to hear it. :)

faizhalde 2y ago

excellent, would love to hear more about resources you used while implementing

Sahil121 2y ago
followed this book - https://build-your-own.org/database/ https://build-your-own.org/database/

twalla 2y ago

Someone asked about resources and I ran into this while evaluating embedded db options for a golang project - it's a collection of db components implemented in golang: https://github.com/thomasjungblut/go-sstables https://github.com/thomasjungblut/go-sstables

tjungblut 2y ago
thank you for referring, feel free to ask any questions you may have

voodooEntity 2y ago

I didnt look into it yet but i already wanted to say - cuz of curiosity i build my own graph database in golang :) and i learned alot. so i absolutly understand why you did it and what experiences you probably made on the way :D congratz !

Sahil121 2y ago
thanks man
varelaseb 2y ago
Is this published anywhere?
- voodooEntity 2y ago
  Kinda. So first i went for a standalone application back than to use it as storage/database. This part is archived now. it was released under: https://github.com/voodooEntity/slingshotdb https://github.com/voodooEntity/slingshotdb At this point i think its important to mention that its nativly an in-memory graph database/storage. At some point (4-5 years ago) i decided to change the way i handle it. Since im using the storage in many private projects, and there was no real traction (while also maintaining a database is quite some work) i decided i gonne change the way i handle it away from a standalone database to a library that works as an in-memory graph storage. Also i added a custom query language including builder (the queries are 100% json compatible so you can in theory build queries from whatever language and transmit it via json. The library is released under : https://github.com/voodooEntity/gits https://github.com/voodooEntity/gits Currently im completly reworking the query parser but since this is a privat project and im the only maintainer it may take some to to finish it. The update will takle some smaller bugs in the query parser that can occur on very complex nested queries and also will optimize query speeds. I cant provide a date when the update will occure i only can say it will (because i need it myself). Finally said: The strength of gits is to be very fast (hence in memory handling) and easy to use from golang. Its threadsafe by default.

bob1029 2y ago

If you are wanting proper SQL command support, you could copy the SQLite parser approach. Properly parsing all valid command texts is not a problem that I would find compelling unless I was being compensated for it. https://www.sqlite.org/lemon.html https://www.sqlite.org/lemon.html You could probably use something like participle, but you'd have to translate the grammar. https://github.com/alecthomas/participle https://github.com/alecthomas/participle

Sahil121 2y ago
yes man have to work on adding the sql cmd support thanks for the links
qaq 2y ago
You can use https://github.com/cockroachdb/cockroachdb-parser https://github.com/cockroachdb/cockroachdb-parser which is basically PGSQL compatible parser implemented in golang
mingodad 2y ago
I do have a big collection of LALR(1) grammars to test/study/develop/document here https://mingodad.github.io/parsertl-playground/playground/ https://mingodad.github.io/parsertl-playground/playground/ including sqlite, tidb, vites, postgresql, mysql, ...

alecco 2y ago

Cool! You watched CMU's intro to Database sytems, right? It's really good and thorough. It will save you of some common pitfalls and can help you navigate the trade-offs. https://www.youtube.com/watch?v=otE2WvX3XdQ&list=PLSE8ODhjZXjYDBpQnSymaectKjxCy6BYq https://www.youtube.com/watch?v=otE2WvX3XdQ&list=PLSE8ODhjZX...

Sahil121 2y ago
sure man
hobs 2y ago
Looks fun, but the intro requirement is "Knows C++" - that's kinda a non starter right?
- alecco 2y ago
  I'm sure that's for the homework.
- sieabahlpark 2y ago
  [dead]
BiraIgnacio 2y ago
Oh thanks for the link, this is really great!
Tostino 2y ago
I really have liked the current Optimizer series. So much amazing info. I hope some company that heavily relies on Postgres sponsors a project to replace the optimizer with a more modern implementation. I feel like for any large applications with olap style queries (even if on the OLTP database) will benefit incredibly from that work. SQL Servers optimizer was light-years ahead of Postgres when I worked with it last over a decade ago.
- alecco 2y ago
  Sadly, most of those corporations would rather spend millions in proprietary solutions than give back 100k to the community. I've seen it first hand many times. They only give money if they absolutely have to.

fipar 2y ago

Congratulations on a great exploratory project! It takes me back to my school years. I never got as far as you (not by any long stretch actually), but I did enjoy creating the storage layer of a database from scratch. To actually have to deal with, instead of just think of, all the edge cases, is quite the transformative experience. As a humble suggestion, since it seems your goal is to understand how relational databases work and not necessarily to write a new database that will compete with others, maybe don't make it an SQL one? We've got enough of those, and not enough of the others. Would be nice to have a new relational DB using Tutorial D as its language for example. Keep hacking!

Sahil121 2y ago
thanks for the suggestion man, will definitely think about this

snissn 2y ago

to also share a reference - here's an on disk hashmap that uses mmap that I made in golang: https://github.com/snissn/gomap https://github.com/snissn/gomap

tapirl 2y ago

Cool! btw, README says Go 1.17+ is required, but go.mod says 1.23+.

cbluth 2y ago
You should trust that the readme is up to date
- tapirl 2y ago
  With Go toolchain [1.17, 1.23.1], it fails to build.

hsnice16 2y ago

This is so nice! I created an email verifier project in Go to check MX records, SMTP server status, and existence in the disposable list. Project - https://github.com/hsnice16/email-verifier https://github.com/hsnice16/email-verifier

curtisszmania 2y ago

[dead]

mdwhatcott 2y ago

Best part: a go.mod with 0 dependencies!

OccamsMirror 2y ago
Now that is impressive. Gotta love zero dependency projects.

dveeden2 2y ago

Other databases written in Go: - TiDB by PingCAP - Vitess by PlanetScale Both are basically only the SQL part as TiDB uses TiKV (written in Rust) and Vitess uses MySQL. For those who want to implement a database in Go but without having to implement a network protocol there is go-mysql, which allows you to do this: https://github.com/go-mysql-org/go-mysql/blob/master/cmd/go-mysqlserver/main.go https://github.com/go-mysql-org/go-mysql/blob/master/cmd/go-... As demonstration I created a networked SQLite: https://github.com/dveeden/go-mysql-examples/blob/main/go-mysqlite/main.go https://github.com/dveeden/go-mysql-examples/blob/main/go-my... Both TiDB and Vitess have parsers that can be used outside standalone. So if you only want to implement your own on disk format, this can help. Note that I'm working for PingCAP on TiDB and I'm also a co-maintainer for go-mysql.

raggi 2y ago
Vitess came from YouTube
- CBLT 2y ago
  Didn't the Vitess team found planetscale?
  - harshitgangal 2y ago
    Yes! The founders of PlanetScale were the co-creators of Vitess at YouTube, where it was built to handle MySQL scalability. PlanetScale builds on Vitess but offers a managed, developer-friendly experience.
aqueueaqueue 2y ago
We use Tidb at work at scale. Great product! Was looking at the source today to understand an error code.

0x3331 2y ago

Looks like a start. Keep it up, keep learning. Definitely check out https://www.youtube.com/c/CMUDatabaseGroup https://www.youtube.com/c/CMUDatabaseGroup Great lectures on relational databases and more.

bosky101 2y ago

Maybe a good place to ask: good sdks or libraries for reading parquet?