3 ms·

We were fortunate to work in AWS which has some of the gnarliest datastructures and algorithms problems in the world, so we got really good there. Spreadsheets

We were fortunate to work in AWS which has some of the gnarliest datastructures and algorithms problems in the world, so we got really good there. Spreadsheets are really just more datastructures and algorithms on the backend. Our frontend guy is brilliant and built a lot of awesome stuff at Airtable, but I can't speak to the craziness that is high-complexity high-performance frontend. But it's absolutely necessary to a good product too!

sidcool 3y ago
Excuse me for probing further. Can you share some of the books that can help with this? Which algorithms were used, how etc. Thank in advance, I am using Row Zero since yesterday and loving it thoroughly.
- gamegoblin 3y ago
  I don't have any great book recommendations because I learned on the job mostly. For a project that will teach you literally everything there is to know about backend development: 1. Write a simple PUT/GET/DELETE REST API network layer, this should be a few hundred lines of code max depending on how much you choose to lean on libraries vs write it yourself 2. Then, write a simple in-memory blob store behind the PUT/GET/DELETE REST API. Just use a very simple hashmap of keys to data. You now have an in-memory S3 mock! 3. Now, rather than in-memory hashmap, start saving the files to disk. Write code that allows you to start and stop the process and recover all the data from disk. Now you have a persistent S3 mock. 4. Now, start finding the limits of this application. Write a program that calls your API with different usage patterns to load test it. Find the limits. 5. Now start optimizing. You can go as far as you want here. This optimization process was my whole career at AWS S3, and most of it at Row Zero. 6. If you want to really learn some advanced data structures and algorithms, stop using the filesystem directly, and write your own storage mechanism. Allocate 1 giant 16GB (or however large) file and store all of your blob data inside that 16GB "partition". You have to do all the serialization and retrieval etc yourself. Your filesystem is already doing this for you when you store as files (look up how the ext4 filesystem works, inodes, pages, etc). But your filesystem is probably optimized for consumer use, not this blob-storage system use. So you can implement your own much faster version if you want. If you want to really learn datastructures and algorithms, try to code a simple Log Structured Merge Tree. This will teach hashes, trees, bloom filters, serialization, deserialization, etc, all with high performance in mind.
  - sidcool 3y ago
    Thanks! Appreciate your detailed response!!

sidcool 3y ago

Excuse me for probing further. Can you share some of the books that can help with this? Which algorithms were used, how etc. Thank in advance, I am using Row Zero since yesterday and loving it thoroughly.

gamegoblin 3y ago
I don't have any great book recommendations because I learned on the job mostly. For a project that will teach you literally everything there is to know about backend development: 1. Write a simple PUT/GET/DELETE REST API network layer, this should be a few hundred lines of code max depending on how much you choose to lean on libraries vs write it yourself 2. Then, write a simple in-memory blob store behind the PUT/GET/DELETE REST API. Just use a very simple hashmap of keys to data. You now have an in-memory S3 mock! 3. Now, rather than in-memory hashmap, start saving the files to disk. Write code that allows you to start and stop the process and recover all the data from disk. Now you have a persistent S3 mock. 4. Now, start finding the limits of this application. Write a program that calls your API with different usage patterns to load test it. Find the limits. 5. Now start optimizing. You can go as far as you want here. This optimization process was my whole career at AWS S3, and most of it at Row Zero. 6. If you want to really learn some advanced data structures and algorithms, stop using the filesystem directly, and write your own storage mechanism. Allocate 1 giant 16GB (or however large) file and store all of your blob data inside that 16GB "partition". You have to do all the serialization and retrieval etc yourself. Your filesystem is already doing this for you when you store as files (look up how the ext4 filesystem works, inodes, pages, etc). But your filesystem is probably optimized for consumer use, not this blob-storage system use. So you can implement your own much faster version if you want. If you want to really learn datastructures and algorithms, try to code a simple Log Structured Merge Tree. This will teach hashes, trees, bloom filters, serialization, deserialization, etc, all with high performance in mind.
- sidcool 3y ago
  Thanks! Appreciate your detailed response!!