3 ms·
I don't have any great book recommendations because I learned on the job mostly. For a project that will teach you literally everything there is to know about
by gamegoblin 3y ago
I don't have any great book recommendations because I learned on the job mostly.
For a project that will teach you literally everything there is to know about backend development:
1. Write a simple PUT/GET/DELETE REST API network layer, this should be a few hundred lines of code max depending on how much you choose to lean on libraries vs write it yourself
2. Then, write a simple in-memory blob store behind the PUT/GET/DELETE REST API. Just use a very simple hashmap of keys to data. You now have an in-memory S3 mock!
3. Now, rather than in-memory hashmap, start saving the files to disk. Write code that allows you to start and stop the process and recover all the data from disk. Now you have a persistent S3 mock.
4. Now, start finding the limits of this application. Write a program that calls your API with different usage patterns to load test it. Find the limits.
5. Now start optimizing. You can go as far as you want here. This optimization process was my whole career at AWS S3, and most of it at Row Zero.
6. If you want to really learn some advanced data structures and algorithms, stop using the filesystem directly, and write your own storage mechanism. Allocate 1 giant 16GB (or however large) file and store all of your blob data inside that 16GB "partition". You have to do all the serialization and retrieval etc yourself.
Your filesystem is already doing this for you when you store as files (look up how the ext4 filesystem works, inodes, pages, etc). But your filesystem is probably optimized for consumer use, not this blob-storage system use.
So you can implement your own much faster version if you want. If you want to really learn datastructures and algorithms, try to code a simple Log Structured Merge Tree. This will teach hashes, trees, bloom filters, serialization, deserialization, etc, all with high performance in mind.
- sidcool 3y agoThanks! Appreciate your detailed response!!