Your first question is how to make what you are working on now faster. Follow a process to find the places where you can make improvements and there are bottle

Your first question is how to make what you are working on now faster. Follow a process to find the places where you can make improvements and there are bottlenecks. I always like to start at the data layer myself, and work my way up the stack. - If using an ORM, dump all the queries that it is creating on your behalf. Check how they compare to what you would hand write, look at the execution plans. - Tune the queries, reduce joins where possible, add/remove indexes etc. Sometimes removing an index helps more then adding new ones if insert/updates are a bottleneck. Don't forget to check these. - In the application layer, check all your translations/transformations to measure and tune those. These can add a lot of time if called repeatedly for many records. - At the UI level find places where there might be a lot more going on then needs be. I so often find places where a lot more data is being requested then needed and then you have to filter through it all and if nothing else it adds memory pressure which can lead to hard to find performance issues. - Cache's are awesome, but they sometimes just add complexity and delays where you don't need them. Small datasets usually can do really well just with some materialized views, data partitions and query tuning. I have actually removed caches before and improved performance after a little query work, don't ignore this. But I wouldn't say that is the norm, but it does exist. - Denormalize in places to remove joins if they are not substantially necessary. When working on a system that had large marketing datasets, I found that by caching a few fields on a highly used table we could eliminate many joins in the application which were on numerous pages (common queries), in some cases giving us 25x improvements. Of course, that adds a little extra time and consideration on inserts/updates but that tradeoff is usually worth it if you are read/query heavy. As for learning to deal with large datasets. It starts with all of the above, you need to really tune the data/queries at the database level first. Postgres is awesome, relational data is great for a ton of things, but there may be a reason to store data in non-relational ways for some things (cache is a great example). This means you might have replicated data in places which adds a little complexity but aids in performance. Always consider how the data is going to be accessed and used by the application(s). Joins are an example of a place where small changes can result in magnitudes greater performance. Large data means small details are way more important, and what I have seen a lot of times is people don't pay attention to a lot of the little details when working with smaller datasets. That's pretty fair most of the time because it really doesn't affect performance on small datasets. But in the end when you get to large data all those details matter. Even little things like picking the right datatype is more critical in large datasets. You don't want to store a number as a string and always be doing conversions on it as an example. It adds a lot of extra wasted cycles. Of course, on small data, you would rarely see a problem with it, which makes people get complacent with their choices many times.