Objects are expensive. If you're dealing with millions of them you definitely want to 1.) use primitives where possible (ie: int, not Integer) and 2.) put those

Objects are expensive. If you're dealing with millions of them you definitely want to 1.) use primitives where possible (ie: int, not Integer) and 2.) put those primitives in native arrays instead of collections (ie: int[] and not List<Integer>). It sounds like in your use case that means going from the collection of objects pattern to the object of collections pattern.