4 ms·
Optimizing Recommendation Systems with JDK's Vector API
- aberoham 7mo ago"the remaining 2% were large batch requests", [which made up 50% of the work] .. who really watches that many shows on Netflix? What was in those batches, if someone is watching that much, why bother with serendipity at all? Most serendipitous thing you could do is shut off their subscription.
- jayd16 7mo agoNote that they likely mean the list of candidates is large not the user history. This is for an API so perhaps 2% of client requests implemented batch requests, providing the opportunity for batch processing of that request.
- lukev 7mo agoI had success on a similar problem by allocating native buffers for the matrices, then using a basic CUDA call. The actual work was 100x faster than my CPU baseline. The bottleneck of course was fetching & loading relevant data to memory to start with.
- zmmmmm 7mo agoThe vector API is really interesting but so frustrating that it has taken so long to materialise. The inability of Java to properly utilise parallel compute - whether it's SIMD or GPU - has been a huge factor in dealing it out of being at the forefront of modern compute.
- pjmlp 7mo agoTornado VM exists for a few years already. Additionally you can always do the Python way, write C++ libs, pack them into a jar, and call it a Java library.
- 7777777phil 7mo agoThis is a JVM infrastructure optimization, not a recommendation architecture breakthrough. At Netflix scale, most of the field runs GPU-accelerated ANN search (FAISS, ScaNN) for candidate retrieval.
- oofbey 7mo agoFirst mover disadvantage: you need to invent a way of doing things (ANN) before any decent tools have been built. 10 years later the world is laughing at you for using Java to do vector math.
- lern_too_spel 7mo agoFrom the sounds of it, they're looking for items that are different from their watch history among some candidate set, which vector dbs aren't optimized for.