The request will be an S3 style API request and the contents will vary depending on the workload. For example, OpenAI trains an LLM, and for every iteration, a

The request will be an S3 style API request and the contents will vary depending on the workload. For example, OpenAI trains an LLM, and for every iteration, a batch of model input data is requested from their object storage. My service would sit in the middle with a subset of the data and respond to the requests faster. Regarding performance, moving data between CPU and GPU is slow but that can be minimized with CUDA unified memory and eliminated with GPUDirect RDMA. The product being a cache for S3 means that the client uses S3 and probably uses other AWS Services for their backend. I can maximize network bandwidth through the co-location of their backend service and my cache service in the same availability zone of AWS. The GPU's computational power is useful for processing batched requests in parallel such as metadata requests and GET/PUT/DELETE requests for small objects. The GPU cache protects data integrity by asynchronously copying changed data to S3 on frequent intervals. Additionally, I am planning to add a WAL option that will leverage the NVME storage of the GPU EC2 instance I am using for the service. There are my thoughts so far on how I would address these issues. I have implemented most of them, and I'm working on improving it over time. I am happy to answer additional questions.