4 ms·
Geniusrise is a modular, loosely-coupled AI-microservices framework. It can be used to perform various tasks, including hosting inference endpoints, performing
by ixaxaar 3y ago
Geniusrise is a modular, loosely-coupled AI-microservices framework.
It can be used to perform various tasks, including hosting inference endpoints, performing bulk inference, fine tune etc with open source models or closed source APIs.
- The framework provides structure for modules and operationalizes and orchestrates them.
- The modular ecosystem provides a layer of abstraction over the myriad of models, libraries, tools, parameters and optimizations underlying the operationalization of modern AI models.
1. Install geniusrise and libs
pip install torch
pip install geniusrise
pip install geniusrise-vision # vision multi-modal models
pip install geniusrise-text # text models, LLMs
pip install geniusrise-audio # audio models
2. Create YAML file
version: '1'
bolts:
my_multimodal_api:
name: VisualQAAPI
state:
type: none
input:
type: batch
args:
input_folder: ./input
output:
type: batch
args:
output_folder: ./output
method: listen
args:
model_name: 'llava-hf/bakLlava-v1-hf'
model_class: 'LlavaForConditionalGeneration'
processor_class: 'AutoProcessor'
device_map: 'cuda:0'
use_cuda: True
precision: 'bfloat16'
quantization: 0
max_memory: None
torchscript: False
compile: False
flash_attention: False
better_transformers: False
endpoint: '*'
port: 3000
cors_domain: 'http://localhost:3000'
username: 'user'
password: 'password'
3. Launch API service
genius rise
MY_IMAGE=/path/to/test/image
(base64 -w 0 $MY_IMAGE | awk '{print "{\"image_base64\": \""$0"\", \"question\": \"<image>\nUSER: Whats the content of the image?\nASSISTANT:\", \"do_sample\": false, \"max_new_tokens\": 128}"}' > /tmp/image_payload.json)
curl -X POST http://localhost:3000/api/v1/answer_question \
-H "Content-Type: application/json" \
-u user:password \
-d @/tmp/image_payload.json | jq
More: https://docs.geniusrise.ai/guides/usage/ https://docs.geniusrise.ai/guides/usage/ and https://github.com/geniusrise/examples https://github.com/geniusrise/examples