Local generative models with GPT4All and LocalAI

Coast Redwoods

So far I tried running models in AWS SageMaker and used the OpenAI APIs. There are some local options too and with only a CPU. The response times are relatively high, and the quality of responses do not match OpenAI but none the less, this is an important step in the future inference on all devices and for use in private/disconnected deployments.

GPT4All has some easy to follow guides. I did not use their installer. Instead, download the a model and you can run a simple python program. Vicuna seems to the trending model to use. It has some fine tuning on top of Facebook LlaMa.

import gpt4all

gpt = gpt4all.GPT4All(model_name='ggml-vicuna-13b-1.1-q4_2.bin', allow_download=False,
                       model_path='/models/')

messages = [{"role": "user", "content": "What other plants can grow well where grapes thrive? Think step by step"}]
response = gpt.chat_completion(messages=messages, default_prompt_header=True, #this instructs the model to be behave like a chatbot
                               default_prompt_footer=False, verbose=True)

print(response)

Additionally there is another project called LocalAI that provides OpenAI compatible wrappers on top of the same model you used with GPT4All. It takes a few minutes to start so be patient and use docker-compose logs to see the progress. There are many errors and warnings, but it does work in the end. Should show something like following when ready:

localai-api-1  | 
localai-api-1  |  ┌───────────────────────────────────────────────────┐ 
localai-api-1  |  │                   Fiber v2.46.0                   │ 
localai-api-1  |  │               http://127.0.0.1:8080               │ 
localai-api-1  |  │       (bound on host 0.0.0.0 and port 8080)       │ 
localai-api-1  |  │                                                   │ 
localai-api-1  |  │ Handlers ............ 23  Processes ........... 1 │ 
localai-api-1  |  │ Prefork ....... Disabled  PID .............. 3662 │ 
localai-api-1  |  └───────────────────────────────────────────────────┘ 
localai-api-1  |

Use curl or Postman at the endpoint /v1/models to see what models you have.

% curl http://localhost:8080/v1/models
{"object":"list","data":[{"id":"ggml-gpt4all-l13b-snoozy.bin","object":"model"},{"id":"ggml-vicuna-13b-1.1-q4_2.bin","object":"model"}]}

Chat follows the similar OpenAI API.

POST http://localhost:8080/v1/chat/completions

{
    "model": "ggml-vicuna-13b-1.1-q4_2.bin",
    "messages": [
        {
            "role": "system",
            "content": "You are a conversational assistant reponding to a customer with accurate answers"
        },
        {
            "role": "user",
            "content": "What other plants can grow well where grapes thrive? Think step by step"
        }
    ],
    "temperature": 0.7
}

{
    "object": "chat.completion",
    "model": "ggml-vicuna-13b-1.1-q4_2.bin",
    "choices": [
        {
            "message": {
                "role": "assistant",
                "content": ".\nAnswer: Grapes typically grow well in regions with mild temperatures, adequate rainfall and well-draining soil. If you want to plant other crops in an area where grapes thrive, you should consider crops that have similar growing requirements. Some examples of crops that can grow well in the same conditions as grapes include:\n1. Fruit trees such as apple, pear and plum\n2. Berries such as raspberries, blackberries and currants\n3. Vegetables such as tomatoes, peppers and cucumbers\n4. Herbs such as basil, oregano and thyme\n5. Flowers such as roses and lavender\nIt is important to note that the specific crop varieties will depend on your local climate and soil conditions. It is best to consult with a local nursery or gardening expert to determine the best crops for your specific location."
            }
        }
    ],
    "usage": {
        "prompt_tokens": 0,
        "completion_tokens": 0,
        "total_tokens": 0
    }
}

These are purely experimental, and also please keep model licensing and biases in mind.

If these topics interest you then reach out to me and I will appreciate any feedback. If you would like to work on such problems you will generally find open roles as well! Please refer to LinkedIn.