4 ms·

Why do none of these apps allow you to set the system prompt? I find these LLM apps kind of useless without being able to refine the way in which the model will

Why do none of these apps allow you to set the system prompt? I find these LLM apps kind of useless without being able to refine the way in which the model will respond to later questions.

wahnfrieden 3y ago
I made a free / mostly open source one for iOS that lets you edit the system prompt https://chatonmac.com https://chatonmac.com
- ionwake 3y ago
  Amazing! Does it submit any data online ?
  - wahnfrieden 3y ago
    No. I definitely do not want any liability of user-generated content or PII or similar. I have no analytics, besides the standard Apple opt-in crash/reporting (not using any 3rd-party service and not sending anything to my own servers). It downloads configuration from GitHub and HuggingFace directly. It also has OpenAI integration, directly to their servers via BYOK.
- ricktdotorg 3y ago
  trying this out! BTW and FYI i need to reduce the font size on my iOS device to be smaller than i like in order to use your add/replace API key key pages. if the font is "larger than normal" i can't see/focus on the box to enter or paste in the API key. just increase your iOS system font size to trigger this. thanks in advance for fixing, will try out the app!
  - wahnfrieden 3y ago
    Thanks for the detailed report - will fix asap, along with releasing the macOS v1.0. I've just soft launched this so far but have more to come so please let me know anything else.
YetAnotherNick 3y ago
Mistral instruct doesn't have system prompt AFAIK. Also llama chat system prompt is very useless in my testing.
- rrr_oh_man 3y ago
  It seems to work well on GPT4All (macOS) with system prompts? Can you link to any doc why it shouldn't work?
- refulgentis 3y ago
  It does; and if it's LLaMa 2 7B Chat stock from Facebook, that was a little rushed imho, doesn't seem as baked in. (GPTs matters but it's _very_ bizarre who it thinks it's coming from)
- nl 3y ago
  Mistral Instruct does use a system prompt. You can see the raw format here: https://www.promptingguide.ai/models/mistral-7b#chat-template-for-mistral-7b-instruct https://www.promptingguide.ai/models/mistral-7b#chat-templat... and you can see how LllamaIndex uses it here (as an example): https://github.com/run-llama/llama_index/blob/1d861a9440cdc9e1b2335320a7a2bf26667b90b2/llama_index/llms/llama_utils.py#L27 https://github.com/run-llama/llama_index/blob/1d861a9440cdc9...
  - sp332 3y ago
    So the system prompt is just part of the first prompt in a conversation? How is that different from not having a system prompt?
    - nl 3y ago
      No, system prompts are surrounded by [INST] [/INST]
      - sp332 3y ago
        No, every user input is surrounded by those tags. Scroll down from that link you posted and read the next two example prompts.
brittlewis12 3y ago
would love for you to give cnvrs a shot! - save characters (system prompt + temperature, and a name & cosmetic color) - download & experiment with models from 1b, 3b, & 7b, and quant options q2k, q4km, q6k - save, search, continue, & export past chats along with smaller touches: - custom theme colors - haptics and more coming soon! https://testflight.apple.com/join/ERFxInZg https://testflight.apple.com/join/ERFxInZg
- sockaddr 3y ago
  Do not download this. I downloaded this on my 14 Pro and it completely locked up the system to the point where even the power button wouldn’t work. I couldn’t use my phone for about 10 minutes.
  - scottbartell 3y ago
    I've used it for a couple weeks on my 15 Pro and I haven't experienced anything like that. (IMO it's well worth the download) The developer is also pretty responsive and actively looking for feedback (which is why it's currently free on TestFlight)
  - brittlewis12 3y ago
    I’m very sorry about your experience. That’s definitely not what I was aiming for, and I can imagine that was a nasty surprise. Any hang like that is unacceptable, full stop. My understanding is Metal is currently causing hangs on devices when there is barely enough RAM to fit the model and prompt, but not quite enough to run. Will work on falling back to CPU to avoid this kind of experience much more aggressively than today. Thank you for taking the time to both try it out and to share your experience; I will use it to ensure it’s better in the future.
    - sockaddr 3y ago
      Thanks for the response. Unfortunately on my device the behavior makes it impossible to report a bug using a screenshot as requested in the app. I can give you more device info if you want to narrow down the cause.
deleted 3y ago
[deleted]

Why do none of these apps allow you to set the system prompt? I find these LLM apps kind of useless without being able to refine the way in which the model will respond to later questions.

wahnfrieden 3y ago

I made a free / mostly open source one for iOS that lets you edit the system prompt https://chatonmac.com https://chatonmac.com

ionwake 3y ago
Amazing! Does it submit any data online ?
- wahnfrieden 3y ago
  No. I definitely do not want any liability of user-generated content or PII or similar. I have no analytics, besides the standard Apple opt-in crash/reporting (not using any 3rd-party service and not sending anything to my own servers). It downloads configuration from GitHub and HuggingFace directly. It also has OpenAI integration, directly to their servers via BYOK.
ricktdotorg 3y ago
trying this out! BTW and FYI i need to reduce the font size on my iOS device to be smaller than i like in order to use your add/replace API key key pages. if the font is "larger than normal" i can't see/focus on the box to enter or paste in the API key. just increase your iOS system font size to trigger this. thanks in advance for fixing, will try out the app!
- wahnfrieden 3y ago
  Thanks for the detailed report - will fix asap, along with releasing the macOS v1.0. I've just soft launched this so far but have more to come so please let me know anything else.

YetAnotherNick 3y ago

Mistral instruct doesn't have system prompt AFAIK. Also llama chat system prompt is very useless in my testing.

rrr_oh_man 3y ago
It seems to work well on GPT4All (macOS) with system prompts? Can you link to any doc why it shouldn't work?
refulgentis 3y ago
It does; and if it's LLaMa 2 7B Chat stock from Facebook, that was a little rushed imho, doesn't seem as baked in. (GPTs matters but it's _very_ bizarre who it thinks it's coming from)
nl 3y ago
Mistral Instruct does use a system prompt. You can see the raw format here: https://www.promptingguide.ai/models/mistral-7b#chat-template-for-mistral-7b-instruct https://www.promptingguide.ai/models/mistral-7b#chat-templat... and you can see how LllamaIndex uses it here (as an example): https://github.com/run-llama/llama_index/blob/1d861a9440cdc9e1b2335320a7a2bf26667b90b2/llama_index/llms/llama_utils.py#L27 https://github.com/run-llama/llama_index/blob/1d861a9440cdc9...
- sp332 3y ago
  So the system prompt is just part of the first prompt in a conversation? How is that different from not having a system prompt?
  - nl 3y ago
    No, system prompts are surrounded by [INST] [/INST]
    - sp332 3y ago
      No, every user input is surrounded by those tags. Scroll down from that link you posted and read the next two example prompts.

brittlewis12 3y ago

would love for you to give cnvrs a shot! - save characters (system prompt + temperature, and a name & cosmetic color) - download & experiment with models from 1b, 3b, & 7b, and quant options q2k, q4km, q6k - save, search, continue, & export past chats along with smaller touches: - custom theme colors - haptics and more coming soon! https://testflight.apple.com/join/ERFxInZg https://testflight.apple.com/join/ERFxInZg

sockaddr 3y ago
Do not download this. I downloaded this on my 14 Pro and it completely locked up the system to the point where even the power button wouldn’t work. I couldn’t use my phone for about 10 minutes.
- scottbartell 3y ago
  I've used it for a couple weeks on my 15 Pro and I haven't experienced anything like that. (IMO it's well worth the download) The developer is also pretty responsive and actively looking for feedback (which is why it's currently free on TestFlight)
- brittlewis12 3y ago
  I’m very sorry about your experience. That’s definitely not what I was aiming for, and I can imagine that was a nasty surprise. Any hang like that is unacceptable, full stop. My understanding is Metal is currently causing hangs on devices when there is barely enough RAM to fit the model and prompt, but not quite enough to run. Will work on falling back to CPU to avoid this kind of experience much more aggressively than today. Thank you for taking the time to both try it out and to share your experience; I will use it to ensure it’s better in the future.
  - sockaddr 3y ago
    Thanks for the response. Unfortunately on my device the behavior makes it impossible to report a bug using a screenshot as requested in the app. I can give you more device info if you want to narrow down the cause.

deleted 3y ago

[deleted]