3 ms·

Llama 2 on togetherAI is as bad of a privacy nightmare as OpenAI

54 points by behnamoh 3y ago

jankovicsandras 3y ago
Better privacy might be running Llama2 locally, offline. If somebody hasn't tried running LLMs yet, here are some lines that do the job in Google Colab or locally. The !s are for Colab, remove them for local terminal. The script downloads the ca. 8GB model, but Llama.cpp can run offline afterwards. ! git clone https://github.com/ggerganov/llama.cpp.git ! wget "https://huggingface.co/TheBloke/CodeLlama-7B-GGUF/resolve/main/codellama-7b.Q8_0.gguf" -P llama.cpp/models ! cd llama.cpp && make ! ./llama.cpp/main -m ./llama.cpp/models/codellama-7b.Q8_0.gguf --color --ctx_size 2048 -n -1 -ins -b 256 --top_k 10000 --temp 0.2 --repeat_penalty 1.1 -t 8
- lardo 3y ago
  https://huggingface.co/TheBloke/CodeLlama-7B-GGUF https://huggingface.co/TheBloke/CodeLlama-7B-GGUF describes codellama-7b.Q8_0.gguf as "very large, extremely low quality loss - not recommended"
  - jmorgan 3y ago
    Q4_0 is often mentioned as being the "tried and true" quantization level to try first. I've heard folks have had good results with 3-bit quantization (Q3_K_M) as well
    - jsemrau 3y ago
      just tried it and its not good. I think one of the better feedback was asking for play an adventure in the style of Space Quest 1
- Patrick_Devine 3y ago
  Or, for the easy way of doing it, just install Ollama and run: `ollama run codellama:7b`. If you want the q8 version (which you probably don't), it's `ollama run codellama:7b-code-q8_0`.
- rvz 3y ago
  Or if you're on macOS just hit 'Download' and use Ollama locally. [0] [0] https://ollama.ai/download https://ollama.ai/download
edgoode 3y ago
You can run models at https://shadeform.ai https://shadeform.ai in your cloud accounts or ours, and we don’t track or keep anything.
Palmik 3y ago
Privacy policy summaries + q&a would be a nice public service. Here's summary of this one: > Personal data collected includes user names, text or image prompts, IP addresses, contact information, and more. The data is used for various purposes, such as providing and improving the service, managing user accounts, contacting users, and analyzing data. Personal data may be shared with service providers, third-party vendors, and other business partners. The company retains personal data for as long as necessary and transfers it to locations where data protection laws may differ. The security of personal data is important, although no method of transmission over the internet is 100% secure. The service is not intended for children under the age of 13, and the company does not knowingly collect their personal information. The privacy policy also includes information about links to other websites, changes to the policy, and supplemental terms and conditions for certain regions. For any questions or to exercise privacy rights, users can contact the company via email. Generated here: https://dstill.ai/hackernews/item/37436498 https://dstill.ai/hackernews/item/37436498
phnofive 3y ago
Any particular problems? Just linking to their policy is a bit confusing. You might consider submitting this as a question and linking to their policy in your summary.
- behnamoh 3y ago
  Read the data collection part: ``` 2. Collecting and Using Your Personal Data 2.1 Types of Data Collected 2.1.1 Personal Data While interacting with the Services, You may provide certain personally identifiable information that could be used to contact or identify You. Personally identifiable information may include, but is not limited to: Your user name for the Services Text or image prompts you input into the Services, or public chats you maintain on the service Your IP address Usage Data Tracking Technologies and Cookies Contact Information Organizational Information like your company title Your Email Cookies Other data that you elect to send to the Company ```
  - yuliyp 3y ago
    That's the normal way a privacy policy is worded. They start by describing the data they get from you, then they describe how that data is used. Obviously you're sending your prompts to them, so they note that and how they'll refer to that data in the rest of their terms. The thing to pay attention to is how they describe the uses and sharing of that data. And yeah, those sections are super-broad amounting to "we can use all of your personal data however we want" (including "improve our Service" and a super wide "for other purposes").
skilled 3y ago
This reads like something that a site using AdSense would use. 2.5 is interesting > Your information, including Personal Data, is processed at the Company's operating offices and in any other places where the parties involved in the processing are located. It means that this information may be transferred to — and maintained on — computers located outside of Your state, province, country or other governmental jurisdiction where the data protection laws may differ from those in Your jurisdiction > Your consent to this Privacy Policy followed by Your submission of such information represents Your agreement to that transfer. And the table in the middle of the policy... > Category Collected? Disclosed? > Identifiers. Name, alias, postal address, unique personal identifier, online identifier, Internet Protocol (IP) address, email address, account name, or other similar identifiers YES YES > Commercial information. Records of personal property, products or services purchased, obtained, or considered, or other purchasing or consuming histories or tendencies. YES YES also interesting... Invested in / partnered with NVIDIA & Hugging Face and others...
- cjonas 3y ago
  I was just talking to a colleague about the need for a service like this. Their pricing is much cheaper than self-hosting on AWS (ml.g5.48xlarge, needed to run 70B Lamma2, is $20.36/hr). But the TOS makes it completely unviable. If someone can figure out how to standup self-hostable, fine-tuneable models with a serverless architecture, it could be a game changer (allowing devs to use something other than OpenAI/Claude).
  - yessen 3y ago
    We offer Llama-2-70B-chat api[0] at Deep Infra. $1 per 1M tokens, support streaming and OpenAI compatible endpoint[1]. 0. https://deepinfra.com/meta-llama/Llama-2-70b-chat-hf https://deepinfra.com/meta-llama/Llama-2-70b-chat-hf 1. https://deepinfra.com/docs/advanced/openai_api https://deepinfra.com/docs/advanced/openai_api
    - cjonas 3y ago
      Nice, awesome that you can just repoint an OpenAI client to this service. I'll give it a try. Do you have future plans to support fine-tuning via pay-per-token billing? Related: Found an interesting Reddit post (rare) discussing how OpenAI manages to do this by loading the LoRA layer at runtime. https://www.reddit.com/r/MachineLearning/comments/15zl7ul/d_how_do_you_think_open_ai_hosts_all_these_fine/ https://www.reddit.com/r/MachineLearning/comments/15zl7ul/d_...
coder543 3y ago
I can't recall hearing about this website before. In fact, only one link to this website has ever gained any traction here[0], and it wasn’t much. Why is it surprising that a random website offering AI services might not have a great privacy policy? I feel like there must be dozens of websites like this that offer hosting of Llama2 by now. My default assumption would be that they’re not as trustworthy as OpenAI, who has a lot to lose, where smaller companies have basically nothing to lose by comparison. If privacy is a primary concern, companies have to work to earn that trust. [0]: https://news.ycombinator.com/item?id=36916762 https://news.ycombinator.com/item?id=36916762
- vikp 3y ago
  This is not a random website - together is a prominent AI research company. Their chief scientist invented flashattention, which is used to train most LLMs. They release open source datasets/models like RedPajama. And they've published research around speeding up model training and newer model architectures.
- jofi1 3y ago
  I don't have any affiliation with them, but Together is hardly random and very important to the open source community, putting out some of the most useful work around things like new architectures, distributed training, datasets.
agnokapathetic 3y ago
It's quite easy to opt-out of OpenAI using your data for training. API and ChatGPT Enterprise are private by default. # OpenAI Products and Privacy - ChatGPT Free - trains on your data unless you Opt Out - ChatGPT Plus - trains on your data unless you Opt Out - ChatGPT Enterprise - does _not_ train on your data - OpenAI API - does _not_ train on your data Opt-out link: https://help.openai.com/en/articles/5722486-how-your-data-is-used-to-improve-model-performance#:~:text=can%20submit%20this%20form https://help.openai.com/en/articles/5722486-how-your-data-is...
- _boffin_ 3y ago
  It’s not just training, but data retention and the ability for employees to access the prompts and completions. You only touch half the problem
- behnamoh 3y ago
  I had no idea! Just submitted the form. That said, the Enterprise version of ChatGPT is not easy to get by individuals. Edit: the wording is confusing: "fill out this form to opt-in to share..." might lead the user to think they must submit this form to opt "out".

54 points by behnamoh 3y ago

Better privacy might be running Llama2 locally, offline. If somebody hasn't tried running LLMs yet, here are some lines that do the job in Google Colab or locally. The !s are for Colab, remove them for local terminal. The script downloads the ca. 8GB model, but Llama.cpp can run offline afterwards. ! git clone https://github.com/ggerganov/llama.cpp.git ! wget "https://huggingface.co/TheBloke/CodeLlama-7B-GGUF/resolve/main/codellama-7b.Q8_0.gguf" -P llama.cpp/models ! cd llama.cpp && make ! ./llama.cpp/main -m ./llama.cpp/models/codellama-7b.Q8_0.gguf --color --ctx_size 2048 -n -1 -ins -b 256 --top_k 10000 --temp 0.2 --repeat_penalty 1.1 -t 8

lardo 3y ago
https://huggingface.co/TheBloke/CodeLlama-7B-GGUF https://huggingface.co/TheBloke/CodeLlama-7B-GGUF describes codellama-7b.Q8_0.gguf as "very large, extremely low quality loss - not recommended"
- jmorgan 3y ago
  Q4_0 is often mentioned as being the "tried and true" quantization level to try first. I've heard folks have had good results with 3-bit quantization (Q3_K_M) as well
  - jsemrau 3y ago
    just tried it and its not good. I think one of the better feedback was asking for play an adventure in the style of Space Quest 1
Patrick_Devine 3y ago
Or, for the easy way of doing it, just install Ollama and run: `ollama run codellama:7b`. If you want the q8 version (which you probably don't), it's `ollama run codellama:7b-code-q8_0`.
rvz 3y ago
Or if you're on macOS just hit 'Download' and use Ollama locally. [0] [0] https://ollama.ai/download https://ollama.ai/download

edgoode 3y ago

You can run models at https://shadeform.ai https://shadeform.ai in your cloud accounts or ours, and we don’t track or keep anything.

Palmik 3y ago

Privacy policy summaries + q&a would be a nice public service. Here's summary of this one: > Personal data collected includes user names, text or image prompts, IP addresses, contact information, and more. The data is used for various purposes, such as providing and improving the service, managing user accounts, contacting users, and analyzing data. Personal data may be shared with service providers, third-party vendors, and other business partners. The company retains personal data for as long as necessary and transfers it to locations where data protection laws may differ. The security of personal data is important, although no method of transmission over the internet is 100% secure. The service is not intended for children under the age of 13, and the company does not knowingly collect their personal information. The privacy policy also includes information about links to other websites, changes to the policy, and supplemental terms and conditions for certain regions. For any questions or to exercise privacy rights, users can contact the company via email. Generated here: https://dstill.ai/hackernews/item/37436498 https://dstill.ai/hackernews/item/37436498

phnofive 3y ago

Any particular problems? Just linking to their policy is a bit confusing. You might consider submitting this as a question and linking to their policy in your summary.

behnamoh 3y ago
Read the data collection part: ``` 2. Collecting and Using Your Personal Data 2.1 Types of Data Collected 2.1.1 Personal Data While interacting with the Services, You may provide certain personally identifiable information that could be used to contact or identify You. Personally identifiable information may include, but is not limited to: Your user name for the Services Text or image prompts you input into the Services, or public chats you maintain on the service Your IP address Usage Data Tracking Technologies and Cookies Contact Information Organizational Information like your company title Your Email Cookies Other data that you elect to send to the Company ```
- yuliyp 3y ago
  That's the normal way a privacy policy is worded. They start by describing the data they get from you, then they describe how that data is used. Obviously you're sending your prompts to them, so they note that and how they'll refer to that data in the rest of their terms. The thing to pay attention to is how they describe the uses and sharing of that data. And yeah, those sections are super-broad amounting to "we can use all of your personal data however we want" (including "improve our Service" and a super wide "for other purposes").

skilled 3y ago

This reads like something that a site using AdSense would use. 2.5 is interesting > Your information, including Personal Data, is processed at the Company's operating offices and in any other places where the parties involved in the processing are located. It means that this information may be transferred to — and maintained on — computers located outside of Your state, province, country or other governmental jurisdiction where the data protection laws may differ from those in Your jurisdiction > Your consent to this Privacy Policy followed by Your submission of such information represents Your agreement to that transfer. And the table in the middle of the policy... > Category Collected? Disclosed? > Identifiers. Name, alias, postal address, unique personal identifier, online identifier, Internet Protocol (IP) address, email address, account name, or other similar identifiers YES YES > Commercial information. Records of personal property, products or services purchased, obtained, or considered, or other purchasing or consuming histories or tendencies. YES YES also interesting... Invested in / partnered with NVIDIA & Hugging Face and others...

cjonas 3y ago
I was just talking to a colleague about the need for a service like this. Their pricing is much cheaper than self-hosting on AWS (ml.g5.48xlarge, needed to run 70B Lamma2, is $20.36/hr). But the TOS makes it completely unviable. If someone can figure out how to standup self-hostable, fine-tuneable models with a serverless architecture, it could be a game changer (allowing devs to use something other than OpenAI/Claude).
- yessen 3y ago
  We offer Llama-2-70B-chat api[0] at Deep Infra. $1 per 1M tokens, support streaming and OpenAI compatible endpoint[1]. 0. https://deepinfra.com/meta-llama/Llama-2-70b-chat-hf https://deepinfra.com/meta-llama/Llama-2-70b-chat-hf 1. https://deepinfra.com/docs/advanced/openai_api https://deepinfra.com/docs/advanced/openai_api
  - cjonas 3y ago
    Nice, awesome that you can just repoint an OpenAI client to this service. I'll give it a try. Do you have future plans to support fine-tuning via pay-per-token billing? Related: Found an interesting Reddit post (rare) discussing how OpenAI manages to do this by loading the LoRA layer at runtime. https://www.reddit.com/r/MachineLearning/comments/15zl7ul/d_how_do_you_think_open_ai_hosts_all_these_fine/ https://www.reddit.com/r/MachineLearning/comments/15zl7ul/d_...

coder543 3y ago

I can't recall hearing about this website before. In fact, only one link to this website has ever gained any traction here[0], and it wasn’t much. Why is it surprising that a random website offering AI services might not have a great privacy policy? I feel like there must be dozens of websites like this that offer hosting of Llama2 by now. My default assumption would be that they’re not as trustworthy as OpenAI, who has a lot to lose, where smaller companies have basically nothing to lose by comparison. If privacy is a primary concern, companies have to work to earn that trust. [0]: https://news.ycombinator.com/item?id=36916762 https://news.ycombinator.com/item?id=36916762

vikp 3y ago
This is not a random website - together is a prominent AI research company. Their chief scientist invented flashattention, which is used to train most LLMs. They release open source datasets/models like RedPajama. And they've published research around speeding up model training and newer model architectures.
jofi1 3y ago
I don't have any affiliation with them, but Together is hardly random and very important to the open source community, putting out some of the most useful work around things like new architectures, distributed training, datasets.

agnokapathetic 3y ago

It's quite easy to opt-out of OpenAI using your data for training. API and ChatGPT Enterprise are private by default. # OpenAI Products and Privacy - ChatGPT Free - trains on your data unless you Opt Out - ChatGPT Plus - trains on your data unless you Opt Out - ChatGPT Enterprise - does _not_ train on your data - OpenAI API - does _not_ train on your data Opt-out link: https://help.openai.com/en/articles/5722486-how-your-data-is-used-to-improve-model-performance#:~:text=can%20submit%20this%20form https://help.openai.com/en/articles/5722486-how-your-data-is...

_boffin_ 3y ago
It’s not just training, but data retention and the ability for employees to access the prompts and completions. You only touch half the problem
behnamoh 3y ago
I had no idea! Just submitted the form. That said, the Enterprise version of ChatGPT is not easy to get by individuals. Edit: the wording is confusing: "fill out this form to opt-in to share..." might lead the user to think they must submit this form to opt "out".