4 ms·

> Parallel inference reaches hundreds of tokens/sec. Marketing claims, meh. It gives normal people the wrong impression. You can’t parallelize your query beca

> Parallel inference reaches hundreds of tokens/sec. Marketing claims, meh. It gives normal people the wrong impression. You can’t parallelize your query because it’s sequential. I think people will be willing to wait the ~200 sec necessary to get 200 words, but it’s best to be up front about this limitation. Also abuse is a problem. Once 4chan realizes they can poison the distributed model, they’ll have a field day. But maybe it’s too much effort for too little reward that trolls won’t bother.

Etheryte 4y ago
There is no poisoning vector, you can only update prompts and adapters hosted locally.
- sillysaurusx 4y ago
  From the site: > you load a small part of the model, then team up with people serving the other parts to run inference or fine-tuning. If multiple people participate in a fine tuning session, you have to trust all of them. You also have to trust everybody for inference too, but at least one of them can’t scramble the model.
  - pshc 4y ago
    Wouldn't untrusted weird input increase loss and be rejected?
  - chatmasta 4y ago
    Maybe this could be solved with opt-in (or opt-out via banning) federation similar to Mastodon. Instead of one network you could have a bunch of different networks each focused on the interests of a different community. Or maybe as someone with a node, you could "subscribe" to different communities that use different filtering and prioritization mechanisms for task assignments. I do love the general direction, and I think it's inevitable that training will move to be more decentralized like this. It's also the best chance we have at disrupting the centralization of "Open"AI and their ilk. I say the earlier we figure this out, the better, but it's not an easy problem to solve cleanly. And, not to be that guy, but maybe we could add some cryptocurrency incentives to the mix... conveniently enough, the crypto miners already have the GPUs ready to go!
  - Etheryte 4y ago
    This is all covered in the docs if you click through past the landing page. If you want to propagate changes to others you need to set up your own swarm, you can't go tuning things on random participants. You can read more at: - https://github.com/bigscience-workshop/petals/wiki/Security,-privacy,-and-AI-safety https://github.com/bigscience-workshop/petals/wiki/Security,... - https://github.com/bigscience-workshop/petals/wiki/Launch-your-own-swarm https://github.com/bigscience-workshop/petals/wiki/Launch-yo...
  - gaogao 4y ago
    You could do attested code in an enclave, which though vulnerable to certain side channels, is probably more robust than the standard case.
robotnikman 4y ago
I think most of 4chan would only want to use it to talk with their anime waifu's
- jquery 4y ago
  That's rather wholesome. Unfortunately, 4chan is barely a Chinese cartoon board anymore, /pol/ seems to have the most external influence which reflects poorly on the whole site.
- chizhik-pyzhik 4y ago
  plural of waifu is wifi, actually
- decremental 4y ago
  It's more their style to get it to recite FBI crime statistics.
- all2 4y ago
  There is a GPT4chan floating around somewhere. Or maybe its Chat4Chan. I don't remember. I try to stay away from that poison.
rohfle 4y ago
From https://github.com/bigscience-workshop/petals/wiki/Security,-privacy,-and-AI-safety https://github.com/bigscience-workshop/petals/wiki/Security,... > Q: Does Petals guarantee that model outputs are correct? > Not by default. A faulty or malicious server could give you incorrect outputs. There are two things you can do about this: > - Verify outputs. Send some of your tensors to two or more peers and check that the answers match. > - Set up a private swarm. You can launch your own swarm hosted by people and organization you trust, who are authorized to process your data. > In future, we plan to implement an automatic verification and a reputation system, so that clients can select servers that they can trust.
- zekrioca 4y ago
  Byzantine problems allover again..
borzunov 4y ago
A Petals dev here. We say up front that "Single-batch inference runs at ≈ 1 sec per step (token)". In turn, "parallel inference" refers to the high-throughput scenario when you generate multiple sequences in parallel. This is useful when you process some large dataset with LLM (e.g. run inference with batch size of 200) or run a beam search with a large beam width. In this case, you can actually get the speed of hundreds of tokens per sec, see our benchmarks for parallel forward passes: https://github.com/bigscience-workshop/petals#benchmarks https://github.com/bigscience-workshop/petals#benchmarks If you have another wording in mind that is more up front, please let us know, we'd be happy to improve the project description. Petals is a non-commercial research project, and we don't want to oversell anything.
- null4bl3 4y ago
  Can it run in a docker-compose container with a set ressource limit? Do each node earn points for supplying resources that can then be spend for greater query / process speed?
  - borzunov 4y ago
    Sure! The point system is being developed, we'll ship it soon. Once it's ready, you'll be able to spend points on high-priority requests to increase the speed.

Etheryte 4y ago

There is no poisoning vector, you can only update prompts and adapters hosted locally.

sillysaurusx 4y ago
From the site: > you load a small part of the model, then team up with people serving the other parts to run inference or fine-tuning. If multiple people participate in a fine tuning session, you have to trust all of them. You also have to trust everybody for inference too, but at least one of them can’t scramble the model.
- pshc 4y ago
  Wouldn't untrusted weird input increase loss and be rejected?
- chatmasta 4y ago
  Maybe this could be solved with opt-in (or opt-out via banning) federation similar to Mastodon. Instead of one network you could have a bunch of different networks each focused on the interests of a different community. Or maybe as someone with a node, you could "subscribe" to different communities that use different filtering and prioritization mechanisms for task assignments. I do love the general direction, and I think it's inevitable that training will move to be more decentralized like this. It's also the best chance we have at disrupting the centralization of "Open"AI and their ilk. I say the earlier we figure this out, the better, but it's not an easy problem to solve cleanly. And, not to be that guy, but maybe we could add some cryptocurrency incentives to the mix... conveniently enough, the crypto miners already have the GPUs ready to go!
- Etheryte 4y ago
  This is all covered in the docs if you click through past the landing page. If you want to propagate changes to others you need to set up your own swarm, you can't go tuning things on random participants. You can read more at: - https://github.com/bigscience-workshop/petals/wiki/Security,-privacy,-and-AI-safety https://github.com/bigscience-workshop/petals/wiki/Security,... - https://github.com/bigscience-workshop/petals/wiki/Launch-your-own-swarm https://github.com/bigscience-workshop/petals/wiki/Launch-yo...
- gaogao 4y ago
  You could do attested code in an enclave, which though vulnerable to certain side channels, is probably more robust than the standard case.

robotnikman 4y ago

I think most of 4chan would only want to use it to talk with their anime waifu's

jquery 4y ago
That's rather wholesome. Unfortunately, 4chan is barely a Chinese cartoon board anymore, /pol/ seems to have the most external influence which reflects poorly on the whole site.
chizhik-pyzhik 4y ago
plural of waifu is wifi, actually
decremental 4y ago
It's more their style to get it to recite FBI crime statistics.
all2 4y ago
There is a GPT4chan floating around somewhere. Or maybe its Chat4Chan. I don't remember. I try to stay away from that poison.

rohfle 4y ago

From https://github.com/bigscience-workshop/petals/wiki/Security,-privacy,-and-AI-safety https://github.com/bigscience-workshop/petals/wiki/Security,... > Q: Does Petals guarantee that model outputs are correct? > Not by default. A faulty or malicious server could give you incorrect outputs. There are two things you can do about this: > - Verify outputs. Send some of your tensors to two or more peers and check that the answers match. > - Set up a private swarm. You can launch your own swarm hosted by people and organization you trust, who are authorized to process your data. > In future, we plan to implement an automatic verification and a reputation system, so that clients can select servers that they can trust.

zekrioca 4y ago
Byzantine problems allover again..

borzunov 4y ago

A Petals dev here. We say up front that "Single-batch inference runs at ≈ 1 sec per step (token)". In turn, "parallel inference" refers to the high-throughput scenario when you generate multiple sequences in parallel. This is useful when you process some large dataset with LLM (e.g. run inference with batch size of 200) or run a beam search with a large beam width. In this case, you can actually get the speed of hundreds of tokens per sec, see our benchmarks for parallel forward passes: https://github.com/bigscience-workshop/petals#benchmarks https://github.com/bigscience-workshop/petals#benchmarks If you have another wording in mind that is more up front, please let us know, we'd be happy to improve the project description. Petals is a non-commercial research project, and we don't want to oversell anything.

null4bl3 4y ago
Can it run in a docker-compose container with a set ressource limit? Do each node earn points for supplying resources that can then be spend for greater query / process speed?
- borzunov 4y ago
  Sure! The point system is being developed, we'll ship it soon. Once it's ready, you'll be able to spend points on high-priority requests to increase the speed.