4 ms·

So if I get access to the Perplexity AI source code (I borrow it from a friend), read all of it, and reproduce it at some level, then Perplexity will be:" sure,

So if I get access to the Perplexity AI source code (I borrow it from a friend), read all of it, and reproduce it at some level, then Perplexity will be:" sure, that's fine no harm, no IP theft, no copyright violation, because you read it so we're good"? No, they would sue me for everything I got, and then some. That's the weird thing about these companies, they are never afraid to use IP law to go after others, but those same laws don't apply to them... because? Just pay the stupid license and if that makes your business unsustainable then it's not much a business is it?

bshacklett 2y ago
If Perplexity’s source code is downloaded from a public web site or other repository, and you take the time to understand the code and produce your own novel implementation, then yes. Now, if you “get it from a friend”, illegally, _or_ you just redeploy the code, without creating a transformative work, then there’s a problem. > Just pay the stupid license and if that makes your business unsustainable then it's not much a business is it? In the persona of a business owner, why pay for something that you don’t legally, need to pay for? The question of how copyright applies to LLMs and other AI is still open. They’d be fools to buy licenses before it’s been decided. More importantly, we’re potentially talking about the entire knowledge of humanity being used in training. There’s no-one on earth with that kind of money. Sure, you can just say that the business model doesn’t work, but we’re discussing new technologies that have real benefit to humanity, and it’s not just businesses that are training models this way. Any decision which hinders businesses from developing models with this data will hinder independent researchers 10 fold, so it’s important that we’re careful about what precedent is set in the name of punishing greedy businessmen.
- flutas 2y ago
  > If Perplexity’s source code is downloaded from a public web site or other repository, and you take the time to understand the code and produce your own novel implementation, then yes. Even that can be considered infringement and get you taken to court. It's one of the reasons reading leaked code is considered bad and you hear terms like cleanroom[0] when discussing reproductions of products. [0]: https://en.wikipedia.org/wiki/Clean_room_design https://en.wikipedia.org/wiki/Clean_room_design
  - bshacklett 2y ago
    It certainly can be, but it's not guaranteed. Clean room design is one way to avoid a legally ambiguous situation. It's not a hard requirement to avoid infringement. For example, the US Supreme Court ruled that Google's use of the Java APIs fell under fair use. My point is: just because certain source material was used in the making of another work does not guarantee that it's infringing on the rights of that original IP.
- mrweasel 2y ago
  > They’d be fools to buy licenses before it’s been decided. They are willingly ignoring licenses until someone sues them? That's still illegal and completely immoral. There is tons of data to train on. The entirety of Wikipedia, all of StackOverflow (at least previously), all of the BSD and MIT licenses source code on Github, the entire Gutenberg project. So much stuff, freely and legally available, yet their feel that they don't need to check licenses?
  - bshacklett 2y ago
    The legality of their behavior is not currently well defined, because it's unprecedented. Fair use permits transformative works. It has yet to be decided whether LLMs and their output qualify as transformative, or even if the training is capable of infringing copyright of an individual work in the first place if they're not reproducing it. In fact, there's a good amount of evidence which indicates that fair use _does_ apply, given how Google operates and what they've argued successfully (https://en.wikipedia.org/wiki/Perfect_10,_Inc._v._Amazon.com,_Inc.#Ninth_Circuit_opinion https://en.wikipedia.org/wiki/Perfect_10,_Inc._v._Amazon.com...). Purchasing licenses when you are already entitled to your current use of the work is just bad business, especially when the legal precedent hasn't been set to know what rights might need to exist in said license. You might not like the idea of your blog posts or other publicly posted materials being used to train LLMs, but that doesn't make it illegal (morality is subjective and I'm not about to argue one way or another). If it's really that much of a problem, you _do_ have the ability to remove your information from public accessibility, or otherwise protect it against LLM ingestion (IP restrictions, etc.). edit: I am not a lawyer (this is likely obvious to any lawyers out there); this is my personal take.
    - halostatue 2y ago
      Note that not all jurisdictions have the concept of "fair use" (use of copyrighted material, regardless of transformation applied, is permitted in certain contexts…ish). Canada, the UK, Australia, and other jurisdictions have "fair dealing" (use of copyrighted material depends on both reason and transformation applied…ish). Other jurisdictions have neither, and only licensed uses are permitted. Because the companies behind large models (diffusion, LLM, etc.) have consumed content created under non-US copyright laws and have presented it to people outside of US copyright law jurisdiction, they are likely liable for misapplication of fair dealing, even if the US ultimately deems what they have done as "fair use" (IMO this is unlikely because of the perfect reproduction problems that plague them all in different ways; there are likely to be the equivalent of trap streets that will make this clearly copyright violation on a large scale). It's worth noting that while models like GitHub Copilot "freely" use MIT, BSD (except BSD0), and Apache licensed software, they are likely violating the licenses every time a reasonable facsimile pops up because of the requirement to include copies of the licensing terms for full or partial distribution or derivation. It's almost as if wholesale copyright violations were the entire business model.
cpursley 2y ago
Funny enough, their prompts leaked: https://www.reddit.com/r/perplexity_ai/s/kn6i20kMLH https://www.reddit.com/r/perplexity_ai/s/kn6i20kMLH And I’ve built a perplexity clone in about a day - it’s not that hard: search -> scrape results -> parse results —> summarize results -> summarize aggregate results into single summary. I’m really not sure I even see their moat.
- emporas 2y ago
  What have you used if i may ask? It seems very simple indeed. What search API is best? Also there is a program called html2text to throw out the html formatting so as to use less tokens. Have you used this or something similar?
  - cpursley 2y ago
    Brave API (Bing is good as well). Here's a little gist (Elixir). It's pretty rudimentary so far and needs refining, but works alright enough (result at bottom): https://gist.github.com/cpursley/b4af2ff3b56c912f659bd5300e422790 https://gist.github.com/cpursley/b4af2ff3b56c912f659bd5300e4... The most useful part is probably the prompt and usage of Phi 3 Mini 128K Instruct for web page summarization and Llama 3 for the final summary (of the summaries). I'm parsing out all but minimal content html but might even remove that to keep context length down.
    - emporas 2y ago
      Very nice, thank you!

bshacklett 2y ago

If Perplexity’s source code is downloaded from a public web site or other repository, and you take the time to understand the code and produce your own novel implementation, then yes. Now, if you “get it from a friend”, illegally, _or_ you just redeploy the code, without creating a transformative work, then there’s a problem. > Just pay the stupid license and if that makes your business unsustainable then it's not much a business is it? In the persona of a business owner, why pay for something that you don’t legally, need to pay for? The question of how copyright applies to LLMs and other AI is still open. They’d be fools to buy licenses before it’s been decided. More importantly, we’re potentially talking about the entire knowledge of humanity being used in training. There’s no-one on earth with that kind of money. Sure, you can just say that the business model doesn’t work, but we’re discussing new technologies that have real benefit to humanity, and it’s not just businesses that are training models this way. Any decision which hinders businesses from developing models with this data will hinder independent researchers 10 fold, so it’s important that we’re careful about what precedent is set in the name of punishing greedy businessmen.

flutas 2y ago
> If Perplexity’s source code is downloaded from a public web site or other repository, and you take the time to understand the code and produce your own novel implementation, then yes. Even that can be considered infringement and get you taken to court. It's one of the reasons reading leaked code is considered bad and you hear terms like cleanroom[0] when discussing reproductions of products. [0]: https://en.wikipedia.org/wiki/Clean_room_design https://en.wikipedia.org/wiki/Clean_room_design
- bshacklett 2y ago
  It certainly can be, but it's not guaranteed. Clean room design is one way to avoid a legally ambiguous situation. It's not a hard requirement to avoid infringement. For example, the US Supreme Court ruled that Google's use of the Java APIs fell under fair use. My point is: just because certain source material was used in the making of another work does not guarantee that it's infringing on the rights of that original IP.
mrweasel 2y ago
> They’d be fools to buy licenses before it’s been decided. They are willingly ignoring licenses until someone sues them? That's still illegal and completely immoral. There is tons of data to train on. The entirety of Wikipedia, all of StackOverflow (at least previously), all of the BSD and MIT licenses source code on Github, the entire Gutenberg project. So much stuff, freely and legally available, yet their feel that they don't need to check licenses?
- bshacklett 2y ago
  The legality of their behavior is not currently well defined, because it's unprecedented. Fair use permits transformative works. It has yet to be decided whether LLMs and their output qualify as transformative, or even if the training is capable of infringing copyright of an individual work in the first place if they're not reproducing it. In fact, there's a good amount of evidence which indicates that fair use _does_ apply, given how Google operates and what they've argued successfully (https://en.wikipedia.org/wiki/Perfect_10,_Inc._v._Amazon.com,_Inc.#Ninth_Circuit_opinion https://en.wikipedia.org/wiki/Perfect_10,_Inc._v._Amazon.com...). Purchasing licenses when you are already entitled to your current use of the work is just bad business, especially when the legal precedent hasn't been set to know what rights might need to exist in said license. You might not like the idea of your blog posts or other publicly posted materials being used to train LLMs, but that doesn't make it illegal (morality is subjective and I'm not about to argue one way or another). If it's really that much of a problem, you _do_ have the ability to remove your information from public accessibility, or otherwise protect it against LLM ingestion (IP restrictions, etc.). edit: I am not a lawyer (this is likely obvious to any lawyers out there); this is my personal take.
  - halostatue 2y ago
    Note that not all jurisdictions have the concept of "fair use" (use of copyrighted material, regardless of transformation applied, is permitted in certain contexts…ish). Canada, the UK, Australia, and other jurisdictions have "fair dealing" (use of copyrighted material depends on both reason and transformation applied…ish). Other jurisdictions have neither, and only licensed uses are permitted. Because the companies behind large models (diffusion, LLM, etc.) have consumed content created under non-US copyright laws and have presented it to people outside of US copyright law jurisdiction, they are likely liable for misapplication of fair dealing, even if the US ultimately deems what they have done as "fair use" (IMO this is unlikely because of the perfect reproduction problems that plague them all in different ways; there are likely to be the equivalent of trap streets that will make this clearly copyright violation on a large scale). It's worth noting that while models like GitHub Copilot "freely" use MIT, BSD (except BSD0), and Apache licensed software, they are likely violating the licenses every time a reasonable facsimile pops up because of the requirement to include copies of the licensing terms for full or partial distribution or derivation. It's almost as if wholesale copyright violations were the entire business model.

cpursley 2y ago

Funny enough, their prompts leaked: https://www.reddit.com/r/perplexity_ai/s/kn6i20kMLH https://www.reddit.com/r/perplexity_ai/s/kn6i20kMLH And I’ve built a perplexity clone in about a day - it’s not that hard: search -> scrape results -> parse results —> summarize results -> summarize aggregate results into single summary. I’m really not sure I even see their moat.

emporas 2y ago
What have you used if i may ask? It seems very simple indeed. What search API is best? Also there is a program called html2text to throw out the html formatting so as to use less tokens. Have you used this or something similar?
- cpursley 2y ago
  Brave API (Bing is good as well). Here's a little gist (Elixir). It's pretty rudimentary so far and needs refining, but works alright enough (result at bottom): https://gist.github.com/cpursley/b4af2ff3b56c912f659bd5300e422790 https://gist.github.com/cpursley/b4af2ff3b56c912f659bd5300e4... The most useful part is probably the prompt and usage of Phi 3 Mini 128K Instruct for web page summarization and Llama 3 for the final summary (of the summaries). I'm parsing out all but minimal content html but might even remove that to keep context length down.
  - emporas 2y ago
    Very nice, thank you!