4 ms·

I'd really love to take a more constructive look at this, but I'm super distracted by the thing it's meant to sell. > We are building AICert, an open-source to

by jchw 3y ago

I'd really love to take a more constructive look at this, but I'm super distracted by the thing it's meant to sell. > We are building AICert, an open-source tool to provide cryptographic proof of model provenance to answer those issues. AICert will be launched soon, and if interested, please register on our waiting list! Hello. Fires are dangerous. Here is how fire burns down a school. Thankfully, we've invented a fire extinguisher. > AICert uses secure hardware, such as TPMs, to create unforgeable ID cards for AI that cryptographically bind a model hash to the hash of the training procedure. > secure hardware, such as TPMs "such as"? Why the uncertainty? So OK. It signs stuff using a TPM of some sort (probably) based on the model hash. So... When and where does the model hash go in? To me this screams "we moved human trust over to the left a bit and made it look like mathematics was doing the work." Let me guess, the training still happens on ordinary GPUs...? It's also "open source". Which part of it? Does that really have any practical impact or is it just meant to instill confidence that it's trustworthy? I'm genuinely unsure. Am I completely missing the idea? I don't think trust in LLMs is all that different from trust in code typically is. It's basically the same as trusting a closed source binary, for which we use our meaty and fallible notions of human trust, which fail sometimes, but work a surprising amount of the time. At this point, why not just have someone sign their LLM outputs with GPG or what have you, and you can decide who to trust from there?

DanyWin 3y ago
There is still a design decision to be made on whether we go for TPMs for integrity only, or go for more recent solutions like Confidential GPUs with H100s, that have both confidentiality and integrity. The trust chain is also different, that is why we are not committing yet. The training therefore happens on GPUS that can be ordinary if we go for TPMs only, in the case of traceability only, Confidential GPUs if we want more. We will make the whole code source open source, which will include the base image of software, and the code to create the proofs using the secure hardware keys to sign that the hash of a specific model comes from a specific training procedure. Of course it is not a silver bullet. But just like signed and audited closed source, we can have parties / software assess the trustworthiness of a piece of code, and if it passes, sign that it answers some security requirements. We intend to do the same thing. It is not up to us to do this check, but we will let the ecosystem do it. Here we focus more on providing tools that actually link the weights to a specific training / audit. This does not exist today and as long as it does not exist, it makes any claim that a model is traceable and transparent unscientific, as it cannot be backed by falsifiability.
- catiopatio 3y ago
  Why does this matter at all?
  - nebulousthree 3y ago
    You go to a jewelry store to buy gold. The salesperson tells you that the piece you want is 18karat gold, and charges you accordingly. How can you confirm the legitimacy of the 18k claim? Both 18k and 9k look just as shiny and golden to your untrained eye. You need a tool and the expertise to be able to tell, so you bring your jeweler friend along to vouch for it. No jeweler friend? Maybe the salesperson can convince you by showing you a certificate of authenticity from a source you recognize. Now replace the gold with a LLM.
    - freeone3000 3y ago
      Why should we trust your certificate more than it looking shiny? What exactly are you certifying and why should we believe you about it?
      - nebulousthree 3y ago
        You shouldn't trust any old certificate more than it looking shiny. But if a third party that you recognise and trust happens to recognise the jewelry or the jeweler themselves, and goes so far as to issue a certificate attesting to that, that becomes another piece of evidence to consider in your decision to purchase.
        ethbr0 3y ago
        Art and antiquities are the better analogy. Anything without an iron-clad chain of provenance should be assumed to be stolen or forged. Because the end product is unprovably authentic in all cases, unless a forger made a detectable error.
    - scrps 3y ago
      If my reading of it is correct this is similar to something like a trusted bootchain where every step is cryptographically verified against the chain and the components. In plain english the final model you load and all the components used to generate that model can be cryptographically verified back to whomever trained it and if any part of that chain can't be verified alarm bells go off, things fail, etc. Someone please correct me if my understanding is off. Edit: typo
samtho 3y ago
> Am I completely missing the idea? I don't think trust in LLMs is all that different from trust in code typically is. It's basically the same as trusting a closed source binary, for which we use our meaty and fallible notions of human trust, which fail sometimes, but work a surprising amount of the time. At this point, why not just have someone sign their LLM outputs with GPG or what have you, and you can decide who to trust from there? This has been my problem with LLMs from day one. Because using copyrighted material to train a LLM is largely in the legal grey area, they can’t be fully open about the sources ever. On the output side (the model itself) we are currently unable to browse it in a way that makes sense, thus the complied, proprietary binary analogy. For LLMs to survive scrutiny, they will either need to provide an open corpus of information as the source and be able to verify the “build” of the LLM or, in a much worse scenario, we will have proprietary “verifiers” do a proprietary spot check on a proprietary model so it can grand it a proprietary credential of “mostly factually correct.” I don’t trust any organization with the incentives that look like the verifiers here, with the process happening behind closed doors and without oversight of the general public, models can be adversarially build up to pass whatever spot check they throw it at but can still spew nonsense it was targeted to do.
- circuit10 3y ago
  > Because using copyrighted material to train a LLM is largely in the legal grey area, they can’t be fully open about the sources ever. I don’t think that’s true, for example some open source LLMs have the training data publicly available, and hiding evidence of something you think could be illegal on purpose sounds too risky for most big companies to do (obviously that happens sometimes but I don’t think it would on that scale)
  - samtho 3y ago
    While there may be some, the most notable ones seem to hide behind the veil of “proprietary training data” but assuming the data is open, the method to generate the model must also be reproducible, thus the toolchain need to be open too. I don’t think there is a lot of incentive to do this.
Retr0id 3y ago
This seems like a classic example of "I have solved the problem by mapping it onto a domain that I do not understand"
ryukoposting 3y ago
Was thinking the same thing - not sure what this accomplishes that couldn't already be done with a GPG signature on a .safetensors file.
jstarfish 3y ago
> Hello. Fires are dangerous. Here is how fire burns down a school. Thankfully, we've invented a fire extinguisher. Heh. Shakedowns are a legitimate way of doing business these days. Invent the threat, sell the solution. Sidestory: I'm convinced the weird "audio glitch" that hit American Airlines in 2022-09 was the work of a cybersecurity firm trying to drum up business for themselves. Their CEO (hello, David) had just a few months earlier personally submitted to AA's CEO a vaguely-worded and entirely-unverifiable incident report suggesting American's inflight wifi provider's payment portal or something had been compromised by The Chinese-- and blamed an unnamed flight attendant for destroying all evidence by forcing him to immediately shut down his laptop. So no evidence, no screenshots, no artifacts verifying he was even on that flight, implied involvement of foreign boogeymen, adverse action taken by malicious/anonymous witnesses, and when pressed for technical details, the reporter dodged questions and feigned ignorance (when asked for his MAC address, he returned one for a virtual adapter and stopped responding). A few months later, AA has a public PA system incident that perplexed everyone and gets attributed to vague "mechanical failure." Could be coincidence, but everything about the former incident screamed of a cybersecurity vendor chasing sales by sowing unverifiable FUD in bad faith. I don't put it past them to engage in "harmless" sabotage.
throwawaaarrgh 3y ago
> It's also "open source". Which part of it? Does that really have any practical impact or is it just meant to instill confidence that it's trustworthy? It means two things: 1) the founders are idealistic techies who like the idea of open source and want to make money off it, 2) they're trying to sell it to other idealistic techie founders B2B. You don't mention things in an elevator pitch unless someone's looking to buy it.

by jchw 3y ago

DanyWin 3y ago

There is still a design decision to be made on whether we go for TPMs for integrity only, or go for more recent solutions like Confidential GPUs with H100s, that have both confidentiality and integrity. The trust chain is also different, that is why we are not committing yet. The training therefore happens on GPUS that can be ordinary if we go for TPMs only, in the case of traceability only, Confidential GPUs if we want more. We will make the whole code source open source, which will include the base image of software, and the code to create the proofs using the secure hardware keys to sign that the hash of a specific model comes from a specific training procedure. Of course it is not a silver bullet. But just like signed and audited closed source, we can have parties / software assess the trustworthiness of a piece of code, and if it passes, sign that it answers some security requirements. We intend to do the same thing. It is not up to us to do this check, but we will let the ecosystem do it. Here we focus more on providing tools that actually link the weights to a specific training / audit. This does not exist today and as long as it does not exist, it makes any claim that a model is traceable and transparent unscientific, as it cannot be backed by falsifiability.

catiopatio 3y ago
Why does this matter at all?
- nebulousthree 3y ago
  You go to a jewelry store to buy gold. The salesperson tells you that the piece you want is 18karat gold, and charges you accordingly. How can you confirm the legitimacy of the 18k claim? Both 18k and 9k look just as shiny and golden to your untrained eye. You need a tool and the expertise to be able to tell, so you bring your jeweler friend along to vouch for it. No jeweler friend? Maybe the salesperson can convince you by showing you a certificate of authenticity from a source you recognize. Now replace the gold with a LLM.
  - freeone3000 3y ago
    Why should we trust your certificate more than it looking shiny? What exactly are you certifying and why should we believe you about it?
    - nebulousthree 3y ago
      You shouldn't trust any old certificate more than it looking shiny. But if a third party that you recognise and trust happens to recognise the jewelry or the jeweler themselves, and goes so far as to issue a certificate attesting to that, that becomes another piece of evidence to consider in your decision to purchase.
      - ethbr0 3y ago
        Art and antiquities are the better analogy. Anything without an iron-clad chain of provenance should be assumed to be stolen or forged. Because the end product is unprovably authentic in all cases, unless a forger made a detectable error.
  - scrps 3y ago
    If my reading of it is correct this is similar to something like a trusted bootchain where every step is cryptographically verified against the chain and the components. In plain english the final model you load and all the components used to generate that model can be cryptographically verified back to whomever trained it and if any part of that chain can't be verified alarm bells go off, things fail, etc. Someone please correct me if my understanding is off. Edit: typo

samtho 3y ago

> Am I completely missing the idea? I don't think trust in LLMs is all that different from trust in code typically is. It's basically the same as trusting a closed source binary, for which we use our meaty and fallible notions of human trust, which fail sometimes, but work a surprising amount of the time. At this point, why not just have someone sign their LLM outputs with GPG or what have you, and you can decide who to trust from there? This has been my problem with LLMs from day one. Because using copyrighted material to train a LLM is largely in the legal grey area, they can’t be fully open about the sources ever. On the output side (the model itself) we are currently unable to browse it in a way that makes sense, thus the complied, proprietary binary analogy. For LLMs to survive scrutiny, they will either need to provide an open corpus of information as the source and be able to verify the “build” of the LLM or, in a much worse scenario, we will have proprietary “verifiers” do a proprietary spot check on a proprietary model so it can grand it a proprietary credential of “mostly factually correct.” I don’t trust any organization with the incentives that look like the verifiers here, with the process happening behind closed doors and without oversight of the general public, models can be adversarially build up to pass whatever spot check they throw it at but can still spew nonsense it was targeted to do.

circuit10 3y ago
> Because using copyrighted material to train a LLM is largely in the legal grey area, they can’t be fully open about the sources ever. I don’t think that’s true, for example some open source LLMs have the training data publicly available, and hiding evidence of something you think could be illegal on purpose sounds too risky for most big companies to do (obviously that happens sometimes but I don’t think it would on that scale)
- samtho 3y ago
  While there may be some, the most notable ones seem to hide behind the veil of “proprietary training data” but assuming the data is open, the method to generate the model must also be reproducible, thus the toolchain need to be open too. I don’t think there is a lot of incentive to do this.

Retr0id 3y ago

This seems like a classic example of "I have solved the problem by mapping it onto a domain that I do not understand"

ryukoposting 3y ago

Was thinking the same thing - not sure what this accomplishes that couldn't already be done with a GPG signature on a .safetensors file.

jstarfish 3y ago

> Hello. Fires are dangerous. Here is how fire burns down a school. Thankfully, we've invented a fire extinguisher. Heh. Shakedowns are a legitimate way of doing business these days. Invent the threat, sell the solution. Sidestory: I'm convinced the weird "audio glitch" that hit American Airlines in 2022-09 was the work of a cybersecurity firm trying to drum up business for themselves. Their CEO (hello, David) had just a few months earlier personally submitted to AA's CEO a vaguely-worded and entirely-unverifiable incident report suggesting American's inflight wifi provider's payment portal or something had been compromised by The Chinese-- and blamed an unnamed flight attendant for destroying all evidence by forcing him to immediately shut down his laptop. So no evidence, no screenshots, no artifacts verifying he was even on that flight, implied involvement of foreign boogeymen, adverse action taken by malicious/anonymous witnesses, and when pressed for technical details, the reporter dodged questions and feigned ignorance (when asked for his MAC address, he returned one for a virtual adapter and stopped responding). A few months later, AA has a public PA system incident that perplexed everyone and gets attributed to vague "mechanical failure." Could be coincidence, but everything about the former incident screamed of a cybersecurity vendor chasing sales by sowing unverifiable FUD in bad faith. I don't put it past them to engage in "harmless" sabotage.

throwawaaarrgh 3y ago

> It's also "open source". Which part of it? Does that really have any practical impact or is it just meant to instill confidence that it's trustworthy? It means two things: 1) the founders are idealistic techies who like the idea of open source and want to make money off it, 2) they're trying to sell it to other idealistic techie founders B2B. You don't mention things in an elevator pitch unless someone's looking to buy it.