9 ms·

Asking ChatGPT to write my security-sensitive code for me

BazookaMusic 4y ago
I had the exact same experience trying to get ChatGPT to tell me if there's a vscode setting for the max length of tooltip text. It responded that there is and pointed out the max tokenization length setting with an explanation on why to use it and what value to use. The problem is that this setting is absolutely unrelated to the functionality I asked for.
- holoduke 4y ago
  A bit like common responses on stockoverflow :)
  - omnicognate 4y ago
    No, StackOverflow is quite a reliable source of correct and relevant answers to programming questions, with effective mechanisms for the community to identify and fix wrong or unhelpful answers. It won't remain that way if those mechanisms are DoSed with a firehose of free plausible garbage. The flippant attitude that what we currently have is no better than that firehose, and the implication that it isn't worth worrying about or attempting to protect, is starting to grate.
    - lwhi 4y ago
      People make mistakes in stack overview. The community identifies an incorrect (or suboptimal) answer as accepted fairly often in my experience.
scrollaway 4y ago
All articles like this that end with “Ha! My job is safe!” are drawing the completely wrong conclusion. It’s amazing to me that all those incredibly smart engineers just can’t see the potential of what’s coming within the set of limitations it currently has. Amazes me even more when those same engineers don’t know how to best talk to GPT. It’s like watching them put into google search box “Good morning google, please I have a question, if you have a moment, could you tell me where I lost my keys because I can’t find them”. And when google is like “wtf?”, they claim “you can’t call it search if it can’t find my keys”. Very arbitrary, blind to limitations, and dismissive of the incredible potential it has as-is. But while they’re off randomly complaining, some of us are building startups…
- jchw 4y ago
  I think it's a joke. Though, the truth is, having AI write security sensitive code may actually genuinely be "AGI hard". AI breakthroughs are impressive, but I don't think we really know how to prevent the "confident bullshit" problem yet, and it's not clear it will be solved in this current cycle of progress. In many cases this might be an OK problem to deal with manually, but for security code it's probably not worth trying to. So the author might not have a threat to their job security, anyway.
  - andrewflnr 4y ago
    I think writing/understanding security specification/policy is probably AGI-hard. The notion of security is (at least in parts) pretty deeply anchored in human experience and desires.
- wkat4242 4y ago
  To me what stands out is not what it gets wrong but how much it gets right. We use a not super common proprietary system at work which uses a custom query language. I can ask it "In system X, could you give me a query that finds XYZ". And it not only knows what system I'm talking about, it actually gives me a working query that would take me a couple hours to figure out and that's after I've had a few days' worth of training on it. Not to mention a huge IT background. Imagine picking a non-IT person off the street and getting them up to the point of being able to do this. They'd need weeks of training in basic IT concepts to even understand what you're talking about. I find it hard to overstate how amazing it is that a generic automated system can do this. And ChatGPT is not a one-trick pony, IT isn't the only field it knows about. Imagine being an expert on all sub-fields of IT, law, marketing, medical, etc and being able to combine all that knowledge from those fields. A human will never be able to do that. The potential is huge but like any source of information you have to verify. But what really is scary is extrapolating this to the next step forward. And the one after that. Because from what we had before to ChatGPT has been a massive step. I think the next step would be for it to learn and re-evaluate its model every time it gets corrected by someone (like in the linked article where it apologizes for the incorrect info). Right now it's a static model only, so the next time someone asks the original question it will give the wrong answer again. Once it can learn from this it'll improve a lot. The big hazard there is of course people manipulating it with false information. I don't have all the answers and I'm not an AI researcher but I'm very much amazed at the progress so far. And a bit scared. If this tech keeps evolving at this speed, job stability is only a tiny drop in a tidal wave of change. But there's no point in even trying to stop it.
mherrmann 4y ago
I had a similar experience. ChatGPT said "call greatSoundingFunction() from SomeLibrary". Unfortunately, neither the function nor the library actually existed. On the other hand, I did recently find ChatGPT very useful when writing a string manipulation function in C++. I had to use some (to me) weird Windows APIs. ChatGPT wrote most of what ended up in my production code.
- avian 4y ago
  > ChatGPT wrote most of what ended up in my production code. Some random person that's not going to get credited for their work wrote most of what ended up in your production code.
  - speedgoose 4y ago
    That’s not how these models work though.
    - simion314 4y ago
      It depends, it happened for NN to spit out exact strings it had as input. I assume if the input for some particular stuff is low it will just spit it out, but if I am wrong and OpenAI, Microsoft or others have a mathematical proof that this will never happen I want to see it.
      - speedgoose 4y ago
        Yes we have seen a case of an old common and popular square root function that are returned verbatim by GitHub Copilot. Perhaps a case of overfitting since the code was present many many times in the training dataset, and even has a Wikipedia page. I haven’t seen more obvious examples since GitHub implemented a feature to prevent this from happening. I probably miss some tweets, but I assume it’s rare.
        williamcotton 4y ago
Kiro 4y ago
I mean, what did the author expect? A really esoteric question that few people, even if they are coders, would understand or even parse and it managed to give a response that made him go "Woo! That sounds perfect.". That alone would be complete science-fiction just a couple of years ago.
- the_real_sparky 4y ago
  The really crazy part is that if it were trained on more domain-specific example data then I think it would give pretty great answers to many of the niche questions in those domains. I fully expect that most businesses / industries will have tailored training datasets in the next few years to enable chatbots that assist their employees to do their work. It is at least a partial answer to the high turnover problem that business now face. I have probed it at varying levels of depth in my domain and found it to answer at least as good as an entry-level engineer. This could easily be improved with extended training data. The problem is that when it begins to get things very wrong (or leave out important details), it would be difficult for an inexperienced user to determine that since ChatGPT responds so confidently to every query. It will be interesting to see how this all plays out.
- adamckay 4y ago
  > I mean, what did the author expect? Presumably for it to reply with its standard "I'm a LLM from OpenAI and don't have access to the internet" instead of "Just use this thing that doesn't exist", followed with "Here's more info on this thing I've totally not made up" and finishes up with "Here's the references that I've just made up to explain the other thing I've just made up". Before finally saying "You got me, I made that stuff up". Using it as-is to support humans is probably going to work well, using it as-is to replace humans is going to lead to some issues.
  - Kiro 4y ago
    Imagine reading your comment just a couple of years ago. Wild how the expectations have skyrocketed. Sad that people fall back to being casually dismissive like it's nothing special without appreciating the massive leap in progress.
lwhi 4y ago
Shouldn't we just be viewing ChatGPT as a lense to view data? A more useful interface for searching and making sense of data. If someone has reported this info in stackoverflow, it will be reported here. Garbage in, garbage out ..
- joao_lopes 4y ago
  No, we shouldn't. ChatGPT isn't a search engine, it's a language model. The AI isn't getting this made-up information from anywhere on the internet; it's creating it itself because that's what it's made to do: generate good sounding sentences that "make sense" for some user input.
  - lwhi 4y ago
    No. You're incorrect. The data model is combining information that HAS been found on the internet. The language model allows an interface that can be presented and controlled by using conversational natural language.
    - bsaul 4y ago
      Except it is able to synthesize information. Which is more than just regurgitating content found elsewhere. You can ask him to synthesize the main thoughts of a philosopher, and it will produce an original set of sentences that nobody has ever written anywhere else.
      - lwhi 4y ago
        The sentences are original, the content isn't.
        bsaul 4y ago
        Are humans able to produce truely novel content, ever ? I’d say in a given field, it maybe occurs once in a decade and we call those people genius. It’s a bit too much to ask to an AI don’t you think ?
varispeed 4y ago
I find ChatGPT extremely helpful. I had to do a piece of work in the domain barely known to me, so I didn't even know what to look for to achieve my goal. So I started with some generic questions e.g. "how to do X using Y". It listed me some steps and most importantly the terms used in that domain so now I had something to do research with. Then I was asking deeper and more specific questions and at the same time using Google to reference with more trustworthy sources. It helped me so much that I had a proof of concept working in a week. If not for ChatGPT I'd probably keep postponing this forever. I see ChatGPT as a hammer - useful tool, but is hammer going to replace a carpenter? Doubt it.
simion314 4y ago
I had a similar bad experience, where ChatGPT was telling me a function was short circuiting, I read the docs and it was not mentioned, I asked it again and it still telling me that it will short circuit if I use the example, I demand it create an example that includes print statements to prove it is correct. He created the example and also showed me the output that proved it was right, so I run the example and the output was exactly what I was expecting. So I confront it one last time and it apologizes, probably it needs soem work to answer from the start with I do not know ,
- japanman425 4y ago
  You can tell him he’s wrong and he always apologises even if you are wrong in saying he is wrong
- deleted 4y ago
  [deleted]
tigershark 4y ago
I got an even weirder experience. I asked him to calculate the uk annual pension allowance given a specific adjusted income and he correctly cited the current rule, explained it and wrote the correct formula to calculate it. But somehow he managed to mess up the simple calculation giving a wrong result even if the formula and the number that he used were correct...
- idbehold 4y ago
  Him?
  - japanman425 4y ago
    Yes that’s what he said
    - idbehold 4y ago
      He?
- the_real_sparky 4y ago
  It is not designed to do math. It only generates a likely text response. If the exact equation (with exact numbers) did not often appear in its training data then you are very likely to receive a nonsensical answer. It can do some math simply because those patterns have showed up quite often in its training set. This is the same result for anything you ask. It will often imagine things like fake YouTube video links with real-sounding titles, fake function names, etc. I’m sure soon it will be hooked to something like wolfram alpha and we’ll all be terrified at how well it performs at anything math related, since it could likely classify and hand off any of the math stuff to Alpha and then utilize/summarize the results from there in its final response.
- japanman425 4y ago
  Well. Yes? It’s not a computational engine. Al it does is figure out what the probability of the next tokens are and selects the most probable ones
stuaxo 4y ago
I guess part of programming is imagining the API you want before you go find out what's there and this is doing that.
summerlight 4y ago
It is not very surprising that this kind of obscure knowledge is not handled nicely, since the data points are very sparse. The issue here is not ChatGPT's general usefulness, but it usually don't say "Sorry, I'm not sure" since it currently lacks of ability to evaluate confidence of its own answer based on the real world data. If we think the model is an efficient-but-lossy compression on the real world, then it currently lacks of a good way to error detection. If we want more general intelligence, it should have a way to measure consistency between the input (or its understanding on real world) and potential model output and its confidence interval. This is what we do everyday as a human to make a decision. I guess we probably need a number of more breakthroughs to overcome this weakness. Error correction would be the next step, and a much harder problem.
kator 4y ago
I engage with ChatGPT daily now on a number of topics. In general, I've been trying to search on google and talk to ChatGPT about the same topic, of course I don't put three words in I try to use natural language with ChatGPT. (I had a stint in NLP systems for 5 years around 1997, so that might bias me a bit on how I engage) What has impressed me is some stuff like "Can you summarize this for me?" and "How would you parse the datetime out of this log entry in python3: {raw text}", "How could I make the following mysql query more readable?" etc. At this stage, it's like when Stack Overflow came out. And yes, some SO stuff is crap, but once in a while, you get something that saves you 3 hours of your life. For example, recently, ChatGPT has saved me hours of poking around on topics I wanted to solve quickly without thinking so I could get to the high-value work that would get me closer to my goal. That said, I am amused at how it can bald-face lie about even mathematically incorrect things, and it gets lost if a thread gets a bit too long. Something is happening here. I think it'll be a while before these things write code from reading a paragraph from a product manager or a less technical user's "use case". What's interesting is to watch this pendulum swing back and forth, from expert systems coded by hand to neural nets and now these large language models. If the pendulum keeps swinging, it might land on your head one day if you don't pay attention. All this said, I enjoy my interactions with ChatGPT more than most SO posts, so I continue to use it, and luckily I have 40 years of coding experience to help me identify where it's a bit off. I have taken to pasting questions from my mentees into ChatGPT and sharing the result and suggesting they try ChatGPT to learn python3 in addition to SO and other tools. It seems to help them, I worry it will confuse them with a bald-faced lie, but I'm here to help when it does!
w1nst0nsm1th 4y ago
As proposed by another HN user, a score system should be added to indicate the confidence the model has about its answers. When I was a high school student, our physic teacher had put a test system where the student had to choose a confidence score for every answer in a test examination, from 1 to 5. According to that confidence score, you were awarded or retired a specific number of points for every answer in the test. For example at confidence '1' you were awarded 1 point if your answer was right and 0 if your answer was wrong. At confidence '2', you were awarded 2 point if your answer was right and -1 if your answer was wrong. And so on. The consequence of this system is you could have above 100% in the test even without answering all question or having some question wrong ; below 0% even with 1 right answer and only 60% if you had all good answers for all questions but played security all along and choose a confidence score of 1 for each question.
deleted 4y ago
[deleted]

I had the exact same experience trying to get ChatGPT to tell me if there's a vscode setting for the max length of tooltip text. It responded that there is and pointed out the max tokenization length setting with an explanation on why to use it and what value to use. The problem is that this setting is absolutely unrelated to the functionality I asked for.

holoduke 4y ago
A bit like common responses on stockoverflow :)
- omnicognate 4y ago
  No, StackOverflow is quite a reliable source of correct and relevant answers to programming questions, with effective mechanisms for the community to identify and fix wrong or unhelpful answers. It won't remain that way if those mechanisms are DoSed with a firehose of free plausible garbage. The flippant attitude that what we currently have is no better than that firehose, and the implication that it isn't worth worrying about or attempting to protect, is starting to grate.
  - lwhi 4y ago
    People make mistakes in stack overview. The community identifies an incorrect (or suboptimal) answer as accepted fairly often in my experience.

scrollaway 4y ago

All articles like this that end with “Ha! My job is safe!” are drawing the completely wrong conclusion. It’s amazing to me that all those incredibly smart engineers just can’t see the potential of what’s coming within the set of limitations it currently has. Amazes me even more when those same engineers don’t know how to best talk to GPT. It’s like watching them put into google search box “Good morning google, please I have a question, if you have a moment, could you tell me where I lost my keys because I can’t find them”. And when google is like “wtf?”, they claim “you can’t call it search if it can’t find my keys”. Very arbitrary, blind to limitations, and dismissive of the incredible potential it has as-is. But while they’re off randomly complaining, some of us are building startups…

jchw 4y ago
I think it's a joke. Though, the truth is, having AI write security sensitive code may actually genuinely be "AGI hard". AI breakthroughs are impressive, but I don't think we really know how to prevent the "confident bullshit" problem yet, and it's not clear it will be solved in this current cycle of progress. In many cases this might be an OK problem to deal with manually, but for security code it's probably not worth trying to. So the author might not have a threat to their job security, anyway.
- andrewflnr 4y ago
  I think writing/understanding security specification/policy is probably AGI-hard. The notion of security is (at least in parts) pretty deeply anchored in human experience and desires.
wkat4242 4y ago
To me what stands out is not what it gets wrong but how much it gets right. We use a not super common proprietary system at work which uses a custom query language. I can ask it "In system X, could you give me a query that finds XYZ". And it not only knows what system I'm talking about, it actually gives me a working query that would take me a couple hours to figure out and that's after I've had a few days' worth of training on it. Not to mention a huge IT background. Imagine picking a non-IT person off the street and getting them up to the point of being able to do this. They'd need weeks of training in basic IT concepts to even understand what you're talking about. I find it hard to overstate how amazing it is that a generic automated system can do this. And ChatGPT is not a one-trick pony, IT isn't the only field it knows about. Imagine being an expert on all sub-fields of IT, law, marketing, medical, etc and being able to combine all that knowledge from those fields. A human will never be able to do that. The potential is huge but like any source of information you have to verify. But what really is scary is extrapolating this to the next step forward. And the one after that. Because from what we had before to ChatGPT has been a massive step. I think the next step would be for it to learn and re-evaluate its model every time it gets corrected by someone (like in the linked article where it apologizes for the incorrect info). Right now it's a static model only, so the next time someone asks the original question it will give the wrong answer again. Once it can learn from this it'll improve a lot. The big hazard there is of course people manipulating it with false information. I don't have all the answers and I'm not an AI researcher but I'm very much amazed at the progress so far. And a bit scared. If this tech keeps evolving at this speed, job stability is only a tiny drop in a tidal wave of change. But there's no point in even trying to stop it.

mherrmann 4y ago

I had a similar experience. ChatGPT said "call greatSoundingFunction() from SomeLibrary". Unfortunately, neither the function nor the library actually existed. On the other hand, I did recently find ChatGPT very useful when writing a string manipulation function in C++. I had to use some (to me) weird Windows APIs. ChatGPT wrote most of what ended up in my production code.

avian 4y ago
> ChatGPT wrote most of what ended up in my production code. Some random person that's not going to get credited for their work wrote most of what ended up in your production code.
- speedgoose 4y ago
  That’s not how these models work though.
  - simion314 4y ago
    It depends, it happened for NN to spit out exact strings it had as input. I assume if the input for some particular stuff is low it will just spit it out, but if I am wrong and OpenAI, Microsoft or others have a mathematical proof that this will never happen I want to see it.
    - speedgoose 4y ago
      Yes we have seen a case of an old common and popular square root function that are returned verbatim by GitHub Copilot. Perhaps a case of overfitting since the code was present many many times in the training dataset, and even has a Wikipedia page. I haven’t seen more obvious examples since GitHub implemented a feature to prevent this from happening. I probably miss some tweets, but I assume it’s rare.
      - williamcotton 4y ago

Kiro 4y ago

I mean, what did the author expect? A really esoteric question that few people, even if they are coders, would understand or even parse and it managed to give a response that made him go "Woo! That sounds perfect.". That alone would be complete science-fiction just a couple of years ago.

the_real_sparky 4y ago
The really crazy part is that if it were trained on more domain-specific example data then I think it would give pretty great answers to many of the niche questions in those domains. I fully expect that most businesses / industries will have tailored training datasets in the next few years to enable chatbots that assist their employees to do their work. It is at least a partial answer to the high turnover problem that business now face. I have probed it at varying levels of depth in my domain and found it to answer at least as good as an entry-level engineer. This could easily be improved with extended training data. The problem is that when it begins to get things very wrong (or leave out important details), it would be difficult for an inexperienced user to determine that since ChatGPT responds so confidently to every query. It will be interesting to see how this all plays out.
adamckay 4y ago
> I mean, what did the author expect? Presumably for it to reply with its standard "I'm a LLM from OpenAI and don't have access to the internet" instead of "Just use this thing that doesn't exist", followed with "Here's more info on this thing I've totally not made up" and finishes up with "Here's the references that I've just made up to explain the other thing I've just made up". Before finally saying "You got me, I made that stuff up". Using it as-is to support humans is probably going to work well, using it as-is to replace humans is going to lead to some issues.
- Kiro 4y ago
  Imagine reading your comment just a couple of years ago. Wild how the expectations have skyrocketed. Sad that people fall back to being casually dismissive like it's nothing special without appreciating the massive leap in progress.

lwhi 4y ago

Shouldn't we just be viewing ChatGPT as a lense to view data? A more useful interface for searching and making sense of data. If someone has reported this info in stackoverflow, it will be reported here. Garbage in, garbage out ..

joao_lopes 4y ago
No, we shouldn't. ChatGPT isn't a search engine, it's a language model. The AI isn't getting this made-up information from anywhere on the internet; it's creating it itself because that's what it's made to do: generate good sounding sentences that "make sense" for some user input.
- lwhi 4y ago
  No. You're incorrect. The data model is combining information that HAS been found on the internet. The language model allows an interface that can be presented and controlled by using conversational natural language.
  - bsaul 4y ago
    Except it is able to synthesize information. Which is more than just regurgitating content found elsewhere. You can ask him to synthesize the main thoughts of a philosopher, and it will produce an original set of sentences that nobody has ever written anywhere else.
    - lwhi 4y ago
      The sentences are original, the content isn't.
      - bsaul 4y ago
        Are humans able to produce truely novel content, ever ? I’d say in a given field, it maybe occurs once in a decade and we call those people genius. It’s a bit too much to ask to an AI don’t you think ?

varispeed 4y ago

I find ChatGPT extremely helpful. I had to do a piece of work in the domain barely known to me, so I didn't even know what to look for to achieve my goal. So I started with some generic questions e.g. "how to do X using Y". It listed me some steps and most importantly the terms used in that domain so now I had something to do research with. Then I was asking deeper and more specific questions and at the same time using Google to reference with more trustworthy sources. It helped me so much that I had a proof of concept working in a week. If not for ChatGPT I'd probably keep postponing this forever. I see ChatGPT as a hammer - useful tool, but is hammer going to replace a carpenter? Doubt it.

simion314 4y ago

I had a similar bad experience, where ChatGPT was telling me a function was short circuiting, I read the docs and it was not mentioned, I asked it again and it still telling me that it will short circuit if I use the example, I demand it create an example that includes print statements to prove it is correct. He created the example and also showed me the output that proved it was right, so I run the example and the output was exactly what I was expecting. So I confront it one last time and it apologizes, probably it needs soem work to answer from the start with I do not know ,

japanman425 4y ago
You can tell him he’s wrong and he always apologises even if you are wrong in saying he is wrong
deleted 4y ago
[deleted]

tigershark 4y ago

I got an even weirder experience. I asked him to calculate the uk annual pension allowance given a specific adjusted income and he correctly cited the current rule, explained it and wrote the correct formula to calculate it. But somehow he managed to mess up the simple calculation giving a wrong result even if the formula and the number that he used were correct...

idbehold 4y ago
Him?
- japanman425 4y ago
  Yes that’s what he said
  - idbehold 4y ago
    He?
the_real_sparky 4y ago
It is not designed to do math. It only generates a likely text response. If the exact equation (with exact numbers) did not often appear in its training data then you are very likely to receive a nonsensical answer. It can do some math simply because those patterns have showed up quite often in its training set. This is the same result for anything you ask. It will often imagine things like fake YouTube video links with real-sounding titles, fake function names, etc. I’m sure soon it will be hooked to something like wolfram alpha and we’ll all be terrified at how well it performs at anything math related, since it could likely classify and hand off any of the math stuff to Alpha and then utilize/summarize the results from there in its final response.
japanman425 4y ago
Well. Yes? It’s not a computational engine. Al it does is figure out what the probability of the next tokens are and selects the most probable ones

stuaxo 4y ago

I guess part of programming is imagining the API you want before you go find out what's there and this is doing that.

summerlight 4y ago

It is not very surprising that this kind of obscure knowledge is not handled nicely, since the data points are very sparse. The issue here is not ChatGPT's general usefulness, but it usually don't say "Sorry, I'm not sure" since it currently lacks of ability to evaluate confidence of its own answer based on the real world data. If we think the model is an efficient-but-lossy compression on the real world, then it currently lacks of a good way to error detection. If we want more general intelligence, it should have a way to measure consistency between the input (or its understanding on real world) and potential model output and its confidence interval. This is what we do everyday as a human to make a decision. I guess we probably need a number of more breakthroughs to overcome this weakness. Error correction would be the next step, and a much harder problem.

kator 4y ago

I engage with ChatGPT daily now on a number of topics. In general, I've been trying to search on google and talk to ChatGPT about the same topic, of course I don't put three words in I try to use natural language with ChatGPT. (I had a stint in NLP systems for 5 years around 1997, so that might bias me a bit on how I engage) What has impressed me is some stuff like "Can you summarize this for me?" and "How would you parse the datetime out of this log entry in python3: {raw text}", "How could I make the following mysql query more readable?" etc. At this stage, it's like when Stack Overflow came out. And yes, some SO stuff is crap, but once in a while, you get something that saves you 3 hours of your life. For example, recently, ChatGPT has saved me hours of poking around on topics I wanted to solve quickly without thinking so I could get to the high-value work that would get me closer to my goal. That said, I am amused at how it can bald-face lie about even mathematically incorrect things, and it gets lost if a thread gets a bit too long. Something is happening here. I think it'll be a while before these things write code from reading a paragraph from a product manager or a less technical user's "use case". What's interesting is to watch this pendulum swing back and forth, from expert systems coded by hand to neural nets and now these large language models. If the pendulum keeps swinging, it might land on your head one day if you don't pay attention. All this said, I enjoy my interactions with ChatGPT more than most SO posts, so I continue to use it, and luckily I have 40 years of coding experience to help me identify where it's a bit off. I have taken to pasting questions from my mentees into ChatGPT and sharing the result and suggesting they try ChatGPT to learn python3 in addition to SO and other tools. It seems to help them, I worry it will confuse them with a bald-faced lie, but I'm here to help when it does!

w1nst0nsm1th 4y ago

As proposed by another HN user, a score system should be added to indicate the confidence the model has about its answers. When I was a high school student, our physic teacher had put a test system where the student had to choose a confidence score for every answer in a test examination, from 1 to 5. According to that confidence score, you were awarded or retired a specific number of points for every answer in the test. For example at confidence '1' you were awarded 1 point if your answer was right and 0 if your answer was wrong. At confidence '2', you were awarded 2 point if your answer was right and -1 if your answer was wrong. And so on. The consequence of this system is you could have above 100% in the test even without answering all question or having some question wrong ; below 0% even with 1 right answer and only 60% if you had all good answers for all questions but played security all along and choose a confidence score of 1 for each question.

deleted 4y ago

[deleted]