5 ms·

This all looks about right, but how would somebody without aws experience be able to validate it without making a mess?

by 8note 3y ago

scarface_74 3y ago
Keep going down the rabbit hole… “I’m a beginner. Walk me through step $n”
scrum-treats 3y ago
Same as pre-GPT: Labs. You still have to practice implementation (for now).
avereveard 3y ago
It still isn't tho, this will work for testing, maybe, but each lambda will be it's own connection and that will exhaust db resources real fast, you're supposed to have a pooling proxy between lambdas and RDS.
- scarface_74 3y ago
  I asked ChatGPT the same question again and then followed up with: “Is there a more efficient way to connect to the database” It then went off the rails. It gave me generic answers about using connection pooling that your language framework provides. Of course that doesn’t help with each Lambda being invoked separately. I then asked “But each Lambda runs based on its own vm” It then correctly said that while each Lambda invocation would have its own pool and couldn’t be shared, it could be shared with multiple connections during one invocation. Which is technically correct. But not the answer I was looking for. In all fairness, it’s the same type of answers I would expect from a junior to mid developer with limited experience with AWS. I would hire a person who could answer that well.
  - deleted 3y ago
    [deleted]
  - BoorishBears 3y ago
    Any time you ask ChatGPT a technical question on something you're not familiar with it should be three parts: - What are the considerations? - Implement it with the considerations - Did we forget anything? It understands the need for a proxy from step 1: https://chat.openai.com/share/7ca37130-a771-457c-8742-a5f941973fb8 https://chat.openai.com/share/7ca37130-a771-457c-8742-a5f941...
    - re5i5tor 3y ago
      Yes! Excellent approach, capturing.
    - re5i5tor 3y ago
      I do short quote-posts on my blog and wrote this one up.
    - scarface_74 3y ago
      That technique didn’t work when I asked it to create a Python script to return all IAM roles with a certain set of policies attached. It still missed using a paginator to handle the list_roles call returning more than 50 roles. Once I pointed it out, it did add pagination support.
      - BoorishBears 3y ago
        I try not to cheat and hint at specific issues (since that relies on prior knowledge) I'd be surprised if even after the last prompt it wouldn't notice that. Saying "Did we miss anything" leaves it open it to re-evaluate both the implementation and the original considerations Edit: There's some non-determinism involved, but GPT-4 caught the pagination from planning stage here: https://chat.openai.com/share/3c356d4f-15d4-4f6e-bd29-af6a0b0095fd https://chat.openai.com/share/3c356d4f-15d4-4f6e-bd29-af6a0b...
        scarface_74 3y ago
        Funny enough, even in my original question, it put pagination as a consideration. But it still didn’t include it.
- Aeolun 3y ago
  Depends, if every lambda creates a single connection you can get to pretty high numbers before it becomes an issue.
- CSDude 3y ago
  And it's in the docs, RDS Proxy, launched in 2020, before GPT training cut-off https://aws.amazon.com/about-aws/whats-new/2020/06/amazon-rds-proxy-generally-available/ https://aws.amazon.com/about-aws/whats-new/2020/06/amazon-rd...
  - avereveard 3y ago
    Cut off date shouldn't matter for RAG
jacurtis 3y ago
This to me is why I don't use ChatGPT. Every time I use it I get answers that are questionable in validity. I'd say that 3/4 of my questions have returned incorrect answers at least partially. Maybe 10% are wildly incorrect with the remainder being partially incorrect. So if it is wrong with MOST of the questions that I am able to validate myself, then how can I trust it on the questions that I am unable to validate myself. The whole reason I started even doing this passive research is because I had an employee late last year who used to be a decent employee (he was never an all-star, but he got the job done to an average/satisfactory level), and started to all of a sudden perform incredibly poorly. He was submitting code and solving problems that were just really bad. He was always just an average producer, and was always teetering on the edge during performance reviews, doing just well enough that we kept him around. But he quickly started to make mistake after mistake. Several code reviews I found really strange artifacts and comments in his code that were blatant mistakes. I confronted him about them over the course of several performance reviews and said he blamed stackoverflow "copy pasta". Eventually I actually fired him, since it was getting to a point that almost everything he submitted or produced was problematic in some way and he was burning more of my time than he was saving. So I ended up firing him. While firing him, in front of HR he finally broke down and admitted that he has been using ChatGPT for everything and he begged us to let him stay and he would stop using it altogether. I of course didn't care at that point and we let him go. But I started to realize the increase of mistakes were all due to ChatGPT leading him astray. That whole experiment really taught me that ChatGPT is not ready for primetime. If you blindly trusted ChatGPT you will find yourself in the wrong place most of the time. The problem is that unless you already know the answer to the question you are asking, it is very difficult to tell where chatgpt's answers might be correct and were they might be incorrect (because it is usually a mixture of both). This makes it entirely useless for asking questions that you are not comfortable validating.
- Aeolun 3y ago
  Why’d he be so reluctant to confess to using ChatGPT? And why would he continue using it despite constant feedback that his performance was getting so much worse?
  - 3y ago
taberiand 3y ago
ChatGPT doesn't need to provide perfect AWS solutions for people without AWS experience; the onus is on the human to do the work necessary on their side to be sure of the solution it produces. It should be thought of as a time saving tool for experts (and people willing to put the work in), not a magic button for lazy people.
scarface_74 3y ago
One miss I see is this > and the appropriate RDS permissions. The role doesn’t need any IAM RDS permissions. It needs network access. But then all of the access to the database is controlled by your standard database permissions - not IAM. I guess there are corner cases like using IAM permissions to connect to your database which can be done. But is not typical

This all looks about right, but how would somebody without aws experience be able to validate it without making a mess?

by 8note 3y ago

This all looks about right, but how would somebody without aws experience be able to validate it without making a mess?

scarface_74 3y ago

Keep going down the rabbit hole… “I’m a beginner. Walk me through step $n”

scrum-treats 3y ago

Same as pre-GPT: Labs. You still have to practice implementation (for now).

avereveard 3y ago

It still isn't tho, this will work for testing, maybe, but each lambda will be it's own connection and that will exhaust db resources real fast, you're supposed to have a pooling proxy between lambdas and RDS.

scarface_74 3y ago
I asked ChatGPT the same question again and then followed up with: “Is there a more efficient way to connect to the database” It then went off the rails. It gave me generic answers about using connection pooling that your language framework provides. Of course that doesn’t help with each Lambda being invoked separately. I then asked “But each Lambda runs based on its own vm” It then correctly said that while each Lambda invocation would have its own pool and couldn’t be shared, it could be shared with multiple connections during one invocation. Which is technically correct. But not the answer I was looking for. In all fairness, it’s the same type of answers I would expect from a junior to mid developer with limited experience with AWS. I would hire a person who could answer that well.
- deleted 3y ago
  [deleted]
- BoorishBears 3y ago
  Any time you ask ChatGPT a technical question on something you're not familiar with it should be three parts: - What are the considerations? - Implement it with the considerations - Did we forget anything? It understands the need for a proxy from step 1: https://chat.openai.com/share/7ca37130-a771-457c-8742-a5f941973fb8 https://chat.openai.com/share/7ca37130-a771-457c-8742-a5f941...
  - re5i5tor 3y ago
    Yes! Excellent approach, capturing.
  - re5i5tor 3y ago
    I do short quote-posts on my blog and wrote this one up.
  - scarface_74 3y ago
    That technique didn’t work when I asked it to create a Python script to return all IAM roles with a certain set of policies attached. It still missed using a paginator to handle the list_roles call returning more than 50 roles. Once I pointed it out, it did add pagination support.
    - BoorishBears 3y ago
      I try not to cheat and hint at specific issues (since that relies on prior knowledge) I'd be surprised if even after the last prompt it wouldn't notice that. Saying "Did we miss anything" leaves it open it to re-evaluate both the implementation and the original considerations Edit: There's some non-determinism involved, but GPT-4 caught the pagination from planning stage here: https://chat.openai.com/share/3c356d4f-15d4-4f6e-bd29-af6a0b0095fd https://chat.openai.com/share/3c356d4f-15d4-4f6e-bd29-af6a0b...
      - scarface_74 3y ago
        Funny enough, even in my original question, it put pagination as a consideration. But it still didn’t include it.
Aeolun 3y ago
Depends, if every lambda creates a single connection you can get to pretty high numbers before it becomes an issue.
CSDude 3y ago
And it's in the docs, RDS Proxy, launched in 2020, before GPT training cut-off https://aws.amazon.com/about-aws/whats-new/2020/06/amazon-rds-proxy-generally-available/ https://aws.amazon.com/about-aws/whats-new/2020/06/amazon-rd...
- avereveard 3y ago
  Cut off date shouldn't matter for RAG

jacurtis 3y ago

This to me is why I don't use ChatGPT. Every time I use it I get answers that are questionable in validity. I'd say that 3/4 of my questions have returned incorrect answers at least partially. Maybe 10% are wildly incorrect with the remainder being partially incorrect. So if it is wrong with MOST of the questions that I am able to validate myself, then how can I trust it on the questions that I am unable to validate myself. The whole reason I started even doing this passive research is because I had an employee late last year who used to be a decent employee (he was never an all-star, but he got the job done to an average/satisfactory level), and started to all of a sudden perform incredibly poorly. He was submitting code and solving problems that were just really bad. He was always just an average producer, and was always teetering on the edge during performance reviews, doing just well enough that we kept him around. But he quickly started to make mistake after mistake. Several code reviews I found really strange artifacts and comments in his code that were blatant mistakes. I confronted him about them over the course of several performance reviews and said he blamed stackoverflow "copy pasta". Eventually I actually fired him, since it was getting to a point that almost everything he submitted or produced was problematic in some way and he was burning more of my time than he was saving. So I ended up firing him. While firing him, in front of HR he finally broke down and admitted that he has been using ChatGPT for everything and he begged us to let him stay and he would stop using it altogether. I of course didn't care at that point and we let him go. But I started to realize the increase of mistakes were all due to ChatGPT leading him astray. That whole experiment really taught me that ChatGPT is not ready for primetime. If you blindly trusted ChatGPT you will find yourself in the wrong place most of the time. The problem is that unless you already know the answer to the question you are asking, it is very difficult to tell where chatgpt's answers might be correct and were they might be incorrect (because it is usually a mixture of both). This makes it entirely useless for asking questions that you are not comfortable validating.

Aeolun 3y ago
Why’d he be so reluctant to confess to using ChatGPT? And why would he continue using it despite constant feedback that his performance was getting so much worse?
- 3y ago

taberiand 3y ago

ChatGPT doesn't need to provide perfect AWS solutions for people without AWS experience; the onus is on the human to do the work necessary on their side to be sure of the solution it produces. It should be thought of as a time saving tool for experts (and people willing to put the work in), not a magic button for lazy people.

scarface_74 3y ago

One miss I see is this > and the appropriate RDS permissions. The role doesn’t need any IAM RDS permissions. It needs network access. But then all of the access to the database is controlled by your standard database permissions - not IAM. I guess there are corner cases like using IAM permissions to connect to your database which can be done. But is not typical