Yes sounds like you should be worried. First thing is to double and triple check your security settings. Under myaccount.google.com click on Security. There

Yes sounds like you should be worried. First thing is to double and triple check your security settings. Under myaccount.google.com click on Security. There look at the 'Your devices' section. See if there are any you don't recognize. You can individually select devices and sign those out. Look 'How you sign in to Google', and the 2FA setting under that. Next, rotate your password. It might be that the attacker has your password and is invoking the 2FA via SMS as a 'fallback'. Change your password.