I have the impression those being better off, with (somewhat) higher social status, education being better connected to the majority population find "Gypsy" to

I have the impression those being better off, with (somewhat) higher social status, education being better connected to the majority population find "Gypsy" to be derogatory. But once I met a poor family referring to themselves this way, so a bit perplexed and curious I rather asked them directly if they consider themselves being Roma or Gypsy... only to get an ambiguous answer. Edit: I met that family in the eastern Slovakia.