5 ms·

Tell HN: Tweets no longer seem to be properly indexed by Google

56 points by consumer451 4y ago

I am a bit fuzzy here, but I recall Google having a deal with Twitter and access to the firehose. I recall finding all kinds of specific tweets, by searching "string," etc. This does not seem to be the case today. This is important because you may be used to relying on Google searches to be exhaustive, and include tweets. Now it seems you have to go to twitter.com as well.

salusinarduis 4y ago
Some other interesting things I've noticed is counters are often wrong, sometimes even with insane values such as -1. Replies missing from retweet views, missing search results, people have reported that replies sometimes don't work or never appear.
- ge96 4y ago
  > counters are often wrong, I heard about "optimistic rendering" -1 does seem bad. Reddit has fluctuating numbers I thought it was to fight bots or maybe using the eventually correct type db.
  - pifm_guy 4y ago
    I never really saw how making the counters random helped fight bots. Anyone who wants to know if their bot armies upvotes are counted can just choose 2000 articles, upvote a random half, then see if the half they upvoted have higher vote tallies than the half they didn't. No amount of delay, quantizing, or adding noise will defeat that tactic. So why try at all?
    - CaveTech 4y ago
      You can't efficiently tell which accounts work and which don't, so a) you'll quickly accumulate more invalid accounts than valid ones, but have the burden of maintaining all of them. b) you'll continue to provide more signal to reddit from your invalid accounts that can be used to burn your valid accounts.
    - derefr 4y ago
      I don’t think the technique would be meant to counter blind voting, but rather strategic voting—i.e. counter-voting some particular post down/up with a random member of your bot army every time it’s voted up/down, to keep its count high/low/at some constant value. I say this, not because it seems like an especially common style of bot to be running, but rather because 1. it would be a rather heavy backend write load on the site if two bots doing this in opposite directions did exist, and ever “clashed” on the same post—and, much more problematically, a never-ending load, as the bots would never be satisfied; and 2. such bots do depend on the exact vote count passing some threshold, so fuzzing votes is a simple way to make such bots confused—not all the time, but probabilistically, enough of the time to make any such “clash” loops eventually quiesce rather than going on forever.
  - JustLurking2022 4y ago
    I would expect it's a database consistency issue across some distributed NoSQL store.
    - hn92726819 4y ago
      I would imagine that's the cause too. But there are ways to prevent that. It is eventual consistency after all, right? As I mentioned in my other comment, I've had the wrong count for multiple years.
- andy_ppp 4y ago
  There are now always exactly 35 new tweets to load... on web anyway. Seems to be an increasingly buggy mess. Isn't it $1bn in interest payments Twitter owes, on what regularity is that paid, every 3 months?
  - rchaud 4y ago
    Yes, they made a $300m payment about a week ago. https://www.bloomberg.com/news/articles/2023-01-30/twitter-said-to-make-first-interest-payment-on-musk-buyout-debt https://www.bloomberg.com/news/articles/2023-01-30/twitter-s...
- 908B64B197 4y ago
  How long until they have a major outage?
- hn92726819 4y ago
  Re: the counter issue; I've had "2" tweets for at least 5 years. I really have 0 tweets, but they mention if you ever delete tweets in bulk, the counter can be off. They confirm this can happen on their site: https://help.twitter.com/en/using-twitter/missing-tweets https://help.twitter.com/en/using-twitter/missing-tweets Makes you wonder if anything security related has glitches like this.
cma 4y ago
They shouldn't index stuff with a login or pay wall anyway, or should put it in a separate tab of the results or something (that is, other than in countries where they were forced to).
- flangola7 4y ago
  Agreed but Twitter isn't a paywall.
  - mcherm 4y ago
    Not yet.
- sva_ 4y ago
  One good change that happened on Twitter recently is that the login nagscreen now has a close-button.
- consumer451 4y ago
  But tweets are on the public web without login.
  - cma 4y ago
    It changes over time, sometimes there will be a dismissable popup, for a while it couldn't be dismissed. It does seem to work now for non-nsfw tweets, but nsfw login-walled ones are indexed by Google too I believe.
  - BelleOfTheBall 4y ago
    Specific tweets, yes, but if you want to scroll a person's profile, you'll need an account after about a page of scrolling.
    - heywhatupboys 4y ago
      no, just close that popup? if it even appears... when is the last time you've used twitter
sourcecodeplz 4y ago
Yeah, also when searching for a personality, the second result would always be a "Tweets by" carousel. Interesting.
ruined 4y ago
google is, however, thoroughly indexing mastodon posts, if you have your server and account configured to allow crawling. enjoy
- pifm_guy 4y ago
  Twitter indexing was basically real-time - you could see a tweet within a few minutes of it being made Mastodon seems to take many days to appear, if at all. I suspect Google just scrapes the webui occasionally and doesn't use any push-based mechanism for notifications. That probably means it misses some too.
  - klyrs 4y ago
    HN is indexed in effectively real time as well. I've googled several phrases I've read here, hoping to find more information, where the HN comment of interest is the sole result.
    - bdcravens 4y ago
      Googled "HN is indexed in effectively real time as well" and your comment is the top result
      - klyrs 4y ago
        Delta of 7 minutes. I thought about putting in a nonsense phrase to test the hypothesis, but apparently that wasn't even necessary. Spooky.
      - pests 4y ago
        Odd the same search from my end brings up 0 results.
      - basch 4y ago
        I just had the opposite happen, was looking for a post I know ive found through google before and had to go through bing and then yandex to find it. Google's historical archive of hn appears wonky, or at least did that day.
wolpoli 4y ago
My recollection was that the deal to index and surface tweets on the SERP ended years ago. Was that the deal you recalled?
- consumer451 4y ago
  Now that I look, yeah the deal I was thinking of dates back to 2015. Did not know that expired. But is the Google(Twitter) search experience the same for you right now as it was in previous months?
deleted 4y ago
[deleted]
LarryMullins 4y ago
I never use general-purpose search engines if I already know which website the thing I'm looking for is on (assuming that website has it's own search functionality.) If I am looking for a wikipedia article, I go to wikipedia and search for it there. If I am looking for a tweet, I go to twitter and search for it there. If I'm searching for something on HN, then it's algolia. Search keywords in firefox make this easy. I can type "wiki train" into my address bar and it will take me directly to https://en.wikipedia.org/w/index.php?search=train https://en.wikipedia.org/w/index.php?search=train From a usability standpoint it's just as streamlined as typing the same into the address bar and expecting Google to find it for you, except you don't need Google. Googling everything by default is a bad habit, but one Google obviously likes to encourage. They want everybody to rely on them for everything.
- bthrn 4y ago
  It’s heavily dependent on the quality of the search offered. For example — Reddit’s search functionality is terrible. I’ve found that Google is much better for searching Reddit.
  - 29athrowaway 4y ago
    Or Amazon. Amazon's search sucks. Results are often completely irrelevant, or dismissive of the search terms. You search for a refrigerator filter from a known refrigerator manufacturer and the first results you get are the toxic offbrand refrigerator filters from companies with randomly generated company names that poison your entire family.
    - waboremo 4y ago
      That combined with Amazon's aggressive category filtering is annoying. Search a generic term like rings, it forces you into the clothing category without the ability to switch. You have to then slap on more keywords in hopes it navigates to the right category, because manually changing the category while including generic searches does not work. It does not matter if you first went to the category you believe your search to be in. Somehow Amazon has managed to make the bloated Aliexpress experience seem more straight forward and easy to navigate.
benevol 4y ago
Google has become a major force of censorship. Twitter (under Elon Musk) is now opposed to that. That should explain what you experience.
- bdcravens 4y ago
  Censorship seems alive and well on Twitter under Musk. https://www.motherjones.com/media/2022/12/elon-musk-twitter-censorship-social-media-mastadon-post/ https://www.motherjones.com/media/2022/12/elon-musk-twitter-... https://www.nytimes.com/2022/12/16/business/media/elon-musk-twitter-journalist-suspension.html https://www.nytimes.com/2022/12/16/business/media/elon-musk-... https://www.msnbc.com/opinion/msnbc-opinion/elon-musk-twitter-censor-left-accounts-rcna59638 https://www.msnbc.com/opinion/msnbc-opinion/elon-musk-twitte...
jahewson 4y ago
Next week, Twitter is starting to charge for their API https://arstechnica.com/tech-policy/2023/02/cash-strapped-twitter-to-start-charging-developers-for-api-access-next-week/amp/ https://arstechnica.com/tech-policy/2023/02/cash-strapped-tw... Could be related? Even if they’re just crawling the public pages I could certainly imagine Elon trying to extract money from Google for the privilege.
- bastawhiz 4y ago
  My understanding was that Google was paying for its firehose access already. Though I suppose they could be forcing a contract renegotiation.
  - consumer451 4y ago
    > Though I suppose they could be forcing a contract renegotiation. This was my attempt at an explanation. I wonder who will blink first and if the public will ever learn the details.
- voytec 4y ago
  Just a few days ago I wrote a comment, that Elon's decisions are pushing Twitter towards becoming a ghost town. But I'm unsure whether he could be so irrational to cut the biggest search engine off. One thing I find important to remember, is that Jack Dorsey gave Elon Musk one bil for the Twitter purchase so maybe there's some kind of Grand Plan that I'm too stupid to see. But would it start with Twitter's bankruptcy?
  - gremlinsinc 4y ago
    I'm not really a conspiracy guy, but it could be a 4d chess move. He can get publicity whenever he wants it, how he wants it. Twitter is the #1 platform for news, and political ideologies to express themselves. If he levels that, it throws a huge wrench since there's really no 'backup'. Everyone has put so much into tweeting on twitter, now that it could go belly up, how does an entire network pick a good replacement? Doesn't even look like they can because of all the polarization. If he had some evil designs, tanking Twitter could be worth losing 45 billion dollars if it gives him more power to do whatever he wants to do. Not saying he is, I'm not an Elon fan, but even I don't think he's this stupid (he's not donald trump) - but I guess he could be - if he isn't doing this on purpose he'd have to be.
dpedu 4y ago
I am finding that this is not the case when I'm searching. Yesterday I read this tweet (below) that talks about some military history relevant to current events. I wanted to read more so I googled `"1957" china balloon` and the first hit was the very same tweet. Testing now, searching for a substring from the tweet in quotes works too. https://twitter.com/NavalInstitute/status/1621639738288373762 https://twitter.com/NavalInstitute/status/162163973828837376...
quench 4y ago
You can add site:twitter.com to the google search to restrict the results. "1957" china balloon site:twitter.com gives 677 results so that might explain why you don't get the result you want with a general search. without a _site_ filter its going to have to pick only one or two tweets
kbrackson 4y ago
[flagged]

56 points by consumer451 4y ago

salusinarduis 4y ago

Some other interesting things I've noticed is counters are often wrong, sometimes even with insane values such as -1. Replies missing from retweet views, missing search results, people have reported that replies sometimes don't work or never appear.

ge96 4y ago
> counters are often wrong, I heard about "optimistic rendering" -1 does seem bad. Reddit has fluctuating numbers I thought it was to fight bots or maybe using the eventually correct type db.
- pifm_guy 4y ago
  I never really saw how making the counters random helped fight bots. Anyone who wants to know if their bot armies upvotes are counted can just choose 2000 articles, upvote a random half, then see if the half they upvoted have higher vote tallies than the half they didn't. No amount of delay, quantizing, or adding noise will defeat that tactic. So why try at all?
  - CaveTech 4y ago
    You can't efficiently tell which accounts work and which don't, so a) you'll quickly accumulate more invalid accounts than valid ones, but have the burden of maintaining all of them. b) you'll continue to provide more signal to reddit from your invalid accounts that can be used to burn your valid accounts.
  - derefr 4y ago
    I don’t think the technique would be meant to counter blind voting, but rather strategic voting—i.e. counter-voting some particular post down/up with a random member of your bot army every time it’s voted up/down, to keep its count high/low/at some constant value. I say this, not because it seems like an especially common style of bot to be running, but rather because 1. it would be a rather heavy backend write load on the site if two bots doing this in opposite directions did exist, and ever “clashed” on the same post—and, much more problematically, a never-ending load, as the bots would never be satisfied; and 2. such bots do depend on the exact vote count passing some threshold, so fuzzing votes is a simple way to make such bots confused—not all the time, but probabilistically, enough of the time to make any such “clash” loops eventually quiesce rather than going on forever.
- JustLurking2022 4y ago
  I would expect it's a database consistency issue across some distributed NoSQL store.
  - hn92726819 4y ago
    I would imagine that's the cause too. But there are ways to prevent that. It is eventual consistency after all, right? As I mentioned in my other comment, I've had the wrong count for multiple years.
andy_ppp 4y ago
There are now always exactly 35 new tweets to load... on web anyway. Seems to be an increasingly buggy mess. Isn't it $1bn in interest payments Twitter owes, on what regularity is that paid, every 3 months?
- rchaud 4y ago
  Yes, they made a $300m payment about a week ago. https://www.bloomberg.com/news/articles/2023-01-30/twitter-said-to-make-first-interest-payment-on-musk-buyout-debt https://www.bloomberg.com/news/articles/2023-01-30/twitter-s...
908B64B197 4y ago
How long until they have a major outage?
hn92726819 4y ago
Re: the counter issue; I've had "2" tweets for at least 5 years. I really have 0 tweets, but they mention if you ever delete tweets in bulk, the counter can be off. They confirm this can happen on their site: https://help.twitter.com/en/using-twitter/missing-tweets https://help.twitter.com/en/using-twitter/missing-tweets Makes you wonder if anything security related has glitches like this.

cma 4y ago

They shouldn't index stuff with a login or pay wall anyway, or should put it in a separate tab of the results or something (that is, other than in countries where they were forced to).

flangola7 4y ago
Agreed but Twitter isn't a paywall.
- mcherm 4y ago
  Not yet.
sva_ 4y ago
One good change that happened on Twitter recently is that the login nagscreen now has a close-button.
consumer451 4y ago
But tweets are on the public web without login.
- cma 4y ago
  It changes over time, sometimes there will be a dismissable popup, for a while it couldn't be dismissed. It does seem to work now for non-nsfw tweets, but nsfw login-walled ones are indexed by Google too I believe.
- BelleOfTheBall 4y ago
  Specific tweets, yes, but if you want to scroll a person's profile, you'll need an account after about a page of scrolling.
  - heywhatupboys 4y ago
    no, just close that popup? if it even appears... when is the last time you've used twitter

sourcecodeplz 4y ago

Yeah, also when searching for a personality, the second result would always be a "Tweets by" carousel. Interesting.

ruined 4y ago

google is, however, thoroughly indexing mastodon posts, if you have your server and account configured to allow crawling. enjoy

pifm_guy 4y ago
Twitter indexing was basically real-time - you could see a tweet within a few minutes of it being made Mastodon seems to take many days to appear, if at all. I suspect Google just scrapes the webui occasionally and doesn't use any push-based mechanism for notifications. That probably means it misses some too.
- klyrs 4y ago
  HN is indexed in effectively real time as well. I've googled several phrases I've read here, hoping to find more information, where the HN comment of interest is the sole result.
  - bdcravens 4y ago
    Googled "HN is indexed in effectively real time as well" and your comment is the top result
    - klyrs 4y ago
      Delta of 7 minutes. I thought about putting in a nonsense phrase to test the hypothesis, but apparently that wasn't even necessary. Spooky.
    - pests 4y ago
      Odd the same search from my end brings up 0 results.
    - basch 4y ago
      I just had the opposite happen, was looking for a post I know ive found through google before and had to go through bing and then yandex to find it. Google's historical archive of hn appears wonky, or at least did that day.

wolpoli 4y ago

My recollection was that the deal to index and surface tweets on the SERP ended years ago. Was that the deal you recalled?

consumer451 4y ago
Now that I look, yeah the deal I was thinking of dates back to 2015. Did not know that expired. But is the Google(Twitter) search experience the same for you right now as it was in previous months?

deleted 4y ago

[deleted]

LarryMullins 4y ago

I never use general-purpose search engines if I already know which website the thing I'm looking for is on (assuming that website has it's own search functionality.) If I am looking for a wikipedia article, I go to wikipedia and search for it there. If I am looking for a tweet, I go to twitter and search for it there. If I'm searching for something on HN, then it's algolia. Search keywords in firefox make this easy. I can type "wiki train" into my address bar and it will take me directly to https://en.wikipedia.org/w/index.php?search=train https://en.wikipedia.org/w/index.php?search=train From a usability standpoint it's just as streamlined as typing the same into the address bar and expecting Google to find it for you, except you don't need Google. Googling everything by default is a bad habit, but one Google obviously likes to encourage. They want everybody to rely on them for everything.

bthrn 4y ago
It’s heavily dependent on the quality of the search offered. For example — Reddit’s search functionality is terrible. I’ve found that Google is much better for searching Reddit.
- 29athrowaway 4y ago
  Or Amazon. Amazon's search sucks. Results are often completely irrelevant, or dismissive of the search terms. You search for a refrigerator filter from a known refrigerator manufacturer and the first results you get are the toxic offbrand refrigerator filters from companies with randomly generated company names that poison your entire family.
  - waboremo 4y ago
    That combined with Amazon's aggressive category filtering is annoying. Search a generic term like rings, it forces you into the clothing category without the ability to switch. You have to then slap on more keywords in hopes it navigates to the right category, because manually changing the category while including generic searches does not work. It does not matter if you first went to the category you believe your search to be in. Somehow Amazon has managed to make the bloated Aliexpress experience seem more straight forward and easy to navigate.

benevol 4y ago

Google has become a major force of censorship. Twitter (under Elon Musk) is now opposed to that. That should explain what you experience.

bdcravens 4y ago
Censorship seems alive and well on Twitter under Musk. https://www.motherjones.com/media/2022/12/elon-musk-twitter-censorship-social-media-mastadon-post/ https://www.motherjones.com/media/2022/12/elon-musk-twitter-... https://www.nytimes.com/2022/12/16/business/media/elon-musk-twitter-journalist-suspension.html https://www.nytimes.com/2022/12/16/business/media/elon-musk-... https://www.msnbc.com/opinion/msnbc-opinion/elon-musk-twitter-censor-left-accounts-rcna59638 https://www.msnbc.com/opinion/msnbc-opinion/elon-musk-twitte...

jahewson 4y ago

Next week, Twitter is starting to charge for their API https://arstechnica.com/tech-policy/2023/02/cash-strapped-twitter-to-start-charging-developers-for-api-access-next-week/amp/ https://arstechnica.com/tech-policy/2023/02/cash-strapped-tw... Could be related? Even if they’re just crawling the public pages I could certainly imagine Elon trying to extract money from Google for the privilege.

bastawhiz 4y ago
My understanding was that Google was paying for its firehose access already. Though I suppose they could be forcing a contract renegotiation.
- consumer451 4y ago
  > Though I suppose they could be forcing a contract renegotiation. This was my attempt at an explanation. I wonder who will blink first and if the public will ever learn the details.
voytec 4y ago
Just a few days ago I wrote a comment, that Elon's decisions are pushing Twitter towards becoming a ghost town. But I'm unsure whether he could be so irrational to cut the biggest search engine off. One thing I find important to remember, is that Jack Dorsey gave Elon Musk one bil for the Twitter purchase so maybe there's some kind of Grand Plan that I'm too stupid to see. But would it start with Twitter's bankruptcy?
- gremlinsinc 4y ago
  I'm not really a conspiracy guy, but it could be a 4d chess move. He can get publicity whenever he wants it, how he wants it. Twitter is the #1 platform for news, and political ideologies to express themselves. If he levels that, it throws a huge wrench since there's really no 'backup'. Everyone has put so much into tweeting on twitter, now that it could go belly up, how does an entire network pick a good replacement? Doesn't even look like they can because of all the polarization. If he had some evil designs, tanking Twitter could be worth losing 45 billion dollars if it gives him more power to do whatever he wants to do. Not saying he is, I'm not an Elon fan, but even I don't think he's this stupid (he's not donald trump) - but I guess he could be - if he isn't doing this on purpose he'd have to be.

dpedu 4y ago

I am finding that this is not the case when I'm searching. Yesterday I read this tweet (below) that talks about some military history relevant to current events. I wanted to read more so I googled `"1957" china balloon` and the first hit was the very same tweet. Testing now, searching for a substring from the tweet in quotes works too. https://twitter.com/NavalInstitute/status/1621639738288373762 https://twitter.com/NavalInstitute/status/162163973828837376...

quench 4y ago

You can add site:twitter.com to the google search to restrict the results. "1957" china balloon site:twitter.com gives 677 results so that might explain why you don't get the result you want with a general search. without a _site_ filter its going to have to pick only one or two tweets

kbrackson 4y ago

[flagged]