5 ms·

Additional details I wrangled for this rabbit hole. I don't think it's t.co doing this intentionally, but rather poor handling of 'do you have our cookies or no

Additional details I wrangled for this rabbit hole. I don't think it's t.co doing this intentionally, but rather poor handling of 'do you have our cookies or not'. Everyone in this thread _proving things_ without taking into account the complexity of the modern web. man curl -b, --cookie <data|filename> (HTTP) Pass the data to the HTTP server in the Cookie header. It is supposedly the data previously received from the server in a "Set-Cookie:" line. ---- Add that option to your curl tests. --- $ time curl -s -b -A "curl/8.2.1" -e ";auto" -L https://t.co/4fs609qwWt -o /dev/null | sha256sum eb9996199e81c3b966fa3d2e98e126516dfdd31f214410317f5bdcc3b241b6a2 - real 0m1.245s user 0m0.087s sys 0m0.034s --- $ time curl -s -b -e ";auto" -L https://t.co/4fs609qwWt -o /dev/null | sha256sum eb9996199e81c3b966fa3d2e98e126516dfdd31f214410317f5bdcc3b241b6a2 - real 0m1.265s user 0m0.103s sys 0m0.023s --- $ time curl -s -b -A "Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0" -e ";auto" -L https://t.co/4fs609qwWt -o /dev/null | sha256sum eb9996199e81c3b966fa3d2e98e126516dfdd31f214410317f5bdcc3b241b6a2 - real 0m1.254s user 0m0.100s sys 0m0.018 ---

mzs 3y ago
oh boy... -b takes an option which in your examples is -A and -e, then what follows is interpreted as a URL and you throw away the warnings: % curl -vgsSIw'> %{time_total}\n' -b -A "curl/8.2.1" https://t.co/DzIiCFp7Ti 2>&1 | grep '^$* WARNING: $\|$Could not resolve host: $\|>' * WARNING: failed to open cookie file "-A" * Could not resolve host: curl curl: (6) Could not resolve host: curl * WARNING: failed to open cookie file "-A" > HEAD /DzIiCFp7Ti HTTP/2 > Host: t.co > User-Agent: curl/8.1.2 > Accept: */* > > 0.013309 > 0.112494
- PenguinCoder 3y ago
  Alright thanks for explaining that . Here's what I see explicitly setting the cookiejar $ time curl -s -b cookies.txt -c cookies.txt -A "Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0" -e ";auto" -L https://t.co/DzIiCFp7Ti [t.co meta refresh page src] real 0m4.635s user 0m0.004s sys 0m0.008s $ time curl -b cookies.txt -c cookies.txt -A "wget/1.23" -e ";auto" -L https://t.co/DzIiCFp7Ti curl: (7) Failed to connect to www.threads.net port 443: Connection refused real 0m4.635s user 0m0.011s sys 0m0.005s $ time curl -b cookies.txt -c cookies.txt -e ";auto" -L https://t.co/DzIiCFp7Ti curl: (7) Failed to connect to www.threads.net port 443 Connection refused real 0m0.129s user 0m0.000s sys 0m0.013s The failed to connects are threads.net likely blocking those user agents but the timing is there which is different than the first UA attempt.
ChrisArchitect 3y ago
Amen
- ChrisArchitect 3y ago
  Good work Penguin. I believe in you
ender7 3y ago
I can replicate this behavior fairly easily in a browser. 1. Open incognito window in Chrome 2. Visit https://t.co/4fs609qwWt -> 5s delay 3. Open a second tab in the same window -> no delay 4. Close window, start a new incognito session 5. Visit https://t.co/4fs609qwWt -> 5s delay returns
- xslowzone 3y ago
  The reason there isn't a delay the second click is because the redirect is cached locally in your browser. Your humble anonymous tipster would appreciate if you do a little legwork.
  - hk__2 3y ago
    > The reason there isn't a delay the second click is because the redirect is cached locally in your browser. No, because it’s not an HTTP redirect. It’s an HTML page that redirects you using a meta tag, something that the browser doesn’t cache.
    - xslowzone 3y ago
      Your humble anonymous tipster notes to their skeptical audience that browsers are capable of caching all sorts of things, even something as peculiar as an HTML page.
      - hk__2 3y ago
        > browsers are capable of caching all sorts of things, even something as peculiar as an HTML page. Yes, and this is irrelevant to your previous comment: caching the HTML doesn’t cache the redirect itself.
        xslowzone 3y ago
        You can lead a horse to water, but you can't make him drink. The delay was not on the HTML page.
dymk 3y ago
If it's not intentional, why are people observing different behavior (no delay) for other domains, but a delay for NYT, bsky etc then?
scientya 3y ago
Amazing that this poor handling of 'do you have our cookies or not' only affects news papers and social media sites that Elon doesn't like! What a coincidence.

mzs 3y ago

oh boy... -b takes an option which in your examples is -A and -e, then what follows is interpreted as a URL and you throw away the warnings: % curl -vgsSIw'> %{time_total}\n' -b -A "curl/8.2.1" https://t.co/DzIiCFp7Ti 2>&1 | grep '^$* WARNING: $\|$Could not resolve host: $\|>' * WARNING: failed to open cookie file "-A" * Could not resolve host: curl curl: (6) Could not resolve host: curl * WARNING: failed to open cookie file "-A" > HEAD /DzIiCFp7Ti HTTP/2 > Host: t.co > User-Agent: curl/8.1.2 > Accept: */* > > 0.013309 > 0.112494

PenguinCoder 3y ago
Alright thanks for explaining that . Here's what I see explicitly setting the cookiejar $ time curl -s -b cookies.txt -c cookies.txt -A "Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0" -e ";auto" -L https://t.co/DzIiCFp7Ti [t.co meta refresh page src] real 0m4.635s user 0m0.004s sys 0m0.008s $ time curl -b cookies.txt -c cookies.txt -A "wget/1.23" -e ";auto" -L https://t.co/DzIiCFp7Ti curl: (7) Failed to connect to www.threads.net port 443: Connection refused real 0m4.635s user 0m0.011s sys 0m0.005s $ time curl -b cookies.txt -c cookies.txt -e ";auto" -L https://t.co/DzIiCFp7Ti curl: (7) Failed to connect to www.threads.net port 443 Connection refused real 0m0.129s user 0m0.000s sys 0m0.013s The failed to connects are threads.net likely blocking those user agents but the timing is there which is different than the first UA attempt.

ChrisArchitect 3y ago

Amen

ChrisArchitect 3y ago
Good work Penguin. I believe in you

ender7 3y ago

I can replicate this behavior fairly easily in a browser. 1. Open incognito window in Chrome 2. Visit https://t.co/4fs609qwWt -> 5s delay 3. Open a second tab in the same window -> no delay 4. Close window, start a new incognito session 5. Visit https://t.co/4fs609qwWt -> 5s delay returns

xslowzone 3y ago
The reason there isn't a delay the second click is because the redirect is cached locally in your browser. Your humble anonymous tipster would appreciate if you do a little legwork.
- hk__2 3y ago
  > The reason there isn't a delay the second click is because the redirect is cached locally in your browser. No, because it’s not an HTTP redirect. It’s an HTML page that redirects you using a meta tag, something that the browser doesn’t cache.
  - xslowzone 3y ago
    Your humble anonymous tipster notes to their skeptical audience that browsers are capable of caching all sorts of things, even something as peculiar as an HTML page.
    - hk__2 3y ago
      > browsers are capable of caching all sorts of things, even something as peculiar as an HTML page. Yes, and this is irrelevant to your previous comment: caching the HTML doesn’t cache the redirect itself.
      - xslowzone 3y ago
        You can lead a horse to water, but you can't make him drink. The delay was not on the HTML page.

dymk 3y ago

If it's not intentional, why are people observing different behavior (no delay) for other domains, but a delay for NYT, bsky etc then?

scientya 3y ago

Amazing that this poor handling of 'do you have our cookies or not' only affects news papers and social media sites that Elon doesn't like! What a coincidence.