3 ms·

Interesting and much needed--I absolutely refuse to go to Twitter links anymore; why bother, they don't usually work unless you're logged in. So, I clicked some

Interesting and much needed--I absolutely refuse to go to Twitter links anymore; why bother, they don't usually work unless you're logged in. So, I clicked some of the links and, at this time, they don't load in a reasonable amount of time, I assume due to the Hug of Death or whatever it is we call that here. This one loaded: https://unlace.app/thread/ianwcrosby/status/1872724231999381790 https://unlace.app/thread/ianwcrosby/status/1872724231999381..., but oddly it's duplicating the thread.

jimmySixDOF 2y ago
I had the same experience and suggest a loading progress bar of some sort. I also didn't see any of the Selenium process on their Github so can't see exactly how the tweets are processed for plain text cleaning. The Twitter/x API is so priced out of the market these types of work arounds are necessary inconveniences so thanks to the dev for putting this together!
- vednig 2y ago
  backend server's code is now available at https://github.com/vednig/x-api https://github.com/vednig/x-api , you can have a look there for more info.
vednig 2y ago
Backend server's source code is now located at https://github.com/vednig/x-api https://github.com/vednig/x-api ,currently the script waits for complete page to load and then extracts info to send out a response. Since each new request takes high time and processing power, it was only logical to store responses and cache them for timely usage. There's also a bit of a workaround required to get videos from X.com as they only load their direct URL when focused hence parsing html couldn't work, so I had to log network traffic to ensure I get their HLS links too.
derdi 2y ago
> I absolutely refuse to go to Twitter links anymore; why bother, they don't usually work unless you're logged in. Replace "x.com" by "xcancel.com" (or some other still working Nitter instance) and you're set. https://xcancel.com/ianwcrosby/status/1872724231999381790 https://xcancel.com/ianwcrosby/status/1872724231999381790
- vednig 2y ago
  Xcancel is also a great service, it's a complete X replacement.
- garciasn 2y ago
  > still working Nitter instance This is the issue with Nitter instances; they seem too ephemeral for reliable use.
  - Akronymus 2y ago
    nitter.poast.org is pretty reliable for me.
    - vednig 2y ago
      I think nitter.poast.org is geo-restricted, I'm getting 403 error
      - KomoD 2y ago
        They're just very strict with IP blocking and ratelimits. Pretty sure VPNs and such are blocked too.
vednig 2y ago
Repetition occurs due to m3u8 requests I'm logging for videos, which require twice the reload of the page, it can be avoided by updating some code. And since it's highly cached based on unique url requests, it persists, I'm thinking of having a purge cache option too, I'd have to think how to implement it so as to prevent abuse as well.
vednig 2y ago
BTW, New Update Alert: fixed this with 4 lines of Python code uniquetwid=[] finalresponse=[] for i in response: if not uniquetwid.__contains__(i['tweet_id']): finalresponse.append(i) print(i) Let's Enjoy, reading unrepeated threads now !

jimmySixDOF 2y ago

I had the same experience and suggest a loading progress bar of some sort. I also didn't see any of the Selenium process on their Github so can't see exactly how the tweets are processed for plain text cleaning. The Twitter/x API is so priced out of the market these types of work arounds are necessary inconveniences so thanks to the dev for putting this together!

vednig 2y ago
backend server's code is now available at https://github.com/vednig/x-api https://github.com/vednig/x-api , you can have a look there for more info.

vednig 2y ago

Backend server's source code is now located at https://github.com/vednig/x-api https://github.com/vednig/x-api ,currently the script waits for complete page to load and then extracts info to send out a response. Since each new request takes high time and processing power, it was only logical to store responses and cache them for timely usage. There's also a bit of a workaround required to get videos from X.com as they only load their direct URL when focused hence parsing html couldn't work, so I had to log network traffic to ensure I get their HLS links too.

derdi 2y ago

> I absolutely refuse to go to Twitter links anymore; why bother, they don't usually work unless you're logged in. Replace "x.com" by "xcancel.com" (or some other still working Nitter instance) and you're set. https://xcancel.com/ianwcrosby/status/1872724231999381790 https://xcancel.com/ianwcrosby/status/1872724231999381790

vednig 2y ago
Xcancel is also a great service, it's a complete X replacement.
garciasn 2y ago
> still working Nitter instance This is the issue with Nitter instances; they seem too ephemeral for reliable use.
- Akronymus 2y ago
  nitter.poast.org is pretty reliable for me.
  - vednig 2y ago
    I think nitter.poast.org is geo-restricted, I'm getting 403 error
    - KomoD 2y ago
      They're just very strict with IP blocking and ratelimits. Pretty sure VPNs and such are blocked too.

vednig 2y ago

Repetition occurs due to m3u8 requests I'm logging for videos, which require twice the reload of the page, it can be avoided by updating some code. And since it's highly cached based on unique url requests, it persists, I'm thinking of having a purge cache option too, I'd have to think how to implement it so as to prevent abuse as well.

vednig 2y ago

BTW, New Update Alert: fixed this with 4 lines of Python code uniquetwid=[] finalresponse=[] for i in response: if not uniquetwid.__contains__(i['tweet_id']): finalresponse.append(i) print(i) Let's Enjoy, reading unrepeated threads now !