5 ms·

They aren’t dead but with more and more build systems that obfuscate everything in the page, including CSS classes and IDs, it’s sometimes hard to do something

by pjerem 4y ago

They aren’t dead but with more and more build systems that obfuscate everything in the page, including CSS classes and IDs, it’s sometimes hard to do something really useful.

saurik 4y ago
Would you mind giving me an example of something you've wanted to do that is really hard (or even nearly impossible) due to the these issues (so I have a standing concrete challenge case)?
- switch007 4y ago
  Trying to block Google’s “people also search for” div (when you navigate back to the search results page) is not easy IIRC
  - potamic 4y ago
    You can try these uBlock filters. Difference is night and day www.google.com##div[role="heading"]:has-text(/^People also ask$/):upward(5) www.google.com##span:has-text(/^People also ask$/):upward(5) www.google.com##div[role="heading"]:has-text(/^Cast$/):upward(5) www.google.com##span:has-text(/^Cast$/):upward(5) www.google.com##h3:has-text(/^Videos/):upward(5) www.google.com##h3:has-text(/^Images/):upward(5) www.google.com##div[role="heading"]:has-text(/^People/):upward(5) www.google.com##span:has-text(/^People/):upward(5) www.google.com##h4:has-text(/^People/):upward(5) www.google.com##span:has-text(Related searches):upward(5) www.google.com##span:has-text(/^Latest/):upward(4) www.google.com##h3:has-text(/^Top stories/):upward(6) www.google.com##div:has-text(/^Trailers/):upward(4) www.google.com##g-scrolling-carousel
  - dceddia 4y ago
    This thing drives me nuts! It’s always popping up JUST at the right time to catch my click when I’m aiming for the second search result. It’s been that way for long enough now that I’ve started to assume it’s like that on purpose, just to keep you from leaving.
- quickthrower2 4y ago
  Try it on a Tailwind & React based site. Lucky for a site like that it used NextJS so it has a lot of JSON state dumped in plaintext.
  - saurik 4y ago
    I am asking for a concrete task challenge, not a kind of site or technology.
    - quickthrower2 4y ago
      Ok scrape data from manifold.markets
      - lgas 4y ago
        Does it change a lot? Because it's pretty easy as is... eg. % curl -s https://manifold.markets https://manifold.markets | htmlq '#__next > div > main > div.items-center.flex.flex-col > div > div.gap-8.flex.flex-col > ul > div > div > div > div.relative.flex-1.gap-3.pr-1.flex.flex-col > div.peer.absolute.-left-6.-top-4.-bottom-4.right-0.z-10 > a' will get you all the anchors for the stories on the home page, and from there it's easy to get any other part of the stories.
        quickthrower2 4y ago
        It changes quite often. Since tailwind is utility based and not semantic it will likely break within the week.
    - christophilus 4y ago
      Probably Tailwindcss.com, then?
      - saurik 4y ago
        No, like "the website X is heavily obfuscated but I wish I had a userscript that could make Y modification". I work on runtime code modification tech, and am mostly known for my work on iOS and jailbreaking. Someone saying "there exist apps that use this weird compiler that are difficult to patch" isn't useful: I need a specific app and a specific target like "the Orchid app is using Flutter, which has its own weird Dart to native compilation step, which is making it almost impossible for me to use the tools we already have available to add a button that automatically builds multi-hop accounts".
- phil294 4y ago
  I have another fun challenge, try to completely disable the auto-zoom-feature on Google Maps. Last time I tried, I stepped through 20 stacks of obfuscated nonsense before eventually giving up. But tbh I can't really imagine any scenario that is near impossible - except for WASM driven web pages which will break just about anything :(
  - lgas 4y ago
    why would WASM driven pages be any different to deal with than non-WASM driven pages?
    - klibertp 4y ago
      You can't scrape HTML out of a rendered bitmap. It's not the only mode WASM can be used, but often all the rendering happens in the compiled code and the page only receives a series of images to display on a canvas. It's Flash all over again.
      - grishka 4y ago
        > It's Flash all over again. Flash was at least easy to decompile. The decompiled code was so good most of the time that you could compile it back into a working swf, with modifications if you want any. But then both WASM and Flash have to get the data from somewhere. That is usually something resembling an API endpoint that returns the result in XML or JSON. Why not sidestep the whole client app thing and go straight for that API endpoint?
      - RobbieGM 4y ago
        Almost nobody is doing this, and almost nobody ever will because it breaks accessibility and a whole host of other features. I would save the alarmism for another day.
        klibertp 4y ago
        You're right about the accessibility and other features breaking. You're wrong in saying that (almost) nobody would do this. What (almost) nobody will do is creating an HTML-based rendering engine for windowing (GTK, Qt, wx, etc.) and graphical (SDL, DirectX, OpenGL, etc.) toolkits. It's technically possible, but maintainers have enough work with supporting just the 3 desktop and 2 mobile platforms. In the absence of a proper HTML rendering frontend, there's really nothing you can do other than compile the whole graphical stack and render into a canvas. That's the default for graphical apps under Emscripten - all games and GUI apps currently work that way. The only apps that have a HTML rendering frontend are CLI tools and REPLs, where it's trivial enough to rewrite the rendering in JS. There are frameworks currently being developed that allow rendering of widget trees in HTML or SVG, but they're still immature and not widely used. Well, WASM itself is not widely used, so it's normal. BTW: were you around in the days of Flash? Because it broke accessibility and a whole host of other features the same way, yet it was still used and hugely popular (there was also ActiveX, which worsened things even more). It began with online games and graphics-rich pages, but it evolved frameworks for generic apps (ie. Flex) and then even CMS engines. Soon enough Flash was used for even blogs and e-commerce. The situation today is different due to the Web having evolved into a more capable and standardized platform, but there's no shortage of devs who would welcome the ability to run their app in the browser, but who dislike HTML/JS so much they would do basically anything just to not have to touch it - accessibility and a lot of other things be damned. Don't underestimate how irrational people and even whole communities can be. As a bonus, here's an example of the "almost" part: http://35.158.218.205/experiments/webDOOM/ http://35.158.218.205/experiments/webDOOM/
- debesyla 4y ago
  Facebook and Instagram ads. Impossible to block through CSS now as the CSS classes are dynamically generated + it's even hard to pick out sponsored posts using .js too.
  - vntok 4y ago
    From js, simply find all <a> tags that have a url containing "utm_medium=paid". Once you get them, for each tag element go up parentElements until the parent is <div role="feed">. Then remove the child.
  - splintercell 4y ago
    I am pretty sure that there is a team of engineers whose job is to office obfuscate ads on Instagram and Facebook.
    - acheron 4y ago
      Surely these people can find something more beneficial to society to work on, such as selling heroin to middle schoolers.
      - geraldwhen 4y ago
        Money is money. Few of the jobs I’ve worked have had a positive impact on society.
  - nonameiguess 4y ago
    Just block the entire feed and go directly to the profile pages of users you want updates from.
- paulgb 4y ago
  ca. 2014 I was able to write a bookmarklet that scraped a twitter thread by accessing the DOM (even causing page navigation) and rendering it as a reply-tree. Two things make this vastly more difficult now: 1. Twitter uses obfuscated css classes that appear to change on every deploy, which makes scraping hard. 2. You used to be able to get around the limit on bookmarklet code size by adding a script tag to the page and loading remote javascript. Now browsers respect HTTP headers that sites can use so that only scripts from whitelisted sites are loaded.
  - tmaly 4y ago
    Do older tweets get these new css classes when viewed after a deploy? I could see using a set of standard tweets you created to sort of benchmark the change.
    - paulgb 4y ago
      I imagine they would. That would be a good approach for data scraping, but difficult to do in a bookmarklet.
- tentacleuno 4y ago
  Anatine[0] was a Twitter client maintained by Sindre Sorhus, a popular open-source maintainer. It made use of a userscript to add extra functionality to Twitter. Sadly, this meant keeping up with Twitter's ever-changing DOM class names. It was eventually abandoned. [0]: https://github.com/sindresorhus/anatine https://github.com/sindresorhus/anatine
- pjerem 4y ago
  As an example, I wanted to create a custom, more compact style sheet for JIRA (I don’t understand all those products with complex interfaces AND white space everywhere). I failed. It’s a PITA to write any selector and it wouldn’t survive the next build from the devs.
  - capableweb 4y ago
    You don't have to select things based on CSS selectors. If you something really resilient, you select something based on the content of the elements (a button that says "Send" will rarely change to say something else, while the styling/DOM can change a lot). Otherwise xpath is another option of the DOM doesn't change a lot while CSS class/ID attributes changes between builds.
- kixiQu 4y ago
  Override the font on all YouTube text elements. You'd probably think it'd be simple, right? This being the kind of thing that Cascading Style Sheets were designed to handle elegantly and gracefully? https://maya.land/user-styles/youtube/ https://maya.land/user-styles/youtube/ I always thought people on here were just cranks complaining about style bloat, because a lot of the "this could have been a text file" arguments were applied to things that were genuinely better with the presentation polish. But this? How can developer ergonomics be worth this? Anyway, user styles and user scripts and bookmarklets are all great and I use them all the time. This level of Big Tech deformation of the basic web technologies makes them more annoying, but still not impossible.
Kye 4y ago
There were these things called API spies back in the VB6 days that would figure out the appropriate way to refer to whatever you clicked. The one from PatorJK would even produce code to do things to it from downloadable templates. It's still around: https://patorjk.com/blog/software/ https://patorjk.com/blog/software/ I wonder why no one ever made something like this for the web.
- LamaOfRuin 4y ago
  It is built into every browser's dev tools, but the point GP was making is that the way many websites are built now means that for an external observer, the way you refer to an element changes with every reload, or with every new build (which may happen at any time and frequently for actively developed sites).

by pjerem 4y ago

They aren’t dead but with more and more build systems that obfuscate everything in the page, including CSS classes and IDs, it’s sometimes hard to do something really useful.

saurik 4y ago

Would you mind giving me an example of something you've wanted to do that is really hard (or even nearly impossible) due to the these issues (so I have a standing concrete challenge case)?

switch007 4y ago
Trying to block Google’s “people also search for” div (when you navigate back to the search results page) is not easy IIRC
- potamic 4y ago
  You can try these uBlock filters. Difference is night and day www.google.com##div[role="heading"]:has-text(/^People also ask$/):upward(5) www.google.com##span:has-text(/^People also ask$/):upward(5) www.google.com##div[role="heading"]:has-text(/^Cast$/):upward(5) www.google.com##span:has-text(/^Cast$/):upward(5) www.google.com##h3:has-text(/^Videos/):upward(5) www.google.com##h3:has-text(/^Images/):upward(5) www.google.com##div[role="heading"]:has-text(/^People/):upward(5) www.google.com##span:has-text(/^People/):upward(5) www.google.com##h4:has-text(/^People/):upward(5) www.google.com##span:has-text(Related searches):upward(5) www.google.com##span:has-text(/^Latest/):upward(4) www.google.com##h3:has-text(/^Top stories/):upward(6) www.google.com##div:has-text(/^Trailers/):upward(4) www.google.com##g-scrolling-carousel
- dceddia 4y ago
  This thing drives me nuts! It’s always popping up JUST at the right time to catch my click when I’m aiming for the second search result. It’s been that way for long enough now that I’ve started to assume it’s like that on purpose, just to keep you from leaving.
quickthrower2 4y ago
Try it on a Tailwind & React based site. Lucky for a site like that it used NextJS so it has a lot of JSON state dumped in plaintext.
- saurik 4y ago
  I am asking for a concrete task challenge, not a kind of site or technology.
  - quickthrower2 4y ago
    Ok scrape data from manifold.markets
    - lgas 4y ago
      Does it change a lot? Because it's pretty easy as is... eg. % curl -s https://manifold.markets https://manifold.markets | htmlq '#__next > div > main > div.items-center.flex.flex-col > div > div.gap-8.flex.flex-col > ul > div > div > div > div.relative.flex-1.gap-3.pr-1.flex.flex-col > div.peer.absolute.-left-6.-top-4.-bottom-4.right-0.z-10 > a' will get you all the anchors for the stories on the home page, and from there it's easy to get any other part of the stories.
      - quickthrower2 4y ago
        It changes quite often. Since tailwind is utility based and not semantic it will likely break within the week.
  - christophilus 4y ago
    Probably Tailwindcss.com, then?
    - saurik 4y ago
      No, like "the website X is heavily obfuscated but I wish I had a userscript that could make Y modification". I work on runtime code modification tech, and am mostly known for my work on iOS and jailbreaking. Someone saying "there exist apps that use this weird compiler that are difficult to patch" isn't useful: I need a specific app and a specific target like "the Orchid app is using Flutter, which has its own weird Dart to native compilation step, which is making it almost impossible for me to use the tools we already have available to add a button that automatically builds multi-hop accounts".
phil294 4y ago
I have another fun challenge, try to completely disable the auto-zoom-feature on Google Maps. Last time I tried, I stepped through 20 stacks of obfuscated nonsense before eventually giving up. But tbh I can't really imagine any scenario that is near impossible - except for WASM driven web pages which will break just about anything :(
- lgas 4y ago
  why would WASM driven pages be any different to deal with than non-WASM driven pages?
  - klibertp 4y ago
    You can't scrape HTML out of a rendered bitmap. It's not the only mode WASM can be used, but often all the rendering happens in the compiled code and the page only receives a series of images to display on a canvas. It's Flash all over again.
    - grishka 4y ago
      > It's Flash all over again. Flash was at least easy to decompile. The decompiled code was so good most of the time that you could compile it back into a working swf, with modifications if you want any. But then both WASM and Flash have to get the data from somewhere. That is usually something resembling an API endpoint that returns the result in XML or JSON. Why not sidestep the whole client app thing and go straight for that API endpoint?
    - RobbieGM 4y ago
      Almost nobody is doing this, and almost nobody ever will because it breaks accessibility and a whole host of other features. I would save the alarmism for another day.
      - klibertp 4y ago
        You're right about the accessibility and other features breaking. You're wrong in saying that (almost) nobody would do this. What (almost) nobody will do is creating an HTML-based rendering engine for windowing (GTK, Qt, wx, etc.) and graphical (SDL, DirectX, OpenGL, etc.) toolkits. It's technically possible, but maintainers have enough work with supporting just the 3 desktop and 2 mobile platforms. In the absence of a proper HTML rendering frontend, there's really nothing you can do other than compile the whole graphical stack and render into a canvas. That's the default for graphical apps under Emscripten - all games and GUI apps currently work that way. The only apps that have a HTML rendering frontend are CLI tools and REPLs, where it's trivial enough to rewrite the rendering in JS. There are frameworks currently being developed that allow rendering of widget trees in HTML or SVG, but they're still immature and not widely used. Well, WASM itself is not widely used, so it's normal. BTW: were you around in the days of Flash? Because it broke accessibility and a whole host of other features the same way, yet it was still used and hugely popular (there was also ActiveX, which worsened things even more). It began with online games and graphics-rich pages, but it evolved frameworks for generic apps (ie. Flex) and then even CMS engines. Soon enough Flash was used for even blogs and e-commerce. The situation today is different due to the Web having evolved into a more capable and standardized platform, but there's no shortage of devs who would welcome the ability to run their app in the browser, but who dislike HTML/JS so much they would do basically anything just to not have to touch it - accessibility and a lot of other things be damned. Don't underestimate how irrational people and even whole communities can be. As a bonus, here's an example of the "almost" part: http://35.158.218.205/experiments/webDOOM/ http://35.158.218.205/experiments/webDOOM/
debesyla 4y ago
Facebook and Instagram ads. Impossible to block through CSS now as the CSS classes are dynamically generated + it's even hard to pick out sponsored posts using .js too.
- vntok 4y ago
  From js, simply find all <a> tags that have a url containing "utm_medium=paid". Once you get them, for each tag element go up parentElements until the parent is <div role="feed">. Then remove the child.
- splintercell 4y ago
  I am pretty sure that there is a team of engineers whose job is to office obfuscate ads on Instagram and Facebook.
  - acheron 4y ago
    Surely these people can find something more beneficial to society to work on, such as selling heroin to middle schoolers.
    - geraldwhen 4y ago
      Money is money. Few of the jobs I’ve worked have had a positive impact on society.
- nonameiguess 4y ago
  Just block the entire feed and go directly to the profile pages of users you want updates from.
paulgb 4y ago
ca. 2014 I was able to write a bookmarklet that scraped a twitter thread by accessing the DOM (even causing page navigation) and rendering it as a reply-tree. Two things make this vastly more difficult now: 1. Twitter uses obfuscated css classes that appear to change on every deploy, which makes scraping hard. 2. You used to be able to get around the limit on bookmarklet code size by adding a script tag to the page and loading remote javascript. Now browsers respect HTTP headers that sites can use so that only scripts from whitelisted sites are loaded.
- tmaly 4y ago
  Do older tweets get these new css classes when viewed after a deploy? I could see using a set of standard tweets you created to sort of benchmark the change.
  - paulgb 4y ago
    I imagine they would. That would be a good approach for data scraping, but difficult to do in a bookmarklet.
tentacleuno 4y ago
Anatine[0] was a Twitter client maintained by Sindre Sorhus, a popular open-source maintainer. It made use of a userscript to add extra functionality to Twitter. Sadly, this meant keeping up with Twitter's ever-changing DOM class names. It was eventually abandoned. [0]: https://github.com/sindresorhus/anatine https://github.com/sindresorhus/anatine
pjerem 4y ago
As an example, I wanted to create a custom, more compact style sheet for JIRA (I don’t understand all those products with complex interfaces AND white space everywhere). I failed. It’s a PITA to write any selector and it wouldn’t survive the next build from the devs.
- capableweb 4y ago
  You don't have to select things based on CSS selectors. If you something really resilient, you select something based on the content of the elements (a button that says "Send" will rarely change to say something else, while the styling/DOM can change a lot). Otherwise xpath is another option of the DOM doesn't change a lot while CSS class/ID attributes changes between builds.
kixiQu 4y ago
Override the font on all YouTube text elements. You'd probably think it'd be simple, right? This being the kind of thing that Cascading Style Sheets were designed to handle elegantly and gracefully? https://maya.land/user-styles/youtube/ https://maya.land/user-styles/youtube/ I always thought people on here were just cranks complaining about style bloat, because a lot of the "this could have been a text file" arguments were applied to things that were genuinely better with the presentation polish. But this? How can developer ergonomics be worth this? Anyway, user styles and user scripts and bookmarklets are all great and I use them all the time. This level of Big Tech deformation of the basic web technologies makes them more annoying, but still not impossible.

Kye 4y ago

There were these things called API spies back in the VB6 days that would figure out the appropriate way to refer to whatever you clicked. The one from PatorJK would even produce code to do things to it from downloadable templates. It's still around: https://patorjk.com/blog/software/ https://patorjk.com/blog/software/ I wonder why no one ever made something like this for the web.

LamaOfRuin 4y ago
It is built into every browser's dev tools, but the point GP was making is that the way many websites are built now means that for an external observer, the way you refer to an element changes with every reload, or with every new build (which may happen at any time and frequently for actively developed sites).