5 ms·

or turn your blog into a frontend/backend combo. keep the frontend as an SPA so that the page has nothing on it. have your backend send data in encrypted format

or turn your blog into a frontend/backend combo. keep the frontend as an SPA so that the page has nothing on it. have your backend send data in encrypted format and the AI scrapers would need to do a tonne of work in order to figure out what your data is. If everyone uses a different key and different encryption algorithm suddenly all their server time is busted decrypting stuff

chii 10mo ago
How does your normal users get access to the same contents? Or are you having the user solve an encryption puzzle to view it?
- vivzkestrel 10mo ago
  - the frontend has a decryption module that ll show users what they want to see, - the backend has an encryption module. - The bots and crawlers will see the encrypted text - Can someone who peeks deeply inside the client side code decrypt it? YES - Will 99% of the scrapers bother doing this? NO - The key can be anything, it could be a per session key agreed upon between the client and the server, a csrf token, or even a fixed key
  - hollowturtle 10mo ago
    Ehm what would stop ai scrapers from using a browser like a normal user would? Google bot already does, it can execute js and can read spa client side generated content, so it proves can be done at scale, and I'm pretty sure some ai scrapers already do
    - vivzkestrel 10mo ago
      rate limit per ip that progressively keeps decreasing req/mins every few mins?
      - chii 10mo ago
        so why not just do that for these scrapers, rather than complicate it by encrypting and decrypting, which is just obfuscation as the private key is clearly available to the end-user?
        vivzkestrel 10mo ago
        tbh i did not encrypt decrypt for the ai scrapers at all, a lot of people were previously trying to download data directly from my API that my frontend uses and this kinda pissed me off a bit. So I added encryption/decryption to the mix and will release the newer version. As I mentioned earlier as well, can someone sit through and decrypt it? yes. Will 99% of them do it? no! Thats where I win
      - prmoustache 10mo ago
        What if scrapers ips are millions of smartphones? If I was as evil as an AI scraper company that is not obeying robots.txt I would totally build/buy thousands of small games/apps for mobiles to use them as jumphosts to scape the web. This is probably happening already.
        vivzkestrel 10mo ago
        in my case my application does not use pagination, it uses infinite scroll, even if you had a million devices that use google chrome, they would all load page 1 and if that req/minute progressively decreaasing thing is implemented, once they start scrolling endlessly they would all hit the rate limits sooner or later, the thing is a human is not going to scroll down a 100 pages but a bot will. once this difference has been factored it, it wont matter how many unique devices they bring into the battle
      - 63stack 10mo ago
        Just skip the whole encryption/decryption shebang then. But no, this does not work, scrapers are using residential ips, and they have enough that they can rotate between them if they get blocked.
    - dirkc 10mo ago
      If you decrypt the content on the client side using an expensive decryption algorithm the scraper needs to spend the computing resource to decrypt.
      - integralid 10mo ago
        Every visitor too. Mobile users are going to love this.
  - PunchyHamster 10mo ago
    the AI will just run chrome instance
- vivzkestrel 10mo ago
  for example this is what my backend renders on the static html page {"z":"gxCit6xEQf0N9IIoG909xfSxypRX7j0BLlXnd5IgWrrzEWzBUxDiS4o4AlIkNYOyuzkY8w4IVoEgUmW02jj84BxhMrNPetK8n6nIn2ORLKQPfIVTS48nGQ1PldtdlpiUNYUm04N+WrMBGGceKYnQoORQO3XbFOVzboFYOWbdMhLdMS2N26YtCUYHwy7jw1AwlXS0Nm1SClb0U1qk2KnDB6s9bcMmpstaOY2RkmGbQ4KMuKHaGzByVzeIPHrtXtNjLnj68cgyLALyO3E5ncqspyjbnuZtfusn2Y49Nu3LVZDDk/JojC6x6GlLZKFEDoiunyfhnqd0SRsDvynKNpFObi3uu+CDrKv2qJgQoyCk392uO4dIqsoJ8iS61DW3hhDoh7nbLsum/E+VdyRdcMo2/0H8cBx6hqxYIOe3hzbMpEsX7YDCy819XYbyi30xQFKe/hGBv3i/LtsD8KdPIHiz2/X/msSh5FfQLVsK2wRx/70FAB4zu4Zj3E9wKMA0HPnDY0BFEtIYFmUNTBMyvFt0k3+MUjGEFoaHT+/LDJNkDzLybsFqpufMhi0RTnkReVVDepU4k9XmxCPyCBYzt74g2BwRS4bgVzmHrJhw9GkS0ZIkHbNhRaUR5iF7mJ/1rIRXDhtLClTEm6DgEO/jAZ/3edLhDrJvCX5gKzZWfsQGgkF2IzHTUdF9HlT5YE/bZ6Bvl9wmHTpnDpnLu3ERGOpuyqkWJG3eji61OB66jx5ZVI2H2o7G4+XDPYRykAs2TJPkxJzR6OU1xaOTmeLjXlqrvG1i7kVf6M+iEJrtvkuPUfDcOSkdSnN+1apnrV+qrNv6Nwhqp89zic2dwvJLYQyL3qT/JQPlwO+TqfhMlklvsPzlMvzECcPgesWTH8LZVdb857FmSbdWjY/rceg4Icg2E6RdCkGEx/JE8XrShAdyl7Udbx8A/+kRALsvTa6+gtd3LxLriMH/wbrasJbply06KaR07SEx7PdWwsdGj8sgmg0l24KRmmqEICKLFfb+k9nCZABUcfjqDlg/x4KDbuRe33Kz+gA++dcq/sLZQm2fRT7UvXFha9RAR9XAEsty1uI3pkjeMcsPRMBIJETkNXG0QUgrjKT44UDYlBSO+mfNexpW6tC1s8gDZJaJBdd/5QzzamnaoAWU9SAksuCc0EkbwmOXxLLqCyXwnXbSZ6LCWqrBpU43BopETh8SBtnrdpTPZWxI8aPHJaF9Qertf7qglqfUWVqnCdALWAO+j+Ma7FkyL04tNX62VmcwqTHQuTQAgZnoo0iZo6wHNPjOgDOxXz+XN8AVA1aIhDEQ8iA+WcPh/+QjDAg/k0wskR2S9MiSP8tVOfD2lMao6A7yuA8FkNK+oOJZVn8IDcIKnGCEem33lC/GFXUGqhi4mh2marxOEHmvYX1V6f3SDlK+NmSMQxbnKVWy3i4A2rJL5Q5l9rZG90rYU53q6ApbK49Zmdy5RyPJpJoDMIa9Py2LtmSEEzW1608Jf7QTUB6WhOLsWBvMlt1fyxtfkOM1h8/WpEga7eDsR+htaqilZgvM/6dyh9C+izWKWK0w+LsJm625e5nHOZs1MQ0DCPA2wu0O/777yOGw0Yw6RTEN6Syy6SY8MpMVAaey21NfCYF9xSMXk15/h4hxdngX65uxNobuE0clCy5BujFbveqrwKHnnDhS79QCgAWQNtm3X5z4dmHxdmqyVBqeu+PMctEXtGdQXOy54nT45FB1MtYSEuejn7q2wJlT31ng6W3Ahb67F33xEhi3gJ14b+RF9mdjoYwfkW/TB4E2/mnYMbVSLHskDEvp/vgYwVCsdHuW/tG7IvN7xG1DVnqeb610HhswMG8qPTtRcHQLeA1mJuvswt1eTPoCRmxfm9qNCGCKI1XoKWr8kLWmxktGLuM+SKomSRJqBJhry56Uj9m3xVgyxRxci6R134jZPT81g7eTknshxj/DVeBlOqFbUitRh8sjFwNJnPXkHVfsmBxdPw/JAnJzORqNcMbJ6adZva/GXg7G+W815X8BlOxU6tG+HOVcJL9eIsxC+rj8+YTrVbv5ynHLxImYTEnCI6Lryo/51uOOJxjgqPlOmlJ7d4bUIDAuCP5QezBkWQka95sT2PckELDtqJL+jgtABaOMEklJeXU4da+rorbdzwGxNkPGw5UZcHt+hwNhWbm6VSgjGB9SkjIsc6I4sKqyg5Dnleh4rJKtapa/kjzSTzFwCJkyd/0AGneMsyTDbzlAgXpvRNZN/9Xv50Q3ZSAj8iVmMmUPQveAPLRxCSuTKyWBwtE0s995Zf3GJ/3VHmdp5yiTHa16qIKuWADBFJs65v+Lov2y1U1tnP/gj4T7Fp4IzfoBbwisGuX//+hxhbtnu+WltoqTg8nmdbsIhJv+YQkBzpGbzfED40I08IAJY6p5WLFCHNSj+GswTG9crBjHTbUkZBFObgVFKXaP61ZSKXq6siqtzRZNAN8bW7nNXXa0cqgxPDSgfsre7nYhhQoKy3UXMwK5weASfrWwJZbDgH5U5Vw8YAICQFn9dmi2p4oGSkaaFpyAQElqj0TBpJiIHaTUz0oXeYW5UGKIuEdqhdMu2JeAfC9Q4NzL6vzPrAP4UW7stwdkHqmKbOQbIaSyef7XVpxc8BJ6dMiFvi1hMBcOtJM21VAcX92ZRz7EmSshVV3lzyFyAJ7LDvx5kxTPUxSEzEDOE8TujJewYdXcWBnGkao8wNtCaDODA0DPn7Btg3ILpgkJZxa4unKILNDTX59guNI9ZFHxi4IZnIgKIRBlZpF1z91a0p6ptm1yoN9DAVAkXFQU+Z53kBXl+QvZlcATgvGMaBV8p5iOFdLxd5r4RPGh5a6xnDOdSqAcPiNBhNFSuYHa1CxZRlAp9hQyvQe5AanhRBuxNkWffBrpzKf0khkXqKTvn9rFrkZb62TDmkKrVkdr3kOcd/qPnAJbzK4FzO+i3Y/4dot0y+7aenjc1QxEmL9BBc8GbQDnT/zQx3keVcfXGNMfekqfoFWzTFOR2BZmkleibnNHcAJY5RZkfkxZVWRyoBl3CCwbAqlyZizrqDOndgogM8KIvlwH89QemyyTCSqrDNaIwe+oXX0l60HBy4mRg9+4vcrMLVyU6ObZo+Ke0GIIuBDJKS8fA8bRc8tmQQ4MLtc/MTqERGBgwgvd+miwZgNlz62pijHAuKp6KUs54t+LRSirCZvjLd6EsKCjAtIBIdE+6eQw4E+snt8j1eX9qYBCD5LauPSSG2nGjli8RDEzKQ3zmp/H5sHiJrSZBN5Ntgw8hoFmyYd/pCFMzJD0dW+TYKiB9Z74dvqYWmnbrjMEHvcq+rc6qp8cV0nQD06zmh/yHV0ZuNPRJPUEKJEkwV+/1RQn7OEF3TOnswxu7VfTXS8M0995YG7442y50PrScx3AIJjWVS++mgkrc+wFh9djaJOcpygjT44RQJxylAzict0vvVMHlIEL0a5BAKTdr5lZyY/mds1ilqHMofCs+mbywdYSfZG3Mtid6J64Z1jos02UPYtcONOS5goWpHZewgO5Mv07bQwra6SKRTg7E9s+JyUTAXnwDT9+MJt5ofbX/pF11WuCElBwvSNK0YAp3Ee/w9te2LEPSK2gP2bE6Z375ERIPBgW2HiZQhpceMngsbEsbXN6uTP15whbesvtzVXI20Cg/tpHHEvW1X/FIWezvf3NJwk8L2pDpY9TyqwgwIatZaddtOYss0z36mImFGe7udNsRFJzGD7qlJ9rIGKeQB+b+c3EHEVTBhhHXYg9DTX8SDoBndNcL1JVTgWnQF2ujUSDS0d94Ge+ErJ/E1L1rwIQIoh4MsL8VHeo68b/Z32EGuqe5LWlAu+/70O/olQ7ghLua1IdH7rZQ0p0iLZQStV5TYTTsmtlGpwaH1tGip6bjunSnCeWcH3K+7QjzSTJvYtCsIZUEbhkFdHihgVsE7ZqPMgnmWr6rU14A7fZGI1Lco3p3Ibn/FKt3eFDihC+yosgdzD2LaYuRQ/vkjHjklkVVVV/kRq1jWSkjrIKjEYz/VQXbLGseih4kToEwmuExdv4OCRFQgTNuoxLacS1O2gDnirIAp7MgwQV2AIduw6mnZ7L9PVuhEpbgRFIMe3xU+G08C7TwWXheQ9djBQgUEVkDOyCALFQG9OsM7xHh/GJKnmHd28oEy+MCmLvRbFAuwiB+iFRLa8aq45idD2tWHv3YcEmkfBc1p3ZqKZ9aCF6TXT9CWjBGFm+eBHTdoN4ueQmo5IJ+Td9N070ZbjavkPMIglk5Cc/9e9iyFKdFinDZjw8B1jeq/rLWhroafgOnOeSF1pecATwlv8vj5IE46V9pJnu2QKeEDfnlEPVYAdtNBHJ4i1lw5ovOf9lqEn485fbJruG778/GLAqz7rVBLqzs9ZuYr5rbla0Vb6aWLLhk6uLth43zRZJ6nqsU3Pd+6dds9qVkjnmPS8NpGHc1p9HxxW8kOyVH6n1b7pEXspbVs4fe1Np/hJuW7R85SkfTFNrVS4cwR9wfF1xaGCVzbfEB4S6HjPEGco9CLh7zEgXGDlsFYQywiY+fpFpoSSZ8yCf9EKMBdrUDnoyrltqyQeAh0oZ3/BgY94aGys52w4/PROpLRx29YV914IVnyLG7ZS1Onk3GF9Uo4r814Db6FwDimByYGKuWMtu4E9SLW/uZ2U/+2RmpcJf40d5duR01ItP4DLPJ5iyLzYd2VEalPGUosL/zLacarpV76gjoTxA3ByVFvq05XV1o9eaw8IhWgaabOj3s46zGOOmlhi5+7nHXwIkl1rGsjnK7b7rLD9D57mUR0RlUK4DFCaPQZuuAFRYXqTWGxupgMbQw44t3kxMTRlUUF3g3iNssGrWj9AR/bL2zbAMDu++IxPYI8jPDzbIdluNwlgeithkPuwkywCbJNqinebzROLBxwxrKe28CbZVYKc2nUOmMH2o6buDUFbu1FuJUDz9ZfrXYacO9n8UCn2c62xFzo456JhBle0cZsd6bUn/ai+Sc+X2RUbbkOHXD4npaUxcBpCFNuJzbHel9rjF+6cCZCH1styqVugLi5i2IN5J8ZDHmVhV2wno7qT3xJrM5D+McXD5sB1P6ocGe6U5cjIe0AVfpmgpPo7xb2aQcbtMpvI7nGf41GL85MUlHfEJN2zMSf2CCsIcog23AgnWJAd36oUV3QB+rAGIU80bv/Hv332zWYxNWpB/bghg/wUeT//9fgj39Wnt65qgwBR2I5fIUULKWtyzkw55ihWJHmumxg6KXClpEA/PiyOqMAyVkvV0sbWy4VDrzPEErfZjqqMFAsB4O0t6SfeifBiDvh5Ga1yJ2FM8e+8/JwczZMvDLvNaMyEm8i/Wx4hgvZa9A8DWoSpBKSvrIeMxtjibmOU8VizDX3WA/wUb5x4uleJ5TgPPP7WHHXm6AZnXCVKMl5NcHRyyR8gLT8AJ44mri26sJTYIs+JzFYh2qqzAvhGuiWTsH2OpHtlp3nqWlB//MldZvllfNOlw1dWf6NGUyxhElsqXbNKK7YFcpYi4VVO8McCRoLN2YV+NtzM3dNMUcrD+llZ7HYtT2VHWST5qnzmSeuMgPUP1gS68kVvpX3cJniqscpGLS34ga9jTX+o0OZdrAV8zdcyD0w0sREd0X1R0n7LNsHcUp3yGo519HA1F3LQMdoqK6uUh7zDmtbo6UgIrk/qgQZEwlMTvVoTFYEBvSlK3dHlrmzc7Oitfta/fsgtaFdcDMHGVvpj4uXd2E1tndfW8C4OQQkuB26idNlIEIPTeTem2vSFiuSQxc7gdbIHEE0PHaUmFyrW+NHKn4I6Zs5mB8Y57oJYW9bKiCBQtzuJlK4QGbu85Qdqn9ypYnrRwl2bmt/ym3l7vZbN7YSFLWUBD5pgtghhJepMTaMeNPWnpRtihMaBkgF2Io4H3TmcY48BlerczHFZJnNrUTpssGZxxH0ioDtP/MH5dW/91pbocDR+faI+hFPWLh7N1Hx0re54HHt6B+BLbPwI2/DrFmPpNPUM17pbvUs02P6BroJoBc7Cz9lm1X8GlDDfqy5sQgHsypEFmTMY06JhAiYlSkLrp9oB1QOXgjXGLfkxlfv90DtREHRB9gBAxUHPeZ56A0mKqDFtCSUVGBA4imygQ2e+l1vU1gphi7W232ihux+TmIHgB9whl0ZvGqaVLHLD0Av3AmySq7gnXfW05GRgDMLyRPGmjKR8ejfylqKRTVI1kVHSu9fN+t+c9IBFga0Di7KHjdaFbz/Rdxs8I74ymky7rP5HwQ3tWlq4lmr5YwJWwGpsLkYWsMETAiJWqQLIzXvjbS21ZHnyIk7k7m3RD9v2RXARyXNlvF75to4LtRG5xqM0pdqD3wc/XoFy9baexA1BheEWL5lYyfqfy6xQp1DDR9dbqgw8g30Ntwyr5cjrfx3uMOZrLFW6X1n+slTZP8j0WiTrvz2RkcvnlU3TTyjBtnlhDl/9kr5e3yMMR8EDwE1F/1rngKwrIrJVCf8FNGNRS6EGSPlxqrhBhIolWx2u7ak5mGhk7Hi9OdQhSKCdddUh+c8QdZj9zmNO+LGfkcSDRfsyK8grPZ51DXv78xZUUIXsEArtv5y7JxYMPacJGsgCC7yrtNXKiO5Mbpuy2l5zpVeZ1tY1JDX095vPjR7UXrUtuBRCZSlgmwJl9rayJ5BUZGkZXeMO9c+0D16cyCP2XHSM2dFODawS5whUBOx6nDICfsVrpCzPY/FMSWFyFpCIHExzcSVArexAoUrRHaDhwiMr0hitDv9Yx9oTo1MAbvbyXXv63juGZIoljgRjCDhoDdtC9Is6qwcIO4pSG80Hh7HZ4rydGfyu3cQtmQ60IusNTyDV22hn1gE7FiM02xX3cSMgd7QkhV5tw2qdc4slSbGo25ggL9vdQymYEKbC4+/UW5h13/YoMpTY0N0EYROjOCVM/Ky9WbQkPqIfxG2zBajIx5ZHdcwLjczmZnTjo69nJlNQQwNGvfaZA1OgxKakDyvkQs40aAhiRJ2WDmDM3ZOb3UCn/fRNDnqcVs/HAQwQNOKQb3n0ybb1a+JhoTlFmUpEcPoE352siGZk173EzHNB1SX/00l0Yw87UDsos52Zb5lf3AIOk3jFd31M5uwA5P2qZWdzvODG4WvUbJEmw8fAZL/m7xeq55i00DZ0vtePoYickXKqDtIyFU1knFaBT3SLvSJbzFzk6RL+nu3mMGHk/SgqoGWqaIBpMyr2h1Ia8U3Tyz1M/pJvShucSPROaLHOD30BuIGE68xYXL4ysEiqmWffvBPkmcAEQ/faPzFFfMfWXrdgGFnQDcyt8gTGLtlUQtwwF7PBBHsMd5p53Eze1PRG4PZC2L+HSNFHY+/DU7EbgbqydDInP8KVmRpkQZJ+Es/YpF20e3ZbndQ5WT20hWqTQU+fAThce7Mwcsj4sPiy5ALcNva0R5SZN2kIXNcmg4IwmLwY53zsUJvn1oX0DYHbSqjcMmSmSzKeSmko1MSA4E5S0oL6jc1JnuF2F3ks28KpKP6+bsfcW+Yc/nDN8WyJDCaTaPPbNOpbB7o6rX1PUy4xSRzzFjlQCp0git/yI9gkRYGLgczhf/QB6pKAYV3iSuLFA9hqPhK1cWKdMB5XAR1cV64bp0zJfVyjgBL5izv40qe40Vn1cKpUqwSeZIF00Xir/kczxvhp6tTgKGxvpTaAM34i/37bHpxnn84N0+e+3vpD9z+3je0YsXbZfc475WJzuLhkZrm+eqjPLgrXvtdOS/X5RFyJynPef0jwettAGyvTYJqHKv5/imlffLfzMtqJHpuyKat2TjZCcMZ5GuJRSsGaPkOFBaUJSCnH75/naWwjaElamH94UJpeOYSi4k3V3whwCzsFRy8rLXU9LuaSOz0OVYUoIG9jdECSWMFajW/p8V8aY53pxKfWtFNtLfTTJh9LOxAQgku2XLtjUe5hb5iGfA74s8pLQECKKRH5VqXnC1Z0YA82TirJOvR2txHCxJm6RsPmoeSKul9Qqy6Rmewo4pUz3/i88ivXsqyh/FgIe5UUVXFG8Lcfa746fHBlaHWJfvYKGa/M2GIftW1FoSH5klnFBuaZMtHVyTafSuXU/R9j+d/AGF0Vpbr6jJquRIHp+hjZOEzSWfgHs/xZEusV8t8dHOS1FuQVd4qZlbHACcxCqImcqpNnTE/5EtU8bFUZFKAfpQWD/czDWIwFBs9nMv4+/MqcTwBEyaCtme++CzdcDy4cI3hiymmcSrLlj82LrHqyMAW3vxXwLh6/XeSWMnb83miU1SIEkco73Mb4pAe2C5RBvAIKB5OR1g6cgqIRWZLsX/0Bwc4z9fEiShytt4GSOyMiqe5qSr5cew6u0YuaJXR2b85q9DB63mZFmrXCop4iXZm/nzA35XOAg6NCDKWw7P75C2X6oumov21NFs11pLkhzFGwu3a8="} only my frontend can figure out what it is
mook 10mo ago
That makes the mistake of thinking they'll care. Most likely they'll just keep downloading the encrypted garbage and never notice.
- lelanthran 10mo ago
  > That makes the mistake of thinking they'll care. Most likely they'll just keep downloading the encrypted garbage and never notice. Not going to happen. They aren't going to add encrypted garbage to their dataset.
venturecruelty 10mo ago
We could also just punish the people who are ruining society.

chii 10mo ago

How does your normal users get access to the same contents? Or are you having the user solve an encryption puzzle to view it?

vivzkestrel 10mo ago
- the frontend has a decryption module that ll show users what they want to see, - the backend has an encryption module. - The bots and crawlers will see the encrypted text - Can someone who peeks deeply inside the client side code decrypt it? YES - Will 99% of the scrapers bother doing this? NO - The key can be anything, it could be a per session key agreed upon between the client and the server, a csrf token, or even a fixed key
- hollowturtle 10mo ago
  Ehm what would stop ai scrapers from using a browser like a normal user would? Google bot already does, it can execute js and can read spa client side generated content, so it proves can be done at scale, and I'm pretty sure some ai scrapers already do
  - vivzkestrel 10mo ago
    rate limit per ip that progressively keeps decreasing req/mins every few mins?
    - chii 10mo ago
      so why not just do that for these scrapers, rather than complicate it by encrypting and decrypting, which is just obfuscation as the private key is clearly available to the end-user?
      - vivzkestrel 10mo ago
        tbh i did not encrypt decrypt for the ai scrapers at all, a lot of people were previously trying to download data directly from my API that my frontend uses and this kinda pissed me off a bit. So I added encryption/decryption to the mix and will release the newer version. As I mentioned earlier as well, can someone sit through and decrypt it? yes. Will 99% of them do it? no! Thats where I win
    - prmoustache 10mo ago
      What if scrapers ips are millions of smartphones? If I was as evil as an AI scraper company that is not obeying robots.txt I would totally build/buy thousands of small games/apps for mobiles to use them as jumphosts to scape the web. This is probably happening already.
      - vivzkestrel 10mo ago
        in my case my application does not use pagination, it uses infinite scroll, even if you had a million devices that use google chrome, they would all load page 1 and if that req/minute progressively decreaasing thing is implemented, once they start scrolling endlessly they would all hit the rate limits sooner or later, the thing is a human is not going to scroll down a 100 pages but a bot will. once this difference has been factored it, it wont matter how many unique devices they bring into the battle
    - 63stack 10mo ago
      Just skip the whole encryption/decryption shebang then. But no, this does not work, scrapers are using residential ips, and they have enough that they can rotate between them if they get blocked.
  - dirkc 10mo ago
    If you decrypt the content on the client side using an expensive decryption algorithm the scraper needs to spend the computing resource to decrypt.
    - integralid 10mo ago
      Every visitor too. Mobile users are going to love this.
- PunchyHamster 10mo ago
  the AI will just run chrome instance
vivzkestrel 10mo ago
for example this is what my backend renders on the static html page {"z":"gxCit6xEQf0N9IIoG909xfSxypRX7j0BLlXnd5IgWrrzEWzBUxDiS4o4AlIkNYOyuzkY8w4IVoEgUmW02jj84BxhMrNPetK8n6nIn2ORLKQPfIVTS48nGQ1PldtdlpiUNYUm04N+WrMBGGceKYnQoORQO3XbFOVzboFYOWbdMhLdMS2N26YtCUYHwy7jw1AwlXS0Nm1SClb0U1qk2KnDB6s9bcMmpstaOY2RkmGbQ4KMuKHaGzByVzeIPHrtXtNjLnj68cgyLALyO3E5ncqspyjbnuZtfusn2Y49Nu3LVZDDk/JojC6x6GlLZKFEDoiunyfhnqd0SRsDvynKNpFObi3uu+CDrKv2qJgQoyCk392uO4dIqsoJ8iS61DW3hhDoh7nbLsum/E+VdyRdcMo2/0H8cBx6hqxYIOe3hzbMpEsX7YDCy819XYbyi30xQFKe/hGBv3i/LtsD8KdPIHiz2/X/msSh5FfQLVsK2wRx/70FAB4zu4Zj3E9wKMA0HPnDY0BFEtIYFmUNTBMyvFt0k3+MUjGEFoaHT+/LDJNkDzLybsFqpufMhi0RTnkReVVDepU4k9XmxCPyCBYzt74g2BwRS4bgVzmHrJhw9GkS0ZIkHbNhRaUR5iF7mJ/1rIRXDhtLClTEm6DgEO/jAZ/3edLhDrJvCX5gKzZWfsQGgkF2IzHTUdF9HlT5YE/bZ6Bvl9wmHTpnDpnLu3ERGOpuyqkWJG3eji61OB66jx5ZVI2H2o7G4+XDPYRykAs2TJPkxJzR6OU1xaOTmeLjXlqrvG1i7kVf6M+iEJrtvkuPUfDcOSkdSnN+1apnrV+qrNv6Nwhqp89zic2dwvJLYQyL3qT/JQPlwO+TqfhMlklvsPzlMvzECcPgesWTH8LZVdb857FmSbdWjY/rceg4Icg2E6RdCkGEx/JE8XrShAdyl7Udbx8A/+kRALsvTa6+gtd3LxLriMH/wbrasJbply06KaR07SEx7PdWwsdGj8sgmg0l24KRmmqEICKLFfb+k9nCZABUcfjqDlg/x4KDbuRe33Kz+gA++dcq/sLZQm2fRT7UvXFha9RAR9XAEsty1uI3pkjeMcsPRMBIJETkNXG0QUgrjKT44UDYlBSO+mfNexpW6tC1s8gDZJaJBdd/5QzzamnaoAWU9SAksuCc0EkbwmOXxLLqCyXwnXbSZ6LCWqrBpU43BopETh8SBtnrdpTPZWxI8aPHJaF9Qertf7qglqfUWVqnCdALWAO+j+Ma7FkyL04tNX62VmcwqTHQuTQAgZnoo0iZo6wHNPjOgDOxXz+XN8AVA1aIhDEQ8iA+WcPh/+QjDAg/k0wskR2S9MiSP8tVOfD2lMao6A7yuA8FkNK+oOJZVn8IDcIKnGCEem33lC/GFXUGqhi4mh2marxOEHmvYX1V6f3SDlK+NmSMQxbnKVWy3i4A2rJL5Q5l9rZG90rYU53q6ApbK49Zmdy5RyPJpJoDMIa9Py2LtmSEEzW1608Jf7QTUB6WhOLsWBvMlt1fyxtfkOM1h8/WpEga7eDsR+htaqilZgvM/6dyh9C+izWKWK0w+LsJm625e5nHOZs1MQ0DCPA2wu0O/777yOGw0Yw6RTEN6Syy6SY8MpMVAaey21NfCYF9xSMXk15/h4hxdngX65uxNobuE0clCy5BujFbveqrwKHnnDhS79QCgAWQNtm3X5z4dmHxdmqyVBqeu+PMctEXtGdQXOy54nT45FB1MtYSEuejn7q2wJlT31ng6W3Ahb67F33xEhi3gJ14b+RF9mdjoYwfkW/TB4E2/mnYMbVSLHskDEvp/vgYwVCsdHuW/tG7IvN7xG1DVnqeb610HhswMG8qPTtRcHQLeA1mJuvswt1eTPoCRmxfm9qNCGCKI1XoKWr8kLWmxktGLuM+SKomSRJqBJhry56Uj9m3xVgyxRxci6R134jZPT81g7eTknshxj/DVeBlOqFbUitRh8sjFwNJnPXkHVfsmBxdPw/JAnJzORqNcMbJ6adZva/GXg7G+W815X8BlOxU6tG+HOVcJL9eIsxC+rj8+YTrVbv5ynHLxImYTEnCI6Lryo/51uOOJxjgqPlOmlJ7d4bUIDAuCP5QezBkWQka95sT2PckELDtqJL+jgtABaOMEklJeXU4da+rorbdzwGxNkPGw5UZcHt+hwNhWbm6VSgjGB9SkjIsc6I4sKqyg5Dnleh4rJKtapa/kjzSTzFwCJkyd/0AGneMsyTDbzlAgXpvRNZN/9Xv50Q3ZSAj8iVmMmUPQveAPLRxCSuTKyWBwtE0s995Zf3GJ/3VHmdp5yiTHa16qIKuWADBFJs65v+Lov2y1U1tnP/gj4T7Fp4IzfoBbwisGuX//+hxhbtnu+WltoqTg8nmdbsIhJv+YQkBzpGbzfED40I08IAJY6p5WLFCHNSj+GswTG9crBjHTbUkZBFObgVFKXaP61ZSKXq6siqtzRZNAN8bW7nNXXa0cqgxPDSgfsre7nYhhQoKy3UXMwK5weASfrWwJZbDgH5U5Vw8YAICQFn9dmi2p4oGSkaaFpyAQElqj0TBpJiIHaTUz0oXeYW5UGKIuEdqhdMu2JeAfC9Q4NzL6vzPrAP4UW7stwdkHqmKbOQbIaSyef7XVpxc8BJ6dMiFvi1hMBcOtJM21VAcX92ZRz7EmSshVV3lzyFyAJ7LDvx5kxTPUxSEzEDOE8TujJewYdXcWBnGkao8wNtCaDODA0DPn7Btg3ILpgkJZxa4unKILNDTX59guNI9ZFHxi4IZnIgKIRBlZpF1z91a0p6ptm1yoN9DAVAkXFQU+Z53kBXl+QvZlcATgvGMaBV8p5iOFdLxd5r4RPGh5a6xnDOdSqAcPiNBhNFSuYHa1CxZRlAp9hQyvQe5AanhRBuxNkWffBrpzKf0khkXqKTvn9rFrkZb62TDmkKrVkdr3kOcd/qPnAJbzK4FzO+i3Y/4dot0y+7aenjc1QxEmL9BBc8GbQDnT/zQx3keVcfXGNMfekqfoFWzTFOR2BZmkleibnNHcAJY5RZkfkxZVWRyoBl3CCwbAqlyZizrqDOndgogM8KIvlwH89QemyyTCSqrDNaIwe+oXX0l60HBy4mRg9+4vcrMLVyU6ObZo+Ke0GIIuBDJKS8fA8bRc8tmQQ4MLtc/MTqERGBgwgvd+miwZgNlz62pijHAuKp6KUs54t+LRSirCZvjLd6EsKCjAtIBIdE+6eQw4E+snt8j1eX9qYBCD5LauPSSG2nGjli8RDEzKQ3zmp/H5sHiJrSZBN5Ntgw8hoFmyYd/pCFMzJD0dW+TYKiB9Z74dvqYWmnbrjMEHvcq+rc6qp8cV0nQD06zmh/yHV0ZuNPRJPUEKJEkwV+/1RQn7OEF3TOnswxu7VfTXS8M0995YG7442y50PrScx3AIJjWVS++mgkrc+wFh9djaJOcpygjT44RQJxylAzict0vvVMHlIEL0a5BAKTdr5lZyY/mds1ilqHMofCs+mbywdYSfZG3Mtid6J64Z1jos02UPYtcONOS5goWpHZewgO5Mv07bQwra6SKRTg7E9s+JyUTAXnwDT9+MJt5ofbX/pF11WuCElBwvSNK0YAp3Ee/w9te2LEPSK2gP2bE6Z375ERIPBgW2HiZQhpceMngsbEsbXN6uTP15whbesvtzVXI20Cg/tpHHEvW1X/FIWezvf3NJwk8L2pDpY9TyqwgwIatZaddtOYss0z36mImFGe7udNsRFJzGD7qlJ9rIGKeQB+b+c3EHEVTBhhHXYg9DTX8SDoBndNcL1JVTgWnQF2ujUSDS0d94Ge+ErJ/E1L1rwIQIoh4MsL8VHeo68b/Z32EGuqe5LWlAu+/70O/olQ7ghLua1IdH7rZQ0p0iLZQStV5TYTTsmtlGpwaH1tGip6bjunSnCeWcH3K+7QjzSTJvYtCsIZUEbhkFdHihgVsE7ZqPMgnmWr6rU14A7fZGI1Lco3p3Ibn/FKt3eFDihC+yosgdzD2LaYuRQ/vkjHjklkVVVV/kRq1jWSkjrIKjEYz/VQXbLGseih4kToEwmuExdv4OCRFQgTNuoxLacS1O2gDnirIAp7MgwQV2AIduw6mnZ7L9PVuhEpbgRFIMe3xU+G08C7TwWXheQ9djBQgUEVkDOyCALFQG9OsM7xHh/GJKnmHd28oEy+MCmLvRbFAuwiB+iFRLa8aq45idD2tWHv3YcEmkfBc1p3ZqKZ9aCF6TXT9CWjBGFm+eBHTdoN4ueQmo5IJ+Td9N070ZbjavkPMIglk5Cc/9e9iyFKdFinDZjw8B1jeq/rLWhroafgOnOeSF1pecATwlv8vj5IE46V9pJnu2QKeEDfnlEPVYAdtNBHJ4i1lw5ovOf9lqEn485fbJruG778/GLAqz7rVBLqzs9ZuYr5rbla0Vb6aWLLhk6uLth43zRZJ6nqsU3Pd+6dds9qVkjnmPS8NpGHc1p9HxxW8kOyVH6n1b7pEXspbVs4fe1Np/hJuW7R85SkfTFNrVS4cwR9wfF1xaGCVzbfEB4S6HjPEGco9CLh7zEgXGDlsFYQywiY+fpFpoSSZ8yCf9EKMBdrUDnoyrltqyQeAh0oZ3/BgY94aGys52w4/PROpLRx29YV914IVnyLG7ZS1Onk3GF9Uo4r814Db6FwDimByYGKuWMtu4E9SLW/uZ2U/+2RmpcJf40d5duR01ItP4DLPJ5iyLzYd2VEalPGUosL/zLacarpV76gjoTxA3ByVFvq05XV1o9eaw8IhWgaabOj3s46zGOOmlhi5+7nHXwIkl1rGsjnK7b7rLD9D57mUR0RlUK4DFCaPQZuuAFRYXqTWGxupgMbQw44t3kxMTRlUUF3g3iNssGrWj9AR/bL2zbAMDu++IxPYI8jPDzbIdluNwlgeithkPuwkywCbJNqinebzROLBxwxrKe28CbZVYKc2nUOmMH2o6buDUFbu1FuJUDz9ZfrXYacO9n8UCn2c62xFzo456JhBle0cZsd6bUn/ai+Sc+X2RUbbkOHXD4npaUxcBpCFNuJzbHel9rjF+6cCZCH1styqVugLi5i2IN5J8ZDHmVhV2wno7qT3xJrM5D+McXD5sB1P6ocGe6U5cjIe0AVfpmgpPo7xb2aQcbtMpvI7nGf41GL85MUlHfEJN2zMSf2CCsIcog23AgnWJAd36oUV3QB+rAGIU80bv/Hv332zWYxNWpB/bghg/wUeT//9fgj39Wnt65qgwBR2I5fIUULKWtyzkw55ihWJHmumxg6KXClpEA/PiyOqMAyVkvV0sbWy4VDrzPEErfZjqqMFAsB4O0t6SfeifBiDvh5Ga1yJ2FM8e+8/JwczZMvDLvNaMyEm8i/Wx4hgvZa9A8DWoSpBKSvrIeMxtjibmOU8VizDX3WA/wUb5x4uleJ5TgPPP7WHHXm6AZnXCVKMl5NcHRyyR8gLT8AJ44mri26sJTYIs+JzFYh2qqzAvhGuiWTsH2OpHtlp3nqWlB//MldZvllfNOlw1dWf6NGUyxhElsqXbNKK7YFcpYi4VVO8McCRoLN2YV+NtzM3dNMUcrD+llZ7HYtT2VHWST5qnzmSeuMgPUP1gS68kVvpX3cJniqscpGLS34ga9jTX+o0OZdrAV8zdcyD0w0sREd0X1R0n7LNsHcUp3yGo519HA1F3LQMdoqK6uUh7zDmtbo6UgIrk/qgQZEwlMTvVoTFYEBvSlK3dHlrmzc7Oitfta/fsgtaFdcDMHGVvpj4uXd2E1tndfW8C4OQQkuB26idNlIEIPTeTem2vSFiuSQxc7gdbIHEE0PHaUmFyrW+NHKn4I6Zs5mB8Y57oJYW9bKiCBQtzuJlK4QGbu85Qdqn9ypYnrRwl2bmt/ym3l7vZbN7YSFLWUBD5pgtghhJepMTaMeNPWnpRtihMaBkgF2Io4H3TmcY48BlerczHFZJnNrUTpssGZxxH0ioDtP/MH5dW/91pbocDR+faI+hFPWLh7N1Hx0re54HHt6B+BLbPwI2/DrFmPpNPUM17pbvUs02P6BroJoBc7Cz9lm1X8GlDDfqy5sQgHsypEFmTMY06JhAiYlSkLrp9oB1QOXgjXGLfkxlfv90DtREHRB9gBAxUHPeZ56A0mKqDFtCSUVGBA4imygQ2e+l1vU1gphi7W232ihux+TmIHgB9whl0ZvGqaVLHLD0Av3AmySq7gnXfW05GRgDMLyRPGmjKR8ejfylqKRTVI1kVHSu9fN+t+c9IBFga0Di7KHjdaFbz/Rdxs8I74ymky7rP5HwQ3tWlq4lmr5YwJWwGpsLkYWsMETAiJWqQLIzXvjbS21ZHnyIk7k7m3RD9v2RXARyXNlvF75to4LtRG5xqM0pdqD3wc/XoFy9baexA1BheEWL5lYyfqfy6xQp1DDR9dbqgw8g30Ntwyr5cjrfx3uMOZrLFW6X1n+slTZP8j0WiTrvz2RkcvnlU3TTyjBtnlhDl/9kr5e3yMMR8EDwE1F/1rngKwrIrJVCf8FNGNRS6EGSPlxqrhBhIolWx2u7ak5mGhk7Hi9OdQhSKCdddUh+c8QdZj9zmNO+LGfkcSDRfsyK8grPZ51DXv78xZUUIXsEArtv5y7JxYMPacJGsgCC7yrtNXKiO5Mbpuy2l5zpVeZ1tY1JDX095vPjR7UXrUtuBRCZSlgmwJl9rayJ5BUZGkZXeMO9c+0D16cyCP2XHSM2dFODawS5whUBOx6nDICfsVrpCzPY/FMSWFyFpCIHExzcSVArexAoUrRHaDhwiMr0hitDv9Yx9oTo1MAbvbyXXv63juGZIoljgRjCDhoDdtC9Is6qwcIO4pSG80Hh7HZ4rydGfyu3cQtmQ60IusNTyDV22hn1gE7FiM02xX3cSMgd7QkhV5tw2qdc4slSbGo25ggL9vdQymYEKbC4+/UW5h13/YoMpTY0N0EYROjOCVM/Ky9WbQkPqIfxG2zBajIx5ZHdcwLjczmZnTjo69nJlNQQwNGvfaZA1OgxKakDyvkQs40aAhiRJ2WDmDM3ZOb3UCn/fRNDnqcVs/HAQwQNOKQb3n0ybb1a+JhoTlFmUpEcPoE352siGZk173EzHNB1SX/00l0Yw87UDsos52Zb5lf3AIOk3jFd31M5uwA5P2qZWdzvODG4WvUbJEmw8fAZL/m7xeq55i00DZ0vtePoYickXKqDtIyFU1knFaBT3SLvSJbzFzk6RL+nu3mMGHk/SgqoGWqaIBpMyr2h1Ia8U3Tyz1M/pJvShucSPROaLHOD30BuIGE68xYXL4ysEiqmWffvBPkmcAEQ/faPzFFfMfWXrdgGFnQDcyt8gTGLtlUQtwwF7PBBHsMd5p53Eze1PRG4PZC2L+HSNFHY+/DU7EbgbqydDInP8KVmRpkQZJ+Es/YpF20e3ZbndQ5WT20hWqTQU+fAThce7Mwcsj4sPiy5ALcNva0R5SZN2kIXNcmg4IwmLwY53zsUJvn1oX0DYHbSqjcMmSmSzKeSmko1MSA4E5S0oL6jc1JnuF2F3ks28KpKP6+bsfcW+Yc/nDN8WyJDCaTaPPbNOpbB7o6rX1PUy4xSRzzFjlQCp0git/yI9gkRYGLgczhf/QB6pKAYV3iSuLFA9hqPhK1cWKdMB5XAR1cV64bp0zJfVyjgBL5izv40qe40Vn1cKpUqwSeZIF00Xir/kczxvhp6tTgKGxvpTaAM34i/37bHpxnn84N0+e+3vpD9z+3je0YsXbZfc475WJzuLhkZrm+eqjPLgrXvtdOS/X5RFyJynPef0jwettAGyvTYJqHKv5/imlffLfzMtqJHpuyKat2TjZCcMZ5GuJRSsGaPkOFBaUJSCnH75/naWwjaElamH94UJpeOYSi4k3V3whwCzsFRy8rLXU9LuaSOz0OVYUoIG9jdECSWMFajW/p8V8aY53pxKfWtFNtLfTTJh9LOxAQgku2XLtjUe5hb5iGfA74s8pLQECKKRH5VqXnC1Z0YA82TirJOvR2txHCxJm6RsPmoeSKul9Qqy6Rmewo4pUz3/i88ivXsqyh/FgIe5UUVXFG8Lcfa746fHBlaHWJfvYKGa/M2GIftW1FoSH5klnFBuaZMtHVyTafSuXU/R9j+d/AGF0Vpbr6jJquRIHp+hjZOEzSWfgHs/xZEusV8t8dHOS1FuQVd4qZlbHACcxCqImcqpNnTE/5EtU8bFUZFKAfpQWD/czDWIwFBs9nMv4+/MqcTwBEyaCtme++CzdcDy4cI3hiymmcSrLlj82LrHqyMAW3vxXwLh6/XeSWMnb83miU1SIEkco73Mb4pAe2C5RBvAIKB5OR1g6cgqIRWZLsX/0Bwc4z9fEiShytt4GSOyMiqe5qSr5cew6u0YuaJXR2b85q9DB63mZFmrXCop4iXZm/nzA35XOAg6NCDKWw7P75C2X6oumov21NFs11pLkhzFGwu3a8="} only my frontend can figure out what it is

mook 10mo ago

That makes the mistake of thinking they'll care. Most likely they'll just keep downloading the encrypted garbage and never notice.

lelanthran 10mo ago
> That makes the mistake of thinking they'll care. Most likely they'll just keep downloading the encrypted garbage and never notice. Not going to happen. They aren't going to add encrypted garbage to their dataset.

venturecruelty 10mo ago

We could also just punish the people who are ruining society.