16 ms·
We accidentally burned through 200GB of proxy bandwidth in 6 hours
- 8organicbits 2y agoWhat infrastructure is this using? Bandwidth seems pretty pricy
- tobyjsullivan 2y agoNo kidding. AWS's notoriously expensive data transfer is only $0.09/GB. Who's charging $2.50/GB? Are they running on a cellular SIM with no data plan?
- mikeocool 2y agoSounds like they are running a web scarping business -- so maybe? Using a cellular connection would be one way to help not get immediately capcha-ed by every site using cloudflare.
- blitzar 2y agoThey should really setup their scraper and (exfil the data) via regular connections.
- ronsor 2y agoResidential rotating proxy providers charge very high rates for data, on the order of $1 - $10 per GB. (These providers often do run their proxies through the cellular network, actually.)
- SteveNuts 2y agoIs this something where end users can get paid for doing nothing other than proxying some traffic through their ISP?
- ronsor 2y agoThat's probably where some of the proxies come from.
- r1ch 2y agoThe end user typically has their device compromised by using free apps where the developers were bribed $$$ to add the proxy "SDK". The botnet operator then rents out the bandwidth at exorbitant rates to anyone who will pay for it. Chrome extensions are also a huge source of this, they look for extensions with a large install base and then make an offer to buy it to turn all the users into proxies.
- slt2021 2y agoend users install shady VPN apps/extensions to watch pirated content, and become part of residential proxy mesh/botnet
- ipython 2y agoIf by “some” traffic you mean botnets, sneaker and ticket scalpers, scammers, content scrapers, credential stuffers … generally scummy stuff, sure. Based on this blog post I would not do any business with Skyvern, if they indeed do business with this underworld of bottom feeders.
- mrguyorama 2y agoSure, if you want a whole bunch of legitimately malicious traffic to be attributed to your internet account.
- floam 2y agoYes. Google “honeygain”
- perks_12 2y ago200GB for $500? What cloud is this?
- hooverd 2y agoapi.skyvern.com is a CNAME to an EC2 ALB, but even using a NAT Gateway ($$$) I can't make more than $1/GB add up.
- intunderflow 2y agoLooks like webshare from the screenshot
- jsnell 2y agoI don't think it's a cloud. It's more likely a residential proxy network, which are typically created by installing malware on users' machines. The operators of these proxy networks want to avoid detection by both the users whose bandwidth they're stealing, and by the companies whose data is being scraped. So they want to make the bandwidth very expensive. And that expensive bandwidth in turn means that their only clients are dodgy as well. Either people looking to scrape data without consent and monetize it, or outright criminals.
- bscphil 2y agoIt's kind of surprising that a presumptively legitimate company (and YC-funded startup) would out themselves as buying black market residential proxy bandwidth, isn't it?
- jsheard 2y agoTheir frontpage also advertises the ability to pass CAPTCHAs, whether by automation or more likely by delegating them to third-world CAPTCHA farms. If that's a major selling point for your automation service then your target market probably ranges from dubious (e.g. data scrapers trying to get around limits) to extremely dubious (e.g. ticket scalpers, spammers, click fraud, etc).
- tristor 2y agoI would have liked to see a bit more of 5 Whys here. It seems like a consistent lesson that startups have to learn over and over is how to manage external dependencies, and particularly the dangers of having Google as a dependency. This is new Chrom(e|ium) behavior, and it has a real cost, both for this company and for users, which may or may not be worth the ROI, but this is what happens when you have a large scale external dependency: stuff moves without your knowledge, consent, or control. Instead of Always. Be. Closing. it should be Always. Be. Mitigating. Dependencies. for startups.
- suchintan 2y agoThis is a great callout. We had an internal discussion about how to manage dependencies effectively, and we made the decision accept the risk that comes with blindly relying on Chrome for now, instead of investing heavily in mitigating that risk today. The main motivator was for us to continue moving fast, and accept that we have a few hard dependencies in our business. The goal is to find product market fit, then allocate time to de-risk some of these hard dependencies. If we fail to find product market fit, this may not matter at all
- tristor 2y agoI think that's a fair strategy. Strong PMF generally overcomes weak execution, the challenge is that when you have hard dependencies on entities like Google or Apple it can easily become existential. Even if you choose to move forward with this dependency you should establish guard rails within your system to ensure you catch shifts faster that may be impactful and have a plan for mitigation. For instance, you should identify key points of integration and possible alternatives even if you choose not to migrate now, so that a future migration is better understood and can be discussed intelligently in the heat of the moment. Even internal documentation can assist as a mitigation for dependency risk.
- suchintan 2y agoYeah exactly. One action item from this is that we need to add anomaly detection to our proxy usage metrics so we can catch this in 15 minutes instead of 6 hours :)
- ang_cire 2y agoBlocking Google from downloading anything onto your computer without consent is always a good idea.
- keepamovin 2y agoyou shouldn’t be paying by the terabyte. Colocate and just pay for the maximum throughout. Far better rates
- skeeter2020 2y agodoesn't work when the sites you're scraping block the IPs/range of your server. They're using a proxy botnet that costs a premium
- keepamovin 2y agomakes sense but you shouldn't be paying a premium for a non-premium service. IP blocks and bandwidth have low unit cost at scale.
- bradley13 2y ago"We run leverage proxy networks and run headful browser instances" Um...say what? I'm pretty broadly based in IT, and I have no idea what that means.
- suchintan 2y agoHaha, apologies for the language! We use residential proxy networks when running Skyvern to help simulate real human behaviour (because that's what Skyvern is trying to do). We run headful browser instances (meaning a real chrome instance running with a real viewport) for the same cause!
- rustdeveloper 2y ago[flagged]
- deleted 2y ago[deleted]
- olliej 2y agoHonestly given many of these stories, $500 seems to be getting off pretty lightly. It’s still absurd to me that many (most?) of these hosting/bandwidth providers don’t seems to allow automatic cut offs and such
- omoikane 2y agoThe discussion linked in the post is from 2022, and the corresponding issue has already been fixed: https://issues.chromium.org/issues/40220332 https://issues.chromium.org/issues/40220332 I wonder if there is a more recent bug related to this?
- TRiG_Ireland 2y agoI think that Chrome is still doing a lot of downloads. They're just no longer showing them to the user.
- fijiaarone 2y agoAnd uploads.
- meindnoch 2y ago>200GB of proxy bandwidth Gigabyte is a measure of information. Bandwidth is information transmitted over time.
- deleted 2y ago[deleted]
- metadat 2y ago200GB is nothing since 2018 when AT&T mass introduced their 1-gig symmetric fiber. Any single common gigabit link can run 200GB in 15 minutes. On any gig link, over the course of 6 hours you can transmit a little more than 4TB one way.. which is 40x more.
- Johnny555 2y agoToo bad AWS didn’t get that memo, 200GB would cost $18 there, and somehow the company in the original post is paying $500 for that bandwidth with whoever their proxy host is.
- suchintan 2y agoHaha unfortunately we use residential proxies under the hood to simulate real users (as you'd expect from AI agents), where bandwidth is significantly more expensive!
- donmcronald 2y agoHow does a residential proxy work? Do people rent out their internet connections to commercial services?
- mike_d 2y agoComputers getting infected with malware, pre-compromised cheap internet devices from Amazon/Wish.com, and game developers monetizing "free" games by running proxies in the background. There are usually a few layers of resellers so technically the proxy provider can throw their hands up in the air and say they are unaware of any malicious activity.
- stevenicr 2y agosomeone in your household adds a 'free vpn app' to their device and that app sells off portions of your bandwidth to others. Someone has a cheap android tv box or similar unpatched thing with your wifi password (one botnet alone has over 1.3 million android boxes working for it as of this week - https://www.securityweek.com/1-3-million-android-tv-boxes-infected-by-vo1d-malware/ https://www.securityweek.com/1-3-million-android-tv-boxes-in... ) Someone start cobbling lists of these so I can block them in a different way.
- patmcc 2y agoI'm now expecting we'll see a couple things in the next few years: 1. An explosion of residential proxy networks and other stuff to circumvent blocking of cloud IP ranges, for all the various AI scraping tools to use. 2. A corresponding explosion of countermeasures to the above. Instead of blocking suspicious IPs, maybe they get a 3GB file on their request to /scrape-target.html
- mrtesthah 2y agoI think that may be against the ToS of most residential ISPs.
- bdcravens 2y agoPerhaps, but it's already fairly prodigious. Among "ethical" providers, it's often bundled as a background service in a lot of clickwrap "freeware". (To say nothing of compromised computers in a botnet)
- bdcravens 2y agoPerhaps an explosion of usage. There's already a few very large residential proxy networks.
- cute_boi 2y agoAnd, I get frequently contacted by Bright data to install their code in my repo.
- monyasau 2y ago[dead]
- tim_at_ping 2y agoHello, A (different) proxy company owner here. This sucks! Sorry that you lost out on so much bandwidth. Feel free to reach out to me at tim@pingproxies.com and I'd be happy to get you set up on our service and credit you with 100GB of free bandwidth to help soften the blow. I'll also be able to get you pricing alittle better than you're currently on if you are interested ;) Within the next few months we're also releasing a bunch of tools to help stop things like this happening on our residential network such as some intelligent routing logic, spend controls and a few other things. You may also want to look into Static Residential ISP Proxies - we charge these per IP address rather than bandwidth and they often end up more economical. We work with carriers like Spectrum, Comcast & AT&T directly to get IP addresses on their networks so they look like residential connections but host them in datacenters - this way you get 99.99%+ availability, 1G+ throughput, stable IP addresses and have unlimited bandwidth. @ everyone else in the thread; if you run a start-up and need proxies then email me - happy to credit you with 50GB free residential bandwidth + give some advice on infra if needed. Cheers, Tim at Ping
- SteveNuts 2y agoI’m interested to know how your residential connections are sourced. It says they’re “ethically sourced”, but it seems like malware/botnet like behavior. Are these residential users aware their traffic is siphoned off for this purpose?
- chimen 2y agoThey are never ethically sourced. Ethically for them means placing a phrase in a 10k word TOS when victims installs app X, game y which loads their sdk. Ethically here means "we warned them in a TOS"
- LargoLasskhyfv 2y agoHuh? > We work with carriers like Spectrum, Comcast & AT&T directly to get IP addresses on their networks so they look like residential connections but host them in datacenters - this way you get 99.99%+ availability, 1G+ throughput, stable IP addresses and have unlimited bandwidth.
- tcfhgj 2y agoPlease, gigabyte isn't a unit of bandwidth. Bandwidth is measured in data/time
- Saris 2y agoTell that to every single ISP and Cell provider.
- tcfhgj 2y agoData volume makes more sense than bandwidth: https://www.telekom.de/prepaid-aktivierung/en/start https://www.telekom.de/prepaid-aktivierung/en/start Their explanation of bandwidth looks fine as well: https://dih.telekom.com/en/glossary/bandwidth https://dih.telekom.com/en/glossary/bandwidth
- elphinstone 2y agoSkyvern is a great name, very evocative. Typical arrogant Google, downloading trash to the user without consent.
- dusted 2y ago" 200GB of proxy bandwidth was approximately $500 burned over the course of 6 hours" The fuck ? So Internet is literally more expensive than buying a drive at amazon, paying for shipping, filling it up putting it on a truck towards a destination anywhere in the world.
- HaZeust 2y agoI mean, yeah, bandwidth costs aren't just about bytes, they're about energy, infrastructure, and routing complexity too.
- dusted 2y agoYes, but, honestly, so is the production of rust spinners :)
- chrisandchris 2y agoWell, one part of the source of the problem is this, where I not even understand all of the words (a bit exaggarating): > Skyvern is an AI agent that helps companies automate workflows in the browser. We run leverage proxy networks and run headful browser instances in the cloud to facilitate most of our automations. So you're doing Selenium, just with Cloud, AI and some other buzzwords you found while Googling?
- hkon 2y agoLiterally means cloudprotection in Norwegian. Thought for a second we had gotten our own cloudflare.
- keremyilmaz 2y agoI was so surprised to read this, especially since we definitely didn't know about it when naming Skyvern. Out of curiosity, I checked both Cambridge's Norwegian-English dictionary and a few other Norwegian sources, but I couldn't find 'Skyvern' listed anywhere. Makes me wonder if Google translate just hallucinated the meaning.
- hkon 2y agoNo, in Norwegian we make words by joining multiple word together (closed compound) in this case sky(cloud) and vern(protection). https://en.m.wikipedia.org/wiki/Compound_(linguistics) https://en.m.wikipedia.org/wiki/Compound_(linguistics)
- keremyilmaz 2y agoAh, now that makes sense! Thanks for sharing it. We definitely didn’t have that in mind when naming it Skyvern, but it's cool to see how it could be interpreted that way
- sam0x17 2y agoGosh I regularly burn through that much just updating games in steam :D. Not proxy bandwidth of course but isn't it funny that the the line between regular usage and $$$ can be what is using the bandwidth. Or rather, isn't it funny that regular consumers expect to be able to use multiple terabytes of data for < $100/mo but the same can still be thousands in other enterprise domains