4 ms·
Show HN: I scraped 3.2B TikTok profiles and 9B posts to build this search engine
B = Billion
Aiming to scrape 1 Trillion TikTok posts by this year.
- avadhesh18 2y agoCould you share how you managed to scrape so much data? I'm especially curious about how you got around rate limits etc.
- IWantAllTheData 2y agoI use proxy.
- bilekas 2y agoI'm here for this. If only to highlight the amount of data leaked willingly. That said, I'd there a breach of service doing this?
- IWantAllTheData 2y agoAll the data were scraped without creating a TikTok account, AKA public data. Google.com has scraped more TikTok profiles and videos than me. In fact, one the way i did it is by scraping google search results itself.
- bilekas 2y agoThere will be some backlash for this, it can seem nefarious when presented this way. That said, can we see your work?
- IWantAllTheData 2y agoYes, I did receive some backlash, which is understandable. However, this is a very useful tool for content creators to find out what's trending on TikTok. It is also used by brands to analyze market sentiment about them. I can't share the specifics, but i can explain the methodology in private. My contacts are on the website.
- paweladamczuk 2y agoI didn't even think this could be controversial before reading the comments. I always assume what I post on social media is public. Do others have different assumptions? Why would people get angry at a random person collecting data they published?
- minhazm 2y agoIt may not be quite that simple, see hiQ Labs vs LinkedIn[1]. [1] https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn
- IWantAllTheData 2y agoWill check it out thanks. I'm always ready to take down the site if TikTok reach out me to.
- Oras 2y agoThat's not about public data. Reference [7] talked about creating fake accounts to scrape. > hiQ had prevailed on the Computer Fraud and Abuse Act (CFAA) “unauthorized access” issue related to public website data but was facing a ruling that it had breached LinkedIn’s User Agreement due to its scraping and creation of fake accounts (subject to its equitable defenses). https://natlawreview.com/article/hiq-and-linkedin-reach-proposed-settlement-landmark-scraping-case https://natlawreview.com/article/hiq-and-linkedin-reach-prop...
- IWantAllTheData 2y agoSo many useful stuffs in the comments. Thank you, will give this a read as well.
- Oras 2y agoFiltering works pretty well, and the design is well executed. For those who complain about scraping, if this is public data, as the OP mentioned, then I don't see how it is different from Google. A clean search would benefit the creators, and give them more visibility. The first thing I had in mind was searching by keyword and number of followers to see which ones would fit a startup to use influencer marketing. Imagine doing that manually on TikTok or through Google.
- IWantAllTheData 2y agoI appreciate the positive comment. Thanks! Fun fact: I did the UI Design, frontend/backend web development, database, servers, scraping and everything else by myself. It took me few months but i guess its worth it.
- costco 2y agoWow this is pretty amazing... I imagine your proxy bill must be pretty huge! I always wondered how companies like Clearview scraped Instagram etc at scale. Do you add user to a queue, get all of that users posts, then add everyone they follow/everyone that is following them to the queue, and repeat? With Twitter I know from experience you can predict what the next snowflake IDs will be so in theory you could enumerate the whole site. If I recall correctly Tiktok has a similar ID scheme but I think people weren't able to figure out what some of the last bits represented.
- IWantAllTheData 2y agoThank you. Yes proxy bill is quite high! Yes you are right, the ID enumeration method doesn't work with TikTok.
- KomoD 2y agoPretty cool, always wanted to do a large scale scraping project like this but never really got around to it. What's your tech stack?
- IWantAllTheData 2y agoAll the scrapers are in Python but im slowly transferring it to Golang, i find it much faster. The website is NextJS.
- alexliu518 2y agoThis is awesome! Quick questions: How do you handle the massive data scale and keep it all up-to-date? Have you faced any ethical challenges with scraping so much public data?
- jayemar 2y agoMaybe this is because I don't use TikTok but I don't know what "Eng. Rate (%)" means, so I'm not sure that something you should appreciate.
- IWantAllTheData 2y agoThanks for letting me know. I will choose a better wording. Eng. Rate (%) is Engagement Rate. Which is a common metric marketers look into when hiring an influencer.