6 ms·

Start mining - FREE 100MM tweet db

12 points by calufa 15y ago

Sup, I have been collecting tweets for 4 days now, using a app that I haven coding for the last 5 months. The reason why I did this app was because I wanted to make user-based recommendations, and other types of data mining using Mahout, and I didnt find enough data for my experiments. About the app I am using a single 8GB-ram Centos Server hosted on the Rackspace cloud with a cost of less than 15 dollars per day. It can process up to 100 (90 - 105) twitter profiles per second. It works with a average of 2GB of ram and 90% CPU. Its completely fault tolerant. It can process other social networks as well using a simple parse-template. I was able to collect 90+ million tweets from more than 6 million -- the db has 20MM users -- users using JAVA, memcache, mysql, php (visualization), a non ACID architecture, using a object-like structure (no-sql?). I hope this datasets helps you get into the big data world. The current sql dump is too big (66GB) to put in one of my servers so please skypeme:calufaxp or email me calufa{a}gmail.com if you want the data. BTW, the data is FREE... If anyone has a server where I can upload this sql and let others download it let me know.

fhsdfh 15y ago
Can someone help a novice and explain what types of things can be achieved with such a dump?
mikelbring 15y ago
Throw it on a torrent?
- calufa 15y ago
  tried that before and I dont know why it doesnt seed. I have had problems with my OSX admin user before... reinstalling everything is not really a option... :/
cstrouse 15y ago
If you upload it to my server I will help you seed it from two locations. Email me for details.
calufa 15y ago
Download the db here. Please dont abuse http://scramblermedia.com/twitter.sql.gz http://scramblermedia.com/twitter.sql.gz
- sidmitra 15y ago
  Awesome, thanks. I'll see if i can seed a torrent. Will post here if i do that.
  - calufa 15y ago
    super
  - deleted 15y ago
    [deleted]
- calufa 15y ago
  I had to remove the file from the server because of bandwidth issues.
calufa 15y ago
BTW, the sql contains: - bio data(7MM) - tweets(90MM) - followers(10MM) - following(10MM) - location(7MM) - profileName(10MM) - relationships (100MM) - websites (4.5MM) - users(20MM) -- 350+MM rows total --
JoachimSchipper 15y ago
So, you are scraping Twitter (likely violating their ToS) to get users' Tweets (likely violating their copyright) and now posting about it on HN? When Twitter is selling chunks of its stream, e.g. via InfoChimps? I don't want to be mean, but this doesn't strike me as a very good idea.
- MrMcDowall 15y ago
  It didn't work out well for the last guys who tried it :( http://discovertext.com/osamabinladen.aspx http://discovertext.com/osamabinladen.aspx
jparicka 15y ago
http://codebiatch.com/ http://codebiatch.com/ .. the file is still uploading if it's not in there yet. Good luck with your project!
- calufa 15y ago
  seems to work fine ;) ... thanks
uptown 15y ago
Thanks for the data. Guess it's time to see whether my ISP has a data cap or not.
calufa 15y ago
Thanks to Jason for putting this up on the archive.org site: http://www.archive.org/details/2011-05-calufa-twitter-sql http://www.archive.org/details/2011-05-calufa-twitter-sql

12 points by calufa 15y ago

fhsdfh 15y ago

Can someone help a novice and explain what types of things can be achieved with such a dump?

mikelbring 15y ago

Throw it on a torrent?

calufa 15y ago
tried that before and I dont know why it doesnt seed. I have had problems with my OSX admin user before... reinstalling everything is not really a option... :/

cstrouse 15y ago

If you upload it to my server I will help you seed it from two locations. Email me for details.

calufa 15y ago

Download the db here. Please dont abuse http://scramblermedia.com/twitter.sql.gz http://scramblermedia.com/twitter.sql.gz

sidmitra 15y ago
Awesome, thanks. I'll see if i can seed a torrent. Will post here if i do that.
- calufa 15y ago
  super
- deleted 15y ago
  [deleted]
calufa 15y ago
I had to remove the file from the server because of bandwidth issues.

calufa 15y ago

BTW, the sql contains: - bio data(7MM) - tweets(90MM) - followers(10MM) - following(10MM) - location(7MM) - profileName(10MM) - relationships (100MM) - websites (4.5MM) - users(20MM) -- 350+MM rows total --

JoachimSchipper 15y ago

So, you are scraping Twitter (likely violating their ToS) to get users' Tweets (likely violating their copyright) and now posting about it on HN? When Twitter is selling chunks of its stream, e.g. via InfoChimps? I don't want to be mean, but this doesn't strike me as a very good idea.

MrMcDowall 15y ago
It didn't work out well for the last guys who tried it :( http://discovertext.com/osamabinladen.aspx http://discovertext.com/osamabinladen.aspx

jparicka 15y ago

http://codebiatch.com/ http://codebiatch.com/ .. the file is still uploading if it's not in there yet. Good luck with your project!

calufa 15y ago
seems to work fine ;) ... thanks

uptown 15y ago

Thanks for the data. Guess it's time to see whether my ISP has a data cap or not.

calufa 15y ago

Thanks to Jason for putting this up on the archive.org site: http://www.archive.org/details/2011-05-calufa-twitter-sql http://www.archive.org/details/2011-05-calufa-twitter-sql