4 ms·

The twitter social graph (follow connections between people) is my data source, I extract it from the API and cache it in a database. The mariadb table storing

by me_bx 12y ago

The twitter social graph (follow connections between people) is my data source, I extract it from the API and cache it in a database. The mariadb table storing this information currently takes a bit more than 500GB, it has about 4 billion rows (based on the statistics, I don't run SELECT count(*) on it anymore). I usually don't use the term "big data" because the buzzword is so popular that it doesn't mean anything anymore.

hdra 12y ago
I'm looking into doing something similar as well, may I know how long have you been collecting the data, and how do you decide which data to collect? I have been collecting data from the twitter API for a few days as well. I wanted to get an idea of average tweeting pattern, but without access to the firehose API, I got a feeling that the sample I have isn't very "neutral" as I have been mostly pulling from the popular and local tweets endpoints. Any advice on how should I approach these kind of data collection?
- me_bx 12y ago
  It all depends on your use case. If user streams are not releavant to you, you may use the `filter` endpoint instead of `sample`, and focus on keywords describing a relevant niche for your analysis. In case you want to limit yourself to tweets geo-located in a certain location, you have to be aware that the bounding box filter of twitter is buggy: it will return you tweets geo-located outside of the desired area, and you are not sure to be getting all tweets.
billyhoffman 12y ago
Won't SELECT count(*) be super slow? Isn't SELECT Count(some_primary_key) a better idea?
- icebraining 12y ago
  It should be the same. count(*) only needs to return the number of columns (regardless of its value), so it can use only indexes, while count(column) must only count non-null values. But since the primary is non-null, it should end up taking the same time.
  - billyhoffman 12y ago
    Wow, just tested this against a table with some pretty wide columns and ~ 7M rows. They take exactly the same time! I would have thought COUNT() would be like SELECT , but I guess the query planner is smart enough to know what to do. Thanks!
- deleted 12y ago
  [deleted]

by me_bx 12y ago

hdra 12y ago

I'm looking into doing something similar as well, may I know how long have you been collecting the data, and how do you decide which data to collect? I have been collecting data from the twitter API for a few days as well. I wanted to get an idea of average tweeting pattern, but without access to the firehose API, I got a feeling that the sample I have isn't very "neutral" as I have been mostly pulling from the popular and local tweets endpoints. Any advice on how should I approach these kind of data collection?

me_bx 12y ago
It all depends on your use case. If user streams are not releavant to you, you may use the `filter` endpoint instead of `sample`, and focus on keywords describing a relevant niche for your analysis. In case you want to limit yourself to tweets geo-located in a certain location, you have to be aware that the bounding box filter of twitter is buggy: it will return you tweets geo-located outside of the desired area, and you are not sure to be getting all tweets.

billyhoffman 12y ago

Won't SELECT count(*) be super slow? Isn't SELECT Count(some_primary_key) a better idea?

icebraining 12y ago
It should be the same. count(*) only needs to return the number of columns (regardless of its value), so it can use only indexes, while count(column) must only count non-null values. But since the primary is non-null, it should end up taking the same time.
- billyhoffman 12y ago
  Wow, just tested this against a table with some pretty wide columns and ~ 7M rows. They take exactly the same time! I would have thought COUNT() would be like SELECT , but I guess the query planner is smart enough to know what to do. Thanks!
deleted 12y ago
[deleted]