8 ms·
It is difficult for me to describe just how angry it makes me that reddit doesn't provide a way for users to even do basic things like "see all of my own commen
by saurik 8y ago
It is difficult for me to describe just how angry it makes me that reddit doesn't provide a way for users to even do basic things like "see all of my own comments" or "see all of the posts made to the subreddit I moderate". They keep nerfing the search APIs and claim it is so they could make the indexes more efficient, but while that might make sense for a full-text search interface, that is entirely unreasonable for basic functionality like "I'm scrolling back through time on my own user page" (where the efficient index is pretty obvious). Both of "see all of the content I posted" and "see all of the content I'm supposedly responsible for" seems like it should be basic, if not required, functionality for any website.
https://www.reddit.com/r/changelog/comments/7tus5f/update_to_search_api/ https://www.reddit.com/r/changelog/comments/7tus5f/update_to...
https://www.reddit.com/r/redditdev/comments/7qpn0h/how_to_retrieve_all_removed_posts_via_api/ https://www.reddit.com/r/redditdev/comments/7qpn0h/how_to_re...
https://www.reddit.com/r/help/comments/1u0scj/get_full_post_history/ https://www.reddit.com/r/help/comments/1u0scj/get_full_post_...
- mustacheemperor 8y agoReddit doesn't even allow users to save more than 1000 posts, and worse does not visibly document this or provide any kind of warning that the limit has been exceeded. Anecdotally, I've read users say that revisiting the saved pages will still show an "unsave" button so the information is recorded somewhere. But once a user exceeds 1000 entries on their "saved" page, adding new ones will silently vaporize old ones. https://www.reddit.com/r/help/comments/6nxqjm/maximum_of_1000_posts_is_it_true/ https://www.reddit.com/r/help/comments/6nxqjm/maximum_of_100...
- Puer 8y agoSaving posts is also notoriously unreliable, in my experience. I've had posts where I've clicked the "save" button and it's flipped to "unsave" but upon refreshing the page the button resets as if it was never pressed (and it never gets added.)
- Deimorz 8y agoIt's a bit weirder than that. It actually does save all the posts, but the "saved" page (like almost every other page on the site) will only show you 1000 items, so there's just no way to access all the older items once they've been "pushed off the end". I posted some more information about it a while ago here: https://www.reddit.com/r/help/comments/7en0uu/my_saved_posts_end_after_a_year/dq69oz3/?context=3 https://www.reddit.com/r/help/comments/7en0uu/my_saved_posts...
- lucb1e 8y agoIf you unsave a newer one, wouldn't the older one pop up again because it updates the index with limit 1000 again? If the data is there, it should find all posts and then truncate, in that order, if I'm understanding this correctly.
- Deimorz 8y agoNope, because the old one's already been pushed off the end of the list and it doesn't re-generate the list when you unsave, just removes the item from the list. Imagine I have a list of max length 5, when I initially fill it up it looks like [5, 4, 3, 2, 1]. If I save one more thing, it adds 6 at the front, then truncates the list and removes the 1 from the end, so now you have [6, 5, 4, 3, 2]. At that point, if I unsave #3, it just removes it from the list so you'd have [6, 5, 4, 2]. #1 is still saved, but nothing happens to pull it back into the list.
- rotred 8y agowhy would they make the limit so low?
- Deimorz 8y agoNot sure, the 1000 limit is in the file from its very first version 10 years ago: https://github.com/reddit-archive/reddit/commit/33fd4e9684ca7668b57aa32ab6a064cd2a86ef6d#diff-8e12a829341215de40b3890544842ed0R14 https://github.com/reddit-archive/reddit/commit/33fd4e9684ca... 1000 probably seemed like a lot at the time or had reasonable performance, and it's just never been changed.
- bluetidepro 8y agoA good work around tip is to set up an https://ifttt.com/ https://ifttt.com/ to trigger when you save to offload your saves to somewhere else. I save all my reddit saves to Evernote, personally. It can help with this known issue!
- minimaxir 8y agoThe big concern I have is that the endpoints used in the article are pretty old and given that Reddit is now making their site more modern like Facebook and Twitter, they'll just close the endpoints holes like they did.
- paulmd 8y agoYeah, Reddit's search is unusable. Some of the important features used to be corralled into CloudSearch but I think they killed that too. I pulled the dumps from Bigquery and am going to load them into PG at some point here, so I can do arbitrary queries without hitting BigQuery every time. Haven't looked at how to do that in realtime though, retrospective queries are mostly what I'm interested in. If you don't want to do that, there's also https://redditsearch.io/ https://redditsearch.io/ - if you want to search back farther, be sure to set the toggle to "all" instead of "day"!
- stuck_in_the_ma 8y agoI built redditsearch.io -- It uses the pushshift API for the back-end. It was thrown together and barely works but hey, I'm just one guy maintaining this as a labor of love. :)
- eat_veggies 8y agoTheir API is so fucked that it is literally impossible to get more than a couple hundred posts from a subreddit. Thankfully, services like pushshift[1] exist, which has a sane API and the option to use plain elasticsearch. [1] https://github.com/pushshift/api https://github.com/pushshift/api
- flashman 8y agoTo answer GP's questions with Pushshift links: "see all of my own comments": http://api.pushshift.io/reddit/comment/search?author=saurik&limit=1000 http://api.pushshift.io/reddit/comment/search?author=saurik&... (use &before=[epoch] for pagination) "see all of the posts made to the subreddit I moderate": https://api.pushshift.io/reddit/search/submission/?subreddit=Cydia https://api.pushshift.io/reddit/search/submission/?subreddit...
- stuck_in_the_ma 8y agoWow, It's so weird recognizing so many people in this discussion. Thanks for giving examples. I have a subreddit (/r/pushshift) that gives examples on how to use the API. I'm always happy to answer questions about the API and take suggestions to improve it. I'm very very close to launching the new API which will have a lot more features and a better design than the current one. Edit: I just realized my username on this site got truncated to uhhh ... wow.
- Deimorz 8y ago(I worked at reddit) It's not really related to search, most of the cause is a pretty bizarre optimization method that reddit decided to implement fairly early on. The database structure is unusual and not very conducive to indexing (it's similar to an EAV model), so at some point they decided to basically write their own "secondary indexing"-like system that stores the "listing indexes" in memcached/Cassandra (with the data itself in PostgreSQL). Whenever something happens that affects any listings (new post created, voting, etc), the site figures out all the listings it needs to update, and where in each listing the affected post now belongs and updates them all. So for example, if you make a new submission to /r/pics, it will go through and add the post's ID in the right spot to the "new posts in /r/pics" listing, the "hot posts in /r/pics" listing, the "new posts by yourusername" listing, and so on. As it's going through and updating all these lists, it also trims each one down to 1000 items. It's conceptually pretty similar to a normal database indexing system, but basically maintains all the indexes "manually" and restricts them all to the top 1000 items. If you're curious enough to dig around in the code, this is probably the main relevant file: https://github.com/reddit-archive/reddit/blob/master/r2/r2/lib/db/queries.py https://github.com/reddit-archive/reddit/blob/master/r2/r2/l...
- codeplea 8y agoThanks for the explanation. The APIs make a lot more sense in light of it. It seems odd they didn't just use Postgres indexing though. Do you know if they benchmarked it at any point?
- Deimorz 8y agoI have no idea, this system was created years before I started working there.
- anarazel 8y agoFor a lot of queries EAV type schemas are really hard to index efficiently. E.g. searching for something like a = ? and b = ? where a and b are dynamic attributes you can't just have a multi-column index when using EAV. So you instead end up with two intermediate query results that then are intersected. If you need even semi-efficient querying EAV usually isn't the answer.