4 ms·

It's not "security" in the hacking sense. Perhaps that was a poor word choice. So when you hit an API, the server can reject your request for any reason, typica

It's not "security" in the hacking sense. Perhaps that was a poor word choice. So when you hit an API, the server can reject your request for any reason, typically a malformed request. Maybe a header it's expecting is missing, or the API key is invalid, something like that. With Robinhood's API, you can't simply provide your bearer token and hit the endpoint. There are MANY headers it is expecting, and if any are missing or incorrect, you'll get a 4xx response. Otherwise, breaking ToS is akin to jaywalking and anybody who sees a side project like this, and their takeaway is "wow they broke ToS" doesn't deserve to be hiring software engineers, and certainly doesn't deserve to be putting them through the ringer such that the first bar to pass is literally Triplebyte (RIP Triplebyte).

justinclift 3y ago
Hopefully you explained the unique weirdness with Robinhood's API parser during the presentation, such that it was necessary? Otherwise it sounds more like a case of "Here's how to get around the rate limiting of Company Foo's API and abuse those limits anyway". Without the explanation, that'd get you an immediate No at lots of places just from the implied legal exposure/liability. Though probably not all. ;)
- gymbeaux 3y ago
  I’m not sure I agree with that. If you’re a technical person, like say a “Senior Software Engineer”, you should pretty much know that’s not what’s happening with something like this. Rate limiting is server-side. “Getting around it” means making fewer requests than what would trigger the rate limiting/temporary blacklisting. You could distribute the requests amongst multiple IP addresses via say a proxy, but that isn’t necessarily going to work, like if the rate limiting is done by bearer token or API key rather than IP address (they should be doing it by the token rather than the IP address), but I made no mention of that so I don’t think anybody would think that’s what I was doing.
  - justinclift 3y ago
    It depends on how it's pitched. :) If you went with something like "using caching to avoid hitting API limits" it sounds legit. Whereas if you went with something like "getting around Company Foo's API rate limits" it sounds like you're exploiting a bug in their system. If you did present the system as spoofing requests to subvert a rate limit / blacklist, as per: #4, my presentation was about how I reverse engineered the Robinhood API, spoofed requests to get around their “security”, and used two levels of caching (Redis and a typical relational DB) to subvert rate limiting/IP blacklisting ... then that doesn't sound like the legit use case scenario. ;) Your further follow up discussion makes it sound like you were just using a self-developed caching system though, and presented it terribly for the target audience.
    - gymbeaux 3y ago
      I don't know that there's a way you can pitch your reasoning to me where I would agree. The two statements sound equivalent to me. API rate limits are just that- rate limits. This is not akin to hacking and a software engineer I doubt would ever confuse it with hacking of any kind. What would be an example of "exploiting a rate limiting bug"? Either the server limits/throttles requests, or it doesn't.
      - justinclift 3y ago
        > What would be an example of "exploiting a rate limiting bug"? All sorts of possibilities spring to mind for that. :) Here's a hypothetical example. Lets say you discovered a logic bug in the API limit code along these lines: "So, it turns out if you have an `X-Forwarded-For` header in the request, then the request is processed properly except it isn't counted towards the limit. Effectively a bypass." Exploiting this one would of course be trivial, as you'd just need to set that header for all your requests.
deleted 3y ago
[deleted]

justinclift 3y ago

Hopefully you explained the unique weirdness with Robinhood's API parser during the presentation, such that it was necessary? Otherwise it sounds more like a case of "Here's how to get around the rate limiting of Company Foo's API and abuse those limits anyway". Without the explanation, that'd get you an immediate No at lots of places just from the implied legal exposure/liability. Though probably not all. ;)

gymbeaux 3y ago
I’m not sure I agree with that. If you’re a technical person, like say a “Senior Software Engineer”, you should pretty much know that’s not what’s happening with something like this. Rate limiting is server-side. “Getting around it” means making fewer requests than what would trigger the rate limiting/temporary blacklisting. You could distribute the requests amongst multiple IP addresses via say a proxy, but that isn’t necessarily going to work, like if the rate limiting is done by bearer token or API key rather than IP address (they should be doing it by the token rather than the IP address), but I made no mention of that so I don’t think anybody would think that’s what I was doing.
- justinclift 3y ago
  It depends on how it's pitched. :) If you went with something like "using caching to avoid hitting API limits" it sounds legit. Whereas if you went with something like "getting around Company Foo's API rate limits" it sounds like you're exploiting a bug in their system. If you did present the system as spoofing requests to subvert a rate limit / blacklist, as per: #4, my presentation was about how I reverse engineered the Robinhood API, spoofed requests to get around their “security”, and used two levels of caching (Redis and a typical relational DB) to subvert rate limiting/IP blacklisting ... then that doesn't sound like the legit use case scenario. ;) Your further follow up discussion makes it sound like you were just using a self-developed caching system though, and presented it terribly for the target audience.
  - gymbeaux 3y ago
    I don't know that there's a way you can pitch your reasoning to me where I would agree. The two statements sound equivalent to me. API rate limits are just that- rate limits. This is not akin to hacking and a software engineer I doubt would ever confuse it with hacking of any kind. What would be an example of "exploiting a rate limiting bug"? Either the server limits/throttles requests, or it doesn't.
    - justinclift 3y ago
      > What would be an example of "exploiting a rate limiting bug"? All sorts of possibilities spring to mind for that. :) Here's a hypothetical example. Lets say you discovered a logic bug in the API limit code along these lines: "So, it turns out if you have an `X-Forwarded-For` header in the request, then the request is processed properly except it isn't counted towards the limit. Effectively a bypass." Exploiting this one would of course be trivial, as you'd just need to set that header for all your requests.

deleted 3y ago

[deleted]