8 ms·

Nginx JSON hacks

178 points by Kenan 15y ago

cdcarter 15y ago
Great article on how to get Nginx to JSONP for you, but I question the idea that setting up a proxy to a paid API with your client key embedded is a good idea... Seems easy to abuse.
- njharman 15y ago
  I often have to create a service/api to serve (and cache) 3rd party content cause of rate limiting. a few servers much < than 10,000's of clients. This makes a few days of work/testing into a few 10's of minutes.
- justincormack 15y ago
  You can set them to "internal" in the nginx config so they can only be called by other nginx config, not directly by clients. You also need to check the error messages do not leak urls. Then there is rate limiting etc, depends on the service.
dreamdu5t 15y ago
Use node.js instead!
- pjscott 15y ago
  It's true that everything in this article could have been done with node.js, but the nginx configuration is pretty trivial, and nginx is good software. It's stable, it's plenty fast, and you don't have to worry about a new release breaking everything. (I have nothing against node, mind you. I've used it for something similar to Google Wave, and it was very straightforward and capable.)
grourk 15y ago
If you need to proxy through nginx on your own server to pad some external api's json (to enable jsonp), wouldn't you be able to put that server on your own domain? Obviating the need for cross-domain jsonp?
olliej 15y ago
JSONP has numerous problems, first it's not JSON -- the object literal is created by executing JS, so you're back to square one with validating your input, the second and more significant problem is that if you ever use JSONP to transmit private data then you've essentially lost the cross origin protection of data present in the browser. Now all an attacker needs to do is get one of your users with an active session to load a page that does <script src="usersprivatedata.jsonp"></script> and they can gather that data.
- jhaglund 15y ago
  I was doing something similar to this article (feeding json directly to JS functions) with PHP. I think this is better for the browser. server side, it's heavier: <?php //inject json //this puts the json in a div //then the js gets the div's innerText and parses it as JSON //so there's an object in js (global) with all the JSON data, with jsonInject's key name if(isset($jsonInject) && count($jsonInject) > 0){ foreach($jsonInject as $i=>$json){ echo "<div style='display:none;' id='jsonInject_{$i}'>{$json}</div>" echo "<script>var {$i} = JSON.parse( $('#jsonInject_{$i}')[0].innerText ); </script>"; } } ?> fill the associative $jsonInject array with json strings in your controller. then this goes in your templater. it makes a hidden div, puts the json string in there. then jquery pulls the innerText value, it gets parsed as json and stored in a js var. depending on use, i'll sometimes just pass the results of JSON.parse to an initialize method. I like the proxy caching. I think apache's mod_proxy and mod_cache could do this but with more config (and resources per request).
- tlrobinson 15y ago
  the second and more significant problem is that if you ever use JSONP to transmit private data then you've essentially lost the cross origin protection of data present in the browser. That's assuming the API is using HTTP cookies for authentication, right? If it's not then theres no way for a 3rd party to construct an authenticated request without knowing the credentials. The real problem here is that a JSONP proxy allows 3rd party sites to access the API from your IP address (and API key if you use the suggestion of an API key proxy). Rate limiting and restricting to specific HTTP Referrers would be prudent.
riffraff 15y ago
Couldn't the last trick be improved? since you have access to params you could possibly use those as in location ^~ /ext_api3/ { echo_before_body $arg_callback //+somehow add "("; proxy_pass http://api.external.com/; echo_after_body ');'; }
- tlrobinson 15y ago
  Indeed, the example code isn't actually JSONP, since it hardcodes the callback.
  - jordansjones 15y ago
    Which I would guess works for his purposes.
tlrobinson 15y ago
I'm not sure I understand how some of these are being used, specifically whether they're being used safely. API keys usually exist for preventing abuse. By exposing an authenticated API proxy you're allowing anyone to abuse someone else's API using your key, likely leading to them banning you. At a minimum you should implement your own rate limiting. Likewise, with a JSONP proxy you're allowing other sites to circumvent the browsers' same origin policy to access that API, which could also lead to abuse. At a minimum you should restrict requests to ones with a recognized HTTP Referer header. Of course with JSONP you also need to trust that the API isn't doing anything malicious, like injecting cookie or other data-stealing JavaScript instead of valid JSON. It would be a good idea to validate that the response is indeed JSON before passing it back to the clients (actually a properly restricted sanitizing JSONP proxy would be a good idea even if the API already provides JSONP) So use these techniques cautiously...
- gnubardt 15y ago
  API keys usually exist for preventing abuse. By exposing an authenticated API proxy you're allowing anyone to abuse someone else's API using your key, likely leading to them banning you. At a minimum you should implement your own rate limiting. The article describes how to set up caching in nginx, so not every request to nginx goes to the external service.
  - tdfx 15y ago
    The potential for abuse remains.
    - RyanKearney 15y ago
      Then don't use it. As someone who has never had much experience dealing with Nginx I found this article to be very informative. Most people are smart enough not make public their third party apis. Those who aren't will quickly learn of the problems this may have. However, there are some cases where this could be extremely useful, such as an intranet.
bartoszpietrzak 15y ago
If anybody is concerned about API rate limiting, here's a nifty solution. http://codetunes.com/2011/07/26/outbound-api-rate-limits-the-nginx-way http://codetunes.com/2011/07/26/outbound-api-rate-limits-the...

178 points by Kenan 15y ago

Great article on how to get Nginx to JSONP for you, but I question the idea that setting up a proxy to a paid API with your client key embedded is a good idea... Seems easy to abuse.

njharman 15y ago
I often have to create a service/api to serve (and cache) 3rd party content cause of rate limiting. a few servers much < than 10,000's of clients. This makes a few days of work/testing into a few 10's of minutes.
justincormack 15y ago
You can set them to "internal" in the nginx config so they can only be called by other nginx config, not directly by clients. You also need to check the error messages do not leak urls. Then there is rate limiting etc, depends on the service.

dreamdu5t 15y ago

Use node.js instead!

pjscott 15y ago
It's true that everything in this article could have been done with node.js, but the nginx configuration is pretty trivial, and nginx is good software. It's stable, it's plenty fast, and you don't have to worry about a new release breaking everything. (I have nothing against node, mind you. I've used it for something similar to Google Wave, and it was very straightforward and capable.)

grourk 15y ago

If you need to proxy through nginx on your own server to pad some external api's json (to enable jsonp), wouldn't you be able to put that server on your own domain? Obviating the need for cross-domain jsonp?

olliej 15y ago

JSONP has numerous problems, first it's not JSON -- the object literal is created by executing JS, so you're back to square one with validating your input, the second and more significant problem is that if you ever use JSONP to transmit private data then you've essentially lost the cross origin protection of data present in the browser. Now all an attacker needs to do is get one of your users with an active session to load a page that does <script src="usersprivatedata.jsonp"></script> and they can gather that data.

jhaglund 15y ago
I was doing something similar to this article (feeding json directly to JS functions) with PHP. I think this is better for the browser. server side, it's heavier: <?php //inject json //this puts the json in a div //then the js gets the div's innerText and parses it as JSON //so there's an object in js (global) with all the JSON data, with jsonInject's key name if(isset($jsonInject) && count($jsonInject) > 0){ foreach($jsonInject as $i=>$json){ echo "<div style='display:none;' id='jsonInject_{$i}'>{$json}</div>" echo "<script>var {$i} = JSON.parse( $('#jsonInject_{$i}')[0].innerText ); </script>"; } } ?> fill the associative $jsonInject array with json strings in your controller. then this goes in your templater. it makes a hidden div, puts the json string in there. then jquery pulls the innerText value, it gets parsed as json and stored in a js var. depending on use, i'll sometimes just pass the results of JSON.parse to an initialize method. I like the proxy caching. I think apache's mod_proxy and mod_cache could do this but with more config (and resources per request).
tlrobinson 15y ago
the second and more significant problem is that if you ever use JSONP to transmit private data then you've essentially lost the cross origin protection of data present in the browser. That's assuming the API is using HTTP cookies for authentication, right? If it's not then theres no way for a 3rd party to construct an authenticated request without knowing the credentials. The real problem here is that a JSONP proxy allows 3rd party sites to access the API from your IP address (and API key if you use the suggestion of an API key proxy). Rate limiting and restricting to specific HTTP Referrers would be prudent.

riffraff 15y ago

Couldn't the last trick be improved? since you have access to params you could possibly use those as in location ^~ /ext_api3/ { echo_before_body $arg_callback //+somehow add "("; proxy_pass http://api.external.com/; echo_after_body ');'; }

tlrobinson 15y ago
Indeed, the example code isn't actually JSONP, since it hardcodes the callback.
- jordansjones 15y ago
  Which I would guess works for his purposes.

tlrobinson 15y ago

I'm not sure I understand how some of these are being used, specifically whether they're being used safely. API keys usually exist for preventing abuse. By exposing an authenticated API proxy you're allowing anyone to abuse someone else's API using your key, likely leading to them banning you. At a minimum you should implement your own rate limiting. Likewise, with a JSONP proxy you're allowing other sites to circumvent the browsers' same origin policy to access that API, which could also lead to abuse. At a minimum you should restrict requests to ones with a recognized HTTP Referer header. Of course with JSONP you also need to trust that the API isn't doing anything malicious, like injecting cookie or other data-stealing JavaScript instead of valid JSON. It would be a good idea to validate that the response is indeed JSON before passing it back to the clients (actually a properly restricted sanitizing JSONP proxy would be a good idea even if the API already provides JSONP) So use these techniques cautiously...

gnubardt 15y ago
API keys usually exist for preventing abuse. By exposing an authenticated API proxy you're allowing anyone to abuse someone else's API using your key, likely leading to them banning you. At a minimum you should implement your own rate limiting. The article describes how to set up caching in nginx, so not every request to nginx goes to the external service.
- tdfx 15y ago
  The potential for abuse remains.
  - RyanKearney 15y ago
    Then don't use it. As someone who has never had much experience dealing with Nginx I found this article to be very informative. Most people are smart enough not make public their third party apis. Those who aren't will quickly learn of the problems this may have. However, there are some cases where this could be extremely useful, such as an intranet.

bartoszpietrzak 15y ago

If anybody is concerned about API rate limiting, here's a nifty solution. http://codetunes.com/2011/07/26/outbound-api-rate-limits-the-nginx-way http://codetunes.com/2011/07/26/outbound-api-rate-limits-the...