4 ms·

Haskell: http-streams and Aeson

71 points by joshrotenberg 13y ago

boothead 13y ago
I've been playing with aeson and http-streams lately too. You might find some of the debugging stuff useful. Here's some code that extracts json from a web page (I hope to be generating haskell modules from API docs eventually.) https://github.com/perurbis/hfreeagent/blob/master/fa-gen-types/Main.hs https://github.com/perurbis/hfreeagent/blob/master/fa-gen-ty...
- joshrotenberg 13y ago
  Great, thanks for the link. I'll dig through that, looks like I can learn a lot from it.
  - boothead 13y ago
    It looks a bit better locally I'll try and push something a bit more coherent tomorrow. Great article by the way!
tikhonj 13y ago
This is a great article, particularly because it's easy to follow. It shows that Haskell is much more accessible to beginners than some people make it out to be! If you've been considering trying Haskell, you should go for it. Perhaps even start with a REST API project like this. You can learn the basics from "Learn You a Haskell"[1], which is a great book available free online, and you can install Haskell with the Haskell Platform[2], which has gotten a new release fairly recently. [1]: http://learnyouahaskell.com/ http://learnyouahaskell.com/ [2]: http://haskell.org/platform http://haskell.org/platform
- joshrotenberg 13y ago
  Thanks, I'm glad you found it easy to follow.
  - ufo 13y ago
    He is not a real Haskell beginner though :P
  - chongli 13y ago
    Haskell is pretty easy to learn for beginners, but this means beginners to programming. People who are highly experienced programmers are not beginners and may have more difficulty learning Haskell because it violates so many of their assumptions. I've been teaching my non-programmer friend Haskell and he's learning very quickly and enjoying it a great deal. He doesn't get frustrated because he has no assumptions about how things "should work".
tel 13y ago
For those who aren't too behind the "JSON as a Serialization of Haskell Types" pattern in Aeson, there's also the generic Value type which represents any JSON object allowing access much more similarly to Python/Ruby/Javascript's recursive Object style member access.
maxpow4h 13y ago
You may want to try using TemplateHaskell. Where you had: data MetaData = MetaData { metadataUrl :: String, metadataTitle :: String ... you can use import Data.Char ... upStr n s = toLower (head (drop n s)) : (drop (n+1) s) $(deriveJSON (upStr 8) ''MetaData) This will splice the `metadataUrl` to use the `url` field from JSON. The deriveJSON function takes a function to change the field names. It's pretty handy.
deleted 13y ago
[deleted]
fujimura 13y ago
Thanks for mentioning me! We got similar ideas ;-)
thu 13y ago
I have a similar blog entry: http://www.hyperedsoftware.com/blog/entries/github-bindings.html http://www.hyperedsoftware.com/blog/entries/github-bindings.....
joeyh 13y ago
chunks <- Streams.toList i let feed = decode (BL.fromChunks chunks) :: Maybe Feed You lose streaming here; this reads the entire response body into memory before sending it to Aeson to decode. Which is a shame since both Aeson and http-steams otherwise seem able to stream arbitrarily large content through without buffering lots of it in memory. (I have not tested how well Aeson manages this; just looking at the types!) This could be used to eg, fold an operation over the list of earthquakes, no matter how long the list becomes. Fixing this is probably beyond the scope of a simple walkthrough. I suspect it could be managed using unsafeInterleaveIO to translate between the io-streams chunks and the lazy bytestring chunks. Might be a nice thing to ask the io-streams developers to add. (Dealing with resource finalization could make it tricky though.) On a less esoteric note, fetchQuakes would be improved by using withConnection. This would make it shorter, and ensures the stream is closed if there's an error.
- enigmo 13y ago
  It's not all that difficult, since io-streams has attoparsec support and aeson is an attoparsec parser... coded directly in the input box, I don't have io-streams installed so I can't see if this typechecks: import Control.Applicative import Data.Aeson import Data.ByteString import System.IO.Streams import System.IO.Streams.Attoparsec (parseFromStream) parseJSONFromStream :: FromJSON a => InputStream ByteString -> IO a parseJSONFromStream = parseJSON <$> parseFromStream json'
  - joshrotenberg 13y ago
    I started looking at attoparsec after the post. Thanks, I'll look closer and see if I can figure it out.
    - joshrotenberg 13y ago
      Found this: https://github.com/RobinKrom/Mtgox/blob/master/src/Mtgox/HttpApi.hs https://github.com/RobinKrom/Mtgox/blob/master/src/Mtgox/Htt... receiveResponse con (\_ i -> parseFromStream json i)) This seems to work but without the benefit of being marshalled into my type.
joshrotenberg 13y ago
Did a quick followup based on some of these comments. Thanks again, very helpful! http://joshrotenberg.com/haskell/2013/06/05/follow-up-http-streams-and-aeson/ http://joshrotenberg.com/haskell/2013/06/05/follow-up-http-s...

71 points by joshrotenberg 13y ago

I've been playing with aeson and http-streams lately too. You might find some of the debugging stuff useful. Here's some code that extracts json from a web page (I hope to be generating haskell modules from API docs eventually.) https://github.com/perurbis/hfreeagent/blob/master/fa-gen-types/Main.hs https://github.com/perurbis/hfreeagent/blob/master/fa-gen-ty...

joshrotenberg 13y ago
Great, thanks for the link. I'll dig through that, looks like I can learn a lot from it.
- boothead 13y ago
  It looks a bit better locally I'll try and push something a bit more coherent tomorrow. Great article by the way!

tikhonj 13y ago

This is a great article, particularly because it's easy to follow. It shows that Haskell is much more accessible to beginners than some people make it out to be! If you've been considering trying Haskell, you should go for it. Perhaps even start with a REST API project like this. You can learn the basics from "Learn You a Haskell"[1], which is a great book available free online, and you can install Haskell with the Haskell Platform[2], which has gotten a new release fairly recently. [1]: http://learnyouahaskell.com/ http://learnyouahaskell.com/ [2]: http://haskell.org/platform http://haskell.org/platform

joshrotenberg 13y ago
Thanks, I'm glad you found it easy to follow.
- ufo 13y ago
  He is not a real Haskell beginner though :P
- chongli 13y ago
  Haskell is pretty easy to learn for beginners, but this means beginners to programming. People who are highly experienced programmers are not beginners and may have more difficulty learning Haskell because it violates so many of their assumptions. I've been teaching my non-programmer friend Haskell and he's learning very quickly and enjoying it a great deal. He doesn't get frustrated because he has no assumptions about how things "should work".

tel 13y ago

For those who aren't too behind the "JSON as a Serialization of Haskell Types" pattern in Aeson, there's also the generic Value type which represents any JSON object allowing access much more similarly to Python/Ruby/Javascript's recursive Object style member access.

maxpow4h 13y ago

You may want to try using TemplateHaskell. Where you had: data MetaData = MetaData { metadataUrl :: String, metadataTitle :: String ... you can use import Data.Char ... upStr n s = toLower (head (drop n s)) : (drop (n+1) s) $(deriveJSON (upStr 8) ''MetaData) This will splice the `metadataUrl` to use the `url` field from JSON. The deriveJSON function takes a function to change the field names. It's pretty handy.

deleted 13y ago

[deleted]

fujimura 13y ago

Thanks for mentioning me! We got similar ideas ;-)

thu 13y ago

I have a similar blog entry: http://www.hyperedsoftware.com/blog/entries/github-bindings.html http://www.hyperedsoftware.com/blog/entries/github-bindings.....

joeyh 13y ago

chunks <- Streams.toList i let feed = decode (BL.fromChunks chunks) :: Maybe Feed You lose streaming here; this reads the entire response body into memory before sending it to Aeson to decode. Which is a shame since both Aeson and http-steams otherwise seem able to stream arbitrarily large content through without buffering lots of it in memory. (I have not tested how well Aeson manages this; just looking at the types!) This could be used to eg, fold an operation over the list of earthquakes, no matter how long the list becomes. Fixing this is probably beyond the scope of a simple walkthrough. I suspect it could be managed using unsafeInterleaveIO to translate between the io-streams chunks and the lazy bytestring chunks. Might be a nice thing to ask the io-streams developers to add. (Dealing with resource finalization could make it tricky though.) On a less esoteric note, fetchQuakes would be improved by using withConnection. This would make it shorter, and ensures the stream is closed if there's an error.

enigmo 13y ago
It's not all that difficult, since io-streams has attoparsec support and aeson is an attoparsec parser... coded directly in the input box, I don't have io-streams installed so I can't see if this typechecks: import Control.Applicative import Data.Aeson import Data.ByteString import System.IO.Streams import System.IO.Streams.Attoparsec (parseFromStream) parseJSONFromStream :: FromJSON a => InputStream ByteString -> IO a parseJSONFromStream = parseJSON <$> parseFromStream json'
- joshrotenberg 13y ago
  I started looking at attoparsec after the post. Thanks, I'll look closer and see if I can figure it out.
  - joshrotenberg 13y ago
    Found this: https://github.com/RobinKrom/Mtgox/blob/master/src/Mtgox/HttpApi.hs https://github.com/RobinKrom/Mtgox/blob/master/src/Mtgox/Htt... receiveResponse con (\_ i -> parseFromStream json i)) This seems to work but without the benefit of being marshalled into my type.

joshrotenberg 13y ago

Did a quick followup based on some of these comments. Thanks again, very helpful! http://joshrotenberg.com/haskell/2013/06/05/follow-up-http-streams-and-aeson/ http://joshrotenberg.com/haskell/2013/06/05/follow-up-http-s...