4 ms·

Show HN: News Extract API – Pull structured data from online news articles

131 points by artembugara 7y ago

Hey HN, It will be more of a "How I released my API without managing a website, servers, users, and payments. With 0$ up-front cost" Over the past year, I have come up with a plan of how I could release my own product without having to deal with managing users and/or dealing with payment processing. It is a 3 steps procedure: 1. Make an API that solves a problem 2. Deploy it with a serverless architecture 3. Distribute through an API Marketplace That took me about 2-3 days to develop an API using Flask, deploy it via Zappa on AWS, and release through RapidAPI. Source code of API: https://github.com/kotartemiy/extract-news-api https://github.com/kotartemiy/extract-news-api Subscribe to API on Rapid: https://rapidapi.com/provider/4109621/apis/extract-news/users https://rapidapi.com/provider/4109621/apis/extract-news/user... I'm on ProductHunt today: https://www.producthunt.com/posts/extract-news-api https://www.producthunt.com/posts/extract-news-api Full article on how I did it: https://towardsdatascience.com/api-as-a-product-how-to-sell-your-work-when-all-you-know-is-a-back-end-bd78b1449119 https://towardsdatascience.com/api-as-a-product-how-to-sell-...

axegon_ 7y ago
I had done this some years ago but it was completely private and I used it for summarizing news and informing me about certain events and shove a stuff in couchbase for future analysis. Kind of lost interest at some point so I scrapped it. Looking at yours(This has become some sort of a habit at this point but first thing I did was scroll through the code) the first thing that struck me were the 55 dependencies. Woah...
- artembugara 7y ago
  There are only 3-4 libraries that I installed when developing. Some libraries require other libraries and so on. Not sure if it is just a Python thing.
  - axegon_ 7y ago
    Yeah, figured. I'm guessing you shoved everything from pip freeze into the requirements. I don't recall flask, newspaper or langdetect having pillow as a requirement for example...
    - deleted 7y ago
      [deleted]
JPKab 7y ago
Thanks for sharing this. I've never played around with Zappa, so looking forward to looking at how you used it. The fact that you're parsing text is icing on the cake, since I love all things NLP.
- artembugara 7y ago
  yeah, Zappa is quite amazing. Serverless seems like another great tool (supports many languages and many platforms): https://github.com/serverless/serverless https://github.com/serverless/serverless
speg 7y ago
Oh man, I need this. I hope it works better than whatever I was trying before. I was using it to grab all my HN upvote/favs and using Postgres full text search on the article text, but most of them ended up being full of gibberish or missing loads of text. A good way for me to answer "where did I see that article last week..."
salmaanp 7y ago
Nice, it has some similarities to what I had done earlier. A reddit bot which reads articles and posts a summary in the comments. It lived entirely on the heroku using free dyno. Didn't know there was an API marketplace! https://github.com/SalmaanP/samacharbot2 https://github.com/SalmaanP/samacharbot2
dennisy 7y ago
The core of the product is a third party lib Newspaper3k. Not sure I would want to pay for the Flask wrapper.
- artembugara 7y ago
  Yeah, that is a product for those who need it deployed and ready to use. For devs, I open-sourced the full code. Also, one thing to understand is that it can be quite expensive for some companies to deploy it. They might need servers, developers, testers, etc. In some cases, it is just cheaper to buy it.
- holler 7y ago
  Interestingly I revisited the repo for news3k and it seems to be inactive with 300+issues, 70+ open PR's, last commit 12 months ago...
  - _____smurf_____ 7y ago
    How is this different from GDELT https://blog.gdeltproject.org/gdelt-2-0-our-global-world-in-realtime/ https://blog.gdeltproject.org/gdelt-2-0-our-global-world-in-...
    - artembugara 7y ago
      it is totally different. You cannot do any of what my code does with GDELT products.
  - thundergolfer 7y ago
    Damn, yeah that's pretty unloved. Weirdly the developer is still advertising consulting services for the project and allowing donations. Given how popular the project is and the owner's absence, community ownership seems like a no-brainer.
    - holler 7y ago
      I'm using newspaper for a new project and this has me wondering how hard it would be to either fork it or create a new repo and let the community run it... hrmm. For me it's not pressing right now but if it really stays stagnant then maybe.
nikhilalmeida 7y ago
Curious what the economics are there in working like this? * What is the market size of developers companies that pay to have such wrapper APIs? * What would pricing for something like this look like? * What is the hit / miss ratio of API ideas? I was thinking of putting together datasets and listing on similar marketplaces. Anyone have any information to share on this?
- polymorph1sm 7y ago
  I googled a put together a few similar services (not affiliated ) and it seems like pricing varies quite a lot paid package start at 150 to 449 per month (all provide free service though). 1) https://currentsapi.services/en https://currentsapi.services/en 2) https://contextualweb.io/news-api/ https://contextualweb.io/news-api/ 3) https://webhose.io/products/news-feeds/ https://webhose.io/products/news-feeds/ 4) https://aylien.com/news-api/ https://aylien.com/news-api/
naikas82 7y ago
I was looking for something like this. But with add-on to summarize the news text as close as human. Any idea???
- artembugara 7y ago
  Hey man. Yeah, text summarization sucks a lot at the moment. I would say there is no such product at the moment. None I am aware about, at least
- jonnyscholes 7y ago
  There's a bot on Reddit that summarizes articles, I see it on r/worldnews. It does a pretty good job most of the time. Looks like it's based on smmry [1] [0] https://www.reddit.com/r/autotldr/comments/31b9fm/faq_autotldr_bot/ https://www.reddit.com/r/autotldr/comments/31b9fm/faq_autotl... [1] https://smmry.com/ https://smmry.com/
rkwz 7y ago
I worked on something similar using a different approach: Fetch the article => Run it through Mozilla Readability library => Extract plain content https://github.com/sheshbabu/readable-scraper/blob/master/index.js https://github.com/sheshbabu/readable-scraper/blob/master/in...
sloev 7y ago
If you want to stay in control of deployment i can recommend Lambdarest instead. Its much simpler: only does routing and input marshaling. Has been in prod since 2017 and is actively maintained (i am doing that ;-) https://github.com/trustpilot/python-lambdarest https://github.com/trustpilot/python-lambdarest
- artembugara 7y ago
  wow, great. I will have a look today

131 points by artembugara 7y ago

axegon_ 7y ago

I had done this some years ago but it was completely private and I used it for summarizing news and informing me about certain events and shove a stuff in couchbase for future analysis. Kind of lost interest at some point so I scrapped it. Looking at yours(This has become some sort of a habit at this point but first thing I did was scroll through the code) the first thing that struck me were the 55 dependencies. Woah...

artembugara 7y ago
There are only 3-4 libraries that I installed when developing. Some libraries require other libraries and so on. Not sure if it is just a Python thing.
- axegon_ 7y ago
  Yeah, figured. I'm guessing you shoved everything from pip freeze into the requirements. I don't recall flask, newspaper or langdetect having pillow as a requirement for example...
  - deleted 7y ago
    [deleted]

JPKab 7y ago

Thanks for sharing this. I've never played around with Zappa, so looking forward to looking at how you used it. The fact that you're parsing text is icing on the cake, since I love all things NLP.

artembugara 7y ago
yeah, Zappa is quite amazing. Serverless seems like another great tool (supports many languages and many platforms): https://github.com/serverless/serverless https://github.com/serverless/serverless

speg 7y ago

Oh man, I need this. I hope it works better than whatever I was trying before. I was using it to grab all my HN upvote/favs and using Postgres full text search on the article text, but most of them ended up being full of gibberish or missing loads of text. A good way for me to answer "where did I see that article last week..."

salmaanp 7y ago

Nice, it has some similarities to what I had done earlier. A reddit bot which reads articles and posts a summary in the comments. It lived entirely on the heroku using free dyno. Didn't know there was an API marketplace! https://github.com/SalmaanP/samacharbot2 https://github.com/SalmaanP/samacharbot2

dennisy 7y ago

The core of the product is a third party lib Newspaper3k. Not sure I would want to pay for the Flask wrapper.

artembugara 7y ago
Yeah, that is a product for those who need it deployed and ready to use. For devs, I open-sourced the full code. Also, one thing to understand is that it can be quite expensive for some companies to deploy it. They might need servers, developers, testers, etc. In some cases, it is just cheaper to buy it.
holler 7y ago
Interestingly I revisited the repo for news3k and it seems to be inactive with 300+issues, 70+ open PR's, last commit 12 months ago...
- _____smurf_____ 7y ago
  How is this different from GDELT https://blog.gdeltproject.org/gdelt-2-0-our-global-world-in-realtime/ https://blog.gdeltproject.org/gdelt-2-0-our-global-world-in-...
  - artembugara 7y ago
    it is totally different. You cannot do any of what my code does with GDELT products.
- thundergolfer 7y ago
  Damn, yeah that's pretty unloved. Weirdly the developer is still advertising consulting services for the project and allowing donations. Given how popular the project is and the owner's absence, community ownership seems like a no-brainer.
  - holler 7y ago
    I'm using newspaper for a new project and this has me wondering how hard it would be to either fork it or create a new repo and let the community run it... hrmm. For me it's not pressing right now but if it really stays stagnant then maybe.

nikhilalmeida 7y ago

Curious what the economics are there in working like this? * What is the market size of developers companies that pay to have such wrapper APIs? * What would pricing for something like this look like? * What is the hit / miss ratio of API ideas? I was thinking of putting together datasets and listing on similar marketplaces. Anyone have any information to share on this?

polymorph1sm 7y ago
I googled a put together a few similar services (not affiliated ) and it seems like pricing varies quite a lot paid package start at 150 to 449 per month (all provide free service though). 1) https://currentsapi.services/en https://currentsapi.services/en 2) https://contextualweb.io/news-api/ https://contextualweb.io/news-api/ 3) https://webhose.io/products/news-feeds/ https://webhose.io/products/news-feeds/ 4) https://aylien.com/news-api/ https://aylien.com/news-api/

naikas82 7y ago

I was looking for something like this. But with add-on to summarize the news text as close as human. Any idea???

artembugara 7y ago
Hey man. Yeah, text summarization sucks a lot at the moment. I would say there is no such product at the moment. None I am aware about, at least
jonnyscholes 7y ago
There's a bot on Reddit that summarizes articles, I see it on r/worldnews. It does a pretty good job most of the time. Looks like it's based on smmry [1] [0] https://www.reddit.com/r/autotldr/comments/31b9fm/faq_autotldr_bot/ https://www.reddit.com/r/autotldr/comments/31b9fm/faq_autotl... [1] https://smmry.com/ https://smmry.com/

rkwz 7y ago

I worked on something similar using a different approach: Fetch the article => Run it through Mozilla Readability library => Extract plain content https://github.com/sheshbabu/readable-scraper/blob/master/index.js https://github.com/sheshbabu/readable-scraper/blob/master/in...

sloev 7y ago

If you want to stay in control of deployment i can recommend Lambdarest instead. Its much simpler: only does routing and input marshaling. Has been in prod since 2017 and is actively maintained (i am doing that ;-) https://github.com/trustpilot/python-lambdarest https://github.com/trustpilot/python-lambdarest

artembugara 7y ago
wow, great. I will have a look today