5 ms·

Show HN: SearQ - A REST API that allows users to search from RSS feeds

32 points by daviducolo 4y ago

daviducolo 4y ago
What is Searq? Built on Ruby on Rails, a popular web application framework that provides a robust foundation for building scalable and maintainable web applications, Searq uses MeiliSearch as its search engine, providing powerful search capabilities through its simple and intuitive API. Searq provides several endpoints that developers can use to interact with the API: Search Endpoint The search endpoint is one of the most important endpoints in Searq. It allows developers to search the index for specific items using a search query. To use the search endpoint, developers can send a GET request to the items endpoint with a search query parameter. Feeds Endpoint The feeds endpoint allows developers to add, delete, and update RSS feeds. To add a feed, developers can simply send a POST request to the feeds endpoint with the URL of the RSS feed. Searq will then fetch the feed and add it to the search index. Tasks Endpoint The tasks endpoint allows developers to view the status of tasks that are running in the background. For example, when a new feed is added, Searq will fetch the feed and update the search index in the background. Developers can use the tasks endpoint to monitor the status of this process. Items Endpoint The items endpoint allows developers to search the index for specific items. Developers can send a GET request to the items endpoint with a search query, and Searq will return a list of items that match the query. Benefits of using Searq There are several benefits to using Searq: Easy to use: Searq provides a simple and intuitive API that developers can use to build their own search engines. Flexible: Searq allows developers to use RSS feeds as the data source, providing flexibility in the types of content that can be searched. Fast: By using MeiliSearch as the search engine, Searq provides lightning-fast search capabilities. Scalable: Searq is built on Ruby on Rails, a framework that provides a robust foundation for building scalable web applications.
kevincox 4y ago
This is a cool idea. It would be interesting to add a spider onto it as well. Check outlinks from known articles and see if they have feeds, then add that to the index. It also seems like feed archiving/pagination is not supported. This means that new feeds won't have much history searchable. It may be a good idea to support archiving and pagination to index old articles. (Although admittedly support for either of these protocols is rare).
- marginalia_nu 4y ago
  Finding "the RSS feed" of a website can be an annoyingly non-trivial problem, since many CMS:es and forum software produces an absolute shit-ton of RSS feeds dynamically.
  - kevincox 4y ago
    I'm sure you can't get 100% of sites that have feeds but support for <link rel=alternate> is very widespread. If you want to be more aggressive you can try fetching links that sound like RSS feeds from link text or URL but in practice I have found that only gets you a small amount more.
    - marginalia_nu 4y ago
      No I mean you'll commonly encounter several RSS feeds on the same page, all rel=alternate. There doesn't seem to be any convention for indicating what the feed is about. You may get an RSS feed for the site, one for each the topic, and one for each comments section of the blog post, etc. These may then be duplicated in various ways since there's more than one path to the same page. It's a real pain in the ass.
- daviducolo 4y ago
  all list are paginated and you can search over the entire dataset.
  - kevincox 4y ago
    I'm talking about the ingested feeds, not the search results.
    - daviducolo 4y ago
      ah ok no, API extracts feeds from March 8, 2023
donatj 4y ago
The homepage of this really needs… More info. Any info really. Does this search feeds I want? It has a contextless `98 feeds` blurb at the bottom. Does it only search those 98? If there's a blog I want to search can this do this? Needs context.
- daviducolo 4y ago
  yes you are right I need to improve in communication. The feeds shown are those from which the items are extracted. 98 for now.
damowangcy 4y ago
So the current site provides API that searches RSS feed of 98 sites (downloadable from the excel sheet or access via the endpoint /api/feeds) to showcase the idea? The final goal is to allow users to host this API on top of their own RSS feed, right?
- daviducolo 4y ago
  yes exactly, for now there are 98 feed sources but anyone can add more to the importer which then automatically updates the feeds every 24 hours.

32 points by daviducolo 4y ago

What is Searq? Built on Ruby on Rails, a popular web application framework that provides a robust foundation for building scalable and maintainable web applications, Searq uses MeiliSearch as its search engine, providing powerful search capabilities through its simple and intuitive API. Searq provides several endpoints that developers can use to interact with the API: Search Endpoint The search endpoint is one of the most important endpoints in Searq. It allows developers to search the index for specific items using a search query. To use the search endpoint, developers can send a GET request to the items endpoint with a search query parameter. Feeds Endpoint The feeds endpoint allows developers to add, delete, and update RSS feeds. To add a feed, developers can simply send a POST request to the feeds endpoint with the URL of the RSS feed. Searq will then fetch the feed and add it to the search index. Tasks Endpoint The tasks endpoint allows developers to view the status of tasks that are running in the background. For example, when a new feed is added, Searq will fetch the feed and update the search index in the background. Developers can use the tasks endpoint to monitor the status of this process. Items Endpoint The items endpoint allows developers to search the index for specific items. Developers can send a GET request to the items endpoint with a search query, and Searq will return a list of items that match the query. Benefits of using Searq There are several benefits to using Searq: Easy to use: Searq provides a simple and intuitive API that developers can use to build their own search engines. Flexible: Searq allows developers to use RSS feeds as the data source, providing flexibility in the types of content that can be searched. Fast: By using MeiliSearch as the search engine, Searq provides lightning-fast search capabilities. Scalable: Searq is built on Ruby on Rails, a framework that provides a robust foundation for building scalable web applications.

kevincox 4y ago

This is a cool idea. It would be interesting to add a spider onto it as well. Check outlinks from known articles and see if they have feeds, then add that to the index. It also seems like feed archiving/pagination is not supported. This means that new feeds won't have much history searchable. It may be a good idea to support archiving and pagination to index old articles. (Although admittedly support for either of these protocols is rare).

marginalia_nu 4y ago
Finding "the RSS feed" of a website can be an annoyingly non-trivial problem, since many CMS:es and forum software produces an absolute shit-ton of RSS feeds dynamically.
- kevincox 4y ago
  I'm sure you can't get 100% of sites that have feeds but support for <link rel=alternate> is very widespread. If you want to be more aggressive you can try fetching links that sound like RSS feeds from link text or URL but in practice I have found that only gets you a small amount more.
  - marginalia_nu 4y ago
    No I mean you'll commonly encounter several RSS feeds on the same page, all rel=alternate. There doesn't seem to be any convention for indicating what the feed is about. You may get an RSS feed for the site, one for each the topic, and one for each comments section of the blog post, etc. These may then be duplicated in various ways since there's more than one path to the same page. It's a real pain in the ass.
daviducolo 4y ago
all list are paginated and you can search over the entire dataset.
- kevincox 4y ago
  I'm talking about the ingested feeds, not the search results.
  - daviducolo 4y ago
    ah ok no, API extracts feeds from March 8, 2023

donatj 4y ago

The homepage of this really needs… More info. Any info really. Does this search feeds I want? It has a contextless `98 feeds` blurb at the bottom. Does it only search those 98? If there's a blog I want to search can this do this? Needs context.

daviducolo 4y ago
yes you are right I need to improve in communication. The feeds shown are those from which the items are extracted. 98 for now.

damowangcy 4y ago

So the current site provides API that searches RSS feed of 98 sites (downloadable from the excel sheet or access via the endpoint /api/feeds) to showcase the idea? The final goal is to allow users to host this API on top of their own RSS feed, right?

daviducolo 4y ago
yes exactly, for now there are 98 feed sources but anyone can add more to the importer which then automatically updates the feeds every 24 hours.