4 ms·

Is there a tech writeup on how the site was built? What software is behind the faceted search? The site and search experience is fast and I'd love to learn mor

Is there a tech writeup on how the site was built? What software is behind the faceted search? The site and search experience is fast and I'd love to learn more about how it was built.

haggy 8y ago
With only ~11,000 entries and what looks to be a pretty defined set of searchable fields, it should be a simple setup. I'd guess the search is powered by either Elasticsearch or Postgres/MySQL. I think the more interesting part of the product is going to be the procurement of available servers which I don't have any guesses outside of public API's + web scraping to do that.
- pheeney 8y ago
  What was interesting to me is the search ID, https://www.serverhunter.com/?search=157-857-0B2 https://www.serverhunter.com/?search=157-857-0B2 for example. Each change results in a new search ID. If you toggle the same option on and off, the search ID is different every time. I originally thought it was a way to reference a unique combination of the facets but it seems to create a new search session for each query.
  - ServerHunter 8y ago
    The search ID is indeed unique, it's a reference to our database which stores your filters, sort order, and column selections. We might replace it with something that doesn't require the backend at all, but the URLs will end up quite long if you have a few filters selected. For now, we figured a short URL is nicer to share with other people. :)
    - zAy0LfpBZLC8mAC 8y ago
      Well, a URL that stays valid would be more important than a short URL? I am assuming that you do garbage collect your DB?
      - ServerHunter 8y ago
        We will change the URL structure in the future, but it'll be backwards compatible - old URLs will continue functioning forever. :)
markbnj 8y ago
It looks to me like the default result set has 11k or so documents, which is very small. A couple of years ago I helped build a specialized vertical search engine that provided faceted search over a similarly-sized dataset. We used django on the front end with elasticsearch on the backend, and it was blazingly fast. I credit our talented django developer with the speed and fluidity of the front end, and I credit elasticsearch with providing amazingly fast aggregated results across that admittedly tiny set of documents.
- pheeney 8y ago
  Doesn't elastic search require a decent size server just to run with no data? Curious about the costs of running elastic search for a small number of documents compared to the low cost of running django. For example the frontend could be static and hosted on S3 for pennies while elastic search seems to recommend at least 8 GB of ram which would be $40/mo on some lost cost VPS providers. Pretty big price difference for a smaller site with a small record set that wants a fast search experience.
  - ServerHunter 8y ago
    I have Elasticsearch running on our development area with just 256 MB of RAM assigned. In production it's quite a bit more, but fortunately, we used Server Hunter to find some great deals for our own infrastructure as well ;)
    - pheeney 8y ago
      So meta haha. How much have you needed in production? According to https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html#_memory https://www.elastic.co/guide/en/elasticsearch/guide/current/... they say its counterproductive to use less then 8 GB but maybe you have seen different results?
      - ServerHunter 8y ago
        I'm not sure how much we've needed, but so far we've had no issues with 4 and 8 GB. Upgrading that soon though, just in case. :)
ServerHunter 8y ago
Not yet! We've actually switched stacks and rewritten the backend a few times already, which might make for a great future story. I'll try to make some time after our upcoming server migration. :)
- pheeney 8y ago
  Is there a TLDR version to tide me over in the meantime?
  - ServerHunter 8y ago
    Sure, at the moment it's just LAMP with Elasticsearch to speed things up. We also have a list of all the libraries and services we use here: https://www.serverhunter.com/licenses/ https://www.serverhunter.com/licenses/ :)

Is there a tech writeup on how the site was built? What software is behind the faceted search? The site and search experience is fast and I'd love to learn more about how it was built.

haggy 8y ago

With only ~11,000 entries and what looks to be a pretty defined set of searchable fields, it should be a simple setup. I'd guess the search is powered by either Elasticsearch or Postgres/MySQL. I think the more interesting part of the product is going to be the procurement of available servers which I don't have any guesses outside of public API's + web scraping to do that.

pheeney 8y ago
What was interesting to me is the search ID, https://www.serverhunter.com/?search=157-857-0B2 https://www.serverhunter.com/?search=157-857-0B2 for example. Each change results in a new search ID. If you toggle the same option on and off, the search ID is different every time. I originally thought it was a way to reference a unique combination of the facets but it seems to create a new search session for each query.
- ServerHunter 8y ago
  The search ID is indeed unique, it's a reference to our database which stores your filters, sort order, and column selections. We might replace it with something that doesn't require the backend at all, but the URLs will end up quite long if you have a few filters selected. For now, we figured a short URL is nicer to share with other people. :)
  - zAy0LfpBZLC8mAC 8y ago
    Well, a URL that stays valid would be more important than a short URL? I am assuming that you do garbage collect your DB?
    - ServerHunter 8y ago
      We will change the URL structure in the future, but it'll be backwards compatible - old URLs will continue functioning forever. :)

markbnj 8y ago

It looks to me like the default result set has 11k or so documents, which is very small. A couple of years ago I helped build a specialized vertical search engine that provided faceted search over a similarly-sized dataset. We used django on the front end with elasticsearch on the backend, and it was blazingly fast. I credit our talented django developer with the speed and fluidity of the front end, and I credit elasticsearch with providing amazingly fast aggregated results across that admittedly tiny set of documents.

pheeney 8y ago
Doesn't elastic search require a decent size server just to run with no data? Curious about the costs of running elastic search for a small number of documents compared to the low cost of running django. For example the frontend could be static and hosted on S3 for pennies while elastic search seems to recommend at least 8 GB of ram which would be $40/mo on some lost cost VPS providers. Pretty big price difference for a smaller site with a small record set that wants a fast search experience.
- ServerHunter 8y ago
  I have Elasticsearch running on our development area with just 256 MB of RAM assigned. In production it's quite a bit more, but fortunately, we used Server Hunter to find some great deals for our own infrastructure as well ;)
  - pheeney 8y ago
    So meta haha. How much have you needed in production? According to https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html#_memory https://www.elastic.co/guide/en/elasticsearch/guide/current/... they say its counterproductive to use less then 8 GB but maybe you have seen different results?
    - ServerHunter 8y ago
      I'm not sure how much we've needed, but so far we've had no issues with 4 and 8 GB. Upgrading that soon though, just in case. :)

ServerHunter 8y ago

Not yet! We've actually switched stacks and rewritten the backend a few times already, which might make for a great future story. I'll try to make some time after our upcoming server migration. :)

pheeney 8y ago
Is there a TLDR version to tide me over in the meantime?
- ServerHunter 8y ago
  Sure, at the moment it's just LAMP with Elasticsearch to speed things up. We also have a list of all the libraries and services we use here: https://www.serverhunter.com/licenses/ https://www.serverhunter.com/licenses/ :)