4 ms·

Doesn't seem like it would be that hard... Couldn't the server just run grep if a user searches? Grep could scan every static file in milliseconds

by umvi 6y ago

djur 6y ago
If you're running grep it isn't a static site anymore.
deleted 6y ago
[deleted]
bule0bird 6y ago
The indexing task can be done during static site generation and results shipped as resources, but the query part (search string parsing, ranking, snippet generation, etc) needs some code running locally (browser, js). We've used Minimal Perfect Hash Function with bloom filter to reduce the mem of a service from ~27G to ~8G, as the data was static. Probably something like that can be used here too, to make indices small.
stevekemp 6y ago
(If you go down this route you might prefer ag/the silver surfer/similar, as being faster than grep.) I've actually reported a couple of security bugs in servers in the past, which were caused by people running grep for searches. Imagine you have a tree of web-content at /var/www, when a user searches for "foo" you run "grep -r -h foo /var/www", then you parse out the resulting filenames. Now imagine what happens if your user searches for "foo /tmp; rm -rf /". Whenever you pass user-input to a command-line application you need to be damn sure you know what you're doing - if the shell is involved in parsing your command you'll have problems.
- kragen 6y ago
  This is a good point. Even if you don't have a shell injection vulnerability, you may end up with a path traversal vulnerability. In this particular case I think you're probably pretty safe as long as you're careful to only pass a single argument to the grep process — but it depends on how exactly grep parses its arguments. If there's a way to sneak ../ in there, you might get hurt. Bleh. I can see why people install WAFs.
  - stavros 6y ago
    I've lately become allergic to things that I can only partially reason about. Sure, I'm probably going to be fine, but I can't guarantee I'll be fine, and the blackhats are always more knowledgeable than me about obscure bits of shell/Unix behavior, so I just shy away and use a less free-form approach.
Nyubis 6y ago
The big difference here is that the bloom filter implementation runs entirely client-side. It saves a roudtrip per search.

Doesn't seem like it would be that hard... Couldn't the server just run grep if a user searches? Grep could scan every static file in milliseconds

by umvi 6y ago

Doesn't seem like it would be that hard... Couldn't the server just run grep if a user searches? Grep could scan every static file in milliseconds

djur 6y ago

If you're running grep it isn't a static site anymore.

deleted 6y ago

[deleted]

bule0bird 6y ago

The indexing task can be done during static site generation and results shipped as resources, but the query part (search string parsing, ranking, snippet generation, etc) needs some code running locally (browser, js). We've used Minimal Perfect Hash Function with bloom filter to reduce the mem of a service from ~27G to ~8G, as the data was static. Probably something like that can be used here too, to make indices small.

stevekemp 6y ago

(If you go down this route you might prefer ag/the silver surfer/similar, as being faster than grep.) I've actually reported a couple of security bugs in servers in the past, which were caused by people running grep for searches. Imagine you have a tree of web-content at /var/www, when a user searches for "foo" you run "grep -r -h foo /var/www", then you parse out the resulting filenames. Now imagine what happens if your user searches for "foo /tmp; rm -rf /". Whenever you pass user-input to a command-line application you need to be damn sure you know what you're doing - if the shell is involved in parsing your command you'll have problems.

kragen 6y ago
This is a good point. Even if you don't have a shell injection vulnerability, you may end up with a path traversal vulnerability. In this particular case I think you're probably pretty safe as long as you're careful to only pass a single argument to the grep process — but it depends on how exactly grep parses its arguments. If there's a way to sneak ../ in there, you might get hurt. Bleh. I can see why people install WAFs.
- stavros 6y ago
  I've lately become allergic to things that I can only partially reason about. Sure, I'm probably going to be fine, but I can't guarantee I'll be fine, and the blackhats are always more knowledgeable than me about obscure bits of shell/Unix behavior, so I just shy away and use a less free-form approach.

Nyubis 6y ago

The big difference here is that the bloom filter implementation runs entirely client-side. It saves a roudtrip per search.