9 ms·
How MDN's Autocomplete Search Works
- ushakov 5y agoi'm wondering how much kb it loads before ready to search? update: 144KB for JSON file a little bit worrying, given their scale and potential bandwidth requirements
- hbcondo714 5y agoYeah I would think this file size will increase well over time. Maybe a part 2 of the article can go over how updates to the file are made when new content is published and possible scaling solutions.
- ourcat 5y agoFor content like this, it's much easier to download the entire search-index.json and run the auto-complete against that. Rather that than hitting a search endpoint (after typing a certain amount of characters).
- simonw 5y agoSadly in 2021 adding 140KB to a page isn't a big deal (given how heavy the rest of the page probably is) - but it really should be. A large chunk of the world's population still pays a locally-expensive rate for mobile bandwidth, and we're increasingly leaving them behind - or worse, pushing them into zero-rating internet plans which mean they can only use Facebook and WhatsApp while avoiding the rest of the web: https://en.wikipedia.org/wiki/Zero-rating https://en.wikipedia.org/wiki/Zero-rating
- earthboundkid 5y agoI miss the old search that let me narrow things down by category.
- peterbe 5y agoWhat do you miss about it? Can you not find what you're looking for?
- earthboundkid 5y agoThe new search tends to give me the CSS property when I want the JavaScript property and the SVG attribute when I want the CSS property etc etc. It’s always choosing the wrong category.
- bityard 5y agoThey hi-jacked the browser's `/` key to focus the field, which is something I hate. As a user, I want `/` to bring up Firefox's quick search bar, especially when reading documentation. They should have just had the search field focused automatically but that would have done away with their "clever" hack to lazy-load the DB containing every page name. Also, I'm confused, I thought https://mdn.dev/ https://mdn.dev/ was the new thing because Mozilla was stepping back from MDN. Is it a fork? They both carry Mozilla logos, so what's going on there?
- ushakov 5y agoimho, they should've opted for CMD/CTRL + K, which Algolia's Doc search uses > They should have just had the search field auto-focused automatically but that would have done away with their "clever" hack to lazy-load the DB containing every page name. this would steal away the focus and is not good for accessibility (unless you're building a search engine)
- est31 5y agoYeah discourse does the same. Sometimes i want to search within a post for some keyword. But ctrl+f redirects you to the global search... that global search only helps if you want to find interesting posts, but it does not support searching inside one, nor does it allow limited search within a thread. So I started using / in discourse discussions. Then that one was being overridden as well. I've heard the recommendation that you turn js off, which gives you a saner experience.
- mikepurvis 5y agoI hate this behaviour in discourse as well, but it hadn't occurred to me to try using it sans JS altogether, since it seemed to be pretty dependent on it. Will give that a shot for sure.
- polar 5y ago> But ctrl+f redirects you to the global search Press ctrl-f twice.
- namanyayg 5y agoIn the code snippet they show the `startAutocomplete()` function checks for the "started" variable being true; but never actually sets it to true.
- peterbe 5y agoIt's pseudo code. The real code is TypeScript React and looks very different and it wouldn't serve the article to take snippets from that code to explain how it works.
- namanyayg 5y agoOf course, otherwise any amateur would be able to see looping network requests right? Just saying that it'd been good to have accurate pseudocode on an MDN article.
- mg 5y agoMy favorite autocomplete library is an ancient version of bootstrap-typeahead.js by Twitter. A single file with less than 400 lines of Javascript. They don't make these anymore :) I use it everywhere where I need autcompletion. For example on the Music-Map: https://www.music-map.com https://www.music-map.com I made a fork of the code which is available here: https://www.gibney.org/0g-typeahead https://www.gibney.org/0g-typeahead
- ourcat 5y agoI did my first autocomplete search UI with that library. These days, due to the rest of the project, I've been using Angular and Material's Autocomplete component, which I've found very easy to customise for in-memory indexes or hits to a remote ElasticSearch 'suggester' proxy endpoint.
- peterbe 5y agoGetting accessibility right is hard. We very much care about that. One of the strong reasons for why we're using Downshift.
- capableweb 5y agoIt seems to be missing the most obvious feature: keyboard navigation. My guess is that accessibility is not well implemented either. Probably why other dropdown implementations are bigger code-size wise.
- mg 5y agoIt has keyboard navigation. It is just not enabled on the Music-Map. Gnoosic uses the same library, and here you can navigate with the cursor keys: https://www.gnoosic.com/faves https://www.gnoosic.com/faves
- flerovium 5y agoI can't wait until FlexSearch reaches 1.0.0. Reading the source code is like reading great literature.
- peterbe 5y ago(author here) We're still on FlexSearch 0.6 and the new 0.7 is a big refactor. I hope we can upgrade some time.
- ranguna 5y agoAny reason for not using semantic versioning? I see in your npm page that despite compatibility "some adjustments" might be needed, aka you broke compatibility. If you did a breaking change, you need to up that major version my man. Please stop with this sentimental versioning, it just causes issues for the rest of us who want to rely on npm's ability to not upgrade stuff on breaking changes, now everyone's gonna have to lock you package version to 0.6 so they don't get your breaking stuff from 0.7.
- pwfcurry 5y agoSemver allows for breaking changes with major version 0: https://semver.org/#spec-item-4 https://semver.org/#spec-item-4
- muxator 5y agoNow I am curious if, in the real MDN production site, serach-index.json loading is triggered by the execution of /static/js/autocomplete.js, when their download should really be started in parallel by the shim. Many websites leave a lot of performance on the table because of such behaviors. My hypothesis is that, since this is easier for the developer, and works good enough, not many people really care. But these things add up, and the web becomes slower and slower.
- peterbe 5y agoMDN has 2 search things: 1. client-side only which downloads a complete list of all titles. 2. full-text search on everything with Elasticsearch.
- yesbabyyes 5y agoFirst of all, great write up and interesting solution, thank you for that! I think GPS question was rather whether the page loads the start autocomplete script on focus, and the script triggers download of the json data, as in the pseudo code, or whether the real code triggers downloading of both in parallel (the script and the json data)?
- winrid 5y agoWe took a similar approach for our documentation search. [0] You can see the "inverted index" is rendered inline in the page, since everything is generated at build time. When you type something that matches a key in the index, we fetch that index key and add it to the results. [1] [2] Obviously we could do a lot better in terms of relevancy, but it's simple and fast. [0] https://docs.fastcomments.com/ https://docs.fastcomments.com/ [1] https://docs.fastcomments.com/index-ublJLBnXgz88.json https://docs.fastcomments.com/index-ublJLBnXgz88.json [2] https://github.com/FastComments/fastcomments-docs/blob/main/src/index.js#L92 https://github.com/FastComments/fastcomments-docs/blob/main/...
- thinkloop 5y agoI thought this was going to be about advanced usage of <datalist>: https://developer.mozilla.org/en-US/docs/Web/HTML/Element/datalist https://developer.mozilla.org/en-US/docs/Web/HTML/Element/da...
- theandrewbailey 5y agoTypos and fuzzy searches cause <datalist> to break.
- peterbe 5y ago<datalist> is awesome! But I find it works better for short options. See https://www.peterbe.com/plog/datalist-looks-great-on-mobile-devices https://www.peterbe.com/plog/datalist-looks-great-on-mobile-...
- ShrigmaMale 5y agoI like mosra's search, implemented in m.css for magnum. He wrote a blog post on it here: https://blog.magnum.graphics/meta/improved-doxygen-documentation-and-search/ https://blog.magnum.graphics/meta/improved-doxygen-documenta... and you can try it on the magnum docs site: https://doc.magnum.graphics/magnum/#search https://doc.magnum.graphics/magnum/#search Fast and can be served from a static site.
- vdm 5y agohttps://react-spectrum.adobe.com/blog/building-a-combobox.html https://react-spectrum.adobe.com/blog/building-a-combobox.ht...
- encryptluks2 5y agoI think adding search to the HTML standard makes more sense overall. The thing I hate about search like this is that they don't work with JS turned off (e.g. terminal browser). Why not just add a JSON search component to HTML itself?
- bastawhiz 5y agoBecause there's no one-size-fits-all solution for search. It's arguable that such a thing wouldn't even suit the needs of a small fraction of folks who were interested in a similar feature. Plus, if it was added to the spec, then Safari users wouldn't be able to use it.
- encryptluks2 5y agoThere is though.. hence why they call them web standards. Nothing I hate more than each website implementing a non-standards way of doing things. If Safari doesn't want to stay current with web standards then so be it. That is their prerogative. If a client-side JSON search component was added to web standards then I have a feeling that eventually they'd have no choice but to adopt.
- sa3dany 5y agoI remember doing something similar a few years ago, I needed autocomplete for a shipping ports field, the data was too big though so I ended up using a csv file in an aws lamda function that filters based on the selected country and returns a much smaller subset. It lazy loaded after the user selected the country. To keep response times low I had to do a binary search on the raw csv bytes. It felt like I was reinventing databases but I liked the idea of it being self contained in a function.
- lootsauce 5y agoHave you looked into s3 select api, you can do sql against a flat file and return just the result all from s3, not too shabby to build a poor mans serverless database.
- brailsafe 5y agoAre they using react for just this one thing on this page!? Honestly it wasn't very clear to me, but they seemed to indicate that. I wonder if that's just because it's the pattern at MDN, but I feel like shipping react along with the JSON has got to be huge.
- nicoburns 5y agoReact's only 100kb (or ~30kb gzipped). You're not likely to notice that if it's fetched in the background.
- brailsafe 5y ago100kb is a lot in the context of making one specific component work. If it's just one thing you could do it will far less overhead. Maybe preact or a native web component or something. If I'm not misunderstanding, then it does seem like how people would include jQuery just to select a few elements and change their classes or make an ajax call.
- eric4smith 5y agoAs an aside here... MDN docs are pretty awesome. I've been learning pure vanilla Javscript on this site more than anywhere else. I used to automatically include jQuery in every project as a habit/reflex. Now, because of MDN, I never do that anymore, unless It makes total sense. Kudos guys!
- randompwd 5y agoso, "144KB over the network" - how much does that equate to in memory? And that's per page where the search has been activated, correct? With no sharing of that dataset between each page?