3 ms·
How do you decide which characters to index? The current Unicode release (15.0) includes 149,186 individual characters. I suppose you can probably ignore U+237C
by voxadam 3y ago
How do you decide which characters to index? The current Unicode release (15.0) includes 149,186 individual characters. I suppose you can probably ignore U+237C (Right Angle with Downwards Zigzag Arrow) seeing as nobody seems to know what it denotes.[0][1]
[0] https://news.ycombinator.com/item?id=31012865 https://news.ycombinator.com/item?id=31012865
[1] https://ionathan.ch/2022/04/09/angzarr.html https://ionathan.ch/2022/04/09/angzarr.html
- NoGravitas 3y agoThat is Ellis.
- PaulHoule 3y agoMost search engines for languages like English are indexing words as opposed to characters so choices as to what characters are indexed are made as part of deciding which words to index. Search engines for CJK languages do tend to work at the character level so a search for “Sona” on a certain site run by (I think) Chinese people will turn up result for “Persona”. I was involved with an A.I. startup where we had lots of meetings about what to do about all the strange Unicode characters and right now in Mastodon there is a lot of concern that screen readers will choke on 𝐮𝐧𝐢𝐜𝐨𝐝𝐞 𝐛𝐨𝐥𝐝 𝐜𝐡𝐚𝐫𝐚𝐜𝐭𝐞𝐫𝐬 while it doesn’t seem that difficult to squash them down to ordinary characters or treat them exactly as <b>unicode bold characters</b>