8 ms·
GitHub switches to Rust based Blackbird code search
- est 3y agoannnnd it's down. https://news.ycombinator.com/item?id=35872835 https://news.ycombinator.com/item?id=35872835 Is this related?
- mellosouls 3y agoConsidering the massive debt GitHub owes to open-source, is this engine also open?
- ZephyrBlu 3y agoCmon bruh.
- azangru 3y agoDoes Github owe debt to open-source, or does open-source owe debt to Github?
- VWWHFSfQ 3y agowhy would any floss owe debt to github
- cortesoft 3y agoBecause they can host their software and community for free?
- woodruffw 3y agoI think it would be accurate to say that the value GitHub extracts from the OSS community (in the form of its core VCS, network effects, ML models, etc.) substantially outstrips the value returned to the community. Which is not to say that the latter is insignificant!
- loeg 3y agoIs that a qualitative or quantitative argument? Like, would you put numbers to either/both sides of that? I'm not confident Github gets more value than it returns (but you may be right).
- ZiiS 3y agoGithub are confident though so...
- cortesoft 3y agoThe OSS projects are also confident that they are getting a better deal, though, since they continue to host on Github. Sometimes both sides win.
- woodruffw 3y agoI don't think this requires belief in a "better" deal, only a good one (or an acceptable one, given the friction and network effects involved in switching).
- blackoil 3y agoBetter measure is if GH is getting very high value in return,some competition should be there working at lesser margins (giving more value for OSS)
- mattnewton 3y agoI think if they did not, and another git host offered free hosting to open source it would have much of GitHub’s share of the market today. They still competed hard on UX and features but it was necessary to win over open source projects, to become the defacto solution for software.
- renewiltord 3y agoYou don't owe debts to open source. That's the point. Or at least that's the point of the free software philosophy. This is also why I think it's unbelievably lucky we had luminaries like RMS in the beginning. To think of software freedom, what a concept! Legend! Meanwhile everyone else is constantly trying to turn everything into a transactional interaction. "what are you going to give me for my code? How are you going to pay your debts?"
- geodel 3y agoIt may or may not be. But magic is in infrastructure and not search engine algorithm per se.
- noname120 3y agoActual press release: https://github.blog/2023-02-06-the-technology-behind-githubs-new-code-search/ https://github.blog/2023-02-06-the-technology-behind-githubs...
- nindalf 3y agoThe linked article contains the news that the feature became generally available. Your link isn't a press release, its an explanation of the tech behind this feature published a few months ago.
- nusaru 3y agoActual, actual press release: https://github.blog/2023-05-08-github-code-search-is-generally-available/ https://github.blog/2023-05-08-github-code-search-is-general... HN discussion from yesterday: https://news.ycombinator.com/item?id=35863175 https://news.ycombinator.com/item?id=35863175
- esjeon 3y agoOh, this is why this news felt very old and wrong. the "0.01 QPS" number for ripgrep is just a on-paper math, not an actual performance number from a running system.
- hboon 3y agoIn 2008, Microsoft acquired Fast Search and Transfer, an enterprise search company that has decent search technologies [1]. It ran on common Linux and Unix systems as well as Windows at the point of the acquisition. [1] I worked there
- AYBABTME 3y agoBlackBird is internally developed by GitHub engineers. I know the engineers who did the research and then wrote up the system.
- hboon 3y agoYeah, I thought so (that it was internally developed), because someone commented that it's not open sourced. I was just posting to say Microsoft have (had?) a perfectly nice search engine many years ago. Either GitHub folks didn't want to use it, it's not good (I think it was good, but maybe it's not suitable, or maybe it isn't maintained anymore shrug) or Microsoft butchered it too much over the years.
- ollien 3y agoI don't know much about the inner workings of Microsoft, but I could also imagine than an acquisition from that long ago just isn't known to those in the Github side of the house. Seems pretty typical for a large company.
- AYBABTME 3y agoBlackBird is a clean slate, code-specific search engine. Not a general purpose fulltext search engine.
- finnh 3y ago> The resulting system can manage about 640 queries per second, compared to about 0.01 queries per second for ripgrep. They are trying to make a point about indexes, but it is overshadowed by the unit mismatch (distributed system QPS vs what sounds like single-machine ripgrep pace)
- deleted 3y ago[deleted]
- wtf_is_up 3y agoAccording to the engineering article [1], that ripgrep number is with 2048 cores combined with 115TB of data in memory. [1] https://github.blog/2023-02-06-the-technology-behind-githubs-new-code-search/ https://github.blog/2023-02-06-the-technology-behind-githubs...
- finnh 3y agoOK, thanks for that. Sounds like it's a valid comparison then.
- secondcoming 3y agoIsn't ripgrep also written in Rust?
- burntsushi 3y agoYes. ripgrep is used as a basis of comparison for that sentence. ripgrep wasn't GitHub's previous search implementation. (It never was. It's not the right tool for that scale.)
- hiyer 3y agoMy current employer uses sourcegraph [1], which also uses n-gram based indexing like github's blackbird. Would be interesting to see a comparative benchmark of the two. 1. https://github.com/sourcegraph/sourcegraph/blob/main/doc/dev/background-information/architecture/index.md https://github.com/sourcegraph/sourcegraph/blob/main/doc/dev...
- faitswulff 3y agoHave you tried Sourcegraph's LLM assistant, Cody, yet?
- hiyer 3y agoNo, the company hasn't enabled it yet for us.
- jason2323 3y agoFix your damn database!
- cupofjoakim 3y agoThis type of comment does not belong on hn. Take your frustrations elsewhere or discuss your point.
- toastal 3y agoI love how Microsoft GitHub doesn’t allow you to search the code base unless you’re authenticated. /s
- pxc 3y agoOn mobile, the search bar is hidden from unauthenticated users, but you can still search. The easiest way is to go to https://github.com/search https://github.com/search That said, I'm also annoyed by various navigability issues, including that one.
- pixelpoet 3y agoLately I feel like the old joke about vegans can be applied to Rust: Q: How do you know if something is written in Rust? A: The author/headline will tell you.
- DonaldPShimoda 3y agoConsidering "it's written in Rust" is synonymous with "it has been statically guaranteed to not contain a wide class of common bugs that frequently cause severe vulnerabilities", I think it's relatively reasonable for people to advertise when it's been used for a large project. Especially considering how much people like to gripe about its use.
- cstrahan 3y agoAll of the above, and also: If it’s written in Rust, the tooling (Cargo; the Rust language server; etc) make it much easier than most other languages to jump in and fix something or add a new feature. I’ve contributed to projects written in Ruby, Python, Java, C#, C++, Haskell and many others, and every Rust project I’ve ever touched has been much easier/quicker to go from `git clone` to PR submitted. That may not be appealing to people who never fix or extend the software they use; for me, as someone who finds bugs and deficiencies in almost everything I lay eyes on, knowing that a piece of software is written in a language that is the least likely to frustrate me and suck up my time when I inevitably have to dive in and tweak something is a huge competitive selling point over similar software written in other languages.
- Capricorn2481 3y agoDoesn't apply to this project, but that can be also read as "it has taken on a class of bugs it wouldn't have been exposed in a GC language, with little benefit." Or even worse, if the app needs to use unsafe, you'd be surprised how many Rust engineers think unsafe somehow "encapsulates" memory bugs inside the block itself. Which is not true. Using unsafe can cause memory issues that bubble up the call stack. I don't feel any guarantees when a product is made in Rust. There are buggy Rust projects
- zvolsky 3y agoRelevant threads: Feb 23, 2020 (https://grep.app https://grep.app) Show HN: Search code in GitHub repos using regular expressions https://news.ycombinator.com/item?id=22396824 https://news.ycombinator.com/item?id=22396824 Dec 8, 2021 Improving GitHub Code Search https://news.ycombinator.com/item?id=29487237 https://news.ycombinator.com/item?id=29487237
- WhereIsTheTruth 3y agoI hate the new search, most of the time it misses simple things and it is infuriating, I always end up just cloning the repo and searching for things using my own tools "about twice as fast", yeah twice as fast but twice less effective.. what's the point