18 ms·

Here's a detailed text outline with key frames for those who don't have time to watch the 36 minute video: https://www.videogist.co/videos/lessons-from-buildin

Here's a detailed text outline with key frames for those who don't have time to watch the 36 minute video: https://www.videogist.co/videos/lessons-from-building-github-code-search-by-luke-francl-strange-loop-2023-1823 https://www.videogist.co/videos/lessons-from-building-github...

ngshiheng 3y ago
this is awesome. i think it'd be super cool if it can read/summarize the comments too
- paradox460 3y ago
  Not sure if it can, but Kagi can: • GitHub's previous code search was slow, limited, and did not support searching Forks due to indexing challenges. A new system called Blackbird was built from scratch to address these issues. • Indexing code poses unique challenges compared to natural language documents, such as handling file changes in version control systems and deduplicating shared code across repositories. • The talk discussed techniques used in Blackbird like trigram tokenization, delta compression, caching, and dynamic shard assignment to improve indexing speed and efficiency at scale. • Architectural decisions like separating indexing from querying and using message queues helped Blackbird scale independently without competing for resources. • Data structures like geometric XOR filters were developed to efficiently estimate differences between codebases and enable features like delta compression. • Iteration speed was improved by making the system easier to change through frequent index version increments without migrations. • Resource usage was optimized through techniques such as document deduplication, caching, and compaction to reduce indexing costs. • Blackbird's design allowed it to efficiently support over 100 million code repositories while the previous system struggled at millions. • Building custom solutions from scratch can be worthwhile when leveraging data structure to outperform generic tools for a domain. • Anticipating and addressing scaling challenges at each magnitude is important to ensure a system remains performant as it grows over time.
  - jve 3y ago
    Those don't look like video comments. Which are: fantastic/well done/wonderful/great talk etc.
ugiox 3y ago
Everyone has 36 minutes to watch a video. Just skip bing watching Netflix. Poor society we are part of, if everything needs to be consumed in 5 minute chunks. One should also think and reflect about the content being presented. Grasp the ideas. It is also about honoring the time the speaker put into the presentation preparing it.
- deleted 3y ago
  [deleted]
100k 3y ago
Nice! Seems like a useful tool for digesting videos.

ngshiheng 3y ago

this is awesome. i think it'd be super cool if it can read/summarize the comments too

paradox460 3y ago
Not sure if it can, but Kagi can: • GitHub's previous code search was slow, limited, and did not support searching Forks due to indexing challenges. A new system called Blackbird was built from scratch to address these issues. • Indexing code poses unique challenges compared to natural language documents, such as handling file changes in version control systems and deduplicating shared code across repositories. • The talk discussed techniques used in Blackbird like trigram tokenization, delta compression, caching, and dynamic shard assignment to improve indexing speed and efficiency at scale. • Architectural decisions like separating indexing from querying and using message queues helped Blackbird scale independently without competing for resources. • Data structures like geometric XOR filters were developed to efficiently estimate differences between codebases and enable features like delta compression. • Iteration speed was improved by making the system easier to change through frequent index version increments without migrations. • Resource usage was optimized through techniques such as document deduplication, caching, and compaction to reduce indexing costs. • Blackbird's design allowed it to efficiently support over 100 million code repositories while the previous system struggled at millions. • Building custom solutions from scratch can be worthwhile when leveraging data structure to outperform generic tools for a domain. • Anticipating and addressing scaling challenges at each magnitude is important to ensure a system remains performant as it grows over time.
- jve 3y ago
  Those don't look like video comments. Which are: fantastic/well done/wonderful/great talk etc.

ugiox 3y ago

Everyone has 36 minutes to watch a video. Just skip bing watching Netflix. Poor society we are part of, if everything needs to be consumed in 5 minute chunks. One should also think and reflect about the content being presented. Grasp the ideas. It is also about honoring the time the speaker put into the presentation preparing it.

deleted 3y ago
[deleted]

100k 3y ago

Nice! Seems like a useful tool for digesting videos.