7 ms·

I don't understand why you think a search engine requires the use of "real engineering skills" like picking and choosing libraries and identifying which opportu

I don't understand why you think a search engine requires the use of "real engineering skills" like picking and choosing libraries and identifying which opportunities will yield fruitful optimization. Literally all of the listed projects, text editors, compilers, operating systems, and ray tracers, can exercise the exact same activities. I'm more inclined to think that your comment is really more revealing about what kind of knowledge is in your own personal wheelhouse. Having worked on a ray tracer that was subsequently used on many feature films, I can assure you that all of your bullet points apply to that project, too. It's much more a matter of whether you want to do something small scale and fun, or whether you want to suck all the joy out of it by applying the same soul crushing constraints we already get paid to do in our day jobs. Bleh.

PH95VuimJjqBqy 3y ago
> I'm more inclined to think that your comment is really more revealing about what kind of knowledge is in your own personal wheelhouse. That's my read as well.
p-e-w 3y ago
> Literally all of the listed projects, text editors, compilers, operating systems, and ray tracers, can exercise the exact same activities. In the linked article, these projects are all explicitly described as opportunities to learn about low-level stuff like how to efficiently store editable text. The difference with a web search engine is that nobody today can build such a thing completely from scratch, therefore it forces you to give up the toxic NIH mentality, which in my experience is usually driven by elitism and ego (on full display in this comment thread), not by some lofty desire to learn something new.
- 59nadir 3y ago
  > The difference with a web search engine is that nobody today can build such a thing completely from scratch I'm sorry, but can you substantiate this claim? I've seen no indication that a search engine is not buildable from scratch at all.
  - fho 3y ago
    Sigh ... Of to the silicone mines it is then ... again /s
    - isoprophlex 3y ago
      You mean, the beach?
      - sethammons 3y ago
        It is a joke based on needing to define "from scratch". Similar to, "how do you bake a cake from scratch? First you must create the universe." Op is gathering the raw ingredients to start fabricating his chips.
        enneff 3y ago
        Yeah, that’s sand. :)
        fho 3y ago
        Fair enough :-)
  - crazygringo 3y ago
    Assuming you get the TCP/IP stack for free, you still need to build fully-featured HTTPS and a "webscale" multi-server database for document storage from scratch. The crawler is easy and so is something like PageRank, but then building the sharded keyword text search engine itself that operates at webscale is a whole other project... The point is that it's too much work for a single person to build all the parts themselves. It's only feasible if you rely on pre-existing HTTPS libraries and database and text search technologies.
    - YetAnotherNick 3y ago
      Simple methods of search like exact matching are very fast using textbook algorithms. There are well known algorithm like suffix tree which could search in millions of documents in milliseconds.
- yu3zhou4 3y ago
  I enjoy reading your discussion and just wanted to add that some people write a big scale software from scratch nowadays - for instance Marginalia for web search and Andreas Kling and team for operating system and web browser
  - cinntaile 3y ago
    I guess the first step in writing large scale software is to become Swedish!
    - marginalia_nu 3y ago
      Long cold winters dimly lit by a flickering CRT.
  - p-e-w 3y ago
    Marginalia, which I'm a big fan of, is not "written from scratch" (it would be stupid to do so). Check the project on GitHub, it has lots of third-party dependencies.
    - marginalia_nu 3y ago
      I definitely lean more toward NIH than what's conventionally considered wise, but most of the time it's not NIH for the sake of NIH. I do pull a lot of libraries, but an enormous amount of what the search engine does is very much built from scratch. The libraries generally deal with parsing common formats, compression, serialization, and various service glue like dependency injection. I think the number of explicit dependencies is a bit inflated by the choice to not use a framework like springboot, which pulls many of the same (or equivalent ones) implicitly. What makes the search engine a search engine, the indexing software (all the way down to database primitives like btrees etc.), a large chunk of the language processing, and so forth; that's all bespoke. I think it needs to be. A lot of existing code just doesn't scale, or has too many customizations that would add unnecessary glue and complexity to my own code. I'm going to echo SerenityOS Andreas and suggest that it's a skill like any other. If you shy away from building custom solutions to hard problems, you will never be good at it; and it will become a self-fulfilling prophecy that these NIH solutions are too hard to build. At the same time, there's a time and a place and you should indeed be judicious as to when to roll your own solutions, but maybe that time and place is exactly in a hobby project like the ones suggested in this thread (and is how my search engine started out; a place to dick around with difficult problems). I'd also add that being able to tackle problems yourself, rather than needing a library to do all the heavy lifting at all times, is a great enabler. Sometimes there is no adequate library, but that doesn't mean the conclusion has to be "welp, I guess we can't do that yet..."
- indigo945 3y ago
  Back when online learn-to-code courses like Codecadamy and Udemy were a fad, I remember that one of them (and unfortunately I don't remember which, and Google, ironically, turns up nothing) taught how to build a search engine in Python from scratch as a first project for complete beginners. I thought it had a reasonable level of complexity for this task. You can still find search-engine-from-scratch courses on Udemy, complete with all the necessary algorithms [1]. [1]: https://www.udemy.com/course/build-a-search-engine-with-python-computer-science-python/ https://www.udemy.com/course/build-a-search-engine-with-pyth...
  - universse 3y ago
    was it udacity's cs101?
    - indigo945 3y ago
      Yes! Thank you.
      - dmoy 3y ago
        Incidentally, the first Google search result for "introduction online cs course that uses Python the build a search engine from scratch" is udemy CS 101 Even through my smartphone autocorrect mistakes of 'introductory -> introduction' and 'the -> to'
- benj111 3y ago
  >elitism and ego You know those car guys who will rebuild their engine, just because? Or those retro computer guys that will recap an ancient board rather than buying a modern pc? For you it is a job, for me it is a hobby. I have no interest in making something 'professional' I want to take it apart to understand how it works. Want to understand how a text editor works? Write one. What component of that is ego? It seems to me you're the elitist, you're not far off saying mere users shouldn't be allowed to modify their own software, shouldn't be allowed to install software that hasn't been okayed by the people who know what they're doing.
  - finnthehuman 3y ago
    > What component of that is ego? Your lack of constraints on personal exploration in software is interpreted as hubris by people who came to software seeking high paying regimented recipe-following.
- vkazanov 3y ago
  A simple search engine is certainly doable from scratch in a matter of weeks, complete with most things expected from a search engine. Similar to compilers or tiny OSes, one can go as hardcore as necessary, or just stick to basic stuff. Of all the typical personal challenge style projects, databases are probable the hardest to build, and even that is not impossible.
  - fragmede 3y ago
    making a database is easy. making a fast database is what's hard.
    - kragen 3y ago
      even implementing all of sql92 or datalog without concern for efficiency is fairly complex
      - convolvatron 3y ago
        implementing a toy datalog should take no more than a week.
        kragen 3y ago
        maybe less
        kragen 3y ago
        but a text editor, a space invaders clone, a tiny basic compiler, or a mini operating system is an afternoon project. a spreadsheet or a video game console emulator is maybe comparable to implementing datalog
    - pointy_hat 3y ago
      Also, making a correct one is not an easy fit.
- AeroNotix 3y ago
  Please can you define "from scratch"?
  - gushogg-blake 3y ago
    I encountered some push back on this in my "code editor from the ground up" post[1]. I think the only reasonable definition of from scratch is: Does not have domain-specific dependencies. So a code editor based on ACE or CodeMirror would not be from scratch, obviously, but one that involves writing all of the domain-specific logic would be. Using generic libraries doesn't stop something being from scratch. (In my case Tree-sitter is arguably domain-specific, but an early version did use a hand-coded JavaScript tokeniser in its place.) [1] https://news.ycombinator.com/item?id=34577246 https://news.ycombinator.com/item?id=34577246
- jvanderbot 3y ago
  If you want to build a compiler from scratch, you must first invent the universe. Peeling back abstractions to see how things could or should work is perfectly fine, even for professionals. Case in point, I've spent a year excising bloated frameworks from my stack at work and replacing the few corners we needed from those frameworks with, e.g. 50 lines of curl calls. The C compiles instantly and is tailored for our tiny use case, produced much quicker delivery on our one related feature we wanted, and removed chains of dependencies. Being reliant on far-too-abstract libraries and frameworks to do simple jobs is also a curse. But nobody at work has the experience to know that curl was sitting right there on our image available for our use. And nobody has that experience because nobody took the time to build something from low level libraries. Now we know how and can make an intelligent decision without defaulting in either direction because we were afraid to try.
  - pipes 3y ago
    Genuine question: what value was added by getting rid of the frameworks?
    - jvanderbot 3y ago
      20,000 fewer lines of code, faster compile / deploy times, significantly less interfacing / translation between "their" types and "their" apis, and a much better control over types and structure across our codebase b/c we didn't need "their" types and structure anywhere. It had crept everywhere. It's just faster, cleaner, and easier in a few cases to do precisely what you need right now, rather than anticipate a million things you might need and refactor your code to adopt a given "solution".
      - cellularmitosis 3y ago
        Yeah, the problem / benefit is never the one framework you added / removed, it is the mindset of reaching for another dependency as a default which leads to a behemoth which no one enjoys working with.
        3y ago
- phkahler 3y ago
  >> The difference with a web search engine is that nobody today can build such a thing completely from scratch Really? I think there are a couple out there. The main issue today is scale, but you could constrain that by limiting your crawler to a fixed set of sites.
- HumblyTossed 3y ago
  Well, if I’m going to build a web search engine from scratch don’t I first need to write a compiler from scratch? Which means I first need to write an editor from scratch…
- Xeamek 3y ago
  While nih is ultimately dependent on multiple different aspects, I would argue that, ironically, it's more likely to create nih-qualifying product with your aproach. Because when you write things from scratch, you actually have the space to innovate. But when building product with preexisting puzzles, there is much less space to actually make any usefull changes that would make your product actually standout from existing alternatives (which nih is all about)
- datadeft 3y ago
  I think searx was largely built by a single person. https://github.com/searx/searx https://github.com/searx/searx
mattarm 3y ago
> It's much more a matter of whether you want to do something small scale and fun, or whether you want to suck all the joy out of it by applying the same soul crushing constraints we already get paid to do in our day jobs. Bleh. Amen. And further, what better prepares a programmer to assess the relative costs of implementing a thing vs using a library providing that thing than having attempted an implementation? Learning by doing is a valid approach, and this can even be called fun.
tamrix 3y ago
How about the ultimate programming challenge. Make a project that makes real money.

PH95VuimJjqBqy 3y ago

> I'm more inclined to think that your comment is really more revealing about what kind of knowledge is in your own personal wheelhouse. That's my read as well.

p-e-w 3y ago

> Literally all of the listed projects, text editors, compilers, operating systems, and ray tracers, can exercise the exact same activities. In the linked article, these projects are all explicitly described as opportunities to learn about low-level stuff like how to efficiently store editable text. The difference with a web search engine is that nobody today can build such a thing completely from scratch, therefore it forces you to give up the toxic NIH mentality, which in my experience is usually driven by elitism and ego (on full display in this comment thread), not by some lofty desire to learn something new.

59nadir 3y ago
> The difference with a web search engine is that nobody today can build such a thing completely from scratch I'm sorry, but can you substantiate this claim? I've seen no indication that a search engine is not buildable from scratch at all.
- fho 3y ago
  Sigh ... Of to the silicone mines it is then ... again /s
  - isoprophlex 3y ago
    You mean, the beach?
    - sethammons 3y ago
      It is a joke based on needing to define "from scratch". Similar to, "how do you bake a cake from scratch? First you must create the universe." Op is gathering the raw ingredients to start fabricating his chips.
      - enneff 3y ago
        Yeah, that’s sand. :)
        fho 3y ago
        Fair enough :-)
- crazygringo 3y ago
  Assuming you get the TCP/IP stack for free, you still need to build fully-featured HTTPS and a "webscale" multi-server database for document storage from scratch. The crawler is easy and so is something like PageRank, but then building the sharded keyword text search engine itself that operates at webscale is a whole other project... The point is that it's too much work for a single person to build all the parts themselves. It's only feasible if you rely on pre-existing HTTPS libraries and database and text search technologies.
  - YetAnotherNick 3y ago
    Simple methods of search like exact matching are very fast using textbook algorithms. There are well known algorithm like suffix tree which could search in millions of documents in milliseconds.
yu3zhou4 3y ago
I enjoy reading your discussion and just wanted to add that some people write a big scale software from scratch nowadays - for instance Marginalia for web search and Andreas Kling and team for operating system and web browser
- cinntaile 3y ago
  I guess the first step in writing large scale software is to become Swedish!
  - marginalia_nu 3y ago
    Long cold winters dimly lit by a flickering CRT.
- p-e-w 3y ago
  Marginalia, which I'm a big fan of, is not "written from scratch" (it would be stupid to do so). Check the project on GitHub, it has lots of third-party dependencies.
  - marginalia_nu 3y ago
    I definitely lean more toward NIH than what's conventionally considered wise, but most of the time it's not NIH for the sake of NIH. I do pull a lot of libraries, but an enormous amount of what the search engine does is very much built from scratch. The libraries generally deal with parsing common formats, compression, serialization, and various service glue like dependency injection. I think the number of explicit dependencies is a bit inflated by the choice to not use a framework like springboot, which pulls many of the same (or equivalent ones) implicitly. What makes the search engine a search engine, the indexing software (all the way down to database primitives like btrees etc.), a large chunk of the language processing, and so forth; that's all bespoke. I think it needs to be. A lot of existing code just doesn't scale, or has too many customizations that would add unnecessary glue and complexity to my own code. I'm going to echo SerenityOS Andreas and suggest that it's a skill like any other. If you shy away from building custom solutions to hard problems, you will never be good at it; and it will become a self-fulfilling prophecy that these NIH solutions are too hard to build. At the same time, there's a time and a place and you should indeed be judicious as to when to roll your own solutions, but maybe that time and place is exactly in a hobby project like the ones suggested in this thread (and is how my search engine started out; a place to dick around with difficult problems). I'd also add that being able to tackle problems yourself, rather than needing a library to do all the heavy lifting at all times, is a great enabler. Sometimes there is no adequate library, but that doesn't mean the conclusion has to be "welp, I guess we can't do that yet..."
indigo945 3y ago
Back when online learn-to-code courses like Codecadamy and Udemy were a fad, I remember that one of them (and unfortunately I don't remember which, and Google, ironically, turns up nothing) taught how to build a search engine in Python from scratch as a first project for complete beginners. I thought it had a reasonable level of complexity for this task. You can still find search-engine-from-scratch courses on Udemy, complete with all the necessary algorithms [1]. [1]: https://www.udemy.com/course/build-a-search-engine-with-python-computer-science-python/ https://www.udemy.com/course/build-a-search-engine-with-pyth...
- universse 3y ago
  was it udacity's cs101?
  - indigo945 3y ago
    Yes! Thank you.
    - dmoy 3y ago
      Incidentally, the first Google search result for "introduction online cs course that uses Python the build a search engine from scratch" is udemy CS 101 Even through my smartphone autocorrect mistakes of 'introductory -> introduction' and 'the -> to'
benj111 3y ago
>elitism and ego You know those car guys who will rebuild their engine, just because? Or those retro computer guys that will recap an ancient board rather than buying a modern pc? For you it is a job, for me it is a hobby. I have no interest in making something 'professional' I want to take it apart to understand how it works. Want to understand how a text editor works? Write one. What component of that is ego? It seems to me you're the elitist, you're not far off saying mere users shouldn't be allowed to modify their own software, shouldn't be allowed to install software that hasn't been okayed by the people who know what they're doing.
- finnthehuman 3y ago
  > What component of that is ego? Your lack of constraints on personal exploration in software is interpreted as hubris by people who came to software seeking high paying regimented recipe-following.
vkazanov 3y ago
A simple search engine is certainly doable from scratch in a matter of weeks, complete with most things expected from a search engine. Similar to compilers or tiny OSes, one can go as hardcore as necessary, or just stick to basic stuff. Of all the typical personal challenge style projects, databases are probable the hardest to build, and even that is not impossible.
- fragmede 3y ago
  making a database is easy. making a fast database is what's hard.
  - kragen 3y ago
    even implementing all of sql92 or datalog without concern for efficiency is fairly complex
    - convolvatron 3y ago
      implementing a toy datalog should take no more than a week.
      - kragen 3y ago
        maybe less
        kragen 3y ago
        but a text editor, a space invaders clone, a tiny basic compiler, or a mini operating system is an afternoon project. a spreadsheet or a video game console emulator is maybe comparable to implementing datalog
  - pointy_hat 3y ago
    Also, making a correct one is not an easy fit.
AeroNotix 3y ago
Please can you define "from scratch"?
- gushogg-blake 3y ago
  I encountered some push back on this in my "code editor from the ground up" post[1]. I think the only reasonable definition of from scratch is: Does not have domain-specific dependencies. So a code editor based on ACE or CodeMirror would not be from scratch, obviously, but one that involves writing all of the domain-specific logic would be. Using generic libraries doesn't stop something being from scratch. (In my case Tree-sitter is arguably domain-specific, but an early version did use a hand-coded JavaScript tokeniser in its place.) [1] https://news.ycombinator.com/item?id=34577246 https://news.ycombinator.com/item?id=34577246
jvanderbot 3y ago
If you want to build a compiler from scratch, you must first invent the universe. Peeling back abstractions to see how things could or should work is perfectly fine, even for professionals. Case in point, I've spent a year excising bloated frameworks from my stack at work and replacing the few corners we needed from those frameworks with, e.g. 50 lines of curl calls. The C compiles instantly and is tailored for our tiny use case, produced much quicker delivery on our one related feature we wanted, and removed chains of dependencies. Being reliant on far-too-abstract libraries and frameworks to do simple jobs is also a curse. But nobody at work has the experience to know that curl was sitting right there on our image available for our use. And nobody has that experience because nobody took the time to build something from low level libraries. Now we know how and can make an intelligent decision without defaulting in either direction because we were afraid to try.
- pipes 3y ago
  Genuine question: what value was added by getting rid of the frameworks?
  - jvanderbot 3y ago
    20,000 fewer lines of code, faster compile / deploy times, significantly less interfacing / translation between "their" types and "their" apis, and a much better control over types and structure across our codebase b/c we didn't need "their" types and structure anywhere. It had crept everywhere. It's just faster, cleaner, and easier in a few cases to do precisely what you need right now, rather than anticipate a million things you might need and refactor your code to adopt a given "solution".
    - cellularmitosis 3y ago
      Yeah, the problem / benefit is never the one framework you added / removed, it is the mindset of reaching for another dependency as a default which leads to a behemoth which no one enjoys working with.
      - 3y ago
phkahler 3y ago
>> The difference with a web search engine is that nobody today can build such a thing completely from scratch Really? I think there are a couple out there. The main issue today is scale, but you could constrain that by limiting your crawler to a fixed set of sites.
HumblyTossed 3y ago
Well, if I’m going to build a web search engine from scratch don’t I first need to write a compiler from scratch? Which means I first need to write an editor from scratch…
Xeamek 3y ago
While nih is ultimately dependent on multiple different aspects, I would argue that, ironically, it's more likely to create nih-qualifying product with your aproach. Because when you write things from scratch, you actually have the space to innovate. But when building product with preexisting puzzles, there is much less space to actually make any usefull changes that would make your product actually standout from existing alternatives (which nih is all about)
datadeft 3y ago
I think searx was largely built by a single person. https://github.com/searx/searx https://github.com/searx/searx

mattarm 3y ago

> It's much more a matter of whether you want to do something small scale and fun, or whether you want to suck all the joy out of it by applying the same soul crushing constraints we already get paid to do in our day jobs. Bleh. Amen. And further, what better prepares a programmer to assess the relative costs of implementing a thing vs using a library providing that thing than having attempted an implementation? Learning by doing is a valid approach, and this can even be called fun.

tamrix 3y ago

How about the ultimate programming challenge. Make a project that makes real money.