7 ms·

Why I chose to write Nuuton in Go

71 points by orangethirty 13y ago

thirsteh 13y ago
To the point that it's developing at a rapid pace: Yes and no. There is a lot of development happening on Go, but the APIs that you can access in the stable releases are not changing behavior. The Go 1 compatibility contract guarantees that no breaking API changes will happen for many years, barring some serious security issue: http://golang.org/doc/go1compat.html http://golang.org/doc/go1compat.html If it compiles now, it should continue to compile and work as expected in the future.
mseepgood 13y ago
I don't know who he is and why he's building a search engine, but I'll upvote this post, since I'm fond of Go as well.
- orangethirty 13y ago
  I'm a nobody. Just some guy having fun pushing his limits. We don't usually need a reason to hack. The act of hacking itself is the reason. (:
coldtea 13y ago
>Google started developing Go for one simple reason: to simplify the act of writing search systems. Not really. For one, Google didn't develop Go (it wasn't a company decision to get a new language, as Java was for SUN, .NET was for MS and Dart was for Google). Go was merely started by some a team working at Google as a side project and it grew a little from there. They do pay several people to work on it nowadays, but it's not some "bet the company" language. Also, Google hasn't mandated replacing any C++/Java stuff with Go, and nobody is touching their search engine code. If it happens, it happens organically and for lesser stuff. Second, Go was developed for "systems" software (albeit in the sense of network servers and such), not search specifically.
- thirsteh 13y ago
  > Go was merely started by some a team working at Google as a side project History shows that a lot of extremely successful things started out that way. > and it grew a little from there That's a bit of an understatement. Go has hundreds of contributors, and is already used by thousands of companies. > They do pay several people to work on it nowadays, but it's not some "bet the company" language. Definitely not. For that matter, neither is Dart. > Also, Google hasn't mandated replacing any C++/Java stuff with Go, and nobody is touching their search engine code. If it happens, it happens organically and for lesser stuff. You make it sound like Go isn't being used at Google, which is disingenuous. No, it probably hasn't replaced the crawler and pagerank logic--it'd be silly to rewrite all of that--but it is continuously being picked for new projects internally. Some of the few such projects that have become public are Vitess, the database frontend that powers YouTube, and the code that powers dl.google.com. (That there is no known mandate to replace existing code isn't indicative of much. It's a bad move to rewrite that much code for any reason.) > Second, Go was developed for "systems" software (albeit in the sense of network servers and such), not search specifically. It started out as a "systems language" (which roughly translates to "for writing backend software", but was commonly misunderstood), however that description has been removed since Go has proved useful in other areas. It is now simply a general-purpose programming language.
  - coldtea 13y ago
    >>Go was merely started by some a team working at Google as a side project >History shows that a lot of extremely successful things started out that way. Also a lot of failures. But both cases are beside the point. I didn't correct him on whether Go is good or not. Just in that it was not started officially by Google. >That's a bit of an understatement. Go has hundreds of contributors, and is already used by thousands of companies. And yet it's still a totally marginal language in the grand scheme of things. Not even 1/100 of what something like Java or .NET is, for example. >Definitely not. For that matter, neither is Dart. Agreed on Dart (and I don't think I said it was). But .NET and Java were. >You make it sound like Go isn't being used at Google, which is disingenuous. No, it probably hasn't replaced the crawler and pagerank logic--it'd be silly to rewrite all of that--but it is continuously being picked for new projects internally. Some of the few such projects that have become public are Vitess, the database frontend that powers YouTube, and the code that powers dl.google.com. Yes, I've read about both, I follow Go since the beginning. Those two quite humble projects are the only things that have become publicly known for 2 years now. A db load balancer (which had some Go issues they had to code around even), and an abandoned part of Google infrastructure in C++ that a Go team member volunteered to rewrite in Go. >however that description has been removed since Go has proved useful in other areas. It is now simply a general-purpose programming language. I believe the reason it that description has been removed was because people complained that "systems software" means specifically OSs, drivers and such, not just backend software.
Trufa 13y ago
Wouldn't it be funny if the search engine that replaces Google is written in a language developed by Google?
blunte 13y ago
The blog post was written in a variant of the English language called "Fragment".
- orangethirty 13y ago
  You are right, it is not very well written. It was late, I was tired.
oskarth 13y ago
Regarding Lisp Things got complicated quickly. I was writing too much code to achieve simple things. Plus I was writing a lot of one-off code. I prefer to write as much reusable code a I can. Mostly because I'm lazy and I like to keep things simple. I'd be interested to hear more details on this. Writing non-reusable code and lisp seems antithetical to me. Do you mean the lack of libraries and interfaces to other services?
- capisce 13y ago
  Yeah, it sounds very dubious to me that the amount of code in Go ends up being much less than what's required to achieve the same in Lisp. From what I've seen Go looks verbose in comparison with the lack of macros and higher level programming.
  - orangethirty 13y ago
    It has proven to be less verbose for this project. Probably due to my lack of Lisp skills. I have been in love with Lisp for years, and study it actively. But I'm not a PG with it. Sadly, my natural programming skills are not that great.
- orangethirty 13y ago
  Its a combination of that factor, and my lack of understanding of the problem. Personally, I did not feel Lisp was very fitting to my work flow and goals. It still is one of my favourite languages, but not the right choice for this project.
coolsunglasses 13y ago
Odd choice. A little more research could've yielded a migration to something like Clojure with the benefits of Lisp and an industry-backed stack (JVM). Of course using an offbeat stack is going to hurt - doesn't mean you have to give up Lisp too. A decision made odder still since being an independent project, one would think maximizing individual productivity would be the only hope of yielding anything novel and useful in a field like search that isn't a fitting of an off-the-shelf to a business problem. (Something I've done before, it was quite neat.) Go is not a language focused around individual productivity in any capacity or scope. It's the language of pyramid-building Egyptian Pharaohs. (Throw 1,000 lives at it, it'll get 'er done...) I say this as someone rather envious of any language that has a native compiler that builds static binaries. (Oh, OCaml...) As we ponder on what orangetide is up to, lets keep in mind he's not explained or revealed anything at all about Nuuton and for all we know - could be having us on for the sake of self-promotion/promoting Go. The incorrect speculation about the origins and purpose of Go make this even stranger. Go exists to make people like Rob Pike happy. The Plan9 and Suckless community of programmers. Those types. That it happened to solve the problems of some Google programmers who were making it was almost incidental.
- capisce 13y ago
  > Go is not a language focused around individual productivity in any capacity or scope. Seems to have been my impression as well. Lisp with its macros can make an individual programmer who doesn't need to worry about coding conventions incredibly productive. I see Rust has a semi-powerful macro system too although not to the extent of Lisp, as it doesn't look like you can evaluate arbitrary code at compile time.
  - pjmlp 13y ago
    I was attracted to Go at the beginning due to Limbo and Oberon influences that the language has. Even did some coding in Go for a few minor projects. But that experience showed me that it actually felt like a time travel back to 1992, when I was using Native Oberon. Sure the language feels like a better C and I find it good that it has native code compilers as the default implementation. But since 1992 we have gained lots of abstractions in mainstream languages that it is pity to throw them out of the window.
danieldk 13y ago
There is no real substance in this blog post. At the end: I chose Go because it is being developed to solve the same problems I have to solve. All by a group of scientists an programmers that are each twice as smart as I have ever been (on my best day). So, what are the problems you are trying to solve? And why can Go solve them better than, Java, Erlang, C++, Scala, or any other language? If you could answer that question, you'd have an interesting blog post. And if the answer is 'concurrency via Goroutines and channels' (it always is), explain why they are better than Erlang or Akka's actors, STM in Clojure or Haskell, or other proposals, and you'd have a stellar blog post. But really, this at the top of Hacker News?
- pjmlp 13y ago
  Go effect. Do you remember when Rails and Node were every day on HN? Now it is time for Go.
  - orangethirty 13y ago
    I actually thought about using those. Rails is a memory hog, and Node is Javascript ( a language I am not fond of).
  - nazka 13y ago
    Now it is time to Go.
  - gabipurcaru 13y ago
    Maybe; but that's the way towards progress
    - pjmlp 13y ago
      We have a different opinion what progress means, it seems.
  - reinhardt 13y ago
    Every language has its 15 weeks of fame.
thomseddon 13y ago
Three rewrites and not yet out of (or even in) beta: http://nuuton.com/ http://nuuton.com/ If you are seriously hoping for this to be a full time business then I would strongly recommend you read "The Lean Startup" or similar. This blind technical pursuit of perfection is a pretty classic approach for us techies and something i'm guilty of myself. It was actually that process that made me realise what my strengths actually were and what I would need to do to start a world class company (work with great people). If this isn't really a commercial project then just ignore the above and happy hacking, perhaps you should consider open sourcing it?
- orangethirty 13y ago
  I'm always telling people here about how they should sell first, and code later. This project has been challenging in that regard, but I have been spending a lot of time figuring out the business aspect of search. It is challenging because one of my main goals is to respect privacy. These days, violating the privacy of the users is a sure way to cash in. But I don't want to do that. As a result, I've had to write a couple of versions to see how it would work out. My current plan incorporates a product line right into the stack. That way, if/when the time comes, I can focus on writing the necessary parts. No need to scramble and try to make it fit. I do follow the lean principle. But one cannot blindly adhere to them. Nuuton is not the sort of project where you just try and get something to work. There had to be a foundation first. But from now on, development will follow the build-measure-learn cycle. I am open sourcing parts of it. Mostly Go libraries that might result from developing it.
ErrantX 13y ago
I wanted it to be pure Python because I love Python. That is until I had to work with threads. After spending some time working with gevent, I decided to simply use another language to deal with that. If I could have a penny...
- orangethirty 13y ago
  May you expand on the comment.
  - ErrantX 13y ago
    It's a common complaint about Python, that's all :)
    - orangethirty 13y ago
      Yes, I know. Had to work on a commercial project that used gevent. Boy was that a mess. The project was a server that had to handle 30K+ hits/second. The client's engineering dept could only get it to do 3K. When I saw they were using gevent (along with a custom framework that lacked any documentation), I knew their goal had a long way to go. Last thing I heard, they were having trouble hiring people to work with it. I'm glad I dodged that bullet.
sramsay 13y ago
And the award for best typo ever goes to: "[Lisp] had been my sweetheart for a log time."
knodi 13y ago
Do the personal whys really matter? Its a good young lang that has a lot of promise.
hal84 13y ago
So I guess if I posted about how I chose C++ and Perl to write Enginuity Search Engine I'd get no love on HN because its boring, stable, scales and just works.
- obviouslygreen 13y ago
  Maybe not. Or maybe you would, if your reasoning were well-presented and reflected a logical approach and conclusion, and if it were insightful in a way that people found interesting and useful.
  - hal84 13y ago
    Well I might have a go at it soon in our engineering section of our company blog. Have some interesting network performance analysis info to share as well.

71 points by orangethirty 13y ago

To the point that it's developing at a rapid pace: Yes and no. There is a lot of development happening on Go, but the APIs that you can access in the stable releases are not changing behavior. The Go 1 compatibility contract guarantees that no breaking API changes will happen for many years, barring some serious security issue: http://golang.org/doc/go1compat.html http://golang.org/doc/go1compat.html If it compiles now, it should continue to compile and work as expected in the future.

mseepgood 13y ago

I don't know who he is and why he's building a search engine, but I'll upvote this post, since I'm fond of Go as well.

orangethirty 13y ago
I'm a nobody. Just some guy having fun pushing his limits. We don't usually need a reason to hack. The act of hacking itself is the reason. (:

coldtea 13y ago

>Google started developing Go for one simple reason: to simplify the act of writing search systems. Not really. For one, Google didn't develop Go (it wasn't a company decision to get a new language, as Java was for SUN, .NET was for MS and Dart was for Google). Go was merely started by some a team working at Google as a side project and it grew a little from there. They do pay several people to work on it nowadays, but it's not some "bet the company" language. Also, Google hasn't mandated replacing any C++/Java stuff with Go, and nobody is touching their search engine code. If it happens, it happens organically and for lesser stuff. Second, Go was developed for "systems" software (albeit in the sense of network servers and such), not search specifically.

thirsteh 13y ago
> Go was merely started by some a team working at Google as a side project History shows that a lot of extremely successful things started out that way. > and it grew a little from there That's a bit of an understatement. Go has hundreds of contributors, and is already used by thousands of companies. > They do pay several people to work on it nowadays, but it's not some "bet the company" language. Definitely not. For that matter, neither is Dart. > Also, Google hasn't mandated replacing any C++/Java stuff with Go, and nobody is touching their search engine code. If it happens, it happens organically and for lesser stuff. You make it sound like Go isn't being used at Google, which is disingenuous. No, it probably hasn't replaced the crawler and pagerank logic--it'd be silly to rewrite all of that--but it is continuously being picked for new projects internally. Some of the few such projects that have become public are Vitess, the database frontend that powers YouTube, and the code that powers dl.google.com. (That there is no known mandate to replace existing code isn't indicative of much. It's a bad move to rewrite that much code for any reason.) > Second, Go was developed for "systems" software (albeit in the sense of network servers and such), not search specifically. It started out as a "systems language" (which roughly translates to "for writing backend software", but was commonly misunderstood), however that description has been removed since Go has proved useful in other areas. It is now simply a general-purpose programming language.
- coldtea 13y ago
  >>Go was merely started by some a team working at Google as a side project >History shows that a lot of extremely successful things started out that way. Also a lot of failures. But both cases are beside the point. I didn't correct him on whether Go is good or not. Just in that it was not started officially by Google. >That's a bit of an understatement. Go has hundreds of contributors, and is already used by thousands of companies. And yet it's still a totally marginal language in the grand scheme of things. Not even 1/100 of what something like Java or .NET is, for example. >Definitely not. For that matter, neither is Dart. Agreed on Dart (and I don't think I said it was). But .NET and Java were. >You make it sound like Go isn't being used at Google, which is disingenuous. No, it probably hasn't replaced the crawler and pagerank logic--it'd be silly to rewrite all of that--but it is continuously being picked for new projects internally. Some of the few such projects that have become public are Vitess, the database frontend that powers YouTube, and the code that powers dl.google.com. Yes, I've read about both, I follow Go since the beginning. Those two quite humble projects are the only things that have become publicly known for 2 years now. A db load balancer (which had some Go issues they had to code around even), and an abandoned part of Google infrastructure in C++ that a Go team member volunteered to rewrite in Go. >however that description has been removed since Go has proved useful in other areas. It is now simply a general-purpose programming language. I believe the reason it that description has been removed was because people complained that "systems software" means specifically OSs, drivers and such, not just backend software.

Trufa 13y ago

Wouldn't it be funny if the search engine that replaces Google is written in a language developed by Google?

blunte 13y ago

The blog post was written in a variant of the English language called "Fragment".

orangethirty 13y ago
You are right, it is not very well written. It was late, I was tired.

oskarth 13y ago

Regarding Lisp Things got complicated quickly. I was writing too much code to achieve simple things. Plus I was writing a lot of one-off code. I prefer to write as much reusable code a I can. Mostly because I'm lazy and I like to keep things simple. I'd be interested to hear more details on this. Writing non-reusable code and lisp seems antithetical to me. Do you mean the lack of libraries and interfaces to other services?

capisce 13y ago
Yeah, it sounds very dubious to me that the amount of code in Go ends up being much less than what's required to achieve the same in Lisp. From what I've seen Go looks verbose in comparison with the lack of macros and higher level programming.
- orangethirty 13y ago
  It has proven to be less verbose for this project. Probably due to my lack of Lisp skills. I have been in love with Lisp for years, and study it actively. But I'm not a PG with it. Sadly, my natural programming skills are not that great.
orangethirty 13y ago
Its a combination of that factor, and my lack of understanding of the problem. Personally, I did not feel Lisp was very fitting to my work flow and goals. It still is one of my favourite languages, but not the right choice for this project.

coolsunglasses 13y ago

Odd choice. A little more research could've yielded a migration to something like Clojure with the benefits of Lisp and an industry-backed stack (JVM). Of course using an offbeat stack is going to hurt - doesn't mean you have to give up Lisp too. A decision made odder still since being an independent project, one would think maximizing individual productivity would be the only hope of yielding anything novel and useful in a field like search that isn't a fitting of an off-the-shelf to a business problem. (Something I've done before, it was quite neat.) Go is not a language focused around individual productivity in any capacity or scope. It's the language of pyramid-building Egyptian Pharaohs. (Throw 1,000 lives at it, it'll get 'er done...) I say this as someone rather envious of any language that has a native compiler that builds static binaries. (Oh, OCaml...) As we ponder on what orangetide is up to, lets keep in mind he's not explained or revealed anything at all about Nuuton and for all we know - could be having us on for the sake of self-promotion/promoting Go. The incorrect speculation about the origins and purpose of Go make this even stranger. Go exists to make people like Rob Pike happy. The Plan9 and Suckless community of programmers. Those types. That it happened to solve the problems of some Google programmers who were making it was almost incidental.

capisce 13y ago
> Go is not a language focused around individual productivity in any capacity or scope. Seems to have been my impression as well. Lisp with its macros can make an individual programmer who doesn't need to worry about coding conventions incredibly productive. I see Rust has a semi-powerful macro system too although not to the extent of Lisp, as it doesn't look like you can evaluate arbitrary code at compile time.
- pjmlp 13y ago
  I was attracted to Go at the beginning due to Limbo and Oberon influences that the language has. Even did some coding in Go for a few minor projects. But that experience showed me that it actually felt like a time travel back to 1992, when I was using Native Oberon. Sure the language feels like a better C and I find it good that it has native code compilers as the default implementation. But since 1992 we have gained lots of abstractions in mainstream languages that it is pity to throw them out of the window.

danieldk 13y ago

There is no real substance in this blog post. At the end: I chose Go because it is being developed to solve the same problems I have to solve. All by a group of scientists an programmers that are each twice as smart as I have ever been (on my best day). So, what are the problems you are trying to solve? And why can Go solve them better than, Java, Erlang, C++, Scala, or any other language? If you could answer that question, you'd have an interesting blog post. And if the answer is 'concurrency via Goroutines and channels' (it always is), explain why they are better than Erlang or Akka's actors, STM in Clojure or Haskell, or other proposals, and you'd have a stellar blog post. But really, this at the top of Hacker News?

pjmlp 13y ago
Go effect. Do you remember when Rails and Node were every day on HN? Now it is time for Go.
- orangethirty 13y ago
  I actually thought about using those. Rails is a memory hog, and Node is Javascript ( a language I am not fond of).
- nazka 13y ago
  Now it is time to Go.
- gabipurcaru 13y ago
  Maybe; but that's the way towards progress
  - pjmlp 13y ago
    We have a different opinion what progress means, it seems.
- reinhardt 13y ago
  Every language has its 15 weeks of fame.

thomseddon 13y ago

Three rewrites and not yet out of (or even in) beta: http://nuuton.com/ http://nuuton.com/ If you are seriously hoping for this to be a full time business then I would strongly recommend you read "The Lean Startup" or similar. This blind technical pursuit of perfection is a pretty classic approach for us techies and something i'm guilty of myself. It was actually that process that made me realise what my strengths actually were and what I would need to do to start a world class company (work with great people). If this isn't really a commercial project then just ignore the above and happy hacking, perhaps you should consider open sourcing it?

orangethirty 13y ago
I'm always telling people here about how they should sell first, and code later. This project has been challenging in that regard, but I have been spending a lot of time figuring out the business aspect of search. It is challenging because one of my main goals is to respect privacy. These days, violating the privacy of the users is a sure way to cash in. But I don't want to do that. As a result, I've had to write a couple of versions to see how it would work out. My current plan incorporates a product line right into the stack. That way, if/when the time comes, I can focus on writing the necessary parts. No need to scramble and try to make it fit. I do follow the lean principle. But one cannot blindly adhere to them. Nuuton is not the sort of project where you just try and get something to work. There had to be a foundation first. But from now on, development will follow the build-measure-learn cycle. I am open sourcing parts of it. Mostly Go libraries that might result from developing it.

ErrantX 13y ago

I wanted it to be pure Python because I love Python. That is until I had to work with threads. After spending some time working with gevent, I decided to simply use another language to deal with that. If I could have a penny...

orangethirty 13y ago
May you expand on the comment.
- ErrantX 13y ago
  It's a common complaint about Python, that's all :)
  - orangethirty 13y ago
    Yes, I know. Had to work on a commercial project that used gevent. Boy was that a mess. The project was a server that had to handle 30K+ hits/second. The client's engineering dept could only get it to do 3K. When I saw they were using gevent (along with a custom framework that lacked any documentation), I knew their goal had a long way to go. Last thing I heard, they were having trouble hiring people to work with it. I'm glad I dodged that bullet.

sramsay 13y ago

And the award for best typo ever goes to: "[Lisp] had been my sweetheart for a log time."

knodi 13y ago

Do the personal whys really matter? Its a good young lang that has a lot of promise.

hal84 13y ago

So I guess if I posted about how I chose C++ and Perl to write Enginuity Search Engine I'd get no love on HN because its boring, stable, scales and just works.

obviouslygreen 13y ago
Maybe not. Or maybe you would, if your reasoning were well-presented and reflected a logical approach and conclusion, and if it were insightful in a way that people found interesting and useful.
- hal84 13y ago
  Well I might have a go at it soon in our engineering section of our company blog. Have some interesting network performance analysis info to share as well.