17 ms·

Using D to Create the World’s Fastest File System

250 points by aldacron 8y ago

dvfjsdhgfv 8y ago
I discovered D is a secret weapon at several companies, but not all of them want to make it widely known...
- deleted 8y ago
  [deleted]
- thwy12321 8y ago
  Is this true? Mind speaking on which types of projects it was mostly used?
  - RobLach 8y ago
    https://medium.com/@NetflixTechBlog/introducing-vectorflow-fe10d7f126b8 https://medium.com/@NetflixTechBlog/introducing-vectorflow-f...
  - yvdriess 8y ago
    Remedy (game developer) uses it in-house, mostly for their animation and AI engines.
  - arunc 8y ago
    Congrats for Weka.IO. They managed to pull it through after the CEO visited the DConf and asked for help. D is a brilliant language. I got drawn into D by this article[1]. I've used D to replace some of the test tools where Java/Python were used. GC is a friend in such cases. I could've written them in C++ as well, but Phobos wins over STL hands down! Initially the "solution" test application for one of our REST services was written in Java. It covered the functional, performance and longevity tests. The REST service itself was written in C++. Ultimately the customers reported that under heavy load the response time of the REST service is too long (> 400 ms) and we fixed the code as well. But during tests we were unable to really load the application to the fullest. 48 cores & 48 threads in Java and we still couldn't overload the REST service. We had no way to prove that our fix actually worked. Ultimately, we rewrote the test app in D and boom, we were finally able to achieve the load that customers created in their production env - both in terms of response times and throughput. I could've written it in C++. But D's std.parallelism + template mixin is a boon, thanks to David Simcha. Based on our tests, the D test app was 20 times faster than the Java equivalent (written by one of the finest and the expert). Sure, there is a lot of visual noise due to @nogc, etc, etc. But beneath that there is a beautiful and readable language that's waiting to unfold, if and only if the D authors realize that they stop the feature creep and revert to sane defaults. On the other hand, I started to convert the REST service from C++ to D last year and it didn't go well. Blame me! There are _very_ rough edges in D as well that are result of over engineering. One such thing is the const'ness in D. It make it impossible to use const, ever, by library authors. I don't care to go and fight in D forum. Manu has already screamed many a times. Lately I found that one of the reasons for poor performance of my D port was also because of auto-decoding! Duh! It was not even GC. (The default should be fast, isn't it?). I spent a considerable amount of time for this port and eventually it failed to meet the performance vs maintainability trade-off. C++ is getting better year by year, at least every 5 years :-). std::vector, std::map, etc are safe for concurrent reads (at least on POSIX). std::vector has erase with iterator. Phobo's std.array doesn't have an equivalent features, albeit it is much more readable than STL. FWIW, D doesn't deserve the negative publicity that it usually gets. It's a fantastic language. Try it and if you like it, enjoy using it for the right use cases. After all choose the right tool for the job at hand. [1] https://wiki.dlang.org/Component_programming_with_ranges https://wiki.dlang.org/Component_programming_with_ranges
    - WalterBright 8y ago
      The difference between const in D and const in C++ is that D's const is transitive, and C++'s is not. Transitive const is necessary for things like pure functions. It's more difficult to use because the compiler really, really means it. But when it's there, you (the user) knows that function is not modifying the const data structure, not no-how, not no-way.
    - nikbackm 8y ago
      Could you elaborate a little on why D libraries cannot use const?
  - p0nce 8y ago
    No it's not true!
  - dvfjsdhgfv 8y ago
    I'm sorry, these are very specific projects. A lot of creative work on the server side.
- yvdriess 8y ago
  Several languages seem to share this curse. You do see companies advertise it on language conferences or websites: Jane Street for OCaml, Remedy Entertainment for D-lang, Naughty Dog and ITA for Common Lisp. But you are often left wondering if it is the tip if the iceberg, or an accurate reflection of industry use. In the case of CL, you see multiple commercial implementations still being supported, indicating much more use than one would suspect from grepping github projects. A pattern I seem to observe is that if the program/framework/platform grows large and successful enough, the company gets acquired for its tech and everything gets rewritten in some more 'industry standard' language. edit: As pointed out, Jane Street advertises its OCaml use, not Haskell. Oops!
  - phonebucket 8y ago
    I’m nitpicking a bit, but Jane Street focuses on OCaml.
  - elemeno 8y ago
    Just a nitpick, but it's OCaml for Jane Street rather than Haskell. Haskell is certainly used as a bit of a secret weapon in some of the Quant groups at banks like Barclays Capital though!
  - vram22 8y ago
    >ITA for Common Lisp. >A pattern I seem to observe is that if the program/framework/platform grows large and successful enough, the company gets acquired for its tech and everything gets rewritten in some more 'industry standard' language. I wonder if that happened for ITA's software after ITA was acquired by Google (for some hundreds of millions, IIRC). I read somewhere that Paul Graham's ViaWeb, which was written in mainly Lisp (or at least some key parts were), was later rewritten in some other language, maybe C++, like the pattern you observe.
    - lispm 8y ago
      Probably it's not that easy to replace Lisp for something else at Google, since it implements a relatively complex search engine core. Thus it's still used. I guess they WOULD be happy to replace it, since Lisp is otherwise not used much at Google, AFAIK. ITA/Google also developed another complex application using Lisp (a reservation system for airlines, initially for Air Canada), but that wasn't successful in the market. At Viaweb Lisp was used for an early web store maker (enables people to have their own webshop) implementation. > company gets acquired for its tech Actually companies get bought for a business model, customers and some amount of tech. Reimplementing earlier approaches, even multiple times, even multiple times with different technologies, seems to be normal in the web business.
- vram22 8y ago
  >D is a secret weapon at several companies That used to be said about Python back in the day, before it became as widely known and used as it is now.
  - dvfjsdhgfv 8y ago
    The difference being with Python you gained in prototyping speed but were losing (at least at the time) on execution speed, whereas in D you get rapid prototyping while keeping very high efficiency.
    - vram22 8y ago
      Good point. But that is partly due to the design of Python, it being a highly dynamic language. I don't want to get into any flame wars, but I'd hazard a guess that Python is somewhat more dynamic in nature than D (I know some D, and Python well), and that at the least, some of the things one can do in Python, would require a lot more knowledge and skill (with D) and acrobatics or contortions (speaking loosely) to do in D. (It would be good to be proven wrong on this, though :) This is not a criticism of D, I really like the language, and am aware of its power. Check out these two posts about D by Dmitry Popov (lead developer and Director), on the Infognition blog: http://www.infognition.com/blog/2014/why_d.html http://www.infognition.com/blog/2014/why_d.html http://www.infognition.com/blog/2014/d_as_scripting_language.html http://www.infognition.com/blog/2014/d_as_scripting_language... (2nd post not loading for me right now) In fact, here are some posts I've written that show how to write some kinds of basic apps for various purposes in D, that I posted about on HN just recently: https://news.ycombinator.com/item?id=18609543 https://news.ycombinator.com/item?id=18609543
      - vram22 8y ago
        Forgot to put the titles for the 2 Infognition posts linked above, here they are again, with titles now: Why D? http://www.infognition.com/blog/2014/why_d.html http://www.infognition.com/blog/2014/why_d.html D as a scripting language: http://www.infognition.com/blog/2014/d_as_scripting_language.html http://www.infognition.com/blog/2014/d_as_scripting_language...
- 2bitencryption 8y ago
  why is it in a company's best interest to not evangelize the publicly-available tools it uses? Microsoft/Amazon/Google are paying $$$ to increase availability of AI/machine learning education, because they want a knowledge pool to hire from. Wouldn't this be the same with languages?
  - WalterBright 8y ago
    Two have told me they did so because D enabled them to write faster code than their competitors, and they were in a line of business where faster code meant making more money. One reason D code is faster is array slicing, where you can refer to a subsection of an array. This pays off for things like strings. Consider the string: s = "/root/me/file.ext"; I want to extract the "file" as a string. In C/C++, because strings are 0-terminated, I have a cache hit, allocate 5 bytes and copy 5 characters and install a 0. In D, name = s[9 .. 13]; I have no cache hit, add 9 to a pointer, copy 5 to the length. (A slice is equivalent to a (length, pointer) pair.)
  - forgottenpass 8y ago
    >why is it in a company's best interest to not evangelize the publicly-available tools it uses? Because they want people that are good with the tool, not people that crash-coursed/bootcamped their way into what they are told is a marketable skill.
iddan 8y ago
Sounds to me like they would be better off with Rust as they are actively avoiding GC. How are the two languages compare in this problem space?
- StreamBright 8y ago
  You can avoid GC with a non-GC standard library that they actually wrote: https://github.com/weka-io/mecca https://github.com/weka-io/mecca
  - ileri62 8y ago
    Here's the thing though: They wrote a new compiler and a new standard library to make the language work for them. How exactly is that good publicity for D? D itself looked and still looks very interesting to me, as it can be very performant, without having insanely ugly syntax such as c++ or rust (inb4 rUsT iS VeRy ReADaBlE).
    - WalterBright 8y ago
      If D didn't have compelling advantages, they wouldn't have invested such effort. Also, when one is really after every erg of performance, one often finds a need to work under the hood to improve it for one's special use cases. For example, many, many, MANY C and C++ users replace malloc/free, and/or write their own custom allocator.
    - vthriller 8y ago
      > D itself looked and still looks very interesting to me, as it can be […] without having insanely ugly syntax such as c++ or rust (inb4 rUsT iS VeRy ReADaBlE). I did a quick dive into aforementioned stdlib only to stumble upon something I would rather prefer any Rust/C++ version over: https://github.com/weka-io/mecca/blob/f5dc6d9f71983ea7b852ad85a99aca2496674f98/src/mecca/lib/exception.d#L284-L293 https://github.com/weka-io/mecca/blob/f5dc6d9f71983ea7b852ad... Some might want to exempt standard library from the "readable code" argument, but I'd argue that readability argument was lost the moment language designers allowed anyone to write things like `whatisthis!"whatever"({t.__ctor(args);})` (regardless of best practices that regular devs might unanimously follow). You can't just say that "A is bad 'cause it allows unreadable code" and follow that with "nobody's doing bad things in B therefore it's much better".
guitarbill 8y ago
> Our previous company, XIV (that IBM acquired in 2007) used a private version of C that had polymorphism, generic programming, and integrated RPC, for code that was a mixture of XML, C, and weird header files. Super interesting. XIV was a great system, but "weird" is a good description for some of the source (IBM of course neglected/ruined it). Thought most of the XIV folk went to Infinidat. In the storage world, C is still king, so nice to see a challenger. But some of the hurdles he describes in the article, ouch. No wonder people stick with un-sexy languages and toolchains.
hardwaresofton 8y ago
I would have liked to hear more about what exactly made D so great -- reading through it it's kind of vague. He mentions generics and finally getting LLVM support working as a target, and being able to use D at a high and low-level but I don't see anything here that's groundbreaking. Could someone who uses D Hate to be that guy, but if you could compare it with rust that would be double A++ good... I know the targets are somewhat different but I find that I can write high level rust and low level rust. Also, I get the feeling that you could very easily have used Go for the same purposes (D was around long before Go IIRC so maybe it was just not being present).
- sargun 8y ago
  One of the nice aspects of D is that it interfaces with C++ really nicely. I'm not overly familiar with how Rust handles this, as the last time I tried to glue together Rust and C++ was more than a year ago, and the tooling was in evolution. From Go / CGo, it can be pretty scary to interface with native C++ code, and it requires lots of annoying boilerplate.
  - WalterBright 8y ago
    "C++ has a very complex ABI, and the Rust ABI is not frozen. However, both C++ and Rust support functions that use the C ABI. Therefore, interoperability between C++ and Rust involves writing things in such a way that C++ sees Rust code as C code and Rust sees C++ code as C code." https://hsivonen.fi/modern-cpp-in-rust/ https://hsivonen.fi/modern-cpp-in-rust/
    - hardwaresofton 8y ago
      Context: this poster is likely Walter Bright the creator and first implementer of D lang itself. Cool to know you also read that article about C++ in rust (I personally read a little bit then left because trying to interface with C++ from rust wasn't a problem I had), but I guess it's obvious that you would.
      - 8y ago
jpz 8y ago
Ah, just looking at this. When I was looking at D back around 2005 it was a closed shop run by one guy - I had the feeling it did not take off because it simply wasn't an open enough project. It is a shame, as since then we've had go, Swift, Rust all come along and it is a lot more competitive for mindshare in that space... not to forget the merits of modern C++ (it is a far better language post 2011)
- fsloth 8y ago
  "it was a closed shop run by one guy" You do realize Walter Bright is a frequent HN resident :) ?
  - jpz 8y ago
    I don't criticise him, I just think the structure of the project meant it failed to really launch. There are people like me (20y+ C++) that when certainly I think of D, I think of it being a closed project. I spent quite a bit of time looking at it and thinking how great it was, but it seemed confusingly proprietary (which is not a great place for a programming language to be.)
jonathanstrange 8y ago
Perhaps an unfortunate name, though. For some reason my brain involuntarily auto-corrects "WekaIO" into "WeakIO". I think advertiser sometimes even play with this auto-correct feature of the brain to make suggestions without stating them.
acd 8y ago
Weka is a name of a Machine learning product from New Zealand. The IBM system had 24xHGST N200 SSD, 830 000 random read, 200 000 random write. that is 24 * 200 000 IOPS=4.8Million random writes. The Matrix system had 64x1.2TB Micron 9100 SSDs. 750 000 random read, 300 0000 random writes. 64 * 300 000=19.2 Million IOPS Ok so you have benchmarked hardware with 4.8 Million write ops vs 19.2 Million write ops.
- billforsternz 8y ago
  I clicked on the story in the hope that there was a New Zealand connection, and was disappointed that it seemed to be an Israeli operation.
fb03 8y ago
Random Fact: I can't help but smile when I read that D's "STL" is called Phobos because of my Doom 2 nostalgia :) Have a nice day, everyone o/
- jordigh 8y ago
  Well, Walter's company is Digital Mars so that's where he gets his naming scheme from. Nothing to do with Doom except they both are using the same Greco-Roman deity astronomical naming scheme.
  - skocznymroczny 8y ago
    Originally D was supposed to be named Mars. The stdlib is named Phobos and there is a project called Deimos which is about providing bindings to C libraries.
- vram22 8y ago
  Apropos of D and Phobos, two posts from my blog: Simple parallel processing in D with std.parallelism (it really did speed things up, ~3x (CPU, not I/O), by my unscientific and small test at least): https://jugad2.blogspot.com/2016/12/simple-parallel-processing-in-d-with.html https://jugad2.blogspot.com/2016/12/simple-parallel-processi... D language front-end merged with GCC 9; some sample D programs: https://jugad2.blogspot.com/2018/10/d-language-front-end-merged-with-gcc-9.html https://jugad2.blogspot.com/2018/10/d-language-front-end-mer... Excerpt from the link above, that says what the post is about: "Here are a handful of D programs that use various features of the language and some of its libraries, on my blog. Most of them are simple command-line utilities to do various things. Readers may find them of use to get a flavor of the language and to know some of the kinds of things it can be used for."
- WalterBright 8y ago
  D's runtime is called Phobos because the language was originally named Mars.
  - eco 8y ago
notacoward 8y ago
For those commenting on the name: weka is not only the name of a storage company and some machine-learning software, but originally a flightless bird native to Aotearoa / New Zealand. http://nzbirdsonline.org.nz/species/weka http://nzbirdsonline.org.nz/species/weka Not to be confused with weta, which is a large scary looking (but actually pretty harmless) insect in the grasshopper family and also the name of another company that did most of the digital effects for the LotR movies.
- regularfry 8y ago
  Weta Digital (https://www.wetafx.co.nz/ https://www.wetafx.co.nz/) are named after the grasshopper, not the bird.
shiftoutbox 8y ago
So worlds fastest is a big claim. - How is it in comparison to a tmpfs on ram ? - How is it in in comparison to ufs/ffs or ext2 - Is it the fastest with some feature ? - Is it portable ? - Is it clusterable , on disk or over the network ? - What is it's target operating system ? - What is it's target market , SAN, NAS, Local Storage, etc ?
tombert 8y ago
I have wondered for awhile now why Rust is getting all this attention of being "Like C but safe and cool!", but no one would mention D. I personally don't do systems programming because I'm not smart enough but the little bit I have done, I found D to be the most-natural feeling (even over Rust), and D has been around a lot longer. I suppose timing is everything for these kinds of things.
- nwah1 8y ago
  D had a closed source reference compiler for a long time, and didn't have any major backers. Rust had Mozilla backing early, and was fully open and transparent in terms of community feedback, and wasn't just making a language that had high level features, but was pioneering a new category of language that added tons of safety guarantees at compile time by default using a new paradigm of enforcing borrowing and ownership. These safety guarantees are not just in regards to memory safety, but also in regards to data races. D seems to allow memory safety guarantees, but not by default, and I believe it is enforced at runtime not compiletime. So Rust was at once more accessible and more exciting.
  - tombert 8y ago
    That's fair, I suspected that people who knew more about this stuff had a reason.
  - jordigh 8y ago
    D's compiler was never "closed source" if by that you mean "source not visible". It had a weird non-open source license for a long time, but the source has always been visible. The weird license did use to scare me away, but that's very much a thing of the past now and now it has a standard free license. https://news.ycombinator.com/item?id=14060846 https://news.ycombinator.com/item?id=14060846 I don't think D's features are any less exciting than Rust's, but Rust has had a lot of RESF backed by Mozilla.
    - ryan_lane 8y ago
      Source code being visible doesn't make it open source. It had a personal-use-only license, which isn't a valid OSSI license (which is what makes something open source).
z3phyr 8y ago
D standard library is so well documented. Its criminally underrated.
lumberjackstian 8y ago
It seems this article is not accurate anymore? (the reference is from March). It's not far off though: https://www.theregister.co.uk/2018/11/30/wekaio/ https://www.theregister.co.uk/2018/11/30/wekaio/ https://www.hpcwire.com/off-the-wire/wekaio-places-in-top-five-of-the-virtual-institutes-io-500-10-node-challenge/ https://www.hpcwire.com/off-the-wire/wekaio-places-in-top-fi...

250 points by aldacron 8y ago

dvfjsdhgfv 8y ago

I discovered D is a secret weapon at several companies, but not all of them want to make it widely known...

deleted 8y ago
[deleted]
thwy12321 8y ago
Is this true? Mind speaking on which types of projects it was mostly used?
- RobLach 8y ago
  https://medium.com/@NetflixTechBlog/introducing-vectorflow-fe10d7f126b8 https://medium.com/@NetflixTechBlog/introducing-vectorflow-f...
- yvdriess 8y ago
  Remedy (game developer) uses it in-house, mostly for their animation and AI engines.
- arunc 8y ago
  Congrats for Weka.IO. They managed to pull it through after the CEO visited the DConf and asked for help. D is a brilliant language. I got drawn into D by this article[1]. I've used D to replace some of the test tools where Java/Python were used. GC is a friend in such cases. I could've written them in C++ as well, but Phobos wins over STL hands down! Initially the "solution" test application for one of our REST services was written in Java. It covered the functional, performance and longevity tests. The REST service itself was written in C++. Ultimately the customers reported that under heavy load the response time of the REST service is too long (> 400 ms) and we fixed the code as well. But during tests we were unable to really load the application to the fullest. 48 cores & 48 threads in Java and we still couldn't overload the REST service. We had no way to prove that our fix actually worked. Ultimately, we rewrote the test app in D and boom, we were finally able to achieve the load that customers created in their production env - both in terms of response times and throughput. I could've written it in C++. But D's std.parallelism + template mixin is a boon, thanks to David Simcha. Based on our tests, the D test app was 20 times faster than the Java equivalent (written by one of the finest and the expert). Sure, there is a lot of visual noise due to @nogc, etc, etc. But beneath that there is a beautiful and readable language that's waiting to unfold, if and only if the D authors realize that they stop the feature creep and revert to sane defaults. On the other hand, I started to convert the REST service from C++ to D last year and it didn't go well. Blame me! There are _very_ rough edges in D as well that are result of over engineering. One such thing is the const'ness in D. It make it impossible to use const, ever, by library authors. I don't care to go and fight in D forum. Manu has already screamed many a times. Lately I found that one of the reasons for poor performance of my D port was also because of auto-decoding! Duh! It was not even GC. (The default should be fast, isn't it?). I spent a considerable amount of time for this port and eventually it failed to meet the performance vs maintainability trade-off. C++ is getting better year by year, at least every 5 years :-). std::vector, std::map, etc are safe for concurrent reads (at least on POSIX). std::vector has erase with iterator. Phobo's std.array doesn't have an equivalent features, albeit it is much more readable than STL. FWIW, D doesn't deserve the negative publicity that it usually gets. It's a fantastic language. Try it and if you like it, enjoy using it for the right use cases. After all choose the right tool for the job at hand. [1] https://wiki.dlang.org/Component_programming_with_ranges https://wiki.dlang.org/Component_programming_with_ranges
  - WalterBright 8y ago
    The difference between const in D and const in C++ is that D's const is transitive, and C++'s is not. Transitive const is necessary for things like pure functions. It's more difficult to use because the compiler really, really means it. But when it's there, you (the user) knows that function is not modifying the const data structure, not no-how, not no-way.
  - nikbackm 8y ago
    Could you elaborate a little on why D libraries cannot use const?
- p0nce 8y ago
  No it's not true!
- dvfjsdhgfv 8y ago
  I'm sorry, these are very specific projects. A lot of creative work on the server side.
yvdriess 8y ago
Several languages seem to share this curse. You do see companies advertise it on language conferences or websites: Jane Street for OCaml, Remedy Entertainment for D-lang, Naughty Dog and ITA for Common Lisp. But you are often left wondering if it is the tip if the iceberg, or an accurate reflection of industry use. In the case of CL, you see multiple commercial implementations still being supported, indicating much more use than one would suspect from grepping github projects. A pattern I seem to observe is that if the program/framework/platform grows large and successful enough, the company gets acquired for its tech and everything gets rewritten in some more 'industry standard' language. edit: As pointed out, Jane Street advertises its OCaml use, not Haskell. Oops!
- phonebucket 8y ago
  I’m nitpicking a bit, but Jane Street focuses on OCaml.
- elemeno 8y ago
  Just a nitpick, but it's OCaml for Jane Street rather than Haskell. Haskell is certainly used as a bit of a secret weapon in some of the Quant groups at banks like Barclays Capital though!
- vram22 8y ago
  >ITA for Common Lisp. >A pattern I seem to observe is that if the program/framework/platform grows large and successful enough, the company gets acquired for its tech and everything gets rewritten in some more 'industry standard' language. I wonder if that happened for ITA's software after ITA was acquired by Google (for some hundreds of millions, IIRC). I read somewhere that Paul Graham's ViaWeb, which was written in mainly Lisp (or at least some key parts were), was later rewritten in some other language, maybe C++, like the pattern you observe.
  - lispm 8y ago
    Probably it's not that easy to replace Lisp for something else at Google, since it implements a relatively complex search engine core. Thus it's still used. I guess they WOULD be happy to replace it, since Lisp is otherwise not used much at Google, AFAIK. ITA/Google also developed another complex application using Lisp (a reservation system for airlines, initially for Air Canada), but that wasn't successful in the market. At Viaweb Lisp was used for an early web store maker (enables people to have their own webshop) implementation. > company gets acquired for its tech Actually companies get bought for a business model, customers and some amount of tech. Reimplementing earlier approaches, even multiple times, even multiple times with different technologies, seems to be normal in the web business.
vram22 8y ago
>D is a secret weapon at several companies That used to be said about Python back in the day, before it became as widely known and used as it is now.
- dvfjsdhgfv 8y ago
  The difference being with Python you gained in prototyping speed but were losing (at least at the time) on execution speed, whereas in D you get rapid prototyping while keeping very high efficiency.
  - vram22 8y ago
    Good point. But that is partly due to the design of Python, it being a highly dynamic language. I don't want to get into any flame wars, but I'd hazard a guess that Python is somewhat more dynamic in nature than D (I know some D, and Python well), and that at the least, some of the things one can do in Python, would require a lot more knowledge and skill (with D) and acrobatics or contortions (speaking loosely) to do in D. (It would be good to be proven wrong on this, though :) This is not a criticism of D, I really like the language, and am aware of its power. Check out these two posts about D by Dmitry Popov (lead developer and Director), on the Infognition blog: http://www.infognition.com/blog/2014/why_d.html http://www.infognition.com/blog/2014/why_d.html http://www.infognition.com/blog/2014/d_as_scripting_language.html http://www.infognition.com/blog/2014/d_as_scripting_language... (2nd post not loading for me right now) In fact, here are some posts I've written that show how to write some kinds of basic apps for various purposes in D, that I posted about on HN just recently: https://news.ycombinator.com/item?id=18609543 https://news.ycombinator.com/item?id=18609543
    - vram22 8y ago
      Forgot to put the titles for the 2 Infognition posts linked above, here they are again, with titles now: Why D? http://www.infognition.com/blog/2014/why_d.html http://www.infognition.com/blog/2014/why_d.html D as a scripting language: http://www.infognition.com/blog/2014/d_as_scripting_language.html http://www.infognition.com/blog/2014/d_as_scripting_language...
2bitencryption 8y ago
why is it in a company's best interest to not evangelize the publicly-available tools it uses? Microsoft/Amazon/Google are paying $$$ to increase availability of AI/machine learning education, because they want a knowledge pool to hire from. Wouldn't this be the same with languages?
- WalterBright 8y ago
  Two have told me they did so because D enabled them to write faster code than their competitors, and they were in a line of business where faster code meant making more money. One reason D code is faster is array slicing, where you can refer to a subsection of an array. This pays off for things like strings. Consider the string: s = "/root/me/file.ext"; I want to extract the "file" as a string. In C/C++, because strings are 0-terminated, I have a cache hit, allocate 5 bytes and copy 5 characters and install a 0. In D, name = s[9 .. 13]; I have no cache hit, add 9 to a pointer, copy 5 to the length. (A slice is equivalent to a (length, pointer) pair.)
- forgottenpass 8y ago
  >why is it in a company's best interest to not evangelize the publicly-available tools it uses? Because they want people that are good with the tool, not people that crash-coursed/bootcamped their way into what they are told is a marketable skill.

iddan 8y ago

Sounds to me like they would be better off with Rust as they are actively avoiding GC. How are the two languages compare in this problem space?

StreamBright 8y ago
You can avoid GC with a non-GC standard library that they actually wrote: https://github.com/weka-io/mecca https://github.com/weka-io/mecca
- ileri62 8y ago
  Here's the thing though: They wrote a new compiler and a new standard library to make the language work for them. How exactly is that good publicity for D? D itself looked and still looks very interesting to me, as it can be very performant, without having insanely ugly syntax such as c++ or rust (inb4 rUsT iS VeRy ReADaBlE).
  - WalterBright 8y ago
    If D didn't have compelling advantages, they wouldn't have invested such effort. Also, when one is really after every erg of performance, one often finds a need to work under the hood to improve it for one's special use cases. For example, many, many, MANY C and C++ users replace malloc/free, and/or write their own custom allocator.
  - vthriller 8y ago
    > D itself looked and still looks very interesting to me, as it can be […] without having insanely ugly syntax such as c++ or rust (inb4 rUsT iS VeRy ReADaBlE). I did a quick dive into aforementioned stdlib only to stumble upon something I would rather prefer any Rust/C++ version over: https://github.com/weka-io/mecca/blob/f5dc6d9f71983ea7b852ad85a99aca2496674f98/src/mecca/lib/exception.d#L284-L293 https://github.com/weka-io/mecca/blob/f5dc6d9f71983ea7b852ad... Some might want to exempt standard library from the "readable code" argument, but I'd argue that readability argument was lost the moment language designers allowed anyone to write things like `whatisthis!"whatever"({t.__ctor(args);})` (regardless of best practices that regular devs might unanimously follow). You can't just say that "A is bad 'cause it allows unreadable code" and follow that with "nobody's doing bad things in B therefore it's much better".

guitarbill 8y ago

> Our previous company, XIV (that IBM acquired in 2007) used a private version of C that had polymorphism, generic programming, and integrated RPC, for code that was a mixture of XML, C, and weird header files. Super interesting. XIV was a great system, but "weird" is a good description for some of the source (IBM of course neglected/ruined it). Thought most of the XIV folk went to Infinidat. In the storage world, C is still king, so nice to see a challenger. But some of the hurdles he describes in the article, ouch. No wonder people stick with un-sexy languages and toolchains.

hardwaresofton 8y ago

I would have liked to hear more about what exactly made D so great -- reading through it it's kind of vague. He mentions generics and finally getting LLVM support working as a target, and being able to use D at a high and low-level but I don't see anything here that's groundbreaking. Could someone who uses D Hate to be that guy, but if you could compare it with rust that would be double A++ good... I know the targets are somewhat different but I find that I can write high level rust and low level rust. Also, I get the feeling that you could very easily have used Go for the same purposes (D was around long before Go IIRC so maybe it was just not being present).

sargun 8y ago
One of the nice aspects of D is that it interfaces with C++ really nicely. I'm not overly familiar with how Rust handles this, as the last time I tried to glue together Rust and C++ was more than a year ago, and the tooling was in evolution. From Go / CGo, it can be pretty scary to interface with native C++ code, and it requires lots of annoying boilerplate.
- WalterBright 8y ago
  "C++ has a very complex ABI, and the Rust ABI is not frozen. However, both C++ and Rust support functions that use the C ABI. Therefore, interoperability between C++ and Rust involves writing things in such a way that C++ sees Rust code as C code and Rust sees C++ code as C code." https://hsivonen.fi/modern-cpp-in-rust/ https://hsivonen.fi/modern-cpp-in-rust/
  - hardwaresofton 8y ago
    Context: this poster is likely Walter Bright the creator and first implementer of D lang itself. Cool to know you also read that article about C++ in rust (I personally read a little bit then left because trying to interface with C++ from rust wasn't a problem I had), but I guess it's obvious that you would.
    - 8y ago

jpz 8y ago

Ah, just looking at this. When I was looking at D back around 2005 it was a closed shop run by one guy - I had the feeling it did not take off because it simply wasn't an open enough project. It is a shame, as since then we've had go, Swift, Rust all come along and it is a lot more competitive for mindshare in that space... not to forget the merits of modern C++ (it is a far better language post 2011)

fsloth 8y ago
"it was a closed shop run by one guy" You do realize Walter Bright is a frequent HN resident :) ?
- jpz 8y ago
  I don't criticise him, I just think the structure of the project meant it failed to really launch. There are people like me (20y+ C++) that when certainly I think of D, I think of it being a closed project. I spent quite a bit of time looking at it and thinking how great it was, but it seemed confusingly proprietary (which is not a great place for a programming language to be.)

jonathanstrange 8y ago

Perhaps an unfortunate name, though. For some reason my brain involuntarily auto-corrects "WekaIO" into "WeakIO". I think advertiser sometimes even play with this auto-correct feature of the brain to make suggestions without stating them.

acd 8y ago

Weka is a name of a Machine learning product from New Zealand. The IBM system had 24xHGST N200 SSD, 830 000 random read, 200 000 random write. that is 24 * 200 000 IOPS=4.8Million random writes. The Matrix system had 64x1.2TB Micron 9100 SSDs. 750 000 random read, 300 0000 random writes. 64 * 300 000=19.2 Million IOPS Ok so you have benchmarked hardware with 4.8 Million write ops vs 19.2 Million write ops.

billforsternz 8y ago
I clicked on the story in the hope that there was a New Zealand connection, and was disappointed that it seemed to be an Israeli operation.

fb03 8y ago

Random Fact: I can't help but smile when I read that D's "STL" is called Phobos because of my Doom 2 nostalgia :) Have a nice day, everyone o/

jordigh 8y ago
Well, Walter's company is Digital Mars so that's where he gets his naming scheme from. Nothing to do with Doom except they both are using the same Greco-Roman deity astronomical naming scheme.
- skocznymroczny 8y ago
  Originally D was supposed to be named Mars. The stdlib is named Phobos and there is a project called Deimos which is about providing bindings to C libraries.
vram22 8y ago
Apropos of D and Phobos, two posts from my blog: Simple parallel processing in D with std.parallelism (it really did speed things up, ~3x (CPU, not I/O), by my unscientific and small test at least): https://jugad2.blogspot.com/2016/12/simple-parallel-processing-in-d-with.html https://jugad2.blogspot.com/2016/12/simple-parallel-processi... D language front-end merged with GCC 9; some sample D programs: https://jugad2.blogspot.com/2018/10/d-language-front-end-merged-with-gcc-9.html https://jugad2.blogspot.com/2018/10/d-language-front-end-mer... Excerpt from the link above, that says what the post is about: "Here are a handful of D programs that use various features of the language and some of its libraries, on my blog. Most of them are simple command-line utilities to do various things. Readers may find them of use to get a flavor of the language and to know some of the kinds of things it can be used for."
WalterBright 8y ago
D's runtime is called Phobos because the language was originally named Mars.
- eco 8y ago

notacoward 8y ago

For those commenting on the name: weka is not only the name of a storage company and some machine-learning software, but originally a flightless bird native to Aotearoa / New Zealand. http://nzbirdsonline.org.nz/species/weka http://nzbirdsonline.org.nz/species/weka Not to be confused with weta, which is a large scary looking (but actually pretty harmless) insect in the grasshopper family and also the name of another company that did most of the digital effects for the LotR movies.

regularfry 8y ago
Weta Digital (https://www.wetafx.co.nz/ https://www.wetafx.co.nz/) are named after the grasshopper, not the bird.

shiftoutbox 8y ago

So worlds fastest is a big claim. - How is it in comparison to a tmpfs on ram ? - How is it in in comparison to ufs/ffs or ext2 - Is it the fastest with some feature ? - Is it portable ? - Is it clusterable , on disk or over the network ? - What is it's target operating system ? - What is it's target market , SAN, NAS, Local Storage, etc ?

tombert 8y ago

I have wondered for awhile now why Rust is getting all this attention of being "Like C but safe and cool!", but no one would mention D. I personally don't do systems programming because I'm not smart enough but the little bit I have done, I found D to be the most-natural feeling (even over Rust), and D has been around a lot longer. I suppose timing is everything for these kinds of things.

nwah1 8y ago
D had a closed source reference compiler for a long time, and didn't have any major backers. Rust had Mozilla backing early, and was fully open and transparent in terms of community feedback, and wasn't just making a language that had high level features, but was pioneering a new category of language that added tons of safety guarantees at compile time by default using a new paradigm of enforcing borrowing and ownership. These safety guarantees are not just in regards to memory safety, but also in regards to data races. D seems to allow memory safety guarantees, but not by default, and I believe it is enforced at runtime not compiletime. So Rust was at once more accessible and more exciting.
- tombert 8y ago
  That's fair, I suspected that people who knew more about this stuff had a reason.
- jordigh 8y ago
  D's compiler was never "closed source" if by that you mean "source not visible". It had a weird non-open source license for a long time, but the source has always been visible. The weird license did use to scare me away, but that's very much a thing of the past now and now it has a standard free license. https://news.ycombinator.com/item?id=14060846 https://news.ycombinator.com/item?id=14060846 I don't think D's features are any less exciting than Rust's, but Rust has had a lot of RESF backed by Mozilla.
  - ryan_lane 8y ago
    Source code being visible doesn't make it open source. It had a personal-use-only license, which isn't a valid OSSI license (which is what makes something open source).

z3phyr 8y ago

D standard library is so well documented. Its criminally underrated.

lumberjackstian 8y ago

It seems this article is not accurate anymore? (the reference is from March). It's not far off though: https://www.theregister.co.uk/2018/11/30/wekaio/ https://www.theregister.co.uk/2018/11/30/wekaio/ https://www.hpcwire.com/off-the-wire/wekaio-places-in-top-five-of-the-virtual-institutes-io-500-10-node-challenge/ https://www.hpcwire.com/off-the-wire/wekaio-places-in-top-fi...