8 ms·
Are there plans to do user studies? 10 minutes watching new users code in Rust will give you better ideas than 10 weeks thinking about the problem in your head.
by modeless 10y ago
Are there plans to do user studies? 10 minutes watching new users code in Rust will give you better ideas than 10 weeks thinking about the problem in your head.
I feel like there's a real lack of user testing in software development tools land. If you're developing software for unsophisticated users it's obvious that you should be doing user testing, but it's an often ignored fact that developers are users too! APIs, compilers, build tools, they could all be vastly improved with some user testing.
- steveklabnik 10y agoWe have done informal ones and may or may not do more formal ones, we'll see.
- modeless 10y agoInformal is fine. The important thing is to do user testing in an ongoing way, not just once. Without a feedback loop it's quite surprising how quickly development priorities diverge from what's really important.
- steveklabnik 10y agoWe are always constantly trying to listen to people and get feedback; for example, it's one of the reasons I pay such close attention to these threads! There's always room for improvement, but iterating in this way is defintely a core value.
- IanCal 10y agoDepending on what group you're trying to attract or improve the experience with, you could potentially setup contracts with the level of person you would like to deal with and pay them to implement something in rust while you monitor their progress & with full access to their work and what they're doing.
- vertex-four 10y agoThe issue is, of course, that the more you focus on specific tasks the more your language becomes "designed for" those tasks, and the language becomes less general. The language already went through some of this with Servo - the DOM would be much easier to implement with an object-oriented language, but that wouldn't fit in well with the rest of Rust and would significantly raise complexity.
- IanCal 10y agoThat is a concern but the other risk is by not doing this the language progresses in areas people happen to currently use it, which will be skewed by what it's already good at. You can then end up never improving for those cases it's weakest. This covers types of programs but also experience levels and histories of the programmers themselves. You are right however that you need to carefully pick the tasks to fit with the original aims of the language. Perhaps a good way of phrasing this is "why aren't X people using rust for Y?".
- deleted 10y ago[deleted]
- kibwen 10y agoMembers of all the Rust subteams are constantly involved in guiding new users through the language. Come to any Boston Rust meetup and you'll find Niko Matsakis giving one-on-one help to people taking their first steps. And it's not just feedback from people in major tech hubs; Brian Anderson and Alex Crichton just got back from mentoring new users at Hack Illinois in Urbana. They go to great lengths to remain plugged-in like this.
- sametmax 10y agoGod yes. If anybody must remember one concept only about ergonomics, it's that.
- joseraul 10y agoThey may not do user studies like this, but for sure they adapt to user feedback. https://thefeedbackloop.xyz/stroustrups-rule-and-layering-over-time/ https://thefeedbackloop.xyz/stroustrups-rule-and-layering-ov...
- tatterdemalion 10y agoWhen we're trying to balance a trade off, I often wish we could just do A/B testing on it, but that's not realistic for languages.
- nostrademons 10y agoWhat you can do is run over a large corpus of programs (crates.io? GitHub?) to identify the places where the different alternatives would impact code, collect metrics on them, and pull some sample programs to investigate the impact on them of the different alternatives. Is anything like that being done? The HTML5 spec was developed in this way, and some of the code-health folks at Google were just starting to operate this way when I left, but I haven't heard of many other language-design efforts working like this. The technology & data exists now, with large public corpora of open-source code, parsing libraries that can identify patterns in it, and big-data tools to run these compiler tools over a large corpus. It's a bit of a paradigm shift from how languages are typically developed, though.
- mattyb 10y agoRuss Cox did this when figuring out how to seamlessly add monotonic elapsed time measurements to Go: https://github.com/golang/proposal/blob/master/design/12914-monotonic.md https://github.com/golang/proposal/blob/master/design/12914-...
- tatterdemalion 10y agoWe use crater to check potentially breaking changes against our entire package ecoystem, but its not as easy to test which of two syntaxes for a new feature will be easier to use by looking at existing code.
- zzzcpan 10y ago"If you're developing software for unsophisticated users it's obvious that you should be doing user testing" Compilers actually have a unique opportunity to introduce some form of opt-out tracking of all of the compilation errors with sources, etc. This could really help to understand the users.
- steveklabnik 10y agoWe have discussed doing this in the past, actually. Opt-in (oops, had "out" here initially) is very important here! I am not sure if it's something that ever landed or not...
- glangdale 10y agoMy comment crossed with the GP post here, but that's a really good idea. Please look into it!
- IanCal 10y agoOpt-in tracking, I really hope you mean. I know the concerns with getting unrepresentative samples but sending what I'm working on to a third party is something that would immediately stop me using an entire language for any form of my work.
- steveklabnik 10y agogah! Yes, I mean "we would never consider tracking you without your express given consent." That's opt in not opt out.
- IanCal 10y ago:) Thought that was the case, seemed odd that people were saying it like it was such a small deal!
- zzzcpan 10y agoIt doesn't have to be strictly opt-in or opt-out. There could be two versions released, a version without tracking could live on the same release page, just a bit lower and not in a bold font or something like that, pushing new users towards the version with opt-out tracking.
- glangdale 10y agoThis would be really great. I constantly see excuses for why no-one should bother doing user studies of programming languages because its really hard. So let's not do it at all and instead focus on approach #2 that you lay out ("10 weeks thinking about the problem in your head"). That being said, this is really hard. Observing a (necessarily small) sample of heterogeneous programmers dealing with (necessarily small) tasks isn't easy, either. I wonder whether users would consent to having their text editors and compilers instrumented to produce a user study of every single character typed and every single build (failed or successful, especially failed). I would imagine that a language designer could learn a huge amount from seeing all the things their compiler is rejecting from real users...
- dllthomas 10y ago> user studies of programming languages I feel like a lot of what I see is people want to establish that language X is much better than language Y, and I think that kind of study almost doesn't make sense with all of the relevant but poorly defined and difficult to control variables. A study around ergonomics of a few options in otherwise the same language is comparatively trivial.
- d33 10y agoYou wouldn't really need to have people install a new text editor. Just plugging this in to the Rust Playground would probably do [0]. [0]: https://play.rust-lang.org/ https://play.rust-lang.org/
- shepmaster 10y agoThis is an interesting idea! In the alternate playground [1], the frontend is React / Redux and so we already track every edit to the code as well as the build results. "All" that would be needed would to save that data somewhere and allow people to opt-in/out. [1]: http://play.integer32.com/ http://play.integer32.com/
- jplasmeier 10y agoI think that a quick survey of the user would be helpful as well- specifically their experience with programming, and their experience with the language in question.
- rattray 10y agoOn this note, I'm actively seeking JavaScript developers interested in doing some user studies of a compile-to-js language I'm working on. It emphasizes ergonomics, actually, as well as making functional best-practices (typed immutability, etc) more accessible. I'm in San Francisco but happy to conduct remotely. I'm targeting people familiar with ES6 and functional styles, but interested in folks of all experience levels. If anyone would be interested in trying it out, shoot me an email at rattray.alex – gmail.
- girvo 10y agoReplying to chase this up tomorrow when I'm not on my phone
- hackuser 10y ago> 10 minutes watching new users code in Rust will give you better ideas than 10 weeks thinking about the problem in your head. I'd add: Do you want to focus on new or experienced users? For example, when implementing systems that will be used 8 hrs/day by its users, we look only at efficiency for experienced users (unless the political situation requires placating the noobs). They will be noobs for a day or maybe a couple weeks; they'll be experienced all day, every day for years. We explain it that way at the beginning and they thank us later. An example many are familiar with is Vim: Steep learning curve, but I'm so happy that the focus is on efficiency for veteran users.
- nathancahill 10y agoWhat's the language that takes that to the extreme? I remember it being used for fintech(?) or spreadsheets and I remember seeing one-liners that look like someone just mashed the keyboard. Apparently incredibly efficient once you are an expert in the language.
- hyperpape 10y agoPerhaps J, which is part of the APL family of languages.
- mickronome 10y agoThat would probably be Q used in kdb+, and J, which the previous response stated is an open source implementation of the same/very similar language. The terseness even visible in the language name.
- throwaway7645 10y agoThe APL family of languages that currently have commercial implementations like Dyalog and Q w/ kdb+ database. J and GNU APL are both free & open source. Dyalog is nice though even though a commercial license sets you back a grand per year (basically what most C# users pay for VS). You get a nice interpreter, support, built-in graphics, full .NET interop, full R interop, heck even DDE which is still useful in finance and my own industry. They seem to take documentation really seriously and have pretty interesting conferences. I don't code in APL, but am considering it as their product just seems pretty agile for my needs (personal user productivity). Q is extremely expensive and is the APL derived language that accompanies the kdb+ database used for time-series analysis on stock quotes...etc. It was built by Arthur Whitney and legend has it the entire source code is 5 pages of C (being an APL guy he has it all scrunched up though...doesn't like to scroll). People seem to like it and be willing to pay a fortune.
- jeffdavis 10y agoWhat programming languages have been developed in this manner? What were the results?
- spenrose 10y agoPython. It was lauded for usability and became a 30 year success: http://python-history.blogspot.com/2009/02/early-language-design-and-development.html http://python-history.blogspot.com/2009/02/early-language-de...
- pjmlp 10y agoIf I recall correctly Microsoft did some regarding VB.
- zzzcpan 10y agoNone. Typical usability testing with watching people do things doesn't apply to programming languages, because of how much more complex and larger languages are in comparison. At best this approach can be used to test usability of tooling, IDEs, but that's it.
- nathan_long 10y agoI've heard it said that the creator of Elm regularly pairs with newbies and adjusts the language based on what they find difficult, but I don't have a citation.
- AsyncAwait 10y ago> 10 minutes watching new users code in Rust will give you better ideas than 10 weeks thinking about the problem in your head. I am not sure whether 10 minutes of Rust would give you actionable info, I mean it certainly takes longer to get familiar with the basics in almost any language, even if the 10 minutes just stands for a short amount of time.
- falcor84 10y agoI assumed "new users" implied relatively new users (before they grew to love the language enough to accept the downsides), not absolute beginners
- pekk 10y agoThings happen even in the first 10 minutes. It's not appropriate to limit user studies only to people who are already experts.
- dllthomas 10y agoBut it's also questionable to limit them to people who are not, depending on what you want to accomplish.
- lomnakkus 10y agoAlthough I think I agree that many current languages do not pay enough attention to ergonomics (hello, Scala!), I think your line of reasoning could be quite dangerous. I don't think there's any empirical support for the idea that a shallow learning curve translates into a powerful/expressive language. EDIT: I'd love to be proved wrong, so if there's anything in the way of evidence, please link me!
- acjohnson55 10y agoAs a Scala dev, I'm amused to see the Scala call-out :) Undoubtedly, it's a language that is a little tricky to get fully fluent, but I'm curious what things pop out to you as bad ergonomics? I think it actually doesn't suffer in many of the ways identified for Rust in this article.
- lomnakkus 10y agoIt's a bit late here, so I won't go into much detail, but just rant a bit. Hope that's OK and gives a general impression of my personal gripes. EDIT: Well, that rant turned out longer than I imagined... and it's even later now, so I'll probably go to bed momentarily. EDIT#n: PS: Scala's still better than Java, but please don't tell me Scala's in any way "coherent" or "well-designed" or "orthogonal". Haven't really tested Kotlin or Ceylon, but I get the feeling that those are really much more c-or-w/d languages for the JVM. Plus, "orthogonal" doesn't mean shit if your ergonomics suffer like they do in Scala. First: See Scala Puzzlers + PaulP's rants... and that's not even half of it. (Yes, I know collections are going to be "fixed", but... not really. I don't think the fact that Odersky still seems to be calling all the shots on that front is in any way encouraging.) Second: An especially annoying thing is the way "_" desugaring works which sometimes leads to really surprising results when it turns out that "no, you cannot use _ to stand for 'identity' in a function parameter context". This was a conflation of 'features' namely that scalac doesn't warn when a unit-returning method is actually with an expression that doesn't return unit. If that explanation doesn't make sense, I apologize -- it's a complicated issue that's hard to explain... which is exactly my problem with some of the problems I've experienced in Scala :) Third: The whole "let's encode typeclasses as implicit parameters" thing is insane. AFAIUI this is actually a thing that was intentional. Unless you have actualy real+usable dependent types it's a huge mistake to get rid of coherence[1]. (Plus, TCs are a semantic-level feature. You don't want to force people to "encode" semantic things you care about in your concrete syntax. That way lies "[OOP] patterns".) Fourth: The whole "for-yield" thing. WTF? Either do monadic notation like it matters or just don't do it. For-yield leads to incredibly noisy code. (It's not even consistent, try starting your for-yield with a "x = ..." line, and you'll see.). The fact that monadic notation starts with a "for" and that collection iteration starts with a "for" is NOT in any way a sign that your language design is "orthogonal" as Odersky loves to claim. It just means that the language designer didn't really understand what he was doing. (As you can probably gather I'm a Haskeller/Idirs'er/etc.). Syntax for these things matters. Fifth: ... and a follow-up to that: @tailrec isn't enough: That fact that explicit trampolining is basically required for any monadic Scala code basically kills the idea of monadic programming right off the bat... unless you're willing to invest huge amounts of time into it for-yield is just unusable. (Yes, I'm aware of the Free monad. One hopes that the performance problems have been solved, but it's still not always enough to have a Free monad.). Honestly I think it would have been better to invest time into implementing tail-rec on the JVM rather than wasting that time on @tailrec. I do mean wasting -- any situation where @tailrec applies, I could have trivially replaced (or "encoded" that "pattern") that code with a while loop and a var. Sixth: Why isn't Monad/Monoid/etc. in the standard library? This goes back to my third point which is that the language doesn't actually encode semantics of typeclasses, and so now we have "scalaz" with its Monad and "non" with its Monad, and "fs2" with its scalaz-stripped-down-copy-of Monad. WTF? This is not needed -- these typeclasses have stood the test of time and it's only because Scala insists on encoding them that we ended up here. (Encoding via various implicit magic tricks, the fact that everything has to be an object, etc. It's fucking ridiculous.) Summary: There's a lot of completely accidental complexity because Scala wasn't really designed as a coherent whole... it just sort of "grew organically". (Like almost every language out there.) [1] Yes, you gain a "choice" of instances via imports. Guess how often that's actually useful vs. harmful? Never. It's absolutely insane that your imports can change your serialization/deserializion code.