15 ms·
Complexity is a source of income in open source ecosystems (2019)
- deleted 5y ago[deleted]
- mushufasa 5y agoThis is true, and a corollary is that the observation that the value proposition for many SaaS companies is just the simplicity over open-source underpinnings. For example, anyone who's tried to implement SAML SSO by hand or self-hosting a shibboleth server will appreciate the relatively simpler services provided by OneLogin / Okta. I'm not sure, however, that this is a problem. The freedom of open source to develop without constraints has to come first, so that someone can later take existing code and impose guardrails to simplify. In other words, freedom itself is at odds with simplicity.
- deleted 5y ago[deleted]
- api 5y agoEconomics, not freedom, is at odds with simplicity. Nothing is actually free, so you have to have a paywall somewhere. For for-profit open source the paywall is usually at the UI/UX layer where that's typically SaaS-based or for enterprise software available in the form of consultants to hold your hand and train your team. Make a free, open, easy to use app and you'll go bankrupt.
- mushufasa 5y agoI meant freedom as in liberty, not free as in 'free beer.' In other words -- if you are free to write code any which way, complexity will result. Or conversely, in order to keep code consistent, someone would need to enforce rules.
- api 5y agoI 100% agree, but the problem is that most people care a lot more about free as in free beer than free as in liberty. I have for quite some time seen the two as being at odds. Free as in free beer undermines free as in liberty by destroying the economic basis for anything other than surveillance or closed business models.
- suifbwish 5y agoI frequently run across codebases that appear to have purposely “broken” tiny things that need to be fixed before the code can be used. This seems to be a type of lock/key mechanism to prevent just anyone from using it even though it’s open source
- the_optimist 5y agoA very unusual claim we have here, on the link-through to Matloff: “The Tidyverse also borrows from other "purist" computer science (CS) philosophies, notably functional programming (FP). The latter is abstract and theoretical, difficult even for CS students, and thus it is clear Tidy is an unwise approach for nonprogrammer students of R.” This seems to be the root of the argument, and it is a completely bizarre statement.
- vharuck 5y agoI agree with Matloff's overall point, though. "Tidy programming" (which came to mean non-standard evaluation) is very hard to understand, even for R professionals. It relies on directly handling symbols, and encourages using new notations to do it. Debugging is even more complex with NST code, and people learning the language will be doing a lot of debugging. I can't imagine a good way to introduce functions to newer users when using NST. You'd have to first mention environments, scoping, and symbols. My rant aside, I did find this next quote from Matloff a better argument for using dplyr: > mean(Nile[80:100]) "printing the mean Nile River flow during a certain range of years. Incredibly, not only would this NOT be in a first lesson with Tidy, the students in a Tidy course may actually never learn how to do this. Typical Tidiers don't consider vectors very important for learners, let alone vector subscipts." With dplyr, you'd subset based on a `filter()`, likely specifying the years to keep. It encourages self-explanatory code. Matloff's vector subscript tells me nothing about why certain elements are kept.
- throwawayboise 5y agoThe popularity of R amazes me. I took a one-week class in R and left with a vow to avoid it at all costs. I have never seen a more confusing, hard to understand, inconsistent software product in my 30 years as an IT professional. It's apparently targeted at scientists and sociologists who are non-programmers. I have no idea how they manage to use it.
- blacktriangle 5y agoMy description of R: It makes hard things easy and easy things hard. Ergonomically it is the absolute worst "programming language" I've touched in my life. However somehow it managed to become the official language of statistics research and has packages to do any type of analysis you can dream of. I think the reason you and I dislike R is because we just work differently than non-programmers. Non programmers think in purely imperative, straight-forward semantics. They write one-off unmaintainable code tying together libraries that solves their immediate problem. Programmers try and write R code as if it was a proper programming language and immediately run into walls. Non-programmers never see the walls because they don't even know there's another way.
- dragontamer 5y agoR > Matlab. But both suck as programming languages. But that's because neither R nor Matlab are primarily programming languages. They're primarily mathematical exploration tools.
- osalberger 5y agoRight. Julia is looking fairly promising as a "real" programming language that is still an excellent Matlab replacement, and possibly to a lesser extent an R replacement, and it does show that there is nothing about filling the exploratory math programming niche that requires it to have the warts that the incumbents have. Fundamentally, R and Matlab are probably best comparable to Perl in terms of how they got popular. While they are rather ugly, they were the first to provide a simple solution to a few specific problems that a lot of people had, which snowballed them into popularity and network effect advantages
- antirez 5y agoOne thing I used to say often was: "Complexity is a form of vendor lock-in"
- throw_m239339 5y agoConsultancies made billions with unnecessarily complex JEE/Enterprise patterns systems that needed hundreds of developer to maintain. So I'm not surprised all that pseudo agile/DRY/SOLID stuff were pushed as "good practices" in the 2000's and the 2010's. "No you can't just use if statements, you need to use a chain of responsibility with many classes cause it's SOLID". It's not a mystery that one of the author of a popular front-end framework was a JEE consultant, and this popular framework now feels like a solution looking for a problem...
- recursivedoubts 5y agoThere's probably something to that. I've accepted that my open source project, htmx, will likely never drive any significant income. It's too simple. Whenever someone starts talking about consulting, I tell them to read the docs and maybe jump on the discord, but that they'll probably figure it out. In addition to the revenue angle, a lot of developers don't like admitting "that seems too complicated" because it can be interpreted as "I'm not smart enough". Arguing for simplicity often requires a lot of backbone (or disagreeableness, according to taste.)
- whalesalad 5y agoComplexity is a source of income everywhere, not just open source. It’s even worse in the Microsoft world. Just because you can’t see the complexity doesn’t mean it isn’t there. The blog post is projecting a lot on the situation. It’s a blog. Take it for what it is.
- throwaway3455 5y ago>It’s even worse in the Microsoft world One of the reasons I stopped using Windows was the complexity pretty much forces a support contract for anything other than a toy business. Funny how distros offering support contracts are taking linux down the same path.
- robertlagrant 5y agoWhen choosing between Office 365 and Google Workspace, you may find the familiarity of Office is an expensive choice with all the IT people you need to hire to tame it.
- api 5y agoIMHO "open" systems that are too complex are not actually open. What is closed source anyway? It just means you get binaries. Binaries are not actually obfuscated. You can disassemble them, and there's even some decompilers on the market that will create functionally equivalent C code with auto-generated function and variable names. Nothing stops you from decompiling and reading a closed-source binary... except the complexity and difficulty of understanding it. Open source systems where the source and installation/admin procedure are absurdly complicated are not actually open... they're just using a different obfuscation technique. If it takes days to install or the source is write-only code, it's no different from closed source software.
- MaxBarraclough 5y ago> What is closed source anyway? This is asked as if there is no correct answer, but there is. Open Source software is software where the source is made available under an OSI-approved Open Source licence. Closed source software is software which is not Open Source software. See https://opensource.org/osd https://opensource.org/osd > It just means you get binaries Sometimes even the binaries are carefully obscured. I imagine it's an uphill battle to dig the binary out of a PS5 game, for instance. > Binaries are not actually obfuscated. Some are. > Nothing stops you from decompiling and reading a closed-source binary... except the complexity and difficulty of understanding it. And possibly the licence, and possibly laws like the DMCA. Also, even if you succeed in reverse-engineering the code, you may be legally constrained in what you can do. If you improve the software, you probably cannot legally distribute the improved version, although you might get away with distributing a delta. See also https://www.gnu.org/philosophy/free-sw.en.html https://www.gnu.org/philosophy/free-sw.en.html > Open source systems where the source and installation/admin procedure are absurdly complicated are not actually open... they're just using a different obfuscation technique. I agree artificial complexity (whether in the code or in the software's usability) could be a way for someone to, say, deliberately subvert copyleft licences. An artificially unmanageable codebase would increase the cost of development for the company, though. As for tricky installation processes, I believe the only way to get the official OpenBSD builds is by buying their DVDs. I don't think most OpenBSD users particularly mind compiling their own system, but I don't know for sure. > If it takes days to install or the source is write-only code, it's no different from closed source software. It is different, because the licence allows others to improve the software.
- fmajid 5y agoAnd this is why Python is eating R’s lunch.
- archibaldJ 5y agoIn mathematics we roughly have two ways to discovering new things: one is the theoretical approach (eg Galois and his theory that proves the none-solutions for fifth degree polynomial equations) and the other is the more adhoc technique-driven approaches (e.g. you can see the ingenuity of that in many of Erdos proofs). Of course the categorization is not always so clear-cut you often get a mixed of both worlds. I think it’s very similar when it comes to writing code. The more theoretical driven approach, the more structures you’ve got at your disposal to reason about things (eg see Mochizuki’s proof of the abc conjecture). For example, should you use monad in a small project? Maybe not. But if you are dealing with thousands lines of code every week you may find that if you reimplement some components as monads then suddenly refactoring gets easier and you can extend things more easily without having to do a global search and modify all the occurrences of a certain thing every time you make changes to the type, etc So ultimately you are offloading cognitive efforts to structures, which are really just constructs to optimize cost-to-transform (though it may increase the cost-to-execute (both computationally as well as mental-visualization/simulation-wise)) So is it worth the effort to work with structures? It really depends on the project you are working on or what you see yourself building in the next 10 years Depending on how you see your career it’s always good to be a bit more ambitious
- sebastialonso 5y agoyou know..."it depends" if a famous (non) answer among (good) software architecture
- hnrj95 5y agohas mochizuki’s proof been formally recognized? i thought it wasn’t yet agreed upon
- jacobolus 5y agoNo, Mochizuki’s “proof” is likely to be fatally flawed. He has almost entirely refused to engage with careful and targeted criticism by the experts who found what they claim is a serious flaw. Mochizuki finally went ahead and published his work in a journal where he is the editor in chief. So much for peer review.
- hinkley 5y agoThese days it seems like people make a library, and they never get to documenting it properly until someone offers them a book deal. And they can't just put all of than knowledge immediately in the docs otherwise who would still buy the book? From outside this looks like they're holding out on us. Some tools require google stalking the maintainer in order to synthesize explanations from six different places into a Theory of the System of how the thing is supposed to work. Apache Ant, for a memorable one. As with many things, you don't know your own opinion of a subject until you have to try to teach it to someone else.
- rabidrat 5y ago> holding out on us They're doing the fun part (coding) for free. It's well-known that developers don't like writing documentation (esp "properly" which is ill-defined and for some definitions can be a much larger task than writing the code in the first place); in fact, does anyone enjoy writing docs as much as an engineer enjoys writing code? So this makes total sense to me. As a user of free software, you can choose: a) pay for a commercial package with comes with docs; b) use free software, and pay for the docs in the form of a book (perhaps waiting a few years until lots of people want to do the same); c) use free software and figure it out from breadcrumbs the developer left in whatever forums they use; d) (c) but then step up and write some docs. Too many people think that free/libre software is supposed to be like commercial software, only better, but it's not the case. It's definitely better in the "libre" way, and of course cheaper in the "price" way, but it's usually not packaged for easy consumption, nor is the developer going to do a bunch of work they don't enjoy for other people who don't want to pay them for it.
- hinkley 5y ago> Too many people think that free/libre software is supposed to be like commercial software, only better, but it's not the case. "It's free so shut up" only works if being free has no consequences. Software is all social, and we pretend that it isn't. Cutting the legs out from underneath commercial software has consequences. Crowding a headspace with entrants has consequences for the next person considering doing the same thing. Implying you have something worth other people's time to look at has consequences. You figure out how to solve all of those problems and you can be above reproach. Until then, if you can't do it moderately right, then don't do it at all (or keep it to yourself).
- furstenheim 5y agoThis is kind of what happens with Postgres. Postgres in itself is great and simple to use. However, managing replication, maintenance, sharding... Is often quoted as very complex in the conference themselves. And that's precisely what most of the companies around Postgres live on
- deleted 5y ago[deleted]
- truth_ 5y agoWhich alternatively will you suggest. I know it all depends on the case, but for a fairly simple use case, what would you suggest? Are MySQL and SQLite still the most reliable ones around?
- whalesalad 5y agoThis is why I dislike commentary like the one you are responding to. Replace "postresql" with literally any tool and the sentence holds true: redis, mysql, mongodb, cassandra, mssql, HDFS, Kubernetes, React, Vue... The fact that cottage industries pop up around powerful tooling is not a bad thing. Some things are just complicated, and there shouldn't be such an aversion to those things.
- osalberger 5y agoThe thing is that SQL was originally built to be an easy accessible DSL for simple database queries, and was not intended to be used to build highly complex stuff. Datalog-like query languages would do the latter much better since they are actually capable of letting you write reusable predicates and libraries for common usecases, while SQL easily grows into a monolithic monster. Sadly a lot of the "NoSQL" hype later on was led by people who had no clue about database theory, and proceeded to throw the baby out with the bathwater and build whatever abomination MongoDB is, rather than taking SQL's theoretical basis and building a language on top of it that is actually designed for complex usecases. There were only a few exceptions like say Datomic in Clojure world that did NoSQL right.
- legulere 5y agoAnother case in point: OAuth
- ExcavateGrandMa 5y agolemme re-iterate :Ð Complexity is a source of useless processing with the only goal to obfuscate simple things... :)
- truth_ 5y ago> RStudio, the company, need to sell their services (their IDE is clunky and will be wiped out if a top of the range product, such as Jetbrains, adds support for R). If R were simple to use, companies would have less need to hire external experts. They are actually fighting for survival I would say. Most of the people using R and RStudio are the people who only know R and RStudio. RStudio's interface feels like it is from the early 90s, and RStudio is impractical and messy to use. Frankly, it is _bad_. R, as a language does only very few amounts of things well. And when some people's use cases that are limited to that, uses it. Data Science wave riders dominantly learn Python (IMO, as they should). Think >90%. There are plenty Data Science Misguiders (influencers) who, just from shallow knowledge, tell gullible learners to learn R OR Python. And still almost everybody chooses the second. Some people I know who uses R amd RStudio uses them as a simple Dashboard, dataviz app, or those who uses it for other reasons write R using Jupyter, VSCode, or good ol' vim. R thrives on the backs of lesser skilled developers amd domain experts using R for some reason. Some argue that the tooling of R projects is better than Python's, but it does so less. A penny is of course more reliable than a train.
- rleigh 5y agoI discovered that PyCharm had some nice R support a few weeks back. Not sure why PyCharm and not other IDEs, but what was there was simple but decent. Even embedded the graphics output in its panels and integrated really nicely. I can see myself using this when I have problems which call for R.
- kuang_eleven 5y agoI would suspect because Python is R's main competitor in data science.
- truth_ 5y agoPython does not have a competitor in Data Science.
- iagovar 5y ago
- gwern 5y ago> Once a collection of complicated packages exist, it is in RStudio’s interest to get as many other packages using them, as quickly as possible. Infect the host quickly, before anybody notices; all the while telling people how much the company is investing in the community that it cares about (making lots of money from). More than RStudio, it's in Tidyverse users' interests to extend and embrace; even without any plan or malice aforethought, it is simply natural that as Tidyverse users do their own thing, their packages and scripts will rely on the Tidyverse ever more, and their downstream users will thenceforth rely on it whether those users like it or not. Thus the situation where you install.package some harmless looking package and now R is installing 20 or 40 packages from the Tidyverse. I dub this general phenomenon 'bitcreep': https://www.gwern.net/Holy-wars https://www.gwern.net/Holy-wars Somewhat ironically, Hadley Wickham himself has been a bit annoyed by bitcreep caused by Mac/Linux users working against Windows users: https://twitter.com/hadleywickham/status/1280340931657564160 https://twitter.com/hadleywickham/status/1280340931657564160
- cwyers 5y agoI missed your article when it first came out, and I'm glad to have encountered it here; thank you, I think it helps explain some things here and more generally. I do think that this explains why the "Base R" people feel so threatened by the Tidyverse. What I find interesting about the quoted excerpt, and the article most broadly, is about how it spends very little time discussing what RStudio's economic interest in propagating the Tidyverse actually is. Because, notably, RStudio doesn't charge for the Tidyverse packages. There's no Tidyverse Enterprise Edition, there's no Tidyverse support plans. They mostly make money off their IDE and server products, and those products don't have a lot of meaningful synergies with the Tidyverse packages; there's few IDE features that integrate directly with the Tidyverse, and those that exist aren't very significant. Meanwhile, nothing about the Tidyverse is written to work better with RStudio than with VS Code using the R language server, or with Emacs, or what have you. This isn't to say that RStudio is doing this for any reason other than economic interest, but the economic interest here is not in convincing R users to use Tidyverse packages; RStudio makes the same amount of money if you do your plots in ggplot or not, and if you switch to ggplot there's nothing about it pushing you to change your IDE/editor. So what is the business model for the Tidyverse, then? It's pretty straightforward: the goal of the Tidyverse from the perspective of RStudio is to drive R adoption, mostly at places who are using commercial closed-source alternatives to R, like Matlab, or SAS or SPSS. (You could argue that they're competing for mindshare with Python, too, but RStudio has moved to treating Python the way that post-Azure Microsoft treats Linux, as a part of its product portfolio. RStudio's flagship products, the IDE and the Connect server, both advertise first-class Python support. Whether or not they have achieved this is another question, but they certainly are trying.) Once you have converted people to using R, you can sell them IDEs and servers. I suspect that the "Base R" people have a hard time grappling with this aspect of the Tidyverse business model, because it implies that Base R is _inherently less popular_ than the Tidyverse, which makes their losing less about the whims of a corporation that they can argue is acting in bad faith, but about the preferences of the R community, which they have no power to gatekeep. Because the point of the Tidyverse is driving adoption among people who were not previously R users, the Tidyverse is able to win over the majority of the community not by persuading holdouts but simply by growing the community with new Tidyverse supporters. I understand how that can feel threatening to someone who was an R user pre-Tidyverse, but you understand why they don't want to focus their message around shrinking the R community. And their efforts to persuade these new people to switch from the Tidyverse to Base R is undermined by the fact that the best argument for Base R over Tidyverse, the familiarity of Base R to someone who learned Base R to begin with, doesn't apply to them at all. They're fighting a losing war with bad weapons.
- fock 5y agoanother example is Moodle. It's horrible spaghetti code (PHP as used in 2000, including addons relying on specific database schemata) and if you google for it you'll find a nice forum thread, where a lot of LMS "consultants" are bashing a guy asking whether there are plans for a rewrite. And yes, as a Moodle user (teacher) you notice quickly that these decisions seriously impact usage!
- turminal 5y agoThe title is actually true for any kind of software ecosystem, open source or not.
- Nasrudith 5y agoI have found that claim mostly projection from closed source rivals in my experiences. Namely because those are the only places I have found support knowledge outright cashgated. I head the claim and it seems as if it was a line from Microsoft back when they were hostile to open source and a minute after it left their mouth managment promptly set up a meaning to ask "Why aren't we doing that?" shortsightedly to the detriment of their own longterm health. Open source ethos have been more "not want to deal with that crap" in ethos and a general tendency against wanting to provide support if it doesn't involve bugfixes. Again back when the community were more stereotypically grumpy greybeard telling you to boost your expertise by RTFM than evangalists with sizable blog and/or social media presence depending on the timeframe, or even novices grinding out tutorial websites as a portfolio item of sorts as an extension of "extracurriculars to get into a good college" side activities. I am not calling that cultural change a bad thing, far from it. I am just noting it has far more energy involved in maintenance of a communal efforts compared to less than easily accessible man files whose gap in time between last update and today could rent a car if it was an ordinary human.
- bigbillheck 5y agoI've had a low opinion of Matloff since at least https://matloff.wordpress.com/2018/06/20/neural-networks-are-essentially-polynomial-regression/ https://matloff.wordpress.com/2018/06/20/neural-networks-are...
- dash2 5y agoNow I'm curious. What was wrong with that paper?
- bigbillheck 5y agoThe central conceit of the paper (https://arxiv.org/abs/1806.06850 https://arxiv.org/abs/1806.06850) is the "NN<->PR correspondence" from section 6, which argues that since nice nonlinear activation functions like tanh can be arbitrarily well-approximated by polynomials, then the whole stack of convolutional layers can be arbitrarily well-approximated by polynomials, and that means the whole network is just a polynomial. And sure, ```technically''' that's true but there are good reasons that numerics aren't just all polynomials all the time (check out Lloyd Trefethen's 'Approximation Theory and Approximation Practice', for example). tanh is nice and smooth but its Taylor series doesn't converge outside a small region, and the whole reason you'd use tanh in the first place is because you'll have things outside that region. And tanh is probably the best case for Matloff's argument! ReLU has that corner, and I don't even want to think about approximating softmax with a polynomial. The blog has comments by, among others, Radford Neal and Kenghagho which I find compelling.
- dash2 5y agoIn general this idea may be true, but for the tidyverse it's just a load of horse. Hadley Wickham is a very talented developer, and what he's particularly talented at is writing interfaces that are easy for people to use. dplyr, a key part of the tidyverse, is a great example of that. It breaks data tidying up into a few simple steps that can be chained together. It's the descendant of an earlier iteration (plyr) and Hadley learned from that and just kept polishing the interface. There's a similar story with tidyr. Reshaping data from wide to long is a complex operation. Base R has a reshape() function and using it has given me permanent PTSD. It was impossible to get right until you read the documentation. After you read the docs - still impossible. Hadley wrote the reshape2 package, which improved things a bit. Then there was tidyr::gather() and spread(). Finally, we got tidyr::pivot_wider() and pivot_longer(), and at last I can be reasonably confident of getting the results I need without too many tries. The tidyverse is hugely popular for just this reason. Without it, I'd probably have abandoned R. I certainly wouldn't dream of teaching it. Ditto Rstudio. Calling these people parasites is absurd.
- macleginn 5y agoI agree with your assessment of the claims made in this post, but I want to note that for me the tidyverse culture became the reason to abandon R for all use cases where it has decent alternatives. R is not pretty, but it is also not super hard, and it has some internal logic. tidyverse's logic is orthogonal to that of R, it is a different data-manipulation paradigm, which I know have to know in addition to having to know R itself---not that you can really skip this part. Also, as it has been already mentioned here, this layering doesn't help with debugging. ggplot2 did a similar thing with plotting. Together these two projects made R more accessible-like and, IMO, seriously damaged the culture around it.
- asdff 5y agotidyverse has ruined stackoverflow. So many questions could be answered with base R, and sometimes you might need to do base R if you want your code widely compatible, but people insist on submitting some arcane ggplot2 or dpylr code instead, and nothing is learned about R.
- schleiss 5y agoI can emphasize with the argument that complexity is a source of income. But, I don't think that it's intentional. Rather, the constant change of developers introduced little inconsistencies that over time accumulate and generate complexity. Recently, I even looked at some open source projects[1] to measure the complexity by counting import/include statements. [1] https://schleiss.io/measuring-potential-complexity-in-open-source-projects https://schleiss.io/measuring-potential-complexity-in-open-s...
- jmcphers 5y ago> The wave is subsiding and they now need to appear to have a viable business (so they can be sold to a bigger fish), which means there has to be a visible market they can sell into. Might need a (2019) tag. As of 2020, RStudio is a Public Benefit Corporation and has a corporate structure that is designed for long-term investment in the scientific community, and isn't susceptible to buyout or IPO. https://blog.rstudio.com/2020/01/29/rstudio-pbc/ https://blog.rstudio.com/2020/01/29/rstudio-pbc/ (disclaimer: I work at RStudio)
- shkkmo 5y agoIt's great that RStudio is a B-corp, but that, by itself, isi a magic bullet. Is there any information you can share on what steps RStudio has taken to reduce susceptibility to buyout and IPO pressure from VC?
- jmcphers 5y agoAbsolutely. The B-corp by itself is indeed not a magic bullet, but the PBC transition (which is separate) does make a big difference. In a typical C corp the corporation is ultimately answerable to shareholders and it is shareholder pressure that causes buyouts and IPOs as shareholders need a return on their investment. A PBC is able to prioritize its mission (in our case, enabling access to scientific tools to everyone, regardless of means) over these demands.
- sytse 5y agoThe incentive to make open source products more complex is worse for consultancy business models than product driven ones.
- js8 5y agoIt should be according to the hypothesis, and it is an observable prediction that could confirm or reject it.
- ellisv 5y agoThis article strikes me as odd. The premise is based on a "pattern of behavior" but to me it feels the author is attributing malice where there likely is none. > For the last nine-months I have noticed that the term Tidyverse is being used more regularly to describe what had been the Hadley-verse. And??? Hadley never liked the name 'Hadleyverse' and the first appearance of the `tidyverse` package on CRAN is from 2016. > RStudio management rode in on the data science wave, raising money from VCs. RStudio is now a Public Benefit Corporation.
- lwhi 5y agoPerhaps there is a perverse incentive to not worry about simplifying now, but I think complexity will eventually piss people off enough to either leave the community or develop an alternative. Atlassian's Jira is a good day example of what the author's gunning for. A complex, unintuitive, hulk of a product .. with seemingly nonsensical UI updates and archives of out of date documentation. Amazing for people providing support and training. As a potential user, I've been annoyed by badly set-up instances so many times I'd gladly turn my back on it.
- agumonkey 5y agoThis is a regular pattern.. it's worth asking if that's not the sole mean for systems to stay alive.
- jedberg 5y agoMy first job was working for Sendmail, Inc. Their entire business model was based on helping people configure open source Sendmail, via consulting and/or a commercial web UI for configuring the software. And of course selling support for the open source software. And it wasn't even intentional. The software existed for almost two decades before the commercial company. The commercial company sprang up because the open source authors needed paychecks for all the support they were doing. And also a bunch of companies were selling Sendmail consulting. So they figured people would rather buy consulting from the people who actually make the software. Sadly, it didn't work out. The company was acquired for a loss to the investors.
- vmarshall23 5y agoHeh. I actually replied below before I saw this, but the sendmail.cf file immediately popped into my head when I read the post’s subject line.
- chovybizzass 5y agois that why i'm tired of programming after 25 years?
- sneeuwpopsneeuw 5y agoBased on the title the first thing that popped in my mind was https://www.aseprite.org/ https://www.aseprite.org/ there source code is freely available to anyone https://github.com/aseprite/aseprite/ https://github.com/aseprite/aseprite/ and if you search some more you can find how to install it at https://github.com/aseprite/aseprite/blob/master/INSTALL.md https://github.com/aseprite/aseprite/blob/master/INSTALL.md But none of these links are easy to find and the installation is not that simple either. Is this way of doing things good/bad? I just do not really know how to feel about it. It's partly better then not having the option available, but, i just don't know to feel about it.
- AzzieElbab 5y agoit is not unique to open source. Complexity is a source of income in software period. Simple software is hard
- zelphirkalt 5y agoI don't have much to say about the R ecosystem, as I am not actively using it or even doing anything using R. > Matloff is also upset about a commercial company swooping in to steal their precious, a common academic complaint (academics swooping in to steal ideas from commercially developed software is, of course, perfectly respectable). This reads like some kind of sarcastic statement. In fact it is indeed perfectly respectable, that academics do this, because it lets our society advance and research progress outside of proprietary software gardens. In my opinion any such kind of action is perfectly respectable, if it shares knowledge with the world. The other way around it is not OK, because from proprietarizing ideas, there is no benefit for society. If a company is worth its salt, it will offer services on top of existing public knowledge, without creating walled gardens. If it cannot do so, then it might be time to realize, that the company should not exist. What is not OK is, if academics take that knowledge and make it seem like they were the ones inventing stuff. One should always give credit where credit is due. Never forget academic honesty. Usually however, a lot of foundational research is done by academics, only to be picked up years later and often by commercial entities dealing in proprietary software.
- tfehring 5y ago> If a company is worth its salt, it will offer services on top of existing public knowledge, without creating walled gardens. I contend that every profitable company leveraging public knowledge does offer services (broadly construed) on top of that knowledge, since otherwise no one would pay them. Maybe the minimal example is that in which the only service provided is marketing, but the range of service levels provided in practice runs the gamut. RStudio clearly goes well beyond that minimum. In some cases, the services offered are so valuable that the company can create a walled garden - which reduces the net value of those services to the customer but provides the company with monopolistic pricing power. (It’s not clear to me that RStudio’s products have significant lock-in, for what it’s worth. R has significant lock-in, like any programming language, and RStudio benefits from the lack of competitors creating good R tooling, but that’s not really a walled garden of RStudio’s making.)
- dls2016 5y ago
- xg15 5y agoThis is why a market-based economy is not good at solving problems. It's great at managing problems, alright, but solving a problem would mean cutting off a source of potential revenue.
- jokethrowaway 5y agoI don't see how you got to this conclusion. It's not good at solving problems in the best possible way according to everyone. Centralisation is great at solving problems in the best possible way according to a single person or circle of people. It's good at solving problems so that the majority of people willing to pledge resources are happy.
- hu3 5y agoThere's an argument that the pharma industry has strong incentives to manage health problems, not solve them.
- breck 5y ago"There are 2 kinds of products: Ones that get more complex over time and ones that no one uses."
- shkkmo 5y agoTrue, but if your business model depends on providing support, you may be less incentized to minimize the growth of complexity as new functionality is added (especially if your software has little competition.)
- asdff 5y agoMeanwhile everyone uses 50 year old bash. Sometimes it's ridiculous to waste effort inventing yet another hammer.
- bigbillheck 5y agoAre you posting from 2039? Because from where I am bash is only 32 years old.
- workallday21 5y agoLooks like R is having their own systemd moment. At this point I would like to read a book about the Confessions of an Open Source Hit Man.
- harikb 5y agoI think the author ignores the fact that creating simple, consistent systems is a very hard problem, and rarely compensated (short term). So most apps, APIs, platforms will eventually become complicated and messy until there is significant push from a competition to force people to do the dirty work of cleaning up architecture So if you want a clean and simple R ecosystem, go support Julia or some alternative like that.
- rossdavidh 5y ago"Never ascribe to conspiracy, that which is adequately explained by incompetence." Not that complexity means the programmer was "incompetent", per se, but not sufficiently competent to keep things simple, because as you point out, keeping things simple is really hard.
- pdfernhout 5y ago"Simple Made Easy" by Rich Hickey: https://www.infoq.com/presentations/Simple-Made-Easy/ https://www.infoq.com/presentations/Simple-Made-Easy/ "We should aim for simplicity because simplicity is a prerequisite for reliability. Simple is often erroneously mistaken for easy. "Easy" means "to be at hand", "to be approachable". "Simple" is the opposite of "complex" which means "being intertwined", "being tied together". Simple != easy. ..."
- failwhaleshark 5y agoOpenStack, K8s. Nuff said.
- vmarshall23 5y agosendmail.cf
- reilly3000 5y agoI have no issue with this basic premise. Its annoying if you want to pay nothing for a mature, nuanced service, but in my mind its the most equitable way to compensate open source development teams. Dual licensing, "enterprise" features, and such seem to be popular, but are really frustrating to work with and buy. No knock on software salespeople, but I've candidly never had a pleasant experience around any sales or marketing interaction with an OSS team. I would far rather pay for expertise than suffer endless email marketing, inside sales lead qualification, waiting for the AE to ask an internal expert, and getting spun around whether a needed feature does/will exist. That entire apparatus seems to exist to extract money from corporate bank accounts while development is a cost center rather than main focus of the team. More fundamentally, complexity has more to do with the problem space than the business model.
- for_i_in_range 5y agoWhat he's touching on is Complexity Fallacy. (i.e. we "over-complexify" things). Why? "We often find it easier to face a complex problem than a simple one."[1] Humans also possess Simplicity Bias. (i.e. we "over-simplify" things). "Simplicity bias is a cognitive bias towards holding views which can be explained by a simple narrative."[2] Ironically, this very article may exemplify the Simplicity Bias to explain a Complexity Fallacy!! It seems the ultimate answer is the perennial, "know thyself." Practice self-awareness; and know which Bias you're leaning on, and more important, why you're leaning on it (in light of the goal). [1]: https://fs.blog/2018/01/complexity-bias/#:~:text=Complexity%20bias%20is%20our%20tendency,problem%20than%20a%20simple%20one https://fs.blog/2018/01/complexity-bias/#:~:text=Complexity%.... [2]: https://issuepedia.org/Simplicity_bias#:~:text=Simplicity%20bias%20is%20a%20cognitive,accurate%2C%20realistic%2C%20and%20balanced https://issuepedia.org/Simplicity_bias#:~:text=Simplicity%20....
- DesiLurker 5y agoIsn't it everywhere?
- Tade0 5y agoWouldn't be the only such project if that was the case. I often joke that Angular(or specifically any app written in it) is a job program for front-end developers. The architecture of this framework is so byzantine that you can spend _years_ working with it and still keep discovering new, often insufficiently documented features.
- ineedasername 5y agotheir IDE is clunky and will be wiped out if a top of the range product, such as Jetbrains, adds support for R I strongly disagree. I've used both and don't find RStudio clunky. And Jetbrains would need to do more than simply add support for another language to compete. RStudio is more than just and IDE. For one, there's Shiny, which makes it trivial to deploy slick web-based data products to end users. Then there's the fact that RStudio has multiple methods of compiling code and analysis into formal document formats suitable for publishing or presentation. RStudio isn't just an IDE, it's an entire ecosystem for coding, analysis, app development, and publication. It's not like Jetbrains couldn't make something like that, but it would essentially be a completely different product, not just one more supported language.
- rossdavidh 5y agoSo, a lot of this reminds me of other languages: 1) javascript and jquery 2) python and numpy (and the scientific stack built atop it) In neither case was the primary incentive, at first anyway, a desire to make money by complexity lock-in. I think there is a real cost to the complexity of having what is, essentially, a whole second language (with its own syntax and ways of doing things) built on top of the original language. But, I think this kind of thing can and does happen regardless of the financial incentives involved.
- rossdavidh 5y ago...plus, I found both jquery and numpy to be quite useful.
- jart 5y agoWell isn't that the whole pay for support business model? You know linux vendors could have made this easier, but there's never been any rational self-interest for them to do so. There's money in bringing order to chaos.
- 908B64B197 5y ago> A complex package ecosystem was probably not part of RStudio's product vision, at least for many years. But sooner or later, RStudio management will have realised that simplicity and ease of use is not in their interest. Or it just grew that way. Which is the most likely explanation.
- tpoacher 5y agoMore than 80% of things stated as facts in this article should have been prefixed with "My highly personal opinion is that [...]"
- mrkramer 5y agoComplexity is a source of income in every business ecosystem. For example Google is super profitable because information on the internet has high degree of complexity and the more complex ecosystem is the more opportunity there is for innovation and revenue.