18 ms·

How Dark Deploys Code in 50ms

269 points by craigkerstiens 7y ago

pbiggar 7y ago
Author here - happy to answer any questions!
- conorgriffin 7y ago
  What is Dark’s biggest weakness (excluding the current lack of adoption)?
  - pbiggar 7y ago
    Wow, what a start :) Probably the biggest one is that you have to leave your editor. If you've spent 20 years carefully configuring your vim or emacs setup, you're not going to be happy to give that up. Don't have a good solution to this right now either.
    - conorgriffin 7y ago
      Haha figured I’d get in early with a hard question. I signed up a long time ago out of curiosity, it’s been a while since then, looking forward to trying it out
    - Sophistifunk 7y ago
      Over all my years it's become clear to me that this attitude of refusing to give up a specific tool (and it's always something married to plain 1d text) is the biggest thing holding out industry back. I don't know if you can convince people to do it, but I'm glad somebody's trying.
      - imglorp 7y ago
        I'm glad someone's trying also. And the reason 1d text is hard to get rid of is not merely an editor (ie, an "app") but a galaxy of languages, tools, and operating systems built around 1d character streams. Everything from the serial driver to the style sheet can be manipulated with primatives for mapping, filtering, and composing character streams and operations on them. There ARE a bunch of benefits to structured code, for example Docker and npm/pip/etc have recently showed us a hint of what composition of standard components can bring. What might be missing from structured code is the rest of the primitives on ASTs. Once we have all that we can talk about backporting to emacs :)
        icebraining 7y ago
        All code is structured, otherwise it couldn't be compiled/interpreted. The Go community, for example, doesn't need to store their code in some sort of custom binary format to apply changes: gofix works fine with plain text. Text is just a serialization format for those ASTs, no different than any binary format.
      - jrockway 7y ago
        I think people don't want to leave their editor because navigating and typing code is a large part of programming. If you try some new editor, it's always going to be pretty wonky. For example, I've been using the Arduino IDE recently. I am used to navigating lines with C-n and C-p. It's something I do thousands of times a day. In the Arduino IDE, those keys create a new sketch (something I want to do maybe once a week) and print the current sketch (something I imagine nobody wants to do ever). The UI is just not thought out at all. I don't think I'm being resistant to change. I think their IDE is just garbage. Some people have done new IDEs well, VSCode comes to mind. But replicating 40 years of features is not going to happen overnight, and some of those features are actually important. Figuring out which ones are the key. If you don't hit a developer's critical set of features, they aren't going to use your thing.
      - icebraining 7y ago
        What do you see we could improve if we only left text? I've read the promise many times, but never actually saw any convincing reasons. Text is just another format anyway.
    - oftenwrong 7y ago
      One editor setup cannot satisfy all of the diverse preferences of all "religious" editor users. You have to support significant customisation to "convert" us. Maybe the editor should be customisable via Dark itself? Alternatively, allow us to use our editor-of-choice and provide a language server to facilitate the creation of structured editing plugins for Dark. I would add that most applications, including most non-vi-or-emacs editors and web browsers, do not allow much customisation of how their input is mapped to actions in the software. For someone like me, a person that is a vi die-hard because of its ergonomics, this is absolutely critical. Post-input customisation, no matter how advanced, cannot satisfy this need.
      - pbiggar 7y ago
        > Maybe the editor should be customisable via Dark itself? That's the plan!
- djm_ 7y ago
  Other languages have large ecosystems of packages that allow users to do common tasks quickly. I can imagine not having something similar would be a sticking point to adoption; how does Dark picture the Dark ecosystem working and competing with other already well-entrenched ecosystems? Thanks!
  - pbiggar 7y ago
    There's a couple of ways of handling this. First is that of course we need to create a package manager and ecosystem, and that will take time (we can accelerate it by making a great experience, but haven't gotten there yet). Dark can talk to other HTTP things, so many of the functions that are done by libraries can be done over HTTP. So you can call other services, even if we don't have an SDK for that yet. And from there it should be easy to package it up so there is an SDK. Finally, it's a question of what you're doing. There's many tasks where the disadvantage of not having an ecosystem is outweighed by Dark's other advantages. And as we grow the ecosystem, the disadvantages go away, unlike the status quo tooling.
- manoji 7y ago
  I am not able to signup for beta with an organization email address. is this intended?
  - pbiggar 7y ago
    No it's not! Sorry about that. Email me direct (see my profile) and I'll try to fix it.
- the_duke 7y ago
  Dark sounds like an interesting concept. I seriously hope though that you will open source the compiler. I can't see Dark going anywhere without a guarantee that you can use it without using the provided infrastructure and that your code will work fine even if Dark goes away or you decide to host it yourself. I certainly won't touch Dark with a 10 foot pole otherwise. And would advise anyone against it strongly.
- ToJans 7y ago
  I see quite some similarities with low-code systems, 4GLs, smalltalk etc, so AFAIK the market is there... Please take into account that your target audience probably is not on HN. If I would be you, I would target web agencies, that can use the dark platform as a competitive advantage, if your system allows you to build stuff faster... IMO it does not make sense to learn all these new paradigms for a small project, you really need scale for this kind of investment from your customers... Right now it feels like you are building a solution that does not know which specific problem it is solving...
- alexashka 7y ago
  Hi Paul, How did the decision to create a whole new language take place? Was using existing languages a serious consideration and if so, what were the trade-offs that made you go in this direction? I'm familiar with the basics of back-ends. Let's say you had a language that compiles each back-end point into a binary executable (each api point is a separate program). All you'd need is versioning via directories, symlinks to the active most up to date version and a router that looks up the executables in a dir and routes to them. Making changes would involve changing the code, pressing "deploy" which pushes to github and gets the backend to pull, re-compile the binaries and update symlinks. Given this setup - what does your approach do that's more than what I've just described? I'm assuming there's some fundamental problem my naive approach does not handle that made you go in the direction that you've just described.
  - pbiggar 7y ago
    We did of course consider it. I don't think it's possible to use existing languages to do this. Presumably, we'd have to use Javascript as our basis cause everyone knows Javascript. But then we'd have people wanting "Dark for python" and "Dark for Ruby" and "Dark for rust", etc, which making a new language neatly sidesteps. So if we built it for javascript, we'd need to decide how to support all of JS's features. How do we support eval? How do we support Typescript and Flow and ReasonML? Then we'd be stuck with all the JS mistakes: promises and futures and callbacks and object model and null/undefined/0/"" comparison tables. This list keeps going: it would have to use a parser. It would need to support npm, etc, etc. I expect once people see Dark that they'll try to build similar features for existing languages, and I hope they succeed! There's some companies that are doing similar things, and I think the scale of their vision is much lower, partially because they are limited by language choice.
    - alexashka 7y ago
      Hm, your answer is not what I was expecting :) Some follow up questions if you are up for it: What are the fundamental features of Dark that are not possible to add to an existing language? What are the fundamental features of an existing language that are not possible to restrict via enforcing a subset, that are fundamental to Dark? Is Dark a very domain specific language? Perhaps these questions will better be answered by a demo that is coming in September.
- leonaves 7y ago
  Is it possible to write code in an offline environment (e.g. on a plane)?
  - pbiggar 7y ago
    Yes, we'll have support for that, including handling sync conflicts when you get back online. It's not gonna be a great experience since Dark is really really designed for working on production, but it should work ok.
deleted 7y ago
[deleted]
devcriollo 7y ago
They show very little, I'm anxious to be able to see.
- pbiggar 7y ago
  We're talking about the design of Dark in these posts. We plan to show what Dark looks like in our launch in September.
sddhhanover 7y ago
Can I take Dark out for a spin?
- pbiggar 7y ago
  Right now, we're in private alpha. If you have a product or service that you want to build, fill out https://darklang.com/subscribe https://darklang.com/subscribe and our CEO Ellen will get back to you to discuss and schedule an onboarding. We're not doing tire kicking yet though :)
  - sddhhanover 7y ago
    Entered my email awhile ago... just updated the mailchimp profile (daniel@orthly.com) to include my use-case. Let me know! Would love to provide feedback - what you're working on sounds awesome
davejstl 7y ago
What are the server/OS/cloud prereqs for Dark?
- pbiggar 7y ago
  There are none! We run the infrastructure in the cloud (we run it on GCP using Postgres and k8s, but will likely go multicloud at some point). Most of the advantages we discuss in this post come from the fact that you're using the Dark editor, language and infra. I don't believe this would be possible with a self-hosted solution.
  - Deimorz 7y ago
    So anyone using Dark is completely locked-in, to the point that they can't even take their code into an editor other than yours? It's a proprietary language, that can only be edited in a proprietary cloud-based editor, that only runs on proprietary infrastructure?
    - tutfbhuf 7y ago
      Yes.
    - pbiggar 7y ago
      Yes, that correct. There are lots of downsides to this, but you also get a lot of things that we don't think are possible without it.
      - nikolay 7y ago
        You're wrong. Everything is possible as open-source. Often the lock-in is not an option. Not at any price. I hope you realize that soon.
        mhenrixon_ 7y ago
        ”You’re wrong” is a sure way to get people to just walk away from a discussion.
voidr 7y ago
> When you write new code in a function or HTTP/event handler, we send a diff of the Abstract Syntax Tree (the representation of your code that our editor and servers use under the hood) to our servers, and then we run that code when requests come in. You can achieve the same result with PHP and FTP, however once you start adding unit tests and branching, that's when things start to take time. Long builds aren't necessarily a problem if they give you enough in return.
- pbiggar 7y ago
  You're exactly right that this style of development would be completely unsafe using PHP+FTP, which is why we had to include all the other stuff: the feature flags, function versioning, DB migrations.
  - voidr 7y ago
    What I was pointing out that hitting save and getting a result on the live server was achievable in the 90s, there is nothing new about it. The reason we stopped doing that is because we wanted version control, we wanted to work in teams and we wanted unit and integration tests, which I doubt that you could run within 50ms. > the feature flags, function versioning I can easily replicate that with at least early 2000s PHP(would probably work with 90s PHP as well). > DB migrations We do have DB migrations in all web frameworks that are worth using.
    - pbiggar 7y ago
      > The reason we stopped doing that is because we wanted version control, we wanted to work in teams and we wanted unit and integration tests, which I doubt that you could run within 50ms. A good way of thinking about Dark is to suppose that instead of stopping doing that, we found a way to make it work. > I can easily replicate that with at least early 2000s PHP(would probably work with 90s PHP as well). If you could replicate it, you'd do it. Except you need version control, unit tests, etc, etc. So you need tooling that has those things to make the feature flags work.
nlawalker 7y ago
Sounds cool, seems like an interesting set of tradeoffs that nobody else has tried yet. From what I've read so far it seems like Dark is a good fit for prototyping, experiments, and small products and teams that "move fast and break things". Would you consider it a good fit for large teams and applications, where different people own different components and reliability is the most important metric? A lot of the accidental complexity in the "standard issue" continuous deployment process detailed in the post is there to prevent accidents and enforce permissions.
- pbiggar 7y ago
  Yeah, all this stuff is actually designed for large teams (though, it'll be a while before Dark is production-ready enough for them). CD matters because it allows you deploy in small chunks which is much less risk than deploying in big chunks. The feature flag is a thing that you might not need if you're small, but it completely changes how large teams work. Same with DB migrations: if you have a small amount of data you can just do a schema migration and the locking wont affect you too badly. We'll have a permissions model at some point that will take care of the ownership problem. We designed Dark to allow extremely fine grained ownership, for example allow a contractor access to a single HTTP route or function. Overall, this is kinda similar to how we went about building CircleCI. We focused on good tooling for companies (eg parallelization and speed) instead of individual projects (eg build matrices).
- lmilcin 7y ago
  The way to go is to have high-trust culture and comprehensive automated testing system. You want developers to behave responsibly and to put all development effort into writing good, reliable code in the first place. I personally prefer pair programming over code review. Code reviews have a lot of downsides but two most important are that you can't get from commit to prod automatically (as there is another inevitable manual step) and that reviewer typically doesn't have enough background/time to perform the review in meaningful way.
ricardobeat 7y ago
EDIT: Ha. My apologies as I just realized my stupidity - I was browsing on mobile and it turns out out I jumped into the 'What is Dark' post [1] and never finished the original! The mailing list links are in that one - I subscribed. This is a lot more interesting, cheers :) [1] https://medium.com/darklang/the-design-of-dark-59f5d38e52d2 https://medium.com/darklang/the-design-of-dark-59f5d38e52d2
- pbiggar 7y ago
  The section that describes how it does it in 50ms is called "#deployless". That describes the mechanics of the deployment. How we made deployless possible is addressed for the rest of the post after that. (The mailing list thing is weird, I only see that link at the bottom. Can you point me at the ones you see in the text?)
  - deleted 7y ago
    [deleted]
adrienjt 7y ago
How coupled is Dark with the Dark language? I understand that the purely functional nature of the language makes its integration with the rest of the platform easier, but the blog post doesn't go into much detail about that. What would it take to integrate Dark with: a) another (existing) purely functional language, and b) a general-purpose language, e.g., Go? Could a code intelligence tool like Sourcegraph help?
- pbiggar 7y ago
  Extremely tightly coupled. That's kinda the point "what can we do if we don't have to couple dark to anything else" (eg editors, other languages). I can't imagine how you'd couple Dark to another language, but I also don't see the point?
  - adrienjt 7y ago
    Thank you. My point was risk mitigation, which I discussed in another reply (which you answered: https://news.ycombinator.com/item?id=20399267 https://news.ycombinator.com/item?id=20399267 )
    - pbiggar 7y ago
      Yeah, the bigger picture here is that there's a ton of valid reasons why people can't use Dark right now. Of course there is, we're a small team with an early product. Once we show that this is really a better/faster/easier way to make backends, then we can start working on handling the risks and constraints of more teams.
      - adrienjt 7y ago
        That makes sense. It does look like you're very close to that first objective though. Congrats! Your blog post very clearly describes what Dark actually is (which was still unclear before I read it).
iLemming 7y ago
I've been seeing a lot of talks about darkland and can't find a single example of code. Can someone show me at least how does it look like?
- gspetr 7y ago
  from darklang.com: "Dark is currently in private alpha." That's probably the primary reason why there's little to no info.
  - amelius 7y ago
    They are leaving us in the dark, so to speak.
    - quickthrower2 7y ago
      It feels like an ICO launch in that regard. Kind of exciting!
z3t4 7y ago
Syntax errors are the easiest errors to fix and almost never get into production. Zero downtime deployments are very rare, and ads a lot of complexity, so it's a big trade-off, compared to just uploading the new code and restarting the service. And having the front-end deal with minor hiccups like service restarts, because users will have such issues all the time due to being on mobile networks, trains going through tunnels etc. You can do as much testing as you want, both manually and automatically, but you still can't detect all issues as efficient as thousands of users in production. So just accept that there will be issues, and instead design your pipeline so that those issues can be fixed within minutes.
- quickthrower2 7y ago
  > You can do as much testing as you want With what Dark is promising, I believe testing would be easier. Deploy to a test environment in 50ms. Then run a suite of tests, then deploy to production in 50ms. There could be some complementary tech that figures out what tests need to be run based on what you change to get the time down.
  - heavenlyblue 7y ago
    They're kind of lying. You only need pip or any package install if your testing image doesn't pre-install the required packages before running the tests. Which I don't myself since I always want to see if my packages work with the latest versions of other packages - and Dark wouldn't be able to handle that anyway. So my test runs actually complete in about 50 ms if I'm not testing any external API functionality. They will also never finish earlier than 5 minutes if I am, even on Dark - since they're based on external services.
- snypox 7y ago
  > Zero downtime deployments are very rare What do you mean? Rolling deployments are a very basic feature of container orchestrators.
  - nijave 7y ago
    You don't even need containers--it's a feature of load balancers containers rely on. Start new container, wait for it to become healthy, add it to the load balancer, remove old container (with an optional, let connections to old container close first)
koliber 7y ago
It's an interesting concept that flips many things on its head. We're getting some real hard-to-achieve benefits out of the box, while sacrificing a lot of things we used to take for granted. One thing that I notice is that Dark requires a completely walled in environment to be able to deliver all of its promises. In reality, any real-life project has to interact with external services, legacy systems, or custom peripherals which are unlikely to be based on Dark. How will Dark be able to handle such cases? Current tooling and processes are able to cope with such things through custom, hand-crafted, and as the article stated, often fragile, systems and processes. The are a pain, but they get the job done. They are very versatile by their nature. While they break more often that we would like to, they can be patched up well enough to band-aid over rough patches and adjust to surprising externalities. Dark seems to have one way of doing things, and I am curious how compatible that way is when it comes to interfacing with external systems.
- btown 7y ago
  It seems that Dark is designed to allow you to wrap external APIs in easy-to-use interfaces, so it should be at home as part of a larger SOA or ecosystem, of course with the caveat that it's still being developed. See the footnote at https://medium.com/darklang/dark-residency-7dbc878a5f0a https://medium.com/darklang/dark-residency-7dbc878a5f0a .
  - pbiggar 7y ago
    Yes, that's exactly right. Dark speaks HTTP (and at some point, will speak GRPC, Thrift, etc) so you can access 3rd party services over HTTP (and wrap them in SDKs in the Dark package manager, same as on npm). We expect we'll have some built-in support for common things (resizing images and videos, or generating pdfs for instance; we already support static assets on a CDN).
    - simonebrunozzi 7y ago
      Hey Paul, nice to see you here on HN. Very promising start! What's the best way to contact you? (I'm $HNusername @ gmail).
quickthrower2 7y ago
I think it is fantastic that this kind of innovation is happening. I get sick of waiting for stuff at work, some kind of build or CI thing. npm i, docker build or whatever else. I hope the language experience is good, and it's a reasonable language (static typing!). It sounds like taking the Go idea of quick builds to the next level.
davedx 7y ago
How does it compare with hot swapping in other languages and platforms like Erlang, JVM/.NET hotswap technologies? I'm guessing the built-in safety functionality from the feature flagging is one thing?
- pbiggar 7y ago
  It's much simpler. Those have complex in-process hot swapping. Dark's "hot swap" is a write to a DB, then new HTTP requests use the new code. As we grow and start to look at compilation and optimization, I think we'll probably get much fancier, but for now we're very into keeping things super simple.
  - davedx 7y ago
    I'm pretty interested in this. Signed up to the mail list. Good luck!
bufferoverflow 7y ago
Code deployment is almost always trivial and fast. It's the databases that are slow and very tricky, especially in distributed systems.
unfunco 7y ago
It sounds like a convoluted version of Geocities.
GordonS 7y ago
As an aside, that big list of tasks and diagram at the end are perfect to show next time a client exclaims about how long it takes to get a simple change into production!
kccqzy 7y ago
Twenty years ago, deployment of a new version of a web app means FTP'ing a new php file to the server.
- eloisant 7y ago
  Well, you're free to deploy like this (but you probably know why it's a very bad idea).
  - somehnguy 7y ago
    Feel free to explain why
    - adrianN 7y ago
      Auditing what was changed and which version is currently in production is a bit harder than it has to be.
    - stabbles 7y ago
      Yet another reason is FTP turned out to be inherently unsafe.
      - jszymborski 7y ago
        SFTP, however, has FTP feature parity and is secure as SSH (since it is ostensibly SSH).
    - viraptor 7y ago
      Copying of multiple files is not atomic. You need to ensure a file with func1 exists before another file calling func1 gets copied. You can do it atomically by having 2 directories and swapping a symlink, but... that's a scriptable pipeline at that point.
- pikzel 7y ago
  Which wasn't a bad thing after all. I've had more incidents with deployment pipelines than with the actual code.
blumomo 7y ago
That's quite a dark name for such a bright idea.
oftenwrong 7y ago
Dark sounds like it would satisfy my preferences well. - statically typed, functional language with no null pointers - safe deploys and rollbacks - structured editing / version control - minimal infrastructure management ...etc. A tool that makes writing a correct program, and deploying it without catastrophe or fuss, easy. I look forward to reading more.
bullen 7y ago
I added the same feature to my Java app server: http://github.com/tinspin/rupy http://github.com/tinspin/rupy it's a simpler design which compiles, zips and hotdeploys over HTTP to a classloader. But it's fast enough as long as you keep each project small.
juandazapata 7y ago
Given how tightly coupled is Dark with its IDE and deployment environment, I'm interested in knowing how much VC money have they taken? What if they get acquired or simply shut down? It looks like there won't be a migration path, and you'll need to rewrite your app from scratch using a different language. Again, it would be nice some transparency around how much VC money they've taken.
- andy_ppp 7y ago
  For there to be 9 of them and 4 open heads I'm guessing it's probably $10-20 million at this stage. I think the only way I'd use this is if I had the option to host it myself i.e. the code was all open sauce.
- pbiggar 7y ago
  The sustainability of Dark is super important to use. We're not fucking around with customers' livelihoods. For our early customers, we have a handshake agreement that we won't leave them hung out to dry. If Dark was to shut down, we would 100% make sure they had the time and ability to port their app off. There would be a time cost to them of course, so it's not perfect. As we get closer to opening up to the public, we'll be doing a bunch of work on the sustainability question so we have good answers to the "acquired or shutdown" problem. We imagine a legal framework to protect customers, but haven't got the details worked out.
  - londons_explore 7y ago
    A legal framework doesn't help much in the "bankruptcy" case. Unless that legal framework means all IP becomes open source and people can self-host I guess.
deleted 7y ago
[deleted]
sciurus 7y ago
How does Dark handle code review? Even if I was willing to accept the other tradeoffs that come with Dark, I'd want another human to approve my changes before that 50ms deploy to production kicks off.
- jmull 7y ago
  I don't think they've detailed that yet, at least not that I've seen. But it is one of the few steps the Dark deployment process retains (see the diagram near the end of the article), so it seems they will support it (hopefully reasonably well).
  - londons_explore 7y ago
    Seems easy enough... Write whatever code you like, directly in production, but feature flags will be forced to zero (or to just your own test users) until code review passes... It requires the language to have a bulletproof feature flag system which can't be bypassed, which it seems is the goal.
- pbiggar 7y ago
  The model is quite different to how people write code today. Instead of a process that takes code from your machine and sends it to prod (and thus has a lot of risk), you code in a sorta sandbox _in prod_. That is then enabled for users via changing the feature flag setting. So to do code review, you write the code behind the feature flag, then ask someone for code review. (There's google-docs style collaboration, so they can see your code if you permit it). After code review, you can "merge" by toggling the feature flag, or changing the setting to "let 10% of users see this" or whatever roll-out strategy you want. just safely. So you're writing code behind a feature flag (but "in prod"), but no users can see it.
  - dkordik 7y ago
    cool- I guess there is some concept of a snapshot/diff to show you the before and after?
keymone 7y ago
The problem is granularity of deployment - the smallest piece you can rollout is, these days, a container with code and dependencies, that has to be build and threaded through your CI/CD stack before it hits prod. I like to imagine what would need to happen to reduce(increase?) that granularity to single function - you change the code, byte-compile it, send that blob to your backend’s hot code swap port and voila - you’re done. A lot has to happen behind the scenes though - a change in dependencies would still require a larger deployment or much smarter code swapping procedure, caller/callee tree would need to be considered, functions need to be pragmatically pure (talking to dB is fine, being part of an “object” - not really), functions need to be extensively specced, etc. But that would be my backend holy grail.
- deleted 7y ago
  [deleted]
- icebraining 7y ago
  What are you looking for, that isn't provided by platforms with hot code-swap (e.g. Erlang) or FaaS systems (e.g. AWS Lambda, OpenFaaS) ?
  - keymone 7y ago
    you're thinking of only surface functions/endpoints, those that are invoked directly from your api route dispatcher. i'm talking about any function in your source code of a large backend with many endpoints. think of function `+` - i want to be able to deploy a change to that function such that no worker process gets respawned. also imagine every time you call `+` in your application - it goes throught the whole FaaS shenanigans with serializing, sending rpc over wire, receiving response, deserializing, dealing with throttling, errors, backoffs. all of that is ridiculous for `(+ 1 2)`. and yes, erlang is a good example of an environment where this could work.
toolslive 7y ago
Back in the day (early 00s), you could do something similar with ZOPE: develop on a production server, in a private session and you could mock about with your version of the code, and when you were satisfied push it through for everybody else. This was because the database that was holding the users, permissions, ... was also holding the code. It was lightyears ahead of J2EE at the time. (Now I wonder what happened to it)
- blunte 7y ago
  I know that Plone, which is built on Zope, found its place in a lot of universities and governments. So I imagine Zope lives on with necessary support from its big users.
  - slinkp 7y ago
    Yup. Plone is still pretty active https://github.com/plone https://github.com/plone
- slinkp 7y ago
  Former professional Zope guy here! (I made a living building things with it in the early-mid 2000s). What happened to Zope? Well, Chris McDonough (creator of Pyramid, and Zope veteran) blogged about this in 2011 and from my perspective he got it exactly right. I still think this history is fascinating and I wish more people knew it. https://web.archive.org/web/20161019153348/http://plope.com/Members/chrism/in_defense_of_zope_libraries https://web.archive.org/web/20161019153348/http://plope.com/... After the history lesson, some of his "lessons learned" bullet points seem very apropos in the context of Dark, particularly the first two: * "Most developers are very, very risk-averse. They like taking small steps, or no steps at all. You have to allow them to consume familiar technologies and allow them to disuse things that get in their way." * "The allure of a completely integrated, monolithic system that effectively prevents the use of alternate development techniques and technologies eventually wears off. And when it does, it wears off with a vengeance."
  - pbiggar 7y ago
KirinDave 7y ago
So... you're 50ms away from a total global outage and/or major feature fault? Why would I want that? How do I control it for green-blue? For rational isolation? Is there native support for canary releases?
- coldtea 7y ago
  Yeah, because speed of deployment prevents CI/tests/checks /s (Seriously, how do people come with such facile counter-arguments?)
  - KirinDave 7y ago
    How does CI actually prevent these kinds of outages? Did CI save Google from the networking outage or tests prevent a global Cloudflare outage? Do you think those organizations don't spend enough energy for the average SV commute on test runs per release? Are you seriously going to pretend that those efforts "solve" reliability and stability in any non-trivial system? It's actually not hard at all to write fast software deployment. Many people do it accidentally. But to tie it directly to editors and stovepipe the infrastructure around testing, that seems to me like a great business model that any rational engineer interested in reliability should immediately reject unless it's coupled with open source, statistically valid canary tooling at the bare minimum. What you actually want at that degree of universal knife-switching is rollbacks.
    - penagwin 7y ago
      > Did CI save Google from the networking outage or tests prevent a global Cloudflare outage? This is not a fair statement. This is like saying "Did seatbelts save the victims who died in car accidents X/Y/Z?" Just because your tests/ci/etc. can't catch every single issue that could go wrong doesn't make them useless. Just because people die every single day in car accidents doesn't mean car's safety features are useless. You're correct in that a CI should not in any way be considered a "perfect solution for outages", but an automated system that tests, deploys, etc. in a predictable and (again) testable manor is at least helpful.
umvi 7y ago
Very interesting, but I don't like the idea of abandoning VSCode to learn some single-purpose IDE
vinceguidry 7y ago
I'm getting really tired of learning new languages / stacks every few months. I want one language to write things in. It has to be a pure-OO language and right now that's ruby. I want to edit that code with one code editor. Syntax awareness is a nice-to-have but not essential. At the moment my editor of choice is Sublime Text 3 but to extend it I need to write python which is unideal. When I start writing websites it'll be hyperstack. Unless your new language can fix all my issues, be open source, and be my One True Human Interface to All Things Machine, I'm going to keep building on ruby, with occasional bash / crystal / rust excursions for speed. Is your language pure-OO? Do you have gradual typing? Does it run on a common VM? Am I going to have to rewrite everything I'd ordinarily bring in a library for? I can accept every single one of your claims but still not be interested in learning a whole new stack because I can already deliver stuff quickly enough with the one I've been iterating on for a decade. Especially when it could end up orphaned in 3 years if it doesn't get enough adoption. I'm not saying your holy grail can't be holy enough. But it has to be pretty darn holy.
taosx 7y ago
Marketing piece with grand plans...not really I would argue with you that this is not such a priority for a software development organization, there are more important things to do. You got so excited...so fast, I think this community may be doing more bad for me than good lately.
vincentmarle 7y ago
This sounds extremely exciting and it looks like a lot of thought has been put into this. I’m looking forward to try it out. A lot of comments here are coming from an anxiety that is caused by many years of dealing with fragile deployment processes, but I like the boldness of Dark to just bypass all that and make the fundamentals safe and solid.
burlesona 7y ago
First I’ve heard of Dark. The blog post is cool, but reading it I felt two main things: (1) no pull reviews would be terrible for a team, and (2) everything being instantly live in prod and automatically versioned etc sounds like a nightmare once you move beyond a simple program. Reading on Darklang.com I see they kind of address this. The ecosystem they’re building is training wheels for developers, a way to make “coding 100x easier.” I can see that working in a sense, and being an entry point that a lot of people use to make neat toys. I could even see it being a gateway that people use to get interested in and learn about programming. But I can’t imagine wanting to build a business with more than one programmer, or any kind of scale on a completely black box system like that. I will also say, there’s a problem when it comes to entry-level systems, trying to teach people to code: The struggle with the complexity is actually important. Dark isn’t actually making all the work of building and running a web application go away, it’s abstracting it all into a platform such that you can’t see it. Suppose a person gets into coding self-taught and learns to work this way. The knowledge isn’t going to be very transferrable - ie when they look at other languages or systems they’re likely to struggle with problems like “what does prod versus dev mean, I just want my program to run for the world...” You usually move knowledge between ecosystems by translating “I did it this way in (toolset A), so what’s the parallel in (toolset b)?” The more you have an idea of the underlying principles the easier this is to figure out. That said, Dark certainly looks neat, and I imagine the implementation is quite cool. The only other nit I’d pick is the name. To me, its not really a language, but maybe a development environment, or framework. I suppose it had a language in it, which I’d probably call DarkScript or something.
- pbiggar 7y ago
  Author and Dark CTO here. > Reading on Darklang.com I see they kind of address this. The ecosystem they’re building is training wheels for developers, a way to make “coding 100x easier.” No no no no no. This is not what we're doing. We're building something for experienced software engineers. (New developers will be able to use it too and it should be much much easier for them than learning in other ways, but that's a secondary audience). > Dark isn’t actually making all the work of building and running a web application go away, it’s abstracting it all into a platform such that you can’t see it. Some of it is being abstracted, but a lot of it being properly removed. Servers are abstracted, but almost all of actual deployment is actually being removed.
module0000 7y ago
Hrm...we don't deploy code in to prod in 50ms, but we certainly deploy to dev/staging/testing in <1s, and promote to prod once we are 100% sure we want to do so. We use the boring, industry-standard tooling and process everyone else uses (Jenkins, k8s, Ansible, nothing bleeding edge). I'm unsure what advantage their product has that makes it stand out among the battle tested tools already in use. Would someone elaborate? disclaimer: working for a l2 network provider, and a mistake costs us infinitely more than an "Oh we're sorry!" can fix.
- ukd1 7y ago
  The point is all that stuff (jenkins, k8s, ansible) is accidental complexity that, I presume someone at your company has-to/had-to think about. I think Dark's angle is it's stuff you shouldn't need to think about in order to solve the actual problem, and removing as much of that as possible is better.
  - deleted 7y ago
    [deleted]
  - alexandercrohde 7y ago
    Okay, I mean conceptually cool, but it's not like an existing startup gonna throw away all their code and rewrite it to save on dev-ops time. I guess the value prop is that if you're starting from scratch, and want to write your whole app in Dark then you can avoid hiring a dev-ops engineer and iterate faster? Seems like a stiff trade-off. If it's an interpreted language you're coming from, setting up envs can be < 30 secs anyways (and same with prod deploys), so that's faster than you need. If it's a compiled language you're coming from (e.g. scala/c++) then it's likely Dark may not match your needs. Plus I guess this assumes every team in your company is using only dark? Like what if it's a SOA and a full environment has a node component?
akhilcacharya 7y ago
Is Dark intended to be a Lisp? I'm surprised by the fact that the team didn't include language examples (from what I can gather)
- pbiggar 7y ago
  No, it's an ML. (You're right that we haven't published samples, we think people get distracted by syntax and we want to focus on the goals at the moment)
  - akhilcacharya 7y ago
    Oh cool! That makes me quite a bit more interested.
deleted 7y ago
[deleted]
jonstaab 7y ago
Very curious about the data migration piece. It sounds like it's designed to handle mapping a row of type a to a row of type a prime. What about situations like a specific combo of type a and c to b and two different records of type c, and optionally delete a record of type d? We occasionally have migrations that are only representable as a function of several entire tables, and which may even leave gaps, e.g. for domain events that affected the system but weren't logged until a later revision.
dcchambers 7y ago
Very interesting and flips mostly everything about CI and CD on it's head. I am disappointed the blog post doesn't include any screenshots or actual examples of the Dark code/editor working. It's kind of hard for me to wrap my head around how this works at a technical level just reading about it...would like to see some concrete examples. I went over to the website to look for a demo/examples but there isn't anything.
jdtangney 7y ago
One question: Security

pbiggar 7y ago

Author here - happy to answer any questions!

conorgriffin 7y ago
What is Dark’s biggest weakness (excluding the current lack of adoption)?
- pbiggar 7y ago
  Wow, what a start :) Probably the biggest one is that you have to leave your editor. If you've spent 20 years carefully configuring your vim or emacs setup, you're not going to be happy to give that up. Don't have a good solution to this right now either.
  - conorgriffin 7y ago
    Haha figured I’d get in early with a hard question. I signed up a long time ago out of curiosity, it’s been a while since then, looking forward to trying it out
  - Sophistifunk 7y ago
    Over all my years it's become clear to me that this attitude of refusing to give up a specific tool (and it's always something married to plain 1d text) is the biggest thing holding out industry back. I don't know if you can convince people to do it, but I'm glad somebody's trying.
    - imglorp 7y ago
      I'm glad someone's trying also. And the reason 1d text is hard to get rid of is not merely an editor (ie, an "app") but a galaxy of languages, tools, and operating systems built around 1d character streams. Everything from the serial driver to the style sheet can be manipulated with primatives for mapping, filtering, and composing character streams and operations on them. There ARE a bunch of benefits to structured code, for example Docker and npm/pip/etc have recently showed us a hint of what composition of standard components can bring. What might be missing from structured code is the rest of the primitives on ASTs. Once we have all that we can talk about backporting to emacs :)
      - icebraining 7y ago
        All code is structured, otherwise it couldn't be compiled/interpreted. The Go community, for example, doesn't need to store their code in some sort of custom binary format to apply changes: gofix works fine with plain text. Text is just a serialization format for those ASTs, no different than any binary format.
    - jrockway 7y ago
      I think people don't want to leave their editor because navigating and typing code is a large part of programming. If you try some new editor, it's always going to be pretty wonky. For example, I've been using the Arduino IDE recently. I am used to navigating lines with C-n and C-p. It's something I do thousands of times a day. In the Arduino IDE, those keys create a new sketch (something I want to do maybe once a week) and print the current sketch (something I imagine nobody wants to do ever). The UI is just not thought out at all. I don't think I'm being resistant to change. I think their IDE is just garbage. Some people have done new IDEs well, VSCode comes to mind. But replicating 40 years of features is not going to happen overnight, and some of those features are actually important. Figuring out which ones are the key. If you don't hit a developer's critical set of features, they aren't going to use your thing.
    - icebraining 7y ago
      What do you see we could improve if we only left text? I've read the promise many times, but never actually saw any convincing reasons. Text is just another format anyway.
  - oftenwrong 7y ago
    One editor setup cannot satisfy all of the diverse preferences of all "religious" editor users. You have to support significant customisation to "convert" us. Maybe the editor should be customisable via Dark itself? Alternatively, allow us to use our editor-of-choice and provide a language server to facilitate the creation of structured editing plugins for Dark. I would add that most applications, including most non-vi-or-emacs editors and web browsers, do not allow much customisation of how their input is mapped to actions in the software. For someone like me, a person that is a vi die-hard because of its ergonomics, this is absolutely critical. Post-input customisation, no matter how advanced, cannot satisfy this need.
    - pbiggar 7y ago
      > Maybe the editor should be customisable via Dark itself? That's the plan!
djm_ 7y ago
Other languages have large ecosystems of packages that allow users to do common tasks quickly. I can imagine not having something similar would be a sticking point to adoption; how does Dark picture the Dark ecosystem working and competing with other already well-entrenched ecosystems? Thanks!
- pbiggar 7y ago
  There's a couple of ways of handling this. First is that of course we need to create a package manager and ecosystem, and that will take time (we can accelerate it by making a great experience, but haven't gotten there yet). Dark can talk to other HTTP things, so many of the functions that are done by libraries can be done over HTTP. So you can call other services, even if we don't have an SDK for that yet. And from there it should be easy to package it up so there is an SDK. Finally, it's a question of what you're doing. There's many tasks where the disadvantage of not having an ecosystem is outweighed by Dark's other advantages. And as we grow the ecosystem, the disadvantages go away, unlike the status quo tooling.
manoji 7y ago
I am not able to signup for beta with an organization email address. is this intended?
- pbiggar 7y ago
  No it's not! Sorry about that. Email me direct (see my profile) and I'll try to fix it.
the_duke 7y ago
Dark sounds like an interesting concept. I seriously hope though that you will open source the compiler. I can't see Dark going anywhere without a guarantee that you can use it without using the provided infrastructure and that your code will work fine even if Dark goes away or you decide to host it yourself. I certainly won't touch Dark with a 10 foot pole otherwise. And would advise anyone against it strongly.
ToJans 7y ago
I see quite some similarities with low-code systems, 4GLs, smalltalk etc, so AFAIK the market is there... Please take into account that your target audience probably is not on HN. If I would be you, I would target web agencies, that can use the dark platform as a competitive advantage, if your system allows you to build stuff faster... IMO it does not make sense to learn all these new paradigms for a small project, you really need scale for this kind of investment from your customers... Right now it feels like you are building a solution that does not know which specific problem it is solving...
alexashka 7y ago
Hi Paul, How did the decision to create a whole new language take place? Was using existing languages a serious consideration and if so, what were the trade-offs that made you go in this direction? I'm familiar with the basics of back-ends. Let's say you had a language that compiles each back-end point into a binary executable (each api point is a separate program). All you'd need is versioning via directories, symlinks to the active most up to date version and a router that looks up the executables in a dir and routes to them. Making changes would involve changing the code, pressing "deploy" which pushes to github and gets the backend to pull, re-compile the binaries and update symlinks. Given this setup - what does your approach do that's more than what I've just described? I'm assuming there's some fundamental problem my naive approach does not handle that made you go in the direction that you've just described.
- pbiggar 7y ago
  We did of course consider it. I don't think it's possible to use existing languages to do this. Presumably, we'd have to use Javascript as our basis cause everyone knows Javascript. But then we'd have people wanting "Dark for python" and "Dark for Ruby" and "Dark for rust", etc, which making a new language neatly sidesteps. So if we built it for javascript, we'd need to decide how to support all of JS's features. How do we support eval? How do we support Typescript and Flow and ReasonML? Then we'd be stuck with all the JS mistakes: promises and futures and callbacks and object model and null/undefined/0/"" comparison tables. This list keeps going: it would have to use a parser. It would need to support npm, etc, etc. I expect once people see Dark that they'll try to build similar features for existing languages, and I hope they succeed! There's some companies that are doing similar things, and I think the scale of their vision is much lower, partially because they are limited by language choice.
  - alexashka 7y ago
    Hm, your answer is not what I was expecting :) Some follow up questions if you are up for it: What are the fundamental features of Dark that are not possible to add to an existing language? What are the fundamental features of an existing language that are not possible to restrict via enforcing a subset, that are fundamental to Dark? Is Dark a very domain specific language? Perhaps these questions will better be answered by a demo that is coming in September.
leonaves 7y ago
Is it possible to write code in an offline environment (e.g. on a plane)?
- pbiggar 7y ago
  Yes, we'll have support for that, including handling sync conflicts when you get back online. It's not gonna be a great experience since Dark is really really designed for working on production, but it should work ok.

deleted 7y ago

[deleted]

devcriollo 7y ago

They show very little, I'm anxious to be able to see.

pbiggar 7y ago
We're talking about the design of Dark in these posts. We plan to show what Dark looks like in our launch in September.

sddhhanover 7y ago

Can I take Dark out for a spin?

pbiggar 7y ago
Right now, we're in private alpha. If you have a product or service that you want to build, fill out https://darklang.com/subscribe https://darklang.com/subscribe and our CEO Ellen will get back to you to discuss and schedule an onboarding. We're not doing tire kicking yet though :)
- sddhhanover 7y ago
  Entered my email awhile ago... just updated the mailchimp profile (daniel@orthly.com) to include my use-case. Let me know! Would love to provide feedback - what you're working on sounds awesome

davejstl 7y ago

What are the server/OS/cloud prereqs for Dark?

pbiggar 7y ago
There are none! We run the infrastructure in the cloud (we run it on GCP using Postgres and k8s, but will likely go multicloud at some point). Most of the advantages we discuss in this post come from the fact that you're using the Dark editor, language and infra. I don't believe this would be possible with a self-hosted solution.
- Deimorz 7y ago
  So anyone using Dark is completely locked-in, to the point that they can't even take their code into an editor other than yours? It's a proprietary language, that can only be edited in a proprietary cloud-based editor, that only runs on proprietary infrastructure?
  - tutfbhuf 7y ago
    Yes.
  - pbiggar 7y ago
    Yes, that correct. There are lots of downsides to this, but you also get a lot of things that we don't think are possible without it.
    - nikolay 7y ago
      You're wrong. Everything is possible as open-source. Often the lock-in is not an option. Not at any price. I hope you realize that soon.
      - mhenrixon_ 7y ago
        ”You’re wrong” is a sure way to get people to just walk away from a discussion.

voidr 7y ago

> When you write new code in a function or HTTP/event handler, we send a diff of the Abstract Syntax Tree (the representation of your code that our editor and servers use under the hood) to our servers, and then we run that code when requests come in. You can achieve the same result with PHP and FTP, however once you start adding unit tests and branching, that's when things start to take time. Long builds aren't necessarily a problem if they give you enough in return.

pbiggar 7y ago
You're exactly right that this style of development would be completely unsafe using PHP+FTP, which is why we had to include all the other stuff: the feature flags, function versioning, DB migrations.
- voidr 7y ago
  What I was pointing out that hitting save and getting a result on the live server was achievable in the 90s, there is nothing new about it. The reason we stopped doing that is because we wanted version control, we wanted to work in teams and we wanted unit and integration tests, which I doubt that you could run within 50ms. > the feature flags, function versioning I can easily replicate that with at least early 2000s PHP(would probably work with 90s PHP as well). > DB migrations We do have DB migrations in all web frameworks that are worth using.
  - pbiggar 7y ago
    > The reason we stopped doing that is because we wanted version control, we wanted to work in teams and we wanted unit and integration tests, which I doubt that you could run within 50ms. A good way of thinking about Dark is to suppose that instead of stopping doing that, we found a way to make it work. > I can easily replicate that with at least early 2000s PHP(would probably work with 90s PHP as well). If you could replicate it, you'd do it. Except you need version control, unit tests, etc, etc. So you need tooling that has those things to make the feature flags work.

nlawalker 7y ago

Sounds cool, seems like an interesting set of tradeoffs that nobody else has tried yet. From what I've read so far it seems like Dark is a good fit for prototyping, experiments, and small products and teams that "move fast and break things". Would you consider it a good fit for large teams and applications, where different people own different components and reliability is the most important metric? A lot of the accidental complexity in the "standard issue" continuous deployment process detailed in the post is there to prevent accidents and enforce permissions.

pbiggar 7y ago
Yeah, all this stuff is actually designed for large teams (though, it'll be a while before Dark is production-ready enough for them). CD matters because it allows you deploy in small chunks which is much less risk than deploying in big chunks. The feature flag is a thing that you might not need if you're small, but it completely changes how large teams work. Same with DB migrations: if you have a small amount of data you can just do a schema migration and the locking wont affect you too badly. We'll have a permissions model at some point that will take care of the ownership problem. We designed Dark to allow extremely fine grained ownership, for example allow a contractor access to a single HTTP route or function. Overall, this is kinda similar to how we went about building CircleCI. We focused on good tooling for companies (eg parallelization and speed) instead of individual projects (eg build matrices).
lmilcin 7y ago
The way to go is to have high-trust culture and comprehensive automated testing system. You want developers to behave responsibly and to put all development effort into writing good, reliable code in the first place. I personally prefer pair programming over code review. Code reviews have a lot of downsides but two most important are that you can't get from commit to prod automatically (as there is another inevitable manual step) and that reviewer typically doesn't have enough background/time to perform the review in meaningful way.

ricardobeat 7y ago

EDIT: Ha. My apologies as I just realized my stupidity - I was browsing on mobile and it turns out out I jumped into the 'What is Dark' post [1] and never finished the original! The mailing list links are in that one - I subscribed. This is a lot more interesting, cheers :) [1] https://medium.com/darklang/the-design-of-dark-59f5d38e52d2 https://medium.com/darklang/the-design-of-dark-59f5d38e52d2

pbiggar 7y ago
The section that describes how it does it in 50ms is called "#deployless". That describes the mechanics of the deployment. How we made deployless possible is addressed for the rest of the post after that. (The mailing list thing is weird, I only see that link at the bottom. Can you point me at the ones you see in the text?)
- deleted 7y ago
  [deleted]

adrienjt 7y ago

How coupled is Dark with the Dark language? I understand that the purely functional nature of the language makes its integration with the rest of the platform easier, but the blog post doesn't go into much detail about that. What would it take to integrate Dark with: a) another (existing) purely functional language, and b) a general-purpose language, e.g., Go? Could a code intelligence tool like Sourcegraph help?

pbiggar 7y ago
Extremely tightly coupled. That's kinda the point "what can we do if we don't have to couple dark to anything else" (eg editors, other languages). I can't imagine how you'd couple Dark to another language, but I also don't see the point?
- adrienjt 7y ago
  Thank you. My point was risk mitigation, which I discussed in another reply (which you answered: https://news.ycombinator.com/item?id=20399267 https://news.ycombinator.com/item?id=20399267 )
  - pbiggar 7y ago
    Yeah, the bigger picture here is that there's a ton of valid reasons why people can't use Dark right now. Of course there is, we're a small team with an early product. Once we show that this is really a better/faster/easier way to make backends, then we can start working on handling the risks and constraints of more teams.
    - adrienjt 7y ago
      That makes sense. It does look like you're very close to that first objective though. Congrats! Your blog post very clearly describes what Dark actually is (which was still unclear before I read it).

iLemming 7y ago

I've been seeing a lot of talks about darkland and can't find a single example of code. Can someone show me at least how does it look like?

gspetr 7y ago
from darklang.com: "Dark is currently in private alpha." That's probably the primary reason why there's little to no info.
- amelius 7y ago
  They are leaving us in the dark, so to speak.
  - quickthrower2 7y ago
    It feels like an ICO launch in that regard. Kind of exciting!

z3t4 7y ago

Syntax errors are the easiest errors to fix and almost never get into production. Zero downtime deployments are very rare, and ads a lot of complexity, so it's a big trade-off, compared to just uploading the new code and restarting the service. And having the front-end deal with minor hiccups like service restarts, because users will have such issues all the time due to being on mobile networks, trains going through tunnels etc. You can do as much testing as you want, both manually and automatically, but you still can't detect all issues as efficient as thousands of users in production. So just accept that there will be issues, and instead design your pipeline so that those issues can be fixed within minutes.

quickthrower2 7y ago
> You can do as much testing as you want With what Dark is promising, I believe testing would be easier. Deploy to a test environment in 50ms. Then run a suite of tests, then deploy to production in 50ms. There could be some complementary tech that figures out what tests need to be run based on what you change to get the time down.
- heavenlyblue 7y ago
  They're kind of lying. You only need pip or any package install if your testing image doesn't pre-install the required packages before running the tests. Which I don't myself since I always want to see if my packages work with the latest versions of other packages - and Dark wouldn't be able to handle that anyway. So my test runs actually complete in about 50 ms if I'm not testing any external API functionality. They will also never finish earlier than 5 minutes if I am, even on Dark - since they're based on external services.
snypox 7y ago
> Zero downtime deployments are very rare What do you mean? Rolling deployments are a very basic feature of container orchestrators.
- nijave 7y ago
  You don't even need containers--it's a feature of load balancers containers rely on. Start new container, wait for it to become healthy, add it to the load balancer, remove old container (with an optional, let connections to old container close first)

koliber 7y ago

It's an interesting concept that flips many things on its head. We're getting some real hard-to-achieve benefits out of the box, while sacrificing a lot of things we used to take for granted. One thing that I notice is that Dark requires a completely walled in environment to be able to deliver all of its promises. In reality, any real-life project has to interact with external services, legacy systems, or custom peripherals which are unlikely to be based on Dark. How will Dark be able to handle such cases? Current tooling and processes are able to cope with such things through custom, hand-crafted, and as the article stated, often fragile, systems and processes. The are a pain, but they get the job done. They are very versatile by their nature. While they break more often that we would like to, they can be patched up well enough to band-aid over rough patches and adjust to surprising externalities. Dark seems to have one way of doing things, and I am curious how compatible that way is when it comes to interfacing with external systems.

btown 7y ago
It seems that Dark is designed to allow you to wrap external APIs in easy-to-use interfaces, so it should be at home as part of a larger SOA or ecosystem, of course with the caveat that it's still being developed. See the footnote at https://medium.com/darklang/dark-residency-7dbc878a5f0a https://medium.com/darklang/dark-residency-7dbc878a5f0a .
- pbiggar 7y ago
  Yes, that's exactly right. Dark speaks HTTP (and at some point, will speak GRPC, Thrift, etc) so you can access 3rd party services over HTTP (and wrap them in SDKs in the Dark package manager, same as on npm). We expect we'll have some built-in support for common things (resizing images and videos, or generating pdfs for instance; we already support static assets on a CDN).
  - simonebrunozzi 7y ago
    Hey Paul, nice to see you here on HN. Very promising start! What's the best way to contact you? (I'm $HNusername @ gmail).

quickthrower2 7y ago

I think it is fantastic that this kind of innovation is happening. I get sick of waiting for stuff at work, some kind of build or CI thing. npm i, docker build or whatever else. I hope the language experience is good, and it's a reasonable language (static typing!). It sounds like taking the Go idea of quick builds to the next level.

davedx 7y ago

How does it compare with hot swapping in other languages and platforms like Erlang, JVM/.NET hotswap technologies? I'm guessing the built-in safety functionality from the feature flagging is one thing?

pbiggar 7y ago
It's much simpler. Those have complex in-process hot swapping. Dark's "hot swap" is a write to a DB, then new HTTP requests use the new code. As we grow and start to look at compilation and optimization, I think we'll probably get much fancier, but for now we're very into keeping things super simple.
- davedx 7y ago
  I'm pretty interested in this. Signed up to the mail list. Good luck!

bufferoverflow 7y ago

Code deployment is almost always trivial and fast. It's the databases that are slow and very tricky, especially in distributed systems.

unfunco 7y ago

It sounds like a convoluted version of Geocities.

GordonS 7y ago

As an aside, that big list of tasks and diagram at the end are perfect to show next time a client exclaims about how long it takes to get a simple change into production!

kccqzy 7y ago

Twenty years ago, deployment of a new version of a web app means FTP'ing a new php file to the server.

eloisant 7y ago
Well, you're free to deploy like this (but you probably know why it's a very bad idea).
- somehnguy 7y ago
  Feel free to explain why
  - adrianN 7y ago
    Auditing what was changed and which version is currently in production is a bit harder than it has to be.
  - stabbles 7y ago
    Yet another reason is FTP turned out to be inherently unsafe.
    - jszymborski 7y ago
      SFTP, however, has FTP feature parity and is secure as SSH (since it is ostensibly SSH).
  - viraptor 7y ago
    Copying of multiple files is not atomic. You need to ensure a file with func1 exists before another file calling func1 gets copied. You can do it atomically by having 2 directories and swapping a symlink, but... that's a scriptable pipeline at that point.
pikzel 7y ago
Which wasn't a bad thing after all. I've had more incidents with deployment pipelines than with the actual code.

blumomo 7y ago

That's quite a dark name for such a bright idea.

oftenwrong 7y ago

Dark sounds like it would satisfy my preferences well. - statically typed, functional language with no null pointers - safe deploys and rollbacks - structured editing / version control - minimal infrastructure management ...etc. A tool that makes writing a correct program, and deploying it without catastrophe or fuss, easy. I look forward to reading more.

bullen 7y ago

I added the same feature to my Java app server: http://github.com/tinspin/rupy http://github.com/tinspin/rupy it's a simpler design which compiles, zips and hotdeploys over HTTP to a classloader. But it's fast enough as long as you keep each project small.

juandazapata 7y ago

Given how tightly coupled is Dark with its IDE and deployment environment, I'm interested in knowing how much VC money have they taken? What if they get acquired or simply shut down? It looks like there won't be a migration path, and you'll need to rewrite your app from scratch using a different language. Again, it would be nice some transparency around how much VC money they've taken.

andy_ppp 7y ago
For there to be 9 of them and 4 open heads I'm guessing it's probably $10-20 million at this stage. I think the only way I'd use this is if I had the option to host it myself i.e. the code was all open sauce.
pbiggar 7y ago
The sustainability of Dark is super important to use. We're not fucking around with customers' livelihoods. For our early customers, we have a handshake agreement that we won't leave them hung out to dry. If Dark was to shut down, we would 100% make sure they had the time and ability to port their app off. There would be a time cost to them of course, so it's not perfect. As we get closer to opening up to the public, we'll be doing a bunch of work on the sustainability question so we have good answers to the "acquired or shutdown" problem. We imagine a legal framework to protect customers, but haven't got the details worked out.
- londons_explore 7y ago
  A legal framework doesn't help much in the "bankruptcy" case. Unless that legal framework means all IP becomes open source and people can self-host I guess.

deleted 7y ago

[deleted]

sciurus 7y ago

How does Dark handle code review? Even if I was willing to accept the other tradeoffs that come with Dark, I'd want another human to approve my changes before that 50ms deploy to production kicks off.

jmull 7y ago
I don't think they've detailed that yet, at least not that I've seen. But it is one of the few steps the Dark deployment process retains (see the diagram near the end of the article), so it seems they will support it (hopefully reasonably well).
- londons_explore 7y ago
  Seems easy enough... Write whatever code you like, directly in production, but feature flags will be forced to zero (or to just your own test users) until code review passes... It requires the language to have a bulletproof feature flag system which can't be bypassed, which it seems is the goal.
pbiggar 7y ago
The model is quite different to how people write code today. Instead of a process that takes code from your machine and sends it to prod (and thus has a lot of risk), you code in a sorta sandbox _in prod_. That is then enabled for users via changing the feature flag setting. So to do code review, you write the code behind the feature flag, then ask someone for code review. (There's google-docs style collaboration, so they can see your code if you permit it). After code review, you can "merge" by toggling the feature flag, or changing the setting to "let 10% of users see this" or whatever roll-out strategy you want. just safely. So you're writing code behind a feature flag (but "in prod"), but no users can see it.
- dkordik 7y ago
  cool- I guess there is some concept of a snapshot/diff to show you the before and after?

keymone 7y ago

The problem is granularity of deployment - the smallest piece you can rollout is, these days, a container with code and dependencies, that has to be build and threaded through your CI/CD stack before it hits prod. I like to imagine what would need to happen to reduce(increase?) that granularity to single function - you change the code, byte-compile it, send that blob to your backend’s hot code swap port and voila - you’re done. A lot has to happen behind the scenes though - a change in dependencies would still require a larger deployment or much smarter code swapping procedure, caller/callee tree would need to be considered, functions need to be pragmatically pure (talking to dB is fine, being part of an “object” - not really), functions need to be extensively specced, etc. But that would be my backend holy grail.

deleted 7y ago
[deleted]
icebraining 7y ago
What are you looking for, that isn't provided by platforms with hot code-swap (e.g. Erlang) or FaaS systems (e.g. AWS Lambda, OpenFaaS) ?
- keymone 7y ago
  you're thinking of only surface functions/endpoints, those that are invoked directly from your api route dispatcher. i'm talking about any function in your source code of a large backend with many endpoints. think of function `+` - i want to be able to deploy a change to that function such that no worker process gets respawned. also imagine every time you call `+` in your application - it goes throught the whole FaaS shenanigans with serializing, sending rpc over wire, receiving response, deserializing, dealing with throttling, errors, backoffs. all of that is ridiculous for `(+ 1 2)`. and yes, erlang is a good example of an environment where this could work.

toolslive 7y ago

Back in the day (early 00s), you could do something similar with ZOPE: develop on a production server, in a private session and you could mock about with your version of the code, and when you were satisfied push it through for everybody else. This was because the database that was holding the users, permissions, ... was also holding the code. It was lightyears ahead of J2EE at the time. (Now I wonder what happened to it)

blunte 7y ago
I know that Plone, which is built on Zope, found its place in a lot of universities and governments. So I imagine Zope lives on with necessary support from its big users.
- slinkp 7y ago
  Yup. Plone is still pretty active https://github.com/plone https://github.com/plone
slinkp 7y ago
Former professional Zope guy here! (I made a living building things with it in the early-mid 2000s). What happened to Zope? Well, Chris McDonough (creator of Pyramid, and Zope veteran) blogged about this in 2011 and from my perspective he got it exactly right. I still think this history is fascinating and I wish more people knew it. https://web.archive.org/web/20161019153348/http://plope.com/Members/chrism/in_defense_of_zope_libraries https://web.archive.org/web/20161019153348/http://plope.com/... After the history lesson, some of his "lessons learned" bullet points seem very apropos in the context of Dark, particularly the first two: * "Most developers are very, very risk-averse. They like taking small steps, or no steps at all. You have to allow them to consume familiar technologies and allow them to disuse things that get in their way." * "The allure of a completely integrated, monolithic system that effectively prevents the use of alternate development techniques and technologies eventually wears off. And when it does, it wears off with a vengeance."
- pbiggar 7y ago

KirinDave 7y ago

So... you're 50ms away from a total global outage and/or major feature fault? Why would I want that? How do I control it for green-blue? For rational isolation? Is there native support for canary releases?

coldtea 7y ago
Yeah, because speed of deployment prevents CI/tests/checks /s (Seriously, how do people come with such facile counter-arguments?)
- KirinDave 7y ago
  How does CI actually prevent these kinds of outages? Did CI save Google from the networking outage or tests prevent a global Cloudflare outage? Do you think those organizations don't spend enough energy for the average SV commute on test runs per release? Are you seriously going to pretend that those efforts "solve" reliability and stability in any non-trivial system? It's actually not hard at all to write fast software deployment. Many people do it accidentally. But to tie it directly to editors and stovepipe the infrastructure around testing, that seems to me like a great business model that any rational engineer interested in reliability should immediately reject unless it's coupled with open source, statistically valid canary tooling at the bare minimum. What you actually want at that degree of universal knife-switching is rollbacks.
  - penagwin 7y ago
    > Did CI save Google from the networking outage or tests prevent a global Cloudflare outage? This is not a fair statement. This is like saying "Did seatbelts save the victims who died in car accidents X/Y/Z?" Just because your tests/ci/etc. can't catch every single issue that could go wrong doesn't make them useless. Just because people die every single day in car accidents doesn't mean car's safety features are useless. You're correct in that a CI should not in any way be considered a "perfect solution for outages", but an automated system that tests, deploys, etc. in a predictable and (again) testable manor is at least helpful.

umvi 7y ago

Very interesting, but I don't like the idea of abandoning VSCode to learn some single-purpose IDE

vinceguidry 7y ago

I'm getting really tired of learning new languages / stacks every few months. I want one language to write things in. It has to be a pure-OO language and right now that's ruby. I want to edit that code with one code editor. Syntax awareness is a nice-to-have but not essential. At the moment my editor of choice is Sublime Text 3 but to extend it I need to write python which is unideal. When I start writing websites it'll be hyperstack. Unless your new language can fix all my issues, be open source, and be my One True Human Interface to All Things Machine, I'm going to keep building on ruby, with occasional bash / crystal / rust excursions for speed. Is your language pure-OO? Do you have gradual typing? Does it run on a common VM? Am I going to have to rewrite everything I'd ordinarily bring in a library for? I can accept every single one of your claims but still not be interested in learning a whole new stack because I can already deliver stuff quickly enough with the one I've been iterating on for a decade. Especially when it could end up orphaned in 3 years if it doesn't get enough adoption. I'm not saying your holy grail can't be holy enough. But it has to be pretty darn holy.

taosx 7y ago

Marketing piece with grand plans...not really I would argue with you that this is not such a priority for a software development organization, there are more important things to do. You got so excited...so fast, I think this community may be doing more bad for me than good lately.

vincentmarle 7y ago

This sounds extremely exciting and it looks like a lot of thought has been put into this. I’m looking forward to try it out. A lot of comments here are coming from an anxiety that is caused by many years of dealing with fragile deployment processes, but I like the boldness of Dark to just bypass all that and make the fundamentals safe and solid.

burlesona 7y ago

First I’ve heard of Dark. The blog post is cool, but reading it I felt two main things: (1) no pull reviews would be terrible for a team, and (2) everything being instantly live in prod and automatically versioned etc sounds like a nightmare once you move beyond a simple program. Reading on Darklang.com I see they kind of address this. The ecosystem they’re building is training wheels for developers, a way to make “coding 100x easier.” I can see that working in a sense, and being an entry point that a lot of people use to make neat toys. I could even see it being a gateway that people use to get interested in and learn about programming. But I can’t imagine wanting to build a business with more than one programmer, or any kind of scale on a completely black box system like that. I will also say, there’s a problem when it comes to entry-level systems, trying to teach people to code: The struggle with the complexity is actually important. Dark isn’t actually making all the work of building and running a web application go away, it’s abstracting it all into a platform such that you can’t see it. Suppose a person gets into coding self-taught and learns to work this way. The knowledge isn’t going to be very transferrable - ie when they look at other languages or systems they’re likely to struggle with problems like “what does prod versus dev mean, I just want my program to run for the world...” You usually move knowledge between ecosystems by translating “I did it this way in (toolset A), so what’s the parallel in (toolset b)?” The more you have an idea of the underlying principles the easier this is to figure out. That said, Dark certainly looks neat, and I imagine the implementation is quite cool. The only other nit I’d pick is the name. To me, its not really a language, but maybe a development environment, or framework. I suppose it had a language in it, which I’d probably call DarkScript or something.

pbiggar 7y ago
Author and Dark CTO here. > Reading on Darklang.com I see they kind of address this. The ecosystem they’re building is training wheels for developers, a way to make “coding 100x easier.” No no no no no. This is not what we're doing. We're building something for experienced software engineers. (New developers will be able to use it too and it should be much much easier for them than learning in other ways, but that's a secondary audience). > Dark isn’t actually making all the work of building and running a web application go away, it’s abstracting it all into a platform such that you can’t see it. Some of it is being abstracted, but a lot of it being properly removed. Servers are abstracted, but almost all of actual deployment is actually being removed.

module0000 7y ago

Hrm...we don't deploy code in to prod in 50ms, but we certainly deploy to dev/staging/testing in <1s, and promote to prod once we are 100% sure we want to do so. We use the boring, industry-standard tooling and process everyone else uses (Jenkins, k8s, Ansible, nothing bleeding edge). I'm unsure what advantage their product has that makes it stand out among the battle tested tools already in use. Would someone elaborate? disclaimer: working for a l2 network provider, and a mistake costs us infinitely more than an "Oh we're sorry!" can fix.

ukd1 7y ago
The point is all that stuff (jenkins, k8s, ansible) is accidental complexity that, I presume someone at your company has-to/had-to think about. I think Dark's angle is it's stuff you shouldn't need to think about in order to solve the actual problem, and removing as much of that as possible is better.
- deleted 7y ago
  [deleted]
- alexandercrohde 7y ago
  Okay, I mean conceptually cool, but it's not like an existing startup gonna throw away all their code and rewrite it to save on dev-ops time. I guess the value prop is that if you're starting from scratch, and want to write your whole app in Dark then you can avoid hiring a dev-ops engineer and iterate faster? Seems like a stiff trade-off. If it's an interpreted language you're coming from, setting up envs can be < 30 secs anyways (and same with prod deploys), so that's faster than you need. If it's a compiled language you're coming from (e.g. scala/c++) then it's likely Dark may not match your needs. Plus I guess this assumes every team in your company is using only dark? Like what if it's a SOA and a full environment has a node component?

akhilcacharya 7y ago

Is Dark intended to be a Lisp? I'm surprised by the fact that the team didn't include language examples (from what I can gather)

pbiggar 7y ago
No, it's an ML. (You're right that we haven't published samples, we think people get distracted by syntax and we want to focus on the goals at the moment)
- akhilcacharya 7y ago
  Oh cool! That makes me quite a bit more interested.

deleted 7y ago

[deleted]

jonstaab 7y ago

Very curious about the data migration piece. It sounds like it's designed to handle mapping a row of type a to a row of type a prime. What about situations like a specific combo of type a and c to b and two different records of type c, and optionally delete a record of type d? We occasionally have migrations that are only representable as a function of several entire tables, and which may even leave gaps, e.g. for domain events that affected the system but weren't logged until a later revision.

dcchambers 7y ago

Very interesting and flips mostly everything about CI and CD on it's head. I am disappointed the blog post doesn't include any screenshots or actual examples of the Dark code/editor working. It's kind of hard for me to wrap my head around how this works at a technical level just reading about it...would like to see some concrete examples. I went over to the website to look for a demo/examples but there isn't anything.

jdtangney 7y ago

One question: Security