4 ms·

We're essentially in agreement--backend development should be the easy part, but it often ends up being needlessly complicated and broken. So many backend syst

We're essentially in agreement--backend development should be the easy part, but it often ends up being needlessly complicated and broken. So many backend systems are made up of the same basic components, but they end up being rewritten all the time because they don't work well together or are poorly documented. Treeline aims to make that 80% of your app that's always the same and basically automate it for you, leaving you to focus on the creative part (the special sauce that makes your app great).

findjashua 12y ago
I laud the effort of making it easier to build apps, but in my experience the backend stuff is mostly building the rest api and cron jobs, which are relatively straightforward. Imho, bulk of the complexity is in user interfaces with lots of moving parts.
- mikermcneil 12y ago
  You're absolutely right. Which is why we should make the backend even easier :)
- jmspring 12y ago
  Reading this, I am curious about the systems you've dealt with? If you are talking the back end basically being crud apps, sure, but most services, the bulk of the work is behind the scenese (rest apiece, etc)
  - findjashua 12y ago
    outside the rest api, mostly crawling 3rd party apis, basic map-reduce aggregations, data sanity checks, the occasional s3 dump. No fancy distributed systems or machine learning stuff everyone else seems to be doing. How about yourself?
- onion2k 12y ago
  It's trivial if your app is just CRUD with domain specific fields, but there can still be some tricky parts; * Sync'ing data between the frontend and backend if your app has any sort of offline capabilities. Conflict resolution is hard if you want anything better than just a dumb overwrite of what's there. * Data munging for anything that isn't coming from a well defined API is a horrible and difficult task. * Scaling and replication on a global scale is still a challenge. It doesn't affect many apps though. You can knock up a decent REST API in Laravel/Express/Django/whatever in a few hours, but getting the data that you push through it still represents some fun and challenging aspects of app development.
  - findjashua 12y ago
    Agree, those are all challenging problems to solve, but I haven't had the chance to work on them 1. we don't support offline mode in our app. Though, quite a few databases have a version number in their documents (es, couch etc), so the user can be shown conflicts if the server responds with a version mismatch 2. Luckily, I haven't had to work with any overly terrible APIs. The documentation is usually bad, but fiddling around with the req params/body usually gets me what I want 3. We're selling to businesses, so our traffic isn't at that scale.
- lazyseq 12y ago
  I build whole systems, and have had specialized jobs ranging from 3D game programming to Machine Learning to Rapid Web App Dev. I've observed many the opposite. I would agree that interfaces have lots of moving parts, but part of what you try to do there is to reduce the complexity because it often means you are passing on too much to the user. I suppose there's a bit of Apple philossphy there. Just cron jobs and REST APIs? Regarding cron jobs, we generally try to avoid them because communicating with them, getting good information, maintaining, and so on sucks. While I've written tons of cron jobs and still do, it's probably 1% of what I've observed happening in the back-end and usually a sign of a duct-taped system. This excludes baked-in unix tools that are actually supposed to work that way of course, rather I am talking things directly involving apps I build. As for REST API, not everything needs to be or should be REST. It introduces all kinds of extra layers of auth, slowdown, caching issues, and so on. For some domains, REST is a bad idea. Of course we use it where it makes sense, but it's not just some magic thing that is and should be used everywhere and anywhere to be representative of the bulk of back-end. Most REST services are built very quickly, it's rather the logic behind them that takes a long time. As for concrete things I've encountered that are actually complex and happen in traditionally the back-end, I'd say none of these are straight-forward: 1. Caching - As it goes, other than naming, this is one of the most difficult things in computer science and happens everywhere in a good application, front, back, let, right, data, service, whatever. 2. Machine Learning, Sentiment Analysis, Analytics, OLAP, etc. - People forget that a big part of building a system is getting information back out of it. 3. Orchestration - this is hugely complicated. So much so, Microsoft created a very complicated and misunderstood product for it (BizTalk). Doing orchestration is still something no one gets. It happens in millions of forms and related manifestations. 4. Deployment - Versioning, machine provisioning, cluster management, and so on are all very complicated. There are tons of these things that fall under what many people call Dev Ops now too as well. I have yet to see magic here in any app. I challenge you to deploy a running system that is accepting live data transactions, then seamlessly roll it back without data loss, down time, corruption, weird states, unforseen bugs, and so on. No one has done this in full which is why this area is becoming increasingly full of solutions and start-ups. Whether it's Google App Engine, Azure, Heroku, your in-house AWS scripts, Chef, anything, I am telling you no one really does this 100% right. Supporting a real app is just as important as the app itself, and often the two have to communicate and be written in ways to make this work. 5. Streaming - Building streaming related back-ends is anything but simple and plug-and-play. No, I am not talking about streaming Tweets. 6. Data Migrations/Updates/Changes/Flow - Very important part of dev, and usually one of the top 5 time sinks. Complete nightmare and again products and libraries that help, but no magic bullets to be found here other than doing the work and putting in the time. As for simple cases, almost everyone deals with getting data in X form, transforming it to Y, and having it change form again after getting a reply from Z service. A huge part of people's time is simply manipulating data structures, at any place in the system and a reason why I tend to favor Clojure for web apps that don't need to be speed demons. As for migrations, ask someone to do one, then double or quadruple their estimate, sit back, and watch. 7. Authentication/Authorization - The world runs on more than just OAuth and Social Auth. Try dealing with people that have secure systems or many other systems that you need to communicate with, especially across network boundaries. 8. Legacy Integration - Happens all the time. Especially when you build a shiny new app very fast, then have to roll out v2 that actually scales. Beyond that, try any business that has been around 5+ years and actually makes money to see just how horrid this can be. 9. Data modeling - If the scenario of ABC person created XYZ app which now struggles to do Foo new thing without huge pains, you haven't worked on apps long enough. Data modeling mistakes and limitations are one of the single biggest problems and time sinks. More obvious examples include simple facts like having to dupe or dump data into other places to consume it the ways people actually want. While your shiny normalized tables with a REST service might work well for returning a simple list of customers, as that list scales and you realize you now have a new place where you need 2 fields from here, 8 from there, and so on and it needs to happen in 100ms or less in total back to the user, then you are in for a treat. 10. Logging - in big systems, the code here can be non-trivial. Often this means more than writing to a log file, but rather providing visibility into parts of the system in real-time or close to it. 11. Scaling - Everyone thinks they are doing this right until they don't do it right. This can involve anything from servers to message transport format to compression to memory management to clustering. We could do this all day, but hopefully this makes a point.
lazyseq 12y ago
Not trying to pick a fight, but again your team comes off as naive and essentially gives the middle-finger to probably the majority of the software developer community with comments like these. It seems like you guys simply have not worked on an actual complex system of any significant size or over-value the complexity of front-ends. I personally respect all facets that go into making software. Anyway, I would argue there's no easy part and dividing things into monoliths at a conceptual-level is a mistake. Putting that aside, whatever part is comparatively (key-word) easier is highly dependent on the implementer. For instance if I build a system that just needs 1 input field, but then performs some sort of machine-learning or triggers some complex background dispatching, then that system is arguably much more complex on the "back-end." I think making big distinctions is a source of trouble to begin with and regardless I do about equal amounts of front and back-end work on my current start-up. As for the special sauce, in many apps the special sauce is actually the back-end. I would agree everyone hates with dealing with 3rd party libs and so on, but the ways you deal with that information is important. I think the way Treeline is described in your website, this article, and video makes weird distinctions and conflations between what is front and back-end work and what are actual problems programmers actually have. I totally get the parts about plugging into another API and not having to write the fetch code, but honestly with a library or two or even implementing integrations with many libraries, the work is minimal. Adding the correct libraries is often an important choice. To give you a concrete example, I do most of my current work in Clojure followed by Haskell, Java, and C. The choice of library authors doing things like using non-standard data structures that I then have to transform or directly consume, using/not using things like async channels, and so on from the ends of their APIs is a huge deal. It's not normally a problem to find a library that works with how you want to do things, but rather important that you pick the right one from the list of choices. If a Clojure library forces me to consume a core.async channel, I might not do it because I want to use something more general like a transducer to abstract that away. Sometimes also the ways that I consume someone else's library are very unique to my app and provide a crucial difference aka special sauce. Consider for example that for years I had a client that had a huge advantage due to code my team created against some public APIs because we were able to stream, batch, and thread our interactions with this particular API in ways most people had not thought of or implemented. The fact that our integrations were based on self-calculated heuristics provided dramatic speed-ups over everyone else meant it looked like we could do things that would cause the client's competitors to think we had some special deal or wonder how it all worked. This is not unique to us, rather it is simply illustrative of the importance of optimizing your business activities in the back-end. Consider that most of wall-street high frequency trading as another example is based on this notion. While I am not claiming that everyone needs to do this, your generalization is the issue here. It is funny because maybe our definitions of "back-end" are the issue. I tend to notice that the wheels on the front-end are invented time and again. How many different UI frameworks, patterns, controls, etc are there that get implemented from scratch? How many people end up composing their apps out of little more than paging, showing pictures, textboxes/checkboxes/pickers, form validation, css positioning, animation, and so on. I'd argue these are the actual repetitive bits that are tedious to write. I am reminded of criticisms of forum software by the authors of discourse. They pointed out a lot of the special sauce things that were wrong with forums, but ended up pissing off a lot of potential users who thought at least the versions in the past few years were essentially a lower functionality version of the same things with fancier loading and scrolling using JavaScript and JS templating. And the critics were at least in part right because not much new was brought to the table beyond looking and feeling slightly better on some basic elements, but the nature of forums software as we know it was hardly shattered so far. It seems to me what you've noticed is that a very particular segment of node.js users who used sails or were involved in the node.js ecosystem were reinventing the wheel or didn't understand the wheel. This is fine as you clearly are trying to address this problem, but you are also making blanket statements about software development. Everyone I have worked with who I consider to be far smarter than me would agree that a common mistake a lot of people make is they think any part of software development can just be replaced by a magic wand wave and that xyz part of the system is just stupid, tedious, and simple. It's like that time before you reach the point in programming where you realize it is not that your code is awesome, it is merely that it sucks slightly less than the next guy's code due to your experience. You guys really need to focus on the concrete things the product does not on philosophical nonsense of magic software development. Presently it just seems like you've made it easier to wrap some node.js APIs, call them in an opinionated way quickly, and slap a visual front-end that includes some canned forms and an impractical editor on top.

findjashua 12y ago

I laud the effort of making it easier to build apps, but in my experience the backend stuff is mostly building the rest api and cron jobs, which are relatively straightforward. Imho, bulk of the complexity is in user interfaces with lots of moving parts.

mikermcneil 12y ago
You're absolutely right. Which is why we should make the backend even easier :)
jmspring 12y ago
Reading this, I am curious about the systems you've dealt with? If you are talking the back end basically being crud apps, sure, but most services, the bulk of the work is behind the scenese (rest apiece, etc)
- findjashua 12y ago
  outside the rest api, mostly crawling 3rd party apis, basic map-reduce aggregations, data sanity checks, the occasional s3 dump. No fancy distributed systems or machine learning stuff everyone else seems to be doing. How about yourself?
onion2k 12y ago
It's trivial if your app is just CRUD with domain specific fields, but there can still be some tricky parts; * Sync'ing data between the frontend and backend if your app has any sort of offline capabilities. Conflict resolution is hard if you want anything better than just a dumb overwrite of what's there. * Data munging for anything that isn't coming from a well defined API is a horrible and difficult task. * Scaling and replication on a global scale is still a challenge. It doesn't affect many apps though. You can knock up a decent REST API in Laravel/Express/Django/whatever in a few hours, but getting the data that you push through it still represents some fun and challenging aspects of app development.
- findjashua 12y ago
  Agree, those are all challenging problems to solve, but I haven't had the chance to work on them 1. we don't support offline mode in our app. Though, quite a few databases have a version number in their documents (es, couch etc), so the user can be shown conflicts if the server responds with a version mismatch 2. Luckily, I haven't had to work with any overly terrible APIs. The documentation is usually bad, but fiddling around with the req params/body usually gets me what I want 3. We're selling to businesses, so our traffic isn't at that scale.
lazyseq 12y ago
I build whole systems, and have had specialized jobs ranging from 3D game programming to Machine Learning to Rapid Web App Dev. I've observed many the opposite. I would agree that interfaces have lots of moving parts, but part of what you try to do there is to reduce the complexity because it often means you are passing on too much to the user. I suppose there's a bit of Apple philossphy there. Just cron jobs and REST APIs? Regarding cron jobs, we generally try to avoid them because communicating with them, getting good information, maintaining, and so on sucks. While I've written tons of cron jobs and still do, it's probably 1% of what I've observed happening in the back-end and usually a sign of a duct-taped system. This excludes baked-in unix tools that are actually supposed to work that way of course, rather I am talking things directly involving apps I build. As for REST API, not everything needs to be or should be REST. It introduces all kinds of extra layers of auth, slowdown, caching issues, and so on. For some domains, REST is a bad idea. Of course we use it where it makes sense, but it's not just some magic thing that is and should be used everywhere and anywhere to be representative of the bulk of back-end. Most REST services are built very quickly, it's rather the logic behind them that takes a long time. As for concrete things I've encountered that are actually complex and happen in traditionally the back-end, I'd say none of these are straight-forward: 1. Caching - As it goes, other than naming, this is one of the most difficult things in computer science and happens everywhere in a good application, front, back, let, right, data, service, whatever. 2. Machine Learning, Sentiment Analysis, Analytics, OLAP, etc. - People forget that a big part of building a system is getting information back out of it. 3. Orchestration - this is hugely complicated. So much so, Microsoft created a very complicated and misunderstood product for it (BizTalk). Doing orchestration is still something no one gets. It happens in millions of forms and related manifestations. 4. Deployment - Versioning, machine provisioning, cluster management, and so on are all very complicated. There are tons of these things that fall under what many people call Dev Ops now too as well. I have yet to see magic here in any app. I challenge you to deploy a running system that is accepting live data transactions, then seamlessly roll it back without data loss, down time, corruption, weird states, unforseen bugs, and so on. No one has done this in full which is why this area is becoming increasingly full of solutions and start-ups. Whether it's Google App Engine, Azure, Heroku, your in-house AWS scripts, Chef, anything, I am telling you no one really does this 100% right. Supporting a real app is just as important as the app itself, and often the two have to communicate and be written in ways to make this work. 5. Streaming - Building streaming related back-ends is anything but simple and plug-and-play. No, I am not talking about streaming Tweets. 6. Data Migrations/Updates/Changes/Flow - Very important part of dev, and usually one of the top 5 time sinks. Complete nightmare and again products and libraries that help, but no magic bullets to be found here other than doing the work and putting in the time. As for simple cases, almost everyone deals with getting data in X form, transforming it to Y, and having it change form again after getting a reply from Z service. A huge part of people's time is simply manipulating data structures, at any place in the system and a reason why I tend to favor Clojure for web apps that don't need to be speed demons. As for migrations, ask someone to do one, then double or quadruple their estimate, sit back, and watch. 7. Authentication/Authorization - The world runs on more than just OAuth and Social Auth. Try dealing with people that have secure systems or many other systems that you need to communicate with, especially across network boundaries. 8. Legacy Integration - Happens all the time. Especially when you build a shiny new app very fast, then have to roll out v2 that actually scales. Beyond that, try any business that has been around 5+ years and actually makes money to see just how horrid this can be. 9. Data modeling - If the scenario of ABC person created XYZ app which now struggles to do Foo new thing without huge pains, you haven't worked on apps long enough. Data modeling mistakes and limitations are one of the single biggest problems and time sinks. More obvious examples include simple facts like having to dupe or dump data into other places to consume it the ways people actually want. While your shiny normalized tables with a REST service might work well for returning a simple list of customers, as that list scales and you realize you now have a new place where you need 2 fields from here, 8 from there, and so on and it needs to happen in 100ms or less in total back to the user, then you are in for a treat. 10. Logging - in big systems, the code here can be non-trivial. Often this means more than writing to a log file, but rather providing visibility into parts of the system in real-time or close to it. 11. Scaling - Everyone thinks they are doing this right until they don't do it right. This can involve anything from servers to message transport format to compression to memory management to clustering. We could do this all day, but hopefully this makes a point.

lazyseq 12y ago

Not trying to pick a fight, but again your team comes off as naive and essentially gives the middle-finger to probably the majority of the software developer community with comments like these. It seems like you guys simply have not worked on an actual complex system of any significant size or over-value the complexity of front-ends. I personally respect all facets that go into making software. Anyway, I would argue there's no easy part and dividing things into monoliths at a conceptual-level is a mistake. Putting that aside, whatever part is comparatively (key-word) easier is highly dependent on the implementer. For instance if I build a system that just needs 1 input field, but then performs some sort of machine-learning or triggers some complex background dispatching, then that system is arguably much more complex on the "back-end." I think making big distinctions is a source of trouble to begin with and regardless I do about equal amounts of front and back-end work on my current start-up. As for the special sauce, in many apps the special sauce is actually the back-end. I would agree everyone hates with dealing with 3rd party libs and so on, but the ways you deal with that information is important. I think the way Treeline is described in your website, this article, and video makes weird distinctions and conflations between what is front and back-end work and what are actual problems programmers actually have. I totally get the parts about plugging into another API and not having to write the fetch code, but honestly with a library or two or even implementing integrations with many libraries, the work is minimal. Adding the correct libraries is often an important choice. To give you a concrete example, I do most of my current work in Clojure followed by Haskell, Java, and C. The choice of library authors doing things like using non-standard data structures that I then have to transform or directly consume, using/not using things like async channels, and so on from the ends of their APIs is a huge deal. It's not normally a problem to find a library that works with how you want to do things, but rather important that you pick the right one from the list of choices. If a Clojure library forces me to consume a core.async channel, I might not do it because I want to use something more general like a transducer to abstract that away. Sometimes also the ways that I consume someone else's library are very unique to my app and provide a crucial difference aka special sauce. Consider for example that for years I had a client that had a huge advantage due to code my team created against some public APIs because we were able to stream, batch, and thread our interactions with this particular API in ways most people had not thought of or implemented. The fact that our integrations were based on self-calculated heuristics provided dramatic speed-ups over everyone else meant it looked like we could do things that would cause the client's competitors to think we had some special deal or wonder how it all worked. This is not unique to us, rather it is simply illustrative of the importance of optimizing your business activities in the back-end. Consider that most of wall-street high frequency trading as another example is based on this notion. While I am not claiming that everyone needs to do this, your generalization is the issue here. It is funny because maybe our definitions of "back-end" are the issue. I tend to notice that the wheels on the front-end are invented time and again. How many different UI frameworks, patterns, controls, etc are there that get implemented from scratch? How many people end up composing their apps out of little more than paging, showing pictures, textboxes/checkboxes/pickers, form validation, css positioning, animation, and so on. I'd argue these are the actual repetitive bits that are tedious to write. I am reminded of criticisms of forum software by the authors of discourse. They pointed out a lot of the special sauce things that were wrong with forums, but ended up pissing off a lot of potential users who thought at least the versions in the past few years were essentially a lower functionality version of the same things with fancier loading and scrolling using JavaScript and JS templating. And the critics were at least in part right because not much new was brought to the table beyond looking and feeling slightly better on some basic elements, but the nature of forums software as we know it was hardly shattered so far. It seems to me what you've noticed is that a very particular segment of node.js users who used sails or were involved in the node.js ecosystem were reinventing the wheel or didn't understand the wheel. This is fine as you clearly are trying to address this problem, but you are also making blanket statements about software development. Everyone I have worked with who I consider to be far smarter than me would agree that a common mistake a lot of people make is they think any part of software development can just be replaced by a magic wand wave and that xyz part of the system is just stupid, tedious, and simple. It's like that time before you reach the point in programming where you realize it is not that your code is awesome, it is merely that it sucks slightly less than the next guy's code due to your experience. You guys really need to focus on the concrete things the product does not on philosophical nonsense of magic software development. Presently it just seems like you've made it easier to wrap some node.js APIs, call them in an opinionated way quickly, and slap a visual front-end that includes some canned forms and an impractical editor on top.