6 ms·

Ask HN: How do you develop internal tools for your organization?

• d--b 5y ago
  I am developing Jig (https://www.jigdev.com https://www.jigdev.com) it’s based on Observable and it’s meant to create dashboards and apps quickly without much fuss. The aim is to have business specialists write the apps and not html/react which is quite tricky to get right. The product is quite rough right now, but I’ve been working with a first client who seem to enjoy it. Happy to demo it.
• Jugurtha 5y ago
  I'm on mobile right now. I'll write something here as soon as I get my laptop. Wait one.
  • siscia 5y ago
    Lovely! Thanks a lot!
    • Jugurtha 5y ago
      On my laptop and writing. It'll take some time.
  • Jugurtha 5y ago
    Please excuse the poor editing and pell-mell thoughts, especially towards the end: >We are considering creating few internal tools to speed up the work of a few departments in our organization, but before starting development heads on, I would love to gather some wisdom from HN. First, what tools are you developing? TL;DR: a machine learning platform that lets you use your own Kubernetes clusters from your cloud provider (GCP, Azure, AWS) and get real-time collaborative notebooks to train, track, package, deploy, and monitor your machine learning models. >How do you handle user authentication? TL;DR: manually added at first, and later added invite links and password-reset flows. Still invite based. >Do you develop one single big application for everything in your organization, or few distinct applications, each with its own specific goal? TL;DR: plugin system with a core that loads extensions at run-time. Each extension is its own separate repository. We could compose different products. >Any specific thing to watch out for? Yes, here we go... Context: consultancy specialized in building end-to-end products for large enterprise. Around 17 people in 2018 working on about 8 projects simultaneously. CEO (co-founder) in one country and CTO (co-founder) in another country with the team. CTO quits a year and a half after I joined. Pretty much everyone quit after that except literally a handful. A colleague and I took over. We killed projects that were going nowhere, unstuck some, tightened the stack (it was Scala-Python-Spark-Kafka-Hadoop-Clojure-Vue-and what have you). Problem: building end-to-end products for several sectors (telcos, energy, banking, transportation, health) meant that in addition to diving into different domains, acquiring data from peculiar data sources, and building the models, we also had to build bespoke "quick-no-time-for-reusability-one-off" applications that used these models and, in some cases, enabled domain experts at client organizations to train models themselves on new data. User authentication and administration, workspaces/groups, etc. This also created a situation where I bumped into a university classmate founding a startup who described his problem, that was exactly the same as a problem we solved for a telco for which we built a product, but we were unable to sell him the product because it was custom-made. It sucked. He needed the exact same features but although they were contained, the containing application was too custom made. The next project we took on, we wanted to change the way we develop product, and the way we worked. I remember the first few commits I made sure to set up a plugin architecture so we could have a minimal core that practically did nothing but load extensions dynamically, and all the functionality would be in these extensions with separate repositories so we could compose and add and remove to create different products. Lesson: It should be as easy to remove things as it is to add things. Next, in 2019, we started experimenting with remote work. It was painful to watch colleagues look at the window to check if their train arrived, or to arrive at the office at 11AM because commute sucked. They were wasting up to 6 hours per day in commute. We experimented with remote work one day a week and saw what broke and fixed it. Then two days a week. Then a whole week here and there. The rationale was the we may have to do it some day and we might as well learn the lessons right then when the stakes are low not to be taken by surprise when the stakes are higher. In the meantime, my colleague (data scientist) had an obligation that put him away for one year. He could read emails but not reply, or open web pages due to the poor internet connection. This introduced a forcing function for asynchroneous communication, tracking, collaboration to be built into the internal platform. We were tired of not knowing which approaches generated the best models, and being unable to easily re-use work from previous projects. The reason they were coming to the office was to train models on a "beefy machine", and it was ridiculous. The next thing to do was to enable our people to do their job remotely on our internal platform. We built a plugin that offered a notebook experience so they could run training jobs that required a GPU and large amounts of memory right from the comfort of their home. Lesson: Just enough engineering and features to get the job done (that's the second important thing). No over building. Get the job to be done right from target users, and implement a slightly more generalized form of the problem while avoiding the "General Problem": https://xkcd.com/974 https://xkcd.com/974. We're not DSL-creating-architecture-astronauts shooting off to far abstraction orbits. We were only left with one data scientist who was already busy working on a project for a client and we lacked feedback from the people (one person, given the other was absent) we were building for. She taught a course at university and had around thirty students in a town that was hit hard by COVID. The university shut down and she asked me if the idea about enabling students to use the platform was still on the table because her students couldn't go to university anymore and use the machines, and practically none of them had, or could afford, a station with GPU/RAM requirements and it put their final year project in machine learning at risk. Here come our users, then. As I said, we needed users because we only had one user and she was busy. We onboarded her students: we created their accounts (yes, no account resetting/forgot password flows), we created a Slack workspace for them, helped them troubleshoot and optimize their code, and guided them through getting the work done. We stayed late. They did two things: - Discover issues we were not aware of. - Put weight on the issues we were aware of in terms of frequency/impact (most important). When the tenth user would complain about something, we knew we really needed to fix it: - We added real-time collaboration in notebooks because they were taking pictures with their phone cameras of the screen and sending them on Slack to get help from their peers or from us. - We added long-running notebook / scheduled background-execution in notebooks because there were resource conflicts: everyone trying to train a model at the same time and getting OOM errors, and users staying late to get resources. Lesson: get initial users as soon as functionally possible. Ask questions. Ruthlessly prioritize issues according to frequency/gravity. Our issue templates on GitLab include an "Instances" section: how many times has this happened and how much does it suck? If it's an imaginary scenario, we won't do it now. If it's a minor inconvenience that happens rarely, we won't do it "now". Action: add documentation (MkDocs: https://www.mkdocs.org https://www.mkdocs.org ) Then we worked hard to fix a lot of bugs whose stack traces were buried in large logs. Action: add application monitoring (Sentry here: https://sentry.io https://sentry.io ) to prioritize bugs and issues. Add incident report templates to make it easier for people to fill them up (in addition to feature/bug templates). Then we needed more feedback to accelerate and solve more generalized versions of the problems we were solving from professionals who were working on projects and problems we were unfamiliar with. Revamp the landing page, add a "Request access" field to get emails. Interact with people and try and get them to try the product. Listen also to why they wouldn't. Send invite links to people we chatted with... Action: add analytics to see sticking points (PostHog: https://posthog.com https://posthog.com ). Ask questions about non-consumption. What's wrong? This lead us to work on load times, Docker image size reduction, fixing bugs, warm starts, and various improvements until we saw usage pick up. When still nothing has happened (we still need feedback and more data points because we're too tiny), we started getting into more conversations with people in a more hands-on approach. Actively look for people online on several platforms and forums. I manually curated around 500 email addresses for whom I got the first name, last name, email address (from Twitter, keybase links to GitHub with `git log | grep Author`. Manually cloned a lot of repos to do that). Then instead of just sharing a link, I qualified early on: most of the professionals who are active in the field, engineers at companies, sometimes CTOs/CEOs (some of them in the middle of mergers and/or acquisitions) had the most useful feedback, got on calls, and wrote huge emails going deep into the product. AI enthusiasts/audience builders/medium writers/people tweeting about AI were often uncouth. The only way to know who was who was to qualify in the initial emails. Lesson: talk with the right people. The right people are more knowledgeable, and those who did something with their life or built a product or a company understand the effort and have been way more generous and courteous. Lesson: even though I started talking with people before and while we were building the product, and the product was for a problem and use cases we intimately know, I would have wanted to talk with people even earlier than that. Lesson: sell the product early on (integrated with Stripe to be able to send invoices). Buying is honest feedback. Product guidelines: - APIs: everything you can do with point and click on the web interfact, you must be able to do with API calls. - The service must be able to die without users being left scrambling to exfiltrate their data with an end of service email (derived from a personal one: I must be able to die without the team being impacted except for nostalgia). - The site must be able to run on a Raspberry PI, but do the work with the right integrations to external compute/storage/networking. - We must be clients of our own product.
    • siscia 5y ago
      Lovely! Thanks a lot for your reply!
• stocktech 5y ago
  I led a team doing this for 5ish years in a F100. We were the only team and we built it from the ground up. We handled all departments. Some of the bigger ones were customer service related such as letter templates/mailing/printing and a phone system automation/integration. Other apps were used by teams of 2-3 people, such as fraud investigation management, accounting automation, or simple 3rd party report generation tools. Over the years, we developed 20-30 apps ranging in complexity. To answer your specific questions: 1) User authentication was a centralized service that managed roles and permissions. We integrated with LDAP so that we would only authenticate current employees or contractors, then the service would look up the appropriate permissions. We delegated management of roles/permissions to a business owner in the department and our IT team handled LDAP. 2) Each app was separate - each had 1 app, 1 database, 1 project board. I think the real benefit with this was a simpler mental model while working with the systems, but this came at the cost of managing shared services/packages. We used frameworks that were more opinionated so that each app made similar assumptions and used the same tooling. I started the team using C# MVC, but we moved to NodeJS on the backend to reduce context switching between FE/BE. The big thing is consistency so that each app's codebase looks and feels the same. When you're jumping between apps, you don't want a day long ramp up period. Similarly, I made sure all devops type tasks were streamlined. We did CI/CD before our product teams did and had full test suites. Preview environments were setup for developers and users. And everything was documented with guides. We could clone our project blueprint repo and deploy it in 5 minutes. A working CRUD app could be done in a few hours. One unnecessary thing I choose to do was create SPA frontends. With our blueprint and shared services, it reduced the complexity, but it really gave the team something to learn/dive into. I think that little bit of "sexy tech" went a long way for morale and recruiting. If I had to do this all over again: 1) I'd do a better job defining the team's value proposition. Those basic CRUD apps could be moved to a no-code platform. The team should be focused on apps where proprietary engineering will benefit the company most. Some of my apps had estimated value-add of $5mm+/year, but we would still get pressure from the execs to make a new app that would save the company 50k. 2) Similarly, I'd make sure my team was separate from rest of the org, as long as it makes sense. When we first got started, I was under a product director. As we grew, we never really found our own space in the org and got pulled in different directions. fwiw, I left the org because they wouldn't promote me. 3) I'd start with each app being separate then move to 1 app per department. We worked towards this as we grew and multiple apps needed to be integrated with each other. I think I moved too late though and our integrations reflected it. Especially as we could have justified a small team for each department, it would have made sense. As a management aside, I wanted to be more agile and form teams around projects - 1 app, 1 database, 1 project board, 1 team. It worked fine, but something something Conway's law. Some things I think I did right: 1) My engineers did everything - project management, roadmaps, engineering, etc. Even when we had dedicated project managers, my engineers were always in the room because the business doesn't know what they want. We had a number of BIG wins because my engineers could come up with non-obvious solutions. I also gave my team the autonomy to solve the business problem whatever way they saw fit. 2) I hired people who cared more about solving problems than the tech. I encouraged doing more design work than engineering and spending time with our users. This led to better UX, better domain knowledge, and very happy users. 3) I advertised the shit out of our work. Multiple departments have similar work and when we'd build an app that did one thing, I'd go to another department and see if it would help them too. Some of our apps started as a copy/paste and evolved in their own right. I'd also send an email to our senior leaders every time a new app went live with a small description, a gif of functionality, and the estimated value-add. Most of time, we'd add something to our roadmap as a result. I was also good at networking within the org, for whatever reason, so I was able to be proactive in some regards.