9 ms·
Show HN: Monte Carlo simulation for predicting project timelines
- virgil_disgr4ce 5y agoI'd like to see some kind of explanation of how exactly you "express the various uncertainties" of the project. I'm not going to click the CTA until I understand what I'd actually be doing to use the product.
- diiq 5y agoThe video on the How It Works page (https://quotes.vistimo.com/how-it-works https://quotes.vistimo.com/how-it-works) has footage of every step you'll take to run an estimate. If you mean you want technical details of the math involved, https://blog.vistimo.com/ https://blog.vistimo.com/
- awillen 5y agoUntil you have some reasonable way to quantify the likelihood that the CEO will see something shiny and insist on changing course, this just doesn't seem like it would be useful in the real world.
- diiq 5y agoThere's an "unknown unknowns" factor specifically for including guesses about new features or last-minute changes. And the great thing about being able to run estimates quickly is that you can tell the CEO how much their new idea will send the timeline off course as soon as they propose it, so they can decide if it's really worth the cost. I've been using this tech for years as a consulting engineer; this is just the dressed up version. It's been useful and effective for me, at least.
- ThouYS 5y agoIntriguing idea
- Grustaf 5y agoInteresting idea, and very nice and simple web page! I have no idea if people are willing to pay for it, but good on you for making an attempt. A question though, isn't Monte Carlo a bit overkill? Would you lose much by modelling each deadline as a normal distribution? Then you could just calculate things analytically.
- diiq 5y agoYou're not wrong; normal distributions are OK but not ideal for estimating durations -- it can lead to weird artifacts if you don't restrict yourself to just the mean. (https://blog.vistimo.com/post/169546687605/estimation-math https://blog.vistimo.com/post/169546687605/estimation-math) And once you're committed to using monte carlo, you're free to include one-off events -- sick days, vacations, social loafing, etc, that are really hard to model analytically, since they are roughly uniform in time -- the longer the project runs, the more days off make the project run longer.
- Grustaf 5y agoI suppose you could model those one-off events with their own normal distributions (with 0 variance), rather than modifying the main distribution? But sure, Monte Carlo is more flexible, and not really harder to implement I suppose, since you're already on a computer!.
- iainmerrick 5y agoThat blog post is very clear, nice work! As another alternative, could you convolve the probability distributions numerically? I’m not sure of the trade-offs between that approach and Monte Carlo, but if it’s something you’ve considered I’d be interested to hear your thoughts.
- diiq 5y agoOnce I took a few steps down the simulation path, it seemed so easy and powerful that I didn't look back. There is some computational cost, but for me it's more than made up for by how easy it is to, for instance, say "I'd like to model sick days"; * let the government tell me how often people are sick (https://www.bls.gov/cps/cpsaat47.htm https://www.bls.gov/cps/cpsaat47.htm, see "Computer and # mathematical occupations", absence rate, due to illness or injury), * make my simulated workers take days off at that rate. And everything else, all the statistics etc, can stay exactly the same. And I can have a lot of trust that it's producing sane results because I can look at a run, and say yes, that's a plausible number of sick days. Monte carlo requires you to be clever once, setting up the simulation, and then you almost never have to be clever later when you want to introduce new twists. Any other statistical model you have to keep being cleverer for every additional complication.
- colincooke 5y agoA simpler, if perhaps less effective, idea would be to just take advantage of ensembling. Get many estimates from everyone qualified to give them, then just fit a normal distribution to your final results. The average of many estimates tends to actually be quite accurate sometimes, with the famous example of visitors to a county fair guessing the weight of an ox[0]: "At a county fair held in Plymouth, 800 visitors had participated in a competition to guess the weight of an ox. While most people's estimates were too high or too low — falling an average of 37 lbs. away from the true weight of 1,198 lbs. — the median of everyone's guess was off by only 9 lbs., or less than 1 percent of the true weight of the ox." Curiously there is also some "within individual" averaging that helps too. If you would ask the same person to estimate the project timeline a few times, spaced temporally, then take the average, it'll probably be more accurate than a single estimate. [0] https://www.npr.org/sections/13.7/2018/03/12/592868569/no-man-is-an-island-the-wisdom-of-deliberating-crowds https://www.npr.org/sections/13.7/2018/03/12/592868569/no-ma...
- diiq 5y agoYou're right! Ensemble forecasting is great! Vistimo proper (https://www.vistimo.com https://www.vistimo.com) does both -- it gets individual high/low estimates from multiple people, and creates a combined distribution from all of them (though each estimate is a log normal, not normal, since it's a time estimate). But Vistimo is a full project management tool; that space is full and switching project management tools is expensive, so it's hard to get people to adopt. Vistimo Quotes is a much lighter-weight way to get many of the same benefits without having to talk your whole team into signing up.
- adam 5y agoI'm curious if you allow the project to enter actuals after the fact, then you can compare them to individuals/aggregate estimates so you know how biased your model/individuals/the organization is on these types of estimation efforts. You could give that as feedback as an additional service: "You're constantly overconfident on X type tasks..."
- 5y ago
- ballmerspeak 5y agoIf those buttons are appealing, I recognize them from this blog post by Josh Comeau https://www.joshwcomeau.com/animation/3d-button/ https://www.joshwcomeau.com/animation/3d-button/
- diiq 5y agoHaha, look closer! These buttons have a continuous curvature profile, rather than a rounded rectangle. It's a big part of what makes them feel so "70's calculator" chic :)
- orlandohill 5y agoJoel Spolsky described this technique, calling it Evidence Based Scheduling. I'm surprised that it didn't become more popular. https://www.joelonsoftware.com/2007/10/26/evidence-based-scheduling/ https://www.joelonsoftware.com/2007/10/26/evidence-based-sch...
- diiq 5y agoI think Steve McConnell was calling for statistical modelling even earlier than that. Definitely not a new idea! I'm working on the assumption that it hasn't caught on because it's a pain to do without proper tooling; but it's also true that many engineers believe estimation is useless, impossible, and a waste of time or worse. It can be hard to overcome the pain of having been burned by a too-tight timeline.
- FourthProtocol 5y agoSteve McConnell wrote the Construx Estimation Tool way back in the '90s. It made use of Monte Carlo simulations. I still have a copy.
- c17r 5y agoCame here to bring up FogBugz having this if someone else didn't already.
- commandlinefan 5y ago> I'm surprised that it didn't become more popular It gave managers answers they didn't want to hear.
- bberenberg 5y agoFeels like a good idea, but the hard part here is the data input. Do you have any plans to import from Jira or something like it? Ideally bi-directional sync after a given simulation is run?
- breakfastduck 5y agoI said a similar thing about having no basic import (even csv or such) function making it completely unusable for large projects. Got a really snarky response from the developer. Big red flag on putting anything serious into it.
- dglass 5y agoI've used this technique before at an employer. I thought it was crazy at first but I came to really like it. No more story points and estimating the size of tickets. The monte carlo simulation handles the forecasting for us so the engineers can focus on completing tasks. The PMs focused on user stories and acceptance criteria. The stakeholders had a continuously updated forecast for when things would likely get done.
- darepublic 5y agoTrying to get correct estimates feels like it violates some basic quantum principle of physics. Were venturing into Schrodinger territory here folks. On a more serious note, good idea and nice site. I may give this a try. I have been thinking along these lines myself, using simulations to complete different dev related tasks
- breakfastduck 5y agoI thought this would be brilliant, so I clicked 'try it out' only to get through and be asked to manually write in the tasks. I have a huge backlog already in an external tool. I can't comprehend I'm expected to write them out one by one with no ability to import that I can see.
- diiq 5y agoHaha, at least tell me which external tool you'd like me to add an import from, and I'll be happy to add it to my backlog.
- breakfastduck 5y agoTool is irrelevant, it’s very surprising you don’t have some basic CSV import, or JSON or any other data interchange format. It’s one thing to type in a few tasks on a small project but it’s completely unrealistic to expect me to manually re type 100s of tasks.
- diiq 5y agoNot trying to be snarky, I just didn't understand what you needed. Json import is not something any other users have brought up, but I'm happy to look into it. I'm working on CSV export, first, but I will absolutely keep this in mind.
- enlightens 5y agoVery interesting! Is there a sample report / projection available to view? I'd like to show this to a few coworkers without having to spend time loading in my own sample data
- asdf_snar 5y agoAs a package this seems perfectly reasonable, but I'm skeptical of the use of Monte Carlo being a technical buzzword rather than a choice made because it was the appropriate tool for the task. I wasn't able to find any technical details in the "How it works" section, but if you assume the distribution of time to complete a task is uniform on the interval [worst case, best case], then at any time point the underlying distribution seems to be a composition of sums and minima of uniform distributions. These are available in closed form. I see in another comment the distributions are assumed to be log-normal -- that's fine too, there are good approximation to these. On the website it says thousands of trajectories are generated. It seems likely that one can straightforwardly compute the result that the Monte Carlo approach would give in the limit of infinitely many generated trajectories.
- diiq 5y agoI promise, this is the result of real constraints! Monte carlo is used because of the limitations of analysis when using log-normal distributions. Normal distributions can be summed algebraically; log normal distributions cannot. https://blog.vistimo.com/post/169546687605/estimation-math https://blog.vistimo.com/post/169546687605/estimation-math Note that Monte-carlo is not used prominently anywhere on the site, just here because I thought HN would find that part of the implementation interesting.
- asdf_snar 5y agoIt is true that there is no closed form expressions for the sum of two independent log normal distributions. However, like I said, good approximations exist. Moreover, if you were to ask me to compute some expectation E[f(X)] where X is the sum of independent one-dimensional log-normals, it seems simplest to convolve the densities and simply compute the expectation by trapezoidal integration.
- diiq 5y agoFair enough! There's more than one way to skin a cat. This way was easier for me, and has additional benefits as described.
- alex_c 5y agoI like this! I would consider using it for some of our projects. Biggest issue I have is with the task list, for any reasonably complicated project we would at least need ability to group tasks into features or milestones. Task lists can get pretty granular, but communication to stakeholders or clients does not usually need to be that detailed, higher level groupings are preferable (and also makes it easier to organize the tasks). As others mentioned, integration with Jira and such would be nice, but not a deal breaker. Also I think I broke my demo project, I changed one estimate and the projections just say "New projections are being calculated". There are only 4 tasks :)
- diiq 5y agoThanks! I totally agree, personally I want more hierarchy; but user testing with milestones was pretty rough. Vistimo proper (https://www.vistimo.com https://www.vistimo.com) is less slick-looking, but has milestones and story-mapping and bugs and enhancements and all kinds of hierarchical organization tools. Most users I've tested with have found Vistimo Quotes' approach easier to pick up and use on existing projects, even if it's less powerful at expressing the full richness of the work. Re: the long running calculations, I will take a look, but it's possibly just a traffic jam. There are a few more "test" simulations running simultaneously than usual this morning.
- skellystudios 5y agoYou should 100% have a sample tool for people to play with on this homepage.
- nscalf 5y agoOr even just a sample video. There’s a good bit of friction before even knowing what this is.
- nwsm 5y agohttps://quotes.vistimo.com/how-it-works https://quotes.vistimo.com/how-it-works
- Raidion 5y agoYep, I'm not giving my email to something before I have a decent understanding of what it does. Even the "Try it out" redirects you instantly to a signup modal.
- journey_16162 5y agoI was interested in it a bit, but not interested enough to fill the form. It's not even about the email address, but just the effort. I know it's ridiculous, but that's how it works. "Don't make me think".
- breakfastduck 5y agoIt also has no data import functionality so if your project is over 50 tasks (basically anything remotely business scale) you’re gonna be sat doing data entry for a while.
- vladsanchez 5y agoHi Sam, your work is admirable and I'm sure lots will pay attention. As a side note, have you seen/heard of (sic) "Vistima" parody? https://youtu.be/4jVQhdlCmjc https://youtu.be/4jVQhdlCmjc Congrats, Stay Thirsty.
- mclouts91 5y agoThere’s a typo on the how it works page... Unkown unkows
- diiq 5y agoThanks, will fix.
- nwsm 5y agoThe idea is interesting, but it appears to involve using Vistimo's UI to break down projects and tasks. It feels incredibly waterfally, and also will involve re-entering mass amounts of project management data. I hope it integrates with Jira/Trello.
- db48x 5y agoNobody can design unilateral phase detractors in just 4 hours.
- tschottdorf 5y agoI tried to use this but once I start entering tasks I don't see how to proceed. There's no way forward from neither there nor the project list page. Am I missing something?
- diiq 5y agoOnce you enter tasks, and estimate at least one, you should see a link to view your "Projections". (Fetching those projections uses websockets, so you may need to refresh if you're behind a particularly restrictive firewall.) If you're still having trouble, drop me a note at sam@sambleckley.com and I'd be happy to help.
- a_zaydak 5y agoThe idea seems simple enough and the math to do these kinds of predictions is fairly straight forward unless there is some secret sauce under the hood that isn't obvious. I guess the value is slick UI / tool that was created around it. One question that I do have about these types of tools... how does one evaluate how well it worked? I guess in the end as long as the timeline wasn't too far off then you call it a win. I think a useful tool for project planning is not necessarily simulating possible time-lines (or a estimated window of time for the project to be completed) however to intelligently identify a set of critical tasks within the list which would have a high probability of totally blowing up your time line their deadlines are missed. In project management / scheduling there are often critical path items listed. However often these critical items are semi-arbitrarily picked by the PM and not backed by and deep insights.
- diiq 5y agoAbout 5 years ago, when I first started using this method, I gave a talk about it that went into detail about how to do it, and the benefits. I did a live demo showing how well it modeled the variance even for really simple tasks like "take off your shoes and then put them back on". The audience reaction was very positive, and I was pleased to have spread the word. No one actually put it into action, though. It turns out if you need better estimation tools, you don't have time to make them -- you're in crunch time, because your estimates were bad. The benefit is not so much slick UI -- that part is just me having fun. The benefit is that the work, however easy, is done. If folks "steal" this idea because it's easy to implement, that's great! It's not a new idea, just an underused one. I just want to stop being hired onto contracts where the first thing I have to do is explain that the deadline is impossible. I agree with you that a linear backlog is not sufficient to do really sophisticated planning -- Vistimo Quotes is a pared-down tool designed for easy adoption, because my kitchen-sink version requires too much buy-in for many teams.
- a_zaydak 5y agoSorry, I did not mean to imply that the only value that the creators had was making a nice UI. I truly meant it as a compliment. There is hard work involved in creating a product out of an idea. Seems like this will be a very useful product. Congrats to all of the people who have worked on it.
- SkyMarshal 5y agoI really like this idea and wish all project management tools included it by default. Fwiw Fogbugz has been using it for over a decade, called Evidence Based Scheduling: - https://www.joelonsoftware.com/2007/10/26/evidence-based-scheduling/ https://www.joelonsoftware.com/2007/10/26/evidence-based-sch... - https://support.fogbugz.com/hc/en-us/articles/360011258994-Evidence-Based-Scheduling-EBS- https://support.fogbugz.com/hc/en-us/articles/360011258994-E... - https://blog.fogbugz.com/evidence-based-scheduling https://blog.fogbugz.com/evidence-based-scheduling There’s at least one Jira app for it: - https://github.com/handshake/pystradamus https://github.com/handshake/pystradamus And Trac also used to have EBS but I’m not sure if it still does, can’t find it atm: - https://trac.edgewall.org/ https://trac.edgewall.org/
- diiq 5y agoVistimo (the full version at https://www.vistimo.com https://www.vistimo.com) does too. It turns out the project management space is full of neat tools with clever abilities, but it's very hard for a team to switch between those tools. It's expensive and time consuming! Vistimo Quotes is an attempt to pull just statistical estimation out on its own, so you can use it no matter what you use for your project management. (A note: this and EBS are pretty different in their actual math -- they just seem similar because the next closest thing is "everyone agrees on a single number per task and we add them up" which is so limiting that just having a statistical model puts EBS, 50/90, and monte-carlo into their own category.)
- BlueTemplar 5y agoHeh, that note is why I decided that "project management" was just a bunch of nonsense (like any other field using maths without taking uncertainty into account...)
- jkaptur 5y agoOh wow, my first job in the industry was building something very much like this! It was actually quite powerful: the UI and simulation were done by Microsoft Project (https://www.microsoft.com/en-us/microsoft-365/project/project-management-software https://www.microsoft.com/en-us/microsoft-365/project/projec...), which I would describe as "the Excel of project management" in that it was an ENORMOUS bag of features for building Gantt charts. The coolest part was that it integrated with Outlook's calendar, so if a task started on June 1st, was assigned to Bob, took two weeks, and Bob was on vacation from June 8th-22nd, it calculated the end date for the task as the end of June. The Monte Carlo part was run by Crystal Ball (which was a startup then, but has since been acquired by Oracle). It was a mature tool, so there was a big library of probability distributions (https://docs.oracle.com/cd/E57185_01/CYBUG/define_assumption_dialog.htm https://docs.oracle.com/cd/E57185_01/CYBUG/define_assumption...), you could enter correlations (if X task takes a long time, Y is going to take a long time too), and you could analyze the output in all sorts of ways (how sensitive is the deadline to this task, how frequently is this task on the critical path, etc.). The implementation was quite a hack - a hidden instance of Excel running Crystal Ball and lots and lots of VBA to manage everything. I'm not sure that it was ever used for real :(
- poxwole 5y agoCan anyone link some research papers on this?
- pton-throw 5y agoThe UI is slick, but why should I trust this black box? The "how does it work" section doesn't even mention * modeling assumptions * training data (if any) * validation (if any) Why is this better than, say, giving the 25-75% range around my "best guess" assuming log-normal duration and the variance of prior project completion times? Can you demonstrate that it's better than that?
- diiq 5y agoIt's not better; it's almost exactly that, as detailed in many comments below. I choose slightly different thresholds -- 75% is not very high, you'll be wrong one time in 4 -- but that's the approach. If you're already doing log normal modeling of your estimates, and including sick days, social loafing, etc, then you don't need Vistimo Quotes. Just keep doing what you're doing!
- pton-throw 5y ago> as detailed in many comments below This is something I would buy if I had confidence in the simulator. Why not write a blog or a brief white paper giving some hint of what's going on behind the scenes? Generally speaking, every bit of energy someone spends being cool (good UI) comes at the expense of energy spent being good. Maybe you've done some serious work on the simulator, but I can't tell from your marketing material.
- deleted 5y ago[deleted]
- Aeolun 5y ago> every bit of energy someone spends being cool (good UI) comes at the expense of energy spent being good Why would you assume that? If your model is already pretty much perfect, not much more you can spend your time on than presentation.
- pton-throw 5y ago
- poodler 5y agoNice service, lots of ways to grow. I think the ability to make pretty pdfs for clients is actually a strong pull. In my experience, statistical modeling of potential timelines is not the bottleneck to accurate quotes: it’s the ability to actually give good guesses for “min time” and “max time” on critical tasks. I.e. the majority of timeline variability is due to just a few “tasks” that totally blow out of proportion because I underestimated how much work was actually involved. Idk, maybe that’s just me. It would be nice to track my estimates vs actual time spent to improve the “human error” part.
- weeblewobble 5y agoThere's a product called LiquidPlanner built around this idea thats been around for years. A friend used to work there but I have no other affiliation. Looks like they're still kickin' https://www.liquidplanner.com/ https://www.liquidplanner.com/
- danielmarkbruce 5y agoOut of interest, why not make this a jira app and sell it in the atlassian marketplace? You get low cost distribution, the tool would fit right into the existing product everyone is using, installation is easy etc, data sync is easy, you can still generate the nice pdfs.
- diiq 5y agoI'm a consultant, and it happens that hardly any of my clients use Jira. So this is the result of scratching my itch (and the itch of people who already give me money) first, and dressing up the result for others to use. Lots of people have expressed interest in Jira integration, and that's definitely on my list to accomplish one way or another. The Atlassian marketplace is like most app stores, though -- if people already know your name, it's easy for them to find and install your app. But if you're a relative nobody, there are 50 other estimation apps, and very little room to pitch what makes you unique.
- Aeolun 5y agoHonestly, the Atlassian app store is pretty bare. There’s pretty much only one or two solutions for everything, and they’re likely 2+ years out of date.
- danielmarkbruce 5y agoInteresting. I appreciate the insight, haven't looked at it in a while.
- impute 5y agoNeat idea. This looks somewhat similar to Actionable Agile except it's up front rather than relying on ticket cycle time. Are you able to share the task list so that each project member is able to fill out their best and worst case estimates on their own in an async fashion, preferably anonymously?
- diiq 5y agoIt's been my experience that most business needs involving estimates can't wait until the job is partway done. "Can we afford this" and "should we do this now" really need some up front answers :) Vistimo Quotes is specifically for producing rapid estimates that don't require team buy-in, sign-up, etc. Vistimo (https://www.vistimo.com https://www.vistimo.com) does anonymous, asynchronous team estimation -- but the additional labor costs (sufficiently describing tasks to async folk understand them, managing team permissions, requesting reestimate when requirements change) have made it not particularly viable as a product. Finding a balance of power and simplicity is never easy.
- impute 5y agoYeah that was the only downside I found with Actionable Agile. It didn't answer the question up-front. However, I think it can work if you're able to estimate the number of tickets instead of time frame. So instead of a best/worst case estimate in time, you'd estimate the best/worst case in number of tickets a task will need to be broken up into. I did this one time when I needed to try to get a better understanding of estimation up front when the work wasn't well defined yet. It's possible that people may give better estimations when thinking about the number of tickets because then you immediately have to think about the number of individual components whereas with time, you're going more by feeling.
- diiq 5y agoThat makes sense! I am constantly interacting with new teams, so I like time because everyone already knows what a day is. A ticket, or a point, or other abstraction changes from team to team and project to project.
- Aeolun 5y agoDoes this have an API to feed it tasks?
- quickthrower2 5y agoEstimation. Great if you can fix (a) who is working on it (b) the visual and technical design and (c) the scope and (d) other interruptions. You can fix those things, have fully signed off design docs, have a CCN process for every little change etc. and watch everything grind to a halt while you deliver the wrong thing on time. I’m not saying don’t estimate. Do! Have a plan and realise it won’t survive contact with reality.