12 ms·

Show HN: AI climbing coach – visualize how to climb any route based on your body

• deleted 2y ago
  [deleted]
• gus_massa 2y ago
  Does the AI create the 3D model from a static camera? Can I choose the color of the avatar? (It's not an important feature, but people will love it.) Does it work with natural walls? (I guess the artificial walls have "handles" with an standard size that makes guesing the scale easier for the computer.)
  • smandava 2y ago
    - it creates a 3D model of you based on the first second of your body in the video. How the video is recorded does affect how the 3D avatar looks. - don't have that functionality but its really easy to implement. - ideally, it works for anything. I will attach some outputs of it climbing outdoor routes. The model doesn't know what a wall means. It just has seen enough data of people climbing that it can somehow correlate certain features in videos to certain human movements.
    • gus_massa 2y ago
      My guess is that for the computer it's easier in an artificial wall becasue features are standarized. (Disclaimer: I never climbed a wall, but from time to time I fall in the https://www.youtube.com/@AlbertOkay https://www.youtube.com/@AlbertOkay rabit hole. In particular this video looks relevant for my question https://www.youtube.com/watch?v=onXrJ5sX9BU https://www.youtube.com/watch?v=onXrJ5sX9BU (but I din't watch it yet.))
      • smandava 2y ago
        Yeah it performs best with indoor bouldering videos with spread out holds and great lighting. I will post examples of it climbing natural walls in the paper. Will send soon.
• smandava 2y ago
  Note: Will attach more example outputs and make a detailed document about how the model was built and the research behind it. If this is interesting to you, feel free to sign up to the waitlist on www.climbing.ai (and make sure to sign up for our Discord!) I originally planned on open sourcing the model, data, weights, and code. Only a few people (me and a couple friends) have access to the hosted model on the web app. If enough people are on the waitlist, I will consider releasing access. This is very expensive to run so was only considering open sourcing it. Note: the model works well sometimes, but most of the times it does not. This is an early research preview. Please tamper any expectations. A really good general model requires millions of videos and much more training time so it is really prohibitively expensive. As mentioned before, if someone has the compute, all they have to do is scale the existing dataset and training pipeline (which I will publish open-source in the coming weeks).
  • ethbr1 2y ago
    High-level details about how you tackled the various sub-problems would be useful. I assume, at minimum, there's: - Segmentation of moving elements - 3d reconstruction from 2d video - Reverse kinematics - Identification of holds - Search for potential climbing path - Generation of kinematics for path
    • hirako2000 2y ago
      Likely looking to see a 100k emails list before really thinking about all these problems to solve and well. Worst case able to build this, scenario, author builds something else entirely.
      • ethbr1 2y ago
        Afaict, author already built a proof of concept.
  • icpmacdo 2y ago
    I've been working on the same thing, there is an interesting quote from back in the day and its either make your program faster or wait 6 months for the computers to get faster. I look at it the same way now with Key Point Analysis and Pose Estimation libraries. I think OSS is best for this category right now.
• jfengel 2y ago
  I look forward to playing with this. It seems astonishing.
  • smandava 2y ago
    I will be open sourcing the codebase and model. Due to compute and training dataset size constraints, the model is not exactly "production ready". I have (and am) cherry picking rare examples of it where it is decent. You can play around with it when I open source it but truly good results require multiple millions of dollars (but it really is a linear scaling of compute and dataset).
• goodmattg 2y ago
  A few thoughts as I've done academic research & built products in this area: - if you're using SMPL body parameters this will have to stay research / open-source - is this leveraging some sort of monocular depth estimation to estimate the wall in 3D space? Also, do you have assumed camera parameters, or is that also estimated? If there isn't any depth information, this will be highly inaccurate on any cliff routes, but still useful on flat wall climbing. Overall, a good idea (that I've also thought about building as a climber) - the tricky part that I'm impressed you have a solution to is path planning up the wall. Even assuming a flat wall with no depth estimation, it's still looks effective.
  • syntaxing 2y ago
    Not OP, but why does it have to be open sourced? Copy left license?
    • goodmattg 2y ago
      commercial license - the research group formed a corporate entity that licenses the body model and all derived work (SMPL-X, etc.): https://meshcapade.com/SMPL https://meshcapade.com/SMPL
  • ethbr1 2y ago
    > the tricky part that I'm impressed you have a solution to is path planning up the wall. I'm assuming this is evolutionary / brute force of some nature, given OP's comments about it being expensive to run.
  • smandava 2y ago
    Yes, this will be open source. This is an end-end system that just takes in video frames. Camera parameters are one of the things that is predicted. It gives promising results for a wide variety of environments (cliffs, diff types of bouldering walls, diff outdoor walls, etc.), though not always accurate. Path planning is also part of the end-end system. Will share more details in the paper.
    • 2y ago
• prions 2y ago
  It looks really interesting, but as an experienced climber I'm not sure if just watching a video of my avatar climbing would really help with skill acquisition. Also, this claims that the wall type or video quality doesn't matter, but I have a hard time understanding how the model would be able to understand that a small crimp could possibly be dual textured and therefore has only a few specific ways of approaching it. So it seems that this is more for visualizing a climb (which is a skill most climbers should develop) and not really for dialing in some sort of microbeta for a problem.
  • etrautmann 2y ago
    Agreed - so much about the detail of how you would climb something comes down to details that would be hard to measure with a camera, like textures, your estimate of friction, etc. Very cool idea though, looks fun to test.
    • sandworm101 2y ago
      And this is deep into "sport climbing", borderline gym rat territory imho. It doesn't model all the other core aspects such as protection, rope management, exposure and rest stops. I imaging if you pointed this at a real cliff and recorded several assents it would quickly become a blurry-twitchy mess as all the movements not touching the rock spoiled the data. Maybe for bouldering, but not for real climbing.
      • soperj 2y ago
        What makes rope climbing "real"?
        • lambdaone 2y ago
          There are lots more variables to consider, particularly in lead climbing, even when you have a bolted route. And trad climbing is even more complex than that.
          • 2y ago
• piterrro 2y ago
  Nice! Cant wait to try it!
• aconz2 2y ago
  Very interesting, do wish we could play with it since it is a show hn, maybe repost when it is open sourced/usable. Looking forward to reading about how it works. How does it know which holds are on? How does it know what the top is? Are those inputs or inferred or is it just end to end completing likely body positions one after the other?
• DanBC 2y ago
  Be interesting to see if climbers like it - I'd love to see Wide Boys testing it out.
• greenie_beans 2y ago
  cool technical project, but as a climber, part of the fun is solving the problem with the help of my own brain and the brains of the other climbers around me.
• kettunen 2y ago
  One of the most fun part of climbing is the problem-solving aspect of it. Figuring your way to get up the wall/boulder. Why would you want to get rid of that?
  • rokkamokka 2y ago
    As with any tutor, this could (possibly) help beginners.
  • _fs 2y ago
    Do you enjoy climbing with others? Personally, I enjoy seeing the different beta on a climb, especially between different body types. It is interesting seeing unique approaches to the same problem.
  • hacksoi 2y ago
    You can use it after you do the climb to see if there is a more optimal solution.
  • NegativeK 2y ago
    Where the fun is is an opinion.
  • crazygringo 2y ago
    What about when you try to solve the problem and you get stuck and can't? Or solve it but in a bad way that requires way too much exertion? The point is to try to solve it yourself, then compare with an expert solution, and therefore learn how to improve. If you're just blindly trying to problem-solve through trial and error without ever comparing against expert feedback, you're going to learn climbling extremely slowly...
    • Tronno 2y ago
      Getting stuck is part of the game. It could take several sessions or it could take several years. It's what makes a climb truly hard, a "project". It's an essential part of the sport - the satisfaction of using one's own body and mind to overcome the seemingly impossible. If you never get stuck, how could you possibly experience it?
• silentsea90 2y ago
  Very cool!! I am unfortunately very far from an AI expert. Can you share how you built this? Did you have to train your own model, or is it an open source model that you finetuned for climbing?
  • smandava 2y ago
    It is a new type of biomechanics AI model. More details in our paper which is coming soon!
• justinnhli 2y ago
  I have had a similar idea for the past few years: less an AI model that takes in a photo and generates a video, more a movement planning tool given a 3D model of the wall. Echoing some of the other comments, I think it would be more interesting to not just see a single avatar climbing the problem, but to see many possible approaches to the same problem, even for climbers with the same body type. This way, even skilled climbers that are stuck on a problem can consider alternate beta/techniques that they may find easier to execute.
• gazelle21 2y ago
  Wow, this is great! I do BJJ and wonder how it could AI models could help. I find watching tape to help, not only myself but everyone who I know has recorded their matches has found it helpful.
• blobbers 2y ago
  Fun! How does it know what route you're climbing, if you can't climb it? >_> Sorry if that's an obvious question. Does it just tell you how it would climb things you can already climb?
• hangonhn 2y ago
  This is really cool from a technology point of view but the main attraction of climbing for me is figuring out how to "solve" the climb. Trying to solve a crux with a buddy is half of the fun for me.
  • eigelc 2y ago
    And a quarter of the other half is asking people in the gym how to do it.
  • deleted 2y ago
    [deleted]
  • crazygringo 2y ago
    Right, but for people in their first couple of years of climbing, it can be really helpful to try to solve it, then see what the AI coach says, examine the diff, and thereby improve your solving and climbing skills. I have definitely discovered in my life that I learn probably 10x faster in situations where I can try something, and then immediately compare it with an expert solution. Rather than just trying and trying and trying on my own. And once you're experienced in climbing (like you probably are), then of course you don't need to compare anymore. The joy is in the solving. But expert solutions are really helpful to get to the point where you can consider yourself experienced, so much more quickly.
    • et-al 2y ago
      With bouldering, the set of moves is fairly limited and what actually prevents someone from pulling harder comes down to flexibility, core strength, and contact strength.
      • crazygringo 2y ago
        OK, but the submission and conversation here isn't about bouldering. This is about wall climbing where there's lots of choice for moves.
        • 2y ago
• alexcaza 2y ago
  Cool idea! My friends and I have chatted about building something similar, though with a coaching focus. Specifically for use with spray walls/kilterboards/etc. It could be interesting to combine something like this with an analysis tool. Analyse a climbers attempts or successes and compare it to the beta figured out by the model. Then offer tips on body positioning or technique based on your weight/height/strength.
• deleted 2y ago
  [deleted]
• programmarchy 2y ago
  Not a climber myself but this is super impressive. Curious about what techniques you’re using. Is this based on a GAN model?
• party_possum 2y ago
  It would be really interesting to build an instance of this model trained on world cup footage for a few reasons. - Like all of these things, your training data matters and the internet is awash with videos of people climbing badly. A lot of people specifically post "I can't climb this, what am I doing wrong?" videos. World cup climbers are, by the nature of the competition, extremely talented and technically proficient climbers. Even when they fail, they fail in smart interesting ways. - There's lots of high quality video footage out there. Heck, the problems are even set with visual clarity in mind which would help when parsing that footage. There's potentially enough video to train instances on individual climbers. You could run side by sides like "How would Tamoa climb this and how would Janja climb this?". - World cup problems are stylistically distinct. They involve lots of moves "typical" climbers will never ever encounter. Many climbers will look at a typical gym problem and think "I have an idea of how to climb this" but will look at a world cup problem and just think "????????". An app that told you how a problem like that should be climbed might be useful. There are drawbacks too. - World cup climbers are outliers, whose physical ability (strength, flexibility, etc.) give them access to kinds of movement that other climbers just don't have. No amount of "knowing the sequence" will get me up a climb that requires a full bat hang (look it up) because I just don't have the ankle strength to do the movement. - World cup "style" is only commonly used at high level comps and in very large commercial gyms. It's probably not extremely relevant to a typical climbing session. - World cup problems are very hard. Mostly v10 and up? It would be hilarious to watch a model trained on genetic monsters crushing the world's hardest boulder problems try to tell a doughy office worker (me) how to climb v2.
  • nl 2y ago
    This was exactly my thought. See https://www.instagram.com/p/Cvg3bJkp30a/ https://www.instagram.com/p/Cvg3bJkp30a/ on a super-hard slab problem at Innsbruck 2023 (Sorato Anraku couldn't do it, so you know it's hard...)
• takk309 2y ago
  Could this be used to grade routes? It would be cool if a photo or short video of a route could be used to better assist in grading climbing routes. Grading is a massively subjective thing, to some extent, so this could be super tricky.
  • et-al 2y ago
    I only watched the video, but without taking in account the texture of the holds, the grading aspect would be limited. One also learns to just recalibrate according to regions. E.g. climbing 11d at one gym, versus 12a at another.
    • mordechai9000 2y ago
      Also, how will the AI know when it is appropriate to inflate the grade to make the climbing seem harder, and when it is appropriate to sandbag so I can pretend a route is easier for me than it actually is?
      • takk309 2y ago
        Easy, it'll just grade everything 5.9+!
• latchkey 2y ago
  Of course everyone on HN is focused on the climbing / beta / technique aspect of this. It is ok that it is a solution looking for a problem. There is obviously no 'business' or 'product' here. It isn't like there is a payment link on the page or anything. What I'd like to see the comments focus on is that we should just be happy that someone is making the effort to learn more about AI and building tooling around it. Experimentation is king. They've put their work out into the open (soon to be open sourced even!), not to be criticized over whether it is useful or not, but just that they created something that could spawn other interesting things that solve real world use cases. Huge kudos for doing this work.
  • iosguyryan 2y ago
    really good example of the culture of climbing too!
  • iosguyryan 2y ago
    If I could “yes and” this… route setters could benefit from this tool to simulate layouts against a number of body types and mobility.
    • ungreased0675 2y ago
      Yes, and help setters create more interesting routes. Maybe a layout with multiple ways to solve it, but all the approximate same difficulty?
    • latchkey 2y ago
      I spent 8 years lead climbing at Mission Cliffs in SF 2-5 days a week (5.12 range was my top). They had excellent route setters there that I spent many hours just watching between climbs. The route setters would effectively do what you're saying, while they were setting the route. They knew exactly what would work and what wouldn't. The holds/wall are really what dictate things and if you're not able to climb it... it isn't really their problem. As far as I can tell, while this project is cool, it really has nothing to do with climbing or route setting. It is an AI project where the developer just kind of made up some task and had AI follow it.
• party_possum 2y ago
  Climbing route setting (when done at a high level) is constantly in conversation with itself. Many climbers don't realize that there are trends or fads in movement styles that sweep across the industry. This has only picked up steam in an era of social media. Just like movie dorks will happily spend hours explaining how an individual shot in a movie is actually an insider reference to another movie, and as a result a statement of intent for the movie as a whole, professional route setters will talk your ear off about the way one of their problems embraces or rejects specific kinds of movement trends of the last 6 months. That intentional rejection is interesting. Many route setters, especially for competitions, are in constant search of novelty. One kind of perfect problem is something that looks confusing and impossible, up until you see it done, at which point it seems almost obvious. It's the feeling of solving a sudoku. But critically, they want climbers to be initially confused. I wonder if AI might actually be better than humans at sequencing these kinds of problems. Humans bring so much context and experience and expectation to the process that we are easily tricked. AI just looks through a few terabytes of video and says "What about this?".
  • chefandy 2y ago
    > I wonder if AI might actually be better than humans at sequencing these kinds of problems Probably a deeply unpopular take here, but without knowing anything about climbing routes, I'm gonna say no. I'm not saying that they won't have excellent quality output that might even solve problems that human output can't, but the process of creating something is meaningful, even commercially. Surely this will be useful in some respects, but I just don't buy the idea that humanity is destined to passively consume automated algorithm-generated utility products-- especially creative ones-- no matter how smooth, cheap, and clever they might be.
    • JadeNB 2y ago
      > the process of creating something is meaningful, even commercially. That's true, but why does it mean that the answer to the more or less objective question "will AI actually be better than humans at sequencing these kinds of problems?" (As stated it's not really objective, but one could easily come up with metrics like, say, total time to a correct solution, or time spent observing the route or other climbers, or ….) One can imagine other, related questions that are less objective (like "will it be a good idea to integrate this AI assistance into climbing competitions?"), but, to me, the answer to the (implicit) original question has nothing to do with whether or not the activity is meaningful, or with humans' destiny one way or the other.
• i_am_a_squirrel 2y ago
  If anyone would actually use this (I don't think they would, because the puzzle, and talking about beta is half the fun of climbing)... ...then this would be very valuable data. Eventually you could train AI to generate routes, and then you could fire all of your master setters! Climb setters should unionize and copyright their art before it's too late (like for painters and software developers and musicians)
• Temporary_31337 2y ago
  I feel like I’m the right audience for this - a not very good climber with unusual distribution of limbs. But I’m not interested - part of the fun in climbing is on sight climbs and solving problems yourself.
  • 31337Logic 2y ago
    Thank you for posting this. :-)
  • johnfn 2y ago
    Man, where is the hacking spirit? OP posts a very cool, impressive hack and all everyone can talk about in the comments is how it actually isn't interesting to them.
    • swores 2y ago
      It's great if someone wants to use the "hacking spirit" to help themself climb, it doesn't obligate everyone else to want to use the same technique; and frankly it's (slightly) more interesting hearing people say why they would or wouldn't use it than to have a bunch of comments just saying "awesome that you're doing this" and ignoring whether they'd find it interesting themselves or not.
      • SOLAR_FIELDS 2y ago
        I would say (opinion) that it’s vastly more interesting to hear whether other people would use Thing I Built. I can nerd snipe myself all day on projects. By that I mean I can easily endlessly entertain myself building things, both useless and useful. Whether they are interesting AND useful for other people is quite a large driver for me both for personal and business projects
    • jajko 2y ago
      If somebody solves the 'hacking' part for you, a lot of fun disappears. Yes they probably had tons of fun, but you lose some of it. Figuring a route for the first time is by far the most rewarding part of climbing (although I love all of it). Same with kids - if they grok something on their own, what a reaction and reward compared to being told how to do it.
• surume 2y ago
  The AI analysed my body and said coldly, "Just take the elevator."
  • amelius 2y ago
    Here is an interesting post about how body mass affects climbing ability. Even if you have a great BMI and muscular build, but if you are heavy, then at some point physics starts working against you. https://www.reddit.com/r/bouldering/comments/595a2m/how_much_does_bodyweight_really_factor_in_to_it/ https://www.reddit.com/r/bouldering/comments/595a2m/how_much... > The basic physics of strength/weight is that strength is reliant on the cross sectional area of muscles (number of fibers) and therefore increases quadratically, whereas weight is a function of volume and increases cubically. So the amount of strength you can gain as your mass increases will always lag behind the amount of mass gained. This is why insects are so strong compared to their bodyweight, and why the creatures with the highest strength/weight ratio's are all small. > If we analzye male pro climbers we'll see that the heaviest of them max out at around 170-180 lbs (and there are VERY few of those, the majority being between 130 and 150), and that's not a fluke. In order to be a great rock climber you need strong forearms, and strong lats, and that's about it. Muscle mass, or fat mass basically anywhere else is mostly just pulling you off the wall (a strong core helps too but it doesn't need to weigh a lot). > (...)
• lambdaone 2y ago
  I'm sceptical about the usefulness of this for a number of reasons, including the points raised by others about hold recognition etc. It's also difficult to see how the system would work out the 3D arrangement of holds. It can clearly try to infer 3D relationships from video of an existing climb, but it would be hard for it to work out the 3D position or orientation of any holds not used by the climber in the video. These two put together make solving the climbing problem even more difficult, because appropriate body positions and moves are often very sensitive to even tiny differences in shape, relative position, and orientation of holds; and occluding volumes, arêtes, cracks and so can make the problem even harder. But as a simplified demo, this is cool, and I salute it.
• giantg2 2y ago
  Probably going to show me the best route to the couch.
• eps 2y ago
  Your demo video is monetized and showing 30 seconds of ads. Might not be a bad idea to switch the ads off.
  • deleted 2y ago
    [deleted]
  • dathos 2y ago
    Pff, they're sharing their work for free and you complain about a 30s ad?
    • eps 2y ago
      I assume it's an oversight rather than an actual attempt to earn few pennies.
• alanzhuly 2y ago
  This is really neat. Do you plan to expand this to other sports or activities as well?
  • smandava 2y ago
    Maybe! Depends on the reception.
• nsagent 2y ago
  Great job! A lot of folks from my CS PhD program are really into climbing (including me), but none of us has conducted research that explored climbing. Rather climbing is our respite from research. Anyway, I think this could work really well as an app. Good luck.
  • smandava 2y ago
    Thanks! Still unsure if we will commercially pursue this specific idea or not. This was moreso meant to be a public progress marker from our broader work pursuing virtual avatar simulations. If enough interest is there, we will think about it. But, for now, this (will be) a paper and some open sourced code.
• colehud 2y ago
  Saw tennis on your LinkedIn - is SABR a nod to the Sneak Attack By Roger?
  • smandava 2y ago
    Ha! That was unintended. "You can't connect the dots looking forward; you can only connect them looking backwards." - someone somewhere (likely Steve Jobs)
• ropable 2y ago
  I'm genuinely curious how the model classifies holds. I could (hypothetically) see this doing something like assigning more utility to certain holds (e.g. a steep sloper that is impossible to use as a handhold but is can be used as a smearing foothold) and thereby giving bad/implausible beta. That being said, it seems like an obvious shortcoming and I'm very interested in learning more!
• nl 2y ago
  I have a side project in a related space. For mine I'm syncing climbing between different videos automatically. I find this pretty useful because it highlights difference. Some examples: https://www.instagram.com/p/Cvg3bJkp30a/ https://www.instagram.com/p/Cvg3bJkp30a/ https://www.instagram.com/p/Cpuv2zvptdW/ https://www.instagram.com/p/Cpuv2zvptdW/ It's actually syncing in 3D so is fairly independent of camera angle (as the second video shows)
• teapourer 2y ago
  Looks cool! It definitely looks technically challenging, especially for expert climbers who have a variety of creative moves at their disposal. But I can definitely see this being useful for novices.
• tuatoru 2y ago
  But does it do Hammersmith Broadway?
• monroewalker 2y ago
  This is cool and all but I can't help but wonder if it'd be even cooler to instead show the beta for climbing out of the negativity found in the comments here. /s But really though, awesome project and congrats on getting it to this point! I'm excited for the day when something like this can be used to critique form
• bryanrasmussen 2y ago
  I guess it's just how my mind works but I thought this will make a great part of some stories I'm working on, only adapted for cat burglars to figure out the optimal route on a building beforehand. Maybe also good thing for the next Mission Impossible movie if they ever make one.
• kriro 2y ago
  Very nice. Please do open it up as much as possible. I'm curious about the data pipeline. I love bouldering and have thought about different ML ideas in that domain. I'm also an avid beta breaker. Finding alternative solutions somehow tickles my hacker-brain. So my first idea was intended beta vs. alternatives but I was only thinking in 2d and images :) Another idea I've played around with is more objective grading but labeling is kind of hard especially with gyms rotating routes so I thought about using Kilterboatd data and transfer learning it or maybe unsupervised approaches. Congrats on a cool idea and most importantly the execution.
• raptorraver 2y ago
  Cool project but please delete it. You’re taking the fun part out from bouldering and just leaving us the labour ;) Edit: apprently I wasnt the first to make this joke. I emphasise that this is Really a cool project. Great job!
  • RunSet 2y ago
    https://i.imgur.com/S7ugEXC.mp4 https://i.imgur.com/S7ugEXC.mp4
• nachteilig 2y ago
  Very cool project. I will try this out when you release it
• some_random 2y ago
  This is rad and I can't wait to try it out for myself
• copypasterepeat 2y ago
  Very cool idea but I wonder how well this can possibly work in practice. So many bouldering problems hinge on small things, such as how a little crimp hold is oriented or how deep the groove in it is. It can be hard to determine this even in person until you get up close or try to grab it. Additionally, this model seems to take into account the body type but not other relevant factors, such as climber's strength and flexibility/mobility. Even in the demo, that suggested move of putting the right foot high and pushing off the jug up and to the right could not be done by a lot of new climbers. This parallels what I've personally seen when watching others climb. For example, some climbers are exceptionally strong and can basically campus their way up a route. Watching their beta is irrelevant to me. I don't have the same strength to pull those moves.
• ericmcer 2y ago
  This has always been a dream, because there is a "perfect" way for your body to climb a route but it would be insanely complex to calculate, especially when factoring in every tiny physiological aspect like ankle strength and hamstring flexibility. This is a really cool step though. Maybe a more useful variant with the current tech would be isolating a single move and seeing every variation the AI can think up.
• southernplaces7 2y ago
  Considering the notoriety of AI models in making confident but blatant mistakes or simply lying about things they don't have real answers to I don't know how smart it is to trust one with something in which bad advice can kill you very quickly and immediately.
• ericzakariasson 2y ago
  can't wait to try haha