5 ms·
Building an AI that watches rugby
- xnx 2y agoBetter title: "LLM OCR on Rugby screenshots to read score and clock"
- djtango 2y agoI don't quite get how diffing frames allows you to find the scores. TFA mentions comparing a frame with and without - but how do you generate that frame without? If you can already do it, what's useful about doing that?
- barbegal 2y agoI think the text is wrong, it's diffing two frames and the areas that are the same are where the scorebaord is as this doesn't change between frames but everything else does.
- nirvael 2y agoI was also confused by this. I think you're right, but in the original text they specifically mention a 'static background' that they remove, so it's not just a simple 'wrong way round' error, it's a fundamental misunderstanding of what's happening. Makes me wonder if the author actually knew what they were doing, or just using an LLM to vibe-code everything.
- sebastiennight 2y agoHe's diffing the frames, and then the only pixels that stay the same are the UI, from which he doesn't directly get the UI (see the example, it's illegible) but he can extract the POSITION of the UI on the screen by finding all the non-red pixels. And then he does a good ol' regular crop on the original image to get the UI excerpt to feed the vision model.
- deleted 2y ago[deleted]
- itissid 2y agoWhy does yolo not work?
- mon_ 2y agoWhy the focus on scorekeeping? I feel like an AI model is overkill here, when you have text-based sources readily available such as news apps, Twitter feeds, and apps such as Livescore which would be easier and cheaper to scrape. They probably cover more matches that aren't televised too. I'd be curious to see what useful insights could be gleamed from the match commentary. You have the main commentator giving play-by-play objective reporting and then a 'colour' commentator giving some subjective analysis during breaks in play. I bet there's a lot of interesting ways this could be used.
- dmurray 2y agoThe only interesting part of the model's output was { "current_play": "ruck", } So the vision model can correctly identify that there's a ruck going on and that the ball is most likely in the ruck. Why not build on this? Which team is in possession? Who was the ball carrier at the start of the ruck, and who tackled him? Who joined the ruck, and how quickly did they get there? How quickly did the attacking team get the ball back in hand, or the defending team turn over possession? What would be a good option for the outhalf if he got the ball right now? All of these except the last would be straightforward enough for a human observer with basic rugby knowledge going through the footage frame by frame, and I bet it would be really valuable to analysts. It seems like computer vision technology is at a stage where this could be automated too.
- MuffinFlavored 2y agoESPN has play by play stuff for free like this on their website for some other sports not sure if it is done by a human or not curious how “an AI can do it” yields much difference in terms of result for the casual watcher
- brookst 2y agoIf this is the final product, not much difference at all. But where the human version is pretty much as far as it’s going to go, this is v0.01 of the AI version. Pretty soon the AI will be predicting what will happen next, commenting on whether this was a good idea (based on statistics), and letting the viewer ask questions about what exactly happened and why.
- walthamstow 2y agoI'm not a rugger bugger but every 5 seconds doesn't really seem like often enough to be taking screenshots. In soccer anyway, a lot can happen in 5 seconds.
- conductr 2y agoMy american football brain had the same reaction. Many of the most pivotal plays are replayed in slow motion as commentators and spectators debate on what actually happened and if the refs got the call right. Also, the average play (ie. 'down') is 4-5 seconds, so not nearly enough data to determine what is going on.
- dncornholio 2y agoSo Rugby is missing a lot of data beside the scoreline, so they created an AI that can extract the scoreline.
- patapong 2y agoI want AIs that clean my apartment while I watch rugby, not AIs that watch rugby while I clean the apartment. ;) In seriousness, this is a cool project and show how sophisticated analysis LLMs can do in a plug and play manner. They may not always be the best solution but a fantastic baseline that can be deployed and adapted to a usecase in less than an hour.
- petesergeant 2y ago> We can’t hire analysts to watch every match and enter data manually. I'm surprised there's not enough fans willing to do that if you could gamify it.
- securingsincity 2y agoThis is a position in baseball. https://www.wbur.org/news/2025/03/30/fenway-park-boston-baseball-score-keeping https://www.wbur.org/news/2025/03/30/fenway-park-boston-base... Here's a radio piece about the official fenway park score keeper from two weeks ago
- goeiedaggoeie 2y agoReading the scoreboard from a TV screen and selling that data is restricted in many jurisdictions. This work is pretty naive I think.
- brookst 2y agoHas there ever been a hacker whose top priority is ensuring compliance with every regulation in every jurisdiction worldwide?
- deleted 2y ago[deleted]
- stronglikedan 2y agoGood thing they're in only one jurisdiction, not many.
- hash872 2y agoI don't think it's possible to be in compliance with every law in every jurisdiction simultaneously. There are over 300,000 federal laws in the US, and apparently no one knows how many laws each of the 50 states has. That's 1 of the world's 195 countries
- arathis 2y agoUgh. You’re the worst
- FiberBundle 2y agoI'd say it isn't in the US though, at least there's precedent that makes me think that: https://en.m.wikipedia.org/wiki/National_Basketball_Ass%27n_v._Motorola,_Inc https://en.m.wikipedia.org/wiki/National_Basketball_Ass%27n_...
- goeiedaggoeie 2y agoI think you misunderstand that case "compiled its scores and statistics by employing people to listen or watch the games, then enter the scores on the computer which transmits the scores to STATS' on-line service, to be sent out to anyone using a SportsTrax pager.[1]" Notice how they watched the game and got the statistics like that. The restrictions are about using the scoreboard and the data displays and reselling/commercialising that data. It is however legal to watch the game and compile and distribute your own stats due to the game entering public domain. Due to this many betting companies and data collection companies have to pay people to watch the game vs just scraping the scoreboard (which is the context from which I learnt about this). ironically at venue OCR is a common way to get scoreboard data.
- glacier5674 2y ago[dead]
- sebastiennight 2y agoI love that as soon as he writes, > The plan was simple. You know you're in for a funny read. More seriously though, the JSON example from a vision language model is interesting but does not take into account how much extrapolation (hallucination) the model will insert over time. For instance, even if not visible in the image, your VLM will probably start inserting details (such as the color of the team's jersey) based on knowing the team's three-letter identifier. So the reliability of the system will go down over time, and it probably compounds if you're using some of that info to feed further steps in the loop.
- hummuscience 2y agoThe moment I started reading this, I got reminded of this recent study: https://arxiv.org/html/2503.10212v1 https://arxiv.org/html/2503.10212v1 The scope is a bit different. The study uses an LLM to interpret pose estimation data and describe the behavior in each frame. The output is text which can be used to create embeddings of behavior. As someone who works in ethology, that's a clever (but maybe expensive) idea. I think the author could use something similar. With multi-person pose estimation models.
- chrsw 2y agoDoes this mean there's probably AI that's already watching high profile football (soccer) matches?
- thom 2y agoDepends on your definition of AI, but yes, lots of them, and not just the high profile matches.
- damnitbuilds 2y agoTL;DR: It extracts the score from the video and gets text from the commentary in the audio. I was hoping for more.
- disjunct 2y agoI wrote a similar script that used a TV tuner during the last World Cup. Since I had an ATSC source, I was able to just pull the CTA-708 captions directly and with little delay.
- numpad0 2y agoaw. I thought this would be about an AI cat that makes wrong commentaries that you can make pointless arguments against. There should be one.
- deleted 2y ago[deleted]
- cewl123 2y agoI want AI that does my job while I watch rugby
- rfdearborn 2y ago> Sending a full-resolution screenshot every five seconds gets expensive fast. For now.
- AllSuperIndians 2y ago[dead]
- 4ndrewl 2y agoMy observation is that watching Rugby on TV is no the same as watching a Rugby match. You're watching something where choices have been made around what you're to see, so your model is already restricted in what it can see. You really need to take a 'full pitch' feed directly from the venue, rather than what is broadcast.