7 ms·
What is Realtalk’s relationship to AI? (2024)
- pmkary 1y agoThere never is anything more satisfying than to see maestro Victor, or maestro Kay being up voted in the hacker news.
- lif 1y agothanks for sharing this, seems like a very idealistic project, had not heard of Bret nor dynamicland before.
- iambateman 1y ago"Stop drawing dead fish" by Bret has stuck with me for a decade – https://www.youtube.com/watch?v=ZfytHvgHybA https://www.youtube.com/watch?v=ZfytHvgHybA
- mumbisChungo 1y agoVery much worth the watch if you haven't seen this one before.
- jasonjmcghee 1y agoHighly recommend one of the most memorable talks I've ever seen... Inventing on Principle: https://www.youtube.com/watch?v=PUv66718DII https://www.youtube.com/watch?v=PUv66718DII
- deleted 1y ago[deleted]
- randomNumber7 1y ago> we aim for a computing system that is fully visible and understandable top-to-bottom I mean even for something that is in theory fully understandable like the linux kernel it is not feasible to actually read the source before using it. To me this really makes no sense. Even for traditional programming we only have so powerful systems because we use a layered approach. You can look into these layers and understand them but it is totally out of scope for a single human being.
- stego-tech 1y agoThat’s because you’re conflating “understanding” with “comprehension”. You can understand every component in a chain and its function, how it works, where its fragilities lay or capabilities are absent, without reviewing the source code for everything you install. To comprehend, however, you must be intimately familiar with the underlying source code, how it compiles, how it speaks to the hardware, etc. I believe this is the crux of what the author is getting at: LLMs are, by their very nature, a black box that cannot ever be understood. You will never understand how an LLM reached its output, because their innate design prohibits that possibility from ever manifesting. These are token prediction machines whose underlying logic would take mathematicians decades to reverse engineer even a single query, by design. I believe that’s what the author was getting at. As we can never understand LLMs in how they reached their output, we cannot rely on them as trustworthy agents of compute or knowledge. Just like we would not trust a human who gives a correct answer much of the time but can never explain how they knew that answer or how they reached that conclusion, so should we not trust LLMs in that same capacity.
- why_at 1y agoI get that LLMs are a black box in ways that most other technologies aren't. It still feels to me like they have to be okay with abstracting out some of the details of how things work. Unless they have a lot of knowledge in electrical engineering/optics, the average user of this isn't going to understand how the camera or projector work except at a very high level. I feel like the problem with LLMs here is more that they are not very predictable in their output and can fail in unexpected ways that are hard to resolve. You can rely on the camera to output some bits corresponding to whatever you're pointing it at even if you don't know anything about its internals.
- gmueckl 1y agoBuilding projection optics is a bench top experiment that we did in 7th grade in school. Electric circuitry isn't exactly rocket science, either. Things like LCD panels for projecting arbitrary images and CCD chips for cameras become harder to understand. But the point is to make users understand the system enough to instill the confidence to change things and explore further. This is important because the true power of computer systems comes from their flexibility and malleability. You can never build that level of confidence with LLMs.
- thatguymike 1y agoI'm sympathetic, but I do think Realtalk could be improved with some simple object recognition and LLMing. One of the challenges I found when I played with RealTalk is interoperability. The aim is to use the "spacial layer" to bootstrap people's intuitions on how programs should work, and interact with the world. It's really cool when this works. But key intuitions about how things interact when combined with each other, only work if the objects have been programmed to be compatible. A balloon wants to "pop if it comes into contact with anything sharp". A cactus wants to say "I am sharp". But if someone else has programmed a needle card to say "I am pointy", then it won't interact with the balloon in a satisfying way. Or, to use one of Dynamicland's favorite examples: say I have an interactive chart which shows populations of different countries when I place the "Mexico card" into the filter spot. What do you think should happen if I put a card showing the Mexican flag in that same spot, or some other card which just says the string "Mexico" on it? Wouldn't it be better if their interaction "just works"? Visual LLMs can aid with this. Even a thin layer which can assign tags or answer binary questions about objects could be used to make programs massively more interoperable.
- rtkwe 1y agoThat's similar to the issue with the whole NFT craze where you'd "take items from one game to another", it requires everything to work with everything. For Dynamicland I get the issue though putting the whole thing through an LLM to make pointy and sharp both trigger the same effects on another card would just hide the interaction entirely. It could or couldn't work for reasons completely opaque to both designer and user.
- smj-edison 1y agoThe way you'd figure this out in dynamic land is you'd look at the balloon, which by custom would have the code taped on somewhere. You'd read that code, figure out what it's looking for, and write said trigger.
- smj-edison 1y agoI just realized that the OP said they had already played around with real talk, but it's too late to edit: sorry for assuming that the printed code is sufficient! Was term mismatch one of the biggest issues you ran into, and if so, was it that the printed code didn't contain enough information?
- Animats 1y agoHere's a video of Dynamicland.[1] The textual description doesn't tell you much. It's still at the cool demo level, though. How do you scale this thing? [1] https://www.youtube.com/watch?v=7wa3nm0qcfM https://www.youtube.com/watch?v=7wa3nm0qcfM
- chubot 1y agoWhat do you mean by “scale”? It’s designed to be decentralized, and promote agency of small, co-located groups of people The typical “scale” mindset is almost the opposite of that — the people doing the scaling are the ones with agency, and the rest get served slop they didn’t choose! If the system is an unreliable demo, then that can promote agency. In the same way that you could fix your car 40 years ago, but you can’t now, because of scaled corporate processes.
- quonn 1y ago> you could fix your car 40 years ago, but you can’t now, because of scaled corporate processes. You can fix your car just fine - just not the electronics. And those were to a large degree added for safety reasons. It is due to the complexity that they are difficult or impossible to fix.
- AnthonyMouse 1y agoThe electronics don't have any more complexity than any other computer system. If you can fix your PC you could fix your car's electronics. Except that they aren't documented. So then your service light comes on, and the car has all kinds of detailed information about why, but the manufacturer doesn't give it to you because they want you to take it to the stealership so they can pick your pocket or try to sell you a new car instead of fixing it yourself or taking it to an independent mechanic. This isn't about the cost; they already pay the cost to write the documentation or software for their own dealerships. It isn't about other carmakers; any company large enough to actually make a car would have no trouble getting a copy of it from one of the dealers. The only reason it's not published on their websites is that they don't want the vehicle owners and independent mechanics to have it, which is spiteful and obnoxious.
- bobajeff 1y agoI've never experienced dynamicaland in person (only seen videos). However, one concern I have about it's demos so far is that they use a projector. So you need a room dark enough to for the projected light and you need to keep your heads, hands, and body out of the way of it.
- fzzzy 1y agoThis is true, but modern laser projectors are very, very bright. I use one as my main computer display with no problems with the blinds open, and the sun shining in. Occlusion is definitely a problem.
- rtkwe 1y agoProjectors have been strong enough to be visible in decently lit rooms for ages. The reason you want the room extremely dark for most projector setups is for contrast, because the darkest thing you can make on a projected image is the ambient surface illumination (and the brightest is that surface under full power from your projector [0]). If you accept that compromise you don't need a super dark room, the recommendation for tight light control is mostly for media viewing where you want reasonable black levels. Do still need to keep hands out of the light to see everything but that can also be part of the interaction too. If we ever get ubiquitous AR glasses or holograms I'm sure Bret will integrate them into DL. [0] Which leads to a bit of a catch 22 you want a surface that looks dark but prefectly reflects all the colors of your projector so you need a white screen which means you ideally want zero other light other than the projector to make the projector act the most like a screen.
- Miraste 1y ago>you need to keep your heads, hands, and body out of the way of it. I've seen systems like this that use multiple projectors from different angles, calibrated for the space and the angle. They're very effective at preventing occlusion, and it takes fewer than you'd think (also see Valve's Lighthouse tech for motion tracking). Unfortunately, doing that is expensive, big, and requires recalibrating whenever it's moved.
- ijk 1y agoThe light level isn't an issue in practice: when I visited the actual installation during the day, the building was brightly lit with natural light and the projections were easily visible, to the point that I didn't think about it at the time.
- deosjr 1y agoSuch an amazing project. I've made a lot of progress recently working on my own homebrew version, running it in the browser in order to share it with people. Planning to take some time soon to take another stab at the real (physical) thing. Progress so far: https://deosjr.github.io/dynamicland/ https://deosjr.github.io/dynamicland/
- why_at 1y agoI'm only just now reading about Dynamicland for the first time, so maybe I'm not understanding something obvious. The text description is not very helpful, as far as I can tell from pictures it's a place where you can move around physical objects and papers to do computer programming type stuff? Under visibility they say: >To empower people to understand and have full agency over the systems they are involved in, we aim for a computing system that is fully visible and understandable top-to-bottom — as simple, transparent, trustable, and non-magical as possible But the programming behind the projector-camera system feels like it would be pretty impenetrable to the average person, right? What is so different about AI?
- rtkwe 1y agoDynamicland is bootstrapped in a sense, [0] the same way you write the first compiler/interpreter for your code in another language then later write it in it's own language. The code running the camera and projector systems is also running from physically printed programs in one of the videos you can see a wall that's the core 'OS' so to speak of Dynamicland. I think the vision is neat but hampered by the projector tech and the cost of setting up a version of your own, since it's so physically tied and Bret is (imo stubbornly) dedicated to the concept there's not a community building on this outside the local area that can make it to DL in person. It'd be neat to have a version for VR for example and maybe some day AR becomes ubiquitous enough to make it work anywhere. [0] Annoyingly it's not open sourced so you can't really build your own version easily or examine it. There have been a few attempts at making similar systems but they haven't lasted as long or been as successful as Bret's Dynamicland.
- why_at 1y agoThat's pretty cool. I figure this is explained in some of the videos but I can't watch them right now. I'm reading more about the "OS" Realtalk >Some operating system engineers might not call Realtalk an operating system, because it’s currently bootstrapped on a kernel which is not (yet) in Realtalk. You definitely couldn't fit the code for an LLM on the wall, so that makes sense. But I still have so many questions. Are they really intending to have a whole kernel written down? How does this work in practice? If you make a change to Realtalk which breaks it, how do you fix it? Do you need a backup version of it running somewhere? You can't boot a computer from paper (unless you're using punch cards or something) so at some level it must exist in a solely digital format, right?
- deleted 1y ago[deleted]
- ijk 1y agoRealTalk has some interesting features that I wish there was a more complete writeup that explained it in detail. Like, you can write a script that talks to functionality that may or may not exist yet. Programming by moving pieces of paper around deservedly gets attention, but there's a lot more to it.
- comeondude 1y agoIm genuinely blown away by llms. I’m an artist who’ve always struggled to learn how to code. I can pick up on computer science concepts, but when I try to sit down and write actual code my brain just pretends it doesn’t exist. Over like 20 years, despite numerous attempts I could never get past few beginner exercises. I viscerally can’t stand the headspace that coding puts me in. Last night I managed to build a custom CDN to deliver cool fonts to my site a la Google fonts, create a gorgeous site with custom code injected CSS and Java (while grokking most of it), and best part … it was FUN! I have never remotely done anything like that in my entire life, and with ChatGPT’s help I managed to it in like 3 hours. It’s bonkers. AI is truly what you make of it, and I think it’s an incredible tool that allows you to learn things in a way that fits how your brain works. I think schools should have curriculum that teaches people how to use AI effectively. It’s truly a force multiplier for creativity. Computers haven’t felt this fun for a long time.
- zx8080 1y ago> a gorgeous site with custom code injected CSS and Java (while grokking most of it) From the context, it's not Java, but Javascript.
- comeondude 1y agoGotcha. Thanks.
- le-mark 1y agoThis was the most surprising/disturbing/enlightening part of the post imo. Surpring; this person literally had no clue! Disturbing; this person literally had no clue? Enlightening; this person literally did not need a clue. My takeaway as an AI skeptic is AI as human augmentation may really have potential?
- comeondude 1y agoJust a vision and goal. I feel like, AI makes learning way more accessible, at least it did for me, where it evoked a childlike sense of curiosity and joy for learning new things. I’m also working on a Trading Card Game, where I feed it my drawings and it renders it into final polished form based on visual style that I spent some time building in chat GPT. It’s like an amplifier / accelerator. I feel like, yes while it can augment us, at the end day it depends on our desire to grow and learn. Otherwise, you will end up with same result as everybody else.
- fouc 1y agoreminds me a bit of http://tunes.org http://tunes.org (and I'm sure there's many more).. so cool to see deep exploration of computing/operating system ideas.