15 ms·
I have written a JVM in Rust
- haspok 3y agoNice project, congrats! One thing struck me as a bit odd: > In particular, it does not support: generics What kind of support is there for generics in the JVM? Maybe I'm too naive to assume that due to type erasure on bytecode level everything is just an Object, ie. a reference type? Or do you mean the class definition parser - but then, you don't really have any checks in place to see if the class file is valid (other than the basic syntax)?
- xxs 3y agopretty much this - generics have (rare) implications to the reflection (but it's unsupported as well) but overall they are replaced with the nearest class/interface when compiled. OTOH lack of string interning is super strange [it's trivial to implement], and w/o it JVM is not a thing. String being equal by reference is important, and part of JLS. Lack of thread makes the entire endeavor a toy project.
- senorrib 3y ago“I want to stress that this is a toy JVM, built for learning purposes and not a serious implementation.”
- xxs 3y agoThe rest of the stuff, incl. I/O is actually on the trivial side - threads do require planning. This is what I meant by being a 'toy' project, threads (and JMM) would be impossible to bolt in later on.
- ncallaway 3y agoThe reason you're being downvoted is you keep dismissing this as a "toy" project, and pointing out that it would be hard to make a real project. But, as the previous commenter attempted to point out to you this project is a *self-described* toy project. On the very page that is linked, the author of the JVM specifically says: "I want to stress that this is a toy JVM, built for learning purposes and not a serious implementation." Thus, absolutely no one disagrees that its a toy JVM. They just want you to stop being dismissive of someone's toy project by repeatedly pointing out its a toy project and not a "not a thing"
- __jem 3y agoRight, just because something is a "toy" doesn't mean it's not still impressive. If someone implemented a "toy" database that could parse and execute SQL queries, distribute data across nodes, etc., you would probably not want to use that in production, but it's still a very impressive project for a single person to pull off, even if it's riddled with bugs. Getting a very complex system to "just barely functional" is still a huge achievement and very cool!
- xxs 3y agoBeing impressing or otherwise is very subjective. SQL (ACID) over multiple non-cache-coherent nodes is extremely difficult to pull with regards to consistency, though.
- xxs 3y agoI've pointed the only part that makes it a toy project is the lack of Threading support, the rest is not hard to add. So the items in list of things missing after 'toy' thing should have totally different weights (with Threads being the added to the last).
- lolinder 3y agoYou're still missing the point—it was always intended to be a toy project, and the author has explicitly declared that they are completely done with it and won't be doing any more work. What does it matter how they sort the list of missing items? It's not a todo list in need of prioritization, it's just an "FYI, these are some of the things I never got to".
- 3cats-in-a-coat 3y agoJava strings are compared by reference, if they do not match, they're compared by value. There's no guarantee every single string has a single instance. That would hurt performance.
- maverwa 3y agoI think op meant "String literals". For those the spec seems to require interning: > Moreover, a string literal always refers to the same instance of class String. This is because string literals - or, more generally, strings that are the values of constant expressions (§15.28) - are "interned" so as to share unique instances, using the method String.intern. And later: > Literal strings within different classes in different packages likewise represent references to the same String object. Source: https://docs.oracle.com/javase/specs/jls/se8/html/jls-3.html#jls-3.10.5 https://docs.oracle.com/javase/specs/jls/se8/html/jls-3.html... But that does - as far as I can see - say nothing for non-literal strings. [edit]: formatting
- 3cats-in-a-coat 3y agoThanks, makes sense yes. Still if the JVM look up in all cases defers to value after ref mismatch, it should work identically, no? Even if interning is mandatory as per spec, I'm not sure how it'd change the outcome of evaluation.
- maverwa 3y agoyeah, I'd assume as much. If it indeed falls back to a by-value comparison it would be slower, but should work.
- xxs 3y agonope - it'd be plain wrong. Literals must be equal by reference, comparing them by value would just break JLS, as they would be equal to any other composed string by reference as well.
- 3y ago
- whizzter 3y agoNot entirely correct, last I checked string interning was ONLY guaranteed for those strings defined in source and read in during class loading, strings created via the String constructor (f.ex. via StringBuilder) CAN duplicate those strings that you hardcoded in your sources, to get the "canonical" string in those cases you have to invoke String.intern() if memory serves me correct. https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/String.html#intern https://docs.oracle.com/en/java/javase/11/docs/api/java.base...() Also interning strings to optimize equality checks to be able to use pointer comparison is dangerous for external inputs since iirc at some point interned strings could permanently be stored (unless implemented by a WeakSet) and attackers could fill up your heap (or cause other GC issues since the entire interning functionality is a cache) by filling up your interning lists with crap.
- xxs 3y ago>String being equal by reference is important, and part of JLS. I never said String must be equal by reference when their content is. However string literals must be equal by reference. I thought Mentioning the JLS would make it obvious, esp. having 'intern' in the context
- ywei3410 3y agoThere's also the new JVM option which eludes me at the moment which sweeps the strings which are promoted to the older generation and interns them. Not certain about whether `String.intern` is permanently stored; I rather suspect that it sweeps the existing strings since iirc the java string has a hash associated with it anyway.
- znpy 3y ago> Lack of thread makes the entire endeavor a toy project. yeah, as stated by the author in the line that says "I want to stress that this is a toy JVM, built for learning purposes and not a serious implementation."
- xxs 3y agoYeah, that's the only part that makes it a toy project - the rest can be added w/o too much of an effort. This is pretty much what makes it a toy project.
- newmana 3y agoThey might be talking about the checkcast operation: https://docs.oracle.com/javase/specs/jvms/se8/html/jvms-6.html#jvms-6.5.checkcast https://docs.oracle.com/javase/specs/jvms/se8/html/jvms-6.ht... This is generated when you do something like: final Main value = list.get(0); http://henrikeichenhardt.blogspot.com/2013/05/how-are-java-generics-implemented-under.html http://henrikeichenhardt.blogspot.com/2013/05/how-are-java-g...
- xxs 3y agoThe cast is added by javac, so it just needs to verify the object on the stack to be compatible w/ the provided class. That part is very simple.
- andreabergia 3y agoThanks! About the generics - some people have pointed out the same on reddit, and yeah, you are correct. The only thing that should be done is to read the Signature attribute that encodes the generic information about classes, methods, and fields (https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.html#jvms-4.7.9 https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.ht...) As a matter of fact, I just did a test and the following code works! :-) public class Generic { public static void main(String[] args) { List<String> strings = new ArrayList<String>(10); strings.add("hey"); strings.add("hackernews"); for (String s : strings) { tempPrint(s); } } private static native void tempPrint(String value); }
- maverwa 3y agoGreat project, congrats! Mad respect. Started working on something very similar a few years back and gave up pretty soon for some stupid reason. Maybe I should try again, am getting better at getting stuff done.
- zote 3y agoPlease do
- cmrdporcupine 3y agoYou should. I've been doing this kind of stuff on the side for 25+ years, and nothing ever gets "done." But the educational value is much higher than you think. And the skills learned can eventually propagate out into interesting paid work. I think of them as retirement projects before I retire. When I actually retire I'll maybe finish them. (That said, I have in the past tried taking jobs that were adjacent to my "research" interests, and found the joy of building these things from scratch is much better than fiddling with the levers on the side of someone else's thing they built from scratch years ago. I like working on and improving production systems, but if they intersect too closely to my personal interests, it can be demoralizing.)
- andreabergia 3y agoThanks! I know the feeling - this project, like most of my other side projects, got abandoned a couple of times. But I was really curious about implementing a GC and, for once, I managed to finish something. I'm glad I did! :-)
- techn00 3y agoSee also https://jacobin.org/ https://jacobin.org/ for JVM 17 written in Go.
- xmcqdpt2 3y agoAlso https://github.com/lihaoyi/Metascala https://github.com/lihaoyi/Metascala for a JVM implemented in Scala running on the JVM.
- dimgl 3y agoSeems… redundant, no?
- cmrdporcupine 3y ago"The goal of Metascala is to create a platform to experiment with the JVM: a 3000 line JVM written in Scala is probably much more approachable than the 1,000,000 lines of C/C++ " Seems like a reasonable goal.
- RcouF1uZ4gsC 3y agoI seriously doubt the ratio of Scala vs C++ for implementing the JVM is 1:300.
- patrec 3y agoHe's not saying that. What he is saying is that are simple, non-production quality implementation in Scala is much more amenable to experimentation than a sophisticated, production-quality implementation in C++ that weighs in at 300x the LOC.
- RcouF1uZ4gsC 3y agoBut a simple non-production quality implementation in C++ would also be amenable to experimentation and not have the bootstrapping issues as well as provide an easier starting point to incorporate more of the existing optimizations as desired.
- cmrdporcupine 3y agoGreat learning project, I'm glad the author is having fun. Implementing a VM from scratch is a blast, and I have learned so much in the past doing that kind of thing. If they're interested in bolting on a GC, it couldn't hurt to look at MMtk. (https://www.mmtk.io/ https://www.mmtk.io/) Some high quality collection algorithms, written to be pluggable to various VMs, and written in Rust.
- sbt567 3y agoUh, first time hearing mmtk. Thanks for the link!
- cmrdporcupine 3y agoI only became aware of it because a former employer (RelationalAI) was heavily interested in replacing Julia's GC with it (for some workloads): https://pretalx.com/juliacon2023/talk/BMBEGY/ https://pretalx.com/juliacon2023/talk/BMBEGY/
- celeritascelery 3y agoNote that MMTK is x86 only. I was going to use it for a toy project but I have a Mac.
- cmrdporcupine 3y agoThat's a bummer -- I guess I never noticed that, when I played with it before it was on an M1 Mac, but compiled into an x86 Julia executable & running through Rosetta (which surprisingly did not suck). Haven't read, but I bet it's likely related to expectations around the x86_64 memory model & atomics. In the long run I see no reason why it couldn't be made portable, but I imagine the authors efforts are elsewhere for now.
- hajmo97 3y agoHow often will this post be reposted on HN ? Recent posts: https://news.ycombinator.com/item?id=36735344 https://news.ycombinator.com/item?id=36735344 - 6 days ago https://news.ycombinator.com/item?id=36717967 https://news.ycombinator.com/item?id=36717967 - 7 days ago https://news.ycombinator.com/item?id=36710803 https://news.ycombinator.com/item?id=36710803 - 7 days ago (OP) Btw. nice project!
- capableweb 3y agoDoesn't really count as a repost if none of the previous submissions didn't get any traction.
- dgb23 3y agoI agree, plenty of technically interesting projects don’t get discussion on hn. There’s always a bit of luck and context involved.
- freedomben 3y agoGetting the attention needed for front page is a huge chance. I've seen great stuff get posted 5 or more times before it makes it out of obscurity. HN is highly non-deterministic in these things.
- bkkz5046 3y agobut more importantly does edbrowse work here?
- tenaf0 3y agoShameless plug of my similar project: https://github.com/tenaf0/rust-jvm3 https://github.com/tenaf0/rust-jvm3
- sproketboy 3y ago[dead]
- FrustratedMonky 3y agoJust for kicks, has anybody tried to Rube-Goldberg it, and see how many VM's can stack on top of each other. Like have Java App running on JVM written in RUST, Running on WASM, Running on JVM, etc.. etc...
- _joel 3y agoIt's turtles, all the way down..
- PTOB 3y agoAnd Logo drives the turtles.
- post-it 3y agoSee also: The Birth & Death of JavaScript, A talk by Gary Bernhardt from PyCon 2014 (https://www.destroyallsoftware.com/talks/the-birth-and-death-of-javascript https://www.destroyallsoftware.com/talks/the-birth-and-death...)
- Zambyte 3y agoThis talk isn't about VMs, but it is about an infinite tower of interpreters. You may find it interesting: https://youtu.be/SrKj4hYic5A https://youtu.be/SrKj4hYic5A
- sylware 3y agoCan I compile it with the rust-written rust compiler?
- encody 3y agoMissed opportunity to call it Just
- deleted 3y ago[deleted]
- orlp 3y agoJust is already 'taken' in the Rust community as it is a command runner utility like make: https://github.com/casey/just https://github.com/casey/just
- entropicdrifter 3y agoOoh, I like that. `just build` feels like a good plea to make to the Rust compiler lol
- 1letterunixname 3y agoHaha. Shout out to homeboy Xoogler Rodarmor.
- brunoborges 3y agoJustVM
- celeritascelery 3y agoI am curious if your ran into limitations due to the lifetimes on this signature fn execute_instruction( &mut self, vm: &mut Vm<'a>, call_stack: &mut CallStack<'a>, instruction: Instruction, ) -> Result<InstructionCompleted<'a>, MethodCallFailed<'a>> When I try to add a lifetime to the `Err` variant of a `Result` and that lifetime is invariant (which it is due to `vm` and `call_stack`) it usually means that I can't use the question mark operator or have early returns in the code[1]. This makes error handling more verbose and less readable. Is that your experience as well? [1] https://users.rust-lang.org/t/nll-and-early-return-not-allowed/85937 https://users.rust-lang.org/t/nll-and-early-return-not-allow...
- celeritascelery 3y agoEDIT: Looks like this is not an issue because the invariant lifetime 'a is not used for the mutable reference of vm or call_stack. So it's not the invariance that is the problem, but rather how Rust reasons about the lifetime of mutable references, which this avoids. In that case I don't understand what the point of 'a is on VM and CallStack. You can create[1][2] those with any unbounded lifetime (including 'static[3]), which means it is not constraining anything. What is the lifetime 'a doing here? Why not remove it? [1] https://github.com/andreabergia/rjvm/blob/be9c54066c64a8287902553dc1bab4b94206086c/vm/src/call_stack.rs#L45-L47 https://github.com/andreabergia/rjvm/blob/be9c54066c64a82879... [2] https://github.com/andreabergia/rjvm/blob/be9c54066c64a8287902553dc1bab4b94206086c/vm/src/vm.rs#L73 https://github.com/andreabergia/rjvm/blob/be9c54066c64a82879... [3] https://github.com/andreabergia/rjvm/blob/be9c54066c64a8287902553dc1bab4b94206086c/vm/tests/integration/real_code_tests.rs#L10-L17 https://github.com/andreabergia/rjvm/blob/be9c54066c64a82879...
- andreabergia 3y agoI wanted to express the fact that everything that gets allocated (call stack, frames, classes, and objects) is alive and valid until the "root" VM is, thus I used 'a more or less everywhere. I also struggled with a got a ton of errors from the borrow checker initially, and I fixed many of those with a lot of explicit lifetimes, but it's not impossible that in some places they are unnecessary.
- nunobrito 3y agoGood work!
- celeritascelery 3y agoI have a few questions about the garbage collection. One of the hard parts of implementing a garbage collector is making sure everything is properly rooted (especially with a moving collector). you have the `do_garbage_collection` method marked unsafe[1], but don't explain what the calling code needs to do to ensure it is safe to call. How do you ensure all references to the heap are rooted? This is not a trivial problem[2][3][4]. Also note that I cloned the repo and tried to run `cargo test` every test fails with 'should be able to add entries to the classpath: InvalidEntry(".../vm/rt.jar")' vm/tests/integration/real_code_tests.rs:15:10 [1] https://github.com/andreabergia/rjvm/blob/be9c54066c64a8287902553dc1bab4b94206086c/vm/src/gc.rs#L131 https://github.com/andreabergia/rjvm/blob/be9c54066c64a82879... [2] https://manishearth.github.io/blog/2021/04/05/a-tour-of-safe-tracing-gc-designs-in-rust/ https://manishearth.github.io/blog/2021/04/05/a-tour-of-safe... [3] https://without.boats/blog/shifgrethor-iii/ https://without.boats/blog/shifgrethor-iii/ [4] https://coredumped.dev/2022/04/11/implementing-a-safe-garbage-collector-in-rust/ https://coredumped.dev/2022/04/11/implementing-a-safe-garbag...
- munificent 3y agoIt's pretty straightforward. Their VM maintains its own notion of a callstack instead of using the native callstack. That lets them iterate over it and find all of the parameters and locals on the VM's callstack and use them as roots. There is a performance cost for a VM having its own virtual callstacks like this, but it makes GC tracing much simpler. (It also makes implementing interesting concurrency and control flow primitives like coroutines or continuations much easier too.)
- celeritascelery 3y agoSeems like that would take care of roots for the bytecode's themselves, but not for "native" functions[1]. Allocating a new object could call gc[2], and native functions are using the native callstack. It seems like it would be easy to allocate in a native function and any unrooted references would be invalidated. In fact I see a case like that here[3]. That method creates a reference with `expect_concrete_object_at` and then calls gc with `new_java_lang_class_object`. It avoids UB by not using `arg` after the call that gc's, but there is nothing stopping you from using `arg` again (and having an invalid reference). [1] https://github.com/andreabergia/rjvm/blob/main/vm/src/native_methods_impl.rs https://github.com/andreabergia/rjvm/blob/main/vm/src/native... [2] https://github.com/andreabergia/rjvm/blob/be9c54066c64a8287902553dc1bab4b94206086c/vm/src/vm.rs#L230-L236 https://github.com/andreabergia/rjvm/blob/be9c54066c64a82879... [3] https://github.com/andreabergia/rjvm/blob/be9c54066c64a8287902553dc1bab4b94206086c/vm/src/native_methods_impl.rs#L237-L246 https://github.com/andreabergia/rjvm/blob/be9c54066c64a82879...
- Agingcoder 3y agoI'm doing a (free) operating system (just a hobby, won't be big and professional like gnu) for 386 (486) AT clones. :-)
- 1letterunixname 3y agoThere's a no-std tutorial on how to write a demo kernel in Rust. https://os.phil-opp.com https://os.phil-opp.com osdev.org, sandpile.org, RBIL, and freevga. The biggest PITA is hardware support. There are many good vintage hardcopy books with recipes for things like reliable port IO and undocumented hardware tricks. - Intel® 64 and IA-32 Architectures Software Developer’s Manual Combined Volumes: 1, 2A, 2B, 2C, 2D, 3A, 3B, 3C, 3D, and 4 - Microsoft MS-DOS Programmer's Reference (also includes real-mode BIOS calls) - PC Interrupts - Undocumented PC - PC Intern - Programmer's Guide To The EGA, VGA, And Super VGA Cards - Graphics Programming Black Book Special Edition Also, it's worth toying with advances in OS dev past the era of monolithic, microkernel, and hybrid. 1. Capability-based like seL4. It has a number of inherent performance and security advantages including capabilities and excellent IPC. 2. POSIX compatibility layer. Even embedded OSes without the concept of threads or processes can implement POSIX. 3. Hypervisor. They're much easier to add with intel's VT-[xd]. Failing that, fall back to emulation. Translational emulation is very performant. 4. Get good at generalizing interrupt handlers, making them fast, avoiding race conditions, and using lock-free patterns. Also: 5. Rewriting or trapping unsupported instructions including x87 and MMX. 6. The failure of pure microkernel was the added complexity and management of sequencing multiple resources in a transactional manner. There are great theoretical security and operational advantages in microkernel architectures but they never caught on widely in a pure form.
- vacuity 3y ago> 2. POSIX compatibility layer. Even embedded OSes without the concept of threads or processes can implement POSIX. Do you have resources about this? I can't quite fathom how it would work, but then again I have no expertise. > 3. Hypervisor. They're much easier to add with intel's VT-[xd]. Failing that, fall back to emulation. Translational emulation is very performant. Speaking of which, QubesOS was on HN recently. The essence is having many VMs to minimize cross-app attack surface and privilege escalation. > 6. The failure of pure microkernel was the added complexity and management of sequencing multiple resources in a transactional manner. There are great theoretical security and operational advantages in microkernel architectures but they never caught on widely in a pure form. I recently saw an interesting brief article[0] about how memory-safe languages can supercede the compartmentalization that microkernels provide. I'm reminded of Theseus[1], written in rUsT, which happens to reflect the sentiment. I've actually been putting off rereading the Theseus USENIX paper. Again, I'm not nearly qualified to answer whether the security of this is comparable to the best of that of microkernels, barring formal verification. Still, I think it should be explored more. [0] https://catern.com/microkernels.html (Write modules, not microkernels) [1] https://www.usenix.org/conference/osdi20/presentation/boos
- appleaday1 3y agoOracle oracle oracle.
- nine_k 3y agoThis project is like the ground floor of the JVM, not the whole tower. I like though how the project's page is direct and clear about that. A lot of the foundation and ground-floor mechanics are pretty interesting though.
- ChuckMcM 3y agoThat is pretty awesome! When I joined the Java effort in '92 (called Oak at the time) the group I was with was looking at writing a full OS in Java. The idea being that you could get to just the minimal set of things needed as "machine code" (aka native methods) you could reduce the attack surface of an embedded OS. (originally Java was targeted to run in things like TV's and other appliances). We were, of course, working in C rather than Rust for the native methods. The JVM in Rust though adds a solid level of memory safety to the entire process.
- mshockwave 3y ago> writing a full OS in Java IMAO, Android kind of achieve that...kind of. They write lots of OS logics in Java (or Kotlin) but mixing lots of system services written in native code at the same time, interconnected by the famous (or infamous?) Bind IPC.
- tgtweak 3y agoEmbedded JVM is actually huge/pervasive and runs things as benign as the chip on your credit card.
- locusofself 3y agoAfter reading your comment I was surprised to find out that credit card chips have any processing capabilities whatsoever, which they apparently do, though at least according to gpt4, they are far too basic to run java/jvm. ?
- tjlingham 3y agoIt feels like that should be true, I get it. However Java Card is very real. https://en.m.wikipedia.org/wiki/Java_Card https://en.m.wikipedia.org/wiki/Java_Card
- _7tgr 3y agoGoogle appears to be significantly more useful than GPT-4 here. [1] is the third result for me for the query "credit card jvm". [2] is the second result and gives a direct (and more importantly, actually correct) answer. That post links to the Oracle documentation for Java Cards [3] which is the fourth result. [1] https://en.m.wikipedia.org/wiki/Java_Card https://en.m.wikipedia.org/wiki/Java_Card [2] https://superuser.com/questions/362567/are-there-any-credit-cards-that-run-java https://superuser.com/questions/362567/are-there-any-credit-... [3] https://www.oracle.com/java/java-card/ https://www.oracle.com/java/java-card/ All of this is just as easy as, if not easier than, using ChatGPT. It's unclear that such a tool even serves this purpose (retrieval of basic facts) adequately, so it should probably be avoided in the future.
- bingemaker 3y agoWhen I see such cool projects, I feel very overwhelmed. How do you get started with Rust and master basics to even attempt doing such a thing? Can OP explain?
- nop_slide 3y agoLikewise. Not to go onto too much of tangent, but on a more personal note I've been generally struggling with this feeling a lot lately. I've been a professional software developer for almost 10 years, and I _know_ I'm competent (and not an impostor) as demonstrated by my current position and ability to ship things. However, lately after viewing developer blogs I become overwhelmed that I actually don't know enough and am not a "real" developer. I seem to have formed a notion of an ideal developer in my head and I compare myself against this imagined construct which leads to these feelings. I admire how these people have so much deep knowledge and can express themselves so clearly and concisely, then wonder why I am not like that. I barely have the energy after work after taking care of my family to do anything further, and I know programming isn't everything but I do have a desire to learn more and improve myself. I recognize this isn't healthy nor is it rational, but it's just a feeling I can't shake lately.
- bingemaker 3y agoThanks for sharing
- theLiminator 3y agoWell, you're probably comparing yourself against the top 1% of developers. It's okay to not be the very best, being in the top 30% of this field already is very rewarding.
- dist1ll 3y agoWhat you're describing is very common amongst developers. So common in fact, that I've written a post about this https://alic.dev/blog/comparisons https://alic.dev/blog/comparisons In short: recognizing your insecurities is the first step. The next step is figuring out what's important to you, shedding impossible to achieve and irrational ambitions, prioritizing your goals in life, and articulating concrete steps to further them.
- aardvark179 3y agoVery well done. Building VMs is always fun, and I’m sure it was an interesting learning experience when combined with Rust’s type system. If you’re looking for a job then ping me on Twitter, Mastodon or my work email, I’m sure you can figure them out from my user id here.
- squirtlebonflow 3y ago[dead]
- fmiras 3y agoYou're awesome, thanks for sharing!
- WhyNotHugo 3y ago> built for learning purposes and not a serious implementation This is a typical way to present a project that ends up replacing the established/existing implementation.