11 ms·
XMODEM in 2022
- ape4 4y agoI remember zmodem was slightly better https://en.wikipedia.org/wiki/ZMODEM https://en.wikipedia.org/wiki/ZMODEM
- blibble 4y agozmodem is still very useful as you can use it with ssh use zssh instead of ssh, then sz on the remote end to send files (and rz to receive)
- outworlder 4y agoIs that for some kind of use-case where you can't use SCP?
- EvanAnderson 4y agoAbsolutely. I used a ton of "sz" when I had a dial-up shell account (and didn't yet know about SLiRP). More recently I've used sz/rz when I've been connected to a router w/ a serial connection and needed to pull down a file from a remote host. The router had an SSH client so I just connected to a Linux box, downloaded my file, then sz'd it to myself over the SSH connection via serial.
- blibble 4y agoworks through as many levels of ssh as you want, and much more convenient that having to open another terminal, figure out the cwd and run the command years ago a load of GUI clients used to have a right click menu to send/receive files that used zmodem (sadly putty wasn't one of them)
- fullstop 4y agoAbsolutely. On some boards where ethernet is not an option you can still run a serial getty and transfer files to / from the device.
- kazinator 4y agoI wrote a zmodem-like protocol many years ago called "sixpack" that worked through anything; I used it through two nested telnet connections going through two terminal servers (serial lines). Sixpack's wire format packs data into six bits, similarly to base64. This was in 1996. In a fit of amazing fortuitousness, a fellow in Japan, almost exactly at the same time, developed a program called Modemu. It is still out there and has not been maintained since. What Modemu does is creaete a pseudo-tty device in your system that you can use in a program like Minicom. This pseudo-TTY device accepts AT commands to "dial" remote hosts and connect to them. So at the time I was able to install my Sixpack commands (sps, spr) into Minicom, and then telnet out using Modemu: ATD<host-namne>, then transfer files to the remote hosts. I would install the receive program by uuencoding it, and then just doing piece by piece copy and paste into the remote session to recover the binary.
- throw0101a 4y ago> zssh TIL: * https://packages.ubuntu.com/search?keywords=zssh https://packages.ubuntu.com/search?keywords=zssh
- tssva 4y agoI used it just last week. Thanks to a bad choice regarding kernel modules on Canonical's part I had an update to Ubuntu 22.04 on a Raspberry Pi leave the RPi inaccessible via Ethernet. Luckily I have the serial console on the RPi connected to another co-located device for exactly this potential situation. I was able to ssh to the other device, launch a serial terminal from there to the RPi and transfer the deb file needed to fix the problem directly to the RPi from my laptop via modem.
- hinkley 4y agozmodem dealt with line noise a lot better, but it seems it was also faster: XMODEM used 128-bytes payloads with a three-byte header and one-byte checksum for a total of 132 bytes per packet. In the era of 300 bps modems, a packet took about four seconds to send, and typical latencies were on the order of 1⁄10 of a second, so the performance overhead was not significant. As speeds increase the problem becomes more problematic; at 2400 bps a packet takes about 1⁄2 to send, so about 1⁄5 of the available bandwidth is wasted waiting for ACKs. At 9600 bps a packet requires only 0.13 seconds to send, so about 1⁄2 of the bandwidth is wasted. ZMODEM addressed these problems by removing the need for ACKs at all, allowing the sender to send data continually as long as the receiver detected no errors. Only NAKs had to be sent, if and only if there was a problem. Since ZMODEM was often used on links with built-in error correction, like X.25, the receiver would often not send a single message back to the sender. As a result, the system would send the entire file in a continual stream, and ZMODEM referred to itself as a "streaming protocol".
- icedchai 4y agoI was a big BBS user (and sysop) from roughly 1988 to 1997 or so. zmodem was much, much faster.
- hinkley 4y agoI was just a user so my memory of why I switched to zmodem was a bit suspect. Now that you mention it, there were definitely largish binaries that I could only download with zmodem. At some point I found a BBS with big files I wanted that only supported ymodem, and after several days of trying and hitting the hour timeout, I gave up.
- flyinghamster 4y agoThe other thing that ZMODEM brought to the table, provided your terminal emulator supported it, was automatic downloads. You didn't have to tell the BBS to send FOO.GIF and then turn around and tell your computer to receive FOO.GIF, you just downloaded it with ZMODEM and the download started immediately.
- EvanAnderson 4y agoXmodem is the right thing for an embedded target for bringup, but zmodem was tons better for BBS'ing. It performed much better than xmodem. Auto-start in zmodem is wonderful. Multi-file transfers are easy and it even sends the filenames. (My old libraries of BBS downloads, pre-zmodem, are a mess of crappy filenames.)
- KerrAvon 4y agoThere has got to be something better than Xmodem for use over a reliable connection. It's not going to be particularly fast.
- EvanAnderson 4y agoI guess it depends how "embedded" the target is, I guess. I'd rather write an xmodem implementation in assembler than I would zmodem. I'm envisioning that you're using xmodem to transfer "brain-stem" type firmware where you're talking to a very constrained system. (I know I've sent firmware to Ethernet switch bootloaders via xmodem over serial. Presumably I was very, very close to the metal in that case.)
- boondaburrah 4y agoExactly. I built my first 8-bit computer after watching Ben Eater's 6502 series and the first thing I did after "Hello! World" was look up what bare minimum serial transfer I could implement that would yell if I fucked up the code in some obvious way (cant check if the data is right on the other end if the RAM is volatile and the transfer code itself is unproven). XModem fit the bill (entire spec on Wikipedia and dumb enough that an assembler beginner can do it).
- amluto 4y agocat works great over a reliable connection as long as you can get it to start and end in the right places. When the connection is reliable, the only thing that needs doing is delimiting the transfer and possibly sending metadata.
- KerrAvon 4y agoZModem was much better once it was available. You'd generally only use Xmodem after that point if you had no other choice, because it failed frequently over unreliable connections, and all phone line connections were essentially unreliable due to line noise (and sometimes other in-band interruptions). IIRC, ZModem was both faster and resumable, so that if your connection terminated in the middle of your two megabyte transfer you didn't have to redownload the first megabyte.
- deleted 4y ago[deleted]
- bluedino 4y agoI remember all kinds of other protocols coming and going. Ymodem-g! Blazing fast. I never had issues with it but friends with worse modems/phone lines were always restarting their transfers.
- ericbarrett 4y agoIf I recall, Ymodem-G (as opposed to Ymodem) was basically an unchecksummed raw file dump with a little metadata header, so it was only usable on a connection without line noise. You could actually get undetected file corruption.
- Mountain_Skies 4y agoSome SysOps would collect and support as many protocols as they could as sort of a form of bragging rights. Even with dozens, almost everyone used zmodem, or once in a while, BiModem if they wanted to chat during a transfer.
- a-dub 4y agoyou can't just go all the way from xmodem to zmodem without stopping at ymodem, ymodem 1k and ymodem 1k with batch support first. also, kermit. who remembers rip graphics?
- thedougd 4y agoI believe I do! Where it'd paint the progress gif or line by line with other formats as it downloaded? You could terminate early if it wasn't what you wanted.
- a-dub 4y agoand full blown point and click guis! of course it was made irrelevant by the fledgling world wide web.
- mmastrac 4y agoRIP script bombs on BBSes were evil. You could black out someone's screen until they exited and dialed back up.
- a-dub 4y agocome to think of it, i don't think there were any private multiline boards in my local toll-free calling area. it was only packet switched commercial services and shell based internet at the libraries/university.
- mmastrac 4y agoThe only multi-line one I remember vividly was running MajorBBS. The inter-user chat was especially vulnerable to sending RIP commands as it didn't filter them out.
- CamperBob2 4y ago"My user number? Why, 99E99, of course."
- floatinglotus 4y agoMaybe BBSs will make a comeback?
- MontagFTB 4y agoPlease share the address for HNMUD when it launches, thanks.
- nope96 4y agoThey might be! I see new ones listed from time to time here: https://old.reddit.com/r/bbs/ https://old.reddit.com/r/bbs/ Weird factoid for the day: Country music star Shooter Jennings runs one! https://old.reddit.com/user/ShooterJennings/submitted/ https://old.reddit.com/user/ShooterJennings/submitted/
- evanelias 4y agoHe also wrote a bbs door game in 2015! http://breakintochat.com/blog/2015/09/18/shooter-jennings-creator-of-from-here-to-eternity/ http://breakintochat.com/blog/2015/09/18/shooter-jennings-cr... Maybe some more recent ones too, I haven't kept up at all.
- jeffbee 4y agoSurprised at the statement that very few new computers have serial ports. I guess true for macs, but most PC motherboards still come with a superIO-type thing that offers 1 or 2 16650A UARTs. [ 0.362434] 00:01: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
- zauguin 4y agoLooking at my last two (non notebook) motherboards (neither of them being particularly new) one of them has a pin header where a serial port could be attached, the other only has some contacts on the board without even pins. Neither of them had a proper port. So even if they could offer these ports and Linux reports them, they might not be usable without special work.
- jabl 4y agoServer platforms tend to have the serial wired to the BMC and going out over the IPMI serial-over-lan on ethernet. But from the host OS it looks like a regular 16550A.
- jeffbee 4y agoUgh, not even populating the header is a terrible way to save one penny. This one I'm using came with the rear panel bracket and everything.
- deckard1 4y agothey are talking about a D-Sub 9 connector, which I haven't seen on new PCs since the early 2000s. If your motherboard has pins they are probably undocumented. And that's likely the best case scenario.
- StillBored 4y agoNah, on actual x86 PC atx/etc motherboards the ports are frequently fully RS-232 compliant, and on all my desktop boards for the past few years are even still documented. Example: I have native, documented DB9/25 on my latest machine, an asus prime x570-pro https://www.asus.com/us/Motherboards-Components/Motherboards/PRIME/PRIME-X570-PRO/ https://www.asus.com/us/Motherboards-Components/Motherboards... which if you download the manual and search for "COM" you will see that it uses the semi standardized 10 pin motherboard header into which a card edge DB9/DB25 can be plugged into. They are overwhelmingly used by hw/firmware/bringup engineers, so, while your right they are frequently undocumented or unpopulated, like JTAG ports its more unusual not to find them. And, yah on a lot of machines they are just whatever logic level the chip supports for IO 1.8V/etc so its common to just wire the headers to FTDI/etc usb converters rather than level shifters.
- AaronFriel 4y agoLet me know if you need the source for QMODEM, I know a guy :)
- thedougd 4y agoYour father? If related and around, please tell them thank you for many years of exploration and learning.
- AaronFriel 4y agoI'll let him know!
- rwl4 4y agoI'll bite! There's a reimplementation called Qodem, but it would be great to see the actual QModem source since AFAIK it has never been released.
- Thoreandan 4y agoAt least a dozen friends I know swore by QModem :^) I was always a Telix user but found QModem a solid term program when I used it! Cheers!
- kazinator 4y agoTelix was nice; I did some scripting in the Salt language. In my first job out of high school, I worked as an IT person in law firm that used Procomm Plus (for connecting to some databases over dial-up, like the land title registry). I used its scripting language to automate logins. I remember Procomm Plus being decent.
- bwanshoom 4y agoWhat is your dad up to these days? Did he go to Microsoft like the rest of Mustang?
- AaronFriel 4y ago
- michaelcampbell 4y agoI once entered a chat on CompuServe with Ward Christensen, the author of Xmodem. Was ~1984/1985. Had a nice chat (at $6/hour) with him, and he was pretty surprised I (nay, anyone) recognized his name and what he had done. Nice guy as I recall.
- ceautery 4y agoThat's awesome! Ironically, Xmodem wouldn't have worked well over CIS's network, since to talk to the X25 nodes you had to have software flow control enabled in your modem connection. Small transfers might work, but when you hit the Xmodem block number that coincides with ^Q, that block would repeatedly NAK until the transfer failed.
- beaugunderson 4y agoI've been reading about Ward in the book that just came out about the BBS era called "The Modem World"; only three chapters in but it's a good read so far.
- PaulHoule 4y agoXMODEM is a terrible protocol. I was needing a reliable protocol to move binary data out of an Arduino into my PC and researching XMODEM I found so many things wrong with it, particularly no sane way to specify the file length and a checksum scheme that is totally inadequate. The buffer size also is a bit big for a machine with tiny RAM. I wound up rolling my own version of https://en.wikipedia.org/wiki/High-Level_Data_Link_Control https://en.wikipedia.org/wiki/High-Level_Data_Link_Control with much smaller packets and a carefully chosen CRC which is adequate for the task. Retrospectively I was shocked that we accepted XMODEM back in the day.
- wvenable 4y agoFor file transfer for my 8bit computer, I also couldn't bring myself to implement XMODEM. I ended up making a protocol similar to XMODEM because I used the standard ASCII characters for framing and escaping but my protocol doesn't have XMODEM's weirdness.
- IncRnd 4y agoPeople well knew of the shortcomings, so they used XMODEM-1K (called YMODEM and had a larger block size) or ZMODEM (a sliding window and retransmission recovery). ZMODEM was definitely the one to use! Once you've used the latter letters you never fallback. You'd never leave z to use y or leave y to use x.
- deepspace 4y agoYes, I am not sure why someone would roll their own protocol when ZMODEM was available and solved most of XMODEM's problems. I was forced to use XMODEM back in the day (because ZMODEM did not exist yet), but as soon as the ZMODEM spec was published in 1988, everyone moved to it and XMODEM became obsolete almost instantly.
- deleted 4y ago[deleted]
- hinkley 4y ago
- glonq 4y agoFor heinous reasons that I won't get into, my team recently finished implementing YModem-over-BLE for our embedded project.
- pantalaimon 4y agoWhy not CoAP?
- qalmakka 4y agoWhy would anyone think someone would implement YMODEM over BLE if they could use anything else? It's one of the crappiest things ever.
- thatfunkymunki 4y agosame team as https://news.ycombinator.com/item?id=31572391 https://news.ycombinator.com/item?id=31572391 ? or is this more common than i imagine
- throw0101a 4y agoOnce HS/Link came out, I never looked back to x/y/zmodem if the BBS had it on their end: * https://en.wikipedia.org/wiki/HS/Link https://en.wikipedia.org/wiki/HS/Link Bidirectional transfers were so handy and a time saver if you were transferring mail files (QWK, SOUP, BlueWave) and your replies. If anyone has some of those old files available, readers are still available: * https://wmcbrine.com/MultiMail/ https://wmcbrine.com/MultiMail/
- gfodor 4y agoHS/Link was also the only protocol I remember that also included a chat between sender and receiver during the download. For the first time you could actually continue your conversation while sending files to one another, instead of having to go dark like you were flying around the moon or something.
- mmastrac 4y agoMost of the time it's easier to unload the driver just to write bytes to the USB endpoints. These devices are basically the easiest possible thing you can talk to over USB.
- zwieback 4y agoMentioned in the article: Saleae logic analyzer. I love that thing, especially the newer mixed-signal version.
- abotsis 4y agoI did this inside idos on an iPad before Apple approved (then later revoked) the ability to drop binaries in the emulator. iDos (dosbox) shipped with a compiler and ability to map a serial port to a tcp socket. So I implemented an XMODEM receiver so I could load my retro warez inside the walled garden :) I spent more time writing the XMODEM receiver than I spent using software I sent over it. It’s never the destination, always the journey.
- qalmakka 4y agoYeah, XMODEM... I had to implement YMODEM for a client of the company I work with a few months ago... Over BLE. It is every bit as bad as it sounds like.
- thatfunkymunki 4y agohttps://news.ycombinator.com/item?id=31571831 https://news.ycombinator.com/item?id=31571831 same team? story time?
- qalmakka 4y agoIt is more common that you imagine. In my case, the story is that some people have literally been porting the same C code base on multiple computers first, and embedded devices later, since the late '80s. Given their lead engineer is literally 80 years old now, the client once again slap YMODEM on their latest device, which uses BLE to literally emulate a good old serial port and pass data on it using YMODEM. It sucks, hard. They were quite puzzled about why it ran so badly, and I didn't have the heart to tell them it was because the whole thing was incredibly idiotic.
- glonq 4y agoDifferent team.
- daneel_w 4y agoI often used YMODEM-G, which offered nice transfer speed improvements. It absolutely needed a clear line since it employed no error correction. I rarely ever had issues with the protocol when dialing local BBSes, but it was a gamble on long-distance calls.
- WalterGR 4y ago> It absolutely needed a clear line since it employed no error correction. YMODEM-G was fast because it didn't do any error detection of its own, so it didn't have the overhead of checksums and such. The usage scenario was that the connection would already have error detection and recovery by virtue of the modem and the connection... protocol? I forget what we called them.
- gruturo 4y ago>error detection and recovery by virtue of the modem and the connection... protocol? I forget what we called them. Yeah protocols. First ones to see widespread use (at least subjectively) were MNP4 and MNP5 for, respectively, Error correction (with a 25% speed increase as a bonus, since it was synchronous and would not therefore waste code space with start and stop bits) and Data Compression (a terrible one which would actually incur an overhead if data was already compressed). They got later replaced by v.42 and v.42bis, with the same respective purposes (but v.42bis was smarter than MNP5 and could switch to transparent mode when sending incompressible data). CARRIER 14400 COMPRESSION: V.42 bis PROTOCOL: LAPM CONNECT 57600/ARQ
- deleted 4y ago[deleted]
- the_only_law 4y agoBelieve I used it last year to transfer files between a Agilent J2300E running Win95 or Win98 and a modern PC. Slow, but convenient given it was already implemented in the terminal emulators used on both end.
- stdcall83 4y agoI implemeted xmodem over uart as a recovery boot for several SOCs in my career. It's a simple protocol you can actually fit into a small boot ROM and test using RTL. We use it only to load the rest of the bootloader which is few hundreds of kb. Then we switch to networking using rj45 for the rest of the bootstrapping. When using xmodem 1k with higher bps you can get to a decent speed.
- lol_catz 4y agoI prefer MobyTurbo for my file transfers
- nickdothutton 4y agoI’d probably reach for ZMODEM 1st, if only because that’s where I ended up in the early 90s running a BBS. I recently had to use Kermit to get some data through a very, very narrow (we are talking a few ‘000 bytes per second) 7-bit only link. The “kids” were in awe.
- jstapels 4y agoFun blog, but I was left hanging as the author never found the issue with the OS level driver (and instead <spoiler>used a workaround library</spoiler>). It's easy to just blame the FTDI driver, but FTDI is used all over the place in the arduino community on MacOS, so I would have have assumed it was working.
- saulrh 4y agoI'd trust the FTDI windows drivers, because almost all industrial use of embedded is on windows, factories and electrical engineering and the like. I'd mostly trust the Linux drivers, because the open-source community uses them. I wouldn't trust the Mac drivers at all.
- cpeterso 4y agoBack in the BBS days, I used to "collect" BBS file transfer protocols. I shared the source code for some of them on GitHub: * JMODEM: https://github.com/cpeterso/jmodem https://github.com/cpeterso/jmodem * SEAlink (System Enhancement Associates): https://github.com/cpeterso/sealink https://github.com/cpeterso/sealink * WXMODEM (Windows XMODEM): https://github.com/cpeterso/wxmodem https://github.com/cpeterso/wxmodem * ZMax: https://github.com/cpeterso/zmax https://github.com/cpeterso/zmax I also remember "Leech Modem", a file transfer protocol that was compatible with XMODEM and YMODEM but designed to bypass a BBS's download quotas. The protocol would NAK the last packet and then abort the file transfer. The user had successfully downloaded the file, but the BBS would mistakenly not count the aborted file transfer against the user's download quota. https://en.wikipedia.org/wiki/LeechModem https://en.wikipedia.org/wiki/LeechModem
- reaperducer 4y agoDon't forget Punter: https://en.wikipedia.org/wiki/Punter_(protocol) https://en.wikipedia.org/wiki/Punter_(protocol) I still think in terms of "SYN/ACK SYN/ACK" pairs when analyzing technology.
- _JamesA_ 4y agoGOO As a teenager I wrote a few implementations of X/Ymodem and Punter for the C64 in machine code back in the day. I was too poor at that age to afford an assembler. I also wrote a native X/Ymodem utility for the IBM AS/400. I had to write the CRC calculations in MI (AS/400 machine interface language) in order to get enough performance to make it work well.
- chipotle_coyote 4y agoWXMODEM is actually "windowed XMODEM", named because it uses a "window" for packet transmission, letting the sender send multiple packets without an acknowledgement. IIRC, this is a technique similar to ZMODEM's, but it's simpler to implement. The very few times I used it back in my BBS days -- yes, I am that old -- it was (by dialup modem standards) very, very fast.
- mrlonglong 4y agoI see your XMODEM and I raise you KERMIT.
- theturtle32 4y agoHaving been a heavy Linux user continuously since about 1999, including a few years as a Linux (and OpenBSD) sysadmin for an ISP back in the day, the biggest unanswered question in my mind at the moment is: why on earth are we using XModem over a serial link to send the initrd to boot a Linux server in 2022?
- rrauenza 4y agoSecureCRT support zmodem. sometimes it nice to be able to just type 'sz foo.log' from a linux ssh session and get the file over conveniently, even if not so quickly...