4 ms·

Open question: in what fields are you working where you really need to work with actual sockets? Most of the network programming I do fall into one of two categ

by kqr 3y ago

Open question: in what fields are you working where you really need to work with actual sockets? Most of the network programming I do fall into one of two categories: 1. A library exists to abstract over the bytes-into-sockets layer, meaning I don't need to deal with it; or 2. A library probably exists to abstract over the bytes-into-sockets layer, but I've chosen to ignore it either because I'm truly ignorant of it, or for intellectual stimulation. (Note that in category 2 I count things like "not using protobuf/thrift/etc. for IPC.) When are these things not true? Genuinely curious! I'm guessing interfacing with existing obscure third parties, or using obscure language environments, but what are examples of that? And what other usages are there?

forrestthewoods 3y ago
There is a whole universe of programming that exists outside of webdev. Firmware, video games, high performance systems, etc.
- JustSomeNobody 3y ago
  This. I think HN leans very heavy on webdev and somehow forgets that the entire rest of the planet runs on software also.
DoingIsLearning 3y ago
Had plenty of use cases where even zmq was overkill. E.g. interfacing a research machine with an OEM machine or a machine to a standalone embedded target, sometimes it makes sense to just serialize, send over a socket, ethernet crossover cable and be done.
- kqr 3y ago
  Your description makes it sound sort of like temporary code to perform a one-time (or few-times) job, and not something that's running and maintained for years. Is that a fair characterisation or am I misunderstanding?
GoblinSlayer 3y ago
At least on Unix and Windows file read and write methods work for sockets too, so streaming bytes isn't different. How would you implement, say, netcat? Launch a jango rest service with kubernetes, which will run in a wm if it's a wrong OS, create a certificate with letsencrypt and access it with a react electron app?
- kqr 3y ago
  Why would I implement netcat when netcat is already implemented? That's sort of why I'm asking – at least in the circles I move, all the low hanging fruit are already picked, and picking the high fruit seems rare, so I'm interested in which people do and what those fruits are!
fredrb 3y ago
I don't work with it either. But I find that learning the abstraction below where you're working at can be quite beneficial to understanding the constraints of your layer, debug, and solve problems. E.g.: Learn the basics Transport Layer protocols (TCP/UDP) if you work with HTTP
dan-robertson 3y ago
1. Writing the libraries you describe 2. Speaking unusual protocols for which (good) libraries don’t exist. 3. I think interacting with the sockets api can also be relevant for tweaking various options to get better performance (though often there are Linux defaults that can be tweaked instead)
- kqr 3y ago
  I'm fully aware of the hypotheticals – what I'm curious about is which your personal experiences are with e.g. those three points you list.
  - dan-robertson 3y ago
    I’ve done/do all three of those things. Knowing about the networking stack (protocols, the various things inside Linux, etc) has been pretty relevant.
synack 3y ago
It's useful to understand the socket API so that you can do things like read the output from strace, notice that accept() returns EAGAIN too often, and know what that indicates.
kragen 3y ago
at knownow i wrote an event-driven nonblocking http server in python with asyncore which could efficiently do comet over hundreds of concurrent http connections; existing http libraries were all blocking at the time i also wrote a similar event-driven nonblocking http client in c for benchmarking and load-testing both that server and a different compatible one being written by another team in c++. this was nonblocking because it was important for the client to be faster than the server. at the time there was no libevent/libev/libuv so i wrote my own at satellogic i maintained the upper layers of our cubesat space protocol stack, which provided a socket-like interface to our coap stack, which i also maintained part of i also had to tweak kernel socket buffering settings for rabbitmq there for wercam i'm experimenting with different approaches to ipc for efficiency; the data being transferred are pixel buffers so i don't think protobuf/thrift/etc. is going to help for the mail server i share with some friends i wrote a milter in python to smtp-reject outside email to certain recipients unless it matched a whitelist; milter doesn't speak pb either and i couldn't find a python library i wrote a multicast file transfer program in python (bccpo/bccpi) for copying files across the lan without having to fiddle around with ip addresses; the data being transferred is just (multicast) the ip and port and (unicast) the filename, file size, and file contents, so again i'm not sure pb would simplify anything. the hard part of getting it working was just multicast i wrote a chat client for icb in python with asyncore, and again i couldn't find a suitable library even when i'm using a library, it's often pretty useful to be able to read strace output or use select or poll with the file descriptor it's trying to hide from me i don't think any of these qualify as either your category 1 or 2 i've also done a huge number of things that fall into category 2, of course, like writing an irc client in bash, writing a web server in assembly, and writing a chat server in three 80-column lines of c, writing a mud, and so on
samtho 3y ago
I often work in a resource constrained environment where my program size is measured in two-digit kilobytes, RAM is measured in the single-digit kilobytes, and it can’t exceed some current measured in mA/h. When I am simply sending a UDP packet, I didn’t need nor can I afford the footprint that these libraries have.
JustSomeNobody 3y ago
I implement protocols to communicate with various hardware (RFID, LiDAR, etc). Most of these communicate over sockets. Some still use serial. I have a library I wrote of socket routines that I use, so it's not like I re-implement that every time, though. And for this, I'd really rather use my library than a 3rd party in case I have to dig in and see what's going on.

by kqr 3y ago

forrestthewoods 3y ago

There is a whole universe of programming that exists outside of webdev. Firmware, video games, high performance systems, etc.

JustSomeNobody 3y ago
This. I think HN leans very heavy on webdev and somehow forgets that the entire rest of the planet runs on software also.

DoingIsLearning 3y ago

Had plenty of use cases where even zmq was overkill. E.g. interfacing a research machine with an OEM machine or a machine to a standalone embedded target, sometimes it makes sense to just serialize, send over a socket, ethernet crossover cable and be done.

kqr 3y ago
Your description makes it sound sort of like temporary code to perform a one-time (or few-times) job, and not something that's running and maintained for years. Is that a fair characterisation or am I misunderstanding?

GoblinSlayer 3y ago

At least on Unix and Windows file read and write methods work for sockets too, so streaming bytes isn't different. How would you implement, say, netcat? Launch a jango rest service with kubernetes, which will run in a wm if it's a wrong OS, create a certificate with letsencrypt and access it with a react electron app?

kqr 3y ago
Why would I implement netcat when netcat is already implemented? That's sort of why I'm asking – at least in the circles I move, all the low hanging fruit are already picked, and picking the high fruit seems rare, so I'm interested in which people do and what those fruits are!

fredrb 3y ago

I don't work with it either. But I find that learning the abstraction below where you're working at can be quite beneficial to understanding the constraints of your layer, debug, and solve problems. E.g.: Learn the basics Transport Layer protocols (TCP/UDP) if you work with HTTP

dan-robertson 3y ago

1. Writing the libraries you describe 2. Speaking unusual protocols for which (good) libraries don’t exist. 3. I think interacting with the sockets api can also be relevant for tweaking various options to get better performance (though often there are Linux defaults that can be tweaked instead)

kqr 3y ago
I'm fully aware of the hypotheticals – what I'm curious about is which your personal experiences are with e.g. those three points you list.
- dan-robertson 3y ago
  I’ve done/do all three of those things. Knowing about the networking stack (protocols, the various things inside Linux, etc) has been pretty relevant.

synack 3y ago

It's useful to understand the socket API so that you can do things like read the output from strace, notice that accept() returns EAGAIN too often, and know what that indicates.

kragen 3y ago

at knownow i wrote an event-driven nonblocking http server in python with asyncore which could efficiently do comet over hundreds of concurrent http connections; existing http libraries were all blocking at the time i also wrote a similar event-driven nonblocking http client in c for benchmarking and load-testing both that server and a different compatible one being written by another team in c++. this was nonblocking because it was important for the client to be faster than the server. at the time there was no libevent/libev/libuv so i wrote my own at satellogic i maintained the upper layers of our cubesat space protocol stack, which provided a socket-like interface to our coap stack, which i also maintained part of i also had to tweak kernel socket buffering settings for rabbitmq there for wercam i'm experimenting with different approaches to ipc for efficiency; the data being transferred are pixel buffers so i don't think protobuf/thrift/etc. is going to help for the mail server i share with some friends i wrote a milter in python to smtp-reject outside email to certain recipients unless it matched a whitelist; milter doesn't speak pb either and i couldn't find a python library i wrote a multicast file transfer program in python (bccpo/bccpi) for copying files across the lan without having to fiddle around with ip addresses; the data being transferred is just (multicast) the ip and port and (unicast) the filename, file size, and file contents, so again i'm not sure pb would simplify anything. the hard part of getting it working was just multicast i wrote a chat client for icb in python with asyncore, and again i couldn't find a suitable library even when i'm using a library, it's often pretty useful to be able to read strace output or use select or poll with the file descriptor it's trying to hide from me i don't think any of these qualify as either your category 1 or 2 i've also done a huge number of things that fall into category 2, of course, like writing an irc client in bash, writing a web server in assembly, and writing a chat server in three 80-column lines of c, writing a mud, and so on

samtho 3y ago

I often work in a resource constrained environment where my program size is measured in two-digit kilobytes, RAM is measured in the single-digit kilobytes, and it can’t exceed some current measured in mA/h. When I am simply sending a UDP packet, I didn’t need nor can I afford the footprint that these libraries have.

JustSomeNobody 3y ago

I implement protocols to communicate with various hardware (RFID, LiDAR, etc). Most of these communicate over sockets. Some still use serial. I have a library I wrote of socket routines that I use, so it's not like I re-implement that every time, though. And for this, I'd really rather use my library than a 3rd party in case I have to dig in and see what's going on.