7 ms·
When the packages and configuration files for the project are more complicated than the python code it would take to do the one simple thing that you need. A l
by CoolGuySteve 7y ago
When the packages and configuration files for the project are more complicated than the python code it would take to do the one simple thing that you need. A lot of frameworks tend to bloat to handle every use case and become overly complicated because of it.
Things like passing protobufs/gRPC when C structs, JSON, or CSV files will do, job scheduling systems that could be replaced by a postgres table, most Kafka applications.
I once worked with a guy who would pattern match whatever problem you had to the trendy Apache or Google library and then spend weeks/months setting it all up. He was easily the least productive person I've ever worked with.
- javajosh 7y ago>...pattern match whatever problem... I agree with your comment. I would add that there are cases where your problem looks really easy right now, but will get increasingly complicated over time in ways you don't expect. In this case, "overkill" 3rd party solutions are actually better. For example, you could write an HTTP client using sockets pretty easily, and it will work in simple cases. But you will, over time, run into edge cases and problems that a mature library has already solved for you. Also, this problem reminds me of having to decide whether to solve the integral, or look it up, back in the day. Solving from scratch was always preferable if you could do it, but sometimes it was just too hard.
- kd5bjo 7y ago> But you will, over time, run into edge cases and problems that a mature library has already solved for It’s entirely possible that the new problems you encounter will be different than the ones the mature library was designed to address, and then you end up hacking around solutions to problems you don’t have in order to fix the ones you do.
- inlined 7y agoI actually did have to write an HTTP library on sockets once. It wasn’t trivial (took a month to productionize), but it was core to our business. We were writing developer tools (Parse) and our file upload library had major problems on the latest iPhone due to an OS bug (if a kernel buffer filled before the modem could upload it, the file transfer would fail). We became the third company I know about to ever solve this problem, but I had built what my colleague called “the hardest to use HTTP library ever”. I was happy to move back to AFNetworking once we could do so without a regression in performance or hitting that bug.
- raverbashing 7y agoOh yeah NIH is a problem, but "we can't invent it here" is also a problem Person tries to find a library that does what one line of code would do (cough left pad cough and others) and then guess what, the library fails at that thing that maybe not 90% of people need but 50% need And that even applies to some corners of famous apps or libraries big sigh
- austincheney 7y agoThis is too common of a problem because many developers are functionally illiterate in their primary programming language. Invented here syndrome: https://dev.to/mortoray/invented-here-syndrome-4mg8 https://dev.to/mortoray/invented-here-syndrome-4mg8
- envolt 7y agoCan you shed a bit more light on this - > job scheduling systems that could be replaced by a postgres table I've never used Postgres in my life ever.
- reledi 7y agoWhat Jon Richards said. Also see https://github.com/collectiveidea/delayed_job https://github.com/collectiveidea/delayed_job which is a popular DB-based scheduling library for Ruby.
- jon_richards 7y agoI think the point is to use a DB as a message broker. Schedule a job by adding a row to a table. Have services running that secure a lock on a row, run the task, and then update/remove the job from the DB. Django-Q (the multiprocessing task queue framework, annoyingly named the same as Django Q objects), allows this as one of the configurable brokers. https://django-q.readthedocs.io/en/latest/brokers.html#django-orm https://django-q.readthedocs.io/en/latest/brokers.html#djang...
- liampronan 7y ago+1. And I think specifically the SKIP LOCKED Postgres feature is relevant for this -https://tnishimura.github.io/articles/queues-in-postgresql/ https://tnishimura.github.io/articles/queues-in-postgresql/
- jon_richards 7y agoOh, I hadn't noticed that. I'm not sure how Django Q manages with a non-postgres DB, since it doesn't mention that as a requirement. I have used nowait before to batch scheduled jobs together for a slow API call.
- orf 7y agoWhy would you pick django-q over Celery, which also supports a database backend?
- closeparen 7y ago>Things like passing protobufs/gRPC when C structs, JSON, or CSV files will do These will always get the job done, but at great cost to future maintainers and callers. Explicitly declared interfaces that give some thought to future evolution aren't that hard, and can save a lot of pain down the road. I vividly remember the outage I caused by changing a struct field name, only to find out it was implicitly a critical part of the API "contract" when serialized to JSON. Another where I relied on a certain field from a dependency's response, only to find it out it was populated in some situations (the examples I used to learn the interface) and not others (revealed by production). Our Thrift services do not have these problems. Anyone can look at a diff and tell a) that the IDL is changing, b) whether the change is breaking, and c) whether a field is required or optional.
- deleted 7y ago[deleted]
- pjc50 7y agoGRPC is great as an API generator, but there's a lot of machinery under the hood in its network layer. It's not small - there was an embedded project where we were considering using it to replace our roll-your-own, then we discovered that the compiled library for GRPC was larger than the entire application. C structs are so brittle they should not be used for any kind of serialisation. You're just setting yourself up for upgrade nightmares. I wish BSON or the ASN.1 toolset were easier to use for this use case.
- fanf2 7y agoI hope the CBOR tooling is better https://cbor.io/tools.html https://cbor.io/tools.html
- CoolGuySteve 7y agoFor C structs, start with a uint32_t 4 char code identifier and a size field. Then you can extend them as long as you only ever append, which is a constraint on protobufs anyways. This is how most media containers have worked since the Amiga days. Meanwhile, their encoding/decoding is infinitely faster than most other serialization formats, allowing you to serialize data back and forth to disk or as IPC faster than anything else.
- RantyDave 7y agoC structs are really, really brittle; CSV is alarmingly complicated (unicode); JSON can't pass "None". I like CBOR, myself.
- mrmonkeyman 7y ago"clue": null
- nyuszika7h 7y ago> JSON can't pass "None" What about null?
- RantyDave 7y agoAhh, then I might be talking crap. I'll check it out, thanks.
- CoolGuySteve 7y agoCBOR has extremely slow encoding/decoding in C/C++ when compared to structs. Nothing is perfect.
- RantyDave 7y agoFor sure, compared to structs. Which you can almost certainly RDMA from one place to another. It's a little bit apples and oranges though since CBOR is self describing (and very lightly compressed). Shameless plug: https://github.com/RantyDave/cppbor https://github.com/RantyDave/cppbor
- acoard 7y agoYou can pass null, undefined, or remove the property. JSON Schema – a whole topic unto itself – gives you further control over locking down how this can be represented (e.g. field MUST be present but can be string matching this regex or null).