14 ms·
Show HN: Kim – A Python serialization and marshaling framework
- sakawa 9y agoIt does look like marshmalllow[1]. How does relate Kim with it? [1]: https://github.com/marshmallow-code/marshmallow/ https://github.com/marshmallow-code/marshmallow/
- tinnet 9y agoObviously no OS developer owes anybody an explanation, but man would I appreciate if more projects had a "why you should use this over related projects" (like e.g. pendulum does https://github.com/sdispater/pendulum/blob/master/README.rst#why-not-arrow https://github.com/sdispater/pendulum/blob/master/README.rst...)
- mikeywaites 9y agoHey. It's a great point and something we will certainly look to add to the documentation. To be honest the docs were the major thing that held up the release of Kim. We made the mistake of leaving them until last so in the end we opted for quality over quantity to get them finished. A great suggestion though, thanks!
- pekk 9y agoI know the pain of searching for software to meet your requirements. But unless you have a friend you can really trust to provide informed recommendations, nobody can take this pain away for you. If all require projects to say negative things about other people's projects while talking up their own, a lot of projects are going to distort the facts. In the end, if we don't have the ability to evaluate the software ourselves, then all we are measuring is who can shout the loudest and who is the most aggressive against other projects. Quiet projects will still be good, but now those would be overlooked even more because they aren't shouting. With this requirement you are making your life easier but you are making life harder on open source developers by forcing them to deal with unnecessary inter-project drama and to divert lots of effort into marketing that could have been put into code. That might make sense in proprietary products, but in open source this kind of demand just hurts the ecosystem. If the pain of choosing is too much then choose something that is standardized, or the most popular thing, or what your trusted friend recommends. People will seek out the very specific projects they need. If you don't even know why you are using something, it isn't the responsibility of someone else to tell you why you are using it!
- tinnet 9y agoI think you're right, when actually using it for production software it's probably wise to not be a trailblazer :) For me this wish for a comparison (that I'd love to be objective and in god spirit of course - naive?) is probably coming more from "shopping around" between projects. Or just when seeing a new thing on HN and wondering if I should investigate adding this particular thing to my toolbox.
- deleted 9y ago[deleted]
- jackqu7 9y ago(I'm Jack, another developer at OSL.) We started writing Kim around the same time as the Marshmallow project began as we found it wasn't suitable for our needs at that time, though it has come a long way since then. They are very similar projects and have similar functionality, but Kim has a focus on making it relatively simple to do unusual or 'advanced' things. For example, Kim supports polymorphism out of the box, if you have an AnimalMapper subclassed by a CatMapper and a DogMapper, passing a Cat and a Dog to AnimalMapper.many.serialize() will automatically do the right thing in a similar way to SQLAlchemy polymorphism. We also have support for complex requirements such as nesting the same object to itself (useful when your JSON representation is nested but your DB representation is flat,) serialising multiple object fields to a single JSON field (eg full_name consisting of obj.first_name and obj.last_name,) a range of security models for marshalling nested objects and a fairly extensible roles system. In general we've followed the philosophy "Simple things should be simple. Complex things should be possible."
- bpicolo 9y agoI've been saddened by Marshmallow on many occasions (I have gripes with the particular way defaults/validation play together. This is true for WTForms too). I'm excited to try out Kim. I've been very close to just writing my own serialization lib on many occasions. It looks like your pipelines might bring a bit of sanity to it. :) It looks like you support a few sorts of validation, but the docs aren't super clear as to what the expected validation strategy is. Could you elaborate on what that looks like? My typical strategy I'd like to do is to just a list of functions that take the input and return a boolean as far as validation goes.
- mikeywaites 9y agoHey that's really great to hear. (that you're keen to use Kim) WTF-Forms and Marshmallow both solve problems and they do it well but it seems like us you wanted something that offered just a bit more flexibility. That's totally the idea behind pipelines in Kim. They are like tiny little computer programmes and are really capable of anything (providing it's possible in Python of course :D ) It's great you asked this question as we noticed part of the documentation was actually broken. here's a link to a pretty basic example of adding extra validation "pipes" to a pipeline http://kim.readthedocs.io/en/latest/user/advanced.html#custom-validation-extra-marshal-pipes http://kim.readthedocs.io/en/latest/user/advanced.html#custo... We'd be more than happy to discuss how to solve more complex requirements if there's something specific you had in mind though. Thanks for the message!
- sametmax 9y agoI think marshmallow primary use case is to unserialize to nested dicts/lists while kim outputs full classes. Did I understood that right ?
- amelius 9y agoCan it serialize cycles?
- mikeywaites 9y agoHey Amelius, thanks for the message. Gonna be honest, I'm not sure what you mean by cycles. Can you elaborate a bit?
- emidln 9y agoa = {} b = {} a["b"] = b b["a"] = a a == deserialize(serialize(a))
- amelius 9y agoRoughly speaking, by cycles I mean a structure that refers to itself somehow. For example: A = {} B = {} A["ref"] = B B["ref"] = A So would it be possible to serialize A and B, and of course to deserialize them? Note that print A gives {'ref': {'ref': {...}}} which is of course not a suitable serialization, since you can't recover the original structure from it.
- jackqu7 9y agoYes, this is possible as long as the second level nested object has a role to stop infinite recursion from occurring. Cycles are not automatically detected. class BaseMapper(Mapper): __type__ = TestType score = Integer() nest = Nested('NestedMapper') __roles__ = {'nested': blacklist('nest')} class NestedMapper(Mapper): __type__ = TestType back = Nested('BaseMapper', role='nested') name = String() obj2 = TestType(name='test') obj = TestType(score=5, nest=obj2) obj2.back = obj >> BaseMapper(obj=obj).serialize() {'nest': {'back': {'score': 5}, 'name': 'test'}, 'score': 5}
- amelius 9y agoThe output isn't clear to me: {'nest': {'back': {'score': 5}, 'name': 'test'}, 'score': 5} How can this be mapped back unambiguously to a cyclic structure? (I might be wrong but it seems to me that the act of serialization has simply expanded the cycle one level deep.)
- siddhant 9y agoCool! Are there any speed comparisons available between this and marshmallow (or other alternatives)?
- mikeywaites 9y agoHi Siddhant, We've not really dug into performance yet, though if you look at the last patch (1.0.2) we yielded a 10% speed up by removing an erroneous try/except block. We've really focussed on features initially and performance is something we're actively researching now. Perhaps we can get some initial benchmarks together and share them with you this week. They will be useful no doubt as we start to plan a release focussed on speed ups. Thanks for reaching out!
- rat87 9y agoI was just looking for something like this or marshmallow.
- sandGorgon 9y agoWe are really looking for serialization libraries that will work with pandas and scikit. This stuff is really all over the place - PMML, Arrow, Dill, pickle. Some stuff won't work with one or the other. I will actually pay for consistency versus performance. There are way too many primitive serialization libraries. Surprisingly none for the higher order ML, etc stuff. Give the kind of people behind Arrow, I would love wrapper that will use Arrow to do all of this...But doesn't matter at the end of the day.
- mikeywaites 9y agoOne of the things we felt very strongly about when developing Kim was that Simple things should be simple. Complex things should be possible. To that end the Pipeline system behind the Field objects really does allow anything to be achieved. Wether thats producing values from composite fields or handling unique or non standard data types. it would be great if you can share some ways that you specifically need serialization to work for something like pandas, or better yet, some ways existing solutions don’t work with pandas. We’ve had some pretty unique requirements ourselves and have not found any blockers yet. Thanks for the message.
- mhneu 9y agoPython's data infrastructure has a huge problem: serialization and thus saving data results. A good serialization library should serialize: - classes/objects (best practice: objects for holding data) - pandas/numpy objects (must have: minimizing space) - namedtuples (currently: a mess, factory implementation) - dicts and lists of dicts (must have: space efficiency) Compare to Matlab: save(f, 'anyobject'); anyobject=load(f) Python is terrible at this and it limits use in real data analysis environments and limits competition with matlab.
- mikeywaites 9y agoThanks for expanding on that mhneu. So our primary focus with Kim has certainly been around serializing/marshaling JSON though we've used it for plenty of other uses cases. It's great to get a view of other problems people are experiencing. Now we've finished wrapping up 1.0.0 we're going to be spending some time on the roadmap of new features. I personally feel variation in use cases from our own is only going to help make Kim better so we'll defo look into this problem some more in the near future. Right now though i couldn't say for sure what Kim would have to offer when working with Pandas etc as we've simply never tried.
- Dowwie 9y agoCool project! In the case of serialization libraries,unless you are validating as part of your (de)serialization, I'd recommend avoiding schema-driven serialization libraries. These Kim-like libraries, such as Marshmallow, introduce quite a bit of overhead. If validation isn't required and performance matters, I recommend choosing a lighter-weight serialization/marshalling alternative, such as that provided by asphalt-serialization: https://github.com/asphalt-framework/asphalt-serialization https://github.com/asphalt-framework/asphalt-serialization Asphalt-serialization supports cbor, msgpack, json, ... and is easy to wire up This recommendation is based on my own experience using Marshmallow for Yosai, analyzing its performance and then refactoring to a ported version of asphalt-serialization.
- mikeywaites 9y agoHey Dowwie! That's a great point and an important distinction to make. As I mentioned in some of the other comments, we have certainly been focussed on features over performance so far but we are actively working on dramatically improving the performance of Kim. I guess it's almost important to pick the right tool for the job. Thanks for sharing the link to asphalt too. I'd not see that before.
- Dowwie 9y agoKeep up the good work, Mikey. :) See you at PyCon, maybe?
- mikeywaites 9y agoOne of the engineers from our team is going to be there for sure. Im certainly keen to go so fingers crossed.
- yeukhon 9y agoNice, but I recommend closing issues https://github.com/mikeywaites/kim/issues https://github.com/mikeywaites/kim/issues which have fixes (some of them show 'merge'). It's one thing I as a user look at choosing whether to adopt a project or not.
- mikeywaites 9y agoabsolutely. Im a bit annoyed at myself that I hadn't got round to that yet but thanks for raising it.
- yeukhon 9y agoI will submit a PR for some doc fixes :-) on the way look out next 24 hours! This is an awesome project for a couple years, great run!
- BuuQu9hu 9y agoSorry, I must be harsh. No. This fundamentally doesn't offer much advantage over a .toJSON() instance method and a .fromJSON() class method. Don't say "security-focused" if you can't handle cyclic object graphs.
- mafro 9y agoPlease elaborate on the reasons for your opinion :)
- limdauto 9y agoI'd like to congratulate the authors regarding the clever naming. I totally get the Eminem's reference. Disclaimer: Posting this comment because my colleague pointed out that I could get some points.
- _e 9y agoKim: A JSON Serialization and Marshaling framework that Mathers
- ziikutv 9y agoSo this takes JSON and maps it to namedtuples? Silly question, what happens with Unicode?
- voidfiles 9y agoI added Kim to my ongoing set of python serialization framework benchmarks here is how it ranks. Library Many Objects One Object --------------------- -------------- ------------ Custom 0.0187769 0.00682402 Strainer 0.0603201 0.0337129 serpy 0.073787 0.038656 Lollipop 0.47821 0.231566 Marshmallow 1.14844 0.598486 Django REST Framework 1.94096 1.3277 kim 2.28477 1.15237 Comments on how to improve the benchmark are appreciated. source: https://voidfiles.github.io/python-serialization-benchmark/ https://voidfiles.github.io/python-serialization-benchmark/
- RussianCow 9y agoJust a minor note: It seems you don't mention anywhere what those numbers actually mean. I'm assuming they are seconds, but I can't know for certain, which makes it really unclear if Kim is the fastest or the slowest.
- makmanalp 9y agoThis is brilliant, exactly what I was looking for. I did a profile recently on some API calls and found that 40-50% was being spent on serialization with marshmallow, which I'm looking to drop. I'll be doing this stuff for myself, but would you be curious in having: a) Support for lima: https://lima.readthedocs.io/en/latest/ https://lima.readthedocs.io/en/latest/ b) more benchmark cases (serializing a larger list of objects)
- mikeywaites 9y agoThanks so much for this Voidfiles. We were under no illusions that we weren't the most performant library out there (yet) This is a great start for us understanding where we need to get to! We've got some work to do :)
- ff7c11 9y agowhy not pickle :) :)
- deleted 9y ago[deleted]