9 ms·

Glom – Restructured Data for Python

313 points by mhashemi 8y ago

amock 8y ago
I haven't played with this yet, but it looks really handy. I deal with much more JSON on the command line than I'd like, so I think having both a single library and command line tool to reshape that data will make that much easier. I've used jq a few times, but when I want to move a little beyond what it does I usually end up writing a Python script. Hopefully this will make that transition smoother.
- mhashemi 8y ago
  Haha, I'm all for console usage, but let me tell you, there's nothing quite like that feeling of moving a working spec into a dedicated application with exception handling, logging, etc. :)
tincholio 8y ago
It looks quite similar in spirit to Clojure's Specter library (https://github.com/nathanmarz/specter https://github.com/nathanmarz/specter), and even seems to have a nod to it (The Spectre of Structure).
- mhashemi 8y ago
  Oh nice! Total coincidence, I assure you. Still, I should read on this and add it to the analogy doc: http://glom.readthedocs.io/en/latest/by_analogy.html http://glom.readthedocs.io/en/latest/by_analogy.html Declarative data transformation generates a lot of comparisons (almost all of them great, though!).
- hessammehr 8y ago
  Ah you beat me to it! :)
faitswulff 8y ago
This is really cool. Did you ever consider an API to do the reverse - to insert a value at a particular point in the data? My interest stems from this issue[0] on the Ruby issue tracker to make a symmetrical method to Hash#dig (which does something similar to, but more limited than glom) called Hash#bury. The problem in the issue was that inserting a value at a given index in an array proved difficult and unnatural in Ruby, so I was wondering if there were other solutions out there. Another question occurs to me - does glom only support string keys? [0]: https://bugs.ruby-lang.org/issues/11747 https://bugs.ruby-lang.org/issues/11747
- mhashemi 8y ago
  glom not only supports more than string keys, it also supports assigning to non-dictionary objects. That's a part of the API we're working on right now, actually. As for the data insertion, mutation may be in the future, but for now glom only transforms and returns new objects. Definitely something to think about though, bookmarked! :)
  - hiccuphippo 8y ago
    Check the updateIn, mergeIn, mergeDeepIn methods in immutable.js. Maybe even asMutable, asImmutable for mutation.
- doublereedkurt 8y ago
  'string-key', T['string-key'], T[not-string-key] :-)
- smittywerben 8y ago
  Perhaps assoc_in or update_in from the Toolz library fits this? https://toolz.readthedocs.io/en/latest/api.html#toolz.dicttoolz.assoc_in https://toolz.readthedocs.io/en/latest/api.html#toolz.dictto... https://toolz.readthedocs.io/en/latest/api.html#toolz.dicttoolz.update_in https://toolz.readthedocs.io/en/latest/api.html#toolz.dictto...
tathougies 8y ago
This seems like lenses for python... neat! I often use python to mess around with things, and almost always miss Haskell's lenses when doing so. This seems like an interesting solution.
hessammehr 8y ago
This might have been unintentional, but I suspect "Spectre of Structure" and "Python's missing piece" refer to Nathan Marz's specter library for clojure [1], similarly touted as clojure's missing piece. I tend to agree in the case of specter, given the mind-boggling types of transformations that are easily (and simply) expressed in it (and often run faster than idiomatic clojure as well). Highly recommended if you ever need to work with deeply nested data structures. [1] https://github.com/nathanmarz/specter https://github.com/nathanmarz/specter
- mhashemi 8y ago
  Total coincidence! Reading the README, Nathan and I are definitely on the same wavelength though. When I get a chance I'll add it to the analogies doc: http://glom.readthedocs.io/en/latest/by_analogy.html http://glom.readthedocs.io/en/latest/by_analogy.html :)
  - mhashemi 8y ago
    What's really interesting is that we're approaching the same ideal state from different directions. Specter goes from Clojure's immutability to something more practical, from Python's super dynamic system to something more declarative and immutable.
lapnitnelav 8y ago
Looks really neat. Striking a balance between ease of use / simplicity and powerful features is a tough exercise but you did well. I can foresee the CLI being quite useful to do away with the run-of-the-mill sed / awk / grep [...] mess. Specifically for the less CLI inclined people out there.
icebraining 8y ago
Can it be used bidirectionally, without having to repeat the work? I have a need to transform between pairs of structures, in both directions, and ever since I found JsonGrammar (https://github.com/MedeaMelana/JsonGrammar2 https://github.com/MedeaMelana/JsonGrammar2) I've been pining for a Python version.
- mhashemi 8y ago
  It depends on the complexity of the spec, but we've already done some programmatic building of glomspecs, so for many cases I think the answer is yes! Once we feel out the patterns I think glom will gain some utilities for this purpose.
  - icebraining 8y ago
    Thanks, that's awesome!
- neuronexmachina 8y ago
  It seems like a subset of glom specs would be uniquely invertable. For example, the spec `{'c': 'a.b'}` could trivially invert to `{'a.b': 'c'}`. I'm not sure how you'd invert more complex specs which make function calls, e.g. sum or len.
boringg 8y ago
Looks slick!
jasonpeacock 8y ago
I'm probably being dense, but I don't see a good description of the input data types supported - the CLI says "json or python". It would be great to have clarification if this is JSON only, or supports other data structures, or parsers could be plugged in?
- mhashemi 8y ago
  From within Python, all objects are supported by default. If you can parse it, you can glom it. You can even register additional behaviors for specific types to keep your specs tight: http://glom.readthedocs.io/en/latest/api.html#setup-and-registration http://glom.readthedocs.io/en/latest/api.html#setup-and-regi... (example: http://glom.readthedocs.io/en/latest/snippets.html#automatic-django-orm-type-handling http://glom.readthedocs.io/en/latest/snippets.html#automatic...) The CLI is in a pretty preliminary state, usable but not as robust as it will be in a few weeks. It only supports built-in parsers (JSON and Python literals) What formats are you thinking? YAML?
  - jasonpeacock 8y ago
    Aha. Thanks! I understand better now, I didn't realize it was so general-purpose :)
nerdponx 8y ago
It's a nice idea, but i never like writing what amounts to a DSL in strings in my code (yes, that applies to in-code SQL as well, although that's often unavoidable). I prefer the `get_in()` method from Toolz: http://toolz.readthedocs.io/en/latest/api.html#toolz.dicttoolz.get_in http://toolz.readthedocs.io/en/latest/api.html#toolz.dicttoo...
- doublereedkurt1 8y ago
  the only string parsing is 'a.b.c', which is mostly equivalent to T.a.b.c so you can completely ignore that capability the very slight difference is that using T you must be explicit about attribute access vs key access whereas 'a.b.c' will try both
- radmin 8y ago
  My thoughts also turned to toolz. Here's a example comparison: glom glom(target, ('system.planets', ['name'])) # ['earth', 'jupiter'] toolz list(pluck('name', get_in(('system', 'planets'), target))) # ['earth', 'jupiter']
  - mhashemi 8y ago
    The real advantage over toolz/get_in is in the breadth of types glom can support (http://glom.readthedocs.io/en/latest/api.html#setup-and-registration http://glom.readthedocs.io/en/latest/api.html#setup-and-regi...), the extensive fallback behavior (http://glom.readthedocs.io/en/latest/api.html#advanced-specifiers http://glom.readthedocs.io/en/latest/api.html#advanced-speci...), and "T" specifier (https://sedimental.org/glom_restructured_data.html#true-python-native https://sedimental.org/glom_restructured_data.html#true-pyth...), which allows performing object-oriented traversals and calls. Check the post for more examples of where path-based access falls short of the mark! (Sidenote: I find it very clunky that get_in() takes a "default" kwarg and _also_ a n"o_default" kwarg. Use a Sentinel object! http://boltons.readthedocs.io/en/latest/typeutils.html#boltons.typeutils.make_sentinel http://boltons.readthedocs.io/en/latest/typeutils.html#bolto... )
    - 8y ago
rodrigoalviani 8y ago
Cool... it looks like... errr... javascript :D
sixdimensional 8y ago
I had a quick look, but I didn't see filtering expressions, only shaping expressions. It seems like glom is more of a result shaper/mapper. Can you filter with glom (maybe with lambdas or something)? I could see the two going together quite well if you were "glomming" a big Python object.
- mhashemi 8y ago
  Filtering is supported through the OMIT value: http://glom.readthedocs.io/en/latest/api.html#glom.OMIT http://glom.readthedocs.io/en/latest/api.html#glom.OMIT (another example: http://glom.readthedocs.io/en/latest/snippets.html#filtered-iteration http://glom.readthedocs.io/en/latest/snippets.html#filtered-... ) Lambdas and functions are always a safe fallback, but glom does its best to keep your specs readable and roundtrippable (gotta love a nice repr()).
  - sixdimensional 8y ago
    That's cool! Thank you for pointing it out!
codezero 8y ago
This is pretty neat and is something I've been thinking about for a current project. Does anyone know if something similar exists in Java/Scala land?
- zaptheimpaler 8y ago
  I think this is similar to lenses in FP languages - check out Monocle for Scala.
zestyping 8y ago
The writing style is just insufferable. Even the API documentation is littered with hyperbole and self-congratulation. We get it, you're proud of your work and extremely proud of yourself. > "as simple and powerful as glom" > "big things come in small packages" > "small API with big functionality" > "power is only surpassed by its intuitiveness" > "simplicity is only surpassed by its utility" > "shortest-named feature may be its most powerful" For heaven's sake, give it a rest! It's a big red flag about your priorities that when I go looking for a precise specification, I can't find answers to simple questions and instead end up wading through incessant marketing phrases. I tried, and I finally gave up halfway through the API doc. It might even be the case that glom is a good idea—but you're making it really hard to trust you as a source of objective information about it. Show, don't tell. My advice to you: you'll generate more interest if you delete every congratulatory word on those pages and focus entirely on helping your readers understand what glom does instead of trying to sell it to them.
- mhashemi 8y ago
  Hey Ka-Ping! Maybe I did get carried away :) How I wish one could publish a dry document and expect people to read all the way to the bottom. I've published enough libraries to know that's not the case. glom's free software so it's all there, as "shown" as can be. But referring you to the code wouldn't be very considerate either. Instead, here's this literate code version that I prepared in advance. Hopefully this will be of more help to you: http://glom.readthedocs.io/en/latest/faq.html#how-does-glom-work http://glom.readthedocs.io/en/latest/faq.html#how-does-glom-...
  - liteye 8y ago
    In http://glom.readthedocs.io/en/latest/tutorial.html#access-granted http://glom.readthedocs.io/en/latest/tutorial.html#access-gr..., > After years of research and countless iterations, the glom team landed on this simple construct: 'years of research', 'countless iterations', 'glom team', really?
    - mhashemi 8y ago
jbritton 8y ago
There is also this Python lens library. https://github.com/ingolemo/python-lenses https://github.com/ingolemo/python-lenses I can't say how they compare, but they have some overlapping features.
staticautomatic 8y ago
Aside: are there any libraries for JSON or dict-like formats with xpath-style querying that are as quick under the hood as lxml is for xml?
harel 8y ago
This reminds me a little of the excellent dpath lib: https://github.com/akesterson/dpath-python https://github.com/akesterson/dpath-python
spedru 8y ago
I'm not really versed in the idioms/social mores of Python, so please take the following with a grain of salt: This seems like it usefully solves a problem, but the invocation pattern is suspect to me -- Instead of "glom" taking the target for picking-apart plus a magic little bit of DSL, what if "glom" took a single parameter, the aforementioned DSL, and returned a function that would perform the corresponding search when called on a target? Even if Python or this package optimises away repeatedly searching (by the same spec|in the same manner), the convention the package prescribes is odd to me, right after the first few paragraphs of intro.
- seanc 8y ago
  Python regex library does this, optionally
  - daturkel 8y ago
    Similarly, statistical distributions in SciPy can be used in "frozen" form (pre-parameterized) or in a more general form where you supply the parameters at the same time you are requesting some attribute it the distribution. Seems to me to be a situation where one is useful if you expect reuse, and the other is useful if you don't.
- mhashemi 8y ago
  The big, classical school of Python definitely prefers top-level functions. Still, I definitely understand that aesthetic, and am on board with not using functools.partial to achieve it. So: https://github.com/mahmoud/glom/issues/14 https://github.com/mahmoud/glom/issues/14 :)
loop0 8y ago
I'm curious about how the author replaced DRF with glom
- mhashemi 8y ago
  DRF still takes care of negotiating formats, etc., but it replaces the serializers. I'll see about getting an example in the repo, stay tuned.
bluemanshoe2 8y ago
Compare/contrast with pstar: https://github.com/iansf/pstar https://github.com/iansf/pstar
agf 8y ago
It seems to me like the advantage to focus on here is the improved error / `None` handling, which will speed debugging and make handling expected edge cases easier. I've seen a lot of inexperienced developers tripped up entirely by this kind of data access, and seen plenty of experienced developers waste time debugging it because of the exact error cases the announcement references. The `T` object, which the article describes as its most powerful, can be a useful pattern in some situations, but it's worth pointing out it isn't new or unique to this project. The author says in another thread here that he first started working on the "stuff leading up to glom" in 2013. One older example, which is virtually identical though less complete, is this Stack Overflow answer I posted in 2012: https://stackoverflow.com/a/9920723/500584 https://stackoverflow.com/a/9920723/500584 I'd seen the general pattern even before that post, if not the Pythonic syntax. I don't think that it's much of an improvement over defining a `lambda`, so again I would say the thing to focus on is the improved debugability and the simpler, dot-notation-as-generic-attribute-or-item-accessor syntax. I think `T` is largely a distraction, or should be reserved for advanced users.
- heavenlyblue 8y ago
  I would like to see the author debugging an application with 10 levels of object wrapping that had one of the middle object’s name misspelled. Libraries like these shine only if they have brilliant tracing and debugging capabilities; otherwise are too easy to reduce to literally a single function.
  - doublereedkurt 8y ago
    http://glom.readthedocs.io/en/latest/api.html#debugging http://glom.readthedocs.io/en/latest/api.html#debugging affordances to add tracing prints, or drop into a pdb at any level The Inspect specifier type provides a way to get visibility into glom’s evaluation of a specification, enabling debugging of those tricky problems that may arise with unexpected data. Inspect can be inserted into an existing spec in one of two ways. First, as a wrapper around the spec in question, or second, as an argument-less placeholder wherever a spec could be. Inspect supports several modes, controlled by keyword arguments. Its default, no-argument mode, simply echos the state of the glom at the point where it appears:
wildleaf 8y ago
Shameless plug: https://www.npmjs.com/package/safely-nested https://www.npmjs.com/package/safely-nested Nothing special, glom just reminded me of it.
derefr 8y ago
My favourite approach to this so far, that I would like other libraries to copy, is Elixir’s Access protocol, which gives you e.g.: foo = %{key: [[1, 2], [3, 4], [5, 6]]} path = [:key, Access.all, Access.at(0)] get_in foo, path # => [1, 3, 5] update_in foo, path, &(&1 * 10) # => %{key: [[10, 2], [30, 4], [50, 6]]} foo |> put_in([:key, Access.all], “foo”) |> put_in([:new_key], “bar”) # => %{key: [“foo”, “foo”, “foo”], new_key: “bar”} That third form is essentially the equivalent of building up a complex object through a series of mutations—but entirely functional.
Corristowolf 8y ago
Can anyone explain the benefit of glom instead of pandas dataframes? For data manipulation pandas is superior in everyway to any library out there. The only thing i think glom would be useful is json parsing as was shown. Sorry if I sound a bit too harsh. I'm looking at it from a data science pov.
wbolster 8y ago
in a similar spirit, i wrote "sanest", sane nested objects, tailored specifically for json fornats: https://sanest.readthedocs.io/ https://sanest.readthedocs.io/ it does not have the exact same feature set though. my focus was mostly on both reading and modifying nested structures in a type safe way.
heavenlyblue 8y ago
I think this project defines the first step at transitioning pip to a js-like repository of single-function modules. Hurray for kool-aid.
pdobsan 8y ago
There is already a well established Gnome project with the same name: http://www.glom.org http://www.glom.org It is a GTK+ front-end to PostgreSQL, similar to Microsoft Access.

313 points by mhashemi 8y ago

I haven't played with this yet, but it looks really handy. I deal with much more JSON on the command line than I'd like, so I think having both a single library and command line tool to reshape that data will make that much easier. I've used jq a few times, but when I want to move a little beyond what it does I usually end up writing a Python script. Hopefully this will make that transition smoother.

mhashemi 8y ago
Haha, I'm all for console usage, but let me tell you, there's nothing quite like that feeling of moving a working spec into a dedicated application with exception handling, logging, etc. :)

tincholio 8y ago

It looks quite similar in spirit to Clojure's Specter library (https://github.com/nathanmarz/specter https://github.com/nathanmarz/specter), and even seems to have a nod to it (The Spectre of Structure).

mhashemi 8y ago
Oh nice! Total coincidence, I assure you. Still, I should read on this and add it to the analogy doc: http://glom.readthedocs.io/en/latest/by_analogy.html http://glom.readthedocs.io/en/latest/by_analogy.html Declarative data transformation generates a lot of comparisons (almost all of them great, though!).
hessammehr 8y ago
Ah you beat me to it! :)

faitswulff 8y ago

This is really cool. Did you ever consider an API to do the reverse - to insert a value at a particular point in the data? My interest stems from this issue[0] on the Ruby issue tracker to make a symmetrical method to Hash#dig (which does something similar to, but more limited than glom) called Hash#bury. The problem in the issue was that inserting a value at a given index in an array proved difficult and unnatural in Ruby, so I was wondering if there were other solutions out there. Another question occurs to me - does glom only support string keys? [0]: https://bugs.ruby-lang.org/issues/11747 https://bugs.ruby-lang.org/issues/11747

mhashemi 8y ago
glom not only supports more than string keys, it also supports assigning to non-dictionary objects. That's a part of the API we're working on right now, actually. As for the data insertion, mutation may be in the future, but for now glom only transforms and returns new objects. Definitely something to think about though, bookmarked! :)
- hiccuphippo 8y ago
  Check the updateIn, mergeIn, mergeDeepIn methods in immutable.js. Maybe even asMutable, asImmutable for mutation.
doublereedkurt 8y ago
'string-key', T['string-key'], T[not-string-key] :-)
smittywerben 8y ago
Perhaps assoc_in or update_in from the Toolz library fits this? https://toolz.readthedocs.io/en/latest/api.html#toolz.dicttoolz.assoc_in https://toolz.readthedocs.io/en/latest/api.html#toolz.dictto... https://toolz.readthedocs.io/en/latest/api.html#toolz.dicttoolz.update_in https://toolz.readthedocs.io/en/latest/api.html#toolz.dictto...

tathougies 8y ago

This seems like lenses for python... neat! I often use python to mess around with things, and almost always miss Haskell's lenses when doing so. This seems like an interesting solution.

hessammehr 8y ago

This might have been unintentional, but I suspect "Spectre of Structure" and "Python's missing piece" refer to Nathan Marz's specter library for clojure [1], similarly touted as clojure's missing piece. I tend to agree in the case of specter, given the mind-boggling types of transformations that are easily (and simply) expressed in it (and often run faster than idiomatic clojure as well). Highly recommended if you ever need to work with deeply nested data structures. [1] https://github.com/nathanmarz/specter https://github.com/nathanmarz/specter

mhashemi 8y ago
Total coincidence! Reading the README, Nathan and I are definitely on the same wavelength though. When I get a chance I'll add it to the analogies doc: http://glom.readthedocs.io/en/latest/by_analogy.html http://glom.readthedocs.io/en/latest/by_analogy.html :)
- mhashemi 8y ago
  What's really interesting is that we're approaching the same ideal state from different directions. Specter goes from Clojure's immutability to something more practical, from Python's super dynamic system to something more declarative and immutable.

lapnitnelav 8y ago

Looks really neat. Striking a balance between ease of use / simplicity and powerful features is a tough exercise but you did well. I can foresee the CLI being quite useful to do away with the run-of-the-mill sed / awk / grep [...] mess. Specifically for the less CLI inclined people out there.

icebraining 8y ago

Can it be used bidirectionally, without having to repeat the work? I have a need to transform between pairs of structures, in both directions, and ever since I found JsonGrammar (https://github.com/MedeaMelana/JsonGrammar2 https://github.com/MedeaMelana/JsonGrammar2) I've been pining for a Python version.

mhashemi 8y ago
It depends on the complexity of the spec, but we've already done some programmatic building of glomspecs, so for many cases I think the answer is yes! Once we feel out the patterns I think glom will gain some utilities for this purpose.
- icebraining 8y ago
  Thanks, that's awesome!
neuronexmachina 8y ago
It seems like a subset of glom specs would be uniquely invertable. For example, the spec `{'c': 'a.b'}` could trivially invert to `{'a.b': 'c'}`. I'm not sure how you'd invert more complex specs which make function calls, e.g. sum or len.

boringg 8y ago

Looks slick!

jasonpeacock 8y ago

I'm probably being dense, but I don't see a good description of the input data types supported - the CLI says "json or python". It would be great to have clarification if this is JSON only, or supports other data structures, or parsers could be plugged in?

mhashemi 8y ago
From within Python, all objects are supported by default. If you can parse it, you can glom it. You can even register additional behaviors for specific types to keep your specs tight: http://glom.readthedocs.io/en/latest/api.html#setup-and-registration http://glom.readthedocs.io/en/latest/api.html#setup-and-regi... (example: http://glom.readthedocs.io/en/latest/snippets.html#automatic-django-orm-type-handling http://glom.readthedocs.io/en/latest/snippets.html#automatic...) The CLI is in a pretty preliminary state, usable but not as robust as it will be in a few weeks. It only supports built-in parsers (JSON and Python literals) What formats are you thinking? YAML?
- jasonpeacock 8y ago
  Aha. Thanks! I understand better now, I didn't realize it was so general-purpose :)

nerdponx 8y ago

It's a nice idea, but i never like writing what amounts to a DSL in strings in my code (yes, that applies to in-code SQL as well, although that's often unavoidable). I prefer the `get_in()` method from Toolz: http://toolz.readthedocs.io/en/latest/api.html#toolz.dicttoolz.get_in http://toolz.readthedocs.io/en/latest/api.html#toolz.dicttoo...

doublereedkurt1 8y ago
the only string parsing is 'a.b.c', which is mostly equivalent to T.a.b.c so you can completely ignore that capability the very slight difference is that using T you must be explicit about attribute access vs key access whereas 'a.b.c' will try both
radmin 8y ago
My thoughts also turned to toolz. Here's a example comparison: glom glom(target, ('system.planets', ['name'])) # ['earth', 'jupiter'] toolz list(pluck('name', get_in(('system', 'planets'), target))) # ['earth', 'jupiter']
- mhashemi 8y ago
  The real advantage over toolz/get_in is in the breadth of types glom can support (http://glom.readthedocs.io/en/latest/api.html#setup-and-registration http://glom.readthedocs.io/en/latest/api.html#setup-and-regi...), the extensive fallback behavior (http://glom.readthedocs.io/en/latest/api.html#advanced-specifiers http://glom.readthedocs.io/en/latest/api.html#advanced-speci...), and "T" specifier (https://sedimental.org/glom_restructured_data.html#true-python-native https://sedimental.org/glom_restructured_data.html#true-pyth...), which allows performing object-oriented traversals and calls. Check the post for more examples of where path-based access falls short of the mark! (Sidenote: I find it very clunky that get_in() takes a "default" kwarg and _also_ a n"o_default" kwarg. Use a Sentinel object! http://boltons.readthedocs.io/en/latest/typeutils.html#boltons.typeutils.make_sentinel http://boltons.readthedocs.io/en/latest/typeutils.html#bolto... )
  - 8y ago

rodrigoalviani 8y ago

Cool... it looks like... errr... javascript :D

sixdimensional 8y ago

I had a quick look, but I didn't see filtering expressions, only shaping expressions. It seems like glom is more of a result shaper/mapper. Can you filter with glom (maybe with lambdas or something)? I could see the two going together quite well if you were "glomming" a big Python object.

mhashemi 8y ago
Filtering is supported through the OMIT value: http://glom.readthedocs.io/en/latest/api.html#glom.OMIT http://glom.readthedocs.io/en/latest/api.html#glom.OMIT (another example: http://glom.readthedocs.io/en/latest/snippets.html#filtered-iteration http://glom.readthedocs.io/en/latest/snippets.html#filtered-... ) Lambdas and functions are always a safe fallback, but glom does its best to keep your specs readable and roundtrippable (gotta love a nice repr()).
- sixdimensional 8y ago
  That's cool! Thank you for pointing it out!

codezero 8y ago

This is pretty neat and is something I've been thinking about for a current project. Does anyone know if something similar exists in Java/Scala land?

zaptheimpaler 8y ago
I think this is similar to lenses in FP languages - check out Monocle for Scala.

zestyping 8y ago

The writing style is just insufferable. Even the API documentation is littered with hyperbole and self-congratulation. We get it, you're proud of your work and extremely proud of yourself. > "as simple and powerful as glom" > "big things come in small packages" > "small API with big functionality" > "power is only surpassed by its intuitiveness" > "simplicity is only surpassed by its utility" > "shortest-named feature may be its most powerful" For heaven's sake, give it a rest! It's a big red flag about your priorities that when I go looking for a precise specification, I can't find answers to simple questions and instead end up wading through incessant marketing phrases. I tried, and I finally gave up halfway through the API doc. It might even be the case that glom is a good idea—but you're making it really hard to trust you as a source of objective information about it. Show, don't tell. My advice to you: you'll generate more interest if you delete every congratulatory word on those pages and focus entirely on helping your readers understand what glom does instead of trying to sell it to them.

mhashemi 8y ago
Hey Ka-Ping! Maybe I did get carried away :) How I wish one could publish a dry document and expect people to read all the way to the bottom. I've published enough libraries to know that's not the case. glom's free software so it's all there, as "shown" as can be. But referring you to the code wouldn't be very considerate either. Instead, here's this literate code version that I prepared in advance. Hopefully this will be of more help to you: http://glom.readthedocs.io/en/latest/faq.html#how-does-glom-work http://glom.readthedocs.io/en/latest/faq.html#how-does-glom-...
- liteye 8y ago
  In http://glom.readthedocs.io/en/latest/tutorial.html#access-granted http://glom.readthedocs.io/en/latest/tutorial.html#access-gr..., > After years of research and countless iterations, the glom team landed on this simple construct: 'years of research', 'countless iterations', 'glom team', really?
  - mhashemi 8y ago

jbritton 8y ago

There is also this Python lens library. https://github.com/ingolemo/python-lenses https://github.com/ingolemo/python-lenses I can't say how they compare, but they have some overlapping features.

staticautomatic 8y ago

Aside: are there any libraries for JSON or dict-like formats with xpath-style querying that are as quick under the hood as lxml is for xml?

harel 8y ago

This reminds me a little of the excellent dpath lib: https://github.com/akesterson/dpath-python https://github.com/akesterson/dpath-python

spedru 8y ago

I'm not really versed in the idioms/social mores of Python, so please take the following with a grain of salt: This seems like it usefully solves a problem, but the invocation pattern is suspect to me -- Instead of "glom" taking the target for picking-apart plus a magic little bit of DSL, what if "glom" took a single parameter, the aforementioned DSL, and returned a function that would perform the corresponding search when called on a target? Even if Python or this package optimises away repeatedly searching (by the same spec|in the same manner), the convention the package prescribes is odd to me, right after the first few paragraphs of intro.

seanc 8y ago
Python regex library does this, optionally
- daturkel 8y ago
  Similarly, statistical distributions in SciPy can be used in "frozen" form (pre-parameterized) or in a more general form where you supply the parameters at the same time you are requesting some attribute it the distribution. Seems to me to be a situation where one is useful if you expect reuse, and the other is useful if you don't.
mhashemi 8y ago
The big, classical school of Python definitely prefers top-level functions. Still, I definitely understand that aesthetic, and am on board with not using functools.partial to achieve it. So: https://github.com/mahmoud/glom/issues/14 https://github.com/mahmoud/glom/issues/14 :)

loop0 8y ago

I'm curious about how the author replaced DRF with glom

mhashemi 8y ago
DRF still takes care of negotiating formats, etc., but it replaces the serializers. I'll see about getting an example in the repo, stay tuned.

bluemanshoe2 8y ago

Compare/contrast with pstar: https://github.com/iansf/pstar https://github.com/iansf/pstar

agf 8y ago

It seems to me like the advantage to focus on here is the improved error / `None` handling, which will speed debugging and make handling expected edge cases easier. I've seen a lot of inexperienced developers tripped up entirely by this kind of data access, and seen plenty of experienced developers waste time debugging it because of the exact error cases the announcement references. The `T` object, which the article describes as its most powerful, can be a useful pattern in some situations, but it's worth pointing out it isn't new or unique to this project. The author says in another thread here that he first started working on the "stuff leading up to glom" in 2013. One older example, which is virtually identical though less complete, is this Stack Overflow answer I posted in 2012: https://stackoverflow.com/a/9920723/500584 https://stackoverflow.com/a/9920723/500584 I'd seen the general pattern even before that post, if not the Pythonic syntax. I don't think that it's much of an improvement over defining a `lambda`, so again I would say the thing to focus on is the improved debugability and the simpler, dot-notation-as-generic-attribute-or-item-accessor syntax. I think `T` is largely a distraction, or should be reserved for advanced users.

heavenlyblue 8y ago
I would like to see the author debugging an application with 10 levels of object wrapping that had one of the middle object’s name misspelled. Libraries like these shine only if they have brilliant tracing and debugging capabilities; otherwise are too easy to reduce to literally a single function.
- doublereedkurt 8y ago
  http://glom.readthedocs.io/en/latest/api.html#debugging http://glom.readthedocs.io/en/latest/api.html#debugging affordances to add tracing prints, or drop into a pdb at any level The Inspect specifier type provides a way to get visibility into glom’s evaluation of a specification, enabling debugging of those tricky problems that may arise with unexpected data. Inspect can be inserted into an existing spec in one of two ways. First, as a wrapper around the spec in question, or second, as an argument-less placeholder wherever a spec could be. Inspect supports several modes, controlled by keyword arguments. Its default, no-argument mode, simply echos the state of the glom at the point where it appears:

wildleaf 8y ago

Shameless plug: https://www.npmjs.com/package/safely-nested https://www.npmjs.com/package/safely-nested Nothing special, glom just reminded me of it.

derefr 8y ago

My favourite approach to this so far, that I would like other libraries to copy, is Elixir’s Access protocol, which gives you e.g.: foo = %{key: [[1, 2], [3, 4], [5, 6]]} path = [:key, Access.all, Access.at(0)] get_in foo, path # => [1, 3, 5] update_in foo, path, &(&1 * 10) # => %{key: [[10, 2], [30, 4], [50, 6]]} foo |> put_in([:key, Access.all], “foo”) |> put_in([:new_key], “bar”) # => %{key: [“foo”, “foo”, “foo”], new_key: “bar”} That third form is essentially the equivalent of building up a complex object through a series of mutations—but entirely functional.

Corristowolf 8y ago

Can anyone explain the benefit of glom instead of pandas dataframes? For data manipulation pandas is superior in everyway to any library out there. The only thing i think glom would be useful is json parsing as was shown. Sorry if I sound a bit too harsh. I'm looking at it from a data science pov.

wbolster 8y ago

in a similar spirit, i wrote "sanest", sane nested objects, tailored specifically for json fornats: https://sanest.readthedocs.io/ https://sanest.readthedocs.io/ it does not have the exact same feature set though. my focus was mostly on both reading and modifying nested structures in a type safe way.

heavenlyblue 8y ago

I think this project defines the first step at transitioning pip to a js-like repository of single-function modules. Hurray for kool-aid.

pdobsan 8y ago

There is already a well established Gnome project with the same name: http://www.glom.org http://www.glom.org It is a GTK+ front-end to PostgreSQL, similar to Microsoft Access.