8 ms·
Step Away from Stack Overflow
- eska 5y agoVeteran advice: don’t try to look smart. Try to solve the problem.
- MadsRC 5y agoThis guy speaks the truth…
- danesparza 5y agoThis is also the best way to navigate any meeting.
- commandlinefan 5y ago> don’t try to look smart. Try to solve the problem. Then see if there's a better way to do it that looks smart.
- yannoninator 5y ago
- unbanned 5y agogroupby
- jve 5y agoI mean, looking at it and knowing python is strong at lists and dictionaries… there sure is a better way which I would search for. So, pythonists, what’s the elegant answer?
- Denvercoder9 5y agoUsing `itertools.groupby` will probably give the cleanest solution. Something like this (untested): groups = itertools.groupby(sorted(mylist, key=lambda x: x['thing']), key=lambda x: x['thing']) newlist = [{**group[0], 'count': sum(item['count'] for item in group)} for group in groups] I'm not overly fond of it, having to sort the list for `groupby` is unpleasant and extracting values from dictonaries is verbose. If this was an array of tuples it could be made much more concise, but of course that doesn't allow storing extra information for each thing, which this solution does.
- deleted 5y ago[deleted]
- eesmith 5y agoDon't know about elegant. The following doesn't have the O(N^2) performance: def merge_counts(mylist): thing_lookup = {} # map thing -> item newlist = [] for item in mylist: thing = item["thing"] if thing in thing_lookup: # seen before; update count. thing_lookup[thing]["count"] += item["count"] else: # first time I've seen this thing newlist.append(item) thing_lookup[thing] = item return newlist It, like the original, mutates the original dictionary. I would prefer: # first time I've seen this thing thing = thing.copy() # don't mutate the original newlist.append(item)
- deleted 5y ago[deleted]
- marginalia_nu 5y agoI'd do GROUP BY in the query to SQLite.
- deepstack 5y agomany of these which can be extremely difficult in imperative, can be very easy in declarative.
- marginalia_nu 5y agoSQL was built to do this type of operation. It's generally a better pattern to fetch the data you need, rather than fetch all the data and then to transform it into what you need in the code.
- deepstack 5y agoand Prologo will be even better and cleaner syntax.
- alexchantavy 5y agoI'm actually working on a similar problem right now so I appreciate your comments. What if the data is already in the Python process so I don't have the ability of doing a `group by` at the database level? Should I perform this functionality in the Python code itself, or should I write to another database table and do the `group by` there? Edit: upon further reflection I think my application is at the tipping point where it needs another database table.
- marginalia_nu 5y agoDepending on your database, perhaps a materialized view is in order?
- detaro 5y agoAgreed with marginalia_ru, if you can get the DB to do it, that's probably the place. if not, I'd go with d = collections.defaultdict(int) for item in mylist: d[item["thing"]] += item["count"] newlist = [{"thing": key, "count": value} for key, value in d.items()] (I'm assuming the order doesn't matter, and the specific output format is needed. E.g. if the next step then iterates over that list and unpacks it, obviously get rid of the last line and use the dict) There is the collections.Counter type, but I think there's not really a beneficial way of using that here. Generally, the "there is one elegant solution" aspect of Python is widely overstated :D
- YurgenJurgensen 5y agoI think this defaultdict solution is the best, or at least most pythonic. I really thought if this was possible to do in a single dictionary comprehension, with some kind of d = {k: sum(v) for k, v in <something>}, but whatever goes in <something> ends up being super ugly.
- runekaagaard 5y agoWhat you do mean ugly :) [{"thing": k, "count": v} for k, v in [[c, c.update({x["thing"]: x["count"]})] for c in [Counter()] for x in response][0][0].items()]
- Denvercoder9 5y agoNote that this approach doesn't work as nicely if you need to store additional information alongside the name and count.
- qsort 5y agoIf you need to store other info, it doesn't make sense to aggregate. Think of it in SQL: SELECT a, b, SUM(c) FROM table GROUP BY a doesn't even compile. You either project away `b` or group over both `a` and `b`.
- 5y ago
- mcv 5y agoI'm not a pythonista, but I'd turn it into a dictionary where the value of `thing` becomes the key, and in the value we sum all those counts. Then turn it back into an array again. That's O(n). In javascript: const mylist = [ {'thing': 'A', 'count': 4}, {'thing': 'B', 'count': 2}, {'thing': 'A', 'count': 6}]; const dict = mylist.reduce(({thing, count}, dict) => ({ [thing] : dict[thing] ? dict[thing]+count : count }, {}); const newList = Object.keys(dict).map(thing => { thing, count: dict[thing] }); (Not tested or anything, so no guarantee that this works right off the bat, but something like this should work.)
- thr0waway164021 5y agoSame algorithm in Python from functools import reduce result = [ {'thing': k, 'count': v} for k, v in reduce( lambda result, item: { **result, item['thing']: item['count'] + result.get(item['thing'], 0) }, mylist, {} ).items() ]
- DangitBobby 5y agoSomething a bit more beginner friendly: agg = {} for item in mylist: if item['thing'] in agg: agg[item['thing']] += item['count'] else: agg[item['thing']] = item['count'] result_list = [] for key, value in agg.items(): result_list.append(dict(thing=key, count=value)) In beginner land, reduce makes your eyes glaze over and dict.get(key, default) is understandable in principle but still confusing in practice.
- Izkata 5y agoAfter almost a decade doing python, I'd probably use a plain loop like this but with ".get(key, default)"
- mcv 5y agoWith the recent changes in Javascript, I've developed an aversion to old fashioned for-loops. I prefer to .map.reduce.filter and if absolutely necessary, .forEach my way through the problem.
- pjc50 5y agoFrom a C#'er: do you people not have LINQ?
- charlieyu1 5y agoI think that is something, maybe collections.Counter, that provides this functionality out of the box. I can’t really remember it, so I’d probably go with defaultdict
- marginalia_nu 5y agoSome 15 years ago when I was an undergraduate, I took a class in Linear Algebra. A month or whatever in we got to the chapter of transformation matrices: Rotation and translation of 3D points. I immediately saw the utility: I could make 3D graphics with these tools! I pretty much ran home and started building a first person spaceship game, pouring over the book trying to work out how to make all these bits fit together. The graphics were rudimentary, I used SDL and most stars were just white squares (SDL_FillRect), but eventually I got to the point where I could travel in "space" and have the stars zoom by. Turns out, software matrix multiplication is not the best way of doing 3D graphics, but it's a great way of learning how the mathematics of it works. In the mean time of doing all this I had gotten linear algebra. LA was one of the first "hard" classes we took, and a lot of people barely passed. I got top grades in that class. The moral of the story is that if you want to learn how something works, then this (what OP is doing), is correct. This is a great way to learn. There probably is a faster or better way of doing it, but in terms of learning-efficiency, this is great.
- gfxgirl 5y ago> Turns out, software matrix multiplication is not the best way of doing 3D graphics ok, what is? all the 3d engines I've ever worked with do tons of software matrix multiplication. You generally need to know where things are for collision checking and other things so doing it all on the gpu is not an option
- marginalia_nu 5y ago> You generally need to know where things are for collision checking and other things so doing it all on the gpu is not an option Sure. You can do some of it in software. That's fine. Point is I didn't use the GPU at all, and did all the rendering in software. No matter how you rotate or translate it, that's not a particularly efficient way of doing it.
- marcosdumay 5y agoAnyway, the GPU usually does it with matrix multiplications.
- Pinus 5y agoIt's probably possible to do this in a one-liner using half of the stuff in itertools, but just changing the output data structure to a dict (at least temporarily; one can always make a list from it at the end) would yield code that is short, understandable and gets the job done without linear searches. Essentially, I'm agreeing with eska's comment, with the addition: "...but consider your data structures!" groupby looks neat, but it requires sorted input, so it's very easy to convert an O(n) problem to an elegant but O(n log n) program. EDIT: one could also do it with a one-liner in pandas, which means that as long as the input is less than ten million lines or so, the "import pandas as pd" statement is going to take longer than the actual program...
- rstuart4133 5y agoPerhaps not one line, maybe not perfect Python but definitely O(n): groupby_dict = collections.defaultdict(int) for l in mylist: groupby_dict[l['thing']] += l['count'] newlist = [] for thing, count in groupby_dict.items(): newlist.append(dict(thing=thing, count=count)) Knowing how do that does require the deep knowledge of data structures I'd expect every professional programmer to have, and doing a quick scan of the languages standard library (maybe an hours work) to see what it offers. After that you're set for solving not only this problem, but almost guaranteed to solve all problems you are likely to hit pretty optimally in Python. I have no idea where Stack Overflow fits into the picture. If you go a step further and do the Python standard tutorial found in the docs, you would discover how a finger weary experienced Python programmer might write those last three lines: newlist = [dict(thing=thing, count=count) for thing, count in groupby_dict.items()] Comprehensions are sweet Python syntactic sugar, but unlike mapping your existing encyclopaedic knowledge of data structures onto the languages standard library they aren't necessary for a casual user of the language. Indeed, some Python'istas will tell you resisting such delights make for clearer code. But trusting a quick Stack Overflow to tell you the optimal way to use a languages data structures - you must be kidding me.
- qsort 5y agoDoesn't detract from any of your points, just as a FYI: Python has a counter data structure in the standard library. The "pythonic" way would be something like this: groupby_dict = Counter() for r in response: groupby_dict.update({r['thing']: r['count']}) Counter is a subclass of dict, no further conversion needed.
- Rainymood 5y agoThis solution is really bad, read it in as a dataframe and use a groupby...
- deleted 5y ago[deleted]
- halfdan 5y agoThat's even worse. You don't need to throw in a massive library like pandas, especially not when OP is build a basic API. Sure, if you're in a data science-y project where pandas is already a dependency, go nuts.
- rbanffy 5y agoIt really depends on the size of the list you want to process. If it's 10 items, pandas is overkill (and probably slower). If it's a million items, pandas is a great solution. I have a nagging feeling there is an easier way to do this, but my quick and dirty solution was def merge_list1(l): other_dict = defaultdict(lambda: 0) for t, c in ((i['thing'], i['count']) for i in l): other_dict[t] += c return ({'thing': k, 'count': other_dict[k]} for k in other_dict) which is still readable, but probably far from optimal.
- Izkata 5y ago> If it's a million items, pandas is a great solution. Possibly not even then, it depends on how much you're doing and I feel like the topic at hand might be around that tipping point. We have some rather slow code that, profiling it, turned out to spend something like 60-70% of its time just converting between python types and native types when moving data in and out of the dataframe.
- rbanffy 5y agoTrue. If there are millions of different “things” conversion times will end up dominating. If they are just a handful, then the libraries will be able to do a lot more work with parallel operations and converting the output will be very quick
- felix_n 5y agoNot to be mean, but TBH I rolled my eyes a bit because awkwardly dancing with the interpreter until something appears to work is kind of exactly like throwing spaghetti at the wall. And as others have pointed out, a quick jaunt to Stack Overflow might have revealed to OP that leaning on the DB is what any seasoned engineer would do, which is probably a better engineering lesson than how to implement a reduce by hand. AFAIK Stack Overflow's mission statement isn't "Be Rip-Offable for Devs Everywhere." Obviously I know about "copy-paste engineers" but that's a character trait, not the result of their tools. I've always treated SO more like a knowledgeable coworker to help when I get stuck.
- masklinn 5y ago> I've always treated SO more like a knowledgeable coworker to help when I get stuck. Not too knowledgeable though, in my experience if you are the knowledgeable colleague in a domain, SO will not help you unless you're in one of the domains with a legendary SO contributor (e.g. C# because of Jon Skeet if you manage to get their attention), and unlike a knowledgeable colleague it won't assist you in looking for a clarification or solution either.
- tonyedgecombe 5y agoEven in C# there are plenty of dodgy answers and stuff that is plain wrong yet accepted and upvoted.
- usrbinbash 5y agoThe difference: SO provides solutions to problems. Experimenting, playing around and "throwing spaghetti at the wall" can lead to an understanding of the problem. SO will tell me to use DISTINCT, and that is correct. But why? Why is this the better? Is it really? Why is the home-brewed reduce slower? How could it be improved? Oh hey, but if I did this to the data upstream I could prevent non.distinct records from being in the DB in the first place, etc. etc. Software Engineering is, among many other things, a craft. Just like carpenting or stone carving or smithing, it must be honed, and mastery can only be achieved if one plays, trys and experiments. ---- Of course, reading about the solutions to a problem can also help in its understanding. For me personally, if I do not understand a problem, I like to do both: Experiment, and read what the "canonical" way to solve it is.
- VimER 5y ago
- dqpb 5y agoThis is great. It’s rare to find a well organized brain dump of a beginner struggling with a simple problem. There is very little record in general of the intermediate states people pass through on their trajectory from novice to expert. I suspect one could build an entirely new framework for education around such data.
- oli5679 5y agoHere is a 1-liner using pandas. I would be interested if it's possible to do it in one-line using something from itertools. import pandas as pd [{'thing':x, 'count':y} for x,y in pd.DataFrame(mylist).groupby('thing' ['count'].sum().to_dict().items()]
- psyklic 5y agoHere is a one-liner standard library version: from itertools import groupby keyfunc = lambda x: x['thing'] [{'thing': name, 'count': sum(t['count'] for t in things)} for name,things in groupby(sorted(mylist, key=keyfunc), keyfunc)]
- oli5679 5y agoThanks this is great
- psyklic 5y agoHere is a more concise Pandas version (though Pandas may be overkill for this task): import pandas as pd pd.DataFrame(mylist).groupby('thing', as_index=False).sum().to_dict('records')
- oli5679 5y agoThanks this is great
- comeonseriously 5y ago>Yup. This looks awful — and is error prone and slow and probably wins some kind of award for being unPythonic — but it works. And I wrote it and I can read it. We can improve tomorrow. That's the key takeaway. Write it once to get it working and understand how to solve the problem, then write it better. "Hence plan to throw one away; you will, anyhow." - Fred Brooks
- hooby 5y agoI kinda disagree with the implied notion that there's only these two options mentioned in the article: A) blindly copy paste whatever code shows up on stack overflow without even trying to understand it - or - B) don't ever look up stack overflow at all. I'll keep doing what I've always done: Look things up on stack overflow (and other sites), compare different answers and check the discussions as well, figure out WHY people are doing what they do - and then do my own experiments based on that. For me personally, this leads to faster learning results than experimenting completely on my own, without any guidance.
- dfxm12 5y agoYeah, this quote: trying to find some kind of shortcut to just get my code working by copy pasta-ing random bits of spagetti and crossing my fingers. Stop. There’s a better way. is about as honest as the Juice Loosener pitch from the Simpsons [0]. It's impossible to be able to just copy and paste something you find online and have it work, especially while ignoring all context and learning nothing. 0 - https://www.youtube.com/watch?v=kDiSWKaS3N0 https://www.youtube.com/watch?v=kDiSWKaS3N0
- gopher_space 5y agoThe article seems geared towards someone working from a book or guide at their leisure. Building things up from first principles is a great way to learn but I personally never have time for the approach on the job.
- DemocracyFTW 5y agoI think what the article does get right is a fundamental technique to solve a problem when you're stuck: reduce to the essentials, get your terms / naming things right, write a teeny self-contained thing (I call it 'building a model') to try it out. When the solution is understood and has proven to work, integrate. Especially when you're hard pressed for time, this may prove faster overall than the approach where you fumble with many screws.
- lostcolony 5y agoYou don't even need to compare answers, per se. I use SO as a way to know what documentation to check. That is, if I want to X, the official documentation is oftentimes extremely difficult to parse through; to do X you have to instantiate a Foo, with an instance of Bar, etc etc etc, and all you have are API level docs. They're usually written the reverse of what I need to solve the problem (given function X -> understand what it does, rather than given what I want to do -> use function X; this gets even worse when you get higher level concepts, where what I want to do -> implies doing 4 different things in this API -> 4 different functions/data types that need to be strung together). SO and its ilk give me an example that I can start to cross reference with the API level docs, to figure out how to pull it all together, and also where to look to find the "gotchas", tweaks, and options.
- aliswe 5y agoi thought this was going to be yet another SO bashing post - I'm disappointed.
- commandlinefan 5y agoSometimes I feel like I was fortunate in a way to have learned programming before the internet - the only way to accomplish something back then was to actually take time to understand the fundamentals of what you were working with.
- da39a3ee 5y agoEee, back when I were a lad… (This has to be said in a Yorkshire accent)
- goto11 5y agoThis approach is great if you are practicing writing algorithms on your own time. If I was writing software professionally, I would think "this sounds like 'group by' in SQL, wonder what the equivalent is in Python", and then I would google for the the most common/idiomatic approach.
- yawnxyz 5y agoSometimes I just want to know what the syntax is... sometimes documentation can be VERY dense, but someone on SO has posed a question or answer that can show me how the syntax works for that one specific thing I'm looking for. Particularly useful for frameworks like Svelte or React where I'm like... I have no idea how that's written.
- oblib 5y ago>>After some time with no progress in figuring out what sticks, it’s time to step away from that approach. That's exactly what motivated me to make a habit of going to SO first. But I don't just go there looking to copy and paste. I go there to learn and always review all the solutions given and the history and comments on them. It was a struggle though because I was ornery and had made a habit of trying to figure everything out on my own. I generally know the logic required but struggled with the syntax to implement it. Since I made that a habit I have learned so much from contributors there and made huge gains in productivity. They don't really give a good or direct reason to "step away" from SO here though. It's really more about crafting code you can read and understand and that's good advice.
- reactspa 5y agoCan someone please clarify why Python decided to go with unconventional names and symbols? E.g., every other language calls them "arrays", and uses `[ ]`, but Python calls (roughly) them "dicts" and uses `{ }`. Every other language calls them "objects", and uses `{ }`, but Python calls (roughly) them "lists" and uses `[ ]`.
- larrik 5y agoI'm not sure what you mean. An Array is a contiguous section of memory with like items. Those like items _could be_ object pointers. The closest thing in Python to this is a tuple. A List is a collection of things with an order, often a linked list or doubly linked list, but could actually be an array as well. Sometimes "array" and "list" are interchangeable. A Dictionary (aka HashMap) is a key/value store. They are denoted in Python with a bounding {} (similar to like JSON objects). Unlike JS or JSON, the keys can be anything hashable, not just strings or ints. I sort of get the impression that "every other language" means javascript, where we sort of abuse the syntax and treat objects as hashmaps (they aren't really) or arrays (also aren't really). Yet, Python maps quite well against JSON objects and lists. Terminology definitely can be different from language to language, but I find Python to be pretty close to the C class of languages for that.
- fellellor 5y agoThe most frequent reason I turn to stack overflow, is for solutions to things I just don’t care about right now. Like, for example, how do I get bootstrap to work in webpacker for rail, etc. I don’t really care about this, I’d rather just be writing ruby code. And even in these cases, SO isn’t that helpful half the time because either the answer is 3 years old and outdated or there used to be a relevant answer but it has been deleted for some violation.
- inkeddeveloper 5y agoUse stack overflow like you use Wikipedia. It’s good to check out but you should really be using it for the references. I find the comments tend to point to documentation quite well.
- mickotron 5y agoI always look for a very fundamental way of doing x. But SO questions and replies almost always are about a very very specific use case. It takes a lot of careful searching to find the one line of code I needed for my use case.
- deleted 5y ago[deleted]
- Kalanos 5y ago`pd.DataFrame.from_records(mylist).groupby(['thing']).sum().to_records()` you can learn a thing or two. people are genuinely here to teach. let's build a new SO because it's owned by china now.
- grandpoobah 5y agoThe specific example in this article reminds me why I like C# so much. var newlist = mylist.GroupBy(i => i.thing).Select(g => new { thing = g.Key, count = g.Sum(i => i.count), }).ToList();
- runekaagaard 5y agoThis does the trick in a (for me) not confusing way: from collections import Counter response = [ { "thing": "A", "count": 4, }, { "thing": "B", "count": 2, }, { "thing": "A", "count": 6, }, ] counter = Counter() for x in response: counter[x["thing"]] += x["count"] newlist = [{"thing": k, "count": v} for k, v in counter.items()] print(newlist)