5 ms·

Python Expertise Level – Self-Assessment

59 points by saeedesmaili 3y ago

__mharrison__ 3y ago
Very few experts out there. Also many folks using Python daily professionally that aren't at the beginner level.
Waterluvian 3y ago
“Can use advanced Python libraries like numpy, pandas, matplotlib.” Interestingly in my 15 years of Python I’ve used numpy sparingly for image processing and haven’t used the others ever. I appreciate this isn’t meant to be an exhaustive checklist but it reminds me how two advanced coders could live in two entirely different drawers of the tool chest.
- joe5150 3y ago
  I'm basically allergic to pandas. My old job used a shared "utility" library written by someone who imported pandas in order to...write Markdown into log files.
- leetrout 3y ago
  Same - I have been using Python since 1998 and I failed the LinkedIn skills quiz on Python because it was over 50% numpy questions and things about matrix math. That's not "python" for me.
kstrauser 3y ago
Eh. Advanced.9: "Can use Python's built-in functions like map(), filter(), reduce()." ...and knows why you should probably use comprehensions instead. Experts.4: "Can use Python's C API to extend Python with C/C++ code." I've written Python professionally for 20+ years. I've extended Python with C for funsies, but have never, not once, needed to do it at work. I'm glad other people have used C (and now Rust) to speed up the Python modules I want to use. I'm equally glad I've been able to use those modules, not have to enhance them with C. Experts.7: "Understands and uses Python's garbage collection system." 99% of what you need to know is "don't write circular references". Experts.10: "Have a good understanding of Python's internals, such as bytecode, the Python interpreter's execution model, and how Python's data types are implemented at the C level." See Experts.4. I've had a nice career writing code in Python, but haven't had to hack on the CPython codebase. I bet there are lots of people who are experts in C who've never written a line in the GCC or Clang codebases.
- diarrhea 3y ago
  Circular references are not an issue, the GC is more than just reference counting.
- selcuka 3y ago
  > ...and knows why you should probably used comprehensions instead. This. The author does not seem to know that (1) reduce is not a built-in function anymore, and (2) map and filter were also planned to be removed from Python 3 [1]: > About 12 years ago, Python aquired lambda, reduce(), filter() and map(), courtesy of (I believe) a Lisp hacker who missed them and submitted working patches. But, despite of the PR value, I think these features should be cut from Python 3000. > Update: lambda, filter and map will stay (the latter two with small changes, returning iterators instead of lists). Only reduce will be removed from the 3.0 standard library. You can import it from functools. [1] https://www.artima.com/weblogs/viewpost.jsp?thread=98196 https://www.artima.com/weblogs/viewpost.jsp?thread=98196
  - deepsquirrelnet 3y ago
    I have only been writing python since Guido wrote that comment, but I have to say I don’t agree with him. I find the map/filter syntax to be very readable and intuitive. List comprehension is ok but to me becomes nearly incomprehensible (see what I did there?) where nesting is involved. It’s easy to chain map and filter statements without losing readability. I think I might be in the minority though.
    - leetrout 3y ago
      It's a "when in Rome..." situation. Theres nothing wrong with using them but what you are going to find across the community are comprehensions. They concisely bring the data container and the iteration together with no fuss. If you need to map and reduce and filter and want to build a chain and/or have reusable predicate functions then go for it. I always encourage my teams to reach for the ecosystem usage before doing things because another language does it different (map/filter/reduce being the best option in JS 90% of the time). But maybe you work at a functional shop and everyone Gets It(tm) and then it is the norm.
drbig 3y ago
Cool and overall sane, so lemme just pitch in from my perch on the critical side: *Advanced* > Can use advanced Python libraries like numpy, pandas, matplotlib. You're a data scientist and this is not "Python". > Can use regular expressions for pattern matching in strings. Not Python really again. > Understands and uses Python's memory management and optimization techniques. Wait, like thinking about how a `list()` looks like from the (false) C-level perspective? Can you fix my CPU transistors while you're at it? (yes, /s) *Experts* > Can use Python's C API to extend Python with C/C++ code. Do you mean whatever FFI Python has? Otherwise this is like asking your mechanic to also make it a plane, and a submarine. After all these are all just vehicles. > Understands and uses Python's garbage collection system. Ref counting? Wait what? > Have a good understanding of Python's internals, such as bytecode, the Python interpreter's execution model, and how Python's data types are implemented at the C level. Submarine :D My perch also lets me see here the aspects I usually don't care about. Thank you!
- iamwpj 3y ago
  I feel similar. No matter how good at Python I get, I doubt I'll ever write an C interface, I just don't need that. I've used pandas and some of these randomly, but TBH I'd rather write a dataclass with static typing rather than re-construct data in pandas. Does that bump me down to a lesser skilled programmer? I guess it's easy to pick on any rubric, but I found the implication of this one be unhelpful. I like the training materials that focus on continual development instead -- I don't recall any off the top of my head, but they provide a contextual learning path, like if you've mastered list comprehension now take a look at decorators.
- dragonwriter 3y ago
  > > Can use advanced Python libraries like numpy, pandas, matplotlib. > You're a data scientist and this is not "Python". Pandas and matplotlib, are a weaker case, maybe, but numpy is fairly broadly used (math being a core part of computing) library outside of just data science. E.g., this tutorial on building a roguelike in Python has two direct dependencies, tcod (a library specifically to support roguelikes) and numpy: https://rogueliketutorials.com/tutorials/tcod/v2/part-0/ https://rogueliketutorials.com/tutorials/tcod/v2/part-0/
swatcoder 3y ago
Self-assessment is so messy that lists like this are just misleading. Among other issues, many of the criteria for "expert" sound more like the criteria for "library developer". Many conventionally-expert application developers and data scientists would smartly not use those features and may self-assess themselves out of the category for lack of active practice. Meanwhile, too many conventionally-junior developers would sophomorically insist that they can and should use them and self-assess themselves as experts.
NoboruWataya 3y ago
> Understands and uses Python's memory management and optimization techniques. What are these?
- 0cf8612b2e1e 3y ago
  Maybe generators? I could kind of see an argument for some tricks like using tuples or slots to save memory. Anything in functools? Those are all under the heading of conserving memory, nothing to do with explicit memory management. If I needed that tight of control, I would not be writing Python.
- JonChesterfield 3y ago
  Rewriting in C++ while regretting prior and current life choices, primarily. Probably a module at a time.
- __loam 3y ago
  Write your program in a language other than Python.
- iamcreasy 3y ago
  One example could be small integer caching [1,2]. [1] https://realpython.com/lessons/small-integer-caching/#description https://realpython.com/lessons/small-integer-caching/#descri... [2] https://docs.python.org/3/c-api/long.html#c.PyLong_FromLong https://docs.python.org/3/c-api/long.html#c.PyLong_FromLong
m463 3y ago
Given he's selling the book at the top for the weird theoretical-but-maybe-not-reality-based "interview process", I think some of the things lack practical feet-on-the-ground expertise levels. Personally I think a few things actual experts should know are: - python packages: pip vs package manager vs conda - python 2 vs 3 differences and porting - os peculiarities, such as windows vs linux etc...
- 0cf8612b2e1e 3y ago
  Python 2 seems like a weird one at this point. I am sure there are plenty of code bases in the wild that are still running Python 2, but what version? You cannot assume 2.7 and there are probably a lot of gotchas I no longer recall from say 2.3 to 2.7. An expert will relearn the differences if/when it needs to be done.
3c6bYDXLMj 3y ago
The author does not know enough about Python, or whats required of “expert-level” Python developer, to be making this content. Typical blind leasing the blind in service of everyone wanting to become a “content creator”.
- 3c6bYDXLMj 3y ago
  Also it need be said that one’s initial shock at the state of numpy/pandas will probably increase with their ‘general’ programming proficiency. The degree to which the ‘Python data science’ scene has their own separate set of conventions, and the degree to which there’s a lot of weird meta-programming going on in these packages, is quite jarring.
john-radio 3y ago
There are a lot of kind of questionable/obscure skills in tiers 2-4, but I think that's fine actually. It's not a software development skill assessment; it purports to be only about "python expertise." There are definitely people out there with strong knowledge of every topic in item 3 and I'd definitely be persuadable that such a person is a "Python expert." I just worry, when I see a list of things tagged "expert" like this, that people will take it as a learning path, which... Could be good, but there are much straighter paths to learning to be effective as a developer.
otteromkram 3y ago
_1. Can use advanced Python libraries like numpy, pandas, matplotlib._ I would disagree that someone's knowledge of third-party libraries helps determine how good they are at Python. Why not ask about the `array` package? What about using the functional builtin packages (eg - itertools, functools)? Memory management and optimizations are negligible, but I'd lump hose in with expert since I rarely see one of the
otteromkram 3y ago
1. Can use advanced Python libraries like numpy, pandas, matplotlib. I don't think this is a very good assessment point of someone's python proficiency. Maybe substitute those with itertools, functools, and array builtin packages.
63 3y ago
> Understands and uses metaclasses In most circumstances, I feel like using metaclasses is a red flag. How many people are there on earth who understand and regularly use meta classes when they're appropriate? I always pegged metaclasses as "you should know this exists in case you ever get unlucky enough to need it but otherwise stay the hell away from it"
- bgribble 3y ago
  The one python pattern I used to use metaclasses for (registry types that keep an index of all their subclasses, very useful for DSL type tasks) is easier with __init_subclass__ and is somewhat more readable. So I haven't used metaclass in a minute. The worst thing about metaclass is that 95% of python programmers look at it and just go "whuzzat?" and require a seminar to catch a clue. It's actively hostile to whoever maintains the code after you.
  - kstrauser 3y ago
    I'm with you completely about __init_subclass__. It covers almost all the uses where I'd otherwise use metaclasses. Junior engineers so that and say "oh, clever!", rather than running off screaming into the night.
  - strbean 3y ago
    I too love attributes that can only be known at runtime and are therefore fundamentally un-type-hintable.
  - nhumrich 3y ago
    Agree, but init_subclass is new to python 3. It almost feels like OPs advice is based on old python and not modern python.
- yen223 3y ago
  I have used metaclasses one time in my career, to simplify the way we generated a schema, in a framework where things are defined using classes. I agree with you in that it's nice to know that metaclasses exist, and that you can potentially reach for them in a pinch. I also think you can go really far in Python development without ever needing to know about metaclasses.
__loam 3y ago
> Understands and uses Python's concurrency and parallelism features. And understands why Python sucks for this use case.
_8j50 3y ago
I needed this, I write python all the time but I know I am not a pro and the language features always change. I just never needed asyncio for example. Maybe it's my C experience in the past speaking but for most use cases I prefer to use simple features anyone who is invested in python can reas easily so that others in the future can maintain it. So, I don't go out of my way to look for special and advanced ways to speed up python or make it more efficient. If performance was that important to me, I would use something like Go, which at least as of 2yrs ago was somehow even simpler than Python and faster.
smitty1e 3y ago
I would consider adding, in Advanced: "Compiles the cpython interpreter from source". Toolchain savvy would seem a precursor to doing C/C++ extensions down in the Expert section.
leetrout 3y ago
I want to work with folks who know the difference between a module and a package. That follow pep8 for most things. People that know what a namespace package is and how __init__.py works. I care more about operating the toolchain efficiently and writing understandable code and avoiding "advanced" patterns until forced to use them for business impacting constraints (performance). Use simple data classes or vanilla classes without a lot of methods. Functions that take objects in and dont mutate them. Readable and understandable by folks new to the org and the team. And juniors. People who care about the impact of maintaining code and put the effort into effective documentation and code comments.
waitforthecream 3y ago
From the comments section, i understand that the author did not do a good job. Can someone please share other resources, that contains an exhaustive list of things to learn in python.
laurent_du 3y ago
I think the goal of the author was to explain how he rates candidates for the positions he is offering, rather than answering properly the question "what makes a beginner / advanced / expert Python developer?". This is made evident by the fact that most contributors to this thread strongly disagree with the author's choice. I personally find it very weird that sets are listed as an advanced feature of the language. And understanding Python's dynamic typing system is somehow an expert-level achievement?
fortyseven 3y ago
A helpful addition to this list would be links to good resource or two, each, where the reader can learn more.

59 points by saeedesmaili 3y ago

Very few experts out there. Also many folks using Python daily professionally that aren't at the beginner level.

“Can use advanced Python libraries like numpy, pandas, matplotlib.” Interestingly in my 15 years of Python I’ve used numpy sparingly for image processing and haven’t used the others ever. I appreciate this isn’t meant to be an exhaustive checklist but it reminds me how two advanced coders could live in two entirely different drawers of the tool chest.

joe5150 3y ago
I'm basically allergic to pandas. My old job used a shared "utility" library written by someone who imported pandas in order to...write Markdown into log files.
leetrout 3y ago
Same - I have been using Python since 1998 and I failed the LinkedIn skills quiz on Python because it was over 50% numpy questions and things about matrix math. That's not "python" for me.

kstrauser 3y ago

Eh. Advanced.9: "Can use Python's built-in functions like map(), filter(), reduce()." ...and knows why you should probably use comprehensions instead. Experts.4: "Can use Python's C API to extend Python with C/C++ code." I've written Python professionally for 20+ years. I've extended Python with C for funsies, but have never, not once, needed to do it at work. I'm glad other people have used C (and now Rust) to speed up the Python modules I want to use. I'm equally glad I've been able to use those modules, not have to enhance them with C. Experts.7: "Understands and uses Python's garbage collection system." 99% of what you need to know is "don't write circular references". Experts.10: "Have a good understanding of Python's internals, such as bytecode, the Python interpreter's execution model, and how Python's data types are implemented at the C level." See Experts.4. I've had a nice career writing code in Python, but haven't had to hack on the CPython codebase. I bet there are lots of people who are experts in C who've never written a line in the GCC or Clang codebases.

diarrhea 3y ago
Circular references are not an issue, the GC is more than just reference counting.
selcuka 3y ago
> ...and knows why you should probably used comprehensions instead. This. The author does not seem to know that (1) reduce is not a built-in function anymore, and (2) map and filter were also planned to be removed from Python 3 [1]: > About 12 years ago, Python aquired lambda, reduce(), filter() and map(), courtesy of (I believe) a Lisp hacker who missed them and submitted working patches. But, despite of the PR value, I think these features should be cut from Python 3000. > Update: lambda, filter and map will stay (the latter two with small changes, returning iterators instead of lists). Only reduce will be removed from the 3.0 standard library. You can import it from functools. [1] https://www.artima.com/weblogs/viewpost.jsp?thread=98196 https://www.artima.com/weblogs/viewpost.jsp?thread=98196
- deepsquirrelnet 3y ago
  I have only been writing python since Guido wrote that comment, but I have to say I don’t agree with him. I find the map/filter syntax to be very readable and intuitive. List comprehension is ok but to me becomes nearly incomprehensible (see what I did there?) where nesting is involved. It’s easy to chain map and filter statements without losing readability. I think I might be in the minority though.
  - leetrout 3y ago
    It's a "when in Rome..." situation. Theres nothing wrong with using them but what you are going to find across the community are comprehensions. They concisely bring the data container and the iteration together with no fuss. If you need to map and reduce and filter and want to build a chain and/or have reusable predicate functions then go for it. I always encourage my teams to reach for the ecosystem usage before doing things because another language does it different (map/filter/reduce being the best option in JS 90% of the time). But maybe you work at a functional shop and everyone Gets It(tm) and then it is the norm.

drbig 3y ago

Cool and overall sane, so lemme just pitch in from my perch on the critical side: *Advanced* > Can use advanced Python libraries like numpy, pandas, matplotlib. You're a data scientist and this is not "Python". > Can use regular expressions for pattern matching in strings. Not Python really again. > Understands and uses Python's memory management and optimization techniques. Wait, like thinking about how a `list()` looks like from the (false) C-level perspective? Can you fix my CPU transistors while you're at it? (yes, /s) *Experts* > Can use Python's C API to extend Python with C/C++ code. Do you mean whatever FFI Python has? Otherwise this is like asking your mechanic to also make it a plane, and a submarine. After all these are all just vehicles. > Understands and uses Python's garbage collection system. Ref counting? Wait what? > Have a good understanding of Python's internals, such as bytecode, the Python interpreter's execution model, and how Python's data types are implemented at the C level. Submarine :D My perch also lets me see here the aspects I usually don't care about. Thank you!

iamwpj 3y ago
I feel similar. No matter how good at Python I get, I doubt I'll ever write an C interface, I just don't need that. I've used pandas and some of these randomly, but TBH I'd rather write a dataclass with static typing rather than re-construct data in pandas. Does that bump me down to a lesser skilled programmer? I guess it's easy to pick on any rubric, but I found the implication of this one be unhelpful. I like the training materials that focus on continual development instead -- I don't recall any off the top of my head, but they provide a contextual learning path, like if you've mastered list comprehension now take a look at decorators.
dragonwriter 3y ago
> > Can use advanced Python libraries like numpy, pandas, matplotlib. > You're a data scientist and this is not "Python". Pandas and matplotlib, are a weaker case, maybe, but numpy is fairly broadly used (math being a core part of computing) library outside of just data science. E.g., this tutorial on building a roguelike in Python has two direct dependencies, tcod (a library specifically to support roguelikes) and numpy: https://rogueliketutorials.com/tutorials/tcod/v2/part-0/ https://rogueliketutorials.com/tutorials/tcod/v2/part-0/

swatcoder 3y ago

Self-assessment is so messy that lists like this are just misleading. Among other issues, many of the criteria for "expert" sound more like the criteria for "library developer". Many conventionally-expert application developers and data scientists would smartly not use those features and may self-assess themselves out of the category for lack of active practice. Meanwhile, too many conventionally-junior developers would sophomorically insist that they can and should use them and self-assess themselves as experts.

NoboruWataya 3y ago

> Understands and uses Python's memory management and optimization techniques. What are these?

0cf8612b2e1e 3y ago
Maybe generators? I could kind of see an argument for some tricks like using tuples or slots to save memory. Anything in functools? Those are all under the heading of conserving memory, nothing to do with explicit memory management. If I needed that tight of control, I would not be writing Python.
JonChesterfield 3y ago
Rewriting in C++ while regretting prior and current life choices, primarily. Probably a module at a time.
__loam 3y ago
Write your program in a language other than Python.
iamcreasy 3y ago
One example could be small integer caching [1,2]. [1] https://realpython.com/lessons/small-integer-caching/#description https://realpython.com/lessons/small-integer-caching/#descri... [2] https://docs.python.org/3/c-api/long.html#c.PyLong_FromLong https://docs.python.org/3/c-api/long.html#c.PyLong_FromLong

m463 3y ago

Given he's selling the book at the top for the weird theoretical-but-maybe-not-reality-based "interview process", I think some of the things lack practical feet-on-the-ground expertise levels. Personally I think a few things actual experts should know are: - python packages: pip vs package manager vs conda - python 2 vs 3 differences and porting - os peculiarities, such as windows vs linux etc...

0cf8612b2e1e 3y ago
Python 2 seems like a weird one at this point. I am sure there are plenty of code bases in the wild that are still running Python 2, but what version? You cannot assume 2.7 and there are probably a lot of gotchas I no longer recall from say 2.3 to 2.7. An expert will relearn the differences if/when it needs to be done.

3c6bYDXLMj 3y ago

The author does not know enough about Python, or whats required of “expert-level” Python developer, to be making this content. Typical blind leasing the blind in service of everyone wanting to become a “content creator”.

3c6bYDXLMj 3y ago
Also it need be said that one’s initial shock at the state of numpy/pandas will probably increase with their ‘general’ programming proficiency. The degree to which the ‘Python data science’ scene has their own separate set of conventions, and the degree to which there’s a lot of weird meta-programming going on in these packages, is quite jarring.

john-radio 3y ago

There are a lot of kind of questionable/obscure skills in tiers 2-4, but I think that's fine actually. It's not a software development skill assessment; it purports to be only about "python expertise." There are definitely people out there with strong knowledge of every topic in item 3 and I'd definitely be persuadable that such a person is a "Python expert." I just worry, when I see a list of things tagged "expert" like this, that people will take it as a learning path, which... Could be good, but there are much straighter paths to learning to be effective as a developer.

otteromkram 3y ago

_1. Can use advanced Python libraries like numpy, pandas, matplotlib._ I would disagree that someone's knowledge of third-party libraries helps determine how good they are at Python. Why not ask about the `array` package? What about using the functional builtin packages (eg - itertools, functools)? Memory management and optimizations are negligible, but I'd lump hose in with expert since I rarely see one of the

otteromkram 3y ago

1. Can use advanced Python libraries like numpy, pandas, matplotlib. I don't think this is a very good assessment point of someone's python proficiency. Maybe substitute those with itertools, functools, and array builtin packages.

63 3y ago

> Understands and uses metaclasses In most circumstances, I feel like using metaclasses is a red flag. How many people are there on earth who understand and regularly use meta classes when they're appropriate? I always pegged metaclasses as "you should know this exists in case you ever get unlucky enough to need it but otherwise stay the hell away from it"

bgribble 3y ago
The one python pattern I used to use metaclasses for (registry types that keep an index of all their subclasses, very useful for DSL type tasks) is easier with __init_subclass__ and is somewhat more readable. So I haven't used metaclass in a minute. The worst thing about metaclass is that 95% of python programmers look at it and just go "whuzzat?" and require a seminar to catch a clue. It's actively hostile to whoever maintains the code after you.
- kstrauser 3y ago
  I'm with you completely about __init_subclass__. It covers almost all the uses where I'd otherwise use metaclasses. Junior engineers so that and say "oh, clever!", rather than running off screaming into the night.
- strbean 3y ago
  I too love attributes that can only be known at runtime and are therefore fundamentally un-type-hintable.
- nhumrich 3y ago
  Agree, but init_subclass is new to python 3. It almost feels like OPs advice is based on old python and not modern python.
yen223 3y ago
I have used metaclasses one time in my career, to simplify the way we generated a schema, in a framework where things are defined using classes. I agree with you in that it's nice to know that metaclasses exist, and that you can potentially reach for them in a pinch. I also think you can go really far in Python development without ever needing to know about metaclasses.

__loam 3y ago

> Understands and uses Python's concurrency and parallelism features. And understands why Python sucks for this use case.

_8j50 3y ago

I needed this, I write python all the time but I know I am not a pro and the language features always change. I just never needed asyncio for example. Maybe it's my C experience in the past speaking but for most use cases I prefer to use simple features anyone who is invested in python can reas easily so that others in the future can maintain it. So, I don't go out of my way to look for special and advanced ways to speed up python or make it more efficient. If performance was that important to me, I would use something like Go, which at least as of 2yrs ago was somehow even simpler than Python and faster.

smitty1e 3y ago

I would consider adding, in Advanced: "Compiles the cpython interpreter from source". Toolchain savvy would seem a precursor to doing C/C++ extensions down in the Expert section.

leetrout 3y ago

I want to work with folks who know the difference between a module and a package. That follow pep8 for most things. People that know what a namespace package is and how __init__.py works. I care more about operating the toolchain efficiently and writing understandable code and avoiding "advanced" patterns until forced to use them for business impacting constraints (performance). Use simple data classes or vanilla classes without a lot of methods. Functions that take objects in and dont mutate them. Readable and understandable by folks new to the org and the team. And juniors. People who care about the impact of maintaining code and put the effort into effective documentation and code comments.

waitforthecream 3y ago

From the comments section, i understand that the author did not do a good job. Can someone please share other resources, that contains an exhaustive list of things to learn in python.

laurent_du 3y ago

I think the goal of the author was to explain how he rates candidates for the positions he is offering, rather than answering properly the question "what makes a beginner / advanced / expert Python developer?". This is made evident by the fact that most contributors to this thread strongly disagree with the author's choice. I personally find it very weird that sets are listed as an advanced feature of the language. And understanding Python's dynamic typing system is somehow an expert-level achievement?

fortyseven 3y ago

A helpful addition to this list would be links to good resource or two, each, where the reader can learn more.