5 ms·

Ask HN: What should go in an Excel-to-Python equivalent of a couch-to-5k?

• westurner 6y ago
  How to write functions in JS / VB script and call them from a cell expression. How to name variables something other than AB3. How to use physical units and datatypes. (How to specify XSD datatype URIs that map to native primitives in an additional frozen header row. e.g. py-moneyed and Pint & NumPy ufuncs) How transitive sort works (is there a tsort to determine what to calculate first (and whether there are cycles) on every modification event?) Which Jupyter platforms do and don't support interactive charts with e.g. ipywidgets? pandas.df.plot(kind=) (matplotlib), seaborne (what are the calculated parameters of this chart?), holoviews, plotly, altair Reproducibility w/ repo2docker / BinderHub: pip freeze > constraints.txt cp constraints.txt requirements.txt conda env export --from-history
  • etothepii 6y ago
    Interesting, your suggestion is that the first thing to do is to learn how to make functions and variable names within excel?
    • westurner 6y ago
      Yeah if you port it to functions and verify that you haven't broken anything, you could then easily port to Python functions that you could call from Excel with an add-on that everyone that opens the sheet needs to have installed; but everyone that opens a sheet that calls Python must have that same extension (and all python package dependencies) installed, too
  • westurner 6y ago
    Also, When is it better to have code in a notebook instead of in a module? How to export notebook cells to a module with nbdev How to write tests to assert the quality of the code and the model: @pytest.mark.parametrize, pytest-notebook, jupyter-pytest-2, pytest-jupyter When is it appropriate to parametrized a notebook with e.g. papermill? How to handle concurrency: dask.distributed + dask-labextension, ipyparallel
• Jugurtha 6y ago
  >My (flippant) reaction to a friend that brought the comment to my attention was unhelpful; "Step 1 day 1, quit." So he has challenged me to write eight helpful blog posts during the remainder of my Garden Leave. One way to do it would be to do impedance matching: maximizing usefulness by bringing Python to the universe the person who asked the question lives in: Excel, Word, PDF, email. This means "Automate the Boring Stuff"[0] to ease into programming with something specific to do that is relevant and useful right off the bat for manipulating files, spreadsheets, PDF documents, etc. Openpyxl[1] for playing more with Excel files, and basically searching "Excel Python" on the internet for more use cases. Python-docx[2] for playing with Word documents, extracting content from tables, etc. PdfPlumber[3] for playing with PDFs. Sometimes you have data in .pptx files, and you can use LibreOffice to convert all of them to PDFs because you hate the pptx format: libreoffice --headless --invisible --convert-to pdf *.pptx Also, if you both are using notebooks, take a look at what we're building at https://iko.ai https://iko.ai. I posted about it a bit earlier[4]. It has no-setup real-time Jupyter notebooks, and all the jazz. Focuses on solving machine learning problems since we built it for ourselves to slash projects' time. - [0]: https://nostarch.com/automatestuff2 https://nostarch.com/automatestuff2 - [1]: https://openpyxl.readthedocs.io/en/stable/ https://openpyxl.readthedocs.io/en/stable/ - [2]: https://python-docx.readthedocs.io/en/latest/ https://python-docx.readthedocs.io/en/latest/ - [3]: https://github.com/jsvine/pdfplumber https://github.com/jsvine/pdfplumber - [4]: https://news.ycombinator.com/item?id=25608098 https://news.ycombinator.com/item?id=25608098
  • etothepii 6y ago
    Using openpyxl to extract data from exist excel workbooks and other documents would seem like a good place to start. I suspect many companies use excel workbooks as "forms" with lots of data at the same cell in multiple workbooks.
    • westurner 6y ago
      > I suspect many companies use excel workbooks as "forms" with lots of data at the same cell in multiple workbooks. Downstream data quality costs can be minimized with data normalized schema and data collection process controls like forms-based data validation. There are established UI/UX design patterns for data validation of user-supplied data: accessible [web] forms with tab-ordered input fields and specific per-input feedback with accessible HTML5 and ARIA. IIUC, Firefox now supports PDF forms, too? Why would we move from a spreadsheet to an actual database? Data integrity: Referential integrity (making sure that record keys actually point to something when creating, updating, or deleting), Columnar datatypes (float, decimal, USD, complex fraction), Access controls (auth(z): authentication and authorization), Auditing (what was the value before and after that) and Disaster Recovery, Organizationally-unified schema development and corresponding validation. Repeatability / Reproducibility: can you replay the steps needed to build the whole sheet? What parameters were entered and how to we script that par so that we can easily assess the relations between the terms of the argument presented?