4 ms·

>My (flippant) reaction to a friend that brought the comment to my attention was unhelpful; "Step 1 day 1, quit." So he has challenged me to write eight helpful

>My (flippant) reaction to a friend that brought the comment to my attention was unhelpful; "Step 1 day 1, quit." So he has challenged me to write eight helpful blog posts during the remainder of my Garden Leave. One way to do it would be to do impedance matching: maximizing usefulness by bringing Python to the universe the person who asked the question lives in: Excel, Word, PDF, email. This means "Automate the Boring Stuff"[0] to ease into programming with something specific to do that is relevant and useful right off the bat for manipulating files, spreadsheets, PDF documents, etc. Openpyxl[1] for playing more with Excel files, and basically searching "Excel Python" on the internet for more use cases. Python-docx[2] for playing with Word documents, extracting content from tables, etc. PdfPlumber[3] for playing with PDFs. Sometimes you have data in .pptx files, and you can use LibreOffice to convert all of them to PDFs because you hate the pptx format: libreoffice --headless --invisible --convert-to pdf *.pptx Also, if you both are using notebooks, take a look at what we're building at https://iko.ai https://iko.ai. I posted about it a bit earlier[4]. It has no-setup real-time Jupyter notebooks, and all the jazz. Focuses on solving machine learning problems since we built it for ourselves to slash projects' time. - [0]: https://nostarch.com/automatestuff2 https://nostarch.com/automatestuff2 - [1]: https://openpyxl.readthedocs.io/en/stable/ https://openpyxl.readthedocs.io/en/stable/ - [2]: https://python-docx.readthedocs.io/en/latest/ https://python-docx.readthedocs.io/en/latest/ - [3]: https://github.com/jsvine/pdfplumber https://github.com/jsvine/pdfplumber - [4]: https://news.ycombinator.com/item?id=25608098 https://news.ycombinator.com/item?id=25608098

etothepii 6y ago
Using openpyxl to extract data from exist excel workbooks and other documents would seem like a good place to start. I suspect many companies use excel workbooks as "forms" with lots of data at the same cell in multiple workbooks.
- westurner 6y ago
  > I suspect many companies use excel workbooks as "forms" with lots of data at the same cell in multiple workbooks. Downstream data quality costs can be minimized with data normalized schema and data collection process controls like forms-based data validation. There are established UI/UX design patterns for data validation of user-supplied data: accessible [web] forms with tab-ordered input fields and specific per-input feedback with accessible HTML5 and ARIA. IIUC, Firefox now supports PDF forms, too? Why would we move from a spreadsheet to an actual database? Data integrity: Referential integrity (making sure that record keys actually point to something when creating, updating, or deleting), Columnar datatypes (float, decimal, USD, complex fraction), Access controls (auth(z): authentication and authorization), Auditing (what was the value before and after that) and Disaster Recovery, Organizationally-unified schema development and corresponding validation. Repeatability / Reproducibility: can you replay the steps needed to build the whole sheet? What parameters were entered and how to we script that par so that we can easily assess the relations between the terms of the argument presented?

etothepii 6y ago

Using openpyxl to extract data from exist excel workbooks and other documents would seem like a good place to start. I suspect many companies use excel workbooks as "forms" with lots of data at the same cell in multiple workbooks.

westurner 6y ago
> I suspect many companies use excel workbooks as "forms" with lots of data at the same cell in multiple workbooks. Downstream data quality costs can be minimized with data normalized schema and data collection process controls like forms-based data validation. There are established UI/UX design patterns for data validation of user-supplied data: accessible [web] forms with tab-ordered input fields and specific per-input feedback with accessible HTML5 and ARIA. IIUC, Firefox now supports PDF forms, too? Why would we move from a spreadsheet to an actual database? Data integrity: Referential integrity (making sure that record keys actually point to something when creating, updating, or deleting), Columnar datatypes (float, decimal, USD, complex fraction), Access controls (auth(z): authentication and authorization), Auditing (what was the value before and after that) and Disaster Recovery, Organizationally-unified schema development and corresponding validation. Repeatability / Reproducibility: can you replay the steps needed to build the whole sheet? What parameters were entered and how to we script that par so that we can easily assess the relations between the terms of the argument presented?