5 ms·

Show HN: A simple lazy Python calculation engine, with spreadsheet demo

123 points by bsdz 5y ago

bsdz 5y ago
This is something I built a little while ago. It's a proof-of-concept and inspired by what I was doing in my day job. To play with the demo spreadsheet you'll need to clone the repo and run the demo script, ie: git clone git@github.com:bsdz/calcengine.git cd calcengine python3 -mvenv --prompt calceng .venv . ./.venv/bin/activate poetry install -E demo python demo/spreadsheet/main.py There's an inline gif showing the spreadsheet in use. Hope it's interesting :)
- doovd 5y ago
  This is a core principle in a lot of the dependency-graph-based frameworks seen in banks (secdb, Athena, quartz) etc. Inspired by those?
  - bsdz 5y ago
    Indeed. Actually with-in the demo samples folder you can find a simple QuantLib example (quantlib.json). Although you'll need to install the QuantLib package separately as I didn't include it in the dependencies.
    - timkpaine 5y ago
      You may also be interested in: https://github.com/timkpaine/tributary/blob/main/docs/examples/lazy/lazy.md https://github.com/timkpaine/tributary/blob/main/docs/exampl...
      - bsdz 5y ago
        Yes I am. Thanks for sharing.
    - wirthjason 5y ago
      The Dag/graph is one of the best parts of those tool chains. That is one library I wish had a good open source equivalent. Partly because it would be nice to use in any project / environment, but also because there would be a lot more documentation and examples. Take Pandas for example, if it stayed internal then it wouldn’t be the powerhouse that it is today. Most dependency graph examples are based around calculations but it’s a versatile pattern and not limited to just math. I find the JS community has really taken hold of the dependency tracking and “reactive framework” idea to build UIs.
- maleldil 5y ago
  Why create a venv and then run poetry install? Poetry already manages virtual environments.
  - bsdz 5y ago
    True. Just an old habit.
- phkahler 5y ago
  Why QT and not tkinter?
  - bsdz 5y ago
    No reason in particular. Qt has some easy to use widgets and is well documented. I've also some experience with Qt.
Tipewryter 5y ago
you can use pip install -r requirements.txt I don't see a requirements.txt file.
- bsdz 5y ago
  Good spot. You can actually just run the following if you don't want to use poetry. pip install git+https://github.com/bsdz/calcengine.git#master Unfortunately, this will only install the core library without the demo spreadsheet files. Even if you were to add the extras modifier "[demo]", it would only install additional dependencies and not the actual demo code. To use the demo you'll need to clone and follow instructions in my other comment.
  - Tipewryter 5y ago
    This outputs many pages of debug information and then dies. The error seems to be this: This package requires Rust >=1.41.0.
    - bsdz 5y ago
      That's very strange. The core library uses pure python and demo uses pyqt5, pandas, matplotlib and pillow. I wasn't aware that any of those projects (or sub projects) depend on Rust. I'm running on Linux / Ubuntu 20.04 with Python 3.8. How about you? (The pip command I just gave should only install the core library and so only depends on base python).
      - eesmith 5y ago
        Does it depend on the cryptography library? https://github.com/pyca/cryptography/issues/5771 https://github.com/pyca/cryptography/issues/5771
        bsdz 5y ago
        It doesn't look like it: $ grep name poetry.lock | cut -d" " -f3 | tr -d '"' | tr '\n' ', ' appdirs,attrs,black,click,cycler,flake8,kiwisolver,matplotlib,mccabe, mypy,mypy-extensions,numpy,pandas,pathspec,pillow,pycodestyle,pyflakes, pyparsing,pyqt5,pyqt5-sip,python-dateutil,pytz,regex,six,toml,typed-ast, typing-extensions
inshadows 5y ago
Nice Python + Qt demo. Gonna learn from it.
- bsdz 5y ago
  Thank you. Hope there's something useful in there for you :-)
dev_snd 5y ago
This is pretty cool. Spreadsheets are so powerful and really great and producing one-off solutions, but when you want to automate them you always need to redo most of the work in a "serious" programming language and wrap all of it in an API etc. I always wondered how one could bridge the gap between creating a PoC computation in a spreadsheet and getting the same code to run in production without the rewrite. This could be a nice first step for a python server application designed in a spreadsheet.
- arthurcolle 5y ago
  There is a tool called wings iirc that lets you pull out DataFrames from an xlsx and vice versa
  - quag 5y ago
    It’s called xlwings. Not a bad tool. https://docs.xlwings.org/en/stable/quickstart.html https://docs.xlwings.org/en/stable/quickstart.html
duckmysick 5y ago
Looks very interesting. Can we plug numpy into this? Or is it not necessary with the assumptions that most steps will be cached and don't have to be recalculated.
- bsdz 5y ago
  Yes, you can use any existing python modules. In the demo you'll see I am using pandas dataframes in individual cells. Pandas is built on numpy.
bradleybuda 5y ago
This is cool! Also gives me a great excuse to post one of my favorite papers, "Build Systems à la Carte", about how make and Excel are solving the same problem in different ways: https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems.pdf https://www.microsoft.com/en-us/research/uploads/prod/2018/0...
- deleted 5y ago
  [deleted]
- O_H_E 5y ago
  > make and Excel are solving the same problem Well, that's a sentence I didn't expect to read today. Count me intrigued.
yssrnjm 5y ago
This is really interesting because I was working recently on a way to have complete spreadsheets using only CSV files. The project is in my Github (https://github.com/yassirnajmaoui/runcsv https://github.com/yassirnajmaoui/runcsv and https://github.com/yassirnajmaoui/runcsv-gui https://github.com/yassirnajmaoui/runcsv-gui) The only (major) thing is that the cells are run in top-bottom left-right instead of running a graph like this repo suggests.

123 points by bsdz 5y ago

bsdz 5y ago

This is something I built a little while ago. It's a proof-of-concept and inspired by what I was doing in my day job. To play with the demo spreadsheet you'll need to clone the repo and run the demo script, ie: git clone git@github.com:bsdz/calcengine.git cd calcengine python3 -mvenv --prompt calceng .venv . ./.venv/bin/activate poetry install -E demo python demo/spreadsheet/main.py There's an inline gif showing the spreadsheet in use. Hope it's interesting :)

doovd 5y ago
This is a core principle in a lot of the dependency-graph-based frameworks seen in banks (secdb, Athena, quartz) etc. Inspired by those?
- bsdz 5y ago
  Indeed. Actually with-in the demo samples folder you can find a simple QuantLib example (quantlib.json). Although you'll need to install the QuantLib package separately as I didn't include it in the dependencies.
  - timkpaine 5y ago
    You may also be interested in: https://github.com/timkpaine/tributary/blob/main/docs/examples/lazy/lazy.md https://github.com/timkpaine/tributary/blob/main/docs/exampl...
    - bsdz 5y ago
      Yes I am. Thanks for sharing.
  - wirthjason 5y ago
    The Dag/graph is one of the best parts of those tool chains. That is one library I wish had a good open source equivalent. Partly because it would be nice to use in any project / environment, but also because there would be a lot more documentation and examples. Take Pandas for example, if it stayed internal then it wouldn’t be the powerhouse that it is today. Most dependency graph examples are based around calculations but it’s a versatile pattern and not limited to just math. I find the JS community has really taken hold of the dependency tracking and “reactive framework” idea to build UIs.
maleldil 5y ago
Why create a venv and then run poetry install? Poetry already manages virtual environments.
- bsdz 5y ago
  True. Just an old habit.
phkahler 5y ago
Why QT and not tkinter?
- bsdz 5y ago
  No reason in particular. Qt has some easy to use widgets and is well documented. I've also some experience with Qt.

Tipewryter 5y ago

you can use pip install -r requirements.txt I don't see a requirements.txt file.

bsdz 5y ago
Good spot. You can actually just run the following if you don't want to use poetry. pip install git+https://github.com/bsdz/calcengine.git#master Unfortunately, this will only install the core library without the demo spreadsheet files. Even if you were to add the extras modifier "[demo]", it would only install additional dependencies and not the actual demo code. To use the demo you'll need to clone and follow instructions in my other comment.
- Tipewryter 5y ago
  This outputs many pages of debug information and then dies. The error seems to be this: This package requires Rust >=1.41.0.
  - bsdz 5y ago
    That's very strange. The core library uses pure python and demo uses pyqt5, pandas, matplotlib and pillow. I wasn't aware that any of those projects (or sub projects) depend on Rust. I'm running on Linux / Ubuntu 20.04 with Python 3.8. How about you? (The pip command I just gave should only install the core library and so only depends on base python).
    - eesmith 5y ago
      Does it depend on the cryptography library? https://github.com/pyca/cryptography/issues/5771 https://github.com/pyca/cryptography/issues/5771
      - bsdz 5y ago
        It doesn't look like it: $ grep name poetry.lock | cut -d" " -f3 | tr -d '"' | tr '\n' ', ' appdirs,attrs,black,click,cycler,flake8,kiwisolver,matplotlib,mccabe, mypy,mypy-extensions,numpy,pandas,pathspec,pillow,pycodestyle,pyflakes, pyparsing,pyqt5,pyqt5-sip,python-dateutil,pytz,regex,six,toml,typed-ast, typing-extensions

inshadows 5y ago

Nice Python + Qt demo. Gonna learn from it.

bsdz 5y ago
Thank you. Hope there's something useful in there for you :-)

dev_snd 5y ago

This is pretty cool. Spreadsheets are so powerful and really great and producing one-off solutions, but when you want to automate them you always need to redo most of the work in a "serious" programming language and wrap all of it in an API etc. I always wondered how one could bridge the gap between creating a PoC computation in a spreadsheet and getting the same code to run in production without the rewrite. This could be a nice first step for a python server application designed in a spreadsheet.

arthurcolle 5y ago
There is a tool called wings iirc that lets you pull out DataFrames from an xlsx and vice versa
- quag 5y ago
  It’s called xlwings. Not a bad tool. https://docs.xlwings.org/en/stable/quickstart.html https://docs.xlwings.org/en/stable/quickstart.html

duckmysick 5y ago

Looks very interesting. Can we plug numpy into this? Or is it not necessary with the assumptions that most steps will be cached and don't have to be recalculated.

bsdz 5y ago
Yes, you can use any existing python modules. In the demo you'll see I am using pandas dataframes in individual cells. Pandas is built on numpy.

bradleybuda 5y ago

This is cool! Also gives me a great excuse to post one of my favorite papers, "Build Systems à la Carte", about how make and Excel are solving the same problem in different ways: https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems.pdf https://www.microsoft.com/en-us/research/uploads/prod/2018/0...

deleted 5y ago
[deleted]
O_H_E 5y ago
> make and Excel are solving the same problem Well, that's a sentence I didn't expect to read today. Count me intrigued.

yssrnjm 5y ago

This is really interesting because I was working recently on a way to have complete spreadsheets using only CSV files. The project is in my Github (https://github.com/yassirnajmaoui/runcsv https://github.com/yassirnajmaoui/runcsv and https://github.com/yassirnajmaoui/runcsv-gui https://github.com/yassirnajmaoui/runcsv-gui) The only (major) thing is that the cells are run in top-bottom left-right instead of running a graph like this repo suggests.