21 ms·
Interactive workflows for C++ with Jupyter
- ddavis 9y agoVery cool! I watched the SciPy talk on xtensor (same developers as xeus) and the project looked pretty awesome. Many people in my field used CINT for a long time and cling is so much better. It's nice to see it used outside of particle physics. One thing I disagree with is the idea that the lack of a good interactive environment for C++ makes it difficult to teach. C++ is a compiled language, so learning how to compile a C++ library or program is _part_ of C++. I feel like C++ beginners should go interactive _after_ learning how to compile a project. I write that with first hand experience; my first programming experience was with writing interpreted C++ in CINT and I feel like it hindered my ability to eventually understand what a real C++ program was.
- pjmlp 9y agoC++ on environments like C++ Builder is relatively easy. The issue is developers having open mind to such environments, luckily it seems to be changing with QtCreator, CLion, VC++, specially thanks to clang being implemented as a library.
- jmabille 9y agoI agree that the compilation process is part of the C++, however (and that's from my teacher's experience), teaching it to beginners as a first lesson may be complicated (and you have to teach it early so they can build their first program). Besides this is not rewarding when your students are familiar with other languages like python where they can write some code and have immediate results without any additional step. So having a good interactive environment is really useful for the first lessons, then you can teach the compilation chaintool and switch to a real environment.
- xfer 9y agoCompiling a project with right incantation of flags is not rocket science but it is time consuming that a build system should be better equipped to do. Interactive development has immense benefits in exploring apis for experts and learning the language for beginners. And C++ is not well-known for its compile-times.
- jcelerier 9y ago> so learning how to compile a C++ library or program is _part_ of C++. I entirely disagree. This is only for historical reasons, the language itself does not care at all about the compilation model. This mindset is what is keeping C++ back, both for the industry AND in students' minds. > I feel like it hindered my ability to eventually understand what a real C++ program was. The moment you have a .o file it's not a C++ program anymore, but a platform-specific object file. You aren't learning C++ but windows / mac / linux's native binary production toolchain.
- AstralStorm 9y agoIndeed, the language does not care which means you should. Object file is not even a compilation unit at times. You get to build the object file, shared objects, dlls what have you. Then use the linker to link it to turn it into an executable. Some modern languages specify the runtime very deep. For example Python or Java - so much that it is hard to separate language from runtime or standard library.
- ddavis 9y ago> The moment you have a .o file it's not a C++ program anymore, but a platform-specific object file. You aren't learning C++ but windows / mac / linux's native binary production toolchain. Then that is at least part of the learning curve of seriously using C++ so I think as of today it's (potentially) dangerous to start with only interpreted code and wait to be introduced to compiling, build tools, etc. They should be taught together. I also think it's not the greatest environment - and I hope it changes, perhaps projects like this will help!
- dguest 9y agoSomeone else commented on the semantic difference between the narrow definition of C++ as a specification and the looser "C++" as a part of the full stack. With regard to the full stack, I tend to agree with you, at least in the academic environment. Organizations that don't have a lot of software engineers will shoot themselves in the foot if they focus on the C++ language rather than the infrastructure around it. An interactive notebook is great to teach someone to add a few lines of code here or there, or to run quick tests, but what we're lacking is people who understand how to plug their jupyter notebook into the production line.
- FlyingSnake 9y agoC++, despite being a really powerful language, has a really primitive build system. Developers have to wade through a complex soup of Makefiles/CMake/Gyp/ninja etc just to get a project set up. There's conan for package management, but it pales when compared to Cargo or Gradle. Moreover setting up Conan is a full time job in itself, and it's not without weird quirks either. I've not tried Bazel yet, and would love to hear if it's as good as Xooglers say. C++ on Jupyter looks amazing, and I really hope it makes starting with C++ easier.
- coherentpony 9y ago> Developers have to wade through a complex soup of Makefiles/CMake/Gyp/ninja etc But you do this precisely once.
- mschuetz 9y agoOnce every week.
- coherentpony 9y agoI don't see how that's possible unless you're just starting out. Do you start a new project every week?
- Matthias247 9y agoI guess it also applies when you want to integrate a new third party library once a week, in cases where the libraries build system doesn't match yours.
- mschuetz 9y ago- I have to start seperate small test and prototype projects every now and then. - I have to include third party libraries occasionally, and sometimes weight if it wouldn't be just easier, faster and better for my mental health to reinvent the wheel instead. - I have to check out, test or use third party projects every now and then. They all use different build systems, different ways to use different build systems and different work around to get around limitations of build systems. I like C++11 and love C++17 but absolutely hate building and managing C++ projects. It's a mess.
- periram 9y agoFinally! Thanks a ton to both Cling developers and Jupyter. Personally, a lot my programming involves prototyping and playing with a lot of different ideas and implementations before settling on a solution. Compiled languages put a steep barrier to this way of thinking and that is why Python has always been my favorite. I have come back to C++ thanks to Cpp14, 17 and great new features like this!!
- rocqua 9y agoThis looks downright amazing. I only wish it had come a few months earlier. I've had to switch from python in a jupyter notebook to C++ for performance reasons, and I really felt the loss of incrementalism.
- cbcoutinho 9y agoYou can create your project into python library by wrapping in ctypes or using the cffi package. I do that with some of my numerical simulation code and it doesn't take away from the interactivity of python. Once I get a feature more or less complete, I re-write it in c/fortran and compile it away. This maintains performance as well as focus, because the 'interesting' thing you'll be working on is quick and dirty using python, and the stuff you've already okay'd is in a compiled language out of sight.
- rocqua 9y agoIt is this transpiling from python to C++ that I want to avoid. Especially because there is a lot of iterative development on the C++. However, I'll essentially never be able to run my code from a notebook because it runs on a cluster I need to SSH into. Thing is, I'd like to develop that code locally and iteratively.
- cbcoutinho 9y agoBut I thought you said you already re-wrote it from python to c++ anyways? I was trying to point out it doesn't have to be either/or. That being said, I think the fact that you need to get access through ssh is another hurdle all together.
- rocqua 9y agoI switched once I needed the performance, but that was not near the end of developtment. The switch was made far from a point where the code, or even the algorithms were finished. Thus it was required to do a lot of incremental development in C++. It wouldn't make sense to add in a manual transpilation at every increment, and I needed the actual C++ code to judge performance.
- BeetleB 9y agoIt would be great if we get C++ on Jupyter. At the moment, I use Org Babel to do these kinds of things. It's a halfway solution.
- mcemilg 9y agoOkay. If its works amazing, so it can be work on other compiled languages? This will be amazing and it can be game changer.
- kasperset 9y agoHow is it different from R notebook implementation of C++ chunks execution?
- peheje 9y agoCool. For anyone looking at using Python when speed is important should also take a look at Numba.
- superbatfish 9y agoYes, numba is great; I've had a good experience with it for some functions. Some problems aren't suited well for it though, especially those that involve Python dictionaries or sets. For those problems, switching to C++ results in MUCH faster code (std::unordered_map is way faster than Python's dict, especially for small elements).
- peheje 9y agoAs I'm aware of Python dictionaries are not supported by Numba, however it will still run in "object mode" which won't give you much performance boost. I tend to always specify the Numba function as njit or objectMode=False. But yea true, Numba does have shortcomings, maybe dictionaries and classes will be supported in the future.
- muxator 9y ago> Most of the progress made in software projects comes from incrementalism. Obstacles to fast iteration hinder progress. I would like to print it in very big letters and put it at the entrance of slow-moving IT departments. "cost of change" is likely one of the most important metrics to measure the effectiveness of a team. Being risk adverse because of fear of change leads to stagnation. "more Jupyter notebooks and less spreadsheets" could be a useful simplification for conveying this change of POV.
- taeric 9y agoAnd yet,"more data in spreadsheets" is a truism. The spreadsheet model is empirically one of the most successful ones we have. Worse, so called "notebooks" are neat, but they are just as bad for engineering best practices as spreadsheets are. Not to say that you shouldn't be able to grow from freeform practices. You should. Just don't mistake them for being somehow superior to spreadsheets.
- jononor 9y agoYou can very easily move some functions out of a notebook and into a proper module, that can be included in a program / production system. Autoreload extension in Python Jupyter notebooks makes it extra nice. It is also easy, and convenient to write py.test style tests inside a notebook, just call them manually inside the block. Then when you move it out to module, you have tests that will automatically be picked up by your testrunner.
- taeric 9y agoYou can. Most don't. And moving something from a free form environment to a tested continuous delivery one, is not automatic. Luckily, the performance constraints of most environments are such that python is not an automatic deal breaker nowadays. That said, correctness of code proofs are usually different from correctness of machine learning algorithms. Such that mixing them seems to just fool both sets of practitioners.
- 9y ago
- aldanor 9y agoI can’t help mentioning my own project - https://github.com/aldanor/ipybind https://github.com/aldanor/ipybind - which provides a lightweight Jupyter interface to pybind11 [1], allowing to quickly sketch working C++ extensions in the notebook; all features of pybind11 like numpy support are immediately available. Been using it myself quite a lot for prototyping C++/Python code. [1] https://github.com/pybind/pybind11 https://github.com/pybind/pybind11
- harry8 9y agoThis is how I'm doing C++ rapid interactive. Combined shell and C++ in a single file that compiles itself and either runs itself or runs objdump on itself. I find it really useful and fast to test something in junkcode. Not quite the same problem as interactive C++ with Jupyter but there's some overlap. Overlap with godbolt as well - find this quicker perhaps because I'm more used to it? https://github.com/hal88/junkcode/blob/master/template_script.cpp https://github.com/hal88/junkcode/blob/master/template_scrip... I post in the hope that it is a useful idea to someone and also that people respond with their similar ideas that might be useful to me.
- rb808 9y agoI'd like to use Jupyter notebooks to do support tasks - eg check on system hosts and do reports, maybe restart processes. I think it could be perfect. I'm just hesitant because no one else seems to use it this way. Any advice out there?
- gcc_programmer 9y agoI think jupyter with C++ is an amazing idea, and I hope it will help more people to start with, in my subjective opinion, one of the most important programming languages today. That being said, I look down on all the commentators who say that C++ has a primitive build system/package manager etc. Lets be clear: C++ has no package manager, and , in my view of the world, no compiled language has a native build system, its just that most compiled languages dont separate compilation from linking and also most languages support modules :) C++ is hard. Throwing some fancy frontend and and a REPL will only make you realise that sooner. The complexity is mostly unjustified, but the language is so powerful that you could do anything with it: from trading strategies to power plant control systems to game engines. I have been studying C++ for 9 years, have used it for 7, and have been a professional developer for only 2.5 years, and I think I have only scratched the surface. I have used python for a lot less time but I feel like I am more comfortable with it because Python is trivial to learn (to a person with a comp sci background ofc). What you think you are cutting away (dependencies/build jnfo) is something essential to learning C++ and how everything comes together. Better learn it sooner than later imho. Rant over
- weberc2 9y ago> no compiled language has a native build system, its just that most compiled languages dont separate compilation from linking and also most languages support modules :) Most modern, mainstream compiled languages have a build system that wraps around the compiler and linker. Invoking the compiler and linker are responsibilities of the build system, and frequently modules are compilation units. Modern C++ is a neat language (I used to be a C++ developer) but it's tooling alone makes it unsuitable for any project where rapid iteration is remotely important.
- blackflame7000 9y agoYou're right about most compilers having a build system MSVC being probably the biggest. Modern C++ when combined with Qt makes rapid iteration very doable. In my opinion Qt should be the standard c++ model.
- mempko 9y agoLooks like the world is finally catching up to the Future that Temple OS lives in.
- Mathnerd314 9y agoTempleOS doesn't do scatter plots.
- westurner 9y agoQuantStack/xeus-cling https://github.com/QuantStack/xeus-cling https://github.com/QuantStack/xeus-cling QuantStack/xwidgets https://github.com/QuantStack/xwidgets https://github.com/QuantStack/xwidgets QuantStack/xplot (bqplot) https://github.com/QuantStack/xplot https://github.com/QuantStack/xplot
- boltzmannbrain 9y agoI really hope this works as promised. I'm curious if they'll add support for boost (www.boost.org) though.
- deepGem 9y agoActually, I've found Xcode to be a very easily usable prototyping tool for C++. I haven't used it in any large project, but it's super simple to just prototype some concepts. Having that in a browser is even better.
- TisenoT 9y ago>This also makes C++ more difficult to teach. The first hours of a C++ class are rarely rewarding as the students must learn how to set up a small project before writing any code. And then, a lot more time is required before their work can result in any visual outcome. Install a c++ compiler, create main.cpp, `g++ main.cpp` What am I missing?
- flukus 9y ago> What am I missing? The students don't know anything about the CLI. They might not know what a directory is or how to cd to it. They may have to install multiple packages. On certain operating systems they'll have to set some PATH variables. I'm not saying it shouldn't be that easy, but due to failures in earlier education it's not.
- ryanpepper 9y agoWhen I was taught C++ at University, we were just told to download Quincy which bundles a (pretty old) version of GCC with a text editor on Windows. Once we had the basics down they introduced Linux, the command line interface to GCC, etc.
- AstralStorm 9y agoFormerly, djgpp package. And the equivalent Windows IDE, can't remember its name but it shipped with Mingw.
- scardine 9y agoI could not make it work on a mac. It almost works but looks like there is no mac package for xeus-cling at the conda repos. Tried also to clone the github repo and compile locally but got tired of fixing dependencies and stuff after a few compiler errors.
- rawland 9y agoTo install this, make sure to start with plain miniconda2, instead of anaconda2. I was fighting against QuantStack's modified software stack for two hours until I discovered this: https://github.com/QuantStack/xeus-cling/issues/74#issuecomment-343681758 https://github.com/QuantStack/xeus-cling/issues/74#issuecomm... Fortunately, they updated the README.md in https://github.com/QuantStack/xeus-cling https://github.com/QuantStack/xeus-cling
- superbatfish 9y agoGood tip! IMHO, I always recommend the Miniconda installer over the full Anaconda installer, except for absolute beginners who would struggle to understand basic concepts like environments or the PATH environment variable, etc.