4 ms·

Thanks for these efforts. I am surprised that you “… were surprised how many built bespoke state management layers and used printlines to debug.” Print is the c

by pama 3y ago

Thanks for these efforts. I am surprised that you “… were surprised how many built bespoke state management layers and used printlines to debug.” Print is the common debug tool since the start of CS, and bespoke state management layers, though not ideal in a hypothetical world, involve less transition effort than a commitment to a new framework not invented here that eventually has to change. Maybe you meant something different?

elijahbenizzy 3y ago
Sure, yes, not going to lie I use printlines all the time. There's nothing inherently wrong with it, and meant no offense! In fact, you can even add a hook with Burr that adds as many printlines as you like :) https://burr.dagworks.io/concepts/hooks/ https://burr.dagworks.io/concepts/hooks/. Meant no offense. That said, logging stuff and using regexes (which is the common way to productionize printlines for live/post-hoc debugging) has a lot of challenges, and this is where we've seen a framework become useful. There's an obvious trade-off between adopting a framework and rolling your own -- we've seen both work in various contexts (we've been in MLOps for quite a while). Our hope is that this will make people more productive by constraining the space they need to think about and allow more mental room for application logic.
- pama 3y ago
  Thanks. Absolutely no offense taken. No worries. I am used to frank communication and it occasionally doesn’t come out well in quick writing on the phone. The main concern I have with frameworks is the inversion of control of the code logic, the lack of composition, the complexity of handling edge cases outside the scope of the framework. Frankly writing apps is not my thing typically. PyTorch is moderately unobtrusive, but other tools on top make tons of assumptions and break more often than they help in the long term. Of course with the current zoo of LLM and APIs and the coming popularity of agents, a lot of people will need some help to get started so I wish you the best of luck in your endeavors. Here is some unsolicited advice: as the LLMs get better, you might want to focus on detailed documentation so that the future AIs can master your framework and help your users. Maybe measure the power of your documentation by finetuning in-house LLMs on it and then evaluate known test cases or create new ones with the finetuned models.
  - krawczstef 3y ago
    > The main concern I have with frameworks is the inversion of control of the code logic, the lack of composition, the complexity of handling edge cases outside the scope of the framework. Frankly writing apps is not my thing typically. PyTorch is moderately unobtrusive, but other tools on top make tons of assumptions and break more often than they help in the long term. 100%; lived that story. In my experience this is usually the result of some poor design assumptions on part of the framework. Other times people want to use something for which it wasn't designed for. In the case of Burr, we still have a few things to iron out, but given our platform backgrounds, we're trying to be cognizant of things that have annoyed us in the past, e.g. two big ones: ability to customize parts given esoteric deployment needs (e.g. persistence layer); adding in observability without it coupling you/getting in the way of logic. > the LLMs get better, you might want to focus on detailed documentation so that the future AIs can master your framework and help your users Even without LLMs I think this is required ;)
    - pama 3y ago
      Agreed that detailed documentation is required anyways. However, LLMs give you the ability to measure the quality of the documentation and the improvements in quality for changes in the documentation without having to wait for complaints by users.
  - elijahbenizzy 3y ago
    Yep! That makes sense. As @krawczstef said, building a wrong abstraciton, an overly rigid abstraction, or an overly complex abstraction can all make things worse. With any abstraction there will also always be people who end up outgrowing it or customizing it. And good point re: good docs for LLMs! On a meta-level there's something very funny about having to write better human-facing docs for the use of computers. Who would have thought that some of our most powerful computing systems emulate how we think!
avereveard 3y ago
Right? People forget the roots. Nowasays logs are super complicated things, but in unix and early Linux days, everything was a file descriptor and it happened to be that 0 was the default one to write to and by convention could be routed by pipes and streams into whatever making it super convenient way to emit informations not to users, but to anything Which is the right thing anyway. If your target are interactive users a log is a terrible thing to do as they won't know where it ends up. And if the target is not interactive users you should have error codes not log messages. I think that is partly because today is hard to just attach a debugger to things and see the state of a program so people try to serialise the program state to english strings with a timestamp and work backwards from that to understand what the program was doing, but then what we need is a differentiable single stack core dump not an english approximation of it. And especially not one that gets stolen by systemd and hidden away. Sorry for the rant but just printing infos and core dumping errors was the best thing ever and it was taken away from me.
- Karrot_Kream 3y ago
  > Right? People forget the roots. Nowadays logs are super complicated things, but in unix and early Linux days, everything was a file descriptor and it happened to be that 0 was the default one to write to and by convention could be routed by pipes and streams into whatever making it super convenient way to emit informations not to users, but to anything No lol, that's not simple, that's just looking at the defaults that unix systems provide. If you've been hacking this long then you're familiar with all the fun issues like: * Are you writing to stdout as a buffered or unbuffered stream? * What happens if your writing process crashes before flushing its buffers if you're using a buffered stream. * Are you redirecting your log stream into a file on a slow volume? * Are collected log files being rotated? What happens if the target volume runs out of space before rotation kicks in? * If you are logging to a file does an error writing to this file block stdout streaming? If so, what happens to your process? These are hard problems which have a variety of solutions. There are more unix-y solutions such as runit's `svlogd` up to more involved solutions like systemd's `journald`, but problems they are nonetheless. I've hit each and every one of these in a system before, though most in non-prod environments. That's not to knock println debugging, which continues to be very popular, it's just that logging is actually a hard problem and pretending like it's not as some appeal to systems of yore isn't helpful. But by definition when you're debugging a system you're not in a production environment anyway so you can definitely deal with the log noise that comes with println debugging.

by pama 3y ago

elijahbenizzy 3y ago

Sure, yes, not going to lie I use printlines all the time. There's nothing inherently wrong with it, and meant no offense! In fact, you can even add a hook with Burr that adds as many printlines as you like :) https://burr.dagworks.io/concepts/hooks/ https://burr.dagworks.io/concepts/hooks/. Meant no offense. That said, logging stuff and using regexes (which is the common way to productionize printlines for live/post-hoc debugging) has a lot of challenges, and this is where we've seen a framework become useful. There's an obvious trade-off between adopting a framework and rolling your own -- we've seen both work in various contexts (we've been in MLOps for quite a while). Our hope is that this will make people more productive by constraining the space they need to think about and allow more mental room for application logic.

pama 3y ago
Thanks. Absolutely no offense taken. No worries. I am used to frank communication and it occasionally doesn’t come out well in quick writing on the phone. The main concern I have with frameworks is the inversion of control of the code logic, the lack of composition, the complexity of handling edge cases outside the scope of the framework. Frankly writing apps is not my thing typically. PyTorch is moderately unobtrusive, but other tools on top make tons of assumptions and break more often than they help in the long term. Of course with the current zoo of LLM and APIs and the coming popularity of agents, a lot of people will need some help to get started so I wish you the best of luck in your endeavors. Here is some unsolicited advice: as the LLMs get better, you might want to focus on detailed documentation so that the future AIs can master your framework and help your users. Maybe measure the power of your documentation by finetuning in-house LLMs on it and then evaluate known test cases or create new ones with the finetuned models.
- krawczstef 3y ago
  > The main concern I have with frameworks is the inversion of control of the code logic, the lack of composition, the complexity of handling edge cases outside the scope of the framework. Frankly writing apps is not my thing typically. PyTorch is moderately unobtrusive, but other tools on top make tons of assumptions and break more often than they help in the long term. 100%; lived that story. In my experience this is usually the result of some poor design assumptions on part of the framework. Other times people want to use something for which it wasn't designed for. In the case of Burr, we still have a few things to iron out, but given our platform backgrounds, we're trying to be cognizant of things that have annoyed us in the past, e.g. two big ones: ability to customize parts given esoteric deployment needs (e.g. persistence layer); adding in observability without it coupling you/getting in the way of logic. > the LLMs get better, you might want to focus on detailed documentation so that the future AIs can master your framework and help your users Even without LLMs I think this is required ;)
  - pama 3y ago
    Agreed that detailed documentation is required anyways. However, LLMs give you the ability to measure the quality of the documentation and the improvements in quality for changes in the documentation without having to wait for complaints by users.
- elijahbenizzy 3y ago
  Yep! That makes sense. As @krawczstef said, building a wrong abstraciton, an overly rigid abstraction, or an overly complex abstraction can all make things worse. With any abstraction there will also always be people who end up outgrowing it or customizing it. And good point re: good docs for LLMs! On a meta-level there's something very funny about having to write better human-facing docs for the use of computers. Who would have thought that some of our most powerful computing systems emulate how we think!

avereveard 3y ago

Right? People forget the roots. Nowasays logs are super complicated things, but in unix and early Linux days, everything was a file descriptor and it happened to be that 0 was the default one to write to and by convention could be routed by pipes and streams into whatever making it super convenient way to emit informations not to users, but to anything Which is the right thing anyway. If your target are interactive users a log is a terrible thing to do as they won't know where it ends up. And if the target is not interactive users you should have error codes not log messages. I think that is partly because today is hard to just attach a debugger to things and see the state of a program so people try to serialise the program state to english strings with a timestamp and work backwards from that to understand what the program was doing, but then what we need is a differentiable single stack core dump not an english approximation of it. And especially not one that gets stolen by systemd and hidden away. Sorry for the rant but just printing infos and core dumping errors was the best thing ever and it was taken away from me.

Karrot_Kream 3y ago
> Right? People forget the roots. Nowadays logs are super complicated things, but in unix and early Linux days, everything was a file descriptor and it happened to be that 0 was the default one to write to and by convention could be routed by pipes and streams into whatever making it super convenient way to emit informations not to users, but to anything No lol, that's not simple, that's just looking at the defaults that unix systems provide. If you've been hacking this long then you're familiar with all the fun issues like: * Are you writing to stdout as a buffered or unbuffered stream? * What happens if your writing process crashes before flushing its buffers if you're using a buffered stream. * Are you redirecting your log stream into a file on a slow volume? * Are collected log files being rotated? What happens if the target volume runs out of space before rotation kicks in? * If you are logging to a file does an error writing to this file block stdout streaming? If so, what happens to your process? These are hard problems which have a variety of solutions. There are more unix-y solutions such as runit's `svlogd` up to more involved solutions like systemd's `journald`, but problems they are nonetheless. I've hit each and every one of these in a system before, though most in non-prod environments. That's not to knock println debugging, which continues to be very popular, it's just that logging is actually a hard problem and pretending like it's not as some appeal to systems of yore isn't helpful. But by definition when you're debugging a system you're not in a production environment anyway so you can definitely deal with the log noise that comes with println debugging.