5 ms·

Pernosco Omniscient Debugger is now available for individual developers

62 points by modeless 6y ago

adamnemecek 6y ago
This is insanely expensive. 20 usd for 5 submissions?
- roca 6y ago
  We often get users estimating that a Pernosco session saves them half an hour or more. Let's say on average it saves 15 minutes. Let's say a reasonable developer's time is worth at least 50 USD an hour. Then Pernosco is giving 12.5 USD of value and we're charging 4 USD. Is that insane?
  - deleted 6y ago
    [deleted]
  - detaro 6y ago
    To me it feels like it's a weird scale. 5 a month is something where I actually have to think when and how to use it, and that's not something I want to do for a tool like that. I assume it's still usable as a local rr-GUI (that hopefully adds something) without submitting? I get that you are still looking for a model, but I would much more expect something like "my employer pays $X00 or $X000 and we get a volume where I don't have to think about using up a scarce resource" (potentially even can have customer systems run it if there's issues) - the market for single licenses feels pretty small to me? EDIT: cleaned up word salad
    - roca 6y ago
      > I assume it's still usable as a local rr-GUI (that hopefully adds something) without submitting? No. > I get that you are still looking for a model, but I would much more expect something like "my employer pays $X00 or $X000 and we don't get a volume where I don't have to think about using up a scarce resource" - the market for single licenses feels pretty small to me? We like that model too but it's been hard to sell that. We're certainly happy to offer such a deal if someone asks for it! People in related businesses have said they've had more success with "bottom-up" individual-developer sales than with enterprisey deals, at least initially, so we are giving it a go.
      - detaro 6y ago
        I guess that makes sense regarding slowly getting a foot in the door and leaving enterprise to "let's talk about it". I don't think it's for me, but I'll try it out and either way good luck, I hope it works out for you!
    - roca 6y ago
      To help with this we've added a new "volume subscription plan" for individual users: 50 submissions per month for $50.
  - antonzabirko 6y ago
    It looks cool but i dont adopt platform services in general. Seems like a profitable business if that port doesnt scare devs away.
  - trevyn 6y ago
    >Is that insane? I have no idea, but I’ve already lost interest.
    - MrBuddyCasino 6y ago
      Same. Punishing tool usage is a really dumb mistake. You want your tool to be used all the time and become indispensable, so you can charge a high monthly fee for it. Corporations are used to and even expect fixed monthly fees. Pernosco doesn't at all understand how large firms work, and what it takes to get a budget for developer productivity tools.
      - sfink 6y ago
        I suggest you consider giving them a little benefit of the doubt? I agree with you that for a larger corporation it makes much more sense for the tool to be prepaid and for the developers to not worry about their usage. There's definitely a learning curve to switching to a new style of debugging, even if you're accustomed to rr, and so encouraging use is more likely to get people to actually get to the point where they're benefiting from the tool. But that's a tough sales cycle, they've tried it, they've said they would be happy if someone came asking for that. You could make the argument that they should hire an enterprise sales person or something, which may be true but also has obvious costs. (Let's just say that it would be a large percentage increase over their current staffing.) I'd think that their reality wins out over your "shoulds". You may be right that they're incompetent about selling into the enterprise, but I'm certain that they've had more experience trying to sell Pernosco to larger corporations than you have and so I'm not going to ding them for this additional purchasing opportunity. Source: I work for a mid-sized company that does have a contract that gives me unlimited access to Pernosco. It's quite useful. I don't know the exact terms. I know I have solved some bugs that I would have been unlikely to ever fix and would be ongoing stability and maintenance concerns. I'm working on a bug right now where the submitter is able to reproduce easily but I am not. They have managed to capture it in an rr recording, so I'm uploading it to Pernosco to take a look even though my desktop's CPU is mismatched enough that I can't just run the recording myself. From past experience, if I have it in Pernosco, I will be able to fix it, even though the root cause is far far removed from the resulting crash. (It's subtle GC corruption.)
  - adamnemecek 6y ago
    OK my workflow might be run, debug, change code, recompile, run, debug... Like many many times over. I can't imagine having to pay every time I use the debugger. Like it's weird that you pay per submission.
    - roca 6y ago
      Yeah, Pernosco may not be a great fit when you have to do that. That's not just due to the pricing model --- which we can change! --- but the entire debugging approach, where we do a lot of analysis up front, speeding the debugging experience but adding latency to the workflow. Now, sometimes people are doing that because to track down a bug they have to go through many cycles of adding instrumentation or logging to their code. In many cases Pernosco can just make those cycles go away, e.g. using https://pernos.co/about/expressions https://pernos.co/about/expressions. However, if you develop new code using cycles of "add new code, compile, debug", and each of those debugging sessions are short, Pernosco isn't a great fit for that currently.
  - JustFinishedBSG 6y ago
    Do you realize how insane things would be if everything was charged as a percentage of value provided ? So you charge, 1/4 of the value given. Should my IDE charge me 1/4 too? My ISP ? My computer OEM ? My electricity provider ? My renter ? My OS ? That's a lot of 1/4th.
  - shawnz 6y ago
    Are you saying that the average session saves the developer 15 minutes, or the average successful session, in which the bug is actually solved, saves 15 minutes? How likely is it that a given bug will be solved with Pernosco? What do I do if I pay and only realize after that the session wasn't helpful?
    - roca 6y ago
      The average session. Of course it's impossible to know the actual average number for any given developer but we often get developers estimating Pernosco saved hours, occasionally days, or occasionally the bug couldn't have been solved any other way. So I just tried to make a conservative guess. > How likely is it that a given bug will be solved with Pernosco? There's no way to put a number on this but I think it's unusual to not eventually figure it out with rr or Pernosco (but Pernosco will be faster) if conditions under the user's control are met: the recording really captures the bug you care about, it's in code in a language we support, and the debuginfo and sources are right. > What do I do if I pay and only realize after that the session wasn't helpful? "Sorry". Hopefully the reasons it wasn't helpful are mostly under your control and the benefits average out in your favour.
- emmanueloga_ 6y ago
  "To use Pernosco, you must consent to collection, transfer, storage and use of your data" I understand the need to monetize but honestly seems like a huge commitment in terms of both trust and money to use a tool like this. Perhaps a nagging screen model would work better? (better => more users using the tool, a percentage of which would def. pay to remove the nag and support the creator/s). A list of successful paid software I can think of from the top of my head: * https://www.reaper.fm/ https://www.reaper.fm/ (nagging screen) * https://www.linqpad.net/ https://www.linqpad.net/ (PRO versions with additional features) * https://www.jetbrains.com/idea https://www.jetbrains.com/idea ("Community" and "Professional" versions) * https://cursive-ide.com/ https://cursive-ide.com/ (30 day trial? I think? Personal, Commercial and Non Commercial licenses avail.) * https://www.sublimetext.com/ https://www.sublimetext.com/ (no limitations but purchase is expected, not enforced) * https://github.com/cognitect-labs/REBL-distro https://github.com/cognitect-labs/REBL-distro (used to _require_ joining a Patreon to use the software, but did not enforce it... Also I think it used to have commercial licenses available).
  - db48x 6y ago
    It's definitely a factor worth considering before purchasing the service. It's going to record an entire program running on your local machine, including all the data that it reads and writes. That includes local files, network connections, events from your window manager, pixels put on the screen, any environment variables that happened to be defined, etc. If any of that is private then you shouldn't upload the recording to Pernosco. If you're working on a public GitHub repository then your concern is pretty much limited only to your local environment. Make sure you don't have security tokens in your environment variables when you record. If your program opens files, make sure it opens something harmless. If your program visits webpages, don't record it visiting a page you want kept private. It's only natural then that the EULA would want you to give Pernosco permission to receive that data, store it for you, and then present it back to you during your debugging sessions. There's also the on-premises option.
    - roca 6y ago
      Note that the pernosco-submit script supports detecting "sensitive" environment variables leaking into the recording: https://github.com/Pernosco/pernosco-submit/blob/a6119a1b291487dd056c38cfeb4280604358ca12/pernosco-submit#L142 https://github.com/Pernosco/pernosco-submit/blob/a6119a1b291... And we accept PRs :-)
- db48x 6y ago
  Worth it. I'd prefer to pay once and then install it on my own computer, even though that would surely cost several thousand dollars. It would still be worth it.
  - sanxiyn 6y ago
    I am not sure I would want to self-host this. It does a lot of analysis on the cloud. https://robert.ocallahan.org/2019/07/auckland-rust-meetup-building.html https://robert.ocallahan.org/2019/07/auckland-rust-meetup-bu... gives the scale of "100s of GB of data produced saturating 36 cores for hour", now that's a heavy compute.
    - db48x 6y ago
      Cool, I hadn't seen these numbers before. I have the disk space, but not the cpus. Do you think I could get it down to just 17 minutes if I buy a dual-Epyc machine to run Pernosco on? I'm asking because I've been looking for an excuse to buy a dual-Epyc machine for a while now…
- deleted 6y ago
  [deleted]
schaefer 6y ago
The video references and recreates a debug session which is originally from a great blog post by Brendan Gregg: “gdb Debugging Full Example” [1]. This original blog post uses only free tools to reach the same conclusions as the Pernosco session. So it’s interesting review and compare them both. [1] http://www.brendangregg.com/blog/2016-08-09/gdb-example-ncurses.html http://www.brendangregg.com/blog/2016-08-09/gdb-example-ncur...
- db48x 6y ago
  Also, that Pernosco session is public so anyone can try their hand at it: https://pernos.co/debug/e9UFXkGq__8KQB-N6n59lA/index.html https://pernos.co/debug/e9UFXkGq__8KQB-N6n59lA/index.html
bigcorp-slave 6y ago
I’ve been an extensive user of time travel debugging for most of my career, and Pernosco looks great. However, please, please change your business model. As is, it is useless for employees of corporations with sane legal departments.
- roca 6y ago
  We would love to talk about on-prem with those corporations --- contact us!
- malux85 6y ago
  What’s the best time travel debugger you’ve ever seen? (any language)
merricksb 6y ago
For the curious, a small discussion about this from last year: https://news.ycombinator.com/item?id=21352380 https://news.ycombinator.com/item?id=21352380
eps 6y ago
Is there a way to compare two sessions? Basically, do a diff and see where the execution flow differed between two sessions? At high level first, then drilling down.
- db48x 6y ago
  They list that feature as a TODO for the future at https://pernos.co/about/vision/#comparing-executions https://pernos.co/about/vision/#comparing-executions
  - eps 6y ago
    If it's done, it will allow cracking software that uses basic protection methods for $10 flat. But for the normal use both the pricing and its off-prem nature renders the whole thing all but useless. A pity really, because the tech looks fantastic.
    - db48x 6y ago
      I agree that comparing different runs of the same program will be amazing. I think that running Pernosco off-premises will be acceptable for many people. I use it a few times per month while working on open-source projects like Reposurgeon and Flex, so I'm not revealing any trade secrets by uploading those recordings. I've learned that the database built by Pernosco from a large recording can be 100GB and take an hour to generate on a 36-core AWS instance. $4 per debugging session is a pretty reasonable price compared to buying that hardware myself. And they are offering an on-premises solution, running on your own hardware. For a tool this good, the price for that is bound to be large enough to give individuals like us reason to think twice, and then probably a third time. Given the potential time savings, I think that it should be easy for even a small company to justify the purchase. On the gripping hand, this is non-Free software. Some will avoid it primarily for that reason, and I can't really fault them for sticking to their principles. I think that it's entirely possible that Pernosco, or debuggers like it, could effectively eliminate traditional debuggers like GDB. GDB would still exist and still work, but without the nearly-omniscient viewpoint that Pernosco offers very few will bother to use it. It's like looking through a keyhole. I've thought about this a lot over the last few months.
db48x 6y ago
Out of curiosity, has anyone else noticed that the videos and screenshots on the About page are not actually videos or even images? They're iframes with the actual Pernosco UI inside them! That is so cool.
KenoFischer 6y ago
We use rr extensively for Julia development [1] (for a sense of scale, I probably look at hundreds of rr traces a month). There's two big challenges for us there 1) Many cloud environments are broken in various ways for rr recording - I spend way too much time guiding our users to the few that do work 2) The GDB frontend really isn't the best way to view all the underlying information Pernosco is well on its way to address the second part of this. Its replayer is quite a bit nicer than the open source rr one and I have high hopes for the platform in general. For the first one, all I can do is plead with the cloud vendors not to mess up the environment. Since this is HN, if you're working for a cloud provider, please ask your engineering team not to break rr. Otherwise I'll have to spend time migrating our users off your platform, which will be fun for neither of us. Anyway, congrats to Robert and Kyle on the launch. Wish you all the best. [1] https://julialang.org/blog/2020/05/rr/ https://julialang.org/blog/2020/05/rr/
- solarengineer 6y ago
  Could you share examples of how cloud providers break rr?
  - roca 6y ago
    Usually they break rr by either blocking access to the hardware PMU entirely (e.g. on AWS, any instance that shares a CPU socket with other instances; on Azure/GCP, everywhere AFAIK), or by introducing subtle errors into the virtualized performance counter values that rr depends on (e.g. VMWare: https://robert.ocallahan.org/2015/11/rr-in-vmware-solved.html https://robert.ocallahan.org/2015/11/rr-in-vmware-solved.htm..., Xen: https://lists.xen.org/archives/html/xen-devel/2017-07/msg02242.html https://lists.xen.org/archives/html/xen-devel/2017-07/msg022...). Another feature rr benefits from is CPUID faulting, which is not virtualized by any of the major cloud providers (though Linux KVM does virtualize it): https://patchwork.kernel.org/project/kvm/patch/20161101013749.4003-8-khuey@kylehuey.com/ https://patchwork.kernel.org/project/kvm/patch/2016110101374.... That'd be "nice to have" but we can live without it.
- mwcampbell 6y ago
  Can you be more specific about which cloud environments do work with rr? In your blog post you said it works on Amazon EC2 instances above a certain size. Which instance types are known to work?
4sak3n 6y ago
Amazing name. Has my Latin nerd stamp of approval.
minetest2048 6y ago
Is it possible to use something like rr for microcontrollers, assuming we can simulate the microcontroller?
- roca 6y ago
  Yes, you can instrument simulators to get record-and-replay. Simics has a record-and-replay feature like that, so does QEMU lately. I don't know of anything specifically for microcontrollers but it certainly could be done.

62 points by modeless 6y ago

adamnemecek 6y ago

This is insanely expensive. 20 usd for 5 submissions?

roca 6y ago
We often get users estimating that a Pernosco session saves them half an hour or more. Let's say on average it saves 15 minutes. Let's say a reasonable developer's time is worth at least 50 USD an hour. Then Pernosco is giving 12.5 USD of value and we're charging 4 USD. Is that insane?
- deleted 6y ago
  [deleted]
- detaro 6y ago
  To me it feels like it's a weird scale. 5 a month is something where I actually have to think when and how to use it, and that's not something I want to do for a tool like that. I assume it's still usable as a local rr-GUI (that hopefully adds something) without submitting? I get that you are still looking for a model, but I would much more expect something like "my employer pays $X00 or $X000 and we get a volume where I don't have to think about using up a scarce resource" (potentially even can have customer systems run it if there's issues) - the market for single licenses feels pretty small to me? EDIT: cleaned up word salad
  - roca 6y ago
    > I assume it's still usable as a local rr-GUI (that hopefully adds something) without submitting? No. > I get that you are still looking for a model, but I would much more expect something like "my employer pays $X00 or $X000 and we don't get a volume where I don't have to think about using up a scarce resource" - the market for single licenses feels pretty small to me? We like that model too but it's been hard to sell that. We're certainly happy to offer such a deal if someone asks for it! People in related businesses have said they've had more success with "bottom-up" individual-developer sales than with enterprisey deals, at least initially, so we are giving it a go.
    - detaro 6y ago
      I guess that makes sense regarding slowly getting a foot in the door and leaving enterprise to "let's talk about it". I don't think it's for me, but I'll try it out and either way good luck, I hope it works out for you!
  - roca 6y ago
    To help with this we've added a new "volume subscription plan" for individual users: 50 submissions per month for $50.
- antonzabirko 6y ago
  It looks cool but i dont adopt platform services in general. Seems like a profitable business if that port doesnt scare devs away.
- trevyn 6y ago
  >Is that insane? I have no idea, but I’ve already lost interest.
  - MrBuddyCasino 6y ago
    Same. Punishing tool usage is a really dumb mistake. You want your tool to be used all the time and become indispensable, so you can charge a high monthly fee for it. Corporations are used to and even expect fixed monthly fees. Pernosco doesn't at all understand how large firms work, and what it takes to get a budget for developer productivity tools.
    - sfink 6y ago
      I suggest you consider giving them a little benefit of the doubt? I agree with you that for a larger corporation it makes much more sense for the tool to be prepaid and for the developers to not worry about their usage. There's definitely a learning curve to switching to a new style of debugging, even if you're accustomed to rr, and so encouraging use is more likely to get people to actually get to the point where they're benefiting from the tool. But that's a tough sales cycle, they've tried it, they've said they would be happy if someone came asking for that. You could make the argument that they should hire an enterprise sales person or something, which may be true but also has obvious costs. (Let's just say that it would be a large percentage increase over their current staffing.) I'd think that their reality wins out over your "shoulds". You may be right that they're incompetent about selling into the enterprise, but I'm certain that they've had more experience trying to sell Pernosco to larger corporations than you have and so I'm not going to ding them for this additional purchasing opportunity. Source: I work for a mid-sized company that does have a contract that gives me unlimited access to Pernosco. It's quite useful. I don't know the exact terms. I know I have solved some bugs that I would have been unlikely to ever fix and would be ongoing stability and maintenance concerns. I'm working on a bug right now where the submitter is able to reproduce easily but I am not. They have managed to capture it in an rr recording, so I'm uploading it to Pernosco to take a look even though my desktop's CPU is mismatched enough that I can't just run the recording myself. From past experience, if I have it in Pernosco, I will be able to fix it, even though the root cause is far far removed from the resulting crash. (It's subtle GC corruption.)
- adamnemecek 6y ago
  OK my workflow might be run, debug, change code, recompile, run, debug... Like many many times over. I can't imagine having to pay every time I use the debugger. Like it's weird that you pay per submission.
  - roca 6y ago
    Yeah, Pernosco may not be a great fit when you have to do that. That's not just due to the pricing model --- which we can change! --- but the entire debugging approach, where we do a lot of analysis up front, speeding the debugging experience but adding latency to the workflow. Now, sometimes people are doing that because to track down a bug they have to go through many cycles of adding instrumentation or logging to their code. In many cases Pernosco can just make those cycles go away, e.g. using https://pernos.co/about/expressions https://pernos.co/about/expressions. However, if you develop new code using cycles of "add new code, compile, debug", and each of those debugging sessions are short, Pernosco isn't a great fit for that currently.
- JustFinishedBSG 6y ago
  Do you realize how insane things would be if everything was charged as a percentage of value provided ? So you charge, 1/4 of the value given. Should my IDE charge me 1/4 too? My ISP ? My computer OEM ? My electricity provider ? My renter ? My OS ? That's a lot of 1/4th.
- shawnz 6y ago
  Are you saying that the average session saves the developer 15 minutes, or the average successful session, in which the bug is actually solved, saves 15 minutes? How likely is it that a given bug will be solved with Pernosco? What do I do if I pay and only realize after that the session wasn't helpful?
  - roca 6y ago
    The average session. Of course it's impossible to know the actual average number for any given developer but we often get developers estimating Pernosco saved hours, occasionally days, or occasionally the bug couldn't have been solved any other way. So I just tried to make a conservative guess. > How likely is it that a given bug will be solved with Pernosco? There's no way to put a number on this but I think it's unusual to not eventually figure it out with rr or Pernosco (but Pernosco will be faster) if conditions under the user's control are met: the recording really captures the bug you care about, it's in code in a language we support, and the debuginfo and sources are right. > What do I do if I pay and only realize after that the session wasn't helpful? "Sorry". Hopefully the reasons it wasn't helpful are mostly under your control and the benefits average out in your favour.
emmanueloga_ 6y ago
"To use Pernosco, you must consent to collection, transfer, storage and use of your data" I understand the need to monetize but honestly seems like a huge commitment in terms of both trust and money to use a tool like this. Perhaps a nagging screen model would work better? (better => more users using the tool, a percentage of which would def. pay to remove the nag and support the creator/s). A list of successful paid software I can think of from the top of my head: * https://www.reaper.fm/ https://www.reaper.fm/ (nagging screen) * https://www.linqpad.net/ https://www.linqpad.net/ (PRO versions with additional features) * https://www.jetbrains.com/idea https://www.jetbrains.com/idea ("Community" and "Professional" versions) * https://cursive-ide.com/ https://cursive-ide.com/ (30 day trial? I think? Personal, Commercial and Non Commercial licenses avail.) * https://www.sublimetext.com/ https://www.sublimetext.com/ (no limitations but purchase is expected, not enforced) * https://github.com/cognitect-labs/REBL-distro https://github.com/cognitect-labs/REBL-distro (used to _require_ joining a Patreon to use the software, but did not enforce it... Also I think it used to have commercial licenses available).
- db48x 6y ago
  It's definitely a factor worth considering before purchasing the service. It's going to record an entire program running on your local machine, including all the data that it reads and writes. That includes local files, network connections, events from your window manager, pixels put on the screen, any environment variables that happened to be defined, etc. If any of that is private then you shouldn't upload the recording to Pernosco. If you're working on a public GitHub repository then your concern is pretty much limited only to your local environment. Make sure you don't have security tokens in your environment variables when you record. If your program opens files, make sure it opens something harmless. If your program visits webpages, don't record it visiting a page you want kept private. It's only natural then that the EULA would want you to give Pernosco permission to receive that data, store it for you, and then present it back to you during your debugging sessions. There's also the on-premises option.
  - roca 6y ago
    Note that the pernosco-submit script supports detecting "sensitive" environment variables leaking into the recording: https://github.com/Pernosco/pernosco-submit/blob/a6119a1b291487dd056c38cfeb4280604358ca12/pernosco-submit#L142 https://github.com/Pernosco/pernosco-submit/blob/a6119a1b291... And we accept PRs :-)
db48x 6y ago
Worth it. I'd prefer to pay once and then install it on my own computer, even though that would surely cost several thousand dollars. It would still be worth it.
- sanxiyn 6y ago
  I am not sure I would want to self-host this. It does a lot of analysis on the cloud. https://robert.ocallahan.org/2019/07/auckland-rust-meetup-building.html https://robert.ocallahan.org/2019/07/auckland-rust-meetup-bu... gives the scale of "100s of GB of data produced saturating 36 cores for hour", now that's a heavy compute.
  - db48x 6y ago
    Cool, I hadn't seen these numbers before. I have the disk space, but not the cpus. Do you think I could get it down to just 17 minutes if I buy a dual-Epyc machine to run Pernosco on? I'm asking because I've been looking for an excuse to buy a dual-Epyc machine for a while now…
deleted 6y ago
[deleted]

schaefer 6y ago

The video references and recreates a debug session which is originally from a great blog post by Brendan Gregg: “gdb Debugging Full Example” [1]. This original blog post uses only free tools to reach the same conclusions as the Pernosco session. So it’s interesting review and compare them both. [1] http://www.brendangregg.com/blog/2016-08-09/gdb-example-ncurses.html http://www.brendangregg.com/blog/2016-08-09/gdb-example-ncur...

db48x 6y ago
Also, that Pernosco session is public so anyone can try their hand at it: https://pernos.co/debug/e9UFXkGq__8KQB-N6n59lA/index.html https://pernos.co/debug/e9UFXkGq__8KQB-N6n59lA/index.html

bigcorp-slave 6y ago

I’ve been an extensive user of time travel debugging for most of my career, and Pernosco looks great. However, please, please change your business model. As is, it is useless for employees of corporations with sane legal departments.

roca 6y ago
We would love to talk about on-prem with those corporations --- contact us!
malux85 6y ago
What’s the best time travel debugger you’ve ever seen? (any language)

merricksb 6y ago

For the curious, a small discussion about this from last year: https://news.ycombinator.com/item?id=21352380 https://news.ycombinator.com/item?id=21352380

eps 6y ago

Is there a way to compare two sessions? Basically, do a diff and see where the execution flow differed between two sessions? At high level first, then drilling down.

db48x 6y ago
They list that feature as a TODO for the future at https://pernos.co/about/vision/#comparing-executions https://pernos.co/about/vision/#comparing-executions
- eps 6y ago
  If it's done, it will allow cracking software that uses basic protection methods for $10 flat. But for the normal use both the pricing and its off-prem nature renders the whole thing all but useless. A pity really, because the tech looks fantastic.
  - db48x 6y ago
    I agree that comparing different runs of the same program will be amazing. I think that running Pernosco off-premises will be acceptable for many people. I use it a few times per month while working on open-source projects like Reposurgeon and Flex, so I'm not revealing any trade secrets by uploading those recordings. I've learned that the database built by Pernosco from a large recording can be 100GB and take an hour to generate on a 36-core AWS instance. $4 per debugging session is a pretty reasonable price compared to buying that hardware myself. And they are offering an on-premises solution, running on your own hardware. For a tool this good, the price for that is bound to be large enough to give individuals like us reason to think twice, and then probably a third time. Given the potential time savings, I think that it should be easy for even a small company to justify the purchase. On the gripping hand, this is non-Free software. Some will avoid it primarily for that reason, and I can't really fault them for sticking to their principles. I think that it's entirely possible that Pernosco, or debuggers like it, could effectively eliminate traditional debuggers like GDB. GDB would still exist and still work, but without the nearly-omniscient viewpoint that Pernosco offers very few will bother to use it. It's like looking through a keyhole. I've thought about this a lot over the last few months.

db48x 6y ago

Out of curiosity, has anyone else noticed that the videos and screenshots on the About page are not actually videos or even images? They're iframes with the actual Pernosco UI inside them! That is so cool.

KenoFischer 6y ago

We use rr extensively for Julia development [1] (for a sense of scale, I probably look at hundreds of rr traces a month). There's two big challenges for us there 1) Many cloud environments are broken in various ways for rr recording - I spend way too much time guiding our users to the few that do work 2) The GDB frontend really isn't the best way to view all the underlying information Pernosco is well on its way to address the second part of this. Its replayer is quite a bit nicer than the open source rr one and I have high hopes for the platform in general. For the first one, all I can do is plead with the cloud vendors not to mess up the environment. Since this is HN, if you're working for a cloud provider, please ask your engineering team not to break rr. Otherwise I'll have to spend time migrating our users off your platform, which will be fun for neither of us. Anyway, congrats to Robert and Kyle on the launch. Wish you all the best. [1] https://julialang.org/blog/2020/05/rr/ https://julialang.org/blog/2020/05/rr/

solarengineer 6y ago
Could you share examples of how cloud providers break rr?
- roca 6y ago
  Usually they break rr by either blocking access to the hardware PMU entirely (e.g. on AWS, any instance that shares a CPU socket with other instances; on Azure/GCP, everywhere AFAIK), or by introducing subtle errors into the virtualized performance counter values that rr depends on (e.g. VMWare: https://robert.ocallahan.org/2015/11/rr-in-vmware-solved.html https://robert.ocallahan.org/2015/11/rr-in-vmware-solved.htm..., Xen: https://lists.xen.org/archives/html/xen-devel/2017-07/msg02242.html https://lists.xen.org/archives/html/xen-devel/2017-07/msg022...). Another feature rr benefits from is CPUID faulting, which is not virtualized by any of the major cloud providers (though Linux KVM does virtualize it): https://patchwork.kernel.org/project/kvm/patch/20161101013749.4003-8-khuey@kylehuey.com/ https://patchwork.kernel.org/project/kvm/patch/2016110101374.... That'd be "nice to have" but we can live without it.
mwcampbell 6y ago
Can you be more specific about which cloud environments do work with rr? In your blog post you said it works on Amazon EC2 instances above a certain size. Which instance types are known to work?

4sak3n 6y ago

Amazing name. Has my Latin nerd stamp of approval.

minetest2048 6y ago

Is it possible to use something like rr for microcontrollers, assuming we can simulate the microcontroller?

roca 6y ago
Yes, you can instrument simulators to get record-and-replay. Simics has a record-and-replay feature like that, so does QEMU lately. I don't know of anything specifically for microcontrollers but it certainly could be done.