8 ms·

Symbols the Microsoft Way (2013)

45 points by harporoeder 6y ago

adzm 6y ago
All that said, in the end you just set the sympath and everything ends up working. It's incredibly refreshing and easy.
the_mitsuhiko 6y ago
I really, really, really wish other platforms would embrace the same concept. We (Sentry) have added support for the same concept for other platforms but sadly there is not a lot of uptick in the general Linux community to start producing indexed debug symbols repositories. Here are all the formats for symbol servers we are currently support for Linux/Mac and Windows: https://getsentry.github.io/symbolicator/advanced/symbol-server-compatibility/ https://getsentry.github.io/symbolicator/advanced/symbol-ser...
- glandium 6y ago
  Thankfully, eventually, enough people will be running debuginfod. https://developers.redhat.com/blog/2019/10/14/introducing-debuginfod-the-elfutils-debuginfo-server/ https://developers.redhat.com/blog/2019/10/14/introducing-de...
  - the_mitsuhiko 6y ago
    I really hope so! I added support for it (https://github.com/getsentry/symbolicator/pull/142 https://github.com/getsentry/symbolicator/pull/142) the moment I saw the announcement :)
layer8 6y ago
For those wondering, “symbols” refers to auxiliary debugging information for binaries: https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/symbols-and-symbol-files https://docs.microsoft.com/en-us/windows-hardware/drivers/de...
luxuryballs 6y ago
This was oddly nostalgic. Reminds me of all the time I spent tinkering with Windows 98 as a kid perhaps.
muststopmyths 6y ago
Another related capability you can use when indexing PDBs for a build is the "source server". It tags the pdb with source control commands to retrieve the source revision at the time of building. Enable the feature in the debugger and you don't have to manually retrieve the right source code when you get a crash dump from a customer for an old version of your software. It embeds executable source control commands in the PDB so you need to make sure you know where your PDBs come from. Reference:https://docs.microsoft.com/en-us/windows/win32/debug/source-server-and-source-indexing https://docs.microsoft.com/en-us/windows/win32/debug/source-...
- brucedawson 6y ago
  Oh absolutely. Source-server/source-indexing is amazing. I use it dozens of times a week when debugging Chrome crash dumps. It's amazing when the correct version of the relevant source code just magically appears. I've blogged about this as well, of course: https://randomascii.wordpress.com/2011/11/11/source-indexing-is-underused-awesomeness/ https://randomascii.wordpress.com/2011/11/11/source-indexing... As you say, knowing the provenance of your PDBs is important. That's part of why I pushed hard for https symbol servers a few years back. Don't ever use http: for a symbol server.
saagarjha 6y ago
The posture towards symbols on different platforms is quite interesting to see: I think all major OSes strip their binaries, but Microsoft provides them as a separate download and Linux of course has *-dbg packages and the upcoming debuginfod. In comparison, macOS is truly an infuriating platform because Apple strips their code and recommends others do so to (it's done automatically as part of deployment postprocessing) for size improvements but because Apple and third party developers are sold on the FUD that providing symbols will ruin their business nobody ever provides them separately. (Aside: if you ship software and run any sort of stripping/minification on it and want to claim it is for the size benefits exclusively, you are honor bound to provide the information you removed, whether that is DWARF or a source map. No exceptions. Personally, I consider this important enough to not do these things at all, but I get it if you fall on the other side of the "size vs. inspectability" spectrum–you just need to also provide the stuff you throw away if you don't want to be annoying.) More on point, though, I can see both Windows and Linux getting the experience of finding debug symbols to be pretty good™–but for the same reasons I cannot see macOS getting this unless attitudes change. Apple of course runs this internally and "forgets" to share the scripts with us, and I have heard of other companies doing similar things, but I don't see any sort of unified effort in this area at all :(
- matheusmoreira 6y ago
  Does symbol stripping actually improve performance though? Or just file size? I'd love it if all my packages came with their symbols by default. Helps a lot when using a debugger. I also just like exploring the binaries.
  - glandium 6y ago
    It improves neither on mac, because debug info is in a separate .dSYM directory. Edit: stripping local symbols does save a little file size, but barely gives a mapping of addresses to symbol names, not actual debug info.
  - Wowfunhappy 6y ago
    Presumably, decreasing the file size of an executable has at least some minimal impact on memory usage while running, no?
jiggawatts 6y ago
Having worked with Citrix a lot, it was always nice that both Microsoft and Citrix provide symbol servers. Despite not having any kernel debugging experience at all, it was always worthwhile setting up WinDbg with those two symbol paths to do some drive-by diagnostics of any blue-screen crash dumps. More than half the time this would instantly identify the "culprit".
MarkSweep 6y ago
One development since that article was written is Source Link: https://docs.microsoft.com/en-us/dotnet/standard/library-guidance/sourcelink https://docs.microsoft.com/en-us/dotnet/standard/library-gui... It makes it easy to publish symbols for you .NET nuget packages. When debugging, Visual Studio will automatically download these symbols and the source code from GitHub, so you can easily step through third party code. There is also support for C++

45 points by harporoeder 6y ago

adzm 6y ago

All that said, in the end you just set the sympath and everything ends up working. It's incredibly refreshing and easy.

I really, really, really wish other platforms would embrace the same concept. We (Sentry) have added support for the same concept for other platforms but sadly there is not a lot of uptick in the general Linux community to start producing indexed debug symbols repositories. Here are all the formats for symbol servers we are currently support for Linux/Mac and Windows: https://getsentry.github.io/symbolicator/advanced/symbol-server-compatibility/ https://getsentry.github.io/symbolicator/advanced/symbol-ser...

glandium 6y ago
Thankfully, eventually, enough people will be running debuginfod. https://developers.redhat.com/blog/2019/10/14/introducing-debuginfod-the-elfutils-debuginfo-server/ https://developers.redhat.com/blog/2019/10/14/introducing-de...
- the_mitsuhiko 6y ago
  I really hope so! I added support for it (https://github.com/getsentry/symbolicator/pull/142 https://github.com/getsentry/symbolicator/pull/142) the moment I saw the announcement :)

layer8 6y ago

For those wondering, “symbols” refers to auxiliary debugging information for binaries: https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/symbols-and-symbol-files https://docs.microsoft.com/en-us/windows-hardware/drivers/de...

luxuryballs 6y ago

This was oddly nostalgic. Reminds me of all the time I spent tinkering with Windows 98 as a kid perhaps.

muststopmyths 6y ago

Another related capability you can use when indexing PDBs for a build is the "source server". It tags the pdb with source control commands to retrieve the source revision at the time of building. Enable the feature in the debugger and you don't have to manually retrieve the right source code when you get a crash dump from a customer for an old version of your software. It embeds executable source control commands in the PDB so you need to make sure you know where your PDBs come from. Reference:https://docs.microsoft.com/en-us/windows/win32/debug/source-server-and-source-indexing https://docs.microsoft.com/en-us/windows/win32/debug/source-...

brucedawson 6y ago
Oh absolutely. Source-server/source-indexing is amazing. I use it dozens of times a week when debugging Chrome crash dumps. It's amazing when the correct version of the relevant source code just magically appears. I've blogged about this as well, of course: https://randomascii.wordpress.com/2011/11/11/source-indexing-is-underused-awesomeness/ https://randomascii.wordpress.com/2011/11/11/source-indexing... As you say, knowing the provenance of your PDBs is important. That's part of why I pushed hard for https symbol servers a few years back. Don't ever use http: for a symbol server.

saagarjha 6y ago

The posture towards symbols on different platforms is quite interesting to see: I think all major OSes strip their binaries, but Microsoft provides them as a separate download and Linux of course has *-dbg packages and the upcoming debuginfod. In comparison, macOS is truly an infuriating platform because Apple strips their code and recommends others do so to (it's done automatically as part of deployment postprocessing) for size improvements but because Apple and third party developers are sold on the FUD that providing symbols will ruin their business nobody ever provides them separately. (Aside: if you ship software and run any sort of stripping/minification on it and want to claim it is for the size benefits exclusively, you are honor bound to provide the information you removed, whether that is DWARF or a source map. No exceptions. Personally, I consider this important enough to not do these things at all, but I get it if you fall on the other side of the "size vs. inspectability" spectrum–you just need to also provide the stuff you throw away if you don't want to be annoying.) More on point, though, I can see both Windows and Linux getting the experience of finding debug symbols to be pretty good™–but for the same reasons I cannot see macOS getting this unless attitudes change. Apple of course runs this internally and "forgets" to share the scripts with us, and I have heard of other companies doing similar things, but I don't see any sort of unified effort in this area at all :(

matheusmoreira 6y ago
Does symbol stripping actually improve performance though? Or just file size? I'd love it if all my packages came with their symbols by default. Helps a lot when using a debugger. I also just like exploring the binaries.
- glandium 6y ago
  It improves neither on mac, because debug info is in a separate .dSYM directory. Edit: stripping local symbols does save a little file size, but barely gives a mapping of addresses to symbol names, not actual debug info.
- Wowfunhappy 6y ago
  Presumably, decreasing the file size of an executable has at least some minimal impact on memory usage while running, no?

jiggawatts 6y ago

Having worked with Citrix a lot, it was always nice that both Microsoft and Citrix provide symbol servers. Despite not having any kernel debugging experience at all, it was always worthwhile setting up WinDbg with those two symbol paths to do some drive-by diagnostics of any blue-screen crash dumps. More than half the time this would instantly identify the "culprit".

MarkSweep 6y ago

One development since that article was written is Source Link: https://docs.microsoft.com/en-us/dotnet/standard/library-guidance/sourcelink https://docs.microsoft.com/en-us/dotnet/standard/library-gui... It makes it easy to publish symbols for you .NET nuget packages. When debugging, Visual Studio will automatically download these symbols and the source code from GitHub, so you can easily step through third party code. There is also support for C++