7 ms·

Sometimes, it is a compiler bug (2022)

103 points by goranmoomin 3y ago

jeffreygoesto 3y ago
Kudos for the extreme perseverence and skill to handle, understand and explain all this! Learned about some seldom visited corners...
lionkor 3y ago
That was a great read! I was not aware you could write extensions in C++, that looks like something I should look into
akoboldfrying 3y ago
Very tenacious debugging! I'm honestly a little surprised that someone would stream this -- so much potential for egg on face for the debugger, and had it been me, much time would have been spent scratching my head, which I don't think would make for compelling viewing. Bonus snark: Fractional line numbers sounds like something Larry Wall would add to Perl, deliberately, as a "feature".
momos 3y ago
As an undergraduate currently still studying, how do you get to the point of being able to do things like this? He seems like a wizard to me...
- jan_Sate 3y ago
  Even if you're able to do things like this, chances are that it isn't a compiler bug when you encounter issues like this one.
- HPsquared 3y ago
  Do students get much practice in debugging and root cause analysis on existing systems? That's something people do a lot of at work, but perhaps not students so much. I'd imagine students work more in a clean "green-field" environment. Kind of like experience gained as a plumber working with dirty clogged real pipes in a basement, vs doing exercises in plumbing school with new pipes and theory about pipe sizes, gradients etc. In other words, school can only give part of the picture.
  - tux3 3y ago
    They get to debug their own code. That's different from debugging a large multi MLOC system, but they're also students, you have to start small.
    - pjmlp 3y ago
      You can get to do that at CERN as summer student, or similar research institute, depending on the luck which department one lands on. And great parties as well.
  - denton-scratch 3y ago
    > Do students get much practice in debugging and root cause analysis on existing systems? It's a waste of a student's time. Deep debugging isn't just one set of skills; each problem is different. It's usually very time-consuming, and you will need to learn new skills and tools. That is: you have to encounter a problem that is a blocker, so that your motivation is that you have to solve the problem. My experience was a long time ago: I had linked a library compiled with Borland C with code compiled with Microsoft C. It wouldn't work. I wss only using one function from the Borland library; that's where the error was occurring. It turned out that the Microsoft compiler required the callee to restore the flag register; the Borland compiler required the caller to do that. Therefore the carry bit wasn't being restored correctly, causing the bug. Took several days to figure out.
- FartyMcFarter 3y ago
  You get there by writing and fixing a lot of your own bugs. Also reading other people's codebases to learn how to navigate large codebases. That's how I got there at least...
- blinkingled 3y ago
  Short answer is start doing things. Somewhat longer answer - develop interest in understanding deeper level things because when things seem hopeless it's the interest/curiosity that will keep you going. Secondly find things to break and fix - this will give you opportunities to learn wider range of things.
- Smaug123 3y ago
  Computers aren't magic, much as they seem to be. In general, if you can reproduce a bug reliably, it is always possible to keep digging until you find the cause. It may take more and more esoteric tools to do so, and you may need to start breaking apart the components you're working with, but at least ask yourself the question "what is preventing me from examining what's going on here" and try to find out how to look inside whatever opaque box you've run up against. Those boxes will start out being like "the function I just wrote", but as your understanding increases and you become familiar with how to look inside the least opaque boxes, the hurdles become more like "the JavaScript runtime", "the C compiler", "the syscalls made to the operating system" or even "the microcode emitted by the CPU" (though I've never needed to go to that level myself!).
  - saagarjha 3y ago
    Nit: CPUs don't typically emit microcode; they are programmed in microcode.
    - colejohnson66 3y ago
      Further nit: Modern x86 CPUs use PLAs to generate microops ("uops") that feed the execution engine. For complex instructions (like CPUID, RDMSR, VM-related, etc.), the microcode ("ucode", which can be updated) generates microops. The execution engine works in discrete[a] operations, hence uops. The microcode is a sequencer that tears apart ISA instructions into those uops. [a]: I'm considering fused operations (like CMP+Jcc) as single "operations" for simplicity.
    - Smaug123 3y ago
      (facepalm, thanks - I have the right structures in my head but not the right words)
- wrsh07 3y ago
  If you're writing a language it supports, use godbolt - https://godbolt.org/ https://godbolt.org/. It's great for settling arguments (do these two versions compile to the same code? If not figure out why! Make sure you're compiling with optimizations) Spend some time learning gdb and other classic cli tools. Don't be afraid of reading books!
- saagarjha 3y ago
  You can do it now! There's nothing magic in what they did. They just went step-by-step trying things out until they found the bug.
- GuB-42 3y ago
  By spending a lot of time on the problem and not panicking, essentially. It you don't rush and focus on making progress, you will find it eventually. There is no wizardry involved, skills will help you find the solution faster, but that's your ability to focus and keep your calm that will get to it in the first place. If you are at work, ignore your boss trying to pressure you (that's not necessary easy), your boss only has the "right" to tell you if you should work or the problem or not, if you are asked to work on the problem, do it as if you have your entire life to do it. With experience, you can make estimates that can help decide if the problem is worth trying to solve or not, but with the correct mindset, one doesn't need experience to solve the problem itself, instead one builds experience by solving the problem.
- HankB99 3y ago
  I learned more about the details of how computer worked in a class on programming an Intel 8080 (on a CP/M system with dual 8" floppy drives) than any other course. Computer science majors were either nascent or non-existent at that time so I don't know what is available today. And microprocessors are a lot more complex than the 8080. Nevertheless, I recommend learning the details of computer architecture by learning some assembly programming. Higher level languages are an abstraction meant to shield the developer from the details of the H/W but it remains useful to understand the details, particularly if the abstraction is broken.
- aardvark179 3y ago
  I personally got there by treating bugs as problems I could try and ask questions about until I had a smaller more precise bug. So either I work out something to ask and answer about the nature of some output, or I try to ask and answer something that helps me more precisely pin down the bug. Sometimes that can be turning an intermittent problem into something I can reproduce consistently, and sometimes it can be constructing new test cases and seeing if their results match my hypothesis. It helps to either have a good memory, or to keep notes so you know what you’ve already explored and tested, and it’s something you get better at with practice. I spent several years on third line product support working out what was going wrong for customers, producing hot fixes, and working with devs to turn those into proper fixes in the next version.
deleted 3y ago
[deleted]
asicsp 3y ago
Previous discussion: https://news.ycombinator.com/item?id=31515982 https://news.ycombinator.com/item?id=31515982 (96 points | May 26, 2022 | 56 comments)
brancz 3y ago
I happen to work on profiling tools where we rely on compilers to output metadata about the machine executable code correctly. It turns out this is surprisingly hard and goes wrong quite often, and sometimes it's even impossible to maintain correctness across multiple rounds of compiler optimizations. For us the way compilers solve this is basically the worst possible scenario, when the compiler can't correctly track it across optimizations, it just starts emitting less or sometimes even no information. These bugs are just awful to understand. At the end of the day compilers are also just software and it's just a matter of how many features you end up being exposed to.
- IshKebab 3y ago
  The sort of thing profilers and debuggers want doesn't really exist at high optimisation levels. In my experience that kind of thing isn't exactly a bug, it's just that the information you want can't even exist in theory. Kind of the same thing as reporting memory usage for programs. You want to be able to say "A uses 3MB, B uses 2MB", but then there's shared memory, file caches, fragmentation, etc. So much complication that "how much memory is this process using?" isn't even a valid question anymore.
  - compiler-guy 3y ago
    Debug info under high optimization is very tricky. A simple example: for (i = 0; i < 4; i++) ... If the compiler unrolls the loop four times the eliminates the induction variable altogether, then combines the four iteration loop into a single vector instruction, what debug info should it emit for the value of i in the loop body? You can argue that it should emit what i would have had at a given point, but now you have four different values for i all at that one vector instruction. Reasoning about that is quite difficult.
    - rcxdude 3y ago
      True, but I would be happy if at least the debugger could point me at the value which is clearly being iterated over in a register, or tell me the value of a standard ABI function's arguments (and no, it hasn't been overwritten by something else).
wheybags 3y ago
A better workaround would be to call __HrLoadAllImportsForDll[1]. Delay loading lets you defer grabbing function pointers from a dll (normally until the first call of that function). But if you call __HrLoadAllImportsForDll from the addon's init code, you can attempt to load them all in one go. That way it doesn't happen during a call with a float argument, so it shouldn't cause problems. 1: https://learn.microsoft.com/en-us/cpp/build/reference/linker-support-for-delay-loaded-dlls?view=msvc-170#load-all-imports-for-a-delay-loaded-dll https://learn.microsoft.com/en-us/cpp/build/reference/linker...
- Const-me 3y ago
  The bug is specific to Win64 builds made with GCC compiler. I’m pretty sure VC++ doesn’t have that bug.
jheriko 3y ago
i found my first compiler bug in my first job in my first few weeks... nobody was keen to believe me except the ms support guy i eventually got in touch with - who was kindly able to tell me that it was caught and fixed in a later version of the compiler that we couldn't use, but then at least gave me the clout to get the issue worked around instead of ignored as the fantasy of a junior programmer.
spandextwins 3y ago
Oh I remember Microport Unix from back in the day. It was pretty much always either a compiler big or an operating system bug. Same with Windows 3.11
deleted 3y ago
[deleted]
coldtea 3y ago
With more recent, still developing languages, it's quite more common too.
wolf550e 3y ago
Is this another one of the endless bugs caused by Microsoft choosing a different ABI for amd64/x86-64/x64 than the unix ABI for amd64/x86-64/x64?
mananaysiempre 3y ago
> $ make install PREFIX=/Projects/binutils-gdb/stragerusr > After a minute, I realized that binutils was being installed in /ucrt64, not into my temporary directory. Oh no! Tip: to install Autotools- or otherwise GNU-Coding-Standards-compliant projects (e.g. CMake-based ones using GNUInstallDirs) into places you haven’t configured them to go to, on the hope they will kinda sorta work, use DESTDIR[1]. The value of PREFIX (et al.) is set at configuration time, and overriding it via make does not work, while DESTDIR is there specifically to be overridden. It’s what Linux package managers use to create a parallel bin/, lib/, usr/, etc. hierarchy that’s then archived up into a package. (The value of DESTDIR is prepended to all destination paths, it does not override them.) Also, support DESTDIR in your build systems. Distro maintainers will thank you. [1] https://www.gnu.org/prep/standards/html_node/DESTDIR.html https://www.gnu.org/prep/standards/html_node/DESTDIR.html
ramshanker 3y ago
It’s a dark art. Or that old saying…. Any science well done, is as good as magic.
- teddyh 3y ago
  “Any technology, no matter how primitive, is magic to those who don’t understand it.” — <http://freefall.purrsia.com/ff300/fv00255.htm http://freefall.purrsia.com/ff300/fv00255.htm>
benced 3y ago
My general philosophy is the compiler or runtime will be to blame about once or twice a career, the library will be to blame a couple times a year (obviously varies with library and language - Node developers find a lot more library bugs because JS) and it’s your fault the rest of the time.
- DylanSp 3y ago
  I've only been developing a few years, and I've already found one compiler bug, in Typescript's type-checker [1]. It's a bit easier to find compiler bugs when new features are getting added pretty frequently. [1] https://github.com/microsoft/TypeScript/issues/35970 https://github.com/microsoft/TypeScript/issues/35970
Sarkie 3y ago
Fantastic full article!
teddyh 3y ago
To be specific, it was a bug in “dlltool”, a part of GNU Binutils used on Windows platforms. The bug was fixed in May 25, 2023, and released in Binutils 2.41 on July 30 2023.
jonasba 3y ago
Oh my god. I have been chasing a null value error in our profiling SDK for the past 3 days. Calling napi_create_double was for some reason always generating a null value for the first entry and I tried everything to reproduce it thinking I have a bug in my code. Thank you, you saved me!
- WirelessGigabit 3y ago
  See if you can update to binutils 2.41. That one contains the bugfix.
torstenvl 3y ago
Not exactly a compiler bug, but Borland Turbo C (which predates ANSI C) had bizarre semantics around character pointers such that char *s was almost a managed string type, and assignment would make a deep copy of the string.... much to the surprise of anyone expecting identical addresses. The manual maybe (?) hints at this behavior: > Using a Character Pointer > The second method you can use to define strings is a character pointer. Edit your program to look like this: main () { char *msg; msg = "Hello, world"; puts(msg); } > The asterisk (*) in front of msg tells the compiler that msg is a pointer to a character; in other words, msg can hold the address of some character. However, the compiler sets aside no space to store characters and does not initialize msg to any particular value. > When the compiler finds the statement msg = "Hello, world\n"; it does two things: - As before, it creates the string "Hello, world\n", followed by a null character somewhere within the object code file. - It assigns to the starting address of that string-the address of the character H-to msg. > The command puts(msg) works just as it did before, printing characters until it encounters the null character. The workaround is of course to use s1 = &s2[0]

103 points by goranmoomin 3y ago

Kudos for the extreme perseverence and skill to handle, understand and explain all this! Learned about some seldom visited corners...

lionkor 3y ago

That was a great read! I was not aware you could write extensions in C++, that looks like something I should look into

akoboldfrying 3y ago

Very tenacious debugging! I'm honestly a little surprised that someone would stream this -- so much potential for egg on face for the debugger, and had it been me, much time would have been spent scratching my head, which I don't think would make for compelling viewing. Bonus snark: Fractional line numbers sounds like something Larry Wall would add to Perl, deliberately, as a "feature".

momos 3y ago

As an undergraduate currently still studying, how do you get to the point of being able to do things like this? He seems like a wizard to me...

jan_Sate 3y ago
Even if you're able to do things like this, chances are that it isn't a compiler bug when you encounter issues like this one.
HPsquared 3y ago
Do students get much practice in debugging and root cause analysis on existing systems? That's something people do a lot of at work, but perhaps not students so much. I'd imagine students work more in a clean "green-field" environment. Kind of like experience gained as a plumber working with dirty clogged real pipes in a basement, vs doing exercises in plumbing school with new pipes and theory about pipe sizes, gradients etc. In other words, school can only give part of the picture.
- tux3 3y ago
  They get to debug their own code. That's different from debugging a large multi MLOC system, but they're also students, you have to start small.
  - pjmlp 3y ago
    You can get to do that at CERN as summer student, or similar research institute, depending on the luck which department one lands on. And great parties as well.
- denton-scratch 3y ago
  > Do students get much practice in debugging and root cause analysis on existing systems? It's a waste of a student's time. Deep debugging isn't just one set of skills; each problem is different. It's usually very time-consuming, and you will need to learn new skills and tools. That is: you have to encounter a problem that is a blocker, so that your motivation is that you have to solve the problem. My experience was a long time ago: I had linked a library compiled with Borland C with code compiled with Microsoft C. It wouldn't work. I wss only using one function from the Borland library; that's where the error was occurring. It turned out that the Microsoft compiler required the callee to restore the flag register; the Borland compiler required the caller to do that. Therefore the carry bit wasn't being restored correctly, causing the bug. Took several days to figure out.
FartyMcFarter 3y ago
You get there by writing and fixing a lot of your own bugs. Also reading other people's codebases to learn how to navigate large codebases. That's how I got there at least...
blinkingled 3y ago
Short answer is start doing things. Somewhat longer answer - develop interest in understanding deeper level things because when things seem hopeless it's the interest/curiosity that will keep you going. Secondly find things to break and fix - this will give you opportunities to learn wider range of things.
Smaug123 3y ago
Computers aren't magic, much as they seem to be. In general, if you can reproduce a bug reliably, it is always possible to keep digging until you find the cause. It may take more and more esoteric tools to do so, and you may need to start breaking apart the components you're working with, but at least ask yourself the question "what is preventing me from examining what's going on here" and try to find out how to look inside whatever opaque box you've run up against. Those boxes will start out being like "the function I just wrote", but as your understanding increases and you become familiar with how to look inside the least opaque boxes, the hurdles become more like "the JavaScript runtime", "the C compiler", "the syscalls made to the operating system" or even "the microcode emitted by the CPU" (though I've never needed to go to that level myself!).
- saagarjha 3y ago
  Nit: CPUs don't typically emit microcode; they are programmed in microcode.
  - colejohnson66 3y ago
    Further nit: Modern x86 CPUs use PLAs to generate microops ("uops") that feed the execution engine. For complex instructions (like CPUID, RDMSR, VM-related, etc.), the microcode ("ucode", which can be updated) generates microops. The execution engine works in discrete[a] operations, hence uops. The microcode is a sequencer that tears apart ISA instructions into those uops. [a]: I'm considering fused operations (like CMP+Jcc) as single "operations" for simplicity.
  - Smaug123 3y ago
    (facepalm, thanks - I have the right structures in my head but not the right words)
wrsh07 3y ago
If you're writing a language it supports, use godbolt - https://godbolt.org/ https://godbolt.org/. It's great for settling arguments (do these two versions compile to the same code? If not figure out why! Make sure you're compiling with optimizations) Spend some time learning gdb and other classic cli tools. Don't be afraid of reading books!
saagarjha 3y ago
You can do it now! There's nothing magic in what they did. They just went step-by-step trying things out until they found the bug.
GuB-42 3y ago
By spending a lot of time on the problem and not panicking, essentially. It you don't rush and focus on making progress, you will find it eventually. There is no wizardry involved, skills will help you find the solution faster, but that's your ability to focus and keep your calm that will get to it in the first place. If you are at work, ignore your boss trying to pressure you (that's not necessary easy), your boss only has the "right" to tell you if you should work or the problem or not, if you are asked to work on the problem, do it as if you have your entire life to do it. With experience, you can make estimates that can help decide if the problem is worth trying to solve or not, but with the correct mindset, one doesn't need experience to solve the problem itself, instead one builds experience by solving the problem.
HankB99 3y ago
I learned more about the details of how computer worked in a class on programming an Intel 8080 (on a CP/M system with dual 8" floppy drives) than any other course. Computer science majors were either nascent or non-existent at that time so I don't know what is available today. And microprocessors are a lot more complex than the 8080. Nevertheless, I recommend learning the details of computer architecture by learning some assembly programming. Higher level languages are an abstraction meant to shield the developer from the details of the H/W but it remains useful to understand the details, particularly if the abstraction is broken.
aardvark179 3y ago
I personally got there by treating bugs as problems I could try and ask questions about until I had a smaller more precise bug. So either I work out something to ask and answer about the nature of some output, or I try to ask and answer something that helps me more precisely pin down the bug. Sometimes that can be turning an intermittent problem into something I can reproduce consistently, and sometimes it can be constructing new test cases and seeing if their results match my hypothesis. It helps to either have a good memory, or to keep notes so you know what you’ve already explored and tested, and it’s something you get better at with practice. I spent several years on third line product support working out what was going wrong for customers, producing hot fixes, and working with devs to turn those into proper fixes in the next version.

deleted 3y ago

[deleted]

asicsp 3y ago

Previous discussion: https://news.ycombinator.com/item?id=31515982 https://news.ycombinator.com/item?id=31515982 (96 points | May 26, 2022 | 56 comments)

brancz 3y ago

I happen to work on profiling tools where we rely on compilers to output metadata about the machine executable code correctly. It turns out this is surprisingly hard and goes wrong quite often, and sometimes it's even impossible to maintain correctness across multiple rounds of compiler optimizations. For us the way compilers solve this is basically the worst possible scenario, when the compiler can't correctly track it across optimizations, it just starts emitting less or sometimes even no information. These bugs are just awful to understand. At the end of the day compilers are also just software and it's just a matter of how many features you end up being exposed to.

IshKebab 3y ago
The sort of thing profilers and debuggers want doesn't really exist at high optimisation levels. In my experience that kind of thing isn't exactly a bug, it's just that the information you want can't even exist in theory. Kind of the same thing as reporting memory usage for programs. You want to be able to say "A uses 3MB, B uses 2MB", but then there's shared memory, file caches, fragmentation, etc. So much complication that "how much memory is this process using?" isn't even a valid question anymore.
- compiler-guy 3y ago
  Debug info under high optimization is very tricky. A simple example: for (i = 0; i < 4; i++) ... If the compiler unrolls the loop four times the eliminates the induction variable altogether, then combines the four iteration loop into a single vector instruction, what debug info should it emit for the value of i in the loop body? You can argue that it should emit what i would have had at a given point, but now you have four different values for i all at that one vector instruction. Reasoning about that is quite difficult.
  - rcxdude 3y ago
    True, but I would be happy if at least the debugger could point me at the value which is clearly being iterated over in a register, or tell me the value of a standard ABI function's arguments (and no, it hasn't been overwritten by something else).

wheybags 3y ago

A better workaround would be to call __HrLoadAllImportsForDll[1]. Delay loading lets you defer grabbing function pointers from a dll (normally until the first call of that function). But if you call __HrLoadAllImportsForDll from the addon's init code, you can attempt to load them all in one go. That way it doesn't happen during a call with a float argument, so it shouldn't cause problems. 1: https://learn.microsoft.com/en-us/cpp/build/reference/linker-support-for-delay-loaded-dlls?view=msvc-170#load-all-imports-for-a-delay-loaded-dll https://learn.microsoft.com/en-us/cpp/build/reference/linker...

Const-me 3y ago
The bug is specific to Win64 builds made with GCC compiler. I’m pretty sure VC++ doesn’t have that bug.

jheriko 3y ago

i found my first compiler bug in my first job in my first few weeks... nobody was keen to believe me except the ms support guy i eventually got in touch with - who was kindly able to tell me that it was caught and fixed in a later version of the compiler that we couldn't use, but then at least gave me the clout to get the issue worked around instead of ignored as the fantasy of a junior programmer.

spandextwins 3y ago

Oh I remember Microport Unix from back in the day. It was pretty much always either a compiler big or an operating system bug. Same with Windows 3.11

deleted 3y ago

[deleted]

coldtea 3y ago

With more recent, still developing languages, it's quite more common too.

wolf550e 3y ago

Is this another one of the endless bugs caused by Microsoft choosing a different ABI for amd64/x86-64/x64 than the unix ABI for amd64/x86-64/x64?

mananaysiempre 3y ago

> $ make install PREFIX=/Projects/binutils-gdb/stragerusr > After a minute, I realized that binutils was being installed in /ucrt64, not into my temporary directory. Oh no! Tip: to install Autotools- or otherwise GNU-Coding-Standards-compliant projects (e.g. CMake-based ones using GNUInstallDirs) into places you haven’t configured them to go to, on the hope they will kinda sorta work, use DESTDIR[1]. The value of PREFIX (et al.) is set at configuration time, and overriding it via make does not work, while DESTDIR is there specifically to be overridden. It’s what Linux package managers use to create a parallel bin/, lib/, usr/, etc. hierarchy that’s then archived up into a package. (The value of DESTDIR is prepended to all destination paths, it does not override them.) Also, support DESTDIR in your build systems. Distro maintainers will thank you. [1] https://www.gnu.org/prep/standards/html_node/DESTDIR.html https://www.gnu.org/prep/standards/html_node/DESTDIR.html

ramshanker 3y ago

It’s a dark art. Or that old saying…. Any science well done, is as good as magic.

teddyh 3y ago
“Any technology, no matter how primitive, is magic to those who don’t understand it.” — <http://freefall.purrsia.com/ff300/fv00255.htm http://freefall.purrsia.com/ff300/fv00255.htm>

benced 3y ago

My general philosophy is the compiler or runtime will be to blame about once or twice a career, the library will be to blame a couple times a year (obviously varies with library and language - Node developers find a lot more library bugs because JS) and it’s your fault the rest of the time.

DylanSp 3y ago
I've only been developing a few years, and I've already found one compiler bug, in Typescript's type-checker [1]. It's a bit easier to find compiler bugs when new features are getting added pretty frequently. [1] https://github.com/microsoft/TypeScript/issues/35970 https://github.com/microsoft/TypeScript/issues/35970

Sarkie 3y ago

Fantastic full article!

teddyh 3y ago

To be specific, it was a bug in “dlltool”, a part of GNU Binutils used on Windows platforms. The bug was fixed in May 25, 2023, and released in Binutils 2.41 on July 30 2023.

jonasba 3y ago

Oh my god. I have been chasing a null value error in our profiling SDK for the past 3 days. Calling napi_create_double was for some reason always generating a null value for the first entry and I tried everything to reproduce it thinking I have a bug in my code. Thank you, you saved me!

WirelessGigabit 3y ago
See if you can update to binutils 2.41. That one contains the bugfix.

torstenvl 3y ago

Not exactly a compiler bug, but Borland Turbo C (which predates ANSI C) had bizarre semantics around character pointers such that char *s was almost a managed string type, and assignment would make a deep copy of the string.... much to the surprise of anyone expecting identical addresses. The manual maybe (?) hints at this behavior: > Using a Character Pointer > The second method you can use to define strings is a character pointer. Edit your program to look like this: main () { char *msg; msg = "Hello, world"; puts(msg); } > The asterisk (*) in front of msg tells the compiler that msg is a pointer to a character; in other words, msg can hold the address of some character. However, the compiler sets aside no space to store characters and does not initialize msg to any particular value. > When the compiler finds the statement msg = "Hello, world\n"; it does two things: - As before, it creates the string "Hello, world\n", followed by a null character somewhere within the object code file. - It assigns to the starting address of that string-the address of the character H-to msg. > The command puts(msg) works just as it did before, printing characters until it encounters the null character. The workaround is of course to use s1 = &s2[0]