5 ms·

Glib is OK except for the fact that it has a memory cache on top of that of malloc. This prevents tools like asan or valgrind from detecting memory related bugs

by lrossi 6y ago

Glib is OK except for the fact that it has a memory cache on top of that of malloc. This prevents tools like asan or valgrind from detecting memory related bugs. It caused my team a lot of grief, to the point that we regretted the choice of using glib in the first place. I am not sure why there is a memory cache in the first place. Malloc may have been slow in the 90s, but these days there is no reason to cache and reuse allocations. It’s also a major security risk, since it nullifies hardening measures from the standard library, as we have seen with openssl/heartbleed recently.

lrossi 6y ago
Another problem is that there is no type checking in the containers. So you have to cast everything to/from void pointers, even for basic structures like lists and arrays. This makes it easy to introduce tricky bugs.
- wahern 6y ago
  The BSDs basically solved this problem decades ago: <sys/queue.h> and <sys/tree.h> provide type-safe core data structures sufficient for most C application needs. The amount of wheel reinvention and dependency complexity outside the BSD universe blows my mind. (Though, FreeBSD projects do have a greater tendency to complicate things, perhaps owing to the stronger corporate-induced feature chasing.) The only real universal sore spot IME has been arrays and vectors. But nobody seems to pitch glib as a way to get a fast and ergonomic FIFO buffer. There are many other areas without simple, go-to solutions, but then that's the nature of C programming. Most of the C programmers I interact with are multi-language programmers, as opposed to many C++, Java, etc engineers who lean toward single-language, monolithic approaches. I can understand using glib for GUI applications, considering it's already a requirement for Gtk, and because of the OOP emphasis in GUI programming. But IMNSHO, in most other areas the right reasons for selecting C as your implementation language are mutually exclusive with the need for cookie-cutter, void-pointer heavy data structure implementations a la glib. EDIT: Removed outdated discussion of systemd + glib.
- attractivechaos 6y ago
  In addition, using void pointers harms performance as this strategy often incurs additional heap allocations, hurts memory locality and prevents compiler optimization. Personally, I avoid glib like a plague.
asveikau 6y ago
Curious for some details. On what layer does that caching occur? I immediately thought g_malloc but it seems to call directly to libc: https://github.com/GNOME/glib/blob/master/glib/gmem.c https://github.com/GNOME/glib/blob/master/glib/gmem.c
the_why_of_y 6y ago
Did you try to disable the slab allocator? https://developer.gnome.org/glib/stable/glib-running.html#G_SLICE https://developer.gnome.org/glib/stable/glib-running.html#G_...
- lrossi 6y ago
  I am aware of that environment variable. The problem is that I think almost nobody is using it, so it could come with a risk of introducing more bugs. By the way, the example shown in the documentation page that you link to seems ridiculous: void *slist = g_slist_alloc (); /* void* gives up type-safety */ g_list_free (slist); /* corruption: sizeof (GSList) != sizeof (GList) */ This bug should actually be caught during compilation if we store the list in a GSList* typed variable instead of void*. You’d think: who does that? Except that this is what you actually end up doing when using any kind of nested glib containers. Elements are always void*, so you have to cast them correctly. So for non-trivial applications, it’s very easy to make mistakes, since you lose the type checking support of the compiler. C is already a tricky language. Removing the type checker makes it even worse.
  - baybal2 6y ago
    You seem to know a lot about Glib. What did you do with it? I looked to port Gtk to MCU, and then gave up. Too much C trickery, and hard to replace, heavy dependencirs. For somebody who knew Qt, and Gtk since 2003, it looks like a miracle now how Qt got smaller, and faster than Gtk, and can even run on an MCU, and quite well! Very big contribution to that was Qt's team willingness to undo the wheel reinvention, and willingness to throw out their hacky attempts to replicate new c++, and standard lib functionality. The Glib+Gtk world, unlike Qt, still lives in ANSI C, and C99 era, and refuses to concede on reinventing functionality of modern standard libraries, language features, and compilers.
    - lrossi 6y ago
      I worked on some Linux apps in another life. I switched to Qt at some point. It was a breath of fresh air! Qt is very reliable and well documented. One can be very productive with it. I think a lot of people have a bad opinion about it as they confuse it with KDE. While KDE does use Qt, the two projects are otherwise independent. I’ve also looked inside the Qt code base a few times. It’s very tidy and quite easy to read. You can tell that their team is very experienced. I used to read their blog as well, they had a lot of good articles. I really hope that they can continue to survive financially. Selling a mostly open source library is not very profitable.
pmiller2 6y ago
I don't know anything about this cache you're referring to, but, is it so pervasive within the library that you can't easily do a patch to remove it? Would such a patch be accepted by upstream? If you're opposed to using the environment variable, those would be my next thoughts if you wanted to get back the ability to use valgrind et al.
- ibotty 6y ago
  You only need an environment variable. It is already implemented. They just did not try it.
inetknght 6y ago
> these days there is no reason to cache and reuse allocations. No, you are wrong. Ideas like that are a big reason that apps get slower despite hardware getting faster.
- stretchcat 6y ago
  Isn't the right way of doing this to use a different implementation of malloc, rather than wrapping malloc with your own meta-malloc?
  - inetknght 6y ago
    > Isn't the right way of doing this to use a different implementation of malloc, rather than wrapping malloc with your own meta-malloc? Yup [0]. There's even other complete libraries like tcmalloc [1] and jemalloc [2]. [0] https://stackoverflow.com/a/262481/1111557 https://stackoverflow.com/a/262481/1111557 [1] https://github.com/google/tcmalloc https://github.com/google/tcmalloc [2] https://github.com/jemalloc/jemalloc https://github.com/jemalloc/jemalloc
    - saagarjha 6y ago
      One reason I can think of wrapping malloc is that C is more likely to give you that than whatever OS-specific API you are going to use to get memory from the kernel (if you even have one!).
- asveikau 6y ago
  I think it becomes an application specific or domain specific thing. For example, I've done some work with audio or video. Nobody working on that goes straight to malloc on every packet or frame. It'd just be asking for pain. But a general purpose allocator doing its own free-list on the assumption that libc is going to suck? I think that's outdated. If you do want to support it, I think it's better to allow the caller to replace the allocator through a function pointer, rather than just do it by default in a library.
  - 6y ago
nn3 6y ago
At least for valgrind it's fairly easy to annotate custom mallocs in the source. Just someone needs to do that.

by lrossi 6y ago

lrossi 6y ago

Another problem is that there is no type checking in the containers. So you have to cast everything to/from void pointers, even for basic structures like lists and arrays. This makes it easy to introduce tricky bugs.

wahern 6y ago
The BSDs basically solved this problem decades ago: <sys/queue.h> and <sys/tree.h> provide type-safe core data structures sufficient for most C application needs. The amount of wheel reinvention and dependency complexity outside the BSD universe blows my mind. (Though, FreeBSD projects do have a greater tendency to complicate things, perhaps owing to the stronger corporate-induced feature chasing.) The only real universal sore spot IME has been arrays and vectors. But nobody seems to pitch glib as a way to get a fast and ergonomic FIFO buffer. There are many other areas without simple, go-to solutions, but then that's the nature of C programming. Most of the C programmers I interact with are multi-language programmers, as opposed to many C++, Java, etc engineers who lean toward single-language, monolithic approaches. I can understand using glib for GUI applications, considering it's already a requirement for Gtk, and because of the OOP emphasis in GUI programming. But IMNSHO, in most other areas the right reasons for selecting C as your implementation language are mutually exclusive with the need for cookie-cutter, void-pointer heavy data structure implementations a la glib. EDIT: Removed outdated discussion of systemd + glib.
attractivechaos 6y ago
In addition, using void pointers harms performance as this strategy often incurs additional heap allocations, hurts memory locality and prevents compiler optimization. Personally, I avoid glib like a plague.

asveikau 6y ago

Curious for some details. On what layer does that caching occur? I immediately thought g_malloc but it seems to call directly to libc: https://github.com/GNOME/glib/blob/master/glib/gmem.c https://github.com/GNOME/glib/blob/master/glib/gmem.c

the_why_of_y 6y ago

Did you try to disable the slab allocator? https://developer.gnome.org/glib/stable/glib-running.html#G_SLICE https://developer.gnome.org/glib/stable/glib-running.html#G_...

lrossi 6y ago
I am aware of that environment variable. The problem is that I think almost nobody is using it, so it could come with a risk of introducing more bugs. By the way, the example shown in the documentation page that you link to seems ridiculous: void *slist = g_slist_alloc (); /* void* gives up type-safety */ g_list_free (slist); /* corruption: sizeof (GSList) != sizeof (GList) */ This bug should actually be caught during compilation if we store the list in a GSList* typed variable instead of void*. You’d think: who does that? Except that this is what you actually end up doing when using any kind of nested glib containers. Elements are always void*, so you have to cast them correctly. So for non-trivial applications, it’s very easy to make mistakes, since you lose the type checking support of the compiler. C is already a tricky language. Removing the type checker makes it even worse.
- baybal2 6y ago
  You seem to know a lot about Glib. What did you do with it? I looked to port Gtk to MCU, and then gave up. Too much C trickery, and hard to replace, heavy dependencirs. For somebody who knew Qt, and Gtk since 2003, it looks like a miracle now how Qt got smaller, and faster than Gtk, and can even run on an MCU, and quite well! Very big contribution to that was Qt's team willingness to undo the wheel reinvention, and willingness to throw out their hacky attempts to replicate new c++, and standard lib functionality. The Glib+Gtk world, unlike Qt, still lives in ANSI C, and C99 era, and refuses to concede on reinventing functionality of modern standard libraries, language features, and compilers.
  - lrossi 6y ago
    I worked on some Linux apps in another life. I switched to Qt at some point. It was a breath of fresh air! Qt is very reliable and well documented. One can be very productive with it. I think a lot of people have a bad opinion about it as they confuse it with KDE. While KDE does use Qt, the two projects are otherwise independent. I’ve also looked inside the Qt code base a few times. It’s very tidy and quite easy to read. You can tell that their team is very experienced. I used to read their blog as well, they had a lot of good articles. I really hope that they can continue to survive financially. Selling a mostly open source library is not very profitable.

pmiller2 6y ago

I don't know anything about this cache you're referring to, but, is it so pervasive within the library that you can't easily do a patch to remove it? Would such a patch be accepted by upstream? If you're opposed to using the environment variable, those would be my next thoughts if you wanted to get back the ability to use valgrind et al.

ibotty 6y ago
You only need an environment variable. It is already implemented. They just did not try it.

inetknght 6y ago

> these days there is no reason to cache and reuse allocations. No, you are wrong. Ideas like that are a big reason that apps get slower despite hardware getting faster.

stretchcat 6y ago
Isn't the right way of doing this to use a different implementation of malloc, rather than wrapping malloc with your own meta-malloc?
- inetknght 6y ago
  > Isn't the right way of doing this to use a different implementation of malloc, rather than wrapping malloc with your own meta-malloc? Yup [0]. There's even other complete libraries like tcmalloc [1] and jemalloc [2]. [0] https://stackoverflow.com/a/262481/1111557 https://stackoverflow.com/a/262481/1111557 [1] https://github.com/google/tcmalloc https://github.com/google/tcmalloc [2] https://github.com/jemalloc/jemalloc https://github.com/jemalloc/jemalloc
  - saagarjha 6y ago
    One reason I can think of wrapping malloc is that C is more likely to give you that than whatever OS-specific API you are going to use to get memory from the kernel (if you even have one!).
asveikau 6y ago
I think it becomes an application specific or domain specific thing. For example, I've done some work with audio or video. Nobody working on that goes straight to malloc on every packet or frame. It'd just be asking for pain. But a general purpose allocator doing its own free-list on the assumption that libc is going to suck? I think that's outdated. If you do want to support it, I think it's better to allow the caller to replace the allocator through a function pointer, rather than just do it by default in a library.
- 6y ago

nn3 6y ago

At least for valgrind it's fairly easy to annotate custom mallocs in the source. Just someone needs to do that.