3 ms·

I'll see to it that this is fixed.

SFjulie1 11y ago
Ohla. You have to rewrite it entirely. Logs should only be built by chunking at the end of the file atomic write of data in at most PIPE_BUF with an "atom" based on this quantity of data. You are free to do whatever serialization inside this constraint as long as a missing chunk cannot corrupt the whole file. (That is the reason I switch off from systemd that stopped respecting this rule.) For your logger to work in a "safe fashion" it has to be designed like an asynchronous HW driver. A part consuming the logs from IRC, that pushes chunks in circular buffer ring of N fixed size data for which buffer overflow HAVE to be checked (per chunk size and for N) (the bottom half). And a driver that is signaled "asynchronuously" to write the chunks into a file when ready. The critical part (writer) should almost be a one liner to avoid interruption and be written in an overly defensive paranoid fashion. You should have a cursor for the writer and the reader on the circular buffer ring and should always check that the consumer does not overrun the writer. Hence the writers' code should be designed to be asymmetrically faster than the reader. It requires IPC or messaging to coordinate both reader and writer. Most people do it in multithreading, I do it with multiprocessing because it is simpler. You should understand that you have thus a Single Point Of Failure (the file) and that because of all this your software will have a domain in which it will fail (if you want to multiply the reader to distribute the load for the consumer to 'scale up'). The file has to be opened with exclusive access. The code must have a "catch all I can" to be able to unlock the file if ever something bad happens and at startup the code should check it can open the file. (atexit callbacks and every "recoverable" interruptions sent by the OS). file.seek is your friend. You should not use any dynamic allocation to make it more robust (yes preallocating fixed memory size with "char circular_ring[N * MAX_SIZE]") Where N and MAX_SIZE are fixed. Well. It requires a complete rewrite for this code to not create a risk for your users. Your code is like full of hidden landmine by lack of design. And you know what? It used to be standard knowledge for introduction to CS for scientific. You know why? Because no users of computers like to lose their precious data made during a very long measurement spilling huge amount of data and costing a lot of resources. (like gold, helium, molybden, electricity, time, wages of qualified operators....) It seems like coders do not care of their users, like a builder thinking it is reasonnable to build 50 store buildings on quicksands because people only judges builder by the look of their creation.
SFJulie2 11y ago
Part 1/2: Resurrecting SFJulie1's original comment here because it seems like it was unfairly killed: Ohla. You have to rewrite it entirely. Logs should only be built by chunking at the end of the file atomic write of data in at most PIPE_BUF with an "atom" based on this quantity of data. You are free to do whatever serialization inside this constraint as long as a missing chunk cannot corrupt the whole file. (That is the reason I switch off from systemd that stopped respecting this rule.) For your logger to work in a "safe fashion" it has to be designed like an asynchronous HW driver. A part consuming the logs from IRC, that pushes chunks in circular buffer ring of N fixed size data for which buffer overflow HAVE to be checked (per chunk size and for N) (the bottom half). And a driver that is signaled "asynchronuously" to write the chunks into a file when ready. The critical part (writer) should almost be a one liner to avoid interruption and be written in an overly defensive paranoid fashion. You should have a cursor for the writer and the reader on the circular buffer ring and should always check that the consumer does not overrun the writer. Hence the writers' code should be designed to be asymmetrically faster than the reader. It requires IPC or messaging to coordinate both reader and writer. Most people do it in multithreading, I do it with multiprocessing because it is simpler. You should understand that you have thus a Single Point Of Failure (the file) and that because of all this your software will have a domain in which it will fail (if you want to multiply the reader to distribute the load for the consumer to 'scale up'). The file has to be opened with exclusive access. The code must have a "catch all I can" to be able to unlock the file if ever something bad happens and at startup the code should check it can open the file. (atexit callbacks and every "recoverable" interruptions sent by the OS).
- deleted 11y ago
  [deleted]

Ohla. You have to rewrite it entirely. Logs should only be built by chunking at the end of the file atomic write of data in at most PIPE_BUF with an "atom" based on this quantity of data. You are free to do whatever serialization inside this constraint as long as a missing chunk cannot corrupt the whole file. (That is the reason I switch off from systemd that stopped respecting this rule.) For your logger to work in a "safe fashion" it has to be designed like an asynchronous HW driver. A part consuming the logs from IRC, that pushes chunks in circular buffer ring of N fixed size data for which buffer overflow HAVE to be checked (per chunk size and for N) (the bottom half). And a driver that is signaled "asynchronuously" to write the chunks into a file when ready. The critical part (writer) should almost be a one liner to avoid interruption and be written in an overly defensive paranoid fashion. You should have a cursor for the writer and the reader on the circular buffer ring and should always check that the consumer does not overrun the writer. Hence the writers' code should be designed to be asymmetrically faster than the reader. It requires IPC or messaging to coordinate both reader and writer. Most people do it in multithreading, I do it with multiprocessing because it is simpler. You should understand that you have thus a Single Point Of Failure (the file) and that because of all this your software will have a domain in which it will fail (if you want to multiply the reader to distribute the load for the consumer to 'scale up'). The file has to be opened with exclusive access. The code must have a "catch all I can" to be able to unlock the file if ever something bad happens and at startup the code should check it can open the file. (atexit callbacks and every "recoverable" interruptions sent by the OS). file.seek is your friend. You should not use any dynamic allocation to make it more robust (yes preallocating fixed memory size with "char circular_ring[N * MAX_SIZE]") Where N and MAX_SIZE are fixed. Well. It requires a complete rewrite for this code to not create a risk for your users. Your code is like full of hidden landmine by lack of design. And you know what? It used to be standard knowledge for introduction to CS for scientific. You know why? Because no users of computers like to lose their precious data made during a very long measurement spilling huge amount of data and costing a lot of resources. (like gold, helium, molybden, electricity, time, wages of qualified operators....) It seems like coders do not care of their users, like a builder thinking it is reasonnable to build 50 store buildings on quicksands because people only judges builder by the look of their creation.

SFJulie2 11y ago

Part 1/2: Resurrecting SFJulie1's original comment here because it seems like it was unfairly killed: Ohla. You have to rewrite it entirely. Logs should only be built by chunking at the end of the file atomic write of data in at most PIPE_BUF with an "atom" based on this quantity of data. You are free to do whatever serialization inside this constraint as long as a missing chunk cannot corrupt the whole file. (That is the reason I switch off from systemd that stopped respecting this rule.) For your logger to work in a "safe fashion" it has to be designed like an asynchronous HW driver. A part consuming the logs from IRC, that pushes chunks in circular buffer ring of N fixed size data for which buffer overflow HAVE to be checked (per chunk size and for N) (the bottom half). And a driver that is signaled "asynchronuously" to write the chunks into a file when ready. The critical part (writer) should almost be a one liner to avoid interruption and be written in an overly defensive paranoid fashion. You should have a cursor for the writer and the reader on the circular buffer ring and should always check that the consumer does not overrun the writer. Hence the writers' code should be designed to be asymmetrically faster than the reader. It requires IPC or messaging to coordinate both reader and writer. Most people do it in multithreading, I do it with multiprocessing because it is simpler. You should understand that you have thus a Single Point Of Failure (the file) and that because of all this your software will have a domain in which it will fail (if you want to multiply the reader to distribute the load for the consumer to 'scale up'). The file has to be opened with exclusive access. The code must have a "catch all I can" to be able to unlock the file if ever something bad happens and at startup the code should check it can open the file. (atexit callbacks and every "recoverable" interruptions sent by the OS).

deleted 11y ago
[deleted]