3 ms·

Fixed some of those. Simple case of just pushing the edit button and sending a PR.

• johndough 5y ago
  The suggested solution https://github.com/Onelinerhub/onelinerhub/blob/34467e427cc679705b19ee0e2774502f3d789060/python/merge_txt_files.md https://github.com/Onelinerhub/onelinerhub/blob/34467e427cc6... still runs out of memory with many files. I believe the original intention behind using mode="a" was to append to the output file while reading the input files at the same time. This way, there is no need for an ever-growing string array. But there are still many other issues like using default platform string encoding instead of detecting it properly or at least using utf-8, checking for ".txt" anywhere in the path instead of at the end, and not closing the input files with a context manager like the output file, which suggests that this code is just pierced together from various sources. A robust solution would require many more lines.
  • mynameismon 5y ago
    Another memory issue (although unlikely) could also be the file itself being larger than the RAM size, which can be avoided by chunking the textfiles.
  • hermitdev 5y ago
    For bonus points: how about properly handling UTF-8 byte order marks (BOM), too? The way this code is written if the files contain a BOM, you're going to pass the BOM right into the destination file. Fine for the first file, not so fine for any other file. Dropping a BOM mark in the middle of a file isn't going to result in valid UTF-8...