3 ms·

Not to make application bugs sound acceptable, but if you're dealing with that many files, you should probably have started in the terminal anyway, whether it b

Not to make application bugs sound acceptable, but if you're dealing with that many files, you should probably have started in the terminal anyway, whether it be Linux or macOS. It is also a good idea to try to organize your way out of large directories (that is, directories with many immediate children). Some filesystems dislike large directories performance-wise, and manual workflows rarely deal well with very long file sequences.

mirimir 8y ago
I've since learned that :) What can I say. I was a n00be, from Windows. And I should add that Debian terminal had no problem with 10^4 or even 10^5 files per directory. For what it's worth, I was massaging several GB of Newsgroup data into SQL. As I recall, I used grep to decompose the data into components. I crunched one year segments. For each year, each type of header, and message bodies, went into a subdirectory, tagged with the message ID. So in each of those subdirectories, there was one file for each message in the data segment. I did it that way, because each type of component needed custom regex to become SQL fields. In the end, the header types and message bodies became tables, indexed and linked by message IDs. So anyway, there were lots of temp files. Why did I do that? You might ask. Well, there was this notorious troll, who was on a vendetta against friends and I. He didn't just post trash. He spoofed our messages, making fun of us, and trying to sow discord. So it was virtually impossible to filter out his shit. So basically I identified many of his personas, going back a decade or two. And eventually I found posts that included unobscured IP addresses and meatspace email addresses. So I emailed him, and threatened to dox him, if he didn't stop trolling us. So he did. And I never actually doxxed him, because that would have violated his privacy. It was Ari Silverstein, by the way.[0] None of my personas posted there, but people were aware of my work. 0) https://www.velocityreviews.com/threads/tracking-ip-addresses-and-usenet-posts.741347/ https://www.velocityreviews.com/threads/tracking-ip-addresse...
- mirimir 8y ago
  Edit: And in case you search about this, just about everything you find will be Ari Silverstein's word salad.

mirimir 8y ago

I've since learned that :) What can I say. I was a n00be, from Windows. And I should add that Debian terminal had no problem with 10^4 or even 10^5 files per directory. For what it's worth, I was massaging several GB of Newsgroup data into SQL. As I recall, I used grep to decompose the data into components. I crunched one year segments. For each year, each type of header, and message bodies, went into a subdirectory, tagged with the message ID. So in each of those subdirectories, there was one file for each message in the data segment. I did it that way, because each type of component needed custom regex to become SQL fields. In the end, the header types and message bodies became tables, indexed and linked by message IDs. So anyway, there were lots of temp files. Why did I do that? You might ask. Well, there was this notorious troll, who was on a vendetta against friends and I. He didn't just post trash. He spoofed our messages, making fun of us, and trying to sow discord. So it was virtually impossible to filter out his shit. So basically I identified many of his personas, going back a decade or two. And eventually I found posts that included unobscured IP addresses and meatspace email addresses. So I emailed him, and threatened to dox him, if he didn't stop trolling us. So he did. And I never actually doxxed him, because that would have violated his privacy. It was Ari Silverstein, by the way.[0] None of my personas posted there, but people were aware of my work. 0) https://www.velocityreviews.com/threads/tracking-ip-addresses-and-usenet-posts.741347/ https://www.velocityreviews.com/threads/tracking-ip-addresse...

mirimir 8y ago
Edit: And in case you search about this, just about everything you find will be Ari Silverstein's word salad.