3 ms·

> In the currently stable version, the main interface of each "adapter" is `fn(Read, Write) -> ()`. To allow custom adapter chaining I have to change it to be `

> In the currently stable version, the main interface of each "adapter" is `fn(Read, Write) -> ()`. To allow custom adapter chaining I have to change it to be `fn(Read) -> Read` where each chained adapter wraps the read stream and converts it while reading. But then I get issues with how to handle threading etc, as well as a random deadlock that I haven't figured out how to solve so far :/ I don't quite grok the problem here. If you file an issue against ripgrep proper with code links and some more details, I can try to assist. Taken literally, ripgrep uses that exact same approach. There are potentially multiple adapters being used. Each adapter is just defined to wrap a `std::io::Read` implementation, and the adapter in turn implements `std::io::Read` so that it can be composed with others. The part that I'm missing is why this has anything to do with threading or deadlocks. I/O adapters shouldn't be having anything to do with synchronization. So I'm probably misunderstanding your problem.

phiresky 6y ago
> If you file an issue against ripgrep proper with code links and some more details Sorry, I don't think I explained my issue very well. In general it has nothing to do with the interaction with ripgrep, that works fine. It's that each adapter (e.g. zip -> list of file streams) needs to have an interface of fn(Read) -> Iter<ReadWithMeta> But then if there's a PDF within the zip, I have to give the returned ReadWithMeta to the PDF adapter - but it can't take ownership, because the Archive file iterators only give borrowed reads. I maybe worked around this by creating a wrapper type [3] and adding an unsafe here [2], but something deadlocks when adapting zip files currently. Also, for external programs, I have to copy the data from the Read into a Write (stdin of the program) - which needs to happen in a separate thread, otherwise the stdout is never read [1], but some Reads I have aren't Send since they come from e.g. zip-rs, so they can't be passed to a thread. [1] https://github.com/phiresky/ripgrep-all/blob/baca166fdab3d2466ac9c482c3c9b8e160014b9a/src/adapters/spawning.rs#L118 https://github.com/phiresky/ripgrep-all/blob/baca166fdab3d24... [2] https://github.com/phiresky/ripgrep-all/blob/baca166fdab3d2466ac9c482c3c9b8e160014b9a/src/recurse.rs#L23 https://github.com/phiresky/ripgrep-all/blob/baca166fdab3d24... [3] https://github.com/phiresky/ripgrep-all/blob/baca166fdab3d2466ac9c482c3c9b8e160014b9a/src/adapters/zip.rs#L52 https://github.com/phiresky/ripgrep-all/blob/baca166fdab3d24...
- burntsushi 6y ago
  I don't have time to dig into this right now, but I just wanted to say that I did at least gather that it wasn't an interaction problem with ripgrep. :-) I just figured the ripgrep issue tracker would be a good place to discuss it. But now that I think about it, the ripgrep-all issue tracker might be a better spot. Maybe post an issue there and ping me? That way we can chat with email notifications and what not.

phiresky 6y ago

> If you file an issue against ripgrep proper with code links and some more details Sorry, I don't think I explained my issue very well. In general it has nothing to do with the interaction with ripgrep, that works fine. It's that each adapter (e.g. zip -> list of file streams) needs to have an interface of fn(Read) -> Iter<ReadWithMeta> But then if there's a PDF within the zip, I have to give the returned ReadWithMeta to the PDF adapter - but it can't take ownership, because the Archive file iterators only give borrowed reads. I maybe worked around this by creating a wrapper type [3] and adding an unsafe here [2], but something deadlocks when adapting zip files currently. Also, for external programs, I have to copy the data from the Read into a Write (stdin of the program) - which needs to happen in a separate thread, otherwise the stdout is never read [1], but some Reads I have aren't Send since they come from e.g. zip-rs, so they can't be passed to a thread. [1] https://github.com/phiresky/ripgrep-all/blob/baca166fdab3d2466ac9c482c3c9b8e160014b9a/src/adapters/spawning.rs#L118 https://github.com/phiresky/ripgrep-all/blob/baca166fdab3d24... [2] https://github.com/phiresky/ripgrep-all/blob/baca166fdab3d2466ac9c482c3c9b8e160014b9a/src/recurse.rs#L23 https://github.com/phiresky/ripgrep-all/blob/baca166fdab3d24... [3] https://github.com/phiresky/ripgrep-all/blob/baca166fdab3d2466ac9c482c3c9b8e160014b9a/src/adapters/zip.rs#L52 https://github.com/phiresky/ripgrep-all/blob/baca166fdab3d24...

burntsushi 6y ago
I don't have time to dig into this right now, but I just wanted to say that I did at least gather that it wasn't an interaction problem with ripgrep. :-) I just figured the ripgrep issue tracker would be a good place to discuss it. But now that I think about it, the ripgrep-all issue tracker might be a better spot. Maybe post an issue there and ping me? That way we can chat with email notifications and what not.