3 ms·
"better document binary and add an intent to remove text data types" For all-binary, the client would have to know about all data types and how to represent th
by jeffdavis 7y ago
"better document binary and add an intent to remove text data types"
For all-binary, the client would have to know about all data types and how to represent them in the host language. But that seems clunky. Consider NUMERIC vs. float vs. int4 vs int8: should the client really know how to parse all of those from binary? It makes more sense to optimize a few common data types to be transferred as binary, and the rest would go through text. That also works better with the extensible type system, where the client driver will never know about all data types the user might want to use. And it works better for things like psql, which need a textual representation.
The main problem with binary is that the "optimize a few columns as binary" can't be done entirely in the driver. The driver knows which types it can parse, but it doesn't know what types a given query will return. The application programmer may know what types the query will return, in which case they can specify to return them in binary if they know which ones are supported by the client driver, but that's ugly (and in libpq, it only supports all-binary or all-text). Postgres could know the data types, but that means that the driver would need to first prepare the query (which is sometimes a good idea anyway, but other times the round trip isn't worth it).
"Not having an inline cancellation (in same TCP/IP connection)"
This is related to another problem, which is that while a query is executing it does not bother to touch the socket at all. That means that the client can disconnect and the query can keep running for a while, which is obviously useless. I tried fixing this at one point but there were a couple problems and I didn't follow through. Detecting client disconnect probably should be done though.
Supporting cancellation would be trickier than just looking for a client disconnect, because there's potentially a lot of data on the socket (pipelined queries), so it would need to read all the messages coming in looking for a cancellation, and would need to save it all somewhere in case there is no cancellation. I think this would screw up query pipelining, because new queries could be coming in faster than they are being executed, and that would lead to continuous memory growth (from all the saved socket data).
So it looks like out-of-band is the only way cancellation will really work, unless I'm missing something.
"Multiple Result Sets... you just call the database alot"
Do pipelined queries help at all here?
"Named parameters are extremely useful for building up query parameters"
+1. No argument there.
"Protocol level language selection"
I'm still trying to wrap my head around this idea. I think I understand what you are saying and it sounds cool. There are some weird implications I'm sure, but it sounds like it's worth exploring.
- anarazel 7y ago> The driver knows which types it can parse, but it doesn't know what types a given query will return. [...] Postgres could know the data types, but that means that the driver would need to first prepare the query (which is sometimes a good idea anyway, but other times the round trip isn't worth it). We return the types of the result set separately even when not preparing. The harder part is doing it without adding roundtrips. I've argued before that we should allow the client to specify which types it wants as binary, unless explicitly specified. IMO that's the only way to solve this incrementally from where we currently are. > Do pipelined queries help at all here? We really need to get the libpq support for pipelining merged :(
- jeffdavis 7y ago"I've argued before that we should allow the client to specify which types it wants as binary, unless explicitly specified. IMO that's the only way to solve this incrementally from where we currently are." I don't understand the idea, can you clarify?
- anarazel 7y agoWe should allow the client, on the protocol level, to add (and presumably remove) types to a server side list of types that can automatically be sent in binary if the result format is specified as 'auto'. That way a driver can add handling for ~10 builtin types (bytea, int[48], float[48], timestamp, ...) that matter from a size/efficiency POV, without needing to handle all the other types. That'd provide a large large fraction of the potential benefit of handling binary for all types. The reason for the server side list is that otherwise I don't see how you could do it without an added roundtrip.