3 ms·
re pipelining: unless Go's network package does this magically under the hood, then sqlite-http doesn't take advantage of that! If you think that would be a nic
by alexgarcia-xyz 4y ago
re pipelining: unless Go's network package does this magically under the hood, then sqlite-http doesn't take advantage of that! If you think that would be a nice feature, file an issue and I'll see what can be done
How complex is your text processing? If I'm working with a small dataset, I typically just save the "raw" responses into a table and create several intermediate tables that extracts the data I need. Something like:
create table raw_responses as http_get('https:/....');
create table intermediate1 as select xxxx(response_body) as message from raw_responses;
create table final as select message from intermediate;
(some CTEs[0] will be very useful here)
The "lambda" pattern described about half-way through this post might be useful as well, if you need to do some complex text processing.
[0] https://www.sqlite.org/lang_with.html https://www.sqlite.org/lang_with.html
- robmccoll 4y agoGo's HTTP client doesn't support pipelining in HTTP 1.1. It will do keep alive / socket re-use. HTTP 2+ have support for request multiplexing which Go supports. Your code appears to create a new client on each request, so I'm not sure if the underlying transport will be re-used or if any multiplexing will take place. I would be surprised if it is.