tobilg

13 ms·

31.
by tobilg 2y ago
I‘m building https://sql-workbench.com in my spare time. It‘s a SQL Workbench in the browser, based on DuckDB WASM. You can query remote and local datasources, such as CSV, JSON or Parquet files. You can also visualize the resul
32.
by tobilg 2y ago
That's a complete other topic. DBOS also doesn't fit in this stack. And what has SAML to do with it?
33.
by tobilg 2y ago
I created a similar project, https://serverlessmaps.com which lets you host PMTiles on S3 and CloudFront, with a Lambda@Edge layer where you could also do auth... With 1TB free traffic from the CDN, and pretty small costs for S3
34.
by tobilg 2y ago
If you store the PMTiles in S3 or any other object store that supports HTTP Range Requests, that's a no-brainer... In a normal disk on you own server, this might become interesting, yes.
35.
by tobilg 2y ago
Thanks for the feedback, are you using dark mode eventually?
36.
by tobilg 2y ago
Exactly, it's "just" extracting the AWS IAM data from the publicly available documentation, stores it in multiple formats and publishes a NPM package containing the data with some wrappers/accessors, so that people can t
37.
by tobilg 2y ago
I publish a daily updated AWS IAM dataset at https://github.com/tobilg/aws-iam-data as well as historized managed IAM policies at https://github.com/tobilg/aws-iam-managed-policies There's al
38.
by tobilg 2y ago
No, it's implemented via parsing the current schema and translating it to a Mermaid ERD, see https://mermaid.js.org/syntax/entityRelationshipDiagram.html
39.
by tobilg 2y ago
Currently it‘s not open sourced. Eventually considering it for the future though.
40.
by tobilg 2y ago
Yes, you can share a query, and also the visualization. See https://tobilg.com/using-duckdb-wasm-for-in-browser-data-eng...
41.
by tobilg 2y ago
There‘s a query history tab, not sure if this what you want? Otherwise, I‘m very open to feedback, just create an issue in the linked GH Repo. Thanks!
42.
by tobilg 2y ago
Nice tool! I built https://sql-workbench.com/ which runs completely in-browser via DuckDB WASM, and enables querying of remote CSV, JSON, Parquet and Arrow data sources, as well as uploaded local files. Charts are supported
43.
by tobilg 2y ago
I agree to the article, but the discussion is nearly two decades old already and more or less the same since I started in classic BI in the early 2000s. I also strongly believe in SQL to be the "glue", or the least common denomina
44.
by tobilg 2y ago
Having a look at the docker-compose.yml, I don't really understand why you'd need Mongo, Postgres, Redis and Clickhouse in the same stack... Could you elaborate?
45.
by tobilg 2y ago
You can do the exact same thing with DuckDB as well...
46.
by tobilg 2y ago
I'm currently rewriting https://github.com/ownstats/ownstats to this model, with a slight difference that I stream Arrow data from a AWS Lambda Function URL into DuckDB WASM in the frontend... Works great. An impr
47.
by tobilg 2y ago
You can try DuckDB in your browser via DuckDB WASM with https://sql-workbench.com
48.
by tobilg 2y ago
I witness the overengineering regarding "big" data tools and pipelines since many years... For a lot of use cases, data warehouses and data lakes are only in the gigabytes or single-digit terabytes range, thus their architecture c
49.
by tobilg 2y ago
Yes, it uses OpenStreetMap data, which you then host on your own AWS account
50.
by tobilg 2y ago
Right, the diagram is generated from the IaC of the project itself, so it contains all AWS resources involved. You could use the Lambda function to verify Access Tokens etc. before returning the tile data if that is a concern. CloudFront as
51.
by tobilg 2y ago
Exactly! I ran this on a r7gd.4xlarge EC2 instance, which took below 3 hours. Then used the much better upload speed from EC2 to S3 as you described. Let me know if you‘re interested in access to the IaC repo of this.
52.
by tobilg 2y ago
That’s not entirely correct. Yes, this project wraps Protomaps/PMTiles and describes the process and delivers Infrastructure as Code to deploy the whole stack to AWS, which the Protomaps website and code repos don’t deliver. Not sure w
53.
by tobilg 2y ago
It uses PMTiles… Range requests are only supported on S3 directly. If you want to use a custom domain, edge level caching etc. you‘ll need to use a CDN like CloudFront
54.
Show HN: ServerlessMaps – Host your own maps in the cloud(github.com)
62 points by tobilg 2y ago|20 comments
55.
Browser-first analytics in natural language with DuckDB and Ollama(tobilg.com)
1 points by tobilg 2y ago|0 comments
56.
by tobilg 2y ago
Author here, thanks for posting! The blog post explains how https://sql-workbench.com can be used for in-browser data engineering and analytics. Let me know if you have questions!
57.
by tobilg 2y ago
The new feature of https://sql-workbench.com uses DuckDB WASM, Langchain & Ollama as well as the DuckDB-NSQL Text-to-SQL model running on the user's local machine to enable querying local and remote data via natural lan
58.
Show HN: Privacy-first analytics in natural language in the browser(tobilg.com)
1 points by tobilg 2y ago|1 comments
59.
by tobilg 3y ago
Thanks you, https://sql-workbench.com author here :-) Let me know if you have some feature requests / bug reports etc. at https://github.com/tobilg/sql-workbench/issues Thanks!
60.
by tobilg 3y ago
Perspective.js is what I use for https://sql-workbench.com grid and charting functionality. You can just do the data manipulations in DuckDB WASM and pipe the Arrow data to Perspective...

More ›