5 ms·
"Databases are usually locked down to a few team members..." Not true in my experience. True at very large / very mature companies, but at any <5-year-old comp
by slotrans 4y ago
"Databases are usually locked down to a few team members..."
Not true in my experience. True at very large / very mature companies, but at any <5-year-old company the DB will be widely accessible, perhaps universally so. Certainly every engineer, every analyst, and every manager will have full access. (I'm sure someone will pipe in to disagree but this has been true for 10 out of the last 10 companies I've worked with). Coming from DigitalOcean and Walmart it's not surprising you would make this mistake, not trying to dunk just want to suggest this might not be as big of a problem as you think.
In that same vein, in a small company environment you don't harass engineers with query requests, you learn SQL. I've seen hundreds of non-technical users do this (and helped many of them), it's not unusual. Visual query-building tools never help. Even good interfaces like Looker, on top of carefully-crafted data warehouses (which most won't have) are still vastly inferior to SQL and so analysts don't use them. Again, just trying to suggest the problem may not be what you think.
Generating queries with a language model won't work. Real data is far messier than you expect. The table names and field names will not be as literal as you need them to be, and half the database will come with "special instructions" (like "oh you need to divide that field by 100 because...") that will not be inferrable from the names or the data. Many DBs will completely lack foreign keys or consistent key naming. There will often be epochal "eras" in the data where everything before YYYY-MM-DD worked like X, and after that it's Y, and the value of that date is recorded nowhere.
"Currently we support Postgres, MySQL, Snowflake, BigQuery, and Redshift."
I know Java isn't hip but consider relying on JDBC for DB access. There are more DBs out there than you can imagine, and 100% of them support JDBC. Otherwise there will always be some DB you're not supporting, and adding that support will be costly.
Anyway just some food for thought. Good luck. (about me: 18 years in data & analytics)
- itake 4y agoAll of my jobs big (1k+ engineers) and small (me), the BE engineers had read access to most databases. The larger companies have some access controls, but if you needed access, it was almost always given.
- debarshri 4y agoWould you mind chat about how you guys manage access? We are building something in the access management space. Would love to get some feedback!
- itake 4y agoAt Grab, we have an internally built system (that I think is backed by Active Directory and leverages experience from ppl from okta). As a "requestor", I choose from a list of services, then I select which role I want, then there are meta attributes (like which country's data I want to access, etc.). This then lets me oAuth into the internal app with a JWT. My access is time limited (max of 90 days or only 2 hours). As a data owner and EM, I approve requests for services and data that I own. As an service creator, I have to register the service in the tool, provide all of my various roles and configurations. This system works with both internal and external authorizations. Like if I need prod AWS write access. It doesn't cover all data (some tools have similar internally built systems), but we will eventually migrate everything to it. This is probably all I have time to share.
- burcs 4y agoThat's a fair comment, If anything them not being locked down makes access to databases easier to connect to Outerbase. I agree that users should learn SQL, even when you use EZQL it returns the actual SQL in a fully-editable IDE that can be modified in whatever way you would want. Do you think that the existing tools out there provide a good environment for those non-technical teams to learn SQL? Maybe I'm biased but I have never been a huge fan of accessing data from the majority of the existing tools as they feel outdated and like tools built mostly for engineers.
- Ntrails 4y ago> Do you think that the existing tools out there provide a good environment for those non-technical teams to learn SQL? I dunno. There are a truly extraordinary number of people I've worked with who query sql by going in to SSMS, object explorer, find table, right click "select top 1000 rows" and then fiddling around with the query to get what they want. Obviously in some schemas this might be useless, but most often there are views which cover obvious uses.
- zie 4y agoI agree with everything, but will push back about everyone having access to everything.. I hope that's mostly read-only or at least a very robust auditing/logging strategy employed. At $WORK, we give basically every employee a DB role in Postgres, but we use table/row level access control(s) and most can only see their own records, or the records of their employees(if they are a manager). We encourage them to learn SQL(most don't). We also do strong audit/logging controls(essentially pg_audit) so we track every DB change, forever. Otherwise I completely agree with you that data is almost always quite messy. I can't imagine their EZSQL stuff would work on our data at all. We have tried various other data finding magic stuff, and all of it without fail is basically useless once attached to real data that has lived in the wild for a while(we have data back to the 1990's in our database).