snidane

21 ms·

211.
by snidane 6y ago
It's actually well understood that privatisation for platform industries is bad, privatisation for businesses running on top of platforms is good. Networks and infrastructure = little to no competition. Eg. networks naturally settle in
212.
Ultra fast OLAP queries with star-tree index(engineering.linkedin.com)
1 points by snidane 6y ago|0 comments
213.
by snidane 6y ago
Much better taxing solution: Single monopoly tax. Make companies/individuals pay proportionally for monopolistic privileges they get from rest of the society. Examples: - network industries (railroads, pipes, energy) will always be nat
214.
by snidane 6y ago
We are probably refering to different scenarios. When purchasing data for analytics, data providers are usually sophisticated enough to know not to modify their data history. With new ones, data delivery format can be negotiated. Data provi
215.
by snidane 6y ago
Source data should not experience deletes or updates, otherwise backfills will not work. Deletes can be handled by mirroring source data. Updates are difficult and will need a full CDC system to capture them. Better is to negotiate with dat
216.
by snidane 6y ago
I had in mind OLAP use cases in environments with lot of both unstructured and structured tabular data. Some kind of scripting is necessary just to structure bunch of text and jsons into tables. Ravendb seems like OLTP NoSQL database on the
217.
by snidane 6y ago
DBT is interesting, but is far from what I'm describing. 1. It is only for structured SQL, not for arbitrary data. I can't use it to unpack raw zipped data for example 2. It couples logic for data transformation and view state man
218.
by snidane 6y ago
Most problems of data engineering of today would be solved in presence of a tool in which I would define arbitrary transformation of a say a single daily data increment and the system would handle the state management and loading of all of
219.
by snidane 6y ago
Can somebody explain Data Mesh in simple terms? I've watched numerous videos and read the article, but still can't understand what it's supposed to say. Is it basically to set up one data lake per analytics team, instead of a
220.
by snidane 6y ago
Ultimatelly there will never be a pure streaming system processing, one record at a time, in real world. Any such system contains a busy loop somewhere inside polling the source, say each 100ms, and unless it shares a lock with the source s
221.
by snidane 6y ago
100% except for one thing. You have to be careful about building things "poorly". There will always be someone who will judge your engineering capabilities by your "poor prototypes". Usually someone non technical from ma
222.
by snidane 6y ago
Or something unexpected happens and you get stuck with low interest rates and a war of some sort. High debt and low interest rate environment don't guarantee your more stable amd secure society. Only for short term while you keep print
223.
by snidane 6y ago
Because graphical programming are just textual programming languages which are just overly complex (xml) and coming with attached visualization engine for its components. They are used only for limited domains such as composing some data fl
224.
by snidane 6y ago
If you have a static website with a lot of data behind it, eg. a map visualizing some larger dataset, then OLAP database will be actually a very good backing storage.
225.
by snidane 6y ago
If you're using sqlite, but for a static website, you might consider using duckdb. It's still evolving, but it has huge potential. It's also an embedded single file database like sqlite, but stores data in columns. Duckdb to
226.
by snidane 6y ago
UiPath and RPA in general seems to be a great complement to no-code IT systems. What you'd normally glue together using code, in no-code environment you just emulate a human clicking at things. I guess the more no-code startups we see
227.
by snidane 6y ago
If you get too good and maintenance, you will be that senior guy keeping thing running while junior will get to build those nice and shiny things, keeping up with latest tech and work on their job transferrable skills. It's always a di
228.
by snidane 6y ago
You often need core computer science skills for the interviewing game.
229.
by snidane 6y ago
That kind of works in theory or for people up there in decision making ranks. For engineers down in the trenches it might not be such a motivating factor because they don't feel their individual contributions matter as much. It might m
230.
by snidane 6y ago
> I don’t think motivated devs want to go home at 2pm. That's the beauty of it. Motivated engineers will use the spare time after 2pm to study and improve things. Or run some errands if they have to, without feeling their butt has t
231.
by snidane 6y ago
The article is correct to point out that in case of no process measurement, politics, not results, will start to dominate. Measuring process inputs will favor Sisyphean work. Appearing to be working, hours punched in and butts-in-seats will
232.
by snidane 6y ago
The best way to motivate devs to build truly robust systems is to let them go home at 2pm when things are finished and production process is running smooth. However most organizations would pile more work onto devs if they finish early, so
233.
by snidane 6y ago
Can I do some advanced xml stuff say using lxml with beautifulsoup using UDFs?
234.
by snidane 6y ago
Thanks for clarification. I either wasn't able to figure this out back then or things have changes since then.
235.
by snidane 6y ago
When I want to process something in Snowflake I would have to copy the data 1. from snowflake storage to s3 2. from s3 to my compute node 3. do the processing 4. store result in s3 5. copy result from s3 to snowflake s
236.
by snidane 6y ago
1. Why tf would javascript be picked as the UDF language when Python dominates the data world. 2. People usually load complex Python libraries for data processing. I wonder if Snowflake UDF would support that or just allow you to use standa
237.
by snidane 6y ago
Why would I store eg. bunch of html files as string columns in Snowflake, only to download them down, process them in python and load back into some other string table. I could do the same in s3 for much cheaper.
238.
by snidane 6y ago
Yeah I thought Snowflake external tables would do this but it is not the case. External table in Snowflake only allows you to ingest data from s3 to their storage (which is also s3 behind the scenes). Perhaps something has changed since the
239.
by snidane 6y ago
With shared-disk the data sits in cold storage (eg. s3) for which you pay cheaply. Only when you have to do some big computation you spin up a temporary compute cluster, do the computation, store resukt back in s3 and kill the cluster. You
240.
by snidane 6y ago
Only after using the data beforehand it gets cached or by actively pushing it into cache. Otherwise everything sits in cold storage far away. And having control over the cache sucks as hell. You can't pin down a table to reside in comp

More ›