faizshah

12 ms·

121.
by faizshah 2y ago
This is a really cool post and that is a massive amount of throughput. In my experience in data engineering, it’s very unlikely you can exceed 500mb/s throughput of your business logic as most libraries you’re using are not optimized t
122.
by faizshah 2y ago
I still didn’t get why they built this, there’s a better explanation of the feature set in the FAQ comparison with parquet: https://docs.arcticdb.io/latest/faq/ > How does ArcticDB differ from Apache Parquet?¶
123.
by faizshah 2y ago
The implementations of zero trust that I have seen involve exposing your service to the public internet with an Authenticating Proxy on top. So instead of trusting the network implicitly you trust the caller’s auth token before they can con
124.
by faizshah 2y ago
I mean this is true but the key part is “securing them for exposure on the internet.” Adding a simple 2FA layer (I think google calls this the Access Proxy or Identity Aware Proxy) on top is usually the way you secure zero trust services. I
125.
by faizshah 2y ago
The converse API in AWS bedrock lets you use function calling across a number of different providers (doesn’t support OpenAI): https://docs.aws.amazon.com/bedrock/latest/userguide/convers... I have been using
126.
by faizshah 2y ago
Same at Amazon, 0 teammates out of the 100+ on my product work at my office or in my timezone but I have to badge into it because they don’t allow virtual positions anymore. So now I have to spend 2.5 hours commuting, worse mood, I'm l
127.
by faizshah 2y ago
I wonder how the debugging experience is, it must be pretty difficult to generate correct source maps.
128.
by faizshah 2y ago
What about from the customer perspective what do enterprise customers want to see happen? I’m guessing you want someone to reach out to you for a video call where they can tell you whether the product works with your Identity Provider, foll
129.
by faizshah 2y ago
If you make it a task prompt instead of a lateral thinking task you get a different “world model” or in my view you are just biasing the output to use a different structure. ``` Can you summarize joe’s schedule for my journal? Joe drove
130.
by faizshah 2y ago
That’s right, if LLMs were really thinking/forming world models etc. we would expect them to be robust against word choice or phrasing. But in practice anyone using RAG can tell you that that is not the case. I’m just a practitioner so
131.
by faizshah 2y ago
This is your human interpretation of what it is doing. Another interpretation could be that it has seen similarly structured sentences in other logic/lateral thinking problems so it raises the likelihood of words that have to do with l
132.
by faizshah 2y ago
Can anyone recommend any other books like this where it is both a history/ethnography and covers the actual technical details/code?
133.
by faizshah 2y ago
The way you want to build and lead an engineering team is so that the individual engineer’s approach to the problem doesn’t matter. The idea is to establish deployment safety and ops mechanisms so Type 2 isn’t even possible and your Sr. Eng
134.
by faizshah 2y ago
This sort of process failure is why there’s a Correction of Errors (postmortem) process at Amazon: https://aws.amazon.com/blogs/mt/why-you-should-develop-a-cor... Specifically asking why did it take so long to det
135.
by faizshah 2y ago
I guess what I’m trying to understand is if they were grounding the AI Overview with the snippets from the google search results then how would the AI overview generate a summary that was that semantically different from the source texts?
136.
by faizshah 2y ago
Does anyone know if they were using RAG for it?
137.
by faizshah 2y ago
Chromium: https://www.chromium.org/developers/design-documents/ Kubernetes: https://github.com/kubernetes/enhancements/tree/master/keps
138.
by faizshah 2y ago
This highly depends on team and project. The more ambiguous your project is the more likely no decision will be made and you will instead need to commit to a follow-up meeting with more docs and more people.
139.
by faizshah 2y ago
Thanks, if anyone else is interested there is an explanation of this feature here: https://subtxt.in/library-data/2016/03/28/json_stream_jq And: https://github.com/jqlang/jq/wik
140.
by faizshah 2y ago
Oh I see, so what I do is I use rg -l aka —-files-with-matches. To produce the relevant files and I use the filepath as the key for the next part of the processing pipeline. So the pattern is to reduce the candidate set of files with rg or
141.
by faizshah 2y ago
I guess I should have written it more explicitly but my two most common techniques are: - ripgrep PATTERN | xargs jq - find PATTERN -exec jq In both cases you have a large amount of data in your file system that is in JSON and you want to e
142.
by faizshah 2y ago
This is a great book, there’s a few tools I would add. Datasette, clickhouse local (cli) and duckdb. I think ripgrep is a big omission, ripgrep | xargs jq and find -exec jq is one of my most common data science workflows cause you can get s
143.
by faizshah 2y ago
As a daily user of ai coding tools, I think if you don’t have criticisms of the tools you probably aren’t using them on production code. I don’t think many would describe Yann Lecun as threatened by these tools, here’s his take on the LLM a
144.
by faizshah 2y ago
90% of my job is spent redesigning and optimizing code that “just transforms XML into JSON and back again.” The problem is when you don’t have a good mental model for how the code works and how it will evolve over time you make bad design d
145.
by faizshah 2y ago
I do the same thing I have never understood the use case and it is included in every tutorial aa if it is something used daily. It’s so much slower to move your eyes and type the number even with relative numbers than it is to just hit w, e
146.
by faizshah 3y ago
Spark probably powers 90% of FANG batch data pipelines. Theres still lots of hive and Hadoop clusters out there that are just being maintained for reporting until they get funding to swap to some fancy DBT setup. Spark is pretty much ubiqui
147.
by faizshah 3y ago
I feel like temporal’s approach of replaying the computation is probably better than trying to serialize the running computation. Serializing the running coroutine gets ugly as is shown here with things like file handles and making pickle a
148.
by faizshah 3y ago
I could be wrong but I don’t think OTP natively implements a load shedding strategy or concurrency controls or request hedging. You would implement a supervisor to do so if you don’t want to overload an external system like a DB or microser
149.
by faizshah 3y ago
I was curious how long it would take with Polars (for scale), apparently 33s: https://github.com/Butch78/1BillionRowChallenge/tree/main I’m kind of interested in the opposite problem, what is the simplest sol
150.
by faizshah 3y ago
So it looks like they are intending this more as a framework that you build your own custom proxy on top of for your team/company. So you just extend the conf for your custom filters and then you add the wrapper systemd Service. I migh

More ›