pitah1

9 ms·

31.
by pitah1 3y ago
This is what I am trying to solve via building Data Catering ( https://data.catering/ ). It gives you the ability to generate data into any database (along with maintaining any relationships between data) via metadata that ca
32.
Creating a Document Answering Chatbot(medium.com)
2 points by pitah1 3y ago|0 comments
33.
by pitah1 3y ago
Interesting feature. One key thing I found when testing is that for you to reproduce the set of steps the user went through, there are a data attribute(s) that need to remain the same. For example, after login, a request for your account in
34.
Providing your own documents to an LLM in your laptop(medium.com)
2 points by pitah1 3y ago|0 comments
35.
by pitah1 3y ago
Grats on building this out. I think there is a lot of potential in this space. I very much understand the challenges of financial/regulatory reporting and data quality :). Couple of things I have noticed. You mention "automates ro
36.
Testing Too Difficult? Automate Your Integration Tests(medium.com)
2 points by pitah1 3y ago|0 comments
37.
Solace Data Generation(medium.com)
1 points by pitah1 3y ago|0 comments
38.
Kafka Data Generation(medium.com)
1 points by pitah1 3y ago|0 comments
39.
MySQL Data Generation with Cleanup(medium.com)
2 points by pitah1 3y ago|0 comments
40.
by pitah1 3y ago
The thoughts here are a bit too focused on the costs of testing. Although deeper questions are asked about your own judgement of whether to test something or not, it doesn't consider the benefits. I wrote about this the other day in an
41.
Postgres Data Generation(medium.com)
3 points by pitah1 3y ago|0 comments
42.
by pitah1 3y ago
It was touched on at the beginnning of the article but something I find overlooked about having tests are: - Ability of other developers to be productive on the project. Having tests tells other developers the intended behaviour and notifie
43.
Cassandra Data Generation(medium.com)
1 points by pitah1 3y ago|0 comments
44.
by pitah1 3y ago
Another side effect to take into consideration when offering notifications to customers is the impact on your services if customers action on those notifications. At a bank I worked at, we enabled salary transaction notifications. Soon we f
45.
by pitah1 3y ago
I was using the following RFC for the formal definition of CSV https://www.rfc-editor.org/rfc/rfc4180.html . When the delimiter is different, you have a different file format. TSV (tab-separated values) is an example of
46.
by pitah1 3y ago
Good in-depth insights into each format. This complements nicely with a site I created called tech-diff ( https://tech-diff.com/file/ ) where it provides a summary of the file formats.
47.
Skills and learnings from starting my own data company(medium.com)
2 points by pitah1 3y ago|0 comments
48.
by pitah1 3y ago
I also had a similar idea when I built tech-diff ( https://tech-diff.com/ ) but rather than being just a catalog of items, it shows what the feature differences are. This is what I would argue is more important when investiga
49.
Show HN: Data Caterer – Data testing tool for any data source(github.com)
2 points by pitah1 3y ago|0 comments
50.
Open Data Contract Standard Joins Linux Foundation(lfaidata.foundation)
1 points by pitah1 3y ago|0 comments
51.
Show HN: Tech Diff – Compare different technologies(tech-diff.com)
4 points by pitah1 3y ago|0 comments
52.
by pitah1 3y ago
Thanks for sharing. Looks simple and easy to use. Could you use this for batch data jobs as well? I would imagine having integration with batch job frameworks like Spark would make this more valuable from an organisation perspective. Also,
53.
by pitah1 3y ago
Agreed. I guess it depends on the target market. If your target is small to medium sized businesses, this is great to start doing analytics. But for large organisations, they generally have ETL/ELT jobs that do all the extraction from
54.
by pitah1 3y ago
Thanks for sharing. Looks very clean and simple to use. Do you plan on supporting non-JSON data types for insertion? For example, inserting CSV files, parquet files, Avro or Protobuf messages?
55.
by pitah1 3y ago
This looks very neat! I like the approach of including all data (API, messaging, database, files) as I went through similar thoughts and implementation recently with my project. A growing pain for data integrations is data validations. The
56.
by pitah1 3y ago
Very neat! Thanks for sharing. Have you thought about how you could export out all of these validations? For example, these validations/contracts can be used as metadata for data consumers who can know and trust your data based on the
57.
by pitah1 3y ago
Thanks for the response. I also noticed there was a mention of data contracts or Pydantic to keep your data clean. Would it make sense to embed that as part of a DLT pipeline or is the recommendation to include it as part of the transformat
58.
by pitah1 3y ago
Do you plan on integrating with metadata sources as well such as Amundsen or Datahub? Or is the plan that DLT will become the metadata source?
59.
Data Generators in 2023(data.catering)
1 points by pitah1 3y ago|0 comments
60.
AWS S3 'Access Denied' (2021)(medium.com)
1 points by pitah1 3y ago|0 comments

More ›