Arimbr

11 ms·

61.
Scaling Data Ingestion Pipelines on Kubernetes(airbyte.com)
2 points by Arimbr 5y ago|0 comments
62.
Airbyte Raises a $150M Series B(airbyte.io)
9 points by Arimbr 5y ago|1 comments
63.
Using an ETL framework vs. writing yet another ETL script(airbyte.io)
114 points by Arimbr 5y ago|74 comments
64.
by Arimbr 5y ago
I like how Prefect is positioned as an orchestrator for the modern data stack. Airflow also started as an orchestrator, but then they tried to cover all sorts of other use cases like ETL/ELT pipelines with transfer and transformation o
65.
ELT pipelines with Prefect, Airbyte and dbt(airbyte.io)
4 points by Arimbr 5y ago|1 comments
66.
Extending the behavior of third-party Docker images on Kubernetes(airbyte.io)
2 points by Arimbr 5y ago|0 comments
67.
Build a Single Customer View with SQL and Open-Source Tools(airbyte.io)
1 points by Arimbr 5y ago|0 comments
68.
by Arimbr 5y ago
Airbyte CDC is based on Debezium, but Airbyte abstracts it away and make it easier to CDC from Postgres, MySQL, MSSQL to any supported destination (included S3). Here is the doc for CDC: https://docs.airbyte.io/understanding
69.
by Arimbr 5y ago
Nice work there! I also think that the next challenge for data teams is all this data documentation and discovery work. I still think that Airflow is great for power data engineers. Airbyte and dbt are positioned to empower data analysts (o
70.
by Arimbr 5y ago
Oh, you should check Materialize. I feel Materialize is like dbt but with an ingestion layer and real-time materialized views. To deliver that you need to centralize data on their the Materialize database, which is may main caveat. With dbt
71.
by Arimbr 5y ago
Really good points! I don't think that Airflow is necessarily a problem if your data engineering team knows how to best use Airflow Operators, Hooks and DAGs for incremental loads. But because Airflow is not an opinionated ETL/ELT
72.
by Arimbr 5y ago
My understanding of dbt is that it builds a DAG based on the interdepencies between models. The interdepencies are parsed from 'ref' functions on the SQL files. The thing with dbt is that you transform the data within a single dat
73.
by Arimbr 5y ago
[author of the article] My main concern about using Airflow for the EL parts is that sources and destinations are highly coupled with Airflow transfer operators (e.g. PostgresToBigQueryOperator). The community needs to provide M * N operato
74.
ETL Pipelines with Airflow: The Good, the Bad and the Ugly(airbyte.io)
147 points by Arimbr 5y ago|86 comments
75.
Three years running Kubernetes on production at Bedrock(tech.bedrockstreaming.com)
1 points by Arimbr 6y ago|0 comments
76.
Show HN: Developers at French startups wrote 600 tech articles(guriosity.com)
3 points by Arimbr 6y ago|0 comments
77.
by Arimbr 6y ago
Hello HN! My name is Ari. I've worked as a data engineer for three French startups. The first, didn't have an engineering blog. The second had one, but i never took the courage to write. The third, didn't consider it a priori
78.
Show HN: A collection of 600 tech articles by developers at French companies(guriosity.com)
1 points by Arimbr 6y ago|1 comments
79.
Why CI/CD for Machine Learning Is Hard – Subreddit Classification Example(blog.valohai.com)
2 points by Arimbr 6y ago|0 comments
80.
by Arimbr 6y ago
Nice, I also think that backend engineering needs more innovation!
81.
French Startups Support Developers to Contribute to Open Source - Here is Why(medium.com)
2 points by Arimbr 6y ago|0 comments
82.
by Arimbr 7y ago
Thanks for your valuable feedback. This indeed answers a question such us "What community is used to hear what you have to say?". It is not really based on your interests, it just takes your text and suggests subreddits where peop
83.
by Arimbr 7y ago
Yes, that is the human solution to the problem. I manually tested a bit on what people ask there :) On the other hand, the machine is faster and lot of people don't get an answer there or can wait for it. The machine is not necessarily
84.
by Arimbr 7y ago
Yes, unfortunately is not part of the 4k subreddits I trained on. I will retrain it with more subreddits :) The list of subreddits and an estimation of the performance for each one is on this Google Spreadsheet https://docs.googl
85.
by Arimbr 7y ago
Nice work @exegete! The Chrome extension idea is great ;) Nice to also see some metrics comparison. I will review your work. Looks like you achieved 0.6 accuracy with 600 classes. I got 0.4 f1-score on 4000 classes, but I have a ton of post
86.
by Arimbr 7y ago
Good feedback. Indeed, it is not clear on the UI. The model concatenates the two, so not sure if changing the label "Message" to "Text" to mimic Reddit UI for text posts would be enough... Or rather just have a single &q
87.
by Arimbr 7y ago
Weird! I just tried "best time tracking app for iOS?" and got "iOSProgramming | 45.0 % swift | 18.0% ProtonMail | 14.0% ios | 4.0% apple | 3.0% iphone | 3.0% FlutterDev | 2.0% freebies | 1.0% jailbreak_ | 1.0% jailbreak | 1.0
88.
by Arimbr 7y ago
Thanks so much for your detailed feedback. There is definitely something going on with the UK thing... Need to dig deeper!
89.
by Arimbr 7y ago
You are right!
90.
by Arimbr 7y ago
Thanks man! Looking forward to working on V2 :)

More ›