ibains

8 ms·

31.
Data Engineering Battle: Python vs. SQL++ vs. Visual=Code
1 points by ibains 5y ago|0 comments
32.
by ibains 5y ago
We’re building a primarily PL startup at Prophecy.io - Low Code Data Engineering Platform. We user parser combinators heavily, code generation and all. Hard to find skilled engineers these days in compilers though. The Low-Code movement see
33.
Ask HN: What is the architectural legacy of Hadoop?
1 points by ibains 5y ago|2 comments
34.
by ibains 5y ago
We should be building Fabs in California, Cars in California, getting the smartest people in the world here, in solid numbers. Perhaps we’re in a global competition and we’ll not be the largest economy soon, perhaps we’re too busy with pron
35.
by ibains 5y ago
Prophecy.io - Building Low-Code Data Engineering product for Spark and Airflow. Compilers, Parsers, Functional Programming, Database internals - you know one of these, or you’re a solid engineer who is interested in them. US | Europe (San F
36.
by ibains 6y ago
Prophecy.io - US/Europe - Full Time - Remote/BayArea in office - Compilers, Functional programming, Parsers, Parser Generators background. Ms, PhD preferred. Pay well. We’re pushing limits of Low Code, applied to Data. We work in
37.
by ibains 6y ago
At Prophecy, we're focused on making data engineers more productive. Visual drag-and-drop on Prophecy produces high quality code on Git. We're releasing our SaaS product today - free forever for small teams (3 users). It is availa
38.
Show HN: Prophecy: Low Code Data Engineering on Spark – free for small teams(prophecy.io)
9 points by ibains 6y ago|1 comments
39.
by ibains 6y ago
Prophecy.io - low code product for data engineering on Spark - is Scala based - we love it! We do compiler work - and packrat parser combinators are great. Basic things are supported well. We use Play. We have GraphQL in web services - use
40.
by ibains 6y ago
packrat parser combinators are recursive descent with infinite look ahead. That’s the problem, engineers are under educated
41.
by ibains 6y ago
There is no such variance and new fashions in compilers every year. There are hardware compilers and tools including language servers primarily. Move to cloud requires source to source compilers for data snd transforms.
42.
by ibains 6y ago
packrat parsers are fast (at expense of some memory)
43.
by ibains 6y ago
Yes, more validation - compilers as taught in universities need improvement. So, a startup of 10 people should hire and train people for months. Raise a $2M seed round, have 12-18 months to show ProductMarket fit and as a founder teach all
44.
by ibains 6y ago
You might believe academia should not impart knowledge students require for their subsequent jobs. Ask the students why they go to university. See what is taught in UCBerkeley with Spark and all coming out of there. I took a systems course
45.
by ibains 6y ago
We pay very very well, all my peers from NVIDIA are above 700k. Why would you presume anything about salary?
46.
by ibains 6y ago
I’ve worked on multiple compilers (optimizations expert) at MSFT on VS and CUDA and gave developed a DSL and worked on Database Compilers. I can’t hire compiler people with right skills. We’re building an IDE and those parsers are written
47.
by ibains 6y ago
Folks, after Snowflake IPO and Dbt funding there is a lot of euphoria and FUD as I talked to some VCs. This doesn't match up when I talk to customers, so trying to clarify what's actually going on.
48.
To VCs: ELT is not the disruption – Data Engineering is(prophecy.io)
10 points by ibains 6y ago|3 comments
49.
by ibains 6y ago
Thus just shows how poor visibility into git is, I hope it gets better. Building a product with poor visibility and ridiculing users for not knowing internals is the worst practice in Computer Science. Hadoop did the same, and has set a rec
50.
by ibains 6y ago
You’re talking as if laws were sent by god, we made laws to serve us, not the other way around. We should change them to be sensible. Also, if all businesses were required to be profitable from day 1, you’d have none. You spend on R&D t
51.
by ibains 6y ago
No, I’m saying that the parent comment suggesting bidding is not a good idea. I’m hiring an H1B since that person has the right skills, and I’m hiring Americans as well (who might take lower salary and stock options initially). Putting some
52.
by ibains 6y ago
I have a startup, 4 in US, 10 in India - top tier engineers. I just hired my first H1B - while we pay well - bidding and salary escrow will just favor the big 5 - they are starving startups of talent in bay area any way. Investors want me
53.
by ibains 6y ago
I love this, I was amongst early engineers on CUDA (compilers). NVIDIA was so well run, but boxed into a smaller graphics card market - ATI and it were forced into low margins since they were made replaceable by OpenGL and DirectX standards
54.
by ibains 6y ago
Prophecy.io | San Francisco bay area | UI Architect | OnSite Prophecy is a Data Engineering product on Spark/Airflow for Fortune 500. We’ve got a unique IDE based on visual studio code (browser) where you can do visual drag and drop, o
55.
by ibains 6y ago
We’re using Go for writing a kubernetes operator to deploy our product. Rest of our stack is in Scala - primarily since it is very good for writing compilers - we have a few in our product. We’re a data engineering product - think Spark IDE
56.
by ibains 6y ago
Very valid points: 1) Agree that Snowflake is far easier to use than Spark. 2) Agree that DBT is a great tool. ETL workflows normally processing 10s of TBs and workflows with large and complex business logic is the context. With Spark code
57.
by ibains 6y ago
Would love to get perspectives from HN community - did you decide between Snowflake and Spark for data engineering? Which one did you pick, and why?
58.
Spark vs. Snowflake: The Cloud Data Engineering (ETL) Debate(prophecy.io)
15 points by ibains 6y ago|4 comments
59.
ETL: Navigating the Cloud Transition (Architectures & Factors to consider)(prophecy.io)
5 points by ibains 6y ago|0 comments
60.
by ibains 6y ago
For the public beta and later our developer version, we only support Databricks. However, for our Enterprise version, we install within your network and this supports EMR, Dataproc, Databricks. This is only relevant where we are spinning up

More ›