ibains

13 ms·

61.
by ibains 6y ago
Raj here, I'm the founder of Prophecy. Previously, I was the product manager for Apache Hive at Hortonworks (and wrote compilers for CUDA at NVIDIA, and Microsoft). The tooling for data engineers is really poor, and I saw many of my cu
62.
Show HN: Prophecy Spark IDE public beta(prophecy.io)
12 points by ibains 6y ago|7 comments
63.
by ibains 6y ago
Super excited to share the public beta with HN community. We think Prophecy is a big step up in Data Engineering tools, and would love to get feedback on the product. The public beta has Spark built-in, so you can just spin up clusters, ple
64.
by ibains 6y ago
Has anyone used Enterprise schedulers - ControlM, Atom and know how it compares to Airflow?
65.
by ibains 6y ago
Maybe the path is to merge visual and code programming. We're doing that for data engineering - you can draw in visual editor, and press a button and you have code. So as long as you constrain the domain, it seems doable. https:/
66.
Show HN: ProphecyHub: Metadata Re-Invented with Git,GraphQL for Data Engineering(medium.com)
22 points by ibains 6y ago|0 comments
67.
Ask HN: How to learn history of Enterprise software
1 points by ibains 6y ago|0 comments
68.
Spark: Column level lineage for Data Engineering(medium.com)
2 points by ibains 7y ago|0 comments
69.
by ibains 7y ago
Thanks for the great inputs, pulling Git info back into the IDE is a great idea. Will definitely add it!
70.
by ibains 7y ago
Excited about sharing the new interface we're building for Apache Spark. We're also trying live screen gifs for the first time - quite an effort to get GIF size down.
71.
Spark deserves a better IDE(medium.com)
13 points by ibains 7y ago|3 comments
72.
by ibains 7y ago
I’ve worked in compilers (optimizations) at Microsoft (vc++) and NVIDIA(CUDA) - register allocators, schedulers and loop optimizers were super fun. Then I worked on database (query) compilers. I managed Apache Hive at Hortonworks, and that
73.
by ibains 7y ago
Prophecy.io - Cloud Native Data Engineering. Hiring for Spark (database) internals and Scala. Offices in San Francisco, Gurgaon, Bangalore. Onsite. Contact Raj at +1.415.310.1638
74.
by ibains 7y ago
Hi everyone! We write parsers and didn't find a good post showing how to solve practical problems using parser combinators in Scala. Sharing our experience and hoping to learn from yours.
75.
Scala Packrat Parser Combinators for DSLs(medium.com)
6 points by ibains 7y ago|1 comments
76.
by ibains 7y ago
I worked on compilers at Microsoft, NVIDIA and there are few jobs for traditional hardware compilers. Some new hardware is being designed though, most fun was new optimizations specific to GPUs different caches such as loop optimizer and re
77.
by ibains 7y ago
We're a scala shop, we get a functional language with features such as parser combinators (packrat parsers) - super useful for building DSLs. Functional-Relational-Mapping (for comprehensions) for SQL. We're able to get good web s
78.
by ibains 7y ago
Hi everyone! in trying to find product market fit, I've learned the great value of customer discovery. I've written it how I'd share with a friend starting her own startup. Look forward to learning from other people's ex
79.
Startup Superpower: Customer Discovery Machine(medium.com)
3 points by ibains 7y ago|1 comments
80.
by ibains 7y ago
This is the workflows written in legacy formats - Ab Initio, Informatica, IBM DataStage - this is a $10B+ industry - so lots of usage out there. They typically have graph formats and Domain Specific Languages to specify transformations. We
81.
by ibains 7y ago
The variation of complexity of transforms across Enterprises and teams within Enterprises is quite large. Spark and Hive (that I was product manager for, and used by hundreds of Enterprises) are used quite heavily in data engineering. Many
82.
by ibains 7y ago
Well, simplest way to put it is - you get an Data Engineering (ETL) product - that'll move your 10s of thousands of ETL workflows from legacy ETL product to Spark, give you a full stack ETL product on Spark, and move you from on-premis
83.
by ibains 7y ago
We're not adding a layer on top of all big data platforms, instead we work on Spark directly. This has advantages, disadvantages. When you use our product, your code looks very similar to writing Spark code in IntelliJ. You get visual
84.
by ibains 7y ago
Foremost, we're focused on Transforms - helping users create and manage complex ETL transforms - this is your business logic. One is the category of schedulers - airflow, astronomer - these are focused on scheduling these Transforms, s
85.
by ibains 7y ago
We're focused on complex transforms for high volume & performance workloads - something you'd find in Enterprise core production ETL workflows. We're a complete replacement for ETL products - you'd use us to replace
86.
by ibains 7y ago
I'd love to understand what your preferred interface looks like, and get your input in building the interface, please send me a mail at raj.bains@prophecy.io
87.
by ibains 7y ago
Hi Everyone! Super excited for our first release!! I was product manager for Apache Hive at Hortonworks and earlier an early engineer at CUDA at NVIDIA - focusing on compiler optimizations. As product manager, I saw so many customers strugg
88.
Show HN: Prophecy.io – Cloud Native Data Engineering(medium.com)
64 points by ibains 7y ago|22 comments
89.
by ibains 7y ago
Architect, Principal Engineer | Seed stage startup | San Francisco, Bangalore, Gurgaon. ONSITE - We’re building Cloud Native Dataflows on Spark and Kubernetes to replace ETL. Need expertise in one or more of Programming Languages/Compi