yeshengm

4 ms·

1. 1.▲
   by yeshengm 6y ago

   Since you need fast lookup, traditional transactional row store like RocksDB seems a better fit.
2. 2.▲
   by yeshengm 6y ago

   Are these execution engines internally using Arrow columnar format or are they just exposing Arrow as a client wire format? AFAIK Spark and Presto does not use Arrow as execution columnar format, but just data sources/sinks.
3. 3.▲
   Ask HN: Why there are so many AI compilers?

   1 points by yeshengm 6y ago|0 comments
4. 4.▲
   by yeshengm 6y ago

   Yeah. Likely a small amount of analytics jobs taking up most memory/CPU footprint, while critical prod OLTP jobs take very little resource comparably.
5. 5.▲
   by yeshengm 6y ago

   Nowadays most Spark users use either PySpark or Spark SQL. Spark is designed in some sense to mimic Scala collection library. Rust is nice but its interop with other open source big data libs (Hadoop/Zookeeper/HBase) can be pain
6. 6.▲
   by yeshengm 6y ago

   I'd say start with some eager functional language such as OCaml without the "object" part. Haskell code is hard to debug cuz you can't rely on good old print. Monad itself is never a silver bullet. Type systems and Compo