6 ms·

I've had these strong feelings and the OP describes it really well. Despite being a polyglot programmer, I really struggle with Python, both in expression and

I've had these strong feelings and the OP describes it really well. Despite being a polyglot programmer, I really struggle with Python, both in expression and performance (unless it's just config for GPUs). Some of this frustration was recently an "Unpopular Opinion" on the Go Time Podcast regarding Python being great for "data exploration" but not for "data engineering": https://changelog.com/gotime/304#t=3196 https://changelog.com/gotime/304#t=3196 I've been yearning for better interactive tooling and ML-related libraries to bridge this gap and started using some even in just the last week: * GoNB (Golang-support for Jupyter notebooks, also from a Googler) https://github.com/janpfeifer/gonb https://github.com/janpfeifer/gonb * That uses Go-Plotly for graphs/UI: https://github.com/MetalBlueberry/go-plotly https://github.com/MetalBlueberry/go-plotly * GoMLX (GoNB author is also on that project, many thanks Jan!) https://github.com/gomlx/gomlx https://github.com/gomlx/gomlx * Hidden at the end of OP is LangChainGo for LLMs, which I haven't used yet: https://github.com/tmc/langchaingo https://github.com/tmc/langchaingo Pick those up and let's make the Go community stronger together!

mountainriver 3y ago
Go will sadly never be viable for ML tasks due to its C interop. Maybe calling an LLM api but often you have a mix of models. Rust also struggles here a bit due to its poor C++ interop. Written but someone who has built ML libraries from scratch in Go
- Art9681 3y ago
  Can you elaborate on this or point me to some more information? I am primarily a Go dev that has been integrating AI/ML into a personal project. Like many other apps, I wrap llama.cpp and various other popular ML projects. Recently I have been getting the itch to go lower level and see if it would be feasible to write an LLM backend in pure Go, or a Stable Diffusion backend. Just for curiosity and learning. But if this is a dead-end I would like to know from someone more experienced than I. Thanks for any advice you can offer in advance.
  - nerdix 3y ago
    Do you know of any resources with details on using llama.cpp with Go?
- neomantra 3y ago
  I'm starting to explore this for Time-Series analysis -- still at the beginning as I mostly used C++ before. Torch.ch (so old now!) used LuaJIT which has excellent FFI. What is the realm of the interop problem there? That one wants to assemble data in Go and then feed it to another API? Or even at a lower level, getting data to a GPU/TPU requires C interop so it's more about the memory sharing? Those libraries come from Google, so hopefully they can help battle these issues in the nether realms.
- LettuceSand12 3y ago
  Why not elixir then? Good c interop, functional, and good performance.
  - skldj28d2 3y ago
    Erlang/Elixir are very slow for most computationally expensive tasks.
    - LettuceSand12 3y ago
      Isn’t python the same? Most of the work is done by c++ or rust libs with python as glue.
      - skldj28d2 3y ago
        Python is the same but there's not nearly enough incentive to move to another slow language. A lot of people that are interested in using something other than Python want a language that doesn't require everything to be written in C, C++ or Fortran and used via bindings. Having most of the ecosystem written in one expressive and relatively fast language has huge benefits.
- cdelsolar 3y ago
  At my last company I built a Go tensorflow inference server.. it worked well
agent281 3y ago
My unpopular opinion is that pandas specifically is terrible for data engineering. The type inference when loading data from files can result in fragile jobs. In my experience, it's also surprisingly slow for a lot of data engineering tasks. IMO, it should just be used for analysis. Python's parallelism story also holds it back, but that may be improving soon. It's unlikely it will be as nice as Go's though.
- isoprophlex 3y ago
  Is that unpopular? I rarely meet people who actually enjoy wading through the baroque, underperforming forest of footguns that is the pandas API.
  - agent281 3y ago
    When I would interview data eng candidates, a large subset would import pandas before reading the problem statement. When I've complained about it, a lot of people are surprised to find a data engineer who doesn't like it. That said, it might just be the companies I've worked at. ¯ \ _ ( ツ ) _ / ¯
    - isoprophlex 3y ago
      Keep on educating them I'd say ;)
wara23arish 3y ago
I also noticed at the last Gophercon in san diego that a lot of talks were regarding data engineering and go. There was even a university professor with a data eng courses that presented about how he teaches most of it in Go and believes its better. (obv since hes at gophercon :) )
GaryNumanVevo 3y ago
I would be really surprised if Go becomes the next ML modelling language. Python has always been a stretch. My money is on Julia in the next 5 years.

mountainriver 3y ago

Go will sadly never be viable for ML tasks due to its C interop. Maybe calling an LLM api but often you have a mix of models. Rust also struggles here a bit due to its poor C++ interop. Written but someone who has built ML libraries from scratch in Go

Art9681 3y ago
Can you elaborate on this or point me to some more information? I am primarily a Go dev that has been integrating AI/ML into a personal project. Like many other apps, I wrap llama.cpp and various other popular ML projects. Recently I have been getting the itch to go lower level and see if it would be feasible to write an LLM backend in pure Go, or a Stable Diffusion backend. Just for curiosity and learning. But if this is a dead-end I would like to know from someone more experienced than I. Thanks for any advice you can offer in advance.
- nerdix 3y ago
  Do you know of any resources with details on using llama.cpp with Go?
neomantra 3y ago
I'm starting to explore this for Time-Series analysis -- still at the beginning as I mostly used C++ before. Torch.ch (so old now!) used LuaJIT which has excellent FFI. What is the realm of the interop problem there? That one wants to assemble data in Go and then feed it to another API? Or even at a lower level, getting data to a GPU/TPU requires C interop so it's more about the memory sharing? Those libraries come from Google, so hopefully they can help battle these issues in the nether realms.
LettuceSand12 3y ago
Why not elixir then? Good c interop, functional, and good performance.
- skldj28d2 3y ago
  Erlang/Elixir are very slow for most computationally expensive tasks.
  - LettuceSand12 3y ago
    Isn’t python the same? Most of the work is done by c++ or rust libs with python as glue.
    - skldj28d2 3y ago
      Python is the same but there's not nearly enough incentive to move to another slow language. A lot of people that are interested in using something other than Python want a language that doesn't require everything to be written in C, C++ or Fortran and used via bindings. Having most of the ecosystem written in one expressive and relatively fast language has huge benefits.
cdelsolar 3y ago
At my last company I built a Go tensorflow inference server.. it worked well

agent281 3y ago

My unpopular opinion is that pandas specifically is terrible for data engineering. The type inference when loading data from files can result in fragile jobs. In my experience, it's also surprisingly slow for a lot of data engineering tasks. IMO, it should just be used for analysis. Python's parallelism story also holds it back, but that may be improving soon. It's unlikely it will be as nice as Go's though.

isoprophlex 3y ago
Is that unpopular? I rarely meet people who actually enjoy wading through the baroque, underperforming forest of footguns that is the pandas API.
- agent281 3y ago
  When I would interview data eng candidates, a large subset would import pandas before reading the problem statement. When I've complained about it, a lot of people are surprised to find a data engineer who doesn't like it. That said, it might just be the companies I've worked at. ¯ \ _ ( ツ ) _ / ¯
  - isoprophlex 3y ago
    Keep on educating them I'd say ;)

wara23arish 3y ago

I also noticed at the last Gophercon in san diego that a lot of talks were regarding data engineering and go. There was even a university professor with a data eng courses that presented about how he teaches most of it in Go and believes its better. (obv since hes at gophercon :) )

GaryNumanVevo 3y ago

I would be really surprised if Go becomes the next ML modelling language. Python has always been a stretch. My money is on Julia in the next 5 years.