6 ms·

I used to work for a Go shop. We dealt with financial data. I found it so annoying that many of my colleagues would use Go for one-off tasks such as aggregating

I used to work for a Go shop. We dealt with financial data. I found it so annoying that many of my colleagues would use Go for one-off tasks such as aggregating CSV files, updating the database with some data, or fetching data from the database, and then trying to make a plot. I saw my colleagues again and again implementing basic algorithms such as rolling median, or finding a maximum. Instead of loading data into Pandas and doing a group by, they would create some kind of loopy solution that would use maps. I totally understand why some people prefer Go in production to Python, but I could never understand why people wouldn't just learn the standard data science tools instead of reinventing the wheel in Go, always debugging their own off-by-one errors. It was difficult for me to trust the results of such analyses, given that I knew how many of the basic functions were written on the fly and probably not even tested. In the end, I didn't think it was a good use of the company's time. It felt more like an ego thing - thinking and showing that Go is sufficient. It reminds me of how people try to use iPads to code - only to show that they can do it.

physicsguy 2y ago
I've had exactly the same experience, it's a nice language but using it for things it's not suited for like data exploration makes no sense to me. Production data pipelines on the other hand, but only after testing them well and as you say, making sure there's good testing if you're implementing things like numerical routines.
- Lyngbakr 2y ago
  Do you have experience implementing ETL pipelines in Go? I think it'd be a better fit for us over our current language, but I'm curious to hear from people who've actually done it.
  - fifilura 2y ago
    What is current language and have considered doing it in SQL? I don't think go will be the right choice. It is just not its strength.
    - physicsguy 2y ago
      It depends on what you're doing right? The commenter here replied to me, and we're processing really large data files that are deliberately not in a SQL database due to size, only artefacts of these files eventually make it into a time series DB. For us Go works well and is performant without any great difficulty. For domain specific analytics we generally use Python, and Go just calls out to an API to do them.
      - fifilura 2y ago
        You are right. And I am mostly a "T" guy so I guess the answer was mostly about the transform. For extracting the data, go is probably a very good choice. But for transforming, pretty often not, although your use case may be suitable. In the end, the question was very open ended.
  - brushyamoeba 2y ago
    I do this at my job.. Disclaimer: I’m a web dev (“architect”) who does some lightweight data engineering tasks to facilitate views in some of my apps. My pipelines are very simple (no DAG-like dependencies across pipelines). I could just have separate scripts, but instead I have a monorepo of pipelines that implement an interface with Extract, Transform, Load methods. I run this as a single process that runs pipelines on a schedule and has an HTTP API for manually triggering pipelines. At some point I felt guilty that I am doing something nobody else seems to do, and that I had rolled my own poor-man’s orchestrator. I played around with Dagster and it was pretty nice but I decided it was overkill for my needs (however I definitely think the actual data analysis team at my company should switch from Jenkins to Dagster heh…) On a separate note, all of my pipelines Load into Elasticsearch, which I’m using as a data warehouse. I’ve realized this is another unconventional decision I’ve made, but it also seems to work well for my use-cases.
  - physicsguy 2y ago
    Yes. It works fairly well. With that said, I've got a feeling that life would be a lot easier to change around if we weren't using it, we end up writing a lot of code to do relatively simple things.
- caeril 2y ago
  > it's a nice language but using it for things it's not suited for like data exploration Pedantic point, but this is an issue of library support, not the language. For whatever reason, data scientists and ML researchers decided to write their libraries and tools in a Kindergartner's language with meaningful whitespace, dynamic typing, and various other juvenile PL features specifically aimed at five year olds and the mentally infirm.
  - physicsguy 2y ago
    Nobody would really use a compiled language for this, the compile-run-edit-cycle just takes too long. Prior to Python people really just used MATLAB and Mathematica for that sort of work in the physics/engineering side, and R and Stata/SPSS on the bio + maths side. MATLAB, Mathematica, Stata and SPSS are all commercial and R has exactly the same problems to Python in environment management and compiled binaries, if you use it today you end up doing a lot of manual compilation of dependencies and putting them in the PATH on Linux at least. Python became popular because the key scientific ecosystem libraries copied the libraries from MATLAB closely which made it easy to pick up, and because it was free. Anaconda made a distribution that was easy to install with all the dependencies compiled for you, which worked on Linux/Mac/Windows which made it much easier to use than R. The other interactive languages around at the time were Ruby which was heavily web dev focused, and Perl. Node didn’t yet exist. Once you have an ecosystem in a language it’s very hard to supplant. You need big resources to go against the grain. That no big company has decided to pour lots of money into alternatives even despite the problems probably tells us that it’s not viewed as being worth it.
arberavdullahu 2y ago
You don’t need to spend time learning standard data science tools. These tasks are well-defined, thoroughly documented in different blogs, and with today’s AI-driven code generation, basic programming knowledge is sufficient. Attempting to manipulate CSV data and generate visualizations in Go would be a waste of time, delivering subpar results.
theshrike79 2y ago
> I used to work for a Go shop. We dealt with financial data. People dealing with finance are THE MOST risk-averse people I know. No new tool will be used without years of vetting and it'll still be blamed if something goes wrong - even if the new tool never touched that bit of the process =) Source: Consulted for a finance company and it was Ye Olde Java and COBOL all the way down =)
- Someone 2y ago
  If they’re risk averse, they shouldn’t, as the OP claimed, implement basic algorithms such as rolling median, or finding a maximum instead of loading data into Pandas and doing a group by, would they?
- classified 2y ago
  I'm surprised they were adventurous enough to use Java.
  - theshrike79 2y ago
    Someone picked Java at the start of the millennium-ish and they never dared to move away. =)
- sanderjd 2y ago
  Using Go for data analysis is not the risk averse choice. If the comment were "everyone was using excel and maybe R and I couldn't get them to use python", then the risk aversion comment would be spot on :)
- lolinder 2y ago
  It sounds like every single data operation in this shop was done with a "new tool" written on the spot in Go.
- zerkten 2y ago
  >> People dealing with finance are THE MOST risk-averse people I know. No new tool will be used without years of vetting and it'll still be blamed if something goes wrong - even if the new tool never touched that bit of the process =) What exactly do you mean by "finance" in your context? Even within a huge organization like Bank of America you can have an extremely conservative part and an innovative part. Obviously, most employees are likely working for the conservative part, but that doesn't mean that there aren't some people playing using modern technology. It just doesn't get evenly distributed, and you can be many layers from that usage in your department.
classified 2y ago
> Instead of loading data into Pandas and doing a group by SQL can do `group by`. No need for overkill with "data science tools".
- sanderjd 2y ago
  If you have a csv, pandas is the "less overkill" route than loading it into a database to use sql.
  - chrisandchris 2y ago
    Each their own tool, but that's nothing awk can't do [1]. I prefer a shell script, because shell is everywhere. [1] https://stackoverflow.com/a/75073649 https://stackoverflow.com/a/75073649
    - sanderjd 2y ago
      Sorry, but no, awk and shell scripts are not a good choice for this kind of work. I'm sorry, but they just aren't! Sure, go for it if you're just doing stuff for your own personal interest. But if you're doing serious data-driven work in a professional environment, this is going to be an awful choice.
- infecto 2y ago
  While definitely doable, using a SQL database to do data discovery is obtuse.
  - orthoxerox 2y ago
    Not really. DuckDB can operate on raw csv without ingesting them. This is as easy if not easier than setting up python and pandas.
    - infecto 2y ago
      Totally agree that it depends on the tools available but given a typical toolset (sql and python). I would lean towards python because some types of analysis are easier to express in code, especially when working on top of the multiple data sets.
wejick 2y ago
I have done the same like your colleagues. Recently I discovered that pandas + jupyter notebook is much better tools. However as gopher that needs occasional data cruncher, I totally relying on LLM chat agent to do my work. But the rest of the work is still go, so the context switching is there but bare able. Imagine a go crawler dumping csv and a python script crunching the csv.
umvi 2y ago
I've mostly seen the opposite - pandas and jupyter notebooks shipped directly to production because the data scientists and AI guys didn't know how to do anything but python. As a result, the solutions were not performant and often had lots of runtime crashes due to python's more loose typing
- Spivak 2y ago
  I'm not sure how you're going to fix this given the data science tools are in Python. Are you gonna implement a half-broken 20% of numpy/scipy for a one-off program and then try to port? These libraries hide a lot of complexity and implementing even a few operators is a project.
  - supermatt 2y ago
    i dont think they are complaining about the libraries, but rather the scratchpads and notebooks that people use for ideation and evaluation being moved directly into a production environment because the authors don't have the experience or time to build more structured, efficient and maintainable code.
  - grahamjameson 2y ago
    Rust + Polars comes to mind.
- philote 2y ago
  If they're shipping notebooks to production and having so many crashes, I'd question that they even know how to do Python.
  - Narhem 2y ago
    When do you ship notebooks to production? Jupiter was never meant for external clients.
    - pclmulqdq 2y ago
      While terrifying, it is not uncommon to see python notebooks make it to production.
whateveracct 2y ago
> I used to work for a Go shop. We dealt with financial data. Hopefully you never forgot to initialize a numeric field and had it default to 0!
- ratorx 2y ago
  I think you are implying that this is incorrect behaviour. Could you explain why? The alternative would be to have it be undefined, which seems clearly worse. Or do you mean that it should be a compiler warning/error instead?
  - sapiogram 2y ago
    I've worked in Go for two years, and I hate zero values with a passion. I would much prefer undefined/nil/whatever as a default value. At least that obviously represents an invalid value, and will crash on use, become `null` in the database and when serialized to JSON. `0`, however, is indistinguishable from "missing data", and will just sit there and slowly poison your production data over time.
    - ratorx 2y ago
      I don’t think you mean that. By undefined, I was referring to C-style undefined behaviour stemming from reading uninitialised memory (which is the only alternative to not setting the memory to a known value when it is declared but not initialised). This is clearly worse than zero values because at least zero values won’t result in initialisation from memory which could be anything. If “null” is better for your use case, that’s not too difficult to emulate in Go (although perhaps not as ergonomic as it is in other languages). What you want is effectively an “optional” type rather than the type itself. The easiest way to represent this in Go is to use pointers to the type, which have the zero-value of nil. Combining this with JSON encoding ‘omitempty’ would get you the properties you were looking for. I don’t think the lack of default nullability is bad (look at all the languages that are trying to slowly phase it out by supporting non-nullable types).
      - 2y ago
RandomThoughts3 2y ago
> rolling median, or finding a maximum The real question is why are you writing code to make a plot when you can load a csv in excel and get a clean plot in 2 minutes.
- noisy_boy 2y ago
  Maybe because they need to do that every day with variations and automating away the monotonous chore is one of the main points of writing code.
  - RandomThoughts3 2y ago
    > I saw my colleagues again and again implementing basic algorithms such as rolling median, or finding a maximum. Emphasis mine so I will go with no regarding your explanation. It’s obvious the discussion was about one shot plotting from the start. The whole Go vs Python discussion for graphing doesn’t make sense in the context of automation.
- rcxdude 2y ago
  Because Excel sucks at plotting, especially if you have non-trivial amounts of data.
pantsforbirds 2y ago
I've almost entirely replaced Pandas with DuckDB in my day-to-day work. I wonder if it would be an easier lift for someone who is familiar with SQL, and doesnt want to pick up Pandas/Polars.
- ansgri 2y ago
  This comment might finally push me to try DuckDB, thanks. I’m quite proficient with scientific python in general, but Pandas API is just alien (and slow). SQL is more natural for its domain.
  - Daishiman 2y ago
    How's DuckDB better than using SQLite on an in-memory DB?
    - pantsforbirds 2y ago
      It's designed for analytics work. Some examples: * You could run a bunch of analytics queries directly on a jsonlines, csv, or parquet file (or even glob of files). * You can output directly to a pandas or polars dataframe * You can use numeric python functions inside of queries * You can also attach to S3 buckets, postgres databases, or sqlite databases and query them directly.
bitbasher 2y ago
For that kind of processing, I prefer to use the database itself. I mean, it is a "data" base and it's pretty good at it. I get it-- sometimes you have a half cooked spreadsheet someone sends you and you need to run some analysis on it.. in that case I don't see any harm in using Go, using visidata, importing it in sqlite or using pandas or whatever.
senkora 2y ago
XTX? I know that they use a lot of Go.
richrichie 2y ago
> I saw my colleagues again and again implementing basic algorithms such as rolling median, or finding a maximum. Instead of loading data into Pandas and doing a group by, they would create some kind of loopy solution that would use maps. Isn’t this trivial? I use these functionalities and maintain my own Go package for such utilities. A software firm should be able to do this without much issue. In fact, it is better done in house. Don’t have to worry about dependency management and downloading the entire Pandas library for just rolling means, etc.

physicsguy 2y ago

I've had exactly the same experience, it's a nice language but using it for things it's not suited for like data exploration makes no sense to me. Production data pipelines on the other hand, but only after testing them well and as you say, making sure there's good testing if you're implementing things like numerical routines.

Lyngbakr 2y ago
Do you have experience implementing ETL pipelines in Go? I think it'd be a better fit for us over our current language, but I'm curious to hear from people who've actually done it.
- fifilura 2y ago
  What is current language and have considered doing it in SQL? I don't think go will be the right choice. It is just not its strength.
  - physicsguy 2y ago
    It depends on what you're doing right? The commenter here replied to me, and we're processing really large data files that are deliberately not in a SQL database due to size, only artefacts of these files eventually make it into a time series DB. For us Go works well and is performant without any great difficulty. For domain specific analytics we generally use Python, and Go just calls out to an API to do them.
    - fifilura 2y ago
      You are right. And I am mostly a "T" guy so I guess the answer was mostly about the transform. For extracting the data, go is probably a very good choice. But for transforming, pretty often not, although your use case may be suitable. In the end, the question was very open ended.
- brushyamoeba 2y ago
  I do this at my job.. Disclaimer: I’m a web dev (“architect”) who does some lightweight data engineering tasks to facilitate views in some of my apps. My pipelines are very simple (no DAG-like dependencies across pipelines). I could just have separate scripts, but instead I have a monorepo of pipelines that implement an interface with Extract, Transform, Load methods. I run this as a single process that runs pipelines on a schedule and has an HTTP API for manually triggering pipelines. At some point I felt guilty that I am doing something nobody else seems to do, and that I had rolled my own poor-man’s orchestrator. I played around with Dagster and it was pretty nice but I decided it was overkill for my needs (however I definitely think the actual data analysis team at my company should switch from Jenkins to Dagster heh…) On a separate note, all of my pipelines Load into Elasticsearch, which I’m using as a data warehouse. I’ve realized this is another unconventional decision I’ve made, but it also seems to work well for my use-cases.
- physicsguy 2y ago
  Yes. It works fairly well. With that said, I've got a feeling that life would be a lot easier to change around if we weren't using it, we end up writing a lot of code to do relatively simple things.
caeril 2y ago
> it's a nice language but using it for things it's not suited for like data exploration Pedantic point, but this is an issue of library support, not the language. For whatever reason, data scientists and ML researchers decided to write their libraries and tools in a Kindergartner's language with meaningful whitespace, dynamic typing, and various other juvenile PL features specifically aimed at five year olds and the mentally infirm.
- physicsguy 2y ago
  Nobody would really use a compiled language for this, the compile-run-edit-cycle just takes too long. Prior to Python people really just used MATLAB and Mathematica for that sort of work in the physics/engineering side, and R and Stata/SPSS on the bio + maths side. MATLAB, Mathematica, Stata and SPSS are all commercial and R has exactly the same problems to Python in environment management and compiled binaries, if you use it today you end up doing a lot of manual compilation of dependencies and putting them in the PATH on Linux at least. Python became popular because the key scientific ecosystem libraries copied the libraries from MATLAB closely which made it easy to pick up, and because it was free. Anaconda made a distribution that was easy to install with all the dependencies compiled for you, which worked on Linux/Mac/Windows which made it much easier to use than R. The other interactive languages around at the time were Ruby which was heavily web dev focused, and Perl. Node didn’t yet exist. Once you have an ecosystem in a language it’s very hard to supplant. You need big resources to go against the grain. That no big company has decided to pour lots of money into alternatives even despite the problems probably tells us that it’s not viewed as being worth it.

arberavdullahu 2y ago

You don’t need to spend time learning standard data science tools. These tasks are well-defined, thoroughly documented in different blogs, and with today’s AI-driven code generation, basic programming knowledge is sufficient. Attempting to manipulate CSV data and generate visualizations in Go would be a waste of time, delivering subpar results.

theshrike79 2y ago

> I used to work for a Go shop. We dealt with financial data. People dealing with finance are THE MOST risk-averse people I know. No new tool will be used without years of vetting and it'll still be blamed if something goes wrong - even if the new tool never touched that bit of the process =) Source: Consulted for a finance company and it was Ye Olde Java and COBOL all the way down =)

Someone 2y ago
If they’re risk averse, they shouldn’t, as the OP claimed, implement basic algorithms such as rolling median, or finding a maximum instead of loading data into Pandas and doing a group by, would they?
classified 2y ago
I'm surprised they were adventurous enough to use Java.
- theshrike79 2y ago
  Someone picked Java at the start of the millennium-ish and they never dared to move away. =)
sanderjd 2y ago
Using Go for data analysis is not the risk averse choice. If the comment were "everyone was using excel and maybe R and I couldn't get them to use python", then the risk aversion comment would be spot on :)
lolinder 2y ago
It sounds like every single data operation in this shop was done with a "new tool" written on the spot in Go.
zerkten 2y ago
>> People dealing with finance are THE MOST risk-averse people I know. No new tool will be used without years of vetting and it'll still be blamed if something goes wrong - even if the new tool never touched that bit of the process =) What exactly do you mean by "finance" in your context? Even within a huge organization like Bank of America you can have an extremely conservative part and an innovative part. Obviously, most employees are likely working for the conservative part, but that doesn't mean that there aren't some people playing using modern technology. It just doesn't get evenly distributed, and you can be many layers from that usage in your department.

classified 2y ago

> Instead of loading data into Pandas and doing a group by SQL can do `group by`. No need for overkill with "data science tools".

sanderjd 2y ago
If you have a csv, pandas is the "less overkill" route than loading it into a database to use sql.
- chrisandchris 2y ago
  Each their own tool, but that's nothing awk can't do [1]. I prefer a shell script, because shell is everywhere. [1] https://stackoverflow.com/a/75073649 https://stackoverflow.com/a/75073649
  - sanderjd 2y ago
    Sorry, but no, awk and shell scripts are not a good choice for this kind of work. I'm sorry, but they just aren't! Sure, go for it if you're just doing stuff for your own personal interest. But if you're doing serious data-driven work in a professional environment, this is going to be an awful choice.
infecto 2y ago
While definitely doable, using a SQL database to do data discovery is obtuse.
- orthoxerox 2y ago
  Not really. DuckDB can operate on raw csv without ingesting them. This is as easy if not easier than setting up python and pandas.
  - infecto 2y ago
    Totally agree that it depends on the tools available but given a typical toolset (sql and python). I would lean towards python because some types of analysis are easier to express in code, especially when working on top of the multiple data sets.

wejick 2y ago

I have done the same like your colleagues. Recently I discovered that pandas + jupyter notebook is much better tools. However as gopher that needs occasional data cruncher, I totally relying on LLM chat agent to do my work. But the rest of the work is still go, so the context switching is there but bare able. Imagine a go crawler dumping csv and a python script crunching the csv.

umvi 2y ago

I've mostly seen the opposite - pandas and jupyter notebooks shipped directly to production because the data scientists and AI guys didn't know how to do anything but python. As a result, the solutions were not performant and often had lots of runtime crashes due to python's more loose typing

Spivak 2y ago
I'm not sure how you're going to fix this given the data science tools are in Python. Are you gonna implement a half-broken 20% of numpy/scipy for a one-off program and then try to port? These libraries hide a lot of complexity and implementing even a few operators is a project.
- supermatt 2y ago
  i dont think they are complaining about the libraries, but rather the scratchpads and notebooks that people use for ideation and evaluation being moved directly into a production environment because the authors don't have the experience or time to build more structured, efficient and maintainable code.
- grahamjameson 2y ago
  Rust + Polars comes to mind.
philote 2y ago
If they're shipping notebooks to production and having so many crashes, I'd question that they even know how to do Python.
- Narhem 2y ago
  When do you ship notebooks to production? Jupiter was never meant for external clients.
  - pclmulqdq 2y ago
    While terrifying, it is not uncommon to see python notebooks make it to production.

whateveracct 2y ago

> I used to work for a Go shop. We dealt with financial data. Hopefully you never forgot to initialize a numeric field and had it default to 0!

ratorx 2y ago
I think you are implying that this is incorrect behaviour. Could you explain why? The alternative would be to have it be undefined, which seems clearly worse. Or do you mean that it should be a compiler warning/error instead?
- sapiogram 2y ago
  I've worked in Go for two years, and I hate zero values with a passion. I would much prefer undefined/nil/whatever as a default value. At least that obviously represents an invalid value, and will crash on use, become `null` in the database and when serialized to JSON. `0`, however, is indistinguishable from "missing data", and will just sit there and slowly poison your production data over time.
  - ratorx 2y ago
    I don’t think you mean that. By undefined, I was referring to C-style undefined behaviour stemming from reading uninitialised memory (which is the only alternative to not setting the memory to a known value when it is declared but not initialised). This is clearly worse than zero values because at least zero values won’t result in initialisation from memory which could be anything. If “null” is better for your use case, that’s not too difficult to emulate in Go (although perhaps not as ergonomic as it is in other languages). What you want is effectively an “optional” type rather than the type itself. The easiest way to represent this in Go is to use pointers to the type, which have the zero-value of nil. Combining this with JSON encoding ‘omitempty’ would get you the properties you were looking for. I don’t think the lack of default nullability is bad (look at all the languages that are trying to slowly phase it out by supporting non-nullable types).
    - 2y ago

RandomThoughts3 2y ago

> rolling median, or finding a maximum The real question is why are you writing code to make a plot when you can load a csv in excel and get a clean plot in 2 minutes.

noisy_boy 2y ago
Maybe because they need to do that every day with variations and automating away the monotonous chore is one of the main points of writing code.
- RandomThoughts3 2y ago
  > I saw my colleagues again and again implementing basic algorithms such as rolling median, or finding a maximum. Emphasis mine so I will go with no regarding your explanation. It’s obvious the discussion was about one shot plotting from the start. The whole Go vs Python discussion for graphing doesn’t make sense in the context of automation.
rcxdude 2y ago
Because Excel sucks at plotting, especially if you have non-trivial amounts of data.

pantsforbirds 2y ago

I've almost entirely replaced Pandas with DuckDB in my day-to-day work. I wonder if it would be an easier lift for someone who is familiar with SQL, and doesnt want to pick up Pandas/Polars.

ansgri 2y ago
This comment might finally push me to try DuckDB, thanks. I’m quite proficient with scientific python in general, but Pandas API is just alien (and slow). SQL is more natural for its domain.
- Daishiman 2y ago
  How's DuckDB better than using SQLite on an in-memory DB?
  - pantsforbirds 2y ago
    It's designed for analytics work. Some examples: * You could run a bunch of analytics queries directly on a jsonlines, csv, or parquet file (or even glob of files). * You can output directly to a pandas or polars dataframe * You can use numeric python functions inside of queries * You can also attach to S3 buckets, postgres databases, or sqlite databases and query them directly.

bitbasher 2y ago

For that kind of processing, I prefer to use the database itself. I mean, it is a "data" base and it's pretty good at it. I get it-- sometimes you have a half cooked spreadsheet someone sends you and you need to run some analysis on it.. in that case I don't see any harm in using Go, using visidata, importing it in sqlite or using pandas or whatever.

senkora 2y ago

XTX? I know that they use a lot of Go.

richrichie 2y ago

> I saw my colleagues again and again implementing basic algorithms such as rolling median, or finding a maximum. Instead of loading data into Pandas and doing a group by, they would create some kind of loopy solution that would use maps. Isn’t this trivial? I use these functionalities and maintain my own Go package for such utilities. A software firm should be able to do this without much issue. In fact, it is better done in house. Don’t have to worry about dependency management and downloading the entire Pandas library for just rolling means, etc.