4 ms·

I'm a data analyst / data application developer for a large humanitarian NGO. What: Lots of small data analysis projects, some large web projects. All projects

I'm a data analyst / data application developer for a large humanitarian NGO. What: Lots of small data analysis projects, some large web projects. All projects are in git repos or in our division's SVN. My work tends to involve ETL data flows, statistical analysis, web development and data visualisation. I'm not "full stack" as I never really play with hardware or do much systems administration. I do get to control a nice big, enterprisey Oracle data warehouse, with a few hundred users, but I've hardly played with nosql, redis, mapreduce, etc. Most of my problems can be solved in-memory, and most of my data fits on a single disk. How: SQL. SQL. More SQL. PL/SQL. Python (Pandas, scipy, numpy). R (these days only when I can't find a python lib). Javascript (just jquery, underscore and D3). Windows 7 while at work, Ubuntu + i3 wm at home. Edit in vim with very few plugins, everything else is done in zsh/powershell. IIS for production, Pyramid/Pylons Project for quick prototyping. A wide variety of RDBMS, but chiefly Oracle. I do lots of my templating/css in-browser using developer tools (FF, Chrome, IE9). My typical web stack is static html and javascript, with json served up out of purpose-specific database views. No need for an MV* framework, since I'm almost never dealing with transactional data. I also use Excel for a) doing quick data transforms b) building interactive, distributable data analysis apps in less than an hour. I'm trying to build an excel replacement with vim bindings and LISPy worksheet formulas.

gjreda 13y ago
My workflow is very similar. All data analysis, all the time. Tools: Lots of SQL (MySQL). AWS Elastic MapReduce + Hive for log analysis. Python (pandas). R (ggplot2 + stat modeling - haven't fully grok'd Python's statsmodels package yet). A little bit of D3, and a lot of command line. Maybe some Flask every once in a while to prototype out a reporting web app. I try to avoid Excel entirely because it always crashes on my MBP, but still find myself in it every so often to simply look at my dataset.
- siddboots 13y ago
  > ... but still find myself in it every so often to simply look at my dataset. Exactly this. I could never understand why there isn't some awesome, lightweight, terminal-based tool for looking at dataframes, and doing basic manipulation.
late2part 13y ago
DevOps for a public company with SaaS cloud offerings. iTerm ssh to machines chef/knife - look at config/status/recipes of servers vi - edit stuff, esp. recipes or scripts (wish I learned emacs, never have) scripts in bash/perl/python/ruby graphite - store data logstash - store logs, sort logs, quantify logs, send log stats and metrics to graphite jenkins - continuos integration, take dev code, run through tests, test deployment postgres - tuning, fighting, debugging, profiling redis - checking sensu/nagios - monitoring lots of good fun!

gjreda 13y ago

My workflow is very similar. All data analysis, all the time. Tools: Lots of SQL (MySQL). AWS Elastic MapReduce + Hive for log analysis. Python (pandas). R (ggplot2 + stat modeling - haven't fully grok'd Python's statsmodels package yet). A little bit of D3, and a lot of command line. Maybe some Flask every once in a while to prototype out a reporting web app. I try to avoid Excel entirely because it always crashes on my MBP, but still find myself in it every so often to simply look at my dataset.

siddboots 13y ago
> ... but still find myself in it every so often to simply look at my dataset. Exactly this. I could never understand why there isn't some awesome, lightweight, terminal-based tool for looking at dataframes, and doing basic manipulation.

late2part 13y ago

DevOps for a public company with SaaS cloud offerings. iTerm ssh to machines chef/knife - look at config/status/recipes of servers vi - edit stuff, esp. recipes or scripts (wish I learned emacs, never have) scripts in bash/perl/python/ruby graphite - store data logstash - store logs, sort logs, quantify logs, send log stats and metrics to graphite jenkins - continuos integration, take dev code, run through tests, test deployment postgres - tuning, fighting, debugging, profiling redis - checking sensu/nagios - monitoring lots of good fun!