3 ms·
So, part of the work i'm doing at the moment (mainly devops for a pre-revenue startup) involves doing an ETL sub-project (scrape data from websites, ingest into
by zemanel 10y ago
So, part of the work i'm doing at the moment (mainly devops for a pre-revenue startup) involves doing an ETL sub-project (scrape data from websites, ingest into a database) which i'm implementing with Django, Django Rest Framework, Scrapy, Celery, Haystack (with ElasticSearch) and Postgres, with the Django Admin configured for data curation/high level job monitoring. It basically took me half a sprint (1 week plus a couple of days) to get a large part of the work done (Scrapy spider work was mosly done by a previous developer), including unit tests for the API/DRF code base, CI integration (Codeship) and a local dev Vagrant setup.
The rest of our stack (the main backend API), which was poorly implemented by a non-proficient developer (which i wouldn't consider a bad developer, merely non-experienced; he eventually moved jobs), is currently done in NodeJS/Express/Sequelize and IMHO should be scrapped and rewritten.
Recently, during a conversation with a new hire frontend developer, he mentioned the ETL sub-project should be done in NodeJS as well, for same language consistency across the stack and i basically replied that if he was willing to rewrite it (with the same features i get from solid, highly featured mature community libraries) in NodeJS himself, he'd be free to do it :-)
Unfortunately, the CTO <rant>which IMHO is pretty null at managing people/software projects and sole aspiration seems to be becoming the CTO of the next Trello</rant>keeps throwing out buzzwords as "Kafka based event data driven application", "snappy websockets driven UI" and "microservices", while in the last year was hardly able to get a buggy over engineered MVP up and doesn't seem to have taken a Q, which means i'm looking for a new job. In the middle of all this NodeJS madness, i'm hopeing i have not become a dungeon Master myself :-)
- alxndr 10y ago> "i'm hopeing i have not become a dungeon Master myself :-)" I have bad news for you...