3 ms·
I was working at a place that had decided that they needed "big data tools" and part of the team was developing a Hadoop data cluster and myriad of other tools
by Tistel 11y ago
I was working at a place that had decided that they needed "big data tools" and part of the team was developing a Hadoop data cluster and myriad of other tools (Tableau, Reddis cache etc).
I put all the campaign (it was an ad tech company) data that we actually used to alter bidding behaviour and it could fit into a sqlite3 DB that was < 5MB. Using python the sqlite DB worked like a charm.
- mrweasel 11y agoPeople like to believe that they have "big problems" and that they need large scale solutions. If they ever manage to get it working correctly they now have big system administration problems on top of their existing problem. Where as doing backups, creating staging/testing environments or moving servers can be done with cp or scp when running SQLite. Agreeably if you pick SQLite and ever manage to reach a scale where it's no longer an appropriate solution, then you will have more trouble migrating of it.
- dtech 11y agoI would say that SQLite transitions quite nicely, as far as transitions go, into a client-server SQL RDBMS.
- sanderjd 11y ago> People like to believe that they have "big problems" and that they need large scale solutions. People like to believe that they will have big problems and will need large scale solutions, and often make the call to plan accordingly. Decision-making for this sort of thing is a tightrope walk across a bunch of unknowables affecting a bunch of trade-offs.