7 ms·

These are great innovations but can we please take a moment to realize 99% of companies are still stuck with a blend of excels, access and sql servers? Why is a

by kfk 6y ago

These are great innovations but can we please take a moment to realize 99% of companies are still stuck with a blend of excels, access and sql servers? Why is adoption of this new tech so poor? Maybe it has something to do with the amount of confusion all the sales pitches about data lakes, big data, ai and company are generating

t0mas88 6y ago
Because most data isn't "big" data. If it fits in Excel on a laptop, why bother to roll out a distributed data lake system like Spark with all the associated ops work.
- noxvilleza 6y ago
  ... and if it doesn't fit on Excel on a laptop, the natural path is just some database running on a desktop, then a server, then a beefier server. It's actually amazing how far just something like postgres will scale for your data needs.
phoe-krk 6y ago
> Why is adoption of this new tech so poor? Because it's unnecessary to those 99% of companies. If company data fits in an Excel spreadsheet, Access database, or a single MySQL/Postgres instance, then introducing all this new tech with all of the associated costs and little return gain is a net loss.
- baskire 6y ago
  Might even see negative gain. Often time the query perf per machine goes down as you move from a single node, indexed, postgresql To a multi node unstructured data. If you don’t have so much data or unstructured data. Why use the more complex and generally slower per FLOP solution.
- kfk 6y ago
  Some of this new tech is simpler to run than MySQL/Postgres like S3/Spectrum. Excel is ok for last mile type of analytics but it doesn't work if you are looking for serious insights. Excel also generates a significant amount of unnecessary manual work and data quality issues. It's not so black and white, what I am saying is that the analytics tech industry direction doesn't seem to address the real issues.
  - phoe-krk 6y ago
    Sure. It's possible to create something that is both good for small data and has a better feature set than Excel/Access/etc., but then you do not have the other main asset of those databases: popularity. It's relatively very easy to find people skilled in these technologies, whereas new technologies need to come either with advocates capable of teaching others on their own or programmers already skilled with them. Otherwise, teaching people new technical skills is costly.
QuesnayJr 6y ago
I actually think the effect of all of this hype will move people past the Excel/Access era. There are simple analyses every company could do with R or scikit-learn that would save or make them money, and they just don't know how. Someone with AI expertise is over-qualified to do this, but they are at least qualified.
hztar 6y ago
The ones I have been talking to stick to their Excel because their little part of the puzzle can solved by Excel. These technologies usually demand data to be collected, but the amount of incentive to do so is low in any classic balkanized F500 organization.
vmsp 6y ago
I know Excel runs the world but are there really that many people using Access?
- Tarq0n 6y ago
  In very technology deficient enterprise organizations it's sometimes the only query tool people are allowed access to.
  - phoe-krk 6y ago
    > are allowed access to. I enjoyed the pun.
fukmbas 6y ago
Because 99% of new tech is bullshit. It changed too much, no one can keep up. These old stacks are still around because they're proven to get the job done.
Draken93 6y ago
There are two scenarios 1. It is not necessary. For smaller datasets excel or sql servers are sufficient. 2. The small dataset has grown over the years and a other solution would be beneficial. But people like the things they are familiar with. They do not want to have the additional work, of learning new tools. Who ever wants to change the existing structures, will usually face high resistance. In the second case, companies often accept the issues of the old solution and they tell themself it is the right way.
magicalhippo 6y ago
Most of our customers have SQL databases in the 5-20GB range. The largest one is around 100GB. Most of them self-host. What would "this new tech" bring us and our customers, and would it be enough to justify the cost of porting?

by kfk 6y ago

t0mas88 6y ago

Because most data isn't "big" data. If it fits in Excel on a laptop, why bother to roll out a distributed data lake system like Spark with all the associated ops work.

noxvilleza 6y ago
... and if it doesn't fit on Excel on a laptop, the natural path is just some database running on a desktop, then a server, then a beefier server. It's actually amazing how far just something like postgres will scale for your data needs.

phoe-krk 6y ago

> Why is adoption of this new tech so poor? Because it's unnecessary to those 99% of companies. If company data fits in an Excel spreadsheet, Access database, or a single MySQL/Postgres instance, then introducing all this new tech with all of the associated costs and little return gain is a net loss.

baskire 6y ago
Might even see negative gain. Often time the query perf per machine goes down as you move from a single node, indexed, postgresql To a multi node unstructured data. If you don’t have so much data or unstructured data. Why use the more complex and generally slower per FLOP solution.
kfk 6y ago
Some of this new tech is simpler to run than MySQL/Postgres like S3/Spectrum. Excel is ok for last mile type of analytics but it doesn't work if you are looking for serious insights. Excel also generates a significant amount of unnecessary manual work and data quality issues. It's not so black and white, what I am saying is that the analytics tech industry direction doesn't seem to address the real issues.
- phoe-krk 6y ago
  Sure. It's possible to create something that is both good for small data and has a better feature set than Excel/Access/etc., but then you do not have the other main asset of those databases: popularity. It's relatively very easy to find people skilled in these technologies, whereas new technologies need to come either with advocates capable of teaching others on their own or programmers already skilled with them. Otherwise, teaching people new technical skills is costly.

QuesnayJr 6y ago

I actually think the effect of all of this hype will move people past the Excel/Access era. There are simple analyses every company could do with R or scikit-learn that would save or make them money, and they just don't know how. Someone with AI expertise is over-qualified to do this, but they are at least qualified.

hztar 6y ago

The ones I have been talking to stick to their Excel because their little part of the puzzle can solved by Excel. These technologies usually demand data to be collected, but the amount of incentive to do so is low in any classic balkanized F500 organization.

vmsp 6y ago

I know Excel runs the world but are there really that many people using Access?

Tarq0n 6y ago
In very technology deficient enterprise organizations it's sometimes the only query tool people are allowed access to.
- phoe-krk 6y ago
  > are allowed access to. I enjoyed the pun.

fukmbas 6y ago

Because 99% of new tech is bullshit. It changed too much, no one can keep up. These old stacks are still around because they're proven to get the job done.

Draken93 6y ago

There are two scenarios 1. It is not necessary. For smaller datasets excel or sql servers are sufficient. 2. The small dataset has grown over the years and a other solution would be beneficial. But people like the things they are familiar with. They do not want to have the additional work, of learning new tools. Who ever wants to change the existing structures, will usually face high resistance. In the second case, companies often accept the issues of the old solution and they tell themself it is the right way.

magicalhippo 6y ago

Most of our customers have SQL databases in the 5-20GB range. The largest one is around 100GB. Most of them self-host. What would "this new tech" bring us and our customers, and would it be enough to justify the cost of porting?