4 ms·

In my experience (which includes large companies from sectors like automotive and utilities) in most cases there is a large, RDBMS backed application doing some

by Pamar 10y ago

In my experience (which includes large companies from sectors like automotive and utilities) in most cases there is a large, RDBMS backed application doing something really critical... And then when you talk with actual users you discover a swarm of Excel files that either support or often second guess the corporate system. And it is not just reporting. In some cases inputting large quantities of tabular data is just done by writing Excel macros that create CSV files to upload to the system (and sometimes, horrifingly, the data are just retyped...).

mtone 10y ago
It's a scenario I've witnessed a few times too many now. Often an unsupported feature X is becoming critical to the team, so users reproduce large chunks of the corporate system in a separate spreadsheet in order to support the feature themselves, then build upon that. Without the ability to react quickly that smaller businesses have, I noticed that some form of resistance from IT is usually a big factor: lack of resources, divergent priorities or visions, unclear benefits from hypothetical feature, limited architecture, end-of-life maintenance states that end up lasting years, etc. Reality changes, systems fall behind, users adapt and spreadsheets are born. One root cause is that it's comparatively hard to add new data in a RDBMS - it usually requires a full development life cycle. On the other hand, spreadsheets are free, provide immediate benefits, allow users to define and refine requirements as they go, and can be abandoned at the drop of a hat (low risk). Quite the conundrum! From the article: > At present, Bakke’s tool enables query construction on an existing database, but it doesn’t enable the direct entry or modification of data. He expects to begin adding that functionality over the next six months Querying data is a problem with many good solutions nowadays, the bigger question is where and how to write that new data the users want to play with (other than in a spreadsheet).
- EGreg 10y ago
  So why can't they use something like Irmin? https://github.com/mirage/irmin https://github.com/mirage/irmin
- intrasight 10y ago
  >it's comparatively hard to add new data in a RDBMS I assume you meant to day "to add new schema". Adding new data is easy and in most enterprise environments is happening automatically every day. As far as "add new schema" goes, I'll claim that the database is just another software component, and if you pursuing an agile strategy, that you should be modifying that schema with every sprint. My client, for whom I do data warehouse consulting, updates the schema every two weeks. Because we are responsive to the business, they never revert to spreadsheet solutions. Of course they are free to create there own custom reports in spreadsheets - we'd never try to take that away. In fact we make it as easy as possible to run business curated queries and dump the data to Excel. Our newest feature for these power users is to allow them to give back to us the Excel reports they built and we will "push and deliver" - put the current data into the raw data sheets and deliver the report - usually to Sharepoint. Getting good business software solutions in place merely requires that the business hire some good developers. Why more don't do so remains a mystery. My best guess, like all things "corporate" is fear and laziness.
  - Pamar 10y ago
    Because we are responsive to the business, they never revert to spreadsheet solutions. Not that I do not believe you but... How hard have you looked? In my experience the largest the corporation the more chances there are that IT just does not realize what the situation is. "Shadow governance by spreadsheet" is something you notice only if you actually work in the same office withe business users for a few days, in my experience (or when the whole thing has grown so much that they cannot really manage it anymore then ask you to fold it back into the main system). Of course, YMMV.
    - intrasight 10y ago
      We work for the business, not for IT. That's a key difference. And that's what I mean by "business hire some good developers". As a developer, I find it much more satisfying.
billybofh 10y ago
I can relate to that - we used to have two related systems which couldn't talk to each other. So the solution was to print out (landscape, tiny font) 1000's of lines from one system, then sit with a ruler and highlighter pen re-typing it into the other. Day in, day out. A newer system which replaced some of the functionality and cost many millions of dollars will only export as .xls and only import from .csv (no API offered). Better than typing it in by hand, but it still pains me every day :-/
- Pamar 10y ago
  Glad (and at the same time sad) to see that plenty of others can relate to my experience.

by Pamar 10y ago

mtone 10y ago

It's a scenario I've witnessed a few times too many now. Often an unsupported feature X is becoming critical to the team, so users reproduce large chunks of the corporate system in a separate spreadsheet in order to support the feature themselves, then build upon that. Without the ability to react quickly that smaller businesses have, I noticed that some form of resistance from IT is usually a big factor: lack of resources, divergent priorities or visions, unclear benefits from hypothetical feature, limited architecture, end-of-life maintenance states that end up lasting years, etc. Reality changes, systems fall behind, users adapt and spreadsheets are born. One root cause is that it's comparatively hard to add new data in a RDBMS - it usually requires a full development life cycle. On the other hand, spreadsheets are free, provide immediate benefits, allow users to define and refine requirements as they go, and can be abandoned at the drop of a hat (low risk). Quite the conundrum! From the article: > At present, Bakke’s tool enables query construction on an existing database, but it doesn’t enable the direct entry or modification of data. He expects to begin adding that functionality over the next six months Querying data is a problem with many good solutions nowadays, the bigger question is where and how to write that new data the users want to play with (other than in a spreadsheet).

EGreg 10y ago
So why can't they use something like Irmin? https://github.com/mirage/irmin https://github.com/mirage/irmin
intrasight 10y ago
>it's comparatively hard to add new data in a RDBMS I assume you meant to day "to add new schema". Adding new data is easy and in most enterprise environments is happening automatically every day. As far as "add new schema" goes, I'll claim that the database is just another software component, and if you pursuing an agile strategy, that you should be modifying that schema with every sprint. My client, for whom I do data warehouse consulting, updates the schema every two weeks. Because we are responsive to the business, they never revert to spreadsheet solutions. Of course they are free to create there own custom reports in spreadsheets - we'd never try to take that away. In fact we make it as easy as possible to run business curated queries and dump the data to Excel. Our newest feature for these power users is to allow them to give back to us the Excel reports they built and we will "push and deliver" - put the current data into the raw data sheets and deliver the report - usually to Sharepoint. Getting good business software solutions in place merely requires that the business hire some good developers. Why more don't do so remains a mystery. My best guess, like all things "corporate" is fear and laziness.
- Pamar 10y ago
  Because we are responsive to the business, they never revert to spreadsheet solutions. Not that I do not believe you but... How hard have you looked? In my experience the largest the corporation the more chances there are that IT just does not realize what the situation is. "Shadow governance by spreadsheet" is something you notice only if you actually work in the same office withe business users for a few days, in my experience (or when the whole thing has grown so much that they cannot really manage it anymore then ask you to fold it back into the main system). Of course, YMMV.
  - intrasight 10y ago
    We work for the business, not for IT. That's a key difference. And that's what I mean by "business hire some good developers". As a developer, I find it much more satisfying.

billybofh 10y ago

I can relate to that - we used to have two related systems which couldn't talk to each other. So the solution was to print out (landscape, tiny font) 1000's of lines from one system, then sit with a ruler and highlighter pen re-typing it into the other. Day in, day out. A newer system which replaced some of the functionality and cost many millions of dollars will only export as .xls and only import from .csv (no API offered). Better than typing it in by hand, but it still pains me every day :-/

Pamar 10y ago
Glad (and at the same time sad) to see that plenty of others can relate to my experience.