40 ms·

Hahahaha. I stumbled upon Flow last year when I switched jobs. Our IT/procurement process is ridiculous, so I scrounged around our intranet resources for licens

by cosmie 8y ago

Hahahaha. I stumbled upon Flow last year when I switched jobs. Our IT/procurement process is ridiculous, so I scrounged around our intranet resources for licenses and tools that I had access to by default or could get/request access to without triggering the Eye of Sauron to gaze down upon me from the tower of IT Security. My team does analytics consulting work for large multi-agency marketing campaigns, and I used Flow and a shared mailbox to create sanity around managing the constant flood of daily/weekly/monthly data snapshots that get emailed and have to be synthesized and combined for end reporting/analysis. I was also able to use Flow and PowerBI to duct tape together a "starter data lake" architecture for our marketing clients. It's been a wild success with clients, due in part because of it's price (in context to their expectations for traditional "data lake pitches") but even more so due to the fact that it gets rubber stamped through any IT review thanks to being all part of Microsoft. It fits most clients workflows and data volumes, and seamlessly transitions to a true data lake by pulling in Azure services as needs mature. Most marketing departments are swimming in data but pretty lacking in their access to technical and process/operations resources to truly manage and leverage it. Flow has been a fantastic tool to introduce structured data management and automation for these clients, as it's low cost enough to slide under the radar, already implicitly allowed software as part of their Office 365 license, and gives them tangible results to show when asking for more budget to develop/mature it further. It's far from my favorite stack to use, but it's good enough to get the job done and breezes through all of the checks and objections and red tape that would make virtually any other stack dead on arrival.

clanrebornsx 8y ago
Yuck so u help those scam MLM companies flourish?
viksit 8y ago
I hope you got recognition for this work!
- cosmie 8y ago
  Thank you! It has certainly been met with recognition, to the point where I'm essentially having to become a people manager again and create a team to support the capability across more clients.
anitil 8y ago
That is such great work. Getting the job done with the tools already around is just such a good feeling.
jac_no_k 8y ago
/me is one of the minions from team "Eye of Sauron". However I'm not stuck in this tower. I'm torn between two worlds. One side applauds the grass roots based solutions and the JFDI attitude. The other side is concerned that these solutions becomes widely used, de facto production, and problems upstream ends up breaking the output. On error, hopefully fails loudly but subtle problems could creep in. If the reports are consumed by other automation, the problem spreads rapidly. Ideally the IT department would take a step back, assess what the users (you the stakeholder) needs are, and provide the solutions. Better Agile. Using the successes to drive more development and operations budget is the way to go.
- lbatx 8y ago
  Until IT departments get infinite resources (that is, never) there will always be a need for these JFDI solutions. Some group will always go starved for attention and have to cobble together their own solutions. IT backlogs are always longer than the resources to do the work.
  - edelans 8y ago
    JFDI = Just Fucking do It (in case other ppl wonder what it means... =) )
    - nirav72 8y ago
      Or what we call 'Shadow IT' in my department.
- cosmie 8y ago
  I completely understand the terrors of hidden dependencies on shadow IT. The work I do tends to grow up organically in different areas of every org, so I've hopped between functional business units, centralized business operations, and IT. When working for business units, I'm careful when I introduce technically advanced process dependencies. I either limit it to the level of technical competency that is generally accessible to the team or else I adopt whatever toolset the IT or development side of the house uses. That way there's always the potential for transition and I don't screw my manager/team over if I leave (or get stuck supporting it in perpetuity while I'm there). This was an example of that. We do client consulting work, but we're only about 2% of the total revenue of our Fortune 500 parent and our IT policies are not conducive to client-driven work requirements. When my manager looked for what we could use when supporting analytics-oriented aspects of client work, tools like Supermetrics[1] were explicitly denied and IT provided virtually no automation/analytics solutions short of a complete Tableau server, which is prohibitively expensive for most of our clients' needs (especially when allocating IT infrastructure & support costs in addition to the license). I'm not a fan of the fragility and monotony of repetitive manual data collection and reporting, nor do clients like to pay for the excessive hours that takes on a recurring basis compared to competing bids (because presumably our competitors can use SaaS tools for automating rote process). I scrounged around and found a pocket of PowerBI users in our IT's SQL Server team. Connected with the director over that team and got access for myself. Created a proof of concept architecture that provided an agile platform to address the full spectrum of our potential needs from "throw in basic analytics for free and eat the cost" to "maintain a full on data lake getting a deluge of real time data streams". Client users are guests invited to our tenant, log in with their (usually) already-existing Office 365/AzureAD user, can have the $10/user/month PowerBI Pro license added to their account via their corporate IT, our corporate IT, or a credit card expense. All of their ETL and data connectors and credentials and PowerBI models are compartmentalized into their own app workspace and subject to corporate DLP policies and accessible/auditable by Office admins. It's technically unsanctioned/shadow IT, but I ran it by the director of that team in case I needed to tap his team for sporadic support needs in the future, and he was both interested in my proposed usage and perfectly fine giving it his blessing (which is great, because his team manages the permissions I need for provisioning users, adding licenses, and creating workspaces). The alternative would have been to either lose business due to non-competitive bids, use a combination of unsanctioned SaaS tools that get expensed and subject to disruption when turnover happens and passwords don't get shared or credit card renewals don't occur (fairly common scenario I've inherited), or using development skills to create custom automation that would be unmaintainable when I left. > Ideally the IT department would take a step back, assess what the users (you the stakeholder) needs are, and provide the solutions. Better Agile. This works well in some cases, but falls flat on it's face just as often. It's often prohibitively expensive from a budgetary standpoint for a business team to pull in resources for a needs analysis and subsequent project. Business users are far cheaper, so the vast majority of drift between needs and systems capabilities/processes go unresolved because it's cheaper to absorb the inefficiency on the business labor for quite a while before it hits a critical point that justifies the expense of pulling in ops/dev staff. For most business teams, shadow IT is a pragmatic stopgap solution to hit their deliverables, similar to the type of tech debt you see in IT projects that gets kicked down the road until it's suddenly not something that can be kicked anymore. And the process of IT stumbling across a grass roots solution acting as a de facto production dependency is basically a litmus test for when the debt finally comes due and triggers a formal needs analysis and IT project. As well, that type of needs analysis generally creates localized optimizations or poor user adoption. The current needs will be a reflection of the processes that evolved within the current system and constraints, and addressing the directly will usually just result in a solution that's "a faster horse". Or it goes to the other extreme, where the solution you provide is state of the art and it turns out they have no ability to transition from riding their current horse to piloting the new fighter jet you provided. A far more effective approach is to creative a collaborative model. Grass roots based solutions exist for a multitude of reasons and won't ever go away. But the concerns of undocumented, unknown, and potentially unmaintained de facto production dependencies existing outside of the purview of IT is a very real concern. So create a migration path. Put in low friction processes, support structures, and policies that give power users on the business side the ability to create sanctioned grass roots solutions. It'll allow business teams to experiment with and build up competencies and informed opinions on newer capabilities and possibilities, allowing for more nuanced and effective needs analysis for major projects. And it also provides enough oversight and management for IT to step in when usage patterns indicate something has become de facto production and warrants a more formal transition to/evaluation by technical and operations stakeholders. And also provides valuable data to ops/IT on where people are needing to create grass root solutions in general, which can better inform their own future systems planning.

by cosmie 8y ago

clanrebornsx 8y ago

Yuck so u help those scam MLM companies flourish?

viksit 8y ago

I hope you got recognition for this work!

cosmie 8y ago
Thank you! It has certainly been met with recognition, to the point where I'm essentially having to become a people manager again and create a team to support the capability across more clients.

anitil 8y ago

That is such great work. Getting the job done with the tools already around is just such a good feeling.

jac_no_k 8y ago

/me is one of the minions from team "Eye of Sauron". However I'm not stuck in this tower. I'm torn between two worlds. One side applauds the grass roots based solutions and the JFDI attitude. The other side is concerned that these solutions becomes widely used, de facto production, and problems upstream ends up breaking the output. On error, hopefully fails loudly but subtle problems could creep in. If the reports are consumed by other automation, the problem spreads rapidly. Ideally the IT department would take a step back, assess what the users (you the stakeholder) needs are, and provide the solutions. Better Agile. Using the successes to drive more development and operations budget is the way to go.

lbatx 8y ago
Until IT departments get infinite resources (that is, never) there will always be a need for these JFDI solutions. Some group will always go starved for attention and have to cobble together their own solutions. IT backlogs are always longer than the resources to do the work.
- edelans 8y ago
  JFDI = Just Fucking do It (in case other ppl wonder what it means... =) )
  - nirav72 8y ago
    Or what we call 'Shadow IT' in my department.
cosmie 8y ago
I completely understand the terrors of hidden dependencies on shadow IT. The work I do tends to grow up organically in different areas of every org, so I've hopped between functional business units, centralized business operations, and IT. When working for business units, I'm careful when I introduce technically advanced process dependencies. I either limit it to the level of technical competency that is generally accessible to the team or else I adopt whatever toolset the IT or development side of the house uses. That way there's always the potential for transition and I don't screw my manager/team over if I leave (or get stuck supporting it in perpetuity while I'm there). This was an example of that. We do client consulting work, but we're only about 2% of the total revenue of our Fortune 500 parent and our IT policies are not conducive to client-driven work requirements. When my manager looked for what we could use when supporting analytics-oriented aspects of client work, tools like Supermetrics[1] were explicitly denied and IT provided virtually no automation/analytics solutions short of a complete Tableau server, which is prohibitively expensive for most of our clients' needs (especially when allocating IT infrastructure & support costs in addition to the license). I'm not a fan of the fragility and monotony of repetitive manual data collection and reporting, nor do clients like to pay for the excessive hours that takes on a recurring basis compared to competing bids (because presumably our competitors can use SaaS tools for automating rote process). I scrounged around and found a pocket of PowerBI users in our IT's SQL Server team. Connected with the director over that team and got access for myself. Created a proof of concept architecture that provided an agile platform to address the full spectrum of our potential needs from "throw in basic analytics for free and eat the cost" to "maintain a full on data lake getting a deluge of real time data streams". Client users are guests invited to our tenant, log in with their (usually) already-existing Office 365/AzureAD user, can have the $10/user/month PowerBI Pro license added to their account via their corporate IT, our corporate IT, or a credit card expense. All of their ETL and data connectors and credentials and PowerBI models are compartmentalized into their own app workspace and subject to corporate DLP policies and accessible/auditable by Office admins. It's technically unsanctioned/shadow IT, but I ran it by the director of that team in case I needed to tap his team for sporadic support needs in the future, and he was both interested in my proposed usage and perfectly fine giving it his blessing (which is great, because his team manages the permissions I need for provisioning users, adding licenses, and creating workspaces). The alternative would have been to either lose business due to non-competitive bids, use a combination of unsanctioned SaaS tools that get expensed and subject to disruption when turnover happens and passwords don't get shared or credit card renewals don't occur (fairly common scenario I've inherited), or using development skills to create custom automation that would be unmaintainable when I left. > Ideally the IT department would take a step back, assess what the users (you the stakeholder) needs are, and provide the solutions. Better Agile. This works well in some cases, but falls flat on it's face just as often. It's often prohibitively expensive from a budgetary standpoint for a business team to pull in resources for a needs analysis and subsequent project. Business users are far cheaper, so the vast majority of drift between needs and systems capabilities/processes go unresolved because it's cheaper to absorb the inefficiency on the business labor for quite a while before it hits a critical point that justifies the expense of pulling in ops/dev staff. For most business teams, shadow IT is a pragmatic stopgap solution to hit their deliverables, similar to the type of tech debt you see in IT projects that gets kicked down the road until it's suddenly not something that can be kicked anymore. And the process of IT stumbling across a grass roots solution acting as a de facto production dependency is basically a litmus test for when the debt finally comes due and triggers a formal needs analysis and IT project. As well, that type of needs analysis generally creates localized optimizations or poor user adoption. The current needs will be a reflection of the processes that evolved within the current system and constraints, and addressing the directly will usually just result in a solution that's "a faster horse". Or it goes to the other extreme, where the solution you provide is state of the art and it turns out they have no ability to transition from riding their current horse to piloting the new fighter jet you provided. A far more effective approach is to creative a collaborative model. Grass roots based solutions exist for a multitude of reasons and won't ever go away. But the concerns of undocumented, unknown, and potentially unmaintained de facto production dependencies existing outside of the purview of IT is a very real concern. So create a migration path. Put in low friction processes, support structures, and policies that give power users on the business side the ability to create sanctioned grass roots solutions. It'll allow business teams to experiment with and build up competencies and informed opinions on newer capabilities and possibilities, allowing for more nuanced and effective needs analysis for major projects. And it also provides enough oversight and management for IT to step in when usage patterns indicate something has become de facto production and warrants a more formal transition to/evaluation by technical and operations stakeholders. And also provides valuable data to ops/IT on where people are needing to create grass root solutions in general, which can better inform their own future systems planning.