7 ms·

20 years ago I was a grad assistant for a couple of professors who'd built a pretty incredible JIT supply chain platform. The US Army started using it to manage

20 years ago I was a grad assistant for a couple of professors who'd built a pretty incredible JIT supply chain platform. The US Army started using it to manage their uniform orders and it worked so well that two entire warehouses were torn down due to lack of use. A new general came in and insisted on using SAP. 6 months later, they had to rebuild both warehouses. EDIT: For some additional context, the system managed to basically eliminate the "bull whip effect" all the way down the supply chain. It's really a fascinating system. Developed by Dr. Bill Kernodle and Dr. Steve Davis from Clemson Apparel Research.

deleted 4y ago
[deleted]
dpflan 4y ago
Are there any documents supporting this? I ask because I am curious what official documents exist around this idea of mandated return to inefficiency -- there must've been a gain elsewhere (or is that too optimistic)? Plus there must be costs documented associated with the first and second program. Genuinely curious about this type of decision (call it "inefficiency thrash"?).
- brightball 4y ago
  Over my career I've seen it communicated as "leadership support" where people implementing different programs know that they are only as effective as the leadership support behind it. I do some Scaled Agile Framework consulting and the implementation a good example, because it can be fantastic but if leadership at the top of the company doesn't get on board the entire process will self destruct or be bastardized. The professors who built the system knew this about every customer they talked to and it was a big learning experience for me. New leaders will come in and often insist on tools that they already know, rather than take the time to learn about better options. If you've ever read Skunk Works (fantastic book), you see lots of this in the military procurement process in the US. It's bad enough that you worry how much it's hurting US national security innovations.
  - Seattle3503 4y ago
    > New leaders will come in and often insist on tools that they already know, rather than take the time to learn about better options. I've seen this with developers. Newly joining developers, particularly junior and mid career developers, tend to insist on using the languages, libraries, and techniques they are familiar with. Senior Developers don't seem to do it as much. Either because they are familiar with many things, or because they have been on the other side of these well intentioned suggestions to know when it is actually productive.
    - stingraycharles 4y ago
      Sounds about right, I went through the whole process, went through a big functional programming romance in the mid of my career, and after 15 years am now like “I’ll use whatever”. I do care about using the right abstractions and high-level interactions, not as much about the language or frameworks. I’ll let others put in their political weight on that, so that I can do it in the areas I believe will really make an impact to the business goals.
- bumby 4y ago
  My (completely speculative) hunch is that they may have been chasing efficiencies beyond the scope of uniforms, or even beyond the scope of the army. For example, the solution may not have worked well with other procurement items and they wanted a single platform. Or the navy already had an SAP contract that could be leveraged without spending money on a separate stand-alone army solution. So it can look like a bad decision that generated local inefficiency while globally it was a better choice. That's not to say SAP actually delivered on those promises.
  - ramesh31 4y ago
    And this is why enterprise software is always so horrid to use when compared to a simplistic web app. It’s about features above all else. If someone can do their job, that’s all that matters. And tightening up the UX around that workflow falls to the very bottom priority against developing new features.
  - tmpz22 4y ago
    Or this small company built by a pair of professors didn't have the customer support arm or continuity guarantees to service an organization as large as the US Army. Its always good to call out corruption, but its also important to realize when a situation is actually crazy complex and while its an awesome story might not be worth formulating a summary judgement from.
    - bumby 4y ago
      That could very well be the case. I looked up the professor and the company he created for this solution and the link is dead. However, for an organization like the army that spends $$$ on logistics, I would bet they would continually fund something like this if it showed enough promise. One thing the DoD and Congress are good at is throwing money at R&D efforts.
      - jacobr1 4y ago
        Another thing is requiring continuity clauses in the contract. Such as, in the case of going out of business or otherwise deciding to discontinue the project, the IP and source code needs to be turned over.
- jwess 4y ago
  Here's a link to a report that has a lot more information about the system: https://apps.dtic.mil/sti/pdfs/ADA393599.pdf https://apps.dtic.mil/sti/pdfs/ADA393599.pdf
  - brightball 4y ago
    How in the world did y’all find that? I saw it posted below. The system was/is called Balanced Flow.
    - latchkey 4y ago
      3rd link (for me): https://www.google.com/search?q=Clemson+Apparel+Research https://www.google.com/search?q=Clemson+Apparel+Research
mardifoufs 4y ago
I'd have thought that the US army would be using its own solution, or at least an American one. Using a foreign made platform to handle such a critical aspect of your military (logistics) sounds a bit weird. Not that I think that those risks, if they even exist, haven't been thoroughly evaluated by tons of experts in the army!
- jupp0r 4y ago
  Gun barrels for the army’s main battle tank are also made in Germany. It’s a close ally. I’d worry more about other countries.
jonhohle 4y ago
The government funding mechanism doesn’t incentivize cost optimization, and in my experience, does the opposite. Money is allocated to be spent and if not spent is “lost” and may be cut from subsequent budget years. This leads to massive end of year purchases to use remaining funds or starting unnecessary projects just because money was allocated, regardless of whether needs have changed. It’s a really broken system that no one seems interested in fixing.
- Lightbody 4y ago
  That problem isn’t unique to government. most sufficiently large public companies do the same thing.
  - wcfields 4y ago
    Sharepoint is similarly entrenched at DOE when I worked with them in 2009. Any project that gets a bid contract can't say specifically that it must use Sharepoint, but the requirements will be written in such a way that Sharepoint is literally the only system that fits all of them.
    - wildzzz 4y ago
      While SharePoint is not the best choice for many applications, it has enough features to make it work. Since there are plenty of SharePoint experts already in the organization, it could be cheaper to just leverage them to build some SharePoint site that just barely works well enough for whatever application. It would cost a lot more to bring in some consultant that knows how to setup the proper software to do whatever new thing the project needs to do along with whatever new service contracts that need to be signed and paid for to cover the software for years. Do you jump into the new software that might be better or do you just use the same old SharePoint that you've already invested heavily in? The efficiencies of both solutions are probably intangible and it would be difficult to price either of them. Eventually someone out there will be the evangelist to convince program management that the new software really is the best choice (Jira, Confluence, Teams, Salesforce, Trello, etc).
solidsnack9000 4y ago
Does it relate to this report in some way? https://apps.dtic.mil/sti/pdfs/ADA393599.pdf https://apps.dtic.mil/sti/pdfs/ADA393599.pdf Authored by Joseph W Kernodle of CLEMSON APPAREL RESEARCH.
- brightball 4y ago
  Wow! Good find. That's probably exactly it (although I haven't seen that document before). I was a GA from 2003-2004.
FredPret 4y ago
SAP, and any other ERP, is just the operating system for your organization. If you run it intelligently, the OS won’t prevent you from getting a good outcome. In this case, it sounds like they should’ve bolted the JIT platform onto SAP
- NohatCoder 4y ago
  But people don't run it intelligently. For a lot of stuff the intelligent thing to do is use SAP as a plain database and program your own logic. But now what you really have is an inferior database with an inferior programming interface, and you pay through your nose for the privilege. Of course getting to somewhere useful is a battle if you have to use SAP consultants instead of real programmers.
  - Arcanum-XIII 4y ago
    Except when used like that it’s dead slow (I’ve seen time series - light, but still - put in SAP). And then you have to understand the codes… So no, not even as a DB, maybe especially as a DB, it should not be used.
  - FredPret 4y ago
    Most use it very intelligently. I’ve seen some amazing automations and I’ve seen countless simpler businesses running vanilla SAP as they should. Basic business logic is built into SAP - but because it covers all aspects of business, it’s a massive tangle of rules. For instance, let’s say you get an invoice from a vendor. Now you have to update your balance with that vendor, and your general ledger. You have to record the new inventory moving in. Perhaps the inventory has a barcode, or a serial number with a warrantee date, or a batch with an expiry date. The inventory movement will be tied to the cost of that invoice - remember to take into account those exchange rates. Also, there’s a variety of ways to account for item cost. You will have had one or more purchase orders - those have to be closed. If the vendor is to be paid in a different currency, you have to do every accounting entry in your currency and theirs, and record the exchange rate differences. Your accounts payable are now in that other currency, which means that has to be updated all the time for accurate bookkeeping. You have to note which employee was the buyer. This buyer has targets and metrics to be tracked. Perhaps there are reports and alerts set up for them. You will have a budget, which may impact this transaction. Of course you also have master data for the vendor, the employee, and the items bought. There is a lot more to this than the above, and this is for the simplest business case - one that happens hundreds or thousands of times a day depending on the business. Usually there are special considerations to program into the above as well. Now imagine what happens when a complicated situation arises, where you have complex transactions spanning months, many countries, and many other businesses. Sometimes there are legal rules built into the logic. Usually those rules only alply to one country. And I haven’t even mentioned freight or tax or landed costs from secondary invoices. SAP is a ball of wire because it simply has to be. If you built a new ERP, you will end up the same.
purpleblue 4y ago
It doesn't make sense for the Army to be in the business of developing software. They should be outsourcing this, and having a well-known vendor like SAP means there's a lot of different avenues to get bids to lower prices.
- anotheracctfo 4y ago
  Why not? They need digital soldiers. They farm spec-ops from the best grunts, so this would give them a pool to recruit their information warefare people from. The Army isn't a business and should never be run as one.
- somat 4y ago
  At some point you hit a scale of operations where it does make sense. It does not make sense for Amazon to run there own shipping fleet... until it does. The other point is that armed forces tend to have a mandate to control the supply chain, this might be a case of that.
major505 4y ago
SAP sucks for custom stuff like uniforms and clotes. I developed a custom engine for things like aluminium rails, and thermal sensors, to generate a new product with the production tree in SAP, so it would be able to decrease the right ammount of raw materials from the warehouse.
phone8675309 4y ago
I wonder how much of a kickback that general got for insisting on it.
deleted 4y ago
[deleted]
mrweasel 4y ago
Similar story in Denmark. The army got a new procurement system, based on SAP, to help reduce cost. I believe Paul-Henning Kamp has a joke in a talk where he claim that they succeeded, because nobody could buy anything for a year. Later I meet a former procurement officer who was reprimanded for excess purchases in the year leading up to the system launch. In the end his team was the only one able to provide equipment for an extended period of time after the new system was introduced. Everyone else had relied in JIT supply chains, but this guy didn't believe that the new system would work and spend a year building up inventory.
m463 4y ago
I remember years ago a friend worked for sun, involved in SAP installs. He said corporations would overpay to install the highest-end (overpriced?) sun systems to get SAP up and running. Apparently the cost savings of something like currency conversions alone would pay for everything almost immediately. And that was just one facet - other efficiencies (taxation?) would have similar savings.

deleted 4y ago

[deleted]

dpflan 4y ago

Are there any documents supporting this? I ask because I am curious what official documents exist around this idea of mandated return to inefficiency -- there must've been a gain elsewhere (or is that too optimistic)? Plus there must be costs documented associated with the first and second program. Genuinely curious about this type of decision (call it "inefficiency thrash"?).

brightball 4y ago
Over my career I've seen it communicated as "leadership support" where people implementing different programs know that they are only as effective as the leadership support behind it. I do some Scaled Agile Framework consulting and the implementation a good example, because it can be fantastic but if leadership at the top of the company doesn't get on board the entire process will self destruct or be bastardized. The professors who built the system knew this about every customer they talked to and it was a big learning experience for me. New leaders will come in and often insist on tools that they already know, rather than take the time to learn about better options. If you've ever read Skunk Works (fantastic book), you see lots of this in the military procurement process in the US. It's bad enough that you worry how much it's hurting US national security innovations.
- Seattle3503 4y ago
  > New leaders will come in and often insist on tools that they already know, rather than take the time to learn about better options. I've seen this with developers. Newly joining developers, particularly junior and mid career developers, tend to insist on using the languages, libraries, and techniques they are familiar with. Senior Developers don't seem to do it as much. Either because they are familiar with many things, or because they have been on the other side of these well intentioned suggestions to know when it is actually productive.
  - stingraycharles 4y ago
    Sounds about right, I went through the whole process, went through a big functional programming romance in the mid of my career, and after 15 years am now like “I’ll use whatever”. I do care about using the right abstractions and high-level interactions, not as much about the language or frameworks. I’ll let others put in their political weight on that, so that I can do it in the areas I believe will really make an impact to the business goals.
bumby 4y ago
My (completely speculative) hunch is that they may have been chasing efficiencies beyond the scope of uniforms, or even beyond the scope of the army. For example, the solution may not have worked well with other procurement items and they wanted a single platform. Or the navy already had an SAP contract that could be leveraged without spending money on a separate stand-alone army solution. So it can look like a bad decision that generated local inefficiency while globally it was a better choice. That's not to say SAP actually delivered on those promises.
- ramesh31 4y ago
  And this is why enterprise software is always so horrid to use when compared to a simplistic web app. It’s about features above all else. If someone can do their job, that’s all that matters. And tightening up the UX around that workflow falls to the very bottom priority against developing new features.
- tmpz22 4y ago
  Or this small company built by a pair of professors didn't have the customer support arm or continuity guarantees to service an organization as large as the US Army. Its always good to call out corruption, but its also important to realize when a situation is actually crazy complex and while its an awesome story might not be worth formulating a summary judgement from.
  - bumby 4y ago
    That could very well be the case. I looked up the professor and the company he created for this solution and the link is dead. However, for an organization like the army that spends $$$ on logistics, I would bet they would continually fund something like this if it showed enough promise. One thing the DoD and Congress are good at is throwing money at R&D efforts.
    - jacobr1 4y ago
      Another thing is requiring continuity clauses in the contract. Such as, in the case of going out of business or otherwise deciding to discontinue the project, the IP and source code needs to be turned over.
jwess 4y ago
Here's a link to a report that has a lot more information about the system: https://apps.dtic.mil/sti/pdfs/ADA393599.pdf https://apps.dtic.mil/sti/pdfs/ADA393599.pdf
- brightball 4y ago
  How in the world did y’all find that? I saw it posted below. The system was/is called Balanced Flow.
  - latchkey 4y ago
    3rd link (for me): https://www.google.com/search?q=Clemson+Apparel+Research https://www.google.com/search?q=Clemson+Apparel+Research

mardifoufs 4y ago

I'd have thought that the US army would be using its own solution, or at least an American one. Using a foreign made platform to handle such a critical aspect of your military (logistics) sounds a bit weird. Not that I think that those risks, if they even exist, haven't been thoroughly evaluated by tons of experts in the army!

jupp0r 4y ago
Gun barrels for the army’s main battle tank are also made in Germany. It’s a close ally. I’d worry more about other countries.

jonhohle 4y ago

The government funding mechanism doesn’t incentivize cost optimization, and in my experience, does the opposite. Money is allocated to be spent and if not spent is “lost” and may be cut from subsequent budget years. This leads to massive end of year purchases to use remaining funds or starting unnecessary projects just because money was allocated, regardless of whether needs have changed. It’s a really broken system that no one seems interested in fixing.

Lightbody 4y ago
That problem isn’t unique to government. most sufficiently large public companies do the same thing.
- wcfields 4y ago
  Sharepoint is similarly entrenched at DOE when I worked with them in 2009. Any project that gets a bid contract can't say specifically that it must use Sharepoint, but the requirements will be written in such a way that Sharepoint is literally the only system that fits all of them.
  - wildzzz 4y ago
    While SharePoint is not the best choice for many applications, it has enough features to make it work. Since there are plenty of SharePoint experts already in the organization, it could be cheaper to just leverage them to build some SharePoint site that just barely works well enough for whatever application. It would cost a lot more to bring in some consultant that knows how to setup the proper software to do whatever new thing the project needs to do along with whatever new service contracts that need to be signed and paid for to cover the software for years. Do you jump into the new software that might be better or do you just use the same old SharePoint that you've already invested heavily in? The efficiencies of both solutions are probably intangible and it would be difficult to price either of them. Eventually someone out there will be the evangelist to convince program management that the new software really is the best choice (Jira, Confluence, Teams, Salesforce, Trello, etc).

solidsnack9000 4y ago

Does it relate to this report in some way? https://apps.dtic.mil/sti/pdfs/ADA393599.pdf https://apps.dtic.mil/sti/pdfs/ADA393599.pdf Authored by Joseph W Kernodle of CLEMSON APPAREL RESEARCH.

brightball 4y ago
Wow! Good find. That's probably exactly it (although I haven't seen that document before). I was a GA from 2003-2004.

FredPret 4y ago

SAP, and any other ERP, is just the operating system for your organization. If you run it intelligently, the OS won’t prevent you from getting a good outcome. In this case, it sounds like they should’ve bolted the JIT platform onto SAP

NohatCoder 4y ago
But people don't run it intelligently. For a lot of stuff the intelligent thing to do is use SAP as a plain database and program your own logic. But now what you really have is an inferior database with an inferior programming interface, and you pay through your nose for the privilege. Of course getting to somewhere useful is a battle if you have to use SAP consultants instead of real programmers.
- Arcanum-XIII 4y ago
  Except when used like that it’s dead slow (I’ve seen time series - light, but still - put in SAP). And then you have to understand the codes… So no, not even as a DB, maybe especially as a DB, it should not be used.
- FredPret 4y ago
  Most use it very intelligently. I’ve seen some amazing automations and I’ve seen countless simpler businesses running vanilla SAP as they should. Basic business logic is built into SAP - but because it covers all aspects of business, it’s a massive tangle of rules. For instance, let’s say you get an invoice from a vendor. Now you have to update your balance with that vendor, and your general ledger. You have to record the new inventory moving in. Perhaps the inventory has a barcode, or a serial number with a warrantee date, or a batch with an expiry date. The inventory movement will be tied to the cost of that invoice - remember to take into account those exchange rates. Also, there’s a variety of ways to account for item cost. You will have had one or more purchase orders - those have to be closed. If the vendor is to be paid in a different currency, you have to do every accounting entry in your currency and theirs, and record the exchange rate differences. Your accounts payable are now in that other currency, which means that has to be updated all the time for accurate bookkeeping. You have to note which employee was the buyer. This buyer has targets and metrics to be tracked. Perhaps there are reports and alerts set up for them. You will have a budget, which may impact this transaction. Of course you also have master data for the vendor, the employee, and the items bought. There is a lot more to this than the above, and this is for the simplest business case - one that happens hundreds or thousands of times a day depending on the business. Usually there are special considerations to program into the above as well. Now imagine what happens when a complicated situation arises, where you have complex transactions spanning months, many countries, and many other businesses. Sometimes there are legal rules built into the logic. Usually those rules only alply to one country. And I haven’t even mentioned freight or tax or landed costs from secondary invoices. SAP is a ball of wire because it simply has to be. If you built a new ERP, you will end up the same.

purpleblue 4y ago

It doesn't make sense for the Army to be in the business of developing software. They should be outsourcing this, and having a well-known vendor like SAP means there's a lot of different avenues to get bids to lower prices.

anotheracctfo 4y ago
Why not? They need digital soldiers. They farm spec-ops from the best grunts, so this would give them a pool to recruit their information warefare people from. The Army isn't a business and should never be run as one.
somat 4y ago
At some point you hit a scale of operations where it does make sense. It does not make sense for Amazon to run there own shipping fleet... until it does. The other point is that armed forces tend to have a mandate to control the supply chain, this might be a case of that.

major505 4y ago

SAP sucks for custom stuff like uniforms and clotes. I developed a custom engine for things like aluminium rails, and thermal sensors, to generate a new product with the production tree in SAP, so it would be able to decrease the right ammount of raw materials from the warehouse.

phone8675309 4y ago

I wonder how much of a kickback that general got for insisting on it.

deleted 4y ago

[deleted]

mrweasel 4y ago

Similar story in Denmark. The army got a new procurement system, based on SAP, to help reduce cost. I believe Paul-Henning Kamp has a joke in a talk where he claim that they succeeded, because nobody could buy anything for a year. Later I meet a former procurement officer who was reprimanded for excess purchases in the year leading up to the system launch. In the end his team was the only one able to provide equipment for an extended period of time after the new system was introduced. Everyone else had relied in JIT supply chains, but this guy didn't believe that the new system would work and spend a year building up inventory.

m463 4y ago

I remember years ago a friend worked for sun, involved in SAP installs. He said corporations would overpay to install the highest-end (overpriced?) sun systems to get SAP up and running. Apparently the cost savings of something like currency conversions alone would pay for everything almost immediately. And that was just one facet - other efficiencies (taxation?) would have similar savings.