6 ms·

True story about IBM Global Services. I was working with a client that was integrating web services with another third party. IBM GS were responsible for managi

by junto 11y ago

True story about IBM Global Services. I was working with a client that was integrating web services with another third party. IBM GS were responsible for managing third party's network. Our client gave IBM the IP addresses to be white listed. They filled in a lengthy time consuming form that got sent through layers of IBM management till some ops team in India finally implemented the change one week later. We started testing. It didn't work. We went back and forth for weeks. Finally we put a conference call together with all parties. Live on the phone we sent a series of SOAP requests over the wire. "Yes, we can see the requests", came the answer. We saw no response. Finally whilst on the phone call, one of the IBM Tech's piped up. "Did you want this traffic to go in AND out?", he asked. "Well, HTTP traffic isn't going to be of much use if we don't get a response to our request you idiots", was the answer I wanted to give. They pulled out the initial firewall request form. The request was to "allow traffic from our IP addresses in through the firewall to reach the client". They took that absolutely literally, and nobody thought to question it. They also absolved themselves of any blame. I detested IBM from that day on.

jlarocco 11y ago
Haha, that's pretty similar to my experience with Indian tech workers in general. Very literal interpretation of English phrases like that. Maybe it's a language barrier?
- deleted 11y ago
  [deleted]
- hkmurakami 11y ago
  Agreed, this sounds like an Indian outsource tech center problem that is universal rather than an ibm specific problem. Cultural differences rather than language?
  - maxxxxx 11y ago
    My experience is with Infosys. They definitely stream information from architect->tech lead->coder one-way. There is no channel back if something doesn't make sense.
shepardrtc 11y ago
That's not just IBM's outsourced vendors. The desire to do the minimal amount of work to adhere to the exact wording of contracts seems to be prevalent. For us, it takes us 3 to 4 weeks to have a single VM created. And even then its usually done wrong. When asking them to do the partitions different, the response is that they're unable to change the script. A month to simply have a script run.
- gaius 11y ago
  They bill by the hour.
- agumonkey 11y ago
  How does .. anything happen ..
  - shepardrtc 11y ago
    It doesn't.
  - bigiain 11y ago
    This goes a long way to explaining that Gartner report from last year which say Marketing departments IT budgets will exceed IT departments IT budgets by 2017. We're seeing this a lot - doing projects directly with marketing department, largely so they can completely circumvent internal IT department's and their apparent stonewalling on seemingly simple requests. (Of course it also means we're seeing things like completely outlandish requests that indicate fundamental security/privacy misunderstandings ("What do you mean you can't import our entire customer database into the WordPress site you're building and hosting cheaply for us?"), as well as last minute compliance intrusions ("Oh, you need this to live on a big-corp.com subdomain? That requires a security audit and risk assessment and mitigation document! How on earth have you got this far without knowing that?")
  - bogomipz 11y ago
    It happens very slowly. I have a similar story but with simple VLAN changes taking weeks and endless forms. Sorry to hear people losing their jobs but yeesh what a horrible company IBM is. This is a company that pivoted to a services company as well.
Johnny555 11y ago
Could also be a compliance thing where they were told "If it's not on the form, don't implement it". And it's up to the requestor to specify everything they want opened. Having the network team second-guess what was requested could leave unwanted security holes when they have to make assumptions about a request. Though I don't understand the problem you experienced -- HTTP runs over a bidirectional TCP connection, if they got your request, then packets for that session were already flowing in both directions (the TCP session wouldn't set up otherwise). The only way I could see what you described would be if they had a proxy or some sort of deep inspection firewall in the middle that was blocking the return response.
- nucleardog 11y ago
  > Could also be a compliance thing where they were told "If it's not on the form, don't implement it". And it's up to the requestor to specify everything they want opened. Sure thing, do exactly what you're told. That doesn't preclude you from firing back a message to say "Hey, this request is unusual and almost certainly wrong, are you sure this is what you wanted?" When a nurse sees an order for 10000 mg of a drug that's never normally administered above 200mg, it's not their job to correct anything, but it is their job to ask the doctor "Hey, did you mean to write 100.00mg? Is this actually correct?"
- DrJokepu 11y ago
  I get the compliance thing but surely in cases where they can see that a request is very likely to be wrong I don't think it would be unreasonable to expect them to send off an email or give you a call to double check if it is really what you wanted.
- hockley 11y ago
  This is not surprising given IBM developed the Consulting Industry Complex standard for blaming the client for failure: The Rational Unified Process.
noselasd 11y ago
So basically like any other big enterprise.
- matwood 11y ago
  I was about to say just this. It is why I will not work in large enterprises again if I can help it. Something that should take 5 minutes can end up taking weeks and still not be done correctly. I'm not sure why you're being down voted if other big enterprises didn't have this issue they would not put up with IBM having this issue. It is all about minimal work, CYA, and minimal thinking. It is why I always tell people that anyone who can really do IT (probably most on this site), will always have a job.
pjmlp 11y ago
It not only IBM, there are quite a few other ones that I could write endless books about such war stories.
- freehunter 11y ago
  Hell, as someone client-facing who works at IBM, I've had experiences with a ton of other vendors who do the same thing. I'm having a hard time getting one firewall change put in through a client's outsourced datacenter from one IP address to one other IP address, one port. Ticket keeps getting closed as complete, until we say it's not and they reopen the ticket. Rinse and repeat. It's been three weeks of this. If you're outsourcing your operations, that's what you have to expect. They're not going to be as invested in your success as your own employees would be.
bbcbasic 11y ago
> "Did you want this traffic to go in AND out?", he asked. Question sounds reasonable - could be a fire & forget scenario. Problem isn't the question but that it takes a week between iterations. If you had to order lunch and it was a week between asking for your steak and being asked if you want it medium or well done then the question isn't the problem
- vacri 11y ago
  UDP perhaps for fire+forget, but TCP is a bidirectional protocol.
tn13 11y ago
That is why they are cheap.
- cwilkes 11y ago
  Maybe cheap in what you get but they are one of the most expensive places around.
deleted 11y ago
[deleted]
epc 11y ago
I was their Corporate Webmaster in the 1990s (effectively CTO, but not quite). The networking support teams had this in place back then. If you did not say you wanted bi–directional traffic, then you would not get bi–directional traffic. They were incredibly dependent on their filter rules, and their process assumed the filter rules were always correct and accurate. One night I got confused about which window I was in and ssh'd into my desktop machine (on the internal 9/8 network) from one of the ibm.com backend servers (on the open internet). A "routine update" caused routes for 9.0.0.0/8 to be advertised on the open internet. Although I had a short escalation path it still took an hour from when I discovered the hole to get anyone in Global Services to accept it as a problem.
ape4 11y ago
That is terrible. Counting the meetings off-shoring actually cost them money.
- ArkyBeagle 11y ago
  It usually does.
  - iofj 11y ago
    Well, it costs their customers money.
geodel 11y ago
Well I had similar experience IBM people. We had one small change request for our IBM websphere MQ setup. It was raised when I joined the project and they could not complete it by the time I left after 1 year. Multiple times they canceled our request ticket. And this whole setup was managed by IBM in their data centers.
leonroy 11y ago
Had the same experience with a large communication equipment manufacturer (who you'd think would understand firewall rules better!). This is pretty normal to be honest. I doubt there are many companies who are 100% on top of things, but I get the sense that's more down to the department or the individuals you're dealing with. There are people who don't think things through everywhere. I certainly wouldn't bear a grudge against IBM for too long!
ArkyBeagle 11y ago
That which is not required is forbidden. That which is forbidden is not required.
maxxxxx 11y ago
I had that happen with Infosys. They do exactly what you tell them to do, good, bad or stupid. You never get feedback that something doesn't make sense. So in the end you look stupid to your managers because you didn't write a good spec. Now I avoid dealing with outsourcers as much as possible unless I have 100% understanding of the issue and can describe it precisely.
padiyar83 11y ago
Hey I am sorry if you are impacted by downsizing, but there are better ways of handling this than disparaging the tech community in India. I have worked in both US and India for more than one company and can tell you there is no discernible difference in the quality of the work that gets done between US and India. And even if I take your point that IBM are fools who doesn't know what they are doing, how to you explain Amazon, Microsoft and Google hiring thousands of developers in India. Microsoft's largest developer shop outside US is in India. Your experience can be due to a variety of different things from person at the other end doing exactly what he is supposed to do just to not poke holes in a secure infra to assuming that the firewall access list is state-full and allows return (HTTP response) traffic to pass by default. And there is no guarantee that a person based in US won't do the same thing.
- hguant 11y ago
  RE 'no discernible difference in quality': there's a fair amount of literature, both anecdotal and factual, that says otherwise. http://www.wsj.com/articles/SB10001424052748703515504576142092863219826 http://www.wsj.com/articles/SB100014240527487035155045761420... http://www.theatlantic.com/international/archive/2013/10/behind-the-bad-indian-coder/280636/ http://www.theatlantic.com/international/archive/2013/10/beh... http://www.rozinskiy.com/why-india-outsourcing-is-doomed/ http://www.rozinskiy.com/why-india-outsourcing-is-doomed/ Those are from the top of the fold of the Google search "quality of Indian tech workers."
  - padiyar83 11y ago
    There are bad coders everywhere and it takes years of deliberate practice to be good at anything, including programming. Those people who did not put in the work, don't make it through grueling interviews at Microsoft, Amazon, Google and other such companies. People who do make it however are just as good as programmers anywhere else in the world.
- slantedview 11y ago
  The difference is in motivation. What motivated me to go into tech is that I liked it. What motivated many of my Indian colleagues over the years to go into tech was, by their own admission, societal pressure. Which one do you think produces better outcomes? This is the case regardless of the occupation, lawyer, plumber, programmer or teacher - if you're passionate, chances are you're better at what you do than someone who isn't.

by junto 11y ago

jlarocco 11y ago

Haha, that's pretty similar to my experience with Indian tech workers in general. Very literal interpretation of English phrases like that. Maybe it's a language barrier?

deleted 11y ago
[deleted]
hkmurakami 11y ago
Agreed, this sounds like an Indian outsource tech center problem that is universal rather than an ibm specific problem. Cultural differences rather than language?
- maxxxxx 11y ago
  My experience is with Infosys. They definitely stream information from architect->tech lead->coder one-way. There is no channel back if something doesn't make sense.

shepardrtc 11y ago

That's not just IBM's outsourced vendors. The desire to do the minimal amount of work to adhere to the exact wording of contracts seems to be prevalent. For us, it takes us 3 to 4 weeks to have a single VM created. And even then its usually done wrong. When asking them to do the partitions different, the response is that they're unable to change the script. A month to simply have a script run.

gaius 11y ago
They bill by the hour.
agumonkey 11y ago
How does .. anything happen ..
- shepardrtc 11y ago
  It doesn't.
- bigiain 11y ago
  This goes a long way to explaining that Gartner report from last year which say Marketing departments IT budgets will exceed IT departments IT budgets by 2017. We're seeing this a lot - doing projects directly with marketing department, largely so they can completely circumvent internal IT department's and their apparent stonewalling on seemingly simple requests. (Of course it also means we're seeing things like completely outlandish requests that indicate fundamental security/privacy misunderstandings ("What do you mean you can't import our entire customer database into the WordPress site you're building and hosting cheaply for us?"), as well as last minute compliance intrusions ("Oh, you need this to live on a big-corp.com subdomain? That requires a security audit and risk assessment and mitigation document! How on earth have you got this far without knowing that?")
- bogomipz 11y ago
  It happens very slowly. I have a similar story but with simple VLAN changes taking weeks and endless forms. Sorry to hear people losing their jobs but yeesh what a horrible company IBM is. This is a company that pivoted to a services company as well.

Johnny555 11y ago

Could also be a compliance thing where they were told "If it's not on the form, don't implement it". And it's up to the requestor to specify everything they want opened. Having the network team second-guess what was requested could leave unwanted security holes when they have to make assumptions about a request. Though I don't understand the problem you experienced -- HTTP runs over a bidirectional TCP connection, if they got your request, then packets for that session were already flowing in both directions (the TCP session wouldn't set up otherwise). The only way I could see what you described would be if they had a proxy or some sort of deep inspection firewall in the middle that was blocking the return response.

nucleardog 11y ago
> Could also be a compliance thing where they were told "If it's not on the form, don't implement it". And it's up to the requestor to specify everything they want opened. Sure thing, do exactly what you're told. That doesn't preclude you from firing back a message to say "Hey, this request is unusual and almost certainly wrong, are you sure this is what you wanted?" When a nurse sees an order for 10000 mg of a drug that's never normally administered above 200mg, it's not their job to correct anything, but it is their job to ask the doctor "Hey, did you mean to write 100.00mg? Is this actually correct?"
DrJokepu 11y ago
I get the compliance thing but surely in cases where they can see that a request is very likely to be wrong I don't think it would be unreasonable to expect them to send off an email or give you a call to double check if it is really what you wanted.
hockley 11y ago
This is not surprising given IBM developed the Consulting Industry Complex standard for blaming the client for failure: The Rational Unified Process.

noselasd 11y ago

So basically like any other big enterprise.

matwood 11y ago
I was about to say just this. It is why I will not work in large enterprises again if I can help it. Something that should take 5 minutes can end up taking weeks and still not be done correctly. I'm not sure why you're being down voted if other big enterprises didn't have this issue they would not put up with IBM having this issue. It is all about minimal work, CYA, and minimal thinking. It is why I always tell people that anyone who can really do IT (probably most on this site), will always have a job.

pjmlp 11y ago

It not only IBM, there are quite a few other ones that I could write endless books about such war stories.

freehunter 11y ago
Hell, as someone client-facing who works at IBM, I've had experiences with a ton of other vendors who do the same thing. I'm having a hard time getting one firewall change put in through a client's outsourced datacenter from one IP address to one other IP address, one port. Ticket keeps getting closed as complete, until we say it's not and they reopen the ticket. Rinse and repeat. It's been three weeks of this. If you're outsourcing your operations, that's what you have to expect. They're not going to be as invested in your success as your own employees would be.

bbcbasic 11y ago

> "Did you want this traffic to go in AND out?", he asked. Question sounds reasonable - could be a fire & forget scenario. Problem isn't the question but that it takes a week between iterations. If you had to order lunch and it was a week between asking for your steak and being asked if you want it medium or well done then the question isn't the problem

vacri 11y ago
UDP perhaps for fire+forget, but TCP is a bidirectional protocol.

tn13 11y ago

That is why they are cheap.

cwilkes 11y ago
Maybe cheap in what you get but they are one of the most expensive places around.

deleted 11y ago

[deleted]

epc 11y ago

I was their Corporate Webmaster in the 1990s (effectively CTO, but not quite). The networking support teams had this in place back then. If you did not say you wanted bi–directional traffic, then you would not get bi–directional traffic. They were incredibly dependent on their filter rules, and their process assumed the filter rules were always correct and accurate. One night I got confused about which window I was in and ssh'd into my desktop machine (on the internal 9/8 network) from one of the ibm.com backend servers (on the open internet). A "routine update" caused routes for 9.0.0.0/8 to be advertised on the open internet. Although I had a short escalation path it still took an hour from when I discovered the hole to get anyone in Global Services to accept it as a problem.

ape4 11y ago

That is terrible. Counting the meetings off-shoring actually cost them money.

ArkyBeagle 11y ago
It usually does.
- iofj 11y ago
  Well, it costs their customers money.

geodel 11y ago

Well I had similar experience IBM people. We had one small change request for our IBM websphere MQ setup. It was raised when I joined the project and they could not complete it by the time I left after 1 year. Multiple times they canceled our request ticket. And this whole setup was managed by IBM in their data centers.

leonroy 11y ago

Had the same experience with a large communication equipment manufacturer (who you'd think would understand firewall rules better!). This is pretty normal to be honest. I doubt there are many companies who are 100% on top of things, but I get the sense that's more down to the department or the individuals you're dealing with. There are people who don't think things through everywhere. I certainly wouldn't bear a grudge against IBM for too long!

ArkyBeagle 11y ago

That which is not required is forbidden. That which is forbidden is not required.

maxxxxx 11y ago

I had that happen with Infosys. They do exactly what you tell them to do, good, bad or stupid. You never get feedback that something doesn't make sense. So in the end you look stupid to your managers because you didn't write a good spec. Now I avoid dealing with outsourcers as much as possible unless I have 100% understanding of the issue and can describe it precisely.

padiyar83 11y ago

Hey I am sorry if you are impacted by downsizing, but there are better ways of handling this than disparaging the tech community in India. I have worked in both US and India for more than one company and can tell you there is no discernible difference in the quality of the work that gets done between US and India. And even if I take your point that IBM are fools who doesn't know what they are doing, how to you explain Amazon, Microsoft and Google hiring thousands of developers in India. Microsoft's largest developer shop outside US is in India. Your experience can be due to a variety of different things from person at the other end doing exactly what he is supposed to do just to not poke holes in a secure infra to assuming that the firewall access list is state-full and allows return (HTTP response) traffic to pass by default. And there is no guarantee that a person based in US won't do the same thing.

hguant 11y ago
RE 'no discernible difference in quality': there's a fair amount of literature, both anecdotal and factual, that says otherwise. http://www.wsj.com/articles/SB10001424052748703515504576142092863219826 http://www.wsj.com/articles/SB100014240527487035155045761420... http://www.theatlantic.com/international/archive/2013/10/behind-the-bad-indian-coder/280636/ http://www.theatlantic.com/international/archive/2013/10/beh... http://www.rozinskiy.com/why-india-outsourcing-is-doomed/ http://www.rozinskiy.com/why-india-outsourcing-is-doomed/ Those are from the top of the fold of the Google search "quality of Indian tech workers."
- padiyar83 11y ago
  There are bad coders everywhere and it takes years of deliberate practice to be good at anything, including programming. Those people who did not put in the work, don't make it through grueling interviews at Microsoft, Amazon, Google and other such companies. People who do make it however are just as good as programmers anywhere else in the world.
slantedview 11y ago
The difference is in motivation. What motivated me to go into tech is that I liked it. What motivated many of my Indian colleagues over the years to go into tech was, by their own admission, societal pressure. Which one do you think produces better outcomes? This is the case regardless of the occupation, lawyer, plumber, programmer or teacher - if you're passionate, chances are you're better at what you do than someone who isn't.