8 ms·

I've never interacted directly with IBM, but I remember about 2 years ago, they had a talk at the Akamai Edge conference about how they dealt with bloated cooki

I've never interacted directly with IBM, but I remember about 2 years ago, they had a talk at the Akamai Edge conference about how they dealt with bloated cookies. The company I work for has the same problem so I sat in to see what their solution was. The problem they were having is that all the various IBM lines of business added so much garbage onto the client's cookie, that eventually their pages would stop loading because they wouldn't be able to parse the cookie. The 'solution' is to detect when that's about to happen, and redirect the client to a page that warns them that their cookie is too big (because IBM made it too big) and give them a button to delete their cookie. They then continue to start over and stuff more garbage into the fresh cookie. That kind of problem solving pretty much made me lose faith in them successfully doing much of anything anymore.

coderdude 8y ago
Those two departments couldn't possibly be related. It seems unfair to base judgement on the entire company for that.
- delinka 8y ago
  I dunno ... seems to me an indication of a larger systemic problem: "we couldn't even provide a company-wide engineering solution for this simple problem, so here's how we used a hack..."
  - philjohn 8y ago
    That's the problem you get with silo'd teams. If they had a core technology team they could handle things like this more gracefully. Or maybe not, this is IBM after all.
    - bnchrch 8y ago
      This is probably just me but I've yet to see a company pull off a core technology team gracefully.
    - semi-extrinsic 8y ago
      Exactly, it's IBM. As a company, what they do is so diversified in both level of complexity and technology type that a "core technology team" is just nonsensical. I mean, in the same company you have on one end of the scale lots of teams doing just outsourced IT support ("my corporate word install is broken, pls fix"). And on the other end for instance the IBM Zürich research lab, who have received multiple Nobel prizes in physics and who developed things like Token ring and Trellis code modulation.
- Urbanxassassin 8y ago
  You are correct so let's talk about how there PCF solution is deprecated, and how they were still using a deprecated container management solution instead of kubernetes, and how if you use Java the for you to use liberty build packs, and how they push XP and TDD practices from he garage yet many of their consultants don't practice what they preach.
- wil421 8y ago
  Sounds like similar stories I’ve heard about IBM but not related to cookies. Sell first and hack it together later which is very similar to a lot of places. Except IBM doesn’t seem to be able to hack it together.
- aiisjustanif 8y ago
  So we have some more info out there: - Their PCF solution is deprecated - They were still using a deprecated container management solution instead of kubernetes until recently - If you use Java the for you to use liberty build packs - They push XP and TDD practices from the garage yet many of their consultants don't practice what they preach.
  - workinthehead 8y ago
    Don't forget their big "blockchain" push (via TV commercials).
  - parasubvert 8y ago
    IBM Cloud Foundry (formerly Bluemix) isn't quite deprecated but the container manager inside CF was stuck on the deprecated DEAs rather than Diego long after Pivotal and others moved on. They moved to Diego last year I think but very slowly. And then they rebranded IBM Spectrum Conductor for Containers (look it up!), a Kubernetes snowflake they came up with, into IBM Cloud Private, because Kubernetes will solve all problems, the issues MUST have been leading with a PaaS and not a CaaS, right? Other observations are spot on Disclaimer, I compete with IBM and have to deal with their account team shenanigans, though parts of IBM are better than others and even can be good partners.
- flukus 8y ago
  If they don't want the company suffering reputation damage by failings in other parts of the company then they should split or at least run under different brands. If they want to accept the positives of the IBM brand then they need to accept all the negatives that come with it. Unfortunately for them that brand is now toxic.
- deleted 8y ago
  [deleted]
defen 8y ago
How did your company solve it?
- akfanta 8y ago
  nice try, IBM.
- darkmighty 8y ago
  Seems fairly "conceptually straightforward" (if not so easy): compress data (internal dependencies), use predictions (external dependencies), save states server-side, simply reduce garbage.
  - scarejunba 8y ago
    Right on the money. We compress the cookie and store the state server side.
- curun1r 8y ago
  At the large company I worked at, it was solved by frequent code audits to any team given subdomain of the company's company.com domain. We were explicitly told to ensure that anything not related to interoperability between company properties was to be set on the ourdomain.company.com subdomain. The team in charge of it had heuristics they used to monitor for unapproved infringements on that policy. If I decided I wanted to visit one of our offshore dev teams, I could book a first-class ticket anytime I wanted to with no approval necessary. If I wanted $2m for some dev effort, I could have the necessary approvals in a day or so. But if I wanted to write 50 bytes to a .company.com cookie, that was weeks of meetings where I needed to justify my team's existence, the value of the feature we were building and prove there was no other way to implement it.
  - sushid 8y ago
    To be fair, it seems like your organization's solution was to make it very difficult to do so, which is kind of a non-solution (aside from trnctng chars, which is a practice even a small ad tech startup that I was at did). Admittedly it is a better practice than continuing to stuff you cookie and direct users to a strange page once in a while.
smartician 8y ago
We have run into the same issue before. Many third party services like Optimizely for example just stuff all their tracking information into a cookie, serialized as JSON, which you as the site owner don't really have much control over. The problem is not that they wouldn't be able to parse the cookie, it's that Akamai edge node web server (Nginx or Apache, don't remember) rejects any request with a header that's above a certain size. Your origin server doesn't even get to know that there ever was such a request. See https://maxchadwick.xyz/blog/http-request-header-size-limits https://maxchadwick.xyz/blog/http-request-header-size-limits for example.
- workinthehead 8y ago
  While you raise a valid issue, you don't know that IBM didn't also have trouble parsing large cookies. And even if they didn't, your information doesn't exactly excuse IBM for using over 8 kilobytes of data to send cookies back and forth.
stefs 8y ago
had to deal with ibm once. we were developing an application for a customer and ibm was working on some new service for them. of course the customer insisted we use the new ibm service, even though it solved a problem we didn't have at all. the ibm product was a website to be used by end users, while our application had to pass our end users data through. so, i asked the ibm guys (via the customer) to give me the docs for their json interface, as i don't necessarily want to parse HTML inside the application to exchange data. there was no json interface - html was all they had -, but they got to work implementing one. what i got was: {"elem" : "html", "children": [ ... {"elem" : "form", "attrs" : {"action": "/an_url", "method" : "post"}, children: [{"input" : {"attrs" : {"type" : "submit", "value" : "..."}}}]} ..., ]} yes, their json interface was the DOM, serialized as json. but it gets better. to submit my data i had to fill in the actual values in their json struct at the appropriate nodes and send it back. i'm a) pretty sure this "json interface" feature cost the customer more than i make in a year, and b) it probably broke the very instant the customer let a frontend/designer guy change the html code months later.
- paganel 8y ago
  > yes, their json interface was the DOM, serialized as json. This is pure thedailywtf material.
  - ryandrake 8y ago
    Did the change request come complete with specific requirements on what the interface should include? Because it sounds like they produced exactly what was asked for: a JSON interface. If the requirements were poorly specified, you can’t lay all the blame on them.
    - therealtomsmith 8y ago
      The act of them doing that alone shows that they either dont get it or that complexity within the organization has brought productivity to a hault. I'd assume it's the second one.
    - manigandham 8y ago
cncrnd 8y ago
I find it hilarious that they made this into a talk! There must be something more to it?
throwaway0255 8y ago
I laughed too hard at this. Reminds me of that famous Jim Reekes story: https://www.youtube.com/watch?v=C5d151lqJsA#t=2m4s https://www.youtube.com/watch?v=C5d151lqJsA#t=2m4s Do you happen to remember what the copy was on that page? I'm sure it had to be something better than "Hey we made your cookie too big. Can we delete it? Ok thanks sorry about that." I bet they went with something subtle and vague, make the problem a little unclear and complicated-sounding, maybe take a little credit for Websphere doing a good job in detecting the problem. Like how I word all my work emails.
- ryanmcbride 8y ago
  I wish I had taken a picture. I don't remember exactly but it was something along the lines of: "We are unable to process your request due to a problem with your cookie, please click below to delete the cookie" And then there was just a normal button input. Their logic around having the user click the button instead of just automatically deleting the cookie and moving on (Something that still shouldn't be necessary if you are using cookies correctly) is that they thought users would be upset at IBM automatically deleting their 'data' without asking. That data just being a bunch of unreadable values for IBMs services.
wallace_f 8y ago
I wonder how long it will take for people to realize politics and bureaucracy does not reward merit
a_bonobo 8y ago
IBM is one of the go-to partners for outsourcing tasks of the Australian government, and it's usually a dumpster fire. - 2016. Australia decided to run a census, population wide (~22 million people), all online. If you didn't enter your stuff you risked a fine. You had one day to get a good 'snapshot'. The service, built by IBM, went down immediately. 'Census outage could have been prevented by turning router on and off again: IBM' http://www.abc.net.au/news/2016-10-25/turning-router-off-and-on-could-have-prevented-census-outage/7963916 http://www.abc.net.au/news/2016-10-25/turning-router-off-and... 'IBM to pay more than $30m in compensation for census fail, Prime Minister Malcolm Turnbull suggests' http://www.abc.net.au/news/2016-11-25/ibm-to-pay-over-$30m-in-compensation-for-census-fail/8057240 http://www.abc.net.au/news/2016-11-25/ibm-to-pay-over-$30m-i... (Have they done this? I don't think so) They said it was an 'overseas DDOS' but there is no proof for that, personally I think it was Australians accessing the page from overseas. Interestingly, this is not on IBM's Wikipedia page. - In Queensland, IBM borked the rollout of a new health payroll system so extremely that the state premier banned the state from outsourcing to IBM (and that was before the census catastrophe!!) >The commission, headed by former supreme court judge Richard Chesterman, tied a number of IBM employees working on the contract bid to serious ethical transgressions, including using leaked information about competitors to gain advantage and attempting to access tender responses by opposing bidders Logica and Accenture. https://www.itnews.com.au/news/queenslands-ibm-ban-lives-on-420969 https://www.itnews.com.au/news/queenslands-ibm-ban-lives-on-... In the end, the state had to carry the costs (1.2 billion) https://www.itnews.com.au/news/qld-health-ibm-payroll-court-battle-officially-over-417921 https://www.itnews.com.au/news/qld-health-ibm-payroll-court-... It was supposed to cost less than 10 million. This is also not on IBM's Wikipedia page.
- piva00 8y ago
  What? $ 1.2 billion for a failed project that in 2010 Queensland's government gave IBM a waiver? This is blatantly corrupt, why the fuck would a government give a company a waiver from their contractual responsibilities on a failed project? That's utterly insane.
  - easytiger 8y ago
    If the project fails then the people who commissioned the project would be at fault, therefore these things rarely truly are considered failures

coderdude 8y ago

Those two departments couldn't possibly be related. It seems unfair to base judgement on the entire company for that.

delinka 8y ago
I dunno ... seems to me an indication of a larger systemic problem: "we couldn't even provide a company-wide engineering solution for this simple problem, so here's how we used a hack..."
- philjohn 8y ago
  That's the problem you get with silo'd teams. If they had a core technology team they could handle things like this more gracefully. Or maybe not, this is IBM after all.
  - bnchrch 8y ago
    This is probably just me but I've yet to see a company pull off a core technology team gracefully.
  - semi-extrinsic 8y ago
    Exactly, it's IBM. As a company, what they do is so diversified in both level of complexity and technology type that a "core technology team" is just nonsensical. I mean, in the same company you have on one end of the scale lots of teams doing just outsourced IT support ("my corporate word install is broken, pls fix"). And on the other end for instance the IBM Zürich research lab, who have received multiple Nobel prizes in physics and who developed things like Token ring and Trellis code modulation.
Urbanxassassin 8y ago
You are correct so let's talk about how there PCF solution is deprecated, and how they were still using a deprecated container management solution instead of kubernetes, and how if you use Java the for you to use liberty build packs, and how they push XP and TDD practices from he garage yet many of their consultants don't practice what they preach.
wil421 8y ago
Sounds like similar stories I’ve heard about IBM but not related to cookies. Sell first and hack it together later which is very similar to a lot of places. Except IBM doesn’t seem to be able to hack it together.
aiisjustanif 8y ago
So we have some more info out there: - Their PCF solution is deprecated - They were still using a deprecated container management solution instead of kubernetes until recently - If you use Java the for you to use liberty build packs - They push XP and TDD practices from the garage yet many of their consultants don't practice what they preach.
- workinthehead 8y ago
  Don't forget their big "blockchain" push (via TV commercials).
- parasubvert 8y ago
  IBM Cloud Foundry (formerly Bluemix) isn't quite deprecated but the container manager inside CF was stuck on the deprecated DEAs rather than Diego long after Pivotal and others moved on. They moved to Diego last year I think but very slowly. And then they rebranded IBM Spectrum Conductor for Containers (look it up!), a Kubernetes snowflake they came up with, into IBM Cloud Private, because Kubernetes will solve all problems, the issues MUST have been leading with a PaaS and not a CaaS, right? Other observations are spot on Disclaimer, I compete with IBM and have to deal with their account team shenanigans, though parts of IBM are better than others and even can be good partners.
flukus 8y ago
If they don't want the company suffering reputation damage by failings in other parts of the company then they should split or at least run under different brands. If they want to accept the positives of the IBM brand then they need to accept all the negatives that come with it. Unfortunately for them that brand is now toxic.
deleted 8y ago
[deleted]

defen 8y ago

How did your company solve it?

akfanta 8y ago
nice try, IBM.
darkmighty 8y ago
Seems fairly "conceptually straightforward" (if not so easy): compress data (internal dependencies), use predictions (external dependencies), save states server-side, simply reduce garbage.
- scarejunba 8y ago
  Right on the money. We compress the cookie and store the state server side.
curun1r 8y ago
At the large company I worked at, it was solved by frequent code audits to any team given subdomain of the company's company.com domain. We were explicitly told to ensure that anything not related to interoperability between company properties was to be set on the ourdomain.company.com subdomain. The team in charge of it had heuristics they used to monitor for unapproved infringements on that policy. If I decided I wanted to visit one of our offshore dev teams, I could book a first-class ticket anytime I wanted to with no approval necessary. If I wanted $2m for some dev effort, I could have the necessary approvals in a day or so. But if I wanted to write 50 bytes to a .company.com cookie, that was weeks of meetings where I needed to justify my team's existence, the value of the feature we were building and prove there was no other way to implement it.
- sushid 8y ago
  To be fair, it seems like your organization's solution was to make it very difficult to do so, which is kind of a non-solution (aside from trnctng chars, which is a practice even a small ad tech startup that I was at did). Admittedly it is a better practice than continuing to stuff you cookie and direct users to a strange page once in a while.

smartician 8y ago

We have run into the same issue before. Many third party services like Optimizely for example just stuff all their tracking information into a cookie, serialized as JSON, which you as the site owner don't really have much control over. The problem is not that they wouldn't be able to parse the cookie, it's that Akamai edge node web server (Nginx or Apache, don't remember) rejects any request with a header that's above a certain size. Your origin server doesn't even get to know that there ever was such a request. See https://maxchadwick.xyz/blog/http-request-header-size-limits https://maxchadwick.xyz/blog/http-request-header-size-limits for example.

workinthehead 8y ago
While you raise a valid issue, you don't know that IBM didn't also have trouble parsing large cookies. And even if they didn't, your information doesn't exactly excuse IBM for using over 8 kilobytes of data to send cookies back and forth.

stefs 8y ago

had to deal with ibm once. we were developing an application for a customer and ibm was working on some new service for them. of course the customer insisted we use the new ibm service, even though it solved a problem we didn't have at all. the ibm product was a website to be used by end users, while our application had to pass our end users data through. so, i asked the ibm guys (via the customer) to give me the docs for their json interface, as i don't necessarily want to parse HTML inside the application to exchange data. there was no json interface - html was all they had -, but they got to work implementing one. what i got was: {"elem" : "html", "children": [ ... {"elem" : "form", "attrs" : {"action": "/an_url", "method" : "post"}, children: [{"input" : {"attrs" : {"type" : "submit", "value" : "..."}}}]} ..., ]} yes, their json interface was the DOM, serialized as json. but it gets better. to submit my data i had to fill in the actual values in their json struct at the appropriate nodes and send it back. i'm a) pretty sure this "json interface" feature cost the customer more than i make in a year, and b) it probably broke the very instant the customer let a frontend/designer guy change the html code months later.

paganel 8y ago
> yes, their json interface was the DOM, serialized as json. This is pure thedailywtf material.
- ryandrake 8y ago
  Did the change request come complete with specific requirements on what the interface should include? Because it sounds like they produced exactly what was asked for: a JSON interface. If the requirements were poorly specified, you can’t lay all the blame on them.
  - therealtomsmith 8y ago
    The act of them doing that alone shows that they either dont get it or that complexity within the organization has brought productivity to a hault. I'd assume it's the second one.
  - manigandham 8y ago

cncrnd 8y ago

I find it hilarious that they made this into a talk! There must be something more to it?

throwaway0255 8y ago

I laughed too hard at this. Reminds me of that famous Jim Reekes story: https://www.youtube.com/watch?v=C5d151lqJsA#t=2m4s https://www.youtube.com/watch?v=C5d151lqJsA#t=2m4s Do you happen to remember what the copy was on that page? I'm sure it had to be something better than "Hey we made your cookie too big. Can we delete it? Ok thanks sorry about that." I bet they went with something subtle and vague, make the problem a little unclear and complicated-sounding, maybe take a little credit for Websphere doing a good job in detecting the problem. Like how I word all my work emails.

ryanmcbride 8y ago
I wish I had taken a picture. I don't remember exactly but it was something along the lines of: "We are unable to process your request due to a problem with your cookie, please click below to delete the cookie" And then there was just a normal button input. Their logic around having the user click the button instead of just automatically deleting the cookie and moving on (Something that still shouldn't be necessary if you are using cookies correctly) is that they thought users would be upset at IBM automatically deleting their 'data' without asking. That data just being a bunch of unreadable values for IBMs services.

wallace_f 8y ago

I wonder how long it will take for people to realize politics and bureaucracy does not reward merit

a_bonobo 8y ago

IBM is one of the go-to partners for outsourcing tasks of the Australian government, and it's usually a dumpster fire. - 2016. Australia decided to run a census, population wide (~22 million people), all online. If you didn't enter your stuff you risked a fine. You had one day to get a good 'snapshot'. The service, built by IBM, went down immediately. 'Census outage could have been prevented by turning router on and off again: IBM' http://www.abc.net.au/news/2016-10-25/turning-router-off-and-on-could-have-prevented-census-outage/7963916 http://www.abc.net.au/news/2016-10-25/turning-router-off-and... 'IBM to pay more than $30m in compensation for census fail, Prime Minister Malcolm Turnbull suggests' http://www.abc.net.au/news/2016-11-25/ibm-to-pay-over-$30m-in-compensation-for-census-fail/8057240 http://www.abc.net.au/news/2016-11-25/ibm-to-pay-over-$30m-i... (Have they done this? I don't think so) They said it was an 'overseas DDOS' but there is no proof for that, personally I think it was Australians accessing the page from overseas. Interestingly, this is not on IBM's Wikipedia page. - In Queensland, IBM borked the rollout of a new health payroll system so extremely that the state premier banned the state from outsourcing to IBM (and that was before the census catastrophe!!) >The commission, headed by former supreme court judge Richard Chesterman, tied a number of IBM employees working on the contract bid to serious ethical transgressions, including using leaked information about competitors to gain advantage and attempting to access tender responses by opposing bidders Logica and Accenture. https://www.itnews.com.au/news/queenslands-ibm-ban-lives-on-420969 https://www.itnews.com.au/news/queenslands-ibm-ban-lives-on-... In the end, the state had to carry the costs (1.2 billion) https://www.itnews.com.au/news/qld-health-ibm-payroll-court-battle-officially-over-417921 https://www.itnews.com.au/news/qld-health-ibm-payroll-court-... It was supposed to cost less than 10 million. This is also not on IBM's Wikipedia page.

piva00 8y ago
What? $ 1.2 billion for a failed project that in 2010 Queensland's government gave IBM a waiver? This is blatantly corrupt, why the fuck would a government give a company a waiver from their contractual responsibilities on a failed project? That's utterly insane.
- easytiger 8y ago
  If the project fails then the people who commissioned the project would be at fault, therefore these things rarely truly are considered failures