4 ms·

Common Application Users Find a Glitch

36 points by asnyder 16y ago

asnyder 16y ago
Situations like this make me sad. 1000 characters is not the same as 150 words. It's very easy to strip out punctuation and then count the number of words in a string. Really really careless. I love the quote: Asked why the problem had not been fixed, Mr. Killion said, “Believe me, if there’s a way to do it, we’d do it. Maybe there’s a way out there we don’t know about.”
- fendrak 16y ago
  This reminds me how most people seem to see programming as some sort of black magic that only happens when the stars align just-so. If only they were aware of how different things actually are... Python solution: def under_the_limit(response): return len(response.split()) <= 150
  - joeld42 16y ago
    This is wrong. They're not checking for 150 words. They need to check that, once typeset on the printable form, the text fits in a box of a certain size. Still, whatever they are using to typeset the form should be able to easily give them that info. Even if, god forbid, they are doing it with HTML they can get that info. Much simpler would be to design the form with a monospaced font for the essay. This is a big reason why manuscripts are still submitted in Courier even though few people use typewriters anymore.
    - ghshephard 16y ago
      YES - I was wondering why nobody had brought up the painfully straight forward solution of "Fixed Width Font" - Then your bounding area is a simple function of the number of characters you typed in, and an error message can pop up if the student exceed it.
      - chc 16y ago
        It's not quite that simple even with a monospace font unless you wrap words unnaturally. For example, the word "heterochromatic" repeated over and over with a space between fills up the HN comment box in 288 characters, but "I DO" makes it to 359.
  - rflrob 16y ago
    There's actually a lot more that could go into figuring out whether a given passage will fit, but arguably a gold standard solution exists, and is freely available: to wit in TeX. While it's probably not trivial to adapt the relevant code, it's not like no solution exists, as the quote implied.
    - waterlesscloud 16y ago
      I really don't understand how an error that fundamentally fails the target use-case can stand for a decade. And how they just can't figure it out. That just seems insane.
- deleted 16y ago
  [deleted]
- URSpider94 16y ago
  I don't think you're understanding the essence of the problem. It's not that they don't know how to count words, it's that the actual limit they are trying to enforce is an x,y bounding box on the page. At the end of the day, this application has to print out on a specified number of pages, in a specified format. If your limit is x,y space, then any "x number of words" limit will, by definition, not match up to reality some fraction of the time. One thing that they could do is switch to a monospace font for the text box, then at least they could specify a fixed number of characters. Now that I think about it, that wouldn't necessarily work either, because of word wrapping. The best case would probably be to tweak the font size to allow a fixed number of words to fit in the box.
  - asnyder 16y ago
    I see, I still don't see an issue. Is there a reason why they can't push text that doesn't fit into another page. I remember when I was younger I did work for a heating company that did boiler inspections in NY. The NYC DOB has specific forms that you had to fill out and specific places for your text. In cases where the text overflowed beyond the 8 or so alloted lines, we were able to create a duplicate page and continue in a copy of the form with the remainder of information. We could this as much as we needed to, so long as the information was in the proper form. This seems like the same logic would be applicable here too. Limit to 150 words, and if it's beyond the page, simply finish on another page. Alternatively, like you said, you could figure out the necessary font-size based on the word-count, length, or even client's sizing information.
    - tibbon 16y ago
      Or better yet, figure out the rough maximum worse case scenario for word length and just leave extra space on the printed area. If every word was 8 characters, and you had 150, then leave room for 1500 characters instead of 1200 on the print area. If it still goes over, dynamically scale the font size down on that section by up to 1pt to make it fit. Or like you said, offer logic to have additional pages printed if needed.
vietor 16y ago
That is seriously embarrassing, but the attitude of "OMG there is no way to fix it!" makes me appreciate the rates that qualified consultants charge. It is in no way an impossible problem to limit it by physical size if it's being printed in a standard format, but apparently it seems that way to some.
- tibbon 16y ago
  Agreed, that isn't the type of thing I'd tell the NYT. Everyone's company/startup has technical issues, but this really could be fixed by a post on StackOverflow if they needed free advice.
  - vietor 16y ago
    Exactly, it's a relatively minor technical issue that could be fixed fairly trivially. Yet rather than doing that it's worked around: "is not new, and had been a reality of the process for more than a decade." From my experience this is a fairly common approach in non-technical institutions. Issues are worked around and people are inconvenienced to a cost far in excess of what it would cost to fix the problem. (The opposite of what occasionally happens in tech startups...) While it's just everyday paperwork for them, it's a life changing event for most applicants and the level of apathy with which they address it is maddening.
pak 16y ago
I think it's a overflow issue. The print preview flows the text in some box on a page, and if you have lots of wide characters it gets cut off because they didn't anticipate the max y-height 150 words can consume. The Times was a bit imprecise in how they used the kid's words to explain the bug (about par for their technical reporting, unfortunately). The way they write it, it sounds a lot like <textarea maxlength=1000></textarea>, but if you think more about the explanation of the wide characters and the "print preview" it can't be that. Overflow issues are actually not trivial to fix, as opposed to the claims some people are making here about "just count the words". Without allowing an excessive amount of space or allowing for more flexibility in the way it lays it out (flowing across multiple pages, etc.), it can be hard to estimate how much space to give for something as vague as a 150-word limit. If the Common App instead specified a 1000-character limit, the programmer/designer could take the widest character like "W" and determine the max y-height for 1000 W's.
- thedz 16y ago
  Maybe one way to approach this would be to have the box expand dynamically vertically until the max word count is reached. Then the text is always going to fit.
- chc 16y ago
  What you're actually saying here is: Overflow issues are not trivial to fix if the only approach you try is guessing how much space the user is likely to want and hoping you're right. That's because guesstimating a length is not really fixing the issue. The flexible layout algorithm you describe in your "Without…" clause is the right thing to do, and it's been a solved problem for decades now. Anyway, even if they don't want to fix the root problem, it would still be trivial to do better than they are doing now. Instead of passing the buck with their "I went over the print preview with a fine-tooth comb" checkbox, it would be pretty close to trivial to algorithmically check whether you're oversetting the type and throw an error along the lines of, "Hey, we didn't allow enough room in the form and aren't competent to make it work right. You'll need to fix this."
Groxx 16y ago
>“A capital W takes up 10 times the space of a period,” he said. “If a student writes 163 characters that include lots of Ws and m’s and g’s and capital letters, their 163 characters are going to take many more inches of space than someone who uses lots of I’s and commas and periods and spaces.” >Asked why the problem had not been fixed, Mr. Killion said, “Believe me, if there’s a way to do it, we’d do it. Maybe there’s a way out there we don’t know about.” ... TeX has been doing this for years. Clearly, their preview does it for them, as it cuts things off where it should. They have a solution staring them in the face. What the hell?
waterlesscloud 16y ago
"Mr. Killion said the issue of “truncation,” as it is known within the Common Application offices, is not new, and had been a reality of the process for more than a decade..." More than a decade? Everyone in that office needs to quit and never work anywhere again.
- patio11 16y ago
  This is the reality in government, academia, and most businesses in which software does not make money. Here's the exact discussion which you'd have if they called up A Consulting Firm Which Did Lots of Academic Work: "By the way, our common application program has a truncation issue." "Ah, yes, I see that." "What would it cost to get it fixed?" "Ten thousand dollars." + "Ten thousand dollars for something which doesn't get graded by US News and World Report or make us look more prestigious than Yale!?! Screw it, we'll put a note in the handbook." + I am open to commentary that this is too high, but disagree. Getting stuff to fit algorithmically is pretty tricky: BCC doesn't get it right 100% of the time (my heuristics are up to 99.X% and doing it "properly" would take weeks), and the amount of work it took me to get from 90 to 99.X% would be in the neighborhood of $10k if charged out at intermediate engineer consulting rates).
  - waterlesscloud 16y ago
    I understand what you're saying, and I don't disagree with your cost estimate at all. I don't think you'd get it perfect in a day, like others have indicated (though I do think you could make it massively better in a few hours). But... This is their product. Their only product. It's what they sell, it's what generates their income. It's not just a cost center, it's the revenue center. And it doesn't serve the customers' needs in a fundamental way. And hasn't in over 10 years. That to me speaks of inexplicable laziness and irresponsibility.
    - macrael 16y ago
      Ahh, but they have no competitors. There is very little reason for them to improve. It's genius! Their entire business model is built on the fact that they don't have competitors. They just have to be good enough for the big schools to sign on, and now that they have that, they have no reason to improve. They are no doubt raking in the cash, but aren't making things better.
davidradcliffe 16y ago
This situation is pretty unbelievable. Almost any halfway decent web developer could build a fix for this in a day.
frisco 16y ago
char_lengths_in_px = {"a": 9, "b": 7, "c": 8, ... } px_per_line = 768 current_length = reduce(lambda x,y: y+char_lengths_in_px[x], my_text, 0) space_left = (px_per_line*num_lines)-current_length FTFY
- mkramlich 16y ago
  I first thought that was a solution written in pseudo-code. Then I realized, "Hey, that's Python!"
- pak 16y ago
  I wish it were this simple, but you forgot about word-wrapping. Also, what to do in the event that space_left is negative, but the user hasn't reached 150 words. (Also, it's layout for print, so better to think in terms of pts or picas.)
  - frisco 16y ago
    This will give you an upper-bound, which is the worst case. You're right that you'll end up not giving the students as much as they "should" have, but its still infinitely better than just truncating it without warning.
    - pak 16y ago
      You mean lower-bound. current_length should be greater than what you calculate given any my_text with >1 char words. So you will wind up giving the students more than you can fit, not less than they should have. That's just as bad as truncating without warning.
w1ntermute 16y ago
Is there any good reason to be using inches instead of characters to calculate the length of an essay submitted digitally?
- rjvir 16y ago
  Yes - the essays have to be printed, on a paper measured by inches, not characters.
drivebyacct2 16y ago
The whole problem is caused by what I consider to be a larger problem: They're trying to print the damn things. In terms of collaborating with others to form an opinion on an applicant, an online no-print system sounds better, more flexible and less prone to these types of problems.
- slapshot 16y ago
  Sitting around a conference table with physical files is a far traditional way to collaborate than doing it all electronically. At most schools, the admissions office staff all works in the same office, and could easily discuss face-to-face. Face-to-face communication is still richer than electronic, based on the ability to judge tone, facial expression, etc. Now, one day maybe iPads could replace physical files when discussing around a conference table, but until then physical files are easier to read (especially in bulk) and easier to wave around.
  - rmc 16y ago
    Sitting around a conference table with physical files is a far traditional way to collaborate than doing it all electronically OK, fair enough. So why don't they just print the whole thing? Some submissions will be on 2 pages? Sounds better than truncating it.
desigooner 16y ago
Higher Education Software sucks mightily for the most part. It's way too rigid and often, the cost of customization is prohibitive for offices working on a certain budget. Also, for some reason, common sense in decision making is not a factor many a times as far as Higher Ed IT goes. I work in higher ed and often see quotes for doing very simple things to be in hundreds and sometimes thousands of dollars. The integration of software from different vendors is a nightmare as no one really provides any sort of an API and such customizations usually end up being ugly hacks since the vendors would charge exorbitantly for doing the same. Also, there's rarely a way to successfully propose a change due to the investment in legacy software and sunken costs. Not to mention the crappy SLAs that the vendors provide. As far as issues like wordcount, it could be something as trivial as changing the parameters on the backend for the system to accept X+Y characters for validation instead of X but still advertise the X character limit on the front end.
colinsidoti 16y ago
Jeesh, the kid goes through so much trouble generating press and says this: "It's kind of ridiculous," he said. "I take computer science. I have a vague idea of how this may or may not work. I think it would be just such an easy thing for an error message, at least, to pop up." So much for taking advantage of the opportunity. Common App is an awful piece of software and there are many places for improvement. There is definitely room for a startup in this space, and the article proves that common app isn't going to put up a fight.

36 points by asnyder 16y ago

Situations like this make me sad. 1000 characters is not the same as 150 words. It's very easy to strip out punctuation and then count the number of words in a string. Really really careless. I love the quote: Asked why the problem had not been fixed, Mr. Killion said, “Believe me, if there’s a way to do it, we’d do it. Maybe there’s a way out there we don’t know about.”

fendrak 16y ago
This reminds me how most people seem to see programming as some sort of black magic that only happens when the stars align just-so. If only they were aware of how different things actually are... Python solution: def under_the_limit(response): return len(response.split()) <= 150
- joeld42 16y ago
  This is wrong. They're not checking for 150 words. They need to check that, once typeset on the printable form, the text fits in a box of a certain size. Still, whatever they are using to typeset the form should be able to easily give them that info. Even if, god forbid, they are doing it with HTML they can get that info. Much simpler would be to design the form with a monospaced font for the essay. This is a big reason why manuscripts are still submitted in Courier even though few people use typewriters anymore.
  - ghshephard 16y ago
    YES - I was wondering why nobody had brought up the painfully straight forward solution of "Fixed Width Font" - Then your bounding area is a simple function of the number of characters you typed in, and an error message can pop up if the student exceed it.
    - chc 16y ago
      It's not quite that simple even with a monospace font unless you wrap words unnaturally. For example, the word "heterochromatic" repeated over and over with a space between fills up the HN comment box in 288 characters, but "I DO" makes it to 359.
- rflrob 16y ago
  There's actually a lot more that could go into figuring out whether a given passage will fit, but arguably a gold standard solution exists, and is freely available: to wit in TeX. While it's probably not trivial to adapt the relevant code, it's not like no solution exists, as the quote implied.
  - waterlesscloud 16y ago
    I really don't understand how an error that fundamentally fails the target use-case can stand for a decade. And how they just can't figure it out. That just seems insane.
deleted 16y ago
[deleted]
URSpider94 16y ago
I don't think you're understanding the essence of the problem. It's not that they don't know how to count words, it's that the actual limit they are trying to enforce is an x,y bounding box on the page. At the end of the day, this application has to print out on a specified number of pages, in a specified format. If your limit is x,y space, then any "x number of words" limit will, by definition, not match up to reality some fraction of the time. One thing that they could do is switch to a monospace font for the text box, then at least they could specify a fixed number of characters. Now that I think about it, that wouldn't necessarily work either, because of word wrapping. The best case would probably be to tweak the font size to allow a fixed number of words to fit in the box.
- asnyder 16y ago
  I see, I still don't see an issue. Is there a reason why they can't push text that doesn't fit into another page. I remember when I was younger I did work for a heating company that did boiler inspections in NY. The NYC DOB has specific forms that you had to fill out and specific places for your text. In cases where the text overflowed beyond the 8 or so alloted lines, we were able to create a duplicate page and continue in a copy of the form with the remainder of information. We could this as much as we needed to, so long as the information was in the proper form. This seems like the same logic would be applicable here too. Limit to 150 words, and if it's beyond the page, simply finish on another page. Alternatively, like you said, you could figure out the necessary font-size based on the word-count, length, or even client's sizing information.
  - tibbon 16y ago
    Or better yet, figure out the rough maximum worse case scenario for word length and just leave extra space on the printed area. If every word was 8 characters, and you had 150, then leave room for 1500 characters instead of 1200 on the print area. If it still goes over, dynamically scale the font size down on that section by up to 1pt to make it fit. Or like you said, offer logic to have additional pages printed if needed.

vietor 16y ago

That is seriously embarrassing, but the attitude of "OMG there is no way to fix it!" makes me appreciate the rates that qualified consultants charge. It is in no way an impossible problem to limit it by physical size if it's being printed in a standard format, but apparently it seems that way to some.

tibbon 16y ago
Agreed, that isn't the type of thing I'd tell the NYT. Everyone's company/startup has technical issues, but this really could be fixed by a post on StackOverflow if they needed free advice.
- vietor 16y ago
  Exactly, it's a relatively minor technical issue that could be fixed fairly trivially. Yet rather than doing that it's worked around: "is not new, and had been a reality of the process for more than a decade." From my experience this is a fairly common approach in non-technical institutions. Issues are worked around and people are inconvenienced to a cost far in excess of what it would cost to fix the problem. (The opposite of what occasionally happens in tech startups...) While it's just everyday paperwork for them, it's a life changing event for most applicants and the level of apathy with which they address it is maddening.

pak 16y ago

I think it's a overflow issue. The print preview flows the text in some box on a page, and if you have lots of wide characters it gets cut off because they didn't anticipate the max y-height 150 words can consume. The Times was a bit imprecise in how they used the kid's words to explain the bug (about par for their technical reporting, unfortunately). The way they write it, it sounds a lot like <textarea maxlength=1000></textarea>, but if you think more about the explanation of the wide characters and the "print preview" it can't be that. Overflow issues are actually not trivial to fix, as opposed to the claims some people are making here about "just count the words". Without allowing an excessive amount of space or allowing for more flexibility in the way it lays it out (flowing across multiple pages, etc.), it can be hard to estimate how much space to give for something as vague as a 150-word limit. If the Common App instead specified a 1000-character limit, the programmer/designer could take the widest character like "W" and determine the max y-height for 1000 W's.

thedz 16y ago
Maybe one way to approach this would be to have the box expand dynamically vertically until the max word count is reached. Then the text is always going to fit.
chc 16y ago
What you're actually saying here is: Overflow issues are not trivial to fix if the only approach you try is guessing how much space the user is likely to want and hoping you're right. That's because guesstimating a length is not really fixing the issue. The flexible layout algorithm you describe in your "Without…" clause is the right thing to do, and it's been a solved problem for decades now. Anyway, even if they don't want to fix the root problem, it would still be trivial to do better than they are doing now. Instead of passing the buck with their "I went over the print preview with a fine-tooth comb" checkbox, it would be pretty close to trivial to algorithmically check whether you're oversetting the type and throw an error along the lines of, "Hey, we didn't allow enough room in the form and aren't competent to make it work right. You'll need to fix this."

Groxx 16y ago

>“A capital W takes up 10 times the space of a period,” he said. “If a student writes 163 characters that include lots of Ws and m’s and g’s and capital letters, their 163 characters are going to take many more inches of space than someone who uses lots of I’s and commas and periods and spaces.” >Asked why the problem had not been fixed, Mr. Killion said, “Believe me, if there’s a way to do it, we’d do it. Maybe there’s a way out there we don’t know about.” ... TeX has been doing this for years. Clearly, their preview does it for them, as it cuts things off where it should. They have a solution staring them in the face. What the hell?

waterlesscloud 16y ago

"Mr. Killion said the issue of “truncation,” as it is known within the Common Application offices, is not new, and had been a reality of the process for more than a decade..." More than a decade? Everyone in that office needs to quit and never work anywhere again.

patio11 16y ago
This is the reality in government, academia, and most businesses in which software does not make money. Here's the exact discussion which you'd have if they called up A Consulting Firm Which Did Lots of Academic Work: "By the way, our common application program has a truncation issue." "Ah, yes, I see that." "What would it cost to get it fixed?" "Ten thousand dollars." + "Ten thousand dollars for something which doesn't get graded by US News and World Report or make us look more prestigious than Yale!?! Screw it, we'll put a note in the handbook." + I am open to commentary that this is too high, but disagree. Getting stuff to fit algorithmically is pretty tricky: BCC doesn't get it right 100% of the time (my heuristics are up to 99.X% and doing it "properly" would take weeks), and the amount of work it took me to get from 90 to 99.X% would be in the neighborhood of $10k if charged out at intermediate engineer consulting rates).
- waterlesscloud 16y ago
  I understand what you're saying, and I don't disagree with your cost estimate at all. I don't think you'd get it perfect in a day, like others have indicated (though I do think you could make it massively better in a few hours). But... This is their product. Their only product. It's what they sell, it's what generates their income. It's not just a cost center, it's the revenue center. And it doesn't serve the customers' needs in a fundamental way. And hasn't in over 10 years. That to me speaks of inexplicable laziness and irresponsibility.
  - macrael 16y ago
    Ahh, but they have no competitors. There is very little reason for them to improve. It's genius! Their entire business model is built on the fact that they don't have competitors. They just have to be good enough for the big schools to sign on, and now that they have that, they have no reason to improve. They are no doubt raking in the cash, but aren't making things better.

davidradcliffe 16y ago

This situation is pretty unbelievable. Almost any halfway decent web developer could build a fix for this in a day.

frisco 16y ago

char_lengths_in_px = {"a": 9, "b": 7, "c": 8, ... } px_per_line = 768 current_length = reduce(lambda x,y: y+char_lengths_in_px[x], my_text, 0) space_left = (px_per_line*num_lines)-current_length FTFY

mkramlich 16y ago
I first thought that was a solution written in pseudo-code. Then I realized, "Hey, that's Python!"
pak 16y ago
I wish it were this simple, but you forgot about word-wrapping. Also, what to do in the event that space_left is negative, but the user hasn't reached 150 words. (Also, it's layout for print, so better to think in terms of pts or picas.)
- frisco 16y ago
  This will give you an upper-bound, which is the worst case. You're right that you'll end up not giving the students as much as they "should" have, but its still infinitely better than just truncating it without warning.
  - pak 16y ago
    You mean lower-bound. current_length should be greater than what you calculate given any my_text with >1 char words. So you will wind up giving the students more than you can fit, not less than they should have. That's just as bad as truncating without warning.

w1ntermute 16y ago

Is there any good reason to be using inches instead of characters to calculate the length of an essay submitted digitally?

rjvir 16y ago
Yes - the essays have to be printed, on a paper measured by inches, not characters.

drivebyacct2 16y ago

The whole problem is caused by what I consider to be a larger problem: They're trying to print the damn things. In terms of collaborating with others to form an opinion on an applicant, an online no-print system sounds better, more flexible and less prone to these types of problems.

slapshot 16y ago
Sitting around a conference table with physical files is a far traditional way to collaborate than doing it all electronically. At most schools, the admissions office staff all works in the same office, and could easily discuss face-to-face. Face-to-face communication is still richer than electronic, based on the ability to judge tone, facial expression, etc. Now, one day maybe iPads could replace physical files when discussing around a conference table, but until then physical files are easier to read (especially in bulk) and easier to wave around.
- rmc 16y ago
  Sitting around a conference table with physical files is a far traditional way to collaborate than doing it all electronically OK, fair enough. So why don't they just print the whole thing? Some submissions will be on 2 pages? Sounds better than truncating it.

desigooner 16y ago

Higher Education Software sucks mightily for the most part. It's way too rigid and often, the cost of customization is prohibitive for offices working on a certain budget. Also, for some reason, common sense in decision making is not a factor many a times as far as Higher Ed IT goes. I work in higher ed and often see quotes for doing very simple things to be in hundreds and sometimes thousands of dollars. The integration of software from different vendors is a nightmare as no one really provides any sort of an API and such customizations usually end up being ugly hacks since the vendors would charge exorbitantly for doing the same. Also, there's rarely a way to successfully propose a change due to the investment in legacy software and sunken costs. Not to mention the crappy SLAs that the vendors provide. As far as issues like wordcount, it could be something as trivial as changing the parameters on the backend for the system to accept X+Y characters for validation instead of X but still advertise the X character limit on the front end.

colinsidoti 16y ago

Jeesh, the kid goes through so much trouble generating press and says this: "It's kind of ridiculous," he said. "I take computer science. I have a vague idea of how this may or may not work. I think it would be just such an easy thing for an error message, at least, to pop up." So much for taking advantage of the opportunity. Common App is an awful piece of software and there are many places for improvement. There is definitely room for a startup in this space, and the article proves that common app isn't going to put up a fight.