7 ms·

One widely discussed study, for instance, judges students on their ability to parse the muddy and semantically tortuous opening of “Bleak House”; this is a litt

One widely discussed study, for instance, judges students on their ability to parse the muddy and semantically tortuous opening of “Bleak House”; this is a little like assessing swimmers on their ability to cross fifty yards of molasses. WTF. Asking university English students to read a book is not asking swimmers to go through molasses.

watwut 1y ago
To be fair, the opening is exactly the sort of clunky writing paragraph that you can normally feel free to skip.
bmiekre 1y ago
This gave a literal LOL
- easterncalculus 1y ago
  I definitely didn't abbreviate WTF when I read that. It gets worse too. “Gas looming through the fog in divers places in the streets, much as the sun may, from the spongey fields, be seen to loom by husbandman and ploughboy,” Dickens wrote. Claude takes a more direct path: “Gas lamps glow dimly through the fog at various spots throughout the streets, much like how the sun might appear to farmers working in misty fields.” Claude doesn't even get this right. The sentence is comparing how the gas lamps and the sun appear to each other, how they both "loom". That's missing completely from Claude's summary.
  - bee_rider 1y ago
    The only tricky thing about the Dickens bit is that it uses that archaic meaning of “divers” that I’m only familiar with because of the Joanna Newsom time travel/romance/trench warfare/birds album.
    - jazzypants 1y ago
      God, I'm so happy I saw this live in concert. She is incredible.
  - KPGv2 1y ago
    "AI can't understand this metaphor" isn't criticizing the thing you think it's criticizing.
    - easterncalculus 1y ago
      Comparison is not metaphor. Ironically metaphor is actually something LLMs are somewhat adept at picking up on provided there's enough training data saying there's a metaphor there, for a seminal work.
      - TeaBrain 1y ago
        Whatever you may wish to call it, simile or metaphor, it's a little silly to complain about it being referred to as a metaphor, considering that similes are a subset of metaphor, even if they often aren't taught this way to children. Also, in common speech and literature, what would be taught as similes to children are almost universally just referred to as metaphors.
  - AudiomaticApp 1y ago
    I don't see what's missing from Claude's summary. Claude doesn't repeat the word "loom", but does explain that Dickens is comparing the appearance of the lamps to that of the sun.
jasonthorsness 1y ago
“A Chancery judge once had the kindness to inform me, as one of a company of some hundred and fifty men and women not labouring under any suspicions of lunacy, that the Court of Chancery, though the shining subject of much popular prejudice (at which point I thought the judge’s eye had a cast in my direction), was almost immaculate.” Not too bad throughout, just a lot of embedded asides/commentary https://www.gutenberg.org/files/1023/1023-h/1023-h.htm#cP https://www.gutenberg.org/files/1023/1023-h/1023-h.htm#cP
- anzumitsu 1y ago
  That’s the preface, the study in question dealt with the opening of Chapter 1
  - jasonthorsness 1y ago
    I like it “London. Michaelmas term lately over, and the Lord Chancellor sitting in Lincoln’s Inn Hall. Implacable November weather. As much mud in the streets as if the waters had but newly retired from the face of the earth, and it would not be wonderful to meet a Megalosaurus, forty feet long or so, waddling like an elephantine lizard up Holborn Hill. Smoke lowering down from chimney-pots, making a soft black drizzle, with flakes of soot in it as big as full-grown snowflakes—gone into mourning, one might imagine, for the death of the sun. Dogs, undistinguishable in mire. Horses, scarcely better; splashed to their very blinkers. Foot passengers, jostling one another’s umbrellas in a general infection of ill temper, and losing their foot-hold at street-corners, where tens of thousands of other foot passengers have been slipping and sliding since the day broke (if this day ever broke), adding new deposits to the crust upon crust of mud, sticking at those points tenaciously to the pavement, and accumulating at compound interest.”
    - datadrivenangel 1y ago
      It's moderately dense literature, as muddy and gloomy as the portrayal of london, but I would expect the majority of people to be able to read this?
anzumitsu 1y ago
Have you read the passage in question? While I would expect English students to be able to work through it it’s not an easy task and would almost certainly require some kind of reference for the anachronistic terms and historical context (which, admittedly, I think the students in the study were allowed, though few took advantage of it).
- haiku2077 1y ago
  Someone posted it below and it doesn't seem that difficult. One old use of the word "wonderful" was all that threw me off
neuroplots 1y ago
Even worse, there are a lot of writers I know who go through the publishing process and the first person to actually read the manuscript is the copy editor. You’d think people in publishing read, but no… A landmark study of this, pertaining to literary agents, was just published today: https://antipodes.substack.com/p/literary-agents-dont-read-how-i-proved https://antipodes.substack.com/p/literary-agents-dont-read-h...
troad 1y ago
Strong agree, and honestly I'm kind of shocked. If an adult English speaker cannot understand the opening of Bleak House - quotes given elsewhere in this thread - they are effectively unable to access the bulk of English literature. This is not someone who belongs in a university English course, this is someone in need of remedial English lessons.
Spivak 1y ago
It really is, students were asked to read a passage in what could be reasonably called a dialect of English that they don't read, write, or speak; that makes reference and uses turns of phrase that would be well-understood by readers at the time but are archaic to a modern reader. If you did that same exact passage but had someone transliterate it into modern English like "foot-passengers" -> "pedestrians" I bet the results would be perfectly acceptable. Why would you test literacy, a very practical skill, using anything other than contemporary language, the kind that they actually use in their day to day?
- KPGv2 1y ago
  > students were asked to read a passage in what could be reasonably called a dialect of English They were asked to read the opening chapter of Bleak House. It's pretty much standard English. > LONDON. Michaelmas Term lately over, and the Lord Chancellor sitting in Lincoln�s Inn Hall. Implacable November weather. As much mud in the streets as if the waters had but newly retired from the face of the earth, and it would not be wonderful to meet a Megalosaurus, forty feet long or so, waddling like an elephantine lizard up Holborn Hill. Smoke lowering down from chimney-pots, making a soft black drizzle, with flakes of soot in it as big as full-grown snow-flakes � gone into mourning, one might imagine, for the death of the sun. The only word I see there that isn't used the same way in bog standard American English is "wonderful." I will grant you that "Michaelmas" is obscure to an American, but "Michaelmas Term [] over" is clear enough: it's a specific span of time that has elapsed. You don't need to know exactly what "Lincoln's Inn Hall" is any more than if you read > I went to Carnegie Deli and didn't know what Carnigie Deli is. It's obviously a deli.
  - zapw 1y ago
    To understand the passage and what follows, it is actually important to know what "Lincoln's Inn Hall" is. And to be clear, it's not an "inn" in the standard modern usage of the term, but rather a rather a professional association for lawyers.
    - allturtles 1y ago
jihadjihad 1y ago
Wait until they encounter Shakespeare!
jihadjihad 1y ago
It reminds me of the now-infamous article that ran in the New York Times a few years ago that argued we should remove the requirement to complete an Algebra course in order to earn a high school diploma. The bar for any amount of academic discipline or rigor may as well be a stripe of masking tape on the floor.
allturtles 1y ago
Not just university students, but English majors [0]. [0]: https://muse.jhu.edu/article/922346 https://muse.jhu.edu/article/922346 It's a very depressing study overall: " When we asked our subjects to name British and American authors and/or works of the nineteenth-century, 48 percent of those from KRU2 and 52 percent of those from KRU1 could recall at most only one author or title on their own."
- jihadjihad 1y ago
  To add to the depression, I wonder how many respondents thought that "nineteenth century" == 1900s.
  - bruce343434 1y ago
    Ain't It Awful
  - rdlw 1y ago
    Wow, this really makes me think. Thanks for making that up and commenting it.
- easterncalculus 1y ago
  Yep, English majors, and on top of that the study was done in 2015, which is not just pre-ChatGPT but pre-TikTok.
- bee_rider 1y ago
  I don’t think I could name many 19’th century engineers, and my list will quickly start to include folks who are more mathematicians, like Chebyshev. And we have the advantage that we literally use equations named after these guys! Names are trivia, imo. The memorization of trivia is… it happens of course, but it is just a symptom of learning.
  - bregma 1y ago
    Isambard Kingdom Brunel? Gustav Eiffel? Robert Stephenson? Nikola Tesla? Alexander Graham Bell? Orville and Wilbur Wright? Thomas Edison? I guess it depends on how much you honour those who lit the way before you.
ofalkaed 1y ago
They were not asked to read a book, they were asked to read 7 paragraphs aloud and then answer questions about it. It would have been nice if the study had a control group of students who sat down and read the same passage to themselves, we interpret for different things when reading aloud and we have no idea how that affected the study but for the study's purpose of highlighting issues with teaching (not proving students are illiterate) the lack of this control group is not a huge issue. English students are students, they are in school to learn things they don't know, not demonstrate what they already know. Looking at the numbers it goes from 0% proficient as freshman to 50% as juniors, drops with seniors but seniors often seem to be an outlier in these sorts of studies, probably because they have a great deal on their mind with the major changes their life is about to undergo.

watwut 1y ago

To be fair, the opening is exactly the sort of clunky writing paragraph that you can normally feel free to skip.

bmiekre 1y ago

This gave a literal LOL

easterncalculus 1y ago
I definitely didn't abbreviate WTF when I read that. It gets worse too. “Gas looming through the fog in divers places in the streets, much as the sun may, from the spongey fields, be seen to loom by husbandman and ploughboy,” Dickens wrote. Claude takes a more direct path: “Gas lamps glow dimly through the fog at various spots throughout the streets, much like how the sun might appear to farmers working in misty fields.” Claude doesn't even get this right. The sentence is comparing how the gas lamps and the sun appear to each other, how they both "loom". That's missing completely from Claude's summary.
- bee_rider 1y ago
  The only tricky thing about the Dickens bit is that it uses that archaic meaning of “divers” that I’m only familiar with because of the Joanna Newsom time travel/romance/trench warfare/birds album.
  - jazzypants 1y ago
    God, I'm so happy I saw this live in concert. She is incredible.
- KPGv2 1y ago
  "AI can't understand this metaphor" isn't criticizing the thing you think it's criticizing.
  - easterncalculus 1y ago
    Comparison is not metaphor. Ironically metaphor is actually something LLMs are somewhat adept at picking up on provided there's enough training data saying there's a metaphor there, for a seminal work.
    - TeaBrain 1y ago
      Whatever you may wish to call it, simile or metaphor, it's a little silly to complain about it being referred to as a metaphor, considering that similes are a subset of metaphor, even if they often aren't taught this way to children. Also, in common speech and literature, what would be taught as similes to children are almost universally just referred to as metaphors.
- AudiomaticApp 1y ago
  I don't see what's missing from Claude's summary. Claude doesn't repeat the word "loom", but does explain that Dickens is comparing the appearance of the lamps to that of the sun.

jasonthorsness 1y ago

“A Chancery judge once had the kindness to inform me, as one of a company of some hundred and fifty men and women not labouring under any suspicions of lunacy, that the Court of Chancery, though the shining subject of much popular prejudice (at which point I thought the judge’s eye had a cast in my direction), was almost immaculate.” Not too bad throughout, just a lot of embedded asides/commentary https://www.gutenberg.org/files/1023/1023-h/1023-h.htm#cP https://www.gutenberg.org/files/1023/1023-h/1023-h.htm#cP

anzumitsu 1y ago
That’s the preface, the study in question dealt with the opening of Chapter 1
- jasonthorsness 1y ago
  I like it “London. Michaelmas term lately over, and the Lord Chancellor sitting in Lincoln’s Inn Hall. Implacable November weather. As much mud in the streets as if the waters had but newly retired from the face of the earth, and it would not be wonderful to meet a Megalosaurus, forty feet long or so, waddling like an elephantine lizard up Holborn Hill. Smoke lowering down from chimney-pots, making a soft black drizzle, with flakes of soot in it as big as full-grown snowflakes—gone into mourning, one might imagine, for the death of the sun. Dogs, undistinguishable in mire. Horses, scarcely better; splashed to their very blinkers. Foot passengers, jostling one another’s umbrellas in a general infection of ill temper, and losing their foot-hold at street-corners, where tens of thousands of other foot passengers have been slipping and sliding since the day broke (if this day ever broke), adding new deposits to the crust upon crust of mud, sticking at those points tenaciously to the pavement, and accumulating at compound interest.”
  - datadrivenangel 1y ago
    It's moderately dense literature, as muddy and gloomy as the portrayal of london, but I would expect the majority of people to be able to read this?

anzumitsu 1y ago

Have you read the passage in question? While I would expect English students to be able to work through it it’s not an easy task and would almost certainly require some kind of reference for the anachronistic terms and historical context (which, admittedly, I think the students in the study were allowed, though few took advantage of it).

haiku2077 1y ago
Someone posted it below and it doesn't seem that difficult. One old use of the word "wonderful" was all that threw me off

neuroplots 1y ago

Even worse, there are a lot of writers I know who go through the publishing process and the first person to actually read the manuscript is the copy editor. You’d think people in publishing read, but no… A landmark study of this, pertaining to literary agents, was just published today: https://antipodes.substack.com/p/literary-agents-dont-read-how-i-proved https://antipodes.substack.com/p/literary-agents-dont-read-h...

troad 1y ago

Strong agree, and honestly I'm kind of shocked. If an adult English speaker cannot understand the opening of Bleak House - quotes given elsewhere in this thread - they are effectively unable to access the bulk of English literature. This is not someone who belongs in a university English course, this is someone in need of remedial English lessons.

Spivak 1y ago

It really is, students were asked to read a passage in what could be reasonably called a dialect of English that they don't read, write, or speak; that makes reference and uses turns of phrase that would be well-understood by readers at the time but are archaic to a modern reader. If you did that same exact passage but had someone transliterate it into modern English like "foot-passengers" -> "pedestrians" I bet the results would be perfectly acceptable. Why would you test literacy, a very practical skill, using anything other than contemporary language, the kind that they actually use in their day to day?

KPGv2 1y ago
> students were asked to read a passage in what could be reasonably called a dialect of English They were asked to read the opening chapter of Bleak House. It's pretty much standard English. > LONDON. Michaelmas Term lately over, and the Lord Chancellor sitting in Lincoln�s Inn Hall. Implacable November weather. As much mud in the streets as if the waters had but newly retired from the face of the earth, and it would not be wonderful to meet a Megalosaurus, forty feet long or so, waddling like an elephantine lizard up Holborn Hill. Smoke lowering down from chimney-pots, making a soft black drizzle, with flakes of soot in it as big as full-grown snow-flakes � gone into mourning, one might imagine, for the death of the sun. The only word I see there that isn't used the same way in bog standard American English is "wonderful." I will grant you that "Michaelmas" is obscure to an American, but "Michaelmas Term [] over" is clear enough: it's a specific span of time that has elapsed. You don't need to know exactly what "Lincoln's Inn Hall" is any more than if you read > I went to Carnegie Deli and didn't know what Carnigie Deli is. It's obviously a deli.
- zapw 1y ago
  To understand the passage and what follows, it is actually important to know what "Lincoln's Inn Hall" is. And to be clear, it's not an "inn" in the standard modern usage of the term, but rather a rather a professional association for lawyers.
  - allturtles 1y ago

jihadjihad 1y ago

Wait until they encounter Shakespeare!

jihadjihad 1y ago

It reminds me of the now-infamous article that ran in the New York Times a few years ago that argued we should remove the requirement to complete an Algebra course in order to earn a high school diploma. The bar for any amount of academic discipline or rigor may as well be a stripe of masking tape on the floor.

allturtles 1y ago

Not just university students, but English majors [0]. [0]: https://muse.jhu.edu/article/922346 https://muse.jhu.edu/article/922346 It's a very depressing study overall: " When we asked our subjects to name British and American authors and/or works of the nineteenth-century, 48 percent of those from KRU2 and 52 percent of those from KRU1 could recall at most only one author or title on their own."

jihadjihad 1y ago
To add to the depression, I wonder how many respondents thought that "nineteenth century" == 1900s.
- bruce343434 1y ago
  Ain't It Awful
- rdlw 1y ago
  Wow, this really makes me think. Thanks for making that up and commenting it.
easterncalculus 1y ago
Yep, English majors, and on top of that the study was done in 2015, which is not just pre-ChatGPT but pre-TikTok.
bee_rider 1y ago
I don’t think I could name many 19’th century engineers, and my list will quickly start to include folks who are more mathematicians, like Chebyshev. And we have the advantage that we literally use equations named after these guys! Names are trivia, imo. The memorization of trivia is… it happens of course, but it is just a symptom of learning.
- bregma 1y ago
  Isambard Kingdom Brunel? Gustav Eiffel? Robert Stephenson? Nikola Tesla? Alexander Graham Bell? Orville and Wilbur Wright? Thomas Edison? I guess it depends on how much you honour those who lit the way before you.

ofalkaed 1y ago

They were not asked to read a book, they were asked to read 7 paragraphs aloud and then answer questions about it. It would have been nice if the study had a control group of students who sat down and read the same passage to themselves, we interpret for different things when reading aloud and we have no idea how that affected the study but for the study's purpose of highlighting issues with teaching (not proving students are illiterate) the lack of this control group is not a huge issue. English students are students, they are in school to learn things they don't know, not demonstrate what they already know. Looking at the numbers it goes from 0% proficient as freshman to 50% as juniors, drops with seniors but seniors often seem to be an outlier in these sorts of studies, probably because they have a great deal on their mind with the major changes their life is about to undergo.