4 ms·

Ask HN: Is this normal? Long-form transformer take home

11 points by bpiche 5y ago

Mostly an ML/NLP engineer, and been interviewing a lot lately, and therefore have been doing a lot of NLP take home tests lately, some more complex than others. Certainly prefer it over any LeetCode tests, though some of these take homes are so complex, they verge on pro bono work and I often feel I should be compensated. One organization recently gave me a take home with the following demands: -- Here are 150k long-form (6000+ words) documents, and a list of labels. Please use a recent transformer model as the vectorization/representation layer to train a multilabel classifier on this data set. You can use CoLab and their free GPU tier, but we won't pay for any GPU/TPU time. Also, please compare this solution to other algorithms (linearSVM, XGBoost) and write 1000 words about the performance tradeoffs. --- I'm not a deep learning expert, and I assumed that transformer models are basically limited to short form text with the exception of the Longformer and Big Bird architecture, and I was under the impression that those solutions were pretty memory intense. Other solutions are limited to only looking at the first 512 characters or so. And I'm not even sure CoLab's free tier can handle this. Is this too much? Part of me is really excited to try this but another part of me is already imagining the compute time/space required to run this thing.

formerkrogemp 5y ago
Why are you working for free? I'm not in machine learning, but if I got this in an accounting field I would think it was a scam and not a legitimate job offer. Just my two cents. Best of luck friend.
- bpiche 5y ago
  From the mouths of babes. You get so used to grinding LeetCode and doing take homes weekend after weekend, you stop questioning reality. Maybe it takes someone from an outside field looking in to see it. Thank you for the refreshing perspective..
  - formerkrogemp 5y ago
    I've enjoyed reading about programming on this site for a few years, so, no, thank you for providing a peek into your reality.
- hjkl0 5y ago
  FWIW, some companies do compensate you for trial work.
deleted 5y ago
[deleted]
jstx1 5y ago
> Here are 150k long-form (6000+ words) documents, and a list of labels. > write 1000 words about the performance tradeoffs. Big no. And it shouldn't be a close decisions for you - this is way too much work to do for someone you aren't working for. Who in 2022 is asking for 1000 words as an explanation? And who is actually reading it??? Fascinating stuff, thanks for sharing. (I kind of wish take homes were less common and more companies were doing LC for data science and ML)
deleted 5y ago
[deleted]
brudgers 5y ago
They are looking for someone with the level of experience that allows treating the problem as trivial. That means plugging in the data to the tool they use all the time, adjusting a few parameters, and writing the 1000 words while waiting for the results…that’s possible because they know the tradeoffs they made as they made them. If it looks like a hard and interesting challenge, that desired candidate isn’t you. Nothing wrong with that. They want someone who can knock this class of problems out before lunch on a Tuesday when the boss asks at 10am.
- p1esk 5y ago
  To put things into perspective, someone like that makes $600k+ at FAANG. At a hot new startup they would be making $300k+ base. And these people are in huge demand currently. If you're expecting this level of compensation I believe a take home task like this is not too much.
  - jstx1 5y ago
    That much? For the task that OP is describing? I know that FAANG salaries are high in the upper bands but I think that OP's take-home is doable at a much lower level. Or I'm very underleveled and underpaid. And of course, the task being doable doesn't make it a good take home, I still think OP should turn it down.
    - p1esk 5y ago
      For the person described in brudgers’ comment.
      - bpiche 4y ago
        This role pays around 170-190k
        p1esk 4y ago
        How many years of ML/NLP experience do you have? 190k is probably reasonable for someone with less than 2 years. If you have more, definitely try more companies. A dollar today is worth is lot less than even 1 year ago, just look at the housing prices.
sjg007 5y ago
Take a pre-trained Bert model and then fine-tune the label classifier part using the documents.
- bpiche 5y ago
  It really is that simple, but I didn't think we were there yet!
  - sjg007 5y ago
    Hey man, google pytorch.
    - bpiche 4y ago
      Spent a lot of time trying to get PyTorch and CUDA to play with my 3070ti yesterday and finally gave up and made a CoLab notebook. Got a pipeline working and trained a multiclass classifier, but the f1 is in the toilet. Might try a different dataset, then maybe a tensorflow implementation next if that doesn't move the needle. Hard to believe the accuracy could be this bad compared to a linearSVM baseline, must be giving it the wrong x/y labels or something.
leobg 5y ago
For the second part, I’d also train a fastText classifier. It’s cheap to do and can often get very usable results at 1/500th the computing power of a BERT/SBERT model. Re: the length problem, some ideas: a) Just use the first part of each doc. For most humans, in most cases, that would be enough to do the classification. b) Mix head and tail of each doc (i.e. cut out as much of the middle as necessary). c) Split each doc into as many parts as needed, classify all of them, and then go with the majority vote. Good luck with the interview!
- bpiche 5y ago
  Your first suggestion seems like the way to go, I read a couple of papers this morning which seemed to confirm that it is a solid strategy. The first one claims that the Longformer architecture is tenable for truly long documents with some minor parameter optimization [0]. The Amazon/MSFT paper claims that just looking at the first part of the doc is superior to most other more complex solutions, like Longformer/Big Bird or hierarchical classification [1]. But here's what's really cool, they concluded that if you still use only 512 tokens, but you don't just look at the first ones, you look at basically an extractive summary of the most salient phrases in the document, it's more accurate. They call it BERT-TextRank, and it's using an extractive summarization algorithm from 2004. Found a tutorial which at least makes classification with RoBERTa pretty straightforward. I honestly didn't think this was possible before today but it's always nice to have your preconceptions challenged. [0] Revisiting Transformer-based Models for Long Document Classification (anonymous ACL submission, published yesterday) https://arxiv.org/abs/2204.06683 https://arxiv.org/abs/2204.06683 [1] Efficient classification of long documents using transformers (Amazon, MSFT, March 12, 2022) https://www.amazon.science/publications/efficient-classification-of-long-documents-using-transformers https://www.amazon.science/publications/efficient-classifica...
parentheses 5y ago
Don’t know ML but this is sounding like a lot of work. I almost always say no to take homes. There’s also a limit to how much the a take home should take. >2h of your time is asking a lot. I philosophically disagree with take home interviews for many reasons.
throwaway81523 5y ago
Yes it is too much. I'd decline to do that much work on an unpaid basis without more commitment from the company. Regarding jstx1's question of who was going to read the 1000 words: obviously, an NLP text summarization package that will spit out a number from 1 to 5.
aborsy 5y ago
More than a couple of hours, no, it’s a scam.
fspear 4y ago
Name and shame

11 points by bpiche 5y ago

formerkrogemp 5y ago

Why are you working for free? I'm not in machine learning, but if I got this in an accounting field I would think it was a scam and not a legitimate job offer. Just my two cents. Best of luck friend.

bpiche 5y ago
From the mouths of babes. You get so used to grinding LeetCode and doing take homes weekend after weekend, you stop questioning reality. Maybe it takes someone from an outside field looking in to see it. Thank you for the refreshing perspective..
- formerkrogemp 5y ago
  I've enjoyed reading about programming on this site for a few years, so, no, thank you for providing a peek into your reality.
hjkl0 5y ago
FWIW, some companies do compensate you for trial work.

deleted 5y ago

[deleted]

jstx1 5y ago

> Here are 150k long-form (6000+ words) documents, and a list of labels. > write 1000 words about the performance tradeoffs. Big no. And it shouldn't be a close decisions for you - this is way too much work to do for someone you aren't working for. Who in 2022 is asking for 1000 words as an explanation? And who is actually reading it??? Fascinating stuff, thanks for sharing. (I kind of wish take homes were less common and more companies were doing LC for data science and ML)

deleted 5y ago

[deleted]

brudgers 5y ago

They are looking for someone with the level of experience that allows treating the problem as trivial. That means plugging in the data to the tool they use all the time, adjusting a few parameters, and writing the 1000 words while waiting for the results…that’s possible because they know the tradeoffs they made as they made them. If it looks like a hard and interesting challenge, that desired candidate isn’t you. Nothing wrong with that. They want someone who can knock this class of problems out before lunch on a Tuesday when the boss asks at 10am.

p1esk 5y ago
To put things into perspective, someone like that makes $600k+ at FAANG. At a hot new startup they would be making $300k+ base. And these people are in huge demand currently. If you're expecting this level of compensation I believe a take home task like this is not too much.
- jstx1 5y ago
  That much? For the task that OP is describing? I know that FAANG salaries are high in the upper bands but I think that OP's take-home is doable at a much lower level. Or I'm very underleveled and underpaid. And of course, the task being doable doesn't make it a good take home, I still think OP should turn it down.
  - p1esk 5y ago
    For the person described in brudgers’ comment.
    - bpiche 4y ago
      This role pays around 170-190k
      - p1esk 4y ago
        How many years of ML/NLP experience do you have? 190k is probably reasonable for someone with less than 2 years. If you have more, definitely try more companies. A dollar today is worth is lot less than even 1 year ago, just look at the housing prices.

sjg007 5y ago

Take a pre-trained Bert model and then fine-tune the label classifier part using the documents.

bpiche 5y ago
It really is that simple, but I didn't think we were there yet!
- sjg007 5y ago
  Hey man, google pytorch.
  - bpiche 4y ago
    Spent a lot of time trying to get PyTorch and CUDA to play with my 3070ti yesterday and finally gave up and made a CoLab notebook. Got a pipeline working and trained a multiclass classifier, but the f1 is in the toilet. Might try a different dataset, then maybe a tensorflow implementation next if that doesn't move the needle. Hard to believe the accuracy could be this bad compared to a linearSVM baseline, must be giving it the wrong x/y labels or something.

leobg 5y ago

For the second part, I’d also train a fastText classifier. It’s cheap to do and can often get very usable results at 1/500th the computing power of a BERT/SBERT model. Re: the length problem, some ideas: a) Just use the first part of each doc. For most humans, in most cases, that would be enough to do the classification. b) Mix head and tail of each doc (i.e. cut out as much of the middle as necessary). c) Split each doc into as many parts as needed, classify all of them, and then go with the majority vote. Good luck with the interview!

bpiche 5y ago
Your first suggestion seems like the way to go, I read a couple of papers this morning which seemed to confirm that it is a solid strategy. The first one claims that the Longformer architecture is tenable for truly long documents with some minor parameter optimization [0]. The Amazon/MSFT paper claims that just looking at the first part of the doc is superior to most other more complex solutions, like Longformer/Big Bird or hierarchical classification [1]. But here's what's really cool, they concluded that if you still use only 512 tokens, but you don't just look at the first ones, you look at basically an extractive summary of the most salient phrases in the document, it's more accurate. They call it BERT-TextRank, and it's using an extractive summarization algorithm from 2004. Found a tutorial which at least makes classification with RoBERTa pretty straightforward. I honestly didn't think this was possible before today but it's always nice to have your preconceptions challenged. [0] Revisiting Transformer-based Models for Long Document Classification (anonymous ACL submission, published yesterday) https://arxiv.org/abs/2204.06683 https://arxiv.org/abs/2204.06683 [1] Efficient classification of long documents using transformers (Amazon, MSFT, March 12, 2022) https://www.amazon.science/publications/efficient-classification-of-long-documents-using-transformers https://www.amazon.science/publications/efficient-classifica...

parentheses 5y ago

Don’t know ML but this is sounding like a lot of work. I almost always say no to take homes. There’s also a limit to how much the a take home should take. >2h of your time is asking a lot. I philosophically disagree with take home interviews for many reasons.

throwaway81523 5y ago

Yes it is too much. I'd decline to do that much work on an unpaid basis without more commitment from the company. Regarding jstx1's question of who was going to read the 1000 words: obviously, an NLP text summarization package that will spit out a number from 1 to 5.

aborsy 5y ago

More than a couple of hours, no, it’s a scam.

fspear 4y ago

Name and shame