4 ms·

I can appreciate that the model might be trained to work on primarily chinese language, but the performance for me really doesnt seem to be there yet. I hate t

I can appreciate that the model might be trained to work on primarily chinese language, but the performance for me really doesnt seem to be there yet. I hate to do the "I asked ai and pasted it here" style of comment, but I really wanted to understand the validity of the claim it surpasses gpt4 in all those benchmarks. I asked gpt4 and qwen-vl-max this prompt to try to get a sense of their capabilities. Nothing even close imo. "tell me a story about relations between people living on one side of a penny floating through space and the other. Use the format of the hero's journey, including rejecting the call, focus on worldbuilding by hinting at further complexity, and do it in a kafkaesque style" quen's story starts with: Once upon a time, there was a penny floating through space. gpt 4: Through the vast emptiness of space drifted an ancient penny, a relic from a long-forgotten Earth. The difference is night and day atm. Ask yourself: would Kafka have started a story with "once upon a time"

offices 3y ago
No, because all of Kafka's books were written in German.
- deleted 3y ago
  [deleted]
andy99 3y ago
I don't think we can extrapolate much from that anecdote, but I'll try. The GPT version was predictably wordy and had the signature precocious high-school essay feeling, e.g. lots of adjectives. It didn't feel Kafkaesque at all, where is the confusion, breathless tone, mysterious K.? Qwen's is pretty unremarkable, just very direct, but I don't see that one is actually following directions better. Maybe further along the difference gets more stark. Also, isn't Qwen a Chinese-first model? How much of the fine tuning was in English? Neither example gets too actual utility, though it could highlight the sort of demo that chatgpt is trained for.
- RugnirViking 3y ago
  that's fair. I just didn't want to paste a massive text block so went with the first line, which showed the extremely basic way of starting quen used. Link to the pastebin below (actually ended up being anonpate, seems like pastebin is down for me?) I felt the overall writing & characterization of gpt4 was far better, including themes of bureaucracy and psychosis whereas quen sounds like a cliched 6 year old writing an aimless story. I'm not saying either of these are amazing stories, but I think any english teacher/creative writing teacher would rate one over the other as things stand right now. Things like the comparing the rows of wheat embossed on the penny to the regimented way of life its people lead are actually half decent. https://anonpaste.pw/v/ccbfb3e8-746f-4311-9db4-cedbf6c3a440#cAtScCMIIST0-zpxj-KFjylZvsbgd9ywJYnbmqAms_A https://anonpaste.pw/v/ccbfb3e8-746f-4311-9db4-cedbf6c3a440#...

offices 3y ago

No, because all of Kafka's books were written in German.

deleted 3y ago
[deleted]

andy99 3y ago

I don't think we can extrapolate much from that anecdote, but I'll try. The GPT version was predictably wordy and had the signature precocious high-school essay feeling, e.g. lots of adjectives. It didn't feel Kafkaesque at all, where is the confusion, breathless tone, mysterious K.? Qwen's is pretty unremarkable, just very direct, but I don't see that one is actually following directions better. Maybe further along the difference gets more stark. Also, isn't Qwen a Chinese-first model? How much of the fine tuning was in English? Neither example gets too actual utility, though it could highlight the sort of demo that chatgpt is trained for.

RugnirViking 3y ago
that's fair. I just didn't want to paste a massive text block so went with the first line, which showed the extremely basic way of starting quen used. Link to the pastebin below (actually ended up being anonpate, seems like pastebin is down for me?) I felt the overall writing & characterization of gpt4 was far better, including themes of bureaucracy and psychosis whereas quen sounds like a cliched 6 year old writing an aimless story. I'm not saying either of these are amazing stories, but I think any english teacher/creative writing teacher would rate one over the other as things stand right now. Things like the comparing the rows of wheat embossed on the penny to the regimented way of life its people lead are actually half decent. https://anonpaste.pw/v/ccbfb3e8-746f-4311-9db4-cedbf6c3a440#cAtScCMIIST0-zpxj-KFjylZvsbgd9ywJYnbmqAms_A https://anonpaste.pw/v/ccbfb3e8-746f-4311-9db4-cedbf6c3a440#...