8 ms·
Alibaba releases 72B LLM with 32k context length
- logicchains 3y agoPerhaps this is the famous "Alibaba Intelligence" that Jack Ma was mocked for in his discussion with Musk.
- deleted 3y ago[deleted]
- qualifiedai 3y agoDoes it still refuse to answer basic history facts (e.g. "What happened in 1989 on Tiananmen square?") like other LLMs from China?
- brucethemoose2 3y agoYi 34B 200K is not like this. In fact, it seems to answer like it is Taiwanese, with way more fervor than base Llama.
- A4ET8a8uTh0 3y agoHow is that any different from US neutering its LLMs to not discuss 'unsafe' topics?
- boppo1 3y agoIt's not. I dislike the US neutering too.
- qualifiedai 3y agoObviously what is being "neutered" is important. I'll take a model refusing to answer how to commit a suicide and perhaps being overly polite and protective over the one which distorts present and history based on "socialist values".
- fullspectrumdev 3y agoPersonally I’ll take neither. Alignment and such is a fucking scam.
- qualifiedai 3y agoNo, alignment should be done. Also, both base and aligned models should be released publicly.
- mensetmanusman 3y agoChoice of data to use is alignment. If you train it on the internet, your LLM will spew SSO garbage, so we align it to be more useful.
- staunton 3y agoAlignment just means that the software does what you want. Of course, any term can be appropriated and abused by whoever can gain from doing so. I still think it's important to have a concrete idea what one means when one uses words. For example, you wouldn't say that "medicine is a fucking scam" to mean that the healthcare industry is broken.
- A4ET8a8uTh0 3y agoNo. The moment you put it upon yourself to decide what is too important to let fragile little minds handle, you have chosen yourself as their thought master. Once anyone chooses that role, they deem themselves more enlightened and therefore more capable than the poor, deluded masses trudging in dirt below from whence they came. I am ok with 'uncensored' LLM, but I am also ok with uncensored internet. The real harm is from people trying to protect me from me, apparently. Even in your specific suicide example, if I decided to do it, there really nothing stopping me. I see no value in that censorship. I only see harm.
- thinkingtoilet 3y ago"Once anyone chooses that role, they deem themselves more enlightened and therefore more capable than the poor, deluded masses trudging in dirt below from whence they came." Or they simply want to sell their models to enterprise clients who don't want to expose certain things to their employees during the work day. This reads a bit like those "I'm a lion" or "I'm the alpha wolf" type posts. Go ahead and try to sell a product that can generate hate speech, nudity, and violent content to enterprises.
- WoahNoun 3y agoOne is a company freely deciding what it's product should and shouldn't do and the other is forced by the government.
- rat9988 3y agoThey are both the same. Edit: to people who downvote, can you tell me why it is different? Aren't companies in USA censoring fearing governmental reaction, and the same for chinese companies?
- bryanlarsen 3y agoThey're censoring fearing the vagaries of fickle advertisers.
- WoahNoun 3y agoNo, they fear that enterprises won't buy their product.
- civilitty 3y ago> Edit: to people who downvote, can you tell me why it is different? Aren't companies in USA censoring fearing governmental reaction, and the same for chinese companies? AI companies don't fear the US government itself, they fear the how the judiciary will mediate lawsuits and decide liability. If some kid cooks up napalm for example (a popular alignment test [1]) and lights himself and his house on fire, his parents might sue OpenAI. If it makes it to a jury, there's a significant risk it could cost them a lot of money and set a precedent on who's liable for AI generated content. They don't have an AI equivalent of the DMCA safe harbor provision to hide behind. [1] https://news.ycombinator.com/item?id=35630801 https://news.ycombinator.com/item?id=35630801
- A4ET8a8uTh0 3y agoYou are on HN, when we discuss encroaching private-public partnerships all the time. Not that long ago, WH got in trouble for encouraging suppression of some news items over others[1]. We are already at a point, where the difference between corps and government is more of a legal abstraction than a reality ( legal abstraction circumvented by that partnership, but that is a different rant ). There is no difference. [1]https://www.cnn.com/2023/02/08/politics/twitter-hearing-house-oversight/index.html https://www.cnn.com/2023/02/08/politics/twitter-hearing-hous...
- mensetmanusman 3y agoHow are western values (historical anomaly) different than authoritarian values (historical norm)?
- campbel 3y agoSeriously, the constant conflating of authoritarian censorship with limiting hate speech and bomb making instructions is tiring.
- A4ET8a8uTh0 3y agoI know right? That those non-aurhoritarian regimes are so hell bent on adopting authoritarian features is mind boggling.
- ok123456 3y agoTry asking ChatGPT any questions about Israel without getting jibberish about "controversy" and it being "complicated."
- mensetmanusman 3y agohttps://chat.openai.com/share/369d8511-e525-4a99-b79a-8a5dc09113ce https://chat.openai.com/share/369d8511-e525-4a99-b79a-8a5dc0... Tried the trick of telling it to take the view of communists?
- I_Am_Nous 3y ago>China generally adheres to a principle of non-interference in the internal affairs of other countries, a stance rooted in its own historical experiences with foreign intervention. (...) Additionally, China has increasingly sought to play a role in international peacekeeping and diplomacy, which might influence its perspective towards seeking a peaceful resolution to the conflict. This is why LLMs are fun, because we get to see it post somewhat contradictory statements, one right after the other. Does China want to interfere, or not?
- 3y ago
- MuffinFlavored 3y agoDo you mean 1989?
- qualifiedai 3y agoSorry, yes :). You can try that question on "aligned" Baichuan2-chat model on HF.
- nkingsy 3y agoTrained on the same number of tokens as the 14b, which was only a little more than the 7b. Seems like they just ran out of good tokens?
- MuffinFlavored 3y agoOne thing I struggle to understand is, are LLMs just a large way of spitting out which token it thinks is best based on scoring? aka it's guessing at best? How will they ever grow to the point where they are nothing more than just really good at guessing what to regurgitate based on what it's already seen? Like when it spits out what would have been a StackOverflow answer (where the community votes if it is right/wrong and it's a living thing with comments to keep it up to date as it goes stale) but mixes up a few of the tokens making the whole thing mildly broken... are these systems just going to get really good? will they ever be trustable/perfect?
- lukev 3y agoNot without extra machinery around the model itself. Token generation is always probabilistic to some degree.
- mrb 3y ago"what to regurgitate based on what it's already seen" ...which is no different than what humans do :-) Our neurons fire in a way only because they fire in patterns they have already seen before.
- porphyra 3y agoOne piece of evidence that LLMs form some sort of internal reasoning model is the stochastic parrot chess engine: https://parrotchess.com/ https://parrotchess.com/ Despite it being statistically improbable for it to have seen every chess position before, it nonetheless plays very strongly.
- MuffinFlavored 3y ago> One piece of evidence that LLMs form some sort of internal reasoning mode Isn't this a fancy way of saying "there's a scoring mechanism" like when you brute-force train a model, and certain chains/patterns of tokens score higher than others? Like in my mind, it can learn English by you showing it a bunch of known-good sentences, telling it they score highly, and bad sentences score poorly. Noun verb structure and all that. Am I greatly oversimplifying it? I feel like I am.
- saurabh20n 3y agoSummary from https://arxiv.org/pdf/2309.16609.pdf https://arxiv.org/pdf/2309.16609.pdf --- (q: how does one format lists on HN?) * qwen-{1.8B,7B,14B}: * 3 trillion tokens; start with BPE tiktoken, cl100k base vocab, augmented with chinese, numbers split into digits, final vocab 152k. * RoPE - rotary positional embedding * context length 2048 * qwen-14b perf percentages: 66.3 MMLU(5), 72.1 CEval(5), 61.8 GSM8K(8), 24.8 MATH(4), 32.3 HumanEval(0), 40.8 MBPP(3), 53.4 BBH(3); beats LLaMA2-13B on all, but behind LLaMA2-70B on all except CEval, MATH and HumanEval (somewhat surprising) * code-qwen-{7B,14B} * additional 90B code tokens over base * context length 8192, flash attention * 14B perf: humaneval 66.4, mbpp 52.4; ok, but not stellar (similar numbers as OSS wizardcoder-py, and lower than gpt-3.5) * math-qwn-{7B,14B}-chat * math instructional dataset * context length 1024 * 14B perf: gsm8k 69.8, MATH 24.2, Math401 85.0, Math23K 78.4 (substantially better than OSS in the same weight class (WizardMath and GAIRMath-Abel) on MATH but same ballpark on GSM8k -- surprising). Math23K is chinese grade school math; and Math401 is arithmetic ability. * comprehensive automatic evaluation in Appendix A.2.1 pg 36 (based on OpenCompass'23) * chat format: <|im_start|>system You are a helpful assistant.<|im_end|> <|im_start|>user Hello!<|im_end|> <|im_start|>assistant Hello! How can I assist you today?<|im_end|>