13 ms·

Replit's new Code LLM: Open Source, 77% smaller than Codex, trained in 1 week

891 points by swyx 4y ago

jeremypress 4y ago
Interesting how this guy has a finance background but knows how to code, especially for emerging technologies like LLMs
- ipsum2 4y ago
  Didn't MosaicML do the training for them?
amasad 4y ago
Some links: - Repo: https://github.com/replit/ReplitLM/tree/main/replit-code-v1-3b https://github.com/replit/ReplitLM/tree/main/replit-code-v1-... - HuggingFace: https://huggingface.co/replit/replit-code-v1-3b https://huggingface.co/replit/replit-code-v1-3b - Demo: https://huggingface.co/spaces/replit/replit-code-v1-3b-demo https://huggingface.co/spaces/replit/replit-code-v1-3b-demo - Early benchmark results: https://twitter.com/amasad/status/1651019556423598081 https://twitter.com/amasad/status/1651019556423598081 A lot about this project was surprising. We knew it was going to be good, but didn't expect to be this good -- especially surprising was the finetuned performance boost, and the fact that the model is decent at language tasks and reasoning (in some cases much better than much larger general-purpose models). It feels like there is a lot more to do with this model, and I have a suspicion you can even make a half-decent chatbot (at least one focused on code) by finetuning it on conversation (and/or instruction) datasets. Will follow up with a more comprehensive technical report and the UL2R version (fill-in-the-middle support).
- pera 4y ago
  Hi there, I have two question: 1 - Why did you choose Markdown? It seems an odd choice for training a model like this. 2 - Have you tried to train only one single PL and then benchmark it against this more general version?
  - amasad 4y ago
    1- We trained on languages that are most popular on Replit. Markdown is important because you need some amount of natural language in the data, and it will act as a sort of "natural language label" for code. 2- I like how portable it is being a single small model doing a lot of languages. Single code models are an approach that models like Salesforce/Codegen did that, but I believe we beat (or get very close) to their mono models on benchmarks.
    - fuzzythinker 4y ago
      Have you thought of finding or creating something like this [0]? I created this as the basis for my origami folding descriptive language. I tried to find something similar, requirements being both well structured and English-like but couldn't find any, so I created it. The origami folding app will hopefully be out in 2 weeks, so you can see how it's used. [0] https://github.com/fuzzthink/mation-spec https://github.com/fuzzthink/mation-spec
  - runnerup 4y ago
    They trained on https://huggingface.co/datasets/bigcode/the-stack-dedup https://huggingface.co/datasets/bigcode/the-stack-dedup which is a massive curated dataset accumulated from GitHub. Details are here: https://www.bigcode-project.org/docs/about/the-stack/ https://www.bigcode-project.org/docs/about/the-stack/ Many of the most-represented "languages" on GitHub are actually things like JSON, XML, HTML, CSV, text, markdown, YAML, and SVG. More details from them here: https://blog.replit.com/llm-training https://blog.replit.com/llm-training
- gbasin 4y ago
  Very exciting, thanks for sharing all this
- letitgo12345 4y ago
  Doesn't the Stack contain HumanEval? So you're basically comparing numbers on the pretraining data.
  - amasad 4y ago
    Can't find it now but pretty sure BigCode said somewhere they explicitly looked for it and removed it. Also subjective measure does match up to the benchmark. Our finetuned model performed +50% on HumanEval and then when using it felt at least that much improved.
    - godelski 4y ago
      You can view the prompts, solutions, and checks here[0]. See my sibling comment (to yours) where I quote the Human Eval paper and do some more analysis. But I think if you look at [0] you'll see that these aren't really unique problems and are likely to have large repetitions in the dataset. I should add to that comment to include the dataset[1] (too late to edit) where they mention that they just scrape all of GitHub (Jan 1 2015 - Mar 31 2022). They do exact and near de-duplicate but near de-duplication is messy. > We implement near-deduplication in our pre-processing pipeline on top of exact deduplication. We first split the files into words/tokens based on non-alphanumeric characters and remove files with fewer than 10 tokens. Next, we compute the MinHash with 256 permutations of all documents, and use Locality Sensitive Hashing to find clusters of duplicates. We further reduce these clusters by ensuring that each file in the original cluster is similar to at least one other file in the reduced cluster. We consider two files similar when their Jaccard similarity exceeds 0.85. Near-duplicates are still difficult to measure. So we should expect duplication, and it should be proportional to the number of samples we have (even if the same variance, but I'd wager higher variance with larger duplications). [0] https://github.com/openai/code-align-evals-data/tree/97446d992c3785d6605f1500b2c9b95d042e7b9c/human_eval https://github.com/openai/code-align-evals-data/tree/97446d9... [1] https://arxiv.org/abs/2211.15533 https://arxiv.org/abs/2211.15533
  - godelski 4y ago
    My favorite line from the HumanEval paper[0] > It is important for these tasks to be hand-written, since our models are trained on a large fraction of GitHub, which already contains solutions to problems from a variety of sources. So to answer your question, yes, the evaluation dataset is spoiled. You can find such unique and never before seen docstrings like > For a given list of input numbers calculate the Mean Absolute Deviation around the mean of this dataset. Mean Absolute Deviation is the absolute difference between each element and a centerpoint (mean in this case)[1] And here's a repo I found that is 8 years old[2]. But how about a more recent one that is even closer?[3] There's plenty more examples[4] (does anyone know how actually limit the date to prior to 2021? `pushed:<2021` doesn't work nor does using the `created` keyword. Date searching doesn't seem to work well). In essence, we can still use this evaluation method to determine how good our model is at doing fuzzy searching. Which, mind you, is still a useful thing. But I would be careful in concluding that this means the model is good at generalizing arbitrary descriptions of code or novel pieces of code. That said, one may be able to argue that not many lines of code are actually that novel. Still, we need to be careful about our conclusions and understand the limitations of our metrics (something I am currently deeply troubled by) [0] https://arxiv.org/abs/2107.03374 https://arxiv.org/abs/2107.03374 [1] https://github.com/openai/code-align-evals-data/blob/97446d992c3785d6605f1500b2c9b95d042e7b9c/human_eval/floats_mean_absolute_deviation.py#L6 https://github.com/openai/code-align-evals-data/blob/97446d9... [2] https://github.com/bertomartin/stat4701/blob/ec2b64f629cbbf6267169302265f73a98edef67d/stck.py#L175 https://github.com/bertomartin/stat4701/blob/ec2b64f629cbbf6... [3] https://github.com/danielwatson6/hate-speech-project/blob/64a2eecce5218373ef5c449eeb1dfb397532eda5/scripts/wordnet_mf.py#L61 https://github.com/danielwatson6/hate-speech-project/blob/64... [4] https://github.com/search?q=abs%28x+-+mean%29+for+language%3APython&type=code https://github.com/search?q=abs%28x+-+mean%29+for+language%3...
- newhouseb 4y ago
  First - thank you for open sourcing this! It's a real gift to the community to have a model intended for "commercial use" that's actually licensed as such. I'd be very interested to hear about the choice/evaluation of the ALiBi approach for positional embedding (perhaps in the technical report). My intuition suggests that while this allows for better generalizability for longer sequence lengths, it penalizes scenarios where an LLM might need to check for things like a function signature far away from where the next token is generated. My initial testing of this model tracks with this intuition but that's by no means a rigorous evaluation.
  - ofirpress 4y ago
    (I wrote ALiBi) You can read the paper here https://arxiv.org/abs/2108.12409 https://arxiv.org/abs/2108.12409 While intuitively it does seem like ALiBi would make it hard for the model to attend to things that are far away, in many scenarios we've tested with different models trained on different datasets, ALiBi always performs better than sinusoidal, rotary, and other embedding types, even when we're not using it to extrapolate to longer sequence lengths. These findings have been confirmed by others, including by the BLOOM open source LM project.
    - newhouseb 4y ago
      Small world! Thanks for the link (which I've now skimmed beyond the abstract). What wasn't obvious to me from the abstract is that different attention heads have different penalty strengths, so if some prediction task requires long range dependencies you might expect one of the less-penalized heads to end up specializing. I wonder what would happen if the penalty for one head is zero? (The paper suggests this might've been tried and just made things worse, but unclear) I must admit that this is a wonderfully elegant (and interpretable) way to do this... much more intuitive (to me at least, a wannabe practitioner) than all of the trig-based embeddings.
      - ofirpress 4y ago
        > so if some prediction task requires long range dependencies you might expect one of the less-penalized heads to end up specializing Exactly. You have heads that focus on content nearby and ones that focus on stuff that is far away. > I wonder what would happen if the penalty for one head is zero? (The paper suggests this might've been tried and just made things worse, but unclear) Yup, this is something we tried. Making one of the heads zero doesn't improve or degrade performance. >I must admit that this is a wonderfully elegant (and interpretable) way to do this... much more intuitive (to me at least, a wannabe practitioner) than all of the trig-based embeddings. Thanks so much!!
- sputknick 4y ago
  What does "fine tuning" mean in this context? Does it mean you fine-tuned it on a specific code repository, or collection of code repositories and then had it do work in those repositories?
  - amasad 4y ago
    Broadly finetuning is any post pretraining training. Most of the time it is used in the context of fitting a more narrow task. In our case, it was the same training objective as the pretraining but meant to be more representative of what Replit users like to code. However, we were surprised by how well it boosted overall performance. Best guess: it's a) novel data and b) the model could take even more training!!
    - spenczar5 4y ago
      How feasible and effective would it be to fine-tune a model against an organization's private source code, resulting in an "internal" model that knows how to work with that org's stuff? Could you, say, fine-tune the model every week with the latest merges? Every hour?
      - pyth0 4y ago
        Finetuning is a relatively quick process. Training the base model is the expensive part (can take weeks and huge amounts of compute), whereas finetuning usually is only on the last few layers and can be done with much less resources. You could definitely have a "nightly" finetune model that is retrained every day or so.
        rattray 4y ago
        Interesting - how would that work for a company that wanted to run their own codex model, on-prem, trained on their own code? Perhaps also trained on their dependencies?
      - naderkhalil 4y ago
        Finetuning a smaller model leading to better performance seems like a significant finding that'll lead to a lot of companies fine-tuning their own internal "ChatGPT"s
- spenczar5 4y ago
  How is this code licensed? I didn't see a license in the repo. It looks interesting!
  - dgacmu 4y ago
    The README indicates: The base model checkpoint is licensed under the Creative Commons license (CC BY-SA-4.0). Under the license, you must give credit to Replit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests that Replit endorses you or your use.
- kir-gadjello 4y ago
  Impressive model, thank you for releasing it under a business-friendly license! Have you considered using Google's sparse "scaling transformer" architecture as the base? Even at 3B scale it can generate 3-4x more tokens per FLOP while being competitive at perplexity with a dense transformer. I think OpenAI uses a variant of it in their ChatGPT-3.5-Turbo product. Here is the paper https://arxiv.org/abs/2111.12763 https://arxiv.org/abs/2111.12763 and the implementation https://github.com/google/trax/blob/master/trax/models/research/terraformer.py https://github.com/google/trax/blob/master/trax/models/resea... if you are interested. Hope you get to look into this!
  - b33j0r 4y ago
    Thank you for releasing the weights along with the announcement. The posts that made great headlines, but “weights are on their way!” Like why did we even get excited? This? Great work.
  - chaxor 4y ago
    I don't think it's a business friendly license?
    - kir-gadjello 4y ago
      It allows for modifications and commercial use: https://creativecommons.org/licenses/by-sa/4.0/ https://creativecommons.org/licenses/by-sa/4.0/ >You are free to: >Share — copy and redistribute the material in any medium or format >Adapt — remix, transform, and build upon the material >for any purpose, even commercially. Compare this to the latest release from StabilityAI lab DeepFloyd, "IF", which in addition to various restrictive clauses strictly prohibits commercial use: https://github.com/deep-floyd/IF/blob/develop/LICENSE-MODEL https://github.com/deep-floyd/IF/blob/develop/LICENSE-MODEL Repl.it's release is as open as it gets these days, in my book.
      - LukeShu 4y ago
        It's a copyleft license; and lots of folks on HN seem to think that copyleft, while being open, isn't business friendly.
- curiousgal 4y ago
  Did any interns help in developing this? If so are you planning on intimidating them as usual? :) Reference: How Replit used legal threats to kill my open-source project https://intuitiveexplanations.com/tech/replit/ https://intuitiveexplanations.com/tech/replit/
  - robertlagrant 4y ago
    Wow. That's extremely poor behaviour if the account is accurate.
    - curiousgal 4y ago
      Oh it is. 4000+ upvotes on HN. https://news.ycombinator.com/item?id=27424195 https://news.ycombinator.com/item?id=27424195
swyx 4y ago
hi HN! back again with an exclusive deep dive with Replit’s head of AI. I attended their developer day last week (https://twitter.com/swyx/status/1650989632413401089 https://twitter.com/swyx/status/1650989632413401089) just expecting a regular fundraise announcement and was totally shocked when they annoucned their own LLM and also said they would open source it. so immediately asked them for a podcast interview and this is the result. my favorite learning is how they are pushing the state of the art - openai’s HumanEval is the industry standard benchmark for code LLMs, but Reza kindly went above and beyond to show how they use “AmjadEval” - using coder intuition to capture human preference on what output is more helpful to coders (see screenshots https://twitter.com/swyx/status/1653791019421569024?s=20 https://twitter.com/swyx/status/1653791019421569024?s=20) please AMA!
- marcodiego 4y ago
  Sorry, I have to ask this: how does this compare to ChatGPT?
  - swyx 4y ago
    it doesn't. replit-code-v1-3b is a code LLM, ChatGPT is an app on top of LLMs. it compares to OpenAI Codex, a small version of which is behind GitHub Copilot.
    - mritchie712 4y ago
      It (replit-code-v1-3b) is already quite good at explaining code: Input: below is a SQL statement: SELECT CAST(DATE_TRUNC('week', "t1"."TIMESTAMP") AS DATE) AS "WEEK_START", COUNT(\*) AS "EVENT_COUNT" FROM "ANALYTICS"."POSTHOG"."POSTHOG_EVENTS" AS "t1" GROUP BY "WEEK_START" ORDER BY "WEEK_START" LIMIT 2000 Explain this SQL. Respond in JSON format with the following keys: TITLE, DESCRIPTION, TABLES JSON response: output: { "title": "Weekly Events Count", "description": "Count of weekly events", "tables": [ { "name": "POSTHOG_EVENTS", "columns": [ "WEEK_START", "EVENT_COUNT" ] } ] }
tarruda 4y ago
3 billion parameters. Does that mean I will be able to run on a 8gb consumer GPU?
- generalizations 4y ago
  Means that once it's incorporated into llama.cpp, you can run it on your laptop.
  - amasad 4y ago
    Hopefully on phones too
- RHab 4y ago
  No, I could only get 2.7B to run on 8GB VRam unfortunatly.
  - amasad 4y ago
    it is 2.7B
    - tarruda 4y ago
      Loading seems to have worked on my laptop's RTX 3070, `nvidia-smi` shows `5188MiB / 8192MiB` in memory usage.
- pera 4y ago
  their pytorch_model.bin is 10.4GB
  - tarruda 4y ago
    I just loaded this on my laptop's RTX 3070 GPU by following the instructions here: https://huggingface.co/replit/replit-code-v1-3b https://huggingface.co/replit/replit-code-v1-3b I don't know how I can test the model, but it seem loading worked. When I run `nvidia-smi` on another terminal, I see `5188MiB / 8192MiB` in the memory-usage column.
tyingq 4y ago
More tools in the field is great! I tried a few things, and it's reasonable, but it does have some quirks that seem to repeat, like: I tried a prompt of: # python function that returns a random integer between min and max And it produced: def random_int(min, max): return random.randint(min, max) # define the size of the grid n = 5 It doesn't add the needed import statement, and I'm unclear why it's "defining the size of the grid".
- circuit10 4y ago
  That's because it's not following instructions like ChatGPT, it's just trying to guess that could plausibly come after what you put, like Copilot or the old GPT-3 models
  - minkzilla 4y ago
    and imports are (almost) always at the top of your file not with this function
    - vharuck 4y ago
      I tried the same input, except wrapping it in triple-quotes instead of commenting it. So that it would match the standard practice for module doc strings. Here's the result: """python function that returns a random integer between min and max""" return random.randint(min, max) def gen_random_float(min, max): """python function that returns a random float between min and max""" return random.uniform( So, it assumed the triple-quote was a function's doc string, despite it not being indented. It then assumes I'll want a similar function for floats (I assume it was cut off by a token limit).
  - jeremyjh 4y ago
    Isn’t ChatGPT also just generating plausible text that could be a response to an instruction?
    - whimsicalism 4y ago
fswd 4y ago
I can barely keep up with this stuff, but quick question. Is there a way to simply change the URL setting of copilot to point to this model? Obviously it needs an endpoint, I could hack something up, but asking if somebody has already done this? Would be nice to cancel my copilot.
- jacobrussell 4y ago
  I don't think it's possible to point Copilot to other models. I don't think Microsoft would benefit much from that feature. You could use existing tools [0] to host your own model which in theory could be used by an extension your IDE uses. But I'm not sure if an extension like that exists. [0] https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui
  - circuit10 4y ago
    Of course it's possible, just not officially See https://github.com/fauxpilot/fauxpilot/blob/main/documentation/client.md#copilot-plugin https://github.com/fauxpilot/fauxpilot/blob/main/documentati...
- circuit10 4y ago
  There's https://github.com/fauxpilot/fauxpilot https://github.com/fauxpilot/fauxpilot but it doesn't use this model
- execveat 4y ago
  It's nowhere close to Codex/Copilot. Try the demo: https://huggingface.co/spaces/replit/replit-code-v1-3b-demo https://huggingface.co/spaces/replit/replit-code-v1-3b-demo
  - tarruda 4y ago
    So they are lying on this tweet? https://twitter.com/Replit/status/1651344186715803648 https://twitter.com/Replit/status/1651344186715803648
waffletower 4y ago
No Clojure. No Julia. No Haskell. No Racket. No Scheme. No Common Lisp. No OCaml. And, as much as I despise Microsoft, No C#. No F#. No Swift. No Objective-C. No Perl. No Datalog. A glaringly lacking choice of languages.
- ebiester 4y ago
  I'm sure that has to do with the dataset available to them.
  - runnerup 4y ago
    Which is a deduplicated version of this: https://www.bigcode-project.org/docs/about/the-stack/ https://www.bigcode-project.org/docs/about/the-stack/ And probably, yes. While it contains 358 programming languages, obviously there's a long tail after the 20 most-represented languages. Some people might not expect without thinking about it for a bit that many of the most-represented "languages" are actually things like JSON, XML, HTML, CSV, text, markdown, YAML, SVG. Also note that it won't be able to parse natural language nearly as well without additionally being trained on something like the LAION dataset, so this version will be more of an autocomplete like Copilot rather than something which can manifest high level business logic from whole cloth like ChatGPT.
- Dayshine 4y ago
  C# was available in the dataset they link, and is the most glaring ommission by global usage...
- ubertaco 4y ago
  I fed it some OCaml and it worked, though the example was trivial: type point = { x: int; y : int } let manhattan_distance (a: point) (b: point) : int = which it completed to type point = { x: int; y : int } let manhattan_distance (a: point) (b: point) : int = abs (a.x - b.x) + abs (a.y - b.y) ...which is a valid and correct OCaml definition of this method: https://try.ocamlpro.com/#code/type'point'='$4'x:'int;'y':'int'$6!let'manhattan_distance'(a:'point)'(b:'point)':'int'=!abs'(a.x'-'b.x)'+'abs'(a.y'-'b.y)!;;!!let'origin'='$4x'='0;'y'='0$6'in!let'dest'='$4x'='2;'y'='3$6'in!Printf.printf'$($+d$('(manhattan_distance'origin'dest) https://try.ocamlpro.com/#code/type'point'='$4'x:'int;'y':'i...
circuit10 4y ago
This probably makes a self-hosted and/or local Copilot a lot more feasible
- FanaHOVA 4y ago
  Yes, something like FauxPilot[0] should be able to use it instead of CodeGen [0] https://github.com/fauxpilot/fauxpilot https://github.com/fauxpilot/fauxpilot
GreedClarifies 4y ago
This is amazing work and bravo on to the people working on redpajama. This is fantastic for the world, this means LLMs will not be controlled by a couple of companies with the associated rents. Yes, private LLMs will likely be a couple of years ahead of 'free' alternatives, but that's OK, we want to incentivize for profit research so long as the services are low priced in time (and in this case in short order). AMAZING WORK.
- swyx 4y ago
  to be clear this work is not based on redpajama - though we did discuss that in the previous episode https://twitter.com/swyx/status/1648080532734087168?s=46&t=90xQ8sGy63D2OtiaoGJuww https://twitter.com/swyx/status/1648080532734087168?s=46&t=9...
  - GreedClarifies 4y ago
    Oh my bad! I thought I read that, is it based upon: https://arxiv.org/abs/2211.15533 https://arxiv.org/abs/2211.15533 (The Stack) ?
    - swyx 4y ago
      partially. Reza discussed their data pipeline in the blogpost that we reference in the show notes
- m3kw9 4y ago
  Have you even tried it? It’s pretty bad
  - GreedClarifies 4y ago
    But that's fine it can be a year or two behind the state of the art. That's not the point. The point is that there will be alternatives and that will reduce the price in time further increasing the impact of the technology. There was a possible future where only MSFT and maybe GOOG and maybe one or two other companies had this technology and extracted massive rents.
love2read 4y ago
Can this be used with the copilot plugins for every ide?
protonbob 4y ago
Darn it doesn't look like it has c sharp.
abxytg 4y ago
[flagged]
LightMachine 4y ago
Any idea how much it cost to train it and how it was trained?
Imnimo 4y ago
Tried it out on the HuggingFace demo, with default settings. Prompt: >def nth_prime(n): Completion: > if n == 1: > return 2 > if n == 2: > return 3 > if n == 3: > return 5 > if n == 4
robertlucas 4y ago
Is there any way to connect these new code focused LLMs into VS Code in order to replace Github Copilot?
vrglvrglvrgl 4y ago
[dead]
sashank_1509 4y ago
Replit: I tried to write HMM Forward function code: ``` # Write a function that performs the forward algorithm of an HMM. Ideally it uses numpy vectorization, # But if you cannot do that it is fine too. Just perform forward inference, given HMM parameters and Observations. # def HMM_forward() : # Write your code here # Hint: # - Remember that HMMs are stochastic, so you will need to multiply transition matrices by observation probabilities # - Remember that HMMs are stochastic, so you will need to multiply emission matrices by observation probabilities # - Remember that HMMs are stochastic, so you will need to multiply emission matrices by transition matrices # - Remember that HMMs are stochastic, so you will need to multiply emission matrices by transition matrices # - Remember that HMMs are stochastic, so ``` It gets stuck in an infinite loop. ChatGPT gets it write, first shot: ``` import numpy as np def forward_algorithm(A, B, pi, obs): """ Performs the forward algorithm for an HMM. Parameters: A: numpy.ndarray, shape (N, N) Transition matrix of the HMM, where N is the number of states. B: numpy.ndarray, shape (N, M) Emission matrix of the HMM, where M is the number of possible observations. pi: numpy.ndarray, shape (N,) Initial probability distribution over states. obs: numpy.ndarray, shape (T,) Sequence of T observations. Returns: alpha: numpy.ndarray, shape (T, N) Forward probabilities for each state at each time step. """ T = obs.shape[0] N = A.shape[0] alpha = np.zeros((T, N)) alpha[0] = pi * B[:, obs[0]] for t in range(1, T): alpha[t] = np.dot(alpha[t-1], A) * B[:, obs[t]] return alpha ``` OpenAI managed to do the important but extremely hard, they moved out of the DL benchmark frame and made something that is general purpose useful. Great effort and congrats to Replit team though, hopefully they can keep iterating on this and reach ChatGPT capabilities someday
- amasad 4y ago
  The model is not RLHF'd or instructed. It's an inline autocomplete model so it will get confused if you talk it like you're talking to a person. Altho it is possible to finetune it this way. To get better full function completion try giving it the function definition and a descriptive docstring as a prompt.
  - deleted 4y ago
m3kw9 4y ago
It just gave me prototypes lol def sieve_eratosthenes(n): ##a function to sort 10 numbers def bubble_sort(a): ##a function to sort 10 numbers def insertion_sort(a): ##a function to sort 10 numbers def quick_sort(a):
- m3kw9 4y ago
  I left the settings. All I added was ##a function to sort 10 numbers. Assuming it would complete it like copilot
- ImprobableTruth 4y ago
  Did you mess around with the settings? I'm getting a correct implementation and since it's deterministic (with default settings) it should be the same for you.
robby_w_g 4y ago
I recognized the name Replit and couldn't remember why. A quick search reminded me: https://news.ycombinator.com/item?id=27424195 https://news.ycombinator.com/item?id=27424195
- stephenjayakar 4y ago
  this feels like an attempt to hive mind against anything cool from this company
  - dimgl 4y ago
    +1, this is unnecessary.
    - aardshark 4y ago
      Alternatively, it's called consequences of your actions. Don't be surprised if shitty behaviour comes back to bite you.
  - ibrarmalik 4y ago
    I think people are smart enough to receive extra information and do whatever they want with that.
  - naillo 4y ago
    Threatening a guy for making an open source version of replit sounds pretty crummy in my eyes.
  - robby_w_g 4y ago
    I think it's fair to evaluate a company's behavior before engaging in business with them. And I personally dislike persons in power abusing their position, which is why I remembered the company name almost two years later. I haven't heard of any similar behavior since then, which is a good sign. But a reputation can be a hard thing to shake. The CEO should have considered that before doing what he did.
- naillo 4y ago
youssefabdelm 4y ago
title is missing: "trained in 1 week, and like most open source LLMs so far... it sucks compared to the closed source alternatives" Great effort of course bla bla bla... Open source really needs some benchmarking, and up their game quality-wise. And yes I know they're expensive as shit to train... let's not keep wasting our money and actually work together, pool our resources, to make a GOOD model. But oh no, everyone wants to put their stamp on it. "Replit did this! Look at us!"
- ImprobableTruth 4y ago
  This is easy to say, but I think the issue is that getting an LLM right isn't easy, so it's not clear who should steward such a project. Something like BLOOM shows that even if you have the necessary compute, you can still get a model that isn't good. I think it will take some time for it to be clear who is a leader in training open source models (maybe it will be the red pajama folks?) and I think they'll get more support after that.
  - youssefabdelm 4y ago
    Fair point
heliophobicdude 4y ago
Important distinction that I'm learning today is that not all LLMs will be interoperable with each others' queries/prompts/inputs. Code LLM right now is not responding how a Chat LLM would respond. ~~~~~ Hats off to the team on the impressive work!
eikenberry 4y ago
Is this a Co-pilot like assistant or something more? Co-pilot is neat but is basically not much more than an automated snippet system. The actual writing of the code is not the part that I want help with, I want an AI system that helps me design better software systems. Something more akin to program mind mapping than some fancy auto-completion system.
- qrio2 4y ago
  I wonder if LLM with something like plantUML would generate anything useful
gowld 4y ago
Can I use repl.it with an external Code LLM, with or without paying repl.it for Ghostwriter ?
- amasad 4y ago
  Yes we have a robust extension system and some are already building alternatives.
  - varunkmohan 4y ago
    Hi from the Codeium team. It's awesome to hear you are allowing other code LLMs to be used on the Replit platform (we're big fans)! We'd love to enable our free chrome extension on Replit.
    - swyx 4y ago
      would love to be able to compare codeium vs ghostwriter inside replit! (or toggle between them based on known strengths or preferences, perhaps by project or by filetype)
SXX 4y ago
Weak spot which I guess similar to other LLMs. If you mention recursion somewhere in comments model sometimes start to recursively generate the same lines over and over again.
doodlesdev 4y ago
The model is way too small, comparing it to Codex feels disingenous. Sure it's 77% smaller, it's also 77% worse. Although, it's a cool project nonetheless. For instance, even this simple snippet generates wrong inline completions: // Only return even numbers bigger than 10 from the array const arrayFilter = (array) => Replit-code-v1: // Only return even numbers bigger than 10 from the array const arrayFilter = (array) => { return array.filter((item) => item > 10); }; Gets it wrong, returns odd numbers. Codeium: // Only return even numbers bigger than 10 from the array const arrayFilter = (array) => { return array.filter((num) => num > 10 && num % 2 === 0); }; ChatGPT (GPT-3.5 Turbo) - Code-only, without the rest of the completion since it's instruction-tuned: const arrayFilter = (array) => { return array.filter(num => num % 2 === 0 && num > 10); } Not comparable at all. For reference if anyone wants to test I ran this through the HuggingFace space using the default parameters, ChatGPT through chat.openai.com, and Codeium through the VSCodium extension on an empty JavaScript file.
- amasad 4y ago
  Interesting. This seems like a weakness of natural language understanding. If you rephrase your prompt slightly it would get it right. Try: // return even numbers that are also more than 10 const arrayFilter = (array) => It would do the right thing. The fine-tuned version gets your prompt right so maybe it benefited from natural language data. Will look more into it.
  - doodlesdev 4y ago
    That's really interesting, indeed I can reproduce this by changing the comment. I also managed to get correct output for this sample by renaming the function.
    - eevilspock 4y ago
      clearly your original comment was unfair.
      - 4y ago
user3939382 4y ago
Unfortunately I'm someone who sometimes can't separate the art from the artist. Replit is the company where the founder sent these nasty pompous threats to their ex-employee for their innocent side project and then tried to double talk his way out of it with a bs non-apology when it got exposed in public. I won't support Replit or anything they make.
davidy123 4y ago
I keep thinking there should be a way to train a copilot against just one set of code libraries. I know LLMs require training against a lot of text to get their smarts, but is there a way to set this up so a model can be created for a specific library by anyone, so it could provide open source support via a transformer + model? Maybe this would be a better approach than a jack of all trades, master of none.
- nl 4y ago
  Yes, this is what fine tuning is for. It's pretty obvious that lots of people will want to take a strong code completion model, then fine tune it on their docs + libraries and then make it available inside their docs/discord/slack as a support thing.
  - davidy123 4y ago
    I guess as soon as a kit is available for this purpose that doesn't require advanced knowledge (say, `aisupport4 my-repo/`) and runs on mainstream-ish hardware and doesn't require a centralized service (even running in the browser via eg transformers.js), things will change considerably.
    - nl 4y ago
      As someone who is very interested in decentralized services (as in my day job involves decentralized databases and I'm actively working on WebGPU support for training) I'd say that the browser-based vision is a fair way off. The software ecosystem is pretty immature, and there are numerous things that need to change before the core technologies are good enough to fine tune competitive LLMs. I do think fine tuning moderate sized LLMs on your own (pretty expensive) hardware using consumer GPUs maybe possible this year. Unfortunately all the evidence is that training (as opposed to inference) requires high-precision, and hence high memory. This is something that consumer GPUs for the most part lack. New techniques are likely to be required (eg better sharing of training on low memory GPUs) but it's hard to predict how they will develop.
chaxor 4y ago
This is a bit hard to believe that the system is decent at producing code which captures complex ideas and higher level structure when the tokens/param value is >30 (it's ~200 here? ) The 'good' models (meaning having lots of 'knowledge' or 'memorization' about the dataset) typically tend to be around 2 tokens/param and models with decent generation of language with less knowledge/memorization are around 30 tokens/param. Perhaps the domain allows for this, but due to the fact that the linguistic interface on the input is still needed... It's hard to believe.
- swyx 4y ago
  this kind of critical thinking is exactly what replit is going to need for their stated goal of doing whole-app generation. right now they only test it on AmjadEval. you… might wanna consider joining them to work on it?
  - chaxor 4y ago
    I'm not sure noticing tokens/params or simplicial modeling properties occur requires much critical thought - perhaps it's just a standard first thought for anyone given an LM now. I've worked tangentially to NLP for about 7 years in academia, but most of my work has been focused on a less known field of mathematics applied to either to outputs or to the NNs, as well as bioinformatics. As such, my expertise may not be as refined as the real players in the field such as Glaese, Finn, Velockovic, etc. but I try typically keep up with the actual key advancements in the field (usually the stuff few people notice). This area takes far too much compute capability for many people to actually become experts in it, so much of my interests have been less on large LMs and more towards algorithms. But I suppose I agree that it is frustrating to see how little knowledge many of the hype-filled crowds possess that are piling into this area. (Not calling anyone specifically out in this thread, just in general across the internet)
- EvgeniyZh 4y ago
  Are you saying the less you train the model the better it is? I'm confused
hinkley 4y ago
I think that 20 years from now, we'll all be sitting around wondering 1) where the fuck are my flying cars, and 2) what were they thinking using computers to write code? And the reason I say this is because these tools are answering a question that we haven't asked yet: what common problems need to be solved in this programming language, and where do I get code to solve that problem? These LLM modules are basically telling us how to duplicate code, and what we need is the opposite: how to stop reinventing the wheel for the 100th time. Instead of writing code for me, tell me if I already have it. If I'm writing it, tell me there's a library for that. If I'm a library writer, give me suggestions for what libraries are missing from the toolkit. All we've done so far is begun the process of automating the production of duplicate code. With absolutely no way to go back in time and correct bugs introduced in earlier iterations. We are likely, for instance, to see 0 day attacks that affect hundreds of applications, but with no simple way to describe which applications are affected. That's going to be a first rate trainwreck.
- seydor 4y ago
  > how to stop reinventing the wheel for the 100th time. The idea of libraries may not have been a good one. It saved human time but no library is perfect because no abstraction is perfect and this causes unnecessary bloat. It seems tha Nature does not use libraries, it uses replication instead, and we can now have that too.
  - sicariusnoctis 4y ago
    Replication does not help in managing completely. That's why we use abstractions, even with the problems they have.
  - x-shadowban 4y ago
    Ha I never wondered what the physical/life version of a shared library is until I read your post so thanks for that.
  - hinkley 4y ago
    So instead everyone who has to solve a problem has to be an expert on that problem, rather than just an informed consumer.
lee101 4y ago
[dead]
dvt 4y ago
I genuinely don't understand how anyone can use something like this and seriously think "oh yeah, this is revolutionary." It's almost complete garbage and can't do anything remotely interesting. # a method that approximates the hyperbolic tangent (clamped tanh) def rational_tanh(x): return (x + 1) / (x - 1) Even gave it the BIG hint of a "clamped" and "rational" tanh, but that ain't it, chief. Forget GPT-4, I would be embarrassed to even show this as a tech demo.
- afro88 4y ago
  I use "something like this" (GPT4) all the time. Use a good model. Can't wait till proper open source models catch up, but they're not there yet. Here's GPT4's response: ``` import math def clamped_tanh(x, n_terms=10): """ Approximate the hyperbolic tangent (tanh) function using a Maclaurin series expansion. Args: x (float): The input value for which to compute the tanh. n_terms (int, optional): The number of terms to use in the Maclaurin series. Default is 10. Returns: float: The approximated tanh value. """ tanh_approx = 0 for n in range(n_terms): coef = ((-1) ** n) * (2 * n + 1) term = coef * (x ** (2 * n + 1)) / math.factorial(2 * n + 1) tanh_approx += term # Clamping the tanh approximation to the range [-1, 1] tanh_approx = max(-1, min(tanh_approx, 1)) return tanh_approx # Example usage x = 0.5 result = clamped_tanh(x) print(f"clamped_tanh({x}) = {result}") ```
  - dvt 4y ago
    I don't mean to get in the weeds here, but this is still bad, as the Padé approximation[1] (which is what you're actually really looking for) is magnitudes better than the Maclaurin series (which is just a special case of the Taylor expansion). Keep in mind that I'm not even an expert (I merely earned a minor in math). In fact, I only know this because I did some research years ago on rational approximations of hyperbolic functions. The right answer gets clipped to -1 or 1 beyond (-3, 3), but it should look like: `x * ( 27 + x * x ) / ( 27 + 9 * x * x )`—though the coefficients (and clip range) can vary. [1] https://en.wikipedia.org/wiki/Pad%C3%A9_approximant https://en.wikipedia.org/wiki/Pad%C3%A9_approximant
deleted 4y ago
[deleted]
markeibes 4y ago
I'm always sad to see these things being trained on a tiny number of programming languages. Makes it harder still for the good languages to compete.
abdellah123 4y ago
"1M concurrent containers" curious about replit containers. Do they run on firecracker?
foverzar 4y ago
> Markdown, Java, JavaScript, Python, TypeScript, PHP, SQL, JSX, reStructuredText, Rust, C, CSS, Go, C++, HTML, Vue, Ruby, Jupyter Notebook, R, Shell No Kotlin T_T I wonder if fine-tuning to a new language would even make sense. AFAIK, it is the core knowledge within the model that really matters, finetuning is essentially specialising.
throwaway290 4y ago
> It's been trained on 525 billion tokens of, of code all permissively licensed code What does "permissively licensed" mean?

891 points by swyx 4y ago

jeremypress 4y ago

Interesting how this guy has a finance background but knows how to code, especially for emerging technologies like LLMs

ipsum2 4y ago
Didn't MosaicML do the training for them?

amasad 4y ago

Some links: - Repo: https://github.com/replit/ReplitLM/tree/main/replit-code-v1-3b https://github.com/replit/ReplitLM/tree/main/replit-code-v1-... - HuggingFace: https://huggingface.co/replit/replit-code-v1-3b https://huggingface.co/replit/replit-code-v1-3b - Demo: https://huggingface.co/spaces/replit/replit-code-v1-3b-demo https://huggingface.co/spaces/replit/replit-code-v1-3b-demo - Early benchmark results: https://twitter.com/amasad/status/1651019556423598081 https://twitter.com/amasad/status/1651019556423598081 A lot about this project was surprising. We knew it was going to be good, but didn't expect to be this good -- especially surprising was the finetuned performance boost, and the fact that the model is decent at language tasks and reasoning (in some cases much better than much larger general-purpose models). It feels like there is a lot more to do with this model, and I have a suspicion you can even make a half-decent chatbot (at least one focused on code) by finetuning it on conversation (and/or instruction) datasets. Will follow up with a more comprehensive technical report and the UL2R version (fill-in-the-middle support).

pera 4y ago
Hi there, I have two question: 1 - Why did you choose Markdown? It seems an odd choice for training a model like this. 2 - Have you tried to train only one single PL and then benchmark it against this more general version?
- amasad 4y ago
  1- We trained on languages that are most popular on Replit. Markdown is important because you need some amount of natural language in the data, and it will act as a sort of "natural language label" for code. 2- I like how portable it is being a single small model doing a lot of languages. Single code models are an approach that models like Salesforce/Codegen did that, but I believe we beat (or get very close) to their mono models on benchmarks.
  - fuzzythinker 4y ago
    Have you thought of finding or creating something like this [0]? I created this as the basis for my origami folding descriptive language. I tried to find something similar, requirements being both well structured and English-like but couldn't find any, so I created it. The origami folding app will hopefully be out in 2 weeks, so you can see how it's used. [0] https://github.com/fuzzthink/mation-spec https://github.com/fuzzthink/mation-spec
- runnerup 4y ago
  They trained on https://huggingface.co/datasets/bigcode/the-stack-dedup https://huggingface.co/datasets/bigcode/the-stack-dedup which is a massive curated dataset accumulated from GitHub. Details are here: https://www.bigcode-project.org/docs/about/the-stack/ https://www.bigcode-project.org/docs/about/the-stack/ Many of the most-represented "languages" on GitHub are actually things like JSON, XML, HTML, CSV, text, markdown, YAML, and SVG. More details from them here: https://blog.replit.com/llm-training https://blog.replit.com/llm-training
gbasin 4y ago
Very exciting, thanks for sharing all this
letitgo12345 4y ago
Doesn't the Stack contain HumanEval? So you're basically comparing numbers on the pretraining data.
- amasad 4y ago
  Can't find it now but pretty sure BigCode said somewhere they explicitly looked for it and removed it. Also subjective measure does match up to the benchmark. Our finetuned model performed +50% on HumanEval and then when using it felt at least that much improved.
  - godelski 4y ago
    You can view the prompts, solutions, and checks here[0]. See my sibling comment (to yours) where I quote the Human Eval paper and do some more analysis. But I think if you look at [0] you'll see that these aren't really unique problems and are likely to have large repetitions in the dataset. I should add to that comment to include the dataset[1] (too late to edit) where they mention that they just scrape all of GitHub (Jan 1 2015 - Mar 31 2022). They do exact and near de-duplicate but near de-duplication is messy. > We implement near-deduplication in our pre-processing pipeline on top of exact deduplication. We first split the files into words/tokens based on non-alphanumeric characters and remove files with fewer than 10 tokens. Next, we compute the MinHash with 256 permutations of all documents, and use Locality Sensitive Hashing to find clusters of duplicates. We further reduce these clusters by ensuring that each file in the original cluster is similar to at least one other file in the reduced cluster. We consider two files similar when their Jaccard similarity exceeds 0.85. Near-duplicates are still difficult to measure. So we should expect duplication, and it should be proportional to the number of samples we have (even if the same variance, but I'd wager higher variance with larger duplications). [0] https://github.com/openai/code-align-evals-data/tree/97446d992c3785d6605f1500b2c9b95d042e7b9c/human_eval https://github.com/openai/code-align-evals-data/tree/97446d9... [1] https://arxiv.org/abs/2211.15533 https://arxiv.org/abs/2211.15533
- godelski 4y ago
  My favorite line from the HumanEval paper[0] > It is important for these tasks to be hand-written, since our models are trained on a large fraction of GitHub, which already contains solutions to problems from a variety of sources. So to answer your question, yes, the evaluation dataset is spoiled. You can find such unique and never before seen docstrings like > For a given list of input numbers calculate the Mean Absolute Deviation around the mean of this dataset. Mean Absolute Deviation is the absolute difference between each element and a centerpoint (mean in this case)[1] And here's a repo I found that is 8 years old[2]. But how about a more recent one that is even closer?[3] There's plenty more examples[4] (does anyone know how actually limit the date to prior to 2021? `pushed:<2021` doesn't work nor does using the `created` keyword. Date searching doesn't seem to work well). In essence, we can still use this evaluation method to determine how good our model is at doing fuzzy searching. Which, mind you, is still a useful thing. But I would be careful in concluding that this means the model is good at generalizing arbitrary descriptions of code or novel pieces of code. That said, one may be able to argue that not many lines of code are actually that novel. Still, we need to be careful about our conclusions and understand the limitations of our metrics (something I am currently deeply troubled by) [0] https://arxiv.org/abs/2107.03374 https://arxiv.org/abs/2107.03374 [1] https://github.com/openai/code-align-evals-data/blob/97446d992c3785d6605f1500b2c9b95d042e7b9c/human_eval/floats_mean_absolute_deviation.py#L6 https://github.com/openai/code-align-evals-data/blob/97446d9... [2] https://github.com/bertomartin/stat4701/blob/ec2b64f629cbbf6267169302265f73a98edef67d/stck.py#L175 https://github.com/bertomartin/stat4701/blob/ec2b64f629cbbf6... [3] https://github.com/danielwatson6/hate-speech-project/blob/64a2eecce5218373ef5c449eeb1dfb397532eda5/scripts/wordnet_mf.py#L61 https://github.com/danielwatson6/hate-speech-project/blob/64... [4] https://github.com/search?q=abs%28x+-+mean%29+for+language%3APython&type=code https://github.com/search?q=abs%28x+-+mean%29+for+language%3...
newhouseb 4y ago
First - thank you for open sourcing this! It's a real gift to the community to have a model intended for "commercial use" that's actually licensed as such. I'd be very interested to hear about the choice/evaluation of the ALiBi approach for positional embedding (perhaps in the technical report). My intuition suggests that while this allows for better generalizability for longer sequence lengths, it penalizes scenarios where an LLM might need to check for things like a function signature far away from where the next token is generated. My initial testing of this model tracks with this intuition but that's by no means a rigorous evaluation.
- ofirpress 4y ago
  (I wrote ALiBi) You can read the paper here https://arxiv.org/abs/2108.12409 https://arxiv.org/abs/2108.12409 While intuitively it does seem like ALiBi would make it hard for the model to attend to things that are far away, in many scenarios we've tested with different models trained on different datasets, ALiBi always performs better than sinusoidal, rotary, and other embedding types, even when we're not using it to extrapolate to longer sequence lengths. These findings have been confirmed by others, including by the BLOOM open source LM project.
  - newhouseb 4y ago
    Small world! Thanks for the link (which I've now skimmed beyond the abstract). What wasn't obvious to me from the abstract is that different attention heads have different penalty strengths, so if some prediction task requires long range dependencies you might expect one of the less-penalized heads to end up specializing. I wonder what would happen if the penalty for one head is zero? (The paper suggests this might've been tried and just made things worse, but unclear) I must admit that this is a wonderfully elegant (and interpretable) way to do this... much more intuitive (to me at least, a wannabe practitioner) than all of the trig-based embeddings.
    - ofirpress 4y ago
      > so if some prediction task requires long range dependencies you might expect one of the less-penalized heads to end up specializing Exactly. You have heads that focus on content nearby and ones that focus on stuff that is far away. > I wonder what would happen if the penalty for one head is zero? (The paper suggests this might've been tried and just made things worse, but unclear) Yup, this is something we tried. Making one of the heads zero doesn't improve or degrade performance. >I must admit that this is a wonderfully elegant (and interpretable) way to do this... much more intuitive (to me at least, a wannabe practitioner) than all of the trig-based embeddings. Thanks so much!!
sputknick 4y ago
What does "fine tuning" mean in this context? Does it mean you fine-tuned it on a specific code repository, or collection of code repositories and then had it do work in those repositories?
- amasad 4y ago
  Broadly finetuning is any post pretraining training. Most of the time it is used in the context of fitting a more narrow task. In our case, it was the same training objective as the pretraining but meant to be more representative of what Replit users like to code. However, we were surprised by how well it boosted overall performance. Best guess: it's a) novel data and b) the model could take even more training!!
  - spenczar5 4y ago
    How feasible and effective would it be to fine-tune a model against an organization's private source code, resulting in an "internal" model that knows how to work with that org's stuff? Could you, say, fine-tune the model every week with the latest merges? Every hour?
    - pyth0 4y ago
      Finetuning is a relatively quick process. Training the base model is the expensive part (can take weeks and huge amounts of compute), whereas finetuning usually is only on the last few layers and can be done with much less resources. You could definitely have a "nightly" finetune model that is retrained every day or so.
      - rattray 4y ago
        Interesting - how would that work for a company that wanted to run their own codex model, on-prem, trained on their own code? Perhaps also trained on their dependencies?
    - naderkhalil 4y ago
      Finetuning a smaller model leading to better performance seems like a significant finding that'll lead to a lot of companies fine-tuning their own internal "ChatGPT"s
spenczar5 4y ago
How is this code licensed? I didn't see a license in the repo. It looks interesting!
- dgacmu 4y ago
  The README indicates: The base model checkpoint is licensed under the Creative Commons license (CC BY-SA-4.0). Under the license, you must give credit to Replit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests that Replit endorses you or your use.
kir-gadjello 4y ago
Impressive model, thank you for releasing it under a business-friendly license! Have you considered using Google's sparse "scaling transformer" architecture as the base? Even at 3B scale it can generate 3-4x more tokens per FLOP while being competitive at perplexity with a dense transformer. I think OpenAI uses a variant of it in their ChatGPT-3.5-Turbo product. Here is the paper https://arxiv.org/abs/2111.12763 https://arxiv.org/abs/2111.12763 and the implementation https://github.com/google/trax/blob/master/trax/models/research/terraformer.py https://github.com/google/trax/blob/master/trax/models/resea... if you are interested. Hope you get to look into this!
- b33j0r 4y ago
  Thank you for releasing the weights along with the announcement. The posts that made great headlines, but “weights are on their way!” Like why did we even get excited? This? Great work.
- chaxor 4y ago
  I don't think it's a business friendly license?
  - kir-gadjello 4y ago
    It allows for modifications and commercial use: https://creativecommons.org/licenses/by-sa/4.0/ https://creativecommons.org/licenses/by-sa/4.0/ >You are free to: >Share — copy and redistribute the material in any medium or format >Adapt — remix, transform, and build upon the material >for any purpose, even commercially. Compare this to the latest release from StabilityAI lab DeepFloyd, "IF", which in addition to various restrictive clauses strictly prohibits commercial use: https://github.com/deep-floyd/IF/blob/develop/LICENSE-MODEL https://github.com/deep-floyd/IF/blob/develop/LICENSE-MODEL Repl.it's release is as open as it gets these days, in my book.
    - LukeShu 4y ago
      It's a copyleft license; and lots of folks on HN seem to think that copyleft, while being open, isn't business friendly.
curiousgal 4y ago
Did any interns help in developing this? If so are you planning on intimidating them as usual? :) Reference: How Replit used legal threats to kill my open-source project https://intuitiveexplanations.com/tech/replit/ https://intuitiveexplanations.com/tech/replit/
- robertlagrant 4y ago
  Wow. That's extremely poor behaviour if the account is accurate.
  - curiousgal 4y ago
    Oh it is. 4000+ upvotes on HN. https://news.ycombinator.com/item?id=27424195 https://news.ycombinator.com/item?id=27424195

swyx 4y ago

hi HN! back again with an exclusive deep dive with Replit’s head of AI. I attended their developer day last week (https://twitter.com/swyx/status/1650989632413401089 https://twitter.com/swyx/status/1650989632413401089) just expecting a regular fundraise announcement and was totally shocked when they annoucned their own LLM and also said they would open source it. so immediately asked them for a podcast interview and this is the result. my favorite learning is how they are pushing the state of the art - openai’s HumanEval is the industry standard benchmark for code LLMs, but Reza kindly went above and beyond to show how they use “AmjadEval” - using coder intuition to capture human preference on what output is more helpful to coders (see screenshots https://twitter.com/swyx/status/1653791019421569024?s=20 https://twitter.com/swyx/status/1653791019421569024?s=20) please AMA!

marcodiego 4y ago
Sorry, I have to ask this: how does this compare to ChatGPT?
- swyx 4y ago
  it doesn't. replit-code-v1-3b is a code LLM, ChatGPT is an app on top of LLMs. it compares to OpenAI Codex, a small version of which is behind GitHub Copilot.
  - mritchie712 4y ago
    It (replit-code-v1-3b) is already quite good at explaining code: Input: below is a SQL statement: SELECT CAST(DATE_TRUNC('week', "t1"."TIMESTAMP") AS DATE) AS "WEEK_START", COUNT(\*) AS "EVENT_COUNT" FROM "ANALYTICS"."POSTHOG"."POSTHOG_EVENTS" AS "t1" GROUP BY "WEEK_START" ORDER BY "WEEK_START" LIMIT 2000 Explain this SQL. Respond in JSON format with the following keys: TITLE, DESCRIPTION, TABLES JSON response: output: { "title": "Weekly Events Count", "description": "Count of weekly events", "tables": [ { "name": "POSTHOG_EVENTS", "columns": [ "WEEK_START", "EVENT_COUNT" ] } ] }

tarruda 4y ago

3 billion parameters. Does that mean I will be able to run on a 8gb consumer GPU?

generalizations 4y ago
Means that once it's incorporated into llama.cpp, you can run it on your laptop.
- amasad 4y ago
  Hopefully on phones too
RHab 4y ago
No, I could only get 2.7B to run on 8GB VRam unfortunatly.
- amasad 4y ago
  it is 2.7B
  - tarruda 4y ago
    Loading seems to have worked on my laptop's RTX 3070, `nvidia-smi` shows `5188MiB / 8192MiB` in memory usage.
pera 4y ago
their pytorch_model.bin is 10.4GB
- tarruda 4y ago
  I just loaded this on my laptop's RTX 3070 GPU by following the instructions here: https://huggingface.co/replit/replit-code-v1-3b https://huggingface.co/replit/replit-code-v1-3b I don't know how I can test the model, but it seem loading worked. When I run `nvidia-smi` on another terminal, I see `5188MiB / 8192MiB` in the memory-usage column.

tyingq 4y ago

More tools in the field is great! I tried a few things, and it's reasonable, but it does have some quirks that seem to repeat, like: I tried a prompt of: # python function that returns a random integer between min and max And it produced: def random_int(min, max): return random.randint(min, max) # define the size of the grid n = 5 It doesn't add the needed import statement, and I'm unclear why it's "defining the size of the grid".

circuit10 4y ago
That's because it's not following instructions like ChatGPT, it's just trying to guess that could plausibly come after what you put, like Copilot or the old GPT-3 models
- minkzilla 4y ago
  and imports are (almost) always at the top of your file not with this function
  - vharuck 4y ago
    I tried the same input, except wrapping it in triple-quotes instead of commenting it. So that it would match the standard practice for module doc strings. Here's the result: """python function that returns a random integer between min and max""" return random.randint(min, max) def gen_random_float(min, max): """python function that returns a random float between min and max""" return random.uniform( So, it assumed the triple-quote was a function's doc string, despite it not being indented. It then assumes I'll want a similar function for floats (I assume it was cut off by a token limit).
- jeremyjh 4y ago
  Isn’t ChatGPT also just generating plausible text that could be a response to an instruction?
  - whimsicalism 4y ago

fswd 4y ago

I can barely keep up with this stuff, but quick question. Is there a way to simply change the URL setting of copilot to point to this model? Obviously it needs an endpoint, I could hack something up, but asking if somebody has already done this? Would be nice to cancel my copilot.

jacobrussell 4y ago
I don't think it's possible to point Copilot to other models. I don't think Microsoft would benefit much from that feature. You could use existing tools [0] to host your own model which in theory could be used by an extension your IDE uses. But I'm not sure if an extension like that exists. [0] https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui
- circuit10 4y ago
  Of course it's possible, just not officially See https://github.com/fauxpilot/fauxpilot/blob/main/documentation/client.md#copilot-plugin https://github.com/fauxpilot/fauxpilot/blob/main/documentati...
circuit10 4y ago
There's https://github.com/fauxpilot/fauxpilot https://github.com/fauxpilot/fauxpilot but it doesn't use this model
execveat 4y ago
It's nowhere close to Codex/Copilot. Try the demo: https://huggingface.co/spaces/replit/replit-code-v1-3b-demo https://huggingface.co/spaces/replit/replit-code-v1-3b-demo
- tarruda 4y ago
  So they are lying on this tweet? https://twitter.com/Replit/status/1651344186715803648 https://twitter.com/Replit/status/1651344186715803648

waffletower 4y ago

No Clojure. No Julia. No Haskell. No Racket. No Scheme. No Common Lisp. No OCaml. And, as much as I despise Microsoft, No C#. No F#. No Swift. No Objective-C. No Perl. No Datalog. A glaringly lacking choice of languages.

ebiester 4y ago
I'm sure that has to do with the dataset available to them.
- runnerup 4y ago
  Which is a deduplicated version of this: https://www.bigcode-project.org/docs/about/the-stack/ https://www.bigcode-project.org/docs/about/the-stack/ And probably, yes. While it contains 358 programming languages, obviously there's a long tail after the 20 most-represented languages. Some people might not expect without thinking about it for a bit that many of the most-represented "languages" are actually things like JSON, XML, HTML, CSV, text, markdown, YAML, SVG. Also note that it won't be able to parse natural language nearly as well without additionally being trained on something like the LAION dataset, so this version will be more of an autocomplete like Copilot rather than something which can manifest high level business logic from whole cloth like ChatGPT.
Dayshine 4y ago
C# was available in the dataset they link, and is the most glaring ommission by global usage...
ubertaco 4y ago
I fed it some OCaml and it worked, though the example was trivial: type point = { x: int; y : int } let manhattan_distance (a: point) (b: point) : int = which it completed to type point = { x: int; y : int } let manhattan_distance (a: point) (b: point) : int = abs (a.x - b.x) + abs (a.y - b.y) ...which is a valid and correct OCaml definition of this method: https://try.ocamlpro.com/#code/type'point'='$4'x:'int;'y':'int'$6!let'manhattan_distance'(a:'point)'(b:'point)':'int'=!abs'(a.x'-'b.x)'+'abs'(a.y'-'b.y)!;;!!let'origin'='$4x'='0;'y'='0$6'in!let'dest'='$4x'='2;'y'='3$6'in!Printf.printf'$($+d$('(manhattan_distance'origin'dest) https://try.ocamlpro.com/#code/type'point'='$4'x:'int;'y':'i...

circuit10 4y ago

This probably makes a self-hosted and/or local Copilot a lot more feasible

FanaHOVA 4y ago
Yes, something like FauxPilot[0] should be able to use it instead of CodeGen [0] https://github.com/fauxpilot/fauxpilot https://github.com/fauxpilot/fauxpilot

GreedClarifies 4y ago

This is amazing work and bravo on to the people working on redpajama. This is fantastic for the world, this means LLMs will not be controlled by a couple of companies with the associated rents. Yes, private LLMs will likely be a couple of years ahead of 'free' alternatives, but that's OK, we want to incentivize for profit research so long as the services are low priced in time (and in this case in short order). AMAZING WORK.

swyx 4y ago
to be clear this work is not based on redpajama - though we did discuss that in the previous episode https://twitter.com/swyx/status/1648080532734087168?s=46&t=90xQ8sGy63D2OtiaoGJuww https://twitter.com/swyx/status/1648080532734087168?s=46&t=9...
- GreedClarifies 4y ago
  Oh my bad! I thought I read that, is it based upon: https://arxiv.org/abs/2211.15533 https://arxiv.org/abs/2211.15533 (The Stack) ?
  - swyx 4y ago
    partially. Reza discussed their data pipeline in the blogpost that we reference in the show notes
m3kw9 4y ago
Have you even tried it? It’s pretty bad
- GreedClarifies 4y ago
  But that's fine it can be a year or two behind the state of the art. That's not the point. The point is that there will be alternatives and that will reduce the price in time further increasing the impact of the technology. There was a possible future where only MSFT and maybe GOOG and maybe one or two other companies had this technology and extracted massive rents.

love2read 4y ago

Can this be used with the copilot plugins for every ide?

protonbob 4y ago

Darn it doesn't look like it has c sharp.

abxytg 4y ago

[flagged]

LightMachine 4y ago

Any idea how much it cost to train it and how it was trained?

Imnimo 4y ago

Tried it out on the HuggingFace demo, with default settings. Prompt: >def nth_prime(n): Completion: > if n == 1: > return 2 > if n == 2: > return 3 > if n == 3: > return 5 > if n == 4

robertlucas 4y ago

Is there any way to connect these new code focused LLMs into VS Code in order to replace Github Copilot?

vrglvrglvrgl 4y ago

[dead]

sashank_1509 4y ago

Replit: I tried to write HMM Forward function code: ``` # Write a function that performs the forward algorithm of an HMM. Ideally it uses numpy vectorization, # But if you cannot do that it is fine too. Just perform forward inference, given HMM parameters and Observations. # def HMM_forward() : # Write your code here # Hint: # - Remember that HMMs are stochastic, so you will need to multiply transition matrices by observation probabilities # - Remember that HMMs are stochastic, so you will need to multiply emission matrices by observation probabilities # - Remember that HMMs are stochastic, so you will need to multiply emission matrices by transition matrices # - Remember that HMMs are stochastic, so you will need to multiply emission matrices by transition matrices # - Remember that HMMs are stochastic, so ``` It gets stuck in an infinite loop. ChatGPT gets it write, first shot: ``` import numpy as np def forward_algorithm(A, B, pi, obs): """ Performs the forward algorithm for an HMM. Parameters: A: numpy.ndarray, shape (N, N) Transition matrix of the HMM, where N is the number of states. B: numpy.ndarray, shape (N, M) Emission matrix of the HMM, where M is the number of possible observations. pi: numpy.ndarray, shape (N,) Initial probability distribution over states. obs: numpy.ndarray, shape (T,) Sequence of T observations. Returns: alpha: numpy.ndarray, shape (T, N) Forward probabilities for each state at each time step. """ T = obs.shape[0] N = A.shape[0] alpha = np.zeros((T, N)) alpha[0] = pi * B[:, obs[0]] for t in range(1, T): alpha[t] = np.dot(alpha[t-1], A) * B[:, obs[t]] return alpha ``` OpenAI managed to do the important but extremely hard, they moved out of the DL benchmark frame and made something that is general purpose useful. Great effort and congrats to Replit team though, hopefully they can keep iterating on this and reach ChatGPT capabilities someday

amasad 4y ago
The model is not RLHF'd or instructed. It's an inline autocomplete model so it will get confused if you talk it like you're talking to a person. Altho it is possible to finetune it this way. To get better full function completion try giving it the function definition and a descriptive docstring as a prompt.
- deleted 4y ago

m3kw9 4y ago

It just gave me prototypes lol def sieve_eratosthenes(n): ##a function to sort 10 numbers def bubble_sort(a): ##a function to sort 10 numbers def insertion_sort(a): ##a function to sort 10 numbers def quick_sort(a):

m3kw9 4y ago
I left the settings. All I added was ##a function to sort 10 numbers. Assuming it would complete it like copilot
ImprobableTruth 4y ago
Did you mess around with the settings? I'm getting a correct implementation and since it's deterministic (with default settings) it should be the same for you.

robby_w_g 4y ago

I recognized the name Replit and couldn't remember why. A quick search reminded me: https://news.ycombinator.com/item?id=27424195 https://news.ycombinator.com/item?id=27424195

stephenjayakar 4y ago
this feels like an attempt to hive mind against anything cool from this company
- dimgl 4y ago
  +1, this is unnecessary.
  - aardshark 4y ago
    Alternatively, it's called consequences of your actions. Don't be surprised if shitty behaviour comes back to bite you.
- ibrarmalik 4y ago
  I think people are smart enough to receive extra information and do whatever they want with that.
- naillo 4y ago
  Threatening a guy for making an open source version of replit sounds pretty crummy in my eyes.
- robby_w_g 4y ago
  I think it's fair to evaluate a company's behavior before engaging in business with them. And I personally dislike persons in power abusing their position, which is why I remembered the company name almost two years later. I haven't heard of any similar behavior since then, which is a good sign. But a reputation can be a hard thing to shake. The CEO should have considered that before doing what he did.
naillo 4y ago

youssefabdelm 4y ago

title is missing: "trained in 1 week, and like most open source LLMs so far... it sucks compared to the closed source alternatives" Great effort of course bla bla bla... Open source really needs some benchmarking, and up their game quality-wise. And yes I know they're expensive as shit to train... let's not keep wasting our money and actually work together, pool our resources, to make a GOOD model. But oh no, everyone wants to put their stamp on it. "Replit did this! Look at us!"

ImprobableTruth 4y ago
This is easy to say, but I think the issue is that getting an LLM right isn't easy, so it's not clear who should steward such a project. Something like BLOOM shows that even if you have the necessary compute, you can still get a model that isn't good. I think it will take some time for it to be clear who is a leader in training open source models (maybe it will be the red pajama folks?) and I think they'll get more support after that.
- youssefabdelm 4y ago
  Fair point

heliophobicdude 4y ago

Important distinction that I'm learning today is that not all LLMs will be interoperable with each others' queries/prompts/inputs. Code LLM right now is not responding how a Chat LLM would respond. ~~~~~ Hats off to the team on the impressive work!

eikenberry 4y ago

Is this a Co-pilot like assistant or something more? Co-pilot is neat but is basically not much more than an automated snippet system. The actual writing of the code is not the part that I want help with, I want an AI system that helps me design better software systems. Something more akin to program mind mapping than some fancy auto-completion system.

qrio2 4y ago
I wonder if LLM with something like plantUML would generate anything useful

gowld 4y ago

Can I use repl.it with an external Code LLM, with or without paying repl.it for Ghostwriter ?

amasad 4y ago
Yes we have a robust extension system and some are already building alternatives.
- varunkmohan 4y ago
  Hi from the Codeium team. It's awesome to hear you are allowing other code LLMs to be used on the Replit platform (we're big fans)! We'd love to enable our free chrome extension on Replit.
  - swyx 4y ago
    would love to be able to compare codeium vs ghostwriter inside replit! (or toggle between them based on known strengths or preferences, perhaps by project or by filetype)

SXX 4y ago

Weak spot which I guess similar to other LLMs. If you mention recursion somewhere in comments model sometimes start to recursively generate the same lines over and over again.

doodlesdev 4y ago

The model is way too small, comparing it to Codex feels disingenous. Sure it's 77% smaller, it's also 77% worse. Although, it's a cool project nonetheless. For instance, even this simple snippet generates wrong inline completions: // Only return even numbers bigger than 10 from the array const arrayFilter = (array) => Replit-code-v1: // Only return even numbers bigger than 10 from the array const arrayFilter = (array) => { return array.filter((item) => item > 10); }; Gets it wrong, returns odd numbers. Codeium: // Only return even numbers bigger than 10 from the array const arrayFilter = (array) => { return array.filter((num) => num > 10 && num % 2 === 0); }; ChatGPT (GPT-3.5 Turbo) - Code-only, without the rest of the completion since it's instruction-tuned: const arrayFilter = (array) => { return array.filter(num => num % 2 === 0 && num > 10); } Not comparable at all. For reference if anyone wants to test I ran this through the HuggingFace space using the default parameters, ChatGPT through chat.openai.com, and Codeium through the VSCodium extension on an empty JavaScript file.

amasad 4y ago
Interesting. This seems like a weakness of natural language understanding. If you rephrase your prompt slightly it would get it right. Try: // return even numbers that are also more than 10 const arrayFilter = (array) => It would do the right thing. The fine-tuned version gets your prompt right so maybe it benefited from natural language data. Will look more into it.
- doodlesdev 4y ago
  That's really interesting, indeed I can reproduce this by changing the comment. I also managed to get correct output for this sample by renaming the function.
  - eevilspock 4y ago
    clearly your original comment was unfair.
    - 4y ago

user3939382 4y ago

Unfortunately I'm someone who sometimes can't separate the art from the artist. Replit is the company where the founder sent these nasty pompous threats to their ex-employee for their innocent side project and then tried to double talk his way out of it with a bs non-apology when it got exposed in public. I won't support Replit or anything they make.

davidy123 4y ago

I keep thinking there should be a way to train a copilot against just one set of code libraries. I know LLMs require training against a lot of text to get their smarts, but is there a way to set this up so a model can be created for a specific library by anyone, so it could provide open source support via a transformer + model? Maybe this would be a better approach than a jack of all trades, master of none.

nl 4y ago
Yes, this is what fine tuning is for. It's pretty obvious that lots of people will want to take a strong code completion model, then fine tune it on their docs + libraries and then make it available inside their docs/discord/slack as a support thing.
- davidy123 4y ago
  I guess as soon as a kit is available for this purpose that doesn't require advanced knowledge (say, `aisupport4 my-repo/`) and runs on mainstream-ish hardware and doesn't require a centralized service (even running in the browser via eg transformers.js), things will change considerably.
  - nl 4y ago
    As someone who is very interested in decentralized services (as in my day job involves decentralized databases and I'm actively working on WebGPU support for training) I'd say that the browser-based vision is a fair way off. The software ecosystem is pretty immature, and there are numerous things that need to change before the core technologies are good enough to fine tune competitive LLMs. I do think fine tuning moderate sized LLMs on your own (pretty expensive) hardware using consumer GPUs maybe possible this year. Unfortunately all the evidence is that training (as opposed to inference) requires high-precision, and hence high memory. This is something that consumer GPUs for the most part lack. New techniques are likely to be required (eg better sharing of training on low memory GPUs) but it's hard to predict how they will develop.

chaxor 4y ago

This is a bit hard to believe that the system is decent at producing code which captures complex ideas and higher level structure when the tokens/param value is >30 (it's ~200 here? ) The 'good' models (meaning having lots of 'knowledge' or 'memorization' about the dataset) typically tend to be around 2 tokens/param and models with decent generation of language with less knowledge/memorization are around 30 tokens/param. Perhaps the domain allows for this, but due to the fact that the linguistic interface on the input is still needed... It's hard to believe.

swyx 4y ago
this kind of critical thinking is exactly what replit is going to need for their stated goal of doing whole-app generation. right now they only test it on AmjadEval. you… might wanna consider joining them to work on it?
- chaxor 4y ago
  I'm not sure noticing tokens/params or simplicial modeling properties occur requires much critical thought - perhaps it's just a standard first thought for anyone given an LM now. I've worked tangentially to NLP for about 7 years in academia, but most of my work has been focused on a less known field of mathematics applied to either to outputs or to the NNs, as well as bioinformatics. As such, my expertise may not be as refined as the real players in the field such as Glaese, Finn, Velockovic, etc. but I try typically keep up with the actual key advancements in the field (usually the stuff few people notice). This area takes far too much compute capability for many people to actually become experts in it, so much of my interests have been less on large LMs and more towards algorithms. But I suppose I agree that it is frustrating to see how little knowledge many of the hype-filled crowds possess that are piling into this area. (Not calling anyone specifically out in this thread, just in general across the internet)
EvgeniyZh 4y ago
Are you saying the less you train the model the better it is? I'm confused

hinkley 4y ago

I think that 20 years from now, we'll all be sitting around wondering 1) where the fuck are my flying cars, and 2) what were they thinking using computers to write code? And the reason I say this is because these tools are answering a question that we haven't asked yet: what common problems need to be solved in this programming language, and where do I get code to solve that problem? These LLM modules are basically telling us how to duplicate code, and what we need is the opposite: how to stop reinventing the wheel for the 100th time. Instead of writing code for me, tell me if I already have it. If I'm writing it, tell me there's a library for that. If I'm a library writer, give me suggestions for what libraries are missing from the toolkit. All we've done so far is begun the process of automating the production of duplicate code. With absolutely no way to go back in time and correct bugs introduced in earlier iterations. We are likely, for instance, to see 0 day attacks that affect hundreds of applications, but with no simple way to describe which applications are affected. That's going to be a first rate trainwreck.

seydor 4y ago
> how to stop reinventing the wheel for the 100th time. The idea of libraries may not have been a good one. It saved human time but no library is perfect because no abstraction is perfect and this causes unnecessary bloat. It seems tha Nature does not use libraries, it uses replication instead, and we can now have that too.
- sicariusnoctis 4y ago
  Replication does not help in managing completely. That's why we use abstractions, even with the problems they have.
- x-shadowban 4y ago
  Ha I never wondered what the physical/life version of a shared library is until I read your post so thanks for that.
- hinkley 4y ago
  So instead everyone who has to solve a problem has to be an expert on that problem, rather than just an informed consumer.

lee101 4y ago

[dead]

dvt 4y ago

I genuinely don't understand how anyone can use something like this and seriously think "oh yeah, this is revolutionary." It's almost complete garbage and can't do anything remotely interesting. # a method that approximates the hyperbolic tangent (clamped tanh) def rational_tanh(x): return (x + 1) / (x - 1) Even gave it the BIG hint of a "clamped" and "rational" tanh, but that ain't it, chief. Forget GPT-4, I would be embarrassed to even show this as a tech demo.

afro88 4y ago
I use "something like this" (GPT4) all the time. Use a good model. Can't wait till proper open source models catch up, but they're not there yet. Here's GPT4's response: ``` import math def clamped_tanh(x, n_terms=10): """ Approximate the hyperbolic tangent (tanh) function using a Maclaurin series expansion. Args: x (float): The input value for which to compute the tanh. n_terms (int, optional): The number of terms to use in the Maclaurin series. Default is 10. Returns: float: The approximated tanh value. """ tanh_approx = 0 for n in range(n_terms): coef = ((-1) ** n) * (2 * n + 1) term = coef * (x ** (2 * n + 1)) / math.factorial(2 * n + 1) tanh_approx += term # Clamping the tanh approximation to the range [-1, 1] tanh_approx = max(-1, min(tanh_approx, 1)) return tanh_approx # Example usage x = 0.5 result = clamped_tanh(x) print(f"clamped_tanh({x}) = {result}") ```
- dvt 4y ago
  I don't mean to get in the weeds here, but this is still bad, as the Padé approximation[1] (which is what you're actually really looking for) is magnitudes better than the Maclaurin series (which is just a special case of the Taylor expansion). Keep in mind that I'm not even an expert (I merely earned a minor in math). In fact, I only know this because I did some research years ago on rational approximations of hyperbolic functions. The right answer gets clipped to -1 or 1 beyond (-3, 3), but it should look like: `x * ( 27 + x * x ) / ( 27 + 9 * x * x )`—though the coefficients (and clip range) can vary. [1] https://en.wikipedia.org/wiki/Pad%C3%A9_approximant https://en.wikipedia.org/wiki/Pad%C3%A9_approximant

deleted 4y ago

[deleted]

markeibes 4y ago

I'm always sad to see these things being trained on a tiny number of programming languages. Makes it harder still for the good languages to compete.

abdellah123 4y ago

"1M concurrent containers" curious about replit containers. Do they run on firecracker?

foverzar 4y ago

> Markdown, Java, JavaScript, Python, TypeScript, PHP, SQL, JSX, reStructuredText, Rust, C, CSS, Go, C++, HTML, Vue, Ruby, Jupyter Notebook, R, Shell No Kotlin T_T I wonder if fine-tuning to a new language would even make sense. AFAIK, it is the core knowledge within the model that really matters, finetuning is essentially specialising.

throwaway290 4y ago

> It's been trained on 525 billion tokens of, of code all permissively licensed code What does "permissively licensed" mean?