11 ms·
Advancements in machine learning for machine learning
- ikers 3y agoFeels like they bury the lede with the first paragraph, but otherwise cool stuff!
- syndicatedjelly 3y ago[flagged]
- anu7df 3y agoGood god. Anyone else who is curious about the human centipede, please don't Google it. Trust me on this one.
- soundarana 3y agoYou might want to google the South Park version of the human centipede, now that you know.
- tommychillfiger 3y agoLOL, agree. I can't think of one without the other. South Park is absolutely iconic - incredible that they've done what they do for so long with as much consistency in quality.
- RyanShook 3y agoThe pace that ML seems to be advancing right now is amazing. I don’t believe in the singularity but it’s changing software and then society in ways no one can predict.
- greatpostman 3y agoI really don’t people will be programming like we do today in five years
- Gabriel_Martin 3y agoI don't people will be either brotha, I don't people will be <3
- greatpostman 3y agoCool
- mortenjorck 3y agoPeople don’t like it is, but it
- HaZeust 3y agoPeople don't think it be like it is, but it do.
- SirMaster 3y agoI don’t see why not. I like programming how I do now. I don’t plan to stop. People do lots of things manually that machines have been able to do for a long time.
- wait_a_minute 3y agoThis + FunSearch make it seem like Singularity is imminent. https://deepmind.google/discover/blog/funsearch-making-new-discoveries-in-mathematical-sciences-using-large-language-models/ https://deepmind.google/discover/blog/funsearch-making-new-d...
- tommychillfiger 3y agoAt great risk of sounding completely ignorant, this approach is basically what I thought the point of machine learning was - cleverly using feedback loops to improve things automatically. The thing that sticks out to me as particularly cool about FunSearch is the use of programs as inputs/outputs and the fact that they managed to automate feedback. I'm pretty naive in terms of granular understanding here as I am barely proficient in Python, to be clear, but when I daydream about things you could solve with machine learning/AI, this is the approach I always think of and I guess is how I thought it already worked. Load it up with the best information we have currently, define the desired results as clearly as possible, implement some form of automatic feedback, and let it run iteratively until it produces something better than what you had before. Is this a case of "well no shit, but actually implementing that effectively is the hard part"? Is it being able to quickly apply it to a wide variety of problems? I guess I'm trying to understand whether this is a novel idea (and if so, what parts are novel), or if the idea has been around and it's a novel implementation.
- IanCal 3y agoThe important thing is "how do you change X so that it heads towards the goal". And "how to do it quickly and efficiently". Otherwise the description is the same as "select randomly, keep the best, iterate". The goal is also complex. You might be thinking of "find the most efficient program" but that's not what we're doing here iiuc. We're trying to get a program that makes other unseen programs more efficient. That's hard to define as a goal.
- Jensson 3y ago> Otherwise the description is the same as "select randomly, keep the best, iterate". That is what they did though. The LLM didn't know what problem it was "solving".
- deleted 3y ago[deleted]
- m3kw9 3y agoI want to see it come out with a cure for a disease that is tough to cure first. Singularity itself is pointless unless it benefits humans which is mainly in health/lower suffering
- educaysean 3y agoI'd say advancement in mathematics, computer science, and heck, even art is far from "pointless". Why does it feel like goalposts get moved everytime there is a significant progress in AI?
- impegh 3y agoI read this same empty “goalposts” lament multiples times a day when reading this website, and though I know what the words mean I’m confused what you all think they mean. https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=Goalposts%20moved%20ai&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... Is this just some performative grousing or do you really think what has been developed to date is “artificial intelligence”? These comments all conveniently fail to define their author’s goalposts that apparently have been reached or surpassed. What were yours?
- quickthrower2 3y agoWell for it to be Artificial and convey Intellgence. I think that goal has been met!
- F-Lexx 3y agoHow do you define intelligence?
- quickthrower2 3y agoI don’t have a exact definition but I will claim that helping me to write code that usually works first time or has a minor bug based on natural language is intelligence.
- DeathArrow 3y agoFor me it's just another gold rush after dotcom, mobile, cloud, VR.
- falcor84 3y agoI'm not sure what the purpose of the word "just" there is. There indeed seems to be quite a lot of gold to be had by whoever gets a foothold.
- Tao3300 3y ago"just" is there to distinguish a gold rush from a singularity
- xbmcuser 3y agoThe first 3 have and did result as of today in trillions in dollars of economic activity. And have changed societies, politics, political participation, access to knowledge etc worldwide for good and bad. So I don't get why you are so dismissive of them.
- rvnx 3y agoAI is definitively here to stay forever. It's not a hype, it's 100% here for the long-term. The hype may be specific to some companies for now, but AI is deeply going to change many industries, especially due to open-source, specialized chips to allow running in local, and new hardware (I strongly hope a clone of H100 A80G comes quickly). The next step is to add limbs to the LLMs. Then we get Tesla bot who is going to help you with daily chores, and to execute tasks in a factory. The bot can ask its own internal knowledge base to know what action to execute next, and because the model can output JSON, the action commands can be sent to motors for in-real-life execution.
- Jensson 3y ago> It's not a hype, it's 100% here for the long-term. You mean it isn't overhyped, hype is just what expectations people have it is underhyped or overhyped that says how those expectations related to reality.
- deleted 3y ago[deleted]
- behnamoh 3y ago[flagged]
- deleted 3y ago[deleted]
- GreedClarifies 3y agoHow’s Gemini looking?
- sbierwagen 3y agoIt is interesting how persistently dominant GPT-4 is: https://twitter.com/lmsysorg/status/1735729398672716114 https://twitter.com/lmsysorg/status/1735729398672716114 Off the top of my head, I can think for at least five foundation models (Llama, Claude, Gemini, Falcon, Mistral) that are all trading blows, but GPT is still a head above them and has been for a year now. Transformer LLMs are simple enough that, demonstrably, anyone with a million bucks of GPU time can make one, but they can't quite catch up with OpenAI. What's their special sauce?
- kccqzy 3y agoTheir only special sauce is the first-mover advantage. Then it attracted users (data), brand recognition, talent and became a positive feedback cycle.
- vitorgrs 3y agoGPT4 was created before most feedback cycle. They had GPT4 ready before ChatGPT launch. If I recall right, GPT4 got done in October. After that, it was RLHF and safety work (Bing starts using GPT4 publicly in February, a month earlier than official launch)
- famouswaffles 3y agoGPT-4 was done training August 2022
- vitorgrs 3y agoThanks!
- kccqzy 3y ago
- beeskneecaps 3y ago[flagged]
- DeathArrow 3y ago[flagged]
- deleted 3y ago[deleted]
- dalbasal 3y agoCan anyone bring this down to earth for me? What's the actual state of these "ML compilers" currently, and what is rhe near term promise?
- voz_ 3y agoCheck out torch.compile
- d3m0t3p 3y agoOne of the easiest approache is torch.compile, it's the latest iteration of pytorch compiler (previous methods were : TorchScript and FX Tracing.) You simply write model = torch.compile(model) "Across these 163 open-source models torch.compile works 93% of time, and the model runs 43% faster in training on an NVIDIA A100 GPU. At Float32 precision, it runs 21% faster on average and at AMP Precision it runs 51% faster on average."[1] What google is trying to do, is to involve more people in the R&D of these kind of methods. [1]https://pytorch.org/get-started/pytorch-2.0/ https://pytorch.org/get-started/pytorch-2.0/
- PartiallyTyped 3y agoThe near term promise is that you can use AMD, CUDA, TPUs, CPUs etc without explicit vendor support for the framework on which the model was developed. Disclaimer: I will be very handwavey, reality is complex. This is achieved by compiling the graph into some intermediate representation. And then implementing the right backend. For projects here, look at stableHLO, IREE, openXLA. You can argue that Jax's jit compiler is a form of such compiler, mapping the traced operations down to XLA, which then does its own bit of magic to make it work on your backend. It's transformations and abstractions all the way down.
- seydor 3y agoWhat about transformer itself, any indication that it is optimal in some way?
- potac 3y agoCan anyone explain how conv works in that graph. You have a tensor of shape [2,4,16] and you convolve with a kernel of shape [4,16,8] and that gives you a [2,8] tensor? How's that possible?
- phillengel 3y agoDoes this help? *1. Input:* * Tensor shape: [2,4,16] * `2`: This represents the *batch size*, meaning there are two independent data samples being processed. * `4`: This is the *input feature dimension*, indicating each sample has 4 features. * `16`: This is the *input channel dimension*, suggesting each feature has 16 channels of information. *2. Kernel:* * Shape: [4,16,8] * `4`: This is the *kernel size*, meaning the filter window used to convolve has a width of 4. * `16`: This matches the *input channel dimension*, ensuring the filter operates on the same number of channels as the input. * `8`: This is the *output channel dimension*, indicating the convolution produces 8 new channels of information per sample. *3. Output:* * Shape: [2,8] * `2`: This remains the *batch size* as the operation is applied to each sample independently. * `8`: This matches the *output channel dimension* of the kernel, signifying the final tensor has 8 new features extracted from the input. *4. How is it possible?* Despite the seemingly mismatched dimensions in the input and output, convolution on graphs works by leveraging the *neighborhood structure* of the graph. Here's a simplified explanation: * The kernel slides across the graph, applying its weights to the features of the current node and its neighbors within a specific radius. * This weighted sum is then aggregated to form a new feature for the current node in each output channel. * As the kernel moves across the graph, it extracts information from the local neighborhood of each node, creating new features that capture relationships and patterns within the graph. *Additional considerations:* * The graph structure and edge weights likely play a role in how information propagates during the convolution process. * Specific details of the convolution implementation, including padding and stride, might also influence the output shape.
- potac 3y agoThanks. What was confusing me is the kernel size 4. Normally in (2D) convolutions you have (in_channels, out_channels, k, k) for a kxk kernel size. In the example above it the k is the first dimension instead of the last. This is in PyTorch, not sure about Keras
- owlbite 3y agoThese ML-compilers are being overhyped. It's all the same trade-off as a traditional compiler: you get a lot more throughput than hiring a specialist performance programmer, but the latter will typically outperform, possibly by orders of magnitude. These things are inferior at many levels: - Algorithmic: These things aren't feeding back to their human masters tips and tricks on how to modify the network to go faster beyond some very basic signals. - Loss of intent: ML network designers are specifying architecture in python, and by the time it's gone through many layers of lowering, you can get some complete garbage. Highly efficient garbage, but still garbage. (recent example, we caught one of these compilers doing a slice update operation by first forming the range of all possible indices to the array, slicing that to get indices to update, and then doing a scatter; we replaced it with a single memcpy call). - Inefficient kernels. Every time we see the output of these compilers go head-to-head with an expert assembly programmer, the compiler loses, often by 30%+. This always seems like the sort of thing that should be easy to solve, but given no-one seems to have cracked it in the past 50 years, it's obviously not as simple as it sounds.
- hotstickyballs 3y agoHardware (and performance) can always be improved without involvement of users so this is actually pretty useful.
- stabbles 3y agoTake a look at the chess engine Stockfish: they tossed out years and years of human written heuristics in board evaluation, to a small neural net that does the same but better. Now consider all the heuristics for inlining, loop unrolling, vectorization etc in compilers, certainly a neural net can be beneficial and possibly easier to maintain than tons of human written heuristics.
- asah 3y agobig +1 - IMHO the future of optimizers (and probably compilers...) are almost certainly ML-based.
- ldjkfkdsjnv 3y ago
- aconz2 3y agosummary: improve prediction of run-time performance of a computation graph using GNN, they use an embedding dictionary for each node's opcode along with some other node features (eg shape, bits, window size, see [1]), they released a big dataset of these graphs in [2] with varying XLA compilation configurations and their resulting perf on TPUs, they did some stuff to improve prediction on bigger graphs than before in [3] by partitioning the graph (METIS graph partition, new to me) and other training things This is only about predicting performance of a given graph and not about improving/suggesting/editing a new equivalent graph. As in FunSearch, models which have decent predictive power could be used with evolutionary search. [1] https://github.com/google-research-datasets/tpu_graphs#features https://github.com/google-research-datasets/tpu_graphs#featu... [2] TpuGraphs: A Performance Prediction Dataset on Large Tensor Computational Graphs https://arxiv.org/abs/2308.13490 https://arxiv.org/abs/2308.13490 [3] Learning Large Graph Property Prediction via Graph Segment Training https://arxiv.org/abs/2305.12322 https://arxiv.org/abs/2305.12322