14 ms·
Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine
- elvinyung 9y agoThis is really cool. Vespa was probably first described in this 2007 paper: https://brage.bibsys.no/xmlui/bitstream/handle/11250/251199/348506_FULLTEXT01.pdf https://brage.bibsys.no/xmlui/bitstream/handle/11250/251199/... Next up, I would really like to see Sherpa/PNUTS (their NoSQL operational database) and Everest (their petabyte-scale Postgres data warehouse) open sourced :)
- FractalNerve 9y agoMay I ask some stupid questions? :/ I don't quite get the diagram of the Vespa Architecture. Is Vespa a middleware between database engine and query parser? This is what puzzles me. If so, are there other such middlewares available for ie. PostgresSQL that allow hooking "Query Templating Models" (that is it?) generated via Machine-Learning Models? Is it way more complicated than that, or did they overengineer the problem into a monolith? EDIT: Looking at https://github.com/vespa-engine/vespa https://github.com/vespa-engine/vespa it seems that it is overengineered, or maybe it consists of individual micro-components like node.js, hmm more questions :( Is GraphQL such middleware or lower-level? Does Vespa replace custom Glue-Code between Backend and Frontend that generates such query-sets for content ranking/positioning? Or what exactly does Vespa solve? I'm sorry, I've read the article, but can't say, yep that's what it is! EDIT: How else could you solve what Vespa does using Rust, Go, or C/C++ libraries? A very simple or general direction would be immensely useful to understand Vespa =) The project makes the simultanous impression of an immense engineering feat and at the same time a huge code debt.
- elvinyung 9y agoI think at a glance, it's basically a much more scalable version of something like Elasticsearch, optimized for very quick wide fanout to a large number of leaf nodes. It's a datastore in its own right (just like ES), but I imagine that e.g. you wouldn't use it to handle transactions.
- FractalNerve 9y agoSo the upsides of Vespa over Elasticsearch are speeding up the rate at which it scales? Ah, that seems reasonable for a company this size, but is there something in there that's of use for Startups? This blog post shows how Elasticsearch was used to reindex a 136TB dataset with 36B documents[1], so I'm unsure exactly where except for Google/Yahoo Scale companies Vespa is of use. I would like to understand howto utilize it though without adding an umnanagable complexity. EDIT: Maybe a Vespa Cloud startup, that abstracts the management and makes "Scalability as a Service" by utilizing other Cloud providers. -- [1] https://thoughts.t37.net/how-we-reindexed-36-billions-documents-in-5-days-within-the-same-elasticsearch-cluster-cd9c054d1db8 https://thoughts.t37.net/how-we-reindexed-36-billions-docume... [2] http://docs.vespa.ai/documentation/vespa-quick-start.html http://docs.vespa.ai/documentation/vespa-quick-start.html
- toast0 9y agoIn my experience running machines with Vespa (ended in 2011) and elastic search (which ended earlier this year), Vespa was a lot more stable, even though my elastic search had many times more hardware and fewer documents. At least once a month, elastic search would take a several minute break to do who knows what, even though there was not even any indexing or anything other than searching going on. In case it matters, I was running elastic as a single node cluster (actually several single node cluster), my production Vespa was multinode, but I think we had a single node (or fewer node anyway) cluster for dev/testing. Anyway, I'm happy that we have more options in this space now.
- FractalNerve 9y ago> How else could you solve what Vespa does using Rust, Go, or C/C++ libraries? Let me try myself answering my own question, I hope someone hops in and tells me where I'm wrong or how else to improve :) 1) Get PostgresSQL exntensions via "package manager" pgxnclient 1.1) pg_bouncer - For connetion pooling 1.2) yoke - As a high-availability cluster manager with auto-failover and automated cluster recovery 1.3) prestodb.io - Distributed SQL query engine for pgsql 1.4) pglogical - Logical streaming replication for using a publish/subscribe model 1.5) pg_lambda - To create your own AWS (meta) Lambda 1.6) pg_strom - To offload tasks to the GPU 1.7) zombodb - To utilize full-text searching via indexes backed by Elasticsearch 2) Put all together with pglogical and presto to seperate GPU/CPU intensive tasks. 2.1) "Build Missing Middleware" - To design/fuse a query visually that combines multiple backends 2.1.1) Create a binary data-stream by integrating pg_lambda, pg_strom, presto and zombodb 2.1.2) "Build Missing Middleware" - A tensor processing extension to use ML Model evaluations 2.1.3) "Use Missing Middleware" - For data-processing via Machine-Learning models 2.1.4) "Use Missing Middleware"- To output ML processed results into the database 2.2) Partition these queries using "pg_lambda + middleware" to create accelerated and fused query results So what's missing to create a Vespa alternative using existing technologies is everything in Point 2) if I'm not mistaken. Torrent based replication isn't exactly neccessary, except at Twitter/Facebook scale, but if you reach that stage you can hire a libtorrent author.
- FractalNerve 9y agoI thik basing this on PostgresSQL was wrong now and believe that a meaningful approach at creating a Vespa alternative yourself is basing this on a Content-Adressable-Storage[1] and adding a DB-Layer ontop (ie. using AUFS). It would have following properties: decentralized, distributed, resilient, highly-available, software-defined storage & retrieval system. According to http://vespa.ai/#featurematrix http://vespa.ai/#featurematrix: FEATURE VESPA ELASTIC SEARCH RELATIONAL DATABASES ACID transactions ••• Optimized for analytics ••• •• Optimized for serving ••• • •• Scalable ••• •• • Easy to operate at scale •• • Text search ••• •• • Machine learned ranking ••• • 2.1.2) - 2.1.4) Middleware logic container ••• 1.4) Live reconfiguration ••• 1.2) And yet I've to admit that even if the Github repository looks quite chaotic, making an alternative, even using existing technologies would be big feat. Initially I would've chosen PostgresSQL as a base, but the "HA-Layer" is something that shouldn't be decoupled and not a later thought. That's why CAS is a much better form of integration. Also integrating the PostgresSQL Engine into a zfs kernel extension ie. would be a mess. And integrating the database engine into a a distributed p2p algorithm would only add compatability issues an no real advantages. [1] https://en.wikipedia.org/wiki/Content-addressable_storage#Open-source_implementations https://en.wikipedia.org/wiki/Content-addressable_storage#Op... PS: Clever aquisition by Docker! "Infinit.sh is a content-addressable and decentralized (peer-to-peer) storage platform that was acquired by Docker Inc." And in my eyes one of the best implementations and easiest targets that allow adding a database-layer ontop.
- 076ae80a-3c97-4 9y ago"When machines are lost or new ones added, data is automatically redistributed over the machines, while continuing serving and accepting writes to the data. Changes to configuration and Java components can be made while serving by deploying a changed application package - no down time or restarts required." That sounds pretty impressive.
- iamalchemist 9y agoLooks impressive!
- cies 9y agoWow this project is humongous! https://github.com/vespa-engine/vespa https://github.com/vespa-engine/vespa I'm really curious how it compares to Lucene/ElasticSearch/ELK, which is currently my tool of choice for (faceted) search and recommendation.
- aidos 9y agoI think that's actually the broadest root folder I've ever seen on a project! It's really hard to know where to start looking. Is anyone familiar with the internals?
- RealJon 9y agoSorry about that - we haven't really optimized the module structure for newcomer comprehension. If you tell me what you want to are looking for I can point you to the right place.
- cies 9y agoFound some comparison (on the root page of their project website, my bad). http://vespa.ai/#featurematrix http://vespa.ai/#featurematrix Next thing I'd really like to know is what existing software it builds on top of.
- martinp 9y agoOutside of general purpose libraries that are found in most software projects it's not based on any existing software. It's built from the ground up by Oath, and the companies that preceded it: Yahoo, FAST, Overture since early-mid 2000s. This article has some more details about the history: https://www.cnbc.com/2017/09/26/yahoo-open-sources-vespa-for-content-recommendations.html https://www.cnbc.com/2017/09/26/yahoo-open-sources-vespa-for... Disclamer: I work on the Vespa team in Trondheim, Norway.
- continuations 9y agoDoug Cutting, the creator of Lucene, was an employee of Yahoo. So does Vespa share any technologies with Lucene?
- martinp 9y agoMaybe this can be merged with the discussion from yesterday: https://news.ycombinator.com/item?id=15339851 https://news.ycombinator.com/item?id=15339851
- gfredtech 9y agoI observe that it was written in Java: https://github.com/vespa-engine/vespa https://github.com/vespa-engine/vespa
- sandGorgon 9y agoI see it has tensor processing built in - http://docs.vespa.ai/documentation/tensor-intro.html http://docs.vespa.ai/documentation/tensor-intro.html Can this be used as a spark+tensorflow replacement ?
- smorgrav 9y agoIt is more like an alternative to tensorflow serving - it does not train models but is good at evaluating them and use the evaluation for ranking.
- gyehuda 9y agoNote: TFoS is also a Yahoo open source project. The teams work together. https://github.com/yahoo/TensorFlowOnSpark https://github.com/yahoo/TensorFlowOnSpark
- sandGorgon 9y agois tfos production-ready right now ? because i thought it was still experimental. is it used inside Yahoo - because Vespa comes with its own tensor processing engine. I wondered who would use one over the other.
- RealJon 9y agoTensorFlow on Spark is for learning, Vespa is for serving. Where Vespa excels is in evaluating a learned model very quickly over lots of documents. We're working on providing support for running models learned with TensorFlow directly. For now people make the translation on their own.
- KGIII 9y agoI don't want to trigger any bot detection by voting all your comments up in a short amount of time. So, I will say thank you and mention that is is contributions from people like you that keep me coming back.
- madmax108 9y agoYahoo time and again releases open source software which is super super helpful to the community at large. But it always makes me wonder why an org with such an amazing engg culture (multiple anecdotes from friends who were at Yahoo, plus the amazing experiences at Yahoo OpenHack each year as a testament to this) could be run into the ground. Really goes to show that engg != business and unless you have a firm business model and growth, an amazing engg team can only get you so far. I know I'm just stating the obvious, but just putting it out there! Vespa looks super interesting (more so since I'm in a company that provides ecommerce search APIs as a product) and I'm sure I'll play with it more. Thanks Yahoo! :)
- reacweb 9y agoIMHO, yahoo has bad investors. One of the proof is the firing of Marissa Meyer. As soon as she was hired, the reputation of yahoo has started to improve. All the fights of investors against her until they make her leave had the opposite effect. It takes years to build a reputation and it takes more years for this reputation to bring dividends. I have a similar feeling for Microsoft. The reputation has (slowly) improved since the arrival of Satya Nadella. I which he succeeds to make it back a big player.
- talad 9y agoMeyer basically destroyed a healthy, but not growing, company in the span of 4 years through short-term number-massaging actions at the cost of their competent engineers and core product development. Nadella isn't making a particularly strong case for MS either with his seeming distaste for any product of theirs that isn't either mobile or cloud-based (like the ones where they actually have a monopoly to build off and no competition worth speaking of), which seems to arise from that 'chasing growth numbers' mindset.
- crispytx 9y agoYeah she was terrible. Their homepage still sucks, freezes up my browser all the time.
- 9y ago
- tallanvor 9y agoWow... I'm guessing (hoping) their lawyers made sure to go over their old agreements with a very fine comb to make sure their license for the software allowed them to open source this. Don't get me wrong, they've added a lot to it, but there's a lot of code in there that could only have come from their purchase of Overture, who had purchased AllTheWeb from FAST (which was itself purchased by Microsoft).
- aargh_aargh 9y ago$ cloc-git https://github.com/vespa-engine/vespa.git http://cloc.sourceforge.net v 1.60 T=64.10 s (224.5 files/s, 28276.3 lines/s) -------------------------------------------------------------------------------- Language files blank comment code -------------------------------------------------------------------------------- Java 6573 106215 102720 537097 C++ 3209 76542 19178 504855 C/C++ Header 2985 42731 57087 158388 XML 389 705 550 139626 Maven 141 133 244 14096 CMake 450 254 560 8452 Perl 57 1124 762 7649 Bourne Shell 196 1257 734 6918 Scala 95 1685 617 6378 Teamcenter def 234 1474 3490 2468 Lisp 4 231 403 2118 HTML 16 211 29 1950 C 7 288 198 1432 Python 6 132 66 556 Ruby 9 39 9 294 Bourne Again Shell 3 35 12 182 Pig Latin 9 39 52 54 make 2 22 8 39 Ant 1 9 17 36 YAML 1 9 1 22 DTD 2 6 6 10 -------------------------------------------------------------------------------- SUM: 14389 233141 186743 1392620 --------------------------------------------------------------------------------
- zimpenfish 9y agoAs an ex-employee, there could not be a better description of Yahoo! development than this.
- sanxiyn 9y agoVespa, using its own malloc: https://github.com/vespa-engine/vespa/tree/master/vespamalloc https://github.com/vespa-engine/vespa/tree/master/vespamallo...
- andreer 9y agoThe reason is simply, for performance. To avoid having to go to the kernel every time we need to allocate memory for a query, and avoid having to clear memory on free/reuse. It is made for Vespa, but also used for other programs. Similar in purpose to Google's TCMalloc: http://goog-perftools.sourceforge.net/doc/tcmalloc.html http://goog-perftools.sourceforge.net/doc/tcmalloc.html
- sanxiyn 9y agoHow is it better than TCMalloc? (If it isn't, it probably should be replaced by TCMalloc.)
- andreer 9y agoIn our tests, vespamalloc has simply been faster. I don't know how in-depth the analysis has been as to why, but obviously vespamalloc is written and tuned for Vespa so that is a likely factor.
- whage 9y agoI don't understand how such a move is profitable for a business. Can someone please point me to some articles that discuss this?
- dysoco 9y agoMy idea is that: 1) It attracts developers who will provide fixes, bug reports, documentation, etc. for free (or even draw potential full-time developers for the project). 2) It makes the project look more "trendy" and appeals to developers who will try not to use proprietary software; this is the same move that .NET did and it seems to have worked very well.
- whage 9y ago2nd point seems valid. 1st one however suggests that people will try to use it to their own benefit - some of them to compete with Yahoo. Why else would they contribute to the project?
- gyehuda 9y agoProprietary code is expensive to maintain. Even though there's a dedicated team of a few dozen people that has been working on Vespa over the years there are thousands of developers who have been contributing to ElasticSearch, Lucene, and other projects in the open source world that are in a similar space. Getting contributions to Vespa will help it grow and evolve to make it better -- much like Yahoo did when it evolved Hadoop and scaled it out, much like Yahoo did when it help caffe, druid, hive, oozie, openstack, pig, storm, shark, spark, and tensorflow and tons of other projects (that it either created, co-created, or took from others and contributed improvements back to help make better for all). Sharing code is fine since we use lots of shared code too. We don't sell code. So if someone wants to use Vespa to make an amazing product and make tons of money, we hope they do. We know that sharing Hadoop helped our competitors, but we also know that the revenue stream comes from ads. So we're glad to share code that makes tech better for everyone. As it turns out, many tech companies feel the same way and openly share code with the industry to help us all get to better tech platforms. In the tech space, it's not about grabbing more of the pie, it's about making a bigger pie. The internet revolution is young and the more we built it in the open, the better it will be for all of us.
- wiradikusuma 9y agoThe quickstart doesn't seem to work, at lease on macOS. I raised a ticket: https://github.com/vespa-engine/vespa/issues/3560 https://github.com/vespa-engine/vespa/issues/3560
- qw 9y agoI just checked the issue, and it seems it was caused by human error. It is closed now.
- abiox 9y agoi wonder if this is able to squeeze something useful out of Yahoo Answers.
- KGIII 9y agoI am not trying to be snarky when I say this, so I'll make it longer than a single word. No. That is, Y! Answers was/is a great idea with terrible results. I'm not sure if it should have been moderated better, or if it should have been marketed better. Hell, maybe it should have had a basic literacy test prior to being allowed to post and answer? I could actually come up with a few hundred ways to have made it better. We know the question and answer format works. It does on many, many sites. It failed there and, largely, that's because of the users. I'm not sure which was worse, the questions or the answers. I'd love to be given that project and tasked with improving it. It's a great idea, but horribly implemented. I'd like to fix it because I'm fond of trying the impossible and like fixing broken things. Improving search for it is absolutely not going to help it. No, that's not going to help in the slightest.
- yesiamyourdad 9y agoYep, the old maxim "garbage in, garbage out". Yahoo! was the biggest internet company out there (maybe AOL was bigger). You had the whole population in all its warty glory using Yahoo as the default search engine, landing page, email, and who knows what else. Plus answers was done at a time when we were extremely naive about the wisdom of crowds and open source knowledge. Later efforts learned from those early 2000s failures. I'm pretty sure Joel Spolsky specifically called out Yahoo Answers when he was talking about what Stack Overflow was going to do right in his early announcements.
- gyehuda 9y agoFWIW, there was a research team at Yahoo who would actually get insights about the way people used language (relevant for contextual search) from Yahoo Answers. They were more active during the earlier years when the content quality was much higher. I don't know if they used Vespa in their mining process, but it would not surprise me if they did. Vespa is used in many projects because it's just that good. So if you are thinking of using it -- for Yahoo Answers, for Quora, whatever, go for it. I know a cancer researcher/engineer who wants to use Vespa for serving clinical reports and trial outcomes. As for the current Answers site, well, we'll see what happens. I know the PM, a delightful person. I don't know the plan. But if there is a plan to make something useful from it, apply. It apparently makes money (otherwise it would have been killed long ago), and that means there's something to work with. When I started at Yahoo I was hoping to get onto the Groups team for the same reason -- a huge challenge to fix something that could be made cool again.
- stereosteve 9y agoHere is a snippet to decodeURIComponent for all the yql examples in the documentation. Makes it a bit easier to see the yql syntax. $('pre:contains("yql=")').each((i, el) => { el.innerText = el.innerText.replace(/\+/g, ' ').replace(/yql=(.+%3B)/, (m, p1) => 'yql=' + decodeURIComponent(p1)) })
- softwaredoug 9y agoIf anyone wants the kind of machine learning based rerabking in Elasticsearch, we've been working with the wikimedia foundation on an Elasticsearch learning to Rank plugin: http://github.com/o19s/elasticsearch-learning-to-rank http://github.com/o19s/elasticsearch-learning-to-rank
- tedd4u 9y agoAt Flickr, we worked closely with the Vespa team from 2011 through 2016 on a wide range of advancements: * partial document refeeding (i.e. expedite indexing a new field to 20+ billion documents without refeeding everything and staying online handling 100M+ free text queries a day) * visual similarity search - check out the tensor ranking features [1] [2] * online elasticity - add/remove replicas / shards online. A must when it could take weeks+ to re-feed from scratch. This is non-trivial to make work smoothly at scale. * latency / tail-latency on complex queries. p90 reduction from 3,000 to 30 ms. This is a major gift to the open-source community of a battle-tested search engine that works reliably without babysitting with very large datasets, and simultaneous high query / high feed volumes. Huge debt of gratitude to the team in Trondheim and Verizon/Oath/Yahoo legal & management teams for making this happen. :+1: [1] http://docs.vespa.ai/documentation/tensor-intro.html http://docs.vespa.ai/documentation/tensor-intro.html [2] http://docs.vespa.ai/documentation/tensor-user-guide.html http://docs.vespa.ai/documentation/tensor-user-guide.html
- benth 9y agoSay I'm using ELK for log aggregation. Would Vespa be a good replacement? One pain point is ingest rate. How many "average" log lines per second can Vespa do per node?
- andreer 9y agoIt could be a replacement for the 'E', but the APIs are different enough that there's no drop-in replacement for the 'L' and 'K' and creating or making those compatible would be a significant effort. Would be great if someone did though :-)
- benth 9y agoGotcha. On the ingest front, do you have any numbers around that? I see some benchmarks that focus on other (important) aspects like QPS but didn't catch anything on ingest.
- RealJon 9y agoWrite speed (add or update) is typically between a few thousand to a few tens of thousands operations per second per node sustained, depending on sizeof data etc. Sustaining throughput over long time is important and often overlooked mentioned in benchmarks.
- bucketman 9y agoHow well does vespa handle time series data, compared to, say, elasticsearch?
- gyehuda 9y ago"Vespa is the single greatest piece of software Yahoo ever built. It's like ElasticSearch but a hundred times better. I am so happy." Laurie Voss Co-founder/COO of @npmjs https://twitter.com/seldo/status/912876700542787585 https://twitter.com/seldo/status/912876700542787585
- elfchief 9y agoNow, if Yahoo would just open source the yinst/opsdb/rolesdb/igor/etc ecosystem. I miss that elegant tooling so fricking much.
- crudbug 9y agoBig open source news in recent times. How is the storage layer designed ? disk format ? Can you extend the layer to support different models such as Property graph ?