13 ms·

Architecture of Nautilus, the new Dropbox search engine

• Tetris1 8y ago
  I remmember when Dropbox released Firefly. It was simple and elegant. Nautilus is a monster in compare. It would be nice to see some pieces of code...
• tomrod 8y ago
  Cool to hear about a revamp. It confuses me why some projects use a well-known open source name. In Linux, it's primary desktop environment (Gnome) uses Nautilus as a file manager. Dropbox even has a package for Dropbox/Nautilus integration.
  • Insanity 8y ago
    Yeah whenever I hear Nautilus, the first thing that comes to mind is the file manager. But I suppose most dropbox users are on mac/windows.
    • hackandtrip 8y ago
      What alternative have Linux users to Dropbox, to using their own server? (That is, for a number of reasons for most people, suboptimal)? You cant really use OneDrive365, and Dropbox offers vast support to Linux, is easy to set and can be used for free too. Is there a reason why Linux users wouldn't use it? Asking for curiosity.
      • barrkel 8y ago
        Every desktop Linux user could use Dropbox and still most users of Dropbox would probably be Windows and Mac.
      • tombowditch 8y ago
        > Is there a reason why Linux users wouldn't use it https://www.theregister.co.uk/2018/08/14/dropbox_encrypted_linux_support/ https://www.theregister.co.uk/2018/08/14/dropbox_encrypted_l... ?
      • groovybits 8y ago
        > Dropbox offers vast support to Linux Dropbox only supports unencrypted ext4 filesystems on Linux, so I would not use the phrase 'vast support'.
        • coldtea 8y ago
          Because there are many popular desktop options for Linux that don't use ext4 and non-encrypted as their default?
          • yjftsjthsd-h 8y ago
            RHEL family is XFS-centric, and quite a few distros (including Ubuntu) offer encryption in the default installer.
            • danieldk 8y ago
              ext4 on encrypted devices, such as dm-crypt/LUKS, is supported. What is not supported are encryption filesystems that are 'filesystem overlays', such as ecryptfs. (Since I was using ZFS, I am still debating whether to stay with Dropbox after November's filesystem apocalypse.)
              • module0000 8y ago
                FYI, you can still use ecrypt fs with dropbox. Put the encrypted store within your dropbox, and mount it outside your dropbox. From the dropbox point of view, you have thousands of files with gibberish as names.
      • dordoka 8y ago
        OwnCloud supports Linux [1]. If you want a SaaS version, they have several hosting partners [2]. [1] https://owncloud.com/client/ https://owncloud.com/client/ [2] https://owncloud.org/hosting-partners/ https://owncloud.org/hosting-partners/
        • pritambaral 8y ago
          Better go with Nextcloud than ownCloud. Nextcloud is the fork by the original developer team, and has quite a few nice improvements compared to ownCloud (e.g.: video and text chat, e2e encryption)
          • dordoka 8y ago
            Didn't know about the fork. Thanks for the suggestion, will check it out.
      • lyk 8y ago
        keybase's kbfs is my dropbox-style replacement. It works really well in my experience
        • danieldk 8y ago
          kbfs is a network filesystem, whereas Dropbox provides file synchronization. kbfs does not work when you are offline, whereas with Dropbox the files are always available locally on your machine.
      • mynewtb 8y ago
        Seafile is great
      • Insanity 8y ago
        Yeah but that is a different group. "A large percentage of linux users, use dropbox" does not equal "A large percentage of dropbox users, use linux".
      • Ylodi 8y ago
        Seafile (https://www.seafile.com/en/home/ https://www.seafile.com/en/home/).
      • cpburns2009 8y ago
        I use SpiderOak One [1] which is a privacy focused alternative to Dropbox. I run it on Ubuntu (and previously on Debian and Arch Linux). There's no free tier like there is with Dropbox though. [1]: https://spideroak.com/one/ https://spideroak.com/one/
      • reacharavindh 8y ago
        There is Syncthing which looks good on paper, but it lacks an iOS client. I'd love it if someone developed even a read-only iOS app for it.
      • minikomi 8y ago
        For a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem. From Windows or Mac, this FTP account could be accessed through built-in software.
    • mywacaday 8y ago
      I'm mostly a windows user but have some exposure to Ubuntu and had never heard of Nautilus
      • Ylodi 8y ago
        You never used a file manager? That was a tiny, little exposure. :-)
        • cpburns2009 8y ago
          I think Ubuntu renames Nautilus to "Files".
          • feborges 8y ago
            GNOME does.
      • phalangion 8y ago
        It's funny because the only reason I know of Nautilus on Ubuntu is from installing Dropbox. To use the Dropbox daemon on Ubuntu you have to install it and then restart Nautilus.
  • M_Bakhtiari 8y ago
    That's why open source projects need to register their trademarks. That's how Gnome managed to stop Groupon from ripping off the name for their own project.
  • acct1771 8y ago
    Clear disconnect from the open source community, which isn't a negative signal, but also is not a positive one.
    • justtopost 8y ago
      Yeah, its enough that I am actively disinterested. Seemz like the opposite of goodwill.
  • buboard 8y ago
    Isnt this a proprietary backend program that users will never know about? Who cares
  • danShumway 8y ago
    Yeah, this seems unnecessarily confusing to me. It's an internal project, so it's not like the name matters for advertising or anything. Surely it's just common courtesy to not step on top of an actively developed, very popular project that is directly related to file management.
• tegansnyder 8y ago
  Is this based on Lucene in any way?
• markpapadakis 8y ago
  I am very much looking forward to forthcoming posts describing the actual architecture and specifics -- this is a great high-level overview, but I hope and expect they will expand on this expose soon.
• arafalov 8y ago
  In 2015, they evaluated Apache Solr and Elasticsearch and decided to build their own (Firefly). They said, other solutions did not scale. So, instead of contributing to scaling (like Apple and Bloomberg and Cloudera did), they went the other way. Now, they seem to be doing it again (at least they are using Tika). In a meanwhile, Solr implemented most of the features they are describing in their architecture document. Specifically: 1)General scaling: https://lucene.apache.org/solr/guide/7_5/introduction-to-scaling-and-distribution.html https://lucene.apache.org/solr/guide/7_5/introduction-to-sca... (using ZooKeeper and SolrCloud) 2) Search Ranking and click-data training: https://lucene.apache.org/solr/guide/7_5/learning-to-rank.html https://lucene.apache.org/solr/guide/7_5/learning-to-rank.ht... (Contributed by Bloomberg) 3) Offline builds with substitution into production: https://lucene.apache.org/solr/guide/7_5/collections-api.html#createalias https://lucene.apache.org/solr/guide/7_5/collections-api.htm... 4) Near-Real-Time: https://lucene.apache.org/solr/guide/7_5/near-real-time-searching.html https://lucene.apache.org/solr/guide/7_5/near-real-time-sear... 5) Sharding specifically: https://lucene.apache.org/solr/guide/7_5/shards-and-indexing-data-in-solrcloud.html https://lucene.apache.org/solr/guide/7_5/shards-and-indexing... 6) Extraction pipeline, they are doing all together. We have: a) pre-Solr extraction (usually done in a stand-alone client, though we do include Tika and DataImportHandler for quick start), b) in-Solr pre-schema processing with Update Request Processors https://lucene.apache.org/solr/guide/7_5/update-request-processors.html https://lucene.apache.org/solr/guide/7_5/update-request-proc... c) Actual per-field text processing pipelines, separate both for index and query (they call query part later "query understanding": https://lucene.apache.org/solr/guide/7_5/understanding-analyzers-tokenizers-and-filters.html https://lucene.apache.org/solr/guide/7_5/understanding-analy... Also, my own: http://www.solr-start.com/info/analyzers/ http://www.solr-start.com/info/analyzers/ 7) Pluggable internal index formats? Here is the latest (FST50): https://lucene.apache.org/solr/guide/7_5/the-tagger-handler.html https://lucene.apache.org/solr/guide/7_5/the-tagger-handler.... 8) Update system configuration live, over API? https://lucene.apache.org/solr/guide/7_5/configuration-apis.html https://lucene.apache.org/solr/guide/7_5/configuration-apis.... https://lucene.apache.org/solr/guide/7_5/schema-api.html https://lucene.apache.org/solr/guide/7_5/schema-api.html 9) Tolerate small failures, but abort if something is definitely not right: http://www.solr-start.com/javadoc/solr-lucene/org/apache/solr/update/processor/TolerantUpdateProcessorFactory.html http://www.solr-start.com/javadoc/solr-lucene/org/apache/sol... 10) Retrieval root: https://lucene.apache.org/solr/guide/7_5/solrcloud-query-routing-and-read-tolerance.html https://lucene.apache.org/solr/guide/7_5/solrcloud-query-rou... 11) Retrieval leaf: That's Solr's basic shard/core 12) The inverted and forward indexes look like standard Lucene index and maybe docValues: https://lucene.apache.org/solr/guide/7_5/docvalues.html https://lucene.apache.org/solr/guide/7_5/docvalues.html 13) Search orchestrator seems to be a couple of features on top of Solr's existing routing linked earlier. There were individual approaches/3rd-party modules doing some of these (shadow, federation, ACL). Some of this is definitely unique to Dropbox though. 14) Precision vs Recall vs Ranking is too many links, but there is a whole book on this: https://www.manning.com/books/relevant-search https://www.manning.com/books/relevant-search (mostly about Elasticsearch, but Solr has added some new features recently to make it even better) 15) BM25, we had it back in 2015: https://opensourceconnections.com/blog/2015/10/16/bm25-the-next-generation-of-lucene-relevation/ https://opensourceconnections.com/blog/2015/10/16/bm25-the-n... 16) (Future for Nautilus): Distance Based embeddings, such as Word2Vec. Commercial offering on top of Solr has it: https://lucidworks.com/2016/11/16/word2vec-fusion-nlp-search/ https://lucidworks.com/2016/11/16/word2vec-fusion-nlp-search... but I remember discussion for Solr as well 17) (Future for Nautilus): Searching images/videos/etc: https://lucidworks.com/2015/08/28/shutterstock-searches-35-million-images-color-using-apache-solr/ https://lucidworks.com/2015/08/28/shutterstock-searches-35-m... And a lot more (Solr Reference manual is more than 1300 pages....). Obviously, this is a bit of a dig at a Dropbox reinventing the wheel again (or perhaps this time actually using Lucene, but forgetting to attribute it so far). But more importantly, it is a message to others that got excited by their architecture post. You can have a similar battle-tested system for yourself, for free. And if something is not perfect, you can fix it and help the rest of the world too. We are always happy to see new contributors. Finally, if you know Apache Solr well, it is not just Dropbox you can work for, but also Lucidworks, Bloomberg, Cloudera, Alfresco, Shutterstock, Dice, CareerBuilder, and many others.
• wiradikusuma 8y ago
  Also worth mentioning: http://vespa.ai/ http://vespa.ai/