6 ms·

Dejavu – Web UI for Elasticsearch

129 points by reactivesearch 9y ago

braderhart 9y ago
I thought for sure that you were going to be a PDF alternative, but then I remembered the spelling is different.
- romwell 9y ago
  Yeah, when I looked at the title, I was confident I've seen that before, but I hadn't. You might say I had a case of, ah, never mind, bad pun. On a serious note, I understand there's only a finite number of words, but a limitless number of software project - and yet surely there are still enough names that aren't used for something well-known and widely used. Yes, the DJVU and Dejavu are spelled differently, but it is the same word.
throwaway2016a 9y ago
> dejavu is the missing web UI for Elasticsearch. Existing web UIs leave much to be desired or are built with server side page rendering techniques (I am looking at you, Kibana). So less so "missing" UI and more "outdated" UI. I personally like Kibana but I am happy to see some innovation in the space. Competition can only help. I love that this can be run as a Docker container or as a Chrome extension. With the quick setup I'll probably give it a try. It looks nice.
- reactivesearch 9y ago
  It's complementary to Kibana, I would say. Dejavu gives the raw data view as well as an ability to add/update data along with mappings. It also comes with an importer view where you can add JSON / CSV files to directly import into ES. Doing this otherwise currently is a major PITA. Kibana is more for dashboarding and visualization, Dejavu gives you control of the raw data views and indexing operations.
- weego 9y ago
  Jesus, since when is server side rendering of server side data supposedly an anti-pattern?
ninjaranter 9y ago
> dejavu uses a websockets based API and subscribes for data changes for the current filtered view. For this to work, the Elasticsearch server needs to support a websockets based publish API Is websockets a requirement just for the real-time updates, or for the entire UI?
- m1sta_ 9y ago
  Why do you ask? Are you planning on accessing an elasticSearch admin tool from a browser that doesn’t have web socket support?
  - milesokeefe 9y ago
    There’s no open source websockets-based ES publish api, you have to use the closed source private service made by the company which created this project.
  - gumoro 9y ago
    Sometimes you want to access the application from outside, available via some corporate proxy, and all do not support WebSockets, and even those that do on paper are so complicated to set up that some admins abandon the idea (looking at you MS Forefront TMG). So it's good to know that you'll get the baseline functions working without WS.
- reactivesearch 9y ago
  Just the realtime updates. There is a "reload" button which lets you update the UI view. Edit: We need to update the GIFs and screenshots - the app has come along much UI/Ux wise since these were originally taken.
burner8309 9y ago
> dejavu is the missing web UI for Elasticsearch. Existing web UIs leave much to be desired or are built with server side page rendering techniques (I am looking at you, Kibana). When did server-side rendering become a bad thing? It tends to be much faster. Kibana feels slow to me, and that's because of the JavaScript it does have.
- rpedela 9y ago
  I don't agree that server-side rendering tends to be faster. I think it largely depends on the use case. I am going to use the word "responsive" rather than "faster" since it is more descriptive of how a UI feels. If you don't need to go to the server, performing an action will be more responsive if done client-side in general. If you do have to go to the server, then you are at the mercy of the network. However the less data transferred, the better in terms of responsiveness. In many cases, asking the server for JSON rather than HTML/JS/CSS is more responsive. Render time is also a factor. If client render time is slower than network/server time, then rendering on the server is more responsive in general.
  - burner8309 9y ago
    How about with JavaScript off?
  - purple-again 9y ago
    This right here. My applications are typically used 8-10 hours a day by a small team of 5-10 users. Everything is done with Ember because Rails could not provide the User experience of a desktop app with the constant server rendering. Everyone loves the SPA when it rolled out and it’s still going strong 3 years later.
  - purerandomness 9y ago
    I agree that client-side UI modifications tend to be faster if no server round-trip is necessary. Rendering HTML will, however, tend to be faster than having a client parse and interpret JSON, and then manipulate the (shadow) DOM accordingly. And then it always felt wrong to me that I have to force countless clients to do the exact same view transformations, which could have been done once, on the server, and cached. For a new project, I'm prerendering HTML fragments via AJAX with IntercoolerJS - it's a pleasure to work with, and the developer and user get both the best experience of those two worlds: A dynamic, blazingly fast UI that feels like an SPA, but degrades gracefully even for the Emacs-w3m user and Googlebot. I also found it's really worth it to move servers or reverse proxies closer to the client - independent of client/server rendering. Users shouldn't wait more than 200ms for an Ajax request to finish.
dabeledo 9y ago
I've been using elasticsearch-head [1] for four years now and it seems it has more features than dejavu? Specially the node/shard visualisation. [1] https://github.com/mobz/elasticsearch-head https://github.com/mobz/elasticsearch-head
- reactivesearch 9y ago
  Yes, we don't focus on admin related features atm. Elasticsearch head or Elasticsearch HQ [1] are good options for doing this. Fun fact: We actually started out with elasticsearch-head, but realized it was really hard to hack on (this was ~2y ago) and decided to create Dejavu. Imo, the key difference is we have focused a lot on how to get the data indexed (mappings, bring your CSV/JSON files) and have the raw data visualized (via UI based filters or DSL queries). [1] https://github.com/ElasticHQ/elasticsearch-HQ https://github.com/ElasticHQ/elasticsearch-HQ
giancarlostoro 9y ago
Ok so maybe someone can tell me what I (we) did wrong at my job we tried using the ELK stack, and it's probably still running but it is such a resource hog. I do not understand why they built Elasticsearch. I've read in a couple places you need like 32GB of RAM[0] just to run this thing to do queries, and having crashed Kibana / Elasticsearch a dozen times I believe it's designed poorly. I had hoped I could drop in MongoDB instead, but saw no indication of this being a fluid change. How many resources are any of you allocating towards your 'ELK' stack (I say 'ELK' cause now they have other software in the mix)? Needless to say, I rather build my own solution for logging instead using a database that's not in-house having experienced all of this. [0]: https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html https://www.elastic.co/guide/en/elasticsearch/guide/current/... Oh wow now it says 64GB of ram is the sweet spot.... What the heck is this thing doing that couldn't of been accomplished with MongoDB or PostgreSQL? I've got busier data sets that don't need 16GB of RAM, and yes we pound the database with logs of sorts and query in all sorts of ways and still I don't get it... I wouldn't recommend this stack to a friend unless they've got plenty of hardware to spare.
- arca_vorago 9y ago
  This is why I never got into the elk craze. Especially coming from oldschool syslog/syslog-ng/rsyslog/journald/etc... and usually searched from the command line. For some reason though the past few years companies and people have become obsessed with gui's, and nagios/cacti were falling out of favor, so people started just dumping elk/graylog so randoms could quickly and easily get that gui... without considering the resource requirements and lack of scalability. (graylog2 fixed a lot of the graylog problems) It seems a lot of managers also started wanting gui-dashboards, which is probably the big reason behind the push. Regardless, I don't think the trend is going away, so the market is ripe for disruption for something that does the same thing but faster and with less resources. The real problem is most of the competitors for some reason decided proprietary was the way to go, and the people using these tools don't want more proprietary bullshit in their stack. The feature that makes them different from other web-gui-graphing tools though is the search/query customization.
- zimbatm 9y ago
  I spent so much time resurrecting ElasticSearch and then developers wouldn't even use it because the query language is based on ngrams and they want grep-like capabilities instead. Unless there are hundreds of GB/day to index it's much simpler to forward the logs using syslog or journald and then use grep on the collector.
forgotcreds 9y ago
>built with server side page rendering techniques (I am looking at you, Kibana). I'm not sure if this is actually accurate: 1. https://github.com/elastic/kibana/search?utf8=&q=React https://github.com/elastic/kibana/search?utf8=&q=React 2. https://elastic.github.io/eui https://elastic.github.io/eui
- reactivesearch 9y ago
  I will have to relook at this - the comment is from a year ago when they weren't using React. On a cursory look, it still seems to use a Node.JS based server [1]. One of the key side effects of needing a server is that there is a more involved distribution + installation process. You just can't run it on a static server like github pages or as a browser extension. [1] https://github.com/elastic/kibana/search?utf8=%E2%9C%93&q=hapi&type= https://github.com/elastic/kibana/search?utf8=%E2%9C%93&q=ha...
tenken 9y ago
Is there anything similar for Solr??
- reactivesearch 9y ago
  Not that I know of, but given the similarities in the underlying APIs - we would be open to supporting this and help guide the implementation if someone is interested in sending a PR. Feel free to file an issue - https://github.com/appbaseio/dejavu/issues/ https://github.com/appbaseio/dejavu/issues/.
artellectual 9y ago
One thing to note for anyone using elasticsearch in production, if you setup your own cluster. It might be wise to put nginx as a reverse proxy. By just doing that we reduced the amount of crashes when the server got hammered. Since nginx will help with request queueing
bestest 9y ago
I've stumbled on a couple of bugs with Dejavu, and wanted to see if I can submit a bug fix. And, well, submit and fix them I can, but boy oh boy is the code terrible. Only use this on your own risk and on a throwaway ES instance, as the code quality in this case kind of implies there's major risk of screwing things up.

I thought for sure that you were going to be a PDF alternative, but then I remembered the spelling is different.

romwell 9y ago
Yeah, when I looked at the title, I was confident I've seen that before, but I hadn't. You might say I had a case of, ah, never mind, bad pun. On a serious note, I understand there's only a finite number of words, but a limitless number of software project - and yet surely there are still enough names that aren't used for something well-known and widely used. Yes, the DJVU and Dejavu are spelled differently, but it is the same word.

throwaway2016a 9y ago

> dejavu is the missing web UI for Elasticsearch. Existing web UIs leave much to be desired or are built with server side page rendering techniques (I am looking at you, Kibana). So less so "missing" UI and more "outdated" UI. I personally like Kibana but I am happy to see some innovation in the space. Competition can only help. I love that this can be run as a Docker container or as a Chrome extension. With the quick setup I'll probably give it a try. It looks nice.

reactivesearch 9y ago
It's complementary to Kibana, I would say. Dejavu gives the raw data view as well as an ability to add/update data along with mappings. It also comes with an importer view where you can add JSON / CSV files to directly import into ES. Doing this otherwise currently is a major PITA. Kibana is more for dashboarding and visualization, Dejavu gives you control of the raw data views and indexing operations.
weego 9y ago
Jesus, since when is server side rendering of server side data supposedly an anti-pattern?

ninjaranter 9y ago

> dejavu uses a websockets based API and subscribes for data changes for the current filtered view. For this to work, the Elasticsearch server needs to support a websockets based publish API Is websockets a requirement just for the real-time updates, or for the entire UI?

m1sta_ 9y ago
Why do you ask? Are you planning on accessing an elasticSearch admin tool from a browser that doesn’t have web socket support?
- milesokeefe 9y ago
  There’s no open source websockets-based ES publish api, you have to use the closed source private service made by the company which created this project.
- gumoro 9y ago
  Sometimes you want to access the application from outside, available via some corporate proxy, and all do not support WebSockets, and even those that do on paper are so complicated to set up that some admins abandon the idea (looking at you MS Forefront TMG). So it's good to know that you'll get the baseline functions working without WS.
reactivesearch 9y ago
Just the realtime updates. There is a "reload" button which lets you update the UI view. Edit: We need to update the GIFs and screenshots - the app has come along much UI/Ux wise since these were originally taken.

burner8309 9y ago

> dejavu is the missing web UI for Elasticsearch. Existing web UIs leave much to be desired or are built with server side page rendering techniques (I am looking at you, Kibana). When did server-side rendering become a bad thing? It tends to be much faster. Kibana feels slow to me, and that's because of the JavaScript it does have.

rpedela 9y ago
I don't agree that server-side rendering tends to be faster. I think it largely depends on the use case. I am going to use the word "responsive" rather than "faster" since it is more descriptive of how a UI feels. If you don't need to go to the server, performing an action will be more responsive if done client-side in general. If you do have to go to the server, then you are at the mercy of the network. However the less data transferred, the better in terms of responsiveness. In many cases, asking the server for JSON rather than HTML/JS/CSS is more responsive. Render time is also a factor. If client render time is slower than network/server time, then rendering on the server is more responsive in general.
- burner8309 9y ago
  How about with JavaScript off?
- purple-again 9y ago
  This right here. My applications are typically used 8-10 hours a day by a small team of 5-10 users. Everything is done with Ember because Rails could not provide the User experience of a desktop app with the constant server rendering. Everyone loves the SPA when it rolled out and it’s still going strong 3 years later.
- purerandomness 9y ago
  I agree that client-side UI modifications tend to be faster if no server round-trip is necessary. Rendering HTML will, however, tend to be faster than having a client parse and interpret JSON, and then manipulate the (shadow) DOM accordingly. And then it always felt wrong to me that I have to force countless clients to do the exact same view transformations, which could have been done once, on the server, and cached. For a new project, I'm prerendering HTML fragments via AJAX with IntercoolerJS - it's a pleasure to work with, and the developer and user get both the best experience of those two worlds: A dynamic, blazingly fast UI that feels like an SPA, but degrades gracefully even for the Emacs-w3m user and Googlebot. I also found it's really worth it to move servers or reverse proxies closer to the client - independent of client/server rendering. Users shouldn't wait more than 200ms for an Ajax request to finish.

dabeledo 9y ago

I've been using elasticsearch-head [1] for four years now and it seems it has more features than dejavu? Specially the node/shard visualisation. [1] https://github.com/mobz/elasticsearch-head https://github.com/mobz/elasticsearch-head

reactivesearch 9y ago
Yes, we don't focus on admin related features atm. Elasticsearch head or Elasticsearch HQ [1] are good options for doing this. Fun fact: We actually started out with elasticsearch-head, but realized it was really hard to hack on (this was ~2y ago) and decided to create Dejavu. Imo, the key difference is we have focused a lot on how to get the data indexed (mappings, bring your CSV/JSON files) and have the raw data visualized (via UI based filters or DSL queries). [1] https://github.com/ElasticHQ/elasticsearch-HQ https://github.com/ElasticHQ/elasticsearch-HQ

giancarlostoro 9y ago

Ok so maybe someone can tell me what I (we) did wrong at my job we tried using the ELK stack, and it's probably still running but it is such a resource hog. I do not understand why they built Elasticsearch. I've read in a couple places you need like 32GB of RAM[0] just to run this thing to do queries, and having crashed Kibana / Elasticsearch a dozen times I believe it's designed poorly. I had hoped I could drop in MongoDB instead, but saw no indication of this being a fluid change. How many resources are any of you allocating towards your 'ELK' stack (I say 'ELK' cause now they have other software in the mix)? Needless to say, I rather build my own solution for logging instead using a database that's not in-house having experienced all of this. [0]: https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html https://www.elastic.co/guide/en/elasticsearch/guide/current/... Oh wow now it says 64GB of ram is the sweet spot.... What the heck is this thing doing that couldn't of been accomplished with MongoDB or PostgreSQL? I've got busier data sets that don't need 16GB of RAM, and yes we pound the database with logs of sorts and query in all sorts of ways and still I don't get it... I wouldn't recommend this stack to a friend unless they've got plenty of hardware to spare.

arca_vorago 9y ago
This is why I never got into the elk craze. Especially coming from oldschool syslog/syslog-ng/rsyslog/journald/etc... and usually searched from the command line. For some reason though the past few years companies and people have become obsessed with gui's, and nagios/cacti were falling out of favor, so people started just dumping elk/graylog so randoms could quickly and easily get that gui... without considering the resource requirements and lack of scalability. (graylog2 fixed a lot of the graylog problems) It seems a lot of managers also started wanting gui-dashboards, which is probably the big reason behind the push. Regardless, I don't think the trend is going away, so the market is ripe for disruption for something that does the same thing but faster and with less resources. The real problem is most of the competitors for some reason decided proprietary was the way to go, and the people using these tools don't want more proprietary bullshit in their stack. The feature that makes them different from other web-gui-graphing tools though is the search/query customization.
zimbatm 9y ago
I spent so much time resurrecting ElasticSearch and then developers wouldn't even use it because the query language is based on ngrams and they want grep-like capabilities instead. Unless there are hundreds of GB/day to index it's much simpler to forward the logs using syslog or journald and then use grep on the collector.

forgotcreds 9y ago

>built with server side page rendering techniques (I am looking at you, Kibana). I'm not sure if this is actually accurate: 1. https://github.com/elastic/kibana/search?utf8=&q=React https://github.com/elastic/kibana/search?utf8=&q=React 2. https://elastic.github.io/eui https://elastic.github.io/eui

reactivesearch 9y ago
I will have to relook at this - the comment is from a year ago when they weren't using React. On a cursory look, it still seems to use a Node.JS based server [1]. One of the key side effects of needing a server is that there is a more involved distribution + installation process. You just can't run it on a static server like github pages or as a browser extension. [1] https://github.com/elastic/kibana/search?utf8=%E2%9C%93&q=hapi&type= https://github.com/elastic/kibana/search?utf8=%E2%9C%93&q=ha...

tenken 9y ago

Is there anything similar for Solr??

reactivesearch 9y ago
Not that I know of, but given the similarities in the underlying APIs - we would be open to supporting this and help guide the implementation if someone is interested in sending a PR. Feel free to file an issue - https://github.com/appbaseio/dejavu/issues/ https://github.com/appbaseio/dejavu/issues/.

artellectual 9y ago

One thing to note for anyone using elasticsearch in production, if you setup your own cluster. It might be wise to put nginx as a reverse proxy. By just doing that we reduced the amount of crashes when the server got hammered. Since nginx will help with request queueing

bestest 9y ago

I've stumbled on a couple of bugs with Dejavu, and wanted to see if I can submit a bug fix. And, well, submit and fix them I can, but boy oh boy is the code terrible. Only use this on your own risk and on a throwaway ES instance, as the code quality in this case kind of implies there's major risk of screwing things up.