13 ms·

Show HN: Spotbot, a Slack bot that monitors your website for visual bugs

179 points by jmtulloss 11y ago

jmtulloss 11y ago
Hi Hacker News! We've been working hard on Spotbot for the past couple months and are looking forward to your feedback on what we've built. @mgc and I will be around all day to answer any questions.
- spenserhale 11y ago
  Justin, People cannot try free, your servers are responding with 503s. I tried to email you from your contact link on website (hi@spotbot.qa) and was undeliverable.
  - jmtulloss 11y ago
    Well the email thing is pretty embarrassing, just fixed. I wrote "hi@spotbot.qa" and set up an email for "hello@spotbot.qa". Face palm. We are experiencing a lot of load (or we were until we went down). We scaled up and things seem to be working better now.
- mtw 11y ago
  "Pool is destroyed"
  - jmtulloss 11y ago
    No longer destroyed!
- lstamour 11y ago
  The site I'm currently maintaining doesn't change very often, but it's approx. 1,500 pages if spidered, plus a large search index. I'm looking at the Fancy plan, but we've got multiple subdomains or internal versions of the site. How are these covered if it's not all under one domain? Are sub-domains part of the same domain? :) Is it possible to integrate with a CI system for on-demand runs, perhaps by supplying a list of changed pages as a hint on where to expect changes and where not? The idea being that to lower your costs, a subset of pages could be run on-demand?
  - jmtulloss 11y ago
    We don't currently have CI integration, but it's coming up fast on the roadmap. What system do you use? We're still hammering out the pricing plans. If you email me at justin@spotbot.qa we can set you up with something that makes sense.
    - lstamour 11y ago
      No specific CI in mind, maybe just Github integration or a webhook to start. I'm currently looking at options to modernize and add tests to a rather manual, SVN-based process. I'll email for sure, I'm interested to see how this will work. Thanks! :)
juecd 11y ago
Super excited to try this — as an early start-up (and the founder in charge of UI), I KNOW our site has UI issues but I don't have the time to do full inspections. What kind of "changes need attention" will Spotbot tell me about?
- jmtulloss 11y ago
  Right now it checks your site every 24 hours, and we try to alert on things that might actually need a person to verify (ie, not animations or videos). We're in the process of building moderation tools so that you can refine what's important to you, and we're really excited about building training a model to be able to highlight changes that are probably broken. We're starting at square one though :)
reustle 11y ago
I understand slack integration for some tools, but is this slackbot-as-a-product thing going to become the new fad? What if I want to use your tool without using slack?
- dkopi 11y ago
  slackbot-monitoring-without-slack-as-a-product
  - jmtulloss 11y ago
    Coming soon to Spotbot :)
- crossman 11y ago
  I was thinking the same thing. I'm interested in trying it out, but do I really feel like setting up slack for a side project when I'm the only one that works on it?
  - jmtulloss 11y ago
    You can have it just post to you in Slack if you don't want to bother other people. What's your preferred notification system?
    - crossman 11y ago
      No I mean a personal side-project that I have no slack set up for because I'm literally the only person that works on it. Not just the only person on a project in a company. Anyway, email would probably be fine for me.
- jmtulloss 11y ago
  We plan on supporting other workflows soon, we just started with Slack cause that's what we use. Next stop is probably email. Any other integration you'd like to see?
5vforest 11y ago
We use Percy.io for this and can't recommend it enough. It integrates with your CI workflow, which makes a whole lot more sense to me than waiting until production. (Or to be fair, staging.)
- jmtulloss 11y ago
  Percy is awesome, no doubt. We're a little different than Percy though. We don't require any code. Setup is just entering a URL. Percy doesn't really require code either, but it relies on you having already written integration tests. We do structural diffing, not pixel diffing. Pixel diffing can be challenging to view as a developer since innocuous changes can cause massive visual pixel diffs. With structural diffing we can have a better idea of exactly what changed and highlight those regions. We're definitely going to be integrating with CI workflows and non-production sites as well :)
dkopi 11y ago
I can see this working for static pages, not sure about pages that require input, user login, or display constantly changing data.
- jmtulloss 11y ago
  We're really excited about tackling those problems. @mgc and I founded Spotbot with the thought that models trained with ML techniques can be immensely useful to developers (novel, I know). So much of coding, debugging, and firefighting is pattern matching, and machines are getting really good at pattern matching. That's how we're going to tackle the user login and dynamic data issues in the long term. We have some short term solutions in the works as well. For dynamic content, we'll have a way to ignore changes that are expected and don't mean anything significant changed. For login, we may implement a token API that will allow us to log in with a bit of work on the developer's side. We can also script logins without too much effort if we are provided with test account credentials. Thanks for giving me an opening to talk about all this :)
  - giovannibajo1 11y ago
    My app is a social network. A social network means that the two most important pages (global stream and user stream) continuously mutate because there is always new user-generated content. Now, humans are very good at catching bugs there; if a the name of the user goes above the avatar picture next to it, humans will see it immediately. But how can a tool like Spotbot not to be confused by all the normal changes in user-generated content, and to just spot structural differences?
    - jmtulloss 11y ago
      Right now we're not great at it. However, the way Spotbot works means that we will be decent at it soon. We do a hash of the structure, so we know exactly what kind of changes there are. For something like your page, the image sources, the text, the names, the votes, and other content change all the time. Soon we'll have a way to ignore those changes while still recognizing that other changes need to be vetted. So, for instance, if the padding between the avatar and the user's status changes, we'll still catch that even though we're ignoring the avatar source and the text content. Soon!
jenniferDewalt 11y ago
This looks very useful! I'm wondering if it's possible to see a timeline of changes, like the history of changes for the last 30 days?
- jmtulloss 11y ago
  It is, although it's not the best UX at the moment. In the details about any page that you're tracking, the past snapshot runs will be listed on the left side. You can go to any point in time, view it, and compare it to the previous time something actually changed.
  - jenniferDewalt 11y ago
    Got it! I can see that being helpful when looking at other site metrics.
ldehaan 11y ago
this is cool, but I just wrote something like this at work using hubot and some node libs, and there are a couple extra options you can setup. if they let me I'll release it OSS, but it was super easy to write, and it pulls diffs from the github project when sees a change. I was actually working on a reverse of that, so if it sees a repo get pushed to it will let you know if the site changed or if it didn't. hubot is so cool, but coffee script is just gross. meh. other than that, it's been incredibly useful.
alecdibble 11y ago
How will this work with a split testing tool like Optimizely?
- jmtulloss 11y ago
  Great question. If you're willing to do a little work on your side, you can craft a URL that will guarantee that the user ends up in a particular bucket. One of the features that we're thinking about is doing visual diffing on A/B test variations. Then you can page through your variations and quickly see how they're different from the control, or if they're broken on any browsers. To do that we would need a deeper integration, either with the user or with the A/B service. I think it'd be pretty cool though!
nir 11y ago
Shameless plug: http://shoov.io/ http://shoov.io/
- jmtulloss 11y ago
  Looks pretty cool! I like that the tool itself is open source. We found that pixel diffs were pretty unwieldy for a lot of different scenarios. How do you deal with, say, a div inserted at the top that pushes all the content on the page down 50px?
  - amitaibu 11y ago
    Shoov uses webdriverCSS under the hood, so you can simply remove/ hide/ exclude that div.
rocky1138 11y ago
Why would I choose this over Browserstack?
- jmtulloss 11y ago
  We actually use Browserstack for some of our devices. Spotbot monitors your site over time, so it'll alert you when things change. With Browserstack, you have to manually run your site in all the devices you care about each and every time you want to check something. Or you can write a Selenium test, but that's a lot more painful than just entering a URL and letting us take care of it for you!
  - rocky1138 11y ago
    Got it, thanks!
DocG 11y ago
Hey It seems that spotbot does not cover for delayed page transistion effects in CSS. I have multiple objects appearing in row and pictures show currently each page in different stage and not fully loaded page :( Other than that, awesome :)
- jmtulloss 11y ago
  Thanks! We do try to deal with CSS transitions but I don't think we have every situation covered yet. What page were you trying it on?
tekklloneer 11y ago
I tried it with our product and it took a snapshot before all of the (wall time intensive) api calls finished. Is there a way to configure it to wait a few seconds?
- jmtulloss 11y ago
  Not currently, but this is a common request. What would you prefer: - A javascript object being injected into your page that you could call to indicate the page was done - A setting within the UI that would allow you to configure a wait time
  - alxndr 11y ago
    Instead of injecting something into the page, what about having the snapshotter running an is-it-done-yet callback periodically to see whether it returns true? That would allow some basic checks like "is this particular DOM element visible yet" or "how many entries are in this list" without needing to modify the production code in the page.
  - tekklloneer 11y ago
    I don't really care about all of the calls, but I'd like to say "here's a regex pattern describing a uri. once any connection with uri pattern closes (within 10 seconds), you can render." Yeah, this isn't great for websockets/long polling or other persistent methods, but it's fine for simple ajax-y calls.
aroc 11y ago
Interesting, but this needs to be integrated into CI, not Slack, to be taken seriously by teams larger than just a few people. And arguably, I tend to set up CI of any serious project before it goes to production.
- jmtulloss 11y ago
  What's your CI system of choice?
  - aroc 11y ago
    I really like Circle CI, but I've used (and enjoyed) most of the other services as well.
xerophyte12932 11y ago
What if my website isn't published yet? Is it possible to run this locally for a locally deployed website/webapp.
Gazoo101 11y ago
I can't really say as to whether this is due to load, or some underlying problem, but it seems that - currently - if you just wait around for the system to get to your requested (free) screenshot, it ticks down to 4 Screenshots ahead of yours, and... Nothing. Not a single one of the boxes seem to update anymore. They all just say 4 screenshots ahead of yours. It seems, I spoke a bit too soon. Something is happening now. I guess my cynicism got the better of me when they all hit 4 and just... Stopped :)

179 points by jmtulloss 11y ago

Hi Hacker News! We've been working hard on Spotbot for the past couple months and are looking forward to your feedback on what we've built. @mgc and I will be around all day to answer any questions.

spenserhale 11y ago
Justin, People cannot try free, your servers are responding with 503s. I tried to email you from your contact link on website (hi@spotbot.qa) and was undeliverable.
- jmtulloss 11y ago
  Well the email thing is pretty embarrassing, just fixed. I wrote "hi@spotbot.qa" and set up an email for "hello@spotbot.qa". Face palm. We are experiencing a lot of load (or we were until we went down). We scaled up and things seem to be working better now.
mtw 11y ago
"Pool is destroyed"
- jmtulloss 11y ago
  No longer destroyed!
lstamour 11y ago
The site I'm currently maintaining doesn't change very often, but it's approx. 1,500 pages if spidered, plus a large search index. I'm looking at the Fancy plan, but we've got multiple subdomains or internal versions of the site. How are these covered if it's not all under one domain? Are sub-domains part of the same domain? :) Is it possible to integrate with a CI system for on-demand runs, perhaps by supplying a list of changed pages as a hint on where to expect changes and where not? The idea being that to lower your costs, a subset of pages could be run on-demand?
- jmtulloss 11y ago
  We don't currently have CI integration, but it's coming up fast on the roadmap. What system do you use? We're still hammering out the pricing plans. If you email me at justin@spotbot.qa we can set you up with something that makes sense.
  - lstamour 11y ago
    No specific CI in mind, maybe just Github integration or a webhook to start. I'm currently looking at options to modernize and add tests to a rather manual, SVN-based process. I'll email for sure, I'm interested to see how this will work. Thanks! :)

juecd 11y ago

Super excited to try this — as an early start-up (and the founder in charge of UI), I KNOW our site has UI issues but I don't have the time to do full inspections. What kind of "changes need attention" will Spotbot tell me about?

jmtulloss 11y ago
Right now it checks your site every 24 hours, and we try to alert on things that might actually need a person to verify (ie, not animations or videos). We're in the process of building moderation tools so that you can refine what's important to you, and we're really excited about building training a model to be able to highlight changes that are probably broken. We're starting at square one though :)

reustle 11y ago

I understand slack integration for some tools, but is this slackbot-as-a-product thing going to become the new fad? What if I want to use your tool without using slack?

dkopi 11y ago
slackbot-monitoring-without-slack-as-a-product
- jmtulloss 11y ago
  Coming soon to Spotbot :)
crossman 11y ago
I was thinking the same thing. I'm interested in trying it out, but do I really feel like setting up slack for a side project when I'm the only one that works on it?
- jmtulloss 11y ago
  You can have it just post to you in Slack if you don't want to bother other people. What's your preferred notification system?
  - crossman 11y ago
    No I mean a personal side-project that I have no slack set up for because I'm literally the only person that works on it. Not just the only person on a project in a company. Anyway, email would probably be fine for me.
jmtulloss 11y ago
We plan on supporting other workflows soon, we just started with Slack cause that's what we use. Next stop is probably email. Any other integration you'd like to see?

5vforest 11y ago

We use Percy.io for this and can't recommend it enough. It integrates with your CI workflow, which makes a whole lot more sense to me than waiting until production. (Or to be fair, staging.)

jmtulloss 11y ago
Percy is awesome, no doubt. We're a little different than Percy though. We don't require any code. Setup is just entering a URL. Percy doesn't really require code either, but it relies on you having already written integration tests. We do structural diffing, not pixel diffing. Pixel diffing can be challenging to view as a developer since innocuous changes can cause massive visual pixel diffs. With structural diffing we can have a better idea of exactly what changed and highlight those regions. We're definitely going to be integrating with CI workflows and non-production sites as well :)

dkopi 11y ago

I can see this working for static pages, not sure about pages that require input, user login, or display constantly changing data.

jmtulloss 11y ago
We're really excited about tackling those problems. @mgc and I founded Spotbot with the thought that models trained with ML techniques can be immensely useful to developers (novel, I know). So much of coding, debugging, and firefighting is pattern matching, and machines are getting really good at pattern matching. That's how we're going to tackle the user login and dynamic data issues in the long term. We have some short term solutions in the works as well. For dynamic content, we'll have a way to ignore changes that are expected and don't mean anything significant changed. For login, we may implement a token API that will allow us to log in with a bit of work on the developer's side. We can also script logins without too much effort if we are provided with test account credentials. Thanks for giving me an opening to talk about all this :)
- giovannibajo1 11y ago
  My app is a social network. A social network means that the two most important pages (global stream and user stream) continuously mutate because there is always new user-generated content. Now, humans are very good at catching bugs there; if a the name of the user goes above the avatar picture next to it, humans will see it immediately. But how can a tool like Spotbot not to be confused by all the normal changes in user-generated content, and to just spot structural differences?
  - jmtulloss 11y ago
    Right now we're not great at it. However, the way Spotbot works means that we will be decent at it soon. We do a hash of the structure, so we know exactly what kind of changes there are. For something like your page, the image sources, the text, the names, the votes, and other content change all the time. Soon we'll have a way to ignore those changes while still recognizing that other changes need to be vetted. So, for instance, if the padding between the avatar and the user's status changes, we'll still catch that even though we're ignoring the avatar source and the text content. Soon!

jenniferDewalt 11y ago

This looks very useful! I'm wondering if it's possible to see a timeline of changes, like the history of changes for the last 30 days?

jmtulloss 11y ago
It is, although it's not the best UX at the moment. In the details about any page that you're tracking, the past snapshot runs will be listed on the left side. You can go to any point in time, view it, and compare it to the previous time something actually changed.
- jenniferDewalt 11y ago
  Got it! I can see that being helpful when looking at other site metrics.

ldehaan 11y ago

this is cool, but I just wrote something like this at work using hubot and some node libs, and there are a couple extra options you can setup. if they let me I'll release it OSS, but it was super easy to write, and it pulls diffs from the github project when sees a change. I was actually working on a reverse of that, so if it sees a repo get pushed to it will let you know if the site changed or if it didn't. hubot is so cool, but coffee script is just gross. meh. other than that, it's been incredibly useful.

alecdibble 11y ago

How will this work with a split testing tool like Optimizely?

jmtulloss 11y ago
Great question. If you're willing to do a little work on your side, you can craft a URL that will guarantee that the user ends up in a particular bucket. One of the features that we're thinking about is doing visual diffing on A/B test variations. Then you can page through your variations and quickly see how they're different from the control, or if they're broken on any browsers. To do that we would need a deeper integration, either with the user or with the A/B service. I think it'd be pretty cool though!

nir 11y ago

Shameless plug: http://shoov.io/ http://shoov.io/

jmtulloss 11y ago
Looks pretty cool! I like that the tool itself is open source. We found that pixel diffs were pretty unwieldy for a lot of different scenarios. How do you deal with, say, a div inserted at the top that pushes all the content on the page down 50px?
- amitaibu 11y ago
  Shoov uses webdriverCSS under the hood, so you can simply remove/ hide/ exclude that div.

rocky1138 11y ago

Why would I choose this over Browserstack?

jmtulloss 11y ago
We actually use Browserstack for some of our devices. Spotbot monitors your site over time, so it'll alert you when things change. With Browserstack, you have to manually run your site in all the devices you care about each and every time you want to check something. Or you can write a Selenium test, but that's a lot more painful than just entering a URL and letting us take care of it for you!
- rocky1138 11y ago
  Got it, thanks!

DocG 11y ago

Hey It seems that spotbot does not cover for delayed page transistion effects in CSS. I have multiple objects appearing in row and pictures show currently each page in different stage and not fully loaded page :( Other than that, awesome :)

jmtulloss 11y ago
Thanks! We do try to deal with CSS transitions but I don't think we have every situation covered yet. What page were you trying it on?

tekklloneer 11y ago

I tried it with our product and it took a snapshot before all of the (wall time intensive) api calls finished. Is there a way to configure it to wait a few seconds?

jmtulloss 11y ago
Not currently, but this is a common request. What would you prefer: - A javascript object being injected into your page that you could call to indicate the page was done - A setting within the UI that would allow you to configure a wait time
- alxndr 11y ago
  Instead of injecting something into the page, what about having the snapshotter running an is-it-done-yet callback periodically to see whether it returns true? That would allow some basic checks like "is this particular DOM element visible yet" or "how many entries are in this list" without needing to modify the production code in the page.
- tekklloneer 11y ago
  I don't really care about all of the calls, but I'd like to say "here's a regex pattern describing a uri. once any connection with uri pattern closes (within 10 seconds), you can render." Yeah, this isn't great for websockets/long polling or other persistent methods, but it's fine for simple ajax-y calls.

aroc 11y ago

Interesting, but this needs to be integrated into CI, not Slack, to be taken seriously by teams larger than just a few people. And arguably, I tend to set up CI of any serious project before it goes to production.

jmtulloss 11y ago
What's your CI system of choice?
- aroc 11y ago
  I really like Circle CI, but I've used (and enjoyed) most of the other services as well.

xerophyte12932 11y ago

What if my website isn't published yet? Is it possible to run this locally for a locally deployed website/webapp.

Gazoo101 11y ago

I can't really say as to whether this is due to load, or some underlying problem, but it seems that - currently - if you just wait around for the system to get to your requested (free) screenshot, it ticks down to 4 Screenshots ahead of yours, and... Nothing. Not a single one of the boxes seem to update anymore. They all just say 4 screenshots ahead of yours. It seems, I spoke a bit too soon. Something is happening now. I guess my cynicism got the better of me when they all hit 4 and just... Stopped :)