17 ms·
What is A/B Testing?
- deleted 13y ago[deleted]
- paraschopra 13y agoThanks for the feedback. I'm the CEO and I'm sorry to hear your feedback. Could you please detail on your feedback? It will help us a lot.
- fekberg 13y agoI think you should re-read what he said.
- zachlatta 13y agoVery neat. Hate the loading screen though. Having to display a loading screen is almost as bad as using flash. Maybe load the assets in the background?
- paraschopra 13y agoActually, we load first 4 screens at first go. Rest is lazily loaded.
- mute 13y agoIt's similar to animation at http://icelandingpagedesign.com http://icelandingpagedesign.com
- siddharthdeswal 13y agoThat's a beautiful landing page. Assuming you work there, how much time did it take to build?
- mute 13y agoAround 6 months. It's inhouse project and I spent as much time it was required to make it good. No idea how many actual hours has been spent.
- chinchang 13y agoYeah..this is just more with images.
- Illniyar 13y agoGreat, another example of a site that is vendor locked to the Chrome browser. Considering your site doesn't work on the majority of browsers (Neither Firefox nor IE), I'm not sure what "testing" you've actually done.
- ankneo 13y agoWorking fine on Firefox 22
- rythie 13y agoWorking fine for me in Firefox 17 (ESR)
- ancarda 13y agoIt also works fine in Safari, although that's likely because of the -webkit- CSS prefixes.
- pre 13y agoFails when Ad Block Plus is running. Don't have time to figure out which bit Ad Block is blocking though I'm afraid.
- 13y ago
- ad93611 13y agoThe animation is very good. Could you explain a bit about how this was done? Did you have to write the code or is there a tool that can generate a page like this?
- bhaisaab 13y agoHeya, this was done by one of our talented interns Kushagra A. (https://twitter.com/kushsolitary https://twitter.com/kushsolitary). We did not use any parallax page making tools, it was written using HTML5, CSS3 and some js library.
- _kushagra 13y agoThanks! I used skrollr.js [1] to create the parallax animation and some JS for scroll handling. I will write a blog post [2] on how I made it in detail soon. [1] https://github.com/Prinzhorn/skrollr https://github.com/Prinzhorn/skrollr [2] http://kushagra.me/blog/ http://kushagra.me/blog/
- akshxy 13y agoGreat job Kushagra!
- alanmeaney 13y agoReally nice work Kushagra - I'm looking forward to your blog post.
- nbouscal 13y agoOne problem for you to be aware of: something about the scroll handling breaks the two-finger swipe Back functionality on my Macbook. It's a minor thing, but very annoying.
- retube 13y agoIE8: "There is a problem with this website's security certificate". Also doesn't render at all well...
- mtrimpe 13y agoReally love the animation, but I can't help but wonder if A/B testing couldn't be explained more powerfully. Within the example of the store for example, the real pain A/B testing addresses is cyclical variation leading to incorrect conclusions. Working from that you could've shown the story of: * the store owner rearranging his display and seeing sales go up * but then the rest of the year sales actually going way down * and when he changes things back to the old layout sales went waaay up again! How can that be? * Cue image of cyclical charts and the facepalm of realizing it wasn't his redesign at all! * Now ... on his website he can send visitors to different versions at the same time! * So he'll never have a year of bad sales again!
- siddharthdeswal 13y agoThose are good points, but we were gunning for the marketer who isn't familiar with A/B testing. This because we see that a lot of our potential customers have usually heard the term somewhere but need a basic, easy-to-understand explanation. Do you think it achieves that purpose?
- e12e 13y agoPerhaps you should deploy an alternate version, and meassure conversion rates? ;-) Seriously, I think you make a great point of selling your service. I also think you gloss over things like cyclic variations -- but I'm not sure it's your job to educate people in the subtle art of fact oriented market research (or what one should call it). For those that don't have the time to learn, changing it probably wouldn't matter. For those that do -- chances are they probably will find it out from a more comprehensive resource (like, say, a good book).
- gbrindisi 13y ago> (like, say, a good book). Can you suggest one?
- siddharthdeswal 13y agoExactly. For someone who's just starting with A/B testing, this serves as a good/interesting introduction. But for someone who really wants to understand the detailed behind-the-scenes, this will obviously fall short. Coming to your suggestion, do you have a good book/website in mind?
- philfrasty 13y agoI personally find it „too much effort“ to scroll down all the way. Nice story, but make it somehow shorter and choose a storyline that is familiar to people so they only have to concentrate on the A/B-aspect because otherwise the story is distracting. Would be great to get some feedback how the site is doing (compared to previous versions).
- paraschopra 13y agoWe do have statistics that 60% of people scroll till the last slide.
- sanju 13y agoawesome animation :) love this way of explanation compare to a video cast
- huhtenberg 13y agoStars of David coming out of the "A/B-Testing" explosion - I bet it's some sort of clever innuendo, but it just seems to escape me :)
- avk 13y agoLikewise, I found it distracting and off-putting.
- felideon 13y agoI'm well aware of it and didn't even notice it the first time. Why would it be off-putting, though?
- goatcurious 13y agoCertainly not a clever innuendo. Looks more like an oversight. There isn't much awareness about Star of David in India. It is the easiest star to sketch -- even my teacher in elementary school used to use the same star in my grading.
- paraschopra 13y agoHmm. We weren't (and aren't) aware of Star of David.
- winter_blue 13y agoFYI it's the flag of Israel...
- deleted 13y ago[deleted]
- felideon 13y agoMaybe the shop owner sells... gefilte fish. :)
- TamDenholm 13y agoI hate to be "that guy" but you've got a spelling mistake. "Fishes sell volumes". The plural of fish, is fish. "Fishes" would be the possessive.
- siddharthdeswal 13y agoMultiple species of fish are usually called "fishes". We used this explanation to verify http://grammarist.com/usage/fish-fishes/ http://grammarist.com/usage/fish-fishes/ Not sure if it is correct though. Do you have a link to a better explanation?
- TamDenholm 13y agoWell consider me corrected. I wasn't aware of that. Learn something new every day.
- ahoy 13y agoThis is correct. You'll also notice the word 'peoples' used when referring to multiple groups of people.
- mortov 13y agoSuch as in The Peoples Front Of Judea and The Judean Peoples Front
- axaxs 13y agoDelivery I think was spelled wrong in one of the slides. I can't be bothered to go back and verify though, but may want to check it out.
- alexcroox 13y agoNice work! We did something on a smaller scale at the bottom of our homepage (with live stats on users choices) http://sidigital.co/ http://sidigital.co/ (you'll have to wait for the liquid to reach there to see the effect of your choice)
- BuddhaSource 13y agoVery neat. It kept me glued till the end.
- chinchang 13y agochecked this a while ago. Clean work!
- IanCal 13y agoWhen say "Bob knows which version brings him more sales", I'm curious about how you extract the relevant information from someone to know if it's a statistically significant change in a user friendly way. Do you find people understand this well? What about multi-variate testing? How do you explain the increase in the number of visitors needed to make a decision? Finally, how do you reconcile giving up-to-date information without compromising on the results? (keeping an experiment going until it's statistically significant is something I can see users doing)
- siddharthdeswal 13y agoThe tool has that built-in. As soon as significance is achieved, we show a "winner", so to answer your question, instead talking about statistical significance we focus on the more positive aspect of an A/B test. Those not initiated in Statistics find this a little difficult to understand. I personally go about it like this: Me: "If you flip a coin 10 times and it shows heads 7 times, can you say with certainty that the coin is loaded?" Them: "Umm, no not really" Me: "Ok, what if I flip it 100 times and it shows head 70 times?" Them: "Yeah maybe" Me : "What if 7000 out of 10,000?" Them: "Of course it's loaded" And that's how I explain the basic concept of statistical significance, but without the math. More data means it becomes more 'obvious'. To take it a notch further, I ask "If your life depended on it, would you still bet that the coin is loaded?" People usually become unsure after this and I joke that their previous decision was based on 95% significance, but if they really want to be sure, they should wait for 99% significance.
- Ovid 13y agoAs soon as significance is achieved, we show a "winner" If you've described that accurately, you're hurting some of your customers. It's not well-known (though it should be obvious, in retrospect), but a test can produce a false positive when, in fact, it's a negative. The problem is that there is always noise in customer data and sometimes you'll randomly have enough conversions such that one of your tests will show a winner when it's not. Matts Einersen has some simulation code to show this: http://www.einarsen.no/is-your-ab-testing-effort-just-chasing-statistical-ghosts/ http://www.einarsen.no/is-your-ab-testing-effort-just-chasin... Evan Miller has a more in-depth (but harder to follow) discussion here: http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/how-not-to-run-an-ab-test.html If, in fact, you show the winner as soon as significance is achieved, it's then trivial to demonstrate that some of your customers will lose money as a result. However, if you take this knowledge and then find a good way to incorporate this into your tool, you may have another great selling point that competitors do not (that being said, the idea is complicated enough that some customers won't understand it).
- modernerd 13y agoWhat motivates web designers to tell stories by repurposing the scroll mechanism instead of, say, using video? This type of "scrollmation" feels incredibly clunky to me. I have never got to the end of one. If you need that much animation on a page to tell your story, isn't it a good sign that video might be more fit for purpose? Video was designed for storytelling, it will work on most devices, and it has well-established UI and conventions, such as a play button and progress bar. If you're going to provide a play button in your scrollmation to automatically advance the action anyway, you've already recognised how irritating it is to manually page through an animated story by scrolling or swiping. So why then go to the trouble of creating a scroll-based animation that repurposes a 20+ year old convention and attempts to reinvent a tiny proportion of what video has to offer? Is it just for the novelty value? Is it because it makes you feel smarter? How is it better for users trying to understand your story? I'd love to hear from anyone who's considered both video and scrollmation and chosen the latter.
- GoldfishCRM 13y agoI think scrolling is better in the way that it does not aquire audio and user can controll the speed of information. This part is boring and I get it, lets skip to the next part. Video has no way for that.
- modernerd 13y ago> I think scrolling is better in the way that it does not aquire audio That's a good point. There is a general expectation that playing a video also means playing audio, and some environments – like offices – aren't great for that. > ...user can control the speed of information. ... Video has no way for that. I frequently skim through video by dragging the head in the progress bar, or skip three or four minutes ahead in longer videos. With scrollmation, the default scroll speed is normally much slower than the equivalent video speed of 30fps, it results in jerky movement, and the speed of the video is directly proportional to how hard I am prepared to work to make it go faster. Good UX designers don't make users do shit work.
- srikrishnan 13y ago
- petenixey 13y agoI love it that you've produced this and I think the simplicity of it is wonderful but I fear the story could leave someone a bit confused. It's confusing first of all because you've chosen a product which people don't generally sell online - fish. If you'd chosen bikes or clothes it would have made sense. The other thing is on the "play" version - it takes about half the animation before you even get to the online bit and when you do, the illustration of how A/B testing actually works only takes up a tiny bit of real estate. For my money I'm not convinced that the offline analogy really adds to it. You know your market better than me (or anyone else here except Pete and Dan) but I feel like you could jump straight in with a story about an online store. "Mike sells bikes online but he's not sure which ones to put on the front page. Will he get the most sales from showing someone bikes on the frontpage or showing them chainsets? He doesn't know and he finds it hard to read his analytics to figure out which one is best. VisualWebsiteOptimiser does all this for him..."
- paraschopra 13y agoThanks for the feedback. Yep, we had debated whether fish is the right product to be sold online. We concluded that probably the choice of what product is sold is not as important as demonstrating the actual confusions during the sales process (both online and offline). We considered selling books, cakes and some other products but with fish, the options were limited and very clearly distinct.
- vinhboy 13y agoAgreed. First off, I was distracted by that slideshow because I kept thinking, "Who the f sells fishes online?" Second, I felt like there were two points in the video because of the whole offline analogy. 1) Selling online is better than offline 2) A/B testing makes you more money
- TwistedWeasel 13y agoApparently you can buy fish online. http://www.pikeplacefish.com/buy/ http://www.pikeplacefish.com/buy/
- speeder 13y ago
- mkhalil 13y agoThanks for this.
- paulodeon 13y agoScrolls Down...Meet Bob...Goodbye Bob...Clicks back
- gordondevoe 13y agoThis was awesome. Great work!
- zekenie 13y agoAre there jewish stars that appear when AB testing is introduced? No issue with this, it just seems a little odd. It reminds me of Hannuka at my parents house! :)
- alxbrun 13y agoJust when you thought Powerpoint was finally dead at last, it comes back disguised in scrolling.. Damned !
- fny 13y agoWhile I appreciate the responsiveness, the page is obnoxious to navigate on a mobile device.
- dbg31415 13y agoSorry, why does this warrant an animated deck? Sheesh, next up, "How to use gravity to fall over."
- nixarn 13y agoA bit off topic. But I've had problems with the visualwebsiteoptimizer, not as a customer but as a user on websites using their system. At two different occasions websites have had problems to render, no matter how many times I try to re-load them and wait. I've then checked the webdevtools to check what's wrong and it's been the websiteoptimizer's script that's been blocking the whole site.
- paraschopra 13y agoI'm sorry to hear your feedback. Was it a recent problem? Did you contact our support? In fact, we're the only A/B testing tool out there with asynchronous code snippet that NEVER blocks the website. Here are more details: http://visualwebsiteoptimizer.com/split-testing-blog/asynchronous-code/ http://visualwebsiteoptimizer.com/split-testing-blog/asynchr...
- nixarn 13y agoIt happend a while back, maybe 3-4 months ago. I didn't contact support. If if happens again on some site, I'll do it. But the script was blocking, it was waiting for the js file to load, and blocking the site while doing so.
- mindslight 13y agoWhat is A/B testing? Local optimization with an assumption that every user is stateless and identical.
- rajington 13y agoBeyond the comments over the presentation style, the metaphor, and browser compatibility I want to see more about your specific app. I like how you did the VWO interface on top of his website, you could easily change the line graph section to show the "Intuitive Reports" feature. Line graphs are easy enough to understand so by including more of your value add features it will still stay simple enough. I know it isn't specifically A/B testing related, but throwing another shoutout to your app at the end like "heatmaps too!" might be nice. It seems people that would look at this slideshow might not be familiar with any web analytics services, and it seems you guys offer many services.