Opticon 2017: Top Insights & Takeaways From a CRO Expert

Written By: Nick Powell, Customer Experience Strategist for Digital Operative

We sponsored and exhibited at Optimizely’s Opticon in Las Vegas, the leading experimentation conference where creativity and science are combined to drive innovation and transform organizations. Even with nearly ten years of conversion rate optimization (CRO) experience, we left feeling inspired and ready to put the knowledge we gained to good use. Before diving back in to our regular work, I spent some time compiling my top insights and takeaways from the many sessions I attended throughout the week. Enjoy!

1) Building a Culture of Testing/Optimization

From the opening keynote address of Opticon 2017, one message rang true above all others: the importance of and proven success in creating a culture of experimentation. Most experts agree that means continuously testing new ideas and making data and analytics a centerpiece to those activities. The focus at Opticon 2017 was very much on how to build a culture of experimentation from the ground up. The team from Optimizely put together a systematic approach to building that type of culture.

The approach starts with having an experimentation hero, an individual within the organization who is focused on facilitating the excitement and rigor necessary to create a truly dynamic experimentation program. By driving communication, consistency and success, multiple teams can achieve true organizational buy-in and produce a culture of experimentation. Dan Siroker, Co-Founder and Executive Chairman, explained: “This means experimentation is pervasive and becomes a universal business practice and every decision within the business is considered an experiment that needs to be validated.”

Earlier this year, I hosted a webinar on “The Culture of Optimization,” so listening to many of the keynotes and sessions at Opticon 2017 felt a little bit like home to me, albeit from an internal perspective. As an agency, the conversation is a little bit different. Our goal is to be “agency experimentation heroes” and to teach our clients to create this culture from the ground up. The path is not easy, however, with an internal champion and a truly dedicated team like the one we have here at Digital Operative, we can create dedicated strategies devoted to experimentation and optimization that will facilitate our clients' growth into an experimentation program and ultimately help them establish a culture of experimentation.

2) New Optimizely Features and Impacts

Optimizely has several new features that really take the X platform to the next level. Let’s take a look at some of my favorites:

• Stats Accelerator - Automates the flow of traffic to experiments to achieve statistical significance up to 300% faster. This means we can accelerate the time it takes to test and get results and realize impacts, which should be a client favorite!
• Support for Dynamic Websites or Single Page Applications (SPAs) - Until now, these sites have been difficult to test, this new feature enables seamless experimentation within Optimizely X.
• Program Management - Experimentation lifecycle management tool that allows direct clientside collaboration and idea management within Optimizely X.

These new features are exciting to see and really a testament to the folks at Optimizely. With artificial intelligence (AI) experimentation becoming more prevalent in the market, Optimizely has taken a big step by launching Stats Accelerator with AI, of which I’m most excited to utilize and pass this valuable new feature on to our clients.

3) The Conference Perspective

Overall, Opticon 2017 was an all out culture of experimentation event. It was great to connect with colleagues in the industry to share ideas and discuss stories of success as well as challenges faced. My top takeaways from Opticon go like this (no particular order):

• Be fearless in pursuit of customer knowledge through experimentation (no fear of failure)
• Every idea is an experimentation opportunity
• Continue to grow in the craft through research and new ideas
• Strive to grow every client’s experimentation program into a culture of experimentation
• Provide internal key performance indicators (KPIs) for experimentation program success validation (velocity and win percentage)




4) Session Overviews

I attended several sessions at Opticon, which were conveniently segmented into the following sections: Bold Ideas for Growth, Creating a culture of Experimentation, and Data, Developers, and Product Management. These sessions provided an array of information from Optimizely technical usage details to program management concepts. Each session was presented by either an Optimizely technical expert, like Pete Kooman, or from an Optimizely user story, like Ari Sheinkin from IBM. Here are some highlights!




Stats Engine, with Pete Kooman, Co-Founder & CTO, Optimizely




Pete discussed how the Optimizely stats engine uses sequential testing in lieu of the traditional “T” test method. The reasoning for this, as Pete explained, is that the Student “T” test was created in 1908, by Guinness (yes, the beer Guinness). Back then, data was expensive and slow and practitioners were highly trained statisticians. These days, data is cheap and in real-time, and because practitioners are not always trained statisticians, a more modern approach to validation was a logical next step to reduce the likelihood of false positives (which with the “T” test method is ~5% every time you peak).

Sequential testing and replacing type I error control with false discovery rate (FDR) control are used to allow users to confidently check experiments in real time, test many variations and decrease the false positive/negative rates. Stats engine does this by computing a local p-value at different points as well as a global p-value (a running minimum of the local p-value).

The important takeaway is that we can now be more confident in the decisions we make based on Stats Engine results.




Building a Culture of Experimentation, with Ari Sheinkin, VP Marketing Analytics, IBM




Ari discussed how IBM made significant progress through experimentation, however, they weren't able to realize success at the scale of the business. On the path to creating a culture of experimentation, he enabled thousands of employees to experiment with ideas. The way forward was to build experimentation into the fabric of the management and to create new roles and clear metrics that held testing accountable. Ari elaborated on this: “Getting leadership buy-in and sponsorship is only the start. You have to get to each individual and give them a taste of how powerful they can be once they become experimental.” The major theme was to promote risk taking and to be unafraid to break things and test the limits. Ari explained: “This happens from the bottom up however top must be on board. Need to trust the experimentation and if we make mistakes, let's make big risks.”

This session was a good example of how to build on success and how to actualize the required steps necessary to building a culture of experimentation.



5) Overall Perspective, Learnings and Next Steps

Optimizely did a great job of promoting the idea and importance of building a culture that has experimentation at the core of every business decision. I completely agree and will continue to work hard to facilitate that culture for each of our clients and future clients. The power of experimentation is real and this message was reiterated throughout Opticon 2017. Overall, it was a great event to meet experimentation-minded professionals, gather research, and learn about new tools and concepts that we can bring back to our client family to continue to help them grow and realize success.



Contact us to learn more about our approach to conversion rate optimization (CRO) and testing/experimentation!