Categories
Getting Started With Analytics

Lessons from Billion-Dollar Analytics that Indies Might Find Useful Too

Hey there. I’m Blake Ellison, and I’ve spent the last five years at PlayStation building a new analytics program there. In that time, we’ve found a number of patterns that we’ve turned into processes for our team.

I’m going to share some of those patterns and processes here and what they mean for indie software creators.

If you’re an indie, then chances are pretty good you’re not yet at that size. But there are lessons from these practices that you can start with now.

Measure Everything!

First and foremost, Measure Everything! Don’t release something without any tracking installed – it will be impossible to validate its success.

Even in big business, people like product managers make the mistake of shipping their feature without analytics, who then have nothing to show for their feature’s success in front of executives. Like it or not, business decisions (that make or break careers) are made from the cold numbers, not the glowing Reddit comments.

Lesson for Indies: You should commit to basic analytics for whatever you release. There are great tools with free tiers.

If need be, explicitly budget in time for analytics as a component of whatever feature you’re working on.

Get the low-hanging fruit

There’s so much you could track: page views, interactions on page, form completions, e-commerce actions, video playback, user identities… there’s a lot.

But you don’t need every single thing for Launch Day.

Lesson for Indies: You can logically put these tracked elements in a priority order and add to your tracking over time.

You need to know some things more than others. For example, you need to know how much revenue you earned much more than how much of your promo video was played. Likewise, you should track which page the user is on, before starting to do additional work to track what interactions users had on each page.

Most analytics tools will by default send a page view event when the standard cut-and-paste JavaScript snippet is fired, which means you get that “for free” (in the technical sense, meaning you don’t have to go to additional work). It’s a great place to start.

Want a complete guide to what to install and in what order? The Analytics for Indies E-book has it. Sign up to be notified the instant it’s published.

Test Your Analytics Too

Data quality matters massively when your app is going out to tens of millions of users.

Our nightmare scenario is that an app goes out with a bug and then millions of users start sending in junk data. Worse than just shipping a bug, that junk data gets commingled with our existing high-quality data. And it’s nearly impossible to remove the junk data – once something is sent in to your analytics tools, it’s in there.

So, our team performs manual QA for every feature that goes out, because we’re highly incentivized to avoid failure.

Lesson for Indies: Test your analytics capture just like you would test your app’s core functionality. If your analytics were to fail, you’d be getting misinformed about the health of your product and what is most popular/successful. Most analytics tools offer debuggers to ensure you’re sending what you expect to. Test this in your dev or staging environment. Just like your core functionality, if you ship something with broken analytics, don’t be afraid to roll back.

Define Your Data

At scale, we operate a “self-serve” model. That means that hundreds of users throughout the company log in to our analytics systems to find out what it is they uniquely want to know. Whether someone is looking for their pet feature’s MAUs or load time performance, everyone wants to clearly understand the specific meaning of the metrics they’re seeing.

Here’s a practical example: most tools have varying definitions for what sequence of events constitutes a “visit”. Did you know that Google Analytics increments a visit when the clock strikes midnight, even if it’s just really one physical visit spanning across midnight?

At PlayStation, we put more and more time into creating data dictionaries that explain data with human-friendly language.

Lesson for Indies: You have so much going on that you shouldn’t have to keep all the ins-and-outs of your analytics tool in your head. Explore your tool’s definitions in the documentation and bookmark them for reference.

Make a Playbook from Your Process

As part of BigCo life, we had to fit into the Agile-like framework used company-wide.

After much negotiation, we settled on a workflow that broke up our work into discrete phases. This workflow had the benefit of being easily understood by all, and could be reflected in JIRA.

The details of that playbook are too boring to put here. But the point is that the process of adding analytics to a feature has a few building-block steps of work. Those steps can be documented, tracked, and taught to new employees.

Lesson for Indies: If you’re the kind of person who has an organized workflow, or finds checklists productive, you can templatize the addition of analytics to your new feature. This makes the work more predictable and controllable.

There’s An E-Book Coming

I’m writing an E-book to walk you step-by-step through setting up flexible, future-ready analytics for your project. Sign up now for four free tips to get started. I’ll also throw in a free cheat sheet that tells you which tools to use.

This E-book will show you, step-by-step, how to set up great analytics for your project. My experience with billion-dollar apps will ensure that your setup is stable and ready for the future.

The book is in progress, and here’s a preview of the chapters planned:

  • Which tools to use, and how to set up the accounts
  • Setting up basic code
  • Tracking revenue and why you should do it
  • Tracking unique users (in a way that respects privacy)
  • Connecting marketing campaigns and traffic sources to Revenue
  • Tracking errors, so that you know what blocks users from paying you
  • Tracking other interactions – is that video trailer valuable?

In all, you can accomplish in a couple hours what an analytics engineer would charge tens of thousands of dollars for.

By Blake Ellison

Blake has spent the last eight years building analytics practices at PlayStation and Rakuten, and now he wants to help independent software creators with their own projects.