Product teams that launch but can’t measure

‘We want a data-driven culture’ are aspirations spoken by companies ranging from start-ups to established enterprises. Being data-driven is important as it provides an objective approach to decision making, removing any bias or incorrectness that ‘we’ perfect / imperfect humans possess. Having worked alongside or coached over 30 Product Managers has surfaced a common gap in Product Manager data culture: confirming prior to feature build and release that data will be available for KPI measurement.

Product teams now release new features at a faster rate than ever before. New platforms and tools enable this fast paced agile and iterative ‘maturing of the product’, introducing risk that must be managed. This risk is prioritizing the reduction of feature launch cycle time over foundational capability (i.e. measuring KPIs) and balancing the two can be challenging. The team must verify that KPIs can actually be measured before launch — this means all metrics must be fully populated with real test data in the actual data pipelines. Don’t let the hype of ‘the feature is released!’ take precedence over not being able to measure, otherwise why are you releasing the feature?

KPIs likely require joining ‘islands of data’ which result directly from how a product is built: connecting specific services that together comprise a solution. Product and features are run by a collection of SaaS, multiple cloud services (if you’re a large enterprise), structured and unstructured data, real-time and batch data, and the list goes on. However, all of these need stitched together with APIs, lakes and warehouses, and scheduled jobs; plus some manual processes that might eventually graduate to the automated big leagues. Once (in all reality before) the data is organized and in the same environment, careful consideration must be given to connecting it on the right keys (values), at the right grain (level of detail), and with unique records (preventing many-to-many joins). Thus, measuring KPIs tends to be more complicated than not.

SaaS UI will only answer some questions, and ‘hoping’ it will answer all of them is dangerous. The sales rep who says measuring KPIs by simply using the SaaS UI sounds great until you realize this out of the box component can’t be customized to meet the need of connecting disparate data sources. They also have limited customizing abilities to show the exact data visual needed to tell the ‘data story’. Moreover, SaaS UIs are challenged with only having the data they produce (i.e. they aren’t a data lake / warehouse that other data can be ETLed into), unless someone like Facebook is in your architecture and ‘custom events’ are incorporated and tracked. Yes, looking at one secluded part of the business and solution is possible with SaaS UI, but a good Product team must evaluate the entire User Journey of data.

Can Excel be the answer to my KPI problems? While Excel is the most common business intelligence tool, it also has short-comings. Excel should only be used for proof of concept, ad hoc, and some small scale analysis. You will see some teams try using Excel to join together data sources, transform data, and the list goes on. This is because Excel has a strong brand, large user base, and tangible intuitive interface that places it as one of the most popular ‘business intelligence’ tools. But always remember, Excel has a row limit of 1,048,576 and once that daily manual process is in place, you’ve reduced time available for net new initiatives.

Including data requirements in 3rd party contracts is often overlooked. A 3rd party in this context carries out tasks between the product team and customers (B2C) or another business (B2B), such as delivering physical mail marketing content. It is natural to have the contract focus be on operational requirements, however including ‘data return’ requirements is very important. Including ‘data return’ requirements helps orient the vendor with what is important. The 3rd party is naturally incentivized to carry out tasks / push forward content to the intended recipient and is not incentivized to return data back to the product team. This results from a 3rd party considering the job done once the content is delivered to the ‘customer’, however 3rd party companies must remember they have customers on both sides, i.e. ones they push content to and ones that pay their bill.

When in doubt, pass along all data. Data engineers do just as their title suggests: engineer the movement and modification of data. While doing so and when scripting their code, the decision of which data is necessary must be made. This decision may get input from others but the prevalent fast-paced cultures make it difficult to thoroughly get input from a full cross-functional team. To account for the requirements of these missing inputs and given cloud processing and storage capacity is inexpensive, always default to passing along as much data as possible. This general approach also reduces the risk of being able to measure new insights arising from product maturity in ways that were unknown prior to launch.

Companies helping solve part of this problem include the usual suspects of Azure, GCP, AWS, MongoDB, Adobe, Amperity, Amplitude, Matillion, fivetran, Alteryx, Jupyter Notebooks & Dataframes, Databricks, and the list goes on. However does your Product and Data culture prioritize subjective measurements?

Your Product success KPI(s) are important, but the tactics to measure those KPIs are more important.

Trust but verify and god speed Product Managers…may someday we all get the data we didn’t know we needed.

Putting square pegs in round holes since the 80s.