After a recent presentation I got a follow-up question from an enthusiastic member of the audience about ‘measuring’ in the context of Lean Agile Product Development. Below is the question and my answer with anonymity preserved:
You mentioned measuring quality and tracking defects in your presentation about Lean Agile. I would love if you don’t mind getting more of an insight in how you do this?
Do you implement measures of success on features you deliver?
The key points you discussed was to take the opportunity to do better and to brainstorm to do better so this motivates the team.
We have implemented measures of success on features but where we have failed is the team feel
a) they are spending too long at calculating a target figure or
b) that it’s just an arbitrary figure so why should we track it against should we just take a check point to see how it is doing?
What we have found is that we are all in agreement that we should be taking the opportunity to see how that feature is doing and if it can be improved.
It would be great to get your insight’s on this.
Product Manager’s Answer
You need to acknowledge that in this scenario your App is just a channel for your company’s Product and having a great App might be responsible for maybe 30% of the overall success of the Product and there are several other factors that have a very important bearing on the overall business success also such as Customer Care, Product Offering etc.
Having said all that, from a Lean perspective you want to try to measure something that correlates with the overall success of the Product and the Business as much as possible.
App Store Ratings
When I was at MTT we used ‘5-star’ App Store ratings as the key metric for all staff. While it is extremely difficult to achieve a 5-star rating for an Airline App, using a common shared metric for the whole company had the effect of aligning everybody in the organisation across multiple functions to achieve the same ‘successful’ outcome. In this scenario the Mobile App is just a distribution channel and not the actual Product that the Airlines are selling, the actual product is the seat on the plane, or more accurately the passenger’s transit from one location to another.
While there are problems with App Store ratings, at least its a constant metric that you can track each quarter. By tracking a broader trend over a longer time period you can take actions involving all functions in the organisation to resolve customer issues and thereby improve your ratings. This aligns with 2 key Lean principles: ‘focus on customer value’ and ‘optimise the whole’.
Considering the ‘optimise the whole’ paradigm in a real world scenario we can think of an example in our own businesses – imagine you’ve created a super-efficient and hyper-productive, cross-functional Product Development team who are doing great measurement, tracking and continuously improving what they do. A few short desk rows away however at the company’s Legal Department the people working there are still using archaic work practices from the 1980’s and refuse to change or adapt. In this scenario the great output of the Product Development team will be constantly blocked by the outdated Legal team and the product, customers and overall business will suffer as a result.
The challenge here is to persuade all of the other departments to buy into the same transformational change by adopting shared metrics that are aligned with all the other teams in the business.
What get’s measured get’s done! (Drucker, Peter)
What if you don’t have App Store ratings every month or quarter?
Noel Tate gave a talk at the last ProductTank event and he talked about using Net Promoter Score to track customer satisfaction. He was able to use this as a barometer of the overall ‘success’ for end-users of the Product/Channel and he successfully brought up the score through a series of initiatives.
You can watch Noel’s excellent talk here: https://www.youtube.com/
Product Owner’s perspective
For a PO working within a Product Development team measuring Story Point velocity and number of Bugs is very important to know where you’re at.
Tracking velocity is a very useful metric for the Product Owner to help with release planning, it shouldn’t be used for classic Project Management style exercises though.
Project Managers always try to slice and dice the numbers and come up with some really dumb ratios that just don’t work, units of labour, points per resource etc.
Never trust anyone who refers to another human being as a resource! 😇
Note: The goal should not just simply be to do more story points, you have to track quality and defects and Live Bugs, the velocity should be brought down to a level where quality is acceptable and predictable, it’s a direct trade off between the speed and quality but bear in mind too that it can take 6-8 weeks after a release for all the Bugs to become known.
Story Point allocation
An important tip on tracking Story Points is that you should only assign Story Points to User Stories, these User Stories must also have acceptance criteria.
Tasks, Bugs, Spikes and everything that’s non-functional must have ‘0’ Points assigned and only have a time estimate associated in hours. I have seen that a lot of Dev teams count in Story Points for all kinds of stuff that is not of any real value to the Product Owner (or End-Users) and this inflates their team’s velocity and distorts the whole picture really because we’re meant to always focus on customer value as our first principle.
Story Points == Customer Value
Being really scientific about Sprint planning, breaking stories into sub-tasks and estimated in hours is a pre-requisite for high performing teams. Everybody takes ownership and responsibility for the estimates and for the commitment to deliver.
Here’s a Sprint Plan template that I created to help the Sprint Planning meetings to go a bit quicker: Macdara – Sprint Planning Template
The team need to have done prep work in advance so most of the items are pre-populated before Sprint Planning and then just the final estimates are agreed as a group with discussion and some of the items are changed as new information emerges. Some Stories and Tasks may also be rejected at this point but ideally these have already been identified before Sprint Planning.
Once the spreadsheet is all done in real-time at the Sprint Planning session the Scrum Master can the go into JIRA afterwards, updates everything and then starts the Sprint.
Also please refer to Noel Tate’s video and his 3:2:1 ratio of 3x Stories : 2x Tech Debt : 1x R& D Innovation for a good steer on sustainable velocity.
Commitment Ceremony & Retro
It’s very important that the Product Development team commits to their estimates and Sprint goals, reinforcing the value of measuring things and tracking them with precision and rigour.
The Sprint Retro is a forum to address problems that prevented the team achieving their Sprint commitment and the Team Lead needs to take this seriously and make a big deal about dropping stories and adjusting the commitment or unblocking the root cause of delays in order to preserve the team’s morale and belief in the system.
Accountability and Trust
As Product Managers who believe in Lean Agile we hand over trust to the Product Development team to do their estimates, make decisions about what’s feasible and have full control over implementing the agreed solution approach (The team includes Dev, PO, QA, UX roles etc.).
The Product Manager lives mostly in the ‘Problem Space’ whereas the team lives mostly in the ‘Solution space’ but obviously its highly collaborative and there’s some cross-over always.
“In God we trust, everyone else must bring Data”. WE Deming