A full-fledged detailed LeSS case study of Poster POS Inc. that happened in times of COVID-19 and the war.
Part 4: "Twenty LeSS Sprints"
First Twenty LeSS Sprints. Observations and Lessons Learned
Perception of Speed - Rushing Makes You Slower
Going Slower is Going Faster
Overall Flat Product Backlog
Product Owner’s Team
Remote Meetings and Multi-Team PBRs
Product Support Process
Engineering Managers Doing Gemba Coding
Managers as Scrum Masters
Engineering Community and Pain of Long-Lived Feature Branches
Lack of Feature Toggling
A Pull for Change
Evolution of the Definition of Done
Feature Onboarding in DoD 1.0
Feature Onboarding in DoD 2.0
On the Path of Continuous Improvements
The Path is the Goal
P.S. Russian Warship, Go F… Yourself!
References
First Twenty LeSS Sprints. Observations and Lessons Learned
Organizational change is not a project with a clear end-date, but rather an ongoing journey towards perfection. This chapter highlights different aspects of the Poster journey (relevant to the LeSS adoption perspective) – the things that have been evolving and dynamics that have been observed during the first six months after the Flip Event.
Perception of Speed - Rushing Makes You Slower
This chapter is a follow-up on the Initial Product Backlog Refinement chapter, analyzing the consequences of the Initial PBR run at the LeSS Flip event.
Eventually, it became apparent a few days in the first Sprint that the selected items had high uncertainty, and some of them just exploded with more work. Obviously, we didn’t invest enough time in the Refinement of those items. And eventually, the Product Owner had to re-prioritize the work mid-Sprint and the teams had to adjust the plans and do some extra coordination activities to redistribute the shared work.
Was it a disaster? Yes and no. Our excuse was that the failures are only failures if they come with no learning. And the learnings of the first Sprint due to lack of PBR activities were immense:
The teams realized that no matter how good is the Sprint plan, it is vital to keep constant coordination because the teams and re-balance the work as the things will likely go wrong due to the reason that could not be foreseen.
The teams had started to practice the most direct way to do the coordination – the so-called “just talk” technique, which required no managers and was merely the teams’ initiative to do the coordination.
The teams had also experimented with the LeSS practice of a Leading Team, where one team would volunteer to start some complex work. Typically, they would do that by biting off a small piece to learn more about that work. Once some clarity was in place, that leading team would be facilitating further Refinement activities, sharing with the other teams their findings, and eventually, coordinating with others for the shared work. This way, three out of six teams were working on very related PBIs already during the second LeSS Sprint.
And finally, the importance of the PBRs: the teams agreed to start investing 4–6 hours weekly on the PBR activities, learning to do it right, not just fast. (The reader may wonder why we’ve chosen a weekly cadence of PBRs. Please refer to Remote Work and LeSS for the details.)
On the positive note, despite all the mentioned shortcomings, the somewhat condensed PBR and Planning activities of the first LeSS Sprint, Poster did send a strong message to everyone, that can be summarized as the following:
we are here to learn to work together, and it will be different,
so no matter how much time we spend analyzing and preparing, it is impossible to have a precise 100%-clear uncertainty-free Sprint plan,
hence, it is totally fine to keep talking and learning from each other within the Sprints, re-adjust the plans when necessary and keep improving our understanding of the product, the work to be done, and the process.
Looking Retrospectively at how the things started at Poster, as a LeSS coach and Scrum Master for the next Sprints, I was pretty happy with those early failures and the resulting learning curve that Poster was on.
Going Slower is Going Faster
It was already clear during the 1st LeSS Sprint, that for the first time in the company’s history, several teams were able to join forces and work together as one on a single and most important business objective. And it was despite the drawbacks of the shortened initial PBR (described above). And over the upcoming Sprints, all the six feature teams were actively contributing to a small set of complex product changes (all related to upcoming changes in the Ukrainian legislation law).
That became possible because:
Transparency: We had a single PBL that got constantly re-prioritized by the PO, and it became apparent to everyone who nothing is more valuable now for the business than the Legislation topic.
Shared work: During the first Sprint, one team pulled the work related to a topic that was new for everyone in the company. They took a bite out of it to see the underlying complexity. Later on, that team served to other teams as a Leading Team (a LeSS practice) serving as an informed helper, and a coordinator, and enabling other Teams to join forces on that.
Shared knowledge: Regularly the Teams were running multi-team PBRs (with mixed groups from each team) that allowed for knowledge to start spreading and enabled Teams to work on what’s important, not on what’s comfortable to them. (The reader may wonder why we’ve chosen a weekly cadence of PBRs. Please refer to Remote Meetings for the details.)
Indeed, there was something to celebrate: the promise of LeSS (high adaptability – low costs of change) was demonstrated already during the very first Sprints.
No doubt, that was yet the only beginning of the journey to mastering LeSS, with deeper changes to be realized. But one thing stood out clearly at the very beginning: Poster, as a company, was now able to focus more teams to learn and work on the high-priority items than ever before in the company’s history.
About a half year later after the 1st LeSS Sprint, at a webinar with Poster employees on LeSS, they were asked by the audience if LeSS accelerates development. Obviously, it is a wrongly formulated and confusing question because, firstly, speed is not an independent variable in a complex system of product development. Secondly, it can improve or degrade over time due to various factors with delayed effects. But what is most important to keep in mind is that the performance (throughput) of an individual developer or a team doesn’t necessarily correlate with the company’s overall throughput. Large systems exert emergent behaviors that can’t be determined by behaviors of its parts. This question is a confusion raised from applying manufacturing ideas to product development. Still, people ask this question a lot.
Looking back at what we have observed at Poster, we could conclude that LeSS made it possible for all the teams to know the global priorities and collaborate on them. What is at the top of the Product Backlog can now be worked on by several teams together. This positively affects the performance of the entire production system.
Had not Poster changed its approach (switching to feature teams and to a single Product Backlog), likely, only a single narrowly specialized subsystem team would have been able to work on that top-priority legislation topic (or event worse – several teams would have worked on it sequentially). To make the things worse, it would have been only a specialized team that had later to support customer issues related to legislation (see Product Support Process for more details on Poster’s approach to support in LeSS). Thus reducing its development capacity to accommodate the fixes. In such an organization, there would have been just one team learning the legislation subject and getting more and more specialized in it, becoming a bottleneck.
So yes, LeSS can speed up things – it facilitates and accelerates the learning. But what about delivery? Was the legislation topic worked on faster than it would have been without LeSS? That depends on what gets measured.
It is not that a LeSS company is producing more code, more features, more output. It is that such a company can now be steered much better to focus on what’s critical. And that stronger focus bound with the abilities of feature teams to learn to work on any product part – it is those things that make such an organization better at learning and hence delivering what’s important.
LeSS adoption at Poster also raised the product quality bar (via extending the Definition of Done). So with LeSS, the Poster developers now were involved in much more requirement clarification activities (PBRs). They did it in collaboration with other ‘downstream’ departments (e.g., feature onboarding, support), producing client documentation and writing more automated tests (again, DoD). They were constantly learning prior unknown pieces of the product and codebase (due to the cross-component nature of feature teams), discussing cross product concerns such as the legacy product architecture and user experience issues (due to the whole product focus)… Clearly, the developers (and the teams) at Poster were doing more work on each feature than before in order to bring the features to a higher level of “done-ness”.
So, do LeSS developers work faster? No. They work slower. They work slower because they are learning to work righter. And working righter results in working faster, in the long run.
Overall Flat Product Backlog
Already during the first PBRs, one issue became very prominent: the PO and his team were not yet sure how to approach the idea of a single Product Backlog (PBL). In theory, from the system thinking viewpoint, the idea of having one PBL for all teams made a lot of sense for them. But once we started working with the PBL, we realized it was, in fact, a multi-level set of backlogs.
The illustration below depicts how the backlogs and their items were structured.
This is how NOT to work with a Product Backlog:
Prioritization on epic-level and multiple sub-backlogs for each epic’s PBIs.
As outlined in the illustration above, the Product Backlog consisted of so-called “epics” (big batches of work aka projects) and each of them had its own sub-backlog where the smaller PBIs were stored. That obviously didn’t allow for prioritizing PBIs across epics. But more over – it was creating unintentional negative dynamics described below.
Iterative and incremental development implies, that one can work item by item, delivering one after the other, inspecting on the fast and early feedback, and adapting to what to get built next. This allows for fine-grained prioritization and constant deliver of high value.
Grouping and gluing the items together, no matter how logically related, impedes this process. Working in these big batches (“projects” or “epics”) creates a false understanding that the entire batch needs to be completed before it can get shipped. Thus invalidating the idea of fine-grained prioritization and slowing down delivery and learning flow.
Since the big batches mean big investments, that calls for tighter control. So, often in such an environment the teams will be asked to provide more accurate estimates and, in its turn, do big analysis and design upfront – that extra work will also make production more expensive. And the bigger the batches are, the more it is so.
The level of uncertainty in a complex domain, such as large-scale digital product development, is inherently high. And batching up the work only increases that uncertainty (i.e., more work with delayed feedback leading to more uncertainty). So, the practice of batching makes it even harder to predict the outcomes of work with any reasonable levels of accuracy, no matter how much time is spent (wasted) on upfront analysis and planning.
So, it is no surprise that no matter how good plans are on the paper, occasionally they would turn out wrong. Those “failures” would make management typically put higher pressure on teams to come up with “more accurate plans” and “more stable architectures”. Thus, making the teams spend more time planning, analyzing and speculating over architecture and estimates – instead of building, shipping, and learning on real customer feedback. All in all, such dynamics creates longer development and feedback cycles and minimizes organizational learning.
In such an obscure environment, managers can only rely on the old-school, heavyweight management techniques such as milestone-driven development with resource allocation and planning. And this added bureaucracy just adds more complexity. And now the organization is in a vicious cycle.
The fear of being wrong with the plans and the added bureaucracy makes everyone focus more on plans and the ever-growing policies than on how to do plans better. In such a “plans creating” organization, everyone is allowed to work only on what has passed the defined planning gates – on what is known, planned, estimated and approved. That kills innovation that must be at the heart of any successful product development.
The downward spiral described above are some reasons why the agile movement calls for iterative and incremental development.
At Poster, we have noticed that dynamics and had to coach the Product Owner and others to restructure the backlogs. And the only way to avoid the above described dynamics was to avoid thinking and working in batches. Structurally, we had to get rid of the epics and create a true single-dimensional Product Backlog with prioritized fine-grained items that would guide iterative and incremental development. We had to reconfigure our backlog management tool (see the chapter on Tooling). Luckily, our tool was flexible enough not to dictate how backlogs must be organized (that’s why, try avoiding Jira and tools alike that prescribe batching).
The true single flat Product Backlog with granular prioritization on PBIs from different themes.
We solved that issue of batching, but created a potential new one: how one doesn’t get lost in a flat Product Backlog for the entire product? And how can one Product Owner manage it? There were several things from the LeSS guides and our experiments that we tried – and with positive outcomes:
We went from batching with epics to tagging individual items with themes to mark which part of the product they belong to.
We preferred slightly larger PBIs (than one would use in a single-team Scrum). This helps to keep the cognitive load of managing the Product Backlog at a feasible level. On average, during the first 10 Sprints, we had around 10-15 PBIs in work every Sprint for all the teams. We were typically refining items for two next Sprints. So adding this all up, and we got 30–45 items on top of the Product Backlog that needed to be understood and prioritized - a manageable job.
Those things together allowed for fine-grained prioritization across the single Product Backlog, that was a door to iterative and incremental development.
Product Owner’s Team
The needed change to the backlog structure (see the chapter above on the Overall Flat Product Backlog), described above, had another systemic root cause. As the people, who before LeSS had been team-level Product Owners, now, in LeSS, were seemed to be losing their power and wanted to get a grip back on to something: if not on a team’s backlog, then at least on an “epic”. If not an “epic”, then a piece of a process.
This chapter described the challenges of the ex-team-level Product Owners (that I have a lot of compassion for) as well as their individual transformational journeys (that I really admire)
As mentioned before, for clarity and to avoid confusion between the term “team-level Product Owner” (as such a role doesn’t exist in LeSS) and the “real scaled” LeSS Product Owner, the former is referred here as “Team Output Owner” (or TOO for short). See the chapter Product Management before LeSS for more analysis of this role and its related dysfunctions.
So, the TOOs at Poster before LeSS could be divided into two main cohorts:
Software developers with extra responsibilities of prioritizing a backlog of a component team.
Business-oriented specialists with domain expertise and a responsibility of prioritizing a backlog of a subsystem team.
See the chapter Product Development before LeSS for analysis of the teams at Poster before adopting LeSS and feature teams.
The 1st cohort of the TOOs were experienced developers, who at some point were given a few other developers to ‘lead’ and ended up managing a backlog for a component that they owned. This was a minority case.
The most TOOs belonged to the 2nd cohort – people with product manager’s ambitions. Before joining Poster, most of them had worked in gastronomy and entered the digital development field as subject-matter experts given a role of team-level Product Owners (essentially: Team Output Owners).
Accountabilities of these specialists before adopting LeSS included the things like:
Managing a team’s backlog
Developing and keeping a deep expertise in a specific part of the product (e.g., fiscalization)
Assisting a team with business analysis related to the backlog items
Defining solutions for customers’ needs
Managing dependencies when product changes required work of multiple teams working sequentially
Working with the CEO and other departments to align and coordinate work
Staying up to date with the legislation and competition landscape
Communicating with the existing customers to understand their needs
From the list above, one can see that the TOOs worked mostly as middle-men, team representors, product designers, business/requirement analysts and team representatives. This was creating a negative dynamics, sucking out teams’ accountability and decreasing teams’ learning, as the TOOs fenced the teams from the problem space and gave the teams task to implement. As with any middle-men, over time a system gets more and more dependent on such role.
In LeSS and with the agile-mindset in general, we believe that “the best architectures, requirements, and designs emerge from self-organizing teams” (11th principle of the Agile Manifesto). Therefore, the teams ought to meet the customers, learn about their problems, and from these interactions generate enough empathy and understanding that will help to come up with a genuine solution – a product experiment to try and learn from. Such a different mindset implies a lot of collaborating and coordination that eventually will be seen as a necessary and inseparable part of engineering – and not a burden to be delegated to external coordinators and middle-men.
After adopting LeSS and switching to the feature teams, most of the activities of TOOs were now either owned by the teams or no longer needed. Sadly, dropping team-level Product Backlog ownership and not having a dedicated team was seen by most of the TOOs as a demotion for them at the beginning of the change.
“Jobs safety over role safety” is an important principle in Lean Production. It means that because we value people, skills and expertise, we can guarantee that whoever was in the company before the structural change can be sure of their jobs. They will stay (and their salary will remain the same). But their role will likely change, and those people will have to learn to apply their skills differently in a new context – learn to add value with what they know and can.
The same principle was applied to the role of the team-level Product Owners, losing their role because of the LeSS adoption. Most of them possessed good knowledge of the domain and the product, had great analytical and communication skills – and those skills were still highly needed. The senior management of Poster wanted these specialists to stay and find a way to keep adding value in the “new world” of self-managing teams without being the middle-men.
My suggestion as a LeSS coach was to try seeing those specialists for what they were – either developers or business analysts. So, why not let them join the feature teams? One of the TOOs with a deep engineering background did, in fact, join a feature team as a developer and did greatly. But for the others, the CEO decided on a “product manager” title. He wanted them to be in his team, helping him to formulate and lay down the product strategy for the overall Product Backlog.
I can’t say that this transition went smooth – an organizational change is the sum of personal changes, and each personal change goes at an individual pace. We had many discussions with the new Product Managers on their new role. And I can recall several attempts when they tried to follow their old habits of managing teams: asking to join team’s Daily Scrums, be added to teams’ internal chat channels, calling for an additional Scrum-of-Scrums event to be organized, etc.
Product Managers felt disconnected from the Sprints and teams (compared to what they were used to before) and attempted to regain control. But the teams who felt the fresh air of self-management did not allow that. They categorically refused to let product managers inside the internal teams’ affairs. The teams claimed that they didn’t need any help until they explicitly asked for it. That was fair. The teams defended their right to self-manage and were improving in this with every Sprint.
Not being able to get engaged deep back with the teams, the product managers continued looking for something to hold on elsewhere. Eventually, they called themselves a “product guild” and started to design and formalize the “discovery process” - a process that they would own. They were claiming “formalization of such a process would professionalize product development, help the teams to avoid misunderstandings, long wasteful discussions and rework”. I, as a coach, saw this emerging and was actively doing nothing for some weeks to come. I was waiting for the right moment to intervene systemically.
The ideas of the need to formalize the “product discovery” was based on several internet-spread articles, including the white paper on how Spotify builds its products with “Think It. Build It. Ship It. Tweak It.” model (see the References for links, and read it with caution). Based on that paradigm, the product managers came up with a staged product development process and asked for feedback. In that described process that saw themselves engaging with the users and developing early prototypes that, once got validated, become a basis for involving the teams and building ‘real’ features on.
And I offered them a two-hour workshop on Lean Thinking. Specifically, we looked into the concepts of flow and process wastes that disrupt a flow: hand-offs, batching and functional groups working in isolation. We also looked at their narrow definition of a prototype, which they saw as an outcome of the discovery phase. Luckily, they were able to agree that any change in the product in fact is an experiment and that any product change is some sort of prototype that is done to validate/invalidate a hypothesis. Early prototypes ought to be lighter and cheaper due to high uncertainty, but it doesn’t mean the feature teams can’t be the ones building them. In fact, involving the teams early on will only improve their understanding of the customer problem and technical feasibility.
To my surprise, a week after, the initiative to “describe the ideal product development with a discovery stage for prototyping” was abandoned by the product managers. And the key influences in their group agreed to work from the ground up instead. They agreed to focus on the product challenges at hand rather than abstract processes, get better at facilitating the PBR events, and most importantly - observe and collect emergent practices and improve processes continuously without making things up.
Eventually, it took about four months for the product managers to find their new place in the company and start seeing the new horizons. They slowly stopped fighting the inevitable and focused on what they were good at – sharing and applying their analytical skills, facilitation, product knowledge and domain expertise. Therefor, weekly Multi-team Product Backlog Refinement (PBR) sessions became that sweet spot where they would collaborate with the teams and pass on their experience.
During the multi-team PBR sessions, the product managers were supporting the teams to understand the underlying business rules and constraints that they were aware of as domain experts and also learned to facilitate user story mapping and other requirement splitting techniques described in the next chapter on Multi-Team PBRs.
And it took about 9 months before the teams could work with real clients on the PBR. That was a mental shift for the product managers, who were slowly turning into “cocktail party hosts” rather than dependent experts and middle-men (see Kent Beck’s podcast).
Remote Meetings and Multi-Team PBRs
Because of the company strategy to hire remotely (see COVID-19 Times and Remote Work) and the fact that the government introduced occasional lock-downs, we had to face the inevitable – learn to run remote LeSS meetings. This chapter highlights the practices of remote meetings that we used, using PBRs as an example.
But first, let me start by listing all the possible drawbacks of remote-run meetings.
With the currently available remote tooling (Zoom-like video conferencing), the non-verbal sources of inter-human interaction are shut down or very limited. Even with the cameras on (we had this as a rule in Poster), one can’t properly read people’s body language or their emotional state. That makes meeting’s interaction low-bandwidth, centralized, and thus results in all sorts of dysfunctions.
Managing participants’ focus and engagement in a physical meeting is getting generally harder with an increased number of participants, but with dozens of people present remotely – it is becoming a real challenge even for seasoned facilitators.
So, when you read online that “remote work is the work of the future”, we can only hope that the future brings better VR/AR-tooling to our remote collaboration. Better tooling would make remote collaboration look more like face-to-face collaboration that we human beings were prepared for by the million years of evolution.
At Poster in 2020, we were using Zoom, Notion, and Miro (see Tooling) as the main tool set for remote work. We’ve experimented a lot with facilitating our meetings, to minimize the drawbacks that the remote work had introduced.
Remote meetings require more energy to stay focused and productive. To run a long meeting (e.g., Multi-Team PBR) is a very challenging activity. So, we decided to run PBRs weekly, limiting them to 3 hours maximum (instead of one PBR per Sprint, as recommended in LeSS). Moreover, the amount of learning and uncertainty of the top-priority items during the first Sprints were so big, that running PBRs more frequently helped us to keep learning faster. It also allowed us to iterate on the process of PBRs.
We did follow the LeSS guides of preferring multi-team over single-team PBRs. And most of our PBR sessions were run in Zoom break-out rooms with mixed groups. During the early LeSS Sprints, we spent about 60 minutes in an Overall PBR where the team representatives and the product managers were laying out a plan on which items which each of the mixed groups were to refine. As we learned and improved, we tried to shorten and simplify that process and eventually ended up with no Overall PBRs needed. The product managers would group closely related items before the event and on the PBRs we would simply open up N break-out rooms (one per each item group) and let the teams form mixed groups just-in-time and refine the items.
It is worth noting that we used a single Miro board for all teams and all the meetings – a shared virtual space. For a specific meeting, we mapped in Miro a map with topics and participants of each break-out room, so that people could know what happened where and could switch between the rooms when required. And in addition to the mentioned tools, all teams also had teams’ internal chats to coordinate while being on a multi-team calls.
It is important to mention that as a preparation for LeSS Flip, about 20-25 volunteers were trained on the basics of facilitation, including PBR facilitation and PBI splitting practices. So, when the teams were creating mixed groups for the PBR sessions, the teams were also making sure that each group had a skillful facilitator.
By failing and learning, we’ve discovered, that nothing makes PBR sessions less productive than having people misaligned on the current product behavior and the context of upcoming changes. So, we introduced a recommendation to start every PBR session (in a break-out room) with someone to demo the actual product to all the room’s participants first. That helped, as people in newly formed feature teams had a fragmented understanding of the product - never underestimate to what extent component development creates many small silos of knowledge.
Despite the above improvements mentioned, at times teams did struggle to split complex large items. And there was one particularly painful situation, where one mixed group had to extend PBRs several times and spent a lot of time on what they called “analysis and architecture”. The reasons being a) getting too technical in the PBRs mixing item Refinement and architecture design; b) not splitting the items properly to be able to apply incremental and iterative development. That failure and the associated lessons shed the light on the importance of applying tools like Story Mapping and my favorite one, “requirements as bubbles” - a visual implementation of the “take a bite” splitting technique.
“Take a bite” splitting pattern.
A “bubble” diagram below depicts of a process of splitting a complex requirement that grew into a “bubble tree” over a series of PBR sessions, where each group of leaves and individual leaves represent smaller PBIs (“bites”) emerged while detailing and splitting. The colors on the leaves indicate a level of clarity and done-ness of the items.
A bubble-like requirements splitting
Product Support Process
The high-priority items at the top of the Product Backlog were not the only work that was there. There was another flow of work: the so-called “support tickets” – client issues that had to be addressed as soon as possible. A chapter above, "High Defect Rate" explained the source of incidents and quality issues the product had before the LeSS adoption. And that didn’t disappear magically by forming new teams.
But now, the support process could be reshaped. Before, only people who knew the affected component, could work on a given support ticket. And now, as component knowledge was spread across multiple feature teams, any team could potentially work on any product issue.
That added a lot of flexibility to the planning (see below) and also drastically reduced developer’s interruptions, as now:
Poster could dedicate an entire feature team or several (not just a knowledgeable individual) to work on support tickets.
Introduce a “support duty role” that teams rotate on a Sprint basis.
There is another possible strategy for dealing with this: each team sends a pair of developers that together form a temporary support team for a Sprint. But because our teams were new, we wanted to make them stay together and jell. And nothing jells teams better than working on critical issues that the customers are facing at the same time protecting all other team’s Sprint plans. So, we introduced a rotating support role that teams could take on.
That is what we did: each Sprint Planning, based on incident metrics, the teams, and the PO decided of how many teams needed to take on the support duty; the reaming teams would work on PBIs in a Sprint cadence. Basically, that was an agreement on how to share the overall teams’ development capacity among these two work queues.
During the very first LeSS Sprints, due to the high incident rate, 50% of the overall development capacity (3 teams out of 6) had to take on the support duty. Such a high incident rate did raise plenty of debates and experiments on bringing that number down:
Better understanding of the effects the issues have on the clients:
Why are there so many issues?
Should all of them be really addressed?
Could we define a better classification of criticality to have a better focus on what was really disrupting the clients?
Less involvement of feature teams via more skillful customer support:
There was the 1st and 2nd line of customer support at Poster, represented by another department. So, what prevented those critical issues to get solved by customer support engineers?
Could we provide better tooling, documentation and what not, so that the customer support engineers get better at addressing those issues without involving feature teams?
Smarter work on the issues by developers:
How can we identify classes of root causes, fixing which would result in a significant decrease in the issue count?
Can we constantly plan product improvements (addressing the issue root causes) as PBIs at the top of the Product Backlog?
What typically slows the developers down? What are the typical touch times on the issues? How can we improve that?
Rigorous work on all the above-mentioned questions, during the Overall Retrospectives and beyond, did yield positive results. Five Sprints later, Poster was able to get down to just 1 out of 6 teams (17% of the overall development capacity) being on the support duty.
One particular aspect (that, to my knowledge, significantly helped to start changing that initial dire situation) was improved collaboration between the customer support department and the feature teams. Namely, I observed the following shift in collaboration:
Representatives of the customer support department were now invited to the Sprint Reviews, where they could better understand the changes being made to the product, understand the impact and timeline of those changes, provide valuable feedback and request a knowledge transfer session.
Representatives of the customer support department were now made regular guests of the PBR sessions, where they collaborated with the teams to learn the most affecting issues and their impacts on the clients.
Those two things above helped to discover that the customer support engineers were lacking some essential tooling and automation to be able to self-serve and resolve a whole class of data-related client issues. After those things were identified and formed as PBIs, they were got built in no time, letting the customer support department to solve a whole class of issues without involving the feature teams.
Engineering Managers Doing Gemba Coding
Engineering Managers was a new role that was introduced with LeSS adoption, with a goal of growing the company’s engineering capabilities rather than managing team work.
Like always with a new role, there were new confusion and questions.
How do managers work with the teams without managing them?
How do managers grow the engineering capabilities, actually?
How do managers stay hands-on?
And how do managers continue being great engineers (they wanted to keep coding)?
What is the difference between this role and Scrum Master?
The realization was that the Engineering Managers need to spend the time with the teams doing real work - that would allow them to learn how the teams are doing the work and help them to get better at it.
So, the engineering managers started to join teams on Sprint-basis as team members. That helped to surface numerous potential improvements, to name a few areas:
Rethinking the testing pyramid and the approach to integration testing.
Accelerating the process of training testers in coding and helping them transition to test automation engineers.
Reconsidering release/branching strategy, that worked 'somewhat OK' when component teams worked within privately owned repos, but now because of the cross-component work a new strategy needs to be applied.
Expanding engineering skills to include basic data science work that each team needed to master to be able to measure the outcome of the features they were building.
And so on.
Managers as Scrum Masters
Right before the LeSS Flip Event, the only Scrum Master at Poster decided to leave the company. That came as a surprise. But event with him staying, we would have been understaffed, as LeSS recommends a Scrum Master to work with 1–3 teams. We had 6 teams, so having several Scrum Masters was necessary.
We tried to hire skilled people but failed. Most Scrum Masters who came for the job interviews knew how to work with a single team only and had close to zero experience with LeSS. We had to pivot.
Meanwhile, in order to better understand and address systemic impediments, I started to work closely with the group of three Engineering Managers and the VP of Engineering. They were eager to work close to the teams and I inspired them to coordination practice called Travelers to join teams on a Sprint basis. They were all experienced developers and could mentor teams on technical questions. Working closely with the teams, they started to collect a broader set of observations that were vital for improving the LeSS adoption.
Eventually, after 6 months, Poster gave up trying to hire new Scrum Masters and realized that the group of the Engineering Managers plus the VPoE had been de facto serving as Scrum Masters for the LeSS adoption.
Such a model of having managers as Scrum Masters has its pros and cons, but in Poster it was the only possibility, and we tried hard to make the benefits outweigh the disadvantages. We shall see yet which hidden and delayed effects such a model has on the organization.
Engineering Community and Pain of Long-Lived Feature Branches
The practice of long-living feature branches (LLFB) breaks the idea of continuous integration, even for a component team owning its repository. This dynamics is well described in Martin’s Fowler’s “Patterns for Managing Source Branches”. But a component team might not experience that pain as bad as several feature teams who are trying to collaborate on the same part of code in a single Sprint. Let me illustrate the LLFB practice in a component team setup and in a feature team setup.
A scenario: isolated component team making changes
Imagine a team starts to change a component by creating a LLFB. What happens if by the end of the Sprint, the changes are not done? The team can drag the changes on and keep the LLFB open, thus making it a bit more longer-living. It is very unlikely other teams would be introducing changes to the same repository (repo), as we are in component ownership modus operandi. When the changes are done, the component team can merge that LLFB back to the trunk. The probability of a merge conflict is low, as no other team contributed to the same repo. This is a dysfunction that breaks the idea of continuous integration, makes incremental development hard and slows down feature releases. But from a view-point of a given component team, those global concerns might be visible or important. It is an example of a local optimization.
A scenario: feature teams collaborating on changes
Now imagine feature teams using the same practice of LLFB. As, any team can now introduce changes to any repo. And now it is not unlikely that a LLFB starts diverging from the trunk. It will inevitably result in potential merge conflicts - a pain that any team can feel and observe the delays caused by this. The severity of the pain is a function of lifetime and number of LLFBs.
At Poster, in one of the early LeSS Sprints, one team was having hard times starting to work on a feature because another team still had a LLFB from a previous Sprint in some of the affected repos. And very soon the developers at Poster realized they had to unlearn the practice of LLFB. But what were the options?
Lack of Feature Toggling
But why do developers use LLFB in the first place? What stops developers from using the trunk-based development practice and avoiding the pain of merging?
At Poster, we realized that the key root-causes of having to use LLFB was lack of feature toggling capabilities. That is, the developers had no way to isolate unfinished features from releasable code but to branch out and keep unfinished work disintegrated. Lack of skills in splitting large Product Backlog items were also contributing to this. As large items took longer to get finished, a probability of having an unfinished feature by the end of a Sprint was always quite high.
A Pull for Change
Those and other findings became hot topics at the emerging engineering community at Poster. And the Engineering Managers now had a good material to work with. Before, they had to convince teams to improve. And now, the teams felt the pain and wanted to improve the way they worked themselves.
We can say that the LeSS adoption created a pull for change: old habits ought to be challenged and addressed, and better ways of developing software were to be uncovered. As the engineering practices started to mature, so did the Definition of Done.
Checking with Poster teams a year later: they have introduced feature toggling, and now a feature gets designed and built in a way that its release can be controlled on a per-user basis.
In-product feature metrics are also now a part of the feature development - teams are learning to build in metrics to measure user interactions and impact of the features.
Evolution of the Definition of Done
This chapter focuses on an aspect of this LeSS adoption - the evolution of the Definition of Done that I’ve witnessed over time at Poster. I do believe that the diff in the DoD over time is one of the greatest metrics to watch for in any Agile adoption. But especially in LeSS, where the expansion of the DoD over time can result in structural changes, e.g. dissolution of so-called “undone departments” (functional silos).
Feature Onboarding in DoD 1.0
A chapter above on Initial Product Backlog Refinement describes our high-level approach to agreeing on the shared DoD. This chapter is dedicated to a specific and probably one of the most important DoD improvements that happened at Poster during the first 10 LeSS Sprints.
Feature onboarding is the Poster’s way to call a chain of processes that needs to happen after features get physically released to production to make sure clients actually start using them. That typically includes things like:
Informing clients about new feature releases.
Providing learning materials and workshops to the clients and also to the internal customer care staff.
Helping clients configure and start using the new features.
Troubleshooting clients’ attempts of feature usage.
Gather early feedback and improve on it fast.
Historically, that pieces of work have never been in the hands of the development teams. It was a dedicated specialist group that covered that area – an ‘undone department’ as we like to call it in LeSS, to highlight a temporal and rather a negative connotation of such groups.
What’s wrong with a dedicated specialist group doing some work that is clearly not software development activity and, hence, is designed not to be a part of the development process? The following dynamics were at play at the pre-LeSS Poster organization:
Feature onboarding was usually done via a no/low-code solutions that send client messages and highlights product interface changes, hence was not seen as development tasks;
… as a consequence, it was not seen as teams’ duty (or part of the DoD 1.0).
The teams controlled physical releasing of features to servers
… but controlling (switching on) feature onboarding information was in the hands of the “onboarding group”
Due to a rather complex feature release process (slow roll-out with canary releases), it was typically unclear for the onboarding group who of the clients have access to the features already and who doesn’t
… and because two different groups have parts of the information, it typically took some time to clarify feature release status
… also to stay safe with displaying feature onboarding info, it was better to be later than early (as not of the clients might have physically gotten the feature yet)
… and all of that caused delays between a) the moment a specific feature was released and b) was known to be released to the users.
That, in turn, delayed the moment the customer started to use new features
… and, of course, delayed feature feedback.
To make things worse, the developers used to start working on the next feature before getting and reacting to the feedback on previously released stuff because it was not clear to them as well how and when onboarding happens.
That’s why this part of the DoD – feature onboarding was so hot when we started to discuss the perfection vision and next steps. Likely, during our initial discussions, it became clear that there was a strong consensus about this issue and its systemic fix – giving parts of the onboarding process to the teams.
Feature Onboarding in DoD 2.0
Although it was clear that the Undone department dealing with onboarding would remain after the Flip Event, the teams agreed to include some tasks of onboarding that were initially out of their DoD:
Teams to improve communication of feature availability.
Teams to develop internal documentation that can be used by the onboarding staff to craft onboarding messages.
A feature is called done, when at least one client is using it successfully.
These were some of the first steps towards the perfection vision.
On the Path of Continuous Improvements
During the early LeSS Sprints, teams embraced the new notion of the onboarding process, teams developed all the necessary documentation, and tried to reach out to the clients directly for onboarding.
That was a historical revelation. But as expected, it has immediately revealed some systemic impediments. One of them is the lack of special tools for teams to select suitable clients from the client base satisfying all the necessary hardware/software requirements. Reminder: Poster is in the B2B business, so clients are restaurants with a rather unique software/hardware ecosystem.
In the spirit of LeSS, that impediment got addressed during one of the early Overall Retrospective. As a result, a cross-functional group of volunteers got together to resolve it. For that particular issue, it was agreed that teams would initiate onboarding by selecting customer IDs (based on some requirements like hardware models and so on). And then involve specialists from the Customer Care department to help with phone calls and arranging for feature testing. Once that connection to the customers was set, then it would be teams to work directly with that customer, help with the set-up, gather feedback and iterate over the solution.
The onboarding department, which had been historically alone on that battlefield, was positively surprised and celebrated that change. Of course, they were still there and were responsible for producing educational content like the YouTube videos in multiple languages. But by expanding the DoD to writing internal and client manuals, and reaching out to the alpha-testing clients, the teams started to minimize hand-offs and speed up feedback cycles.
A short video clip below (in Russian) illustrates one of the first features Poster built, released, and onboarded just a few Sprints after the Flip Event.
The Path is the Goal
It is needless to say that we are still at the very beginning of a long and exciting journey ahead of us. We are committed to experimenting and applying critical thinking to assess our progress. And hopefully, one day we will be able to add a chapter: “50 LeSS Sprints Later…”.
And to highlight that we are not done and that the path to perfection vision is infinite, below are some observations from Poster colleagues on the recently observed challenges.
Kirill Reketskiy, Product Manager, says:
We had recently a precedent when an item that was taken to a Sprint was identified by a team as badly refined. So, it was taken off the Sprint for another PBR session to consider the new knowledge the team acquired on how to refine it better.
Furthermore, we have seen cases when at the PBRs we cannot find suitable product solutions or to find any solutions. We have introduced a new status for PBIs in the overall Product Backlog called “Prepare for PBR”. That is when a product manager needs to research a problem, works out the underlying reasons, motives of the necessary change, analyze whether there are similar cases within the product so that the UX stays congruent…
P.S. Russian Warship, Go F… Yourself!
Obviously, this wasn’t a part of the growth strategy, but on the 1st of March 2022 (a week after Russia started the war on Ukraine on the 24th of February 2022), Poster blocked the accounts of 10’000 Russian taxpaying businesses. The blocking feature didn’t just block Russian taxpayers to access the product, but was also educating them on the real matter of things – it was the war of the Russian Federation against the sovereign country of Ukraine.
It took just several days for the company to change its global strategy and exited its largest market, losing a good half of its customers. You never know when and how you will need to adapt. So be always ready can pay off anytime!
In the fall of 2023, half a year after losing 50% of its customers and within 200 days of the war, Poster returned to being a profitable business.
High adaptivity comes with strong resilience.
References
Change process:
John Kotter: https://www.kotterinc.com/8-steps-process-for-leading-change/
Ahmad Fahmy: Gemba Sprints
Craig Larman: Larman's Laws of Organizational Behavior
Alexey Krivitsky: Studying LeSS Adoption at Poster with Org Topologies Scans
Dr. W. Edwards Deming: The 14 Points and “Out of Crises”
Alexey Krivitsky Extract Team Leads
LeSS Flip:
Ahmad Fahmy: How to form teams? A story of self-designing teams
Mark Bregenzer: Teams Self Designed
Alexey Krivitsky: Org Topologies study of org archetypes at Poster
Technical:
Martin Fowler: Feature Toggles (Feature Flags)
Martin Fowler: Patterns for Managing Source Branches
Paul Hammant: Trunk-Based Development
Requirements
Alexey Krivitsky: What if I tell you requirements are like bubbles?
Alexey Krivitsky: Try Impact-Driven Product Backlog
Product:
Kent Beck: Product Thinking podcast
Henrik Kniberg: “How Spotify builds products” (use with caution!)
Comments